Models & Evaluation

Neural Scaling Laws

Neural scaling laws are the empirical finding that a language model's test loss falls as a smooth power law in three quantities: the number of parameters, the size of the training dataset, and the compute spent. They let researchers predict the performance of large models from cheap small-scale runs.

What neural scaling laws describe

A neural scaling law is an empirical relationship showing that the test loss of a neural network, typically the cross-entropy loss of an autoregressive language model, decreases predictably as a power law when one of three resources grows: the number of model parameters (N), the number of training tokens (D), or the total training compute (C). A power law means that plotting loss against the resource on log-log axes produces a straight line over many orders of magnitude, so doubling a resource yields a fixed fractional reduction in loss rather than a fixed absolute one.

The practical value of such laws is prediction. Because the trend holds across a wide range, a team can fit the law on a family of small, cheap models and extrapolate to estimate how a much larger and far more expensive model will perform before committing the budget to train it. This turned large-model development from guesswork into a forecastable engineering exercise and shaped how labs allocate parameters, data, and GPU hours.

Scaling laws are descriptive, not mechanistic. They summarize what is observed without explaining why the exponents take the values they do, and they describe the smooth decline of the loss rather than the behavior of any specific downstream task. A skill measured by accuracy on a benchmark can stay flat while loss falls steadily, then change sharply, so loss curves and task curves must be read separately.

Loss falls as a power law in parameters N, dataset size D, and compute C.
On log-log axes the relationship appears as a near-straight line over many orders of magnitude.
The laws are empirical and predictive, fit on small runs and extrapolated to large ones.
They describe smooth loss, not the per-task accuracy that practitioners ultimately care about.

Kaplan et al. 2020: the original power laws

The foundational study is "Scaling Laws for Neural Language Models" by Jared Kaplan, Sam McCandlish, and colleagues at OpenAI, posted to arXiv in January 2020. Training many Transformer language models across a wide range of sizes and dataset sizes, they found that loss scales as a power law with model size, dataset size, and compute, with trends spanning more than seven orders of magnitude. When the other factors are not bottlenecks, each resource on its own predicts the loss.

For model size, the paper reports the form L(N) is approximately (N_c / N) raised to the power alpha_N, with the exponent alpha_N about 0.076 and the scale constant N_c about 8.8 times ten to the thirteenth. The small exponent encodes a hard truth: a tenfold increase in parameters lowers loss only modestly, because ten to the power of negative 0.076 is roughly 0.84, so ten times the parameters cuts loss by about sixteen percent. Analogous power laws hold for dataset size and for compute.

Kaplan and colleagues also concluded that, for a fixed compute budget, the most efficient strategy was to train very large models and stop well before convergence, spending relatively little of the budget on data. This recommendation pointed toward large, comparatively data-light models and directly influenced the design of systems such as GPT-3. That conclusion was later revised.

L(N) ≈ (N_c / N)^α_N,    α_N ≈ 0.076,  N_c ≈ 8.8×10¹³

Kaplan et al. 2020 power law for loss as a function of model size N, when data and compute are not limiting.

Paper: "Scaling Laws for Neural Language Models", Kaplan et al., OpenAI, January 2020.
Reported L(N) follows a power law with exponent alpha_N about 0.076.
A tenfold parameter increase reduces loss by only about sixteen percent.
Original advice favored very large, undertrained models, later corrected by Chinchilla.

The Chinchilla correction

In 2022, DeepMind revisited the question in "Training Compute-Optimal Large Language Models" by Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, and colleagues. Sweeping over four hundred models trained with varied parameter counts and token budgets, they found that for compute-optimal training the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. This contradicted the earlier emphasis on parameters over data.

The headline result was a controlled comparison. Chinchilla, a 70 billion parameter model trained on 1.4 trillion tokens, used roughly the same compute as the 280 billion parameter Gopher but four times the data, and it uniformly and significantly outperformed Gopher across downstream tasks, reaching 67.5 percent average accuracy on the MMLU benchmark. The conclusion was that earlier flagship models, Gopher and GPT-3 among them, had been substantially undertrained: too many parameters fed too little data.

From the compute-optimal frontier comes the often-quoted rule of thumb of roughly 20 training tokens per parameter. A 70 billion parameter model implies about 1.4 trillion tokens, which is exactly the Chinchilla recipe. The ratio is a smooth optimum derived from the fitted laws rather than a sharp threshold, and the precise value shifts with the dataset and the fitting procedure, but the qualitative lesson, balance data against parameters, reshaped the field.

L(N, D) = E + A / N^α + B / D^β,    E≈1.69,  A≈406.4,  α≈0.34,  B≈410.7,  β≈0.28

Chinchilla parametric loss. E is the irreducible loss (the entropy of natural language); the two power-law terms vanish as N and D grow.

Paper: "Training Compute-Optimal Large Language Models", Hoffmann et al., DeepMind, 2022.
Compute-optimal scaling grows N and D roughly equally, about 20 tokens per parameter.
Chinchilla 70B on 1.4T tokens beat Gopher 280B at matched compute, 67.5 percent MMLU.
Implication: pre-Chinchilla flagship models were undertrained relative to their size.

Reading and fitting a power law

A power law L is approximately a constant times x to a negative exponent becomes a straight line under logarithms: log L equals log of the constant minus the exponent times log x. This is why scaling-law plots use log-log axes, and why the slope of that line is the exponent. Fitting a law is therefore, in its simplest form, a linear regression in log space, which is enough to recover the exponent from a handful of training runs.

The Chinchilla parametric form L(N, D) = E + A/N^alpha + B/D^beta adds the irreducible loss term E, which represents the entropy of natural language, the floor that no amount of scaling can pass. With E present the fit is no longer a single straight line, so practitioners estimate E, A, B, alpha, and beta jointly by minimizing a fitting error over observed runs rather than by plain linear regression. The published constants are E about 1.69, A about 406.4, alpha about 0.34, B about 410.7, and beta about 0.28.

The simple log-log fit below recovers an exponent from synthetic single-variable data. It illustrates the mechanic that makes scaling laws usable in practice: a cheap regression on small models yields a slope that extrapolates to predict large ones. Real fits add the irreducible term and pool many runs, but the underlying straight-line-in-log-space intuition is the same.

python

import numpy as np

# Synthetic data: loss = (Nc / N) ** alpha, observed at several model sizes N
N = np.array([1e6, 1e7, 1e8, 1e9, 1e10])
alpha_true, Nc = 0.076, 8.8e13
loss = (Nc / N) ** alpha_true

# Fit a straight line in log-log space: log L = log(c) - alpha * log(N)
slope, intercept = np.polyfit(np.log(N), np.log(loss), 1)
alpha_hat = -slope
print(f"recovered exponent alpha = {alpha_hat:.3f}")  # ~0.076

Recovering a Kaplan-style exponent with a numpy log-log linear fit. Real Chinchilla fits add an irreducible term E and fit N and D jointly.

Taking logs turns a power law into a straight line whose slope is the exponent.
A single-variable exponent can be recovered by ordinary least squares in log space.
The Chinchilla form adds an irreducible-loss floor E, requiring a joint nonlinear fit.
Cheap small-model fits extrapolate to forecast expensive large-model loss.

Practical consequences and inference-aware scaling

Scaling laws turned model planning into a budgeting problem. Given a fixed compute budget, the laws say how to split it between making the model bigger and feeding it more data, and the Chinchilla frontier gives the compute-optimal split. Before launching an expensive run, a lab can fit the law on a ladder of small models and forecast the final loss, choosing the configuration expected to land on the frontier.

Compute-optimal in the Chinchilla sense, however, means optimal for training compute alone. It ignores the cost of serving the model afterward. A model that will answer billions of queries is cheaper overall if it is smaller, because every inference call is cheaper, even if reaching a target quality then demands far more training tokens than the 20-per-parameter rule suggests. This logic drives deliberate over-training: small models such as the Llama family are trained on token counts well past their Chinchilla-optimal point to lower per-query inference cost.

The shift reflects a broader move from training-centric to total-cost-of-ownership thinking. When inference dominates the lifetime bill, the right model is often smaller and trained longer than a pure training-compute analysis would pick. Scaling laws still set the frontier; what changed is which point on it is chosen once serving cost enters the objective.

Chinchilla optimality minimizes training compute, not the cost of serving the model.
Heavy inference demand favors smaller models trained well past 20 tokens per parameter.
Deliberate over-training, as with the Llama models, lowers per-query inference cost.
The objective shifts from training compute to total cost of ownership including serving.

Limits, caveats, and open questions

Scaling laws are empirical regularities, not physical constants, and they can break. The fitted exponents depend on the architecture, the dataset, the tokenizer, and the optimization recipe, so a law calibrated on one setup may not transfer to another. A 2024 study reconciling the Kaplan and Chinchilla results traced their apparent disagreement largely to Kaplan et al. counting only non-embedding parameters and fitting at smaller model scale, a reminder that the precise constants are sensitive to methodology.

A second caveat is that loss is not capability. Downstream metrics, exact-match accuracy or pass rates on coded tasks, do not always track the smooth fall of cross-entropy loss; some abilities appear to change abruptly with scale, and the choice of metric can make that change look sharp or gradual. Predicting a benchmark score from a loss extrapolation is therefore far less reliable than predicting the loss itself.

Finally, the data side of the laws faces a physical ceiling. The Chinchilla recipe consumes tokens roughly in proportion to parameters, and the supply of high-quality human-written text is finite, raising the prospect of data exhaustion for the largest runs. Responses under active study include training on synthetic data, repeating data across multiple passes, and laws that scale test-time compute rather than only training compute. None of these removes the underlying constraint that the empirical laws describe a regime that may not hold without limit.

Exponents depend on architecture, data, tokenizer, and optimizer; laws do not always transfer.
Kaplan and Chinchilla differences trace largely to Kaplan counting non-embedding parameters and fitting at smaller scale.
Loss extrapolations predict cross-entropy well but downstream task accuracy poorly.
Finite high-quality text raises data-exhaustion concerns at the largest scales.

Key takeaways

Neural scaling laws state that test loss falls as a power law in model size N, dataset size D, and compute C, predictable over many orders of magnitude.
Kaplan et al. 2020 reported L(N) approximately (N_c/N)^alpha_N with alpha_N about 0.076, so a tenfold parameter increase cuts loss only about sixteen percent.
Chinchilla (Hoffmann et al. 2022) showed earlier models were undertrained and that N and D should scale equally, roughly 20 tokens per parameter.
Chinchilla 70B trained on 1.4 trillion tokens beat the 280B Gopher at matched compute, reaching 67.5 percent on MMLU.
Compute-optimal targets training cost only; heavy inference demand favors smaller, deliberately over-trained models for cheaper serving.
The laws are empirical and can break: exponents are setup-dependent, downstream accuracy differs from loss, and high-quality data is finite.

Frequently asked questions

They are empirical rules showing that a model's test loss drops in a smooth, predictable way as you increase its parameters, its training data, or the compute spent on training. On log-log axes the trend looks like a straight line, so the same proportional gain repeats with each doubling of a resource. Because the trend is stable over many orders of magnitude, researchers fit it on small models and extrapolate to forecast large ones.

Kaplan et al. 2020 emphasized making models very large and stopping before convergence, spending relatively little of the budget on data. Chinchilla (Hoffmann et al. 2022) found this undertrained models and that, for a fixed compute budget, parameters and tokens should grow together at roughly 20 tokens per parameter. A 2024 follow-up showed much of the apparent conflict came from Kaplan counting only non-embedding parameters and fitting at smaller model scale.

It is the Chinchilla rule of thumb that a compute-optimal model should see about 20 training tokens for every parameter it has, so a 70 billion parameter model implies roughly 1.4 trillion tokens. The figure comes from the compute-optimal frontier fitted across more than four hundred models, not from a sharp threshold. The exact value shifts with the dataset and the fitting method, but the lesson is to balance data against model size.

Chinchilla used the same training compute as the 280 billion parameter Gopher but spent it on a smaller 70 billion parameter model trained on four times the data, 1.4 trillion tokens. Because Gopher had been undertrained for its size, the better-balanced Chinchilla outperformed it across downstream tasks, reaching 67.5 percent average accuracy on MMLU. The result demonstrated that more data, not just more parameters, was being left on the table.

They predict cross-entropy loss reliably but downstream task accuracy much less so. Some capabilities stay flat as loss falls and then change sharply, and the choice of metric can make that change look abrupt or gradual. Extrapolating a benchmark score from a loss curve is therefore far riskier than extrapolating the loss itself, which is why loss and task curves should be evaluated separately.

They are empirical, so they hold only within the regime where they were measured and can break outside it. The Chinchilla recipe consumes tokens in proportion to parameters, and the supply of high-quality human text is finite, raising data-exhaustion concerns at the largest scales. Active responses include synthetic data, repeating data across passes, and scaling test-time compute, though none removes the underlying limit.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free