Scaling laws

A scaling law is a fitted curve plus a decision about what is being held fixed. The curves have been good. The decisions have been revised twice, and the second revision was forced by an arithmetic error in the paper everyone was quoting.

2020: spend it on parameters. Kaplan et al., published 23 January 2020, found that cross-entropy loss "scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude," and drew the operational conclusion that "optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence." GPT-3 is that sentence executed: 175 billion parameters, 300 billion tokens. The GPT-3 paper says so in the caption of its own scaling figure — "Based on the analysis in Scaling Laws For Neural Language Models we train much larger models on many fewer tokens than is typical."

2022: no, spend it equally. Hoffmann et al., 29 March 2022, trained "over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens" and concluded that "for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled." The demonstration was Chinchilla — 70 billion parameters, four times Gopher's data, the same compute budget — beating Gopher at 280 billion, GPT-3 at 175 billion, Jurassic-1 at 178 billion and Megatron-Turing NLG at 530 billion — four models each larger than the one that beat them.

2024: the disagreement was engineering, and one of the fits was broken. Two papers took the pair apart in the same year.

Besiroglu, Erdil, Barnett and You (15 April 2024) tried to replicate Hoffmann's third estimation method and could not. The published confidence intervals were too narrow to be possible: reproducing them "would need to have access to the results from nearly 240×2116=600,000 training runs," where the authors "likely had between 400 and 500 data points." After the replication appeared, a lead author of the Chinchilla paper supplied the cause — the minimiser was averaging Huber loss values over examples instead of summing them, which raised the loss scale and terminated L-BFGS-B early during both the fit and the bootstrap, leaving the bootstrapped parameters near their initialisation. The headline rule survived; the third method's error bars did not.

Porian, Wortsman, Jitsev, Schmidt and Carmon (27 June 2024) went after the other end and reproduced Kaplan's law on two datasets, then removed the disagreement with three corrections: counting the decoding layer's compute (Hoffmann did, Kaplan did not), shortening a fixed-length warm-up that was too long for small models and inflated their optimal token counts, and tuning learning rate, batch size and the AdamW β₂ parameter per model size. With those fixed, Kaplan's setup agrees with Chinchilla. The two laws were never measuring different worlds; they were measuring the same world with different training defaults.

And the objective moved. LLaMA, 27 February 2023, trained deliberately past the compute-optimal point — the smallest model on a trillion tokens, the largest on 1.4 trillion — with the reasoning stated plainly: "Although it may be cheaper to train a large model to reach a certain level of performance, a smaller one trained longer will ultimately be cheaper at inference." Chinchilla optimality answers "what is the best model I can train for this compute." A model that will be served answers a different question, and the optimum moves.

The useful reading is that "the scaling law" is three separable things: an empirical power law, which has held; a fit, which was wrong in a checkable way and got corrected; and a choice of objective, which is a business decision and has changed at least once. Arguments that cite the first to settle the third are skipping two steps.