Emergent abilities

The claim that made emergence interesting was never "big models are better." It was that some abilities arrive discontinuously and at an unforeseeable size. Wei and fifteen co-authors set the definition on 15 June 2022: "We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models." Both halves of the appeal are in that sentence — sharpness and unpredictability — and both are properties of a plotted curve.

Rylan Schaeffer, Brando Miranda and Sanmi Koyejo pointed out on 28 April 2023 what else that curve depends on. Fix the model outputs; change only the scoring function. Exact String Match awards nothing for a fifty-character answer with one character wrong, so a model steadily improving its per-character accuracy scores zero, zero, zero, then suddenly non-zero. Token edit distance on the same outputs rises smoothly the whole way. Their claim is narrow and testable: "nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance."

The supporting evidence is what makes the paper hard to wave away. A meta-analysis of BIG-Bench found emergence under 4 of 39 hand-annotated metrics, and that two metrics — Multiple Choice Grade and Exact String Match — "account for >92% of claimed emergent abilities." Then, having predicted the effect, they manufactured it: never-before-reported emergent abilities produced on demand in convolutional, fully connected and self-attentional networks, including a LeNet on MNIST digits, shallow autoencoders reconstructing CIFAR100, and autoregressive transformers classifying Omniglot characters. No language model, no new scale frontier, no new training run. Only a metric swapped for a discontinuous one.

What survives is worth stating precisely, because the paper is often cited as having shown emergence is fake. It did not. It showed that the sharpness and the unpredictability — the two properties that made the concept load-bearing — are properties of the researcher's choice of metric for a fixed set of model outputs. Capabilities still improve with scale, and the underlying improvement is smooth and, on continuous measures, extrapolable. What evaporates is the discontinuity.

The practical consequence is a question to ask of any emergence claim: on what metric, and does the same data plotted with a continuous one still have a kink in it? The same question separates grokking's apparent phase change from the gradual circuit formation underneath it, and the two literatures are the same finding about different objects — a jump in a plot is not evidence of a jump in a network.