Temperature and top-p

The two knobs every inference API exposes were not designed together and do not do the same kind of thing. Temperature rescales the whole distribution: divide every logit by t before the softmax, and the ordering is untouched while the gaps between candidates widen or narrow. Top-p truncates it: keep only the smallest set of highest-probability tokens whose cumulative mass reaches p, renormalize, sample from that. One changes how much the model's own confidence counts; the other changes which tokens are eligible at all.

Why sampling exists at all is the part usually skipped, and Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi established it on 22 April 2019. The intuitive decoding rule is to take the most likely continuation — greedy, or beam search over several. It produces text that loops. Their diagnosis is a measurement rather than an aesthetic judgement: the probability the model assigns to real human continuations is "on average, much lower" than the probability of what beam search produces, and "natural language rarely remains in a high probability zone for multiple consecutive time steps, instead veering into lower-probability but more informative tokens." Human writing does not maximize the model's likelihood. A decoder that maximizes it is therefore not approximating human text more closely as it gets better at maximizing; it is walking away from it.

The repetition numbers make the failure concrete on GPT-2 Large. Greedy decoding repeats at 73.66%; beam search at width 16 at 28.94%; nucleus sampling at p=0.95 at 0.36%. The paper's own figure caption puts the contrast as "the increased variance that characterizes human text, in contrast with the endless repetition of text decoded by Beam Search."

The paired perplexity numbers are the reason to prefer truncation over temperature, and they are what an API call's settings are actually trading against. Text generated with nucleus sampling at p=0.95 has perplexity 13.13 against human text's 12.38 — close on both sides. A temperature-scaled top-k configuration (t=0.7, k=40) cuts repetition to 8.86% but lands at perplexity 3.48: text several times more predictable than anything a person writes. Lowering temperature buys coherence by making the output more probable than human writing is, and the cost is not visible as an error. It is visible as a register — flatter, more expected, more like the mode.

The practical consequence follows from the two mechanisms being different. Temperature never sets a token's probability to zero — it only makes the tail relatively lighter, so any token remains reachable at any t above zero. Top-p removes tokens from consideration outright. Turning both down at once therefore compounds rather than duplicating, which is why a value tuned for one is not a substitute for the other. And neither knob touches what the model knows. A model asked for a fact it never reliably learned will not produce it at any temperature; the sampler decides how the uncertainty is expressed, not whether it exists.