Rotary Position Embedding

Position has to enter attention somehow, and the dot product between a query and a key is indifferent to where either came from. RoPE's answer: absolute position is encoded with a rotation matrix applied to the query and key vectors, which puts explicit relative position dependency into the self-attention formulationsource, accessed 2026-08-28. Rotate the query at position m and the key at position n by angles proportional to their indices, and the angle between them depends on m − n alone. Nothing is added to the residual stream, no parameters are learned, and the operation is a pairwise rotation of adjacent channels — cheap enough to sit inside the attention kernel.

That is the design almost every open-weight decoder now ships, and it displaced learned absolute embeddings for a specific reason: a learned table has a last row.

What the paper claimed, in its own words. RoPE offers the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping linear self-attention with relative position encodingsource, accessed 2026-08-28. The first of those is the one that entered general belief, and it is the one that did not survive contact with longer inputs.

What was measured. Chen and colleagues (2023-06-27) state the failure directly: extrapolating beyond the trained length may lead to catastrophically high attention scores that completely ruin the self-attention mechanismsource, accessed 2026-08-28. Their theoretical version of the same point is a ratio rather than an adjective — the upper bound of interpolation is at least around 600x smaller than that of extrapolationsource, accessed 2026-08-28. Two months later YaRN opens by asserting the general case: these models fail to generalize past the sequence length they were trained onsource, accessed 2026-08-28. The formula is defined at every position; the model is not trained at every position, and a rotation angle it has never seen produces a score distribution it has never seen.

The fix inverted the claim. Instead of running positions off the end of the trained range, Position Interpolation linearly down-scales the position indices so a longer input lands inside the range the model already knows — the sequence is squeezed, not extended. It is cheap: LLaMA models from 7B to 65B stretched to 32768 positions within 1000 fine-tuning steps, with quality on short inputs relatively well preservedsource, accessed 2026-08-28. YaRN then treated the rotation frequencies separately rather than scaling them uniformly, reporting 10x fewer tokens and 2.5x fewer training steps than previous context-extension methodssource, accessed 2026-08-28 and, unlike plain interpolation, an ability to run past the length its own fine-tuning data contained.

The distinction worth carrying away is between a definition and a behaviour. "Any sequence length" was true of the equations from the first day and false of the models for years afterwards, and every context-extension paper since has been an attempt to close that gap by paying training compute for it. What such a model does with the positions it gains is a separate measurement again.