KV cache

Attention at decode step n needs the key and value vectors of every earlier position. Recomputing them each step is quadratic in the sequence; storing them is linear, so every serving stack stores them. That store is the KV cache.

The property that governs everything downstream is that it is per request. Model weights are loaded once and shared by every request on the box; each concurrent request holds its own cache, sized two tensors per layer per position — layers × key-value heads × head dimension × bytes per element — and it grows with every token that request has seen or produced. This, not the parameter count, usually decides how many requests a GPU can serve at once.

Three lines of attack, all of them ordinary practice in serving stacks today.

Make it smaller in the architecture. Multi-query attention gives all query heads a single shared key-value head, shrinking the cache by the head count at some quality cost. Grouped-query attention (Ainslie et al., 2023-05-22) interpolates — more than one key-value head, fewer than all — and its practical contribution is a recipe for converting an existing multi-head checkpoint using 5% of the compute that pre-trained it. Nobody had to retrain from scratch to get the smaller cache, which is why the technique spread within months.

Manage it like memory. Before 2023, serving systems handed each request a contiguous slab sized for the longest output it might produce. The vLLM authors measured what that cost: 60-80% of KV memory lost to fragmentation and over-reservation. PagedAttention (SOSP 2023) applies operating-system paging — fixed-size blocks, a block table per sequence, non-contiguous physical storage — and brings waste under 4%. Note where the reported 2-4x throughput improvement over FasterTransformer and Orca came from: an allocator, not a faster kernel. Reclaimed memory becomes batch size, and batch size is throughput.

Reuse it between requests. Two requests that share a prefix can share its cache blocks — prefix caching in vLLM, RadixAttention in SGLang.

The part users see, and misread. Provider prompt caching is this same mechanism sold through the API, and it is not "the provider remembers your prompt". On Anthropic's API a cache entry is written only where you place a cache_control breakpoint, and it writes exactly one entry: a hash of the prefix ending at that block. A later request hits only if that prefix hash already exists; if the breakpoint itself misses, the system walks backward — at most 20 blockssource, accessed 2026-08-28, counting the breakpoint as the first — looking for an entry some earlier request wrote, then stops. The breakpoint budget is 4 per requestsource, accessed 2026-08-28. The default entry lives 5 minutessource, accessed 2026-08-28, timed from the start of the request that wrote or read it — so a reply that streams for four minutes leaves about one minute for the follow-up to arrive. The classic zero-hit-rate bug falls straight out of this: put the breakpoint on the block that changes every request, and every write is a new hash that nothing will ever match. OpenAI's version supports both modes: below a minimum cacheable prompt length nothing is cached at all, and above it "you can choose where to place cache breakpoints explicitly, or let OpenAI choose their locations implicitly." Retention is at least 30 minutes after the latest write or reusesource, accessed 2026-08-28 on current models, and caches are not shared across organizations.

Cache lifetimes are measured in minutes rather than days because a cached prefix occupies the same GPU memory every other request is competing for. Prompt caching is not storage. It is a reservation.