tool · tool/vllm

vLLM

activestable

Also called PagedAttention

Facts

license: Apache-2.0source, accessed 2026-08-28
origin: originally developed in the Sky Computing Lab at UC Berkeleysource, accessed 2026-08-28
launch throughput claim: at release, 14x-24x the throughput of HuggingFace Transformers and 2.2x-3.5x that of Text Generation Inference, on LLaMA 7B/13B with ShareGPT request distributionssource, accessed 2026-08-28
speculative decoding methods: a range of speculation methods, model-based ones such as EAGLE, MTP, draft models, PARD and MLP alongside simpler ones such as n-gram and suffix decodingsource, accessed 2026-08-29

2023-09-12
PagedAttention published at SOSP 2023source
2023-06-20
released with PagedAttention, reporting KV cache waste cut from 60-80% to under 4%source