Retrieval-Augmented Generation

The paper that named this is not describing the thing the name now denotes, and the difference is the training.

What was published. Lewis and colleagues (2020-05-22) combined a parametric memory — a pretrained seq2seq model — with a non-parametric one: a dense vector index of 21M 100-word chunks from a December 2018 Wikipedia dumpsource, accessed 2026-08-28. The paper compares two formulations: RAG-Sequence conditions the whole generated sequence on the same retrieved passages, and RAG-Token can use a different passage for each token. Crucially, a general-purpose fine-tuning recipe: the query encoder and the seq2seq generator are trained jointly, and the document encoder is notsource, accessed 2026-08-28. The generator learns to use retrieval; retrieval learns which passages that generator can use.

On open-domain question answering the result was 44.5 exact match on Natural Questions for RAG-Sequence and 44.1 for RAG-Token, against 41.5 for DPR, 40.4 for REALM and 36.6 for T5-11B with salient span maskingsource, accessed 2026-08-28 — a generative model beating the extractive retrieve-and-read architectures that had owned the benchmark, while also being editable: swap the index, change what the model knows, no gradient step.

The shortcut the authors found, and everyone kept. The obvious design would learn the document encoder too, so that the index adapts to the task along with everything else. The authors did not: updating the document encoder requires periodically re-indexing every document; the authors report they do not find this step necessary for strong performancesource, accessed 2026-08-28. That sentence is why the technique propagated. The expensive half was measured to be optional, which left a design any team could build from an off-the-shelf embedding model and a vector store.

What the word means now. Follow the current mainstream framing — load documents, split them, embed the chunks into a vector store, retrieve per question, and put the retrieved text in the prompt — inference-time access to data, with nothing trainedsource, accessed 2026-08-28 — and note what has gone: the query encoder is no longer trained against the generator, and the generator is not trained at all. Deployed RAG kept the paper's diagram and dropped its recipe. It is a prompt-construction strategy that inherited a name from a fine-tuning result.

The assumption that inherits badly. A trained generator learned to attend to what retrieval handed it. An untrained one only receives it, and position turns out to matter: accuracy is highest when the relevant passage sits at the beginning or the end of the input and degrades significantly when it must be read from the middle, even for explicitly long-context modelssource, accessed 2026-08-28. That measurement bears directly on the two easiest knobs in a retrieval pipeline. Increasing the number of retrieved passages pushes middling-rank passages into the middle of the prompt, where they are read worst; a reranker earns its cost mainly by moving the right passage to an end. Neither behaviour is visible in a retrieval metric — recall counts the passage as retrieved either way.