Low-Rank Adaptation

Freeze the pretrained weight matrix and learn a rank-decomposed update beside it: two thin matrices whose product has the shape of the weight but a tiny fraction of its parameters. The bet is that adaptation to a downstream task has low intrinsic rank, and the paper's own ablation is the evidence — a rank as small as one sufficed for adapting the query and value projections on the datasets testedsource, accessed 2026-08-28.

What that buys. Against full fine-tuning of GPT-3 175B with Adam, the paper reports 10,000x fewer trainable parameters and 3x less GPU memory than fine-tuning GPT-3 175B with Adamsource, accessed 2026-08-28, and the concrete form of it is the number that changed practice: checkpoint 350GB to 35MB, training VRAM 1.2TB to 350GB, and a 25% training speedup on GPT-3 175Bsource, accessed 2026-08-28. A per-task artifact that fits in email rather than in a datacenter is what made serving hundreds of fine-tunes of one base model an ordinary thing to do.

The zero-latency claim has a condition. LoRA adds no inference latency because the update can be folded into the frozen weight before serving — the served matrix is just a matrix again. That is also the trade: the paper's own limitations note that it is not straightforward to batch inputs to different tasks with different adapters in one forward pass, if the adapter is absorbed into the weights to eliminate the added latencysource, accessed 2026-08-28. You can have merged weights and no overhead, or unmerged adapters and mixed-task batching, and serving stacks that offer many adapters at once are paying for the second.

QLoRA moved the floor. Dettmers and colleagues (2023-05-23) backpropagated through a frozen 4-bit base model into the adapters, with three memory tricks — NormalFloat, quantizing the quantization constants, and paged optimizer states — and reported a 65B model finetuned on a single 48GB GPU while preserving 16-bit finetuning task performancesource, accessed 2026-08-28. Their demonstration model was the headline: Guanaco reached 99.3% of the performance level of ChatGPT on the Vicuna benchmark after 24 hours of finetuning on one GPUsource, accessed 2026-08-28.

Where the original claim does not hold. LoRA's abstract says it performs on-par or better than fine-tuning; that was measured on RoBERTa, DeBERTa, GPT-2 and GPT-3 adapting to tasks close to what they already knew. Biderman and colleagues (2024-05-15) asked the harder question — teaching a model a domain it does not have — and report that in the standard low-rank settings LoRA substantially underperforms full finetuning on programming and mathematics, across instruction finetuning of roughly 100K prompt-response pairs and continued pretraining of 20B unstructured tokenssource, accessed 2026-08-28. Their explanation is mechanical rather than rhetorical: full finetuning learns perturbations with a rank 10-100x greater than typical LoRA configurationssource, accessed 2026-08-28. The update genuinely is low-rank; the learning is not.

The same study found the mirror of that result. LoRA holds on to the base model's abilities outside the target domain better than full fine-tuning does, and better than weight decay or dropout, while keeping generations more diverse. So the honest summary is not "cheaper and just as good". It is that constraining the update to low rank acts as a strong regularizer: less is learned, and less is destroyed, and which of those matters depends on whether the job is to adjust a model's behaviour or to give it a new subject.