Pruning

The claim SparseGPT made on 2023-01-02 is easy to state and was not obviously true: at least 50% sparsity in one shot without any retraining at minimal loss of accuracy, and up to 60% unstructured sparsity with negligible increase in perplexity — more than 100 billion weights ignored at inference timesource, accessed 2026-08-28. No gradient step, no fine-tuning run to recover, and under 4.5 hours to prune OPT-175B or BLOOM-176B, the largest open models available at the timesource, accessed 2026-08-28.

How half a model becomes optional. Pruning is solved one layer at a time against a small calibration set: choose which weights in a column to drop, then adjust the weights that remain so the layer's output on the calibration data moves as little as possible. The compensation is where the accuracy comes from, and Wanda's contribution in June was to show how much of it you can skip — prune the weights with the smallest magnitude multiplied by the corresponding input activation, per output — no retraining and no weight update, and the pruned model is used as issource, accessed 2026-08-28 — while still performing competitively against the methods that do the intensive weight update. Its motivation is the observation of emergent large-magnitude features in these models: a weight's importance is approximated well enough by its own size times the size of what it multiplies.

The number that gets misread. The measurement on the largest model is raw-WikiText2 perplexity for OPT-175B: 8.35 dense, 8.21 at 50% unstructured, 8.45 at 4:8 and 8.74 at 2:4source, accessed 2026-08-28. Half the weights removed, and perplexity slightly better than dense. The authors do not lean on it, and the caution is theirs, not a reviewer's: at the very largest scale there is even a slight accuracy improvement over the dense baseline, which however seems to be dataset specificsource, accessed 2026-08-28.

The pattern that keeps quality is the one silicon ignores. Read the same table in the other direction and the ordering is uncomfortable. The best result comes from unstructured sparsity, where a zero may sit anywhere — and a dense matrix multiply does not run faster because some of its inputs are zero. The pattern hardware does accelerate is 2:4, two non-zeros in every group of four, supported from NVIDIA's Ampere generation onward, and 2:4 is the worst row in the table. The delivered speed is smaller than the pattern promises, too: 2:4 sparsity gives roughly 1.54x-1.79x on the matrix shapes of OPT-175B on NVIDIA Ampere GPUs against a 2x theoretical ceiling; unstructured sparsity gave 1.57x, 1.82x and 2.16x end-to-end at 40%, 50% and 60% on CPU with DeepSparsesource, accessed 2026-08-28. Note where the larger end-to-end gains in that figure come from — a CPU inference engine, not the accelerators these models are served on.

So there are two honest sentences about pruning a large language model, and they are usually merged into one dishonest one. Half the weights can be removed with almost no measurable quality cost. Removing them makes the model smaller, and makes it meaningfully faster only in the arrangement that costs the most quality, on hardware that supports that arrangement.