Proximal Policy Optimization

PPO was published for simulated robotic locomotion and Atari game playing, not for language models. Its contribution is a surrogate objective that enables multiple epochs of minibatch updates per batch of sampled data, keeping some of the benefits of trust region policy optimization while being much simpler to implementsource, accessed 2026-08-28: the update is clipped so that the new policy cannot move far from the one that produced the samples, which is what makes reusing a batch safe.

That property is why it took over language-model alignment. In this setting a sample is a full generation scored by a reward model — expensive to produce, and worth several gradient epochs each. The loop holds four networks: the policy being trained, a frozen reference the policy is penalised for drifting from, the reward model standing in for a human, and a value network estimating what a partial generation is worth. The reward model is the thing being optimised against, and it is a model, so the pressure to exploit it is why the reference penalty is there at all.

What it bought, measured. InstructGPT (2022-03-04) is the reference result. The headline is a parameter upset: outputs from the 1.3B InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameterssource, accessed 2026-08-28. At equal size the margin is larger still — 175B InstructGPT outputs are preferred to 175B GPT-3 outputs 85 ± 3% of the timesource, accessed 2026-08-28 — and two of the specifics matter more than the preference score, because they name behaviours rather than tastes: 21% against GPT-3's 41% on closed-domain tasks — making up information not present in the input about half as oftensource, accessed 2026-08-28, and about 25% fewer toxic outputs than GPT-3 when prompted to be respectfulsource, accessed 2026-08-28.

The tax, named by the people who paid it. Aligned models regressed on public NLP benchmarks, and the paper does not hide the trade — it engineers around it: mixing PPO updates with updates that increase the log likelihood of the pretraining distribution, called PPO-ptx, substantially reduces regressions on public NLP datasets without compromising preference scoressource, accessed 2026-08-28. The fix is worth looking at squarely. The reinforcement learning stage pulls the model away from its pretraining distribution, and the repair is to pull it back with a term of the original pretraining objective, mixed into the same updates.

The number nobody quotes. training the 175B PPO-ptx model took 60 petaflops/s-days, against 3,640 petaflops/s-days for pretraining GPT-3source, accessed 2026-08-28 — under two per cent of the pretraining bill, for the step that turned a completion engine into something people could give instructions to. The scarce input was never compute: it was about 40 contractors; roughly 13k prompts for supervised fine-tuning, 33k for the reward model and 31k for the PPO stagesource, accessed 2026-08-28. Capability was bought with a pretraining run. Usefulness was bought with a few tens of thousands of ranked comparisons and a team small enough to fit in one room.