Reinforcement Learning from Human Feedback

RLHF is a pipeline with three stages, and the name belongs to all three together rather than to any one of them: preference comparisons, then a reward model fitted to them, then policy optimisation against that model constrained to stay near the starting policysource, accessed 2026-09-11. Each stage exists because the previous one ran out: demonstrations show one good answer but cannot say which of two acceptable answers is better, so comparisons are collected instead; raters are too slow to consult during training, so a reward model stands in for them; and showing correct outputs is impossible where nobody can write the right answer down, so the policy is scored rather than shown.

None of this began with language models. In June 2017 the same pipeline ran on simulated robots and Atari — Christiano, Leike and colleagues define goals as (non-expert) human preferences between pairs of trajectory segments and solve Atari games and simulated robot locomotion with no access to the reward function, training novel behaviors on about an hour of human timesource, accessed 2026-09-11. What made it the default for assistants came five years later, when InstructGPT walked the full staircase on GPT-3: supervised fine-tuning on demonstrations first, then the preference-to-reward-to-policy loop — and outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters, with improvements in truthfulness and reductions in toxic outputs at minimal regressions on public NLP datasetssource, accessed 2026-09-11.

The distinction this entry exists to hold. PPO is the algorithm most often filling the pipeline's third slot, not a synonym for the pipeline. Proximal Policy Optimization is one way to optimise a policy against a reward model; the proof that the slot is interchangeable is that it has been replaced without touching the other two stages: Direct Preference Optimization solves the standard RLHF problem with a classification loss — no reward model trained, no sampling from the model during fine-tuning — exceeding PPO-based RLHF at sentiment control and matching or improving it on summarization and single-turn dialoguesource, accessed 2026-09-11. See Direct Preference Optimization for the substitution worked through. A reader who meets "RLHF" and lands on one optimiser has been sent to a part; the pipeline is the thing the word means.