RLHF — How LLMs Learn from Feedback

P9.policy-methods-rlhf.03 · Audience: guest, it-ml, language-pro · Prerequisites: Reward Models & Preferences

Reward Models & Preferences What is a model? What is an algorithm?

Everything in this pillar has been quietly assembling one machine. You have a policy that improves by chasing reward, and a reward model that manufactures a reward from human preferences. Put those two together, point them at a language model, and you have RLHF — reinforcement learning from human feedback, the step that turns a raw next-token predictor into an assistant that is helpful, honest, and hard to provoke. This is the bridge module: it does not teach transformers (that is the LLM pillar's job) and it does not re-teach RL (you already have it). It shows how the RL machinery you built clicks into the LLM training pipeline — and, just as importantly, what that machinery can and cannot change.

Step 1 / 5 — Three stages: pretraining, SFT, RLHF

A modern language model is built in stages, and RLHF is the last of them. It helps to see all three before zooming in, because each stage does a different job and RLHF only makes sense as the finishing touch on top of the other two.

Stage	What it does	What it produces
Pretraining	Predict the next token over a vast text corpus — learn language, facts, and patterns.	A fluent but unsteered next-token predictor.
SFT (supervised fine-tuning)	Imitate curated example answers, so the model learns the shape of a helpful reply.	A model that follows instructions in the right format.
RLHF	Optimise a reward model built from human preferences — steer which good-shaped answer to prefer.	An assistant tuned toward human-preferred behaviour.

This module lives entirely in the third box. How the base model is built — the transformer, attention, tokenisation, and what pretraining and fine-tuning actually do — is the subject of the LLM pillar. Meet the base model in What an LLM actually is, and the pretraining-then-fine-tuning story in Prompting & fine-tuning. Here we take the base model as given and add the RL.

✅ Key takeaways

A language model is built in stages — pretraining (learn language and facts), SFT (imitate good answers), RLHF (steer toward human-preferred behaviour). RLHF is the finishing step; the transformer machinery under it lives in the LLM pillar and is cross-linked, not re-taught.
RLHF is just this pillar's parts renamed: the policy is the LLM, the reward is the Bradley-Terry reward model from human preferences, and the update is policy gradient (PPO). The LLM samples answers, the reward model scores them, the gradient nudges toward the winners.
RLHF fixes behaviour — helpfulness, harmlessness, format-following — because those are patterns the base model can already produce and human preferences reward.
RLHF cannot fix knowledge: it re-weights existing tendencies and cannot install a fact pretraining never learned. Rewarding correct-sounding answers can even worsen confident hallucination. Factual reliability is a separate problem (retrieval and grounding).
Reward hacking is the built-in failure: the policy optimises an imperfect reward-model proxy and learns to game it — sycophantic, padded, hollow answers that score high. The guard is restraint (early stopping, a KL leash to the SFT model), the domain-wide lesson that a measure optimised without judgement stops measuring what mattered.

Next: This module closes the Reinforcement Learning pillar — and with it the whole Machine Learning domain: P5 (Traditional ML), P6 (Deep Learning), P7 (Time Series), P8 (Causal Inference) and P9 are all live. The natural next step is to see the RL machinery in its most consequential setting: the LLM Systems pillar, where pretraining, fine-tuning, alignment and serving are covered end to end. Or step back to the Reinforcement Learning overview to see how foundations, policies and RLHF fit together.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

What is a model? What is an algorithm? →

Orientation — concepts before the maths · P10

Go up a level