Reward Models & Preferences
P9.policy-methods-rlhf.02 · Audience: guest, it-ml, language-pro · Prerequisites: From Values to Policies
The previous module ended on a policy that improves by chasing reward. But that quietly assumed the one thing an LLM does not have: a reward. Nobody can write down a formula for "how helpful was this answer". You cannot score usefulness the way a game scores points. So where does the reward come from? The honest answer is: from people — but not by asking them to grade. Humans are unreliable graders and reliable judges. This module is about turning many small human judgements — "this answer is better than that one" — into a single scalar reward the policy can climb. That machine is the reward model, and its engine is a century-old idea called Bradley-Terry.
In a gridworld the reward is obvious: +1 at the goal, −1 in the pit. In language there is no such number. "How helpful, honest, and harmless was this reply?" has no formula — it lives in human judgement. The only source of the signal is people.
ⓘ Concept: No formula means ask a human — but ask the right question
Why it matters — You cannot hand-author a reward function for open-ended language: the qualities you care about are exactly the ones that resist being written as an equation. So the reward must be elicited from humans. But how you ask changes everything — asking a person to put a number on quality and asking them which of two answers is better are very different requests, and only one of them gives you data you can trust.
The rest of this module is about asking humans the reliable question, and then doing arithmetic on their answers.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
🎓 Practice ladder
3 graded rungs · ~27 minBuild the reward-model core with your own hands: feel why ranks survive when ratings drift, fit Bradley-Terry scores from a preference table by hand, then push those scores through to the behaviour they induce.
Rung 1 — Rank, don't rate (explorer)
Loading exercise…
Rung 2 — Bradley-Terry by hand
Loading exercise…
Rung 3 — Preferences to reward to behaviour
Loading exercise…
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Policy Methods & RLHF