Policy Methods & RLHF
A track of P9 · Reinforcement Learning.
From value tables to policies to preference-trained reward models — the RL machinery that teaches LLMs to be helpful.
Value tables work beautifully on a gridworld and not at all on a world with more states than atoms. This track makes the jump from reading a value table to learning a policy directly — a function from situation to action — and then follows that idea all the way to how large language models are aligned.
It starts with why enumerate-and-max breaks and what replaces it: the policy as the learned object, nudged toward actions that earned more reward (policy gradients, kept at the level of a traced two-action example — the calculus lives in the neural-network pillar). Then reward models: humans are unreliable at scoring an answer out of ten but reliable at saying which of two is better, so you turn pairwise preferences into a scalar reward with a Bradley-Terry fit you build by hand. The pillar closes on RLHF — the reward model provides the signal, the policy is the language model, and a reinforcement step pushes it toward helpful, harmless, format-following behaviour. You will see exactly what that can fix (behaviour) and what it cannot (facts the base model never learned), and where it goes wrong when the policy games its own reward.