P9 · Reinforcement Learning
bandits → MDPs → policy methods; bridges to RLHF
The data is not given here — your own actions generate it, and feedback arrives as reward, not labels. From bandits and gridworlds to the reward models that teach large language models to be helpful.
Every model in this discipline so far was handed its data: a table of labelled rows, a corpus of text, a cohort to analyse. Reinforcement learning removes that gift. Here the agent must act to see anything at all — and what it sees is not a label saying "the right answer was X", but a reward saying "that was worth 3". The data is a consequence of the agent's own choices, which creates the problem that defines the field: to learn what is good you must try things, but trying a bad thing costs you, so when do you stop exploring and start exploiting what you already know?
The first track builds the machinery from the ground a learner can check by hand. Multi-armed bandits pose the explore-exploit dilemma in its purest form — a row of slot machines, no state, just the question of which arm to pull. Markov decision processes add state and delayed reward, so a good move now may only pay off later. Value iteration solves a small gridworld exactly with the Bellman equation; Q-learning learns the same answer from experience alone, and you watch a value map converge one backup at a time.
The second track climbs to where modern AI actually uses this. Policies — the learned map from situation to action — replace value tables when the world is too big to enumerate. Reward models turn human pairwise preferences into a scalar signal (the Bradley-Terry fit you will build by hand). And the pillar closes on RLHF: how a large language model is nudged toward being helpful by exactly this machinery — a reward model over human preferences, a policy that is the LLM, and the reinforcement step that ties them together. It is the bridge from this pillar straight into the systems that made transformers useful.