From Values to Policies

P9.policy-methods-rlhf.01 · Audience: guest, it-ml, language-pro · Prerequisites: Value Iteration & Q-Learning

Value Iteration & Q-Learning Reward Models & Preferences

Q-learning left you with a habit: build a table of values, then at every state look up each action and take the best one. That habit is quietly built on an assumption — that you can list every action and read off a number for each. When the world is a 4×3 gridworld, that is effortless. When the world is a language model choosing among fifty thousand possible next tokens, the table never fits and the lookup never finishes. This module is about the move that rescues you: stop learning what everything is worth and start learning what to do directly. That directly-learned object is the policy, and it is the thing every modern method — including the one that trains LLMs — actually optimises.

Step 1 / 5 — The value-table habit

Everything you built in the foundations track ends the same way. You learn a number for each state-action pair, and then you behave by enumerating the actions at your current state and taking the one with the highest value. Look up, compare, pick the max. The behaviour is never learned directly — it is squeezed out of the value table as a last step.

ⓘ Concept: Enumerate-and-max: behaviour as a by-product of values

action(s) = argmax over a of Q(s, a) — scan every action, keep the best

Why it matters — This is the defining move of value-based RL. The agent never stores 'what to do' — it stores 'what each option is worth' and re-derives the action every time by scanning all of them and keeping the best. That is cheap and exact when the actions are few and countable, which is exactly why gridworld makes it look free. The cost is hidden in a single word: enumerate.

Hold on to that word enumerate. It is doing an enormous amount of silent work, and the next step is about what happens when it can no longer be done.

✅ Key takeaways

Value-based RL behaves by enumerate-and-max: scan every action at the current state and take the highest-valued one. That is exact and cheap only when the actions are few and countable.
Enumerate-and-max breaks when states or actions explode — continuous controls, or a language model choosing among ~50,000 tokens. The argmax has nothing finite to scan.
A policy π is the directly-learned object: a function from state to action (or to a distribution over actions) that answers 'what should I do?' without enumerating alternatives — exactly the kind of function a network can be.
A stochastic policy π(a|s) outputs probabilities and samples from them, which keeps exploration alive and — being smooth — lets you improve it by shifting probabilities a little at a time. A language model is this: a probability over the vocabulary at each step.
Policy gradient, at the level of intuition, is one sentence: push probability toward actions that earned more reward, away from those that earned less. The exact step size is a gradient — the Deep Learning pillar's machinery, cross-linked and not re-derived here.

Next: P9.policy-methods-rlhf.02 — Reward Models & Preferences

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Reward Models & Preferences →

Policy Methods & RLHF · P9

Later in Policy Methods & RLHF

Go up a level