Alignment & adaptation
P10.llm-systems-frontier.04 · Audience: it-ml · Prerequisites: Frontier architectures
Turning a pretrained next-token predictor into a helpful assistant and adapting it cheaply, one idea at a time: the RLHF pipeline, DPO and RLAIF alternatives, and parameter-efficient fine-tuning — each with a worked example. Interview depth for the LLM-systems track.
ⓘ Concept: One mental model to keep
Why alignment exists. Pretraining optimizes next-token likelihood, not being helpful and safe. Alignment is a post-training stage — usually SFT then preference optimization — that steers behaviour toward human preferences.
Why it matters — Pretraining and alignment optimise different targets, which is why a fluent base model still needs a post-training stage before it behaves like an assistant.
Three stages: SFT on instruction→response demos; a reward model trained on human preference comparisons (Bradley-Terry ranking loss); then PPO to maximise the reward minus a per-token KL penalty against the SFT reference. The KL term stops the policy over-optimizing the reward model (reward hacking) into off-distribution text.
Worked example — the PPO objective, and why KL matters
objective = E[ r(x, y) ] - beta * KL( policy || sft_reference )
with beta = 0.1:
reward r = +5.0, KL drift = 8.0
score = 5.0 - 0.1 * 8.0 = 4.2
# push reward higher, but the KL term taxes drifting off the SFT modelWith beta = 0.1, a high reward (+5.0) is discounted by the KL drift (−0.8) to 4.2 — so the policy can't win by drifting far from the trusted SFT model to game the RM. Failure modes to name: reward hacking (Goodhart), sycophancy, length bias, and PPO's instability/cost (live RM + reference + value model).
⚡ Interview Ref — the quick-scan Reference face
Aligning and adapting pretrained LLMs: the RLHF pipeline, DPO and RLAIF/constitutional alternatives, and parameter-efficient fine-tuning. Why alignment exists: pretraining optimizes next-token likelihood, not being helpful and safe — a post-training stage (SFT then preference optimization) steers the model toward human-preferred behaviour.
1 — The classic RLHF pipeline (q19)
Three stages: SFT on instruction→response demos; a reward model trained on human preference comparisons (Bradley-Terry ranking loss); PPO to maximise reward minus a per-token KL penalty against the SFT reference. The KL keeps the policy close to the trusted SFT model so it can't over-optimize the RM (reward hacking). Failure modes: reward hacking / over-optimization (Goodhart), sycophancy, verbosity / length bias, RM mis-generalization, and PPO's instability + cost (live RM + reference + value model).
2 — DPO vs RLHF/PPO vs Constitutional AI / RLAIF (q33)
RLHF (PPO) = human comparisons → reward model → on-policy RL + KL; powerful but complex. DPO exploits a closed-form optimal policy to skip the RM and RL, minimising a classification-style loss directly on the same preference pairs (chosen up, rejected down, reference model implicit) — simpler, more stable, now a common default; trade-off: off-policy, less expressive at the frontier. RLAIF / Constitutional AI replaces most human labels with AI-generated preferences from a written constitution — scalable, auditable.
3 — Parameter-efficient fine-tuning: LoRA / QLoRA (q34)
Full fine-tuning stores an optimizer state per parameter and a full model copy
per task. LoRA freezes base W and learns a low-rank update ΔW = B·A
(rank r ≪ d): h = Wx + B(Ax) — often <1% of parameters, tiny checkpoints,
hot-swappable, no inference latency once merged, and forgetting-resistant (base
frozen). QLoRA trains adapters on a frozen 4-bit base to fit very large
models on one GPU. Other PEFT: adapters, prefix / prompt tuning, (IA)³.
Interview one-liners
- RLHF = SFT → reward model → PPO with a KL penalty; the KL stops reward hacking / over-optimization.
- DPO = same preference data, no RM/RL, a direct classification-style loss — simpler and stable; RLAIF/Constitutional swaps human labels for AI feedback.
- LoRA freezes the base and learns a low-rank delta (<1% params) — cheap, swappable, forgetting-resistant; QLoRA adds a 4-bit frozen base.
📚 Go Further
Curated resources on alignment and parameter-efficient fine-tuning.
| Type | Resource |
|---|---|
| Reference | Ouyang et al. (2022), Training language models to follow instructions with human feedback — InstructGPT |
| Reference | Rafailov et al. (2023), Direct Preference Optimization |
| Reference | Bai et al. (2022), Constitutional AI: Harmlessness from AI Feedback |
| Reference | Hu et al. (2021), LoRA; Dettmers et al. (2023), QLoRA |
| In-app | P10.llm-systems-frontier.05 — Production LLM systems, for shipping the aligned model |
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Where next?
Later in LLM Systems & Frontier
Go up a level