Frontier architectures
P10.llm-systems-frontier.03 · Audience: it-ml · Prerequisites: Training at scale
Two frontier topics, paced out: Mixture-of-Experts (adding parameters without adding per-token compute — the idea, then the load-balancing problem) and the mechanistic view of in-context learning (induction heads). Each step has a worked example. Interview depth for the LLM-systems track.
ⓘ Concept: One mental model to keep
MoE decouples total parameters from compute per token; induction heads are a concrete circuit behind in-context learning. Each step turns the idea into numbers or a trace.
Why it matters — Both ideas break an assumption you might carry from earlier modules: that capacity costs compute, and that in-context learning must be some kind of hidden training.
Replace a dense FFN layer with many parallel expert FFNs plus a router that sends each token to only top-k experts (usually k=1 or 2). This decouples total parameters (all experts) from compute per token (only the active experts) — a sparsely activated model. Routing is per-token, per-MoE-layer, so different tokens take different paths.
Worked example — Mixtral 8×7B's sparse activation
8 experts per MoE layer, router picks top-2 per token
total parameters ≈ 47B (all 8 experts resident)
active per token ≈ 13B (only 2 of 8 experts run)
-> ~4x the params of a 13B dense model, but ~13B of compute per tokenMixtral holds ~47B parameters but activates only ~13B per token — you pay 13B-model compute for far more capacity. The router is a small gating network that softmax-scores the experts and combines the top-2 outputs weighted by the gate.
⚡ Interview Ref — the quick-scan Reference face
Two frontier topics: Mixture-of-Experts (adding parameters without adding per-token compute) and the mechanistic view of in-context learning (induction heads).
1 — Mixture-of-Experts (q23)
Replace a dense FFN with many parallel expert FFNs plus a router that sends each token to only top-k experts (k=1 or 2). This decouples total parameters from compute per token — a sparsely activated model (Mixtral 8×7B: ~47B params, ~13B active per token). Routing is per token, per MoE layer. Load balancing is the central difficulty: without it the router collapses onto favourite experts — fixed by an auxiliary loss, a capacity factor (overflow tokens dropped via the residual), and noisy routing. Trade-offs: high memory (all experts resident), all-to-all communication, training instability. MoE buys parameters cheaply in FLOPs but pays in memory, communication, and load-balancing complexity.
2 — Mechanistic in-context learning: induction heads (q29)
ICL picks up a pattern from the prompt without weight updates. A leading
mechanistic answer is the induction head, implementing [A][B] … [A] → [B]:
a previous-token head writes info about the prior token, then an induction
head does match-then-copy — its query matches the previous occurrence of
the current token, and it copies what came next (a QK match composed with an OV
copy). These heads form abruptly at a training phase change where ICL jumps,
and generalise beyond literal repeats. ICL is, at least in part, concrete
circuits, not gradient descent at inference.
Interview one-liners
- MoE decouples total params from per-token compute; the hard part is load balancing (aux loss + capacity factor) and the cost is memory + all-to-all communication.
- Induction heads are a two-head 'match-then-copy' circuit implementing
[A][B]…[A]→[B]; they appear at a training phase change and help explain in-context learning.
📚 Go Further
Curated resources on MoE and mechanistic interpretability.
| Type | Resource |
|---|---|
| Reference | Shazeer et al. (2017), Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer |
| Reference | Fedus et al. (2021), Switch Transformers; Jiang et al. (2024), Mixtral of Experts |
| Reference | Olsson et al. (2022), In-context Learning and Induction Heads |
| Reference | Elhage et al. (2021), A Mathematical Framework for Transformer Circuits |
| In-app | P10.llm-systems-frontier.04 — Alignment & adaptation, for post-training these architectures |
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Where next?
Later in LLM Systems & Frontier
Go up a level