Inference & serving
P10.llm-systems-frontier.01 · Audience: it-ml
How decoder-only LLMs generate text efficiently, one idea at a time: the KV cache, multi-query / grouped-query attention, speculative decoding, and continuous batching with paged attention — each with a small worked calculation. Interview depth for the LLM-systems track.
ⓘ Concept: One mental model to keep
Generation is two regimes: a one-off prefill (the whole prompt processed in parallel), then decode (one token at a time, each step depending on all previous tokens). The decode phase is memory-bandwidth bound, and almost every trick below exists to make it cheaper.
Why it matters — Once you see decode as memory-bandwidth bound, every serving trick below becomes one goal in disguise: move fewer bytes per generated token, or keep the GPU busy despite them.
During decoding, attention at step needs the keys and values of every previous position. They don't change once computed, so you cache them and compute Q/K/V only for the new token. That turns O(n²) recomputation into cheap per-step work — but the cache grows linearly and must be read every step, making decode memory-bandwidth bound.
Why decode-only: prefill already has all tokens and runs in parallel, so the cache is built during prefill and read + appended to during decode. KV-cache size — not FLOPs — is usually what caps how many concurrent long sequences a GPU can serve.
Worked example — size a KV cache
Cache size is roughly
For a 32-layer model, 32 KV heads, 2048 tokens, head_dim 128, fp16:
2 * 32 * 32 * 2048 * 128 * 2 bytes
= 1,073,741,824 bytes
≈ 1 GB per sequence (just the KV cache!)That's ≈ 1 GB per sequence before the weights — so on a 40 GB GPU the KV cache, not FLOPs, caps how many concurrent long sequences you can serve. Everything else on this page shrinks that cache or keeps the GPU busy despite it.
⚡ Interview Ref — the quick-scan Reference face
How decoder-only LLMs are served efficiently: the KV cache, multi-query / grouped-query attention, speculative decoding, and continuous batching with paged attention. Autoregressive generation is two regimes — a one-off prefill (whole prompt in parallel), then decode (one token at a time); almost every trick makes the memory-bound decode phase cheaper.
1 — The KV cache (q11)
During decoding, attention needs the K and V of every previous position;
they don't change, so you cache them and compute Q/K/V only for the new
token. Cache shape ≈ 2 × n_layers × n_kv_heads × seq_len × head_dim per
sequence. It turns O(n²) recomputation into cheap per-step work, but must be
read every step, making decode memory-bandwidth bound — KV-cache size,
not FLOPs, caps concurrency.
2 — Multi-query & grouped-query attention (q25)
The cache is dominated by the number of KV heads. MHA keeps one K/V per query head; MQA shares a single K/V across all heads (~1/n_heads of MHA, slight quality drop); GQA shares one K/V per group (tunable middle ground, near-MHA quality). GQA is the modern default (LLaMA-2/3, Mistral). Smaller cache = fewer bytes per decode step = higher throughput and longer contexts.
3 — Speculative decoding (q26)
A small draft model proposes k tokens; the big target model verifies them in one parallel forward pass, accepting the longest agreeing prefix and resampling on the first disagreement. A rejection-sampling rule makes it lossless — same output distribution, fewer sequential steps. Speedup rides on the draft's acceptance rate (2–3× typical); Medusa and n-gram drafting avoid a separate draft model. It changes latency, not the distribution.
4 — Continuous batching & paged attention / vLLM (q27)
Static batching waits for the slowest sequence, idling finished slots. Continuous batching admits a queued request the moment a slot frees, keeping the GPU saturated. Paged attention splits the KV cache into fixed blocks mapped by a block table, allocated on demand (near-zero waste) and shareable copy-on-write across a common prefix — why vLLM far outpaces naive batching.
Interview one-liners
- KV cache trades memory/bandwidth for compute; it makes decode memory-bound and caps concurrency.
- GQA ≈ MHA quality at a fraction of the KV cache (MQA is the 1-KV-head extreme).
- Speculative decoding is lossless — draft proposes, target verifies in parallel.
- Continuous batching + paged attention (vLLM) keep the GPU saturated and eliminate KV fragmentation.
📚 Go Further
Curated resources on LLM inference and serving.
| Type | Resource |
|---|---|
| Reference | Shazeer (2019), Fast Transformer Decoding: One Write-Head is All You Need — MQA |
| Reference | Ainslie et al. (2023), GQA: Training Generalized Multi-Query Transformer Models |
| Reference | Leviathan et al. (2023), Fast Inference from Transformers via Speculative Decoding |
| Reference | Kwon et al. (2023), Efficient Memory Management for LLM Serving with PagedAttention — vLLM |
| In-app | P10.llm-systems-frontier.02 — Training at scale, for the quantization that shrinks these caches further |
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
🎓 Practice ladder
3 graded rungs · ~28 minServing is arithmetic before it is engineering — size the cache, do the quantization maths, then reason about batching throughput. The workspace is three panels — instructions, a code editor, and live output with test results. Run checks the visible tests; Submit grades against hidden edge cases; the reference solution unlocks once you pass.
Rung 1 — Size the KV cache
Loading exercise…
Rung 2 — Quantization arithmetic
Loading exercise…
Senior rung — Continuous-batching throughput
Loading exercise…
Where next?
Later in LLM Systems & Frontier
Go up a level