Production LLM systems
P10.llm-systems-frontier.05 · Audience: it-ml · Prerequisites: Alignment & adaptation
Shipping an LLM feature, one idea at a time: evaluating it (offline vs online), mitigating hallucination, managing latency and cost, and defending against prompt injection — each with a worked example. Interview depth for the LLM-systems track.
ⓘ Concept: One mental model to keep
The model is the easy part; the system around it — evals, grounding, latency/cost, and untrusted input — is where products succeed or fail. Each step makes one of those concrete.
Why it matters — Interview answers that focus only on the model miss where real LLM products break; the differentiator is the discipline around it.
Benchmarks (MMLU) rarely predict product success — evaluate on your task, your data. Offline eval (a curated set scored automatically or by an LLM-as-judge) is cheap, fast, reproducible but a proxy; online eval (A/B on real traffic) measures real impact but is slow and risky. The discipline: gate offline, confirm on a small online A/B, mine production failures back into the offline set.
Worked example — the gate → confirm workflow
1. offline: 200 cases (representative + adversarial), LLM-as-judge rubric
-> regression test on every change; must not drop vs baseline
2. online: ship to 1% traffic, A/B vs control
-> task success, thumbs-up, latency, cost, guardrail refusals
3. mine production failures -> add to the 200-case offline setThe 200 cases gate every change offline; only survivors go to a 1% online A/B before full rollout, and real failures feed back into the eval set. Watch for eval leakage (test data in training) and Goodharting a single metric.
⚡ Interview Ref — the quick-scan Reference face
Shipping an LLM feature: evaluation (offline vs online), mitigating hallucination (RAG vs fine-tuning), managing latency and cost, and defending against prompt injection. The model is the easy part; the system around it is where products succeed or fail.
1 — Evaluating an LLM for a product (q30)
Benchmarks rarely predict product success — evaluate on your task, your data. Offline eval (curated set, automatic or LLM-as-judge) is cheap, fast, reproducible but a proxy; human eval catches what automation misses; online eval (A/B on real traffic) measures real impact but is slow, noisy, risky. Discipline: gate offline, confirm on a small online A/B, mine production failures back into the offline set. Watch eval leakage and Goodharting a single metric.
2 — Mitigating hallucination: RAG vs fine-tuning (q31)
Hallucination = fluent, confident, unsupported output, inherent to a model that predicts likely text. RAG fixes stale / long-tail / private facts and enables citations (update the index, not the weights); fine-tuning fixes format, style, and skills, not volatile facts. Rule of thumb: RAG for knowledge, fine-tuning for behaviour. Extra levers: say "I don't know", constrain to retrieved context, and measure faithfulness / groundedness, not just fluency.
3 — Managing latency & cost (q32)
TTFT (time to first token — prefill + queueing) and TPOT (inter-token — memory-bound decode); total = TTFT + TPOT × output_tokens. Output tokens dominate latency and cost (sequential, often priced higher). Levers: smaller / routed model, shorten outputs (biggest lever), prompt (KV) + response caching, serving efficiency (continuous batching, speculative decoding, GQA), streaming, trimmed RAG context. Reach for caching + shorter outputs first.
4 — Prompt injection & defending agents / RAG (q36)
Prompt injection = untrusted text carries instructions that hijack the model; root cause is no hard boundary between instructions and data. Indirect injection (instructions inside content the model ingests) is the scary one for agents. Defenses (layered, none complete): least privilege + human-in-the-loop for consequential actions, separate trust levels, input/ output filtering + allow-listing, sandboxing + egress control, dual-LLM / quarantine patterns. Treat ingested content as untrusted (like SQL injection) and contain blast radius.
Interview one-liners
- Gate changes offline, confirm on a small online A/B, and mine production failures back into the eval set; watch eval leakage + Goodharting.
- RAG for knowledge, fine-tuning for behaviour; measure faithfulness, not just fluency.
- Output tokens dominate latency & cost (total = TTFT + TPOT × out) — shorten outputs + cache before exotic tricks.
- Prompt injection: no instruction/data boundary — defend with least privilege, human-in-the-loop, and sandboxing, not prompt pleading.
📚 Go Further
Curated resources on production LLM systems.
| Type | Resource |
|---|---|
| Guide | Chip Huyen — Building LLM applications for production & Evaluation |
| Reference | Lewis et al. (2020), Retrieval-Augmented Generation |
| Guide | OWASP — Top 10 for LLM Applications (LLM01: Prompt Injection) |
| Guide | Simon Willison — Prompt injection series & the dual-LLM pattern |
| In-app | P10.llm-systems-frontier.01 — Inference & serving, for the latency levers (KV cache, batching) |
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Where next?
Go up a level