LLMOps

The same prompt cost $0.002 yesterday and$0.04 today. Nobody touched your code. So who changed what? Perhaps the router started falling back to the expensive tier when the cheap model's error rate ticked up; perhaps a colleague grew the prompt template by 800 tokens; perhaps retries doubled because a guardrail began tripping on a new class of input. In classical serving, cost per request is a constant you measure once. In an LLM system it is a variable with many owners — and finding yesterday's twenty-fold jump is a routine Tuesday, not an incident.

This track is deliberately a synthesis: it pulls together pieces taught elsewhere — inference and KV-cache economics from P10, latency budgets and batching from P11, LLM-as-judge evaluation from P5, and the versioned eval sets and tracked runs of P13's own lifecycle track — and composes them around one operational question: how do you serve many requests, cheaply, at a quality you can defend? The central mechanism is routing across a model fleet: send each request to the cheapest model that meets the quality bar, and be able to say, with evidence, where that bar sits.

The evidence is the interesting part. Offline eval feeds the routing policy — eval-in-the-loop, not eval-as-a-report — so a model only earns traffic by scoring for it, and loses traffic the same way. Around that sit the incident playbooks generative systems need: fallback tiers when a provider degrades, guardrail trips, retry budgets that do not silently double your bill. Every routing policy is a cost/quality trade; the discipline this track teaches is making that trade explicit, measured, and reversible — which is exactly what the $0.04 morning demands.