Operations

The demo answered in 80 milliseconds. With 200 concurrent users it takes four seconds, and profiling the model finds nothing — inference is exactly as fast as it ever was. The time went somewhere the profiler does not look: into the queue. Once request arrivals start colliding, waiting time is governed by utilisation, and as utilisation creeps toward 1 the wait explodes — a law of arithmetic, not a bug. This track exists so that the four-second surprise never surprises you again.

The serving half puts numbers on all of it. You set a latency budget and split it across stages; you learn why the p99 — not the mean — is the number an SLO lives or dies by; you size capacity with Little's law (L = lambda times W) and tune batching to the largest size whose tail latency still fits the budget. Because real HTTP serving needs hardware the sandbox does not have, you build the behaviour instead: a single-server FIFO queue simulator, written by you, that shows the tail emerging from colliding arrivals — which is precisely where it comes from in production.

The monitoring half faces the slower failure. A model that passed every offline gate still decays in production, not because the code changed but because the world did. Monitoring makes that decay measurable — PSI and the two-sample KS test on live inputs against the training distribution — and then does the part dashboards usually dodge: a decision policy that converts a drift score into no-shift, monitor, or retrain. Operating a model is these two motions on repeat: hold the latency budget, watch the distributions, decide.