Backtesting & Anomaly Detection

P7.forecasting.05 · Audience: guest, it-ml, language-pro · Prerequisites: Baselines You Must Beat

Forecasting as Supervised Learning Why Randomize?

LLM grading (this page):Real gradingOff — no LLM calls

Five forecasters now sit on the bench: three baselines, the smoothing family, ARIMA, and the lag-feature regressors. This closing module builds the machinery that picks a winner honestly — rolling-origin backtesting — and names every way the picking goes wrong: the leakage catalogue. Then it turns the machinery around: a forecast that knows what should happen next is also a detector for what should not — the residual-based anomaly flag.

Step 1 / 6 — One honest split is not enough

You already know the split must respect time: train strictly before test. But a single time split grades every forecaster on one exam sat on one date. Maybe the test window landed on a calm fortnight, or a freak one — with one number per model you cannot tell skill from luck of the calendar.

The fix is to hold the exam many times. Pick an early cut-off — the forecast origin — train on everything before it, forecast the next few steps, and score the miss. Then roll the origin forward, retrain on the now-longer history, forecast, score again. Each roll is called a fold; the model's skill is the average score across folds, and the spread across folds tells you how much that skill wobbles.

ⓘ Concept: Rolling-origin backtesting

Why it matters — Backtesting simulates deployment: at every fold the model stands where it will stand in production — history behind, unknown future ahead — and is graded only on that future. It is cross-validation rebuilt for ordered data: many test sets for statistical confidence, with the train-before-test rule enforced in every single fold. The expanding-window form grows the training set at each roll; a sliding window keeps its length fixed to favour recent behaviour.

Note what stays constant: the metric. Each fold reports MASE against the naive denominator, so fold scores are comparable across folds, across models, and across series — module 01's scale-free honesty doing its job inside the tournament.

✅ Key takeaways

One time split is one exam on one date. Rolling-origin backtesting rolls the forecast origin forward — train on the past, forecast, score, roll — simulating deployment at every fold.
Judge models by the average and the spread of per-fold MASE: a model that collapses on recent folds is worse than its mean score suggests.
Leakage wears many disguises: shuffled splits, peeking windows, preprocessing fitted on everything, hindsight covariates, tuning against the test folds. The universal test — would I have had this number at the forecast origin?
Residuals are what the model's picture of normal cannot explain. Standardise them and flag |z| > 3: an offline anomaly detector whose quality is exactly the forecaster's quality.
Offline flags on a finished series are this pillar's job; the same loop on live traffic is P13's monitoring problem. And when the question turns from what will happen to what caused it, forecasting hands over to P8.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

4 graded rungs · ~43 min

Run the honest tournament yourself: spot the lying evaluation, build the rolling-origin split generator, backtest the baselines with it, then flag injected anomalies with residual z-scores.

Rung 1 — Spot the leak (explorer)

Loading exercise…

Rung 2 — Rolling-origin splits

Loading exercise…

Rung 3 — Backtest the baselines

Loading exercise…

Rung 4 — Flag the anomalies (senior)

Loading exercise…

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Why Randomize? →

Experimentation · P8

Go up a level