When It Goes Wrong — Incident Thinking

P15.governance-safety.04 · Audience: guest, it-ml, language-pro · Prerequisites: LLM Safety — Failure Modes & Misuse

Evaluating for Safety What Machine Learning Is (and Isn't)

LLM grading (this page):Real gradingOff — no LLM calls

Everything so far has been prevention: map the risk, know the failure modes, evaluate before shipping. This module is about the day prevention was not enough — when a system that passed its checks is already in production, already affecting real people, and someone realises it has been quietly doing harm. That day comes to every serious team, and the difference between a responsible organisation and a negligent one is not that the first never has incidents; it is that the first knows exactly what to do when it does. We walk one incident end to end — the biased screening model from the start of this pillar, now live and rejecting people — and turn the response into a playbook you can reuse.

🗺️ So far you have built: the whole preventive arc of Responsible AI — where bias enters (fairness-privacy.01), the governance map and its risk tiers (governance-safety.01), the LLM failure modes to guard against (02), and how to evaluate for safety without mistaking a pass for a proof (03). This is the module for when all of that was not enough.

In this module you will: incident thinking: what a responsible team does once a deployed system is already causing harm. You will follow one case — the biased CV-screening model from fairness-privacy.01, now in production — through a five-step responsible-AI playbook (detect, contain, communicate, remediate, prevent), and see where P15's principles hand off to P13's operational runbook. This module closes the Responsible-AI pillar, and it is where you practise: two graded rungs on the incident case.

Step 1 / 5 — Incidents are inevitable; unpreparedness is a choice

No amount of pre-launch care makes incidents impossible. Coverage gaps, shifting inputs, spec-gamed metrics, plain human error — the previous modules named all the reasons a passing system can still fail a real person. Incident thinking accepts that and prepares for it: it is the practised, ordered response you reach for when the failure is no longer hypothetical but live.

ⓘ Concept: Incident thinking: an ordered response you rehearse before you need it

incident thinking = a rehearsed sequence (detect → contain → communicate → remediate → prevent) that replaces panic with process

Why it matters — Under the pressure of a live harm — angry users, a journalist calling, a regulator asking — teams improvise badly: they hide, they delete evidence, they fix the wrong thing first, they tell no one until it leaks. A rehearsed playbook replaces improvisation with sequence: contain the harm before you diagnose it, tell the affected people before they find out, fix the root cause not just the symptom. The mark of a mature organisation is not the absence of incidents but the presence of this reflex — which is exactly why governance frameworks require incident-response processes for high-risk systems, and why 'we didn't have a plan' is itself a finding.

✅ Key takeaways

Incidents are inevitable; unpreparedness is a choice. Incident thinking is a rehearsed, ordered response — detect, contain, communicate, remediate, prevent — that replaces panic with process. Governance requires it for high-risk systems, and 'we had no plan' is itself a finding.
The case names the failure precisely: a high-risk screening model, trained on skewed history, has for three months produced a discriminatory outcome against a protected group at scale. Precise naming — not 'the model is a bit off' — drives the whole response.
Detect confirms and scopes the harm; contain stops it now — human fallback or off-switch — before diagnosis, because every hour the system runs adds victims. This is the human oversight from module 01 finally used as the genuine off-switch it was meant to be.
Communicate is an obligation, not a PR problem: tell the affected people, the owners, and often the regulator, honestly and early — a cover-up is a worse failure than the original bias. Remediate its partner: re-review the wrongly-rejected applicants, repair the harm, not just the model.
Prevent maps each root cause to a specific measure and a named owner — production per-group monitoring, a real oversight point, a tier-matched evaluation bar. Designing what to detect is P15; building the monitoring and on-call runbook that detects it is P13's operational job.

Next: Responsible AI (P15) complete — step back to the pillar and on to MLOps (P13)

You have reached the end of the governance-safety track and the close of the Responsible AI pillar (P15). From where bias enters, through fairness metrics, privacy, model documentation, the governance map, LLM failure modes and safety evaluation, to this incident playbook — you now hold the normative core of building AI that is fair, private, documented and governable. Step back to the P15 pillar overview to see the fairness-privacy and governance-safety tracks together, or follow the handoff into MLOps (P13), where the guardrails this pillar specifies become the monitoring and on-call runbook that keeps them true in production. With P15 closed, the Data Science & AI discipline is very nearly complete — the surrounding practice that a model needs to be trustworthy at work.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

2 graded rungs · ~20 min

Now work the incident yourself. These two rungs are mentor-graded free-form exercises on the screening-model case: the left panel gives you the scenario, you write your analysis, and the mentor grades it against a rubric (visible-tests-only with a "mentor unavailable" notice when you are offline). Rung 1 has you triage the live incident; Rung 2 writes the full post-mortem. (The third rung of this ladder — writing the model card the system should have had — lives on the model-cards module in the fairness-privacy track.)

Rung 1 — Triage the incident (explorer)

Loading exercise…

Rung 2 — The post-mortem (senior)

Loading exercise…

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

What Machine Learning Is (and Isn't) →

ML Literacy for Leaders · P16

Go up a level