Where Bias Enters

P15.fairness-privacy.01 · Audience: guest, it-ml, language-pro

Writing About Numbers Fairness Metrics — and Why They Conflict

A large company built a model to screen job applicants. It read a résumé and scored how likely the person was to be a strong hire, and it was meant to save recruiters time by surfacing the best candidates first. It did save time. It also learned to mark women down. Résumés that mentioned a women's chess club or a women's college scored lower; the very word women's became, to the model, a faint signal of a weaker candidate. Nobody wrote that rule. No engineer typed "prefer men". The model was simply shown a decade of past hiring and asked to predict more of the same — and the past it was shown had hired mostly men. When the harm surfaced, the tool was quietly scrapped.

That is the failure this whole pillar exists to prevent — not a crash, not a wrong number, but a system that worked, technically, while treating one group of people worse than another. Everything normative about a model — is it fair, is it private, is it documented, is it accountable at all — is the subject of Responsible AI, and it is a separate question from whether the model is accurate. A model can be accurate on average and still be unjust to the people it sorts. This module opens the pillar with the first and most important question: where does bias actually get in? The answer is not "the algorithm" — it is a chain of ordinary decisions, and we will walk that chain end to end on the hiring story above.

🗺️ So far you have built: the ML craft to build a model and read whether it works — training a classifier and judging it with accuracy, precision and recall (Machine Learning foundations and Evaluation, P5), in whatever order you met them. Those tools answer 'is the model accurate?'. This pillar adds a question they cannot answer on their own: 'is the model fair, private and accountable?' — and it needs no prior fairness background, only the willingness to follow one story carefully.

In this module you will: where bias enters a model — not as a single villainous line of code, but as a chain: the data it learns from, the labels we teach it to copy, the objective we ask it to optimise, and the way it is finally deployed. Each link is an ordinary engineering decision that can quietly encode a past injustice, and we trace all four on one worked example: the résumé screener that learned to mark women down.

Step 1 / 6 — Bias is not a bug — it is inherited

The instinct, when we hear "the model was biased", is to look for the line of code that did it — the sabotage, the mistake, the switch someone flipped the wrong way. There is almost never such a line. A machine-learning model does not invent its preferences; it inherits them, from the examples we show it and the goal we set it. The screener was never told to prefer men. It was told: here are ten years of people we hired, learn to spot more like them. It did exactly that — and because the people we hired were mostly men, "more like them" quietly meant "more men".

ⓘ Concept: A model is a mirror of its inputs, not an oracle

model learns: 'reproduce these examples, optimise this goal' → it faithfully reproduces whatever bias the examples and the goal already contained

Why it matters — This reframing is the whole foundation of the pillar. If bias were a bug, you would fix it by finding the faulty line and correcting it. But inherited bias has no faulty line — it is distributed across the data, the labels, the objective and the deployment, each of them individually reasonable. That is why fairness cannot be bolted on at the end with a single patch: you have to inspect every place where a human decision entered the model, because any of them can carry a past injustice forward as if it were a fact about the world.

So the useful question is not "did someone make the model biased?" but "where along the way could bias have entered?" There are four such places, and the rest of this module visits each one. None of them requires any mathematics — each is a decision a person made, in plain sight, that a fairer process would have made differently.

✅ Key takeaways

The failure Responsible AI exists to prevent is not a crash but a model that works on average while treating one group worse. A résumé screener trained on a decade of male-skewed hiring learned to mark women down — no engineer wrote that rule; the model inherited it.
Bias is inherited, not coded. A model reproduces whatever bias its examples and objective already contain, so fairness cannot be fixed with one patch — you must inspect every place a human decision entered the model.
Data: models learn the biased past, and deleting the protected column does not help — proxies (a postcode, a name, a club) smuggle the trait back in, removing only your ability to measure the bias.
Labels: a model can only be as fair as the target it is taught to predict; a biased answer key ('strong hire' judged in a biased culture) makes a perfectly accurate model a perfectly biased one.
Objective: overall accuracy is a size-weighted average that rewards serving the majority, so it actively incentivises per-group harm — which is why evaluation must be per group, never in aggregate.
Deployment: a model's biased decisions become next year's training data, so bias compounds in a feedback loop. This is why Responsible AI is a continuous practice, not a pre-launch checkbox.

Next: P15.fairness-privacy.02 — Fairness Metrics — and Why They Conflict

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Fairness Metrics — and Why They Conflict →

Fairness & Privacy · P15

Later in Fairness & Privacy

This module unlocks

Go up a level