Data Validation & Quality Checks

P12.data-quality-sql.04 · Audience: guest, it-ml, language-pro · Prerequisites: SQL for Analytics I — Select, Filter, Group

Columnar Thinking with polars Lineage — Where Did This Number Come From?

LLM grading (this page):Real gradingOff — no LLM calls

The last three modules assumed the data was clean. It never is. A null where a city should be, a negative quantity, a product listed twice, an order pointing at a product that does not exist — each is a small corruption that, as the joins module showed, can ride silently into a total and out the other side as a plausible wrong number. Validation is the discipline of catching those corruptions on purpose, at the door, and refusing to let a bad batch through quietly. This module builds the four rule types that catch the overwhelming majority of data defects, runs them on the deliberately-corrupted retail dataset, and makes the case for the loudest possible failure.

Step 1 / 4 — Fail loudly, or not at all

There is a temptation, when a pipeline meets a bad row, to be forgiving: skip it, coerce it, fill in a default, keep going. That instinct is exactly backwards. A row that is silently dropped or quietly patched becomes a discrepancy nobody can explain three weeks later — the revenue is off by a bit, and the reason is buried in a coercion no log recorded. The engineering virtue is the opposite: fail loudly. Detect the violation, name it, and stop — or at minimum, quarantine the batch and alert a human.

ⓘ Concept: Loud failure is cheaper than silent corruption

detect → name the violation → halt or quarantine → alert (never: skip, coerce, and continue in silence)

Why it matters — A pipeline that halts on bad data costs you an alert and an hour; a pipeline that swallows bad data costs you a wrong decision and a week of forensic archaeology to find out why. Silent handling — the try/except that logs nothing, the fillna that hides a missing join — trades a visible problem now for an invisible one later, and the later one is always more expensive because the evidence has scattered. Validation is the practice of paying the small, loud cost up front, on purpose.

🗣️ From a linguist's perspective: The smoke detector you don't disconnect

A smoke detector that chirps at 3 a.m. is annoying, and the tempting fix is to pull the battery. But the chirp is the point — it is the cheap, loud signal that something needs attention while the problem is still small. Silencing a validation check to make a pipeline 'just run' is pulling the battery: the pipeline goes quiet, and so does your only warning that the numbers have started to lie.

✅ Key takeaways

Fail loudly: detect a violation, name it, and halt or quarantine — never silently skip or coerce. Loud failure costs an alert and an hour; silent corruption costs a wrong decision and a week of forensics, because the evidence scatters.
Four rule types catch most defects — null (required value present?), range (number within bounds?), uniqueness (key actually unique?), referential integrity (does every reference point at a real row?). Each guards a specific downstream disaster from the earlier modules.
On the corrupted retail dataset the report flags exactly four: city_not_null = [c3], quantity_in_range = [o4], product_id_unique = [p1], order_product_fk = [p99]. Referential integrity is asymmetric — a child orphan is an error, a childless parent is fine.
A good report returns keys, not counts: city_not_null = [c3] is a work item; '4 errors' is a shrug. A violation is an identifier, in input order, so the report reads like a to-do list.
A quality gate composes the rules into one PASS / WARN / FAIL verdict by severity — halting on join-poisoning corruption while merely flagging the cosmetic. That is loud failure made operational, and a trusted FAIL is one that only fires on real corruption.

Next: P12.data-quality-sql.05 — Lineage — Where Did This Number Come From?

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

4 graded rungs · ~40 min

Now catch the defects yourself. Each rung is a three-panel workspace: instructions on the left, a code editor in the middle, output and test results on the right. Run checks the visible tests; Submit grades against hidden edge cases. The explorer rung profiles the corrupted batch and asks which column fails the range rule; the practitioner rungs implement the rules and handle referential integrity's asymmetry; the senior rung composes them into a configurable quality gate.

Rung 1 — Count the damage (explorer)

Loading exercise…

Rung 2 — Write the rules

Loading exercise…

Rung 3 — Referential integrity

Loading exercise…

Rung 4 — The quality gate (senior)

Loading exercise…

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Lineage — Where Did This Number Come From? →

Data Quality & SQL · P12

Later in Data Quality & SQL

Lineage — Where Did This Number Come From?

Go up a level