Lineage — Where Did This Number Come From?

P12.data-quality-sql.05 · Audience: guest, it-ml, language-pro · Prerequisites: Data Validation & Quality Checks

Data Validation & Quality Checks Evaluation infrastructure & experiment tracking

Someone points at a dashboard and asks the question every data engineer dreads and should welcome: "Where did this number come from?" If the honest answer is a shrug, the number is not trustworthy no matter how carefully it was computed — because nobody can check it, reproduce it, or say who is allowed to change it. Lineage is the answer to that question: the traceable path from a figure on a screen back through every transformation to the source rows that produced it. This module — the pillar's closer — traces one number, Paris revenue = 40, all the way to its origins, then draws the line where the data engineer's responsibility ends and the model's begins.

Step 1 / 4 — The question a number must be able to answer

A number on a dashboard is the end of a process, and by the time you read it, all the steps that made it are invisible. Lineage is the practice of keeping those steps recoverable — so that any figure can answer three questions: what source data went into it, what transformations shaped it, and who owns each step along the way. A number that can answer them is defensible; one that cannot is just a rumour with good typography.

ⓘ Concept: Lineage: the traceable path from figure to source

lineage(figure) = { source rows → transformations applied → owner of each step }

Why it matters — Provenance is what separates a trustworthy number from a plausible one. The same value — Paris revenue = 40 — is worth completely different amounts of confidence depending on whether you can trace it to the exact orders that produced it or merely believe it looks about right. Lineage makes a number reproducible (someone else can rerun the path and get the same figure), auditable (a reviewer can inspect each step), and debuggable (when it is wrong, you can walk the path to the corruption). Without it, every downstream decision rests on faith.

Notice how directly this depends on everything the pillar built. A number is only as traceable as its least-documented step — a silent coercion, an un-validated join, a query nobody saved all sever the chain. Lineage is not a separate tool bolted on at the end; it is what you get for free when every earlier step was done in the open.

✅ Key takeaways

Lineage is the traceable path from a figure back to its source — answering what data, what transformations, and who owns each step. A number that can answer those is defensible; one that cannot is a rumour with good typography.
Paris revenue = 40 traces cleanly: the GROUP BY summed 20 + 20, each priced by an orders→products join, back to source orders o1 (2 × 10) and o2 (1 × 20), both belonging to customer c1 = Paris. The same trace is the map you follow when a number goes wrong.
Trust needs a governed path, not just a traceable one: a data contract states the schema, guarantees, owner and change process of a table — and the validation rules from the previous module are that contract written in code (product_id is unique is both a check and a promise).
Follow lineage forward and the number becomes a feature-table row: ownership passes from this pillar (data trustworthy upstream) to MLOps (model trustworthy in production). Knowing where your responsibility ends prevents both the validation gap and the overreach.
That is the whole craft of data engineering: not just moving data, but making every number traceable, validated and owned — so the trust survives the handoff. Downstream of the feature table it becomes P13's problem; what crosses the line is a number the model team can trace.

Next: P13 — MLOps: downstream of the feature table, the model's lifecycle begins

You have reached the end of the data-quality-sql track and the close of the Data Engineering pillar. Step back to the P12 pillar overview to see the two tracks together — pipelines and quality — or follow the number across the boundary into MLOps (P13), where the trustworthy feature table becomes a model's problem. The craft you built here — SQL that does not lie, joins that do not multiply, validation that fails loudly, and lineage that can always answer "where did this come from?" — is the foundation everything downstream stands on.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Evaluation infrastructure & experiment tracking →

ML Lifecycle · P13

Go up a level