From Notebook to Pipeline
P12.data-pipelines.01 · Audience: guest, it-ml, language-pro
A monthly revenue figure went to the board at €4.2 million. It was wrong. Not by a rounding error — the number was nearly double the truth, because a report notebook had quietly counted a duplicated row twice where two tables were joined, and nobody noticed for a quarter. No code crashed. No red error appeared. A plausible-looking number sat in a slide, and everyone downstream trusted it.
That is the failure this whole pillar exists to prevent. Everything upstream of a model or a report — where the data comes from, how it is shaped, whether it can be trusted — is data engineering, and it decides whether the numbers everyone else builds on deserve that trust. This pillar has two tracks: the pipelines that move and reshape data reliably, and the quality machinery — SQL, validation, lineage — that makes a single number defensible. This module opens the pillar with its most basic and most violated idea: a computation that ran once, by hand, is not yet engineering.
A notebook is a wonderful place to discover something. You load a file, try a join, plot a chart, and a number appears. The trouble begins the moment someone else needs that number again — next month, from fresh data, without you sitting there running cells in the right order. A notebook that produced the right answer once has proven almost nothing about whether it will produce the right answer tomorrow.
| A notebook that ran once | A pipeline |
|---|---|
| Cells run in whatever order you clicked | Steps run in a fixed, declared order |
| Works on the data that happened to be loaded | Works on next month's data too |
| A person watches it and fixes hiccups by hand | Runs unattended; failures are visible, not silent |
| Run it twice and who knows what happens | Run it twice and the result is unchanged |
ⓘ Concept: Engineering is what survives the second run
Why it matters — The gap between a script and a pipeline is not fancier tools — it is a set of guarantees about what happens when you are not watching. Will it run in the right order? Will it work on data it has never seen? Will running it again be safe? A notebook answers none of these; a pipeline answers all of them. Every idea in this pillar is one of those guarantees made concrete.
None of this asks you to abandon notebooks — they remain the right tool for exploring. It asks you to notice when a computation has graduated from "I am figuring this out" to "the business depends on this running", because that is the moment it needs the guarantees below. The Python you already have (Python for Data, P4) is exactly the material a pipeline is built from; this pillar adds the reliability around it.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Data Pipelines
This module unlocks
Go up a level