Data Pipelines
A track of P12 · Data Engineering.
From a notebook that ran once to a pipeline that runs every night: formats, schemas, batch-vs-stream thinking and orchestration - tool-agnostic on purpose.
A notebook that produces the right number once is not a pipeline. A pipeline runs tonight, and tomorrow night, and the night a batch arrives twice or a day late — and it produces the right number every time. This track is the discipline that makes that true, built from the one property everything rests on: idempotency, so re-running a job never double-counts and never corrupts.
You will start there — why "it ran once" is not engineering — then work through file formats as contracts (row-oriented CSV versus columnar Parquet, schema-on-read versus schema-on-write, and the quiet ways a type lies when a number is stored as text or a null becomes the string "NA"). The batch module adds the two failure modes that define real pipelines: the duplicate that a naive re-append silently doubles, fixed by an upsert keyed on identity; and late data — a correction that lands after the day closed and must restate that day's total without touching the others. The track closes on orchestration as a tool-agnostic idea: a DAG of jobs, retries, backfills and schedules, minus any particular logo.