P12 · Data Engineering

A dashboard shows revenue up 12%. A vice-president repeats it in a board meeting. A week later someone notices the join that produced the number double-counted every order whose product appeared twice in the catalogue — the real figure was flat. Nobody wrote a bad model. The failure was entirely upstream: in the pipeline that moved the data and the checks that never ran. This pillar is that upstream world — the engineering that decides whether a number deserves the trust people place in it.

The first track is the pipeline itself. The difference between a notebook that ran once and a job that runs every night is idempotency: re-run it and nothing double-counts, nothing breaks. You will meet file formats as contracts (CSV versus columnar Parquet, and the ways types silently lie), the mental model of batch versus streaming and the late data that restates yesterday's totals, and orchestration as a tool-agnostic idea — a DAG of jobs with retries and backfills, minus the logo soup.

The second track is where trust is earned or lost. SQL, hands-on in an in-browser database, from the humble GROUP BY up to the joins that silently multiply rows and the window functions that compute running totals and latest-record dedup. polars and columnar thinking for analytics at scale. Then the quality machinery: validation rules that fail loudly on nulls, bad ranges, duplicates and orphaned keys — and lineage, the audit trail that lets you answer the only question that matters when a number is challenged: where did this come from?