Batch, Streams & Late Data

P12.data-pipelines.03 · Audience: guest, it-ml, language-pro · Prerequisites: Formats & Schemas

Formats & Schemas Orchestration Without the Logo Soup

LLM grading (this page):Real gradingOff — no LLM calls

A pipeline rarely loads all of history every night — that would be slow and wasteful. Instead it loads what is new: yesterday's orders, this hour's events. The moment you load data incrementally, a trap opens that has caught almost every data team at least once. Re-run yesterday's load — because a job glitched, or a colleague wasn't sure it finished — and if you simply append it again, yesterday's numbers are now counted twice. And a second problem arrives with the mail: what do you do when a correction for Monday shows up on Wednesday, after Monday's total was already reported? This module builds the mental model and the one technique — idempotent merging — that defuses both.

Step 1 / 5 — Batch or stream — two mental models

There are two ways to move data through a pipeline, and it helps to hold both as pictures. Batch processing gathers data into chunks and processes a chunk at a time: "every night, load yesterday's orders". Stream processing handles each record as it arrives, continuously: "the moment an order is placed, update the running total". Batch is a bus that leaves on a schedule; a stream is a moving walkway that never stops.

Aspect	Batch	Stream
Unit of work	A chunk (a day, an hour)	One record
When it runs	On a schedule	Continuously
Freshness	As old as the last run	Near real-time
Simplicity	Simpler to build & reason about	More moving parts

ⓘ Concept: Batch and stream are the same job at different grain

batch: process a chunk on a schedule → simple, re-runnable, slightly stale · stream: process each record continuously → fresh, more moving parts

Why it matters — A stream is, in a sense, batch processing with a batch size of one and no wait — and most of the hard problems (duplicates, ordering, late data) show up in both. Batch is the right default: it is simpler, cheaper, and easy to re-run, and most reporting does not need up-to-the-second freshness. Reach for streaming only when the value of fresh data justifies the extra machinery. Everything this module teaches next is framed in batch because it is clearer there, but the ideas carry straight over to streams.

✅ Key takeaways

Batch processes data in chunks on a schedule; streaming processes each record continuously. Batch is the simpler, re-runnable default; the same problems — duplicates, ordering, late data — appear in both, so the module teaches them in the clearer batch frame.
Incremental loads add only what is new. Doing that by appending is not idempotent: re-run yesterday's batch of 3 orders and the table becomes 6 — every total doubled, silently, exactly the pillar-opener failure.
The fix is to merge by a key (upsert): for each incoming row, replace the row with that key or insert it. Re-running the same batch overwrites each row with an identical copy, so the table stays at 3 — the load is now idempotent.
The same keyed merge handles late data: a correction that arrives after a period closed overwrites exactly its own key. Monday, closed at 10, is restated to 25 by a late Monday correction while Tuesday is untouched — restate the one period, touch nothing else.
One operation — an idempotent, keyed merge — makes incremental loading safe to re-run, defuses the duplicate trap, and restates late data cleanly and auditably, at either batch or stream grain.

Next: P12.data-pipelines.04 — Orchestration Without the Logo Soup

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

🎓 Practice ladder

3 graded rungs · ~29 min

Now build the merge yourself. Each rung is a three-panel workspace: instructions on the left, a code editor in the middle, output and test results on the right. Run checks the visible tests; Submit grades against hidden edge cases. You start by spotting the duplicates a naive append creates, then write an idempotent upsert-by-key merge, and finally handle a late-arriving correction and report exactly which days it restated.

Rung 1 — Spot the duplicates (explorer)

Loading exercise…

Rung 2 — Idempotent merge

Loading exercise…

Rung 3 — Late data arrives (senior)

Loading exercise…

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Orchestration Without the Logo Soup →

Data Pipelines · P12

Later in Data Pipelines

Orchestration Without the Logo Soup

Go up a level