Formats & Schemas

P12.data-pipelines.02 · Audience: guest, it-ml, language-pro · Prerequisites: From Notebook to Pipeline

From Notebook to Pipeline Batch, Streams & Late Data

Before a pipeline can transform data, it has to read it from somewhere and write it somewhere — and the file format it uses is a real engineering decision, not an afterthought. A million-row export saved the convenient way can be ten times larger and ten times slower to read than the same data saved the deliberate way. Worse, a format can quietly let a column lie about what it holds: a price stored as text, a missing value that turned into the literal word NA. This module is about how data is stored — and how a schema turns a loose pile of bytes into a contract you can actually trust.

Step 1 / 5 — Rows on disk, or columns on disk

All three formats hold the same table — the difference is how the bytes are laid out on disk, and that one choice drives everything else. A row-oriented format writes one whole record after another: all of row 1's fields, then all of row 2's. A columnar format writes one whole column after another: every city, then every price, then every quantity.

Why care? Because most analytics reads a few columns across all rows — "sum the revenue", "average the quantity" — not whole records. When a column sits together on disk, the reader grabs exactly that column and skips the rest; and because a column holds all-similar values, it compresses far better than a row of mixed types. Row formats are the opposite bet: cheap to append one record at a time and easy for a human to read, but the reader must walk every field of every row even to sum a single column.

ⓘ Concept: Row-oriented vs columnar is the root trade-off

row-oriented: whole records together \to easy to read & append, slow & large to scan \cdot columnar: whole columns together \to fast & small to scan, binary & write-once

Why it matters — Almost every difference between these formats follows from this one layout choice. Columnar storage (Parquet) wins analytics: read only the columns you need, compress each column hard, scan huge tables fast — at the cost of being a binary format no one can open in a text editor and slower to write a single new row. Row storage (CSV, JSON) wins for readability, easy appends and interchange, at the cost of size and scan speed. Choosing a format is really choosing which of these you are optimising for.

✅ Key takeaways

The file format is an engineering decision. The root trade-off is layout: row-oriented (CSV, JSON) keeps whole records together — readable, easy to append, but large and slow to scan; columnar (Parquet) keeps whole columns together — small, fast to scan and highly compressible, but binary and write-once.
CSV is the universal human-readable interchange; JSON handles nested or ragged records; Parquet is the pipeline's own storage — smallest, fastest, and it carries a typed schema. Rule of thumb: CSV/JSON at the edges, Parquet in the middle.
A schema is the declared shape of a table. Schema-on-write (Parquet) checks and stores types with the data, so the file is a self-describing contract; schema-on-read (CSV/JSON) stores only text and makes every reader re-declare the types.
Without a schema, types silently lie: a number stored as text sums to glued strings ('10.0' + '20.0' → '10.020.0'); a null exported as the word 'NA' turns the whole column into text. Nothing crashes — you get a plausible wrong answer.
The defence is cheap: declare the types, decide what a missing value looks like, and make a violation fail loudly and early. That question — what is this really, and what did null become? — is where the data-quality track begins.

Next: P12.data-pipelines.03 — Batch, Streams & Late Data

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Batch, Streams & Late Data →

Data Pipelines · P12

Later in Data Pipelines

Go up a level