BPE Tokenization

P10.text-representation.04 · Audience: guest, it-ml, language-pro · Prerequisites: Tokenization

Byte-Pair Encoding — how a GPT-style tokenizer learns its vocabulary one merge at a time.

Step 1 / 6 — Why Subword Tokenization?

🗣️ From a linguist's perspective: Morphemes — the reusable pieces inside words

A translator does not memorise every inflected form of a verb separately. They recognise a stem and its endings — travaill-er, travaill-ons, travaill-ait share one root. BPE discovers the same reusable pieces automatically: instead of storing newer, newest, and newborn as three unrelated symbols, it learns new as a shared unit and glues the rest on. The tokenizer ends up with a vocabulary of sub-word morphemes it was never explicitly taught.

ⓘ Concept: Three levels of tokenization

Word-level — split on spaces. Simple, but the vocabulary explodes and any unseen word becomes a single unknown token. photosynthesis → [UNK].

Character-level — one token per character. Never fails on unseen words, but sequences become very long and each token carries almost no meaning.

Subword (BPE) — start from characters and learn which adjacent pieces to glue together based on frequency. Common words become single tokens; rare words fall back to a handful of subword pieces. This is the compromise every modern LLM uses.

Why it matters — BPE began life as a text-compression algorithm and was adapted for NLP by Sennrich et al. (2016). GPT-2, GPT-4, Llama, and RoBERTa all tokenize with a BPE variant — the vocabulary you are about to build is the very first thing that runs when you type a prompt into any of them.

Reference: Sennrich et al. 2016 — Neural Machine Translation of Rare Words with Subword Units

Used by: GPT-2, GPT-4, Llama, RoBERTa. The four stages ahead — initialise, count pairs, merge, infer — are the entire algorithm.

Check your understanding

Why is subword tokenization preferred over word-level tokenization for a word the tokenizer has never seen before?

Two pairs each appear once, but one is inside a word with frequency 5 and the other inside a word with frequency 1. Which does BPE merge first?

The merge rules are applied in a fixed order during inference. What determines that order?

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

WordPiece Tokenization →

Text Representation · P10

Later in Text Representation

Go up a level