BPE Tokenization
P10.text-representation.04 · Audience: guest, it-ml, language-pro · Prerequisites: Tokenization
If a tokenizer stores every word it has ever seen, its vocabulary explodes and any new word still breaks it; if it stores only single characters, nothing ever breaks but every sentence becomes a long string of nearly meaningless pieces. Byte-Pair Encoding resolves this tension by learning which character sequences occur often enough to deserve a token of their own, so common words stay whole while rare words fall back to reusable pieces. This module builds a GPT-style BPE tokenizer from scratch, one merge at a time, and then uses it to tokenize text it has never seen.
🗣️ From a linguist's perspective: Morphemes — the reusable pieces inside words
ⓘ Concept: Three levels of tokenization
Word-level — split on spaces. Simple, but the vocabulary explodes and any unseen word becomes a single unknown token. photosynthesis → [UNK].
Character-level — one token per character. Never fails on unseen words, but sequences become very long and each token carries almost no meaning.
Subword (BPE) — start from characters and learn which adjacent pieces to glue together based on frequency. Common words become single tokens; rare words fall back to a handful of subword pieces. This is the compromise every modern LLM uses.
Why it matters — BPE began life as a text-compression algorithm and was adapted for NLP by Sennrich et al. (2016). GPT-2, GPT-4, Llama, and RoBERTa all tokenize with a BPE variant — the vocabulary you are about to build is the very first thing that runs when you type a prompt into any of them.
Reference: Sennrich et al. 2016 — Neural Machine Translation of Rare Words with Subword Units
Used by: GPT-2, GPT-4, Llama, RoBERTa. The four stages ahead — initialise, count pairs, merge, infer — are the entire algorithm.
Reading the corpus…
Open in Labs: BPE against WordPiece
Check your understanding
Self-check
Why is subword tokenization preferred over word-level tokenization for a word the tokenizer has never seen before?
Self-check
Two pairs each appear once, but one is inside a word with frequency 5 and the other inside a word with frequency 1. Which does BPE merge first?
Self-check
The merge rules are applied in a fixed order during inference. What determines that order?
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Keeping your files on this device
Off by default. The mentor always gets your file; this only decides whether your own copy stays here. Copies live in this browser only - they do not follow you to another device, and clearing site data removes them.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
My notes on this module
Loading your notes...
Where next?
Later in Text Representation
Go up a level