WordPiece Tokenization

P10.text-representation.05 · Audience: guest, it-ml, language-pro · Prerequisites: BPE Tokenization

Merging the most frequent pair, as BPE does, is not the only reasonable way to grow a vocabulary: a pair of pieces can be rare yet so exclusive that they almost never appear apart, and that exclusivity is strong evidence of a real linguistic unit. WordPiece, the tokenizer behind BERT, keeps BPE's merge loop but ranks candidate pairs by exactly that specificity instead of raw frequency. This module traces WordPiece stage by stage on the same corpus as the previous module, so you can see precisely where the two tokenizers diverge and why BERT and GPT split the same word differently.

Step 1 / 6 — Same Loop, Different Score

🗣️ From a linguist's perspective: Specificity over raw frequency

A terminologist building a glossary does not add a phrase just because its two words are individually common — machine and learning are both frequent, so their co-occurrence is unremarkable. They add a phrase when the pairing is distinctive: two words that almost only appear together point to a real term. WordPiece scores merges the same way — it rewards a pair whose parts are mutually exclusive, even if the pair itself is rare.

ⓘ Concept: Three differences from BPE

WordPiece runs the same iterative merge loop as BPE, but:

	BPE	WordPiece
Word boundary	`</w>` suffix	`##` prefix on non-initial tokens
Merge score	`freq(AB)`	`freq(AB) / (freq(A) × freq(B))`
Inference	replay merge rules in order	longest-match-first in the vocabulary

The ## prefix inverts the mental model: BPE marks where a word ends; WordPiece marks this token is glued to whatever came before it.

Why it matters — Because the score divides by freq(A) × freq(B), a pair whose parts appear only together scores 1.0 — the maximum — even if it occurs once. BPE would never pick it. That single change is why BERT and GPT produce different tokens for the same word.

Reference: Schuster & Nakamura 2012 — Japanese and Korean Voice Search (WordPiece)

Used by: BERT, DistilBERT, mBERT. The capstone compares WordPiece with BPE live.

Reading the corpus…

Open in Labs: BPE against WordPiece

Check your understanding

Self-check

WordPiece and BPE run the same merge loop. Name the one quantity they rank candidate pairs by that differs.

Self-check

A pair appears only once, but its two tokens only ever appear together. What is its WordPiece score, and will BPE merge it early?

Self-check

BPE stores its ordered merge rules; WordPiece does not. What does WordPiece use at inference time instead?

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Images, PDF or text. Kept on this device only.

Keeping your files on this device

Keep a copy of the files I attach, in this browser

Off by default. The mentor always gets your file; this only decides whether your own copy stays here. Copies live in this browser only - they do not follow you to another device, and clearing site data removes them.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

My notes on this module

Loading your notes...

Where next?

Continue

Learned Embeddings →

Text Representation · P10

Later in Text Representation

Learned Embeddings

Go up a level