WordPiece Tokenization

P10.text-representation.05 · Audience: guest, it-ml, language-pro · Prerequisites: BPE Tokenization

WordPiece — how a BERT-style tokenizer scores merges by specificity, and where it diverges from BPE on the same corpus.

Step 1 / 6 — Same Loop, Different Score

🗣️ From a linguist's perspective: Specificity over raw frequency

A terminologist building a glossary does not add a phrase just because its two words are individually common — machine and learning are both frequent, so their co-occurrence is unremarkable. They add a phrase when the pairing is distinctive: two words that almost only appear together point to a real term. WordPiece scores merges the same way — it rewards a pair whose parts are mutually exclusive, even if the pair itself is rare.

ⓘ Concept: Three differences from BPE

WordPiece runs the same iterative merge loop as BPE, but:

	BPE	WordPiece
Word boundary	`</w>` suffix	`##` prefix on non-initial tokens
Merge score	`freq(AB)`	`freq(AB) / (freq(A) × freq(B))`
Inference	replay merge rules in order	longest-match-first in the vocabulary

The ## prefix inverts the mental model: BPE marks where a word ends; WordPiece marks this token is glued to whatever came before it.

Why it matters — Because the score divides by freq(A) × freq(B), a pair whose parts appear only together scores 1.0 — the maximum — even if it occurs once. BPE would never pick it. That single change is why BERT and GPT produce different tokens for the same word.

Reference: Schuster & Nakamura 2012 — Japanese and Korean Voice Search (WordPiece)

Used by: BERT, DistilBERT, mBERT. The capstone compares WordPiece with BPE live.

Check your understanding

WordPiece and BPE run the same merge loop. Name the one quantity they rank candidate pairs by that differs.

A pair appears only once, but its two tokens only ever appear together. What is its WordPiece score, and will BPE merge it early?

BPE stores its ordered merge rules; WordPiece does not. What does WordPiece use at inference time instead?

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Learned Embeddings →

Text Representation · P10

Later in Text Representation

Learned Embeddings

Go up a level