Tokenization
P10.text-representation.02 · Audience: guest, it-ml, language-pro · Prerequisites: Mathematical Foundations
We need to associate words with mathematical representations like vectors, but we don't map words to vectors directly. We first split the words into subwords, and it is this list of subwords — derived from the corpus's words — that we will use to represent words and sentences. In this module you will watch that splitting happen step by step on the active corpus: the subword vocabulary is built from all the corpus sentences at once, while the pipeline itself is traced on the single sentence you select, so you can follow your own example all the way from raw characters to token IDs — the numbered entries these subwords (the pipeline calls them tokens) get in the vocabulary.
🗣️ From a linguist's perspective: the corpus
ⓘ Concept: One shared corpus for the whole tutorial
Why it matters — Every number in every later module — co-occurrence counts, SVD embeddings, attention weights — traces back to the corpus shown here, so you can follow one word end to end. Switch to the eight-sentence demo corpus to follow it by hand.
characters
0
words (approx.)
0
corpus sentences
0
Unicode code points of the selected sentence
| char | code point | decimal |
|---|
🎓 Interview deep-dive
WordPiece vs byte-level BPE. BERT's WordPiece scores merges by likelihood gain (specificity), GPT's byte-level BPE by raw frequency over bytes — byte-level guarantees any string is tokenizable with no [UNK] at all. Failure modes: numbers fragment unpredictably (why arithmetic is hard for LLMs), and low-resource languages pay a "token tax" — the same sentence costs several times more tokens than in English.
📚 Go further
- Sennrich et al. 2016 — Subword units
- Devlin et al. 2018 — BERT
- Hugging Face — Tokenizers course chapter
Check your understanding
Self-check
Why is the vocabulary built from the whole corpus rather than the selected sentence?
Self-check
What does coverage measure?
Self-check
What weight does a [PAD] position receive after masked softmax, and why?
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Keeping your files on this device
Off by default. The mentor always gets your file; this only decides whether your own copy stays here. Copies live in this browser only - they do not follow you to another device, and clearing site data removes them.
🎓 Practice ladder
2 graded rungs · ~20 minNow build the representation maths yourself. Each rung is a three-panel workspace: instructions on the left, a code editor in the middle, output + test results on the right. Run checks the visible tests; Submit grades against hidden edge cases — the co-occurrence matrix must stay symmetric as the window widens, and TF-IDF must handle a term absent from a document and a term present in every document (smallest IDF, never zero).
Rung 1 — Build the co-occurrence matrix
Loading exercise…
Rung 2 — Rank terms by TF-IDF
Loading exercise…
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
My notes on this module
Loading your notes...
Where next?
Later in Text Representation
This module unlocks
Go up a level