Tokenization
P10.text-representation.02 · Audience: guest, it-ml, language-pro · Prerequisites: Mathematical Foundations
From the demo corpus to token IDs — every step made visible. The vocabulary is built from all corpus sentences; the pipeline is traced on the one you select.
🗣️ From a linguist's perspective: the corpus
ⓘ Concept: One shared corpus for the whole tutorial
Why it matters — Every number in every later module — co-occurrence counts, SVD embeddings, attention weights — traces back to these eight sentences, so you can follow one word end to end.
characters
0
words (approx.)
0
corpus sentences
0
Unicode code points of the selected sentence
| char | code point | decimal |
|---|
🎓 Interview deep-dive
WordPiece vs byte-level BPE. BERT's WordPiece scores merges by likelihood gain (specificity), GPT's byte-level BPE by raw frequency over bytes — byte-level guarantees any string is tokenizable with no [UNK] at all. Failure modes: numbers fragment unpredictably (why arithmetic is hard for LLMs), and low-resource languages pay a "token tax" — the same sentence costs several times more tokens than in English.
📚 Go further
- Sennrich et al. 2016 — Subword units
- Devlin et al. 2018 — BERT
- Hugging Face — Tokenizers course chapter
Check your understanding
Why is the vocabulary built from the whole corpus rather than the selected sentence?
What does coverage measure?
What weight does a [PAD] position receive after masked softmax, and why?
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
🎓 Practice ladder
2 graded rungs · ~20 minNow build the representation maths yourself. Each rung is a three-panel workspace: instructions on the left, a code editor in the middle, output + test results on the right. Run checks the visible tests; Submit grades against hidden edge cases — the co-occurrence matrix must stay symmetric as the window widens, and TF-IDF must handle a term absent from a document and a term present in every document (smallest IDF, never zero).
Rung 1 — Build the co-occurrence matrix
Loading exercise…
Rung 2 — Rank terms by TF-IDF
Loading exercise…
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Text Representation
This module unlocks
Go up a level