Learned Embeddings

P10.text-representation.06 · Audience: guest, it-ml, language-pro · Prerequisites: Embeddings

WordPiece Tokenization Positional Encoding

From a masked token to a weight update — watching gradient descent teach the embedding table, one BERT-style training step traced end to end.

sentencetoken to masklearning rate

Step 1 / 5 — Static vs Contextual Embeddings

🗣️ From a linguist's perspective: why context changes meaning

A polysemous word like bank holds several senses at once; only the sentence disambiguates. A static table must average all senses into one vector; a trained encoder produces a different representation for each context — the dictionary entry versus the word-in-use.

ⓘ Concept: Two paradigms

The training loop: mask a token → predict it from context → measure the error → push the error back through the network → nudge the table. This page traces one such step with real numbers.

e_{\mathrm{bank}} = E[\mathrm{bank}] \;(\text{fixed}) \qquad \text{vs} \qquad h_{\mathrm{bank}} = \text{Encoder}(E, \text{context})

Why it matters — The leap from static (T2) to learned + contextual is the leap from LSA-era NLP to BERT and GPT.

Reference: Devlin et al. 2018 — BERT

Computing…

🎓 Interview deep-dive

BERT vs GPT pretraining. BERT masks 15% of tokens (with the 80/10/10 trick: 80% [MASK], 10% random token, 10% unchanged) and predicts them bidirectionally; GPT predicts every next token left-to-right. GPT's objective yields a loss signal at every position (dense), BERT's only at masked ones (sparse) — one reason autoregressive scaling proved so effective. This page's variant feeds the true token (CBOW-with-attention) so the gradient reaches the true row directly — same dynamic, no special-token bookkeeping.

📚 Go further

Devlin et al. 2018 — BERT
Jay Alammar — The Illustrated BERT
Andrej Karpathy — Let's build GPT

Check your understanding

What three components sum to form the BERT input embedding?

Why does dL/dlogits sum to exactly zero?

What happens to the update if you raise the learning rate from 0.01 to 0.5?

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

Positional Encoding →

Positional Encoding · P10

Go up a level