Positional Encoding
P10.positional-encoding.01 · Audience: guest, it-ml, language-pro · Prerequisites: Embeddings
Injecting sequence order into permutation-invariant token embeddings — from the problem to a position-aware embedding matrix, on the real demo corpus.
🗣️ From a linguist's perspective: order as signal
ⓘ Concept: The permutation-invariance problem
Why it matters — Attention is a content-only weighted sum: shuffle the input rows and the dot products are unchanged. Without position information a Transformer reads a bag of words.
Reference: Vaswani et al. 2017, §3.5
Computing…
🎓 Interview deep-dive
Permutation equivariance, precisely. Without PE, attention satisfies Attention(P·X) = P·Attention(X) for any permutation matrix P: the output is a content-only weighted sum over an unordered set of (key, value) pairs, so reordering inputs merely reorders outputs. PE breaks that symmetry on purpose — it is the only thing telling the model that position 3 differs from position 7.
📚 Go further
- Vaswani et al. 2017, §3.5 — Attention Is All You Need
- Amirhossein Kazemnejad — Transformer Architecture: The Positional Encoding
- Harvard NLP — The Annotated Transformer
- Andrej Karpathy — Let's build GPT
Check your understanding
Without position information, can a Transformer distinguish the same words in a different order?
For d_model = 4, what is PE(1, 0) to four decimals?
What is PE(0, 2i) for every i?
A 4-token sentence with d_model = 8: what is the shape after adding PE?
Why do identical words at different positions end up with different vectors?
For d_model = 4, what is PE(0, 1)?
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
🎓 Practice ladder
1 graded rung · ~6 minBuild the encoding yourself. The rung is a three-panel workspace: instructions, a code editor, and output + test results. Run checks the visible tests; Submit grades against hidden edge cases — even dimensions use sine, odd use cosine, the sine/cosine pair shares a frequency, and position 0 is all sines-to-0 / cosines-to-1.
Rung 1 — Fill the 4×4 PE table
Loading exercise…
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Positional Encoding
This module unlocks
Go up a level