Modern Positional Encodings
P10.positional-encoding.02 · Audience: guest, it-ml, language-pro · Prerequisites: Positional Encoding
Sinusoidal positional encoding solved the word-order problem, but it turned out to have real limits: a model trained on short texts cannot make sense of positions it has never seen, and language mostly cares about how far apart two words are, not about their absolute seat numbers. Modern language models therefore replaced it with two different ideas: RoPE, which rotates vectors inside attention so only relative distance survives, and ALiBi, which simply makes attention fade with distance. This module explains both in plain language, through analogies and pictures — no matrix maths required.
Think of sinusoidal PE as theatre tickets: every seat has a printed row-and-seat number. It works perfectly — for the theatre it was printed for. Ask the ticketing system about row 200 in a theatre built with 32 rows and it has never seen such a ticket; the numbers it prints are technically defined but mean nothing reliable.
That is the model's problem with sequences longer than training: the sinusoidal signal keeps oscillating past the boundary, but the model never learned what those values imply.
Open in Labs: Positions encoded and compared
The second limit: the tickets are absolute. Language mostly cares about relative distance — “the adjective just before its noun” matters, “the adjective at absolute position 17” rarely does.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Keeping your files on this device
Off by default. The mentor always gets your file; this only decides whether your own copy stays here. Copies live in this browser only - they do not follow you to another device, and clearing site data removes them.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
My notes on this module
Loading your notes...
Where next?
Go up a level