Modern Positional Encodings
P10.positional-encoding.02 · Audience: guest, it-ml, language-pro · Prerequisites: Positional Encoding
How RoPE and ALiBi solve the limitations of sinusoidal PE — no matrix maths required.
Think of sinusoidal PE as theatre tickets: every seat has a printed row-and-seat number. It works perfectly — for the theatre it was printed for. Ask the ticketing system about row 200 in a theatre built with 32 rows and it has never seen such a ticket; the numbers it prints are technically defined but mean nothing reliable.
That is the model's problem with sequences longer than training: the sinusoidal signal keeps oscillating past the boundary, but the model never learned what those values imply.
The second limit: the tickets are absolute. Language mostly cares about relative distance — “the adjective just before its noun” matters, “the adjective at absolute position 17” rarely does.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Positional Encoding
Go up a level