Modern PE (Deep Dive)
P10.positional-encoding.03 · Audience: it-ml · Prerequisites: Modern Positional Encodings
Beyond sinusoidal PE: how RoPE and ALiBi address its limitations in modern large language models — with the actual matrices, computed live.
🗣️ From a linguist's perspective: long-discourse reference
ⓘ Concept: Two limitations of sinusoidal PE
Why it matters — These two gaps — poor length extrapolation and absolute-only position — are precisely what RoPE and ALiBi were designed to close.
Recall — the sinusoidal formulas (P1)
🎓 Interview deep-dive
Extending context length. ALiBi extrapolates natively. RoPE models stretch further via position interpolation (compress positions into the trained range), NTK-aware scaling, or YaRN. Orthogonally, sparse/sliding-window attention (Longformer, BigBird, Mistral's SWA) and ring attention cut the O(n²) cost. The recurring trade-off: long-range faithfulness vs compute and memory.
📚 Go further
- Press et al. 2022 — ALiBi: Train Short, Test Long
- Su et al. 2023 — RoFormer (RoPE)
- Kazemnejad — The Positional Encoding
- Lilian Weng — The Transformer Family v2.0
Check your understanding
Name one limitation of fixed sinusoidal PE on sequences longer than training.
Which method rotates Q and K inside the attention dot product?
Does ALiBi make attention to distant tokens more or less likely?
Which method has no learned parameters AND a linear distance penalty?
After RoPE, does the Q·K dot product depend on absolute or relative position?
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Go up a level