Positional Encoding

Take "dog bites man" and "man bites dog". Same three words — and after the embedding step, the very same three vectors. To everything you built in the text-representation track, the headline and the non-story are indistinguishable: a bag of identical vectors with order nowhere to be found. Attention, waiting next door, only makes this worse — it mixes vectors without caring where they sit. So where does word order live? Out of the box: nowhere. It has to be put in deliberately, and this track is about the strange, elegant way that is done.

The trick is to give every slot in the sentence its own signature — a vector that says "I am position three" — and simply add it onto the word's embedding. After that, each token's vector carries both what it is and where it stands, and the difference between biter and bitten is back in the numbers. The obvious follow-up — why signatures built from waves instead of just counting 1, 2, 3? — has a genuinely satisfying answer, and you will see it plotted rather than asserted.

From there the track walks forward to the present day. The classic scheme from the original Transformer paper comes first, computed and visualised until its stripe patterns stop looking like abstract art. Two further modules then take on the rotary and relative schemes running inside current LLMs — and why the field moved on, which turns out to be a story about sentences longer than any the model was trained on.

The corpus below holds the sentences these position signatures get stamped onto — edit it and every pattern in the track recomputes for your words.