Labs · Inside the block

Positions encoded and compared

The sinusoidal table and why PE·PEᵀ depends only on the offset; the same property as a rotation (RoPE) and as a penalty on distance (ALiBi).

\mathrm{PE}_{p,2i} = \sin\!\left(p / b^{2i/d}\right),\quad \mathrm{PE}_{p,2i+1} = \cos\!\left(p / b^{2i/d}\right)

32 positions × d_model = 16

The paper's table: fast columns on the left, slow on the right.

The encoding, and the property that makes it usable

Reads /api/labs/positions — loading

Computing…