What is Attention?
P10.attention.01 · Audience: guest, it-ml, language-pro
By the end of Blocks 4–5, every word of the sentence has become a vector that encodes what the word means (its SVD embedding) and where it stands (its positional encoding). The whole sentence is one matrix — one row per word, d_model numbers per row.
Computing…
But every row is still context-blind: the vector for bank is identical in "river bank" and "savings bank". The question Attention answers is:
For each word, which other words matter right now — and how do we mix their meanings into its representation?
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Attention
Go up a level