Sentence Embeddings

P10.transformer-architecture.02 · Audience: guest, it-ml, language-pro · Prerequisites: Encoder Block

Pooling token representations into one sentence vector — the reference tutorial's capstone: raw text in, one comparable vector out.

Step 1 / 7 — From Tokens to Sentences

🗣️ From a linguist's perspective: from words to the sentence

A corpus linguist moves between granularities: word-level concordances, then sentence-level summaries. Pooling is that move made numeric — many token rows in, one sentence profile out.

ⓘ Concept: From token representations to sentence vectors

Three classic strategies: take the first row (CLS), average all rows (mean), or take each dimension's maximum (max).

\text{sentence\_vec} = \text{Pool}\!\left( \text{EncoderOutput} \right)\in \mathbb{R}^{d_{\text{model}}}

Why it matters — Search, clustering, and semantic comparison all need ONE fixed-size vector per text — pooling is the bridge from token space to sentence space.

Reference: Reimers & Gurevych 2019 — Sentence-BERT

sentenced_modelnum_heads

Computing…

🎓 Interview deep-dive

Adapting a pre-trained model. BERT-style: fine-tune with a task head on [CLS] (or mean-pool for sentence tasks — SBERT). GPT-style: prompt with in-context/few-shot examples, no weight updates. Between them: parameter-efficient fine-tuning (LoRA/PEFT — train tiny adapters), instruction tuning, and RAG (retrieve with pooled embeddings exactly like step 6, then generate grounded on the hits).

📚 Go further

Reimers & Gurevych 2019 — Sentence-BERT
Jay Alammar — The Illustrated BERT
Sentence-Transformers — pretrained models
Devlin et al. 2019 — BERT
Hugging Face — NLP course, ch. 5–6

Check your understanding

What is the encoder-output shape for 3 tokens with d_model = 8, and after pooling?

Which token does BERT use as the sentence embedding?

Which pooling strategy is most sensitive to a single highly-activated token?

Two sentence vectors have cosine similarity 0.95 — similar or dissimilar?

What advantage does mean pooling have over CLS without fine-tuning?

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

What an LLM actually is →

LLM Essentials · P10

This module unlocks

Go up a level