Embedding Evaluation

P5.evaluation.02 · Audience: guest, it-ml, language-pro · Prerequisites: Evaluation Metrics for NLP

Evaluation Metrics for NLP LLM & Generation Evaluation

An embedding is just a list of coordinates, and no coordinate is right or wrong on its own — so the metrics of the previous module, which compare predictions against a ground truth, have nothing to grab onto here. The only way to judge an embedding space is indirectly, and there are two honest ways to do it: ask whether its geometry matches human judgements of word meaning, or ask whether the vectors actually help on a real task downstream — and the two verdicts often disagree. In this module you work through both modes, including a full cosine-similarity matrix computed over the eight demo sentences, and meet the standard benchmarks that score sentence embeddings at scale. The embeddings on trial are the very ones the P10 Transformers pillar builds.

Step 1 / 4 — Intrinsic Evaluation — Similarity Benchmarks and Analogies

Intrinsic evaluation measures embedding quality directly — without a downstream task. The two canonical methods are word-similarity benchmarks and analogy tasks.

Word-similarity benchmarks. A human-annotated dataset provides pairs of words and a similarity score:

Dataset	Pairs	Score range	What is rated
WordSim-353	353	0–10	relatedness (cat ↔ pet)
SimLex-999	999	0–10	similarity (cat ↔ feline, not cat ↔ fur)
MEN	3,000	0–50	association

Evaluation: compute cosine similarity between the model's word embeddings for each pair, then measure Spearman rank correlation (ρ) between model scores and human scores. Higher ρ is better.

Intrinsic check on the demo-corpus vocabulary. The demo corpus shares words across sentences. Four pairs to inspect:

Word A	Word B	Expected relatedness
lions	animals	highly related (both wild-life words)
population	community	moderately related (both group/society words)
algorithm	tokenization	weakly related (both CS terms)
king	food	unrelated

A good embedding model should assign higher cosine similarity to the first pair than the last. WordSim-353 provides the authoritative human scores to compare against.

Analogy tasks. The classic analogy test: king − man + woman ≈ queen. In embedding space, the direction from man → woman should be the same as the direction from king → queen:

\vec{v}(\text{king}) - \vec{v}(\text{man}) + \vec{v}(\text{woman}) \approx \vec{v}(\text{queen})

Evaluation: for each analogy (a is to b as c is to ?), compute the result vector v(b) − v(a) + v(c) and find the nearest vocabulary word (excluding a, b, c). Score = fraction of analogies answered correctly. The Google Analogy Dataset contains 19,544 analogies across semantic categories (capitals, currencies, family relations) and syntactic categories (adjective ↔ adverb, verb tenses).

A demo-corpus analogy. From the demo corpus, one possible analogy direction: lions is to dangerous as algorithm is to ? A model trained on the demo corpus is too small to answer this reliably, but it illustrates the test format. The P10 Transformers pillar's embedding modules — Embeddings and Learned Embeddings — run vector arithmetic of exactly this kind over the demo-corpus embedding space.

Limitations of intrinsic evaluation:

Low correlation with downstream tasks — a high WordSim-353 score does not guarantee good classification accuracy.
Benchmark saturation — models now score higher than human inter-annotator agreement on WordSim-353; the benchmark is no longer discriminative.
Context insensitivity — word-level intrinsic metrics miss polysemy ('bank' has very different embeddings depending on context in BERT).
Annotation bias — human scores encode cultural and linguistic biases that propagate into models trained to match them.

📚 Go Further

The benchmarks and papers this module draws on.

Type	Resource
Paper	Cer et al. 2017 — SemEval-2017 Task 1: Semantic Textual Similarity (the STS-Benchmark)
Paper	Muennighoff et al. 2022 — MTEB: Massive Text Embedding Benchmark
Paper	Faruqui et al. 2016 — Problems With Evaluation of Word Embeddings Using Word Similarity Tasks (the intrinsic/extrinsic gap)
In-app	P10 — Embeddings and Learned Embeddings, for the embedding spaces being evaluated
In-app	P10 — Sentence Embeddings, for the sentence encoders that STS-B and MTEB rank

✅ Key takeaways

Intrinsic evaluation tests the embedding space directly: Spearman correlation of model cosine similarities against human word-similarity judgements (WordSim-353, SimLex-999, MEN) and analogy arithmetic (king − man + woman ≈ queen, scored on the 19,544-analogy Google dataset).
Extrinsic evaluation tests embeddings on a real task: freeze the embeddings, train a linear probe, report F1 — or rank documents and report MRR@k, where each query contributes the reciprocal rank of its first relevant result (worked example: MRR@3 = (1.000 + 0.500) / 2 = 0.750).
The intrinsic/extrinsic gap is real: the correlation between WordSim-353 scores and downstream task F1 is near zero across models (Faruqui et al. 2016) — always report at least one extrinsic benchmark.
Sentence-level benchmarks: STS-B correlates model cosine similarity with human 0–5 equivalence judgements; MTEB averages 56 datasets across 8 task categories for a far more robust signal than any single benchmark.
On the demo corpus, bag-of-words cosine captures lexical overlap only — the top pair S2 ↔ S6 (0.621) is driven by function words. Trained sentence encoders close the semantic gap: on STS-B, ρ ≈ 0.50 (BoW) → 0.65 (TF-IDF) → 0.87 (SBERT).

Next: P5.evaluation.03 — LLM & Generation Evaluation: BERTScore, benchmark suites (MMLU, HELM, BIG-Bench), LLM-as-judge, RLHF reward models, and human annotation protocols.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Images, PDF or text. Kept on this device only.

Keeping your files on this device

Keep a copy of the files I attach, in this browser

Off by default. The mentor always gets your file; this only decides whether your own copy stays here. Copies live in this browser only - they do not follow you to another device, and clearing site data removes them.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

My notes on this module

Loading your notes...

Where next?

Continue

LLM & Generation Evaluation →

Evaluation · P5

Later in Evaluation

LLM & Generation Evaluation

Go up a level