Embedding Evaluation

P5.evaluation.02 · Audience: guest, it-ml, language-pro · Prerequisites: Evaluation Metrics for NLP

Evaluation Metrics for NLP LLM & Generation Evaluation

How do you know your word or sentence embeddings are any good? This module covers intrinsic evaluation (word-similarity benchmarks and analogy tasks), extrinsic evaluation (downstream classification and retrieval MRR@k), the sentence-embedding benchmarks STS-B and MTEB, and a worked cosine-similarity matrix built from the demo corpus. The embeddings being judged are the ones the P10 Transformers pillar builds — Embeddings and Sentence Embeddings.

Step 1 / 4 — Intrinsic Evaluation — Similarity Benchmarks and Analogies

Intrinsic evaluation measures embedding quality directly — without a downstream task. The two canonical methods are word-similarity benchmarks and analogy tasks.

Word-similarity benchmarks. A human-annotated dataset provides pairs of words and a similarity score:

Dataset	Pairs	Score range	What is rated
WordSim-353	353	0–10	relatedness (cat ↔ pet)
SimLex-999	999	0–10	similarity (cat ↔ feline, not cat ↔ fur)
MEN	3,000	0–50	association

Evaluation: compute cosine similarity between the model's word embeddings for each pair, then measure Spearman rank correlation (ρ) between model scores and human scores. Higher ρ is better.

Intrinsic check on the demo-corpus vocabulary. The demo corpus shares words across sentences. Four pairs to inspect:

Word A	Word B	Expected relatedness
lions	animals	highly related (both wild-life words)
population	community	moderately related (both group/society words)
algorithm	tokenization	weakly related (both CS terms)
king	food	unrelated

A good embedding model should assign higher cosine similarity to the first pair than the last. WordSim-353 provides the authoritative human scores to compare against.

Analogy tasks. The classic analogy test: king − man + woman ≈ queen. In embedding space, the direction from man → woman should be the same as the direction from king → queen:

\vec{v}(\text{king}) - \vec{v}(\text{man}) + \vec{v}(\text{woman}) \approx \vec{v}(\text{queen})

Evaluation: for each analogy (a is to b as c is to ?), compute the result vector v(b) − v(a) + v(c) and find the nearest vocabulary word (excluding a, b, c). Score = fraction of analogies answered correctly. The Google Analogy Dataset contains 19,544 analogies across semantic categories (capitals, currencies, family relations) and syntactic categories (adjective ↔ adverb, verb tenses).

A demo-corpus analogy. From the demo corpus, one possible analogy direction: lions is to dangerous as algorithm is to ? A model trained on the demo corpus is too small to answer this reliably, but it illustrates the test format. The P10 Transformers pillar's embedding modules — Embeddings and Learned Embeddings — run vector arithmetic of exactly this kind over the demo-corpus embedding space.

Limitations of intrinsic evaluation:

Low correlation with downstream tasks — a high WordSim-353 score does not guarantee good classification accuracy.
Benchmark saturation — models now score higher than human inter-annotator agreement on WordSim-353; the benchmark is no longer discriminative.
Context insensitivity — word-level intrinsic metrics miss polysemy ('bank' has very different embeddings depending on context in BERT).
Annotation bias — human scores encode cultural and linguistic biases that propagate into models trained to match them.

📚 Go Further

The benchmarks and papers this module draws on.

Type	Resource
Paper	Cer et al. 2017 — SemEval-2017 Task 1: Semantic Textual Similarity (the STS-Benchmark)
Paper	Muennighoff et al. 2022 — MTEB: Massive Text Embedding Benchmark
Paper	Faruqui et al. 2016 — Problems With Evaluation of Word Embeddings Using Word Similarity Tasks (the intrinsic/extrinsic gap)
In-app	P10 — Embeddings and Learned Embeddings, for the embedding spaces being evaluated
In-app	P10 — Sentence Embeddings, for the sentence encoders that STS-B and MTEB rank

✅ Key takeaways

Intrinsic evaluation tests the embedding space directly: Spearman correlation of model cosine similarities against human word-similarity judgements (WordSim-353, SimLex-999, MEN) and analogy arithmetic (king − man + woman ≈ queen, scored on the 19,544-analogy Google dataset).
Extrinsic evaluation tests embeddings on a real task: freeze the embeddings, train a linear probe, report F1 — or rank documents and report MRR@k, where each query contributes the reciprocal rank of its first relevant result (worked example: MRR@3 = (1.000 + 0.500) / 2 = 0.750).
The intrinsic/extrinsic gap is real: the correlation between WordSim-353 scores and downstream task F1 is near zero across models (Faruqui et al. 2016) — always report at least one extrinsic benchmark.
Sentence-level benchmarks: STS-B correlates model cosine similarity with human 0–5 equivalence judgements; MTEB averages 56 datasets across 8 task categories for a far more robust signal than any single benchmark.
On the demo corpus, bag-of-words cosine captures lexical overlap only — the top pair S2 ↔ S6 (0.621) is driven by function words. Trained sentence encoders close the semantic gap: on STS-B, ρ ≈ 0.50 (BoW) → 0.65 (TF-IDF) → 0.87 (SBERT).

Next: P5.evaluation.03 — LLM & Generation Evaluation: BERTScore, benchmark suites (MMLU, HELM, BIG-Bench), LLM-as-judge, RLHF reward models, and human annotation protocols.

Ask the mentor about this module

Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.

Ctrl/Cmd + Enter to send

Try it yourself

A scratch console for this page's ideas — ungraded, nothing you run here is recorded.

Scratch console

A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.

Output appears here.

Where next?

Continue

LLM & Generation Evaluation →

Evaluation · P5

Later in Evaluation

LLM & Generation Evaluation

Go up a level