LLM & Generation Evaluation

BLEU and ROUGE measure surface overlap — matching tokens between hypothesis and reference. For abstractive summarisation, paraphrase, and free-form generation, a correct output may share zero tokens with the reference and still score 0.

The paraphrase problem:

Row 1: the hypothesis is a perfect paraphrase of the reference but BLEU-1 = 0 because no tokens match. BLEU cannot distinguish a wrong answer from a correct paraphrase.

BERTScore (Zhang et al. 2020) replaces exact token matching with contextual embedding similarity — the contextual vectors come from a pretrained BERT-style encoder, the machinery built up in the P10 Transformers pillar:

1. Embed all m hypothesis tokens and all n reference tokens with a pretrained BERT-style model.
2. Precision: for each hypothesis token, take its maximum cosine similarity to any reference token:
   $$P_{\text{BERT}} = \frac{1}{m} \sum_{j=1}^{m} \max_{i} \cos(\hat{r}_j, r_i)$$
3. Recall: for each reference token, take its maximum cosine similarity to any hypothesis token:
   $$R_{\text{BERT}} = \frac{1}{n} \sum_{i=1}^{n} \max_{j} \cos(r_i, \hat{r}_j)$$
4. F1: the harmonic mean of P_BERT and R_BERT.

Key property: 'cat' and 'feline' have high cosine similarity in BERT space, so a paraphrase scores near 1.0 even with no token overlap.

Worked BERTScore example. Hypothesis: 'a canine ran fast'; reference: 'a dog sprinted'. Suppose a BERT model produces the following pairwise cosine similarities (rows = hypothesis tokens, columns = reference tokens):

Precision (row max, then mean): a → 1.00, canine → 0.92, ran → 0.88, fast → 0.72, so P = 0.8800. Recall (column max, then mean): a → 1.00, dog → 0.92, sprinted → 0.88, so R = 0.9333.

BERTScore F1 = 2 × 0.8800 × 0.9333 / (0.8800 + 0.9333) = 0.9059. Compare to BLEU-1: only 'a' matches, so BLEU-1 ≈ 0.25 × BP, well below 0.25. BERTScore captures that 'canine'/'dog' and 'ran'/'sprinted' are semantically equivalent.

Other embedding-based metrics:

• MoverScore (Zhao et al. 2019): uses Earth Mover's Distance (Wasserstein distance) between the embedding distributions of hypothesis and reference — captures meaning transportation rather than just token alignment.
• BLEURT (Sellam et al. 2020): fine-tunes BERT specifically to predict human translation-quality judgements — a learned metric that correlates better with human scores than BERTScore on MT.
• COMET (Rei et al. 2020): another learned metric for MT, trained on direct-assessment human scores from WMT — currently state-of-the-art for MT evaluation.