Re-ranking & IR metrics
P11.ir-rag.02 · Audience: guest, it-ml, language-pro · Prerequisites: Information retrieval & hybrid search
One retriever cannot be both fast enough to scan everything and accurate enough to order the top. The resolution is a funnel of progressively more expensive re-rankers — and three metrics that tell you whether it works: recall@k, MRR, and nDCG@k.
- Tune for recall (don't miss the right doc) → it must stay cheap → shallow scorer → good shortlist, mediocre order.
- Tune for precision (perfect order) → the scorer gets expensive → too slow over millions.
🎯 Split the job across stages: a cheap high-recall retriever builds a shortlist, then a more expensive re-ranker sharpens that much smaller set. Recall is won early and cannot be recovered later; precision is won late, where you can afford it.
⚡ Interview Ref — the quick-scan Reference face
- Why stages: one retriever can't be both cheap-enough-to-scan-all and accurate-enough-to-order-the-top. Recall early (cannot be recovered), precision late (affordable on a small set).
- Funnel counts: millions → thousands (candidate gen: ANN+BM25 fused) → tens (cross-encoder) → few (optional LLM). ~10× shrink per stage.
- Bi- vs cross-encoder: bi encodes query/doc separately → precompute doc vectors offline → scans millions; cross encodes them together (full cross-attention) → no precompute → one pass/candidate → re-ranks hundreds.
- recall@k: |relevant ∩ top-k| / |relevant| — the ceiling a re-ranker can't raise.
- MRR: 1 / rank-of-first-relevant, averaged. Best when there's one right answer.
- nDCG@k: DCG (log₂ position discount) / IDCG (ideal order). Re-ranking a few positions moves it even at fixed recall.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.
Try it yourself
A scratch console for this page's ideas — ungraded, nothing you run here is recorded.
Scratch console
A scratch console with the scientific stack (pandas, numpy, scikit-learn). Runs on the server — no network, resource-limited and measured.
Output appears here.
Where next?
Later in Information Retrieval & RAG
This module unlocks