Evaluation

Your spam filter is 99% accurate, and your inbox is still full of spam. How? Easily: if only one email in a hundred is spam, a filter that waves everything through is 99% accurate too. Accuracy answered a question you never asked. The questions you did ask — of the spam it saw, how much did it catch? of what it flagged, how much was really spam? — have different names, different numbers, and a habit of disagreeing with each other. A single score is never the truth about a model; it is one camera angle, and someone chose where to point it.

This track is about choosing the angles deliberately. The first module builds the classic instruments by hand, on examples small enough to verify — precision and recall, F1, ROC curves, perplexity for language models, BLEU and ROUGE for translation and summaries — and then asks the quieter question: when the model says it is 90% sure, is it right 90% of the time? From there, evaluation moves to targets that resist a red pen. How do you grade sentence embeddings, where the output is a list of numbers claiming to capture meaning? And how do you grade generated text, where there is no single right answer at all and today's benchmark is tomorrow's homework leak?

Evaluation is the measurement layer of the entire platform: the pipeline builds embeddings, models make predictions, and this track decides whether any of it is good. It is also where self-deception is cheapest — a favourable split, a metric that flatters, a test set the model has quietly seen — so the methodology half of the track (splits, cross-validation, multiple-comparison honesty) matters as much as the formulas. Everything is computed live on the demo corpus or a hand-checkable toy, and the statistics you built in P1 return here with a job: deciding whether a two-point improvement is a result or a coin flip.