tau

01what it measures, and how

Kendall's tau-b between the judge's scores and the experts' ratings of an article's 16 summaries, averaged over the four dimensions.

Per article the judge scores 16 summaries on coherence, consistency, fluency and relevance. For each dimension its scores are correlated with the expert ratings by tau-b, which corrects for ties on both sides (an LLM judge's integer scores tie often); a dimension the judge did not return, or a constant side, counts 0. tau is the mean of the four, then over articles: agreement with people on ordering summaries, not a summary's quality.

02on summeval_100

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold