Change the ruler, not the run.
Every model output in these evals is frozen, so when a better way to score them arrives the eval can be measured again for the price of the scoring alone. We did it twice: a person corrected disputed labels, and SummEval-validated LLM judges re-scored the summaries. Both moved a headline result. Then one reader's blind check went the other way from the judges, and judges at 41× the price agreed with the first ones, not with the reader. On SummEval's expert ratings and on crowd ratings of 2022 LLM summaries, judges agree with people far better than the metric this leaderboard used, though not system by system; on this eval's own outputs, nobody has yet shown which of the two to believe. That is the lesson: a judge is worth having, and worth checking.
The metric was the weak link.
The summarisation eval ranks by coverage, an overlap score against reference summaries. SummEval measures how well a scorer agrees with expert annotators, as Kendall τ: 1 when the scorer orders an article's summaries as the experts do, 0 when unrelated. Coverage agrees with them at τ 0.10. The two LLM judges used below agree at τ 0.54 and 0.52, against 0.10 for the metric the leaderboard was sorted by.
The best judges agree with the experts about 93% as well as an expert does, the best of 24 chosen on the same articles, and 92% held out. That is the case for judging: when the cheap metric barely tracks what people think, a validated judge is the better instrument on that data, and it can re-score outputs you already have. Two caveats from the same report: the judges rank all 16 summaries of an article in one call, which this measure rewards, and a no-model lookup of which system wrote each summary already reaches 63% of an expert, so part of a judge's score may be recognising which system wrote a summary; how much is not measured.
First on coverage. 25th on the judges.
The two judges read every article with all 29 arms' summaries, names hidden and order shuffled: 960 calls, $0.3109, 197 of 200 articles scored. On their score every hosted LLM ranks above every BART model. bart_l, first on coverage, scores 3.87 of 5 and ranks 25th of 29, significantly below all 24 hosted LLMs. The hosted LLMs score 4.53–4.98, bunched near the top of a 5-point scale, so the order among them, anthropic_l first (18th on coverage), is not a finding.
The judges agree closely with each other: Spearman 0.93 over arm means. Re-ordering the summaries moved per-article scores by 0.11–0.21, a repeat call by 0.03–0.12. 3 of 960 replies did not parse; an article counts only when every block of every judge parsed, and 197 of 200 did.
Asked instead to pick the best of all 29 summaries in one call (judge-summ-pick-v1, 198 of 200 articles, $0.17), the two judges named the same winner on 0.36 of the articles; on 20 articles shown again in a new order, each judge kept its pick on 0.55 and 0.45 of them, and on a repeat call 0.75 and 0.95. A pick among 29 depends on their order about half the time: it ranks what the judges prefer, beside the grades, not instead of them.
14 to 6 the other way.
The one blind human check went the other way.
One reader, the author and not an expert, compared 20 pairs of
bart_landdeepseek_msummaries blind and preferredbart_l14 to 6 (sign test p = 0.115: a split at least this lopsided would turn up that often from coin flips, so not significant). The judges preferreddeepseek_mon all 20. Scored one summary at a time instead of in lists (unvalidated), they still rateddeepseek_mhigher on 192 of 200 articles and never lower. Two limits on the check itself: the reader saw only the first 170-220 words of each article, which can favour a summary that stays close to the lead; and the judged leader,anthropic_l, was not in it.A published check, on three 2022 systems.
On three other systems, the GPT-3, BART-based BRIO and T0 summaries of Goyal, Li & Durrett (2022, CC BY 4.0; 100 CNN articles, three annotators each), the judges agree with people about as well as people agree with each other: picking the best of the three, against one annotator they score 0.498 (gemma_l) and 0.526 (qwen_s), annotators against each other 0.454, chance 0.333. On the same articles they prefer the GPT-3 summary to BRIO's more often than the annotators do (72.7% and 77.0% against 63.6% and 63.2%; each rate's 95% interval is computed on its own, not as a paired test, and the intervals overlap for both judges). So they lean towards the LLM summary, within noise, in one comparison. The reader above was asked a different question, which summary is better overall; the judges score four dimensions.
So neither instrument is confirmed, and coverage stays primary.
The judges' ranking is directional at most. The one check against it is one reader on truncated articles, also not conclusive. Possible reasons the two differ, none tested here: the judges were validated on 2019 summaries and checked on 2022 ones (§05), never on these arms; the truncated articles can favour a summary that copies the lead; one reader is a small sample. Two judges agreeing with each other is not two judges agreeing with people.
Same validation score, 41× the price, the same verdict.
Were the first judges simply too small? The two highest-scoring SummEval judges from vendors not in the first pair, openai_l (τ 0.54) and anthropic_l (0.53; the first pair's gemma_l scored 0.54), validated by their committed SummEval runs, judged the same judging set with the same prompt, reasoning off: 960 calls, $12.92, 200 of 200 articles, no reply unparsed. They are larger and dearer judges, not clearly better ones.
They reach the same verdict. Every hosted LLM again ranks above every BART model; bart_l scores 3.82 of 5 and ranks 25th of 29, significantly below all 24 hosted LLMs on their score too. The two scorers order the arms alike (Spearman 0.85 over arm means), and on the 20 human-check pairs the frontier judges also prefer deepseek_m, 20 to 0. On the Goyal et al. systems they score 0.466 (openai_l) and 0.488 (anthropic_l) against one annotator, annotators against each other 0.454; on the articles both decided, openai_l prefers the GPT-3 summary on 70.4% against the annotators' 61.7%, and anthropic_l on 64.2% against 64.2%.
So the disagreement with the reader is not a small-judge problem: judges of very different size and price agree with each other, and not with the one reader. Either LLM judges share a preference that reader does not, or the reader's check is too small to tell; §05 tests the first. One thing grows with the frontier judges: own-family scores. openai_l rated its own vendor's arms +0.53 above the others and anthropic_l +0.59, against +0.18/+0.30 for the first pair: raw gaps, not bias estimates, but the judged leader, anthropic_l, is one judge's own family. Paying 41× as much did not settle it; only people can.
Checked against people, on LLM-era summaries.
SummEval's summaries are from 2019. Zhang et al. (2024) released crowd ratings of 19 summarisers on 100 CNN/DailyMail articles: GPT-3-era LLMs, base and instruction-tuned; the fine-tunes BRIO and Pegasus; and the reference highlights, three ratings each. All four judges scored them. Each is measured against the mean of two ratings with the third held out, and so is one rater against the other two; the ratings themselves are not stored here, only the judges' scores. Agreement is Kendall τ per article: 1 when an instrument orders the summaries as the raters do, 0 when it is unrelated, −1 when reversed.
Every judge agrees with the raters' consensus better than one rater does, ahead on 72–80 of the articles, and better than coverage, ahead on 88–94; every comparison p < 0.001. The cheap pair does as well as the frontier pair, for $0.11 against $4.35. On faithfulness alone they trail one rater (0.48–0.61 against 0.64).
And on the question this page keeps returning to: ranking the systems by mean coherence and relevance, the raters put BRIO, a BART-based fine-tune, 5th of 19, after four instruction-tuned LLMs; the judges rank it 1st or 2nd (Spearman 0.71–0.79 between the judges' and the raters' rankings of the systems; 1 is the same order). So these judges show no blanket preference for LLM-written summaries over a BART-style one. They deviate from the raters system by system in both directions: the raters rank the reference highlights 16th, the judges 3rd to 6th.
The limits. One crowd rating is a noisy baseline, and these τ are against crowd raters on another set, so they do not compare with SummEval's experts. The 19 systems span a wide range of quality, which makes a high Spearman easier. The articles and ratings have been public since 2023, so a judge may have seen them. And these are 2022's systems, not this eval's arms: this is evidence about the judges, not a check of the ranking in §02.
How much of the fine-tunes' lead was 17 labels?
On AG News three small fine-tunes led every hosted LLM, concentrated in 17 items that at least 80% of the hosted field got wrong. One reader, the author, adjudicated by topic those items and every item a fine-tune missed, 29 in all, shown each item's text, its gold label and the hosted majority's answer. The reader judged the gold wrong on all 17, and right on every item the fine-tunes missed. The reader's reason: the labels follow the source feed's section, so company news from a technology desk is Sci/Tech and non-US news, sport included, is World.
Re-scored against those labels for $0, every hosted LLM has a higher accuracy than every fine-tune. The fine-tunes fall from 1, 2, 3 to 25–27 of 30, and anthropic_l rises from 5th to first at 0.9850; it differs significantly from 9, including all three fine-tunes (which tie at 0.9050), and is not significantly different from 20 hosted arms: most of the new order is not significant. On DBpedia 3 labels were judged wrong; the top stays a crowd: gemma_m and qwen_m lead at 0.9964, gemma_m differs significantly from only 5 of 27, and bert_base_fy moves from 3rd to 11th at an unchanged 0.9857.
The instrument here was a person, with two limits. The reader saw the hosted answer, and every item the hosted field missed was resolved the hosted field's way, which is also what anchoring on that answer would produce. And only items some arm disagreed on were read, so an error the gold and every arm share would not be seen. What it shows for certain is how much the ranking depends on those labels: a fine-tune trained on a dataset learns its conventions. If your labels follow a house convention, that is what you want. If they mean what the words say, it is not.
Both stay beside the primary. Neither replaces it.
A re-measurement is written beside the original, never over it, and an eval's primary metric changes only by a recorded decision with the old one kept. Both decisions here kept the original:
AG News and DBpedia keep the dataset's gold: anyone can repeat it, and the adjudication is one reader's, made while seeing the hosted answer. Summarisation keeps coverage: the judges' ranking has not been confirmed by people on these summaries, and the one check that exists went the other way.
Both re-measurements are in the catalog, each as its own group beside the primary: the judged arms on the summarisation goal, the corrected labels on the AG News goal. The inspector tests either, arm against arm.
It matters, in both directions. On the primary metrics a free model is first, or not significantly different from first, in 5 of the 6 evals, and the dearest model is first in none. Taken at face value, the two re-measurements would make that 3 of 6, and put the dearest model, anthropic_l, first in 2 of 6: in AG News and summarisation it is first on the point estimate of both re-measured rankings, though not significantly different from most of the field in AG News; DBpedia's top stays a crowd. One of them rests on one reader who saw the hosted answer, the other on a ranking one reader's check went against. That is how much a measurement decision can move a headline, and why the decision is recorded rather than made by whichever number came last.
Five rules the harness enforces, or asks you to.
Validate it against people, on outputs like yours.
The gate refuses a judge with no committed validation run, and one below the scorer's floor if the scorer sets one (set it to the best free scorer's score on the same human ratings). What it cannot check is whether those ratings look like your outputs. Here the first validation was on 2019 summaries; a second, on crowd ratings of LLM-era summaries (§05), agrees with it, and neither is this eval's own outputs.
Use two judges from different vendors, blind.
Each judge is reported, and how far they agree. Arm names are hidden and the order is shuffled per article, so a preference for a name is impossible and a preference for a slot is spread across the arms instead of following one. Arms from a judge's own vendor family are flagged: gemma_l rated them +0.18 against the others; qwen_s rated them +0.30 against the others, a raw gap, not a bias estimate, since families differ in real quality.
Measure the judge's own noise and position bias.
A second shuffle and a repeat on a subset. Here the reshuffle moved per-article scores by 0.11–0.21 and a repeat by 0.03–0.12. Those are per-article numbers; whether a gap between two arms' means is real is what the paired tests answer, run on the judged scores like any other.
Keep every verdict.
Each reply is stored with its scores, so the pass resumes, costs $0 to re-read, and anyone can check what the judge said. A reply that is not bare JSON scores, which can carry the source's text, is kept only as its hash.
Read some yourself before you trust the ranking.
Agreement between two judges is not agreement with people. Reading 20 pairs is what produced the disagreement here; the primary metric changes only after a check like that agrees, and a check worth that weight needs more than one reader.
Re-measure what you already have.
A judge scorer is one file: two or more judges, the task they score (what an item is, what the outputs are, the dimensions, the scale), the prompt, and the runs that validated each judge against people. The harness refuses one that is not validated, freezes a blinded judging set, stores every verdict and writes scored runs the leaderboard and the paired tests read unchanged. It is optional: nothing else in the harness needs it.
# price it: builds the judging set, calls nothing $ evh judge-score --dataset-id my_v1 --scorer my-judge --dry-run # judge, capped and resumable; refused until validated $ evh judge-score --dataset-id my_v1 --scorer my-judge # rank and test on the judged scores $ EVAL_RUNS_DIR=data/runs-judged/my-judge evh leaderboard --dataset-id my_v1