On mean τ, 23 of 24 LLM judges score above a trained free evaluator.
Grade machine summaries the way experts do. The best LLM judges reach 93% of a human expert's agreement (best of 24, same articles), far ahead of the free evaluator trained for the task (64%), which scores one summary at a time while the judges rank all 16 at once; and an open-weight judge is not significantly different from the leader.
The report's pick, as run here. Use an LLM judge, and start with gemma_l; before relying on any judge, check it against a small set of human judgements on your own outputs.
The goal's rule, in the catalog. A scorer that agrees with experts on summaries (SummEval) (frugal, on tau) recommends gemma_l, the promoted baseline. The goal page, with its reasons → An agent gets the same answer from the MCP server: the tool goals with goal: summary_judge_summeval at https://evals.humancopilot.eu/mcp (how to connect).
Every system, one axis.
What the money bought.
What this eval found.
On mean τ, 23 of 24 LLM judges score above a free evaluator trained for the task.
The best free scorer, unievalpub (UniEval, on its own inputs), scores 0.3717; the weakest judge, llama_s, 0.3533. In the paired test UniEval is significantly below 23 of the 24 judges and not significantly different from the weakest, on either set of inputs. The judges rank all 16 summaries in one call while UniEval scores one at a time, and a pointwise LLM judge was not run.
An open-weight judge is not significantly different from the leader.
gemma_l scores 0.5384 against openai_l's 0.5398, and the leader is not significantly different from it or from 4 others. No provider was pinned: one judge, run once on each of two providers, scored 0.0113 apart and billed 3.7× apart.
The dearest model finished 3rd of 33.
anthropic_l billed $2.01 for 100 items. The cheapest model not significantly different from the leader, gemma_l, cost 72× less.
The best judges come close to a single expert, and stay below one.
Scored against the mean of two experts, as one expert is against the other two, the best judge reaches 0.5195 — 93% of one expert's 0.5587 (95% interval 89–97%), and the gap excludes zero. It is the best of 24, chosen on the same articles (92% when chosen on half and scored on the other half). UniEval reaches 64%, BARTScore 44%; a no-model lookup of which system wrote each summary reaches 63%. The summaries are 2019-era and reasoning was off.
The order is stable; the winner is not.
Two disjoint 20-article halves agree at ρ = 0.90 on the order of the 24 judges, but name the same unique winner only 21% of the time.
This repository's own metric agrees with experts less than ROUGE does.
coverage agrees with the experts at τ 0.1011, below ROUGE-1's 0.1487.
These judges, put to work.
Two of these judges, gemma_l (τ 0.54) and qwen_s (0.52), re-measured the summarisation eval's summaries, where its own metric agrees with the experts at τ 0.10. They re-ranked the field; one reader's blind check of 20 pairs went the other way, not significantly. On crowd ratings of 2022 LLM summaries (Zhang et al.) the same judges track people better than one rater does on coherence and relevance, trail on faithfulness, and do not rank the systems the same way; on the summarisation eval's own outputs neither is confirmed.
Silver is not gold.
The best of the 24 judges agrees with the experts at τ 0.5398 — better than any free scorer measured, and, against two experts, 93% of one expert's agreement with the other two. An LLM judge is a silver reference. This eval is what checking one against a gold set looks like.
This habit, in the method →Check it yourself.
First, with nothing installed: test every arm against openai_l, item by item, in the catalog's inspector, with the same paired test and Holm correction as this page; anthropic_l, the most expensive model, among them. The goal page has the rest: what the goal recommends, and why.
The 9 free arms rerun on a laptop CPU for $0, pinned to the weights that were measured. The 24 hosted arms billed $7.03 when measured and rerun with one OpenRouter key.
make compare writes a replication card: each rerun against its committed run, item by item, and every significant difference and tie on this page re-tested with your reruns in place.
$ make plan EXP=summeval $ make data EXP=summeval $ make reproduce EXP=summeval LEVEL=local # with OPENROUTER_API_KEY set $ make reproduce EXP=summeval LEVEL=hosted $ make compare EXP=summeval