eval-harness

goal

A scorer that agrees with experts on summaries (SummEval)

Find a summary scorer whose ranking of summaries agrees with expert judges, so it can stand in for them.

01what it is for

decision
Which scorers (overlap metrics, learned metrics, LLM judges) agree with experts, and how close they come to one expert agreeing with the others.
audience
anyone choosing a metric or an LLM judge to score summaries
decides
tau, higher is better
weighs
quality (tau); cost per item; speed per item
strategy
frugal: contenders: anything not significantly different from the leader by the paired test; the cheaper wins
tie-break
cost, speed
output contract
scores
status
active
guardrails
  • check against people: Before relying on any judge, check it against a small set of human judgements on your own outputs. (research/REPORT_SUMMEVAL.md §4)
  • provider: No provider was pinned: gemma_l scored 0.0113 apart and billed 3.7× apart on two providers, one run each. (research/REPORT_SUMMEVAL.md §3.3)
links

02the answer so far

summeval_100 · baseline promoted 2026-10-25 · scored against gold

se_gemma_l_n100_v1

gemma_l (Gemma 4 31B, open weights, $0.0278 for 100 articles) is the cheapest judge the leader openai_l is not significantly different from. Check any judge against human judgements on your own outputs first. REPORT_SUMMEVAL, executive summary.

deploy sheet: what production must match
measured by
se_gemma_l_n100_v1_20261018T094137Z on summeval_100
against
gold/summeval_100
model sent
eval-gemma-31b
model record
gemma4-31b
deployment
hosted · homelab LiteLLM (the fleet's gateway) · eval-gemma-31b
server
not recorded
build
none recorded (a hosted model has no digest)
settings sent
max_tokens: 1500
model: eval-gemma-31b
prompt_file: prompt.txt
provider: litellm
reasoning: {"enabled": false}
temperature: 0.0
not sent
bookkeeping: family, price_tier, usd_per_mtok_in, usd_per_mtok_out
the record sets reasoning_effort: none; this arm did not send it
declared
settings.prompt_sha256: 6c02d00929048d9c3fea511af40c3ddf90fdf1550afb7713d3f5f4d86e44f5ee
settings.reasoning: {'enabled': False}
settings.temperature: 0.0
deployment.alias: eval-gemma-31b
deployment.endpoint: http://homelab:4001
deployment.id: openrouter/google/gemma-4-31b-it
deployment.id_source: proxy-model-info
deployment.provider_routing: None
reported per item
finish_reason: stop
provider: CoreWeave, DeepInfra
response_model: eval-gemma-31b
adapter
examples/summeval-judging/adapter.py b757ca07e338
harness
90b88fd29bd7

No capture here says what production ran (the adapter's production_ran).

Command: evh deploy-sheet --config se_gemma_l_n100_v1 [--capture C]

inspect it

03how we got here: rounds

Rounds are optional: each one sets out to learn something, changes a few arms, drops what lost and says why, and ends in a conclusion. Decisions are made with evh round-open, round-retire and round-close; evh rounds says the same as this page. Open a round for what changed, what was retired and why, and what to do next.

r2 closed
from r1
Pinned to CoreWeave on UniEval's text, gemma_l scores 0.5397 against its 0.5384 (+0.0013, not significant); routed to SambaNova it scored 0.5510 and billed 3.7× as much: consistent with a provider effect, one run per provider. (REPORT_SUMMEVAL §3.3.)
Does UniEval's text move the judge, and does the provider?
0 arms (2 retired)

2 retired

arms

  • retired 2: a paired side run, not on the leaderboard; data/runs is the record
    se_gemma_l_pubtext_n100_v1, se_gemma_l_pubtextcw_n100_v1
r1 closed
Scored against two experts, the best judges reach 0.5195, 93% of one expert's agreement with the other two (0.5587); every free scorer is far behind (UniEval 64%, BARTScore 44%, ROUGE-1 26%). Use an LLM judge, and start with gemma_l ($0.0278 for 100 articles), the cheapest judge the leader openai_l is not significantly different from. (REPORT_SUMMEVAL, executive summary.)
Which of 24 hosted LLM judges and nine free scorers agrees best with SummEval's expert scores, and at what cost?
33 arms inspect
best se_openai_l_n100_v1 0.540

04summeval_100 · test

summeval_100: 100 items · test

all 100 annotated articles

references: gold/summeval_100

Mean tau over each arm's items, in groups by what each arm is scored against: numbers from different references, or from a judge, are different measurements and are never ranked together. A mean is not a verdict: the inspector tests each arm against its group's leader (Holm across them). Flagged arms are never the leader. filed: the arm's config names this goal (goal:); an arm without it is listed because it records the measure.

scored against gold/summeval_100 gold

Test every arm in this group against its leader

Recommended: se_gemma_l_n100_v1

  • Strategy frugal. Quality leader: se_openai_l_n100_v1 (tau 0.5398). Contenders beside it (not significantly different by the paired test): se_gemma_l_n100_v1, se_anthropic_l_n100_v1, se_openai_m_n100_v1, se_qwen37max_n100_v1, se_qwen_s_n100_v1.
  • By cost (cost_usd): se_gemma_l_n100_v1 $0.000278, se_qwen_s_n100_v1 $0.000385, se_qwen37max_n100_v1 $0.004436, se_openai_l_n100_v1 $0.006350, se_openai_m_n100_v1 $0.014153, se_anthropic_l_n100_v1 $0.020077.
  • Recommended: se_gemma_l_n100_v1.

Strategy frugal (contenders: anything not significantly different from the leader by the paired test; the cheaper wins). Weighs quality (tau); cost per item; speed per item. Among the contenders on quality, the tie-break is cost, speed. The rule recommends; promoting is evh run-promote; the goal chooses its strategy (strategy:).

armmean tauitemscost per itemspeed per itemflags
se_gemma_l_pubtext_n100_v1 · pair filed retired in r20.551100$0.001013 · 2,235 tokp50 2,417 · p95 4,134 ms—test vs leader
se_openai_l_n100_v1 filed0.540100$0.006350 · 2,106 tokp50 2,205 · p95 2,643 ms—leader
se_gemma_l_pubtextcw_n100_v1 · pair filed retired in r20.540100$0.000276 · 2,241 tokp50 2,639 · p95 6,340 ms—test vs leader
se_gemma_l_n100_v1 filed baseline recommended0.538100$0.000278 · 2,251 tokp50 6,204 · p95 10,768 ms—test vs leader
se_anthropic_l_n100_v1 filed0.528100$0.020077 · 3,151 tokp50 3,378 · p95 3,975 ms—test vs leader
se_openai_m_n100_v1 filed0.528100$0.014153 · 2,106 tokp50 2,667 · p95 3,133 ms—test vs leader
se_qwen37max_n100_v1 filed0.520100$0.004436 · 2,261 tokp50 4,416 · p95 5,240 ms—test vs leader
se_qwen_s_n100_v1 filed0.519100$0.000385 · 2,297 tokp50 3,672 · p95 4,710 ms—test vs leader
se_glm_m_n100_v1 filed0.517100$0.001144 · 2,198 tokp50 3,360 · p95 4,908 ms—test vs leader
se_qwen_m_n100_v1 filed0.507100$0.001084 · 2,386 tokp50 7,054 · p95 7,789 ms—test vs leader
se_glm_l_n100_v1 filed0.492100$0.002451 · 2,201 tokp50 3,130 · p95 6,118 ms—test vs leader
se_deepseek_m_n100_v1 filed0.486100$0.000771 · 2,213 tokp50 877 · p95 2,130 ms—test vs leader
se_anthropic_s_n100_v1 filed0.484100$0.003237 · 2,333 tokp50 2,315 · p95 3,192 ms—test vs leader
se_anthropic_m_n100_v1 filed0.480100$0.008048 · 3,153 tokp50 2,966 · p95 3,359 ms—test vs leader
se_gemma_m_n100_v1 filed0.477100$0.000252 · 2,247 tokp50 4,583 · p95 6,532 ms—test vs leader
se_deepseek_l_n100_v1 filed0.469100$0.000258 · 2,220 tokp50 5,028 · p95 6,637 ms—test vs leader
se_mistral_m_n100_v1 filed0.468100$0.001191 · 2,293 tokp50 1,836 · p95 2,678 ms—test vs leader
se_deepseek_s_n100_v1 filed0.459100$0.000307 · 2,217 tokp50 2,535 · p95 9,703 ms—test vs leader
se_mistral_l_n100_v1 filed0.455100$0.001257 · 2,281 tokp50 3,072 · p95 50,068 ms—test vs leader
se_llama_l_n100_v1 filed0.454100$0.000594 · 2,250 tokp50 11,823 · p95 29,054 ms—test vs leader
se_glm_s_n100_v1 filed0.446100$0.000481 · 2,282 tokp50 5,649 · p95 7,549 ms—test vs leader
se_mistral_s_n100_v1 filed0.427100$0.000283 · 2,435 tokp50 8,940 · p95 14,768 ms—test vs leader
se_gemma_s_n100_v1 filed0.420100$0.000279 · 2,329 tokp50 9,191 · p95 25,300 ms—test vs leader
se_openai_s_n100_v1 filed0.411100$0.002124 · 2,106 tokp50 1,373 · p95 1,834 ms—test vs leader
se_llama_m_n100_v1 filed0.406100$0.000517 · 2,209 tokp50 2,971 · p95 13,790 ms—test vs leader
se_unievalpub_n100_v1 filed0.372100$0.000000p50 47,259 · p95 63,387 ms—test vs leader
se_llama_s_n100_v1 filed0.353100$0.000297 · 2,313 tokp50 6,512 · p95 9,588 ms—test vs leader
se_unieval_n100_v1 filed0.348100$0.000000p50 42,677 · p95 60,322 ms—test vs leader
se_bartscore_n100_v1 filed0.256100$0.000000p50 16,061 · p95 23,628 ms—test vs leader
se_rouge2_n100_v1 filed0.158100$0.000000p50 174 · p95 239 ms—test vs leader
se_rouge1_n100_v1 filed0.149100$0.000000p50 177 · p95 281 ms—test vs leader
se_rougeL_n100_v1 filed0.143100$0.000000p50 258 · p95 434 ms—test vs leader
se_coverage_n100_v1 filed0.101100$0.000000p50 877 · p95 1,128 ms—test vs leader
se_length_n100_v1 filed0.039100$0.000000p50 0.18 · p95 0.3 ms—test vs leader
se_random_n100_v1 filed0.003100$0.000000p50 0.12 · p95 0.21 ms—test vs leader
$0.0000$0.0201cost (USD per item)0.540.0025se_openai_l_n100_v1: tau 0.540, cost 0.006se_gemma_l_n100_v1: tau 0.538, cost 0.000se_anthropic_l_n100_v1: tau 0.528, cost 0.020se_openai_m_n100_v1: tau 0.528, cost 0.014se_qwen37max_n100_v1: tau 0.520, cost 0.004se_qwen_s_n100_v1: tau 0.519, cost 0.000se_glm_m_n100_v1: tau 0.517, cost 0.001se_qwen_m_n100_v1: tau 0.507, cost 0.001se_glm_l_n100_v1: tau 0.492, cost 0.002se_deepseek_m_n100_v1: tau 0.486, cost 0.001se_anthropic_s_n100_v1: tau 0.484, cost 0.003se_anthropic_m_n100_v1: tau 0.480, cost 0.008se_gemma_m_n100_v1: tau 0.477, cost 0.000se_deepseek_l_n100_v1: tau 0.469, cost 0.000se_mistral_m_n100_v1: tau 0.468, cost 0.001se_deepseek_s_n100_v1: tau 0.459, cost 0.000se_mistral_l_n100_v1: tau 0.455, cost 0.001se_llama_l_n100_v1: tau 0.454, cost 0.001se_glm_s_n100_v1: tau 0.446, cost 0.000se_mistral_s_n100_v1: tau 0.427, cost 0.000se_gemma_s_n100_v1: tau 0.420, cost 0.000se_openai_s_n100_v1: tau 0.411, cost 0.002se_llama_m_n100_v1: tau 0.406, cost 0.001se_unievalpub_n100_v1: tau 0.372, cost 0.000se_llama_s_n100_v1: tau 0.353, cost 0.000se_unieval_n100_v1: tau 0.348, cost 0.000se_bartscore_n100_v1: tau 0.256, cost 0.000se_rouge2_n100_v1: tau 0.158, cost 0.000se_rouge1_n100_v1: tau 0.149, cost 0.000se_rougeL_n100_v1: tau 0.143, cost 0.000se_coverage_n100_v1: tau 0.101, cost 0.000se_length_n100_v1: tau 0.039, cost 0.000se_random_n100_v1: tau 0.003, cost 0.000se_openai_l_n100_v1se_gemma_l_n100_v1se_gemma_m_n100_v1se_unievalpub_n100_v1
tau against cost (USD per item): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.20763386.6speed p95 (ms)0.540.0025se_openai_l_n100_v1: tau 0.540, speed 2643.2se_gemma_l_n100_v1: tau 0.538, speed 10768.4se_anthropic_l_n100_v1: tau 0.528, speed 3974.7se_openai_m_n100_v1: tau 0.528, speed 3133.2se_qwen37max_n100_v1: tau 0.520, speed 5239.9se_qwen_s_n100_v1: tau 0.519, speed 4709.6se_glm_m_n100_v1: tau 0.517, speed 4908.4se_qwen_m_n100_v1: tau 0.507, speed 7789.2se_glm_l_n100_v1: tau 0.492, speed 6117.8se_deepseek_m_n100_v1: tau 0.486, speed 2129.5se_anthropic_s_n100_v1: tau 0.484, speed 3192.5se_anthropic_m_n100_v1: tau 0.480, speed 3358.9se_gemma_m_n100_v1: tau 0.477, speed 6531.6se_deepseek_l_n100_v1: tau 0.469, speed 6636.8se_mistral_m_n100_v1: tau 0.468, speed 2678.4se_deepseek_s_n100_v1: tau 0.459, speed 9703.0se_mistral_l_n100_v1: tau 0.455, speed 50068.3se_llama_l_n100_v1: tau 0.454, speed 29053.7se_glm_s_n100_v1: tau 0.446, speed 7549.2se_mistral_s_n100_v1: tau 0.427, speed 14767.9se_gemma_s_n100_v1: tau 0.420, speed 25299.9se_openai_s_n100_v1: tau 0.411, speed 1833.8se_llama_m_n100_v1: tau 0.406, speed 13790.4se_unievalpub_n100_v1: tau 0.372, speed 63386.6se_llama_s_n100_v1: tau 0.353, speed 9588.0se_unieval_n100_v1: tau 0.348, speed 60322.4se_bartscore_n100_v1: tau 0.256, speed 23628.2se_rouge2_n100_v1: tau 0.158, speed 238.9se_rouge1_n100_v1: tau 0.149, speed 281.4se_rougeL_n100_v1: tau 0.143, speed 433.9se_coverage_n100_v1: tau 0.101, speed 1127.5se_length_n100_v1: tau 0.039, speed 0.300se_random_n100_v1: tau 0.003, speed 0.207se_openai_l_n100_v1se_deepseek_m_n100_v1se_openai_s_n100_v1se_rouge2_n100_v1se_length_n100_v1se_random_n100_v1
tau against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.