dataset
summeval_100
100 items · 41 arms. Headline: each arm's own primary metric, mean over its items; arms with different primary metrics are not ranked against each other.
baseline, promoted 2026-10-25
gemma_l (Gemma 4 31B, open weights, $0.0278 for 100 articles) is the cheapest judge the leader openai_l is not significantly different from. Check any judge against human judgements on your own outputs first. REPORT_SUMMEVAL, executive summary.
Inspect the 35 arms on tau, scored against gold/summeval_100Inspect the 3 arms on pair_agreement, scored against gold/summeval_100Inspect the 3 arms on top1, scored against gold/summeval_100Open its sheet
goals: A scorer that agrees with experts on summaries (SummEval)
01arms
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
41 shown
| arm | metric | mean | items | cost per item | speed per item | runs | scored against | |
|---|---|---|---|---|---|---|---|---|
| se_gemma_l_pubtext_n100_v1 · pair run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.551 | 100 | $0.001013 · 2,235 tok | p50 2,417 · p95 4,134 ms | 1 | gold/summeval_100 gold | inspect |
| se_openai_l_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.540 | 100 | $0.006350 · 2,106 tok | p50 2,205 · p95 2,643 ms | 1 | gold/summeval_100 gold | inspect |
| se_gemma_l_pubtextcw_n100_v1 · pair run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.540 | 100 | $0.000276 · 2,241 tok | p50 2,639 · p95 6,340 ms | 1 | gold/summeval_100 gold | inspect |
| se_gemma_l_n100_v1 baseline run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.538 | 100 | $0.000278 · 2,251 tok | p50 6,204 · p95 10,768 ms | 1 | gold/summeval_100 gold | inspect |
| se_anthropic_l_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.528 | 100 | $0.020077 · 3,151 tok | p50 3,378 · p95 3,975 ms | 1 | gold/summeval_100 gold | inspect |
| se_openai_m_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.528 | 100 | $0.014153 · 2,106 tok | p50 2,667 · p95 3,133 ms | 1 | gold/summeval_100 gold | inspect |
| se_qwen37max_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.520 | 100 | $0.004436 · 2,261 tok | p50 4,416 · p95 5,240 ms | 1 | gold/summeval_100 gold | inspect |
| se_qwen_s_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.519 | 100 | $0.000385 · 2,297 tok | p50 3,672 · p95 4,710 ms | 1 | gold/summeval_100 gold | inspect |
| se_glm_m_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.517 | 100 | $0.001144 · 2,198 tok | p50 3,360 · p95 4,908 ms | 1 | gold/summeval_100 gold | inspect |
| se_qwen_m_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.507 | 100 | $0.001084 · 2,386 tok | p50 7,054 · p95 7,789 ms | 1 | gold/summeval_100 gold | inspect |
| se_glm_l_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.492 | 100 | $0.002451 · 2,201 tok | p50 3,130 · p95 6,118 ms | 1 | gold/summeval_100 gold | inspect |
| se_deepseek_m_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.486 | 100 | $0.000771 · 2,213 tok | p50 877 · p95 2,130 ms | 1 | gold/summeval_100 gold | inspect |
| se_anthropic_s_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.484 | 100 | $0.003237 · 2,333 tok | p50 2,315 · p95 3,192 ms | 1 | gold/summeval_100 gold | inspect |
| se_anthropic_m_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.480 | 100 | $0.008048 · 3,153 tok | p50 2,966 · p95 3,359 ms | 1 | gold/summeval_100 gold | inspect |
| se_gemma_m_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.477 | 100 | $0.000252 · 2,247 tok | p50 4,583 · p95 6,532 ms | 1 | gold/summeval_100 gold | inspect |
| se_deepseek_l_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.469 | 100 | $0.000258 · 2,220 tok | p50 5,028 · p95 6,637 ms | 1 | gold/summeval_100 gold | inspect |
| se_mistral_m_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.468 | 100 | $0.001191 · 2,293 tok | p50 1,836 · p95 2,678 ms | 1 | gold/summeval_100 gold | inspect |
| se_deepseek_s_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.459 | 100 | $0.000307 · 2,217 tok | p50 2,535 · p95 9,703 ms | 1 | gold/summeval_100 gold | inspect |
| se_mistral_l_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.455 | 100 | $0.001257 · 2,281 tok | p50 3,072 · p95 50,068 ms | 1 | gold/summeval_100 gold | inspect |
| se_llama_l_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.454 | 100 | $0.000594 · 2,250 tok | p50 11,823 · p95 29,054 ms | 1 | gold/summeval_100 gold | inspect |
| se_glm_s_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.446 | 100 | $0.000481 · 2,282 tok | p50 5,649 · p95 7,549 ms | 1 | gold/summeval_100 gold | inspect |
| se_mistral_s_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.427 | 100 | $0.000283 · 2,435 tok | p50 8,940 · p95 14,768 ms | 1 | gold/summeval_100 gold | inspect |
| se_gemma_s_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.420 | 100 | $0.000279 · 2,329 tok | p50 9,191 · p95 25,300 ms | 1 | gold/summeval_100 gold | inspect |
| se_openai_s_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.411 | 100 | $0.002124 · 2,106 tok | p50 1,373 · p95 1,834 ms | 1 | gold/summeval_100 gold | inspect |
| se_llama_m_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.406 | 100 | $0.000517 · 2,209 tok | p50 2,971 · p95 13,790 ms | 1 | gold/summeval_100 gold | inspect |
| se_unievalpub_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.372 | 100 | $0.000000 | p50 47,259 · p95 63,387 ms | 1 | gold/summeval_100 gold | inspect |
| se_llama_s_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.353 | 100 | $0.000297 · 2,313 tok | p50 6,512 · p95 9,588 ms | 1 | gold/summeval_100 gold | inspect |
| se_unieval_n100_v1 run on omarchy · Linux/x86_64, 32 CPUs | tau | 0.348 | 100 | $0.000000 | p50 42,677 · p95 60,322 ms | 1 | gold/summeval_100 gold | inspect |
| se_bartscore_n100_v1 run on Linux/x86_64, 4 CPUs | tau | 0.256 | 100 | $0.000000 | p50 16,061 · p95 23,628 ms | 1 | gold/summeval_100 gold | inspect |
| se_rouge2_n100_v1 run on Linux/x86_64, 4 CPUs | tau | 0.158 | 100 | $0.000000 | p50 174 · p95 239 ms | 1 | gold/summeval_100 gold | inspect |
| se_rouge1_n100_v1 run on Linux/x86_64, 4 CPUs | tau | 0.149 | 100 | $0.000000 | p50 177 · p95 281 ms | 1 | gold/summeval_100 gold | inspect |
| se_rougeL_n100_v1 run on Linux/x86_64, 4 CPUs | tau | 0.143 | 100 | $0.000000 | p50 258 · p95 434 ms | 1 | gold/summeval_100 gold | inspect |
| se_coverage_n100_v1 run on Linux/x86_64, 4 CPUs | tau | 0.101 | 100 | $0.000000 | p50 877 · p95 1,128 ms | 1 | gold/summeval_100 gold | inspect |
| se_length_n100_v1 run on Linux/x86_64, 4 CPUs | tau | 0.039 | 100 | $0.000000 | p50 0.18 · p95 0.3 ms | 1 | gold/summeval_100 gold | inspect |
| se_random_n100_v1 run on Linux/x86_64, 4 CPUs | tau | 0.003 | 100 | $0.000000 | p50 0.12 · p95 0.21 ms | 1 | gold/summeval_100 gold | inspect |
| se_gemma_l_n100_pairwise_v1 run on mac-dev · Linux/x86_64, 32 CPUs | pair_agreement | 0.825 | 100 | $0.000764 · 7,503 tok | p50 5,979 · p95 10,637 ms | 1 | gold/summeval_100 gold | inspect |
| se_qwen_s_n100_pairwise_v1 run on mac-dev · Linux/x86_64, 32 CPUs | pair_agreement | 0.812 | 100 | $0.000531 · 7,669 tok | p50 11,556 · p95 59,941 ms | 1 | gold/summeval_100 gold | inspect |
| se_random_n100_pairwise_v1 run on Linux/x86_64, 32 CPUs | pair_agreement | 0.473 | 100 | $0.000000 | p50 0.064 · p95 0.12 ms | 1 | gold/summeval_100 gold | inspect |
| se_gemma_l_n100_pick_v1 run on mac-dev · Linux/x86_64, 32 CPUs | top1 | 0.364 | 100 | $0.000209 · 2,063 tok | p50 1,124 · p95 7,045 ms | 1 | gold/summeval_100 gold | inspect |
| se_qwen_s_n100_pick_v1 run on mac-dev · Linux/x86_64, 32 CPUs | top1 | 0.263 | 100 | $0.000282 · 2,090 tok | p50 1,283 · p95 2,276 ms | 1 | gold/summeval_100 gold | inspect |
| se_random_n100_pick_v1 run on Linux/x86_64, 32 CPUs | top1 | 0.070 | 100 | $0.000000 | p50 0.037 · p95 0.063 ms | 1 | gold/summeval_100 gold | inspect |
02what's inside
100 items, json, 1276–2344 words (median 1813, middle half 1620–2000).
Source: mteb/summeval (test split). The text stays there; each item below links to its row.
the gold
gold/summeval_100: a record per item: coherence, consistency, fluency, relevance
100 shown
| item | words | gold | open |
|---|---|---|---|
cnn-test-404 | 1714 | 4 fields | source ↗ · inspect |
cnn-test-476 | 1700 | 4 fields | source ↗ · inspect |
cnn-test-513 | 1581 | 4 fields | source ↗ · inspect |
cnn-test-88c | 1536 | 4 fields | source ↗ · inspect |
cnn-test-a02 | 1362 | 4 fields | source ↗ · inspect |
cnn-test-b1c | 1779 | 4 fields | source ↗ · inspect |
cnn-test-b8b | 1630 | 4 fields | source ↗ · inspect |
cnn-test-bcd | 1320 | 4 fields | source ↗ · inspect |
cnn-test-c05 | 1620 | 4 fields | source ↗ · inspect |
cnn-test-d75 | 1477 | 4 fields | source ↗ · inspect |
cnn-test-e2b | 1833 | 4 fields | source ↗ · inspect |
cnn-test-e49 | 1809 | 4 fields | source ↗ · inspect |
cnn-test-fbb | 1488 | 4 fields | source ↗ · inspect |
dm-test-02c9 | 1611 | 4 fields | source ↗ · inspect |
dm-test-03e2 | 1807 | 4 fields | source ↗ · inspect |
dm-test-051d | 2344 | 4 fields | source ↗ · inspect |
dm-test-0783 | 1541 | 4 fields | source ↗ · inspect |
dm-test-096e | 1629 | 4 fields | source ↗ · inspect |
dm-test-0cd1 | 2124 | 4 fields | source ↗ · inspect |
dm-test-0cdd | 1691 | 4 fields | source ↗ · inspect |
dm-test-0f07 | 1994 | 4 fields | source ↗ · inspect |
dm-test-130f | 1420 | 4 fields | source ↗ · inspect |
dm-test-14c8 | 1984 | 4 fields | source ↗ · inspect |
dm-test-1557 | 1632 | 4 fields | source ↗ · inspect |
dm-test-1747 | 1881 | 4 fields | source ↗ · inspect |
dm-test-1749 | 1993 | 4 fields | source ↗ · inspect |
dm-test-1824 | 1931 | 4 fields | source ↗ · inspect |
dm-test-207d | 2156 | 4 fields | source ↗ · inspect |
dm-test-2377 | 2051 | 4 fields | source ↗ · inspect |
dm-test-23e8 | 1455 | 4 fields | source ↗ · inspect |
dm-test-26e4 | 1636 | 4 fields | source ↗ · inspect |
dm-test-2a48 | 1696 | 4 fields | source ↗ · inspect |
dm-test-2abe | 1604 | 4 fields | source ↗ · inspect |
dm-test-2c37 | 2108 | 4 fields | source ↗ · inspect |
dm-test-2c66 | 2138 | 4 fields | source ↗ · inspect |
dm-test-2cf8 | 1540 | 4 fields | source ↗ · inspect |
dm-test-2cfc | 1859 | 4 fields | source ↗ · inspect |
dm-test-2fec | 1813 | 4 fields | source ↗ · inspect |
dm-test-3111 | 1874 | 4 fields | source ↗ · inspect |
dm-test-3635 | 1840 | 4 fields | source ↗ · inspect |
dm-test-3785 | 1818 | 4 fields | source ↗ · inspect |
dm-test-3ba3 | 1940 | 4 fields | source ↗ · inspect |
dm-test-4001 | 1567 | 4 fields | source ↗ · inspect |
dm-test-4660 | 1342 | 4 fields | source ↗ · inspect |
dm-test-4a59 | 1770 | 4 fields | source ↗ · inspect |
dm-test-4e0d | 1881 | 4 fields | source ↗ · inspect |
dm-test-4f3b | 2045 | 4 fields | source ↗ · inspect |
dm-test-4fb6 | 1987 | 4 fields | source ↗ · inspect |
dm-test-54b8 | 1834 | 4 fields | source ↗ · inspect |
dm-test-5be0 | 2050 | 4 fields | source ↗ · inspect |
dm-test-632c | 1388 | 4 fields | source ↗ · inspect |
dm-test-6372 | 2017 | 4 fields | source ↗ · inspect |
dm-test-6721 | 1989 | 4 fields | source ↗ · inspect |
dm-test-6c13 | 1647 | 4 fields | source ↗ · inspect |
dm-test-6f02 | 1654 | 4 fields | source ↗ · inspect |
dm-test-7cba | 1500 | 4 fields | source ↗ · inspect |
dm-test-8017 | 1549 | 4 fields | source ↗ · inspect |
dm-test-808a | 1759 | 4 fields | source ↗ · inspect |
dm-test-81bd | 1972 | 4 fields | source ↗ · inspect |
dm-test-8764 | 1669 | 4 fields | source ↗ · inspect |
dm-test-8847 | 2092 | 4 fields | source ↗ · inspect |
dm-test-8b51 | 1811 | 4 fields | source ↗ · inspect |
dm-test-8b59 | 1470 | 4 fields | source ↗ · inspect |
dm-test-8bc8 | 1633 | 4 fields | source ↗ · inspect |
dm-test-8e6c | 2250 | 4 fields | source ↗ · inspect |
dm-test-97bb | 2000 | 4 fields | source ↗ · inspect |
dm-test-998c | 1353 | 4 fields | source ↗ · inspect |
dm-test-9afd | 2024 | 4 fields | source ↗ · inspect |
dm-test-9b1a | 2008 | 4 fields | source ↗ · inspect |
dm-test-9b5a | 2012 | 4 fields | source ↗ · inspect |
dm-test-9bea | 2008 | 4 fields | source ↗ · inspect |
dm-test-9d0f | 1684 | 4 fields | source ↗ · inspect |
dm-test-a3b6 | 2059 | 4 fields | source ↗ · inspect |
dm-test-b1f6 | 2069 | 4 fields | source ↗ · inspect |
dm-test-b255 | 2167 | 4 fields | source ↗ · inspect |
dm-test-b592 | 1972 | 4 fields | source ↗ · inspect |
dm-test-b5bc | 1879 | 4 fields | source ↗ · inspect |
dm-test-bd46 | 1431 | 4 fields | source ↗ · inspect |
dm-test-c16c | 1816 | 4 fields | source ↗ · inspect |
dm-test-c50d | 1912 | 4 fields | source ↗ · inspect |
dm-test-c666 | 1747 | 4 fields | source ↗ · inspect |
dm-test-cf4b | 1646 | 4 fields | source ↗ · inspect |
dm-test-d296 | 2106 | 4 fields | source ↗ · inspect |
dm-test-d3c9 | 1784 | 4 fields | source ↗ · inspect |
dm-test-d680 | 1996 | 4 fields | source ↗ · inspect |
dm-test-d83b | 1955 | 4 fields | source ↗ · inspect |
dm-test-d89d | 1718 | 4 fields | source ↗ · inspect |
dm-test-dea0 | 2322 | 4 fields | source ↗ · inspect |
dm-test-e06f | 1586 | 4 fields | source ↗ · inspect |
dm-test-e428 | 1493 | 4 fields | source ↗ · inspect |
dm-test-e43e | 1493 | 4 fields | source ↗ · inspect |
dm-test-e470 | 2222 | 4 fields | source ↗ · inspect |
dm-test-e7d8 | 2082 | 4 fields | source ↗ · inspect |
dm-test-e880 | 2006 | 4 fields | source ↗ · inspect |
dm-test-eeef | 1321 | 4 fields | source ↗ · inspect |
dm-test-f26d | 1879 | 4 fields | source ↗ · inspect |
dm-test-f37f | 2107 | 4 fields | source ↗ · inspect |
dm-test-f468 | 1647 | 4 fields | source ↗ · inspect |
dm-test-f5fe | 1276 | 4 fields | source ↗ · inspect |
dm-test-fada | 2062 | 4 fields | source ↗ · inspect |
Each row is the profile evh dataset-items --dataset-id summeval_100 prints; inspect opens the item in the inspector with the dataset's largest group of comparable arms, and its text where the inspector runs on a machine that has it.
03reference sets
- gold/summeval_100 gold · promoted from person the dataset's authors
04labels
No labels yet. A sheet to fill: evh label-sheet --dataset-id summeval_100 --out sheet.csv; a run's outputs as a model set: evh label-from-run --run R --set S.