eval-harness

goal

Short news summaries (CNN)

Pick a model that writes a short summary of a CNN-style news article, at a known cost, for a reader who wants the story's main points.

01what it is for

decision
Which models can be told apart on summary quality, and what each costs per article.
audience
a reader of news who wants the story's main points in a few sentences
decides
coverage, higher is better
weighs
quality (coverage); cost per item; speed per item; rouge1
strategy
balanced: contenders: not significantly different from the leader by the paired test AND within the quality margin (default)
tie-break
cost, speed
output contract
paragraph · about the reference highlights' length (coverage clips longer summaries) · en
status
active
guardrails
  • the metric: coverage agrees with expert judgement poorly (τ 0.1011 on SummEval, below ROUGE-1's 0.1487), and judges validated on SummEval rank every hosted LLM above every BART. Read section 3.8 before choosing. (research/REPORT_SUMMARIZATION.md, executive summary and §3.8)
  • the text: bart_l's result is specific to CNN-style text. (research/REPORT_SUMMARIZATION.md, executive summary)
links

02the answer so far

cnn_dailymail_200 · baseline promoted 2026-10-25 · scored against gold

cnn_bart_l_n200_v1

bart_l (BART-large-CNN, MIT, $0 on a CPU) leads on coverage at 0.3461, not significantly different from deepseek_m (0.3460, p = 0.99), for CNN-style text; rent deepseek_m if you need speed. REPORT_SUMMARIZATION, executive summary.

deploy sheet: what production must match
measured by
cnn_bart_l_n200_v1_20260926T153757Z on cnn_dailymail_200
against
gold/cnn_dailymail_200
model sent
facebook/bart-large-cnn
model record
facebook-bart-large-cnn
server
not recorded
build
37f520fa929c961707657b28798b30c003dd100b
settings sent
device: cpu
early_stopping: true
length_penalty: 2.0
max_length: 142
min_length: 56
model: facebook/bart-large-cnn
no_repeat_ngram_size: 3
num_beams: 4
precision: fp32
provider: hf_local
not sent
bookkeeping: family, price_tier
declared
deployment.weight_files: ['model.safetensors:1625222120']
reported per item
encoder_window: 1024
adapter
examples/summarization-cnn-dailymail/adapter.py 51a7b06b1a2e
harness
fb7366a0a694

No capture here says what production ran (the adapter's production_ran).

Command: evh deploy-sheet --config cnn_bart_l_n200_v1 [--capture C]

inspect it

cnn_dailymail_20

No decision yet. When the tests separate an arm worth keeping, promote its run with the reason they give: evh run-promote --run <run_id> --reason "…"

cnn_dailymail_dm200

No decision yet. When the tests separate an arm worth keeping, promote its run with the reason they give: evh run-promote --run <run_id> --reason "…"

03how we got here: rounds

Rounds are optional: each one sets out to learn something, changes a few arms, drops what lost and says why, and ends in a conclusion. Decisions are made with evh round-open, round-retire and round-close; evh rounds says the same as this page. Open a round for what changed, what was retired and why, and what to do next.

r3 closed
from r2
bart_s re-ran 200 of 200 outputs byte-identical on the same machine (three times, counting the resumed run). (REPORT_SUMMARIZATION.)
Does a local arm's output change when it is run again on the same machine?
0 arms (1 retired)

1 retired

arms

r2 closed
from r1
All five local arms reproduced 200 of 200 outputs byte-identically (bart_m, bart_s and bart_l_xsum on the same Linux machine; bart_l and lead3 on a different one): their run-to-run variance is zero. (REPORT_SUMMARIZATION, the local contrast.)
Do the local arms reproduce, and how fast are they, re-run on one Linux machine?
0 arms (5 retired)

5 retired

arms

r1 closed
On coverage, bart_l (BART-large-CNN, $0, 11.0 s per article on a 12-core Mac CPU) scores 0.3461 and the best hosted arm, deepseek_m ($191 per 1M articles, 1.6 s), 0.3460: not significant (p = 0.99). Use bart_l if you can run a free 1.6 GB model on CPU and your text looks like CNN news; rent deepseek_m if you need speed; do not buy anthropic_l. (REPORT_SUMMARIZATION, executive summary.)
Which of 24 hosted LLMs (8 vendors × 3 price tiers) and local BART summarisers covers a CNN article's gold highlights best, and at what cost?
29 arms inspect
best cnn_bart_l_n200_v1 0.346

04cnn_dailymail_200 · test

cnn_dailymail_200: 200 items · test

first 200 test rows

references: gold/cnn_dailymail_200, silver/cnn_dailymail_200 (written by cnn_anthropic_l_n200_v1)

Mean coverage over each arm's items, in groups by what each arm is scored against: numbers from different references, or from a judge, are different measurements and are never ranked together. A mean is not a verdict: the inspector tests each arm against its group's leader (Holm across them). Flagged arms are never the leader. filed: the arm's config names this goal (goal:); an arm without it is listed because it records the measure.

scored against gold/cnn_dailymail_200 gold

Test every arm in this group against its leader

Recommended: cnn_bart_l_n200_v1

  • Strategy balanced. Quality leader: cnn_bart_l_n200_v1 (coverage 0.3461). Contenders beside it (not significantly different by the paired test and within 0.006922 (2% of the leader)): cnn_deepseek_m_n200_v1. Not significantly different but further off, so not contenders: cnn_llama_l_n200_v1, cnn_llama_m_n200_v1, cnn_bart_m_n200_v1, cnn_llama_s_n200_v1, cnn_deepseek_l_n200_v1, cnn_glm_m_n200_v1, cnn_deepseek_s_n200_v1, cnn_bart_s_n200_v1, cnn_openai_m_n200_v1, cnn_openai_l_n200_v1, cnn_mistral_s_n200_v1, cnn_anthropic_s_n200_v1, cnn_anthropic_m_n200_v1, cnn_gemma_l_n200_v1.
  • By cost (cost_usd): cnn_bart_l_n200_v1 $0.000000, cnn_deepseek_m_n200_v1 $0.000191.
  • Recommended: cnn_bart_l_n200_v1.

Strategy balanced (contenders: not significantly different from the leader by the paired test AND within the quality margin (default)). Weighs quality (coverage); cost per item; speed per item; rouge1. Among the contenders on quality, the tie-break is cost, speed. The rule recommends; promoting is evh run-promote; the goal chooses its strategy (strategy:).

armmean coverageitemscost per itemspeed per itemrouge1flags
cnn_bart_l_n200_v1 filed baseline recommended0.346200$0.000000 · 763 tokp50 10,724 · p95 14,371 ms0.3477—leader
cnn_bart_l_n200_v1 · linux filed retired in r20.346200$0.000000 · 763 tokp50 6,257 · p95 9,188 ms0.3477—test vs leader
cnn_deepseek_m_n200_v1 filed0.346200$0.000191 · 812 tokp50 1,497 · p95 2,924 ms0.3612—test vs leader
cnn_llama_l_n200_v1 filed0.339200$0.000195 · 805 tokp50 3,646 · p95 5,531 ms0.3636—test vs leader
cnn_llama_m_n200_v1 filed0.338200$0.000165 · 809 tokp50 2,682 · p95 11,240 ms0.3477—test vs leader
cnn_bart_m_n200_v1 filed0.337200$0.000000 · 766 tokp50 4,649 · p95 7,223 ms0.3457—test vs leader
cnn_bart_m_n200_v1 · linux filed retired in r20.337200$0.000000 · 766 tokp50 4,108 · p95 6,463 ms0.3457—test vs leader
cnn_llama_s_n200_v1 filed0.333200$0.000111 · 814 tokp50 2,204 · p95 4,041 ms0.3526—test vs leader
cnn_deepseek_l_n200_v1 filed0.332200$0.000694 · 802 tokp50 2,354 · p95 4,380 ms0.3547—test vs leader
cnn_glm_m_n200_v1 filed0.332200$0.000497 · 803 tokp50 1,848 · p95 7,145 ms0.3517—test vs leader
cnn_deepseek_s_n200_v1 filed0.328200$0.000069 · 800 tokp50 2,237 · p95 5,644 ms0.3504—test vs leader
cnn_bart_s_n200_v1 filed0.328200$0.000000 · 761 tokp50 3,683 · p95 5,055 ms0.3418—test vs leader
cnn_bart_s_n200_v1 · linux filed retired in r20.328200$0.000000 · 761 tokp50 3,387 · p95 4,550 ms0.3418—test vs leader
cnn_bart_s_n200_v1 · repeats filed retired in r30.328200$0.000000 · 761 tokp50 3,708 · p95 4,822 ms0.3418—test vs leader
cnn_openai_m_n200_v1 filed0.327200$0.005358 · 786 tokp50 1,719 · p95 2,126 ms0.3599—test vs leader
cnn_openai_l_n200_v1 filed0.327200$0.002181 · 786 tokp50 1,735 · p95 2,381 ms0.362—test vs leader
cnn_mistral_s_n200_v1 filed0.326200$0.000089 · 833 tokp50 2,573 · p95 5,142 ms0.3285—test vs leader
cnn_anthropic_s_n200_v1 filed0.324200$0.001305 · 897 tokp50 1,982 · p95 2,834 ms0.3259—test vs leader
cnn_anthropic_m_n200_v1 filed0.324200$0.003739 · 1,282 tokp50 2,992 · p95 3,675 ms0.3347—test vs leader
cnn_gemma_l_n200_v1 filed0.320200$0.000119 · 810 tokp50 1,817 · p95 6,172 ms0.3511—test vs leader
cnn_openai_s_n200_v1 filed0.317200$0.000836 · 793 tokp50 1,238 · p95 1,573 ms0.3506—test vs leader
cnn_anthropic_l_n200_v1 filed0.315200$0.009826 · 1,301 tokp50 3,259 · p95 4,718 ms0.3289—test vs leader
cnn_glm_s_n200_v1 filed0.314200$0.000152 · 786 tokp50 1,580 · p95 2,341 ms0.3489—test vs leader
cnn_gemma_m_n200_v1 filed0.314200$0.000077 · 812 tokp50 1,388 · p95 4,483 ms0.3442—test vs leader
cnn_glm_l_n200_v1 filed0.312200$0.000766 · 818 tokp50 2,351 · p95 12,758 ms0.3334—test vs leader
cnn_gemma_s_n200_v1 filed0.312200$0.000100 · 820 tokp50 5,933 · p95 12,255 ms0.3215—test vs leader
cnn_qwen_l_n200_v1 filed0.311200$0.000844 · 828 tokp50 2,732 · p95 4,428 ms0.3309—test vs leader
cnn_mistral_l_n200_v1 filed0.308200$0.000486 · 828 tokp50 2,272 · p95 51,619 ms0.3188—test vs leader
cnn_qwen_s_n200_v1 filed0.307200$0.000135 · 825 tokp50 7,855 · p95 108,807 ms0.336—test vs leader
cnn_mistral_m_n200_v1 filed0.304200$0.000435 · 832 tokp50 2,495 · p95 4,801 ms0.3197—test vs leader
cnn_qwen_m_n200_v1 filed0.303200$0.000331 · 826 tokp50 1,531 · p95 1,911 ms0.3164—test vs leader
cnn_lead3_n200_v1 filed0.272200$0.000000p50 0.057 · p95 0.13 ms0.302—test vs leader
cnn_lead3_n200_v1 · linux filed retired in r20.272200$0.000000p50 0.063 · p95 0.15 ms0.302—test vs leader
cnn_bart_l_xsum_n200_v1 filed0.207200$0.000000 · 725 tokp50 4,433 · p95 7,325 ms0.2836—test vs leader
cnn_bart_l_xsum_n200_v1 · linux filed retired in r20.207200$0.000000 · 725 tokp50 3,934 · p95 7,054 ms0.2836—test vs leader
$0.0000$0.0098cost (USD per item)0.3460.207cnn_bart_l_n200_v1: coverage 0.346, cost 0.000cnn_deepseek_m_n200_v1: coverage 0.346, cost 0.000cnn_llama_l_n200_v1: coverage 0.339, cost 0.000cnn_llama_m_n200_v1: coverage 0.338, cost 0.000cnn_bart_m_n200_v1: coverage 0.337, cost 0.000cnn_llama_s_n200_v1: coverage 0.333, cost 0.000cnn_deepseek_l_n200_v1: coverage 0.332, cost 0.001cnn_glm_m_n200_v1: coverage 0.332, cost 0.000cnn_deepseek_s_n200_v1: coverage 0.328, cost 0.000cnn_bart_s_n200_v1: coverage 0.328, cost 0.000cnn_openai_m_n200_v1: coverage 0.327, cost 0.005cnn_openai_l_n200_v1: coverage 0.327, cost 0.002cnn_mistral_s_n200_v1: coverage 0.326, cost 0.000cnn_anthropic_s_n200_v1: coverage 0.324, cost 0.001cnn_anthropic_m_n200_v1: coverage 0.324, cost 0.004cnn_gemma_l_n200_v1: coverage 0.320, cost 0.000cnn_openai_s_n200_v1: coverage 0.317, cost 0.001cnn_anthropic_l_n200_v1: coverage 0.315, cost 0.010cnn_glm_s_n200_v1: coverage 0.314, cost 0.000cnn_gemma_m_n200_v1: coverage 0.314, cost 0.000cnn_glm_l_n200_v1: coverage 0.312, cost 0.001cnn_gemma_s_n200_v1: coverage 0.312, cost 0.000cnn_qwen_l_n200_v1: coverage 0.311, cost 0.001cnn_mistral_l_n200_v1: coverage 0.308, cost 0.000cnn_qwen_s_n200_v1: coverage 0.307, cost 0.000cnn_mistral_m_n200_v1: coverage 0.304, cost 0.000cnn_qwen_m_n200_v1: coverage 0.303, cost 0.000cnn_lead3_n200_v1: coverage 0.272, cost 0.000cnn_bart_l_xsum_n200_v1: coverage 0.207, cost 0.000cnn_bart_l_n200_v1
coverage against cost (USD per item): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.129108807.3speed p95 (ms)0.3460.207cnn_bart_l_n200_v1: coverage 0.346, speed 14370.8cnn_deepseek_m_n200_v1: coverage 0.346, speed 2923.6cnn_llama_l_n200_v1: coverage 0.339, speed 5530.8cnn_llama_m_n200_v1: coverage 0.338, speed 11239.9cnn_bart_m_n200_v1: coverage 0.337, speed 7223.5cnn_llama_s_n200_v1: coverage 0.333, speed 4040.6cnn_deepseek_l_n200_v1: coverage 0.332, speed 4380.2cnn_glm_m_n200_v1: coverage 0.332, speed 7145.5cnn_deepseek_s_n200_v1: coverage 0.328, speed 5643.6cnn_bart_s_n200_v1: coverage 0.328, speed 5054.7cnn_openai_m_n200_v1: coverage 0.327, speed 2126.3cnn_openai_l_n200_v1: coverage 0.327, speed 2380.7cnn_mistral_s_n200_v1: coverage 0.326, speed 5142.2cnn_anthropic_s_n200_v1: coverage 0.324, speed 2833.6cnn_anthropic_m_n200_v1: coverage 0.324, speed 3674.6cnn_gemma_l_n200_v1: coverage 0.320, speed 6171.8cnn_openai_s_n200_v1: coverage 0.317, speed 1573.0cnn_anthropic_l_n200_v1: coverage 0.315, speed 4717.9cnn_glm_s_n200_v1: coverage 0.314, speed 2341.2cnn_gemma_m_n200_v1: coverage 0.314, speed 4482.6cnn_glm_l_n200_v1: coverage 0.312, speed 12757.6cnn_gemma_s_n200_v1: coverage 0.312, speed 12255.1cnn_qwen_l_n200_v1: coverage 0.311, speed 4428.1cnn_mistral_l_n200_v1: coverage 0.308, speed 51618.8cnn_qwen_s_n200_v1: coverage 0.307, speed 108807.3cnn_mistral_m_n200_v1: coverage 0.304, speed 4800.7cnn_qwen_m_n200_v1: coverage 0.303, speed 1910.8cnn_lead3_n200_v1: coverage 0.272, speed 0.129cnn_bart_l_xsum_n200_v1: coverage 0.207, speed 7325.1cnn_bart_l_n200_v1cnn_deepseek_m_n200_v1cnn_openai_m_n200_v1cnn_openai_s_n200_v1cnn_lead3_n200_v1
coverage against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.2840.364rouge10.3460.207cnn_bart_l_n200_v1: coverage 0.346, rouge1 0.348cnn_deepseek_m_n200_v1: coverage 0.346, rouge1 0.361cnn_llama_l_n200_v1: coverage 0.339, rouge1 0.364cnn_llama_m_n200_v1: coverage 0.338, rouge1 0.348cnn_bart_m_n200_v1: coverage 0.337, rouge1 0.346cnn_llama_s_n200_v1: coverage 0.333, rouge1 0.353cnn_deepseek_l_n200_v1: coverage 0.332, rouge1 0.355cnn_glm_m_n200_v1: coverage 0.332, rouge1 0.352cnn_deepseek_s_n200_v1: coverage 0.328, rouge1 0.350cnn_bart_s_n200_v1: coverage 0.328, rouge1 0.342cnn_openai_m_n200_v1: coverage 0.327, rouge1 0.360cnn_openai_l_n200_v1: coverage 0.327, rouge1 0.362cnn_mistral_s_n200_v1: coverage 0.326, rouge1 0.329cnn_anthropic_s_n200_v1: coverage 0.324, rouge1 0.326cnn_anthropic_m_n200_v1: coverage 0.324, rouge1 0.335cnn_gemma_l_n200_v1: coverage 0.320, rouge1 0.351cnn_openai_s_n200_v1: coverage 0.317, rouge1 0.351cnn_anthropic_l_n200_v1: coverage 0.315, rouge1 0.329cnn_glm_s_n200_v1: coverage 0.314, rouge1 0.349cnn_gemma_m_n200_v1: coverage 0.314, rouge1 0.344cnn_glm_l_n200_v1: coverage 0.312, rouge1 0.333cnn_gemma_s_n200_v1: coverage 0.312, rouge1 0.322cnn_qwen_l_n200_v1: coverage 0.311, rouge1 0.331cnn_mistral_l_n200_v1: coverage 0.308, rouge1 0.319cnn_qwen_s_n200_v1: coverage 0.307, rouge1 0.336cnn_mistral_m_n200_v1: coverage 0.304, rouge1 0.320cnn_qwen_m_n200_v1: coverage 0.303, rouge1 0.316cnn_lead3_n200_v1: coverage 0.272, rouge1 0.302cnn_bart_l_xsum_n200_v1: coverage 0.207, rouge1 0.284cnn_bart_l_n200_v1cnn_deepseek_m_n200_v1cnn_llama_l_n200_v1
coverage against rouge1: better is right top. Named and joined: the frontier, the arms no other arm beats on both.

scored against silver/cnn_dailymail_200 silver

Test every arm in this group against its leader

Shown, not recommended

  • Strategy balanced. Quality leader: cnn_anthropic_m_n200_v1 · silver (coverage 0.4913). Contenders beside it (not significantly different by the paired test and within 0.009825 (2% of the leader)): none.
  • Scored against silver, beside gold/cnn_dailymail_200: gold decides on cnn_dailymail_200; this group is shown, not recommended. Silver is not what the goal decides on: a model's opinion (its author's family ranks higher on it) or what production recorded afterwards; its caveat says which.
armmean coverageitemscost per itemspeed per itemrouge1flags
cnn_anthropic_m_n200_v1 · silver filed0.491200$0.003739 · 1,282 tokp50 2,992 · p95 3,675 ms0.5945—leader
cnn_deepseek_m_n200_v1 · silver filed0.471200$0.000191 · 812 tokp50 1,497 · p95 2,924 ms0.5904—test vs leader
cnn_anthropic_s_n200_v1 · silver filed0.469200$0.001305 · 897 tokp50 1,982 · p95 2,834 ms0.5616—test vs leader
cnn_deepseek_l_n200_v1 · silver filed0.399200$0.000694 · 802 tokp50 2,354 · p95 4,380 ms0.5284—test vs leader
cnn_qwen_l_n200_v1 · silver filed0.398200$0.000844 · 828 tokp50 2,732 · p95 4,428 ms0.5167—test vs leader
cnn_glm_m_n200_v1 · silver filed0.389200$0.000497 · 803 tokp50 1,848 · p95 7,145 ms0.5148—test vs leader
cnn_llama_s_n200_v1 · silver filed0.388200$0.000111 · 814 tokp50 2,204 · p95 4,041 ms0.5003—test vs leader
cnn_deepseek_s_n200_v1 · silver filed0.388200$0.000069 · 800 tokp50 2,237 · p95 5,644 ms0.5281—test vs leader
cnn_openai_s_n200_v1 · silver filed0.384200$0.000836 · 793 tokp50 1,238 · p95 1,573 ms0.5211—test vs leader
cnn_mistral_s_n200_v1 · silver filed0.382200$0.000089 · 833 tokp50 2,573 · p95 5,142 ms0.4993—test vs leader
cnn_qwen_m_n200_v1 · silver filed0.381200$0.000331 · 826 tokp50 1,531 · p95 1,911 ms0.4967—test vs leader
cnn_glm_l_n200_v1 · silver filed0.375200$0.000766 · 818 tokp50 2,351 · p95 12,758 ms0.4875—test vs leader
cnn_openai_l_n200_v1 · silver filed0.371200$0.002181 · 786 tokp50 1,735 · p95 2,381 ms0.5248—test vs leader
cnn_llama_l_n200_v1 · silver filed0.370200$0.000195 · 805 tokp50 3,646 · p95 5,531 ms0.4993—test vs leader
cnn_openai_m_n200_v1 · silver filed0.359200$0.005358 · 786 tokp50 1,719 · p95 2,126 ms0.5065—test vs leader
cnn_llama_m_n200_v1 · silver filed0.358200$0.000165 · 809 tokp50 2,682 · p95 11,240 ms0.4819—test vs leader
cnn_gemma_s_n200_v1 · silver filed0.350200$0.000100 · 820 tokp50 5,933 · p95 12,255 ms0.4701—test vs leader
cnn_gemma_m_n200_v1 · silver filed0.342200$0.000077 · 812 tokp50 1,388 · p95 4,483 ms0.4823—test vs leader
cnn_gemma_l_n200_v1 · silver filed0.342200$0.000119 · 810 tokp50 1,817 · p95 6,172 ms0.4816—test vs leader
cnn_mistral_l_n200_v1 · silver filed0.337200$0.000486 · 828 tokp50 2,272 · p95 51,619 ms0.4679—test vs leader
cnn_qwen_s_n200_v1 · silver filed0.334200$0.000135 · 825 tokp50 7,855 · p95 108,807 ms0.4709—test vs leader
cnn_mistral_m_n200_v1 · silver filed0.320200$0.000435 · 832 tokp50 2,495 · p95 4,801 ms0.4535—test vs leader
cnn_bart_m_n200_v1 · silver filed0.320200$0.000000 · 766 tokp50 4,649 · p95 7,223 ms0.4218—test vs leader
cnn_bart_l_n200_v1 · silver filed0.317200$0.000000 · 763 tokp50 10,724 · p95 14,371 ms0.4207—test vs leader
cnn_bart_s_n200_v1 · silver filed0.306200$0.000000 · 761 tokp50 3,683 · p95 5,055 ms0.4138—test vs leader
cnn_glm_s_n200_v1 · silver filed0.303200$0.000152 · 786 tokp50 1,580 · p95 2,341 ms0.4443—test vs leader
cnn_lead3_n200_v1 · silver filed0.300200$0.000000p50 0.057 · p95 0.13 ms0.3882—test vs leader
cnn_bart_l_xsum_n200_v1 · silver filed0.137200$0.000000 · 725 tokp50 4,433 · p95 7,325 ms0.2553—test vs leader
cnn_anthropic_l_n200_v1 · silver filed1.000200$0.009826 · 1,301 tokp50 3,259 · p95 4,718 ms1.00wrote the reference it is scored againsttest vs leader
$0.0000$0.0054cost (USD per item)0.4910.137cnn_anthropic_m_n200_v1 · silver: coverage 0.491, cost 0.004cnn_deepseek_m_n200_v1 · silver: coverage 0.471, cost 0.000cnn_anthropic_s_n200_v1 · silver: coverage 0.469, cost 0.001cnn_deepseek_l_n200_v1 · silver: coverage 0.399, cost 0.001cnn_qwen_l_n200_v1 · silver: coverage 0.398, cost 0.001cnn_glm_m_n200_v1 · silver: coverage 0.389, cost 0.000cnn_llama_s_n200_v1 · silver: coverage 0.388, cost 0.000cnn_deepseek_s_n200_v1 · silver: coverage 0.388, cost 0.000cnn_openai_s_n200_v1 · silver: coverage 0.384, cost 0.001cnn_mistral_s_n200_v1 · silver: coverage 0.382, cost 0.000cnn_qwen_m_n200_v1 · silver: coverage 0.381, cost 0.000cnn_glm_l_n200_v1 · silver: coverage 0.375, cost 0.001cnn_openai_l_n200_v1 · silver: coverage 0.371, cost 0.002cnn_llama_l_n200_v1 · silver: coverage 0.370, cost 0.000cnn_openai_m_n200_v1 · silver: coverage 0.359, cost 0.005cnn_llama_m_n200_v1 · silver: coverage 0.358, cost 0.000cnn_gemma_s_n200_v1 · silver: coverage 0.350, cost 0.000cnn_gemma_m_n200_v1 · silver: coverage 0.342, cost 0.000cnn_gemma_l_n200_v1 · silver: coverage 0.342, cost 0.000cnn_mistral_l_n200_v1 · silver: coverage 0.337, cost 0.000cnn_qwen_s_n200_v1 · silver: coverage 0.334, cost 0.000cnn_mistral_m_n200_v1 · silver: coverage 0.320, cost 0.000cnn_bart_m_n200_v1 · silver: coverage 0.320, cost 0.000cnn_bart_l_n200_v1 · silver: coverage 0.317, cost 0.000cnn_bart_s_n200_v1 · silver: coverage 0.306, cost 0.000cnn_glm_s_n200_v1 · silver: coverage 0.303, cost 0.000cnn_lead3_n200_v1 · silver: coverage 0.300, cost 0.000cnn_bart_l_xsum_n200_v1 · silver: coverage 0.137, cost 0.000cnn_anthropic_m_n200_v1 · silvercnn_deepseek_m_n200_v1 · silvercnn_llama_s_n200_v1 · silvercnn_deepseek_s_n200_v1 · silvercnn_bart_m_n200_v1 · silver
coverage against cost (USD per item): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.129108807.3speed p95 (ms)0.4910.137cnn_anthropic_m_n200_v1 · silver: coverage 0.491, speed 3674.6cnn_deepseek_m_n200_v1 · silver: coverage 0.471, speed 2923.6cnn_anthropic_s_n200_v1 · silver: coverage 0.469, speed 2833.6cnn_deepseek_l_n200_v1 · silver: coverage 0.399, speed 4380.2cnn_qwen_l_n200_v1 · silver: coverage 0.398, speed 4428.1cnn_glm_m_n200_v1 · silver: coverage 0.389, speed 7145.5cnn_llama_s_n200_v1 · silver: coverage 0.388, speed 4040.6cnn_deepseek_s_n200_v1 · silver: coverage 0.388, speed 5643.6cnn_openai_s_n200_v1 · silver: coverage 0.384, speed 1573.0cnn_mistral_s_n200_v1 · silver: coverage 0.382, speed 5142.2cnn_qwen_m_n200_v1 · silver: coverage 0.381, speed 1910.8cnn_glm_l_n200_v1 · silver: coverage 0.375, speed 12757.6cnn_openai_l_n200_v1 · silver: coverage 0.371, speed 2380.7cnn_llama_l_n200_v1 · silver: coverage 0.370, speed 5530.8cnn_openai_m_n200_v1 · silver: coverage 0.359, speed 2126.3cnn_llama_m_n200_v1 · silver: coverage 0.358, speed 11239.9cnn_gemma_s_n200_v1 · silver: coverage 0.350, speed 12255.1cnn_gemma_m_n200_v1 · silver: coverage 0.342, speed 4482.6cnn_gemma_l_n200_v1 · silver: coverage 0.342, speed 6171.8cnn_mistral_l_n200_v1 · silver: coverage 0.337, speed 51618.8cnn_qwen_s_n200_v1 · silver: coverage 0.334, speed 108807.3cnn_mistral_m_n200_v1 · silver: coverage 0.320, speed 4800.7cnn_bart_m_n200_v1 · silver: coverage 0.320, speed 7223.5cnn_bart_l_n200_v1 · silver: coverage 0.317, speed 14370.8cnn_bart_s_n200_v1 · silver: coverage 0.306, speed 5054.7cnn_glm_s_n200_v1 · silver: coverage 0.303, speed 2341.2cnn_lead3_n200_v1 · silver: coverage 0.300, speed 0.129cnn_bart_l_xsum_n200_v1 · silver: coverage 0.137, speed 7325.1cnn_anthropic_m_n200_v1 · silvercnn_deepseek_m_n200_v1 · silvercnn_anthropic_s_n200_v1 · silvercnn_openai_s_n200_v1 · silvercnn_lead3_n200_v1 · silver
coverage against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.2550.595rouge10.4910.137cnn_anthropic_m_n200_v1 · silver: coverage 0.491, rouge1 0.595cnn_deepseek_m_n200_v1 · silver: coverage 0.471, rouge1 0.590cnn_anthropic_s_n200_v1 · silver: coverage 0.469, rouge1 0.562cnn_deepseek_l_n200_v1 · silver: coverage 0.399, rouge1 0.528cnn_qwen_l_n200_v1 · silver: coverage 0.398, rouge1 0.517cnn_glm_m_n200_v1 · silver: coverage 0.389, rouge1 0.515cnn_llama_s_n200_v1 · silver: coverage 0.388, rouge1 0.500cnn_deepseek_s_n200_v1 · silver: coverage 0.388, rouge1 0.528cnn_openai_s_n200_v1 · silver: coverage 0.384, rouge1 0.521cnn_mistral_s_n200_v1 · silver: coverage 0.382, rouge1 0.499cnn_qwen_m_n200_v1 · silver: coverage 0.381, rouge1 0.497cnn_glm_l_n200_v1 · silver: coverage 0.375, rouge1 0.487cnn_openai_l_n200_v1 · silver: coverage 0.371, rouge1 0.525cnn_llama_l_n200_v1 · silver: coverage 0.370, rouge1 0.499cnn_openai_m_n200_v1 · silver: coverage 0.359, rouge1 0.507cnn_llama_m_n200_v1 · silver: coverage 0.358, rouge1 0.482cnn_gemma_s_n200_v1 · silver: coverage 0.350, rouge1 0.470cnn_gemma_m_n200_v1 · silver: coverage 0.342, rouge1 0.482cnn_gemma_l_n200_v1 · silver: coverage 0.342, rouge1 0.482cnn_mistral_l_n200_v1 · silver: coverage 0.337, rouge1 0.468cnn_qwen_s_n200_v1 · silver: coverage 0.334, rouge1 0.471cnn_mistral_m_n200_v1 · silver: coverage 0.320, rouge1 0.453cnn_bart_m_n200_v1 · silver: coverage 0.320, rouge1 0.422cnn_bart_l_n200_v1 · silver: coverage 0.317, rouge1 0.421cnn_bart_s_n200_v1 · silver: coverage 0.306, rouge1 0.414cnn_glm_s_n200_v1 · silver: coverage 0.303, rouge1 0.444cnn_lead3_n200_v1 · silver: coverage 0.300, rouge1 0.388cnn_bart_l_xsum_n200_v1 · silver: coverage 0.137, rouge1 0.255cnn_anthropic_m_n200_v1 · silver
coverage against rouge1: better is right top. Named and joined: the frontier, the arms no other arm beats on both.

scored by judge judge-summ-pick-v1

Test every arm in this group against its leader

Shown, not recommended

  • Strategy balanced. Quality leader: cnn_anthropic_l_n200_v1 · judged/judge-summ-pick-v1 (judge_won 0.4019). Contenders beside it (not significantly different by the paired test and within 0.008039 (2% of the leader)): none.
  • Read on judge_won: the goal decides on coverage; this group is shown, not recommended.
armmean judge_wonitemscost per itemspeed per itemflags
cnn_anthropic_l_n200_v1 · judged/judge-summ-pick-v1 filed0.402198$0.009783 · 1,293 tokp50 3,259 · p95 4,722 ms—leader
cnn_anthropic_s_n200_v1 · judged/judge-summ-pick-v1 filed0.226198$0.001299 · 892 tokp50 1,980 · p95 2,836 ms—test vs leader
cnn_anthropic_m_n200_v1 · judged/judge-summ-pick-v1 filed0.144198$0.003724 · 1,275 tokp50 2,992 · p95 3,677 ms—test vs leader
cnn_deepseek_m_n200_v1 · judged/judge-summ-pick-v1 filed0.083198$0.000189 · 807 tokp50 1,497 · p95 2,927 ms—test vs leader
cnn_qwen_l_n200_v1 · judged/judge-summ-pick-v1 filed0.059198$0.000840 · 823 tokp50 2,732 · p95 4,435 ms—test vs leader
cnn_qwen_m_n200_v1 · judged/judge-summ-pick-v1 filed0.020198$0.000329 · 821 tokp50 1,528 · p95 1,912 ms—test vs leader
cnn_deepseek_l_n200_v1 · judged/judge-summ-pick-v1 filed0.014198$0.000699 · 797 tokp50 2,348 · p95 4,384 ms—test vs leader
cnn_openai_s_n200_v1 · judged/judge-summ-pick-v1 filed0.008198$0.000833 · 788 tokp50 1,238 · p95 1,583 ms—test vs leader
cnn_glm_l_n200_v1 · judged/judge-summ-pick-v1 filed0.006198$0.000765 · 813 tokp50 2,351 · p95 12,765 ms—test vs leader
cnn_gemma_m_n200_v1 · judged/judge-summ-pick-v1 filed0.005198$0.000076 · 807 tokp50 1,388 · p95 4,485 ms—test vs leader
cnn_glm_m_n200_v1 · judged/judge-summ-pick-v1 filed0.005198$0.000496 · 798 tokp50 1,848 · p95 7,151 ms—test vs leader
cnn_openai_l_n200_v1 · judged/judge-summ-pick-v1 filed0.004198$0.002169 · 781 tokp50 1,731 · p95 2,385 ms—test vs leader
cnn_mistral_s_n200_v1 · judged/judge-summ-pick-v1 filed0.004198$0.000089 · 828 tokp50 2,573 · p95 5,085 ms—test vs leader
cnn_deepseek_s_n200_v1 · judged/judge-summ-pick-v1 filed0.004198$0.000068 · 795 tokp50 2,227 · p95 5,599 ms—test vs leader
cnn_openai_m_n200_v1 · judged/judge-summ-pick-v1 filed0.004198$0.005335 · 781 tokp50 1,719 · p95 2,130 ms—test vs leader
cnn_llama_l_n200_v1 · judged/judge-summ-pick-v1 filed0.003198$0.000194 · 800 tokp50 3,646 · p95 5,538 ms—test vs leader
cnn_mistral_l_n200_v1 · judged/judge-summ-pick-v1 filed0.003198$0.000484 · 823 tokp50 2,236 · p95 51,435 ms—test vs leader
cnn_gemma_s_n200_v1 · judged/judge-summ-pick-v1 filed0.002198$0.000099 · 816 tokp50 5,887 · p95 12,319 ms—test vs leader
cnn_llama_s_n200_v1 · judged/judge-summ-pick-v1 filed0.001198$0.000111 · 809 tokp50 2,191 · p95 4,013 ms—test vs leader
cnn_qwen_s_n200_v1 · judged/judge-summ-pick-v1 filed0.001198$0.000134 · 820 tokp50 7,855 · p95 109,634 ms—test vs leader
cnn_glm_s_n200_v1 · judged/judge-summ-pick-v1 filed0.001198$0.000151 · 781 tokp50 1,580 · p95 2,342 ms—test vs leader
cnn_gemma_l_n200_v1 · judged/judge-summ-pick-v1 filed0.001198$0.000118 · 805 tokp50 1,817 · p95 6,291 ms—test vs leader
cnn_llama_m_n200_v1 · judged/judge-summ-pick-v1 filed0.001198$0.000163 · 804 tokp50 2,682 · p95 11,251 ms—test vs leader
cnn_bart_l_n200_v1 · judged/judge-summ-pick-v1 filed0.000198$0.000000 · 759 tokp50 10,690 · p95 14,372 ms—test vs leader
cnn_bart_l_xsum_n200_v1 · judged/judge-summ-pick-v1 filed0.000198$0.000000 · 721 tokp50 4,392 · p95 7,327 ms—test vs leader
cnn_bart_m_n200_v1 · judged/judge-summ-pick-v1 filed0.000198$0.000000 · 761 tokp50 4,623 · p95 7,234 ms—test vs leader
cnn_bart_s_n200_v1 · judged/judge-summ-pick-v1 filed0.000198$0.000000 · 757 tokp50 3,675 · p95 5,057 ms—test vs leader
cnn_lead3_n200_v1 · judged/judge-summ-pick-v1 filed0.000198$0.000000p50 0.057 · p95 0.13 ms—test vs leader
cnn_mistral_m_n200_v1 · judged/judge-summ-pick-v1 filed0.000198$0.000433 · 827 tokp50 2,482 · p95 4,807 ms—test vs leader
$0.0000$0.0098cost (USD per item)0.4020cnn_anthropic_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.402, cost 0.010cnn_anthropic_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.226, cost 0.001cnn_anthropic_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.144, cost 0.004cnn_deepseek_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.083, cost 0.000cnn_qwen_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.059, cost 0.001cnn_qwen_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.020, cost 0.000cnn_deepseek_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.014, cost 0.001cnn_openai_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.008, cost 0.001cnn_glm_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.006, cost 0.001cnn_gemma_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.005, cost 0.000cnn_glm_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.005, cost 0.000cnn_openai_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.004, cost 0.002cnn_mistral_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.004, cost 0.000cnn_deepseek_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.004, cost 0.000cnn_openai_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.004, cost 0.005cnn_llama_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.003, cost 0.000cnn_mistral_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.003, cost 0.000cnn_gemma_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.002, cost 0.000cnn_llama_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.001, cost 0.000cnn_qwen_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.001, cost 0.000cnn_glm_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.001, cost 0.000cnn_gemma_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.001, cost 0.000cnn_llama_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.001, cost 0.000cnn_bart_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.000, cost 0.000cnn_bart_l_xsum_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.000, cost 0.000cnn_bart_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.000, cost 0.000cnn_bart_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.000, cost 0.000cnn_lead3_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.000, cost 0.000cnn_mistral_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.000, cost 0.000cnn_anthropic_l_n200_v1 · judged/judge-summ-pick-v1cnn_anthropic_s_n200_v1 · judged/judge-summ-pick-v1cnn_deepseek_m_n200_v1 · judged/judge-summ-pick-v1cnn_gemma_m_n200_v1 · judged/judge-summ-pick-v1cnn_deepseek_s_n200_v1 · judged/judge-summ-pick-v1cnn_bart_l_n200_v1 · judged/judge-summ-pick-v1cnn_bart_l_xsum_n200_v1 · judged/judge-summ-pick-v1cnn_bart_m_n200_v1 · judged/judge-summ-pick-v1cnn_bart_s_n200_v1 · judged/judge-summ-pick-v1cnn_lead3_n200_v1 · judged/judge-summ-pick-v1
judge_won against cost (USD per item): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.129109634.3speed p95 (ms)0.4020cnn_anthropic_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.402, speed 4722.0cnn_anthropic_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.226, speed 2836.3cnn_anthropic_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.144, speed 3677.0cnn_deepseek_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.083, speed 2927.1cnn_qwen_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.059, speed 4435.0cnn_qwen_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.020, speed 1912.4cnn_deepseek_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.014, speed 4384.1cnn_openai_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.008, speed 1582.8cnn_glm_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.006, speed 12765.2cnn_gemma_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.005, speed 4485.5cnn_glm_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.005, speed 7150.7cnn_openai_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.004, speed 2385.5cnn_mistral_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.004, speed 5085.1cnn_deepseek_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.004, speed 5598.8cnn_openai_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.004, speed 2130.3cnn_llama_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.003, speed 5537.8cnn_mistral_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.003, speed 51434.7cnn_gemma_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.002, speed 12318.7cnn_llama_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.001, speed 4012.7cnn_qwen_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.001, speed 109634.3cnn_glm_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.001, speed 2342.4cnn_gemma_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.001, speed 6291.4cnn_llama_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.001, speed 11251.1cnn_bart_l_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.000, speed 14371.8cnn_bart_l_xsum_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.000, speed 7326.7cnn_bart_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.000, speed 7234.1cnn_bart_s_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.000, speed 5056.8cnn_lead3_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.000, speed 0.129cnn_mistral_m_n200_v1 · judged/judge-summ-pick-v1: judge_won 0.000, speed 4806.8cnn_anthropic_l_n200_v1 · judged/judge-summ-pick-v1cnn_anthropic_s_n200_v1 · judged/judge-summ-pick-v1cnn_qwen_m_n200_v1 · judged/judge-summ-pick-v1cnn_openai_s_n200_v1 · judged/judge-summ-pick-v1cnn_lead3_n200_v1 · judged/judge-summ-pick-v1
judge_won against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.

scored by judge judge-summ-pointwise-v1

Test every arm in this group against its leader

Shown, not recommended

  • Strategy balanced. Quality leader: cnn_deepseek_m_n200_v1 · judged/judge-summ-pointwise-v1 (judge_score_unvalidated 4.94). Contenders beside it (not significantly different by the paired test and within 0.0988 (2% of the leader)): none.
  • judge_score_unvalidated: judges never compared with people for this use, a trial rather than a measurement; this group is shown, not recommended.
armmean judge_score_unvalidateditemscost per itemspeed per itemflags
cnn_deepseek_m_n200_v1 · judged/judge-summ-pointwise-v1 filed4.940200$0.000191 · 812 tokp50 1,497 · p95 2,924 ms—leader
cnn_bart_l_n200_v1 · judged/judge-summ-pointwise-v1 filed4.181200$0.000000 · 763 tokp50 10,724 · p95 14,371 ms—test vs leader
$0.0000$0.0002cost (USD per item)4.944.181cnn_deepseek_m_n200_v1 · judged/judge-summ-pointwise-v1: judge_score_unvalidated 4.940, cost 0.000cnn_bart_l_n200_v1 · judged/judge-summ-pointwise-v1: judge_score_unvalidated 4.181, cost 0.000cnn_deepseek_m_n200_v1 · judged/judge-summ-pointwise-v1cnn_bart_l_n200_v1 · judged/judge-summ-pointwise-v1
judge_score_unvalidated against cost (USD per item): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
2923.614370.8speed p95 (ms)4.944.181cnn_deepseek_m_n200_v1 · judged/judge-summ-pointwise-v1: judge_score_unvalidated 4.940, speed 2923.6cnn_bart_l_n200_v1 · judged/judge-summ-pointwise-v1: judge_score_unvalidated 4.181, speed 14370.8cnn_deepseek_m_n200_v1 · judged/judge-summ-pointwise-v1
judge_score_unvalidated against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.

scored by judge judge-summ-v1

Test every arm in this group against its leader

Shown, not recommended

  • Strategy balanced. Quality leader: cnn_anthropic_l_n200_v1 · judged/judge-summ-v1 (judge_score 4.98). Contenders beside it (not significantly different by the paired test and within 0.09952 (2% of the leader)): cnn_anthropic_m_n200_v1 · judged/judge-summ-v1, cnn_anthropic_s_n200_v1 · judged/judge-summ-v1.
  • By cost (cost_usd): cnn_anthropic_s_n200_v1 · judged/judge-summ-v1 $0.001301, cnn_anthropic_m_n200_v1 · judged/judge-summ-v1 $0.003730, cnn_anthropic_l_n200_v1 · judged/judge-summ-v1 $0.009811.
  • Read on judge_score: the goal decides on coverage; this group is shown, not recommended.
armmean judge_scoreitemscost per itemspeed per itemflags
cnn_anthropic_l_n200_v1 · judged/judge-summ-v1 filed4.976197$0.009811 · 1,298 tokp50 3,258 · p95 4,724 ms—leader
cnn_anthropic_m_n200_v1 · judged/judge-summ-v1 filed4.963197$0.003730 · 1,278 tokp50 2,992 · p95 3,678 ms—test vs leader
cnn_anthropic_s_n200_v1 · judged/judge-summ-v1 filed4.952197$0.001301 · 894 tokp50 1,981 · p95 2,838 ms—test vs leader
cnn_deepseek_m_n200_v1 · judged/judge-summ-v1 filed4.923197$0.000191 · 809 tokp50 1,489 · p95 2,889 ms—test vs leader
cnn_deepseek_l_n200_v1 · judged/judge-summ-v1 filed4.877197$0.000685 · 799 tokp50 2,350 · p95 4,386 ms—test vs leader
cnn_qwen_m_n200_v1 · judged/judge-summ-v1 filed4.876197$0.000330 · 824 tokp50 1,529 · p95 1,913 ms—test vs leader
cnn_glm_m_n200_v1 · judged/judge-summ-v1 filed4.861197$0.000497 · 801 tokp50 1,853 · p95 7,153 ms—test vs leader
cnn_openai_s_n200_v1 · judged/judge-summ-v1 filed4.825197$0.000834 · 791 tokp50 1,238 · p95 1,588 ms—test vs leader
cnn_deepseek_s_n200_v1 · judged/judge-summ-v1 filed4.818197$0.000069 · 797 tokp50 2,247 · p95 5,697 ms—test vs leader
cnn_qwen_l_n200_v1 · judged/judge-summ-v1 filed4.812197$0.000842 · 826 tokp50 2,725 · p95 4,438 ms—test vs leader
cnn_gemma_s_n200_v1 · judged/judge-summ-v1 filed4.801197$0.000100 · 818 tokp50 5,952 · p95 12,351 ms—test vs leader
cnn_mistral_s_n200_v1 · judged/judge-summ-v1 filed4.798197$0.000089 · 831 tokp50 2,575 · p95 5,148 ms—test vs leader
cnn_openai_m_n200_v1 · judged/judge-summ-v1 filed4.795197$0.005346 · 784 tokp50 1,720 · p95 2,132 ms—test vs leader
cnn_openai_l_n200_v1 · judged/judge-summ-v1 filed4.773197$0.002175 · 783 tokp50 1,733 · p95 2,376 ms—test vs leader
cnn_gemma_l_n200_v1 · judged/judge-summ-v1 filed4.715197$0.000119 · 808 tokp50 1,826 · p95 6,351 ms—test vs leader
cnn_mistral_m_n200_v1 · judged/judge-summ-v1 filed4.708197$0.000434 · 830 tokp50 2,496 · p95 4,810 ms—test vs leader
cnn_gemma_m_n200_v1 · judged/judge-summ-v1 filed4.707197$0.000077 · 810 tokp50 1,386 · p95 4,487 ms—test vs leader
cnn_llama_s_n200_v1 · judged/judge-summ-v1 filed4.704197$0.000111 · 812 tokp50 2,195 · p95 4,062 ms—test vs leader
cnn_llama_l_n200_v1 · judged/judge-summ-v1 filed4.700197$0.000193 · 803 tokp50 3,657 · p95 5,541 ms—test vs leader
cnn_mistral_l_n200_v1 · judged/judge-summ-v1 filed4.698197$0.000485 · 826 tokp50 2,240 · p95 51,633 ms—test vs leader
cnn_llama_m_n200_v1 · judged/judge-summ-v1 filed4.674197$0.000165 · 807 tokp50 2,692 · p95 11,257 ms—test vs leader
cnn_qwen_s_n200_v1 · judged/judge-summ-v1 filed4.609197$0.000135 · 823 tokp50 7,347 · p95 110,048 ms—test vs leader
cnn_glm_l_n200_v1 · judged/judge-summ-v1 filed4.607197$0.000763 · 816 tokp50 2,351 · p95 12,769 ms—test vs leader
cnn_glm_s_n200_v1 · judged/judge-summ-v1 filed4.530197$0.000152 · 783 tokp50 1,587 · p95 2,343 ms—test vs leader
cnn_bart_l_n200_v1 · judged/judge-summ-v1 filed3.868197$0.000000 · 761 tokp50 10,700 · p95 14,266 ms—test vs leader
cnn_bart_m_n200_v1 · judged/judge-summ-v1 filed3.564197$0.000000 · 764 tokp50 4,621 · p95 7,239 ms—test vs leader
cnn_bart_l_xsum_n200_v1 · judged/judge-summ-v1 filed3.516197$0.000000 · 723 tokp50 4,368 · p95 7,328 ms—test vs leader
cnn_lead3_n200_v1 · judged/judge-summ-v1 filed3.421197$0.000000p50 0.057 · p95 0.13 ms—test vs leader
cnn_bart_s_n200_v1 · judged/judge-summ-v1 filed3.185197$0.000000 · 759 tokp50 3,672 · p95 5,058 ms—test vs leader
$0.0000$0.0098cost (USD per item)4.9763.185cnn_anthropic_l_n200_v1 · judged/judge-summ-v1: judge_score 4.976, cost 0.010cnn_anthropic_m_n200_v1 · judged/judge-summ-v1: judge_score 4.963, cost 0.004cnn_anthropic_s_n200_v1 · judged/judge-summ-v1: judge_score 4.952, cost 0.001cnn_deepseek_m_n200_v1 · judged/judge-summ-v1: judge_score 4.923, cost 0.000cnn_deepseek_l_n200_v1 · judged/judge-summ-v1: judge_score 4.877, cost 0.001cnn_qwen_m_n200_v1 · judged/judge-summ-v1: judge_score 4.876, cost 0.000cnn_glm_m_n200_v1 · judged/judge-summ-v1: judge_score 4.861, cost 0.000cnn_openai_s_n200_v1 · judged/judge-summ-v1: judge_score 4.825, cost 0.001cnn_deepseek_s_n200_v1 · judged/judge-summ-v1: judge_score 4.818, cost 0.000cnn_qwen_l_n200_v1 · judged/judge-summ-v1: judge_score 4.812, cost 0.001cnn_gemma_s_n200_v1 · judged/judge-summ-v1: judge_score 4.801, cost 0.000cnn_mistral_s_n200_v1 · judged/judge-summ-v1: judge_score 4.798, cost 0.000cnn_openai_m_n200_v1 · judged/judge-summ-v1: judge_score 4.795, cost 0.005cnn_openai_l_n200_v1 · judged/judge-summ-v1: judge_score 4.773, cost 0.002cnn_gemma_l_n200_v1 · judged/judge-summ-v1: judge_score 4.715, cost 0.000cnn_mistral_m_n200_v1 · judged/judge-summ-v1: judge_score 4.708, cost 0.000cnn_gemma_m_n200_v1 · judged/judge-summ-v1: judge_score 4.707, cost 0.000cnn_llama_s_n200_v1 · judged/judge-summ-v1: judge_score 4.704, cost 0.000cnn_llama_l_n200_v1 · judged/judge-summ-v1: judge_score 4.700, cost 0.000cnn_mistral_l_n200_v1 · judged/judge-summ-v1: judge_score 4.698, cost 0.000cnn_llama_m_n200_v1 · judged/judge-summ-v1: judge_score 4.674, cost 0.000cnn_qwen_s_n200_v1 · judged/judge-summ-v1: judge_score 4.609, cost 0.000cnn_glm_l_n200_v1 · judged/judge-summ-v1: judge_score 4.607, cost 0.001cnn_glm_s_n200_v1 · judged/judge-summ-v1: judge_score 4.530, cost 0.000cnn_bart_l_n200_v1 · judged/judge-summ-v1: judge_score 3.868, cost 0.000cnn_bart_m_n200_v1 · judged/judge-summ-v1: judge_score 3.564, cost 0.000cnn_bart_l_xsum_n200_v1 · judged/judge-summ-v1: judge_score 3.516, cost 0.000cnn_lead3_n200_v1 · judged/judge-summ-v1: judge_score 3.421, cost 0.000cnn_bart_s_n200_v1 · judged/judge-summ-v1: judge_score 3.185, cost 0.000cnn_anthropic_l_n200_v1 · judged/judge-summ-v1cnn_anthropic_m_n200_v1 · judged/judge-su…cnn_anthropic_s_n200_v1 · judged/judge-summ-v1cnn_deepseek_m_n200_v1 · judged/judge-summ-v1cnn_deepseek_s_n200_v1 · judged/judge-summ-v1cnn_bart_l_n200_v1 · judged/judge-summ-v1
judge_score against cost (USD per item): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.13110047.9speed p95 (ms)4.9763.185cnn_anthropic_l_n200_v1 · judged/judge-summ-v1: judge_score 4.976, speed 4724.0cnn_anthropic_m_n200_v1 · judged/judge-summ-v1: judge_score 4.963, speed 3678.2cnn_anthropic_s_n200_v1 · judged/judge-summ-v1: judge_score 4.952, speed 2837.6cnn_deepseek_m_n200_v1 · judged/judge-summ-v1: judge_score 4.923, speed 2889.0cnn_deepseek_l_n200_v1 · judged/judge-summ-v1: judge_score 4.877, speed 4386.0cnn_qwen_m_n200_v1 · judged/judge-summ-v1: judge_score 4.876, speed 1913.2cnn_glm_m_n200_v1 · judged/judge-summ-v1: judge_score 4.861, speed 7153.3cnn_openai_s_n200_v1 · judged/judge-summ-v1: judge_score 4.825, speed 1587.7cnn_deepseek_s_n200_v1 · judged/judge-summ-v1: judge_score 4.818, speed 5696.9cnn_qwen_l_n200_v1 · judged/judge-summ-v1: judge_score 4.812, speed 4438.4cnn_gemma_s_n200_v1 · judged/judge-summ-v1: judge_score 4.801, speed 12350.6cnn_mistral_s_n200_v1 · judged/judge-summ-v1: judge_score 4.798, speed 5147.5cnn_openai_m_n200_v1 · judged/judge-summ-v1: judge_score 4.795, speed 2132.4cnn_openai_l_n200_v1 · judged/judge-summ-v1: judge_score 4.773, speed 2375.9cnn_gemma_l_n200_v1 · judged/judge-summ-v1: judge_score 4.715, speed 6351.2cnn_mistral_m_n200_v1 · judged/judge-summ-v1: judge_score 4.708, speed 4809.9cnn_gemma_m_n200_v1 · judged/judge-summ-v1: judge_score 4.707, speed 4486.9cnn_llama_s_n200_v1 · judged/judge-summ-v1: judge_score 4.704, speed 4062.1cnn_llama_l_n200_v1 · judged/judge-summ-v1: judge_score 4.700, speed 5541.3cnn_mistral_l_n200_v1 · judged/judge-summ-v1: judge_score 4.698, speed 51632.8cnn_llama_m_n200_v1 · judged/judge-summ-v1: judge_score 4.674, speed 11256.7cnn_qwen_s_n200_v1 · judged/judge-summ-v1: judge_score 4.609, speed 110047.9cnn_glm_l_n200_v1 · judged/judge-summ-v1: judge_score 4.607, speed 12769.0cnn_glm_s_n200_v1 · judged/judge-summ-v1: judge_score 4.530, speed 2343.0cnn_bart_l_n200_v1 · judged/judge-summ-v1: judge_score 3.868, speed 14265.6cnn_bart_m_n200_v1 · judged/judge-summ-v1: judge_score 3.564, speed 7239.4cnn_bart_l_xsum_n200_v1 · judged/judge-summ-v1: judge_score 3.516, speed 7327.5cnn_lead3_n200_v1 · judged/judge-summ-v1: judge_score 3.421, speed 0.130cnn_bart_s_n200_v1 · judged/judge-summ-v1: judge_score 3.185, speed 5057.8cnn_anthropic_l_n200_v1 · judged/judge-summ-v1cnn_anthropic_m_n200_v1 · judged/judge-summ-v1cnn_anthropic_s_n200_v1 · judged/judge-summ-v1cnn_qwen_m_n200_v1 · judged/judge-summ-v1cnn_openai_s_n200_v1 · judged/judge-summ-v1cnn_lead3_n200_v1 · judged/judge-summ-v1
judge_score against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.

scored by judge judge-summ-v2

Test every arm in this group against its leader

Shown, not recommended

  • Strategy balanced. Quality leader: cnn_anthropic_l_n200_v1 · judged/judge-summ-v2 (judge_score 4.89). Contenders beside it (not significantly different by the paired test and within 0.09776 (2% of the leader)): cnn_deepseek_m_n200_v1 · judged/judge-summ-v2.
  • By cost (cost_usd): cnn_deepseek_m_n200_v1 · judged/judge-summ-v2 $0.000191, cnn_anthropic_l_n200_v1 · judged/judge-summ-v2 $0.009826.
  • Read on judge_score: the goal decides on coverage; this group is shown, not recommended.
armmean judge_scoreitemscost per itemspeed per itemflags
cnn_anthropic_l_n200_v1 · judged/judge-summ-v2 filed4.888200$0.009826 · 1,301 tokp50 3,259 · p95 4,718 ms—leader
cnn_deepseek_m_n200_v1 · judged/judge-summ-v2 filed4.854200$0.000191 · 812 tokp50 1,497 · p95 2,924 ms—test vs leader
cnn_openai_l_n200_v1 · judged/judge-summ-v2 filed4.847200$0.002181 · 786 tokp50 1,735 · p95 2,381 ms—test vs leader
cnn_openai_m_n200_v1 · judged/judge-summ-v2 filed4.811200$0.005358 · 786 tokp50 1,719 · p95 2,126 ms—test vs leader
cnn_anthropic_m_n200_v1 · judged/judge-summ-v2 filed4.809200$0.003739 · 1,282 tokp50 2,992 · p95 3,675 ms—test vs leader
cnn_deepseek_l_n200_v1 · judged/judge-summ-v2 filed4.763200$0.000694 · 802 tokp50 2,354 · p95 4,380 ms—test vs leader
cnn_openai_s_n200_v1 · judged/judge-summ-v2 filed4.747200$0.000836 · 793 tokp50 1,238 · p95 1,573 ms—test vs leader
cnn_anthropic_s_n200_v1 · judged/judge-summ-v2 filed4.728200$0.001305 · 897 tokp50 1,982 · p95 2,834 ms—test vs leader
cnn_glm_m_n200_v1 · judged/judge-summ-v2 filed4.718200$0.000497 · 803 tokp50 1,848 · p95 7,145 ms—test vs leader
cnn_deepseek_s_n200_v1 · judged/judge-summ-v2 filed4.644200$0.000069 · 800 tokp50 2,237 · p95 5,644 ms—test vs leader
cnn_gemma_l_n200_v1 · judged/judge-summ-v2 filed4.603200$0.000119 · 810 tokp50 1,817 · p95 6,172 ms—test vs leader
cnn_qwen_m_n200_v1 · judged/judge-summ-v2 filed4.600200$0.000331 · 826 tokp50 1,531 · p95 1,911 ms—test vs leader
cnn_gemma_m_n200_v1 · judged/judge-summ-v2 filed4.585200$0.000077 · 812 tokp50 1,388 · p95 4,483 ms—test vs leader
cnn_qwen_l_n200_v1 · judged/judge-summ-v2 filed4.573200$0.000844 · 828 tokp50 2,732 · p95 4,428 ms—test vs leader
cnn_llama_l_n200_v1 · judged/judge-summ-v2 filed4.529200$0.000195 · 805 tokp50 3,646 · p95 5,531 ms—test vs leader
cnn_llama_m_n200_v1 · judged/judge-summ-v2 filed4.488200$0.000165 · 809 tokp50 2,682 · p95 11,240 ms—test vs leader
cnn_llama_s_n200_v1 · judged/judge-summ-v2 filed4.469200$0.000111 · 814 tokp50 2,204 · p95 4,041 ms—test vs leader
cnn_qwen_s_n200_v1 · judged/judge-summ-v2 filed4.461200$0.000135 · 825 tokp50 7,855 · p95 108,807 ms—test vs leader
cnn_gemma_s_n200_v1 · judged/judge-summ-v2 filed4.429200$0.000100 · 820 tokp50 5,933 · p95 12,255 ms—test vs leader
cnn_glm_s_n200_v1 · judged/judge-summ-v2 filed4.420200$0.000152 · 786 tokp50 1,580 · p95 2,341 ms—test vs leader
cnn_mistral_s_n200_v1 · judged/judge-summ-v2 filed4.408200$0.000089 · 833 tokp50 2,573 · p95 5,142 ms—test vs leader
cnn_glm_l_n200_v1 · judged/judge-summ-v2 filed4.384200$0.000766 · 818 tokp50 2,351 · p95 12,758 ms—test vs leader
cnn_mistral_m_n200_v1 · judged/judge-summ-v2 filed4.309200$0.000435 · 832 tokp50 2,495 · p95 4,801 ms—test vs leader
cnn_mistral_l_n200_v1 · judged/judge-summ-v2 filed4.037200$0.000486 · 828 tokp50 2,272 · p95 51,619 ms—test vs leader
cnn_bart_l_n200_v1 · judged/judge-summ-v2 filed3.821200$0.000000 · 763 tokp50 10,724 · p95 14,371 ms—test vs leader
cnn_bart_l_xsum_n200_v1 · judged/judge-summ-v2 filed3.366200$0.000000 · 725 tokp50 4,433 · p95 7,325 ms—test vs leader
cnn_lead3_n200_v1 · judged/judge-summ-v2 filed3.339200$0.000000p50 0.057 · p95 0.13 ms—test vs leader
cnn_bart_m_n200_v1 · judged/judge-summ-v2 filed3.235200$0.000000 · 766 tokp50 4,649 · p95 7,223 ms—test vs leader
cnn_bart_s_n200_v1 · judged/judge-summ-v2 filed2.946200$0.000000 · 761 tokp50 3,683 · p95 5,055 ms—test vs leader
$0.0000$0.0098cost (USD per item)4.8882.946cnn_anthropic_l_n200_v1 · judged/judge-summ-v2: judge_score 4.888, cost 0.010cnn_deepseek_m_n200_v1 · judged/judge-summ-v2: judge_score 4.854, cost 0.000cnn_openai_l_n200_v1 · judged/judge-summ-v2: judge_score 4.847, cost 0.002cnn_openai_m_n200_v1 · judged/judge-summ-v2: judge_score 4.811, cost 0.005cnn_anthropic_m_n200_v1 · judged/judge-summ-v2: judge_score 4.809, cost 0.004cnn_deepseek_l_n200_v1 · judged/judge-summ-v2: judge_score 4.763, cost 0.001cnn_openai_s_n200_v1 · judged/judge-summ-v2: judge_score 4.747, cost 0.001cnn_anthropic_s_n200_v1 · judged/judge-summ-v2: judge_score 4.728, cost 0.001cnn_glm_m_n200_v1 · judged/judge-summ-v2: judge_score 4.718, cost 0.000cnn_deepseek_s_n200_v1 · judged/judge-summ-v2: judge_score 4.644, cost 0.000cnn_gemma_l_n200_v1 · judged/judge-summ-v2: judge_score 4.603, cost 0.000cnn_qwen_m_n200_v1 · judged/judge-summ-v2: judge_score 4.600, cost 0.000cnn_gemma_m_n200_v1 · judged/judge-summ-v2: judge_score 4.585, cost 0.000cnn_qwen_l_n200_v1 · judged/judge-summ-v2: judge_score 4.573, cost 0.001cnn_llama_l_n200_v1 · judged/judge-summ-v2: judge_score 4.529, cost 0.000cnn_llama_m_n200_v1 · judged/judge-summ-v2: judge_score 4.488, cost 0.000cnn_llama_s_n200_v1 · judged/judge-summ-v2: judge_score 4.469, cost 0.000cnn_qwen_s_n200_v1 · judged/judge-summ-v2: judge_score 4.461, cost 0.000cnn_gemma_s_n200_v1 · judged/judge-summ-v2: judge_score 4.429, cost 0.000cnn_glm_s_n200_v1 · judged/judge-summ-v2: judge_score 4.420, cost 0.000cnn_mistral_s_n200_v1 · judged/judge-summ-v2: judge_score 4.408, cost 0.000cnn_glm_l_n200_v1 · judged/judge-summ-v2: judge_score 4.384, cost 0.001cnn_mistral_m_n200_v1 · judged/judge-summ-v2: judge_score 4.309, cost 0.000cnn_mistral_l_n200_v1 · judged/judge-summ-v2: judge_score 4.037, cost 0.000cnn_bart_l_n200_v1 · judged/judge-summ-v2: judge_score 3.821, cost 0.000cnn_bart_l_xsum_n200_v1 · judged/judge-summ-v2: judge_score 3.366, cost 0.000cnn_lead3_n200_v1 · judged/judge-summ-v2: judge_score 3.339, cost 0.000cnn_bart_m_n200_v1 · judged/judge-summ-v2: judge_score 3.235, cost 0.000cnn_bart_s_n200_v1 · judged/judge-summ-v2: judge_score 2.946, cost 0.000cnn_anthropic_l_n200_v1 · judged/judge-summ-v2cnn_deepseek_m_n200_v1 · judged/judge-summ-v2cnn_deepseek_s_n200_v1 · judged/judge-summ-v2cnn_bart_l_n200_v1 · judged/judge-summ-v2
judge_score against cost (USD per item): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.129108807.3speed p95 (ms)4.8882.946cnn_anthropic_l_n200_v1 · judged/judge-summ-v2: judge_score 4.888, speed 4717.9cnn_deepseek_m_n200_v1 · judged/judge-summ-v2: judge_score 4.854, speed 2923.6cnn_openai_l_n200_v1 · judged/judge-summ-v2: judge_score 4.847, speed 2380.7cnn_openai_m_n200_v1 · judged/judge-summ-v2: judge_score 4.811, speed 2126.3cnn_anthropic_m_n200_v1 · judged/judge-summ-v2: judge_score 4.809, speed 3674.6cnn_deepseek_l_n200_v1 · judged/judge-summ-v2: judge_score 4.763, speed 4380.2cnn_openai_s_n200_v1 · judged/judge-summ-v2: judge_score 4.747, speed 1573.0cnn_anthropic_s_n200_v1 · judged/judge-summ-v2: judge_score 4.728, speed 2833.6cnn_glm_m_n200_v1 · judged/judge-summ-v2: judge_score 4.718, speed 7145.5cnn_deepseek_s_n200_v1 · judged/judge-summ-v2: judge_score 4.644, speed 5643.6cnn_gemma_l_n200_v1 · judged/judge-summ-v2: judge_score 4.603, speed 6171.8cnn_qwen_m_n200_v1 · judged/judge-summ-v2: judge_score 4.600, speed 1910.8cnn_gemma_m_n200_v1 · judged/judge-summ-v2: judge_score 4.585, speed 4482.6cnn_qwen_l_n200_v1 · judged/judge-summ-v2: judge_score 4.573, speed 4428.1cnn_llama_l_n200_v1 · judged/judge-summ-v2: judge_score 4.529, speed 5530.8cnn_llama_m_n200_v1 · judged/judge-summ-v2: judge_score 4.488, speed 11239.9cnn_llama_s_n200_v1 · judged/judge-summ-v2: judge_score 4.469, speed 4040.6cnn_qwen_s_n200_v1 · judged/judge-summ-v2: judge_score 4.461, speed 108807.3cnn_gemma_s_n200_v1 · judged/judge-summ-v2: judge_score 4.429, speed 12255.1cnn_glm_s_n200_v1 · judged/judge-summ-v2: judge_score 4.420, speed 2341.2cnn_mistral_s_n200_v1 · judged/judge-summ-v2: judge_score 4.408, speed 5142.2cnn_glm_l_n200_v1 · judged/judge-summ-v2: judge_score 4.384, speed 12757.6cnn_mistral_m_n200_v1 · judged/judge-summ-v2: judge_score 4.309, speed 4800.7cnn_mistral_l_n200_v1 · judged/judge-summ-v2: judge_score 4.037, speed 51618.8cnn_bart_l_n200_v1 · judged/judge-summ-v2: judge_score 3.821, speed 14370.8cnn_bart_l_xsum_n200_v1 · judged/judge-summ-v2: judge_score 3.366, speed 7325.1cnn_lead3_n200_v1 · judged/judge-summ-v2: judge_score 3.339, speed 0.129cnn_bart_m_n200_v1 · judged/judge-summ-v2: judge_score 3.235, speed 7223.5cnn_bart_s_n200_v1 · judged/judge-summ-v2: judge_score 2.946, speed 5054.7cnn_anthropic_l_n200_v1 · judged/judge-summ-v2cnn_deepseek_m_n200_v1 · judged/judge-summ-v2cnn_openai_l_n200_v1 · judged/judge-summ-v2cnn_openai_m_n200_v1 · judged/judge-summ-v2cnn_openai_s_n200_v1 · judged/judge-summ-v2cnn_lead3_n200_v1 · judged/judge-summ-v2
judge_score against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.

05cnn_dailymail_20 · pilot

cnn_dailymail_20: 20 items · pilot

nested in the 200; prompts tuned here only

references: gold/cnn_dailymail_20

scored against gold/cnn_dailymail_20 gold

Test every arm in this group against its leader

Shown, not recommended

  • Strategy balanced. Quality leader: cnn_bart_l_v1 (coverage 0.366). Contenders beside it (not significantly different by the paired test and within 0.00732 (2% of the leader)): none.
  • The goal does not decide on cnn_dailymail_20 (pilot): shown, not recommended.
armmean coverageitemscost per itemspeed per itemrouge1flags
cnn_bart_l_v1 filed0.36620$0.000000 · 732 tokp50 11,468 · p95 17,811 ms0.3584—leader
cnn_lead3_v1 filed0.26220$0.000000p50 0.069 · p95 0.13 ms0.3115—test vs leader
0.13217810.7speed p95 (ms)0.3660.262cnn_bart_l_v1: coverage 0.366, speed 17810.7cnn_lead3_v1: coverage 0.262, speed 0.132cnn_bart_l_v1cnn_lead3_v1
coverage against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.3120.358rouge10.3660.262cnn_bart_l_v1: coverage 0.366, rouge1 0.358cnn_lead3_v1: coverage 0.262, rouge1 0.312cnn_bart_l_v1
coverage against rouge1: better is right top. Named and joined: the frontier, the arms no other arm beats on both.

cost: $0 for every arm, so no trade-off to draw.

06cnn_dailymail_dm200 · generalization

cnn_dailymail_dm200: 200 items · generalization

200 seeded DailyMail articles; a check that never decides

references: gold/cnn_dailymail_dm200

scored against gold/cnn_dailymail_dm200 gold

Test every arm in this group against its leader

Shown, not recommended

  • Strategy balanced. Quality leader: cnn_bart_l_dm200_v1 (coverage 0.443). Contenders beside it (not significantly different by the paired test and within 0.00886 (2% of the leader)): cnn_bart_m_dm200_v1.
  • By speed (latency_ms p95): cnn_bart_m_dm200_v1 11,473 ms, cnn_bart_l_dm200_v1 16,948 ms.
  • The goal does not decide on cnn_dailymail_dm200 (generalization): shown, not recommended.
armmean coverageitemscost per itemspeed per itemrouge1flags
cnn_bart_l_dm200_v1 filed0.443200$0.000000 · 934 tokp50 10,251 · p95 16,948 ms0.4616—leader
cnn_bart_m_dm200_v1 filed0.443200$0.000000 · 940 tokp50 6,927 · p95 11,473 ms0.4695—test vs leader
cnn_bart_s_dm200_v1 filed0.405200$0.000000 · 914 tokp50 4,207 · p95 5,571 ms0.4478—test vs leader
cnn_lead3_dm200_v1 filed0.370200$0.000000p50 0.1 · p95 0.27 ms0.422—test vs leader
cnn_bart_l_xsum_dm200_v1 filed0.152200$0.000000 · 874 tokp50 5,103 · p95 8,017 ms0.2472—test vs leader
0.27216948.4speed p95 (ms)0.4430.152cnn_bart_l_dm200_v1: coverage 0.443, speed 16948.4cnn_bart_m_dm200_v1: coverage 0.443, speed 11472.6cnn_bart_s_dm200_v1: coverage 0.405, speed 5570.8cnn_lead3_dm200_v1: coverage 0.370, speed 0.272cnn_bart_l_xsum_dm200_v1: coverage 0.152, speed 8016.5cnn_bart_l_dm200_v1cnn_bart_m_dm200_v1cnn_bart_s_dm200_v1cnn_lead3_dm200_v1
coverage against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.2470.47rouge10.4430.152cnn_bart_l_dm200_v1: coverage 0.443, rouge1 0.462cnn_bart_m_dm200_v1: coverage 0.443, rouge1 0.470cnn_bart_s_dm200_v1: coverage 0.405, rouge1 0.448cnn_lead3_dm200_v1: coverage 0.370, rouge1 0.422cnn_bart_l_xsum_dm200_v1: coverage 0.152, rouge1 0.247cnn_bart_l_dm200_v1cnn_bart_m_dm200_v1
coverage against rouge1: better is right top. Named and joined: the frontier, the arms no other arm beats on both.

cost: $0 for every arm, so no trade-off to draw.