A free model not significantly different from the best paid one — on CNN's news.
Summarise a news article in two or three sentences. A 1.6 GB fine-tune is not significantly different from the best paid model (0.00008 apart) — on CNN's articles.
The report's pick, as run here. Use bart_l if you can run a free 1.6 GB model on CPU and your text looks like CNN news; rent deepseek_m ($191 per 1M articles) if you need speed; do not buy anthropic_l.
The goal's rule, in the catalog. Short news summaries (CNN) (balanced, on coverage) recommends bart_l, the promoted baseline. The goal page, with its reasons → An agent gets the same answer from the MCP server: the tool goals with goal: news_summary_cnn at https://evals.humancopilot.eu/mcp (how to connect).
Re-measured. SummEval-validated LLM judges, cheap and frontier, rank every hosted LLM above every BART model; one reader's check went the other way. Neither is confirmed. Below · what it teaches
Every system, one axis.
What the money bought.
What this eval found.
A 1.6 GB model fine-tuned on this corpus is not significantly different from the best paid model.
bart_l scores 0.3461 on coverage, deepseek_m 0.3460 — a gap of 0.00008. It is not significantly different from 15 of the other 28 arms.
The dearest model finished 18th of 29.
anthropic_l billed $1.97 for 200 items. The cheapest model not significantly different from the leader, deepseek_s, cost 143× less.
The same architecture trained on another corpus finishes last.
BART fine-tuned on XSum's one-sentence BBC summaries scores 0.2071 — 29th of 29, below lead-3. bart_l's lead is consistent with learning CNN's house style; the XSum control also decodes shorter, so it does not isolate style.
The winner depends on the metric.
On ROUGE-2 the three CNN-trained BARTs take the top three places; on ROUGE-1 the LLMs lead and bart_l is 12th.
Scope: these are CNN's articles.
All 200 come from CNN's 9% of the test set. On 200 DailyMail articles the five local models kept every verdict (bart_l 0.4430, lead-3 0.3697); the paid models were not re-run.
Does it hold on DailyMail?
The committed sample is all CNN. On 200 DailyMail articles drawn at random, the five local summarisers keep every verdict among themselves. The hosted models were not re-run there.
Re-measured by SummEval-validated LLM judges, and checked by a person.
Scored again by two LLM judges validated on SummEval, every hosted LLM ranks above every BART model: bart_l falls from 1st to 25th of 29 (the hosted scores bunch near the top of the scale, so their own order is not a finding). One reader's blind check of 20 pairs on truncated articles went the other way, 14 to 6 for bart_l over deepseek_m, not significant either. Two frontier judges at 41× the price agreed with the first pair, 20 to 0 on the same pairs. On crowd ratings of 2022 LLM summaries the judges track people better than one, noisy, rater does on coherence and relevance (not on faithfulness), while still ranking systems differently in both directions; on these outputs neither is confirmed, so coverage stays the primary metric.
The data is the instrument.
The committed sample turned out to be all CNN's articles. A calibration row — copying the first three sentences on the full test set, against its published score — is what showed it.
This habit, in the method →Check it yourself.
First, with nothing installed: test every arm against bart_l, item by item, in the catalog's inspector, with the same paired test and Holm correction as this page; anthropic_l, the most expensive model, among them. The goal page has the rest: what the goal recommends, and why.
The 5 free arms rerun on a laptop CPU for $0, pinned to the weights that were measured. The 24 hosted arms billed $5.74 when measured and rerun with one OpenRouter key.
make compare writes a replication card: each rerun against its committed run, item by item, and every significant difference and tie on this page re-tested with your reruns in place.
$ make plan EXP=summarization $ make data EXP=summarization $ make reproduce EXP=summarization LEVEL=local # with OPENROUTER_API_KEY set $ make reproduce EXP=summarization LEVEL=hosted $ make compare EXP=summarization