Summarisation · CNN/DailyMail

A free model ties the frontier — on CNN's news.

Summarise a news article in two or three sentences. A 1.6 GB fine-tune ties the best paid model — on CNN's articles.

Question

Summarise a news article in two or three short sentences, in a wire-service style.

Data

200 items from the CNN/DailyMail test split

Systems

29 compared: 24 hosted models billed per call; 5 local models and baselines, free

Metric

coverage — how much of the reference summary's wording the summary recovers, at the reference's length (ROUGE-Lsum recall, clipped); this repo's own, see the protocol. Higher is better

Calibration

LEAD-3, ROUGE-1 39.94 vs 40.34 published

Billed

$5.74 for the whole sweep

Read the full report Protocol All experiments

01The field

Every system, one axis.

hosted · proprietaryhosted · open weightslocal · freestatistically tied with the leaderhigher score: further right →

Each dot is one system, by coverage. The ringed dot is the most expensive hosted model; the shaded band is the group the leader cannot be statistically separated from. A dot left of the ⫽ break sits far below the rest, off the scale; tap it for its score.

02Cost

What the money bought.

hosted · proprietaryhosted · open weightslocal · freebetter: up · cheaper: left

Billed cost per 1,000 items, read from the provider's response, against coverage. Free models sit in the left lane. A dot below the ⫽ break sits far below the rest, off the scale; tap it for its score.

03Findings

What this experiment found.

A 1.6 GB model fine-tuned on this corpus ties the best paid model.
bart_l scores 0.3461 on coverage, deepseek_m 0.3460 — a gap of 0.0001. It cannot be separated from 15 of the other 28 arms.
The dearest model finished 18th of 29.
anthropic_l billed $1.97 for 200 items. The cheapest model statistically tied with the leader, deepseek_s, cost 143× less.
The same architecture trained on another corpus finishes last.
BART fine-tuned on XSum's one-sentence BBC summaries scores 0.2071 — 29th of 29, below lead-3. What bart_l has is CNN's house style.
The winner depends on the metric.
On ROUGE-2 the three CNN-trained BARTs take the top three places; on ROUGE-1 the LLMs lead and bart_l is 12th.
Scope: these are CNN's articles.
All 200 come from CNN's 9% of the test set. On 200 DailyMail articles the five local models kept every verdict (bart_l 0.4430, lead-3 0.3697); the paid models were not re-run.

04Outside CNN

Does it hold on DailyMail?

The committed sample is all CNN. On 200 DailyMail articles drawn at random, the five local summarisers keep every verdict among themselves. The hosted models were not re-run there.

05The method, here

The data is the instrument.

The committed sample turned out to be all CNN's articles. A calibration row — copying the first three sentences on the full test set, against its published score — is what showed it.

This habit, in the method →

06Reproduce it

Check it yourself.

The 5 free arms rerun on a laptop CPU for $0, pinned to the weights that were measured. The 24 hosted arms billed $5.74 when measured and rerun with one OpenRouter key.

make compare writes a replication card: each rerun against its committed run, item by item, and every separation and tie on this page re-tested with your reruns in place.

The guide This example's code

$ make plan      EXP=summarization
$ make data      EXP=summarization
$ make reproduce EXP=summarization LEVEL=local
# with OPENROUTER_API_KEY set
$ make reproduce EXP=summarization LEVEL=hosted
$ make compare   EXP=summarization

A free model ties the frontier — on CNN's news.

Every system, one axis.

What the money bought.

What this experiment found.

A 1.6 GB model fine-tuned on this corpus ties the best paid model.

The dearest model finished 18th of 29.

The same architecture trained on another corpus finishes last.

The winner depends on the metric.

Scope: these are CNN's articles.

Does it hold on DailyMail?

The data is the instrument.

Check it yourself.