24 frontier and open-weight LLMs, small fine-tuned models and the baselines they have to beat, run through five public benchmarks. Every comparison is tested for significance, every cost is read off the provider's bill, and the instrument itself is checked against the published literature.
5/5
experiments where the dearest model was not the best
4/5
leaderboards where first place is a statistical tie
$0
for a model that is first or tied for first, in 5 of the 5
7/7
calibration checks that match the published numbers
01Price
The price tag told you nothing.
Each row is one benchmark; each dot is one system, placed by its score. The ringed dot is the most expensive hosted model. In all five it finished behind something cheaper — and in four, behind something free.
hosted · proprietaryhosted · open weightslocal · freestatistically tied with the leaderhigher score: further right →
Baselines (lead-3, keyword rules, empty output) are left off the axis to keep the field readable; they are in each experiment's page. A dot left of the ⫽ break sits far below the rest, off the scale. Hover or tap any dot for its score, rank and bill.
02Ties
The top is a crowd, not a podium.
A leaderboard sorts, so something is always first. The question is whether first is distinguishable from second. Here every leader was tested against every other arm on the same items, with a paired permutation test and a correction for running many comparisons at once.
In 4 of 5 experiments the leader cannot be told apart from a group of others — on summarisation, from 15 of them. The cheapest paid model inside that group cost a fraction of the dearest: 143× less on summarisation.
The one clean winner is a 499 MB tagger fine-tuned on Few-NERD's own training data, separated from all 27 rivals.
Tied with 1st: other systems the leader cannot be statistically separated from (paired permutation test, Holm-corrected). …of which paid: how many of those are hosted models that bill per call; the rest are free. Cheapest paid tie: the cheapest of them, and how many times less it cost than the most expensive model in the field. 0 tied means a clean winner: separated from every rival.
03Fine-tuning
Train a small model on your data. Yours.
The best fine-tuned model was first, or tied for first, in 5 of 5.
Every one is under 2 GB and needs no GPU — $0 per item — against hosted models billing up to $1.97 for 200 items (anthropic_l, CNN/DailyMail).
But only on text like the text it was trained on.
The same BART architecture, fine-tuned on XSum's one-sentence BBC summaries instead of CNN/DailyMail's, finishes last of 29 — below copying the first three sentences.
And part of a fine-tune's lead is learning the dataset's own conventions.
On AG News, nearly the whole lead of the three fine-tunes sits in 17 items whose gold labels most of the hosted field disputes. On Few-NERD, 57% of the winner's margin is one entity type whose meaning exists only in that corpus. The leaders are better at the benchmark; how much of that is better at the task is a separate question, measured in each report.
04Trust
Can you trust any of this?
Every experiment scores a sample of 200–280 items, not a full test set. So each one also runs a calibration: one well-known model, through this repo's own code, on the benchmark's full test set (or a large seeded sample), compared with the number its authors published.
Our figure minus the published one, in points of the metric (ROUGE-1 F1 for summarisation, accuracy, micro-F1 or nDCG@10 ×100 elsewhere). Bars are our 95% interval. 7 of 7 sit on the published value.
Calibration found one real error — in our sample, not our scorer.
The summarisation experiment took the first 200 articles of the CNN/DailyMail test set. The test set lists CNN's articles first, so all 200 were CNN's, while the benchmark is 91% DailyMail. Lead-3 scored ROUGE-1 30.20 on that slice against 39.94 on the full test set; CNN articles are simply harder to summarise from their lead.
Re-run on 200 DailyMail articles drawn at random, the five local summarisers kept every verdict they had on CNN: the fine-tune still beats lead-3 by the same margin, and the XSum model is still last. The hosted models were not re-run, so the summarisation price findings are scoped to CNN's articles — and say so.
coverage of the five local summarisers on each slice. Dashed: BART trained on XSum.
Nothing on this site is typed in by hand. Every figure is recomputed from the committed runs each time the site is built, and every number the prose quotes must appear in the report it comes from, or the build fails. Behind the reports, a checker verifies 82 claims against the runs on every commit.
05The experiments
Five benchmarks, one harness.
The same 24 hosted models — eight vendors, three price tiers each — on every task, beside the strongest free models each task allows.
The harness, every run, every model output and every analysis script are in the repository. Five levels of checking, each optional, each a few commands.
00
Verify
Every number in every report, recomputed from the committed predictions.
$0 · one command
01
Data
Fetch the public benchmarks and hash-check every item against what was measured.
$0
02
Local
Rerun the 26 free arms — local models and baselines — on your CPU, pinned to the same weights.
$0 · CPU hours
03
Hosted
Rerun the 24 LLMs with one OpenRouter key. The test: does every separation still hold?
$12.05 billed, all five
04
Extend
Add a model of your own and test it against the whole committed field.
one config
# every number, from the committed runs$ make verify
# rerun an experiment and check its findings$ make plan EXP=scifact
$ make data EXP=scifact
$ make reproduce EXP=scifact LEVEL=local
$ make compare EXP=scifact
Reviewed in five adversarial rounds, every correction kept: the lab notebook records what was wrong, how it was found, and what changed.
Who built this
Built by Marko Dragoljevic.
I have led engineering teams for twenty years — at ServiceNow, EclecticIQ and Backbase — and through Human Copilot I now advise technology leaders. Which model, build or buy, own or rent: the questions this harness answers are the ones I help teams answer, on their own data.
