The evals · worked examples, ongoing · 6 tasks · 167 measured runs · $19.07 billed
The most expensive model never won.
24 hosted LLMs, proprietary and open-weight from small to frontier tiers, small fine-tuned models and the baselines they have to beat, run through six public benchmarks as worked examples of the harness and its method. Every comparison is tested for significance, every cost is read off the provider's bill, and the instrument itself is checked against the published literature. The headline is on each eval's primary metric. Re-measured, against labels a person corrected and by LLM judges, the dearest model, anthropic_l, is first on the point estimate in two: AG News and CNN/DailyMail; neither re-measurement is confirmed.
5/5
evals with frontier models where the dearest model was not the best (the same model each time; SciFact ran only the twelve cheapest hosted models)
5/6
leaderboards where the leader is not significantly different from others (at 100–280 items)
$0
for a model that is first, or not significantly different from first, in 5 of the 6; 3 of 6 if the re-measured scores are taken at face value
11/11
calibration checks that agree with the published numbers, 10 inside their own interval
01Price
The dearest model was never the best.
Each row is one benchmark; each dot is one system, placed by its score. The ringed dot is the most expensive hosted model. In all six it finished behind something cheaper — and in three, behind something free. On SciFact only the twelve cheapest hosted models ran, so there it is the dearest of those; no Anthropic or OpenAI arm was run.
hosted · proprietaryhosted · open weightslocal · freenot significantly different from the leaderhigher score: further right →
Baselines (lead-3, keyword rules, empty output) are left off the axis to keep the field readable; they are in each eval's page. A dot left of the ⫽ break sits far below the rest, off the scale. Hover or tap any dot for its score, rank and bill.
02Contenders
The top is a crowd, not a podium.
A leaderboard sorts, so something is always first. The question is whether first is distinguishable from second. Here every leader was tested against every other arm on the same items, with a paired permutation test and a correction for running many comparisons at once.
In 5 of 6 evals the leader cannot be told apart from a group of others — on summarisation, from 15 of them. The cheapest paid model inside that group cost a fraction of the dearest: 143× less on summarisation.
The one clean winner is a 499 MB tagger fine-tuned on Few-NERD's own training data, significantly different from all 27 rivals.
Not significantly different from 1st: other systems the leader is not significantly different from (paired permutation test, Holm-corrected). …of which paid: how many of those are hosted models that bill per call; the rest are free. Cheapest paid one: the cheapest of them, and how many times less it cost than the most expensive model in the field. 0 means a clean winner: significantly different from every rival.
03Fine-tuning
Train a small model on your data. Yours.
A free local model was first, or not significantly different from first, in 5 of 6.
Against hosted LLMs run with one prompt, no examples and no reasoning. Every one is under 2 GB and needs no GPU — $0 per item — against hosted models billing up to $2.01 for 100 items (anthropic_l, SummEval).
The clearest wins were on text like the text it was trained on.
The same BART architecture, fine-tuned on XSum's one-sentence BBC summaries instead of CNN/DailyMail's, finishes last of 29 — below copying the first three sentences. The XSum control also decodes shorter, so it does not isolate style.
On one task, even a free model trained for it lost to LLM judges — and the dearest judge was not needed.
Judging summaries, scored against the mean of two experts as one expert is against the other two, the best judges reach 93% of a human's agreement (the best of 24, chosen on the same articles; 92% held out); UniEval, the best free scorer and an evaluator trained for the task, reaches 64%. The judges rank all 16 summaries in one call while UniEval scores one at a time, and a pointwise LLM judge was not run; a no-model lookup of which system wrote each summary reaches 63%. The cheapest judge the leader is not significantly different from, gemma_l, has open weights and bills $278 per million articles — 1/23 of the leader. One task and one trained evaluator, and why it lost is not tested: the direction, not the size.
And part of a fine-tune's lead is learning the dataset's own conventions.
On AG News, the lead of the three fine-tunes is concentrated in 17 items the hosted field mostly gets wrong. Adjudicated by topic by one reader shown the hosted answer, the gold was wrong on all 17, and on that gold every hosted LLM scores above every fine-tune. On Few-NERD, 57% of the winner's margin over openai_m is one entity type the prompt never defined (30–57% across the hosted arms). The leaders are better at the benchmark; how much of that is better at the task is a separate question, measured in each report.
04Trust
Can you trust any of this?
Five evals score a sample of 200–280 items, not a full test set, and SummEval scores the whole of its 100. So each one also runs a calibration: one well-known model or metric, through this repo's own code, on the benchmark's full test set (or a large seeded sample), compared with the number its authors published.
Our figure minus the published one, in points of the metric (ROUGE-1 F1 for summarisation; accuracy, micro-F1, nDCG@10 or Kendall τ ×100 elsewhere). Bars are our 95% interval. 11 of 11 agree with the published value within that interval plus a 2-point allowance for protocol differences; 10 are inside the interval itself.
Calibration found one real error — in our sample, not our scorer.
The summarisation eval took the first 200 articles of the CNN/DailyMail test set. The test set lists CNN's articles first, so all 200 were CNN's, while the benchmark is 91% DailyMail. Lead-3 scored ROUGE-1 30.20 on that slice against 39.94 on the full test set; CNN articles are simply harder to summarise from their lead.
Re-run on 200 DailyMail articles drawn at random, the five local summarisers kept every verdict they had on CNN: the fine-tune still beats lead-3 by the same margin, and the XSum model is still last. The hosted models were not re-run, so the summarisation price findings are scoped to CNN's articles — and say so.
coverage of the five local summarisers on each slice. Dashed: BART trained on XSum.
No figure on this site is typed in by hand. Every figure is recomputed from the committed runs each time the site is built, and every number the prose quotes must appear in the report it comes from, or the build fails. Behind the reports, a checker verifies 200 claims against the runs on every commit.
Or test it yourself, in your browser. Every run behind these charts is in the catalog, and its inspector re-runs any comparison on this site, item by item, with the same paired test. Each eval's page opens it on every arm against its leader. Or ask your agent: the same views answer over MCP at https://evals.humancopilot.eu/mcp, read-only, no sign-in (how to connect).
05The evals
Six benchmarks, one harness.
The same 24 hosted models — eight vendors, three price tiers each — on every task (the cheapest twelve on SciFact), beside the strongest free models each task allows.
The harness, every run, every model output and every analysis script are in the repository. Before any of it, the catalog shows every run and its inspector re-tests any comparison, with nothing installed. Then five levels of checking, each optional, each a few commands.
00
Verify
Every number in every report, recomputed from the committed predictions.
$0 · one command
01
Data
Fetch the public benchmarks and hash-check every item against what was measured.
$0
02
Local
Rerun the 35 free arms — local models and baselines — on your CPU, pinned to the same weights.
$0 · CPU hours
03
Hosted
Rerun the 24 LLMs with one OpenRouter key. The test: does every significant difference still hold?
$19.07 billed, all six
04
Extend
Add a model of your own and test it against the whole committed field.
one config
# every number, from the committed runs$ make verify
# rerun an eval and check its findings$ make plan EXP=scifact
$ make data EXP=scifact
$ make reproduce EXP=scifact LEVEL=local
$ make compare EXP=scifact
Reviewed in adversarial rounds: the lab notebook records what was wrong, how it was found, and what changed; the reports state today's figures, and git keeps every earlier version.
07The hardware
Where the runs were made.
Every run on this site records the machine it ran on; the counts below are read from those records, and each machine's description from the machine itself. Most of it is a 2018 Mac mini. No open-weight LLM was run on any of these machines for these results: the "own it" figures elsewhere are provider-served scores, set against what would fit a given memory budget.
machine
what it is
runs on this site it measured
role
Mac mini (2018)
Intel Core i7-8700B, 6 cores / 12 threads, 32 GB
128 20 local, 108 hosted
the homelab host: it ran the first five evals, local arms on its CPU and hosted arms as API calls through a LiteLLM gateway it also hosts
Cloud runner
4-core Linux (x86_64) containers; a session's container, so not captured
13 13 local, 0 hosted
4-core Linux containers from the operator's cloud coding sessions (the fingerprints record two kernel releases); they ran the six local models that need torch 2.6 or later (the distilled and XSum BARTs, the three bert-base fine-tunes) and SummEval's seven free scorers, and re-ran nine local arms on one machine for timing (harness/data/runs-linux)
GMKtec EVO-X3
AMD Ryzen AI Max+ 395 (Strix Halo) with Radeon 8060S, 32 threads, 128 GB unified memory
26 2 local, 24 hosted
the workstation SummEval's 24 hosted judges were called from, and where its two UniEval arms ran on CPU; local LLMs for a planned next eval are run here, and none is in the committed results
NVIDIA DGX Spark
NVIDIA GB10, 20 Arm cores, 128 GB unified memory
none
the homelab's GPU: it serves local models and is the workhorse for the runs of the operator's other projects (the podcast pipeline's models among them); none of this site's committed runs was measured on it
MacBook Pro (M4 Pro)
Apple M4 Pro, 14 cores, 48 GB unified memory
none
the laptop the work is driven from, and a workhorse for the runs of the operator's other projects; none of this site's committed runs was measured on it
Hosted runs are API calls made from that machine; the model ran at the provider. Local runs ran on the machine's own CPU. The DGX Spark and the MacBook are the workhorses for the runs of the operator's other projects; none of these runs was measured on them. Captured 2026-10-18: docs/evidence/hardware.json.
Who built this
Built by Marko Dragoljevic.
I have led engineering teams for twenty years — at ServiceNow, EclecticIQ and Backbase — and through Human Copilot I now advise technology leaders. Which model, build or buy, own or rent: the questions this harness answers are the ones I help teams answer, on their own data.
