Small models lead, largely on disputed labels.
Four-way topic labels. Three small fine-tunes lead on the point estimate, LLMs as run here; the lead is concentrated in 17 items the hosted field mostly gets wrong.
The report's pick, as run here. If your labels follow a house convention, fine-tune on them, starting with bert_mini; if they mean what the words say, use a hosted LLM. If you rent, start with gemma_m ($8 per 1M items).
The goal's rule, in the catalog. News topic classification (AG News) (balanced, on correct) recommends no single arm yet. Speed was not used to break the tie: the arms ran on different machines. Still undecided after the tie-break: bert_base_ta, bert_base_fy, bert_mini. The baseline a person promoted is bert_mini, with the reason recorded: “bert_mini (44 MB, $0 marginal), for labels that follow AG News's house convention: 0.9450, not significantly different from bert_base_ta (0.9600, p = 0.55), and 11× faster on one Linux machine (4.3 vs 49 ms median). REPORT_CLASSIFICATION, executive summary and §3.1.” The goal page, with its reasons → An agent gets the same answer from the MCP server: the tool goals with goal: news_topic_ag at https://evals.humancopilot.eu/mcp (how to connect).
Re-measured. On labels one reader corrected, every hosted LLM has a higher accuracy than every fine-tune, so the fine-tune pick holds only if your labels follow AG News's filing convention. Below · what it teaches
Every system, one axis.
What the money bought.
What this eval found.
Three small fine-tuned classifiers lead on the point estimate; only the first differs significantly from all but one paid arm.
bert_base_ta leads at 0.9600; the best hosted model, anthropic_m, scores 0.9100.
The dearest model finished 5th of 30.
anthropic_l billed $0.1750 for 200 items. The cheapest model not significantly different from the leader, anthropic_m, cost 2.5× less.
The lead is concentrated in 17 items the hosted field mostly gets wrong.
17 items are labelled against the judgement of most of the hosted field — business news about technology companies filed as Sci/Tech, Olympic results filed as World. The fine-tunes agree with the gold on most of them. On the other 183 items the arms cannot be told apart — anthropic_l 0.984, bert_base_ta 0.973 (discordant 3 vs 1, exact sign p = 0.625) — and that cut is selected by the hosted arms' own errors, so it flatters them. Against gold adjudicated by topic by one reader, who was shown the hosted field's answer (29 items reviewed from both sides, 17 changed), every hosted LLM scores above every fine-tune: anthropic_l leads at 0.9850 and differs significantly from all three (tied at 0.9050). The primary numbers stay on the dataset's own gold.
Re-measured against labels a person corrected.
One reader, the author, adjudicated by topic 29 items, shown the hosted answer: the 17 disputed ones, which at least 80% of the hosted LLMs got wrong, and the 12 others a fine-tune got wrong. The gold was wrong on all 17 disputed items and right on the 12. On that gold every hosted LLM has a higher accuracy than every fine-tune: the fine-tunes fall from 1–3 to 25–27 of 30, and anthropic_l rises to first (significantly different from the fine-tunes, not from most hosted arms). The dataset's own gold stays primary.
Silver is not gold.
Nearly the whole lead of the three fine-tunes sits in 17 items whose gold labels most of the hosted field disputes.
This habit, in the method →Check it yourself.
First, with nothing installed: test every arm against bert_base_ta, item by item, in the catalog's inspector, with the same paired test and Holm correction as this page; anthropic_l, the most expensive model, among them. The goal page has the rest: what the goal recommends, and why.
The 6 free arms rerun on a laptop CPU for $0, pinned to the weights that were measured. The 24 hosted arms billed $0.58 when measured and rerun with one OpenRouter key.
make compare writes a replication card: each rerun against its committed run, item by item, and every significant difference and tie on this page re-tested with your reruns in place.
$ make plan EXP=ag-news $ make data EXP=ag-news $ make reproduce EXP=ag-news LEVEL=local # with OPENROUTER_API_KEY set $ make reproduce EXP=ag-news LEVEL=hosted $ make compare EXP=ag-news