eval-harness
Classification · AG News

Small models win by learning the labels.

Four-way topic labels. Three small fine-tunes take the top three places, largely by learning the dataset's labelling conventions.

Question
Label a news snippet World, Sports, Business or Sci/Tech.
Data
200 items from the AG News test split
Systems
30 compared: 24 hosted models billed per call; 6 local models and baselines, free
Metric
accuracy — the field's standard; higher is better
Calibration
bert_mini 0.9338 vs 0.9339 published; bert_base_ta 0.9514 vs 0.9514 published; bert_base_fy 0.9375 vs 0.9375 published
Billed
$0.5808 for the whole sweep
Read the full reportProtocolAll experiments
01The field

Every system, one axis.

hosted · proprietaryhosted · open weightslocal · freestatistically tied with the leaderhigher score: further right →
bert_base_ta · local · free accuracy 0.9600 · rank 1/30 free: runs locally, nothing billedbert_base_fy · local · free accuracy 0.9500 · rank 2/30 free: runs locally, nothing billedbert_mini · local · free accuracy 0.9450 · rank 3/30 free: runs locally, nothing billedanthropic_m · hosted · proprietary accuracy 0.9100 · rank 4/30 billed $0.0703 for 200 itemsopenai_m · hosted · proprietary accuracy 0.8950 · rank 6/30 billed $0.1272 for 200 itemsopenai_l · hosted · proprietary accuracy 0.8900 · rank 7/30 billed $0.0487 for 200 itemsqwen_m · hosted · proprietary accuracy 0.8850 · rank 8/30 billed $0.0070 for 200 itemsllama_m · hosted · open weights accuracy 0.8800 · rank 9/30 billed $0.0040 for 200 itemsgemma_s · hosted · open weights accuracy 0.8750 · rank 10/30 billed $0.0021 for 200 itemsmistral_l · hosted · open weights accuracy 0.8750 · rank 11/30 billed $0.0103 for 200 itemsqwen_s · hosted · proprietary accuracy 0.8750 · rank 12/30 billed $0.0036 for 200 itemsdeepseek_l · hosted · open weights accuracy 0.8700 · rank 13/30 billed $0.0114 for 200 itemsgemma_l · hosted · open weights accuracy 0.8700 · rank 14/30 billed $0.0028 for 200 itemsqwen_l · hosted · proprietary accuracy 0.8700 · rank 15/30 billed $0.0164 for 200 itemsglm_l · hosted · open weights accuracy 0.8600 · rank 16/30 billed $0.0186 for 200 itemsglm_m · hosted · open weights accuracy 0.8600 · rank 17/30 billed $0.0107 for 200 itemsmistral_m · hosted · proprietary accuracy 0.8600 · rank 18/30 billed $0.0096 for 200 itemsmistral_s · hosted · open weights accuracy 0.8600 · rank 19/30 billed $0.0019 for 200 itemsopenai_s · hosted · proprietary accuracy 0.8600 · rank 20/30 billed $0.0190 for 200 itemsgemma_m · hosted · open weights accuracy 0.8550 · rank 21/30 billed $0.0016 for 200 itemsglm_s · hosted · open weights accuracy 0.8500 · rank 22/30 billed $0.0027 for 200 itemsdeepseek_m · hosted · open weights accuracy 0.8450 · rank 23/30 billed $0.0043 for 200 itemsanthropic_s · hosted · proprietary accuracy 0.8400 · rank 24/30 billed $0.0254 for 200 itemsdeepseek_s · hosted · open weights accuracy 0.8400 · rank 25/30 billed $0.0014 for 200 itemsllama_l · hosted · open weights accuracy 0.8400 · rank 26/30 billed $0.0043 for 200 itemsllama_s · hosted · open weights accuracy 0.8350 · rank 27/30 billed $0.0025 for 200 itemsbart_mnli · local · free accuracy 0.7000 · rank 28/30 free: runs locally, nothing billedanthropic_l · hosted · proprietary accuracy 0.9000 · rank 5/30 billed $0.1750 for 200 itemsdearest · 5th · 0.900bert_base_ta · 1st · 0.960
bert_base_ta · local · free accuracy 0.9600 · rank 1/30 free: runs locally, nothing billedbert_base_fy · local · free accuracy 0.9500 · rank 2/30 free: runs locally, nothing billedbert_mini · local · free accuracy 0.9450 · rank 3/30 free: runs locally, nothing billedanthropic_m · hosted · proprietary accuracy 0.9100 · rank 4/30 billed $0.0703 for 200 itemsopenai_m · hosted · proprietary accuracy 0.8950 · rank 6/30 billed $0.1272 for 200 itemsopenai_l · hosted · proprietary accuracy 0.8900 · rank 7/30 billed $0.0487 for 200 itemsqwen_m · hosted · proprietary accuracy 0.8850 · rank 8/30 billed $0.0070 for 200 itemsllama_m · hosted · open weights accuracy 0.8800 · rank 9/30 billed $0.0040 for 200 itemsgemma_s · hosted · open weights accuracy 0.8750 · rank 10/30 billed $0.0021 for 200 itemsmistral_l · hosted · open weights accuracy 0.8750 · rank 11/30 billed $0.0103 for 200 itemsqwen_s · hosted · proprietary accuracy 0.8750 · rank 12/30 billed $0.0036 for 200 itemsdeepseek_l · hosted · open weights accuracy 0.8700 · rank 13/30 billed $0.0114 for 200 itemsgemma_l · hosted · open weights accuracy 0.8700 · rank 14/30 billed $0.0028 for 200 itemsqwen_l · hosted · proprietary accuracy 0.8700 · rank 15/30 billed $0.0164 for 200 itemsglm_l · hosted · open weights accuracy 0.8600 · rank 16/30 billed $0.0186 for 200 itemsglm_m · hosted · open weights accuracy 0.8600 · rank 17/30 billed $0.0107 for 200 itemsmistral_m · hosted · proprietary accuracy 0.8600 · rank 18/30 billed $0.0096 for 200 itemsmistral_s · hosted · open weights accuracy 0.8600 · rank 19/30 billed $0.0019 for 200 itemsopenai_s · hosted · proprietary accuracy 0.8600 · rank 20/30 billed $0.0190 for 200 itemsgemma_m · hosted · open weights accuracy 0.8550 · rank 21/30 billed $0.0016 for 200 itemsglm_s · hosted · open weights accuracy 0.8500 · rank 22/30 billed $0.0027 for 200 itemsdeepseek_m · hosted · open weights accuracy 0.8450 · rank 23/30 billed $0.0043 for 200 itemsanthropic_s · hosted · proprietary accuracy 0.8400 · rank 24/30 billed $0.0254 for 200 itemsdeepseek_s · hosted · open weights accuracy 0.8400 · rank 25/30 billed $0.0014 for 200 itemsllama_l · hosted · open weights accuracy 0.8400 · rank 26/30 billed $0.0043 for 200 itemsllama_s · hosted · open weights accuracy 0.8350 · rank 27/30 billed $0.0025 for 200 itemsbart_mnli · local · free accuracy 0.7000 · rank 28/30 free: runs locally, nothing billedanthropic_l · hosted · proprietary accuracy 0.9000 · rank 5/30 billed $0.1750 for 200 itemsdearest · 5th · 0.900bert_base_ta · 1st · 0.960
Each dot is one system, by accuracy. The ringed dot is the most expensive hosted model; the shaded band is the group the leader cannot be statistically separated from.
02Cost

What the money bought.

hosted · proprietaryhosted · open weightslocal · freebetter: up · cheaper: left
$0.001$0.01$0.1$10.7000.8300.960billed cost per 1,000 items (log scale)↑ accuracyfreebert_base_ta · local · free accuracy 0.9600 · $0bert_base_fy · local · free accuracy 0.9500 · $0bert_mini · local · free accuracy 0.9450 · $0bart_mnli · local · free accuracy 0.7000 · $0anthropic_m · hosted · proprietary accuracy 0.9100 $0.3514 per 1,000 itemsanthropic_l · hosted · proprietary accuracy 0.9000 $0.8749 per 1,000 itemsopenai_m · hosted · proprietary accuracy 0.8950 $0.6361 per 1,000 itemsopenai_l · hosted · proprietary accuracy 0.8900 $0.2435 per 1,000 itemsqwen_m · hosted · proprietary accuracy 0.8850 $0.0348 per 1,000 itemsllama_m · hosted · open weights accuracy 0.8800 $0.0198 per 1,000 itemsgemma_s · hosted · open weights accuracy 0.8750 $0.0107 per 1,000 itemsmistral_l · hosted · open weights accuracy 0.8750 $0.0516 per 1,000 itemsqwen_s · hosted · proprietary accuracy 0.8750 $0.0180 per 1,000 itemsdeepseek_l · hosted · open weights accuracy 0.8700 $0.0571 per 1,000 itemsgemma_l · hosted · open weights accuracy 0.8700 $0.0140 per 1,000 itemsqwen_l · hosted · proprietary accuracy 0.8700 $0.0821 per 1,000 itemsglm_l · hosted · open weights accuracy 0.8600 $0.0930 per 1,000 itemsglm_m · hosted · open weights accuracy 0.8600 $0.0533 per 1,000 itemsmistral_m · hosted · proprietary accuracy 0.8600 $0.0480 per 1,000 itemsmistral_s · hosted · open weights accuracy 0.8600 $0.0095 per 1,000 itemsopenai_s · hosted · proprietary accuracy 0.8600 $0.0950 per 1,000 itemsgemma_m · hosted · open weights accuracy 0.8550 $0.0079 per 1,000 itemsglm_s · hosted · open weights accuracy 0.8500 $0.0137 per 1,000 itemsdeepseek_m · hosted · open weights accuracy 0.8450 $0.0216 per 1,000 itemsanthropic_s · hosted · proprietary accuracy 0.8400 $0.1268 per 1,000 itemsdeepseek_s · hosted · open weights accuracy 0.8400 $0.0070 per 1,000 itemsllama_l · hosted · open weights accuracy 0.8400 $0.0214 per 1,000 itemsllama_s · hosted · open weights accuracy 0.8350 $0.0127 per 1,000 itemsanthropic_l · dearestbert_base_ta · 1st
$0.001$0.01$0.1$10.7000.8300.960cost per 1,000 items (log)↑ accuracyfreebert_base_ta · local · free accuracy 0.9600 · $0bert_base_fy · local · free accuracy 0.9500 · $0bert_mini · local · free accuracy 0.9450 · $0bart_mnli · local · free accuracy 0.7000 · $0anthropic_m · hosted · proprietary accuracy 0.9100 $0.3514 per 1,000 itemsanthropic_l · hosted · proprietary accuracy 0.9000 $0.8749 per 1,000 itemsopenai_m · hosted · proprietary accuracy 0.8950 $0.6361 per 1,000 itemsopenai_l · hosted · proprietary accuracy 0.8900 $0.2435 per 1,000 itemsqwen_m · hosted · proprietary accuracy 0.8850 $0.0348 per 1,000 itemsllama_m · hosted · open weights accuracy 0.8800 $0.0198 per 1,000 itemsgemma_s · hosted · open weights accuracy 0.8750 $0.0107 per 1,000 itemsmistral_l · hosted · open weights accuracy 0.8750 $0.0516 per 1,000 itemsqwen_s · hosted · proprietary accuracy 0.8750 $0.0180 per 1,000 itemsdeepseek_l · hosted · open weights accuracy 0.8700 $0.0571 per 1,000 itemsgemma_l · hosted · open weights accuracy 0.8700 $0.0140 per 1,000 itemsqwen_l · hosted · proprietary accuracy 0.8700 $0.0821 per 1,000 itemsglm_l · hosted · open weights accuracy 0.8600 $0.0930 per 1,000 itemsglm_m · hosted · open weights accuracy 0.8600 $0.0533 per 1,000 itemsmistral_m · hosted · proprietary accuracy 0.8600 $0.0480 per 1,000 itemsmistral_s · hosted · open weights accuracy 0.8600 $0.0095 per 1,000 itemsopenai_s · hosted · proprietary accuracy 0.8600 $0.0950 per 1,000 itemsgemma_m · hosted · open weights accuracy 0.8550 $0.0079 per 1,000 itemsglm_s · hosted · open weights accuracy 0.8500 $0.0137 per 1,000 itemsdeepseek_m · hosted · open weights accuracy 0.8450 $0.0216 per 1,000 itemsanthropic_s · hosted · proprietary accuracy 0.8400 $0.1268 per 1,000 itemsdeepseek_s · hosted · open weights accuracy 0.8400 $0.0070 per 1,000 itemsllama_l · hosted · open weights accuracy 0.8400 $0.0214 per 1,000 itemsllama_s · hosted · open weights accuracy 0.8350 $0.0127 per 1,000 itemsanthropic_l · dearestbert_base_ta · 1st
Billed cost per 1,000 items, read from the provider's response, against accuracy. Free models sit in the left lane.
03Findings

What this experiment found.

  1. Three small fine-tuned classifiers take the top three places.

    bert_base_ta leads at 0.9600; the best hosted model, anthropic_m, scores 0.9100.

  2. The dearest model finished 5th of 30.

    anthropic_l billed $0.1750 for 200 items. The cheapest model statistically tied with the leader, anthropic_m, cost 2.5× less.

  3. Most of the lead is the dataset's labelling convention.

    17 items are labelled against the judgement of most of the hosted field — business news about technology companies filed as Sci/Tech, Olympic results filed as World. The fine-tunes agree with the gold on most of them.

04The method, here

Silver is not gold.

Nearly the whole lead of the three fine-tunes sits in 17 items whose gold labels most of the hosted field disputes.

This habit, in the method →
05Reproduce it

Check it yourself.

The 6 free arms rerun on a laptop CPU for $0, pinned to the weights that were measured. The 24 hosted arms billed $0.58 when measured and rerun with one OpenRouter key.

make compare writes a replication card: each rerun against its committed run, item by item, and every separation and tie on this page re-tested with your reruns in place.

The guideThis example's code
$ make plan      EXP=ag-news
$ make data      EXP=ag-news
$ make reproduce EXP=ag-news LEVEL=local
# with OPENROUTER_API_KEY set
$ make reproduce EXP=ag-news LEVEL=hosted
$ make compare   EXP=ag-news