hosted · proprietaryhosted · open weightslocal · freestatistically tied with the leaderhigher score: further right →
Each dot is one system, by accuracy. The ringed dot is the most expensive hosted model; the shaded band is the group the leader cannot be statistically separated from. A dot left of the ⫽ break sits far below the rest, off the scale; tap it for its score.
02Cost
What the money bought.
hosted · proprietaryhosted · open weightslocal · freebetter: up · cheaper: left
Billed cost per 1,000 items, read from the provider's response, against accuracy. Free models sit in the left lane. A dot below the ⫽ break sits far below the rest, off the scale; tap it for its score.
03Findings
What this experiment found.
No winner: a group of nineteen.
qwen_m leads at 0.9929, statistically tied with 19 others, including the free fine-tune.
The dearest model finished 2nd of 28.
anthropic_l billed $0.3784 for 280 items. The cheapest model statistically tied with the leader, gemma_m, cost 115× less.
The benchmark is saturated, and the gold is sometimes wrong.
Three items are disputed by 24 to 25 of 26 arms, and reading them, the gold label is the error.
04The method, here
Tested, not read.
19 arms cannot be separated from the leader. A saturated benchmark sorts; it does not rank — and the table still prints them first, second, third.
The 4 free arms rerun on a laptop CPU for $0, pinned to the weights that were measured. The 24 hosted arms billed $1.17 when measured and rerun with one OpenRouter key.
make compare writes a replication card: each rerun against its committed run, item by item, and every separation and tie on this page re-tested with your reruns in place.
$ make plan EXP=dbpedia
$ make data EXP=dbpedia
$ make reproduce EXP=dbpedia LEVEL=local
# with OPENROUTER_API_KEY set$ make reproduce EXP=dbpedia LEVEL=hosted
$ make compare EXP=dbpedia
