hosted · proprietaryhosted · open weightslocal · freestatistically tied with the leaderhigher score: further right →
Each dot is one system, by F1. The ringed dot is the most expensive hosted model; the shaded band is the group the leader cannot be statistically separated from.
02Cost
What the money bought.
hosted · proprietaryhosted · open weightslocal · freebetter: up · cheaper: left
Billed cost per 1,000 items, read from the provider's response, against F1. Free models sit in the left lane.
03Findings
What this experiment found.
The only unambiguous winner in the repo.
span_marker, fine-tuned on Few-NERD's training split, scores 0.7674 and is separated from all 27 other arms.
The dearest model finished 3rd of 28.
anthropic_l billed $0.6897 for 280 items. No paid model is statistically tied with the leader.
But its margin is partly the corpus's own conventions.
57% of its margin is one entity type, other, whose meaning exists only in this corpus, and 38% of its lead over the best hosted model is agreeing with annotation the rest of the field rejects.
04The method, here
Silver is not gold.
57% of the winner's margin is one entity type whose meaning exists only in that corpus. A clean win can still be partly a win at the annotation, not the task.
The 4 free arms rerun on a laptop CPU for $0, pinned to the weights that were measured. The 24 hosted arms billed $2.61 when measured and rerun with one OpenRouter key.
make compare writes a replication card: each rerun against its committed run, item by item, and every separation and tie on this page re-tested with your reruns in place.
$ make plan EXP=few-nerd
$ make data EXP=few-nerd
$ make reproduce EXP=few-nerd LEVEL=local
# with OPENROUTER_API_KEY set$ make reproduce EXP=few-nerd LEVEL=hosted
$ make compare EXP=few-nerd
