The one clean win.
Find and type entities in a sentence. The only clean winner in the repo: a fine-tuned tagger, significantly different from every rival.
The report's pick, as run here. If you have labelled data in a fixed taxonomy, fine-tune a small tagger (span_marker); do not rent a zero-shot hosted LLM for this job. If you do rent, take the cheapest, gemma_m.
The goal's rule, in the catalog. Named-entity extraction (Few-NERD) (balanced, on f1) recommends span_marker, the promoted baseline. The goal page, with its reasons → An agent gets the same answer from the MCP server: the tool goals with goal: entities_few_nerd at https://evals.humancopilot.eu/mcp (how to connect).
Every system, one axis.
What the money bought.
What this eval found.
The one significant win in the repo.
span_marker, fine-tuned on Few-NERD's training split, scores 0.7674 and differs significantly from all 27 other arms.
The dearest model finished 3rd of 28.
anthropic_l billed $0.6897 for 280 items. The leader differs significantly from every paid model.
But its margin is partly the corpus's own conventions.
Against openai_m, 57% of its margin is one entity type, other, which the prompt never defined (30–57% across the hosted arms), and up to 38% of its lead is agreement with annotation the rest of the field rejects.
Silver is not gold.
57% of the winner's margin over openai_m is one entity type the prompt never defined (30–57% across the hosted arms). A clean win can still be partly a win at the annotation, not the task.
This habit, in the method →Check it yourself.
First, with nothing installed: test every arm against span_marker, item by item, in the catalog's inspector, with the same paired test and Holm correction as this page; anthropic_l, the most expensive model, among them. The goal page has the rest: what the goal recommends, and why.
The 4 free arms rerun on a laptop CPU for $0, pinned to the weights that were measured. The 24 hosted arms billed $2.61 when measured and rerun with one OpenRouter key.
make compare writes a replication card: each rerun against its committed run, item by item, and every significant difference and tie on this page re-tested with your reruns in place.
$ make plan EXP=few-nerd $ make data EXP=few-nerd $ make reproduce EXP=few-nerd LEVEL=local # with OPENROUTER_API_KEY set $ make reproduce EXP=few-nerd LEVEL=hosted $ make compare EXP=few-nerd