Classification · DBpedia-14

The leader is not significantly different from 19 others.

Fourteen-way ontology labels on a saturated benchmark: the leader is not significantly different from nineteen others (1 to 8 items behind), and the gold label is sometimes in dispute.

The report's pick, as run here. Take the cheapest arm you can run at 0.9857: gemma_m (~$12 per 1M items), qwen_s (~$26) or the fine-tuned bert_base_fy.

The goal's rule, in the catalog. Encyclopedia topic classification (DBpedia) (frugal, on correct) recommends bert_base_fy, the promoted baseline. The goal page, with its reasons → An agent gets the same answer from the MCP server: the tool goals with goal: entity_topic_dbpedia at https://evals.humancopilot.eu/mcp (how to connect).

Re-measured. Three disputed labels were corrected; the top stays a crowd and the pick stands. Below · what it teaches

Question

Label a Wikipedia abstract with one of 14 ontology classes.

Data

280 items from the DBpedia-14 test split

Systems

28 compared: 24 hosted models billed per call; 4 local models and baselines, free

Metric

accuracy — the field's standard; higher is better

Calibration

bert_base_fy 0.9879 vs 0.9903 published

Billed

$1.17 for the whole sweep

Read the full report In the catalog Protocol All evals

01The field

Every system, one axis.

hosted · proprietaryhosted · open weightslocal · freenot significantly different from the leaderhigher score: further right →

Each dot is one system, by accuracy. The ringed dot is the most expensive hosted model; the shaded band is the group the leader is not significantly different from. A dot left of the ⫽ break sits far below the rest, off the scale; tap it for its score. Every system has its own page in the catalog: its run, cost, speed and fingerprint.

02Cost

What the money bought.

hosted · proprietaryhosted · open weightslocal · freebetter: up · cheaper: left

Billed cost per 1,000 items, read from the provider's response, against accuracy. Free models sit in the left lane. A dot below the ⫽ break sits far below the rest, off the scale; tap it for its score.

03Findings

What this eval found.

No winner: the leader and nineteen others.
qwen_m leads at 0.9929, not significantly different from 19 others, including the free fine-tune.
The dearest model finished 2nd of 28.
anthropic_l billed $0.3784 for 280 items. The cheapest model not significantly different from the leader, gemma_m, cost 115× less.
The benchmark is saturated, and the gold is sometimes disputed.
Three items are disputed by 24 to 25 of 26 arms. Adjudicated by one reader shown the hosted answer, the gold was wrong on all three; the dataset's own gold stays primary.

04Re-measured

Re-measured against labels a person corrected.

One reader, the author, adjudicated 5 items, shown the hosted answer: the 3 disputed ones, which at least 90% of the hosted LLMs got wrong, and the 2 others bert_base_fy got wrong. The gold was wrong on all 3 disputed items and right on the 2. The top stays a crowd, and bert_base_fy moves from 3rd to 11th at an unchanged 0.9857. The dataset's own gold stays primary.

What re-measuring changed, and what it teaches →

05The method, here

Tested, not read.

19 arms are not significantly different from the leader. A saturated benchmark sorts; it does not rank — and the table still prints them first, second, third.

This habit, in the method →

06Reproduce it

Check it yourself.

First, with nothing installed: test every arm against qwen_m, item by item, in the catalog's inspector, with the same paired test and Holm correction as this page; anthropic_l, the most expensive model, among them. The goal page has the rest: what the goal recommends, and why.

The 4 free arms rerun on a laptop CPU for $0, pinned to the weights that were measured. The 24 hosted arms billed $1.17 when measured and rerun with one OpenRouter key.

make compare writes a replication card: each rerun against its committed run, item by item, and every significant difference and tie on this page re-tested with your reruns in place.

The guide This example's code

$ make plan      EXP=dbpedia
$ make data      EXP=dbpedia
$ make reproduce EXP=dbpedia LEVEL=local
# with OPENROUTER_API_KEY set
$ make reproduce EXP=dbpedia LEVEL=hosted
$ make compare   EXP=dbpedia