eval-harness

goal

Encyclopedia topic classification (DBpedia)

Pick a classifier that labels an encyclopedia abstract with one of fourteen classes, at a known cost per item.

01what it is for

decision
Which classifiers can be told apart on accuracy across many classes, and at what cost.
audience
a pipeline that files encyclopedia entries by class
decides
correct, higher is better
weighs
quality (correct); cost per item; speed per item
strategy
frugal: contenders: anything not significantly different from the leader by the paired test; the cheaper wins
tie-break
cost, speed
output contract
label
status
active
guardrails
  • licence: DBpedia-14 is CC-BY-SA 3.0 + GFDL, inherited from Wikipedia: anything redistributed that contains its text is share-alike. (docs/DATA.md)
  • not shown is not ruled out: At 280 items no gap under 10 items was significant: a gain of up to about 3 points from paying more is not ruled out, only not shown. (research/REPORT_CLASSIFICATION.md, executive summary)
links

02the answer so far

dbpedia_280 · baseline promoted 2026-10-25 · scored against gold

db_bert_base_fy_n200_v1

The cheapest arm at 0.9857: bert_base_fy (438 MB, $0 marginal, 87 ms per item on a 4-core Linux CPU), two items behind the leader qwen_m and not significantly different from it, as gemma_m and qwen_s are. REPORT_CLASSIFICATION, executive summary.

deploy sheet: what production must match
measured by
db_bert_base_fy_n200_v1_20260930T091006Z on dbpedia_280
against
gold/dbpedia_280
model sent
fabriceyhc/bert-base-uncased-dbpedia_14
model record
fabriceyhc-bert-base-uncased-dbpedia_14
server
not recorded
build
none recorded (a hosted model has no digest)
settings sent
device: cpu
label_order: ["Company", "EducationalInstitution", "Artist", "Athlete", "OfficeHolder", "MeanOfTransportation", "Building", "NaturalPlace", "Village", "Animal", "Plant", "Album", "Film", "WrittenWork"]
model: fabriceyhc/bert-base-uncased-dbpedia_14
precision: fp32
provider: hf_local
not sent
bookkeeping: family, price_tier
declared
deployment.encoder_window: 512
deployment.label_order: ['Company', 'EducationalInstitution', 'Artist', 'Athlete', 'OfficeHolder', 'MeanOfTransportation', 'Building', 'NaturalPlace', 'Village', 'Animal', 'Plant', 'Album', 'Film', 'WrittenWork']
deployment.labels: ['Company', 'EducationalInstitution', 'Artist', 'Athlete', 'OfficeHolder', 'MeanOfTransportation', 'Building', 'NaturalPlace', 'Village', 'Animal', 'Plant', 'Album', 'Film', 'WrittenWork']
deployment.labels_sha256: d3a33437c1e2b2fe45fdb7a425b14796d89d31a0d8919d899149e27f5c6f899d
deployment.parser_sha256: 4ad032219af088a127d97f358471c648a50d9665e3b32cd2f3e2f2dca7d8ded2
deployment.provider: hf_local
adapter
examples/classification-dbpedia-14/adapter.py 58b55c964f8b
harness
4068ea23007d

No capture here says what production ran (the adapter's production_ran).

Command: evh deploy-sheet --config db_bert_base_fy_n200_v1 [--capture C]

inspect it

dbpedia_56

No decision yet. When the tests separate an arm worth keeping, promote its run with the reason they give: evh run-promote --run <run_id> --reason "…"

03how we got here: rounds

Rounds are optional: each one sets out to learn something, changes a few arms, drops what lost and says why, and ends in a conclusion. Decisions are made with evh round-open, round-retire and round-close; evh rounds says the same as this page. Open a round for what changed, what was retired and why, and what to do next.

r2 closed
from r1
bert_base_fy runs at 87 ms per item on a 4-core Linux CPU (438 MB, $0 marginal). (REPORT_CLASSIFICATION, the decision table.)
How fast is the fine-tuned arm timed on the Linux machine?
0 arms (1 retired)

1 retired

arms

r1 closed
The top eleven arms (0.9786–0.9929) sit within 4 items of each other across a 115× price range; the leader qwen_m differs significantly from only 8 of the 27 others. Take the cheapest arm you can run at 0.9857: gemma_m (~$12 per 1M items), qwen_s (~$26) or the fine-tuned bert_base_fy ($0 marginal). (REPORT_CLASSIFICATION, executive summary.)
Which of 24 hosted LLMs (8 vendors × 3 price tiers), a fine-tuned BERT, a zero-shot NLI model, regex rules and a constant labels DBpedia-14 best, and at what cost?
28 arms inspect
best db_qwen_m_n200_v1 0.993

04dbpedia_280 · test

dbpedia_280: 280 items · test

20 per class

references: gold/dbpedia_280

Mean correct over each arm's items, in groups by what each arm is scored against: numbers from different references, or from a judge, are different measurements and are never ranked together. A mean is not a verdict: the inspector tests each arm against its group's leader (Holm across them). Flagged arms are never the leader. filed: the arm's config names this goal (goal:); an arm without it is listed because it records the measure.

scored against gold/dbpedia_280 gold

Test every arm in this group against its leader

Recommended: db_bert_base_fy_n200_v1

  • Strategy frugal. Quality leader: db_qwen_m_n200_v1 (correct 0.9929). Contenders beside it (not significantly different by the paired test): 19 arms
    whichdb_anthropic_l_n200_v1, db_bert_base_fy_n200_v1, db_deepseek_l_n200_v1, db_gemma_m_n200_v1, db_glm_m_n200_v1, db_qwen_s_n200_v1, db_anthropic_m_n200_v1, db_openai_m_n200_v1, db_anthropic_s_n200_v1, db_llama_m_n200_v1, db_deepseek_m_n200_v1, db_gemma_l_n200_v1, db_gemma_s_n200_v1, db_llama_l_n200_v1, db_openai_l_n200_v1, db_openai_s_n200_v1, db_glm_l_n200_v1, db_qwen_l_n200_v1, db_mistral_l_n200_v1
  • By cost (cost_usd): db_bert_base_fy_n200_v1 $0.000000, db_gemma_m_n200_v1 $0.000012, db_gemma_s_n200_v1 $0.000012, db_gemma_l_n200_v1 $0.000016, db_qwen_s_n200_v1 $0.000026, db_deepseek_m_n200_v1 $0.000028, db_llama_m_n200_v1 $0.000030, db_llama_l_n200_v1 $0.000032, db_qwen_m_n200_v1 $0.000051, db_glm_m_n200_v1 $0.000057, db_deepseek_l_n200_v1 $0.000074, db_mistral_l_n200_v1 $0.000075, db_qwen_l_n200_v1 $0.000122, db_glm_l_n200_v1 $0.000128, db_openai_s_n200_v1 $0.000128, db_anthropic_s_n200_v1 $0.000192, db_openai_l_n200_v1 $0.000329, db_anthropic_m_n200_v1 $0.000541, db_openai_m_n200_v1 $0.000850, db_anthropic_l_n200_v1 $0.001351.
  • Recommended: db_bert_base_fy_n200_v1.

Strategy frugal (contenders: anything not significantly different from the leader by the paired test; the cheaper wins). Weighs quality (correct); cost per item; speed per item. Among the contenders on quality, the tie-break is cost, speed. The rule recommends; promoting is evh run-promote; the goal chooses its strategy (strategy:).

armmean correctitemscost per itemspeed per itemflags
db_qwen_m_n200_v1 filed0.993280$0.000051 · 154 tokp50 818 · p95 1,115 ms—leader
db_anthropic_l_n200_v1 filed0.989280$0.001351 · 245 tokp50 3,244 · p95 3,704 ms—test vs leader
db_bert_base_fy_n200_v1 filed baseline recommended0.986280$0.000000p50 75 · p95 119 ms—test vs leader
db_bert_base_fy_n200_v1 · linux filed retired in r20.986280$0.000000p50 61 · p95 175 ms—test vs leader
db_deepseek_l_n200_v1 filed0.986280$0.000074 · 141 tokp50 1,497 · p95 3,617 ms—test vs leader
db_gemma_m_n200_v1 filed0.986280$0.000012 · 150 tokp50 651 · p95 1,549 ms—test vs leader
db_glm_m_n200_v1 filed0.986280$0.000057 · 147 tokp50 1,073 · p95 4,958 ms—test vs leader
db_qwen_s_n200_v1 filed0.986280$0.000026 · 167 tokp50 8,595 · p95 34,672 ms—test vs leader
db_anthropic_m_n200_v1 filed0.982280$0.000541 · 245 tokp50 2,203 · p95 2,748 ms—test vs leader
db_openai_m_n200_v1 filed0.982280$0.000850 · 142 tokp50 1,087 · p95 1,661 ms—test vs leader
db_anthropic_s_n200_v1 filed0.979280$0.000192 · 169 tokp50 981 · p95 1,740 ms—test vs leader
db_llama_m_n200_v1 filed0.979280$0.000030 · 155 tokp50 795 · p95 5,859 ms—test vs leader
db_deepseek_m_n200_v1 filed0.975280$0.000028 · 141 tokp50 994 · p95 1,662 ms—test vs leader
db_gemma_l_n200_v1 filed0.975280$0.000016 · 150 tokp50 662 · p95 3,346 ms—test vs leader
db_gemma_s_n200_v1 filed0.975280$0.000012 · 147 tokp50 586 · p95 997 ms—test vs leader
db_llama_l_n200_v1 filed0.975280$0.000032 · 145 tokp50 621 · p95 1,156 ms—test vs leader
db_openai_l_n200_v1 filed0.975280$0.000329 · 142 tokp50 1,227 · p95 1,650 ms—test vs leader
db_openai_s_n200_v1 filed0.975280$0.000128 · 143 tokp50 842 · p95 1,335 ms—test vs leader
db_glm_l_n200_v1 filed0.968280$0.000128 · 151 tokp50 1,508 · p95 3,595 ms—test vs leader
db_qwen_l_n200_v1 filed0.968280$0.000122 · 149 tokp50 1,152 · p95 3,564 ms—test vs leader
db_mistral_l_n200_v1 filed0.964280$0.000075 · 144 tokp50 620 · p95 57,125 ms—test vs leader
db_llama_s_n200_v1 filed0.957280$0.000027 · 145 tokp50 1,178 · p95 2,351 ms—test vs leader
db_mistral_m_n200_v1 filed0.957280$0.000067 · 156 tokp50 402 · p95 758 ms—test vs leader
db_mistral_s_n200_v1 filed0.957280$0.000014 · 144 tokp50 770 · p95 1,864 ms—test vs leader
db_deepseek_s_n200_v1 filed0.943280$0.000010 · 141 tokp50 1,188 · p95 3,065 ms—test vs leader
db_glm_s_n200_v1 filed0.939280$0.000018 · 147 tokp50 1,080 · p95 1,606 ms—test vs leader
db_keyword_n200_v1 filed0.707280$0.000000p50 0.076 · p95 0.22 ms—test vs leader
db_bart_mnli_n200_v1 filed0.629280$0.000000p50 5,354 · p95 8,420 ms—test vs leader
db_constant_n200_v1 filed0.071280$0.000000p50 0.003 · p95 0.006 ms—test vs leader
$0.0000$0.0014cost (USD per item)0.9930.071db_qwen_m_n200_v1: correct 0.993, cost 0.000db_anthropic_l_n200_v1: correct 0.989, cost 0.001db_bert_base_fy_n200_v1: correct 0.986, cost 0.000db_deepseek_l_n200_v1: correct 0.986, cost 0.000db_gemma_m_n200_v1: correct 0.986, cost 0.000db_glm_m_n200_v1: correct 0.986, cost 0.000db_qwen_s_n200_v1: correct 0.986, cost 0.000db_anthropic_m_n200_v1: correct 0.982, cost 0.001db_openai_m_n200_v1: correct 0.982, cost 0.001db_anthropic_s_n200_v1: correct 0.979, cost 0.000db_llama_m_n200_v1: correct 0.979, cost 0.000db_deepseek_m_n200_v1: correct 0.975, cost 0.000db_gemma_l_n200_v1: correct 0.975, cost 0.000db_gemma_s_n200_v1: correct 0.975, cost 0.000db_llama_l_n200_v1: correct 0.975, cost 0.000db_openai_l_n200_v1: correct 0.975, cost 0.000db_openai_s_n200_v1: correct 0.975, cost 0.000db_glm_l_n200_v1: correct 0.968, cost 0.000db_qwen_l_n200_v1: correct 0.968, cost 0.000db_mistral_l_n200_v1: correct 0.964, cost 0.000db_llama_s_n200_v1: correct 0.957, cost 0.000db_mistral_m_n200_v1: correct 0.957, cost 0.000db_mistral_s_n200_v1: correct 0.957, cost 0.000db_deepseek_s_n200_v1: correct 0.943, cost 0.000db_glm_s_n200_v1: correct 0.939, cost 0.000db_keyword_n200_v1: correct 0.707, cost 0.000db_bart_mnli_n200_v1: correct 0.629, cost 0.000db_constant_n200_v1: correct 0.071, cost 0.000db_qwen_m_n200_v1db_bert_base_fy_n200_v1
correct against cost (USD per item): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.00657125.4speed p95 (ms)0.9930.071db_qwen_m_n200_v1: correct 0.993, speed 1115.2db_anthropic_l_n200_v1: correct 0.989, speed 3704.0db_bert_base_fy_n200_v1: correct 0.986, speed 118.6db_deepseek_l_n200_v1: correct 0.986, speed 3616.8db_gemma_m_n200_v1: correct 0.986, speed 1549.3db_glm_m_n200_v1: correct 0.986, speed 4958.4db_qwen_s_n200_v1: correct 0.986, speed 34672.1db_anthropic_m_n200_v1: correct 0.982, speed 2748.4db_openai_m_n200_v1: correct 0.982, speed 1661.0db_anthropic_s_n200_v1: correct 0.979, speed 1739.7db_llama_m_n200_v1: correct 0.979, speed 5859.0db_deepseek_m_n200_v1: correct 0.975, speed 1661.5db_gemma_l_n200_v1: correct 0.975, speed 3345.8db_gemma_s_n200_v1: correct 0.975, speed 997.4db_llama_l_n200_v1: correct 0.975, speed 1156.0db_openai_l_n200_v1: correct 0.975, speed 1650.5db_openai_s_n200_v1: correct 0.975, speed 1335.0db_glm_l_n200_v1: correct 0.968, speed 3594.7db_qwen_l_n200_v1: correct 0.968, speed 3563.6db_mistral_l_n200_v1: correct 0.964, speed 57125.4db_llama_s_n200_v1: correct 0.957, speed 2350.9db_mistral_m_n200_v1: correct 0.957, speed 758.2db_mistral_s_n200_v1: correct 0.957, speed 1864.0db_deepseek_s_n200_v1: correct 0.943, speed 3064.6db_glm_s_n200_v1: correct 0.939, speed 1605.6db_keyword_n200_v1: correct 0.707, speed 0.221db_bart_mnli_n200_v1: correct 0.629, speed 8420.1db_constant_n200_v1: correct 0.071, speed 0.006db_qwen_m_n200_v1db_bert_base_fy_n200_v1db_keyword_n200_v1db_constant_n200_v1
correct against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.

05dbpedia_56 · pilot

dbpedia_56: 56 items · pilot

the pilot slice; all 56 of its items are in dbpedia_280

references: gold/dbpedia_56

No arm on this dataset records correct.