eval-harness

goal

News topic classification (AG News)

Pick a classifier that labels a news headline and snippet with one of four topics, at a known cost per item.

01what it is for

decision
Which classifiers can be told apart on accuracy, and what each costs and how fast it runs.
audience
a pipeline that routes news items by topic
decides
correct, higher is better
weighs
quality (correct); cost per item; speed per item
strategy
balanced: contenders: not significantly different from the leader by the paired test AND within the quality margin (default)
tie-break
cost, speed
output contract
label
status
active
guardrails
  • labelling convention: The fine-tunes' lead is AG News's filing convention: against gold adjudicated by topic, every hosted LLM scores above every fine-tune. Read section 3.4 before choosing. (research/REPORT_CLASSIFICATION.md §3.4)
  • licence: AG News is licensed `unknown`, for research and non-commercial use; bert_base_ta and bert_mini state no licence. (examples/classification-ag-news/README.md)
links

02the answer so far

ag_news_200 · baseline promoted 2026-10-25 · scored against gold

ag_bert_mini_n200_v1

bert_mini (44 MB, $0 marginal), for labels that follow AG News's house convention: 0.9450, not significantly different from bert_base_ta (0.9600, p = 0.55), and 11× faster on one Linux machine (4.3 vs 49 ms median). REPORT_CLASSIFICATION, executive summary and §3.1.

deploy sheet: what production must match
measured by
ag_bert_mini_n200_v1_20260927T090317Z on ag_news_200
against
gold/ag_news_200
model sent
mrm8488/bert-mini-finetuned-age_news-classification
model record
mrm8488-bert-mini-finetuned-age_news-classification
server
not recorded
build
e94b474cce6f842ee5a90daa7352f5152fac4572
settings sent
device: cpu
label_order: ["World", "Sports", "Business", "Sci/Tech"]
model: mrm8488/bert-mini-finetuned-age_news-classification
precision: fp32
provider: hf_local
not sent
bookkeeping: family, price_tier
declared
deployment.encoder_window: 512
deployment.label_order: ['World', 'Sports', 'Business', 'Sci/Tech']
deployment.labels: ['World', 'Sports', 'Business', 'Sci/Tech']
deployment.labels_sha256: c7a1ee508b78c0e92f28d159d340a6537ac31ff70ff566430fe2507da58464a3
deployment.parser_sha256: 87f75633ea16670ab5a3169fe3125c91107f84959e79c56dd2f17c66a418246d
deployment.provider: hf_local
deployment.weight_files: ['model.safetensors:44698846']
adapter
examples/classification-ag-news/adapter.py 8e4dffce3505
harness
1e80be6531e2

No capture here says what production ran (the adapter's production_ran).

Command: evh deploy-sheet --config ag_bert_mini_n200_v1 [--capture C]

inspect it

ag_news_20

No decision yet. When the tests separate an arm worth keeping, promote its run with the reason they give: evh run-promote --run <run_id> --reason "…"

03how we got here: rounds

Rounds are optional: each one sets out to learn something, changes a few arms, drops what lost and says why, and ends in a conclusion. Decisions are made with evh round-open, round-retire and round-close; evh rounds says the same as this page. Open a round for what changed, what was retired and why, and what to do next.

r2 closed
from r1
Re-timed together on the Linux machine (medians): bert_base_ta 49 ms, bert_base_fy 55 ms, bert_mini 4.3 ms. bert_mini is 11× faster than bert_base_ta, at accuracy not significantly different from it (p = 0.55). (REPORT_CLASSIFICATION §3.1.)
How fast is each local arm when the three are timed together on one Linux machine?
0 arms (3 retired)

3 retired

arms

r1 closed
Against the dataset's gold, three public BERT fine-tunes (0.9600, 0.9500, 0.9450) score above all 24 hosted LLMs (0.835–0.910); that lead is AG News's filing convention. If your labels follow a house convention, fine-tune, starting with bert_mini (44 MB, $0 marginal); if they mean what the words say, use a hosted LLM; if you rent, start with gemma_m ($8 per 1M items). (REPORT_CLASSIFICATION, executive summary.)
Which of 24 hosted LLMs (8 vendors × 3 price tiers), three fine-tuned BERTs, a zero-shot NLI model, regex rules and a constant labels AG News best, and at what cost?
30 arms inspect
best ag_anthropic_l_n200_v1 0.985

04ag_news_200 · test

ag_news_200: 200 items · test

first 50 per class from the first 1

references: gold/ag_news_200, gold/ag_news_200_adj1

Mean correct over each arm's items, in groups by what each arm is scored against: numbers from different references, or from a judge, are different measurements and are never ranked together. A mean is not a verdict: the inspector tests each arm against its group's leader (Holm across them). Flagged arms are never the leader. filed: the arm's config names this goal (goal:); an arm without it is listed because it records the measure.

scored against gold/ag_news_200 gold

Test every arm in this group against its leader

No single arm recommended yet

  • Strategy balanced. Quality leader: ag_bert_base_ta_n200_v1 (correct 0.96). Contenders beside it (not significantly different by the paired test and within 0.0192 (2% of the leader)): ag_bert_base_fy_n200_v1, ag_bert_mini_n200_v1. Not significantly different but further off, so not contenders: ag_anthropic_m_n200_v1.
  • Speed was not used to break the tie: the arms ran on different machines.
  • Still undecided after the tie-break: all 3 contenders.

Strategy balanced (contenders: not significantly different from the leader by the paired test AND within the quality margin (default)). Weighs quality (correct); cost per item; speed per item. Among the contenders on quality, the tie-break is cost, speed. The rule recommends; promoting is evh run-promote; the goal chooses its strategy (strategy:).

armmean correctitemscost per itemspeed per itemflags
ag_bert_base_ta_n200_v1 filed0.960200$0.000000p50 64 · p95 114 ms—leader
ag_bert_base_ta_n200_v1 · linux filed retired in r20.960200$0.000000p50 49 · p95 91 ms—test vs leader
ag_bert_base_fy_n200_v1 filed0.950200$0.000000p50 69 · p95 118 ms—test vs leader
ag_bert_base_fy_n200_v1 · linux filed retired in r20.950200$0.000000p50 55 · p95 322 ms—test vs leader
ag_bert_mini_n200_v1 filed baseline0.945200$0.000000p50 6.3 · p95 12 ms—test vs leader
ag_bert_mini_n200_v1 · linux filed retired in r20.945200$0.000000p50 4.3 · p95 7 ms—test vs leader
ag_anthropic_m_n200_v1 filed0.910200$0.000351 · 155 tokp50 1,884 · p95 2,224 ms—test vs leader
ag_anthropic_l_n200_v1 filed0.900200$0.000875 · 155 tokp50 1,881 · p95 2,411 ms—test vs leader
ag_openai_m_n200_v1 filed0.895200$0.000636 · 100 tokp50 1,079 · p95 1,380 ms—test vs leader
ag_openai_l_n200_v1 filed0.890200$0.000244 · 100 tokp50 1,153 · p95 1,626 ms—test vs leader
ag_qwen_m_n200_v1 filed0.885200$0.000035 · 104 tokp50 905 · p95 1,176 ms—test vs leader
ag_llama_m_n200_v1 filed0.880200$0.000020 · 106 tokp50 682 · p95 3,527 ms—test vs leader
ag_gemma_s_n200_v1 filed0.875200$0.000011 · 103 tokp50 561 · p95 1,396 ms—test vs leader
ag_mistral_l_n200_v1 filed0.875200$0.000052 · 98 tokp50 595 · p95 56,011 ms—test vs leader
ag_qwen_s_n200_v1 filed0.875200$0.000018 · 117 tokp50 971 · p95 1,803 ms—test vs leader
ag_deepseek_l_n200_v1 filed0.870200$0.000057 · 96 tokp50 1,242 · p95 2,329 ms—test vs leader
ag_gemma_l_n200_v1 filed0.870200$0.000014 · 106 tokp50 519 · p95 1,749 ms—test vs leader
ag_qwen_l_n200_v1 filed0.870200$0.000082 · 100 tokp50 2,895 · p95 4,011 ms—test vs leader
ag_glm_l_n200_v1 filed0.860200$0.000093 · 104 tokp50 1,322 · p95 3,893 ms—test vs leader
ag_glm_m_n200_v1 filed0.860200$0.000053 · 99 tokp50 1,130 · p95 5,862 ms—test vs leader
ag_mistral_m_n200_v1 filed0.860200$0.000048 · 110 tokp50 397 · p95 718 ms—test vs leader
ag_mistral_s_n200_v1 filed0.860200$0.000010 · 98 tokp50 661 · p95 1,224 ms—test vs leader
ag_openai_s_n200_v1 filed0.860200$0.000095 · 100 tokp50 828 · p95 1,100 ms—test vs leader
ag_gemma_m_n200_v1 filed0.855200$0.000008 · 106 tokp50 523 · p95 1,164 ms—test vs leader
ag_glm_s_n200_v1 filed0.850200$0.000014 · 100 tokp50 1,084 · p95 1,946 ms—test vs leader
ag_deepseek_m_n200_v1 filed0.845200$0.000022 · 96 tokp50 663 · p95 1,538 ms—test vs leader
ag_anthropic_s_n200_v1 filed0.840200$0.000127 · 109 tokp50 937 · p95 1,218 ms—test vs leader
ag_deepseek_s_n200_v1 filed0.840200$0.000007 · 96 tokp50 1,229 · p95 3,523 ms—test vs leader
ag_llama_l_n200_v1 filed0.840200$0.000021 · 100 tokp50 635 · p95 1,582 ms—test vs leader
ag_llama_s_n200_v1 filed0.835200$0.000013 · 100 tokp50 699 · p95 1,540 ms—test vs leader
ag_bart_mnli_n200_v1 filed0.700200$0.000000p50 1,286 · p95 2,403 ms—test vs leader
ag_keyword_n200_v1 filed0.670200$0.000000p50 0.038 · p95 0.089 ms—test vs leader
ag_constant_n200_v1 filed0.250200$0.000000p50 0.002 · p95 0.002 ms—test vs leader
$0.0000$0.0009cost (USD per item)0.960.25ag_bert_base_ta_n200_v1: correct 0.960, cost 0.000ag_bert_base_fy_n200_v1: correct 0.950, cost 0.000ag_bert_mini_n200_v1: correct 0.945, cost 0.000ag_anthropic_m_n200_v1: correct 0.910, cost 0.000ag_anthropic_l_n200_v1: correct 0.900, cost 0.001ag_openai_m_n200_v1: correct 0.895, cost 0.001ag_openai_l_n200_v1: correct 0.890, cost 0.000ag_qwen_m_n200_v1: correct 0.885, cost 0.000ag_llama_m_n200_v1: correct 0.880, cost 0.000ag_gemma_s_n200_v1: correct 0.875, cost 0.000ag_mistral_l_n200_v1: correct 0.875, cost 0.000ag_qwen_s_n200_v1: correct 0.875, cost 0.000ag_deepseek_l_n200_v1: correct 0.870, cost 0.000ag_gemma_l_n200_v1: correct 0.870, cost 0.000ag_qwen_l_n200_v1: correct 0.870, cost 0.000ag_glm_l_n200_v1: correct 0.860, cost 0.000ag_glm_m_n200_v1: correct 0.860, cost 0.000ag_mistral_m_n200_v1: correct 0.860, cost 0.000ag_mistral_s_n200_v1: correct 0.860, cost 0.000ag_openai_s_n200_v1: correct 0.860, cost 0.000ag_gemma_m_n200_v1: correct 0.855, cost 0.000ag_glm_s_n200_v1: correct 0.850, cost 0.000ag_deepseek_m_n200_v1: correct 0.845, cost 0.000ag_anthropic_s_n200_v1: correct 0.840, cost 0.000ag_deepseek_s_n200_v1: correct 0.840, cost 0.000ag_llama_l_n200_v1: correct 0.840, cost 0.000ag_llama_s_n200_v1: correct 0.835, cost 0.000ag_bart_mnli_n200_v1: correct 0.700, cost 0.000ag_keyword_n200_v1: correct 0.670, cost 0.000ag_constant_n200_v1: correct 0.250, cost 0.000ag_bert_base_ta_n200_v1
correct against cost (USD per item): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.00256011.2speed p95 (ms)0.960.25ag_bert_base_ta_n200_v1: correct 0.960, speed 114.2ag_bert_base_fy_n200_v1: correct 0.950, speed 118.2ag_bert_mini_n200_v1: correct 0.945, speed 11.9ag_anthropic_m_n200_v1: correct 0.910, speed 2224.0ag_anthropic_l_n200_v1: correct 0.900, speed 2411.0ag_openai_m_n200_v1: correct 0.895, speed 1379.7ag_openai_l_n200_v1: correct 0.890, speed 1626.2ag_qwen_m_n200_v1: correct 0.885, speed 1175.5ag_llama_m_n200_v1: correct 0.880, speed 3527.2ag_gemma_s_n200_v1: correct 0.875, speed 1396.0ag_mistral_l_n200_v1: correct 0.875, speed 56011.2ag_qwen_s_n200_v1: correct 0.875, speed 1802.8ag_deepseek_l_n200_v1: correct 0.870, speed 2329.4ag_gemma_l_n200_v1: correct 0.870, speed 1748.9ag_qwen_l_n200_v1: correct 0.870, speed 4010.8ag_glm_l_n200_v1: correct 0.860, speed 3893.3ag_glm_m_n200_v1: correct 0.860, speed 5862.0ag_mistral_m_n200_v1: correct 0.860, speed 717.7ag_mistral_s_n200_v1: correct 0.860, speed 1223.8ag_openai_s_n200_v1: correct 0.860, speed 1100.4ag_gemma_m_n200_v1: correct 0.855, speed 1163.9ag_glm_s_n200_v1: correct 0.850, speed 1945.5ag_deepseek_m_n200_v1: correct 0.845, speed 1537.9ag_anthropic_s_n200_v1: correct 0.840, speed 1217.6ag_deepseek_s_n200_v1: correct 0.840, speed 3522.9ag_llama_l_n200_v1: correct 0.840, speed 1582.2ag_llama_s_n200_v1: correct 0.835, speed 1540.4ag_bart_mnli_n200_v1: correct 0.700, speed 2403.0ag_keyword_n200_v1: correct 0.670, speed 0.089ag_constant_n200_v1: correct 0.250, speed 0.002ag_bert_base_ta_n200_v1ag_bert_mini_n200_v1ag_keyword_n200_v1ag_constant_n200_v1
correct against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.

scored against gold/ag_news_200_adj1 gold

Test every arm in this group against its leader

Recommended: ag_openai_l_n200_v1 · adjudicated

  • Strategy balanced. Quality leader: ag_anthropic_l_n200_v1 · adjudicated (correct 0.985). Contenders beside it (not significantly different by the paired test and within 0.0197 (2% of the leader)): ag_openai_m_n200_v1 · adjudicated, ag_openai_l_n200_v1 · adjudicated. Not significantly different but further off, so not contenders: ag_anthropic_m_n200_v1 · adjudicated, ag_llama_m_n200_v1 · adjudicated, ag_gemma_s_n200_v1 · adjudicated, ag_qwen_m_n200_v1 · adjudicated, ag_qwen_s_n200_v1 · adjudicated, ag_deepseek_l_n200_v1 · adjudicated, ag_gemma_l_n200_v1 · adjudicated, ag_qwen_l_n200_v1 · adjudicated, ag_mistral_l_n200_v1 · adjudicated, ag_glm_l_n200_v1 · adjudicated, ag_glm_m_n200_v1 · adjudicated, ag_mistral_m_n200_v1 · adjudicated, ag_openai_s_n200_v1 · adjudicated, ag_gemma_m_n200_v1 · adjudicated, ag_glm_s_n200_v1 · adjudicated, ag_mistral_s_n200_v1 · adjudicated, ag_anthropic_s_n200_v1 · adjudicated, ag_llama_l_n200_v1 · adjudicated.
  • By cost (cost_usd): ag_openai_l_n200_v1 · adjudicated $0.000244, ag_openai_m_n200_v1 · adjudicated $0.000636, ag_anthropic_l_n200_v1 · adjudicated $0.000875.
  • Recommended: ag_openai_l_n200_v1 · adjudicated.
armmean correctitemscost per itemspeed per itemflags
ag_anthropic_l_n200_v1 · adjudicated filed0.985200$0.000875 · 155 tokp50 1,881 · p95 2,411 ms—leader
ag_openai_m_n200_v1 · adjudicated filed0.980200$0.000636 · 100 tokp50 1,079 · p95 1,380 ms—test vs leader
ag_openai_l_n200_v1 · adjudicated filed recommended0.975200$0.000244 · 100 tokp50 1,153 · p95 1,626 ms—test vs leader
ag_anthropic_m_n200_v1 · adjudicated filed0.965200$0.000351 · 155 tokp50 1,884 · p95 2,224 ms—test vs leader
ag_llama_m_n200_v1 · adjudicated filed0.965200$0.000020 · 106 tokp50 682 · p95 3,527 ms—test vs leader
ag_gemma_s_n200_v1 · adjudicated filed0.960200$0.000011 · 103 tokp50 561 · p95 1,396 ms—test vs leader
ag_qwen_m_n200_v1 · adjudicated filed0.960200$0.000035 · 104 tokp50 905 · p95 1,176 ms—test vs leader
ag_qwen_s_n200_v1 · adjudicated filed0.960200$0.000018 · 117 tokp50 971 · p95 1,803 ms—test vs leader
ag_deepseek_l_n200_v1 · adjudicated filed0.955200$0.000057 · 96 tokp50 1,242 · p95 2,329 ms—test vs leader
ag_gemma_l_n200_v1 · adjudicated filed0.955200$0.000014 · 106 tokp50 519 · p95 1,749 ms—test vs leader
ag_qwen_l_n200_v1 · adjudicated filed0.955200$0.000082 · 100 tokp50 2,895 · p95 4,011 ms—test vs leader
ag_mistral_l_n200_v1 · adjudicated filed0.950200$0.000052 · 98 tokp50 595 · p95 56,011 ms—test vs leader
ag_glm_l_n200_v1 · adjudicated filed0.945200$0.000093 · 104 tokp50 1,322 · p95 3,893 ms—test vs leader
ag_glm_m_n200_v1 · adjudicated filed0.945200$0.000053 · 99 tokp50 1,130 · p95 5,862 ms—test vs leader
ag_mistral_m_n200_v1 · adjudicated filed0.945200$0.000048 · 110 tokp50 397 · p95 718 ms—test vs leader
ag_openai_s_n200_v1 · adjudicated filed0.945200$0.000095 · 100 tokp50 828 · p95 1,100 ms—test vs leader
ag_gemma_m_n200_v1 · adjudicated filed0.940200$0.000008 · 106 tokp50 523 · p95 1,164 ms—test vs leader
ag_glm_s_n200_v1 · adjudicated filed0.935200$0.000014 · 100 tokp50 1,084 · p95 1,946 ms—test vs leader
ag_mistral_s_n200_v1 · adjudicated filed0.930200$0.000010 · 98 tokp50 661 · p95 1,224 ms—test vs leader
ag_anthropic_s_n200_v1 · adjudicated filed0.925200$0.000127 · 109 tokp50 937 · p95 1,218 ms—test vs leader
ag_deepseek_m_n200_v1 · adjudicated filed0.925200$0.000022 · 96 tokp50 663 · p95 1,538 ms—test vs leader
ag_deepseek_s_n200_v1 · adjudicated filed0.925200$0.000007 · 96 tokp50 1,229 · p95 3,523 ms—test vs leader
ag_llama_l_n200_v1 · adjudicated filed0.925200$0.000021 · 100 tokp50 635 · p95 1,582 ms—test vs leader
ag_llama_s_n200_v1 · adjudicated filed0.920200$0.000013 · 100 tokp50 699 · p95 1,540 ms—test vs leader
ag_bert_base_fy_n200_v1 · adjudicated filed0.905200$0.000000p50 69 · p95 118 ms—test vs leader
ag_bert_base_ta_n200_v1 · adjudicated filed0.905200$0.000000p50 64 · p95 114 ms—test vs leader
ag_bert_mini_n200_v1 · adjudicated filed0.905200$0.000000p50 6.3 · p95 12 ms—test vs leader
ag_bart_mnli_n200_v1 · adjudicated filed0.755200$0.000000p50 1,286 · p95 2,403 ms—test vs leader
ag_keyword_n200_v1 · adjudicated filed0.670200$0.000000p50 0.038 · p95 0.089 ms—test vs leader
ag_constant_n200_v1 · adjudicated filed0.225200$0.000000p50 0.002 · p95 0.002 ms—test vs leader
$0.0000$0.0009cost (USD per item)0.9850.225ag_anthropic_l_n200_v1 · adjudicated: correct 0.985, cost 0.001ag_openai_m_n200_v1 · adjudicated: correct 0.980, cost 0.001ag_openai_l_n200_v1 · adjudicated: correct 0.975, cost 0.000ag_anthropic_m_n200_v1 · adjudicated: correct 0.965, cost 0.000ag_llama_m_n200_v1 · adjudicated: correct 0.965, cost 0.000ag_gemma_s_n200_v1 · adjudicated: correct 0.960, cost 0.000ag_qwen_m_n200_v1 · adjudicated: correct 0.960, cost 0.000ag_qwen_s_n200_v1 · adjudicated: correct 0.960, cost 0.000ag_deepseek_l_n200_v1 · adjudicated: correct 0.955, cost 0.000ag_gemma_l_n200_v1 · adjudicated: correct 0.955, cost 0.000ag_qwen_l_n200_v1 · adjudicated: correct 0.955, cost 0.000ag_mistral_l_n200_v1 · adjudicated: correct 0.950, cost 0.000ag_glm_l_n200_v1 · adjudicated: correct 0.945, cost 0.000ag_glm_m_n200_v1 · adjudicated: correct 0.945, cost 0.000ag_mistral_m_n200_v1 · adjudicated: correct 0.945, cost 0.000ag_openai_s_n200_v1 · adjudicated: correct 0.945, cost 0.000ag_gemma_m_n200_v1 · adjudicated: correct 0.940, cost 0.000ag_glm_s_n200_v1 · adjudicated: correct 0.935, cost 0.000ag_mistral_s_n200_v1 · adjudicated: correct 0.930, cost 0.000ag_anthropic_s_n200_v1 · adjudicated: correct 0.925, cost 0.000ag_deepseek_m_n200_v1 · adjudicated: correct 0.925, cost 0.000ag_deepseek_s_n200_v1 · adjudicated: correct 0.925, cost 0.000ag_llama_l_n200_v1 · adjudicated: correct 0.925, cost 0.000ag_llama_s_n200_v1 · adjudicated: correct 0.920, cost 0.000ag_bert_base_fy_n200_v1 · adjudicated: correct 0.905, cost 0.000ag_bert_base_ta_n200_v1 · adjudicated: correct 0.905, cost 0.000ag_bert_mini_n200_v1 · adjudicated: correct 0.905, cost 0.000ag_bart_mnli_n200_v1 · adjudicated: correct 0.755, cost 0.000ag_keyword_n200_v1 · adjudicated: correct 0.670, cost 0.000ag_constant_n200_v1 · adjudicated: correct 0.225, cost 0.000ag_anthropic_l_n200_v1 · adjudicatedag_openai_m_n200_v1 · adjudicatedag_openai_l_n200_v1 · adjudicatedag_llama_m_n200_v1 · adjudicatedag_gemma_s_n200_v1 · adjudicatedag_gemma_m_n200_v1 · adjudicatedag_deepseek_s_n200_v1 · adjudicatedag_bert_base_fy_n200_v1 · adjudicatedag_bert_base_ta_n200_v1 · adjudicatedag_bert_mini_n200_v1 · adjudicated
correct against cost (USD per item): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.00256011.2speed p95 (ms)0.9850.225ag_anthropic_l_n200_v1 · adjudicated: correct 0.985, speed 2411.0ag_openai_m_n200_v1 · adjudicated: correct 0.980, speed 1379.7ag_openai_l_n200_v1 · adjudicated: correct 0.975, speed 1626.2ag_anthropic_m_n200_v1 · adjudicated: correct 0.965, speed 2224.0ag_llama_m_n200_v1 · adjudicated: correct 0.965, speed 3527.2ag_gemma_s_n200_v1 · adjudicated: correct 0.960, speed 1396.0ag_qwen_m_n200_v1 · adjudicated: correct 0.960, speed 1175.5ag_qwen_s_n200_v1 · adjudicated: correct 0.960, speed 1802.8ag_deepseek_l_n200_v1 · adjudicated: correct 0.955, speed 2329.4ag_gemma_l_n200_v1 · adjudicated: correct 0.955, speed 1748.9ag_qwen_l_n200_v1 · adjudicated: correct 0.955, speed 4010.8ag_mistral_l_n200_v1 · adjudicated: correct 0.950, speed 56011.2ag_glm_l_n200_v1 · adjudicated: correct 0.945, speed 3893.3ag_glm_m_n200_v1 · adjudicated: correct 0.945, speed 5862.0ag_mistral_m_n200_v1 · adjudicated: correct 0.945, speed 717.7ag_openai_s_n200_v1 · adjudicated: correct 0.945, speed 1100.4ag_gemma_m_n200_v1 · adjudicated: correct 0.940, speed 1163.9ag_glm_s_n200_v1 · adjudicated: correct 0.935, speed 1945.5ag_mistral_s_n200_v1 · adjudicated: correct 0.930, speed 1223.8ag_anthropic_s_n200_v1 · adjudicated: correct 0.925, speed 1217.6ag_deepseek_m_n200_v1 · adjudicated: correct 0.925, speed 1537.9ag_deepseek_s_n200_v1 · adjudicated: correct 0.925, speed 3522.9ag_llama_l_n200_v1 · adjudicated: correct 0.925, speed 1582.2ag_llama_s_n200_v1 · adjudicated: correct 0.920, speed 1540.4ag_bert_base_fy_n200_v1 · adjudicated: correct 0.905, speed 118.2ag_bert_base_ta_n200_v1 · adjudicated: correct 0.905, speed 114.2ag_bert_mini_n200_v1 · adjudicated: correct 0.905, speed 11.9ag_bart_mnli_n200_v1 · adjudicated: correct 0.755, speed 2403.0ag_keyword_n200_v1 · adjudicated: correct 0.670, speed 0.089ag_constant_n200_v1 · adjudicated: correct 0.225, speed 0.002ag_anthropic_l_n200_v1 · adjudicatedag_openai_m_n200_v1 · adjudicatedag_qwen_m_n200_v1 · adjudicatedag_mistral_m_n200_v1 · adjudicatedag_bert_mini_n200_v1 · adjudicatedag_keyword_n200_v1 · adjudicatedag_constant_n200_v1 · adjudicated
correct against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.

05ag_news_20 · pilot

ag_news_20: 20 items · pilot

the pilot slice; all 20 of its items are in ag_news_200

references: gold/ag_news_20

No arm on this dataset records correct.