eval-harness

datasets

Every run, by dataset

686 arms on 16 datasets, from 267 configs. An arm is one config on one dataset scored one way, its repeats averaged per item, exactly as pair-test reads it: a config rescored against silver or by a judge is an arm of its own. Open a dataset to compare its arms in the inspector.

all time, 2026-09-25 to 2026-11-02

01datasets

group by

16 shown
datasettagsitemsarmsruns in periodruns per dayspend in periodreference setsgoalsbaselinelast run
orrery_triage_v1_labelled
capture orrery_2026_10, where labelled:yes, one per fingerprint
environment prodhf chipi-nbgd/orrery-triage@v1project orrerypurpose decidessource triage-fleet6533691$11.74gold/orrery_triage_v1_labelled, silver/orrery_triage_v1_labelled_hand, silver/orrery_triage_v1_labelled_outcomesTriage decisions on Orrery's error signalstriage_baseline_pro_v12026-11-02
cnn_dailymail_200—20015335$19.24gold/cnn_dailymail_200, silver/cnn_dailymail_200Short news summaries (CNN)cnn_bart_l_n200_v12026-11-02
ag_news_200—2006333$0.5808gold/ag_news_200, gold/ag_news_200_adj1News topic classification (AG News)ag_bert_mini_n200_v12026-10-25
summeval_100—1004141$7.33gold/summeval_100A scorer that agrees with experts on summaries (SummEval)se_gemma_l_n100_v12026-11-02
dbpedia_280—2802929$1.17gold/dbpedia_280Encyclopedia topic classification (DBpedia)db_bert_base_fy_n200_v12026-09-30
few_nerd_280—2802828$2.61gold/few_nerd_280Named-entity extraction (Few-NERD)fn_span_marker_n200_v12026-10-01
scifact_200—2002626$2.97gold/scifact_200Evidence retrieval for scientific claims (SciFact)sf_e5_base_n200_v12026-09-29
cnn_dailymail_dm200—20055$0.0000gold/cnn_dailymail_dm200Short news summaries (CNN)—2026-10-15
cnn_dailymail_20—2022$0.0000gold/cnn_dailymail_20Short news summaries (CNN)—2026-09-26
fleet_dedup_v1
capture fleet_dedup_2026_08
hf chipi-nbgd/signal-fleet-dedup@v1purpose regression-guardsource triage-fleet10022—silver/fleet_dedup_v1——2026-10-30
smoke_v1—511$0.0000silver/smoke_v1—demo_v12026-10-23
ag_news_20—2000—gold/ag_news_20News topic classification (AG News)——
dbpedia_56—5600—gold/dbpedia_56Encyclopedia topic classification (DBpedia)——
few_nerd_56—5600—gold/few_nerd_56Named-entity extraction (Few-NERD)——
orrery_triage_smoke
capture orrery_2026_10, where labelled:yes, where evidence_level:exact, one per fingerprint
purpose smokesource triage-fleet200—gold/orrery_triage_smoke———
scifact_40—4000—gold/scifact_40Evidence retrieval for scientific claims (SciFact)——