eval-harness

dataset

few_nerd_280

280 items · 28 arms. Headline: each arm's own primary metric, mean over its items; arms with different primary metrics are not ranked against each other.

baseline, promoted 2026-10-25

fn_span_marker_n200_v1

span_marker (0.7674, $0, 499 MB on a CPU) differs significantly from all 27 other arms: with labelled data in a fixed taxonomy, fine-tune a small tagger. REPORT_NER, executive summary. Promoted with --allow-dirty: this rescore was written from a dirty tree on 2026-10-01; on 2026-10-25 today's clean code re-scored it from its stored outputs identically, item by item.

Inspect the 28 arms on f1, scored against gold/few_nerd_280Open its sheet

goals: Named-entity extraction (Few-NERD)

01arms

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

28 shown
armmetricmeanitemscost per itemspeed per itemrunsscored against
fn_span_marker_n200_v1 baselinef10.767280$0.000000p50 630 · p95 1,217 ms1gold/few_nerd_280 goldinspect
fn_openai_m_n200_v1f10.686280$0.002324 · 174 tokp50 1,490 · p95 2,322 ms1gold/few_nerd_280 goldinspect
fn_anthropic_l_n200_v1f10.680280$0.002463 · 233 tokp50 2,651 · p95 4,020 ms1gold/few_nerd_280 goldinspect
fn_gemma_m_n200_v1f10.673280$0.000032 · 203 tokp50 1,759 · p95 10,254 ms1gold/few_nerd_280 goldinspect
fn_openai_l_n200_v1f10.672280$0.000612 · 154 tokp50 1,804 · p95 2,890 ms1gold/few_nerd_280 goldinspect
fn_anthropic_m_n200_v1f10.666280$0.001016 · 236 tokp50 2,511 · p95 3,392 ms1gold/few_nerd_280 goldinspect
fn_qwen_s_n200_v1f10.665280$0.000056 · 213 tokp50 1,893 · p95 3,747 ms1gold/few_nerd_280 goldinspect
fn_qwen_m_n200_v1f10.655280$0.000142 · 204 tokp50 2,094 · p95 4,116 ms1gold/few_nerd_280 goldinspect
fn_gemma_l_n200_v1f10.648280$0.000048 · 206 tokp50 2,458 · p95 7,003 ms1gold/few_nerd_280 goldinspect
fn_qwen_l_n200_v1f10.624280$0.000258 · 161 tokp50 1,548 · p95 2,671 ms1gold/few_nerd_280 goldinspect
fn_deepseek_m_n200_v1f10.615280$0.000071 · 172 tokp50 919 · p95 3,161 ms1gold/few_nerd_280 goldinspect
fn_llama_l_n200_v1f10.599280$0.000063 · 176 tokp50 3,713 · p95 12,665 ms1gold/few_nerd_280 goldinspect
fn_mistral_l_n200_v1f10.598280$0.000150 · 176 tokp50 1,548 · p95 57,451 ms1gold/few_nerd_280 goldinspect
fn_openai_s_n200_v1f10.595280$0.000280 · 158 tokp50 1,121 · p95 1,571 ms1gold/few_nerd_280 goldinspect
fn_glm_m_n200_v1f10.594280$0.000159 · 177 tokp50 2,167 · p95 7,803 ms1gold/few_nerd_280 goldinspect
fn_mistral_m_n200_v1f10.591280$0.000165 · 183 tokp50 739 · p95 1,294 ms1gold/few_nerd_280 goldinspect
fn_deepseek_l_n200_v1f10.587280$0.000305 · 180 tokp50 3,036 · p95 7,752 ms1gold/few_nerd_280 goldinspect
fn_anthropic_s_n200_v1f10.584280$0.000554 · 210 tokp50 1,241 · p95 1,902 ms1gold/few_nerd_280 goldinspect
fn_deepseek_s_n200_v1f10.576280$0.000032 · 176 tokp50 1,782 · p95 5,277 ms1gold/few_nerd_280 goldinspect
fn_llama_m_n200_v1f10.574280$0.000069 · 207 tokp50 3,203 · p95 15,183 ms1gold/few_nerd_280 goldinspect
fn_llama_s_n200_v1f10.569280$0.000037 · 185 tokp50 2,354 · p95 5,653 ms1gold/few_nerd_280 goldinspect
fn_glm_l_n200_v1f10.566280$0.000365 · 252 tokp50 2,096 · p95 8,249 ms1gold/few_nerd_280 goldinspect
fn_glm_s_n200_v1f10.553280$0.000069 · 184 tokp50 1,966 · p95 3,882 ms1gold/few_nerd_280 goldinspect
fn_gemma_s_n200_v1f10.550280$0.000033 · 215 tokp50 3,792 · p95 13,451 ms1gold/few_nerd_280 goldinspect
fn_mistral_s_n200_v1f10.524280$0.000027 · 182 tokp50 1,827 · p95 4,100 ms1gold/few_nerd_280 goldinspect
fn_gliner_n200_v1f10.454280$0.000000p50 158 · p95 239 ms1gold/few_nerd_280 goldinspect
fn_capitalized_n200_v1f10.191280$0.000000p50 0.032 · p95 0.066 ms1gold/few_nerd_280 goldinspect
fn_nothing_n200_v1f10.125280$0.000000p50 0.002 · p95 0.004 ms1gold/few_nerd_280 goldinspect

02what's inside

280 items, text, 5–99 words (median 23, middle half 16–33).

Source: DFKI-SLT/few-nerd (test split). The text stays there; each item below links to its row.

01002005–14 words: 5015–24 words: 10825–34 words: 6435–44 words: 3645–54 words: 1455–64 words: 465–74 words: 175–84 words: 295–104 words: 15–1415–2425–3435–4445–5455–6465–7475–8485–9495–104

the gold

gold/few_nerd_280: a list per item, 0–16 entries, median 2

280 shown
itemwordsgoldopen
0040170d9b30482 entriessource ↗ · inspect
009e011601e7322 entriessource ↗ · inspect
0137582f0bcf191 entriessource ↗ · inspect
0194f3cecd17180 entriessource ↗ · inspect
03c8cd8517ab321 entriessource ↗ · inspect
05053bf2e3ed261 entriessource ↗ · inspect
067318c51b8c141 entriessource ↗ · inspect
083ec69d6b6e234 entriessource ↗ · inspect
0989903e3330230 entriessource ↗ · inspect
0a78fa046ef5121 entriessource ↗ · inspect
0ad4d8989c16271 entriessource ↗ · inspect
0bb82665c509224 entriessource ↗ · inspect
0c903c6a3b98182 entriessource ↗ · inspect
0d9666f7f385230 entriessource ↗ · inspect
0dcec3fe6689233 entriessource ↗ · inspect
0e25cec61da2295 entriessource ↗ · inspect
0e88aabc5a04170 entriessource ↗ · inspect
0efd80adbf45111 entriessource ↗ · inspect
0f581f9ac261274 entriessource ↗ · inspect
0fd56d48992d50 entriessource ↗ · inspect
0fdb825dcd51223 entriessource ↗ · inspect
100f09d9ab92234 entriessource ↗ · inspect
114599c4db48272 entriessource ↗ · inspect
11e07d52a178141 entriessource ↗ · inspect
13d8a0d6cc6d342 entriessource ↗ · inspect
1428be85fb66241 entriessource ↗ · inspect
14f359df5982274 entriessource ↗ · inspect
154350ae6e7f334 entriessource ↗ · inspect
16e664a92aea190 entriessource ↗ · inspect
17160ad9c17570 entriessource ↗ · inspect
17656d31bd1b203 entriessource ↗ · inspect
184098570107112 entriessource ↗ · inspect
191fb6f80dc36210 entriessource ↗ · inspect
1985ab09043b162 entriessource ↗ · inspect
1996f8ce65e3365 entriessource ↗ · inspect
1aa19b265dcd71 entriessource ↗ · inspect
1c73e93d13a6284 entriessource ↗ · inspect
1d43e323929b7010 entriessource ↗ · inspect
1d54602b9f08293 entriessource ↗ · inspect
1e099a2582be172 entriessource ↗ · inspect
1e39a6ccdcf8314 entriessource ↗ · inspect
1e6e5439bc7a549 entriessource ↗ · inspect
1f007e7e7bc7362 entriessource ↗ · inspect
201c411a53a3222 entriessource ↗ · inspect
20985463eeaf190 entriessource ↗ · inspect
212f1acc404a211 entriessource ↗ · inspect
225a2c68e43e81 entriessource ↗ · inspect
245c51d60e1c122 entriessource ↗ · inspect
25559a78d027102 entriessource ↗ · inspect
259eccd9c019285 entriessource ↗ · inspect
25b1e07b4d6a376 entriessource ↗ · inspect
265bfdd6edbf191 entriessource ↗ · inspect
2708ce3cf54a162 entriessource ↗ · inspect
2a2d69db61de303 entriessource ↗ · inspect
2a41dc156fa9111 entriessource ↗ · inspect
2c36bfd000fa132 entriessource ↗ · inspect
2c45b98b4318273 entriessource ↗ · inspect
2cc020e83e55397 entriessource ↗ · inspect
2dcaef03824a222 entriessource ↗ · inspect
2ec56f793990152 entriessource ↗ · inspect
2f8accae25ed171 entriessource ↗ · inspect
2f8f4864df79438 entriessource ↗ · inspect
31ceaaed5c87142 entriessource ↗ · inspect
32b918b99570350 entriessource ↗ · inspect
32f8c5f654ce264 entriessource ↗ · inspect
3508a3a3b41d263 entriessource ↗ · inspect
353f0f464182111 entriessource ↗ · inspect
3aa2864f93c3484 entriessource ↗ · inspect
3abb3b9915ef242 entriessource ↗ · inspect
3e6e1ccfe1d8482 entriessource ↗ · inspect
3e7678086081403 entriessource ↗ · inspect
3eb07e5b98fd819 entriessource ↗ · inspect
3edbd33a357f163 entriessource ↗ · inspect
3f06d1f8c9e3130 entriessource ↗ · inspect
40ec1785b682172 entriessource ↗ · inspect
4307ddbc8248281 entriessource ↗ · inspect
44933125796d194 entriessource ↗ · inspect
45389d305dd9170 entriessource ↗ · inspect
45a806c770c26113 entriessource ↗ · inspect
47bbd512a5d9214 entriessource ↗ · inspect
488caab0d6cf182 entriessource ↗ · inspect
4924b1a7eb90515 entriessource ↗ · inspect
4925c6d7e0f9423 entriessource ↗ · inspect
4a0ddf96df1d161 entriessource ↗ · inspect
4a265ae7ba2d300 entriessource ↗ · inspect
4a6a00d4b489232 entriessource ↗ · inspect
4b3d74eeaaa1402 entriessource ↗ · inspect
4b6ab1b30460174 entriessource ↗ · inspect
4d67a87176ef253 entriessource ↗ · inspect
4d8d103fe230330 entriessource ↗ · inspect
52b5c23b52a8111 entriessource ↗ · inspect
5418a048089c324 entriessource ↗ · inspect
5436c6772746274 entriessource ↗ · inspect
5657509145c2401 entriessource ↗ · inspect
59ff807150be503 entriessource ↗ · inspect
5a648df06391220 entriessource ↗ · inspect
5cd2277018b6214 entriessource ↗ · inspect
5d3a33863fd551 entriessource ↗ · inspect
5d4b857b40dd3810 entriessource ↗ · inspect
5e02814ebfbf173 entriessource ↗ · inspect
5e14be5285ab200 entriessource ↗ · inspect
5efd0213d373152 entriessource ↗ · inspect
5f673f46db31204 entriessource ↗ · inspect
601026c0db12385 entriessource ↗ · inspect
60441076c08b202 entriessource ↗ · inspect
61d02cfeea14183 entriessource ↗ · inspect
628edb1d87c0204 entriessource ↗ · inspect
6358593ec938454 entriessource ↗ · inspect
63a55a54089c426 entriessource ↗ · inspect
63fe12cde22d132 entriessource ↗ · inspect
65b0430bce0b240 entriessource ↗ · inspect
6630795fd877374 entriessource ↗ · inspect
6639c198ec31102 entriessource ↗ · inspect
66b8394647b6435 entriessource ↗ · inspect
675c0e87cbd9331 entriessource ↗ · inspect
683c42353d58201 entriessource ↗ · inspect
68495a46e487164 entriessource ↗ · inspect
69497f3e63cd91 entriessource ↗ · inspect
6d642f8fe7e6346 entriessource ↗ · inspect
6fa496748157202 entriessource ↗ · inspect
6fd65c53c1c0433 entriessource ↗ · inspect
70ce5e72e409384 entriessource ↗ · inspect
70d7fbebada8572 entriessource ↗ · inspect
7134935983e3260 entriessource ↗ · inspect
73dddefd1b60334 entriessource ↗ · inspect
744ae5193538373 entriessource ↗ · inspect
7458c45a6274233 entriessource ↗ · inspect
748d91bcba2e253 entriessource ↗ · inspect
756f937ac1ca243 entriessource ↗ · inspect
767fd1265c5e163 entriessource ↗ · inspect
7779c6667d6d252 entriessource ↗ · inspect
78742e22f60b262 entriessource ↗ · inspect
78b771857c23234 entriessource ↗ · inspect
78e1761fe603262 entriessource ↗ · inspect
78ee7593820c162 entriessource ↗ · inspect
7904a3595023233 entriessource ↗ · inspect
7b5a12ac4834142 entriessource ↗ · inspect
7c08539ada08201 entriessource ↗ · inspect
7cf247ca97b3130 entriessource ↗ · inspect
7e634dad973c343 entriessource ↗ · inspect
7fecb639c51b455 entriessource ↗ · inspect
803ec09793e7459 entriessource ↗ · inspect
815027679a00342 entriessource ↗ · inspect
8660aa136df8408 entriessource ↗ · inspect
87024c1d50f0273 entriessource ↗ · inspect
87914135ea59182 entriessource ↗ · inspect
895c8a4b9789152 entriessource ↗ · inspect
8b2b5c6b09b9639 entriessource ↗ · inspect
8b7dd08ea443151 entriessource ↗ · inspect
8c2c8d0bda71274 entriessource ↗ · inspect
8d8ce4d2ac75183 entriessource ↗ · inspect
8e8454ea5075284 entriessource ↗ · inspect
8f0ac2c2956681 entriessource ↗ · inspect
90585b9cd0c7263 entriessource ↗ · inspect
907ca3986e16181 entriessource ↗ · inspect
923c51398600448 entriessource ↗ · inspect
93584819852f343 entriessource ↗ · inspect
94343f6ff0aa295 entriessource ↗ · inspect
96d4aa82ecda223 entriessource ↗ · inspect
96f1eb0567e7364 entriessource ↗ · inspect
975737891102274 entriessource ↗ · inspect
9785726b5c82192 entriessource ↗ · inspect
99550cb6459a142 entriessource ↗ · inspect
9c5d3f2ab860132 entriessource ↗ · inspect
9d55b2eabb8081 entriessource ↗ · inspect
9dfc589dc05d202 entriessource ↗ · inspect
9e1ac71e83f9436 entriessource ↗ · inspect
9e654509adb6120 entriessource ↗ · inspect
9e6f8f7c28a6152 entriessource ↗ · inspect
a111755e72cc111 entriessource ↗ · inspect
a1c67bbfe0ea312 entriessource ↗ · inspect
a35043bbe9ab242 entriessource ↗ · inspect
a3b05d9b5fd1192 entriessource ↗ · inspect
a40ed39cc49e180 entriessource ↗ · inspect
a53ff238af77405 entriessource ↗ · inspect
a565341e2cf6192 entriessource ↗ · inspect
a585127dfafe312 entriessource ↗ · inspect
a66963eb19ce243 entriessource ↗ · inspect
a66d86d02034325 entriessource ↗ · inspect
a68981540c14162 entriessource ↗ · inspect
a85af769c43f396 entriessource ↗ · inspect
a870a786978991 entriessource ↗ · inspect
aa6b05814dc6214 entriessource ↗ · inspect
aacf783d640e213 entriessource ↗ · inspect
ab3fc740bde8213 entriessource ↗ · inspect
aba9cc96a538233 entriessource ↗ · inspect
ac3383156b9f264 entriessource ↗ · inspect
acb8a8dd627a410 entriessource ↗ · inspect
afbe679fa70a383 entriessource ↗ · inspect
b0d96c600ed9436 entriessource ↗ · inspect
b1bd7568935c102 entriessource ↗ · inspect
b1dfb240c31f333 entriessource ↗ · inspect
b2283442b299313 entriessource ↗ · inspect
b36fbc0e97c5251 entriessource ↗ · inspect
b39bb44fcc9580 entriessource ↗ · inspect
b46d72a8d008132 entriessource ↗ · inspect
b7488593d27e142 entriessource ↗ · inspect
b8bc97b4949d263 entriessource ↗ · inspect
ba15dd01dff4182 entriessource ↗ · inspect
ba470c6c35f8211 entriessource ↗ · inspect
bae3817bf50e211 entriessource ↗ · inspect
bb6551b8be12315 entriessource ↗ · inspect
bdc7068e5022224 entriessource ↗ · inspect
be34c5ea1240101 entriessource ↗ · inspect
c12a0d572623536 entriessource ↗ · inspect
c14222f4ee92233 entriessource ↗ · inspect
c14f93ca2ac0181 entriessource ↗ · inspect
c4272731364590 entriessource ↗ · inspect
c463913bc497430 entriessource ↗ · inspect
c4c75211ccec120 entriessource ↗ · inspect
c4d889811a89483 entriessource ↗ · inspect
c51442586d8e154 entriessource ↗ · inspect
c5401edd1a73213 entriessource ↗ · inspect
c5bd0ae169a6233 entriessource ↗ · inspect
c652a25da2f9545 entriessource ↗ · inspect
c7d2d8aa9755233 entriessource ↗ · inspect
c96910312e39162 entriessource ↗ · inspect
c99546ed4a15141 entriessource ↗ · inspect
caf8b8d2760c264 entriessource ↗ · inspect
cc5e7845bf29112 entriessource ↗ · inspect
cdab9daa03c3111 entriessource ↗ · inspect
ce037b805aa6220 entriessource ↗ · inspect
ce3acb23131f320 entriessource ↗ · inspect
cfc67fcbf20b182 entriessource ↗ · inspect
d222a725821c182 entriessource ↗ · inspect
d27225312792142 entriessource ↗ · inspect
d2bb81825498418 entriessource ↗ · inspect
d2f88f44058d252 entriessource ↗ · inspect
d51f8dd7a677214 entriessource ↗ · inspect
d6606492e7a3183 entriessource ↗ · inspect
d7c5de16e532240 entriessource ↗ · inspect
d837cd062513211 entriessource ↗ · inspect
d8f98b8b5b95120 entriessource ↗ · inspect
d9836306134d446 entriessource ↗ · inspect
d9d3911536f2110 entriessource ↗ · inspect
dad6c2f564ac242 entriessource ↗ · inspect
dcc5b5ca25ad241 entriessource ↗ · inspect
de3b49d1b113282 entriessource ↗ · inspect
e115fab643e7222 entriessource ↗ · inspect
e122e5c79be3171 entriessource ↗ · inspect
e1da206d02a7424 entriessource ↗ · inspect
e2059038b2ec3916 entriessource ↗ · inspect
e23230083c87152 entriessource ↗ · inspect
e2931e27a24c342 entriessource ↗ · inspect
e2c081c836c5160 entriessource ↗ · inspect
e2e24815a6a4162 entriessource ↗ · inspect
e562fd65ba86273 entriessource ↗ · inspect
e68137f0b4aa162 entriessource ↗ · inspect
e6ff2af00600221 entriessource ↗ · inspect
e83f20b42aa0161 entriessource ↗ · inspect
e99d096e68ff181 entriessource ↗ · inspect
ed3a906df426994 entriessource ↗ · inspect
ee24c587d35871 entriessource ↗ · inspect
ee90f2323bf7122 entriessource ↗ · inspect
ee9dc43c47ec394 entriessource ↗ · inspect
eec80e631b98111 entriessource ↗ · inspect
eed6f7718603474 entriessource ↗ · inspect
ef205046da4e254 entriessource ↗ · inspect
efd76ea1d6ce101 entriessource ↗ · inspect
efe4bd647c5c354 entriessource ↗ · inspect
f095bc7697c2140 entriessource ↗ · inspect
f0adf6452ffa150 entriessource ↗ · inspect
f2930615ebb5131 entriessource ↗ · inspect
f33b03c8b4d3284 entriessource ↗ · inspect
f3d617de2179444 entriessource ↗ · inspect
f56f1708001a536 entriessource ↗ · inspect
f5708db0e156303 entriessource ↗ · inspect
f6ede995512b4310 entriessource ↗ · inspect
f7a4f59e82c3253 entriessource ↗ · inspect
f8003bcfe7f9343 entriessource ↗ · inspect
f950156bd03d223 entriessource ↗ · inspect
f9a45bf94f0a211 entriessource ↗ · inspect
fac5fa7ae1d1172 entriessource ↗ · inspect
fb746240b0e2838 entriessource ↗ · inspect
fc4e6e3dd004272 entriessource ↗ · inspect
fd597abec6c6120 entriessource ↗ · inspect
fdc5ee241a49341 entriessource ↗ · inspect
fefd35d0ca6a223 entriessource ↗ · inspect
ff03bcfa33ec264 entriessource ↗ · inspect
fff972f0193f193 entriessource ↗ · inspect

Each row is the profile evh dataset-items --dataset-id few_nerd_280 prints; inspect opens the item in the inspector with the dataset's largest group of comparable arms, and its text where the inspector runs on a machine that has it.

03reference sets

04labels

No labels yet. A sheet to fill: evh label-sheet --dataset-id few_nerd_280 --out sheet.csv; a run's outputs as a model set: evh label-from-run --run R --set S.