eval-harness

goal

Named-entity extraction (Few-NERD)

Pick an extractor that finds the named entities in a sentence and types them into eight coarse types.

01what it is for

decision
Which extractors can be told apart on entity F1, and at what cost.
audience
a pipeline that indexes people, places and organisations mentioned in text
decides
f1, higher is better
weighs
quality (f1); cost per item; speed per item
strategy
balanced: contenders: not significantly different from the leader by the paired test AND within the quality margin (default)
tie-break
cost, speed
output contract
entity_list
status
active
guardrails
  • what was not measured: Labelling and training cost, an LLM given definitions or examples, and contamination of the hosted models were not measured. (research/REPORT_NER.md, executive summary)
links

02the answer so far

few_nerd_280 · baseline promoted 2026-10-25 · scored against gold

fn_span_marker_n200_v1

span_marker (0.7674, $0, 499 MB on a CPU) differs significantly from all 27 other arms: with labelled data in a fixed taxonomy, fine-tune a small tagger. REPORT_NER, executive summary. Promoted with --allow-dirty: this rescore was written from a dirty tree on 2026-10-01; on 2026-10-25 today's clean code re-scored it from its stored outputs identically, item by item.

deploy sheet: what production must match
measured by
fn_span_marker_n200_v1_20260928T070856Z on few_nerd_280
against
gold/few_nerd_280
model sent
guishe/span-marker-generic-ner-v1-fewnerd-fine-super
model record
guishe-span-marker-generic-ner-v1-fewnerd-fine-super
server
not recorded
build
f836dfa9b6b0e6a6379607a8e35166103ac41aee
settings sent
coarsen: true
device: cpu
model: guishe/span-marker-generic-ner-v1-fewnerd-fine-super
precision: fp32
provider: span_marker
not sent
bookkeeping: family, price_tier
declared
deployment.coarsen: True
deployment.labels: ['art', 'building', 'event', 'location', 'organization', 'other', 'person', 'product']
deployment.normalizer_sha256: a7c22b054099a72f91d772754dcd49b748fa739340ea4f38f2934eb18950a942
deployment.parser_sha256: 101110f889f26b800af7eacbd887bab82aef1556713b869ab27d86f78a605181
deployment.provider: span_marker
deployment.weight_files: ['model.safetensors:499028932']
adapter
examples/ner-few-nerd/adapter.py d147da551073
harness
b3cc5d861cf2

No capture here says what production ran (the adapter's production_ran).

Command: evh deploy-sheet --config fn_span_marker_n200_v1 [--capture C]

inspect it

few_nerd_56

No decision yet. When the tests separate an arm worth keeping, promote its run with the reason they give: evh run-promote --run <run_id> --reason "…"

03how we got here: rounds

Rounds are optional: each one sets out to learn something, changes a few arms, drops what lost and says why, and ends in a conclusion. Decisions are made with evh round-open, round-retire and round-close; evh rounds says the same as this page. Open a round for what changed, what was retired and why, and what to do next.

r1 closed
span_marker (fine-tuned on Few-NERD's training split; 499 MB, CPU, $0) scores 0.7674 against 0.6864 for the best of 24 zero-shot hosted LLMs (openai_m) and differs significantly from all 27 other arms. With labelled data in a fixed taxonomy, fine-tune a small tagger; if you rent, take the cheapest of the arms openai_m is not significantly different from, gemma_m. (REPORT_NER, executive summary.)
Which of 24 hosted LLMs, a fine-tuned and a zero-shot span tagger and two non-learned baselines tags Few-NERD's eight coarse types best, and at what cost?
28 arms inspect
best fn_span_marker_n200_v1 0.767

04few_nerd_280 · test

few_nerd_280: 280 items · test

280 sentences

references: gold/few_nerd_280

Mean f1 over each arm's items, in groups by what each arm is scored against: numbers from different references, or from a judge, are different measurements and are never ranked together. A mean is not a verdict: the inspector tests each arm against its group's leader (Holm across them). Flagged arms are never the leader. filed: the arm's config names this goal (goal:); an arm without it is listed because it records the measure.

scored against gold/few_nerd_280 gold

Test every arm in this group against its leader

Recommended: fn_span_marker_n200_v1

  • Strategy balanced. Quality leader: fn_span_marker_n200_v1 (f1 0.7674). Contenders beside it (not significantly different by the paired test and within 0.01535 (2% of the leader)): none.
  • Recommended: fn_span_marker_n200_v1.

Strategy balanced (contenders: not significantly different from the leader by the paired test AND within the quality margin (default)). Weighs quality (f1); cost per item; speed per item. Among the contenders on quality, the tie-break is cost, speed. The rule recommends; promoting is evh run-promote; the goal chooses its strategy (strategy:).

armmean f1itemscost per itemspeed per itemflags
fn_span_marker_n200_v1 filed baseline recommended0.767280$0.000000p50 630 · p95 1,217 ms—leader
fn_openai_m_n200_v1 filed0.686280$0.002324 · 174 tokp50 1,490 · p95 2,322 ms—test vs leader
fn_anthropic_l_n200_v1 filed0.680280$0.002463 · 233 tokp50 2,651 · p95 4,020 ms—test vs leader
fn_gemma_m_n200_v1 filed0.673280$0.000032 · 203 tokp50 1,759 · p95 10,254 ms—test vs leader
fn_openai_l_n200_v1 filed0.672280$0.000612 · 154 tokp50 1,804 · p95 2,890 ms—test vs leader
fn_anthropic_m_n200_v1 filed0.666280$0.001016 · 236 tokp50 2,511 · p95 3,392 ms—test vs leader
fn_qwen_s_n200_v1 filed0.665280$0.000056 · 213 tokp50 1,893 · p95 3,747 ms—test vs leader
fn_qwen_m_n200_v1 filed0.655280$0.000142 · 204 tokp50 2,094 · p95 4,116 ms—test vs leader
fn_gemma_l_n200_v1 filed0.648280$0.000048 · 206 tokp50 2,458 · p95 7,003 ms—test vs leader
fn_qwen_l_n200_v1 filed0.624280$0.000258 · 161 tokp50 1,548 · p95 2,671 ms—test vs leader
fn_deepseek_m_n200_v1 filed0.615280$0.000071 · 172 tokp50 919 · p95 3,161 ms—test vs leader
fn_llama_l_n200_v1 filed0.599280$0.000063 · 176 tokp50 3,713 · p95 12,665 ms—test vs leader
fn_mistral_l_n200_v1 filed0.598280$0.000150 · 176 tokp50 1,548 · p95 57,451 ms—test vs leader
fn_openai_s_n200_v1 filed0.595280$0.000280 · 158 tokp50 1,121 · p95 1,571 ms—test vs leader
fn_glm_m_n200_v1 filed0.594280$0.000159 · 177 tokp50 2,167 · p95 7,803 ms—test vs leader
fn_mistral_m_n200_v1 filed0.591280$0.000165 · 183 tokp50 739 · p95 1,294 ms—test vs leader
fn_deepseek_l_n200_v1 filed0.587280$0.000305 · 180 tokp50 3,036 · p95 7,752 ms—test vs leader
fn_anthropic_s_n200_v1 filed0.584280$0.000554 · 210 tokp50 1,241 · p95 1,902 ms—test vs leader
fn_deepseek_s_n200_v1 filed0.576280$0.000032 · 176 tokp50 1,782 · p95 5,277 ms—test vs leader
fn_llama_m_n200_v1 filed0.574280$0.000069 · 207 tokp50 3,203 · p95 15,183 ms—test vs leader
fn_llama_s_n200_v1 filed0.569280$0.000037 · 185 tokp50 2,354 · p95 5,653 ms—test vs leader
fn_glm_l_n200_v1 filed0.566280$0.000365 · 252 tokp50 2,096 · p95 8,249 ms—test vs leader
fn_glm_s_n200_v1 filed0.553280$0.000069 · 184 tokp50 1,966 · p95 3,882 ms—test vs leader
fn_gemma_s_n200_v1 filed0.550280$0.000033 · 215 tokp50 3,792 · p95 13,451 ms—test vs leader
fn_mistral_s_n200_v1 filed0.524280$0.000027 · 182 tokp50 1,827 · p95 4,100 ms—test vs leader
fn_gliner_n200_v1 filed0.454280$0.000000p50 158 · p95 239 ms—test vs leader
fn_capitalized_n200_v1 filed0.191280$0.000000p50 0.032 · p95 0.066 ms—test vs leader
fn_nothing_n200_v1 filed0.125280$0.000000p50 0.002 · p95 0.004 ms—test vs leader
$0.0000$0.0025cost (USD per item)0.7670.125fn_span_marker_n200_v1: f1 0.767, cost 0.000fn_openai_m_n200_v1: f1 0.686, cost 0.002fn_anthropic_l_n200_v1: f1 0.680, cost 0.002fn_gemma_m_n200_v1: f1 0.673, cost 0.000fn_openai_l_n200_v1: f1 0.672, cost 0.001fn_anthropic_m_n200_v1: f1 0.666, cost 0.001fn_qwen_s_n200_v1: f1 0.665, cost 0.000fn_qwen_m_n200_v1: f1 0.655, cost 0.000fn_gemma_l_n200_v1: f1 0.648, cost 0.000fn_qwen_l_n200_v1: f1 0.624, cost 0.000fn_deepseek_m_n200_v1: f1 0.615, cost 0.000fn_llama_l_n200_v1: f1 0.599, cost 0.000fn_mistral_l_n200_v1: f1 0.598, cost 0.000fn_openai_s_n200_v1: f1 0.595, cost 0.000fn_glm_m_n200_v1: f1 0.594, cost 0.000fn_mistral_m_n200_v1: f1 0.591, cost 0.000fn_deepseek_l_n200_v1: f1 0.587, cost 0.000fn_anthropic_s_n200_v1: f1 0.584, cost 0.001fn_deepseek_s_n200_v1: f1 0.576, cost 0.000fn_llama_m_n200_v1: f1 0.574, cost 0.000fn_llama_s_n200_v1: f1 0.569, cost 0.000fn_glm_l_n200_v1: f1 0.566, cost 0.000fn_glm_s_n200_v1: f1 0.553, cost 0.000fn_gemma_s_n200_v1: f1 0.550, cost 0.000fn_mistral_s_n200_v1: f1 0.524, cost 0.000fn_gliner_n200_v1: f1 0.454, cost 0.000fn_capitalized_n200_v1: f1 0.191, cost 0.000fn_nothing_n200_v1: f1 0.125, cost 0.000fn_span_marker_n200_v1
f1 against cost (USD per item): better is left top. Named and joined: the frontier, the arms no other arm beats on both.
0.00457450.5speed p95 (ms)0.7670.125fn_span_marker_n200_v1: f1 0.767, speed 1216.7fn_openai_m_n200_v1: f1 0.686, speed 2321.6fn_anthropic_l_n200_v1: f1 0.680, speed 4020.2fn_gemma_m_n200_v1: f1 0.673, speed 10254.5fn_openai_l_n200_v1: f1 0.672, speed 2890.1fn_anthropic_m_n200_v1: f1 0.666, speed 3392.4fn_qwen_s_n200_v1: f1 0.665, speed 3746.6fn_qwen_m_n200_v1: f1 0.655, speed 4116.2fn_gemma_l_n200_v1: f1 0.648, speed 7003.0fn_qwen_l_n200_v1: f1 0.624, speed 2670.8fn_deepseek_m_n200_v1: f1 0.615, speed 3161.3fn_llama_l_n200_v1: f1 0.599, speed 12665.1fn_mistral_l_n200_v1: f1 0.598, speed 57450.5fn_openai_s_n200_v1: f1 0.595, speed 1571.4fn_glm_m_n200_v1: f1 0.594, speed 7802.9fn_mistral_m_n200_v1: f1 0.591, speed 1293.9fn_deepseek_l_n200_v1: f1 0.587, speed 7751.9fn_anthropic_s_n200_v1: f1 0.584, speed 1901.8fn_deepseek_s_n200_v1: f1 0.576, speed 5276.7fn_llama_m_n200_v1: f1 0.574, speed 15183.2fn_llama_s_n200_v1: f1 0.569, speed 5652.6fn_glm_l_n200_v1: f1 0.566, speed 8248.7fn_glm_s_n200_v1: f1 0.553, speed 3881.7fn_gemma_s_n200_v1: f1 0.550, speed 13451.2fn_mistral_s_n200_v1: f1 0.524, speed 4100.1fn_gliner_n200_v1: f1 0.454, speed 238.7fn_capitalized_n200_v1: f1 0.191, speed 0.066fn_nothing_n200_v1: f1 0.125, speed 0.004fn_span_marker_n200_v1fn_gliner_n200_v1fn_capitalized_n200_v1fn_nothing_n200_v1
f1 against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.

05few_nerd_56 · pilot

few_nerd_56: 56 items · pilot

the pilot slice; all 56 of its items are in few_nerd_280

references: gold/few_nerd_56

No arm on this dataset records f1.