eval-harness

metric

fp

better
neutral: a description
kind
descriptive: says what the output was, never whether it was good
decides
no goal
weighed by
no goal
recorded on
few_nerd_280

01what it measures, and how

Typed false positives: predicted entities that matched no gold one.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"fp": {"summary": "…", "details": "…"}}.

02on few_nerd_280

gold/few_nerd_280fn_capitalized_n200_v1: 3.114fn_mistral_s_n200_v1: 1.914fn_gemma_s_n200_v1: 1.868fn_gliner_n200_v1: 1.679fn_deepseek_s_n200_v1: 1.579fn_deepseek_m_n200_v1: 1.525fn_llama_l_n200_v1: 1.521fn_llama_s_n200_v1: 1.468fn_deepseek_l_n200_v1: 1.461fn_glm_m_n200_v1: 1.461fn_openai_s_n200_v1: 1.411fn_mistral_m_n200_v1: 1.389fn_anthropic_s_n200_v1: 1.371fn_glm_s_n200_v1: 1.343fn_llama_m_n200_v1: 1.25fn_qwen_l_n200_v1: 1.214fn_mistral_l_n200_v1: 1.211fn_qwen_m_n200_v1: 1.182fn_glm_l_n200_v1: 1.15fn_qwen_s_n200_v1: 1.05fn_gemma_l_n200_v1: 1.032fn_openai_l_n200_v1: 1.007fn_anthropic_l_n200_v1: 0.961fn_openai_m_n200_v1: 0.954fn_gemma_m_n200_v1: 0.936fn_anthropic_m_n200_v1: 0.868fn_span_marker_n200_v1: 0.593fn_nothing_n200_v1: 00better · 3.114

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean fpitemsscored against
fn_capitalized_n200_v13.114280gold/few_nerd_280 gold
fn_mistral_s_n200_v11.914280gold/few_nerd_280 gold
fn_gemma_s_n200_v11.868280gold/few_nerd_280 gold
fn_gliner_n200_v11.679280gold/few_nerd_280 gold
fn_deepseek_s_n200_v11.579280gold/few_nerd_280 gold
fn_deepseek_m_n200_v11.525280gold/few_nerd_280 gold
fn_llama_l_n200_v11.521280gold/few_nerd_280 gold
fn_llama_s_n200_v11.468280gold/few_nerd_280 gold
fn_deepseek_l_n200_v11.461280gold/few_nerd_280 gold
fn_glm_m_n200_v11.461280gold/few_nerd_280 gold
fn_openai_s_n200_v11.411280gold/few_nerd_280 gold
fn_mistral_m_n200_v11.389280gold/few_nerd_280 gold
fn_anthropic_s_n200_v11.371280gold/few_nerd_280 gold
fn_glm_s_n200_v11.343280gold/few_nerd_280 gold
fn_llama_m_n200_v11.250280gold/few_nerd_280 gold
fn_qwen_l_n200_v11.214280gold/few_nerd_280 gold
fn_mistral_l_n200_v11.211280gold/few_nerd_280 gold
fn_qwen_m_n200_v11.182280gold/few_nerd_280 gold
fn_glm_l_n200_v11.150280gold/few_nerd_280 gold
fn_qwen_s_n200_v11.050280gold/few_nerd_280 gold
fn_gemma_l_n200_v11.032280gold/few_nerd_280 gold
fn_openai_l_n200_v11.007280gold/few_nerd_280 gold
fn_anthropic_l_n200_v10.961280gold/few_nerd_280 gold
fn_openai_m_n200_v10.954280gold/few_nerd_280 gold
fn_gemma_m_n200_v10.936280gold/few_nerd_280 gold
fn_anthropic_m_n200_v10.868280gold/few_nerd_280 gold
fn_span_marker_n200_v10.593280gold/few_nerd_280 gold
fn_nothing_n200_v10.000280gold/few_nerd_280 gold