eval-harness

metric

untyped_fp

better
neutral: a description
kind
descriptive: says what the output was, never whether it was good
decides
no goal
weighed by
no goal
recorded on
few_nerd_280

01what it measures, and how

False positives on texts alone, types ignored.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"untyped_fp": {"summary": "…", "details": "…"}}.

02on few_nerd_280

gold/few_nerd_280fn_capitalized_n200_v1: 1.625fn_mistral_s_n200_v1: 1.471fn_gemma_s_n200_v1: 1.429fn_gliner_n200_v1: 1.396fn_deepseek_s_n200_v1: 1.161fn_openai_s_n200_v1: 1.146fn_llama_l_n200_v1: 1.129fn_deepseek_l_n200_v1: 1.121fn_deepseek_m_n200_v1: 1.107fn_llama_s_n200_v1: 1.082fn_mistral_m_n200_v1: 1.082fn_glm_m_n200_v1: 1.011fn_llama_m_n200_v1: 0.939fn_mistral_l_n200_v1: 0.904fn_glm_s_n200_v1: 0.886fn_qwen_l_n200_v1: 0.886fn_anthropic_s_n200_v1: 0.879fn_qwen_m_n200_v1: 0.879fn_glm_l_n200_v1: 0.821fn_qwen_s_n200_v1: 0.771fn_openai_l_n200_v1: 0.736fn_openai_m_n200_v1: 0.679fn_anthropic_l_n200_v1: 0.668fn_gemma_l_n200_v1: 0.664fn_gemma_m_n200_v1: 0.625fn_anthropic_m_n200_v1: 0.564fn_span_marker_n200_v1: 0.414fn_nothing_n200_v1: 00better · 1.625

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean untyped_fpitemsscored against
fn_capitalized_n200_v11.625280gold/few_nerd_280 gold
fn_mistral_s_n200_v11.471280gold/few_nerd_280 gold
fn_gemma_s_n200_v11.429280gold/few_nerd_280 gold
fn_gliner_n200_v11.396280gold/few_nerd_280 gold
fn_deepseek_s_n200_v11.161280gold/few_nerd_280 gold
fn_openai_s_n200_v11.146280gold/few_nerd_280 gold
fn_llama_l_n200_v11.129280gold/few_nerd_280 gold
fn_deepseek_l_n200_v11.121280gold/few_nerd_280 gold
fn_deepseek_m_n200_v11.107280gold/few_nerd_280 gold
fn_llama_s_n200_v11.082280gold/few_nerd_280 gold
fn_mistral_m_n200_v11.082280gold/few_nerd_280 gold
fn_glm_m_n200_v11.011280gold/few_nerd_280 gold
fn_llama_m_n200_v10.939280gold/few_nerd_280 gold
fn_mistral_l_n200_v10.904280gold/few_nerd_280 gold
fn_glm_s_n200_v10.886280gold/few_nerd_280 gold
fn_qwen_l_n200_v10.886280gold/few_nerd_280 gold
fn_anthropic_s_n200_v10.879280gold/few_nerd_280 gold
fn_qwen_m_n200_v10.879280gold/few_nerd_280 gold
fn_glm_l_n200_v10.821280gold/few_nerd_280 gold
fn_qwen_s_n200_v10.771280gold/few_nerd_280 gold
fn_openai_l_n200_v10.736280gold/few_nerd_280 gold
fn_openai_m_n200_v10.679280gold/few_nerd_280 gold
fn_anthropic_l_n200_v10.668280gold/few_nerd_280 gold
fn_gemma_l_n200_v10.664280gold/few_nerd_280 gold
fn_gemma_m_n200_v10.625280gold/few_nerd_280 gold
fn_anthropic_m_n200_v10.564280gold/few_nerd_280 gold
fn_span_marker_n200_v10.414280gold/few_nerd_280 gold
fn_nothing_n200_v10.000280gold/few_nerd_280 gold