eval-harness

metric

fn

better
neutral: a description
kind
descriptive: says what the output was, never whether it was good
decides
no goal
weighed by
no goal
recorded on
few_nerd_280

01what it measures, and how

Typed false negatives: gold entities no prediction matched.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"fn": {"summary": "…", "details": "…"}}.

02on few_nerd_280

gold/few_nerd_280fn_nothing_n200_v1: 2.736fn_capitalized_n200_v1: 2.118fn_gliner_n200_v1: 1.189fn_glm_s_n200_v1: 1.154fn_glm_l_n200_v1: 1.089fn_anthropic_s_n200_v1: 1.007fn_mistral_s_n200_v1: 0.989fn_llama_s_n200_v1: 0.971fn_qwen_l_n200_v1: 0.95fn_glm_m_n200_v1: 0.936fn_deepseek_s_n200_v1: 0.904fn_llama_l_n200_v1: 0.896fn_mistral_m_n200_v1: 0.896fn_deepseek_m_n200_v1: 0.875fn_gemma_s_n200_v1: 0.875fn_llama_m_n200_v1: 0.864fn_anthropic_m_n200_v1: 0.861fn_mistral_l_n200_v1: 0.854fn_gemma_l_n200_v1: 0.836fn_deepseek_l_n200_v1: 0.793fn_gemma_m_n200_v1: 0.793fn_openai_s_n200_v1: 0.789fn_qwen_m_n200_v1: 0.771fn_qwen_s_n200_v1: 0.771fn_openai_m_n200_v1: 0.746fn_anthropic_l_n200_v1: 0.729fn_openai_l_n200_v1: 0.718fn_span_marker_n200_v1: 0.5390.539better · 2.736

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean fnitemsscored against
fn_nothing_n200_v12.736280gold/few_nerd_280 gold
fn_capitalized_n200_v12.118280gold/few_nerd_280 gold
fn_gliner_n200_v11.189280gold/few_nerd_280 gold
fn_glm_s_n200_v11.154280gold/few_nerd_280 gold
fn_glm_l_n200_v11.089280gold/few_nerd_280 gold
fn_anthropic_s_n200_v11.007280gold/few_nerd_280 gold
fn_mistral_s_n200_v10.989280gold/few_nerd_280 gold
fn_llama_s_n200_v10.971280gold/few_nerd_280 gold
fn_qwen_l_n200_v10.950280gold/few_nerd_280 gold
fn_glm_m_n200_v10.936280gold/few_nerd_280 gold
fn_deepseek_s_n200_v10.904280gold/few_nerd_280 gold
fn_llama_l_n200_v10.896280gold/few_nerd_280 gold
fn_mistral_m_n200_v10.896280gold/few_nerd_280 gold
fn_deepseek_m_n200_v10.875280gold/few_nerd_280 gold
fn_gemma_s_n200_v10.875280gold/few_nerd_280 gold
fn_llama_m_n200_v10.864280gold/few_nerd_280 gold
fn_anthropic_m_n200_v10.861280gold/few_nerd_280 gold
fn_mistral_l_n200_v10.854280gold/few_nerd_280 gold
fn_gemma_l_n200_v10.836280gold/few_nerd_280 gold
fn_deepseek_l_n200_v10.793280gold/few_nerd_280 gold
fn_gemma_m_n200_v10.793280gold/few_nerd_280 gold
fn_openai_s_n200_v10.789280gold/few_nerd_280 gold
fn_qwen_m_n200_v10.771280gold/few_nerd_280 gold
fn_qwen_s_n200_v10.771280gold/few_nerd_280 gold
fn_openai_m_n200_v10.746280gold/few_nerd_280 gold
fn_anthropic_l_n200_v10.729280gold/few_nerd_280 gold
fn_openai_l_n200_v10.718280gold/few_nerd_280 gold
fn_span_marker_n200_v10.539280gold/few_nerd_280 gold