eval-harness

metric

untyped_tp

better
neutral: a description
kind
descriptive: says what the output was, never whether it was good
decides
no goal
weighed by
no goal
recorded on
few_nerd_280

01what it measures, and how

True positives on texts alone, types ignored.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"untyped_tp": {"summary": "…", "details": "…"}}.

02on few_nerd_280

gold/few_nerd_280fn_span_marker_n200_v1: 2.375fn_anthropic_l_n200_v1: 2.3fn_gemma_s_n200_v1: 2.3fn_openai_l_n200_v1: 2.289fn_deepseek_l_n200_v1: 2.282fn_deepseek_m_n200_v1: 2.279fn_gemma_l_n200_v1: 2.268fn_qwen_m_n200_v1: 2.268fn_openai_m_n200_v1: 2.264fn_gemma_m_n200_v1: 2.254fn_deepseek_s_n200_v1: 2.25fn_glm_m_n200_v1: 2.25fn_qwen_s_n200_v1: 2.243fn_llama_l_n200_v1: 2.232fn_anthropic_s_n200_v1: 2.221fn_openai_s_n200_v1: 2.211fn_mistral_l_n200_v1: 2.189fn_mistral_s_n200_v1: 2.189fn_llama_m_n200_v1: 2.182fn_anthropic_m_n200_v1: 2.179fn_llama_s_n200_v1: 2.15fn_mistral_m_n200_v1: 2.146fn_qwen_l_n200_v1: 2.114fn_capitalized_n200_v1: 2.107fn_glm_s_n200_v1: 2.039fn_glm_l_n200_v1: 1.975fn_gliner_n200_v1: 1.829fn_nothing_n200_v1: 00better · 2.375

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean untyped_tpitemsscored against
fn_span_marker_n200_v12.375280gold/few_nerd_280 gold
fn_anthropic_l_n200_v12.300280gold/few_nerd_280 gold
fn_gemma_s_n200_v12.300280gold/few_nerd_280 gold
fn_openai_l_n200_v12.289280gold/few_nerd_280 gold
fn_deepseek_l_n200_v12.282280gold/few_nerd_280 gold
fn_deepseek_m_n200_v12.279280gold/few_nerd_280 gold
fn_gemma_l_n200_v12.268280gold/few_nerd_280 gold
fn_qwen_m_n200_v12.268280gold/few_nerd_280 gold
fn_openai_m_n200_v12.264280gold/few_nerd_280 gold
fn_gemma_m_n200_v12.254280gold/few_nerd_280 gold
fn_deepseek_s_n200_v12.250280gold/few_nerd_280 gold
fn_glm_m_n200_v12.250280gold/few_nerd_280 gold
fn_qwen_s_n200_v12.243280gold/few_nerd_280 gold
fn_llama_l_n200_v12.232280gold/few_nerd_280 gold
fn_anthropic_s_n200_v12.221280gold/few_nerd_280 gold
fn_openai_s_n200_v12.211280gold/few_nerd_280 gold
fn_mistral_l_n200_v12.189280gold/few_nerd_280 gold
fn_mistral_s_n200_v12.189280gold/few_nerd_280 gold
fn_llama_m_n200_v12.182280gold/few_nerd_280 gold
fn_anthropic_m_n200_v12.179280gold/few_nerd_280 gold
fn_llama_s_n200_v12.150280gold/few_nerd_280 gold
fn_mistral_m_n200_v12.146280gold/few_nerd_280 gold
fn_qwen_l_n200_v12.114280gold/few_nerd_280 gold
fn_capitalized_n200_v12.107280gold/few_nerd_280 gold
fn_glm_s_n200_v12.039280gold/few_nerd_280 gold
fn_glm_l_n200_v11.975280gold/few_nerd_280 gold
fn_gliner_n200_v11.829280gold/few_nerd_280 gold
fn_nothing_n200_v10.000280gold/few_nerd_280 gold