eval-harness

metric

n_predicted

better
neutral: a description
kind
descriptive: says what the output was, never whether it was good
decides
no goal
weighed by
no goal
recorded on
few_nerd_280

01what it measures, and how

How many entities the arm returned for the item, after normalisation.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"n_predicted": {"summary": "…", "details": "…"}}.

02on few_nerd_280

gold/few_nerd_280fn_capitalized_n200_v1: 3.732fn_gemma_s_n200_v1: 3.729fn_mistral_s_n200_v1: 3.661fn_deepseek_s_n200_v1: 3.411fn_deepseek_l_n200_v1: 3.404fn_deepseek_m_n200_v1: 3.386fn_llama_l_n200_v1: 3.361fn_openai_s_n200_v1: 3.357fn_glm_m_n200_v1: 3.261fn_llama_s_n200_v1: 3.232fn_mistral_m_n200_v1: 3.229fn_gliner_n200_v1: 3.225fn_qwen_m_n200_v1: 3.146fn_llama_m_n200_v1: 3.121fn_anthropic_s_n200_v1: 3.1fn_mistral_l_n200_v1: 3.093fn_openai_l_n200_v1: 3.025fn_qwen_s_n200_v1: 3.014fn_qwen_l_n200_v1: 3fn_anthropic_l_n200_v1: 2.968fn_openai_m_n200_v1: 2.943fn_gemma_l_n200_v1: 2.932fn_glm_s_n200_v1: 2.925fn_gemma_m_n200_v1: 2.879fn_glm_l_n200_v1: 2.796fn_span_marker_n200_v1: 2.789fn_anthropic_m_n200_v1: 2.743fn_nothing_n200_v1: 00better · 3.732

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean n_predicteditemsscored against
fn_capitalized_n200_v13.732280gold/few_nerd_280 gold
fn_gemma_s_n200_v13.729280gold/few_nerd_280 gold
fn_mistral_s_n200_v13.661280gold/few_nerd_280 gold
fn_deepseek_s_n200_v13.411280gold/few_nerd_280 gold
fn_deepseek_l_n200_v13.404280gold/few_nerd_280 gold
fn_deepseek_m_n200_v13.386280gold/few_nerd_280 gold
fn_llama_l_n200_v13.361280gold/few_nerd_280 gold
fn_openai_s_n200_v13.357280gold/few_nerd_280 gold
fn_glm_m_n200_v13.261280gold/few_nerd_280 gold
fn_llama_s_n200_v13.232280gold/few_nerd_280 gold
fn_mistral_m_n200_v13.229280gold/few_nerd_280 gold
fn_gliner_n200_v13.225280gold/few_nerd_280 gold
fn_qwen_m_n200_v13.146280gold/few_nerd_280 gold
fn_llama_m_n200_v13.121280gold/few_nerd_280 gold
fn_anthropic_s_n200_v13.100280gold/few_nerd_280 gold
fn_mistral_l_n200_v13.093280gold/few_nerd_280 gold
fn_openai_l_n200_v13.025280gold/few_nerd_280 gold
fn_qwen_s_n200_v13.014280gold/few_nerd_280 gold
fn_qwen_l_n200_v13.000280gold/few_nerd_280 gold
fn_anthropic_l_n200_v12.968280gold/few_nerd_280 gold
fn_openai_m_n200_v12.943280gold/few_nerd_280 gold
fn_gemma_l_n200_v12.932280gold/few_nerd_280 gold
fn_glm_s_n200_v12.925280gold/few_nerd_280 gold
fn_gemma_m_n200_v12.879280gold/few_nerd_280 gold
fn_glm_l_n200_v12.796280gold/few_nerd_280 gold
fn_span_marker_n200_v12.789280gold/few_nerd_280 gold
fn_anthropic_m_n200_v12.743280gold/few_nerd_280 gold
fn_nothing_n200_v10.000280gold/few_nerd_280 gold