eval-harness

metric

n_gold

better
neutral: a description
kind
descriptive: says what the output was, never whether it was good
decides
no goal
weighed by
no goal
recorded on
few_nerd_280

01what it measures, and how

How many gold entities the item has, after normalisation.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"n_gold": {"summary": "…", "details": "…"}}.

02on few_nerd_280

gold/few_nerd_280fn_anthropic_l_n200_v1: 2.736fn_anthropic_m_n200_v1: 2.736fn_anthropic_s_n200_v1: 2.736fn_capitalized_n200_v1: 2.736fn_deepseek_l_n200_v1: 2.736fn_deepseek_m_n200_v1: 2.736fn_deepseek_s_n200_v1: 2.736fn_gemma_l_n200_v1: 2.736fn_gemma_m_n200_v1: 2.736fn_gemma_s_n200_v1: 2.736fn_gliner_n200_v1: 2.736fn_glm_l_n200_v1: 2.736fn_glm_m_n200_v1: 2.736fn_glm_s_n200_v1: 2.736fn_llama_l_n200_v1: 2.736fn_llama_m_n200_v1: 2.736fn_llama_s_n200_v1: 2.736fn_mistral_l_n200_v1: 2.736fn_mistral_m_n200_v1: 2.736fn_mistral_s_n200_v1: 2.736fn_nothing_n200_v1: 2.736fn_openai_l_n200_v1: 2.736fn_openai_m_n200_v1: 2.736fn_openai_s_n200_v1: 2.736fn_qwen_l_n200_v1: 2.736fn_qwen_m_n200_v1: 2.736fn_qwen_s_n200_v1: 2.736fn_span_marker_n200_v1: 2.7362.236better · 3.236

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean n_golditemsscored against
fn_anthropic_l_n200_v12.736280gold/few_nerd_280 gold
fn_anthropic_m_n200_v12.736280gold/few_nerd_280 gold
fn_anthropic_s_n200_v12.736280gold/few_nerd_280 gold
fn_capitalized_n200_v12.736280gold/few_nerd_280 gold
fn_deepseek_l_n200_v12.736280gold/few_nerd_280 gold
fn_deepseek_m_n200_v12.736280gold/few_nerd_280 gold
fn_deepseek_s_n200_v12.736280gold/few_nerd_280 gold
fn_gemma_l_n200_v12.736280gold/few_nerd_280 gold
fn_gemma_m_n200_v12.736280gold/few_nerd_280 gold
fn_gemma_s_n200_v12.736280gold/few_nerd_280 gold
fn_gliner_n200_v12.736280gold/few_nerd_280 gold
fn_glm_l_n200_v12.736280gold/few_nerd_280 gold
fn_glm_m_n200_v12.736280gold/few_nerd_280 gold
fn_glm_s_n200_v12.736280gold/few_nerd_280 gold
fn_llama_l_n200_v12.736280gold/few_nerd_280 gold
fn_llama_m_n200_v12.736280gold/few_nerd_280 gold
fn_llama_s_n200_v12.736280gold/few_nerd_280 gold
fn_mistral_l_n200_v12.736280gold/few_nerd_280 gold
fn_mistral_m_n200_v12.736280gold/few_nerd_280 gold
fn_mistral_s_n200_v12.736280gold/few_nerd_280 gold
fn_nothing_n200_v12.736280gold/few_nerd_280 gold
fn_openai_l_n200_v12.736280gold/few_nerd_280 gold
fn_openai_m_n200_v12.736280gold/few_nerd_280 gold
fn_openai_s_n200_v12.736280gold/few_nerd_280 gold
fn_qwen_l_n200_v12.736280gold/few_nerd_280 gold
fn_qwen_m_n200_v12.736280gold/few_nerd_280 gold
fn_qwen_s_n200_v12.736280gold/few_nerd_280 gold
fn_span_marker_n200_v12.736280gold/few_nerd_280 gold