eval-harness

metric

concision

better
higher
kind
quality: measured against a reference or by a judge
decides
no goal
weighed by
no goal
recorded on
cnn_dailymail_20, cnn_dailymail_200, cnn_dailymail_dm200

01what it measures, and how

ROUGE-Lsum precision over the whole summary: the share of what was written that matches the reference. Padding lowers it.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"concision": {"summary": "…", "details": "…"}}.

02on cnn_dailymail_20

gold/cnn_dailymail_20cnn_bart_l_v1: 0.282cnn_lead3_v1: 0.2250.225better · 0.282

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean concisionitemsscored against
cnn_bart_l_v10.28220gold/cnn_dailymail_20 gold
cnn_lead3_v10.22520gold/cnn_dailymail_20 gold

03on cnn_dailymail_200

gold/cnn_dailymail_200cnn_bart_l_xsum_n200_v1: 0.312cnn_bart_l_xsum_n200_v1 · linux: 0.312cnn_glm_s_n200_v1: 0.28cnn_openai_l_n200_v1: 0.28cnn_openai_m_n200_v1: 0.279cnn_gemma_l_n200_v1: 0.274cnn_llama_l_n200_v1: 0.271cnn_gemma_m_n200_v1: 0.266cnn_deepseek_s_n200_v1: 0.263cnn_llama_m_n200_v1: 0.262cnn_bart_l_n200_v1: 0.261cnn_bart_l_n200_v1 · linux: 0.261cnn_qwen_s_n200_v1: 0.26cnn_bart_s_n200_v1: 0.26cnn_bart_s_n200_v1 · linux: 0.26cnn_bart_s_n200_v1 · repeats: 0.26cnn_deepseek_l_n200_v1: 0.259cnn_bart_m_n200_v1: 0.258cnn_bart_m_n200_v1 · linux: 0.258cnn_glm_m_n200_v1: 0.257cnn_deepseek_m_n200_v1: 0.254cnn_openai_s_n200_v1: 0.251cnn_mistral_m_n200_v1: 0.251cnn_llama_s_n200_v1: 0.247cnn_glm_l_n200_v1: 0.246cnn_mistral_l_n200_v1: 0.236cnn_mistral_s_n200_v1: 0.235cnn_gemma_s_n200_v1: 0.233cnn_qwen_l_n200_v1: 0.231cnn_anthropic_m_n200_v1: 0.221cnn_qwen_m_n200_v1: 0.221cnn_lead3_n200_v1: 0.215cnn_lead3_n200_v1 · linux: 0.215cnn_anthropic_l_n200_v1: 0.209cnn_anthropic_s_n200_v1: 0.208silver/cnn_dailymail_200cnn_openai_l_n200_v1 · silver: 0.642cnn_openai_m_n200_v1 · silver: 0.63cnn_deepseek_m_n200_v1 · silver: 0.611cnn_deepseek_s_n200_v1 · silver: 0.603cnn_gemma_l_n200_v1 · silver: 0.595cnn_deepseek_l_n200_v1 · silver: 0.587cnn_openai_s_n200_v1 · silver: 0.583cnn_qwen_s_n200_v1 · silver: 0.573cnn_anthropic_m_n200_v1 · silver: 0.572cnn_glm_m_n200_v1 · silver: 0.571cnn_glm_s_n200_v1 · silver: 0.57cnn_gemma_m_n200_v1 · silver: 0.569cnn_llama_l_n200_v1 · silver: 0.558cnn_mistral_m_n200_v1 · silver: 0.544cnn_qwen_l_n200_v1 · silver: 0.541cnn_llama_m_n200_v1 · silver: 0.54cnn_glm_l_n200_v1 · silver: 0.536cnn_mistral_s_n200_v1 · silver: 0.521cnn_llama_s_n200_v1 · silver: 0.515cnn_mistral_l_n200_v1 · silver: 0.508cnn_anthropic_s_n200_v1 · silver: 0.506cnn_qwen_m_n200_v1 · silver: 0.5cnn_gemma_s_n200_v1 · silver: 0.494cnn_bart_l_xsum_n200_v1 · silver: 0.479cnn_bart_l_n200_v1 · silver: 0.462cnn_bart_s_n200_v1 · silver: 0.454cnn_bart_m_n200_v1 · silver: 0.453cnn_lead3_n200_v1 · silver: 0.3760.208better · 0.642

Not drawn: cnn_anthropic_l_n200_v1 · silver, scored against the reference it wrote (its row is in the table).

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean concisionitemsscored against
cnn_bart_l_xsum_n200_v10.312200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v1 · linux0.312200gold/cnn_dailymail_200 gold
cnn_glm_s_n200_v10.280200gold/cnn_dailymail_200 gold
cnn_openai_l_n200_v10.280200gold/cnn_dailymail_200 gold
cnn_openai_m_n200_v10.279200gold/cnn_dailymail_200 gold
cnn_gemma_l_n200_v10.274200gold/cnn_dailymail_200 gold
cnn_llama_l_n200_v10.271200gold/cnn_dailymail_200 gold
cnn_gemma_m_n200_v10.266200gold/cnn_dailymail_200 gold
cnn_deepseek_s_n200_v10.263200gold/cnn_dailymail_200 gold
cnn_llama_m_n200_v10.262200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v10.261200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v1 · linux0.261200gold/cnn_dailymail_200 gold
cnn_qwen_s_n200_v10.260200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v10.260200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · linux0.260200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · repeats0.260200gold/cnn_dailymail_200 gold
cnn_deepseek_l_n200_v10.259200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v10.258200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v1 · linux0.258200gold/cnn_dailymail_200 gold
cnn_glm_m_n200_v10.257200gold/cnn_dailymail_200 gold
cnn_deepseek_m_n200_v10.254200gold/cnn_dailymail_200 gold
cnn_openai_s_n200_v10.251200gold/cnn_dailymail_200 gold
cnn_mistral_m_n200_v10.251200gold/cnn_dailymail_200 gold
cnn_llama_s_n200_v10.247200gold/cnn_dailymail_200 gold
cnn_glm_l_n200_v10.246200gold/cnn_dailymail_200 gold
cnn_mistral_l_n200_v10.236200gold/cnn_dailymail_200 gold
cnn_mistral_s_n200_v10.235200gold/cnn_dailymail_200 gold
cnn_gemma_s_n200_v10.233200gold/cnn_dailymail_200 gold
cnn_qwen_l_n200_v10.231200gold/cnn_dailymail_200 gold
cnn_anthropic_m_n200_v10.221200gold/cnn_dailymail_200 gold
cnn_qwen_m_n200_v10.221200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v10.215200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v1 · linux0.215200gold/cnn_dailymail_200 gold
cnn_anthropic_l_n200_v10.209200gold/cnn_dailymail_200 gold
cnn_anthropic_s_n200_v10.208200gold/cnn_dailymail_200 gold
cnn_anthropic_l_n200_v1 · silver1.000200silver/cnn_dailymail_200 silver
cnn_openai_l_n200_v1 · silver0.642200silver/cnn_dailymail_200 silver
cnn_openai_m_n200_v1 · silver0.630200silver/cnn_dailymail_200 silver
cnn_deepseek_m_n200_v1 · silver0.611200silver/cnn_dailymail_200 silver
cnn_deepseek_s_n200_v1 · silver0.603200silver/cnn_dailymail_200 silver
cnn_gemma_l_n200_v1 · silver0.595200silver/cnn_dailymail_200 silver
cnn_deepseek_l_n200_v1 · silver0.587200silver/cnn_dailymail_200 silver
cnn_openai_s_n200_v1 · silver0.583200silver/cnn_dailymail_200 silver
cnn_qwen_s_n200_v1 · silver0.573200silver/cnn_dailymail_200 silver
cnn_anthropic_m_n200_v1 · silver0.572200silver/cnn_dailymail_200 silver
cnn_glm_m_n200_v1 · silver0.571200silver/cnn_dailymail_200 silver
cnn_glm_s_n200_v1 · silver0.570200silver/cnn_dailymail_200 silver
cnn_gemma_m_n200_v1 · silver0.569200silver/cnn_dailymail_200 silver
cnn_llama_l_n200_v1 · silver0.558200silver/cnn_dailymail_200 silver
cnn_mistral_m_n200_v1 · silver0.544200silver/cnn_dailymail_200 silver
cnn_qwen_l_n200_v1 · silver0.541200silver/cnn_dailymail_200 silver
cnn_llama_m_n200_v1 · silver0.540200silver/cnn_dailymail_200 silver
cnn_glm_l_n200_v1 · silver0.536200silver/cnn_dailymail_200 silver
cnn_mistral_s_n200_v1 · silver0.521200silver/cnn_dailymail_200 silver
cnn_llama_s_n200_v1 · silver0.515200silver/cnn_dailymail_200 silver
cnn_mistral_l_n200_v1 · silver0.508200silver/cnn_dailymail_200 silver
cnn_anthropic_s_n200_v1 · silver0.506200silver/cnn_dailymail_200 silver
cnn_qwen_m_n200_v1 · silver0.500200silver/cnn_dailymail_200 silver
cnn_gemma_s_n200_v1 · silver0.494200silver/cnn_dailymail_200 silver
cnn_bart_l_xsum_n200_v1 · silver0.479200silver/cnn_dailymail_200 silver
cnn_bart_l_n200_v1 · silver0.462200silver/cnn_dailymail_200 silver
cnn_bart_s_n200_v1 · silver0.454200silver/cnn_dailymail_200 silver
cnn_bart_m_n200_v1 · silver0.453200silver/cnn_dailymail_200 silver
cnn_lead3_n200_v1 · silver0.376200silver/cnn_dailymail_200 silver

04on cnn_dailymail_dm200

gold/cnn_dailymail_dm200cnn_bart_s_dm200_v1: 0.427cnn_bart_l_dm200_v1: 0.397cnn_bart_m_dm200_v1: 0.391cnn_bart_l_xsum_dm200_v1: 0.382cnn_lead3_dm200_v1: 0.3340.334better · 0.427

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean concisionitemsscored against
cnn_bart_s_dm200_v10.427200gold/cnn_dailymail_dm200 gold
cnn_bart_l_dm200_v10.397200gold/cnn_dailymail_dm200 gold
cnn_bart_m_dm200_v10.391200gold/cnn_dailymail_dm200 gold
cnn_bart_l_xsum_dm200_v10.382200gold/cnn_dailymail_dm200 gold
cnn_lead3_dm200_v10.334200gold/cnn_dailymail_dm200 gold