eval-harness

metric

rouge1

better
higher
kind
quality: measured against a reference or by a judge
decides
no goal
weighed by
Short news summaries (CNN)
recorded on
cnn_dailymail_20, cnn_dailymail_200, cnn_dailymail_dm200

01what it measures, and how

ROUGE-1 F1 against the reference, for comparison with published CNN/DailyMail results. Rises with length; read after coverage and concision.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"rouge1": {"summary": "…", "details": "…"}}.

02on cnn_dailymail_20

gold/cnn_dailymail_20cnn_bart_l_v1: 0.358cnn_lead3_v1: 0.3120.312better · 0.358

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean rouge1itemsscored against
cnn_bart_l_v10.35820gold/cnn_dailymail_20 gold
cnn_lead3_v10.31220gold/cnn_dailymail_20 gold

03on cnn_dailymail_200

gold/cnn_dailymail_200cnn_llama_l_n200_v1: 0.364cnn_openai_l_n200_v1: 0.362cnn_deepseek_m_n200_v1: 0.361cnn_openai_m_n200_v1: 0.36cnn_deepseek_l_n200_v1: 0.355cnn_llama_s_n200_v1: 0.353cnn_glm_m_n200_v1: 0.352cnn_gemma_l_n200_v1: 0.351cnn_openai_s_n200_v1: 0.351cnn_deepseek_s_n200_v1: 0.35cnn_glm_s_n200_v1: 0.349cnn_bart_l_n200_v1: 0.348cnn_bart_l_n200_v1 · linux: 0.348cnn_llama_m_n200_v1: 0.348cnn_bart_m_n200_v1: 0.346cnn_bart_m_n200_v1 · linux: 0.346cnn_gemma_m_n200_v1: 0.344cnn_bart_s_n200_v1: 0.342cnn_bart_s_n200_v1 · linux: 0.342cnn_bart_s_n200_v1 · repeats: 0.342cnn_qwen_s_n200_v1: 0.336cnn_anthropic_m_n200_v1: 0.335cnn_glm_l_n200_v1: 0.333cnn_qwen_l_n200_v1: 0.331cnn_anthropic_l_n200_v1: 0.329cnn_mistral_s_n200_v1: 0.329cnn_anthropic_s_n200_v1: 0.326cnn_gemma_s_n200_v1: 0.322cnn_mistral_m_n200_v1: 0.32cnn_mistral_l_n200_v1: 0.319cnn_qwen_m_n200_v1: 0.316cnn_lead3_n200_v1: 0.302cnn_lead3_n200_v1 · linux: 0.302cnn_bart_l_xsum_n200_v1: 0.284cnn_bart_l_xsum_n200_v1 · linux: 0.284silver/cnn_dailymail_200cnn_anthropic_m_n200_v1 · silver: 0.595cnn_deepseek_m_n200_v1 · silver: 0.59cnn_anthropic_s_n200_v1 · silver: 0.562cnn_deepseek_l_n200_v1 · silver: 0.528cnn_deepseek_s_n200_v1 · silver: 0.528cnn_openai_l_n200_v1 · silver: 0.525cnn_openai_s_n200_v1 · silver: 0.521cnn_qwen_l_n200_v1 · silver: 0.517cnn_glm_m_n200_v1 · silver: 0.515cnn_openai_m_n200_v1 · silver: 0.507cnn_llama_s_n200_v1 · silver: 0.5cnn_mistral_s_n200_v1 · silver: 0.499cnn_llama_l_n200_v1 · silver: 0.499cnn_qwen_m_n200_v1 · silver: 0.497cnn_glm_l_n200_v1 · silver: 0.487cnn_gemma_m_n200_v1 · silver: 0.482cnn_llama_m_n200_v1 · silver: 0.482cnn_gemma_l_n200_v1 · silver: 0.482cnn_qwen_s_n200_v1 · silver: 0.471cnn_gemma_s_n200_v1 · silver: 0.47cnn_mistral_l_n200_v1 · silver: 0.468cnn_mistral_m_n200_v1 · silver: 0.453cnn_glm_s_n200_v1 · silver: 0.444cnn_bart_m_n200_v1 · silver: 0.422cnn_bart_l_n200_v1 · silver: 0.421cnn_bart_s_n200_v1 · silver: 0.414cnn_lead3_n200_v1 · silver: 0.388cnn_bart_l_xsum_n200_v1 · silver: 0.2550.255better · 0.595

Not drawn: cnn_anthropic_l_n200_v1 · silver, scored against the reference it wrote (its row is in the table).

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean rouge1itemsscored against
cnn_llama_l_n200_v10.364200gold/cnn_dailymail_200 gold
cnn_openai_l_n200_v10.362200gold/cnn_dailymail_200 gold
cnn_deepseek_m_n200_v10.361200gold/cnn_dailymail_200 gold
cnn_openai_m_n200_v10.360200gold/cnn_dailymail_200 gold
cnn_deepseek_l_n200_v10.355200gold/cnn_dailymail_200 gold
cnn_llama_s_n200_v10.353200gold/cnn_dailymail_200 gold
cnn_glm_m_n200_v10.352200gold/cnn_dailymail_200 gold
cnn_gemma_l_n200_v10.351200gold/cnn_dailymail_200 gold
cnn_openai_s_n200_v10.351200gold/cnn_dailymail_200 gold
cnn_deepseek_s_n200_v10.350200gold/cnn_dailymail_200 gold
cnn_glm_s_n200_v10.349200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v10.348200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v1 · linux0.348200gold/cnn_dailymail_200 gold
cnn_llama_m_n200_v10.348200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v10.346200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v1 · linux0.346200gold/cnn_dailymail_200 gold
cnn_gemma_m_n200_v10.344200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v10.342200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · linux0.342200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · repeats0.342200gold/cnn_dailymail_200 gold
cnn_qwen_s_n200_v10.336200gold/cnn_dailymail_200 gold
cnn_anthropic_m_n200_v10.335200gold/cnn_dailymail_200 gold
cnn_glm_l_n200_v10.333200gold/cnn_dailymail_200 gold
cnn_qwen_l_n200_v10.331200gold/cnn_dailymail_200 gold
cnn_anthropic_l_n200_v10.329200gold/cnn_dailymail_200 gold
cnn_mistral_s_n200_v10.329200gold/cnn_dailymail_200 gold
cnn_anthropic_s_n200_v10.326200gold/cnn_dailymail_200 gold
cnn_gemma_s_n200_v10.322200gold/cnn_dailymail_200 gold
cnn_mistral_m_n200_v10.320200gold/cnn_dailymail_200 gold
cnn_mistral_l_n200_v10.319200gold/cnn_dailymail_200 gold
cnn_qwen_m_n200_v10.316200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v10.302200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v1 · linux0.302200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v10.284200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v1 · linux0.284200gold/cnn_dailymail_200 gold
cnn_anthropic_l_n200_v1 · silver1.000200silver/cnn_dailymail_200 silver
cnn_anthropic_m_n200_v1 · silver0.595200silver/cnn_dailymail_200 silver
cnn_deepseek_m_n200_v1 · silver0.590200silver/cnn_dailymail_200 silver
cnn_anthropic_s_n200_v1 · silver0.562200silver/cnn_dailymail_200 silver
cnn_deepseek_l_n200_v1 · silver0.528200silver/cnn_dailymail_200 silver
cnn_deepseek_s_n200_v1 · silver0.528200silver/cnn_dailymail_200 silver
cnn_openai_l_n200_v1 · silver0.525200silver/cnn_dailymail_200 silver
cnn_openai_s_n200_v1 · silver0.521200silver/cnn_dailymail_200 silver
cnn_qwen_l_n200_v1 · silver0.517200silver/cnn_dailymail_200 silver
cnn_glm_m_n200_v1 · silver0.515200silver/cnn_dailymail_200 silver
cnn_openai_m_n200_v1 · silver0.507200silver/cnn_dailymail_200 silver
cnn_llama_s_n200_v1 · silver0.500200silver/cnn_dailymail_200 silver
cnn_mistral_s_n200_v1 · silver0.499200silver/cnn_dailymail_200 silver
cnn_llama_l_n200_v1 · silver0.499200silver/cnn_dailymail_200 silver
cnn_qwen_m_n200_v1 · silver0.497200silver/cnn_dailymail_200 silver
cnn_glm_l_n200_v1 · silver0.487200silver/cnn_dailymail_200 silver
cnn_gemma_m_n200_v1 · silver0.482200silver/cnn_dailymail_200 silver
cnn_llama_m_n200_v1 · silver0.482200silver/cnn_dailymail_200 silver
cnn_gemma_l_n200_v1 · silver0.482200silver/cnn_dailymail_200 silver
cnn_qwen_s_n200_v1 · silver0.471200silver/cnn_dailymail_200 silver
cnn_gemma_s_n200_v1 · silver0.470200silver/cnn_dailymail_200 silver
cnn_mistral_l_n200_v1 · silver0.468200silver/cnn_dailymail_200 silver
cnn_mistral_m_n200_v1 · silver0.453200silver/cnn_dailymail_200 silver
cnn_glm_s_n200_v1 · silver0.444200silver/cnn_dailymail_200 silver
cnn_bart_m_n200_v1 · silver0.422200silver/cnn_dailymail_200 silver
cnn_bart_l_n200_v1 · silver0.421200silver/cnn_dailymail_200 silver
cnn_bart_s_n200_v1 · silver0.414200silver/cnn_dailymail_200 silver
cnn_lead3_n200_v1 · silver0.388200silver/cnn_dailymail_200 silver
cnn_bart_l_xsum_n200_v1 · silver0.255200silver/cnn_dailymail_200 silver

04on cnn_dailymail_dm200

gold/cnn_dailymail_dm200cnn_bart_m_dm200_v1: 0.47cnn_bart_l_dm200_v1: 0.462cnn_bart_s_dm200_v1: 0.448cnn_lead3_dm200_v1: 0.422cnn_bart_l_xsum_dm200_v1: 0.2470.247better · 0.47

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean rouge1itemsscored against
cnn_bart_m_dm200_v10.470200gold/cnn_dailymail_dm200 gold
cnn_bart_l_dm200_v10.462200gold/cnn_dailymail_dm200 gold
cnn_bart_s_dm200_v10.448200gold/cnn_dailymail_dm200 gold
cnn_lead3_dm200_v10.422200gold/cnn_dailymail_dm200 gold
cnn_bart_l_xsum_dm200_v10.247200gold/cnn_dailymail_dm200 gold