eval-harness

metric

rougeLsum

better
higher
kind
quality: measured against a reference or by a judge
decides
no goal
weighed by
no goal
recorded on
cnn_dailymail_20, cnn_dailymail_200, cnn_dailymail_dm200

01what it measures, and how

ROUGE-Lsum F1 against the reference (per-sentence longest common subsequence), the CNN/DailyMail literature's figure. Rises with length.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"rougeLsum": {"summary": "…", "details": "…"}}.

02on cnn_dailymail_20

gold/cnn_dailymail_20cnn_bart_l_v1: 0.33cnn_lead3_v1: 0.280.28better · 0.33

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean rougeLsumitemsscored against
cnn_bart_l_v10.33020gold/cnn_dailymail_20 gold
cnn_lead3_v10.28020gold/cnn_dailymail_20 gold

03on cnn_dailymail_200

gold/cnn_dailymail_200cnn_llama_l_n200_v1: 0.322cnn_deepseek_m_n200_v1: 0.319cnn_bart_l_n200_v1: 0.316cnn_bart_l_n200_v1 · linux: 0.316cnn_bart_m_n200_v1: 0.315cnn_bart_m_n200_v1 · linux: 0.315cnn_openai_l_n200_v1: 0.315cnn_openai_m_n200_v1: 0.312cnn_deepseek_l_n200_v1: 0.312cnn_bart_s_n200_v1: 0.311cnn_bart_s_n200_v1 · linux: 0.311cnn_bart_s_n200_v1 · repeats: 0.311cnn_llama_m_n200_v1: 0.31cnn_gemma_l_n200_v1: 0.309cnn_deepseek_s_n200_v1: 0.308cnn_llama_s_n200_v1: 0.308cnn_glm_m_n200_v1: 0.308cnn_gemma_m_n200_v1: 0.302cnn_glm_s_n200_v1: 0.301cnn_openai_s_n200_v1: 0.298cnn_glm_l_n200_v1: 0.294cnn_anthropic_m_n200_v1: 0.29cnn_mistral_s_n200_v1: 0.289cnn_anthropic_l_n200_v1: 0.288cnn_qwen_s_n200_v1: 0.287cnn_gemma_s_n200_v1: 0.283cnn_anthropic_s_n200_v1: 0.282cnn_mistral_m_n200_v1: 0.281cnn_qwen_l_n200_v1: 0.28cnn_mistral_l_n200_v1: 0.28cnn_qwen_m_n200_v1: 0.274cnn_lead3_n200_v1: 0.27cnn_lead3_n200_v1 · linux: 0.27cnn_bart_l_xsum_n200_v1: 0.246cnn_bart_l_xsum_n200_v1 · linux: 0.246silver/cnn_dailymail_200cnn_deepseek_m_n200_v1 · silver: 0.528cnn_anthropic_m_n200_v1 · silver: 0.527cnn_anthropic_s_n200_v1 · silver: 0.489cnn_deepseek_l_n200_v1 · silver: 0.47cnn_deepseek_s_n200_v1 · silver: 0.467cnn_openai_l_n200_v1 · silver: 0.466cnn_openai_s_n200_v1 · silver: 0.459cnn_glm_m_n200_v1 · silver: 0.458cnn_openai_m_n200_v1 · silver: 0.454cnn_qwen_l_n200_v1 · silver: 0.45cnn_llama_l_n200_v1 · silver: 0.441cnn_llama_s_n200_v1 · silver: 0.438cnn_mistral_s_n200_v1 · silver: 0.437cnn_glm_l_n200_v1 · silver: 0.435cnn_gemma_l_n200_v1 · silver: 0.431cnn_qwen_m_n200_v1 · silver: 0.428cnn_llama_m_n200_v1 · silver: 0.426cnn_gemma_m_n200_v1 · silver: 0.423cnn_qwen_s_n200_v1 · silver: 0.414cnn_gemma_s_n200_v1 · silver: 0.406cnn_mistral_l_n200_v1 · silver: 0.402cnn_mistral_m_n200_v1 · silver: 0.397cnn_glm_s_n200_v1 · silver: 0.391cnn_bart_m_n200_v1 · silver: 0.373cnn_bart_l_n200_v1 · silver: 0.373cnn_bart_s_n200_v1 · silver: 0.363cnn_lead3_n200_v1 · silver: 0.332cnn_bart_l_xsum_n200_v1 · silver: 0.2110.211better · 0.528

Not drawn: cnn_anthropic_l_n200_v1 · silver, scored against the reference it wrote (its row is in the table).

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean rougeLsumitemsscored against
cnn_llama_l_n200_v10.322200gold/cnn_dailymail_200 gold
cnn_deepseek_m_n200_v10.319200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v10.316200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v1 · linux0.316200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v10.315200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v1 · linux0.315200gold/cnn_dailymail_200 gold
cnn_openai_l_n200_v10.315200gold/cnn_dailymail_200 gold
cnn_openai_m_n200_v10.312200gold/cnn_dailymail_200 gold
cnn_deepseek_l_n200_v10.312200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v10.311200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · linux0.311200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · repeats0.311200gold/cnn_dailymail_200 gold
cnn_llama_m_n200_v10.310200gold/cnn_dailymail_200 gold
cnn_gemma_l_n200_v10.309200gold/cnn_dailymail_200 gold
cnn_deepseek_s_n200_v10.308200gold/cnn_dailymail_200 gold
cnn_llama_s_n200_v10.308200gold/cnn_dailymail_200 gold
cnn_glm_m_n200_v10.308200gold/cnn_dailymail_200 gold
cnn_gemma_m_n200_v10.302200gold/cnn_dailymail_200 gold
cnn_glm_s_n200_v10.301200gold/cnn_dailymail_200 gold
cnn_openai_s_n200_v10.298200gold/cnn_dailymail_200 gold
cnn_glm_l_n200_v10.294200gold/cnn_dailymail_200 gold
cnn_anthropic_m_n200_v10.290200gold/cnn_dailymail_200 gold
cnn_mistral_s_n200_v10.289200gold/cnn_dailymail_200 gold
cnn_anthropic_l_n200_v10.288200gold/cnn_dailymail_200 gold
cnn_qwen_s_n200_v10.287200gold/cnn_dailymail_200 gold
cnn_gemma_s_n200_v10.283200gold/cnn_dailymail_200 gold
cnn_anthropic_s_n200_v10.282200gold/cnn_dailymail_200 gold
cnn_mistral_m_n200_v10.281200gold/cnn_dailymail_200 gold
cnn_qwen_l_n200_v10.280200gold/cnn_dailymail_200 gold
cnn_mistral_l_n200_v10.280200gold/cnn_dailymail_200 gold
cnn_qwen_m_n200_v10.274200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v10.270200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v1 · linux0.270200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v10.246200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v1 · linux0.246200gold/cnn_dailymail_200 gold
cnn_anthropic_l_n200_v1 · silver1.000200silver/cnn_dailymail_200 silver
cnn_deepseek_m_n200_v1 · silver0.528200silver/cnn_dailymail_200 silver
cnn_anthropic_m_n200_v1 · silver0.527200silver/cnn_dailymail_200 silver
cnn_anthropic_s_n200_v1 · silver0.489200silver/cnn_dailymail_200 silver
cnn_deepseek_l_n200_v1 · silver0.470200silver/cnn_dailymail_200 silver
cnn_deepseek_s_n200_v1 · silver0.467200silver/cnn_dailymail_200 silver
cnn_openai_l_n200_v1 · silver0.466200silver/cnn_dailymail_200 silver
cnn_openai_s_n200_v1 · silver0.459200silver/cnn_dailymail_200 silver
cnn_glm_m_n200_v1 · silver0.458200silver/cnn_dailymail_200 silver
cnn_openai_m_n200_v1 · silver0.454200silver/cnn_dailymail_200 silver
cnn_qwen_l_n200_v1 · silver0.450200silver/cnn_dailymail_200 silver
cnn_llama_l_n200_v1 · silver0.441200silver/cnn_dailymail_200 silver
cnn_llama_s_n200_v1 · silver0.438200silver/cnn_dailymail_200 silver
cnn_mistral_s_n200_v1 · silver0.437200silver/cnn_dailymail_200 silver
cnn_glm_l_n200_v1 · silver0.435200silver/cnn_dailymail_200 silver
cnn_gemma_l_n200_v1 · silver0.431200silver/cnn_dailymail_200 silver
cnn_qwen_m_n200_v1 · silver0.428200silver/cnn_dailymail_200 silver
cnn_llama_m_n200_v1 · silver0.426200silver/cnn_dailymail_200 silver
cnn_gemma_m_n200_v1 · silver0.423200silver/cnn_dailymail_200 silver
cnn_qwen_s_n200_v1 · silver0.414200silver/cnn_dailymail_200 silver
cnn_gemma_s_n200_v1 · silver0.406200silver/cnn_dailymail_200 silver
cnn_mistral_l_n200_v1 · silver0.402200silver/cnn_dailymail_200 silver
cnn_mistral_m_n200_v1 · silver0.397200silver/cnn_dailymail_200 silver
cnn_glm_s_n200_v1 · silver0.391200silver/cnn_dailymail_200 silver
cnn_bart_m_n200_v1 · silver0.373200silver/cnn_dailymail_200 silver
cnn_bart_l_n200_v1 · silver0.373200silver/cnn_dailymail_200 silver
cnn_bart_s_n200_v1 · silver0.363200silver/cnn_dailymail_200 silver
cnn_lead3_n200_v1 · silver0.332200silver/cnn_dailymail_200 silver
cnn_bart_l_xsum_n200_v1 · silver0.211200silver/cnn_dailymail_200 silver

04on cnn_dailymail_dm200

gold/cnn_dailymail_dm200cnn_bart_m_dm200_v1: 0.439cnn_bart_l_dm200_v1: 0.432cnn_bart_s_dm200_v1: 0.417cnn_lead3_dm200_v1: 0.382cnn_bart_l_xsum_dm200_v1: 0.2120.212better · 0.439

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean rougeLsumitemsscored against
cnn_bart_m_dm200_v10.439200gold/cnn_dailymail_dm200 gold
cnn_bart_l_dm200_v10.432200gold/cnn_dailymail_dm200 gold
cnn_bart_s_dm200_v10.417200gold/cnn_dailymail_dm200 gold
cnn_lead3_dm200_v10.382200gold/cnn_dailymail_dm200 gold
cnn_bart_l_xsum_dm200_v10.212200gold/cnn_dailymail_dm200 gold