eval-harness

metric

compression

better
neutral: a description
kind
descriptive: says what the output was, never whether it was good
decides
no goal
weighed by
no goal
recorded on
cnn_dailymail_20, cnn_dailymail_200, cnn_dailymail_dm200

01what it measures, and how

Summary words divided by article words: a description, not a quality.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"compression": {"summary": "…", "details": "…"}}.

02on cnn_dailymail_20

gold/cnn_dailymail_20cnn_lead3_v1: 0.173cnn_bart_l_v1: 0.1340.134better · 0.173

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean compressionitemsscored against
cnn_lead3_v10.17320gold/cnn_dailymail_20 gold
cnn_bart_l_v10.13420gold/cnn_dailymail_20 gold

03on cnn_dailymail_200

gold/cnn_dailymail_200cnn_anthropic_l_n200_v1: 0.187cnn_anthropic_s_n200_v1: 0.176cnn_glm_l_n200_v1: 0.167cnn_anthropic_m_n200_v1: 0.16cnn_lead3_n200_v1: 0.159cnn_lead3_n200_v1 · linux: 0.159cnn_deepseek_m_n200_v1: 0.145cnn_bart_m_n200_v1: 0.143cnn_bart_m_n200_v1 · linux: 0.143cnn_qwen_m_n200_v1: 0.143cnn_llama_s_n200_v1: 0.142cnn_mistral_s_n200_v1: 0.138cnn_qwen_l_n200_v1: 0.137cnn_bart_s_n200_v1: 0.136cnn_bart_s_n200_v1 · linux: 0.136cnn_bart_s_n200_v1 · repeats: 0.136cnn_bart_l_n200_v1: 0.132cnn_bart_l_n200_v1 · linux: 0.132cnn_gemma_s_n200_v1: 0.131cnn_glm_m_n200_v1: 0.128cnn_deepseek_l_n200_v1: 0.126cnn_openai_s_n200_v1: 0.126cnn_llama_l_n200_v1: 0.124cnn_llama_m_n200_v1: 0.122cnn_mistral_l_n200_v1: 0.122cnn_deepseek_s_n200_v1: 0.121cnn_gemma_m_n200_v1: 0.113cnn_qwen_s_n200_v1: 0.11cnn_mistral_m_n200_v1: 0.109cnn_openai_l_n200_v1: 0.109cnn_gemma_l_n200_v1: 0.108cnn_openai_m_n200_v1: 0.107cnn_glm_s_n200_v1: 0.1cnn_bart_l_xsum_n200_v1: 0.054cnn_bart_l_xsum_n200_v1 · linux: 0.054silver/cnn_dailymail_200cnn_anthropic_s_n200_v1 · silver: 0.176cnn_glm_l_n200_v1 · silver: 0.167cnn_anthropic_m_n200_v1 · silver: 0.16cnn_lead3_n200_v1 · silver: 0.159cnn_deepseek_m_n200_v1 · silver: 0.145cnn_bart_m_n200_v1 · silver: 0.143cnn_qwen_m_n200_v1 · silver: 0.143cnn_llama_s_n200_v1 · silver: 0.142cnn_mistral_s_n200_v1 · silver: 0.138cnn_qwen_l_n200_v1 · silver: 0.137cnn_bart_s_n200_v1 · silver: 0.136cnn_bart_l_n200_v1 · silver: 0.132cnn_gemma_s_n200_v1 · silver: 0.131cnn_glm_m_n200_v1 · silver: 0.128cnn_deepseek_l_n200_v1 · silver: 0.126cnn_openai_s_n200_v1 · silver: 0.126cnn_llama_l_n200_v1 · silver: 0.124cnn_llama_m_n200_v1 · silver: 0.122cnn_mistral_l_n200_v1 · silver: 0.122cnn_deepseek_s_n200_v1 · silver: 0.121cnn_gemma_m_n200_v1 · silver: 0.113cnn_qwen_s_n200_v1 · silver: 0.11cnn_mistral_m_n200_v1 · silver: 0.109cnn_openai_l_n200_v1 · silver: 0.109cnn_gemma_l_n200_v1 · silver: 0.108cnn_openai_m_n200_v1 · silver: 0.107cnn_glm_s_n200_v1 · silver: 0.1cnn_bart_l_xsum_n200_v1 · silver: 0.0540.054better · 0.187

Not drawn: cnn_anthropic_l_n200_v1 · silver, scored against the reference it wrote (its row is in the table).

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean compressionitemsscored against
cnn_anthropic_l_n200_v10.187200gold/cnn_dailymail_200 gold
cnn_anthropic_s_n200_v10.176200gold/cnn_dailymail_200 gold
cnn_glm_l_n200_v10.167200gold/cnn_dailymail_200 gold
cnn_anthropic_m_n200_v10.160200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v10.159200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v1 · linux0.159200gold/cnn_dailymail_200 gold
cnn_deepseek_m_n200_v10.145200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v10.143200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v1 · linux0.143200gold/cnn_dailymail_200 gold
cnn_qwen_m_n200_v10.143200gold/cnn_dailymail_200 gold
cnn_llama_s_n200_v10.142200gold/cnn_dailymail_200 gold
cnn_mistral_s_n200_v10.138200gold/cnn_dailymail_200 gold
cnn_qwen_l_n200_v10.137200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v10.136200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · linux0.136200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · repeats0.136200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v10.132200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v1 · linux0.132200gold/cnn_dailymail_200 gold
cnn_gemma_s_n200_v10.131200gold/cnn_dailymail_200 gold
cnn_glm_m_n200_v10.128200gold/cnn_dailymail_200 gold
cnn_deepseek_l_n200_v10.126200gold/cnn_dailymail_200 gold
cnn_openai_s_n200_v10.126200gold/cnn_dailymail_200 gold
cnn_llama_l_n200_v10.124200gold/cnn_dailymail_200 gold
cnn_llama_m_n200_v10.122200gold/cnn_dailymail_200 gold
cnn_mistral_l_n200_v10.122200gold/cnn_dailymail_200 gold
cnn_deepseek_s_n200_v10.121200gold/cnn_dailymail_200 gold
cnn_gemma_m_n200_v10.113200gold/cnn_dailymail_200 gold
cnn_qwen_s_n200_v10.110200gold/cnn_dailymail_200 gold
cnn_mistral_m_n200_v10.109200gold/cnn_dailymail_200 gold
cnn_openai_l_n200_v10.109200gold/cnn_dailymail_200 gold
cnn_gemma_l_n200_v10.108200gold/cnn_dailymail_200 gold
cnn_openai_m_n200_v10.107200gold/cnn_dailymail_200 gold
cnn_glm_s_n200_v10.100200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v10.054200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v1 · linux0.054200gold/cnn_dailymail_200 gold
cnn_anthropic_l_n200_v1 · silver0.187200silver/cnn_dailymail_200 silver
cnn_anthropic_s_n200_v1 · silver0.176200silver/cnn_dailymail_200 silver
cnn_glm_l_n200_v1 · silver0.167200silver/cnn_dailymail_200 silver
cnn_anthropic_m_n200_v1 · silver0.160200silver/cnn_dailymail_200 silver
cnn_lead3_n200_v1 · silver0.159200silver/cnn_dailymail_200 silver
cnn_deepseek_m_n200_v1 · silver0.145200silver/cnn_dailymail_200 silver
cnn_bart_m_n200_v1 · silver0.143200silver/cnn_dailymail_200 silver
cnn_qwen_m_n200_v1 · silver0.143200silver/cnn_dailymail_200 silver
cnn_llama_s_n200_v1 · silver0.142200silver/cnn_dailymail_200 silver
cnn_mistral_s_n200_v1 · silver0.138200silver/cnn_dailymail_200 silver
cnn_qwen_l_n200_v1 · silver0.137200silver/cnn_dailymail_200 silver
cnn_bart_s_n200_v1 · silver0.136200silver/cnn_dailymail_200 silver
cnn_bart_l_n200_v1 · silver0.132200silver/cnn_dailymail_200 silver
cnn_gemma_s_n200_v1 · silver0.131200silver/cnn_dailymail_200 silver
cnn_glm_m_n200_v1 · silver0.128200silver/cnn_dailymail_200 silver
cnn_deepseek_l_n200_v1 · silver0.126200silver/cnn_dailymail_200 silver
cnn_openai_s_n200_v1 · silver0.126200silver/cnn_dailymail_200 silver
cnn_llama_l_n200_v1 · silver0.124200silver/cnn_dailymail_200 silver
cnn_llama_m_n200_v1 · silver0.122200silver/cnn_dailymail_200 silver
cnn_mistral_l_n200_v1 · silver0.122200silver/cnn_dailymail_200 silver
cnn_deepseek_s_n200_v1 · silver0.121200silver/cnn_dailymail_200 silver
cnn_gemma_m_n200_v1 · silver0.113200silver/cnn_dailymail_200 silver
cnn_qwen_s_n200_v1 · silver0.110200silver/cnn_dailymail_200 silver
cnn_mistral_m_n200_v1 · silver0.109200silver/cnn_dailymail_200 silver
cnn_openai_l_n200_v1 · silver0.109200silver/cnn_dailymail_200 silver
cnn_gemma_l_n200_v1 · silver0.108200silver/cnn_dailymail_200 silver
cnn_openai_m_n200_v1 · silver0.107200silver/cnn_dailymail_200 silver
cnn_glm_s_n200_v1 · silver0.100200silver/cnn_dailymail_200 silver
cnn_bart_l_xsum_n200_v1 · silver0.054200silver/cnn_dailymail_200 silver

04on cnn_dailymail_dm200

gold/cnn_dailymail_dm200cnn_lead3_dm200_v1: 0.148cnn_bart_m_dm200_v1: 0.14cnn_bart_l_dm200_v1: 0.126cnn_bart_s_dm200_v1: 0.106cnn_bart_l_xsum_dm200_v1: 0.0390.039better · 0.148

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean compressionitemsscored against
cnn_lead3_dm200_v10.148200gold/cnn_dailymail_dm200 gold
cnn_bart_m_dm200_v10.140200gold/cnn_dailymail_dm200 gold
cnn_bart_l_dm200_v10.126200gold/cnn_dailymail_dm200 gold
cnn_bart_s_dm200_v10.106200gold/cnn_dailymail_dm200 gold
cnn_bart_l_xsum_dm200_v10.039200gold/cnn_dailymail_dm200 gold