eval-harness

metric

summary_words

better
neutral: a description
kind
descriptive: says what the output was, never whether it was good
decides
no goal
weighed by
no goal
recorded on
cnn_dailymail_20, cnn_dailymail_200, cnn_dailymail_dm200

01what it measures, and how

How many whitespace-separated words the summary had.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"summary_words": {"summary": "…", "details": "…"}}.

02on cnn_dailymail_20

gold/cnn_dailymail_20cnn_lead3_v1: 61.4cnn_bart_l_v1: 50.350.3better · 61.4

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean summary_wordsitemsscored against
cnn_lead3_v161.420gold/cnn_dailymail_20 gold
cnn_bart_l_v150.320gold/cnn_dailymail_20 gold

03on cnn_dailymail_200

gold/cnn_dailymail_200cnn_anthropic_l_n200_v1: 74.5cnn_anthropic_s_n200_v1: 70.7cnn_glm_l_n200_v1: 64.8cnn_anthropic_m_n200_v1: 64.4cnn_lead3_n200_v1: 62cnn_lead3_n200_v1 · linux: 62cnn_deepseek_m_n200_v1: 57.1cnn_qwen_m_n200_v1: 56.7cnn_llama_s_n200_v1: 56.5cnn_bart_m_n200_v1: 55.4cnn_bart_m_n200_v1 · linux: 55.4cnn_qwen_l_n200_v1: 54.7cnn_mistral_s_n200_v1: 53.8cnn_gemma_s_n200_v1: 52.5cnn_bart_s_n200_v1: 51.7cnn_bart_s_n200_v1 · linux: 51.7cnn_bart_s_n200_v1 · repeats: 51.7cnn_bart_l_n200_v1: 50.8cnn_bart_l_n200_v1 · linux: 50.8cnn_deepseek_l_n200_v1: 50.2cnn_glm_m_n200_v1: 50.2cnn_llama_m_n200_v1: 49.2cnn_llama_l_n200_v1: 48.7cnn_mistral_l_n200_v1: 48.5cnn_openai_s_n200_v1: 48.1cnn_deepseek_s_n200_v1: 47.9cnn_gemma_m_n200_v1: 44.0cnn_mistral_m_n200_v1: 43.3cnn_qwen_s_n200_v1: 42.9cnn_gemma_l_n200_v1: 42.5cnn_openai_l_n200_v1: 42.5cnn_openai_m_n200_v1: 41.6cnn_glm_s_n200_v1: 38.7cnn_bart_l_xsum_n200_v1: 21.2cnn_bart_l_xsum_n200_v1 · linux: 21.2silver/cnn_dailymail_200cnn_anthropic_s_n200_v1 · silver: 70.7cnn_glm_l_n200_v1 · silver: 64.8cnn_anthropic_m_n200_v1 · silver: 64.4cnn_lead3_n200_v1 · silver: 62cnn_deepseek_m_n200_v1 · silver: 57.1cnn_qwen_m_n200_v1 · silver: 56.7cnn_llama_s_n200_v1 · silver: 56.5cnn_bart_m_n200_v1 · silver: 55.4cnn_qwen_l_n200_v1 · silver: 54.7cnn_mistral_s_n200_v1 · silver: 53.8cnn_gemma_s_n200_v1 · silver: 52.5cnn_bart_s_n200_v1 · silver: 51.7cnn_bart_l_n200_v1 · silver: 50.8cnn_deepseek_l_n200_v1 · silver: 50.2cnn_glm_m_n200_v1 · silver: 50.2cnn_llama_m_n200_v1 · silver: 49.2cnn_llama_l_n200_v1 · silver: 48.7cnn_mistral_l_n200_v1 · silver: 48.5cnn_openai_s_n200_v1 · silver: 48.1cnn_deepseek_s_n200_v1 · silver: 47.9cnn_gemma_m_n200_v1 · silver: 44.0cnn_mistral_m_n200_v1 · silver: 43.3cnn_qwen_s_n200_v1 · silver: 42.9cnn_gemma_l_n200_v1 · silver: 42.5cnn_openai_l_n200_v1 · silver: 42.5cnn_openai_m_n200_v1 · silver: 41.6cnn_glm_s_n200_v1 · silver: 38.7cnn_bart_l_xsum_n200_v1 · silver: 21.221.2better · 74.5

Not drawn: cnn_anthropic_l_n200_v1 · silver, scored against the reference it wrote (its row is in the table).

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean summary_wordsitemsscored against
cnn_anthropic_l_n200_v174.5200gold/cnn_dailymail_200 gold
cnn_anthropic_s_n200_v170.7200gold/cnn_dailymail_200 gold
cnn_glm_l_n200_v164.8200gold/cnn_dailymail_200 gold
cnn_anthropic_m_n200_v164.4200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v162.0200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v1 · linux62.0200gold/cnn_dailymail_200 gold
cnn_deepseek_m_n200_v157.1200gold/cnn_dailymail_200 gold
cnn_qwen_m_n200_v156.7200gold/cnn_dailymail_200 gold
cnn_llama_s_n200_v156.5200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v155.4200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v1 · linux55.4200gold/cnn_dailymail_200 gold
cnn_qwen_l_n200_v154.7200gold/cnn_dailymail_200 gold
cnn_mistral_s_n200_v153.8200gold/cnn_dailymail_200 gold
cnn_gemma_s_n200_v152.5200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v151.7200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · linux51.7200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · repeats51.7200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v150.8200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v1 · linux50.8200gold/cnn_dailymail_200 gold
cnn_deepseek_l_n200_v150.2200gold/cnn_dailymail_200 gold
cnn_glm_m_n200_v150.2200gold/cnn_dailymail_200 gold
cnn_llama_m_n200_v149.2200gold/cnn_dailymail_200 gold
cnn_llama_l_n200_v148.7200gold/cnn_dailymail_200 gold
cnn_mistral_l_n200_v148.5200gold/cnn_dailymail_200 gold
cnn_openai_s_n200_v148.1200gold/cnn_dailymail_200 gold
cnn_deepseek_s_n200_v147.9200gold/cnn_dailymail_200 gold
cnn_gemma_m_n200_v144.0200gold/cnn_dailymail_200 gold
cnn_mistral_m_n200_v143.3200gold/cnn_dailymail_200 gold
cnn_qwen_s_n200_v142.9200gold/cnn_dailymail_200 gold
cnn_gemma_l_n200_v142.5200gold/cnn_dailymail_200 gold
cnn_openai_l_n200_v142.5200gold/cnn_dailymail_200 gold
cnn_openai_m_n200_v141.6200gold/cnn_dailymail_200 gold
cnn_glm_s_n200_v138.7200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v121.2200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v1 · linux21.2200gold/cnn_dailymail_200 gold
cnn_anthropic_l_n200_v1 · silver74.5200silver/cnn_dailymail_200 silver
cnn_anthropic_s_n200_v1 · silver70.7200silver/cnn_dailymail_200 silver
cnn_glm_l_n200_v1 · silver64.8200silver/cnn_dailymail_200 silver
cnn_anthropic_m_n200_v1 · silver64.4200silver/cnn_dailymail_200 silver
cnn_lead3_n200_v1 · silver62.0200silver/cnn_dailymail_200 silver
cnn_deepseek_m_n200_v1 · silver57.1200silver/cnn_dailymail_200 silver
cnn_qwen_m_n200_v1 · silver56.7200silver/cnn_dailymail_200 silver
cnn_llama_s_n200_v1 · silver56.5200silver/cnn_dailymail_200 silver
cnn_bart_m_n200_v1 · silver55.4200silver/cnn_dailymail_200 silver
cnn_qwen_l_n200_v1 · silver54.7200silver/cnn_dailymail_200 silver
cnn_mistral_s_n200_v1 · silver53.8200silver/cnn_dailymail_200 silver
cnn_gemma_s_n200_v1 · silver52.5200silver/cnn_dailymail_200 silver
cnn_bart_s_n200_v1 · silver51.7200silver/cnn_dailymail_200 silver
cnn_bart_l_n200_v1 · silver50.8200silver/cnn_dailymail_200 silver
cnn_deepseek_l_n200_v1 · silver50.2200silver/cnn_dailymail_200 silver
cnn_glm_m_n200_v1 · silver50.2200silver/cnn_dailymail_200 silver
cnn_llama_m_n200_v1 · silver49.2200silver/cnn_dailymail_200 silver
cnn_llama_l_n200_v1 · silver48.7200silver/cnn_dailymail_200 silver
cnn_mistral_l_n200_v1 · silver48.5200silver/cnn_dailymail_200 silver
cnn_openai_s_n200_v1 · silver48.1200silver/cnn_dailymail_200 silver
cnn_deepseek_s_n200_v1 · silver47.9200silver/cnn_dailymail_200 silver
cnn_gemma_m_n200_v1 · silver44.0200silver/cnn_dailymail_200 silver
cnn_mistral_m_n200_v1 · silver43.3200silver/cnn_dailymail_200 silver
cnn_qwen_s_n200_v1 · silver42.9200silver/cnn_dailymail_200 silver
cnn_gemma_l_n200_v1 · silver42.5200silver/cnn_dailymail_200 silver
cnn_openai_l_n200_v1 · silver42.5200silver/cnn_dailymail_200 silver
cnn_openai_m_n200_v1 · silver41.6200silver/cnn_dailymail_200 silver
cnn_glm_s_n200_v1 · silver38.7200silver/cnn_dailymail_200 silver
cnn_bart_l_xsum_n200_v1 · silver21.2200silver/cnn_dailymail_200 silver

04on cnn_dailymail_dm200

gold/cnn_dailymail_dm200cnn_lead3_dm200_v1: 77.3cnn_bart_m_dm200_v1: 75.8cnn_bart_l_dm200_v1: 67.3cnn_bart_s_dm200_v1: 53.8cnn_bart_l_xsum_dm200_v1: 20.320.3better · 77.3

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean summary_wordsitemsscored against
cnn_lead3_dm200_v177.3200gold/cnn_dailymail_dm200 gold
cnn_bart_m_dm200_v175.8200gold/cnn_dailymail_dm200 gold
cnn_bart_l_dm200_v167.3200gold/cnn_dailymail_dm200 gold
cnn_bart_s_dm200_v153.8200gold/cnn_dailymail_dm200 gold
cnn_bart_l_xsum_dm200_v120.3200gold/cnn_dailymail_dm200 gold