eval-harness

metric

length_vs_reference

better
neutral: a description
kind
descriptive: says what the output was, never whether it was good
decides
no goal
weighed by
no goal
recorded on
cnn_dailymail_20, cnn_dailymail_200, cnn_dailymail_dm200

01what it measures, and how

Summary words divided by reference words: 1 means as long as the human's. A verbose winner shows here.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"length_vs_reference": {"summary": "…", "details": "…"}}.

02on cnn_dailymail_20

gold/cnn_dailymail_20cnn_lead3_v1: 1.716cnn_bart_l_v1: 1.4671.467better · 1.716

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean length_vs_referenceitemsscored against
cnn_lead3_v11.71620gold/cnn_dailymail_20 gold
cnn_bart_l_v11.46720gold/cnn_dailymail_20 gold

03on cnn_dailymail_200

gold/cnn_dailymail_200cnn_anthropic_l_n200_v1: 2.302cnn_anthropic_s_n200_v1: 2.188cnn_glm_l_n200_v1: 2.024cnn_anthropic_m_n200_v1: 1.984cnn_lead3_n200_v1: 1.905cnn_lead3_n200_v1 · linux: 1.905cnn_deepseek_m_n200_v1: 1.767cnn_llama_s_n200_v1: 1.762cnn_qwen_m_n200_v1: 1.758cnn_bart_m_n200_v1: 1.722cnn_bart_m_n200_v1 · linux: 1.722cnn_qwen_l_n200_v1: 1.697cnn_mistral_s_n200_v1: 1.672cnn_gemma_s_n200_v1: 1.635cnn_bart_s_n200_v1: 1.613cnn_bart_s_n200_v1 · linux: 1.613cnn_bart_s_n200_v1 · repeats: 1.613cnn_bart_l_n200_v1: 1.579cnn_bart_l_n200_v1 · linux: 1.579cnn_glm_m_n200_v1: 1.556cnn_deepseek_l_n200_v1: 1.546cnn_llama_m_n200_v1: 1.527cnn_llama_l_n200_v1: 1.521cnn_mistral_l_n200_v1: 1.512cnn_openai_s_n200_v1: 1.502cnn_deepseek_s_n200_v1: 1.487cnn_gemma_m_n200_v1: 1.371cnn_mistral_m_n200_v1: 1.353cnn_qwen_s_n200_v1: 1.343cnn_openai_l_n200_v1: 1.324cnn_gemma_l_n200_v1: 1.309cnn_openai_m_n200_v1: 1.296cnn_glm_s_n200_v1: 1.21cnn_bart_l_xsum_n200_v1: 0.657cnn_bart_l_xsum_n200_v1 · linux: 0.657silver/cnn_dailymail_200cnn_anthropic_s_n200_v1 · silver: 0.965cnn_glm_l_n200_v1 · silver: 0.89cnn_anthropic_m_n200_v1 · silver: 0.877cnn_lead3_n200_v1 · silver: 0.852cnn_deepseek_m_n200_v1 · silver: 0.78cnn_llama_s_n200_v1 · silver: 0.776cnn_qwen_m_n200_v1 · silver: 0.775cnn_bart_m_n200_v1 · silver: 0.763cnn_qwen_l_n200_v1 · silver: 0.744cnn_mistral_s_n200_v1 · silver: 0.739cnn_gemma_s_n200_v1 · silver: 0.721cnn_bart_s_n200_v1 · silver: 0.714cnn_bart_l_n200_v1 · silver: 0.701cnn_deepseek_l_n200_v1 · silver: 0.686cnn_glm_m_n200_v1 · silver: 0.686cnn_llama_m_n200_v1 · silver: 0.673cnn_llama_l_n200_v1 · silver: 0.666cnn_mistral_l_n200_v1 · silver: 0.662cnn_openai_s_n200_v1 · silver: 0.662cnn_deepseek_s_n200_v1 · silver: 0.655cnn_gemma_m_n200_v1 · silver: 0.603cnn_mistral_m_n200_v1 · silver: 0.594cnn_qwen_s_n200_v1 · silver: 0.589cnn_openai_l_n200_v1 · silver: 0.582cnn_gemma_l_n200_v1 · silver: 0.578cnn_openai_m_n200_v1 · silver: 0.572cnn_glm_s_n200_v1 · silver: 0.533cnn_bart_l_xsum_n200_v1 · silver: 0.2910.291better · 2.302

Not drawn: cnn_anthropic_l_n200_v1 · silver, scored against the reference it wrote (its row is in the table).

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean length_vs_referenceitemsscored against
cnn_anthropic_l_n200_v12.302200gold/cnn_dailymail_200 gold
cnn_anthropic_s_n200_v12.188200gold/cnn_dailymail_200 gold
cnn_glm_l_n200_v12.024200gold/cnn_dailymail_200 gold
cnn_anthropic_m_n200_v11.984200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v11.905200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v1 · linux1.905200gold/cnn_dailymail_200 gold
cnn_deepseek_m_n200_v11.767200gold/cnn_dailymail_200 gold
cnn_llama_s_n200_v11.762200gold/cnn_dailymail_200 gold
cnn_qwen_m_n200_v11.758200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v11.722200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v1 · linux1.722200gold/cnn_dailymail_200 gold
cnn_qwen_l_n200_v11.697200gold/cnn_dailymail_200 gold
cnn_mistral_s_n200_v11.672200gold/cnn_dailymail_200 gold
cnn_gemma_s_n200_v11.635200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v11.613200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · linux1.613200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · repeats1.613200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v11.579200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v1 · linux1.579200gold/cnn_dailymail_200 gold
cnn_glm_m_n200_v11.556200gold/cnn_dailymail_200 gold
cnn_deepseek_l_n200_v11.546200gold/cnn_dailymail_200 gold
cnn_llama_m_n200_v11.527200gold/cnn_dailymail_200 gold
cnn_llama_l_n200_v11.521200gold/cnn_dailymail_200 gold
cnn_mistral_l_n200_v11.512200gold/cnn_dailymail_200 gold
cnn_openai_s_n200_v11.502200gold/cnn_dailymail_200 gold
cnn_deepseek_s_n200_v11.487200gold/cnn_dailymail_200 gold
cnn_gemma_m_n200_v11.371200gold/cnn_dailymail_200 gold
cnn_mistral_m_n200_v11.353200gold/cnn_dailymail_200 gold
cnn_qwen_s_n200_v11.343200gold/cnn_dailymail_200 gold
cnn_openai_l_n200_v11.324200gold/cnn_dailymail_200 gold
cnn_gemma_l_n200_v11.309200gold/cnn_dailymail_200 gold
cnn_openai_m_n200_v11.296200gold/cnn_dailymail_200 gold
cnn_glm_s_n200_v11.210200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v10.657200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v1 · linux0.657200gold/cnn_dailymail_200 gold
cnn_anthropic_l_n200_v1 · silver1.000200silver/cnn_dailymail_200 silver
cnn_anthropic_s_n200_v1 · silver0.965200silver/cnn_dailymail_200 silver
cnn_glm_l_n200_v1 · silver0.890200silver/cnn_dailymail_200 silver
cnn_anthropic_m_n200_v1 · silver0.877200silver/cnn_dailymail_200 silver
cnn_lead3_n200_v1 · silver0.852200silver/cnn_dailymail_200 silver
cnn_deepseek_m_n200_v1 · silver0.780200silver/cnn_dailymail_200 silver
cnn_llama_s_n200_v1 · silver0.776200silver/cnn_dailymail_200 silver
cnn_qwen_m_n200_v1 · silver0.775200silver/cnn_dailymail_200 silver
cnn_bart_m_n200_v1 · silver0.763200silver/cnn_dailymail_200 silver
cnn_qwen_l_n200_v1 · silver0.744200silver/cnn_dailymail_200 silver
cnn_mistral_s_n200_v1 · silver0.739200silver/cnn_dailymail_200 silver
cnn_gemma_s_n200_v1 · silver0.721200silver/cnn_dailymail_200 silver
cnn_bart_s_n200_v1 · silver0.714200silver/cnn_dailymail_200 silver
cnn_bart_l_n200_v1 · silver0.701200silver/cnn_dailymail_200 silver
cnn_deepseek_l_n200_v1 · silver0.686200silver/cnn_dailymail_200 silver
cnn_glm_m_n200_v1 · silver0.686200silver/cnn_dailymail_200 silver
cnn_llama_m_n200_v1 · silver0.673200silver/cnn_dailymail_200 silver
cnn_llama_l_n200_v1 · silver0.666200silver/cnn_dailymail_200 silver
cnn_mistral_l_n200_v1 · silver0.662200silver/cnn_dailymail_200 silver
cnn_openai_s_n200_v1 · silver0.662200silver/cnn_dailymail_200 silver
cnn_deepseek_s_n200_v1 · silver0.655200silver/cnn_dailymail_200 silver
cnn_gemma_m_n200_v1 · silver0.603200silver/cnn_dailymail_200 silver
cnn_mistral_m_n200_v1 · silver0.594200silver/cnn_dailymail_200 silver
cnn_qwen_s_n200_v1 · silver0.589200silver/cnn_dailymail_200 silver
cnn_openai_l_n200_v1 · silver0.582200silver/cnn_dailymail_200 silver
cnn_gemma_l_n200_v1 · silver0.578200silver/cnn_dailymail_200 silver
cnn_openai_m_n200_v1 · silver0.572200silver/cnn_dailymail_200 silver
cnn_glm_s_n200_v1 · silver0.533200silver/cnn_dailymail_200 silver
cnn_bart_l_xsum_n200_v1 · silver0.291200silver/cnn_dailymail_200 silver

04on cnn_dailymail_dm200

gold/cnn_dailymail_dm200cnn_lead3_dm200_v1: 1.464cnn_bart_m_dm200_v1: 1.393cnn_bart_l_dm200_v1: 1.236cnn_bart_s_dm200_v1: 1.022cnn_bart_l_xsum_dm200_v1: 0.3840.384better · 1.464

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean length_vs_referenceitemsscored against
cnn_lead3_dm200_v11.464200gold/cnn_dailymail_dm200 gold
cnn_bart_m_dm200_v11.393200gold/cnn_dailymail_dm200 gold
cnn_bart_l_dm200_v11.236200gold/cnn_dailymail_dm200 gold
cnn_bart_s_dm200_v11.022200gold/cnn_dailymail_dm200 gold
cnn_bart_l_xsum_dm200_v10.384200gold/cnn_dailymail_dm200 gold