metric

`fmt_overlong`

better: neutral: a description
kind: descriptive: says what the output was, never whether it was good
decides: no goal
weighed by: no goal
recorded on: cnn_dailymail_20, cnn_dailymail_200, cnn_dailymail_dm200

01what it measures, and how

1 when the output runs past 90 words where the prompt asks for 2-3 short sentences.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"fmt_overlong": {"summary": "…", "details": "…"}}.

02on cnn_dailymail_20

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

arm	mean fmt_overlong	items	scored against
cnn_lead3_v1	0.050	20	gold/cnn_dailymail_20 gold
cnn_bart_l_v1	0.000	20	gold/cnn_dailymail_20 gold

03on cnn_dailymail_200

Not drawn: cnn_anthropic_l_n200_v1 · silver, scored against the reference it wrote (its row is in the table).

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

arm	mean fmt_overlong	items	scored against
cnn_anthropic_l_n200_v1	0.115	200	gold/cnn_dailymail_200 gold
cnn_glm_l_n200_v1	0.100	200	gold/cnn_dailymail_200 gold
cnn_lead3_n200_v1	0.085	200	gold/cnn_dailymail_200 gold
cnn_lead3_n200_v1 · linux	0.085	200	gold/cnn_dailymail_200 gold
cnn_anthropic_s_n200_v1	0.060	200	gold/cnn_dailymail_200 gold
cnn_anthropic_m_n200_v1	0.015	200	gold/cnn_dailymail_200 gold
cnn_deepseek_m_n200_v1	0.010	200	gold/cnn_dailymail_200 gold
cnn_qwen_l_n200_v1	0.005	200	gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_deepseek_l_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_deepseek_s_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_gemma_l_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_gemma_m_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_gemma_s_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_glm_m_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_glm_s_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_llama_l_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_llama_m_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_llama_s_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_mistral_l_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_mistral_m_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_mistral_s_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_openai_l_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_openai_m_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_openai_s_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_qwen_m_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_qwen_s_n200_v1	0.000	200	gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v1 · linux	0.000	200	gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v1 · linux	0.000	200	gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v1 · linux	0.000	200	gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · linux	0.000	200	gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · repeats	0.000	200	gold/cnn_dailymail_200 gold
cnn_anthropic_l_n200_v1 · silver	0.115	200	silver/cnn_dailymail_200 silver
cnn_glm_l_n200_v1 · silver	0.100	200	silver/cnn_dailymail_200 silver
cnn_lead3_n200_v1 · silver	0.085	200	silver/cnn_dailymail_200 silver
cnn_anthropic_s_n200_v1 · silver	0.060	200	silver/cnn_dailymail_200 silver
cnn_anthropic_m_n200_v1 · silver	0.015	200	silver/cnn_dailymail_200 silver
cnn_deepseek_m_n200_v1 · silver	0.010	200	silver/cnn_dailymail_200 silver
cnn_qwen_l_n200_v1 · silver	0.005	200	silver/cnn_dailymail_200 silver
cnn_bart_l_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_bart_l_xsum_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_bart_m_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_bart_s_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_deepseek_l_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_deepseek_s_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_gemma_l_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_gemma_m_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_gemma_s_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_glm_m_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_glm_s_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_llama_l_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_llama_m_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_llama_s_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_mistral_l_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_mistral_m_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_mistral_s_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_openai_l_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_openai_m_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_openai_s_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_qwen_m_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver
cnn_qwen_s_n200_v1 · silver	0.000	200	silver/cnn_dailymail_200 silver

04on cnn_dailymail_dm200

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

arm	mean fmt_overlong	items	scored against
cnn_bart_m_dm200_v1	0.200	200	gold/cnn_dailymail_dm200 gold
cnn_lead3_dm200_v1	0.200	200	gold/cnn_dailymail_dm200 gold
cnn_bart_l_dm200_v1	0.120	200	gold/cnn_dailymail_dm200 gold
cnn_bart_l_xsum_dm200_v1	0.000	200	gold/cnn_dailymail_dm200 gold
cnn_bart_s_dm200_v1	0.000	200	gold/cnn_dailymail_dm200 gold