eval-harness

metric

fmt_narration

better
neutral: a description
kind
descriptive: says what the output was, never whether it was good
decides
no goal
weighed by
no goal
recorded on
cnn_dailymail_20, cnn_dailymail_200, cnn_dailymail_dm200

01what it measures, and how

1 when the output thinks out loud ("the user wants", "let me draft", think tags) instead of only summarising: its quality numbers describe something else.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"fmt_narration": {"summary": "…", "details": "…"}}.

02on cnn_dailymail_20

gold/cnn_dailymail_20cnn_bart_l_v1: 0cnn_lead3_v1: 0-0.5better · 0.5

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean fmt_narrationitemsscored against
cnn_bart_l_v10.00020gold/cnn_dailymail_20 gold
cnn_lead3_v10.00020gold/cnn_dailymail_20 gold

03on cnn_dailymail_200

gold/cnn_dailymail_200cnn_glm_l_n200_v1: 0.105cnn_anthropic_l_n200_v1: 0.01cnn_anthropic_m_n200_v1: 0.01cnn_bart_l_n200_v1: 0.01cnn_bart_l_xsum_n200_v1: 0.01cnn_bart_m_n200_v1: 0.01cnn_gemma_s_n200_v1: 0.01cnn_lead3_n200_v1: 0.01cnn_mistral_m_n200_v1: 0.01cnn_openai_l_n200_v1: 0.01cnn_openai_s_n200_v1: 0.01cnn_bart_l_n200_v1 · linux: 0.01cnn_bart_l_xsum_n200_v1 · linux: 0.01cnn_bart_m_n200_v1 · linux: 0.01cnn_lead3_n200_v1 · linux: 0.01cnn_bart_s_n200_v1: 0.005cnn_deepseek_s_n200_v1: 0.005cnn_glm_m_n200_v1: 0.005cnn_llama_m_n200_v1: 0.005cnn_mistral_l_n200_v1: 0.005cnn_bart_s_n200_v1 · linux: 0.005cnn_bart_s_n200_v1 · repeats: 0.005cnn_anthropic_s_n200_v1: 0cnn_deepseek_l_n200_v1: 0cnn_deepseek_m_n200_v1: 0cnn_gemma_l_n200_v1: 0cnn_gemma_m_n200_v1: 0cnn_glm_s_n200_v1: 0cnn_llama_l_n200_v1: 0cnn_llama_s_n200_v1: 0cnn_mistral_s_n200_v1: 0cnn_openai_m_n200_v1: 0cnn_qwen_l_n200_v1: 0cnn_qwen_m_n200_v1: 0cnn_qwen_s_n200_v1: 0silver/cnn_dailymail_200cnn_glm_l_n200_v1 · silver: 0.105cnn_anthropic_m_n200_v1 · silver: 0.01cnn_bart_l_n200_v1 · silver: 0.01cnn_bart_l_xsum_n200_v1 · silver: 0.01cnn_bart_m_n200_v1 · silver: 0.01cnn_gemma_s_n200_v1 · silver: 0.01cnn_lead3_n200_v1 · silver: 0.01cnn_mistral_m_n200_v1 · silver: 0.01cnn_openai_l_n200_v1 · silver: 0.01cnn_openai_s_n200_v1 · silver: 0.01cnn_bart_s_n200_v1 · silver: 0.005cnn_deepseek_s_n200_v1 · silver: 0.005cnn_glm_m_n200_v1 · silver: 0.005cnn_llama_m_n200_v1 · silver: 0.005cnn_mistral_l_n200_v1 · silver: 0.005cnn_anthropic_s_n200_v1 · silver: 0cnn_deepseek_l_n200_v1 · silver: 0cnn_deepseek_m_n200_v1 · silver: 0cnn_gemma_l_n200_v1 · silver: 0cnn_gemma_m_n200_v1 · silver: 0cnn_glm_s_n200_v1 · silver: 0cnn_llama_l_n200_v1 · silver: 0cnn_llama_s_n200_v1 · silver: 0cnn_mistral_s_n200_v1 · silver: 0cnn_openai_m_n200_v1 · silver: 0cnn_qwen_l_n200_v1 · silver: 0cnn_qwen_m_n200_v1 · silver: 0cnn_qwen_s_n200_v1 · silver: 00better · 0.105

Not drawn: cnn_anthropic_l_n200_v1 · silver, scored against the reference it wrote (its row is in the table).

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean fmt_narrationitemsscored against
cnn_glm_l_n200_v10.105200gold/cnn_dailymail_200 gold
cnn_anthropic_l_n200_v10.010200gold/cnn_dailymail_200 gold
cnn_anthropic_m_n200_v10.010200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v10.010200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v10.010200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v10.010200gold/cnn_dailymail_200 gold
cnn_gemma_s_n200_v10.010200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v10.010200gold/cnn_dailymail_200 gold
cnn_mistral_m_n200_v10.010200gold/cnn_dailymail_200 gold
cnn_openai_l_n200_v10.010200gold/cnn_dailymail_200 gold
cnn_openai_s_n200_v10.010200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v1 · linux0.010200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v1 · linux0.010200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v1 · linux0.010200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v1 · linux0.010200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v10.005200gold/cnn_dailymail_200 gold
cnn_deepseek_s_n200_v10.005200gold/cnn_dailymail_200 gold
cnn_glm_m_n200_v10.005200gold/cnn_dailymail_200 gold
cnn_llama_m_n200_v10.005200gold/cnn_dailymail_200 gold
cnn_mistral_l_n200_v10.005200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · linux0.005200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · repeats0.005200gold/cnn_dailymail_200 gold
cnn_anthropic_s_n200_v10.000200gold/cnn_dailymail_200 gold
cnn_deepseek_l_n200_v10.000200gold/cnn_dailymail_200 gold
cnn_deepseek_m_n200_v10.000200gold/cnn_dailymail_200 gold
cnn_gemma_l_n200_v10.000200gold/cnn_dailymail_200 gold
cnn_gemma_m_n200_v10.000200gold/cnn_dailymail_200 gold
cnn_glm_s_n200_v10.000200gold/cnn_dailymail_200 gold
cnn_llama_l_n200_v10.000200gold/cnn_dailymail_200 gold
cnn_llama_s_n200_v10.000200gold/cnn_dailymail_200 gold
cnn_mistral_s_n200_v10.000200gold/cnn_dailymail_200 gold
cnn_openai_m_n200_v10.000200gold/cnn_dailymail_200 gold
cnn_qwen_l_n200_v10.000200gold/cnn_dailymail_200 gold
cnn_qwen_m_n200_v10.000200gold/cnn_dailymail_200 gold
cnn_qwen_s_n200_v10.000200gold/cnn_dailymail_200 gold
cnn_glm_l_n200_v1 · silver0.105200silver/cnn_dailymail_200 silver
cnn_anthropic_l_n200_v1 · silver0.010200silver/cnn_dailymail_200 silver
cnn_anthropic_m_n200_v1 · silver0.010200silver/cnn_dailymail_200 silver
cnn_bart_l_n200_v1 · silver0.010200silver/cnn_dailymail_200 silver
cnn_bart_l_xsum_n200_v1 · silver0.010200silver/cnn_dailymail_200 silver
cnn_bart_m_n200_v1 · silver0.010200silver/cnn_dailymail_200 silver
cnn_gemma_s_n200_v1 · silver0.010200silver/cnn_dailymail_200 silver
cnn_lead3_n200_v1 · silver0.010200silver/cnn_dailymail_200 silver
cnn_mistral_m_n200_v1 · silver0.010200silver/cnn_dailymail_200 silver
cnn_openai_l_n200_v1 · silver0.010200silver/cnn_dailymail_200 silver
cnn_openai_s_n200_v1 · silver0.010200silver/cnn_dailymail_200 silver
cnn_bart_s_n200_v1 · silver0.005200silver/cnn_dailymail_200 silver
cnn_deepseek_s_n200_v1 · silver0.005200silver/cnn_dailymail_200 silver
cnn_glm_m_n200_v1 · silver0.005200silver/cnn_dailymail_200 silver
cnn_llama_m_n200_v1 · silver0.005200silver/cnn_dailymail_200 silver
cnn_mistral_l_n200_v1 · silver0.005200silver/cnn_dailymail_200 silver
cnn_anthropic_s_n200_v1 · silver0.000200silver/cnn_dailymail_200 silver
cnn_deepseek_l_n200_v1 · silver0.000200silver/cnn_dailymail_200 silver
cnn_deepseek_m_n200_v1 · silver0.000200silver/cnn_dailymail_200 silver
cnn_gemma_l_n200_v1 · silver0.000200silver/cnn_dailymail_200 silver
cnn_gemma_m_n200_v1 · silver0.000200silver/cnn_dailymail_200 silver
cnn_glm_s_n200_v1 · silver0.000200silver/cnn_dailymail_200 silver
cnn_llama_l_n200_v1 · silver0.000200silver/cnn_dailymail_200 silver
cnn_llama_s_n200_v1 · silver0.000200silver/cnn_dailymail_200 silver
cnn_mistral_s_n200_v1 · silver0.000200silver/cnn_dailymail_200 silver
cnn_openai_m_n200_v1 · silver0.000200silver/cnn_dailymail_200 silver
cnn_qwen_l_n200_v1 · silver0.000200silver/cnn_dailymail_200 silver
cnn_qwen_m_n200_v1 · silver0.000200silver/cnn_dailymail_200 silver
cnn_qwen_s_n200_v1 · silver0.000200silver/cnn_dailymail_200 silver

04on cnn_dailymail_dm200

gold/cnn_dailymail_dm200cnn_bart_m_dm200_v1: 0.02cnn_lead3_dm200_v1: 0.02cnn_bart_l_dm200_v1: 0.015cnn_bart_l_xsum_dm200_v1: 0.005cnn_bart_s_dm200_v1: 0.0050.005better · 0.02

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean fmt_narrationitemsscored against
cnn_bart_m_dm200_v10.020200gold/cnn_dailymail_dm200 gold
cnn_lead3_dm200_v10.020200gold/cnn_dailymail_dm200 gold
cnn_bart_l_dm200_v10.015200gold/cnn_dailymail_dm200 gold
cnn_bart_l_xsum_dm200_v10.005200gold/cnn_dailymail_dm200 gold
cnn_bart_s_dm200_v10.005200gold/cnn_dailymail_dm200 gold