eval-harness

metric

grounding

better
higher
kind
quality: measured against a reference or by a judge
decides
no goal
weighed by
no goal
recorded on
cnn_dailymail_20, cnn_dailymail_200, cnn_dailymail_dm200

01what it measures, and how

Share of the summary's distinct word bigrams that appear in the article: high is extractive, low means the summary left the source (a paraphrase or a fabrication).

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"grounding": {"summary": "…", "details": "…"}}.

02on cnn_dailymail_20

gold/cnn_dailymail_20cnn_lead3_v1: 1cnn_bart_l_v1: 0.8710.871better · 1

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean groundingitemsscored against
cnn_lead3_v11.00020gold/cnn_dailymail_20 gold
cnn_bart_l_v10.87120gold/cnn_dailymail_20 gold

03on cnn_dailymail_200

gold/cnn_dailymail_200cnn_lead3_n200_v1: 1cnn_lead3_n200_v1 · linux: 1cnn_bart_l_n200_v1: 0.884cnn_bart_l_n200_v1 · linux: 0.884cnn_bart_s_n200_v1: 0.877cnn_bart_s_n200_v1 · linux: 0.877cnn_bart_s_n200_v1 · repeats: 0.877cnn_bart_m_n200_v1: 0.853cnn_bart_m_n200_v1 · linux: 0.853cnn_deepseek_m_n200_v1: 0.434cnn_bart_l_xsum_n200_v1: 0.422cnn_bart_l_xsum_n200_v1 · linux: 0.422cnn_llama_s_n200_v1: 0.41cnn_anthropic_l_n200_v1: 0.402cnn_llama_l_n200_v1: 0.396cnn_openai_l_n200_v1: 0.374cnn_llama_m_n200_v1: 0.369cnn_anthropic_s_n200_v1: 0.357cnn_deepseek_s_n200_v1: 0.353cnn_glm_s_n200_v1: 0.353cnn_anthropic_m_n200_v1: 0.349cnn_gemma_m_n200_v1: 0.345cnn_gemma_l_n200_v1: 0.345cnn_glm_m_n200_v1: 0.343cnn_openai_s_n200_v1: 0.34cnn_openai_m_n200_v1: 0.338cnn_deepseek_l_n200_v1: 0.333cnn_qwen_s_n200_v1: 0.332cnn_glm_l_n200_v1: 0.325cnn_qwen_l_n200_v1: 0.31cnn_mistral_s_n200_v1: 0.297cnn_qwen_m_n200_v1: 0.281cnn_mistral_m_n200_v1: 0.271cnn_gemma_s_n200_v1: 0.259cnn_mistral_l_n200_v1: 0.248silver/cnn_dailymail_200cnn_lead3_n200_v1 · silver: 1cnn_bart_l_n200_v1 · silver: 0.884cnn_bart_s_n200_v1 · silver: 0.877cnn_bart_m_n200_v1 · silver: 0.853cnn_deepseek_m_n200_v1 · silver: 0.434cnn_bart_l_xsum_n200_v1 · silver: 0.422cnn_llama_s_n200_v1 · silver: 0.41cnn_llama_l_n200_v1 · silver: 0.396cnn_openai_l_n200_v1 · silver: 0.374cnn_llama_m_n200_v1 · silver: 0.369cnn_anthropic_s_n200_v1 · silver: 0.357cnn_deepseek_s_n200_v1 · silver: 0.353cnn_glm_s_n200_v1 · silver: 0.353cnn_anthropic_m_n200_v1 · silver: 0.349cnn_gemma_m_n200_v1 · silver: 0.345cnn_gemma_l_n200_v1 · silver: 0.345cnn_glm_m_n200_v1 · silver: 0.343cnn_openai_s_n200_v1 · silver: 0.34cnn_openai_m_n200_v1 · silver: 0.338cnn_deepseek_l_n200_v1 · silver: 0.333cnn_qwen_s_n200_v1 · silver: 0.332cnn_glm_l_n200_v1 · silver: 0.325cnn_qwen_l_n200_v1 · silver: 0.31cnn_mistral_s_n200_v1 · silver: 0.297cnn_qwen_m_n200_v1 · silver: 0.281cnn_mistral_m_n200_v1 · silver: 0.271cnn_gemma_s_n200_v1 · silver: 0.259cnn_mistral_l_n200_v1 · silver: 0.2480.248better · 1

Not drawn: cnn_anthropic_l_n200_v1 · silver, scored against the reference it wrote (its row is in the table).

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean groundingitemsscored against
cnn_lead3_n200_v11.000200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v1 · linux1.000200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v10.884200gold/cnn_dailymail_200 gold
cnn_bart_l_n200_v1 · linux0.884200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v10.877200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · linux0.877200gold/cnn_dailymail_200 gold
cnn_bart_s_n200_v1 · repeats0.877200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v10.853200gold/cnn_dailymail_200 gold
cnn_bart_m_n200_v1 · linux0.853200gold/cnn_dailymail_200 gold
cnn_deepseek_m_n200_v10.434200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v10.422200gold/cnn_dailymail_200 gold
cnn_bart_l_xsum_n200_v1 · linux0.422200gold/cnn_dailymail_200 gold
cnn_llama_s_n200_v10.410200gold/cnn_dailymail_200 gold
cnn_anthropic_l_n200_v10.402200gold/cnn_dailymail_200 gold
cnn_llama_l_n200_v10.396200gold/cnn_dailymail_200 gold
cnn_openai_l_n200_v10.374200gold/cnn_dailymail_200 gold
cnn_llama_m_n200_v10.369200gold/cnn_dailymail_200 gold
cnn_anthropic_s_n200_v10.357200gold/cnn_dailymail_200 gold
cnn_deepseek_s_n200_v10.353200gold/cnn_dailymail_200 gold
cnn_glm_s_n200_v10.353199gold/cnn_dailymail_200 gold
cnn_anthropic_m_n200_v10.349200gold/cnn_dailymail_200 gold
cnn_gemma_m_n200_v10.345200gold/cnn_dailymail_200 gold
cnn_gemma_l_n200_v10.345200gold/cnn_dailymail_200 gold
cnn_glm_m_n200_v10.343200gold/cnn_dailymail_200 gold
cnn_openai_s_n200_v10.340200gold/cnn_dailymail_200 gold
cnn_openai_m_n200_v10.338200gold/cnn_dailymail_200 gold
cnn_deepseek_l_n200_v10.333200gold/cnn_dailymail_200 gold
cnn_qwen_s_n200_v10.332200gold/cnn_dailymail_200 gold
cnn_glm_l_n200_v10.325199gold/cnn_dailymail_200 gold
cnn_qwen_l_n200_v10.310200gold/cnn_dailymail_200 gold
cnn_mistral_s_n200_v10.297200gold/cnn_dailymail_200 gold
cnn_qwen_m_n200_v10.281200gold/cnn_dailymail_200 gold
cnn_mistral_m_n200_v10.271200gold/cnn_dailymail_200 gold
cnn_gemma_s_n200_v10.259200gold/cnn_dailymail_200 gold
cnn_mistral_l_n200_v10.248200gold/cnn_dailymail_200 gold
cnn_lead3_n200_v1 · silver1.000200silver/cnn_dailymail_200 silver
cnn_bart_l_n200_v1 · silver0.884200silver/cnn_dailymail_200 silver
cnn_bart_s_n200_v1 · silver0.877200silver/cnn_dailymail_200 silver
cnn_bart_m_n200_v1 · silver0.853200silver/cnn_dailymail_200 silver
cnn_deepseek_m_n200_v1 · silver0.434200silver/cnn_dailymail_200 silver
cnn_bart_l_xsum_n200_v1 · silver0.422200silver/cnn_dailymail_200 silver
cnn_llama_s_n200_v1 · silver0.410200silver/cnn_dailymail_200 silver
cnn_anthropic_l_n200_v1 · silver0.402200silver/cnn_dailymail_200 silver
cnn_llama_l_n200_v1 · silver0.396200silver/cnn_dailymail_200 silver
cnn_openai_l_n200_v1 · silver0.374200silver/cnn_dailymail_200 silver
cnn_llama_m_n200_v1 · silver0.369200silver/cnn_dailymail_200 silver
cnn_anthropic_s_n200_v1 · silver0.357200silver/cnn_dailymail_200 silver
cnn_deepseek_s_n200_v1 · silver0.353200silver/cnn_dailymail_200 silver
cnn_glm_s_n200_v1 · silver0.353199silver/cnn_dailymail_200 silver
cnn_anthropic_m_n200_v1 · silver0.349200silver/cnn_dailymail_200 silver
cnn_gemma_m_n200_v1 · silver0.345200silver/cnn_dailymail_200 silver
cnn_gemma_l_n200_v1 · silver0.345200silver/cnn_dailymail_200 silver
cnn_glm_m_n200_v1 · silver0.343200silver/cnn_dailymail_200 silver
cnn_openai_s_n200_v1 · silver0.340200silver/cnn_dailymail_200 silver
cnn_openai_m_n200_v1 · silver0.338200silver/cnn_dailymail_200 silver
cnn_deepseek_l_n200_v1 · silver0.333200silver/cnn_dailymail_200 silver
cnn_qwen_s_n200_v1 · silver0.332200silver/cnn_dailymail_200 silver
cnn_glm_l_n200_v1 · silver0.325199silver/cnn_dailymail_200 silver
cnn_qwen_l_n200_v1 · silver0.310200silver/cnn_dailymail_200 silver
cnn_mistral_s_n200_v1 · silver0.297200silver/cnn_dailymail_200 silver
cnn_qwen_m_n200_v1 · silver0.281200silver/cnn_dailymail_200 silver
cnn_mistral_m_n200_v1 · silver0.271200silver/cnn_dailymail_200 silver
cnn_gemma_s_n200_v1 · silver0.259200silver/cnn_dailymail_200 silver
cnn_mistral_l_n200_v1 · silver0.248200silver/cnn_dailymail_200 silver

04on cnn_dailymail_dm200

gold/cnn_dailymail_dm200cnn_lead3_dm200_v1: 1cnn_bart_s_dm200_v1: 0.903cnn_bart_l_dm200_v1: 0.899cnn_bart_m_dm200_v1: 0.88cnn_bart_l_xsum_dm200_v1: 0.5510.551better · 1

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean groundingitemsscored against
cnn_lead3_dm200_v11.000200gold/cnn_dailymail_dm200 gold
cnn_bart_s_dm200_v10.903200gold/cnn_dailymail_dm200 gold
cnn_bart_l_dm200_v10.899200gold/cnn_dailymail_dm200 gold
cnn_bart_m_dm200_v10.880200gold/cnn_dailymail_dm200 gold
cnn_bart_l_xsum_dm200_v10.551200gold/cnn_dailymail_dm200 gold