metric

`rho_coherence`

better: higher
kind: quality: measured against a reference or by a judge
decides: no goal
weighed by: no goal
recorded on: summeval_100

01what it measures, and how

Spearman's rho with the experts' coherence ratings across an article's 16 summaries (0 when the judge returned no coherence scores).

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"rho_coherence": {"summary": "…", "details": "…"}}.

02on summeval_100

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

arm	mean rho_coherence	items	scored against
se_gemma_l_pubtext_n100_v1 · pair	0.669	100	gold/summeval_100 gold
se_gemma_l_n100_v1	0.659	100	gold/summeval_100 gold
se_gemma_l_pubtextcw_n100_v1 · pair	0.654	100	gold/summeval_100 gold
se_openai_l_n100_v1	0.646	100	gold/summeval_100 gold
se_qwen_m_n100_v1	0.625	100	gold/summeval_100 gold
se_anthropic_l_n100_v1	0.606	100	gold/summeval_100 gold
se_qwen37max_n100_v1	0.598	100	gold/summeval_100 gold
se_anthropic_m_n100_v1	0.594	100	gold/summeval_100 gold
se_qwen_s_n100_v1	0.576	100	gold/summeval_100 gold
se_mistral_m_n100_v1	0.576	100	gold/summeval_100 gold
se_unievalpub_n100_v1	0.575	100	gold/summeval_100 gold
se_openai_m_n100_v1	0.574	100	gold/summeval_100 gold
se_glm_m_n100_v1	0.562	100	gold/summeval_100 gold
se_glm_l_n100_v1	0.558	100	gold/summeval_100 gold
se_mistral_l_n100_v1	0.552	100	gold/summeval_100 gold
se_unieval_n100_v1	0.552	100	gold/summeval_100 gold
se_anthropic_s_n100_v1	0.548	100	gold/summeval_100 gold
se_deepseek_m_n100_v1	0.543	100	gold/summeval_100 gold
se_deepseek_l_n100_v1	0.536	100	gold/summeval_100 gold
se_gemma_m_n100_v1	0.532	100	gold/summeval_100 gold
se_glm_s_n100_v1	0.524	100	gold/summeval_100 gold
se_llama_l_n100_v1	0.510	100	gold/summeval_100 gold
se_deepseek_s_n100_v1	0.509	100	gold/summeval_100 gold
se_bartscore_n100_v1	0.492	100	gold/summeval_100 gold
se_gemma_s_n100_v1	0.450	100	gold/summeval_100 gold
se_mistral_s_n100_v1	0.420	100	gold/summeval_100 gold
se_llama_m_n100_v1	0.419	100	gold/summeval_100 gold
se_openai_s_n100_v1	0.400	100	gold/summeval_100 gold
se_llama_s_n100_v1	0.375	100	gold/summeval_100 gold
se_rougeL_n100_v1	0.205	100	gold/summeval_100 gold
se_rouge2_n100_v1	0.183	100	gold/summeval_100 gold
se_rouge1_n100_v1	0.172	100	gold/summeval_100 gold
se_coverage_n100_v1	0.125	100	gold/summeval_100 gold
se_random_n100_v1	0.014	100	gold/summeval_100 gold
se_length_n100_v1	-0.020	100	gold/summeval_100 gold