metric

`tau_coherence`

better: higher
kind: quality: measured against a reference or by a judge
decides: no goal
weighed by: no goal
recorded on: summeval_100

01what it measures, and how

Kendall's tau-b with the experts' coherence ratings across an article's 16 summaries (0 when the judge returned no coherence scores).

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"tau_coherence": {"summary": "…", "details": "…"}}.

02on summeval_100

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

arm	mean tau_coherence	items	scored against
se_gemma_l_pubtext_n100_v1 · pair	0.575	100	gold/summeval_100 gold
se_gemma_l_n100_v1	0.566	100	gold/summeval_100 gold
se_gemma_l_pubtextcw_n100_v1 · pair	0.562	100	gold/summeval_100 gold
se_openai_l_n100_v1	0.550	100	gold/summeval_100 gold
se_qwen_m_n100_v1	0.529	100	gold/summeval_100 gold
se_anthropic_l_n100_v1	0.521	100	gold/summeval_100 gold
se_qwen37max_n100_v1	0.509	100	gold/summeval_100 gold
se_anthropic_m_n100_v1	0.504	100	gold/summeval_100 gold
se_openai_m_n100_v1	0.490	100	gold/summeval_100 gold
se_mistral_m_n100_v1	0.489	100	gold/summeval_100 gold
se_qwen_s_n100_v1	0.484	100	gold/summeval_100 gold
se_glm_l_n100_v1	0.481	100	gold/summeval_100 gold
se_glm_m_n100_v1	0.480	100	gold/summeval_100 gold
se_mistral_l_n100_v1	0.470	100	gold/summeval_100 gold
se_deepseek_m_n100_v1	0.467	100	gold/summeval_100 gold
se_anthropic_s_n100_v1	0.466	100	gold/summeval_100 gold
se_deepseek_l_n100_v1	0.457	100	gold/summeval_100 gold
se_gemma_m_n100_v1	0.455	100	gold/summeval_100 gold
se_glm_s_n100_v1	0.447	100	gold/summeval_100 gold
se_unievalpub_n100_v1	0.442	100	gold/summeval_100 gold
se_deepseek_s_n100_v1	0.441	100	gold/summeval_100 gold
se_llama_l_n100_v1	0.433	100	gold/summeval_100 gold
se_unieval_n100_v1	0.424	100	gold/summeval_100 gold
se_gemma_s_n100_v1	0.386	100	gold/summeval_100 gold
se_bartscore_n100_v1	0.380	100	gold/summeval_100 gold
se_mistral_s_n100_v1	0.359	100	gold/summeval_100 gold
se_llama_m_n100_v1	0.354	100	gold/summeval_100 gold
se_openai_s_n100_v1	0.340	100	gold/summeval_100 gold
se_llama_s_n100_v1	0.313	100	gold/summeval_100 gold
se_rougeL_n100_v1	0.153	100	gold/summeval_100 gold
se_rouge2_n100_v1	0.138	100	gold/summeval_100 gold
se_rouge1_n100_v1	0.130	100	gold/summeval_100 gold
se_coverage_n100_v1	0.094	100	gold/summeval_100 gold
se_random_n100_v1	0.010	100	gold/summeval_100 gold
se_length_n100_v1	-0.014	100	gold/summeval_100 gold