metric

`judge_score`

better: higher
kind: quality: measured against a reference or by a judge
decides: no goal
weighed by: no goal
recorded on: cnn_dailymail_200

01what it measures, and how

The judges' mean score for the output on the scorer's scale, over its dimensions and judges (evh judge-score). The judges were validated against human ratings.

evh judge-score shows each item and the outputs of several arms to two or more judges from different vendor families, blinded and in a seeded order, in blocks. Each judge scores every output on each of the scorer's dimensions; an arm's judge_score for an item is the mean over the dimensions and the judges. The scorer's validation names the committed runs and the agreement with people they reached; the pass also re-asks a fraction of items in another order (position) and again (noise), and reports how much the scores moved.

02on cnn_dailymail_200

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

arm	mean judge_score	items	scored against
cnn_anthropic_l_n200_v1 · judged/judge-summ-v1	4.976	197	judge `judge-summ-v1`
cnn_anthropic_m_n200_v1 · judged/judge-summ-v1	4.963	197	judge `judge-summ-v1`
cnn_anthropic_s_n200_v1 · judged/judge-summ-v1	4.952	197	judge `judge-summ-v1`
cnn_deepseek_m_n200_v1 · judged/judge-summ-v1	4.923	197	judge `judge-summ-v1`
cnn_anthropic_l_n200_v1 · judged/judge-summ-v2	4.888	200	judge `judge-summ-v2`
cnn_deepseek_l_n200_v1 · judged/judge-summ-v1	4.877	197	judge `judge-summ-v1`
cnn_qwen_m_n200_v1 · judged/judge-summ-v1	4.876	197	judge `judge-summ-v1`
cnn_glm_m_n200_v1 · judged/judge-summ-v1	4.861	197	judge `judge-summ-v1`
cnn_deepseek_m_n200_v1 · judged/judge-summ-v2	4.854	200	judge `judge-summ-v2`
cnn_openai_l_n200_v1 · judged/judge-summ-v2	4.847	200	judge `judge-summ-v2`
cnn_openai_s_n200_v1 · judged/judge-summ-v1	4.825	197	judge `judge-summ-v1`
cnn_deepseek_s_n200_v1 · judged/judge-summ-v1	4.818	197	judge `judge-summ-v1`
cnn_qwen_l_n200_v1 · judged/judge-summ-v1	4.812	197	judge `judge-summ-v1`
cnn_openai_m_n200_v1 · judged/judge-summ-v2	4.811	200	judge `judge-summ-v2`
cnn_anthropic_m_n200_v1 · judged/judge-summ-v2	4.809	200	judge `judge-summ-v2`
cnn_gemma_s_n200_v1 · judged/judge-summ-v1	4.801	197	judge `judge-summ-v1`
cnn_mistral_s_n200_v1 · judged/judge-summ-v1	4.798	197	judge `judge-summ-v1`
cnn_openai_m_n200_v1 · judged/judge-summ-v1	4.795	197	judge `judge-summ-v1`
cnn_openai_l_n200_v1 · judged/judge-summ-v1	4.773	197	judge `judge-summ-v1`
cnn_deepseek_l_n200_v1 · judged/judge-summ-v2	4.763	200	judge `judge-summ-v2`
cnn_openai_s_n200_v1 · judged/judge-summ-v2	4.747	200	judge `judge-summ-v2`
cnn_anthropic_s_n200_v1 · judged/judge-summ-v2	4.728	200	judge `judge-summ-v2`
cnn_glm_m_n200_v1 · judged/judge-summ-v2	4.718	200	judge `judge-summ-v2`
cnn_gemma_l_n200_v1 · judged/judge-summ-v1	4.715	197	judge `judge-summ-v1`
cnn_mistral_m_n200_v1 · judged/judge-summ-v1	4.708	197	judge `judge-summ-v1`
cnn_gemma_m_n200_v1 · judged/judge-summ-v1	4.707	197	judge `judge-summ-v1`
cnn_llama_s_n200_v1 · judged/judge-summ-v1	4.704	197	judge `judge-summ-v1`
cnn_llama_l_n200_v1 · judged/judge-summ-v1	4.700	197	judge `judge-summ-v1`
cnn_mistral_l_n200_v1 · judged/judge-summ-v1	4.698	197	judge `judge-summ-v1`
cnn_llama_m_n200_v1 · judged/judge-summ-v1	4.674	197	judge `judge-summ-v1`
cnn_deepseek_s_n200_v1 · judged/judge-summ-v2	4.644	200	judge `judge-summ-v2`
cnn_qwen_s_n200_v1 · judged/judge-summ-v1	4.609	197	judge `judge-summ-v1`
cnn_glm_l_n200_v1 · judged/judge-summ-v1	4.607	197	judge `judge-summ-v1`
cnn_gemma_l_n200_v1 · judged/judge-summ-v2	4.603	200	judge `judge-summ-v2`
cnn_qwen_m_n200_v1 · judged/judge-summ-v2	4.600	200	judge `judge-summ-v2`
cnn_gemma_m_n200_v1 · judged/judge-summ-v2	4.585	200	judge `judge-summ-v2`
cnn_qwen_l_n200_v1 · judged/judge-summ-v2	4.573	200	judge `judge-summ-v2`
cnn_glm_s_n200_v1 · judged/judge-summ-v1	4.530	197	judge `judge-summ-v1`
cnn_llama_l_n200_v1 · judged/judge-summ-v2	4.529	200	judge `judge-summ-v2`
cnn_llama_m_n200_v1 · judged/judge-summ-v2	4.488	200	judge `judge-summ-v2`
cnn_llama_s_n200_v1 · judged/judge-summ-v2	4.469	200	judge `judge-summ-v2`
cnn_qwen_s_n200_v1 · judged/judge-summ-v2	4.461	200	judge `judge-summ-v2`
cnn_gemma_s_n200_v1 · judged/judge-summ-v2	4.429	200	judge `judge-summ-v2`
cnn_glm_s_n200_v1 · judged/judge-summ-v2	4.420	200	judge `judge-summ-v2`
cnn_mistral_s_n200_v1 · judged/judge-summ-v2	4.408	200	judge `judge-summ-v2`
cnn_glm_l_n200_v1 · judged/judge-summ-v2	4.384	200	judge `judge-summ-v2`
cnn_mistral_m_n200_v1 · judged/judge-summ-v2	4.309	200	judge `judge-summ-v2`
cnn_mistral_l_n200_v1 · judged/judge-summ-v2	4.037	200	judge `judge-summ-v2`
cnn_bart_l_n200_v1 · judged/judge-summ-v1	3.868	197	judge `judge-summ-v1`
cnn_bart_l_n200_v1 · judged/judge-summ-v2	3.821	200	judge `judge-summ-v2`
cnn_bart_m_n200_v1 · judged/judge-summ-v1	3.564	197	judge `judge-summ-v1`
cnn_bart_l_xsum_n200_v1 · judged/judge-summ-v1	3.516	197	judge `judge-summ-v1`
cnn_lead3_n200_v1 · judged/judge-summ-v1	3.421	197	judge `judge-summ-v1`
cnn_bart_l_xsum_n200_v1 · judged/judge-summ-v2	3.366	200	judge `judge-summ-v2`
cnn_lead3_n200_v1 · judged/judge-summ-v2	3.339	200	judge `judge-summ-v2`
cnn_bart_m_n200_v1 · judged/judge-summ-v2	3.235	200	judge `judge-summ-v2`
cnn_bart_s_n200_v1 · judged/judge-summ-v1	3.185	197	judge `judge-summ-v1`
cnn_bart_s_n200_v1 · judged/judge-summ-v2	2.946	200	judge `judge-summ-v2`