metric

`judge_qwen_s`

better: higher
kind: quality: measured against a reference or by a judge
decides: no goal
weighed by: no goal
recorded on: cnn_dailymail_200

01what it measures, and how

Judge qwen_s's mean score for the output, on the scorer's scale.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"judge_qwen_s": {"summary": "…", "details": "…"}}.

02on cnn_dailymail_200

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

arm	mean judge_qwen_s	items	scored against
cnn_anthropic_l_n200_v1 · judged/judge-summ-v1	4.957	197	judge `judge-summ-v1`
cnn_anthropic_m_n200_v1 · judged/judge-summ-v1	4.940	197	judge `judge-summ-v1`
cnn_anthropic_s_n200_v1 · judged/judge-summ-v1	4.928	197	judge `judge-summ-v1`
cnn_deepseek_m_n200_v1 · judged/judge-summ-v1	4.879	197	judge `judge-summ-v1`
cnn_qwen_m_n200_v1 · judged/judge-summ-v1	4.830	197	judge `judge-summ-v1`
cnn_deepseek_l_n200_v1 · judged/judge-summ-v1	4.817	197	judge `judge-summ-v1`
cnn_glm_m_n200_v1 · judged/judge-summ-v1	4.786	197	judge `judge-summ-v1`
cnn_qwen_l_n200_v1 · judged/judge-summ-v1	4.773	197	judge `judge-summ-v1`
cnn_openai_s_n200_v1 · judged/judge-summ-v1	4.753	197	judge `judge-summ-v1`
cnn_deepseek_s_n200_v1 · judged/judge-summ-v1	4.730	197	judge `judge-summ-v1`
cnn_openai_m_n200_v1 · judged/judge-summ-v1	4.713	197	judge `judge-summ-v1`
cnn_gemma_s_n200_v1 · judged/judge-summ-v1	4.702	197	judge `judge-summ-v1`
cnn_mistral_s_n200_v1 · judged/judge-summ-v1	4.694	197	judge `judge-summ-v1`
cnn_openai_l_n200_v1 · judged/judge-summ-v1	4.678	197	judge `judge-summ-v1`
cnn_gemma_m_n200_v1 · judged/judge-summ-v1	4.586	197	judge `judge-summ-v1`
cnn_llama_s_n200_v1 · judged/judge-summ-v1	4.576	197	judge `judge-summ-v1`
cnn_mistral_m_n200_v1 · judged/judge-summ-v1	4.576	197	judge `judge-summ-v1`
cnn_gemma_l_n200_v1 · judged/judge-summ-v1	4.575	197	judge `judge-summ-v1`
cnn_llama_l_n200_v1 · judged/judge-summ-v1	4.560	197	judge `judge-summ-v1`
cnn_glm_l_n200_v1 · judged/judge-summ-v1	4.537	197	judge `judge-summ-v1`
cnn_llama_m_n200_v1 · judged/judge-summ-v1	4.519	197	judge `judge-summ-v1`
cnn_mistral_l_n200_v1 · judged/judge-summ-v1	4.509	197	judge `judge-summ-v1`
cnn_qwen_s_n200_v1 · judged/judge-summ-v1	4.489	197	judge `judge-summ-v1`
cnn_glm_s_n200_v1 · judged/judge-summ-v1	4.338	197	judge `judge-summ-v1`
cnn_bart_l_n200_v1 · judged/judge-summ-v1	3.607	197	judge `judge-summ-v1`
cnn_bart_m_n200_v1 · judged/judge-summ-v1	3.171	197	judge `judge-summ-v1`
cnn_bart_l_xsum_n200_v1 · judged/judge-summ-v1	3.145	197	judge `judge-summ-v1`
cnn_lead3_n200_v1 · judged/judge-summ-v1	3.122	197	judge `judge-summ-v1`
cnn_bart_s_n200_v1 · judged/judge-summ-v1	2.829	197	judge `judge-summ-v1`
cnn_anthropic_l_n200_v1 · judged/judge-summ-pick-v1	0.419	198	judge `judge-summ-pick-v1`
cnn_anthropic_s_n200_v1 · judged/judge-summ-pick-v1	0.212	198	judge `judge-summ-pick-v1`
cnn_anthropic_m_n200_v1 · judged/judge-summ-pick-v1	0.141	198	judge `judge-summ-pick-v1`
cnn_deepseek_m_n200_v1 · judged/judge-summ-pick-v1	0.076	198	judge `judge-summ-pick-v1`
cnn_qwen_l_n200_v1 · judged/judge-summ-pick-v1	0.071	198	judge `judge-summ-pick-v1`
cnn_qwen_m_n200_v1 · judged/judge-summ-pick-v1	0.025	198	judge `judge-summ-pick-v1`
cnn_deepseek_l_n200_v1 · judged/judge-summ-pick-v1	0.020	198	judge `judge-summ-pick-v1`
cnn_openai_s_n200_v1 · judged/judge-summ-pick-v1	0.010	198	judge `judge-summ-pick-v1`
cnn_gemma_m_n200_v1 · judged/judge-summ-pick-v1	0.005	198	judge `judge-summ-pick-v1`
cnn_llama_l_n200_v1 · judged/judge-summ-pick-v1	0.005	198	judge `judge-summ-pick-v1`
cnn_mistral_l_n200_v1 · judged/judge-summ-pick-v1	0.005	198	judge `judge-summ-pick-v1`
cnn_openai_l_n200_v1 · judged/judge-summ-pick-v1	0.005	198	judge `judge-summ-pick-v1`
cnn_openai_m_n200_v1 · judged/judge-summ-pick-v1	0.005	198	judge `judge-summ-pick-v1`
cnn_bart_l_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_bart_l_xsum_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_bart_m_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_bart_s_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_deepseek_s_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_gemma_l_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_gemma_s_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_glm_l_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_glm_m_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_glm_s_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_lead3_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_llama_m_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_llama_s_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_mistral_m_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_mistral_s_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`
cnn_qwen_s_n200_v1 · judged/judge-summ-pick-v1	0.000	198	judge `judge-summ-pick-v1`