metric
judge_score_unvalidated
- better
- higher
- kind
- quality: measured against a reference or by a judge
- decides
- no goal
- weighed by
- no goal
- recorded on
- cnn_dailymail_200
01what it measures, and how
The judges' mean score for the output on the scorer's scale, over its dimensions and judges (evh judge-score). The judges were not validated against human ratings: a trial, not a measurement.
Computed as judge_score is, by judges that were never compared with human ratings for this use. It ranks arms by what the judges prefer, which may not be what people prefer: read it as a trial. A validated scorer turns it into judge_score.
02on cnn_dailymail_200
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | mean judge_score_unvalidated | items | scored against |
|---|---|---|---|
| cnn_deepseek_m_n200_v1 · judged/judge-summ-pointwise-v1 | 4.940 | 200 | judge judge-summ-pointwise-v1 |
| cnn_bart_l_n200_v1 · judged/judge-summ-pointwise-v1 | 4.181 | 200 | judge judge-summ-pointwise-v1 |