metric
pair_agreement
- better
- higher
- kind
- quality: measured against a reference or by a judge
- decides
- no goal
- weighed by
- no goal
- recorded on
- summeval_100
01what it measures, and how
Pairwise mode: among the sampled pairs of an article's summaries that the experts rank apart, the share where the judge preferred the summary the experts prefer; a tie counts half and an unreadable answer 0. Chance is 0.5.
A project explains its own metric at length with METRIC_DESCRIPTIONS = {"pair_agreement": {"summary": "…", "details": "…"}}.
02on summeval_100
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | mean pair_agreement | items | scored against |
|---|---|---|---|
| se_gemma_l_n100_pairwise_v1 | 0.825 | 100 | gold/summeval_100 gold |
| se_qwen_s_n100_pairwise_v1 | 0.812 | 100 | gold/summeval_100 gold |
| se_random_n100_pairwise_v1 | 0.473 | 100 | gold/summeval_100 gold |