metric
n_pairs
- better
- neutral: a description
- kind
- descriptive: says what the output was, never whether it was good
- decides
- no goal
- weighed by
- no goal
- recorded on
- summeval_100
01what it measures, and how
Pairwise mode: how many of the article's sampled pairs the experts ranked apart, the pairs pair_agreement is averaged over.
A project explains its own metric at length with METRIC_DESCRIPTIONS = {"n_pairs": {"summary": "…", "details": "…"}}.
02on summeval_100
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | mean n_pairs | items | scored against |
|---|---|---|---|
| se_gemma_l_n100_pairwise_v1 | 7.490 | 100 | gold/summeval_100 gold |
| se_qwen_s_n100_pairwise_v1 | 7.490 | 100 | gold/summeval_100 gold |
| se_random_n100_pairwise_v1 | 7.490 | 100 | gold/summeval_100 gold |