metric
top1_chance
- better
- neutral: a description
- kind
- descriptive: says what the output was, never whether it was good
- decides
- no goal
- weighed by
- no goal
- recorded on
- summeval_100
01what it measures, and how
Pick mode: what a random pick scores on the article, the share of its 16 summaries the experts rate best (|best set| / 16). A judge's top1 is read against its mean.
A project explains its own metric at length with METRIC_DESCRIPTIONS = {"top1_chance": {"summary": "…", "details": "…"}}.
02on summeval_100
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | mean top1_chance | items | scored against |
|---|---|---|---|
| se_gemma_l_n100_pick_v1 | 0.088 | 100 | gold/summeval_100 gold |
| se_qwen_s_n100_pick_v1 | 0.088 | 100 | gold/summeval_100 gold |
| se_random_n100_pick_v1 | 0.088 | 100 | gold/summeval_100 gold |