metric
top1
- better
- higher
- kind
- quality: measured against a reference or by a judge
- decides
- no goal
- weighed by
- no goal
- recorded on
- summeval_100
01what it measures, and how
Pick mode: how often the judge's pick of the best of an article's 16 summaries is a summary the experts rate best (highest mean over the four dimensions; ties among the experts form the best set). 1 for a correct single pick; a pick of several scores the share of them that are correct; an unreadable pick scores 0.
A project explains its own metric at length with METRIC_DESCRIPTIONS = {"top1": {"summary": "…", "details": "…"}}.
02on summeval_100
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | mean top1 | items | scored against |
|---|---|---|---|
| se_gemma_l_n100_pick_v1 | 0.364 | 100 | gold/summeval_100 gold |
| se_qwen_s_n100_pick_v1 | 0.263 | 100 | gold/summeval_100 gold |
| se_random_n100_pick_v1 | 0.070 | 100 | gold/summeval_100 gold |