judge_won

01what it measures, and how

The share of the item the arm won, by judges who picked the best output (1, or 1/k when k share the best) or compared outputs two at a time (the share of its pairs won): evh judge-score, mode pick or pairwise. The judges were validated in that mode against human preferences.

evh judge-score in pick mode shows every arm's output for an item at once, blind and shuffled, and each judge names the best; in pairwise mode it shows two at a time and the judge says which is better, for every pair. An arm's judge_won for an item is its share of that item (1 alone best, 1/k shared; its share of the pairs won), the mean over judges. Its mean over items is the arm's win rate; the summary adds Bradley-Terry strengths with intervals, and the position and noise checks.

02on cnn_dailymail_200

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold