dataset
cnn_dailymail_20
20 items · 2 arms. Headline: each arm's own primary metric, mean over its items; arms with different primary metrics are not ranked against each other.
Inspect the 2 arms on coverage, scored against gold/cnn_dailymail_20Open its sheet
goals: Short news summaries (CNN)
01arms
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | metric | mean | items | cost per item | speed per item | runs | scored against | |
|---|---|---|---|---|---|---|---|---|
| cnn_bart_l_v1 | coverage | 0.366 | 20 | $0.000000 · 732 tok | p50 11,468 · p95 17,811 ms | 1 | gold/cnn_dailymail_20 gold | inspect |
| cnn_lead3_v1 | coverage | 0.262 | 20 | $0.000000 | p50 0.069 · p95 0.13 ms | 1 | gold/cnn_dailymail_20 gold | inspect |
02what's inside
20 items, text, 111–1133 words (median 465, middle half 361–738).
Source: abisee/cnn_dailymail (test split). The text stays there; each item below links to its row.
the gold
gold/cnn_dailymail_20: text, 18–56 words, median 37
| item | words | gold | open |
|---|---|---|---|
0d3c8c276d07 | 361 | 44 words | source ↗ · inspect |
1b2cc634e2bf | 440 | 24 words | source ↗ · inspect |
1dd00c89d71a | 954 | 45 words | source ↗ · inspect |
203886369fee | 465 | 32 words | source ↗ · inspect |
230c52285499 | 411 | 44 words | source ↗ · inspect |
2bd8ada1de6a | 113 | 21 words | source ↗ · inspect |
384175be1c8d | 498 | 27 words | source ↗ · inspect |
4495ba8f3a34 | 704 | 38 words | source ↗ · inspect |
469c6ac05092 | 1133 | 52 words | source ↗ · inspect |
6222f33c2c79 | 289 | 28 words | source ↗ · inspect |
88e7dac40f3e | 805 | 29 words | source ↗ · inspect |
a38e72fed886 | 202 | 45 words | source ↗ · inspect |
c222979bd1cf | 738 | 37 words | source ↗ · inspect |
c27cf1b136cc | 364 | 43 words | source ↗ · inspect |
dc833f8b55e3 | 1028 | 47 words | source ↗ · inspect |
e2706dce6cf2 | 773 | 56 words | source ↗ · inspect |
ec4b5fef725b | 389 | 35 words | source ↗ · inspect |
ee17dfb574fe | 111 | 33 words | source ↗ · inspect |
eeafdc8b2d81 | 262 | 18 words | source ↗ · inspect |
f001ec5c4704 | 567 | 36 words | source ↗ · inspect |
Each row is the profile evh dataset-items --dataset-id cnn_dailymail_20 prints; inspect opens the item in the inspector with the dataset's largest group of comparable arms, and its text where the inspector runs on a machine that has it.
03reference sets
- gold/cnn_dailymail_20 gold · promoted from person the dataset's authors
04labels
No labels yet. A sheet to fill: evh label-sheet --dataset-id cnn_dailymail_20 --out sheet.csv; a run's outputs as a model set: evh label-from-run --run R --set S.