dataset
scifact_40
40 items · 0 arms. Headline: each arm's own primary metric, mean over its items; arms with different primary metrics are not ranked against each other.
01arms
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | metric | mean | items | cost per item | speed per item | runs | scored against |
|---|
02what's inside
40 items, text, 6–25 words (median 13, middle half 10–15).
Source: BeIR/scifact (queries split). The text stays there; each item below links to its row.
the gold
gold/scifact_40: a list per item, 1–5 entries, median 1
| item | words | gold | open |
|---|---|---|---|
1024 | 11 | 1 entries | source ↗ |
1041 | 15 | 2 entries | source ↗ |
1121 | 13 | 1 entries | source ↗ |
1130 | 17 | 1 entries | source ↗ |
1144 | 17 | 1 entries | source ↗ |
1175 | 8 | 1 entries | source ↗ |
118 | 12 | 1 entries | source ↗ |
1191 | 11 | 1 entries | source ↗ |
1196 | 12 | 1 entries | source ↗ |
1199 | 18 | 1 entries | source ↗ |
127 | 10 | 1 entries | source ↗ |
1270 | 14 | 1 entries | source ↗ |
1273 | 11 | 1 entries | source ↗ |
1274 | 25 | 3 entries | source ↗ |
1278 | 15 | 1 entries | source ↗ |
133 | 19 | 5 entries | source ↗ |
1389 | 8 | 1 entries | source ↗ |
180 | 19 | 1 entries | source ↗ |
218 | 8 | 1 entries | source ↗ |
239 | 8 | 1 entries | source ↗ |
294 | 12 | 1 entries | source ↗ |
295 | 18 | 1 entries | source ↗ |
303 | 13 | 1 entries | source ↗ |
362 | 24 | 1 entries | source ↗ |
399 | 11 | 1 entries | source ↗ |
415 | 13 | 1 entries | source ↗ |
439 | 13 | 1 entries | source ↗ |
501 | 7 | 1 entries | source ↗ |
507 | 15 | 1 entries | source ↗ |
533 | 7 | 1 entries | source ↗ |
619 | 15 | 2 entries | source ↗ |
659 | 7 | 1 entries | source ↗ |
783 | 13 | 1 entries | source ↗ |
785 | 15 | 1 entries | source ↗ |
800 | 19 | 1 entries | source ↗ |
832 | 6 | 1 entries | source ↗ |
834 | 10 | 1 entries | source ↗ |
847 | 18 | 1 entries | source ↗ |
882 | 10 | 1 entries | source ↗ |
914 | 7 | 1 entries | source ↗ |
Each row is the profile evh dataset-items --dataset-id scifact_40 prints; inspect opens the item in the inspector with the dataset's largest group of comparable arms, and its text where the inspector runs on a machine that has it.
03reference sets
- gold/scifact_40 gold · promoted from person the dataset's authors
04labels
No labels yet. A sheet to fill: evh label-sheet --dataset-id scifact_40 --out sheet.csv; a run's outputs as a model set: evh label-from-run --run R --set S.