dataset
smoke_v1
5 items · 1 arms. Headline: each arm's own primary metric, mean over its items; arms with different primary metrics are not ranked against each other.
baseline, promoted 2026-10-23
The worked example a fresh clone ships: the bundled demo arm on smoke_v1, so V6 and the walked loop resolve inside the committed set.
01arms
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | metric | mean | items | cost per item | speed per item | runs | scored against | |
|---|---|---|---|---|---|---|---|---|
| demo_v1 baseline | overlap_f1 | 0.416 | 5 | $0.000000 | p50 0.007 · p95 0.015 ms | 1 | silver/smoke_v1 silver | inspect |
02what's inside
5 items, text, 6–42 words (median 42, middle half 42–42).
| item | words | gold | open |
|---|---|---|---|
item_01 | 42 | — | — |
item_02 | 42 | — | — |
item_03 | 42 | — | — |
item_04 | 42 | — | — |
item_05 | 6 | — | — |
Each row is the profile evh dataset-items --dataset-id smoke_v1 prints; inspect opens the item in the inspector with the dataset's largest group of comparable arms, and its text where the inspector runs on a machine that has it.
03reference sets
- silver/smoke_v1 silver · promoted from model
silver_smoke_v1the arm:reference_opus_v1
04labels
| set | made on | coverage | guideline | made | promoted |
|---|---|---|---|---|---|
model silver_smoke_v1 | this one | 5 of 5 | none | model 2026-11-01 | silver/smoke_v1 silver |