metric
overlap_f1
- better
- higher
- kind
- quality: measured against a reference or by a judge
- decides
- no goal
- weighed by
- no goal
- recorded on
- smoke_v1
01what it measures, and how
Token-overlap F1 against the reference: the bundled adapter's placeholder, enough to run the loop, not to defend a decision.
The bundled adapter's score: the output and the reference as sets of lowercased words, precision is the share of the output's words in the reference, recall the share of the reference's words in the output, and F1 their harmonic mean. Word order, synonyms and repetition do not count. Replace score() with what your task rewards.
02on smoke_v1
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | mean overlap_f1 | items | scored against |
|---|---|---|---|
| demo_v1 | 0.416 | 5 | silver/smoke_v1 silver |