metric
type_penalty
- better
- neutral: a description
- kind
- descriptive: says what the output was, never whether it was good
- decides
- no goal
- weighed by
- no goal
- recorded on
- few_nerd_280
01what it measures, and how
untyped_f1 minus f1: the credit lost only to mislabelling entities the arm did find. Large means a typing problem, not a detection problem.
A project explains its own metric at length with METRIC_DESCRIPTIONS = {"type_penalty": {"summary": "…", "details": "…"}}.
02on few_nerd_280
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold