metric
parsed
- better
- higher
- kind
- quality: measured against a reference or by a judge (descriptive on summeval_100)
- decides
- no goal
- weighed by
- no goal
- recorded on
- ag_news_200, dbpedia_280, few_nerd_280, scifact_200, summeval_100
01what it measures, and how
Whether the reply could be read as an answer in the task's format: 1 or 0 per item, so its mean is the share readable (for the SummEval judges, the share of the four dimensions that came back readable). An unreadable reply scores 0 on the quality metrics, never skipped.
A project explains its own metric at length with METRIC_DESCRIPTIONS = {"parsed": {"summary": "…", "details": "…"}}.
02on ag_news_200
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
03on dbpedia_280
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
04on few_nerd_280
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
05on scifact_200
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
06on summeval_100
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold