metric
confidence
- better
- neutral: a description
- kind
- descriptive: says what the output was, never whether it was good
- decides
- no goal
- weighed by
- no goal
- recorded on
- ag_news_200, dbpedia_280
01what it measures, and how
The local classifier's probability for the label it chose (the top score of the fine-tune or the zero-shot NLI model). Not a quality: an input to calibration questions.
A project explains its own metric at length with METRIC_DESCRIPTIONS = {"confidence": {"summary": "…", "details": "…"}}.
02on ag_news_200
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | mean confidence | items | scored against |
|---|---|---|---|
| ag_bert_base_fy_n200_v1 | 0.989 | 200 | gold/ag_news_200 gold |
| ag_bert_base_fy_n200_v1 · linux | 0.989 | 200 | gold/ag_news_200 gold |
| ag_bert_base_ta_n200_v1 | 0.986 | 200 | gold/ag_news_200 gold |
| ag_bert_base_ta_n200_v1 · linux | 0.986 | 200 | gold/ag_news_200 gold |
| ag_bert_mini_n200_v1 · linux | 0.957 | 200 | gold/ag_news_200 gold |
| ag_bert_mini_n200_v1 | 0.957 | 200 | gold/ag_news_200 gold |
| ag_bart_mnli_n200_v1 | 0.572 | 200 | gold/ag_news_200 gold |
03on dbpedia_280
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | mean confidence | items | scored against |
|---|---|---|---|
| db_bert_base_fy_n200_v1 | 0.997 | 280 | gold/dbpedia_280 gold |
| db_bert_base_fy_n200_v1 · linux | 0.997 | 280 | gold/dbpedia_280 gold |
| db_bart_mnli_n200_v1 | 0.317 | 280 | gold/dbpedia_280 gold |