metric
input_truncated
- better
- neutral: a description
- kind
- descriptive: says what the output was, never whether it was good
- decides
- no goal
- weighed by
- no goal
- recorded on
- cnn_dailymail_20, cnn_dailymail_200, cnn_dailymail_dm200, summeval_100
01what it measures, and how
1 when a local model saw only part of the article, because it was longer than the model's encoder window (1,024 tokens for BART) and was cut, else 0. Those items are a different task from the hosted arms' full-article ones.
A project explains its own metric at length with METRIC_DESCRIPTIONS = {"input_truncated": {"summary": "…", "details": "…"}}.
02on cnn_dailymail_20
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | mean input_truncated | items | scored against |
|---|---|---|---|
| cnn_bart_l_v1 | 0.200 | 20 | gold/cnn_dailymail_20 gold |
03on cnn_dailymail_200
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | mean input_truncated | items | scored against |
|---|---|---|---|
| cnn_bart_s_n200_v1 · repeats | 0.750 | 4 | gold/cnn_dailymail_200 gold |
| cnn_bart_l_n200_v1 | 0.230 | 200 | gold/cnn_dailymail_200 gold |
| cnn_bart_l_xsum_n200_v1 | 0.230 | 200 | gold/cnn_dailymail_200 gold |
| cnn_bart_m_n200_v1 | 0.230 | 200 | gold/cnn_dailymail_200 gold |
| cnn_bart_s_n200_v1 | 0.230 | 200 | gold/cnn_dailymail_200 gold |
| cnn_bart_l_n200_v1 · linux | 0.230 | 200 | gold/cnn_dailymail_200 gold |
| cnn_bart_l_xsum_n200_v1 · linux | 0.230 | 200 | gold/cnn_dailymail_200 gold |
| cnn_bart_m_n200_v1 · linux | 0.230 | 200 | gold/cnn_dailymail_200 gold |
| cnn_bart_s_n200_v1 · linux | 0.230 | 200 | gold/cnn_dailymail_200 gold |
04on cnn_dailymail_dm200
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | mean input_truncated | items | scored against |
|---|---|---|---|
| cnn_bart_l_dm200_v1 | 0.290 | 200 | gold/cnn_dailymail_dm200 gold |
| cnn_bart_l_xsum_dm200_v1 | 0.290 | 200 | gold/cnn_dailymail_dm200 gold |
| cnn_bart_m_dm200_v1 | 0.290 | 200 | gold/cnn_dailymail_dm200 gold |
| cnn_bart_s_dm200_v1 | 0.290 | 200 | gold/cnn_dailymail_dm200 gold |
05on summeval_100
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
| arm | mean input_truncated | items | scored against |
|---|---|---|---|
| se_bartscore_n100_v1 | 0.000 | 100 | gold/summeval_100 gold |