metrics

Metrics

What each number on these pages measures, which way is better, where it is recorded, and which goals decide on it. A project describes its own metrics with METRIC_DESCRIPTIONS in its adapter; the harness describes its own.

all time, 2026-09-25 to 2026-11-02

01metrics

group by

96 shown

metric	role	what it measures	better	recorded on	decides · weighs	arms recording it in period
`answer_words`	describes	How many whitespace-separated words the reply had: a description, not a quality.	neutral: a description	ag_news_200 63 arms dbpedia_280 29 arms	—	62
`class_match`	describes	1 when all three flags (operational, test, low_signal) are the pinned code's, else 0.	higher	fleet_dedup_v1 1 arm	—	1
`compression`	describes	Summary words divided by article words: a description, not a quality.	neutral: a description	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	—	42
`concision`	describes	ROUGE-Lsum precision over the whole summary: the share of what was written that matches the reference. Padding lowers it.	higher	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	—	42
`confidence`	describes	The local classifier's probability for the label it chose (the top score of the fine-tune or the zero-shot NLI model). Not a quality: an input to calibration questions.	neutral: a description	ag_news_200 7 arms dbpedia_280 3 arms	—	10
`correct`	decides	1 when the decision is one the label accepts (file, dismiss, escalate or cleanup), else 0.	higher	ag_news_200 63 arms dbpedia_280 29 arms orrery_triage_v1_labelled 336 arms	Encyclopedia topic classification (DBpedia) News topic classification (AG News)	145
`correct_after_repair`	describes	1 when the answer is correct only because the parser read it: the label was right but the reply was not exactly the label. A large share means the accuracy partly measures the parser.	neutral: a description	ag_news_200 63 arms dbpedia_280 29 arms	—	62
`cost_usd`	weighs	What one item cost, in USD: the provider's bill when the response carries one, else the tokens times the config's prices.	lower	ag_news_200 63 arms cnn_dailymail_20 2 arms cnn_dailymail_200 153 arms cnn_dailymail_dm200 5 arms dbpedia_280 29 arms few_nerd_280 28 arms orrery_triage_v1_labelled 326 arms scifact_200 26 arms smoke_v1 1 arm summeval_100 41 arms	weighed by Named-entity extraction (Few-NERD), Encyclopedia topic classification (DBpedia), Short news summaries (CNN), News topic classification (AG News), Evidence retrieval for scientific claims (SciFact), A scorer that agrees with experts on summaries (SummEval), Triage decisions on Orrery's error signals	281
`coverage`	decides	ROUGE-Lsum recall with the summary cut to the reference's length: did it carry the story.	higher	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	Short news summaries (CNN)	42
`deferred`	describes	1 when the fleet deferred the decision twice for the same reason, so none was reached.	lower	orrery_triage_v1_labelled 107 arms	—	32
`duplicate_ids`	describes	Ids the ranking repeated; only the first occurrence counts.	neutral: a description	scifact_200 26 arms	—	26
`escalated`	describes	1 when the decision hands the signal to a person (escalate), labelled or not.	lower	orrery_triage_v1_labelled 336 arms	—	83
`f1`	decides	Typed entity F1 for the item: an entity counts when its text and its type both match a gold one.	higher	few_nerd_280 28 arms	Named-entity extraction (Few-NERD)	28
`false_alarm`	weighs	1 when noise (the label: dismiss or cleanup) was filed or escalated.	lower	orrery_triage_v1_labelled 336 arms	weighed by Triage decisions on Orrery's error signals	83
`false_dismiss`	describes	1 when a real problem (the label: file or escalate) was dismissed or cleaned up: a miss.	lower	orrery_triage_v1_labelled 336 arms	—	83
`fmt_bullets`	describes	1 when the output has bullet lines where the prompt asks for sentences.	neutral: a description	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	—	42
`fmt_label`	describes	1 when the output opens with a label such as "Summary:" the prompt did not ask for.	neutral: a description	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	—	42
`fmt_markdown`	describes	1 when the output opens with markdown (bold, a heading, a quote) the prompt did not ask for.	neutral: a description	ag_news_200 63 arms cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms dbpedia_280 29 arms	—	104
`fmt_narration`	describes	1 when the output thinks out loud ("the user wants", "let me draft", think tags) instead of only summarising: its quality numbers describe something else.	neutral: a description	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	—	42
`fmt_overlong`	describes	1 when the output runs past 90 words where the prompt asks for 2-3 short sentences.	neutral: a description	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	—	42
`fmt_paragraphs`	describes	1 when the output has more than one paragraph where the prompt asks for 2-3 short sentences.	neutral: a description	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	—	42
`fmt_verbose`	describes	1 when the reply ran past two words, where the prompt asks for the label alone.	neutral: a description	ag_news_200 63 arms dbpedia_280 29 arms	—	62
`fn`	describes	Typed false negatives: gold entities no prediction matched.	neutral: a description	few_nerd_280 28 arms	—	28
`found_any`	describes	1 when any relevant document appears anywhere in the returned list.	neutral: a description	scifact_200 26 arms	—	26
`fp`	describes	Typed false positives: predicted entities that matched no gold one.	neutral: a description	few_nerd_280 28 arms	—	28
`grounding`	describes	Share of the summary's distinct word bigrams that appear in the article: high is extractive, low means the summary left the source (a paraphrase or a fabrication).	higher	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	—	42
`input_truncated`	describes	1 when a local model saw only part of the article, because it was longer than the model's encoder window (1,024 tokens for BART) and was cut, else 0. Those items are a different task from the hosted arms' full-article ones.	neutral: a description	cnn_dailymail_20 1 arm cnn_dailymail_200 9 arms cnn_dailymail_dm200 4 arms summeval_100 1 arm	—	15
`judge_anthropic_l`	describes	Judge anthropic_l's mean score for the output, on the scorer's scale.	higher	cnn_dailymail_200 29 arms	—	0
`judge_gemma_l`	describes	Judge gemma_l's mean score for the output, on the scorer's scale.	higher	cnn_dailymail_200 58 arms	—	0
`judge_gemma_l_unvalidated`	describes	Judge gemma_l's mean score for the output, on the scorer's scale, not validated against human ratings.	higher	cnn_dailymail_200 2 arms	—	0
`judge_openai_l`	describes	Judge openai_l's mean score for the output, on the scorer's scale.	higher	cnn_dailymail_200 29 arms	—	0
`judge_qwen_s`	describes	Judge qwen_s's mean score for the output, on the scorer's scale.	higher	cnn_dailymail_200 58 arms	—	0
`judge_qwen_s_unvalidated`	describes	Judge qwen_s's mean score for the output, on the scorer's scale, not validated against human ratings.	higher	cnn_dailymail_200 2 arms	—	0
`judge_score`	describes	The judges' mean score for the output on the scorer's scale, over its dimensions and judges (evh judge-score). The judges were validated against human ratings.	higher	cnn_dailymail_200 58 arms	—	0
`judge_score_unvalidated`	describes	The judges' mean score for the output on the scorer's scale, over its dimensions and judges (evh judge-score). The judges were not validated against human ratings: a trial, not a measurement.	higher	cnn_dailymail_200 2 arms	—	0
`judge_won`	describes	The share of the item the arm won, by judges who picked the best output (1, or 1/k when k share the best) or compared outputs two at a time (the share of its pairs won): evh judge-score, mode pick or pairwise. The judges were validated in that mode against human preferences.	higher	cnn_dailymail_200 29 arms	—	0
`key_match`	describes	1 when the signal's normalized key is the one the pinned fleet code gives it, else 0.	higher	fleet_dedup_v1 1 arm	—	1
`latency_ms`	weighs	Wall-clock time of one call, in milliseconds.	lower	ag_news_200 63 arms cnn_dailymail_20 2 arms cnn_dailymail_200 153 arms cnn_dailymail_dm200 5 arms dbpedia_280 29 arms few_nerd_280 28 arms fleet_dedup_v1 2 arms orrery_triage_v1_labelled 336 arms scifact_200 26 arms smoke_v1 1 arm summeval_100 41 arms	weighed by Named-entity extraction (Few-NERD), Encyclopedia topic classification (DBpedia), Short news summaries (CNN), News topic classification (AG News), Evidence retrieval for scientific claims (SciFact), A scorer that agrees with experts on summaries (SummEval), Triage decisions on Orrery's error signals	285
`length_vs_reference`	describes	Summary words divided by reference words: 1 means as long as the human's. A verbose winner shows here.	neutral: a description	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	—	42
`llm_named_unknown`	describes	Ids the reranking LLM named that were not among the candidates it was shown: invented or out-of-list documents, which are dropped from its ordering.	neutral: a description	scifact_200 19 arms	—	19
`llm_parsed`	describes	1 when the reranking LLM's reply was a readable ordering. The pipeline always ranks (BM25's order stands when the LLM fails), so this says whether the LLM contributed.	neutral: a description	scifact_200 19 arms	—	19
`model_correct`	describes	correct, for the model's own decision before the fleet's gates.	higher	orrery_triage_v1_labelled 336 arms	—	83
`model_escalated`	describes	escalated, for the model's own decision before the fleet's gates.	lower	orrery_triage_v1_labelled 336 arms	—	83
`model_false_alarm`	describes	false_alarm of the model's own decision, before the fleet's gates.	lower	orrery_triage_v1_labelled 336 arms	—	83
`model_false_dismiss`	describes	false_dismiss of the model's own decision, before the fleet's gates.	lower	orrery_triage_v1_labelled 336 arms	—	83
`model_triage_cost`	describes	triage_cost of the model's own decision, before the fleet's gates changed it.	lower	orrery_triage_v1_labelled 317 arms	—	64
`mrr_10`	describes	Reciprocal rank of the first relevant document if it is in the first 10, else 0.	higher	scifact_200 26 arms	—	26
`n_gold`	describes	How many gold entities the item has, after normalisation.	neutral: a description	few_nerd_280 28 arms	—	28
`n_pairs`	describes	Pairwise mode: how many of the article's sampled pairs the experts ranked apart, the pairs pair_agreement is averaged over.	neutral: a description	summeval_100 3 arms	—	3
`n_predicted`	describes	How many entities the arm returned for the item, after normalisation.	neutral: a description	few_nerd_280 28 arms	—	28
`n_probes`	describes	How many probes the model asked for before it decided.	higher	orrery_triage_v1_labelled 336 arms	—	83
`n_relevant`	describes	How many documents are judged relevant (grade above 0) for the query.	neutral: a description	scifact_200 26 arms	—	26
`n_returned`	describes	How many distinct document ids the ranking returned.	neutral: a description	scifact_200 26 arms	—	26
`ndcg_10`	decides	nDCG@10: how high the relevant documents sit in the first 10, against the best possible order.	higher	scifact_200 26 arms	Evidence retrieval for scientific claims (SciFact)	26
`no_decision`	describes	1 when the fleet reached no decision (an empty last reply, or the same deferral twice); scored as wrong both ways.	lower	orrery_triage_v1_labelled 230 arms	—	60
`output_words`	describes	How many words the output has: a description of the output, not a quality.	neutral: a description	smoke_v1 1 arm	—	1
`overlap_f1`	describes	Token-overlap F1 against the reference: the bundled adapter's placeholder, enough to run the loop, not to defend a decision.	higher	smoke_v1 1 arm	—	1
`pair_agreement`	describes	Pairwise mode: among the sampled pairs of an article's summaries that the experts rank apart, the share where the judge preferred the summary the experts prefer; a tie counts half and an unreadable answer 0. Chance is 0.5.	higher	summeval_100 3 arms	—	3
`parsed`	describes	Whether the reply could be read as an answer in the task's format: 1 or 0 per item, so its mean is the share readable (for the SummEval judges, the share of the four dimensions that came back readable). An unreadable reply scores 0 on the quality metrics, never skipped.	higher	ag_news_200 63 arms dbpedia_280 29 arms few_nerd_280 28 arms scifact_200 26 arms summeval_100 41 arms	—	157
`precision`	describes	Typed precision: the share of the predicted entities that match a gold one in text and type.	higher	few_nerd_280 28 arms	—	28
`rank_of_first`	describes	Rank of the first relevant document anywhere in the list, 0 when none was returned. Descriptive: 0 means never found, so it must not be averaged as a rank.	neutral: a description	scifact_200 26 arms	—	26
`reasoning_tokens`	describes	Tokens a reasoning model spent thinking before it answered, as the provider reported them: a description of the call, not a quality.	neutral: a description	ag_news_200 48 arms cnn_dailymail_200 121 arms dbpedia_280 24 arms few_nerd_280 24 arms scifact_200 19 arms summeval_100 30 arms	—	145
`recall`	describes	Typed recall: the share of the gold entities found with the right type.	higher	few_nerd_280 28 arms	—	28
`recall_10`	describes	Share of the query's relevant documents found in the first 10.	higher	scifact_200 26 arms	—	26
`recall_100`	describes	Share of the query's relevant documents found in the first 100.	higher	scifact_200 26 arms	—	26
`reference_words`	describes	How many words the gold summary has.	neutral: a description	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	—	42
`rho`	describes	Spearman's rho between the judge's scores and the experts' ratings of an article's 16 summaries, averaged over the four dimensions: kept for comparison with the literature.	higher	summeval_100 35 arms	—	35
`rho_coherence`	describes	Spearman's rho with the experts' coherence ratings across an article's 16 summaries (0 when the judge returned no coherence scores).	higher	summeval_100 35 arms	—	35
`rho_consistency`	describes	Spearman's rho with the experts' consistency ratings across an article's 16 summaries (0 when the judge returned no consistency scores).	higher	summeval_100 35 arms	—	35
`rho_fluency`	describes	Spearman's rho with the experts' fluency ratings across an article's 16 summaries (0 when the judge returned no fluency scores).	higher	summeval_100 35 arms	—	35
`rho_relevance`	describes	Spearman's rho with the experts' relevance ratings across an article's 16 summaries (0 when the judge returned no relevance scores).	higher	summeval_100 35 arms	—	35
`rouge1`	weighs	ROUGE-1 F1 against the reference, for comparison with published CNN/DailyMail results. Rises with length; read after coverage and concision.	higher	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	weighed by Short news summaries (CNN)	42
`rougeLsum`	describes	ROUGE-Lsum F1 against the reference (per-sentence longest common subsequence), the CNN/DailyMail literature's figure. Rises with length.	higher	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	—	42
`summary_words`	describes	How many whitespace-separated words the summary had.	neutral: a description	cnn_dailymail_20 2 arms cnn_dailymail_200 64 arms cnn_dailymail_dm200 5 arms	—	42
`table_miss`	describes	1 when the model asked for a probe the item's frozen probe table does not hold.	lower	orrery_triage_v1_labelled 336 arms	—	83
`tau`	decides	Kendall's tau-b between the judge's scores and the experts' ratings of an article's 16 summaries, averaged over the four dimensions.	higher	summeval_100 35 arms	A scorer that agrees with experts on summaries (SummEval)	35
`tau_coherence`	describes	Kendall's tau-b with the experts' coherence ratings across an article's 16 summaries (0 when the judge returned no coherence scores).	higher	summeval_100 35 arms	—	35
`tau_consistency`	describes	Kendall's tau-b with the experts' consistency ratings across an article's 16 summaries (0 when the judge returned no consistency scores).	higher	summeval_100 35 arms	—	35
`tau_fluency`	describes	Kendall's tau-b with the experts' fluency ratings across an article's 16 summaries (0 when the judge returned no fluency scores).	higher	summeval_100 35 arms	—	35
`tau_relevance`	describes	Kendall's tau-b with the experts' relevance ratings across an article's 16 summaries (0 when the judge returned no relevance scores).	higher	summeval_100 35 arms	—	35
`tie_rate`	describes	Share of the 120 summary pairs the judge scored equal, averaged over the dimensions it returned: how often it could not tell two summaries apart.	neutral: a description	summeval_100 35 arms	—	35
`tokens_in`	describes	Prompt tokens per item, as the provider counted them.	lower	ag_news_200 48 arms cnn_dailymail_20 1 arm cnn_dailymail_200 147 arms cnn_dailymail_dm200 4 arms dbpedia_280 24 arms few_nerd_280 24 arms orrery_triage_v1_labelled 336 arms scifact_200 19 arms summeval_100 30 arms	—	242
`tokens_out`	describes	Completion tokens per item, as the provider counted them.	lower	ag_news_200 48 arms cnn_dailymail_20 1 arm cnn_dailymail_200 147 arms cnn_dailymail_dm200 4 arms dbpedia_280 24 arms few_nerd_280 24 arms orrery_triage_v1_labelled 336 arms scifact_200 19 arms summeval_100 30 arms	—	242
`top1`	describes	Pick mode: how often the judge's pick of the best of an article's 16 summaries is a summary the experts rate best (highest mean over the four dimensions; ties among the experts form the best set). 1 for a correct single pick; a pick of several scores the share of them that are correct; an unreadable pick scores 0.	higher	summeval_100 3 arms	—	3
`top1_chance`	describes	Pick mode: what a random pick scores on the article, the share of its 16 summaries the experts rate best (\|best set\| / 16). A judge's top1 is read against its mean.	neutral: a description	summeval_100 3 arms	—	3
`tp`	describes	Typed true positives: predicted entities matched to a gold one (text and type).	neutral: a description	few_nerd_280 28 arms	—	28
`triage_cost`	decides	One decision's cost: 5 for a missed real problem, 1 for a false alarm, 0 when right; lower is better.	lower	orrery_triage_v1_labelled 317 arms	Triage decisions on Orrery's error signals	64
`triage_cost_p10`	describes	triage_cost with a missed real problem priced at 10 false alarms instead of 5.	lower	orrery_triage_v1_labelled 317 arms	—	64
`triage_cost_p3`	describes	triage_cost with a missed real problem priced at 3 false alarms instead of 5.	lower	orrery_triage_v1_labelled 317 arms	—	64
`truncated`	describes	1 when the provider cut the reply off at the token ceiling (finish_reason "length", and in most adapters a completion that reached 98% of max_tokens), else 0. An arm with truncated above 0 has scores that partly describe the ceiling, not the model.	neutral: a description	ag_news_200 48 arms cnn_dailymail_200 121 arms dbpedia_280 24 arms few_nerd_280 24 arms scifact_200 19 arms summeval_100 30 arms	—	145
`type_penalty`	describes	untyped_f1 minus f1: the credit lost only to mislabelling entities the arm did find. Large means a typing problem, not a detection problem.	neutral: a description	few_nerd_280 28 arms	—	28
`unknown_ids`	describes	Ids in the ranking that are not corpus documents: an LLM inventing documents rather than reordering them. They keep their slot.	neutral: a description	scifact_200 26 arms	—	26
`untyped_f1`	describes	F1 on the entity texts alone, types ignored: did the arm find the entity at all.	higher	few_nerd_280 28 arms	—	28
`untyped_fn`	describes	False negatives on texts alone, types ignored.	neutral: a description	few_nerd_280 28 arms	—	28
`untyped_fp`	describes	False positives on texts alone, types ignored.	neutral: a description	few_nerd_280 28 arms	—	28
`untyped_tp`	describes	True positives on texts alone, types ignored.	neutral: a description	few_nerd_280 28 arms	—	28