metric

`deferred`

better: lower
kind: quality: measured against a reference or by a judge
decides: no goal
weighed by: no goal
recorded on: orrery_triage_v1_labelled

01what it measures, and how

1 when the fleet deferred the decision twice for the same reason, so none was reached.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"deferred": {"summary": "…", "details": "…"}}.

02on orrery_triage_v1_labelled

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

arm	mean deferred	items	scored against
triage_baseline_pro_noreason_v1	0.000	65	gold/orrery_triage_v1_labelled gold
triage_deepseek_41flash_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_gpt_6_luna_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_hosted_gemma4_31b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_hosted_gpt_oss_120b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_hosted_qwen35_122b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_kimi_k31_pro_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gemma4_12b_noreason_v1	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gemma4_31b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gpt_oss_120b_v1	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gpt_oss_120b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gpt_oss_20b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_llama31_8b_v1	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_nemotron_35_bolt_e4b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_nemotron_35_lightning_30b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_nemotron_35_super_120b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_qwen35_122b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_qwen38_27b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_qwen3_30b_a3b_v1	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_gemma4_12b_noreason_v1	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_gemma4_31b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_gpt_oss_20b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_nemotron_35_bolt_e4b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_nemotron_35_lightning_30b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_qwen38_27b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_gemma4_31b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_gpt_oss_20b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_nemotron_35_bolt_e4b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_nemotron_35_lightning_30b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_qwen38_27b_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_mistral_large_4_v2	0.000	65	gold/orrery_triage_v1_labelled gold
triage_baseline_pro_noreason_v1 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_deepseek_41flash_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_gpt_6_luna_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_hosted_gemma4_31b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_hosted_gpt_oss_120b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_hosted_qwen35_122b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_kimi_k31_pro_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gemma4_12b_noreason_v1 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gemma4_31b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gpt_oss_120b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gpt_oss_20b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_llama31_8b_v1 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_nemotron_35_bolt_e4b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_nemotron_35_lightning_30b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_nemotron_35_super_120b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_qwen35_122b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_qwen38_27b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_qwen3_30b_a3b_v1 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_gemma4_12b_noreason_v1 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_gemma4_31b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_gpt_oss_20b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_nemotron_35_bolt_e4b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_nemotron_35_lightning_30b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_qwen38_27b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_gemma4_31b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_gpt_oss_20b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_nemotron_35_bolt_e4b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_nemotron_35_lightning_30b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_qwen38_27b_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_mistral_large_4_v2 · rescored 3d3a6afd	0.000	65	gold/orrery_triage_v1_labelled gold
triage_baseline_pro_noreason_v1 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_deepseek_41flash_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_gpt_6_luna_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_hosted_gemma4_31b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_hosted_gpt_oss_120b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_hosted_qwen35_122b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_kimi_k31_pro_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gemma4_12b_noreason_v1 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gemma4_31b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gpt_oss_120b_v1 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gpt_oss_120b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gpt_oss_20b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_llama31_8b_v1 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_nemotron_35_bolt_e4b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_nemotron_35_lightning_30b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_nemotron_35_super_120b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_qwen35_122b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_qwen38_27b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_qwen3_30b_a3b_v1 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_gemma4_12b_noreason_v1 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_gemma4_31b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_gpt_oss_20b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_nemotron_35_bolt_e4b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_nemotron_35_lightning_30b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_qwen38_27b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_gemma4_31b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_gpt_oss_20b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_nemotron_35_bolt_e4b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_nemotron_35_lightning_30b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_x3_qwen38_27b_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_mistral_large_4_v2 · rescored 5f2c057c	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gemma4_12b_noreason_v1 · rescored 77adc97b	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gpt_oss_120b_v1 · rescored 77adc97b	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_llama31_8b_v1 · rescored 77adc97b	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_qwen3_30b_a3b_v1 · rescored 77adc97b	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_gemma4_12b_noreason_v1 · rescored 77adc97b	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gemma4_12b_noreason_v1 · rescored cd35dc1f	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_gpt_oss_120b_v1 · rescored cd35dc1f	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_llama31_8b_v1 · rescored cd35dc1f	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_dgx_qwen3_30b_a3b_v1 · rescored cd35dc1f	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_gemma4_12b_noreason_v1 · rescored cd35dc1f	0.000	65	gold/orrery_triage_v1_labelled gold
triage_local_mac_ministral_3_14b_v1	0.032	31	gold/orrery_triage_v1_labelled gold
triage_local_mac_ministral_3_14b_v1 · rescored 3d3a6afd	0.032	31	gold/orrery_triage_v1_labelled gold
triage_local_mac_ministral_3_14b_v1 · rescored 5f2c057c	0.032	31	gold/orrery_triage_v1_labelled gold
triage_local_mac_ministral_3_14b_v1 · rescored 77adc97b	0.032	31	gold/orrery_triage_v1_labelled gold
triage_local_mac_ministral_3_14b_v1 · rescored cd35dc1f	0.032	31	gold/orrery_triage_v1_labelled gold