goal
Triage decisions on Orrery's error signals
Decide for each error signal whether to file an issue, dismiss it, escalate it to a person, or clean it up, the way the person who later closed the issue would have wanted.
01what it is for
- decision
- Which model the triage agent runs.
- audience
- the operator of the triage fleet
- decides
- triage_cost, lower is better
- min items
- 30
- weighs
- quality (triage_cost); cost per item; speed per item; false_alarm · no worse than the baseline (paired test)
- strategy
- frugal: contenders: anything not significantly different from the leader by the paired test; the cheaper wins
- tie-break
- cost, speed
- status
- active
02the answer so far
orrery_triage_v1_labelled · round r3 concluded 2026-11-01
Onboarding was the fix round 1 needed, not a winner. The 25 arms re-run with their model records' settings (reasoning off, or low where a model cannot stop; a reply cap; gpt-oss on Ollama read through reply_in_tool_call) all answered, none deferred. On the combined cost the paired test still finds no arm significantly different from the leader, the vLLM qwen3-30b-a3b on the DGX (0.400), except two significantly worse after Holm: llama4_scout and qwen3.5:122b on the DGX; 50 are not significantly different. The false-alarm limit (no worse than production, each arm on its own) rules out the arm that always escalates, Nemotron 3.5 Lightning v2 on all three machines, and llama4_scout. Production's model with reasoning off raises fewer false alarms than production (significant at 0.05) and misses more real problems (not significant); on the combined cost it is not significantly different from production (+0.37). Nothing is promoted: 65 decisions cannot tell this field apart (a power estimate put it at about 135-175 independent decisions). The next step is more labelled decisions, not more arms.
baseline: triage_baseline_pro_v1, promoted 2026-10-31 · scored against gold
why it was promoted
Nothing is decided at n = 65: on the combined cost the paired test finds no arm under the false-alarm ceiling significantly different from the leader, the arm that always dismisses among them; the ceiling (production's 4-pass mean, 21.25 of 31) was set after the results and sits within one or two false alarms of 18 rows; a no-decision scored as an escalation, a path production does not have. Production's own model is promoted as the baseline the next round measures against (owner, 2026-10-31). Next: a no-decision counts as wrong on both sides, the ceiling becomes no worse than this baseline by the paired test, and the gold grows to the 93 owner-labelled decisions before machines are booked.deploy sheet: what production must match
- measured by
triage_baseline_pro_v1_20261030T132352Zon orrery_triage_v1_labelled- against
- gold/orrery_triage_v1_labelled
- model sent
eval-fleet-triage-pro- model record
fleet-triage-pro- deployment
- hosted · homelab LiteLLM (the fleet's gateway) · eval-fleet-triage-pro
- server
- not recorded
- build
- none recorded (a hosted model has no digest)
- settings sent
model: eval-fleet-triage-proprovider: gateway- not sent
- bookkeeping:
usd_per_mtok_in, usd_per_mtok_out
the record setsreasoning_effort: none; this arm did not send it - declared
settings.prompt_sha256: 77f26d87ad9f4d117cd1cbdbd5be9bbb5fcd9666294055c37376fe6938bd1131deployment.fleet_code_sha256: 248e25030011aab82f5798eeb7c74fd03f2d7cfd04866b8b017d3740663727d9deployment.fleet_commit: f07efa9- reported per item
model: eval-fleet-triage-proprompt_sha: c2ece738- adapter
examples/triage-fleet/adapter.pyed10c946cea7- harness
- 0.5.0 44863aeef2ff
No capture here says what production ran (the adapter's production_ran).
Command: evh deploy-sheet --config triage_baseline_pro_v1 [--capture C]
03how we got here: rounds
Rounds are optional: each one sets out to learn something, changes a few arms, drops what lost and says why, and ends in a conclusion. Decisions are made with evh round-open, round-retire and round-close; evh rounds says the same as this page. Open a round for what changed, what was retired and why, and what to do next.
r3 closed
from r2the answer so far, above
Onboarding was the fix round 1 needed, not a winner. The 25 arms re-run with their model records' settings (reasoning off, or low where a model cannot stop; a reply cap; gpt-oss on Ollama read through reply_in_tool_call) all answered, none deferred. On the combined cost the paired test still finds no arm significantly different from the leader, the vLLM qwen3-30b-a3b on the DGX (0.400), except two significantly worse after Holm: llama4_scout and qwen3.5:122b on the DGX; 50 are not significantly different. The false-alarm limit (no worse than production, each arm on its own) rules out the arm that always escalates, Nemotron 3.5 Lightning v2 on all three machines, and llama4_scout. Production's model with reasoning off raises fewer false alarms than production (significant at 0.05) and misses more real problems (not significant); on the combined cost it is not significantly different from production (+0.37). Nothing is promoted: 65 decisions cannot tell this field apart (a power estimate put it at about 135-175 independent decisions). The next step is more labelled decisions, not more arms.Round 1's arms re-run as onboarding found they should be set up, and whether production's model does better with reasoning off
2 change(s) · 25 retired · 27 carried
from r2
the answer so far, above
Onboarding was the fix round 1 needed, not a winner. The 25 arms re-run with their model records' settings (reasoning off, or low where a model cannot stop; a reply cap; gpt-oss on Ollama read through reply_in_tool_call) all answered, none deferred. On the combined cost the paired test still finds no arm significantly different from the leader, the vLLM qwen3-30b-a3b on the DGX (0.400), except two significantly worse after Holm: llama4_scout and qwen3.5:122b on the DGX; 50 are not significantly different. The false-alarm limit (no worse than production, each arm on its own) rules out the arm that always escalates, Nemotron 3.5 Lightning v2 on all three machines, and llama4_scout. Production's model with reasoning off raises fewer false alarms than production (significant at 0.05) and misses more real problems (not significant); on the combined cost it is not significantly different from production (+0.37). Nothing is promoted: 65 decisions cannot tell this field apart (a power estimate put it at about 135-175 independent decisions). The next step is more labelled decisions, not more arms.Round 1's arms re-run as onboarding found they should be set up, and whether production's model does better with reasoning off
2 change(s) · 25 retired · 27 carried
what changed
- 25 arms re-run with their model records' settings (reasoning_effort none, or low where it cannot stop), each smoked first; a 1024-token reply cap
- a new candidate: production's model with reasoning off
arms
- retired 25: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings
triage_deepseek_41flash_v1, triage_gpt_6_luna_v1, triage_hosted_gemma4_31b_v1, triage_hosted_gpt_oss_120b_v1, triage_hosted_qwen35_122b_v1, triage_kimi_k31_pro_v1, triage_local_dgx_gemma4_31b_v1, triage_local_dgx_gpt_oss_120b_v1, triage_local_dgx_gpt_oss_20b_v1, triage_local_dgx_nemotron_35_bolt_e4b_v1, triage_local_dgx_nemotron_35_lightning_30b_v1, triage_local_dgx_nemotron_35_super_120b_v1, triage_local_dgx_qwen35_122b_v1, triage_local_dgx_qwen38_27b_v1, triage_local_mac_gemma4_31b_v1, triage_local_mac_gpt_oss_20b_v1, triage_local_mac_nemotron_35_bolt_e4b_v1, triage_local_mac_nemotron_35_lightning_30b_v1, triage_local_mac_qwen38_27b_v1, triage_local_x3_gemma4_31b_v1, triage_local_x3_gpt_oss_20b_v1, triage_local_x3_nemotron_35_bolt_e4b_v1, triage_local_x3_nemotron_35_lightning_30b_v1, triage_local_x3_qwen38_27b_v1, triage_mistral_large_4_v1 carried in unchanged 27
triage_baseline_pro_v1, triage_gemini_35_flash_lite_v1, triage_gemini_4_argon_v1, triage_gpt_61_sol_v1, triage_haiku_55_v1, triage_hosted_mistral_medium_35_v1, triage_hosted_qwen3_30b_a3b_v1, triage_local_dgx_gemma4_12b_noreason_v1, triage_local_dgx_llama31_8b_v1, triage_local_dgx_llama4_scout_v1, triage_local_dgx_ministral_3_14b_v1, triage_local_dgx_mistral_medium_35_128b_v1, triage_local_dgx_phi4_14b_v1, triage_local_dgx_qwen3_30b_a3b_v1, triage_local_dgx_vllm_qwen3_30b_a3b_nvfp4_v1, triage_local_mac_gemma4_12b_noreason_v1, triage_local_mac_llama31_8b_v1, triage_local_mac_ministral_3_14b_v1, triage_local_mac_phi4_14b_v1, triage_local_mac_qwen3_30b_a3b_v1, triage_local_x3_llama31_8b_v1, triage_local_x3_ministral_3_14b_v1, triage_local_x3_phi4_14b_v1, triage_local_x3_qwen3_30b_a3b_v1, triage_sonnet_55_v1, triage_stub_dismiss_v1, triage_stub_escalate_v1
scoring
- 2026-11-01: round 3's arms were scored by three versions of the triage adapter (reply_in_tool_call and the gateway's reasoning flag changed its calling code, not its scoring); one version for the round's evidence: score-preview moved no arm (56 at +0.0000) (
3d3a6afdb978, 62 run(s) re-scored into runs-rescored/3d3a6afdb978/runs, runs-rescored/3d3a6afdb978/runs-repeats) - 2026-11-02: Labels can accept more than one answer (owner, 2026-11-02): the gold's file-or-escalate is {"accept": ["escalate", "file"]}, and the adapter reads it with answers(); one adapter version and one gold for the round's evidence (
5f2c057cdaa5, 62 run(s) re-scored into runs-rescored/5f2c057cdaa5/runs, runs-rescored/5f2c057cdaa5/runs-repeats)
r2 closed
from r1The owner's rules of 2026-10-31, applied to round 1's runs at $0: the gpt-oss:20b rows fall from the top once a no-decision counts against them, the false-alarm gate rules out the arm that always escalates and llama4_scout on the DGX, and the paired test still finds none of the other 54 significantly different from the leader. Onboarding then showed 32 of round 1's arms ran at the server's default reasoning for models that reason by default: round 3 re-runs 25 of them with the onboarded settings.What the owner's rules of 2026-10-31 change in round 1's evidence, before any machine is booked
1 change(s) · 52 carried · judged by 77adc97b06a8
from r1
The owner's rules of 2026-10-31, applied to round 1's runs at $0: the gpt-oss:20b rows fall from the top once a no-decision counts against them, the false-alarm gate rules out the arm that always escalates and llama4_scout on the DGX, and the paired test still finds none of the other 54 significantly different from the leader. Onboarding then showed 32 of round 1's arms ran at the server's default reasoning for models that reason by default: round 3 re-runs 25 of them with the onboarded settings.
What the owner's rules of 2026-10-31 change in round 1's evidence, before any machine is booked
1 change(s) · 52 carried · judged by 77adc97b06a8
what changed
- scoring: a no-decision counts as a miss on a real problem and a false alarm on noise (production delivers nothing); the false-alarm ceiling is no worse than production, the baseline, by the paired test, each arm on its own
arms
carried in unchanged 52
triage_baseline_pro_v1, triage_deepseek_41flash_v1, triage_gemini_35_flash_lite_v1, triage_gemini_4_argon_v1, triage_gpt_61_sol_v1, triage_gpt_6_luna_v1, triage_haiku_55_v1, triage_hosted_gemma4_31b_v1, triage_hosted_gpt_oss_120b_v1, triage_hosted_mistral_medium_35_v1, triage_hosted_qwen35_122b_v1, triage_hosted_qwen3_30b_a3b_v1, triage_kimi_k31_pro_v1, triage_local_dgx_gemma4_12b_noreason_v1, triage_local_dgx_gemma4_31b_v1, triage_local_dgx_gpt_oss_120b_v1, triage_local_dgx_gpt_oss_20b_v1, triage_local_dgx_llama31_8b_v1, triage_local_dgx_llama4_scout_v1, triage_local_dgx_ministral_3_14b_v1, triage_local_dgx_mistral_medium_35_128b_v1, triage_local_dgx_nemotron_35_bolt_e4b_v1, triage_local_dgx_nemotron_35_lightning_30b_v1, triage_local_dgx_nemotron_35_super_120b_v1, triage_local_dgx_phi4_14b_v1, triage_local_dgx_qwen35_122b_v1, triage_local_dgx_qwen38_27b_v1, triage_local_dgx_qwen3_30b_a3b_v1, triage_local_dgx_vllm_qwen3_30b_a3b_nvfp4_v1, triage_local_mac_gemma4_12b_noreason_v1, triage_local_mac_gemma4_31b_v1, triage_local_mac_gpt_oss_20b_v1, triage_local_mac_llama31_8b_v1, triage_local_mac_ministral_3_14b_v1, triage_local_mac_nemotron_35_bolt_e4b_v1, triage_local_mac_nemotron_35_lightning_30b_v1, triage_local_mac_phi4_14b_v1, triage_local_mac_qwen38_27b_v1, triage_local_mac_qwen3_30b_a3b_v1, triage_local_x3_gemma4_31b_v1, triage_local_x3_gpt_oss_20b_v1, triage_local_x3_llama31_8b_v1, triage_local_x3_ministral_3_14b_v1, triage_local_x3_nemotron_35_bolt_e4b_v1, triage_local_x3_nemotron_35_lightning_30b_v1, triage_local_x3_phi4_14b_v1, triage_local_x3_qwen38_27b_v1, triage_local_x3_qwen3_30b_a3b_v1, triage_mistral_large_4_v1, triage_sonnet_55_v1, triage_stub_dismiss_v1, triage_stub_escalate_v1
scoring
- judged by
77adc97b06a8 - 2026-10-31: a no-decision counts as a miss on a real problem and a false alarm on noise (production delivers nothing); the false-alarm ceiling is no worse than production, the baseline, by the paired test, each arm on its own (
77adc97b06a8, 65 run(s) re-scored into runs-rescored/77adc97b06a8/runs, runs-rescored/77adc97b06a8/runs-repeats) - 2026-11-02: Labels can accept more than one answer (owner, 2026-11-02): the gold's file-or-escalate is {"accept": ["escalate", "file"]}, and the adapter reads it with answers(); one adapter version and one gold for the round's evidence (
5f2c057cdaa5, 65 run(s) re-scored into runs-rescored/5f2c057cdaa5/runs, runs-rescored/5f2c057cdaa5/runs-repeats)
r1 closedthe answer so far, above
Nothing is decided at n = 65: on the combined cost the paired test finds no arm under the false-alarm ceiling significantly different from the leader, the arm that always dismisses among them; the ceiling (production's 4-pass mean, 21.25 of 31) was set after the results and sits within one or two false alarms of 18 rows; a no-decision scored as an escalation, a path production does not have. Production's own model is promoted as the baseline the next round measures against (owner, 2026-10-31). Next: a no-decision counts as wrong on both sides, the ceiling becomes no worse than this baseline by the paired test, and the gold grows to the 93 owner-labelled decisions before machines are booked.Which model should the triage fleet run: every candidate, hosted and self-hosted on three machines, on the 65 decisions the owner labelled, against production's own model
52 carried · judged by cd35dc1ffccc
the answer so far, above
Nothing is decided at n = 65: on the combined cost the paired test finds no arm under the false-alarm ceiling significantly different from the leader, the arm that always dismisses among them; the ceiling (production's 4-pass mean, 21.25 of 31) was set after the results and sits within one or two false alarms of 18 rows; a no-decision scored as an escalation, a path production does not have. Production's own model is promoted as the baseline the next round measures against (owner, 2026-10-31). Next: a no-decision counts as wrong on both sides, the ceiling becomes no worse than this baseline by the paired test, and the gold grows to the 93 owner-labelled decisions before machines are booked.Which model should the triage fleet run: every candidate, hosted and self-hosted on three machines, on the 65 decisions the owner labelled, against production's own model
52 carried · judged by cd35dc1ffccc
arms
carried in unchanged 52
triage_baseline_pro_v1, triage_deepseek_41flash_v1, triage_gemini_35_flash_lite_v1, triage_gemini_4_argon_v1, triage_gpt_61_sol_v1, triage_gpt_6_luna_v1, triage_haiku_55_v1, triage_hosted_gemma4_31b_v1, triage_hosted_gpt_oss_120b_v1, triage_hosted_mistral_medium_35_v1, triage_hosted_qwen35_122b_v1, triage_hosted_qwen3_30b_a3b_v1, triage_kimi_k31_pro_v1, triage_local_dgx_gemma4_12b_noreason_v1, triage_local_dgx_gemma4_31b_v1, triage_local_dgx_gpt_oss_120b_v1, triage_local_dgx_gpt_oss_20b_v1, triage_local_dgx_llama31_8b_v1, triage_local_dgx_llama4_scout_v1, triage_local_dgx_ministral_3_14b_v1, triage_local_dgx_mistral_medium_35_128b_v1, triage_local_dgx_nemotron_35_bolt_e4b_v1, triage_local_dgx_nemotron_35_lightning_30b_v1, triage_local_dgx_nemotron_35_super_120b_v1, triage_local_dgx_phi4_14b_v1, triage_local_dgx_qwen35_122b_v1, triage_local_dgx_qwen38_27b_v1, triage_local_dgx_qwen3_30b_a3b_v1, triage_local_dgx_vllm_qwen3_30b_a3b_nvfp4_v1, triage_local_mac_gemma4_12b_noreason_v1, triage_local_mac_gemma4_31b_v1, triage_local_mac_gpt_oss_20b_v1, triage_local_mac_llama31_8b_v1, triage_local_mac_ministral_3_14b_v1, triage_local_mac_nemotron_35_bolt_e4b_v1, triage_local_mac_nemotron_35_lightning_30b_v1, triage_local_mac_phi4_14b_v1, triage_local_mac_qwen38_27b_v1, triage_local_mac_qwen3_30b_a3b_v1, triage_local_x3_gemma4_31b_v1, triage_local_x3_gpt_oss_20b_v1, triage_local_x3_llama31_8b_v1, triage_local_x3_ministral_3_14b_v1, triage_local_x3_nemotron_35_bolt_e4b_v1, triage_local_x3_nemotron_35_lightning_30b_v1, triage_local_x3_phi4_14b_v1, triage_local_x3_qwen38_27b_v1, triage_local_x3_qwen3_30b_a3b_v1, triage_mistral_large_4_v1, triage_sonnet_55_v1, triage_stub_dismiss_v1, triage_stub_escalate_v1
scoring
- judged by
cd35dc1ffccc - 2026-10-31: the adapter's score() changed seven times while the round ran, so its runs were re-scored onto one version, the combined cost (owner 2026-10-30); recorded by hand: the re-score was made before RFC-022, by evh rescore --out, and moved to its designated place (
cd35dc1ffccc, 65 run(s) re-scored into runs-rescored/cd35dc1ffccc/runs, runs-rescored/cd35dc1ffccc/runs-repeats) - 2026-11-02: Labels can accept more than one answer (owner, 2026-11-02): the gold's file-or-escalate is {"accept": ["escalate", "file"]}, and the adapter reads it with answers(); one adapter version and one gold for the round's evidence (
5f2c057cdaa5, 65 run(s) re-scored into runs-rescored/5f2c057cdaa5/runs, runs-rescored/5f2c057cdaa5/runs-repeats)
04orrery_triage_v1_labelled · validation
orrery_triage_v1_labelled: 65 items · validation
65 decisions the owner labelled, one per fingerprint, 2026-07-25 to 2026-10-29; mostly rebuilt evidence. 83 arm(s) are read from their re-score with the current scorer, not their first score (the measured run keeps its own page).
references: gold/orrery_triage_v1_labelled, silver/orrery_triage_v1_labelled_hand, silver/orrery_triage_v1_labelled_outcomes
Mean triage_cost over each arm's items, in groups by what each arm is scored against: numbers from different references, or from a judge, are different measurements and are never ranked together. A mean is not a verdict: the inspector tests each arm against its group's leader (Holm across them). Flagged arms are never the leader. filed: the arm's config names this goal (goal:); an arm without it is listed because it records the measure.
scored against gold/orrery_triage_v1_labelled gold
Test every arm in this group against its leader
No single arm recommended yet
- Every arm here is read through its re-score by scorer 5f2c057c (RFC-022).
- triage_stub_escalate_v1 is out: false_alarm worse than the baseline triage_baseline_pro_v1 (paired test, p 0.00145).
- triage_local_dgx_nemotron_35_lightning_30b_v2 is out: false_alarm worse than the baseline triage_baseline_pro_v1 (paired test, p 0.0209).
- triage_local_x3_nemotron_35_lightning_30b_v2 is out: false_alarm worse than the baseline triage_baseline_pro_v1 (paired test, p 0.0067).
- triage_local_mac_nemotron_35_lightning_30b_v2 is out: false_alarm worse than the baseline triage_baseline_pro_v1 (paired test, p 0.0035).
- triage_local_dgx_llama4_scout_v1 is out: false_alarm worse than the baseline triage_baseline_pro_v1 (paired test, p 0.0329).
- Strategy frugal. Quality leader: triage_local_dgx_vllm_qwen3_30b_a3b_nvfp4_v1 (triage_cost 0.4). Contenders beside it (not significantly different by the paired test): 51 arms
which
triage_local_dgx_llama31_8b_v1, triage_local_mac_llama31_8b_v1, triage_local_x3_llama31_8b_v1, triage_gemini_35_flash_lite_v1, triage_gpt_6_luna_v2, triage_local_x3_nemotron_35_bolt_e4b_v2, triage_gpt_61_sol_v1 · repeats ×3, triage_haiku_55_v1, triage_local_dgx_gemma4_12b_noreason_v1, triage_local_mac_gemma4_12b_noreason_v1, triage_local_mac_nemotron_35_bolt_e4b_v2, triage_local_x3_nemotron_35_bolt_e4b_v1 · repeats, triage_haiku_55_v1 · repeats ×3, triage_gpt_6_luna_v1 · repeats ×3, triage_hosted_gpt_oss_120b_v2, triage_gemini_4_argon_v1, triage_local_dgx_nemotron_35_bolt_e4b_v2, triage_gpt_61_sol_v1, triage_local_dgx_gpt_oss_120b_v2, triage_local_dgx_phi4_14b_v1, triage_local_dgx_nemotron_35_super_120b_v2, triage_hosted_qwen3_30b_a3b_v1, triage_local_mac_gpt_oss_20b_v2, triage_local_x3_gemma4_31b_v2, triage_sonnet_55_v1, triage_hosted_gemma4_31b_v2, triage_kimi_k31_pro_v2, triage_local_dgx_gemma4_31b_v2, triage_local_dgx_gpt_oss_20b_v2, triage_local_x3_qwen3_30b_a3b_v1, triage_local_mac_phi4_14b_v1, triage_baseline_pro_v1 · repeats ×3, triage_local_x3_phi4_14b_v1, triage_baseline_pro_v1, triage_local_mac_gemma4_31b_v2, triage_mistral_large_4_v2, triage_local_x3_gpt_oss_20b_v2, triage_local_x3_ministral_3_14b_v1, triage_local_dgx_ministral_3_14b_v1, triage_local_mac_qwen3_30b_a3b_v1, triage_stub_dismiss_v1, triage_hosted_mistral_medium_35_v1, triage_hosted_qwen35_122b_v2, triage_local_dgx_qwen3_30b_a3b_v1, triage_deepseek_41flash_v2, triage_local_mac_ministral_3_14b_v1, triage_local_dgx_mistral_medium_35_128b_v1, triage_local_dgx_qwen38_27b_v2, triage_local_mac_qwen38_27b_v2, triage_baseline_pro_noreason_v1, triage_local_x3_qwen38_27b_v2 - cost is not recorded for triage_stub_dismiss_v1: it cannot break the tie.
- Speed was not used to break the tie: the arms ran on different machines.
- Still undecided after the tie-break: all 52 contenders.
Strategy frugal (contenders: anything not significantly different from the leader by the paired test; the cheaper wins). Weighs quality (triage_cost); cost per item; speed per item; false_alarm · no worse than the baseline (paired test). Among the contenders on quality, the tie-break is cost, speed. The rule recommends; promoting is evh run-promote; the goal chooses its strategy (strategy:).
| arm | mean triage_cost | items | cost per item | speed per item | false_alarm no worse than the baseline (paired test) | flags | |
|---|---|---|---|---|---|---|---|
| triage_ | 0.400 | 65 | $0.000000 · 4,813 tok | p50 7,347 · p95 11,042 ms | 0.6774 | — | leader |
| triage_ | 0.415 | 65 | $0.000000 · 5,545 tok | p50 5,179 · p95 7,879 ms | 0.871 | — | test vs leader |
| triage_ | 0.415 | 65 | $0.000000 · 5,565 tok | p50 7,125 · p95 10,541 ms | 0.871 | — | test vs leader |
| triage_ | 0.415 | 65 | $0.000000 · 5,490 tok | p50 5,389 · p95 8,370 ms | 0.871 | — | test vs leader |
| triage_ | 0.477 | 65 | 1,100 tok | p50 0.085 · p95 0.13 ms | 1.00 | false_alarm worse than the baseline triage_baseline_pro_v1 (paired test, p 0.00145) | test vs leader |
| triage_ | 0.523 | 65 | $0.000000 · 6,295 tok | p50 14,013 · p95 22,893 ms | 0.6129 | — | test vs leader |
| triage_ | 0.523 | 65 | $0.000000 · 7,352 tok | p50 33,430 · p95 80,784 ms | 0.6129 | — | test vs leader |
| triage_ | 0.538 | 65 | $0.001368 · 3,125 tok | p50 3,353 · p95 5,695 ms | 0.6452 | — | test vs leader |
| triage_ | 0.554 | 65 | $0.000668 · 5,346 tok | p50 8,324 · p95 11,239 ms | 0.6774 | — | test vs leader |
| triage_ | 0.554 | 65 | $0.000000 · 5,306 tok | p50 5,534 · p95 7,717 ms | 0.5161 | — | test vs leader |
| triage_ | 0.585 | 65 | $0.010918 · 4,160 tok | p50 11,696 · p95 18,138 ms | 0.7419 | — | test vs leader |
| triage_ | 0.585 | 65 | $0.000715 · 4,764 tok | p50 11,518 · p95 21,428 ms | 0.7419 | — | test vs leader |
| triage_ | 0.585 | 65 | $0.000998 · 5,684 tok | p50 6,867 · p95 13,887 ms | 0.5806 | — | test vs leader |
| triage_ | 0.585 | 65 | $0.000000 · 3,389 tok | p50 5,603 · p95 8,411 ms | 0.5806 | — | test vs leader |
| triage_ | 0.585 | 65 | $0.000000 · 3,311 tok | p50 8,599 · p95 13,904 ms | 0.5806 | — | test vs leader |
| triage_ | 0.585 | 65 | $0.000000 · 5,696 tok | p50 5,737 · p95 7,706 ms | 0.5806 | — | test vs leader |
| triage_ | 0.600 | 65 | $0.000000 · 6,171 tok | p50 21,363 · p95 33,502 ms | 0.6129 | — | test vs leader |
| triage_ | 0.600 | 65 | $0.000000 · 6,171 tok | p50 20,275 · p95 32,697 ms | 0.6129 | — | test vs leader |
| triage_ | 0.610 | 65 | $0.001024 · 5,815 tok | p50 7,273 · p95 12,605 ms | 0.6344 | — | test vs leader |
| triage_ | 0.626 | 65 | $0.000684 · 4,621 tok | p50 10,962 · p95 18,893 ms | 0.6667 | — | test vs leader |
| triage_ | 0.631 | 65 | $0.000627 · 3,330 tok | p50 6,448 · p95 11,349 ms | 0.6774 | — | test vs leader |
| triage_ | 0.646 | 65 | $0.013617 · 4,751 tok | p50 23,508 · p95 61,827 ms | 0.7097 | — | test vs leader |
| triage_ | 0.646 | 65 | $0.019130 · 5,576 tok | p50 43,158 · p95 79,162 ms | 0.7097 | — | test vs leader |
| triage_ | 0.646 | 65 | $0.000000 · 5,506 tok | p50 3,850 · p95 5,195 ms | 0.7097 | — | test vs leader |
| triage_ | 0.646 | 65 | $0.000000 · 7,756 tok | p50 34,999 · p95 73,667 ms | 0.7097 | — | test vs leader |
| triage_ | 0.646 | 65 | $0.000000 · 7,484 tok | p50 55,843 · p95 129,188 ms | 0.7097 | — | test vs leader |
| triage_ | 0.662 | 65 | $0.010847 · 4,156 tok | p50 12,311 · p95 19,302 ms | 0.7419 | — | test vs leader |
| triage_ | 0.662 | 65 | $0.000000 · 3,135 tok | p50 7,719 · p95 10,098 ms | 0.5806 | — | test vs leader |
| triage_ | 0.662 | 65 | $0.000000 · 6,064 tok | p50 17,140 · p95 23,284 ms | 0.7419 | — | test vs leader |
| triage_ | 0.677 | 65 | $0.000849 · 3,854 tok | p50 16,361 · p95 31,807 ms | 0.6129 | — | test vs leader |
| triage_ | 0.677 | 65 | $0.000000 · 5,384 tok | p50 46,681 · p95 127,215 ms | 0.6129 | — | test vs leader |
| triage_ | 0.677 | 65 | $0.000000 · 6,286 tok | p50 6,762 · p95 8,238 ms | 0.9355 | false_alarm worse than the baseline triage_baseline_pro_v1 (paired test, p 0.0209) | test vs leader |
| triage_ | 0.677 | 65 | $0.000000 · 4,514 tok | p50 36,513 · p95 70,263 ms | 0.6129 | — | test vs leader |
| triage_ | 0.677 | 65 | $0.000000 · 5,246 tok | p50 15,795 · p95 21,883 ms | 0.7742 | — | test vs leader |
| triage_ | 0.677 | 65 | $0.000000 · 5,271 tok | p50 91,096 · p95 182,205 ms | 0.6129 | — | test vs leader |
| triage_ | 0.677 | 65 | $0.000000 · 6,288 tok | p50 7,933 · p95 9,459 ms | 0.9355 | false_alarm worse than the baseline triage_baseline_pro_v1 (paired test, p 0.0067) | test vs leader |
| triage_ | 0.692 | 65 | $0.000000 · 3,194 tok | p50 13,206 · p95 24,077 ms | 0.6452 | — | test vs leader |
| triage_ | 0.692 | 65 | $0.000000 · 6,265 tok | p50 11,237 · p95 14,132 ms | 0.9677 | false_alarm worse than the baseline triage_baseline_pro_v1 (paired test, p 0.0035) | test vs leader |
| triage_ | 0.692 | 65 | $0.000000 · 5,386 tok | p50 60,463 · p95 159,924 ms | 0.6452 | — | test vs leader |
| triage_ | 0.708 | 65 | $0.000700 · 6,111 tok | p50 16,982 · p95 35,517 ms | 0.5161 | — | test vs leader |
| triage_ | 0.708 | 65 | $0.000000 · 3,270 tok | p50 5,856 · p95 8,006 ms | 0.6774 | — | test vs leader |
| triage_ | 0.708 | 65 | $0.000000 · 4,219 tok | p50 14,550 · p95 21,031 ms | 0.5161 | — | test vs leader |
| triage_ | 0.708 | 65 | $0.022659 · 7,149 tok | p50 12,111 · p95 20,225 ms | 0.6774 | — | test vs leader |
| triage_ | 0.723 | 65 | $0.000903 · 4,276 tok | p50 9,818 · p95 19,215 ms | 0.5484 | — | test vs leader |
| triage_ | 0.723 | 65 | $0.000897 · 4,242 tok | p50 6,789 · p95 32,394 ms | 0.5484 | — | test vs leader |
| triage_ | 0.723 | 65 | $0.012041 · 4,605 tok | p50 22,561 · p95 34,524 ms | 0.7097 | — | test vs leader |
| triage_ | 0.723 | 65 | $0.000000 · 4,333 tok | p50 12,714 · p95 18,314 ms | 0.5484 | — | test vs leader |
| triage_ | 0.723 | 65 | $0.000000 · 2,977 tok | p50 4,298 · p95 6,526 ms | 0.871 | — | test vs leader |
| triage_ | 0.723 | 65 | $0.000000 · 6,231 tok | p50 6,349 · p95 10,328 ms | 0.3871 | — | test vs leader |
| triage_ | 0.754 | 65 | $0.000862 · 5,180 tok | p50 6,905 · p95 27,865 ms | 0.6129 | — | test vs leader |
| triage_ | 0.754 | 65 | $0.000000 · 6,321 tok | p50 21,874 · p95 33,411 ms | 0.6129 | — | test vs leader |
| triage_ | 0.754 | 65 | $0.000000 · 5,888 tok | p50 22,493 · p95 27,539 ms | 0.7742 | — | test vs leader |
| triage_ | 0.764 | 65 | $0.005104 · 4,916 tok | p50 21,498 · p95 55,678 ms | 0.6882 | — | test vs leader |
| triage_ | 0.769 | 65 | $0.000000 · 5,949 tok | p50 16,420 · p95 21,153 ms | 0.8065 | — | test vs leader |
| triage_ | 0.785 | 65 | $0.004992 · 4,790 tok | p50 19,965 · p95 47,741 ms | 0.6774 | — | test vs leader |
| triage_ | 0.800 | 65 | $0.000000 · 4,345 tok | p50 24,585 · p95 36,311 ms | 0.5484 | — | test vs leader |
| triage_ | 0.800 | 65 | $0.006867 · 6,704 tok | p50 15,459 · p95 30,084 ms | 0.7097 | — | test vs leader |
| triage_ | 0.800 | 65 | $0.006948 · 6,760 tok | p50 19,158 · p95 35,678 ms | 0.7097 | — | test vs leader |
| triage_ | 0.815 | 65 | $0.000000 · 3,225 tok | p50 6,092 · p95 8,461 ms | 0.7419 | — | test vs leader |
| triage_ | 0.815 | 65 | $0.000000 · 6,617 tok | p50 22,275 · p95 31,162 ms | 0.5806 | — | test vs leader |
| triage_ | 0.831 | 65 | $0.000000 · 6,489 tok | p50 19,704 · p95 29,658 ms | 0.4516 | — | test vs leader |
| triage_ | 0.831 | 65 | $0.000000 · 4,536 tok | p50 33,735 · p95 60,085 ms | 0.6129 | — | test vs leader |
| triage_ | 0.831 | 65 | $0.000000 · 6,279 tok | p50 8,898 · p95 13,952 ms | 0.4516 | — | test vs leader |
| triage_ | 0.862 | 65 | 1,100 tok | p50 0.092 · p95 0.17 ms | 0.6774 | — | test vs leader |
| triage_ | 0.877 | 65 | $0.011539 · 6,292 tok | p50 6,080 · p95 8,340 ms | 0.5484 | — | test vs leader |
| triage_ | 0.877 | 65 | $0.001654 · 4,247 tok | p50 6,774 · p95 9,273 ms | 0.5484 | — | test vs leader |
| triage_ | 0.892 | 65 | $0.000000 · 6,384 tok | p50 5,362 · p95 8,443 ms | 0.4194 | — | test vs leader |
| triage_ | 0.908 | 65 | $0.000355 · 4,756 tok | p50 2,897 · p95 6,148 ms | 0.6129 | — | test vs leader |
| triage_ | 0.908 | 65 | $0.000000 · 6,497 tok | p50 27,548 · p95 39,331 ms | 0.6129 | — | test vs leader |
| triage_ | 0.908 | 65 | $0.000000 · 4,388 tok | p50 43,283 · p95 76,811 ms | 0.6129 | — | test vs leader |
| triage_ | 1.000 | 65 | $0.000000 · 5,941 tok | p50 112,303 · p95 167,929 ms | 0.6452 | — | test vs leader |
| triage_ | 1.015 | 65 | $0.000000 · 3,538 tok | p50 12,488 · p95 21,540 ms | 0.6774 | — | test vs leader |
| triage_ | 1.046 | 65 | $0.000000 · 4,483 tok | p50 62,523 · p95 121,029 ms | 0.5806 | — | test vs leader |
| triage_ | 1.077 | 65 | $0.003350 · 5,474 tok | p50 15,600 · p95 33,509 ms | 0.6452 | — | test vs leader |
| triage_ | 1.092 | 65 | $0.000000 · 3,429 tok | p50 17,421 · p95 30,731 ms | 0.6774 | — | test vs leader |
| triage_ | 1.092 | 65 | $0.000000 · 3,619 tok | p50 16,207 · p95 29,798 ms | 0.6774 | — | test vs leader |
| triage_ | 1.138 | 65 | $0.000000 · 5,318 tok | p50 35,972 · p95 57,784 ms | 0.6129 | — | test vs leader |
| triage_ | 1.138 | 65 | $0.000000 · 4,585 tok | p50 15,771 · p95 24,424 ms | 0.6129 | — | test vs leader |
| triage_ | 1.138 | 65 | $0.000000 · 4,569 tok | p50 32,196 · p95 46,026 ms | 0.6129 | — | test vs leader |
| triage_ | 1.154 | 65 | $0.004862 · 5,590 tok | p50 7,710 · p95 14,703 ms | 0.4839 | — | test vs leader |
| triage_ | 1.262 | 65 | $0.000000 · 5,629 tok | p50 13,116 · p95 17,645 ms | 0.871 | false_alarm worse than the baseline triage_baseline_pro_v1 (paired test, p 0.0329) | test vs leader |
| triage_ | 1.292 | 65 | $0.000000 · 4,523 tok | p50 19,431 · p95 31,568 ms | 0.6129 | — | test vs leader |
| triage_ | 1.354 | 65 | $0.000000 · 4,180 tok | p50 15,011 · p95 19,803 ms | 0.5806 | — | test vs leader |