arms
Arms and configs
686 arms from 267 configs on 16 datasets. An arm is one config on one dataset scored one way: measured, rescored against another reference, judged, or run again (the how column says which; group by config to see them together). Pick arms on one dataset to open them in the inspector.
all time, 2026-09-25 to 2026-11-02
01every arm
group by
only
686 shown
| arm | how measured | dataset | model · provider | deciding measure | repeats | cost / item | latency | runs in period | spend in period | |
|---|---|---|---|---|---|---|---|---|---|---|
| ag_anthropic_l_n200_v1 | measured | ag_news_200 | eval-opus-5 litellm | correct 0.900 | 1 | $0.0009 | 1965 ms | 1 | — | |
| ag_anthropic_l_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.900 → 0.985 (+0.085) vs measured | ag_news_200 | eval-opus-5 litellm | correct 0.985 | 1 | $0.0009 | 1965 ms | 0 | — | |
| ag_anthropic_m_n200_v1 | measured | ag_news_200 | eval-claude-sonnet litellm | correct 0.910 | 1 | $0.0004 | 1916 ms | 1 | — | |
| ag_anthropic_m_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.910 → 0.965 (+0.055) vs measured | ag_news_200 | eval-claude-sonnet litellm | correct 0.965 | 1 | $0.0004 | 1916 ms | 0 | — | |
| ag_anthropic_s_n200_v1 | measured | ag_news_200 | eval-claude-haiku litellm | correct 0.840 | 1 | $0.0001 | 987 ms | 1 | — | |
| ag_anthropic_s_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.840 → 0.925 (+0.085) vs measured | ag_news_200 | eval-claude-haiku litellm | correct 0.925 | 1 | $0.0001 | 987 ms | 0 | — | |
| ag_bart_mnli_n200_v1 | measured | ag_news_200 | facebook/bart-large-mnli hf_zeroshot · local | correct 0.700 | 1 | — | 1387 ms | 1 | — | |
| ag_bart_mnli_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.700 → 0.755 (+0.055) vs measured | ag_news_200 | facebook/bart-large-mnli hf_zeroshot · local | correct 0.755 | 1 | — | 1387 ms | 0 | — | |
| ag_bert_base_fy_n200_v1 | measured | ag_news_200 | fabriceyhc/bert-base-uncased-ag_news hf_local · local | correct 0.950 | 1 | — | 90 ms | 1 | — | |
| ag_bert_base_fy_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.950 → 0.905 (-0.045) vs measured | ag_news_200 | fabriceyhc/bert-base-uncased-ag_news hf_local · local | correct 0.905 | 1 | — | 90 ms | 0 | — | |
| ag_bert_base_fy_n200_v1 · linux retired in r2 | run again (linux) 0.950 → 0.950 (+0.000) vs measured | ag_news_200 | fabriceyhc/bert-base-uncased-ag_news hf_local · local | correct 0.950 | 1 | — | 202 ms | 1 | — | |
| ag_bert_base_ta_n200_v1 leader | measured | ag_news_200 | textattack/bert-base-uncased-ag-news hf_local · local | correct 0.960 | 1 | — | 76 ms | 1 | — | |
| ag_bert_base_ta_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.960 → 0.905 (-0.055) vs measured | ag_news_200 | textattack/bert-base-uncased-ag-news hf_local · local | correct 0.905 | 1 | — | 76 ms | 0 | — | |
| ag_bert_base_ta_n200_v1 · linux retired in r2 | run again (linux) 0.960 → 0.960 (+0.000) vs measured | ag_news_200 | textattack/bert-base-uncased-ag-news hf_local · local | correct 0.960 | 1 | — | 57 ms | 1 | — | |
| ag_bert_mini_n200_v1 baseline | measured | ag_news_200 | mrm8488/bert-mini-finetuned-age_news-classification hf_local · local | correct 0.945 | 1 | — | 7 ms | 1 | — | |
| ag_bert_mini_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.945 → 0.905 (-0.040) vs measured | ag_news_200 | mrm8488/bert-mini-finetuned-age_news-classification hf_local · local | correct 0.905 | 1 | — | 7 ms | 0 | — | |
| ag_bert_mini_n200_v1 · linux retired in r2 | run again (linux) 0.945 → 0.945 (+0.000) vs measured | ag_news_200 | mrm8488/bert-mini-finetuned-age_news-classification hf_local · local | correct 0.945 | 1 | — | 21 ms | 1 | — | |
| ag_constant_n200_v1 | measured | ag_news_200 | — constant · local | correct 0.250 | 1 | — | 0 ms | 1 | — | |
| ag_constant_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.250 → 0.225 (-0.025) vs measured | ag_news_200 | — constant · local | correct 0.225 | 1 | — | 0 ms | 0 | — | |
| ag_deepseek_l_n200_v1 | measured | ag_news_200 | eval-deepseek-pro litellm | correct 0.870 | 1 | $0.0001 | 1323 ms | 1 | — | |
| ag_deepseek_l_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.870 → 0.955 (+0.085) vs measured | ag_news_200 | eval-deepseek-pro litellm | correct 0.955 | 1 | $0.0001 | 1323 ms | 0 | — | |
| ag_deepseek_m_n200_v1 | measured | ag_news_200 | eval-deepseek-41flash litellm | correct 0.845 | 1 | $0.0000 | 890 ms | 1 | — | |
| ag_deepseek_m_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.845 → 0.925 (+0.080) vs measured | ag_news_200 | eval-deepseek-41flash litellm | correct 0.925 | 1 | $0.0000 | 890 ms | 0 | — | |
| ag_deepseek_s_n200_v1 | measured | ag_news_200 | eval-deepseek-flash litellm | correct 0.840 | 1 | $0.0000 | 1602 ms | 1 | — | |
| ag_deepseek_s_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.840 → 0.925 (+0.085) vs measured | ag_news_200 | eval-deepseek-flash litellm | correct 0.925 | 1 | $0.0000 | 1602 ms | 0 | — | |
| ag_gemma_l_n200_v1 | measured | ag_news_200 | eval-gemma-31b litellm | correct 0.870 | 1 | $0.0000 | 911 ms | 1 | — | |
| ag_gemma_l_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.870 → 0.955 (+0.085) vs measured | ag_news_200 | eval-gemma-31b litellm | correct 0.955 | 1 | $0.0000 | 911 ms | 0 | — | |
| ag_gemma_m_n200_v1 | measured | ag_news_200 | eval-gemma-26b litellm | correct 0.855 | 1 | $0.0000 | 666 ms | 1 | — | |
| ag_gemma_m_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.855 → 0.940 (+0.085) vs measured | ag_news_200 | eval-gemma-26b litellm | correct 0.940 | 1 | $0.0000 | 666 ms | 0 | — | |
| ag_gemma_s_n200_v1 | measured | ag_news_200 | eval-gemma-3-27b litellm | correct 0.875 | 1 | $0.0000 | 664 ms | 1 | — | |
| ag_gemma_s_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.875 → 0.960 (+0.085) vs measured | ag_news_200 | eval-gemma-3-27b litellm | correct 0.960 | 1 | $0.0000 | 664 ms | 0 | — | |
| ag_glm_l_n200_v1 | measured | ag_news_200 | eval-glm-5 litellm | correct 0.860 | 1 | $0.0001 | 1871 ms | 1 | — | |
| ag_glm_l_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.860 → 0.945 (+0.085) vs measured | ag_news_200 | eval-glm-5 litellm | correct 0.945 | 1 | $0.0001 | 1871 ms | 0 | — | |
| ag_glm_m_n200_v1 | measured | ag_news_200 | eval-glm-46 litellm | correct 0.860 | 1 | $0.0001 | 2262 ms | 1 | — | |
| ag_glm_m_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.860 → 0.945 (+0.085) vs measured | ag_news_200 | eval-glm-46 litellm | correct 0.945 | 1 | $0.0001 | 2262 ms | 0 | — | |
| ag_glm_s_n200_v1 | measured | ag_news_200 | eval-glm-45-air litellm | correct 0.850 | 1 | $0.0000 | 1154 ms | 1 | — | |
| ag_glm_s_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.850 → 0.935 (+0.085) vs measured | ag_news_200 | eval-glm-45-air litellm | correct 0.935 | 1 | $0.0000 | 1154 ms | 0 | — | |
| ag_keyword_n200_v1 | measured | ag_news_200 | — keyword · local | correct 0.670 | 1 | — | 0 ms | 1 | — | |
| ag_keyword_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.670 → 0.670 (+0.000) vs measured | ag_news_200 | — keyword · local | correct 0.670 | 1 | — | 0 ms | 0 | — | |
| ag_llama_l_n200_v1 | measured | ag_news_200 | eval-llama-maverick litellm | correct 0.840 | 1 | $0.0000 | 804 ms | 1 | — | |
| ag_llama_l_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.840 → 0.925 (+0.085) vs measured | ag_news_200 | eval-llama-maverick litellm | correct 0.925 | 1 | $0.0000 | 804 ms | 0 | — | |
| ag_llama_m_n200_v1 | measured | ag_news_200 | eval-llama-70b litellm | correct 0.880 | 1 | $0.0000 | 1193 ms | 1 | — | |
| ag_llama_m_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.880 → 0.965 (+0.085) vs measured | ag_news_200 | eval-llama-70b litellm | correct 0.965 | 1 | $0.0000 | 1193 ms | 0 | — | |
| ag_llama_s_n200_v1 | measured | ag_news_200 | eval-llama-scout litellm | correct 0.835 | 1 | $0.0000 | 784 ms | 1 | — | |
| ag_llama_s_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.835 → 0.920 (+0.085) vs measured | ag_news_200 | eval-llama-scout litellm | correct 0.920 | 1 | $0.0000 | 784 ms | 0 | — | |
| ag_mistral_l_n200_v1 | measured | ag_news_200 | eval-mistral-large litellm | correct 0.875 | 1 | $0.0001 | 7662 ms | 1 | — | |
| ag_mistral_l_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.875 → 0.950 (+0.075) vs measured | ag_news_200 | eval-mistral-large litellm | correct 0.950 | 1 | $0.0001 | 7662 ms | 0 | — | |
| ag_mistral_m_n200_v1 | measured | ag_news_200 | eval-mistral-medium litellm | correct 0.860 | 1 | $0.0000 | 437 ms | 1 | — | |
| ag_mistral_m_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.860 → 0.945 (+0.085) vs measured | ag_news_200 | eval-mistral-medium litellm | correct 0.945 | 1 | $0.0000 | 437 ms | 0 | — | |
| ag_mistral_s_n200_v1 | measured | ag_news_200 | eval-mistral-small litellm | correct 0.860 | 1 | $0.0000 | 732 ms | 1 | — | |
| ag_mistral_s_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.860 → 0.930 (+0.070) vs measured | ag_news_200 | eval-mistral-small litellm | correct 0.930 | 1 | $0.0000 | 732 ms | 0 | — | |
| ag_openai_l_n200_v1 | measured | ag_news_200 | eval-gpt-6sol litellm | correct 0.890 | 1 | $0.0002 | 1207 ms | 1 | — | |
| ag_openai_l_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.890 → 0.975 (+0.085) vs measured | ag_news_200 | eval-gpt-6sol litellm | correct 0.975 | 1 | $0.0002 | 1207 ms | 0 | — | |
| ag_openai_m_n200_v1 | measured | ag_news_200 | eval-gpt-55 litellm | correct 0.895 | 1 | $0.0006 | 1105 ms | 1 | — | |
| ag_openai_m_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.895 → 0.980 (+0.085) vs measured | ag_news_200 | eval-gpt-55 litellm | correct 0.980 | 1 | $0.0006 | 1105 ms | 0 | — | |
| ag_openai_s_n200_v1 | measured | ag_news_200 | eval-gpt-mini litellm | correct 0.860 | 1 | $0.0001 | 859 ms | 1 | — | |
| ag_openai_s_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.860 → 0.945 (+0.085) vs measured | ag_news_200 | eval-gpt-mini litellm | correct 0.945 | 1 | $0.0001 | 859 ms | 0 | — | |
| ag_qwen_l_n200_v1 | measured | ag_news_200 | eval-qwen-3max litellm | correct 0.870 | 1 | $0.0001 | 2849 ms | 1 | — | |
| ag_qwen_l_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.870 → 0.955 (+0.085) vs measured | ag_news_200 | eval-qwen-3max litellm | correct 0.955 | 1 | $0.0001 | 2849 ms | 0 | — | |
| ag_qwen_m_n200_v1 | measured | ag_news_200 | eval-qwen-37plus litellm | correct 0.885 | 1 | $0.0000 | 911 ms | 1 | — | |
| ag_qwen_m_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.885 → 0.960 (+0.075) vs measured | ag_news_200 | eval-qwen-37plus litellm | correct 0.960 | 1 | $0.0000 | 911 ms | 0 | — | |
| ag_qwen_s_n200_v1 | measured | ag_news_200 | eval-qwen-flash litellm | correct 0.875 | 1 | $0.0000 | 1144 ms | 1 | — | |
| ag_qwen_s_n200_v1 · adjudicated | rescored against gold/ag_news_200_adj1 0.875 → 0.960 (+0.085) vs measured | ag_news_200 | eval-qwen-flash litellm | correct 0.960 | 1 | $0.0000 | 1144 ms | 0 | — | |
| cnn_bart_l_v1 | measured | cnn_dailymail_20 | facebook/bart-large-cnn hf_local · local | coverage 0.366 | 1 | — | 11861 ms | 1 | — | |
| cnn_lead3_v1 | measured | cnn_dailymail_20 | — lead_k · local | coverage 0.262 | 1 | — | 0 ms | 1 | — | |
| cnn_anthropic_l_n200_v1 | measured | cnn_dailymail_200 | eval-opus-5 litellm | coverage 0.315 | 1 | $0.0098 | 3402 ms | 1 | — | |
| cnn_anthropic_l_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-opus-5 litellm | judge_won 0.402 | 1 | $0.0098 | 3399 ms | 0 | — | |
| cnn_anthropic_l_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-opus-5 litellm | judge_score 4.976 | 1 | $0.0098 | 3402 ms | 0 | — | |
| cnn_anthropic_l_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-opus-5 litellm | judge_score 4.888 | 1 | $0.0098 | 3402 ms | 0 | — | |
| cnn_anthropic_l_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.315 → 1.000 (+0.685) vs measured | cnn_dailymail_200 | eval-opus-5 litellm | coverage 1.000 | 1 | $0.0098 | 3402 ms | 0 | — | |
| cnn_anthropic_m_n200_v1 | measured | cnn_dailymail_200 | eval-claude-sonnet litellm | coverage 0.324 | 1 | $0.0037 | 3034 ms | 1 | — | |
| cnn_anthropic_m_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-claude-sonnet litellm | judge_won 0.144 | 1 | $0.0037 | 3033 ms | 0 | — | |
| cnn_anthropic_m_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-claude-sonnet litellm | judge_score 4.963 | 1 | $0.0037 | 3032 ms | 0 | — | |
| cnn_anthropic_m_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-claude-sonnet litellm | judge_score 4.809 | 1 | $0.0037 | 3034 ms | 0 | — | |
| cnn_anthropic_m_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.324 → 0.491 (+0.168) vs measured | cnn_dailymail_200 | eval-claude-sonnet litellm | coverage 0.491 | 1 | $0.0037 | 3034 ms | 0 | — | |
| cnn_anthropic_s_n200_v1 | measured | cnn_dailymail_200 | eval-claude-haiku litellm | coverage 0.324 | 1 | $0.0013 | 2025 ms | 1 | — | |
| cnn_anthropic_s_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-claude-haiku litellm | judge_won 0.226 | 1 | $0.0013 | 2023 ms | 0 | — | |
| cnn_anthropic_s_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-claude-haiku litellm | judge_score 4.952 | 1 | $0.0013 | 2020 ms | 0 | — | |
| cnn_anthropic_s_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-claude-haiku litellm | judge_score 4.728 | 1 | $0.0013 | 2025 ms | 0 | — | |
| cnn_anthropic_s_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.324 → 0.469 (+0.145) vs measured | cnn_dailymail_200 | eval-claude-haiku litellm | coverage 0.469 | 1 | $0.0013 | 2025 ms | 0 | — | |
| cnn_bart_l_n200_v1 baselineleaderrecommended | measured | cnn_dailymail_200 | facebook/bart-large-cnn hf_local · local | coverage 0.346 | 1 | — | 11010 ms | 1 | — | |
| cnn_bart_l_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | facebook/bart-large-cnn hf_local · local | judge_won 0.000 | 1 | — | 11007 ms | 0 | — | |
| cnn_bart_l_n200_v1 · judged/judge-summ-pointwise-v1 | judged by judge-summ-pointwise-v1 | cnn_dailymail_200 | facebook/bart-large-cnn hf_local · local | judge_score_unvalidated 4.181 | 1 | — | 11010 ms | 0 | — | |
| cnn_bart_l_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | facebook/bart-large-cnn hf_local · local | judge_score 3.868 | 1 | — | 10996 ms | 0 | — | |
| cnn_bart_l_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | facebook/bart-large-cnn hf_local · local | judge_score 3.821 | 1 | — | 11010 ms | 0 | — | |
| cnn_bart_l_n200_v1 · linux retired in r2 | run again (linux) 0.346 → 0.346 (+0.000) vs measured | cnn_dailymail_200 | facebook/bart-large-cnn hf_local · local | coverage 0.346 | 1 | — | 6507 ms | 1 | — | |
| cnn_bart_l_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.346 → 0.317 (-0.029) vs measured | cnn_dailymail_200 | facebook/bart-large-cnn hf_local · local | coverage 0.317 | 1 | — | 11010 ms | 0 | — | |
| cnn_bart_l_xsum_n200_v1 | measured | cnn_dailymail_200 | facebook/bart-large-xsum hf_local · local | coverage 0.207 | 1 | — | 4621 ms | 1 | — | |
| cnn_bart_l_xsum_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | facebook/bart-large-xsum hf_local · local | judge_won 0.000 | 1 | — | 4609 ms | 0 | — | |
| cnn_bart_l_xsum_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | facebook/bart-large-xsum hf_local · local | judge_score 3.516 | 1 | — | 4605 ms | 0 | — | |
| cnn_bart_l_xsum_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | facebook/bart-large-xsum hf_local · local | judge_score 3.366 | 1 | — | 4621 ms | 0 | — | |
| cnn_bart_l_xsum_n200_v1 · linux retired in r2 | run again (linux) 0.207 → 0.207 (+0.000) vs measured | cnn_dailymail_200 | facebook/bart-large-xsum hf_local · local | coverage 0.207 | 1 | — | 4248 ms | 1 | — | |
| cnn_bart_l_xsum_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.207 → 0.137 (-0.070) vs measured | cnn_dailymail_200 | facebook/bart-large-xsum hf_local · local | coverage 0.137 | 1 | — | 4621 ms | 0 | — | |
| cnn_bart_m_n200_v1 | measured | cnn_dailymail_200 | sshleifer/distilbart-cnn-12-6 hf_local · local | coverage 0.337 | 1 | — | 5062 ms | 1 | — | |
| cnn_bart_m_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | sshleifer/distilbart-cnn-12-6 hf_local · local | judge_won 0.000 | 1 | — | 5048 ms | 0 | — | |
| cnn_bart_m_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | sshleifer/distilbart-cnn-12-6 hf_local · local | judge_score 3.564 | 1 | — | 5054 ms | 0 | — | |
| cnn_bart_m_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | sshleifer/distilbart-cnn-12-6 hf_local · local | judge_score 3.235 | 1 | — | 5062 ms | 0 | — | |
| cnn_bart_m_n200_v1 · linux retired in r2 | run again (linux) 0.337 → 0.337 (+0.000) vs measured | cnn_dailymail_200 | sshleifer/distilbart-cnn-12-6 hf_local · local | coverage 0.337 | 1 | — | 4375 ms | 1 | — | |
| cnn_bart_m_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.337 → 0.320 (-0.017) vs measured | cnn_dailymail_200 | sshleifer/distilbart-cnn-12-6 hf_local · local | coverage 0.320 | 1 | — | 5062 ms | 0 | — | |
| cnn_bart_s_n200_v1 | measured | cnn_dailymail_200 | sshleifer/distilbart-cnn-6-6 hf_local · local | coverage 0.328 | 1 | — | 4484 ms | 1 | — | |
| cnn_bart_s_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | sshleifer/distilbart-cnn-6-6 hf_local · local | judge_won 0.000 | 1 | — | 4485 ms | 0 | — | |
| cnn_bart_s_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | sshleifer/distilbart-cnn-6-6 hf_local · local | judge_score 3.185 | 1 | — | 4489 ms | 0 | — | |
| cnn_bart_s_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | sshleifer/distilbart-cnn-6-6 hf_local · local | judge_score 2.946 | 1 | — | 4484 ms | 0 | — | |
| cnn_bart_s_n200_v1 · linux retired in r2 | run again (linux) 0.328 → 0.328 (+0.000) vs measured | cnn_dailymail_200 | sshleifer/distilbart-cnn-6-6 hf_local · local | coverage 0.328 | 1 | — | 3420 ms | 1 | — | |
| cnn_bart_s_n200_v1 · repeats retired in r3 | run again (repeats) 0.328 → 0.328 (+0.000) vs measured | cnn_dailymail_200 | sshleifer/distilbart-cnn-6-6 hf_local · local | coverage 0.328 | 1 | — | 3766 ms | 1 | — | |
| cnn_bart_s_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.328 → 0.306 (-0.022) vs measured | cnn_dailymail_200 | sshleifer/distilbart-cnn-6-6 hf_local · local | coverage 0.306 | 1 | — | 4484 ms | 0 | — | |
| cnn_deepseek_l_n200_v1 | measured | cnn_dailymail_200 | eval-deepseek-pro litellm | coverage 0.332 | 1 | $0.0007 | 2665 ms | 1 | — | |
| cnn_deepseek_l_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-deepseek-pro litellm | judge_won 0.014 | 1 | $0.0007 | 2659 ms | 0 | — | |
| cnn_deepseek_l_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-deepseek-pro litellm | judge_score 4.877 | 1 | $0.0007 | 2666 ms | 0 | — | |
| cnn_deepseek_l_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-deepseek-pro litellm | judge_score 4.763 | 1 | $0.0007 | 2665 ms | 0 | — | |
| cnn_deepseek_l_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.332 → 0.399 (+0.066) vs measured | cnn_dailymail_200 | eval-deepseek-pro litellm | coverage 0.399 | 1 | $0.0007 | 2665 ms | 0 | — | |
| cnn_deepseek_m_n200_v1 | measured | cnn_dailymail_200 | eval-deepseek-41flash litellm | coverage 0.346 | 1 | $0.0002 | 1595 ms | 1 | — | |
| cnn_deepseek_m_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-deepseek-41flash litellm | judge_won 0.083 | 1 | $0.0002 | 1592 ms | 0 | — | |
| cnn_deepseek_m_n200_v1 · judged/judge-summ-pointwise-v1 | judged by judge-summ-pointwise-v1 | cnn_dailymail_200 | eval-deepseek-41flash litellm | judge_score_unvalidated 4.940 | 1 | $0.0002 | 1595 ms | 0 | — | |
| cnn_deepseek_m_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-deepseek-41flash litellm | judge_score 4.923 | 1 | $0.0002 | 1589 ms | 0 | — | |
| cnn_deepseek_m_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-deepseek-41flash litellm | judge_score 4.854 | 1 | $0.0002 | 1595 ms | 0 | — | |
| cnn_deepseek_m_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.346 → 0.471 (+0.125) vs measured | cnn_dailymail_200 | eval-deepseek-41flash litellm | coverage 0.471 | 1 | $0.0002 | 1595 ms | 0 | — | |
| cnn_deepseek_s_n200_v1 | measured | cnn_dailymail_200 | eval-deepseek-flash litellm | coverage 0.328 | 1 | $0.0001 | 2753 ms | 1 | — | |
| cnn_deepseek_s_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-deepseek-flash litellm | judge_won 0.004 | 1 | $0.0001 | 2714 ms | 0 | — | |
| cnn_deepseek_s_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-deepseek-flash litellm | judge_score 4.818 | 1 | $0.0001 | 2753 ms | 0 | — | |
| cnn_deepseek_s_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-deepseek-flash litellm | judge_score 4.644 | 1 | $0.0001 | 2753 ms | 0 | — | |
| cnn_deepseek_s_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.328 → 0.388 (+0.059) vs measured | cnn_dailymail_200 | eval-deepseek-flash litellm | coverage 0.388 | 1 | $0.0001 | 2753 ms | 0 | — | |
| cnn_gemma_l_n200_v1 | measured | cnn_dailymail_200 | eval-gemma-31b litellm | coverage 0.320 | 1 | $0.0001 | 2591 ms | 1 | — | |
| cnn_gemma_l_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-gemma-31b litellm | judge_won 0.001 | 1 | $0.0001 | 2600 ms | 0 | — | |
| cnn_gemma_l_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-gemma-31b litellm | judge_score 4.715 | 1 | $0.0001 | 2601 ms | 0 | — | |
| cnn_gemma_l_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-gemma-31b litellm | judge_score 4.603 | 1 | $0.0001 | 2591 ms | 0 | — | |
| cnn_gemma_l_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.320 → 0.342 (+0.021) vs measured | cnn_dailymail_200 | eval-gemma-31b litellm | coverage 0.342 | 1 | $0.0001 | 2591 ms | 0 | — | |
| cnn_gemma_m_n200_v1 | measured | cnn_dailymail_200 | eval-gemma-26b litellm | coverage 0.314 | 1 | $0.0001 | 1873 ms | 1 | — | |
| cnn_gemma_m_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-gemma-26b litellm | judge_won 0.005 | 1 | $0.0001 | 1871 ms | 0 | — | |
| cnn_gemma_m_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-gemma-26b litellm | judge_score 4.707 | 1 | $0.0001 | 1873 ms | 0 | — | |
| cnn_gemma_m_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-gemma-26b litellm | judge_score 4.585 | 1 | $0.0001 | 1873 ms | 0 | — | |
| cnn_gemma_m_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.314 → 0.342 (+0.029) vs measured | cnn_dailymail_200 | eval-gemma-26b litellm | coverage 0.342 | 1 | $0.0001 | 1873 ms | 0 | — | |
| cnn_gemma_s_n200_v1 | measured | cnn_dailymail_200 | eval-gemma-3-27b litellm | coverage 0.312 | 1 | $0.0001 | 6956 ms | 1 | — | |
| cnn_gemma_s_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-gemma-3-27b litellm | judge_won 0.002 | 1 | $0.0001 | 6932 ms | 0 | — | |
| cnn_gemma_s_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-gemma-3-27b litellm | judge_score 4.801 | 1 | $0.0001 | 6989 ms | 0 | — | |
| cnn_gemma_s_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-gemma-3-27b litellm | judge_score 4.429 | 1 | $0.0001 | 6956 ms | 0 | — | |
| cnn_gemma_s_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.312 → 0.350 (+0.039) vs measured | cnn_dailymail_200 | eval-gemma-3-27b litellm | coverage 0.350 | 1 | $0.0001 | 6956 ms | 0 | — | |
| cnn_glm_l_n200_v1 | measured | cnn_dailymail_200 | eval-glm-5 litellm | coverage 0.312 | 1 | $0.0008 | 5689 ms | 1 | — | |
| cnn_glm_l_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-glm-5 litellm | judge_won 0.006 | 1 | $0.0008 | 5718 ms | 0 | — | |
| cnn_glm_l_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-glm-5 litellm | judge_score 4.607 | 1 | $0.0008 | 5739 ms | 0 | — | |
| cnn_glm_l_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-glm-5 litellm | judge_score 4.384 | 1 | $0.0008 | 5689 ms | 0 | — | |
| cnn_glm_l_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.312 → 0.375 (+0.063) vs measured | cnn_dailymail_200 | eval-glm-5 litellm | coverage 0.375 | 1 | $0.0008 | 5689 ms | 0 | — | |
| cnn_glm_m_n200_v1 | measured | cnn_dailymail_200 | eval-glm-46 litellm | coverage 0.332 | 1 | $0.0005 | 3230 ms | 1 | — | |
| cnn_glm_m_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-glm-46 litellm | judge_won 0.005 | 1 | $0.0005 | 3243 ms | 0 | — | |
| cnn_glm_m_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-glm-46 litellm | judge_score 4.861 | 1 | $0.0005 | 3234 ms | 0 | — | |
| cnn_glm_m_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-glm-46 litellm | judge_score 4.718 | 1 | $0.0005 | 3230 ms | 0 | — | |
| cnn_glm_m_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.332 → 0.389 (+0.057) vs measured | cnn_dailymail_200 | eval-glm-46 litellm | coverage 0.389 | 1 | $0.0005 | 3230 ms | 0 | — | |
| cnn_glm_s_n200_v1 | measured | cnn_dailymail_200 | eval-glm-45-air litellm | coverage 0.314 | 1 | $0.0002 | 1610 ms | 1 | — | |
| cnn_glm_s_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-glm-45-air litellm | judge_won 0.001 | 1 | $0.0002 | 1612 ms | 0 | — | |
| cnn_glm_s_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-glm-45-air litellm | judge_score 4.530 | 1 | $0.0002 | 1615 ms | 0 | — | |
| cnn_glm_s_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-glm-45-air litellm | judge_score 4.420 | 1 | $0.0002 | 1610 ms | 0 | — | |
| cnn_glm_s_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.314 → 0.303 (-0.011) vs measured | cnn_dailymail_200 | eval-glm-45-air litellm | coverage 0.303 | 1 | $0.0002 | 1610 ms | 0 | — | |
| cnn_lead3_n200_v1 | measured | cnn_dailymail_200 | — lead_k · local | coverage 0.272 | 1 | — | 0 ms | 1 | — | |
| cnn_lead3_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | — lead_k · local | judge_won 0.000 | 1 | — | 0 ms | 0 | — | |
| cnn_lead3_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | — lead_k · local | judge_score 3.421 | 1 | — | 0 ms | 0 | — | |
| cnn_lead3_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | — lead_k · local | judge_score 3.339 | 1 | — | 0 ms | 0 | — | |
| cnn_lead3_n200_v1 · linux retired in r2 | run again (linux) 0.272 → 0.272 (+0.000) vs measured | cnn_dailymail_200 | — lead_k · local | coverage 0.272 | 1 | — | 0 ms | 1 | — | |
| cnn_lead3_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.272 → 0.300 (+0.029) vs measured | cnn_dailymail_200 | — lead_k · local | coverage 0.300 | 1 | — | 0 ms | 0 | — | |
| cnn_llama_l_n200_v1 | measured | cnn_dailymail_200 | eval-llama-maverick litellm | coverage 0.339 | 1 | $0.0002 | 3473 ms | 1 | — | |
| cnn_llama_l_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-llama-maverick litellm | judge_won 0.003 | 1 | $0.0002 | 3474 ms | 0 | — | |
| cnn_llama_l_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-llama-maverick litellm | judge_score 4.700 | 1 | $0.0002 | 3495 ms | 0 | — | |
| cnn_llama_l_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-llama-maverick litellm | judge_score 4.529 | 1 | $0.0002 | 3473 ms | 0 | — | |
| cnn_llama_l_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.339 → 0.370 (+0.031) vs measured | cnn_dailymail_200 | eval-llama-maverick litellm | coverage 0.370 | 1 | $0.0002 | 3473 ms | 0 | — | |
| cnn_llama_m_n200_v1 | measured | cnn_dailymail_200 | eval-llama-70b litellm | coverage 0.338 | 1 | $0.0002 | 4045 ms | 1 | — | |
| cnn_llama_m_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-llama-70b litellm | judge_won 0.001 | 1 | $0.0002 | 4066 ms | 0 | — | |
| cnn_llama_m_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-llama-70b litellm | judge_score 4.674 | 1 | $0.0002 | 4072 ms | 0 | — | |
| cnn_llama_m_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-llama-70b litellm | judge_score 4.488 | 1 | $0.0002 | 4045 ms | 0 | — | |
| cnn_llama_m_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.338 → 0.358 (+0.019) vs measured | cnn_dailymail_200 | eval-llama-70b litellm | coverage 0.358 | 1 | $0.0002 | 4045 ms | 0 | — | |
| cnn_llama_s_n200_v1 | measured | cnn_dailymail_200 | eval-llama-scout litellm | coverage 0.333 | 1 | $0.0001 | 2388 ms | 1 | — | |
| cnn_llama_s_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-llama-scout litellm | judge_won 0.001 | 1 | $0.0001 | 2370 ms | 0 | — | |
| cnn_llama_s_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-llama-scout litellm | judge_score 4.704 | 1 | $0.0001 | 2392 ms | 0 | — | |
| cnn_llama_s_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-llama-scout litellm | judge_score 4.469 | 1 | $0.0001 | 2388 ms | 0 | — | |
| cnn_llama_s_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.333 → 0.388 (+0.055) vs measured | cnn_dailymail_200 | eval-llama-scout litellm | coverage 0.388 | 1 | $0.0001 | 2388 ms | 0 | — | |
| cnn_mistral_l_n200_v1 | measured | cnn_dailymail_200 | eval-mistral-large litellm | coverage 0.308 | 1 | $0.0005 | 11045 ms | 1 | — | |
| cnn_mistral_l_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-mistral-large litellm | judge_won 0.003 | 1 | $0.0005 | 10529 ms | 0 | — | |
| cnn_mistral_l_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-mistral-large litellm | judge_score 4.698 | 1 | $0.0005 | 11180 ms | 0 | — | |
| cnn_mistral_l_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-mistral-large litellm | judge_score 4.037 | 1 | $0.0005 | 11045 ms | 0 | — | |
| cnn_mistral_l_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.308 → 0.337 (+0.029) vs measured | cnn_dailymail_200 | eval-mistral-large litellm | coverage 0.337 | 1 | $0.0005 | 11045 ms | 0 | — | |
| cnn_mistral_m_n200_v1 | measured | cnn_dailymail_200 | eval-mistral-medium litellm | coverage 0.304 | 1 | $0.0004 | 2654 ms | 1 | — | |
| cnn_mistral_m_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-mistral-medium litellm | judge_won 0.000 | 1 | $0.0004 | 2648 ms | 0 | — | |
| cnn_mistral_m_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-mistral-medium litellm | judge_score 4.708 | 1 | $0.0004 | 2659 ms | 0 | — | |
| cnn_mistral_m_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-mistral-medium litellm | judge_score 4.309 | 1 | $0.0004 | 2654 ms | 0 | — | |
| cnn_mistral_m_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.304 → 0.320 (+0.016) vs measured | cnn_dailymail_200 | eval-mistral-medium litellm | coverage 0.320 | 1 | $0.0004 | 2654 ms | 0 | — | |
| cnn_mistral_s_n200_v1 | measured | cnn_dailymail_200 | eval-mistral-small litellm | coverage 0.326 | 1 | $0.0001 | 2820 ms | 1 | — | |
| cnn_mistral_s_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-mistral-small litellm | judge_won 0.004 | 1 | $0.0001 | 2810 ms | 0 | — | |
| cnn_mistral_s_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-mistral-small litellm | judge_score 4.798 | 1 | $0.0001 | 2826 ms | 0 | — | |
| cnn_mistral_s_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-mistral-small litellm | judge_score 4.408 | 1 | $0.0001 | 2820 ms | 0 | — | |
| cnn_mistral_s_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.326 → 0.382 (+0.057) vs measured | cnn_dailymail_200 | eval-mistral-small litellm | coverage 0.382 | 1 | $0.0001 | 2820 ms | 0 | — | |
| cnn_openai_l_n200_v1 | measured | cnn_dailymail_200 | eval-gpt-6sol litellm | coverage 0.327 | 1 | $0.0022 | 1775 ms | 1 | — | |
| cnn_openai_l_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-gpt-6sol litellm | judge_won 0.004 | 1 | $0.0022 | 1775 ms | 0 | — | |
| cnn_openai_l_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-gpt-6sol litellm | judge_score 4.773 | 1 | $0.0022 | 1769 ms | 0 | — | |
| cnn_openai_l_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-gpt-6sol litellm | judge_score 4.847 | 1 | $0.0022 | 1775 ms | 0 | — | |
| cnn_openai_l_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.327 → 0.371 (+0.044) vs measured | cnn_dailymail_200 | eval-gpt-6sol litellm | coverage 0.371 | 1 | $0.0022 | 1775 ms | 0 | — | |
| cnn_openai_m_n200_v1 | measured | cnn_dailymail_200 | eval-gpt-55 litellm | coverage 0.327 | 1 | $0.0054 | 1745 ms | 1 | — | |
| cnn_openai_m_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-gpt-55 litellm | judge_won 0.004 | 1 | $0.0053 | 1746 ms | 0 | — | |
| cnn_openai_m_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-gpt-55 litellm | judge_score 4.795 | 1 | $0.0053 | 1746 ms | 0 | — | |
| cnn_openai_m_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-gpt-55 litellm | judge_score 4.811 | 1 | $0.0054 | 1745 ms | 0 | — | |
| cnn_openai_m_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.327 → 0.359 (+0.033) vs measured | cnn_dailymail_200 | eval-gpt-55 litellm | coverage 0.359 | 1 | $0.0054 | 1745 ms | 0 | — | |
| cnn_openai_s_n200_v1 | measured | cnn_dailymail_200 | eval-gpt-mini litellm | coverage 0.317 | 1 | $0.0008 | 1277 ms | 1 | — | |
| cnn_openai_s_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-gpt-mini litellm | judge_won 0.008 | 1 | $0.0008 | 1277 ms | 0 | — | |
| cnn_openai_s_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-gpt-mini litellm | judge_score 4.825 | 1 | $0.0008 | 1277 ms | 0 | — | |
| cnn_openai_s_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-gpt-mini litellm | judge_score 4.747 | 1 | $0.0008 | 1277 ms | 0 | — | |
| cnn_openai_s_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.317 → 0.384 (+0.067) vs measured | cnn_dailymail_200 | eval-gpt-mini litellm | coverage 0.384 | 1 | $0.0008 | 1277 ms | 0 | — | |
| cnn_qwen_l_n200_v1 | measured | cnn_dailymail_200 | eval-qwen-3max litellm | coverage 0.311 | 1 | $0.0008 | 2945 ms | 1 | — | |
| cnn_qwen_l_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-qwen-3max litellm | judge_won 0.059 | 1 | $0.0008 | 2943 ms | 0 | — | |
| cnn_qwen_l_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-qwen-3max litellm | judge_score 4.812 | 1 | $0.0008 | 2946 ms | 0 | — | |
| cnn_qwen_l_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-qwen-3max litellm | judge_score 4.573 | 1 | $0.0008 | 2945 ms | 0 | — | |
| cnn_qwen_l_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.311 → 0.398 (+0.087) vs measured | cnn_dailymail_200 | eval-qwen-3max litellm | coverage 0.398 | 1 | $0.0008 | 2945 ms | 0 | — | |
| cnn_qwen_m_n200_v1 | measured | cnn_dailymail_200 | eval-qwen-37plus litellm | coverage 0.303 | 1 | $0.0003 | 1534 ms | 1 | — | |
| cnn_qwen_m_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-qwen-37plus litellm | judge_won 0.020 | 1 | $0.0003 | 1533 ms | 0 | — | |
| cnn_qwen_m_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-qwen-37plus litellm | judge_score 4.876 | 1 | $0.0003 | 1532 ms | 0 | — | |
| cnn_qwen_m_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-qwen-37plus litellm | judge_score 4.600 | 1 | $0.0003 | 1534 ms | 0 | — | |
| cnn_qwen_m_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.303 → 0.381 (+0.078) vs measured | cnn_dailymail_200 | eval-qwen-37plus litellm | coverage 0.381 | 1 | $0.0003 | 1534 ms | 0 | — | |
| cnn_qwen_s_n200_v1 | measured | cnn_dailymail_200 | eval-qwen-flash litellm | coverage 0.307 | 1 | $0.0001 | 25038 ms | 1 | — | |
| cnn_qwen_s_n200_v1 · judged/judge-summ-pick-v1 | judged by judge-summ-pick-v1 | cnn_dailymail_200 | eval-qwen-flash litellm | judge_won 0.001 | 1 | $0.0001 | 25218 ms | 0 | — | |
| cnn_qwen_s_n200_v1 · judged/judge-summ-v1 | judged by judge-summ-v1 | cnn_dailymail_200 | eval-qwen-flash litellm | judge_score 4.609 | 1 | $0.0001 | 24872 ms | 0 | — | |
| cnn_qwen_s_n200_v1 · judged/judge-summ-v2 | judged by judge-summ-v2 | cnn_dailymail_200 | eval-qwen-flash litellm | judge_score 4.461 | 1 | $0.0001 | 25038 ms | 0 | — | |
| cnn_qwen_s_n200_v1 · silver | rescored against silver/cnn_dailymail_200 0.307 → 0.334 (+0.027) vs measured | cnn_dailymail_200 | eval-qwen-flash litellm | coverage 0.334 | 1 | $0.0001 | 25038 ms | 0 | — | |
| cnn_bart_l_dm200_v1 | measured | cnn_dailymail_dm200 | facebook/bart-large-cnn hf_local · local | coverage 0.443 | 1 | — | 11260 ms | 1 | — | |
| cnn_bart_l_xsum_dm200_v1 | measured | cnn_dailymail_dm200 | facebook/bart-large-xsum hf_local · local | coverage 0.152 | 1 | — | 5362 ms | 1 | — | |
| cnn_bart_m_dm200_v1 | measured | cnn_dailymail_dm200 | sshleifer/distilbart-cnn-12-6 hf_local · local | coverage 0.443 | 1 | — | 7292 ms | 1 | — | |
| cnn_bart_s_dm200_v1 | measured | cnn_dailymail_dm200 | sshleifer/distilbart-cnn-6-6 hf_local · local | coverage 0.405 | 1 | — | 4302 ms | 1 | — | |
| cnn_lead3_dm200_v1 | measured | cnn_dailymail_dm200 | — lead_k · local | coverage 0.370 | 1 | — | 0 ms | 1 | — | |
| db_anthropic_l_n200_v1 | measured | dbpedia_280 | eval-opus-5 litellm | correct 0.989 | 1 | $0.0014 | 3101 ms | 1 | — | |
| db_anthropic_m_n200_v1 | measured | dbpedia_280 | eval-claude-sonnet litellm | correct 0.982 | 1 | $0.0005 | 2258 ms | 1 | — | |
| db_anthropic_s_n200_v1 | measured | dbpedia_280 | eval-claude-haiku litellm | correct 0.979 | 1 | $0.0002 | 1083 ms | 1 | — | |
| db_bart_mnli_n200_v1 | measured | dbpedia_280 | facebook/bart-large-mnli hf_zeroshot · local | correct 0.629 | 1 | — | 5718 ms | 1 | — | |
| db_bert_base_fy_n200_v1 baselinerecommended | measured | dbpedia_280 | fabriceyhc/bert-base-uncased-dbpedia_14 hf_local · local | correct 0.986 | 1 | — | 87 ms | 1 | — | |
| db_bert_base_fy_n200_v1 · linux retired in r2 | run again (linux) 0.986 → 0.986 (+0.000) vs measured | dbpedia_280 | fabriceyhc/bert-base-uncased-dbpedia_14 hf_local · local | correct 0.986 | 1 | — | 166 ms | 1 | — | |
| db_constant_n200_v1 | measured | dbpedia_280 | — constant · local | correct 0.071 | 1 | — | 0 ms | 1 | — | |
| db_deepseek_l_n200_v1 | measured | dbpedia_280 | eval-deepseek-pro litellm | correct 0.986 | 1 | $0.0001 | 1782 ms | 1 | — | |
| db_deepseek_m_n200_v1 | measured | dbpedia_280 | eval-deepseek-41flash litellm | correct 0.975 | 1 | $0.0000 | 975 ms | 1 | — | |
| db_deepseek_s_n200_v1 | measured | dbpedia_280 | eval-deepseek-flash litellm | correct 0.943 | 1 | $0.0000 | 1548 ms | 1 | — | |
| db_gemma_l_n200_v1 | measured | dbpedia_280 | eval-gemma-31b litellm | correct 0.975 | 1 | $0.0000 | 1145 ms | 1 | — | |
| db_gemma_m_n200_v1 | measured | dbpedia_280 | eval-gemma-26b litellm | correct 0.986 | 1 | $0.0000 | 856 ms | 1 | — | |
| db_gemma_s_n200_v1 | measured | dbpedia_280 | eval-gemma-3-27b litellm | correct 0.975 | 1 | $0.0000 | 647 ms | 1 | — | |
| db_glm_l_n200_v1 | measured | dbpedia_280 | eval-glm-5 litellm | correct 0.968 | 1 | $0.0001 | 1978 ms | 1 | — | |
| db_glm_m_n200_v1 | measured | dbpedia_280 | eval-glm-46 litellm | correct 0.986 | 1 | $0.0001 | 1930 ms | 1 | — | |
| db_glm_s_n200_v1 | measured | dbpedia_280 | eval-glm-45-air litellm | correct 0.939 | 1 | $0.0000 | 1183 ms | 1 | — | |
| db_keyword_n200_v1 | measured | dbpedia_280 | — keyword · local | correct 0.707 | 1 | — | 0 ms | 1 | — | |
| db_llama_l_n200_v1 | measured | dbpedia_280 | eval-llama-maverick litellm | correct 0.975 | 1 | $0.0000 | 660 ms | 1 | — | |
| db_llama_m_n200_v1 | measured | dbpedia_280 | eval-llama-70b litellm | correct 0.979 | 1 | $0.0000 | 1927 ms | 1 | — | |
| db_llama_s_n200_v1 | measured | dbpedia_280 | eval-llama-scout litellm | correct 0.957 | 1 | $0.0000 | 1374 ms | 1 | — | |
| db_mistral_l_n200_v1 | measured | dbpedia_280 | eval-mistral-large litellm | correct 0.964 | 1 | $0.0001 | 10565 ms | 1 | — | |
| db_mistral_m_n200_v1 | measured | dbpedia_280 | eval-mistral-medium litellm | correct 0.957 | 1 | $0.0001 | 439 ms | 1 | — | |
| db_mistral_s_n200_v1 | measured | dbpedia_280 | eval-mistral-small litellm | correct 0.957 | 1 | $0.0000 | 943 ms | 1 | — | |
| db_openai_l_n200_v1 | measured | dbpedia_280 | eval-gpt-6sol litellm | correct 0.975 | 1 | $0.0003 | 1266 ms | 1 | — | |
| db_openai_m_n200_v1 | measured | dbpedia_280 | eval-gpt-55 litellm | correct 0.982 | 1 | $0.0009 | 1178 ms | 1 | — | |
| db_openai_s_n200_v1 | measured | dbpedia_280 | eval-gpt-mini litellm | correct 0.975 | 1 | $0.0001 | 910 ms | 1 | — | |
| db_qwen_l_n200_v1 | measured | dbpedia_280 | eval-qwen-3max litellm | correct 0.968 | 1 | $0.0001 | 1660 ms | 1 | — | |
| db_qwen_m_n200_v1 leader | measured | dbpedia_280 | eval-qwen-37plus litellm | correct 0.993 | 1 | $0.0001 | 853 ms | 1 | — | |
| db_qwen_s_n200_v1 | measured | dbpedia_280 | eval-qwen-flash litellm | correct 0.986 | 1 | $0.0000 | 12349 ms | 1 | — | |
| fn_anthropic_l_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-opus-5 litellm | f1 0.680 | 1 | $0.0025 | 2759 ms | 1 | — | |
| fn_anthropic_m_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-claude-sonnet litellm | f1 0.666 | 1 | $0.0010 | 2692 ms | 1 | — | |
| fn_anthropic_s_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-claude-haiku litellm | f1 0.584 | 1 | $0.0006 | 1338 ms | 1 | — | |
| fn_capitalized_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | — capitalized · local | f1 0.191 | 1 | — | 0 ms | 1 | — | |
| fn_deepseek_l_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-deepseek-pro litellm | f1 0.587 | 1 | $0.0003 | 3727 ms | 1 | — | |
| fn_deepseek_m_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-deepseek-41flash litellm | f1 0.615 | 1 | $0.0001 | 1229 ms | 1 | — | |
| fn_deepseek_s_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-deepseek-flash litellm | f1 0.576 | 1 | $0.0000 | 2277 ms | 1 | — | |
| fn_gemma_l_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-gemma-31b litellm | f1 0.648 | 1 | $0.0000 | 2910 ms | 1 | — | |
| fn_gemma_m_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-gemma-26b litellm | f1 0.673 | 1 | $0.0000 | 3050 ms | 1 | — | |
| fn_gemma_s_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-gemma-3-27b litellm | f1 0.550 | 1 | $0.0000 | 5941 ms | 1 | — | |
| fn_gliner_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | urchade/gliner_medium-v2.1 gliner · local | f1 0.454 | 1 | — | 169 ms | 1 | — | |
| fn_glm_l_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-glm-5 litellm | f1 0.566 | 1 | $0.0004 | 3014 ms | 1 | — | |
| fn_glm_m_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-glm-46 litellm | f1 0.594 | 1 | $0.0002 | 3659 ms | 1 | — | |
| fn_glm_s_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-glm-45-air litellm | f1 0.553 | 1 | $0.0001 | 2111 ms | 1 | — | |
| fn_llama_l_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-llama-maverick litellm | f1 0.599 | 1 | $0.0001 | 4797 ms | 1 | — | |
| fn_llama_m_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-llama-70b litellm | f1 0.574 | 1 | $0.0001 | 5402 ms | 1 | — | |
| fn_llama_s_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-llama-scout litellm | f1 0.569 | 1 | $0.0000 | 2638 ms | 1 | — | |
| fn_mistral_l_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-mistral-large litellm | f1 0.598 | 1 | $0.0001 | 10247 ms | 1 | — | |
| fn_mistral_m_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-mistral-medium litellm | f1 0.591 | 1 | $0.0002 | 794 ms | 1 | — | |
| fn_mistral_s_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-mistral-small litellm | f1 0.524 | 1 | $0.0000 | 2064 ms | 1 | — | |
| fn_nothing_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | — nothing · local | f1 0.125 | 1 | — | 0 ms | 1 | — | |
| fn_openai_l_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-gpt-6sol litellm | f1 0.672 | 1 | $0.0006 | 1903 ms | 1 | — | |
| fn_openai_m_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-gpt-55 litellm | f1 0.686 | 1 | $0.0023 | 1598 ms | 1 | — | |
| fn_openai_s_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-gpt-mini litellm | f1 0.595 | 1 | $0.0003 | 1206 ms | 1 | — | |
| fn_qwen_l_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-qwen-3max litellm | f1 0.624 | 1 | $0.0003 | 1667 ms | 1 | — | |
| fn_qwen_m_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-qwen-37plus litellm | f1 0.655 | 1 | $0.0001 | 2241 ms | 1 | — | |
| fn_qwen_s_n200_v1 | rescored against gold/few_nerd_280 | few_nerd_280 | eval-qwen-flash litellm | f1 0.665 | 1 | $0.0001 | 2154 ms | 1 | — | |
| fn_span_marker_n200_v1 baselineleaderrecommended | rescored against gold/few_nerd_280 | few_nerd_280 | guishe/span-marker-generic-ner-v1-fewnerd-fine-super span_marker · local | f1 0.767 | 1 | — | 758 ms | 1 | — | |
| dedup_guard_v1 | measured | fleet_dedup_v1 | — — · local | key_match 1.000 | 1 | — | 0 ms | 1 | — | |
| dedup_pinned_f07efa9_v1 | measured | fleet_dedup_v1 | — — · local | — | 1 | — | 0 ms | 1 | — | |
| triage_baseline_pro_noreason_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-fleet-triage-pro @ hosted smoked gateway | triage_cost 1.154 | 1 | $0.0049 | 8352 ms | 1 | — | |
| triage_baseline_pro_noreason_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 1.154 → 1.154 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-fleet-triage-pro @ hosted smoked gateway | triage_cost 1.154 | 1 | $0.0049 | 8352 ms | 0 | — | |
| triage_baseline_pro_noreason_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 1.154 → 1.154 (+0.000) vs measured | orrery_triage_v1_labelled | eval-fleet-triage-pro @ hosted smoked gateway | triage_cost 1.154 | 1 | $0.0049 | 8352 ms | 0 | — | |
| triage_baseline_pro_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-fleet-triage-pro gateway | false_dismiss 0.176 | 1 | $0.0050 | 22736 ms | 1 | — | |
| triage_baseline_pro_v1 · repeats ×3 | run again (repeats) 0.176 → 0.167 (-0.010) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-fleet-triage-pro gateway | false_dismiss 0.167 | 3 | $0.0051 | 26285 ms | 3 | — | |
| triage_baseline_pro_v1 · rescored 3d3a6afd, repeats ×3 | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-fleet-triage-pro gateway | triage_cost 0.764 | 3 | $0.0051 | 26285 ms | 0 | — | |
| triage_baseline_pro_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-fleet-triage-pro gateway | triage_cost 0.785 | 1 | $0.0050 | 22736 ms | 0 | — | |
| triage_baseline_pro_v1 · rescored 5f2c057c, repeats ×3 | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-fleet-triage-pro gateway | triage_cost 0.764 | 3 | $0.0051 | 26285 ms | 0 | — | |
| triage_baseline_pro_v1 · rescored 5f2c057c baseline | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-fleet-triage-pro gateway | triage_cost 0.785 | 1 | $0.0050 | 22736 ms | 0 | — | |
| triage_baseline_pro_v1 · rescored 77adc97b, repeats ×3 | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-fleet-triage-pro gateway | triage_cost 0.764 | 3 | $0.0051 | 26285 ms | 0 | — | |
| triage_baseline_pro_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-fleet-triage-pro gateway | triage_cost 0.785 | 1 | $0.0050 | 22736 ms | 0 | — | |
| triage_baseline_pro_v1 · rescored cd35dc1f, repeats ×3 | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-fleet-triage-pro gateway | triage_cost 0.764 | 3 | $0.0051 | 26285 ms | 0 | — | |
| triage_baseline_pro_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-fleet-triage-pro gateway | triage_cost 0.785 | 1 | $0.0050 | 22736 ms | 0 | — | |
| triage_deepseek_41flash_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-deepseek-41flash gateway | false_dismiss 0.176 | 1 | $0.0009 | 9833 ms | 1 | — | |
| triage_deepseek_41flash_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-deepseek-41flash gateway | triage_cost 0.754 | 1 | $0.0009 | 9833 ms | 0 | — | |
| triage_deepseek_41flash_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-deepseek-41flash gateway | triage_cost 0.754 | 1 | $0.0009 | 9833 ms | 0 | — | |
| triage_deepseek_41flash_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-deepseek-41flash gateway | triage_cost 0.754 | 1 | $0.0009 | 9833 ms | 0 | — | |
| triage_deepseek_41flash_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-deepseek-41flash @ hosted smoked gateway | triage_cost 0.908 | 1 | $0.0004 | 3282 ms | 1 | — | |
| triage_deepseek_41flash_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.908 → 0.908 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-deepseek-41flash @ hosted smoked gateway | triage_cost 0.908 | 1 | $0.0004 | 3282 ms | 0 | — | |
| triage_deepseek_41flash_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.908 → 0.908 (+0.000) vs measured | orrery_triage_v1_labelled | eval-deepseek-41flash @ hosted smoked gateway | triage_cost 0.908 | 1 | $0.0004 | 3282 ms | 0 | — | |
| triage_gemini_35_flash_lite_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-gemini-35-flash-lite gateway | false_dismiss 0.088 | 1 | $0.0014 | 3487 ms | 1 | — | |
| triage_gemini_35_flash_lite_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gemini-35-flash-lite gateway | triage_cost 0.538 | 1 | $0.0014 | 3487 ms | 0 | — | |
| triage_gemini_35_flash_lite_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-gemini-35-flash-lite gateway | triage_cost 0.538 | 1 | $0.0014 | 3487 ms | 0 | — | |
| triage_gemini_35_flash_lite_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gemini-35-flash-lite gateway | triage_cost 0.538 | 1 | $0.0014 | 3487 ms | 0 | — | |
| triage_gemini_35_flash_lite_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gemini-35-flash-lite gateway | triage_cost 0.538 | 1 | $0.0014 | 3487 ms | 0 | — | |
| triage_gemini_4_argon_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-gemini-4-argon gateway | false_dismiss 0.118 | 1 | $0.0136 | 30118 ms | 1 | — | |
| triage_gemini_4_argon_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gemini-4-argon gateway | triage_cost 0.646 | 1 | $0.0136 | 30118 ms | 0 | — | |
| triage_gemini_4_argon_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-gemini-4-argon gateway | triage_cost 0.646 | 1 | $0.0136 | 30118 ms | 0 | — | |
| triage_gemini_4_argon_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gemini-4-argon gateway | triage_cost 0.646 | 1 | $0.0136 | 30118 ms | 0 | — | |
| triage_gemini_4_argon_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gemini-4-argon gateway | triage_cost 0.646 | 1 | $0.0136 | 30118 ms | 0 | — | |
| triage_gpt_61_sol_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-61-sol gateway | false_dismiss 0.118 | 1 | $0.0108 | 12241 ms | 1 | — | |
| triage_gpt_61_sol_v1 · repeats ×3 | run again (repeats) 0.118 → 0.088 (-0.029) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-61-sol gateway | false_dismiss 0.088 | 3 | $0.0109 | 12006 ms | 3 | — | |
| triage_gpt_61_sol_v1 · rescored 3d3a6afd, repeats ×3 | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-61-sol gateway | triage_cost 0.585 | 3 | $0.0109 | 12006 ms | 0 | — | |
| triage_gpt_61_sol_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-61-sol gateway | triage_cost 0.662 | 1 | $0.0108 | 12241 ms | 0 | — | |
| triage_gpt_61_sol_v1 · rescored 5f2c057c, repeats ×3 | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-gpt-61-sol gateway | triage_cost 0.585 | 3 | $0.0109 | 12006 ms | 0 | — | |
| triage_gpt_61_sol_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-gpt-61-sol gateway | triage_cost 0.662 | 1 | $0.0108 | 12241 ms | 0 | — | |
| triage_gpt_61_sol_v1 · rescored 77adc97b, repeats ×3 | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-61-sol gateway | triage_cost 0.585 | 3 | $0.0109 | 12006 ms | 0 | — | |
| triage_gpt_61_sol_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-61-sol gateway | triage_cost 0.662 | 1 | $0.0108 | 12241 ms | 0 | — | |
| triage_gpt_61_sol_v1 · rescored cd35dc1f, repeats ×3 | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-61-sol gateway | triage_cost 0.585 | 3 | $0.0109 | 12006 ms | 0 | — | |
| triage_gpt_61_sol_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-61-sol gateway | triage_cost 0.662 | 1 | $0.0108 | 12241 ms | 0 | — | |
| triage_gpt_6_luna_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-6-luna gateway | false_dismiss 0.088 | 1 | $0.0007 | 12200 ms | 1 | — | |
| triage_gpt_6_luna_v1 · repeats ×3 | run again (repeats) 0.088 → 0.118 (+0.029) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-6-luna gateway | false_dismiss 0.118 | 3 | $0.0007 | 11598 ms | 3 | — | |
| triage_gpt_6_luna_v1 · rescored 5f2c057c, repeats ×3 | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-gpt-6-luna gateway | triage_cost 0.626 | 3 | $0.0007 | 11598 ms | 0 | — | |
| triage_gpt_6_luna_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-gpt-6-luna gateway | triage_cost 0.585 | 1 | $0.0007 | 12200 ms | 0 | — | |
| triage_gpt_6_luna_v1 · rescored 77adc97b, repeats ×3 | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-6-luna gateway | triage_cost 0.626 | 3 | $0.0007 | 11598 ms | 0 | — | |
| triage_gpt_6_luna_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-6-luna gateway | triage_cost 0.585 | 1 | $0.0007 | 12200 ms | 0 | — | |
| triage_gpt_6_luna_v1 · rescored cd35dc1f, repeats ×3 | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-6-luna gateway | triage_cost 0.626 | 3 | $0.0007 | 11598 ms | 0 | — | |
| triage_gpt_6_luna_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-6-luna gateway | triage_cost 0.585 | 1 | $0.0007 | 12200 ms | 0 | — | |
| triage_gpt_6_luna_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-6-luna @ hosted smoked gateway | triage_cost 0.554 | 1 | $0.0007 | 7973 ms | 1 | — | |
| triage_gpt_6_luna_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.554 → 0.554 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-6-luna @ hosted smoked gateway | triage_cost 0.554 | 1 | $0.0007 | 7973 ms | 0 | — | |
| triage_gpt_6_luna_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.554 → 0.554 (+0.000) vs measured | orrery_triage_v1_labelled | eval-gpt-6-luna @ hosted smoked gateway | triage_cost 0.554 | 1 | $0.0007 | 7973 ms | 0 | — | |
| triage_haiku_55_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-claude-haiku-55 gateway | false_dismiss 0.118 | 1 | $0.0010 | 7626 ms | 1 | — | |
| triage_haiku_55_v1 · repeats ×3 | run again (repeats) 0.118 → 0.118 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-claude-haiku-55 gateway | false_dismiss 0.118 | 3 | $0.0010 | 7529 ms | 3 | — | |
| triage_haiku_55_v1 · rescored 3d3a6afd, repeats ×3 | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-claude-haiku-55 gateway | triage_cost 0.610 | 3 | $0.0010 | 7529 ms | 0 | — | |
| triage_haiku_55_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-claude-haiku-55 gateway | triage_cost 0.585 | 1 | $0.0010 | 7626 ms | 0 | — | |
| triage_haiku_55_v1 · rescored 5f2c057c, repeats ×3 | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-claude-haiku-55 gateway | triage_cost 0.610 | 3 | $0.0010 | 7529 ms | 0 | — | |
| triage_haiku_55_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-claude-haiku-55 gateway | triage_cost 0.585 | 1 | $0.0010 | 7626 ms | 0 | — | |
| triage_haiku_55_v1 · rescored 77adc97b, repeats ×3 | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-claude-haiku-55 gateway | triage_cost 0.610 | 3 | $0.0010 | 7529 ms | 0 | — | |
| triage_haiku_55_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-claude-haiku-55 gateway | triage_cost 0.585 | 1 | $0.0010 | 7626 ms | 0 | — | |
| triage_haiku_55_v1 · rescored cd35dc1f, repeats ×3 | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-claude-haiku-55 gateway | triage_cost 0.610 | 3 | $0.0010 | 7529 ms | 0 | — | |
| triage_haiku_55_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-claude-haiku-55 gateway | triage_cost 0.585 | 1 | $0.0010 | 7626 ms | 0 | — | |
| triage_hosted_gemma4_31b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-gemma-31b gateway | false_dismiss 0.176 | 1 | $0.0009 | 10510 ms | 1 | — | |
| triage_hosted_gemma4_31b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-gemma-31b gateway | triage_cost 0.723 | 1 | $0.0009 | 10510 ms | 0 | — | |
| triage_hosted_gemma4_31b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gemma-31b gateway | triage_cost 0.723 | 1 | $0.0009 | 10510 ms | 0 | — | |
| triage_hosted_gemma4_31b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gemma-31b gateway | triage_cost 0.723 | 1 | $0.0009 | 10510 ms | 0 | — | |
| triage_hosted_gemma4_31b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-gemma-31b @ hosted smoked gateway | triage_cost 0.723 | 1 | $0.0009 | 10550 ms | 1 | — | |
| triage_hosted_gemma4_31b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.723 → 0.723 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-gemma-31b @ hosted smoked gateway | triage_cost 0.723 | 1 | $0.0009 | 10550 ms | 0 | — | |
| triage_hosted_gemma4_31b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.723 → 0.723 (+0.000) vs measured | orrery_triage_v1_labelled | eval-gemma-31b @ hosted smoked gateway | triage_cost 0.723 | 1 | $0.0009 | 10550 ms | 0 | — | |
| triage_hosted_gpt_oss_120b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-oss-120b gateway | false_dismiss 0.147 | 1 | $0.0008 | 17021 ms | 1 | — | |
| triage_hosted_gpt_oss_120b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-gpt-oss-120b gateway | triage_cost 0.677 | 1 | $0.0008 | 17021 ms | 0 | — | |
| triage_hosted_gpt_oss_120b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-oss-120b gateway | triage_cost 0.677 | 1 | $0.0008 | 17021 ms | 0 | — | |
| triage_hosted_gpt_oss_120b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-oss-120b gateway | triage_cost 0.677 | 1 | $0.0008 | 17021 ms | 0 | — | |
| triage_hosted_gpt_oss_120b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-oss-120b @ hosted smoked gateway | triage_cost 0.631 | 1 | $0.0006 | 6541 ms | 1 | — | |
| triage_hosted_gpt_oss_120b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.631 → 0.631 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-gpt-oss-120b @ hosted smoked gateway | triage_cost 0.631 | 1 | $0.0006 | 6541 ms | 0 | — | |
| triage_hosted_gpt_oss_120b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.631 → 0.631 (+0.000) vs measured | orrery_triage_v1_labelled | eval-gpt-oss-120b @ hosted smoked gateway | triage_cost 0.631 | 1 | $0.0006 | 6541 ms | 0 | — | |
| triage_hosted_mistral_medium_35_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-mistral-medium-35 gateway | false_dismiss 0.235 | 1 | $0.0115 | 6103 ms | 1 | — | |
| triage_hosted_mistral_medium_35_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-mistral-medium-35 gateway | triage_cost 0.877 | 1 | $0.0115 | 6103 ms | 0 | — | |
| triage_hosted_mistral_medium_35_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-mistral-medium-35 gateway | triage_cost 0.877 | 1 | $0.0115 | 6103 ms | 0 | — | |
| triage_hosted_mistral_medium_35_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-mistral-medium-35 gateway | triage_cost 0.877 | 1 | $0.0115 | 6103 ms | 0 | — | |
| triage_hosted_mistral_medium_35_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-mistral-medium-35 gateway | triage_cost 0.877 | 1 | $0.0115 | 6103 ms | 0 | — | |
| triage_hosted_qwen35_122b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-qwen35-122b gateway | false_dismiss 0.294 | 1 | $0.0033 | 17662 ms | 1 | — | |
| triage_hosted_qwen35_122b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-qwen35-122b gateway | triage_cost 1.077 | 1 | $0.0033 | 17662 ms | 0 | — | |
| triage_hosted_qwen35_122b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-qwen35-122b gateway | triage_cost 1.077 | 1 | $0.0033 | 17662 ms | 0 | — | |
| triage_hosted_qwen35_122b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-qwen35-122b gateway | triage_cost 1.077 | 1 | $0.0033 | 17662 ms | 0 | — | |
| triage_hosted_qwen35_122b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-qwen35-122b @ hosted smoked gateway | triage_cost 0.877 | 1 | $0.0017 | 6681 ms | 1 | — | |
| triage_hosted_qwen35_122b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.877 → 0.877 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-qwen35-122b @ hosted smoked gateway | triage_cost 0.877 | 1 | $0.0017 | 6681 ms | 0 | — | |
| triage_hosted_qwen35_122b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.877 → 0.877 (+0.000) vs measured | orrery_triage_v1_labelled | eval-qwen35-122b @ hosted smoked gateway | triage_cost 0.877 | 1 | $0.0017 | 6681 ms | 0 | — | |
| triage_hosted_qwen3_30b_a3b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-qwen3-30b-a3b gateway | triage_cost 0.708 | 1 | $0.0007 | 17976 ms | 1 | — | |
| triage_hosted_qwen3_30b_a3b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.708 → 0.708 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-qwen3-30b-a3b gateway | triage_cost 0.708 | 1 | $0.0007 | 17976 ms | 0 | — | |
| triage_hosted_qwen3_30b_a3b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.708 → 0.708 (+0.000) vs measured | orrery_triage_v1_labelled | eval-qwen3-30b-a3b gateway | triage_cost 0.708 | 1 | $0.0007 | 17976 ms | 0 | — | |
| triage_hosted_qwen3_30b_a3b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.708 → 0.708 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-qwen3-30b-a3b gateway | triage_cost 0.708 | 1 | $0.0007 | 17976 ms | 0 | — | |
| triage_hosted_qwen3_30b_a3b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.708 → 0.708 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-qwen3-30b-a3b gateway | triage_cost 0.708 | 1 | $0.0007 | 17976 ms | 0 | — | |
| triage_kimi_k31_pro_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-kimi-k31-pro gateway | false_dismiss 0.118 | 1 | $0.0191 | 46089 ms | 1 | — | |
| triage_kimi_k31_pro_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-kimi-k31-pro gateway | triage_cost 0.646 | 1 | $0.0191 | 46089 ms | 0 | — | |
| triage_kimi_k31_pro_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-kimi-k31-pro gateway | triage_cost 0.646 | 1 | $0.0191 | 46089 ms | 0 | — | |
| triage_kimi_k31_pro_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-kimi-k31-pro gateway | triage_cost 0.646 | 1 | $0.0191 | 46089 ms | 0 | — | |
| triage_kimi_k31_pro_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-kimi-k31-pro @ hosted smoked gateway | triage_cost 0.723 | 1 | $0.0120 | 21533 ms | 1 | — | |
| triage_kimi_k31_pro_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.723 → 0.723 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-kimi-k31-pro @ hosted smoked gateway | triage_cost 0.723 | 1 | $0.0120 | 21533 ms | 0 | — | |
| triage_kimi_k31_pro_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.723 → 0.723 (+0.000) vs measured | orrery_triage_v1_labelled | eval-kimi-k31-pro @ hosted smoked gateway | triage_cost 0.723 | 1 | $0.0120 | 21533 ms | 0 | — | |
| triage_local_dgx_gemma4_12b_noreason_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:12b gateway | triage_cost 0.585 | 1 | — | 5697 ms | 1 | — | |
| triage_local_dgx_gemma4_12b_noreason_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.585 → 0.585 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:12b gateway | triage_cost 0.585 | 1 | — | 5697 ms | 0 | — | |
| triage_local_dgx_gemma4_12b_noreason_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.585 → 0.585 (+0.000) vs measured | orrery_triage_v1_labelled | gemma4:12b gateway | triage_cost 0.585 | 1 | — | 5697 ms | 0 | — | |
| triage_local_dgx_gemma4_12b_noreason_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.585 → 0.585 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:12b gateway | triage_cost 0.585 | 1 | — | 5697 ms | 0 | — | |
| triage_local_dgx_gemma4_12b_noreason_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.585 → 0.585 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:12b gateway | triage_cost 0.585 | 1 | — | 5697 ms | 0 | — | |
| triage_local_dgx_gemma4_31b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b gateway | triage_cost 0.677 | 1 | — | 58863 ms | 1 | — | |
| triage_local_dgx_gemma4_31b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured | orrery_triage_v1_labelled | gemma4:31b gateway | triage_cost 0.677 | 1 | — | 58863 ms | 0 | — | |
| triage_local_dgx_gemma4_31b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b gateway | triage_cost 0.677 | 1 | — | 58863 ms | 0 | — | |
| triage_local_dgx_gemma4_31b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b gateway | triage_cost 0.677 | 1 | — | 58863 ms | 0 | — | |
| triage_local_dgx_gemma4_31b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b @ dgx smoked gateway | triage_cost 0.723 | 1 | — | 13030 ms | 1 | — | |
| triage_local_dgx_gemma4_31b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.723 → 0.723 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b @ dgx smoked gateway | triage_cost 0.723 | 1 | — | 13030 ms | 0 | — | |
| triage_local_dgx_gemma4_31b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.723 → 0.723 (+0.000) vs measured | orrery_triage_v1_labelled | gemma4:31b @ dgx smoked gateway | triage_cost 0.723 | 1 | — | 13030 ms | 0 | — | |
| triage_local_dgx_gpt_oss_120b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:120b gateway | triage_cost 0.692 | 1 | — | 14766 ms | 1 | — | |
| triage_local_dgx_gpt_oss_120b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.692 → 0.692 (+0.000) vs measured | orrery_triage_v1_labelled | gpt-oss:120b gateway | triage_cost 0.692 | 1 | — | 14766 ms | 0 | — | |
| triage_local_dgx_gpt_oss_120b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.692 → 0.692 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:120b gateway | triage_cost 0.692 | 1 | — | 14766 ms | 0 | — | |
| triage_local_dgx_gpt_oss_120b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.692 → 0.692 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:120b gateway | triage_cost 0.692 | 1 | — | 14766 ms | 0 | — | |
| triage_local_dgx_gpt_oss_120b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:120b @ dgx smoke: check gateway | triage_cost 0.662 | 1 | — | 7391 ms | 1 | — | |
| triage_local_dgx_gpt_oss_120b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.662 → 0.662 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:120b @ dgx smoke: check gateway | triage_cost 0.662 | 1 | — | 7391 ms | 0 | — | |
| triage_local_dgx_gpt_oss_120b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.662 → 0.662 (+0.000) vs measured | orrery_triage_v1_labelled | gpt-oss:120b @ dgx smoke: check gateway | triage_cost 0.662 | 1 | — | 7391 ms | 0 | — | |
| triage_local_dgx_gpt_oss_20b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b gateway | triage_cost 0.477 | 1 | — | 13456 ms | 1 | — | |
| triage_local_dgx_gpt_oss_20b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.477 → 1.015 (+0.538) vs measured | orrery_triage_v1_labelled | gpt-oss:20b gateway | triage_cost 1.015 | 1 | — | 13456 ms | 0 | — | |
| triage_local_dgx_gpt_oss_20b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.477 → 1.015 (+0.538) vs measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b gateway | triage_cost 1.015 | 1 | — | 13456 ms | 0 | — | |
| triage_local_dgx_gpt_oss_20b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.477 → 0.477 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b gateway | triage_cost 0.477 | 1 | — | 13456 ms | 0 | — | |
| triage_local_dgx_gpt_oss_20b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b @ dgx smoke: check gateway | triage_cost 0.723 | 1 | — | 4681 ms | 1 | — | |
| triage_local_dgx_gpt_oss_20b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.723 → 0.723 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b @ dgx smoke: check gateway | triage_cost 0.723 | 1 | — | 4681 ms | 0 | — | |
| triage_local_dgx_gpt_oss_20b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.723 → 0.723 (+0.000) vs measured | orrery_triage_v1_labelled | gpt-oss:20b @ dgx smoke: check gateway | triage_cost 0.723 | 1 | — | 4681 ms | 0 | — | |
| triage_local_dgx_llama31_8b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 5766 ms | 1 | — | |
| triage_local_dgx_llama31_8b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.415 → 0.415 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 5766 ms | 0 | — | |
| triage_local_dgx_llama31_8b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.415 → 0.415 (+0.000) vs measured | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 5766 ms | 0 | — | |
| triage_local_dgx_llama31_8b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.415 → 0.415 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 5766 ms | 0 | — | |
| triage_local_dgx_llama31_8b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.415 → 0.415 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 5766 ms | 0 | — | |
| triage_local_dgx_llama4_scout_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | llama4:scout gateway | triage_cost 1.262 | 1 | — | 13354 ms | 1 | — | |
| triage_local_dgx_llama4_scout_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 1.262 → 1.262 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | llama4:scout gateway | triage_cost 1.262 | 1 | — | 13354 ms | 0 | — | |
| triage_local_dgx_llama4_scout_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 1.262 → 1.262 (+0.000) vs measured | orrery_triage_v1_labelled | llama4:scout gateway | triage_cost 1.262 | 1 | — | 13354 ms | 0 | — | |
| triage_local_dgx_llama4_scout_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 1.262 → 1.262 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | llama4:scout gateway | triage_cost 1.262 | 1 | — | 13354 ms | 0 | — | |
| triage_local_dgx_llama4_scout_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 1.262 → 1.262 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | llama4:scout gateway | triage_cost 1.262 | 1 | — | 13354 ms | 0 | — | |
| triage_local_dgx_ministral_3_14b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.831 | 1 | — | 20452 ms | 1 | — | |
| triage_local_dgx_ministral_3_14b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.831 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.831 | 1 | — | 20452 ms | 0 | — | |
| triage_local_dgx_ministral_3_14b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.831 (+0.000) vs measured | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.831 | 1 | — | 20452 ms | 0 | — | |
| triage_local_dgx_ministral_3_14b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.831 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.831 | 1 | — | 20452 ms | 0 | — | |
| triage_local_dgx_ministral_3_14b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.831 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.831 | 1 | — | 20452 ms | 0 | — | |
| triage_local_dgx_mistral_medium_35_128b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | mistral-medium-3.5:128b gateway | triage_cost 1.000 | 1 | — | 117538 ms | 1 | — | |
| triage_local_dgx_mistral_medium_35_128b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 1.000 → 1.000 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | mistral-medium-3.5:128b gateway | triage_cost 1.000 | 1 | — | 117538 ms | 0 | — | |
| triage_local_dgx_mistral_medium_35_128b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 1.000 → 1.000 (+0.000) vs measured | orrery_triage_v1_labelled | mistral-medium-3.5:128b gateway | triage_cost 1.000 | 1 | — | 117538 ms | 0 | — | |
| triage_local_dgx_mistral_medium_35_128b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 1.000 → 1.000 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | mistral-medium-3.5:128b gateway | triage_cost 1.000 | 1 | — | 117538 ms | 0 | — | |
| triage_local_dgx_mistral_medium_35_128b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 1.000 → 1.000 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | mistral-medium-3.5:128b gateway | triage_cost 1.000 | 1 | — | 117538 ms | 0 | — | |
| triage_local_dgx_nemotron_35_bolt_e4b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.523 | 1 | — | 15259 ms | 1 | — | |
| triage_local_dgx_nemotron_35_bolt_e4b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.523 → 0.523 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.523 | 1 | — | 15259 ms | 0 | — | |
| triage_local_dgx_nemotron_35_bolt_e4b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.523 → 0.523 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.523 | 1 | — | 15259 ms | 0 | — | |
| triage_local_dgx_nemotron_35_bolt_e4b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.523 → 0.523 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.523 | 1 | — | 15259 ms | 0 | — | |
| triage_local_dgx_nemotron_35_bolt_e4b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M @ dgx smoke: check gateway | triage_cost 0.646 | 1 | — | 3881 ms | 1 | — | |
| triage_local_dgx_nemotron_35_bolt_e4b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.646 → 0.646 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M @ dgx smoke: check gateway | triage_cost 0.646 | 1 | — | 3881 ms | 0 | — | |
| triage_local_dgx_nemotron_35_bolt_e4b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.646 → 0.646 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M @ dgx smoke: check gateway | triage_cost 0.646 | 1 | — | 3881 ms | 0 | — | |
| triage_local_dgx_nemotron_35_lightning_30b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b gateway | triage_cost 0.646 | 1 | — | 39979 ms | 1 | — | |
| triage_local_dgx_nemotron_35_lightning_30b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.646 → 0.646 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b gateway | triage_cost 0.646 | 1 | — | 39979 ms | 0 | — | |
| triage_local_dgx_nemotron_35_lightning_30b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.646 → 0.646 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b gateway | triage_cost 0.646 | 1 | — | 39979 ms | 0 | — | |
| triage_local_dgx_nemotron_35_lightning_30b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.646 → 0.646 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b gateway | triage_cost 0.646 | 1 | — | 39979 ms | 0 | — | |
| triage_local_dgx_nemotron_35_lightning_30b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b @ dgx smoked gateway | triage_cost 0.677 | 1 | — | 6786 ms | 1 | — | |
| triage_local_dgx_nemotron_35_lightning_30b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b @ dgx smoked gateway | triage_cost 0.677 | 1 | — | 6786 ms | 0 | — | |
| triage_local_dgx_nemotron_35_lightning_30b_v2 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b @ dgx smoked gateway | triage_cost 0.677 | 1 | — | 6786 ms | 0 | — | |
| triage_local_dgx_nemotron_35_super_120b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-super:120b gateway | triage_cost 0.677 | 1 | — | 41019 ms | 1 | — | |
| triage_local_dgx_nemotron_35_super_120b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-super:120b gateway | triage_cost 0.677 | 1 | — | 41019 ms | 0 | — | |
| triage_local_dgx_nemotron_35_super_120b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-super:120b gateway | triage_cost 0.677 | 1 | — | 41019 ms | 0 | — | |
| triage_local_dgx_nemotron_35_super_120b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-super:120b gateway | triage_cost 0.677 | 1 | — | 41019 ms | 0 | — | |
| triage_local_dgx_nemotron_35_super_120b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-super:120b @ dgx smoked gateway | triage_cost 0.677 | 1 | — | 15898 ms | 1 | — | |
| triage_local_dgx_nemotron_35_super_120b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-super:120b @ dgx smoked gateway | triage_cost 0.677 | 1 | — | 15898 ms | 0 | — | |
| triage_local_dgx_nemotron_35_super_120b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-super:120b @ dgx smoked gateway | triage_cost 0.677 | 1 | — | 15898 ms | 0 | — | |
| triage_local_dgx_phi4_14b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.662 | 1 | — | 16973 ms | 1 | — | |
| triage_local_dgx_phi4_14b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.662 → 0.662 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.662 | 1 | — | 16973 ms | 0 | — | |
| triage_local_dgx_phi4_14b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.662 → 0.662 (+0.000) vs measured | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.662 | 1 | — | 16973 ms | 0 | — | |
| triage_local_dgx_phi4_14b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.662 → 0.662 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.662 | 1 | — | 16973 ms | 0 | — | |
| triage_local_dgx_phi4_14b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.662 → 0.662 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.662 | 1 | — | 16973 ms | 0 | — | |
| triage_local_dgx_qwen35_122b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.5:122b gateway | triage_cost 1.138 | 1 | — | 38007 ms | 1 | — | |
| triage_local_dgx_qwen35_122b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 1.138 → 1.138 (+0.000) vs measured | orrery_triage_v1_labelled | qwen3.5:122b gateway | triage_cost 1.138 | 1 | — | 38007 ms | 0 | — | |
| triage_local_dgx_qwen35_122b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 1.138 → 1.138 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.5:122b gateway | triage_cost 1.138 | 1 | — | 38007 ms | 0 | — | |
| triage_local_dgx_qwen35_122b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 1.138 → 1.138 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.5:122b gateway | triage_cost 1.138 | 1 | — | 38007 ms | 0 | — | |
| triage_local_dgx_qwen35_122b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.5:122b @ dgx smoked gateway | triage_cost 1.354 | 1 | — | 14511 ms | 1 | — | |
| triage_local_dgx_qwen35_122b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 1.354 → 1.354 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.5:122b @ dgx smoked gateway | triage_cost 1.354 | 1 | — | 14511 ms | 0 | — | |
| triage_local_dgx_qwen35_122b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 1.354 → 1.354 (+0.000) vs measured | orrery_triage_v1_labelled | qwen3.5:122b @ dgx smoked gateway | triage_cost 1.354 | 1 | — | 14511 ms | 0 | — | |
| triage_local_dgx_qwen38_27b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b gateway | triage_cost 0.831 | 1 | — | 37826 ms | 1 | — | |
| triage_local_dgx_qwen38_27b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.831 (+0.000) vs measured | orrery_triage_v1_labelled | qwen3.8:27b gateway | triage_cost 0.831 | 1 | — | 37826 ms | 0 | — | |
| triage_local_dgx_qwen38_27b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.831 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b gateway | triage_cost 0.831 | 1 | — | 37826 ms | 0 | — | |
| triage_local_dgx_qwen38_27b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.831 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b gateway | triage_cost 0.831 | 1 | — | 37826 ms | 0 | — | |
| triage_local_dgx_qwen38_27b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b @ dgx smoked gateway | triage_cost 1.138 | 1 | — | 16559 ms | 1 | — | |
| triage_local_dgx_qwen38_27b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 1.138 → 1.138 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b @ dgx smoked gateway | triage_cost 1.138 | 1 | — | 16559 ms | 0 | — | |
| triage_local_dgx_qwen38_27b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 1.138 → 1.138 (+0.000) vs measured | orrery_triage_v1_labelled | qwen3.8:27b @ dgx smoked gateway | triage_cost 1.138 | 1 | — | 16559 ms | 0 | — | |
| triage_local_dgx_qwen3_30b_a3b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.892 | 1 | — | 5859 ms | 1 | — | |
| triage_local_dgx_qwen3_30b_a3b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.892 → 0.892 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.892 | 1 | — | 5859 ms | 0 | — | |
| triage_local_dgx_qwen3_30b_a3b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.892 → 0.892 (+0.000) vs measured | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.892 | 1 | — | 5859 ms | 0 | — | |
| triage_local_dgx_qwen3_30b_a3b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.892 → 0.892 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.892 | 1 | — | 5859 ms | 0 | — | |
| triage_local_dgx_qwen3_30b_a3b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.892 → 0.892 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.892 | 1 | — | 5859 ms | 0 | — | |
| triage_local_dgx_vllm_qwen3_30b_a3b_nvfp4_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | NVFP4/Qwen3-30B-A3B-Instruct-2507-FP4 gateway | triage_cost 0.400 | 1 | — | 7824 ms | 1 | — | |
| triage_local_dgx_vllm_qwen3_30b_a3b_nvfp4_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.400 → 0.400 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | NVFP4/Qwen3-30B-A3B-Instruct-2507-FP4 gateway | triage_cost 0.400 | 1 | — | 7824 ms | 0 | — | |
| triage_local_dgx_vllm_qwen3_30b_a3b_nvfp4_v1 · rescored 5f2c057c leader | rescored against gold/orrery_triage_v1_labelled 0.400 → 0.400 (+0.000) vs measured | orrery_triage_v1_labelled | NVFP4/Qwen3-30B-A3B-Instruct-2507-FP4 gateway | triage_cost 0.400 | 1 | — | 7824 ms | 0 | — | |
| triage_local_dgx_vllm_qwen3_30b_a3b_nvfp4_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.400 → 0.400 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | NVFP4/Qwen3-30B-A3B-Instruct-2507-FP4 gateway | triage_cost 0.400 | 1 | — | 7824 ms | 0 | — | |
| triage_local_dgx_vllm_qwen3_30b_a3b_nvfp4_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.400 → 0.400 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | NVFP4/Qwen3-30B-A3B-Instruct-2507-FP4 gateway | triage_cost 0.400 | 1 | — | 7824 ms | 0 | — | |
| triage_local_mac_gemma4_12b_noreason_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:12b gateway | triage_cost 0.585 | 1 | — | 9174 ms | 1 | — | |
| triage_local_mac_gemma4_12b_noreason_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.585 → 0.585 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:12b gateway | triage_cost 0.585 | 1 | — | 9174 ms | 0 | — | |
| triage_local_mac_gemma4_12b_noreason_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.585 → 0.585 (+0.000) vs measured | orrery_triage_v1_labelled | gemma4:12b gateway | triage_cost 0.585 | 1 | — | 9174 ms | 0 | — | |
| triage_local_mac_gemma4_12b_noreason_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.585 → 0.585 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:12b gateway | triage_cost 0.585 | 1 | — | 9174 ms | 0 | — | |
| triage_local_mac_gemma4_12b_noreason_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.585 → 0.585 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:12b gateway | triage_cost 0.585 | 1 | — | 9174 ms | 0 | — | |
| triage_local_mac_gemma4_31b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b-it-q4_K_M gateway | triage_cost 0.677 | 1 | — | 100889 ms | 1 | — | |
| triage_local_mac_gemma4_31b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured | orrery_triage_v1_labelled | gemma4:31b-it-q4_K_M gateway | triage_cost 0.677 | 1 | — | 100889 ms | 0 | — | |
| triage_local_mac_gemma4_31b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b-it-q4_K_M gateway | triage_cost 0.677 | 1 | — | 100889 ms | 0 | — | |
| triage_local_mac_gemma4_31b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b-it-q4_K_M gateway | triage_cost 0.677 | 1 | — | 100889 ms | 0 | — | |
| triage_local_mac_gemma4_31b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b-it-q4_K_M @ mac smoked gateway | triage_cost 0.800 | 1 | — | 25063 ms | 1 | — | |
| triage_local_mac_gemma4_31b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.800 → 0.800 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b-it-q4_K_M @ mac smoked gateway | triage_cost 0.800 | 1 | — | 25063 ms | 0 | — | |
| triage_local_mac_gemma4_31b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.800 → 0.800 (+0.000) vs measured | orrery_triage_v1_labelled | gemma4:31b-it-q4_K_M @ mac smoked gateway | triage_cost 0.800 | 1 | — | 25063 ms | 0 | — | |
| triage_local_mac_gpt_oss_20b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b gateway | triage_cost 0.554 | 1 | — | 19953 ms | 1 | — | |
| triage_local_mac_gpt_oss_20b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.554 → 1.092 (+0.538) vs measured | orrery_triage_v1_labelled | gpt-oss:20b gateway | triage_cost 1.092 | 1 | — | 19953 ms | 0 | — | |
| triage_local_mac_gpt_oss_20b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.554 → 1.092 (+0.538) vs measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b gateway | triage_cost 1.092 | 1 | — | 19953 ms | 0 | — | |
| triage_local_mac_gpt_oss_20b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.554 → 0.554 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b gateway | triage_cost 0.554 | 1 | — | 19953 ms | 0 | — | |
| triage_local_mac_gpt_oss_20b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b @ mac smoke: check gateway | triage_cost 0.708 | 1 | — | 5802 ms | 1 | — | |
| triage_local_mac_gpt_oss_20b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.708 → 0.708 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b @ mac smoke: check gateway | triage_cost 0.708 | 1 | — | 5802 ms | 0 | — | |
| triage_local_mac_gpt_oss_20b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.708 → 0.708 (+0.000) vs measured | orrery_triage_v1_labelled | gpt-oss:20b @ mac smoke: check gateway | triage_cost 0.708 | 1 | — | 5802 ms | 0 | — | |
| triage_local_mac_llama31_8b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 7780 ms | 1 | — | |
| triage_local_mac_llama31_8b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.415 → 0.415 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 7780 ms | 0 | — | |
| triage_local_mac_llama31_8b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.415 → 0.415 (+0.000) vs measured | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 7780 ms | 0 | — | |
| triage_local_mac_llama31_8b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.415 → 0.415 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 7780 ms | 0 | — | |
| triage_local_mac_llama31_8b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.415 → 0.415 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 7780 ms | 0 | — | |
| triage_local_mac_ministral_3_14b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.831 | 1 | — | 27378 ms | 1 | — | |
| triage_local_mac_ministral_3_14b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.908 (+0.077) vs measured the goals read its re-score | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.908 | 1 | — | 27378 ms | 0 | — | |
| triage_local_mac_ministral_3_14b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.908 (+0.077) vs measured | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.908 | 1 | — | 27378 ms | 0 | — | |
| triage_local_mac_ministral_3_14b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.908 (+0.077) vs measured the goals read its re-score | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.908 | 1 | — | 27378 ms | 0 | — | |
| triage_local_mac_ministral_3_14b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.831 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.831 | 1 | — | 27378 ms | 0 | — | |
| triage_local_mac_nemotron_35_bolt_e4b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.754 | 1 | — | 23126 ms | 1 | — | |
| triage_local_mac_nemotron_35_bolt_e4b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.754 → 0.754 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.754 | 1 | — | 23126 ms | 0 | — | |
| triage_local_mac_nemotron_35_bolt_e4b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.754 → 0.754 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.754 | 1 | — | 23126 ms | 0 | — | |
| triage_local_mac_nemotron_35_bolt_e4b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.754 → 0.754 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.754 | 1 | — | 23126 ms | 0 | — | |
| triage_local_mac_nemotron_35_bolt_e4b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M @ mac smoked gateway | triage_cost 0.585 | 1 | — | 5883 ms | 1 | — | |
| triage_local_mac_nemotron_35_bolt_e4b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.585 → 0.585 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M @ mac smoked gateway | triage_cost 0.585 | 1 | — | 5883 ms | 0 | — | |
| triage_local_mac_nemotron_35_bolt_e4b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.585 → 0.585 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M @ mac smoked gateway | triage_cost 0.585 | 1 | — | 5883 ms | 0 | — | |
| triage_local_mac_nemotron_35_lightning_30b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b-a3b-q4_K_M gateway | triage_cost 0.646 | 1 | — | 66004 ms | 1 | — | |
| triage_local_mac_nemotron_35_lightning_30b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.646 → 0.646 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b-a3b-q4_K_M gateway | triage_cost 0.646 | 1 | — | 66004 ms | 0 | — | |
| triage_local_mac_nemotron_35_lightning_30b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.646 → 0.646 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b-a3b-q4_K_M gateway | triage_cost 0.646 | 1 | — | 66004 ms | 0 | — | |
| triage_local_mac_nemotron_35_lightning_30b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.646 → 0.646 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b-a3b-q4_K_M gateway | triage_cost 0.646 | 1 | — | 66004 ms | 0 | — | |
| triage_local_mac_nemotron_35_lightning_30b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b-a3b-q4_K_M @ mac smoked gateway | triage_cost 0.692 | 1 | — | 11065 ms | 1 | — | |
| triage_local_mac_nemotron_35_lightning_30b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.692 → 0.692 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b-a3b-q4_K_M @ mac smoked gateway | triage_cost 0.692 | 1 | — | 11065 ms | 0 | — | |
| triage_local_mac_nemotron_35_lightning_30b_v2 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.692 → 0.692 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b-a3b-q4_K_M @ mac smoked gateway | triage_cost 0.692 | 1 | — | 11065 ms | 0 | — | |
| triage_local_mac_phi4_14b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.754 | 1 | — | 21773 ms | 1 | — | |
| triage_local_mac_phi4_14b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.754 → 0.754 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.754 | 1 | — | 21773 ms | 0 | — | |
| triage_local_mac_phi4_14b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.754 → 0.754 (+0.000) vs measured | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.754 | 1 | — | 21773 ms | 0 | — | |
| triage_local_mac_phi4_14b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.754 → 0.754 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.754 | 1 | — | 21773 ms | 0 | — | |
| triage_local_mac_phi4_14b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.754 → 0.754 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.754 | 1 | — | 21773 ms | 0 | — | |
| triage_local_mac_qwen38_27b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b gateway | triage_cost 1.046 | 1 | — | 69674 ms | 1 | — | |
| triage_local_mac_qwen38_27b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 1.046 → 1.046 (+0.000) vs measured | orrery_triage_v1_labelled | qwen3.8:27b gateway | triage_cost 1.046 | 1 | — | 69674 ms | 0 | — | |
| triage_local_mac_qwen38_27b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 1.046 → 1.046 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b gateway | triage_cost 1.046 | 1 | — | 69674 ms | 0 | — | |
| triage_local_mac_qwen38_27b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 1.046 → 1.046 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b gateway | triage_cost 1.046 | 1 | — | 69674 ms | 0 | — | |
| triage_local_mac_qwen38_27b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b @ mac smoked gateway | triage_cost 1.138 | 1 | — | 33395 ms | 1 | — | |
| triage_local_mac_qwen38_27b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 1.138 → 1.138 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b @ mac smoked gateway | triage_cost 1.138 | 1 | — | 33395 ms | 0 | — | |
| triage_local_mac_qwen38_27b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 1.138 → 1.138 (+0.000) vs measured | orrery_triage_v1_labelled | qwen3.8:27b @ mac smoked gateway | triage_cost 1.138 | 1 | — | 33395 ms | 0 | — | |
| triage_local_mac_qwen3_30b_a3b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.831 | 1 | — | 8996 ms | 1 | — | |
| triage_local_mac_qwen3_30b_a3b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.831 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.831 | 1 | — | 8996 ms | 0 | — | |
| triage_local_mac_qwen3_30b_a3b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.831 (+0.000) vs measured | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.831 | 1 | — | 8996 ms | 0 | — | |
| triage_local_mac_qwen3_30b_a3b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.831 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.831 | 1 | — | 8996 ms | 0 | — | |
| triage_local_mac_qwen3_30b_a3b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.831 → 0.831 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.831 | 1 | — | 8996 ms | 0 | — | |
| triage_local_x3_gemma4_31b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b gateway | triage_cost 0.692 | 1 | — | 72058 ms | 1 | — | |
| triage_local_x3_gemma4_31b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.692 → 0.692 (+0.000) vs measured | orrery_triage_v1_labelled | gemma4:31b gateway | triage_cost 0.692 | 1 | — | 72058 ms | 0 | — | |
| triage_local_x3_gemma4_31b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.692 → 0.692 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b gateway | triage_cost 0.692 | 1 | — | 72058 ms | 0 | — | |
| triage_local_x3_gemma4_31b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.692 → 0.692 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b gateway | triage_cost 0.692 | 1 | — | 72058 ms | 0 | — | |
| triage_local_x3_gemma4_31b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b @ x3 smoke: check gateway | triage_cost 0.708 | 1 | — | 14320 ms | 1 | — | |
| triage_local_x3_gemma4_31b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.708 → 0.708 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gemma4:31b @ x3 smoke: check gateway | triage_cost 0.708 | 1 | — | 14320 ms | 0 | — | |
| triage_local_x3_gemma4_31b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.708 → 0.708 (+0.000) vs measured | orrery_triage_v1_labelled | gemma4:31b @ x3 smoke: check gateway | triage_cost 0.708 | 1 | — | 14320 ms | 0 | — | |
| triage_local_x3_gpt_oss_20b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b gateway | triage_cost 0.400 | 1 | — | 17235 ms | 1 | — | |
| triage_local_x3_gpt_oss_20b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.400 → 1.092 (+0.692) vs measured | orrery_triage_v1_labelled | gpt-oss:20b gateway | triage_cost 1.092 | 1 | — | 17235 ms | 0 | — | |
| triage_local_x3_gpt_oss_20b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.400 → 1.092 (+0.692) vs measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b gateway | triage_cost 1.092 | 1 | — | 17235 ms | 0 | — | |
| triage_local_x3_gpt_oss_20b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.400 → 0.400 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b gateway | triage_cost 0.400 | 1 | — | 17235 ms | 0 | — | |
| triage_local_x3_gpt_oss_20b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b @ x3 smoke: check gateway | triage_cost 0.815 | 1 | — | 5864 ms | 1 | — | |
| triage_local_x3_gpt_oss_20b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.815 → 0.815 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | gpt-oss:20b @ x3 smoke: check gateway | triage_cost 0.815 | 1 | — | 5864 ms | 0 | — | |
| triage_local_x3_gpt_oss_20b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.815 → 0.815 (+0.000) vs measured | orrery_triage_v1_labelled | gpt-oss:20b @ x3 smoke: check gateway | triage_cost 0.815 | 1 | — | 5864 ms | 0 | — | |
| triage_local_x3_llama31_8b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | llama3.1:8b gateway | false_dismiss 0.000 | 1 | — | 5837 ms | 1 | — | |
| triage_local_x3_llama31_8b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 5837 ms | 0 | — | |
| triage_local_x3_llama31_8b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 5837 ms | 0 | — | |
| triage_local_x3_llama31_8b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 5837 ms | 0 | — | |
| triage_local_x3_llama31_8b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | llama3.1:8b gateway | triage_cost 0.415 | 1 | — | 5837 ms | 0 | — | |
| triage_local_x3_ministral_3_14b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.815 | 1 | — | 22057 ms | 1 | — | |
| triage_local_x3_ministral_3_14b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.815 → 0.815 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.815 | 1 | — | 22057 ms | 0 | — | |
| triage_local_x3_ministral_3_14b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.815 → 0.815 (+0.000) vs measured | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.815 | 1 | — | 22057 ms | 0 | — | |
| triage_local_x3_ministral_3_14b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.815 → 0.815 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.815 | 1 | — | 22057 ms | 0 | — | |
| triage_local_x3_ministral_3_14b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.815 → 0.815 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | ministral-3:14b gateway | triage_cost 0.815 | 1 | — | 22057 ms | 0 | — | |
| triage_local_x3_nemotron_35_bolt_e4b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.600 | 1 | — | 22252 ms | 1 | — | |
| triage_local_x3_nemotron_35_bolt_e4b_v1 · repeats | run again (repeats) 0.600 → 0.600 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.600 | 1 | — | 21266 ms | 1 | — | |
| triage_local_x3_nemotron_35_bolt_e4b_v1 · rescored 5f2c057c, repeats | rescored against gold/orrery_triage_v1_labelled 0.600 → 0.600 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.600 | 1 | — | 21266 ms | 0 | — | |
| triage_local_x3_nemotron_35_bolt_e4b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.600 → 0.600 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.600 | 1 | — | 22252 ms | 0 | — | |
| triage_local_x3_nemotron_35_bolt_e4b_v1 · rescored 77adc97b, repeats | rescored against gold/orrery_triage_v1_labelled 0.600 → 0.600 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.600 | 1 | — | 21266 ms | 0 | — | |
| triage_local_x3_nemotron_35_bolt_e4b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.600 → 0.600 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.600 | 1 | — | 22252 ms | 0 | — | |
| triage_local_x3_nemotron_35_bolt_e4b_v1 · rescored cd35dc1f, repeats | rescored against gold/orrery_triage_v1_labelled 0.600 → 0.600 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.600 | 1 | — | 21266 ms | 0 | — | |
| triage_local_x3_nemotron_35_bolt_e4b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.600 → 0.600 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M gateway | triage_cost 0.600 | 1 | — | 22252 ms | 0 | — | |
| triage_local_x3_nemotron_35_bolt_e4b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M @ x3 smoke: check gateway | triage_cost 0.554 | 1 | — | 5646 ms | 1 | — | |
| triage_local_x3_nemotron_35_bolt_e4b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.554 → 0.554 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M @ x3 smoke: check gateway | triage_cost 0.554 | 1 | — | 5646 ms | 0 | — | |
| triage_local_x3_nemotron_35_bolt_e4b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.554 → 0.554 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-bolt:e4b-q4_K_M @ x3 smoke: check gateway | triage_cost 0.554 | 1 | — | 5646 ms | 0 | — | |
| triage_local_x3_nemotron_35_lightning_30b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b gateway | triage_cost 0.523 | 1 | — | 39458 ms | 1 | — | |
| triage_local_x3_nemotron_35_lightning_30b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.523 → 0.523 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b gateway | triage_cost 0.523 | 1 | — | 39458 ms | 0 | — | |
| triage_local_x3_nemotron_35_lightning_30b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.523 → 0.523 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b gateway | triage_cost 0.523 | 1 | — | 39458 ms | 0 | — | |
| triage_local_x3_nemotron_35_lightning_30b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.523 → 0.523 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b gateway | triage_cost 0.523 | 1 | — | 39458 ms | 0 | — | |
| triage_local_x3_nemotron_35_lightning_30b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b @ x3 smoke: check gateway | triage_cost 0.677 | 1 | — | 7797 ms | 1 | — | |
| triage_local_x3_nemotron_35_lightning_30b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b @ x3 smoke: check gateway | triage_cost 0.677 | 1 | — | 7797 ms | 0 | — | |
| triage_local_x3_nemotron_35_lightning_30b_v2 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.677 → 0.677 (+0.000) vs measured | orrery_triage_v1_labelled | nemotron-3.5-lightning:30b @ x3 smoke: check gateway | triage_cost 0.677 | 1 | — | 7797 ms | 0 | — | |
| triage_local_x3_phi4_14b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.769 | 1 | — | 16911 ms | 1 | — | |
| triage_local_x3_phi4_14b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.769 → 0.769 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.769 | 1 | — | 16911 ms | 0 | — | |
| triage_local_x3_phi4_14b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.769 → 0.769 (+0.000) vs measured | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.769 | 1 | — | 16911 ms | 0 | — | |
| triage_local_x3_phi4_14b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.769 → 0.769 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.769 | 1 | — | 16911 ms | 0 | — | |
| triage_local_x3_phi4_14b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.769 → 0.769 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | phi4:14b gateway | triage_cost 0.769 | 1 | — | 16911 ms | 0 | — | |
| triage_local_x3_qwen38_27b_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b gateway | triage_cost 0.908 | 1 | — | 46161 ms | 1 | — | |
| triage_local_x3_qwen38_27b_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.908 → 0.908 (+0.000) vs measured | orrery_triage_v1_labelled | qwen3.8:27b gateway | triage_cost 0.908 | 1 | — | 46161 ms | 0 | — | |
| triage_local_x3_qwen38_27b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.908 → 0.908 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b gateway | triage_cost 0.908 | 1 | — | 46161 ms | 0 | — | |
| triage_local_x3_qwen38_27b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.908 → 0.908 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b gateway | triage_cost 0.908 | 1 | — | 46161 ms | 0 | — | |
| triage_local_x3_qwen38_27b_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b @ x3 smoke: check gateway | triage_cost 1.292 | 1 | — | 20630 ms | 1 | — | |
| triage_local_x3_qwen38_27b_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 1.292 → 1.292 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3.8:27b @ x3 smoke: check gateway | triage_cost 1.292 | 1 | — | 20630 ms | 0 | — | |
| triage_local_x3_qwen38_27b_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 1.292 → 1.292 (+0.000) vs measured | orrery_triage_v1_labelled | qwen3.8:27b @ x3 smoke: check gateway | triage_cost 1.292 | 1 | — | 20630 ms | 0 | — | |
| triage_local_x3_qwen3_30b_a3b_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.723 | 1 | — | 6932 ms | 1 | — | |
| triage_local_x3_qwen3_30b_a3b_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.723 → 0.723 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.723 | 1 | — | 6932 ms | 0 | — | |
| triage_local_x3_qwen3_30b_a3b_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.723 → 0.723 (+0.000) vs measured | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.723 | 1 | — | 6932 ms | 0 | — | |
| triage_local_x3_qwen3_30b_a3b_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.723 → 0.723 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.723 | 1 | — | 6932 ms | 0 | — | |
| triage_local_x3_qwen3_30b_a3b_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.723 → 0.723 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | qwen3:30b-a3b-instruct-2507-q4_K_M gateway | triage_cost 0.723 | 1 | — | 6932 ms | 0 | — | |
| triage_mistral_large_4_v1 retired in r3 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-mistral-large-4 gateway | false_dismiss 0.176 | 1 | $0.0069 | 17290 ms | 1 | — | |
| triage_mistral_large_4_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-mistral-large-4 gateway | triage_cost 0.800 | 1 | $0.0069 | 17290 ms | 0 | — | |
| triage_mistral_large_4_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-mistral-large-4 gateway | triage_cost 0.800 | 1 | $0.0069 | 17290 ms | 0 | — | |
| triage_mistral_large_4_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-mistral-large-4 gateway | triage_cost 0.800 | 1 | $0.0069 | 17290 ms | 0 | — | |
| triage_mistral_large_4_v2 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-mistral-large-4 @ hosted smoked gateway | triage_cost 0.800 | 1 | $0.0069 | 20489 ms | 1 | — | |
| triage_mistral_large_4_v2 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.800 → 0.800 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | eval-mistral-large-4 @ hosted smoked gateway | triage_cost 0.800 | 1 | $0.0069 | 20489 ms | 0 | — | |
| triage_mistral_large_4_v2 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.800 → 0.800 (+0.000) vs measured | orrery_triage_v1_labelled | eval-mistral-large-4 @ hosted smoked gateway | triage_cost 0.800 | 1 | $0.0069 | 20489 ms | 0 | — | |
| triage_sonnet_55_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | eval-claude-sonnet-55 gateway | false_dismiss 0.147 | 1 | $0.0227 | 12035 ms | 1 | — | |
| triage_sonnet_55_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-claude-sonnet-55 gateway | triage_cost 0.708 | 1 | $0.0227 | 12035 ms | 0 | — | |
| triage_sonnet_55_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled | orrery_triage_v1_labelled | eval-claude-sonnet-55 gateway | triage_cost 0.708 | 1 | $0.0227 | 12035 ms | 0 | — | |
| triage_sonnet_55_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-claude-sonnet-55 gateway | triage_cost 0.708 | 1 | $0.0227 | 12035 ms | 0 | — | |
| triage_sonnet_55_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled the goals read its re-score | orrery_triage_v1_labelled | eval-claude-sonnet-55 gateway | triage_cost 0.708 | 1 | $0.0227 | 12035 ms | 0 | — | |
| triage_stub_dismiss_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | — stub · local | triage_cost 0.862 | 1 | — | 0 ms | 1 | — | |
| triage_stub_dismiss_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.862 → 0.862 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | — stub · local | triage_cost 0.862 | 1 | — | 0 ms | 0 | — | |
| triage_stub_dismiss_v1 · rescored 5f2c057c | rescored against gold/orrery_triage_v1_labelled 0.862 → 0.862 (+0.000) vs measured | orrery_triage_v1_labelled | — stub · local | triage_cost 0.862 | 1 | — | 0 ms | 0 | — | |
| triage_stub_dismiss_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.862 → 0.862 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | — stub · local | triage_cost 0.862 | 1 | — | 0 ms | 0 | — | |
| triage_stub_dismiss_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.862 → 0.862 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | — stub · local | triage_cost 0.862 | 1 | — | 0 ms | 0 | — | |
| triage_stub_escalate_v1 | measured the goals read its re-score | orrery_triage_v1_labelled | — stub · local | triage_cost 0.477 | 1 | — | 0 ms | 1 | — | |
| triage_stub_escalate_v1 · rescored 3d3a6afd | rescored against gold/orrery_triage_v1_labelled 0.477 → 0.477 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | — stub · local | triage_cost 0.477 | 1 | — | 0 ms | 0 | — | |
| triage_stub_escalate_v1 · rescored 5f2c057c not eligible | rescored against gold/orrery_triage_v1_labelled 0.477 → 0.477 (+0.000) vs measured | orrery_triage_v1_labelled | — stub · local | triage_cost 0.477 | 1 | — | 0 ms | 0 | — | |
| triage_stub_escalate_v1 · rescored 77adc97b | rescored against gold/orrery_triage_v1_labelled 0.477 → 0.477 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | — stub · local | triage_cost 0.477 | 1 | — | 0 ms | 0 | — | |
| triage_stub_escalate_v1 · rescored cd35dc1f | rescored against gold/orrery_triage_v1_labelled 0.477 → 0.477 (+0.000) vs measured the goals read its re-score | orrery_triage_v1_labelled | — stub · local | triage_cost 0.477 | 1 | — | 0 ms | 0 | — | |
| sf_bge_small_n200_v1 | re-parsed | scifact_200 | BAAI/bge-small-en-v1.5 dense · local | ndcg_10 0.710 | 1 | — | 29 ms | 1 | — | |
| sf_bm25_n200_v1 | re-parsed | scifact_200 | — bm25 · local | ndcg_10 0.645 | 1 | — | 22 ms | 1 | — | |
| sf_deepseek_m_n200_v1 | re-parsed | scifact_200 | eval-deepseek-41flash rerank | ndcg_10 0.740 | 1 | $0.0012 | 1145 ms | 1 | — | |
| sf_deepseek_s_n200_v1 | re-parsed | scifact_200 | eval-deepseek-flash rerank | ndcg_10 0.738 | 1 | $0.0003 | 3453 ms | 1 | — | |
| sf_e5_base_n200_v1 baselinerecommended | re-parsed | scifact_200 | intfloat/e5-base-v2 dense · local | ndcg_10 0.719 | 1 | — | 59 ms | 1 | — | |
| sf_first_k_n200_v1 | re-parsed | scifact_200 | — first_k · local | ndcg_10 0.000 | 1 | — | 0 ms | 1 | — | |
| sf_gemma_l_n200_v1 | re-parsed | scifact_200 | eval-gemma-31b rerank | ndcg_10 0.736 | 1 | $0.0009 | 9701 ms | 1 | — | |
| sf_gemma_m_n200_v1 | re-parsed | scifact_200 | eval-gemma-26b rerank | ndcg_10 0.736 | 1 | $0.0004 | 5664 ms | 1 | — | |
| sf_gemma_s_n200_v1 | re-parsed | scifact_200 | eval-gemma-3-27b rerank | ndcg_10 0.702 | 1 | $0.0005 | 7071 ms | 1 | — | |
| sf_glm_m_n200_v1 | re-parsed | scifact_200 | eval-glm-46 rerank | ndcg_10 0.742 | 1 | $0.0023 | 6920 ms | 1 | — | |
| sf_glm_s_bm25first_n200_v1 · pair retired in r2 | re-parsed | scifact_200 | eval-glm-45-air rerank | ndcg_10 0.742 | 1 | $0.0007 | 3024 ms | 1 | — | |
| sf_glm_s_e5first_n200_v1 · pair retired in r2 | re-parsed | scifact_200 | eval-glm-45-air rerank | ndcg_10 0.789 | 1 | $0.0007 | 3127 ms | 1 | — | |
| sf_glm_s_n200_v1 leader | re-parsed | scifact_200 | eval-glm-45-air rerank | ndcg_10 0.744 | 1 | $0.0007 | 3120 ms | 1 | — | |
| sf_llama_l_n200_v1 | re-parsed | scifact_200 | eval-llama-maverick rerank | ndcg_10 0.728 | 1 | $0.0009 | 4554 ms | 1 | — | |
| sf_llama_l_n200_v1 · repeats retired in r3 | re-parsed | scifact_200 | eval-llama-maverick rerank | ndcg_10 0.715 | 1 | $0.0008 | 11386 ms | 1 | — | |
| sf_llama_m_n200_v1 | re-parsed | scifact_200 | eval-llama-70b rerank | ndcg_10 0.709 | 1 | $0.0007 | 4064 ms | 1 | — | |
| sf_llama_s_n200_v1 | re-parsed | scifact_200 | eval-llama-scout rerank | ndcg_10 0.717 | 1 | $0.0007 | 17482 ms | 1 | — | |
| sf_llama_s_n200_v1 · repeats retired in r3 | re-parsed | scifact_200 | eval-llama-scout rerank | ndcg_10 0.713 | 1 | $0.0005 | 4874 ms | 1 | — | |
| sf_minilm_n200_v1 | re-parsed | scifact_200 | sentence-transformers/all-MiniLM-L6-v2 dense · local | ndcg_10 0.653 | 1 | — | 17 ms | 1 | — | |
| sf_mistral_s_n200_v1 | re-parsed | scifact_200 | eval-mistral-small rerank | ndcg_10 0.707 | 1 | $0.0004 | 7224 ms | 1 | — | |
| sf_mpnet_n200_v1 | re-parsed | scifact_200 | sentence-transformers/all-mpnet-base-v2 dense · local | ndcg_10 0.664 | 1 | — | 65 ms | 1 | — | |
| sf_qwen_s_bm25first_n200_v1 · pair retired in r2 | re-parsed | scifact_200 | eval-qwen-flash rerank | ndcg_10 0.714 | 1 | $0.0008 | 6196 ms | 1 | — | |
| sf_qwen_s_e5first_n200_v1 · pair retired in r2 | re-parsed | scifact_200 | eval-qwen-flash rerank | ndcg_10 0.771 | 1 | $0.0008 | 6145 ms | 1 | — | |
| sf_qwen_s_n200_v1 | re-parsed | scifact_200 | eval-qwen-flash rerank | ndcg_10 0.718 | 1 | $0.0008 | 5651 ms | 1 | — | |
| sf_qwen_s_n200_v1 · repeats retired in r3 | re-parsed | scifact_200 | eval-qwen-flash rerank | ndcg_10 0.719 | 1 | $0.0008 | 5976 ms | 1 | — | |
| sf_random_n200_v1 | re-parsed | scifact_200 | — random · local | ndcg_10 0.001 | 1 | — | 2 ms | 1 | — | |
| demo_v1 | measured | smoke_v1 | — echo · local | overlap_f1 0.416 | 1 | — | 0 ms | 1 | — | |
| se_anthropic_l_n100_v1 | measured | summeval_100 | eval-opus-5 litellm | tau 0.528 | 1 | $0.0201 | 3427 ms | 1 | — | |
| se_anthropic_m_n100_v1 | measured | summeval_100 | eval-claude-sonnet litellm | tau 0.480 | 1 | $0.0080 | 3008 ms | 1 | — | |
| se_anthropic_s_n100_v1 | measured | summeval_100 | eval-claude-haiku litellm | tau 0.484 | 1 | $0.0032 | 2433 ms | 1 | — | |
| se_bartscore_n100_v1 | measured | summeval_100 | facebook/bart-large-cnn bartscore · local | tau 0.256 | 1 | — | 15662 ms | 1 | — | |
| se_coverage_n100_v1 | measured | summeval_100 | — coverage · local | tau 0.101 | 1 | — | 892 ms | 1 | — | |
| se_deepseek_l_n100_v1 | measured | summeval_100 | eval-deepseek-pro litellm | tau 0.469 | 1 | $0.0003 | 5535 ms | 1 | — | |
| se_deepseek_m_n100_v1 | measured | summeval_100 | eval-deepseek-41flash litellm | tau 0.486 | 1 | $0.0008 | 1120 ms | 1 | — | |
| se_deepseek_s_n100_v1 | measured | summeval_100 | eval-deepseek-flash litellm | tau 0.459 | 1 | $0.0003 | 3985 ms | 1 | — | |
| se_gemma_l_n100_pairwise_v1 | measured | summeval_100 | eval-gemma-31b litellm | pair_agreement 0.825 | 1 | $0.0008 | 6981 ms | 1 | — | |
| se_gemma_l_n100_pick_v1 | measured | summeval_100 | eval-gemma-31b litellm | top1 0.364 | 1 | $0.0002 | 1813 ms | 1 | — | |
| se_gemma_l_n100_v1 baselinerecommended | measured | summeval_100 | eval-gemma-31b litellm | tau 0.538 | 1 | $0.0003 | 6670 ms | 1 | — | |
| se_gemma_l_pubtext_n100_v1 · pair retired in r2 | run again (pair) | summeval_100 | eval-gemma-31b litellm | tau 0.551 | 1 | $0.0010 | 2614 ms | 1 | — | |
| se_gemma_l_pubtextcw_n100_v1 · pair retired in r2 | run again (pair) | summeval_100 | eval-gemma-31b litellm | tau 0.540 | 1 | $0.0003 | 3268 ms | 1 | — | |
| se_gemma_m_n100_v1 | measured | summeval_100 | eval-gemma-26b litellm | tau 0.477 | 1 | $0.0003 | 4308 ms | 1 | — | |
| se_gemma_s_n100_v1 | measured | summeval_100 | eval-gemma-3-27b litellm | tau 0.420 | 1 | $0.0003 | 11572 ms | 1 | — | |
| se_glm_l_n100_v1 | measured | summeval_100 | eval-glm-5 litellm | tau 0.492 | 1 | $0.0025 | 3590 ms | 1 | — | |
| se_glm_m_n100_v1 | measured | summeval_100 | eval-glm-46 litellm | tau 0.517 | 1 | $0.0011 | 3609 ms | 1 | — | |
| se_glm_s_n100_v1 | measured | summeval_100 | eval-glm-45-air litellm | tau 0.446 | 1 | $0.0005 | 5617 ms | 1 | — | |
| se_length_n100_v1 | measured | summeval_100 | — length · local | tau 0.039 | 1 | — | 0 ms | 1 | — | |
| se_llama_l_n100_v1 | measured | summeval_100 | eval-llama-maverick litellm | tau 0.454 | 1 | $0.0006 | 12346 ms | 1 | — | |
| se_llama_m_n100_v1 | measured | summeval_100 | eval-llama-70b litellm | tau 0.406 | 1 | $0.0005 | 4834 ms | 1 | — | |
| se_llama_s_n100_v1 | measured | summeval_100 | eval-llama-scout litellm | tau 0.353 | 1 | $0.0003 | 7141 ms | 1 | — | |
| se_mistral_l_n100_v1 | measured | summeval_100 | eval-mistral-large litellm | tau 0.455 | 1 | $0.0013 | 10885 ms | 1 | — | |
| se_mistral_m_n100_v1 | measured | summeval_100 | eval-mistral-medium litellm | tau 0.468 | 1 | $0.0012 | 1946 ms | 1 | — | |
| se_mistral_s_n100_v1 | measured | summeval_100 | eval-mistral-small litellm | tau 0.427 | 1 | $0.0003 | 9185 ms | 1 | — | |
| se_openai_l_n100_v1 leader | measured | summeval_100 | eval-gpt-6sol litellm | tau 0.540 | 1 | $0.0063 | 2228 ms | 1 | — | |
| se_openai_m_n100_v1 | measured | summeval_100 | eval-gpt-55 litellm | tau 0.528 | 1 | $0.0142 | 2690 ms | 1 | — | |
| se_openai_s_n100_v1 | measured | summeval_100 | eval-gpt-mini litellm | tau 0.411 | 1 | $0.0021 | 1551 ms | 1 | — | |
| se_qwen37max_n100_v1 | measured | summeval_100 | eval-qwen-37max litellm | tau 0.520 | 1 | $0.0044 | 4514 ms | 1 | — | |
| se_qwen_m_n100_v1 | measured | summeval_100 | eval-qwen-37plus litellm | tau 0.507 | 1 | $0.0011 | 6770 ms | 1 | — | |
| se_qwen_s_n100_pairwise_v1 | measured | summeval_100 | eval-qwen-flash litellm | pair_agreement 0.812 | 1 | $0.0005 | 20081 ms | 1 | — | |
| se_qwen_s_n100_pick_v1 | measured | summeval_100 | eval-qwen-flash litellm | top1 0.263 | 1 | $0.0003 | 1492 ms | 1 | — | |
| se_qwen_s_n100_v1 | measured | summeval_100 | eval-qwen-flash litellm | tau 0.519 | 1 | $0.0004 | 3954 ms | 1 | — | |
| se_random_n100_pairwise_v1 | measured | summeval_100 | — random · local | pair_agreement 0.473 | 1 | — | 0 ms | 1 | — | |
| se_random_n100_pick_v1 | measured | summeval_100 | — random · local | top1 0.070 | 1 | — | 0 ms | 1 | — | |
| se_random_n100_v1 | measured | summeval_100 | — random · local | tau 0.003 | 1 | — | 0 ms | 1 | — | |
| se_rouge1_n100_v1 | measured | summeval_100 | — rouge · local | tau 0.149 | 1 | — | 196 ms | 1 | — | |
| se_rouge2_n100_v1 | measured | summeval_100 | — rouge · local | tau 0.158 | 1 | — | 181 ms | 1 | — | |
| se_rougeL_n100_v1 | measured | summeval_100 | — rouge · local | tau 0.143 | 1 | — | 274 ms | 1 | — | |
| se_unieval_n100_v1 | measured | summeval_100 | MingZhong/unieval-sum unieval · local | tau 0.348 | 1 | — | 41676 ms | 1 | — | |
| se_unievalpub_n100_v1 | measured | summeval_100 | MingZhong/unieval-sum unieval · local | tau 0.372 | 1 | — | 44694 ms | 1 | — |
02configs
One row per config, across every dataset it ran on: where it ranks on each goal is its best arm among the arms the goal's decision weighs.
267 shown
| config | model · provider | datasets | arms | spend | rank on each goal | runs in period | spend in period |
|---|---|---|---|---|---|---|---|
ag_anthropic_l_n200_v1 | eval-opus-5litellm | ag_news_200 | 2 | $0.1750 | News topic classification (AG News) #5 of 30 | 1 | — |
ag_anthropic_m_n200_v1 | eval-claude-sonnetlitellm | ag_news_200 | 2 | $0.0703 | News topic classification (AG News) #4 of 30 | 1 | — |
ag_anthropic_s_n200_v1 | eval-claude-haikulitellm | ag_news_200 | 2 | $0.0254 | News topic classification (AG News) #24 of 30 | 1 | — |
ag_bart_mnli_n200_v1 | facebook/bart-large-mnlihf_zeroshot · local | ag_news_200 | 2 | — | News topic classification (AG News) #28 of 30 | 1 | — |
ag_bert_base_fy_n200_v1 | fabriceyhc/bert-base-uncased-ag_newshf_local · local | ag_news_200 | 3 | — | News topic classification (AG News) #2 of 30 | 2 | — |
ag_bert_base_ta_n200_v1 | textattack/bert-base-uncased-ag-newshf_local · local | ag_news_200 | 3 | — | News topic classification (AG News) #1 of 30 | 2 | — |
ag_bert_mini_n200_v1 | mrm8488/bert-mini-finetuned-age_news-classificationhf_local · local | ag_news_200 | 3 | — | News topic classification (AG News) #3 of 30 | 2 | — |
ag_constant_n200_v1 | — constant · local | ag_news_200 | 2 | — | News topic classification (AG News) #30 of 30 | 1 | — |
ag_deepseek_l_n200_v1 | eval-deepseek-prolitellm | ag_news_200 | 2 | $0.0114 | News topic classification (AG News) #13 of 30 | 1 | — |
ag_deepseek_m_n200_v1 | eval-deepseek-41flashlitellm | ag_news_200 | 2 | $0.0043 | News topic classification (AG News) #23 of 30 | 1 | — |
ag_deepseek_s_n200_v1 | eval-deepseek-flashlitellm | ag_news_200 | 2 | $0.0014 | News topic classification (AG News) #25 of 30 | 1 | — |
ag_gemma_l_n200_v1 | eval-gemma-31blitellm | ag_news_200 | 2 | $0.0028 | News topic classification (AG News) #14 of 30 | 1 | — |
ag_gemma_m_n200_v1 | eval-gemma-26blitellm | ag_news_200 | 2 | $0.0016 | News topic classification (AG News) #21 of 30 | 1 | — |
ag_gemma_s_n200_v1 | eval-gemma-3-27blitellm | ag_news_200 | 2 | $0.0021 | News topic classification (AG News) #10 of 30 | 1 | — |
ag_glm_l_n200_v1 | eval-glm-5litellm | ag_news_200 | 2 | $0.0186 | News topic classification (AG News) #16 of 30 | 1 | — |
ag_glm_m_n200_v1 | eval-glm-46litellm | ag_news_200 | 2 | $0.0107 | News topic classification (AG News) #17 of 30 | 1 | — |
ag_glm_s_n200_v1 | eval-glm-45-airlitellm | ag_news_200 | 2 | $0.0027 | News topic classification (AG News) #22 of 30 | 1 | — |
ag_keyword_n200_v1 | — keyword · local | ag_news_200 | 2 | — | News topic classification (AG News) #29 of 30 | 1 | — |
ag_llama_l_n200_v1 | eval-llama-mavericklitellm | ag_news_200 | 2 | $0.0043 | News topic classification (AG News) #26 of 30 | 1 | — |
ag_llama_m_n200_v1 | eval-llama-70blitellm | ag_news_200 | 2 | $0.0040 | News topic classification (AG News) #9 of 30 | 1 | — |
ag_llama_s_n200_v1 | eval-llama-scoutlitellm | ag_news_200 | 2 | $0.0025 | News topic classification (AG News) #27 of 30 | 1 | — |
ag_mistral_l_n200_v1 | eval-mistral-largelitellm | ag_news_200 | 2 | $0.0103 | News topic classification (AG News) #11 of 30 | 1 | — |
ag_mistral_m_n200_v1 | eval-mistral-mediumlitellm | ag_news_200 | 2 | $0.0096 | News topic classification (AG News) #18 of 30 | 1 | — |
ag_mistral_s_n200_v1 | eval-mistral-smalllitellm | ag_news_200 | 2 | $0.0019 | News topic classification (AG News) #19 of 30 | 1 | — |
ag_openai_l_n200_v1 | eval-gpt-6sollitellm | ag_news_200 | 2 | $0.0487 | News topic classification (AG News) #7 of 30 | 1 | — |
ag_openai_m_n200_v1 | eval-gpt-55litellm | ag_news_200 | 2 | $0.1272 | News topic classification (AG News) #6 of 30 | 1 | — |
ag_openai_s_n200_v1 | eval-gpt-minilitellm | ag_news_200 | 2 | $0.0190 | News topic classification (AG News) #20 of 30 | 1 | — |
ag_qwen_l_n200_v1 | eval-qwen-3maxlitellm | ag_news_200 | 2 | $0.0164 | News topic classification (AG News) #15 of 30 | 1 | — |
ag_qwen_m_n200_v1 | eval-qwen-37pluslitellm | ag_news_200 | 2 | $0.0070 | News topic classification (AG News) #8 of 30 | 1 | — |
ag_qwen_s_n200_v1 | eval-qwen-flashlitellm | ag_news_200 | 2 | $0.0036 | News topic classification (AG News) #12 of 30 | 1 | — |
cnn_anthropic_l_n200_v1 | eval-opus-5litellm | cnn_dailymail_200 | 5 | $1.97 | Short news summaries (CNN) #18 of 29 | 1 | — |
cnn_anthropic_m_n200_v1 | eval-claude-sonnetlitellm | cnn_dailymail_200 | 5 | $0.7478 | Short news summaries (CNN) #15 of 29 | 1 | — |
cnn_anthropic_s_n200_v1 | eval-claude-haikulitellm | cnn_dailymail_200 | 5 | $0.2611 | Short news summaries (CNN) #14 of 29 | 1 | — |
cnn_bart_l_dm200_v1 | facebook/bart-large-cnnhf_local · local | cnn_dailymail_dm200 | 1 | — | — | 1 | — |
cnn_bart_l_n200_v1 | facebook/bart-large-cnnhf_local · local | cnn_dailymail_200 | 7 | — | Short news summaries (CNN) #1 of 29 | 2 | — |
cnn_bart_l_v1 | facebook/bart-large-cnnhf_local · local | cnn_dailymail_20 | 1 | — | — | 1 | — |
cnn_bart_l_xsum_dm200_v1 | facebook/bart-large-xsumhf_local · local | cnn_dailymail_dm200 | 1 | — | — | 1 | — |
cnn_bart_l_xsum_n200_v1 | facebook/bart-large-xsumhf_local · local | cnn_dailymail_200 | 6 | — | Short news summaries (CNN) #29 of 29 | 2 | — |
cnn_bart_m_dm200_v1 | sshleifer/distilbart-cnn-12-6hf_local · local | cnn_dailymail_dm200 | 1 | — | — | 1 | — |
cnn_bart_m_n200_v1 | sshleifer/distilbart-cnn-12-6hf_local · local | cnn_dailymail_200 | 6 | — | Short news summaries (CNN) #5 of 29 | 2 | — |
cnn_bart_s_dm200_v1 | sshleifer/distilbart-cnn-6-6hf_local · local | cnn_dailymail_dm200 | 1 | — | — | 1 | — |
cnn_bart_s_n200_v1 | sshleifer/distilbart-cnn-6-6hf_local · local | cnn_dailymail_200 | 7 | — | Short news summaries (CNN) #10 of 29 | 3 | — |
cnn_deepseek_l_n200_v1 | eval-deepseek-prolitellm | cnn_dailymail_200 | 5 | $0.1389 | Short news summaries (CNN) #7 of 29 | 1 | — |
cnn_deepseek_m_n200_v1 | eval-deepseek-41flashlitellm | cnn_dailymail_200 | 6 | $0.0381 | Short news summaries (CNN) #2 of 29 | 1 | — |
cnn_deepseek_s_n200_v1 | eval-deepseek-flashlitellm | cnn_dailymail_200 | 5 | $0.0137 | Short news summaries (CNN) #9 of 29 | 1 | — |
cnn_gemma_l_n200_v1 | eval-gemma-31blitellm | cnn_dailymail_200 | 5 | $0.0237 | Short news summaries (CNN) #16 of 29 | 1 | — |
cnn_gemma_m_n200_v1 | eval-gemma-26blitellm | cnn_dailymail_200 | 5 | $0.0154 | Short news summaries (CNN) #20 of 29 | 1 | — |
cnn_gemma_s_n200_v1 | eval-gemma-3-27blitellm | cnn_dailymail_200 | 5 | $0.0200 | Short news summaries (CNN) #22 of 29 | 1 | — |
cnn_glm_l_n200_v1 | eval-glm-5litellm | cnn_dailymail_200 | 5 | $0.1532 | Short news summaries (CNN) #21 of 29 | 1 | — |
cnn_glm_m_n200_v1 | eval-glm-46litellm | cnn_dailymail_200 | 5 | $0.0995 | Short news summaries (CNN) #8 of 29 | 1 | — |
cnn_glm_s_n200_v1 | eval-glm-45-airlitellm | cnn_dailymail_200 | 5 | $0.0304 | Short news summaries (CNN) #19 of 29 | 1 | — |
cnn_lead3_dm200_v1 | — lead_k · local | cnn_dailymail_dm200 | 1 | — | — | 1 | — |
cnn_lead3_n200_v1 | — lead_k · local | cnn_dailymail_200 | 6 | — | Short news summaries (CNN) #28 of 29 | 2 | — |
cnn_lead3_v1 | — lead_k · local | cnn_dailymail_20 | 1 | — | — | 1 | — |
cnn_llama_l_n200_v1 | eval-llama-mavericklitellm | cnn_dailymail_200 | 5 | $0.0389 | Short news summaries (CNN) #3 of 29 | 1 | — |
cnn_llama_m_n200_v1 | eval-llama-70blitellm | cnn_dailymail_200 | 5 | $0.0330 | Short news summaries (CNN) #4 of 29 | 1 | — |
cnn_llama_s_n200_v1 | eval-llama-scoutlitellm | cnn_dailymail_200 | 5 | $0.0222 | Short news summaries (CNN) #6 of 29 | 1 | — |
cnn_mistral_l_n200_v1 | eval-mistral-largelitellm | cnn_dailymail_200 | 5 | $0.0973 | Short news summaries (CNN) #24 of 29 | 1 | — |
cnn_mistral_m_n200_v1 | eval-mistral-mediumlitellm | cnn_dailymail_200 | 5 | $0.0871 | Short news summaries (CNN) #26 of 29 | 1 | — |
cnn_mistral_s_n200_v1 | eval-mistral-smalllitellm | cnn_dailymail_200 | 5 | $0.0179 | Short news summaries (CNN) #13 of 29 | 1 | — |
cnn_openai_l_n200_v1 | eval-gpt-6sollitellm | cnn_dailymail_200 | 5 | $0.4362 | Short news summaries (CNN) #12 of 29 | 1 | — |
cnn_openai_m_n200_v1 | eval-gpt-55litellm | cnn_dailymail_200 | 5 | $1.07 | Short news summaries (CNN) #11 of 29 | 1 | — |
cnn_openai_s_n200_v1 | eval-gpt-minilitellm | cnn_dailymail_200 | 5 | $0.1672 | Short news summaries (CNN) #17 of 29 | 1 | — |
cnn_qwen_l_n200_v1 | eval-qwen-3maxlitellm | cnn_dailymail_200 | 5 | $0.1689 | Short news summaries (CNN) #23 of 29 | 1 | — |
cnn_qwen_m_n200_v1 | eval-qwen-37pluslitellm | cnn_dailymail_200 | 5 | $0.0661 | Short news summaries (CNN) #27 of 29 | 1 | — |
cnn_qwen_s_n200_v1 | eval-qwen-flashlitellm | cnn_dailymail_200 | 5 | $0.0270 | Short news summaries (CNN) #25 of 29 | 1 | — |
db_anthropic_l_n200_v1 | eval-opus-5litellm | dbpedia_280 | 1 | $0.3784 | Encyclopedia topic classification (DBpedia) #2 of 28 | 1 | — |
db_anthropic_m_n200_v1 | eval-claude-sonnetlitellm | dbpedia_280 | 1 | $0.1514 | Encyclopedia topic classification (DBpedia) #8 of 28 | 1 | — |
db_anthropic_s_n200_v1 | eval-claude-haikulitellm | dbpedia_280 | 1 | $0.0536 | Encyclopedia topic classification (DBpedia) #10 of 28 | 1 | — |
db_bart_mnli_n200_v1 | facebook/bart-large-mnlihf_zeroshot · local | dbpedia_280 | 1 | — | Encyclopedia topic classification (DBpedia) #27 of 28 | 1 | — |
db_bert_base_fy_n200_v1 | fabriceyhc/bert-base-uncased-dbpedia_14hf_local · local | dbpedia_280 | 2 | — | Encyclopedia topic classification (DBpedia) #3 of 28 | 2 | — |
db_constant_n200_v1 | — constant · local | dbpedia_280 | 1 | — | Encyclopedia topic classification (DBpedia) #28 of 28 | 1 | — |
db_deepseek_l_n200_v1 | eval-deepseek-prolitellm | dbpedia_280 | 1 | $0.0209 | Encyclopedia topic classification (DBpedia) #4 of 28 | 1 | — |
db_deepseek_m_n200_v1 | eval-deepseek-41flashlitellm | dbpedia_280 | 1 | $0.0079 | Encyclopedia topic classification (DBpedia) #12 of 28 | 1 | — |
db_deepseek_s_n200_v1 | eval-deepseek-flashlitellm | dbpedia_280 | 1 | $0.0028 | Encyclopedia topic classification (DBpedia) #24 of 28 | 1 | — |
db_gemma_l_n200_v1 | eval-gemma-31blitellm | dbpedia_280 | 1 | $0.0046 | Encyclopedia topic classification (DBpedia) #13 of 28 | 1 | — |
db_gemma_m_n200_v1 | eval-gemma-26blitellm | dbpedia_280 | 1 | $0.0033 | Encyclopedia topic classification (DBpedia) #5 of 28 | 1 | — |
db_gemma_s_n200_v1 | eval-gemma-3-27blitellm | dbpedia_280 | 1 | $0.0035 | Encyclopedia topic classification (DBpedia) #14 of 28 | 1 | — |
db_glm_l_n200_v1 | eval-glm-5litellm | dbpedia_280 | 1 | $0.0359 | Encyclopedia topic classification (DBpedia) #18 of 28 | 1 | — |
db_glm_m_n200_v1 | eval-glm-46litellm | dbpedia_280 | 1 | $0.0159 | Encyclopedia topic classification (DBpedia) #6 of 28 | 1 | — |
db_glm_s_n200_v1 | eval-glm-45-airlitellm | dbpedia_280 | 1 | $0.0052 | Encyclopedia topic classification (DBpedia) #25 of 28 | 1 | — |
db_keyword_n200_v1 | — keyword · local | dbpedia_280 | 1 | — | Encyclopedia topic classification (DBpedia) #26 of 28 | 1 | — |
db_llama_l_n200_v1 | eval-llama-mavericklitellm | dbpedia_280 | 1 | $0.0089 | Encyclopedia topic classification (DBpedia) #15 of 28 | 1 | — |
db_llama_m_n200_v1 | eval-llama-70blitellm | dbpedia_280 | 1 | $0.0084 | Encyclopedia topic classification (DBpedia) #11 of 28 | 1 | — |
db_llama_s_n200_v1 | eval-llama-scoutlitellm | dbpedia_280 | 1 | $0.0076 | Encyclopedia topic classification (DBpedia) #21 of 28 | 1 | — |
db_mistral_l_n200_v1 | eval-mistral-largelitellm | dbpedia_280 | 1 | $0.0210 | Encyclopedia topic classification (DBpedia) #20 of 28 | 1 | — |
db_mistral_m_n200_v1 | eval-mistral-mediumlitellm | dbpedia_280 | 1 | $0.0188 | Encyclopedia topic classification (DBpedia) #22 of 28 | 1 | — |
db_mistral_s_n200_v1 | eval-mistral-smalllitellm | dbpedia_280 | 1 | $0.0039 | Encyclopedia topic classification (DBpedia) #23 of 28 | 1 | — |
db_openai_l_n200_v1 | eval-gpt-6sollitellm | dbpedia_280 | 1 | $0.0921 | Encyclopedia topic classification (DBpedia) #16 of 28 | 1 | — |
db_openai_m_n200_v1 | eval-gpt-55litellm | dbpedia_280 | 1 | $0.2380 | Encyclopedia topic classification (DBpedia) #9 of 28 | 1 | — |
db_openai_s_n200_v1 | eval-gpt-minilitellm | dbpedia_280 | 1 | $0.0360 | Encyclopedia topic classification (DBpedia) #17 of 28 | 1 | — |
db_qwen_l_n200_v1 | eval-qwen-3maxlitellm | dbpedia_280 | 1 | $0.0342 | Encyclopedia topic classification (DBpedia) #19 of 28 | 1 | — |
db_qwen_m_n200_v1 | eval-qwen-37pluslitellm | dbpedia_280 | 1 | $0.0143 | Encyclopedia topic classification (DBpedia) #1 of 28 | 1 | — |
db_qwen_s_n200_v1 | eval-qwen-flashlitellm | dbpedia_280 | 1 | $0.0072 | Encyclopedia topic classification (DBpedia) #7 of 28 | 1 | — |
dedup_guard_v1 | — — · local | fleet_dedup_v1 | 1 | — | — | 1 | — |
dedup_pinned_f07efa9_v1 | — — · local | fleet_dedup_v1 | 1 | — | — | 1 | — |
demo_v1 | — echo · local | smoke_v1 | 1 | — | — | 1 | — |
fn_anthropic_l_n200_v1 | eval-opus-5litellm | few_nerd_280 | 1 | $0.6897 | Named-entity extraction (Few-NERD) #3 of 28 | 1 | — |
fn_anthropic_m_n200_v1 | eval-claude-sonnetlitellm | few_nerd_280 | 1 | $0.2846 | Named-entity extraction (Few-NERD) #6 of 28 | 1 | — |
fn_anthropic_s_n200_v1 | eval-claude-haikulitellm | few_nerd_280 | 1 | $0.1552 | Named-entity extraction (Few-NERD) #18 of 28 | 1 | — |
fn_capitalized_n200_v1 | — capitalized · local | few_nerd_280 | 1 | — | Named-entity extraction (Few-NERD) #27 of 28 | 1 | — |
fn_deepseek_l_n200_v1 | eval-deepseek-prolitellm | few_nerd_280 | 1 | $0.0853 | Named-entity extraction (Few-NERD) #17 of 28 | 1 | — |
fn_deepseek_m_n200_v1 | eval-deepseek-41flashlitellm | few_nerd_280 | 1 | $0.0200 | Named-entity extraction (Few-NERD) #11 of 28 | 1 | — |
fn_deepseek_s_n200_v1 | eval-deepseek-flashlitellm | few_nerd_280 | 1 | $0.0091 | Named-entity extraction (Few-NERD) #19 of 28 | 1 | — |
fn_gemma_l_n200_v1 | eval-gemma-31blitellm | few_nerd_280 | 1 | $0.0134 | Named-entity extraction (Few-NERD) #9 of 28 | 1 | — |
fn_gemma_m_n200_v1 | eval-gemma-26blitellm | few_nerd_280 | 1 | $0.0088 | Named-entity extraction (Few-NERD) #4 of 28 | 1 | — |
fn_gemma_s_n200_v1 | eval-gemma-3-27blitellm | few_nerd_280 | 1 | $0.0093 | Named-entity extraction (Few-NERD) #24 of 28 | 1 | — |
fn_gliner_n200_v1 | urchade/gliner_medium-v2.1gliner · local | few_nerd_280 | 1 | — | Named-entity extraction (Few-NERD) #26 of 28 | 1 | — |
fn_glm_l_n200_v1 | eval-glm-5litellm | few_nerd_280 | 1 | $0.1023 | Named-entity extraction (Few-NERD) #22 of 28 | 1 | — |
fn_glm_m_n200_v1 | eval-glm-46litellm | few_nerd_280 | 1 | $0.0445 | Named-entity extraction (Few-NERD) #15 of 28 | 1 | — |
fn_glm_s_n200_v1 | eval-glm-45-airlitellm | few_nerd_280 | 1 | $0.0194 | Named-entity extraction (Few-NERD) #23 of 28 | 1 | — |
fn_llama_l_n200_v1 | eval-llama-mavericklitellm | few_nerd_280 | 1 | $0.0176 | Named-entity extraction (Few-NERD) #12 of 28 | 1 | — |
fn_llama_m_n200_v1 | eval-llama-70blitellm | few_nerd_280 | 1 | $0.0194 | Named-entity extraction (Few-NERD) #20 of 28 | 1 | — |
fn_llama_s_n200_v1 | eval-llama-scoutlitellm | few_nerd_280 | 1 | $0.0103 | Named-entity extraction (Few-NERD) #21 of 28 | 1 | — |
fn_mistral_l_n200_v1 | eval-mistral-largelitellm | few_nerd_280 | 1 | $0.0419 | Named-entity extraction (Few-NERD) #13 of 28 | 1 | — |
fn_mistral_m_n200_v1 | eval-mistral-mediumlitellm | few_nerd_280 | 1 | $0.0462 | Named-entity extraction (Few-NERD) #16 of 28 | 1 | — |
fn_mistral_s_n200_v1 | eval-mistral-smalllitellm | few_nerd_280 | 1 | $0.0076 | Named-entity extraction (Few-NERD) #25 of 28 | 1 | — |
fn_nothing_n200_v1 | — nothing · local | few_nerd_280 | 1 | — | Named-entity extraction (Few-NERD) #28 of 28 | 1 | — |
fn_openai_l_n200_v1 | eval-gpt-6sollitellm | few_nerd_280 | 1 | $0.1715 | Named-entity extraction (Few-NERD) #5 of 28 | 1 | — |
fn_openai_m_n200_v1 | eval-gpt-55litellm | few_nerd_280 | 1 | $0.6507 | Named-entity extraction (Few-NERD) #2 of 28 | 1 | — |
fn_openai_s_n200_v1 | eval-gpt-minilitellm | few_nerd_280 | 1 | $0.0784 | Named-entity extraction (Few-NERD) #14 of 28 | 1 | — |
fn_qwen_l_n200_v1 | eval-qwen-3maxlitellm | few_nerd_280 | 1 | $0.0722 | Named-entity extraction (Few-NERD) #10 of 28 | 1 | — |
fn_qwen_m_n200_v1 | eval-qwen-37pluslitellm | few_nerd_280 | 1 | $0.0396 | Named-entity extraction (Few-NERD) #8 of 28 | 1 | — |
fn_qwen_s_n200_v1 | eval-qwen-flashlitellm | few_nerd_280 | 1 | $0.0157 | Named-entity extraction (Few-NERD) #7 of 28 | 1 | — |
fn_span_marker_n200_v1 | guishe/span-marker-generic-ner-v1-fewnerd-fine-superspan_marker · local | few_nerd_280 | 1 | — | Named-entity extraction (Few-NERD) #1 of 28 | 1 | — |
se_anthropic_l_n100_v1 | eval-opus-5litellm | summeval_100 | 1 | $2.01 | A scorer that agrees with experts on summaries (SummEval) #3 of 33 | 1 | — |
se_anthropic_m_n100_v1 | eval-claude-sonnetlitellm | summeval_100 | 1 | $0.8048 | A scorer that agrees with experts on summaries (SummEval) #12 of 33 | 1 | — |
se_anthropic_s_n100_v1 | eval-claude-haikulitellm | summeval_100 | 1 | $0.3237 | A scorer that agrees with experts on summaries (SummEval) #11 of 33 | 1 | — |
se_bartscore_n100_v1 | facebook/bart-large-cnnbartscore · local | summeval_100 | 1 | — | A scorer that agrees with experts on summaries (SummEval) #27 of 33 | 1 | — |
se_coverage_n100_v1 | — coverage · local | summeval_100 | 1 | — | A scorer that agrees with experts on summaries (SummEval) #31 of 33 | 1 | — |
se_deepseek_l_n100_v1 | eval-deepseek-prolitellm | summeval_100 | 1 | $0.0258 | A scorer that agrees with experts on summaries (SummEval) #14 of 33 | 1 | — |
se_deepseek_m_n100_v1 | eval-deepseek-41flashlitellm | summeval_100 | 1 | $0.0771 | A scorer that agrees with experts on summaries (SummEval) #10 of 33 | 1 | — |
se_deepseek_s_n100_v1 | eval-deepseek-flashlitellm | summeval_100 | 1 | $0.0307 | A scorer that agrees with experts on summaries (SummEval) #16 of 33 | 1 | — |
se_gemma_l_n100_pairwise_v1 | eval-gemma-31blitellm | summeval_100 | 1 | $0.0764 | — | 1 | — |
se_gemma_l_n100_pick_v1 | eval-gemma-31blitellm | summeval_100 | 1 | $0.0209 | — | 1 | — |
se_gemma_l_n100_v1 | eval-gemma-31blitellm | summeval_100 | 1 | $0.0278 | A scorer that agrees with experts on summaries (SummEval) #2 of 33 | 1 | — |
se_gemma_l_pubtext_n100_v1 | eval-gemma-31blitellm | summeval_100 | 1 | $0.1013 | — | 1 | — |
se_gemma_l_pubtextcw_n100_v1 | eval-gemma-31blitellm | summeval_100 | 1 | $0.0276 | — | 1 | — |
se_gemma_m_n100_v1 | eval-gemma-26blitellm | summeval_100 | 1 | $0.0252 | A scorer that agrees with experts on summaries (SummEval) #13 of 33 | 1 | — |
se_gemma_s_n100_v1 | eval-gemma-3-27blitellm | summeval_100 | 1 | $0.0279 | A scorer that agrees with experts on summaries (SummEval) #21 of 33 | 1 | — |
se_glm_l_n100_v1 | eval-glm-5litellm | summeval_100 | 1 | $0.2451 | A scorer that agrees with experts on summaries (SummEval) #9 of 33 | 1 | — |
se_glm_m_n100_v1 | eval-glm-46litellm | summeval_100 | 1 | $0.1144 | A scorer that agrees with experts on summaries (SummEval) #7 of 33 | 1 | — |
se_glm_s_n100_v1 | eval-glm-45-airlitellm | summeval_100 | 1 | $0.0481 | A scorer that agrees with experts on summaries (SummEval) #19 of 33 | 1 | — |
se_length_n100_v1 | — length · local | summeval_100 | 1 | — | A scorer that agrees with experts on summaries (SummEval) #32 of 33 | 1 | — |
se_llama_l_n100_v1 | eval-llama-mavericklitellm | summeval_100 | 1 | $0.0594 | A scorer that agrees with experts on summaries (SummEval) #18 of 33 | 1 | — |
se_llama_m_n100_v1 | eval-llama-70blitellm | summeval_100 | 1 | $0.0517 | A scorer that agrees with experts on summaries (SummEval) #23 of 33 | 1 | — |
se_llama_s_n100_v1 | eval-llama-scoutlitellm | summeval_100 | 1 | $0.0297 | A scorer that agrees with experts on summaries (SummEval) #25 of 33 | 1 | — |
se_mistral_l_n100_v1 | eval-mistral-largelitellm | summeval_100 | 1 | $0.1257 | A scorer that agrees with experts on summaries (SummEval) #17 of 33 | 1 | — |
se_mistral_m_n100_v1 | eval-mistral-mediumlitellm | summeval_100 | 1 | $0.1191 | A scorer that agrees with experts on summaries (SummEval) #15 of 33 | 1 | — |
se_mistral_s_n100_v1 | eval-mistral-smalllitellm | summeval_100 | 1 | $0.0283 | A scorer that agrees with experts on summaries (SummEval) #20 of 33 | 1 | — |
se_openai_l_n100_v1 | eval-gpt-6sollitellm | summeval_100 | 1 | $0.6350 | A scorer that agrees with experts on summaries (SummEval) #1 of 33 | 1 | — |
se_openai_m_n100_v1 | eval-gpt-55litellm | summeval_100 | 1 | $1.42 | A scorer that agrees with experts on summaries (SummEval) #4 of 33 | 1 | — |
se_openai_s_n100_v1 | eval-gpt-minilitellm | summeval_100 | 1 | $0.2124 | A scorer that agrees with experts on summaries (SummEval) #22 of 33 | 1 | — |
se_qwen37max_n100_v1 | eval-qwen-37maxlitellm | summeval_100 | 1 | $0.4436 | A scorer that agrees with experts on summaries (SummEval) #5 of 33 | 1 | — |
se_qwen_m_n100_v1 | eval-qwen-37pluslitellm | summeval_100 | 1 | $0.1084 | A scorer that agrees with experts on summaries (SummEval) #8 of 33 | 1 | — |
se_qwen_s_n100_pairwise_v1 | eval-qwen-flashlitellm | summeval_100 | 1 | $0.0531 | — | 1 | — |
se_qwen_s_n100_pick_v1 | eval-qwen-flashlitellm | summeval_100 | 1 | $0.0282 | — | 1 | — |
se_qwen_s_n100_v1 | eval-qwen-flashlitellm | summeval_100 | 1 | $0.0385 | A scorer that agrees with experts on summaries (SummEval) #6 of 33 | 1 | — |
se_random_n100_pairwise_v1 | — random · local | summeval_100 | 1 | — | — | 1 | — |
se_random_n100_pick_v1 | — random · local | summeval_100 | 1 | — | — | 1 | — |
se_random_n100_v1 | — random · local | summeval_100 | 1 | — | A scorer that agrees with experts on summaries (SummEval) #33 of 33 | 1 | — |
se_rouge1_n100_v1 | — rouge · local | summeval_100 | 1 | — | A scorer that agrees with experts on summaries (SummEval) #29 of 33 | 1 | — |
se_rouge2_n100_v1 | — rouge · local | summeval_100 | 1 | — | A scorer that agrees with experts on summaries (SummEval) #28 of 33 | 1 | — |
se_rougeL_n100_v1 | — rouge · local | summeval_100 | 1 | — | A scorer that agrees with experts on summaries (SummEval) #30 of 33 | 1 | — |
se_unieval_n100_v1 | MingZhong/unieval-sumunieval · local | summeval_100 | 1 | — | A scorer that agrees with experts on summaries (SummEval) #26 of 33 | 1 | — |
se_unievalpub_n100_v1 | MingZhong/unieval-sumunieval · local | summeval_100 | 1 | — | A scorer that agrees with experts on summaries (SummEval) #24 of 33 | 1 | — |
sf_bge_small_n200_v1 | BAAI/bge-small-en-v1.5dense · local | scifact_200 | 1 | — | Evidence retrieval for scientific claims (SciFact) #11 of 19 | 1 | — |
sf_bm25_n200_v1 | — bm25 · local | scifact_200 | 1 | — | Evidence retrieval for scientific claims (SciFact) #17 of 19 | 1 | — |
sf_deepseek_m_n200_v1 | eval-deepseek-41flashrerank | scifact_200 | 1 | $0.2381 | Evidence retrieval for scientific claims (SciFact) #3 of 19 | 1 | — |
sf_deepseek_s_n200_v1 | eval-deepseek-flashrerank | scifact_200 | 1 | $0.0600 | Evidence retrieval for scientific claims (SciFact) #4 of 19 | 1 | — |
sf_e5_base_n200_v1 | intfloat/e5-base-v2dense · local | scifact_200 | 1 | — | Evidence retrieval for scientific claims (SciFact) #8 of 19 | 1 | — |
sf_first_k_n200_v1 | — first_k · local | scifact_200 | 1 | — | Evidence retrieval for scientific claims (SciFact) #19 of 19 | 1 | — |
sf_gemma_l_n200_v1 | eval-gemma-31brerank | scifact_200 | 1 | $0.1701 | Evidence retrieval for scientific claims (SciFact) #6 of 19 | 1 | — |
sf_gemma_m_n200_v1 | eval-gemma-26brerank | scifact_200 | 1 | $0.0721 | Evidence retrieval for scientific claims (SciFact) #5 of 19 | 1 | — |
sf_gemma_s_n200_v1 | eval-gemma-3-27brerank | scifact_200 | 1 | $0.0962 | Evidence retrieval for scientific claims (SciFact) #14 of 19 | 1 | — |
sf_glm_m_n200_v1 | eval-glm-46rerank | scifact_200 | 1 | $0.4576 | Evidence retrieval for scientific claims (SciFact) #2 of 19 | 1 | — |
sf_glm_s_bm25first_n200_v1 | eval-glm-45-airrerank | scifact_200 | 1 | $0.1481 | — | 1 | — |
sf_glm_s_e5first_n200_v1 | eval-glm-45-airrerank | scifact_200 | 1 | $0.1464 | — | 1 | — |
sf_glm_s_n200_v1 | eval-glm-45-airrerank | scifact_200 | 1 | $0.1421 | Evidence retrieval for scientific claims (SciFact) #1 of 19 | 1 | — |
sf_llama_l_n200_v1 | eval-llama-maverickrerank | scifact_200 | 2 | $0.3422 | Evidence retrieval for scientific claims (SciFact) #7 of 19 | 2 | — |
sf_llama_m_n200_v1 | eval-llama-70brerank | scifact_200 | 1 | $0.1434 | Evidence retrieval for scientific claims (SciFact) #12 of 19 | 1 | — |
sf_llama_s_n200_v1 | eval-llama-scoutrerank | scifact_200 | 2 | $0.2506 | Evidence retrieval for scientific claims (SciFact) #10 of 19 | 2 | — |
sf_minilm_n200_v1 | sentence-transformers/all-MiniLM-L6-v2dense · local | scifact_200 | 1 | — | Evidence retrieval for scientific claims (SciFact) #16 of 19 | 1 | — |
sf_mistral_s_n200_v1 | eval-mistral-smallrerank | scifact_200 | 1 | $0.0852 | Evidence retrieval for scientific claims (SciFact) #13 of 19 | 1 | — |
sf_mpnet_n200_v1 | sentence-transformers/all-mpnet-base-v2dense · local | scifact_200 | 1 | — | Evidence retrieval for scientific claims (SciFact) #15 of 19 | 1 | — |
sf_qwen_s_bm25first_n200_v1 | eval-qwen-flashrerank | scifact_200 | 1 | $0.1551 | — | 1 | — |
sf_qwen_s_e5first_n200_v1 | eval-qwen-flashrerank | scifact_200 | 1 | $0.1508 | — | 1 | — |
sf_qwen_s_n200_v1 | eval-qwen-flashrerank | scifact_200 | 2 | $0.3117 | Evidence retrieval for scientific claims (SciFact) #9 of 19 | 2 | — |
sf_random_n200_v1 | — random · local | scifact_200 | 1 | — | Evidence retrieval for scientific claims (SciFact) #18 of 19 | 1 | — |
triage_baseline_pro_noreason_v1 | eval-fleet-triage-progateway | orrery_triage_v1_labelled | 3 | $0.3160 | Triage decisions on Orrery's error signals #52 of 55 | 1 | — |
triage_baseline_pro_v1 | eval-fleet-triage-progateway | orrery_triage_v1_labelled | 10 | $1.32 | Triage decisions on Orrery's error signals #35 of 55 | 4 | — |
triage_deepseek_41flash_v1 | eval-deepseek-41flashgateway | orrery_triage_v1_labelled | 4 | $0.0560 | — | 1 | — |
triage_deepseek_41flash_v2 | eval-deepseek-41flashgateway | orrery_triage_v1_labelled | 3 | $0.0231 | Triage decisions on Orrery's error signals #47 of 55 | 1 | — |
triage_gemini_35_flash_lite_v1 | eval-gemini-35-flash-litegateway | orrery_triage_v1_labelled | 5 | $0.0889 | Triage decisions on Orrery's error signals #6 of 55 | 1 | — |
triage_gemini_4_argon_v1 | eval-gemini-4-argongateway | orrery_triage_v1_labelled | 5 | $0.8851 | Triage decisions on Orrery's error signals #17 of 55 | 1 | — |
triage_gpt_61_sol_v1 | eval-gpt-61-solgateway | orrery_triage_v1_labelled | 10 | $2.83 | Triage decisions on Orrery's error signals #9 of 55 | 4 | — |
triage_gpt_6_luna_v1 | eval-gpt-6-lunagateway | orrery_triage_v1_labelled | 8 | $0.1798 | Triage decisions on Orrery's error signals #15 of 55 | 4 | — |
triage_gpt_6_luna_v2 | eval-gpt-6-lunagateway | orrery_triage_v1_labelled | 3 | $0.0434 | Triage decisions on Orrery's error signals #7 of 55 | 1 | — |
triage_haiku_55_v1 | eval-claude-haiku-55gateway | orrery_triage_v1_labelled | 10 | $0.2646 | Triage decisions on Orrery's error signals #10 of 55 | 4 | — |
triage_hosted_gemma4_31b_v1 | eval-gemma-31bgateway | orrery_triage_v1_labelled | 4 | $0.0587 | — | 1 | — |
triage_hosted_gemma4_31b_v2 | eval-gemma-31bgateway | orrery_triage_v1_labelled | 3 | $0.0583 | Triage decisions on Orrery's error signals #29 of 55 | 1 | — |
triage_hosted_gpt_oss_120b_v1 | eval-gpt-oss-120bgateway | orrery_triage_v1_labelled | 4 | $0.0552 | — | 1 | — |
triage_hosted_gpt_oss_120b_v2 | eval-gpt-oss-120bgateway | orrery_triage_v1_labelled | 3 | $0.0407 | Triage decisions on Orrery's error signals #16 of 55 | 1 | — |
triage_hosted_mistral_medium_35_v1 | eval-mistral-medium-35gateway | orrery_triage_v1_labelled | 5 | $0.7500 | Triage decisions on Orrery's error signals #44 of 55 | 1 | — |
triage_hosted_qwen35_122b_v1 | eval-qwen35-122bgateway | orrery_triage_v1_labelled | 4 | $0.2177 | — | 1 | — |
triage_hosted_qwen35_122b_v2 | eval-qwen35-122bgateway | orrery_triage_v1_labelled | 3 | $0.1075 | Triage decisions on Orrery's error signals #45 of 55 | 1 | — |
triage_hosted_qwen3_30b_a3b_v1 | eval-qwen3-30b-a3bgateway | orrery_triage_v1_labelled | 5 | $0.0455 | Triage decisions on Orrery's error signals #25 of 55 | 1 | — |
triage_kimi_k31_pro_v1 | eval-kimi-k31-progateway | orrery_triage_v1_labelled | 4 | $1.24 | — | 1 | — |
triage_kimi_k31_pro_v2 | eval-kimi-k31-progateway | orrery_triage_v1_labelled | 3 | $0.7826 | Triage decisions on Orrery's error signals #30 of 55 | 1 | — |
triage_local_dgx_gemma4_12b_noreason_v1 | gemma4:12bgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #11 of 55 | 1 | — |
triage_local_dgx_gemma4_31b_v1 | gemma4:31bgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_dgx_gemma4_31b_v2 | gemma4:31bgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #31 of 55 | 1 | — |
triage_local_dgx_gpt_oss_120b_v1 | gpt-oss:120bgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_dgx_gpt_oss_120b_v2 | gpt-oss:120bgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #19 of 55 | 1 | — |
triage_local_dgx_gpt_oss_20b_v1 | gpt-oss:20bgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_dgx_gpt_oss_20b_v2 | gpt-oss:20bgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #32 of 55 | 1 | — |
triage_local_dgx_llama31_8b_v1 | llama3.1:8bgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #2 of 55 | 1 | — |
triage_local_dgx_llama4_scout_v1 | llama4:scoutgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #53 of 55 | 1 | — |
triage_local_dgx_ministral_3_14b_v1 | ministral-3:14bgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #41 of 55 | 1 | — |
triage_local_dgx_mistral_medium_35_128b_v1 | mistral-medium-3.5:128bgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #49 of 55 | 1 | — |
triage_local_dgx_nemotron_35_bolt_e4b_v1 | nemotron-3.5-bolt:e4b-q4_K_Mgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_dgx_nemotron_35_bolt_e4b_v2 | nemotron-3.5-bolt:e4b-q4_K_Mgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #18 of 55 | 1 | — |
triage_local_dgx_nemotron_35_lightning_30b_v1 | nemotron-3.5-lightning:30bgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_dgx_nemotron_35_lightning_30b_v2 | nemotron-3.5-lightning:30bgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #21 of 55 | 1 | — |
triage_local_dgx_nemotron_35_super_120b_v1 | nemotron-3.5-super:120bgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_dgx_nemotron_35_super_120b_v2 | nemotron-3.5-super:120bgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #22 of 55 | 1 | — |
triage_local_dgx_phi4_14b_v1 | phi4:14bgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #20 of 55 | 1 | — |
triage_local_dgx_qwen35_122b_v1 | qwen3.5:122bgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_dgx_qwen35_122b_v2 | qwen3.5:122bgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #55 of 55 | 1 | — |
triage_local_dgx_qwen38_27b_v1 | qwen3.8:27bgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_dgx_qwen38_27b_v2 | qwen3.8:27bgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #50 of 55 | 1 | — |
triage_local_dgx_qwen3_30b_a3b_v1 | qwen3:30b-a3b-instruct-2507-q4_K_Mgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #46 of 55 | 1 | — |
triage_local_dgx_vllm_qwen3_30b_a3b_nvfp4_v1 | NVFP4/Qwen3-30B-A3B-Instruct-2507-FP4gateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #1 of 55 | 1 | — |
triage_local_mac_gemma4_12b_noreason_v1 | gemma4:12bgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #12 of 55 | 1 | — |
triage_local_mac_gemma4_31b_v1 | gemma4:31b-it-q4_K_Mgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_mac_gemma4_31b_v2 | gemma4:31b-it-q4_K_Mgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #37 of 55 | 1 | — |
triage_local_mac_gpt_oss_20b_v1 | gpt-oss:20bgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_mac_gpt_oss_20b_v2 | gpt-oss:20bgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #26 of 55 | 1 | — |
triage_local_mac_llama31_8b_v1 | llama3.1:8bgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #3 of 55 | 1 | — |
triage_local_mac_ministral_3_14b_v1 | ministral-3:14bgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #48 of 55 | 1 | — |
triage_local_mac_nemotron_35_bolt_e4b_v1 | nemotron-3.5-bolt:e4b-q4_K_Mgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_mac_nemotron_35_bolt_e4b_v2 | nemotron-3.5-bolt:e4b-q4_K_Mgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #13 of 55 | 1 | — |
triage_local_mac_nemotron_35_lightning_30b_v1 | nemotron-3.5-lightning:30b-a3b-q4_K_Mgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_mac_nemotron_35_lightning_30b_v2 | nemotron-3.5-lightning:30b-a3b-q4_K_Mgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #24 of 55 | 1 | — |
triage_local_mac_phi4_14b_v1 | phi4:14bgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #34 of 55 | 1 | — |
triage_local_mac_qwen38_27b_v1 | qwen3.8:27bgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_mac_qwen38_27b_v2 | qwen3.8:27bgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #51 of 55 | 1 | — |
triage_local_mac_qwen3_30b_a3b_v1 | qwen3:30b-a3b-instruct-2507-q4_K_Mgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #42 of 55 | 1 | — |
triage_local_x3_gemma4_31b_v1 | gemma4:31bgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_x3_gemma4_31b_v2 | gemma4:31bgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #27 of 55 | 1 | — |
triage_local_x3_gpt_oss_20b_v1 | gpt-oss:20bgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_x3_gpt_oss_20b_v2 | gpt-oss:20bgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #39 of 55 | 1 | — |
triage_local_x3_llama31_8b_v1 | llama3.1:8bgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #4 of 55 | 1 | — |
triage_local_x3_ministral_3_14b_v1 | ministral-3:14bgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #40 of 55 | 1 | — |
triage_local_x3_nemotron_35_bolt_e4b_v1 | nemotron-3.5-bolt:e4b-q4_K_Mgateway | orrery_triage_v1_labelled | 8 | — | Triage decisions on Orrery's error signals #14 of 55 | 2 | — |
triage_local_x3_nemotron_35_bolt_e4b_v2 | nemotron-3.5-bolt:e4b-q4_K_Mgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #8 of 55 | 1 | — |
triage_local_x3_nemotron_35_lightning_30b_v1 | nemotron-3.5-lightning:30bgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_x3_nemotron_35_lightning_30b_v2 | nemotron-3.5-lightning:30bgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #23 of 55 | 1 | — |
triage_local_x3_phi4_14b_v1 | phi4:14bgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #36 of 55 | 1 | — |
triage_local_x3_qwen38_27b_v1 | qwen3.8:27bgateway | orrery_triage_v1_labelled | 4 | — | — | 1 | — |
triage_local_x3_qwen38_27b_v2 | qwen3.8:27bgateway | orrery_triage_v1_labelled | 3 | — | Triage decisions on Orrery's error signals #54 of 55 | 1 | — |
triage_local_x3_qwen3_30b_a3b_v1 | qwen3:30b-a3b-instruct-2507-q4_K_Mgateway | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #33 of 55 | 1 | — |
triage_mistral_large_4_v1 | eval-mistral-large-4gateway | orrery_triage_v1_labelled | 4 | $0.4464 | — | 1 | — |
triage_mistral_large_4_v2 | eval-mistral-large-4gateway | orrery_triage_v1_labelled | 3 | $0.4516 | Triage decisions on Orrery's error signals #38 of 55 | 1 | — |
triage_sonnet_55_v1 | eval-claude-sonnet-55gateway | orrery_triage_v1_labelled | 5 | $1.47 | Triage decisions on Orrery's error signals #28 of 55 | 1 | — |
triage_stub_dismiss_v1 | — stub · local | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #43 of 55 | 1 | — |
triage_stub_escalate_v1 | — stub · local | orrery_triage_v1_labelled | 5 | — | Triage decisions on Orrery's error signals #5 of 55 | 1 | — |
03what changed in the period
Nothing changed for an arm in this period.
| when | arm | what |
|---|---|---|
| 2026-11-02 | 29 arms on cnn_dailymail_200 | judged by judge-summ-pick-v1 |
| 2026-11-02 | 83 arms on orrery_triage_v1_labelled | rescored against gold/orrery_triage_v1_labelled |
| 2026-11-02 | se_gemma_l_n100_pairwise_v1 | measured |
| 2026-11-02 | se_gemma_l_n100_pick_v1 | measured |
| 2026-11-02 | se_qwen_s_n100_pairwise_v1 | measured |
| 2026-11-02 | se_qwen_s_n100_pick_v1 | measured |
| 2026-11-02 | se_random_n100_pairwise_v1 | measured |
| 2026-11-02 | se_random_n100_pick_v1 | measured |
| 2026-11-01 | 56 arms on orrery_triage_v1_labelled | rescored against gold/orrery_triage_v1_labelled |
| 2026-11-01 | triage_local_mac_gemma4_31b_v2 | measured |
| 2026-11-01 | triage_deepseek_41flash_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_gpt_6_luna_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_hosted_gemma4_31b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_hosted_gpt_oss_120b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_hosted_qwen35_122b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_kimi_k31_pro_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_dgx_gemma4_31b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_dgx_gpt_oss_120b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_dgx_gpt_oss_20b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_dgx_nemotron_35_bolt_e4b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_dgx_nemotron_35_lightning_30b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_dgx_nemotron_35_super_120b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_dgx_qwen35_122b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_dgx_qwen38_27b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_mac_gemma4_31b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_mac_gpt_oss_20b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_mac_nemotron_35_bolt_e4b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_mac_nemotron_35_lightning_30b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_mac_qwen38_27b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_x3_gemma4_31b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_x3_gpt_oss_20b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_x3_nemotron_35_bolt_e4b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_x3_nemotron_35_lightning_30b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_local_x3_qwen38_27b_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-11-01 | triage_mistral_large_4_v1 | retired in r3: run at the server's default reasoning for a model that reasons by default; re-run in this round as _v2 with the model's onboarded settings |
| 2026-10-31 | triage_baseline_pro_noreason_v1 | measured |
| 2026-10-31 | triage_deepseek_41flash_v2 | measured |
| 2026-10-31 | triage_gpt_6_luna_v2 | measured |
| 2026-10-31 | triage_hosted_gemma4_31b_v2 | measured |
| 2026-10-31 | triage_hosted_gpt_oss_120b_v2 | measured |
| 2026-10-31 | triage_hosted_qwen35_122b_v2 | measured |
| 2026-10-31 | triage_kimi_k31_pro_v2 | measured |
| 2026-10-31 | triage_local_dgx_gemma4_12b_noreason_v1 | measured |
| 2026-10-31 | triage_local_dgx_gemma4_31b_v1 | measured |
| 2026-10-31 | triage_local_dgx_gemma4_31b_v2 | measured |
| 2026-10-31 | triage_local_dgx_gpt_oss_120b_v1 | measured |
| 2026-10-31 | triage_local_dgx_gpt_oss_120b_v2 | measured |
| 2026-10-31 | triage_local_dgx_gpt_oss_20b_v2 | measured |
| 2026-10-31 | triage_local_dgx_llama31_8b_v1 | measured |
| 2026-10-31 | triage_local_dgx_ministral_3_14b_v1 | measured |
| 2026-10-31 | triage_local_dgx_nemotron_35_bolt_e4b_v2 | measured |
| 2026-10-31 | triage_local_dgx_nemotron_35_lightning_30b_v2 | measured |
| 2026-10-31 | triage_local_dgx_nemotron_35_super_120b_v2 | measured |
| 2026-10-31 | triage_local_dgx_phi4_14b_v1 | measured |
| 2026-10-31 | triage_local_dgx_qwen35_122b_v2 | measured |
| 2026-10-31 | triage_local_dgx_qwen38_27b_v2 | measured |
| 2026-10-31 | triage_local_dgx_qwen3_30b_a3b_v1 | measured |
| 2026-10-31 | triage_local_dgx_vllm_qwen3_30b_a3b_nvfp4_v1 | measured |
| 2026-10-31 | triage_local_mac_gemma4_12b_noreason_v1 | measured |
| 2026-10-31 | triage_local_mac_gpt_oss_20b_v2 | measured |
| 2026-10-31 | triage_local_mac_ministral_3_14b_v1 | measured |
| 2026-10-31 | triage_local_mac_nemotron_35_bolt_e4b_v2 | measured |
| 2026-10-31 | triage_local_mac_nemotron_35_lightning_30b_v2 | measured |
| 2026-10-31 | triage_local_mac_qwen38_27b_v2 | measured |
| 2026-10-31 | triage_local_mac_qwen3_30b_a3b_v1 | measured |
| 2026-10-31 | triage_local_x3_gemma4_31b_v2 | measured |
| 2026-10-31 | triage_local_x3_gpt_oss_20b_v2 | measured |
| 2026-10-31 | triage_local_x3_nemotron_35_bolt_e4b_v2 | measured |
| 2026-10-31 | triage_local_x3_nemotron_35_lightning_30b_v2 | measured |
| 2026-10-31 | triage_local_x3_qwen38_27b_v2 | measured |
| 2026-10-31 | triage_local_x3_qwen3_30b_a3b_v1 | measured |
| 2026-10-31 | triage_mistral_large_4_v2 | measured |
| 2026-10-31 | triage_stub_dismiss_v1 | measured |
| 2026-10-31 | triage_stub_escalate_v1 | measured |
| 2026-10-31 | 114 arms on orrery_triage_v1_labelled | rescored against gold/orrery_triage_v1_labelled |
| 2026-10-31 | triage_baseline_pro_v1 | promoted as the baseline |
| 2026-10-30 | dedup_guard_v1 | measured |
| 2026-10-30 | dedup_pinned_f07efa9_v1 | measured |
| 2026-10-30 | triage_baseline_pro_v1 | measured |
| 2026-10-30 | triage_deepseek_41flash_v1 | measured |
| 2026-10-30 | triage_gemini_35_flash_lite_v1 | measured |
| 2026-10-30 | triage_gemini_4_argon_v1 | measured |
| 2026-10-30 | triage_gpt_61_sol_v1 | measured |
| 2026-10-30 | triage_gpt_6_luna_v1 | measured |
| 2026-10-30 | triage_haiku_55_v1 | measured |
| 2026-10-30 | triage_hosted_gemma4_31b_v1 | measured |
| 2026-10-30 | triage_hosted_gpt_oss_120b_v1 | measured |
| 2026-10-30 | triage_hosted_mistral_medium_35_v1 | measured |
| 2026-10-30 | triage_hosted_qwen35_122b_v1 | measured |
| 2026-10-30 | triage_hosted_qwen3_30b_a3b_v1 | measured |
| 2026-10-30 | triage_kimi_k31_pro_v1 | measured |
| 2026-10-30 | triage_local_dgx_gpt_oss_20b_v1 | measured |
| 2026-10-30 | triage_local_dgx_llama4_scout_v1 | measured |
| 2026-10-30 | triage_local_dgx_mistral_medium_35_128b_v1 | measured |
| 2026-10-30 | triage_local_dgx_nemotron_35_bolt_e4b_v1 | measured |
| 2026-10-30 | triage_local_dgx_nemotron_35_lightning_30b_v1 | measured |
| 2026-10-30 | triage_local_dgx_nemotron_35_super_120b_v1 | measured |
| 2026-10-30 | triage_local_dgx_qwen35_122b_v1 | measured |
| 2026-10-30 | triage_local_dgx_qwen38_27b_v1 | measured |
| 2026-10-30 | triage_local_mac_gemma4_31b_v1 | measured |
| 2026-10-30 | triage_local_mac_gpt_oss_20b_v1 | measured |
| 2026-10-30 | triage_local_mac_llama31_8b_v1 | measured |
| 2026-10-30 | triage_local_mac_nemotron_35_bolt_e4b_v1 | measured |
| 2026-10-30 | triage_local_mac_nemotron_35_lightning_30b_v1 | measured |
| 2026-10-30 | triage_local_mac_phi4_14b_v1 | measured |
| 2026-10-30 | triage_local_mac_qwen38_27b_v1 | measured |
| 2026-10-30 | triage_local_x3_gemma4_31b_v1 | measured |
| 2026-10-30 | triage_local_x3_gpt_oss_20b_v1 | measured |
| 2026-10-30 | triage_local_x3_llama31_8b_v1 | measured |
| 2026-10-30 | triage_local_x3_ministral_3_14b_v1 | measured |
| 2026-10-30 | triage_local_x3_nemotron_35_bolt_e4b_v1 | measured |
| 2026-10-30 | triage_local_x3_nemotron_35_lightning_30b_v1 | measured |
| 2026-10-30 | triage_local_x3_phi4_14b_v1 | measured |
| 2026-10-30 | triage_local_x3_qwen38_27b_v1 | measured |
| 2026-10-30 | triage_mistral_large_4_v1 | measured |
| 2026-10-30 | triage_sonnet_55_v1 | measured |
| 2026-10-30 | 5 arms on orrery_triage_v1_labelled | run again (repeats) |
| 2026-10-25 | 30 arms on ag_news_200 | rescored against gold/ag_news_200_adj1 |
| 2026-10-25 | 29 arms on cnn_dailymail_200 | judged by judge-summ-v1 |
| 2026-10-25 | 29 arms on cnn_dailymail_200 | judged by judge-summ-v2 |
| 2026-10-25 | 29 arms on cnn_dailymail_200 | rescored against silver/cnn_dailymail_200 |
| 2026-10-25 | cnn_bart_l_n200_v1 · judged/judge-summ-pointwise-v1 | judged by judge-summ-pointwise-v1 |
| 2026-10-25 | cnn_deepseek_m_n200_v1 · judged/judge-summ-pointwise-v1 | judged by judge-summ-pointwise-v1 |
| 2026-10-25 | ag_bert_mini_n200_v1 | promoted as the baseline |
| 2026-10-25 | cnn_bart_l_n200_v1 | promoted as the baseline |
| 2026-10-25 | db_bert_base_fy_n200_v1 | promoted as the baseline |
| 2026-10-25 | fn_span_marker_n200_v1 | promoted as the baseline |
| 2026-10-25 | se_gemma_l_n100_v1 | promoted as the baseline |
| 2026-10-25 | sf_e5_base_n200_v1 | promoted as the baseline |
| 2026-10-23 | demo_v1 | measured |
| 2026-10-23 | demo_v1 | promoted as the baseline |
| 2026-10-19 | se_gemma_l_pubtext_n100_v1 · pair | run again (pair) |
| 2026-10-19 | se_gemma_l_pubtextcw_n100_v1 · pair | run again (pair) |
| 2026-10-19 | se_gemma_l_pubtext_n100_v1 · pair | retired in r2: a paired side run, not on the leaderboard; data/runs is the record |
| 2026-10-19 | se_gemma_l_pubtextcw_n100_v1 · pair | retired in r2: a paired side run, not on the leaderboard; data/runs is the record |
| 2026-10-18 | se_anthropic_l_n100_v1 | measured |
| 2026-10-18 | se_anthropic_m_n100_v1 | measured |
| 2026-10-18 | se_anthropic_s_n100_v1 | measured |
| 2026-10-18 | se_deepseek_l_n100_v1 | measured |
| 2026-10-18 | se_deepseek_m_n100_v1 | measured |
| 2026-10-18 | se_deepseek_s_n100_v1 | measured |
| 2026-10-18 | se_gemma_l_n100_v1 | measured |
| 2026-10-18 | se_gemma_m_n100_v1 | measured |
| 2026-10-18 | se_gemma_s_n100_v1 | measured |
| 2026-10-18 | se_glm_l_n100_v1 | measured |
| 2026-10-18 | se_glm_m_n100_v1 | measured |
| 2026-10-18 | se_glm_s_n100_v1 | measured |
| 2026-10-18 | se_llama_l_n100_v1 | measured |
| 2026-10-18 | se_llama_m_n100_v1 | measured |
| 2026-10-18 | se_llama_s_n100_v1 | measured |
| 2026-10-18 | se_mistral_l_n100_v1 | measured |
| 2026-10-18 | se_mistral_m_n100_v1 | measured |
| 2026-10-18 | se_mistral_s_n100_v1 | measured |
| 2026-10-18 | se_openai_l_n100_v1 | measured |
| 2026-10-18 | se_openai_m_n100_v1 | measured |
| 2026-10-18 | se_openai_s_n100_v1 | measured |
| 2026-10-18 | se_qwen37max_n100_v1 | measured |
| 2026-10-18 | se_qwen_m_n100_v1 | measured |
| 2026-10-18 | se_qwen_s_n100_v1 | measured |
| 2026-10-18 | se_unieval_n100_v1 | measured |
| 2026-10-18 | se_unievalpub_n100_v1 | measured |
| 2026-10-15 | cnn_bart_s_dm200_v1 | measured |
| 2026-10-15 | se_bartscore_n100_v1 | measured |
| 2026-10-15 | se_coverage_n100_v1 | measured |
| 2026-10-15 | se_length_n100_v1 | measured |
| 2026-10-15 | se_random_n100_v1 | measured |
| 2026-10-15 | se_rouge1_n100_v1 | measured |
| 2026-10-15 | se_rouge2_n100_v1 | measured |
| 2026-10-15 | se_rougeL_n100_v1 | measured |
| 2026-10-14 | cnn_bart_l_dm200_v1 | measured |
| 2026-10-14 | cnn_bart_l_xsum_dm200_v1 | measured |
| 2026-10-14 | cnn_bart_m_dm200_v1 | measured |
| 2026-10-14 | cnn_lead3_dm200_v1 | measured |
| 2026-10-01 | fn_mistral_l_n200_v1 | rescored against gold/few_nerd_280 |
| 2026-09-30 | ag_bert_base_fy_n200_v1 | measured |
| 2026-09-30 | ag_bert_base_ta_n200_v1 | measured |
| 2026-09-30 | 3 arms on ag_news_200 | run again (linux) |
| 2026-09-30 | 5 arms on cnn_dailymail_200 | run again (linux) |
| 2026-09-30 | cnn_bart_l_xsum_n200_v1 | measured |
| 2026-09-30 | cnn_bart_m_n200_v1 | measured |
| 2026-09-30 | cnn_bart_s_n200_v1 | measured |
| 2026-09-30 | cnn_bart_s_n200_v1 · repeats | run again (repeats) |
| 2026-09-30 | db_bert_base_fy_n200_v1 | measured |
| 2026-09-30 | db_bert_base_fy_n200_v1 · linux | run again (linux) |
| 2026-09-30 | db_bert_base_fy_n200_v1 | retired in r2: a re-timing of the record's config on one Linux machine; data/runs is the record |
| 2026-09-30 | cnn_bart_l_n200_v1 | retired in r2: a re-timing of the record's config on one Linux machine; data/runs is the record |
| 2026-09-30 | cnn_bart_l_xsum_n200_v1 | retired in r2: a re-timing of the record's config on one Linux machine; data/runs is the record |
| 2026-09-30 | cnn_bart_m_n200_v1 | retired in r2: a re-timing of the record's config on one Linux machine; data/runs is the record |
| 2026-09-30 | cnn_bart_s_n200_v1 | retired in r2: a re-timing of the record's config on one Linux machine; data/runs is the record |
| 2026-09-30 | cnn_lead3_n200_v1 | retired in r2: a re-timing of the record's config on one Linux machine; data/runs is the record |
| 2026-09-30 | cnn_bart_s_n200_v1 | retired in r3: a second run of the record's config; data/runs is the record |
| 2026-09-30 | ag_bert_base_fy_n200_v1 | retired in r2: a re-timing of the record's config on one Linux machine; data/runs is the record |
| 2026-09-30 | ag_bert_base_ta_n200_v1 | retired in r2: a re-timing of the record's config on one Linux machine; data/runs is the record |
| 2026-09-30 | ag_bert_mini_n200_v1 | retired in r2: a re-timing of the record's config on one Linux machine; data/runs is the record |
| 2026-09-29 | 3 arms on scifact_200 | re-parsed |
| 2026-09-29 | sf_llama_l_n200_v1 | retired in r3: a second run of the record's config; data/runs is the record |
| 2026-09-29 | sf_llama_s_n200_v1 | retired in r3: a second run of the record's config; data/runs is the record |
| 2026-09-29 | sf_qwen_s_n200_v1 | retired in r3: a second run of the record's config; data/runs is the record |
| 2026-09-28 | 27 arms on few_nerd_280 | rescored against gold/few_nerd_280 |
| 2026-09-28 | 23 arms on scifact_200 | re-parsed |
| 2026-09-28 | sf_glm_s_bm25first_n200_v1 · pair | retired in r2: a paired side run, not on the leaderboard; data/runs is the record |
| 2026-09-28 | sf_glm_s_e5first_n200_v1 · pair | retired in r2: a paired side run, not on the leaderboard; data/runs is the record |
| 2026-09-28 | sf_qwen_s_bm25first_n200_v1 · pair | retired in r2: a paired side run, not on the leaderboard; data/runs is the record |
| 2026-09-28 | sf_qwen_s_e5first_n200_v1 · pair | retired in r2: a paired side run, not on the leaderboard; data/runs is the record |
| 2026-09-27 | ag_anthropic_l_n200_v1 | measured |
| 2026-09-27 | ag_anthropic_m_n200_v1 | measured |
| 2026-09-27 | ag_anthropic_s_n200_v1 | measured |
| 2026-09-27 | ag_bart_mnli_n200_v1 | measured |
| 2026-09-27 | ag_bert_mini_n200_v1 | measured |
| 2026-09-27 | ag_constant_n200_v1 | measured |
| 2026-09-27 | ag_deepseek_l_n200_v1 | measured |
| 2026-09-27 | ag_deepseek_m_n200_v1 | measured |
| 2026-09-27 | ag_deepseek_s_n200_v1 | measured |
| 2026-09-27 | ag_gemma_l_n200_v1 | measured |
| 2026-09-27 | ag_gemma_m_n200_v1 | measured |
| 2026-09-27 | ag_gemma_s_n200_v1 | measured |
| 2026-09-27 | ag_glm_l_n200_v1 | measured |
| 2026-09-27 | ag_glm_m_n200_v1 | measured |
| 2026-09-27 | ag_glm_s_n200_v1 | measured |
| 2026-09-27 | ag_keyword_n200_v1 | measured |
| 2026-09-27 | ag_llama_l_n200_v1 | measured |
| 2026-09-27 | ag_llama_m_n200_v1 | measured |
| 2026-09-27 | ag_llama_s_n200_v1 | measured |
| 2026-09-27 | ag_mistral_l_n200_v1 | measured |
| 2026-09-27 | ag_mistral_m_n200_v1 | measured |
| 2026-09-27 | ag_mistral_s_n200_v1 | measured |
| 2026-09-27 | ag_openai_l_n200_v1 | measured |
| 2026-09-27 | ag_openai_m_n200_v1 | measured |
| 2026-09-27 | ag_openai_s_n200_v1 | measured |
| 2026-09-27 | ag_qwen_l_n200_v1 | measured |
| 2026-09-27 | ag_qwen_m_n200_v1 | measured |
| 2026-09-27 | ag_qwen_s_n200_v1 | measured |
| 2026-09-27 | db_anthropic_l_n200_v1 | measured |
| 2026-09-27 | db_anthropic_m_n200_v1 | measured |
| 2026-09-27 | db_anthropic_s_n200_v1 | measured |
| 2026-09-27 | db_bart_mnli_n200_v1 | measured |
| 2026-09-27 | db_constant_n200_v1 | measured |
| 2026-09-27 | db_deepseek_l_n200_v1 | measured |
| 2026-09-27 | db_deepseek_m_n200_v1 | measured |
| 2026-09-27 | db_deepseek_s_n200_v1 | measured |
| 2026-09-27 | db_gemma_l_n200_v1 | measured |
| 2026-09-27 | db_gemma_m_n200_v1 | measured |
| 2026-09-27 | db_gemma_s_n200_v1 | measured |
| 2026-09-27 | db_glm_l_n200_v1 | measured |
| 2026-09-27 | db_glm_m_n200_v1 | measured |
| 2026-09-27 | db_glm_s_n200_v1 | measured |
| 2026-09-27 | db_keyword_n200_v1 | measured |
| 2026-09-27 | db_llama_l_n200_v1 | measured |
| 2026-09-27 | db_llama_m_n200_v1 | measured |
| 2026-09-27 | db_llama_s_n200_v1 | measured |
| 2026-09-27 | db_mistral_l_n200_v1 | measured |
| 2026-09-27 | db_mistral_m_n200_v1 | measured |
| 2026-09-27 | db_mistral_s_n200_v1 | measured |
| 2026-09-27 | db_openai_l_n200_v1 | measured |
| 2026-09-27 | db_openai_m_n200_v1 | measured |
| 2026-09-27 | db_openai_s_n200_v1 | measured |
| 2026-09-27 | db_qwen_l_n200_v1 | measured |
| 2026-09-27 | db_qwen_m_n200_v1 | measured |
| 2026-09-27 | db_qwen_s_n200_v1 | measured |
| 2026-09-26 | cnn_bart_l_v1 | measured |
| 2026-09-26 | cnn_lead3_v1 | measured |
| 2026-09-26 | cnn_anthropic_l_n200_v1 | measured |
| 2026-09-26 | cnn_anthropic_m_n200_v1 | measured |
| 2026-09-26 | cnn_anthropic_s_n200_v1 | measured |
| 2026-09-26 | cnn_bart_l_n200_v1 | measured |
| 2026-09-26 | cnn_gemma_l_n200_v1 | measured |
| 2026-09-26 | cnn_gemma_m_n200_v1 | measured |
| 2026-09-26 | cnn_gemma_s_n200_v1 | measured |
| 2026-09-26 | cnn_glm_l_n200_v1 | measured |
| 2026-09-26 | cnn_glm_m_n200_v1 | measured |
| 2026-09-26 | cnn_glm_s_n200_v1 | measured |
| 2026-09-26 | cnn_lead3_n200_v1 | measured |
| 2026-09-26 | cnn_llama_l_n200_v1 | measured |
| 2026-09-26 | cnn_llama_m_n200_v1 | measured |
| 2026-09-26 | cnn_llama_s_n200_v1 | measured |
| 2026-09-26 | cnn_mistral_l_n200_v1 | measured |
| 2026-09-26 | cnn_mistral_m_n200_v1 | measured |
| 2026-09-26 | cnn_mistral_s_n200_v1 | measured |
| 2026-09-26 | cnn_openai_l_n200_v1 | measured |
| 2026-09-26 | cnn_openai_m_n200_v1 | measured |
| 2026-09-26 | cnn_openai_s_n200_v1 | measured |
| 2026-09-26 | cnn_qwen_l_n200_v1 | measured |
| 2026-09-26 | cnn_qwen_s_n200_v1 | measured |
| 2026-09-25 | cnn_deepseek_l_n200_v1 | measured |
| 2026-09-25 | cnn_deepseek_m_n200_v1 | measured |
| 2026-09-25 | cnn_deepseek_s_n200_v1 | measured |
| 2026-09-25 | cnn_qwen_m_n200_v1 | measured |
04cost and quality, by goal