models

Models

64 models: 11 vetted, 12 onboarded with no arm smoked, 1 to check, 40 not onboarded yet. A model is onboarded once (its deployment on every machine, what it can do, the settings chosen for it) and vetted when an arm passes its smoke on the task's own items; then any eval's arms use it.

all time, 2026-09-25 to 2026-11-02

01every model

64 shown

model	build	where	settings	onboarding	arms	goals	best result
DeepSeek 4.1 Flash llm	`eval-deepseek-41flash`	hosted	reasoning off	pass 2026-10-31	19	7 goals Triage decisions on Orrery's error signals +6	#2 of 29 Short news summaries (CNN) +6 more
Gemma 4 31B llm	`gemma4:31b@17ba34c06c80` +2	x3, dgx, mac, hosted	reasoning off	pass 2026-10-31	43	7 goals A scorer that agrees with experts on summaries (SummEval) +6	#2 of 33 A scorer that agrees with experts on summaries (SummEval) +6 more
GPT-6 Luna llm	`eval-gpt-6-luna`	hosted	reasoning off	pass 2026-10-31	11	Triage decisions on Orrery's error signals	#7 of 55 Triage decisions on Orrery's error signals
gpt-oss 120B llm	`gpt-oss:120b@714d86958b6e` +1	x3, dgx, hosted	reasoning low	pass 2026-10-31	14	Triage decisions on Orrery's error signals	#16 of 55 Triage decisions on Orrery's error signals
Kimi K3.1 Pro llm	`eval-kimi-k31-pro`	hosted	reasoning off	pass 2026-10-31	7	Triage decisions on Orrery's error signals	#30 of 55 Triage decisions on Orrery's error signals
Mistral Large 4 llm	`eval-mistral-large-4`	hosted	reasoning low	pass 2026-10-31	7	Triage decisions on Orrery's error signals	#38 of 55 Triage decisions on Orrery's error signals
Nemotron 3.5 Bolt e4b llm	`nemotron-3.5-bolt:e4b-q4_K_M@69f2be39b309`	x3, dgx, mac	reasoning off	pass 2026-10-31	25	Triage decisions on Orrery's error signals	#8 of 55 Triage decisions on Orrery's error signals
Nemotron 3.5 Lightning 30B llm	`nemotron-3.5-lightning:30b@e6905fffe2e2` +1	x3, dgx, mac	reasoning off	pass 2026-10-31	21	Triage decisions on Orrery's error signals	#21 of 55 Triage decisions on Orrery's error signals
Qwen 3.5 122B llm	`qwen3.5:122b@8b9d11d807c5` +1	x3, dgx, hosted	reasoning off	pass 2026-10-31	14	Triage decisions on Orrery's error signals	#45 of 55 Triage decisions on Orrery's error signals
Qwen 3.8 27B llm	`qwen3.8:27b@aaee06c39dcf`	x3, dgx, mac	reasoning off	pass 2026-10-31	21	Triage decisions on Orrery's error signals	#50 of 55 Triage decisions on Orrery's error signals
The fleet's production triage model llm	`eval-fleet-triage-pro`	hosted	reasoning off	pass 2026-10-31	13	Triage decisions on Orrery's error signals	#35 of 55 Triage decisions on Orrery's error signals
Claude Haiku 5.5 llm	`eval-claude-haiku-55`	hosted	defaults	probed 2026-10-31	10	Triage decisions on Orrery's error signals	#10 of 55 Triage decisions on Orrery's error signals
Claude Sonnet 5.5 llm	`eval-claude-sonnet-55`	hosted	defaults	probed 2026-10-31	5	Triage decisions on Orrery's error signals	#28 of 55 Triage decisions on Orrery's error signals
Gemini 3.5 Flash Lite llm	`eval-gemini-35-flash-lite`	hosted	defaults	probed 2026-10-31	5	Triage decisions on Orrery's error signals	#6 of 55 Triage decisions on Orrery's error signals
Gemini 4 Argon llm	`eval-gemini-4-argon`	hosted	defaults	probed 2026-10-31	5	Triage decisions on Orrery's error signals	#17 of 55 Triage decisions on Orrery's error signals
Gemma 4 12B llm	`gemma4:12b@6114515d63c1`	x3, dgx, mac	reasoning off	probed 2026-10-31	10	Triage decisions on Orrery's error signals	#11 of 55 Triage decisions on Orrery's error signals
GPT-6.1 Sol llm	`eval-gpt-61-sol`	hosted	defaults	probed 2026-10-31	10	Triage decisions on Orrery's error signals	#9 of 55 Triage decisions on Orrery's error signals
gpt-oss 20B llm	`gpt-oss:20b@f38aa0c53da5`	x3, dgx, mac	reasoning low	probed 2026-10-31	21	Triage decisions on Orrery's error signals	#26 of 55 Triage decisions on Orrery's error signals
Llama 3.1 8B llm	`llama3.1:8b@46e0c10c039e`	x3, dgx, mac	defaults	probed 2026-10-31	15	Triage decisions on Orrery's error signals	#2 of 55 Triage decisions on Orrery's error signals
Llama 4 Scout llm	`llama4:scout@bf31604e25c2`	x3, dgx	defaults	probed 2026-10-31	5	Triage decisions on Orrery's error signals	#53 of 55 Triage decisions on Orrery's error signals
Mistral Medium 3.5 llm	`mistral-medium-3.5:128b@0341632adb05` +1	x3, dgx, hosted	defaults	probed 2026-10-31	10	Triage decisions on Orrery's error signals	#44 of 55 Triage decisions on Orrery's error signals
Phi-4 14B llm	`phi4:14b@ac896e5b8b34`	x3, dgx, mac	defaults	probed 2026-10-31	15	Triage decisions on Orrery's error signals	#20 of 55 Triage decisions on Orrery's error signals
Qwen3 30B A3B Instruct 2507 llm	`qwen3:30b-a3b-instruct-2507-q4_K_M@19e422b02313` +2	x3, dgx, mac, dgx-vllm, hosted	defaults	probed 2026-10-31	25	Triage decisions on Orrery's error signals	#1 of 55 Triage decisions on Orrery's error signals
Ministral 3 14B llm	`ministral-3:14b@4760c35aeb9d`	x3, dgx, mac	defaults	check 2026-10-31	15	Triage decisions on Orrery's error signals	#40 of 55 Triage decisions on Orrery's error signals
baai-bge-small-en-v1.5 ml	—	—	defaults	not onboarded	1	Evidence retrieval for scientific claims (SciFact)	#11 of 19 Evidence retrieval for scientific claims (SciFact)
eval-claude-haiku llm	—	—	defaults	not onboarded	10	5 goals News topic classification (AG News) +4	#11 of 33 A scorer that agrees with experts on summaries (SummEval) +4 more
eval-claude-sonnet llm	—	—	defaults	not onboarded	10	5 goals News topic classification (AG News) +4	#4 of 30 News topic classification (AG News) +4 more
eval-deepseek-flash llm	—	—	defaults	not onboarded	11	6 goals News topic classification (AG News) +5	#4 of 19 Evidence retrieval for scientific claims (SciFact) +5 more
eval-deepseek-pro llm	—	—	defaults	not onboarded	10	5 goals News topic classification (AG News) +4	#4 of 28 Encyclopedia topic classification (DBpedia) +4 more
eval-gemma-26b llm	—	—	defaults	not onboarded	11	6 goals News topic classification (AG News) +5	#4 of 28 Named-entity extraction (Few-NERD) +5 more
eval-gemma-3-27b llm	—	—	defaults	not onboarded	11	6 goals News topic classification (AG News) +5	#10 of 30 News topic classification (AG News) +5 more
eval-glm-45-air llm	—	—	defaults	not onboarded	13	6 goals Evidence retrieval for scientific claims (SciFact) +5	#1 of 19 Evidence retrieval for scientific claims (SciFact) +5 more
eval-glm-46 llm	—	—	defaults	not onboarded	11	6 goals News topic classification (AG News) +5	#2 of 19 Evidence retrieval for scientific claims (SciFact) +5 more
eval-glm-5 llm	—	—	defaults	not onboarded	10	5 goals News topic classification (AG News) +4	#9 of 33 A scorer that agrees with experts on summaries (SummEval) +4 more
eval-gpt-55 llm	—	—	defaults	not onboarded	10	5 goals News topic classification (AG News) +4	#2 of 28 Named-entity extraction (Few-NERD) +4 more
eval-gpt-6sol llm	—	—	defaults	not onboarded	10	5 goals News topic classification (AG News) +4	#1 of 33 A scorer that agrees with experts on summaries (SummEval) +4 more
eval-gpt-mini llm	—	—	defaults	not onboarded	10	5 goals News topic classification (AG News) +4	#14 of 28 Named-entity extraction (Few-NERD) +4 more
eval-llama-70b llm	—	—	defaults	not onboarded	11	6 goals News topic classification (AG News) +5	#4 of 29 Short news summaries (CNN) +5 more
eval-llama-maverick llm	—	—	defaults	not onboarded	12	6 goals Evidence retrieval for scientific claims (SciFact) +5	#3 of 29 Short news summaries (CNN) +5 more
eval-llama-scout llm	—	—	defaults	not onboarded	12	6 goals Evidence retrieval for scientific claims (SciFact) +5	#6 of 29 Short news summaries (CNN) +5 more
eval-mistral-large llm	—	—	defaults	not onboarded	10	5 goals News topic classification (AG News) +4	#11 of 30 News topic classification (AG News) +4 more
eval-mistral-medium llm	—	—	defaults	not onboarded	10	5 goals News topic classification (AG News) +4	#15 of 33 A scorer that agrees with experts on summaries (SummEval) +4 more
eval-mistral-small llm	—	—	defaults	not onboarded	11	6 goals News topic classification (AG News) +5	#13 of 29 Short news summaries (CNN) +5 more
eval-opus-5 llm	—	—	defaults	not onboarded	10	5 goals News topic classification (AG News) +4	#2 of 28 Encyclopedia topic classification (DBpedia) +4 more
eval-qwen-37max llm	—	—	defaults	not onboarded	1	A scorer that agrees with experts on summaries (SummEval)	#5 of 33 A scorer that agrees with experts on summaries (SummEval)
eval-qwen-37plus llm	—	—	defaults	not onboarded	10	5 goals News topic classification (AG News) +4	#1 of 28 Encyclopedia topic classification (DBpedia) +4 more
eval-qwen-3max llm	—	—	defaults	not onboarded	9	4 goals News topic classification (AG News) +3	#10 of 28 Named-entity extraction (Few-NERD) +3 more
eval-qwen-flash llm	—	—	defaults	not onboarded	16	6 goals Evidence retrieval for scientific claims (SciFact) +5	#6 of 33 A scorer that agrees with experts on summaries (SummEval) +5 more
fabriceyhc-bert-base-uncased-ag_news ml	—	—	defaults	not onboarded	3	News topic classification (AG News)	#2 of 30 News topic classification (AG News)
fabriceyhc-bert-base-uncased-dbpedia_14 ml	—	—	defaults	not onboarded	2	Encyclopedia topic classification (DBpedia)	#3 of 28 Encyclopedia topic classification (DBpedia)
facebook-bart-large-cnn ml	—	—	defaults	not onboarded	10	2 goals Short news summaries (CNN) +1	#1 of 29 Short news summaries (CNN) +1 more
facebook-bart-large-mnli ml	—	—	defaults	not onboarded	3	2 goals News topic classification (AG News) +1	#28 of 30 News topic classification (AG News) +1 more
facebook-bart-large-xsum ml	—	—	defaults	not onboarded	7	Short news summaries (CNN)	#29 of 29 Short news summaries (CNN)
guishe-span-marker-generic-ner-v1-fewnerd-fine-super ml	—	—	defaults	not onboarded	1	Named-entity extraction (Few-NERD)	#1 of 28 Named-entity extraction (Few-NERD)
intfloat-e5-base-v2 ml	—	—	defaults	not onboarded	1	Evidence retrieval for scientific claims (SciFact)	#8 of 19 Evidence retrieval for scientific claims (SciFact)
mingzhong-unieval-sum ml	—	—	defaults	not onboarded	2	A scorer that agrees with experts on summaries (SummEval)	#24 of 33 A scorer that agrees with experts on summaries (SummEval)
mrm8488-bert-mini-finetuned-age_news-classification ml	—	—	defaults	not onboarded	3	News topic classification (AG News)	#3 of 30 News topic classification (AG News)
Nemotron 3.5 Super 120B llm	`nemotron-3.5-super:120b@8a58f3e60082`	x3, dgx	reasoning off	not onboarded	7	Triage decisions on Orrery's error signals	#22 of 55 Triage decisions on Orrery's error signals
sentence-transformers-all-minilm-l6-v2 ml	—	—	defaults	not onboarded	1	Evidence retrieval for scientific claims (SciFact)	#16 of 19 Evidence retrieval for scientific claims (SciFact)
sentence-transformers-all-mpnet-base-v2 ml	—	—	defaults	not onboarded	1	Evidence retrieval for scientific claims (SciFact)	#15 of 19 Evidence retrieval for scientific claims (SciFact)
sshleifer-distilbart-cnn-12-6 ml	—	—	defaults	not onboarded	7	Short news summaries (CNN)	#5 of 29 Short news summaries (CNN)
sshleifer-distilbart-cnn-6-6 ml	—	—	defaults	not onboarded	8	Short news summaries (CNN)	#10 of 29 Short news summaries (CNN)
textattack-bert-base-uncased-ag-news ml	—	—	defaults	not onboarded	3	News topic classification (AG News)	#1 of 30 News topic classification (AG News)
urchade-gliner_medium-v2.1 ml	—	—	defaults	not onboarded	1	Named-entity extraction (Few-NERD)	#26 of 28 Named-entity extraction (Few-NERD)

02what changed in the period

when	model	what
2026-10-31	deepseek-4.1-flash	hosted onboarding: pass
2026-10-31	deepseek-4.1-flash	reasoning_effort (server default) → none (it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all)
2026-10-31	gemma4-31b	x3 onboarding: pass
2026-10-31	gemma4-31b	dgx onboarding: pass
2026-10-31	gemma4-31b	mac onboarding: pass
2026-10-31	gemma4-31b	hosted onboarding: pass
2026-10-31	gemma4-31b	reasoning_effort (server default) → none (it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all)
2026-10-31	gpt-6-luna	hosted onboarding: pass
2026-10-31	gpt-6-luna	reasoning_effort (server default) → none (it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all)
2026-10-31	gpt-oss-120b	x3 onboarding: pass
2026-10-31	gpt-oss-120b	dgx onboarding: pass
2026-10-31	gpt-oss-120b	hosted onboarding: pass
2026-10-31	gpt-oss-120b	reasoning_effort (server default) → none (it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all)
2026-10-31	gpt-oss-120b	reasoning_effort none → low (reasoning cannot be turned off: with none it still reasoned on every machine, and the hosted route answers "Reasoning is mandatory" (onboarding, 2026-10-31); low is the least it takes)
2026-10-31	kimi-k3.1-pro	hosted onboarding: pass
2026-10-31	kimi-k3.1-pro	reasoning_effort (server default) → none (it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all)
2026-10-31	mistral-large-4	hosted onboarding: pass
2026-10-31	mistral-large-4	reasoning_effort (server default) → none (it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all)
2026-10-31	mistral-large-4	reasoning_effort none → low (its provider refused reasoning_effort none (onboarding, 2026-10-31); low is the least it is asked)
2026-10-31	nemotron-3.5-bolt-e4b	x3 onboarding: pass
2026-10-31	nemotron-3.5-bolt-e4b	dgx onboarding: pass
2026-10-31	nemotron-3.5-bolt-e4b	mac onboarding: pass
2026-10-31	nemotron-3.5-bolt-e4b	reasoning_effort (server default) → none (it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all)
2026-10-31	nemotron-3.5-lightning-30b	x3 onboarding: pass
2026-10-31	nemotron-3.5-lightning-30b	dgx onboarding: pass
2026-10-31	nemotron-3.5-lightning-30b	mac onboarding: pass
2026-10-31	nemotron-3.5-lightning-30b	reasoning_effort (server default) → none (it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all)
2026-10-31	qwen3.5-122b	x3 onboarding: pass
2026-10-31	qwen3.5-122b	dgx onboarding: pass
2026-10-31	qwen3.5-122b	hosted onboarding: pass
2026-10-31	qwen3.5-122b	reasoning_effort (server default) → none (it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all)
2026-10-31	qwen3.8-27b	x3 onboarding: pass
2026-10-31	qwen3.8-27b	dgx onboarding: pass
2026-10-31	qwen3.8-27b	mac onboarding: pass
2026-10-31	qwen3.8-27b	reasoning_effort (server default) → none (it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all)
2026-10-31	fleet-triage-pro	hosted onboarding: pass
2026-10-31	fleet-triage-pro	reasoning_effort (server default) → none (it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all)
2026-10-31	claude-haiku-5.5	hosted onboarding: pass
2026-10-31	claude-sonnet-5.5	hosted onboarding: pass
2026-10-31	gemini-3.5-flash-lite	hosted onboarding: pass
2026-10-31	gemini-4-argon	hosted onboarding: pass
2026-10-31	gemma4-12b	x3 onboarding: pass
2026-10-31	gemma4-12b	dgx onboarding: pass
2026-10-31	gemma4-12b	mac onboarding: pass
2026-10-31	gemma4-12b	reasoning_effort (server default) → none (gemma4:12b looped in its reasoning on the triage prompt; with none it answered in 7.7 s)
2026-10-31	gemma4-12b	max_tokens, call_timeout_s 1024, 30 s → (the arms') (a reply cap and a call limit are a task's; the triage arms set them (review E6))
2026-10-31	gpt-6.1-sol	hosted onboarding: pass
2026-10-31	gpt-oss-20b	x3 onboarding: pass
2026-10-31	gpt-oss-20b	dgx onboarding: pass
2026-10-31	gpt-oss-20b	mac onboarding: pass
2026-10-31	gpt-oss-20b	reasoning_effort (server default) → none (it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all)
2026-10-31	gpt-oss-20b	reasoning_effort none → low (reasoning cannot be turned off: with none it still reasoned on every machine (onboarding, 2026-10-31); low is the least it takes)
2026-10-31	llama3.1-8b	x3 onboarding: pass
2026-10-31	llama3.1-8b	dgx onboarding: pass
2026-10-31	llama3.1-8b	mac onboarding: pass
2026-10-31	llama4-scout	x3 onboarding: pass
2026-10-31	llama4-scout	dgx onboarding: pass
2026-10-31	mistral-medium-3.5	x3 onboarding: pass
2026-10-31	mistral-medium-3.5	dgx onboarding: pass
2026-10-31	mistral-medium-3.5	hosted onboarding: pass
2026-10-31	phi4-14b	x3 onboarding: pass
2026-10-31	phi4-14b	dgx onboarding: pass
2026-10-31	phi4-14b	mac onboarding: pass
2026-10-31	qwen3-30b-a3b	x3 onboarding: pass
2026-10-31	qwen3-30b-a3b	dgx onboarding: pass
2026-10-31	qwen3-30b-a3b	mac onboarding: pass
2026-10-31	qwen3-30b-a3b	dgx-vllm onboarding: pass
2026-10-31	qwen3-30b-a3b	hosted onboarding: pass
2026-10-31	ministral-3-14b	x3 onboarding: check
2026-10-31	ministral-3-14b	dgx onboarding: check
2026-10-31	ministral-3-14b	mac onboarding: pass
2026-10-31	nemotron-3.5-super-120b	dgx onboarding: pass
2026-10-31	nemotron-3.5-super-120b	reasoning_effort (server default) → none (it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all)