eval-harness

model

Gemma 4 31B llm

A model record: onboarded once for the project and used by any eval's arms.

weights
open
licence
Apache-2.0
aliases
gemma4:31b gemma4:31b-it-q4_K_M eval-gemma-31b
settings
reasoning_effort none it reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all
onboarding
vetted 2026-10-31
best result per goal
#2 of 33 A scorer that agrees with experts on summaries (SummEval)
#6 of 19 Evidence retrieval for scientific claims (SciFact)
#9 of 28 Named-entity extraction (Few-NERD)
#13 of 28 Encyclopedia topic classification (DBpedia)
#14 of 30 News topic classification (AG News)
#27 of 55 Triage decisions on Orrery's error signals
#16 of 29 Short news summaries (CNN)

Every model

01deployments and their onboarding

x3 · x3 · ollama 0.40.13 · gemma4:31b · onboarding pass 2026-10-31 · 20.1 tokens/s
where
x3
server or device
ollama 0.40.13
build
gemma4:31b
17ba34c06c801ba22f6047bab21ced9429c6d43e0e42ecf339b0f3e5a61e8d97
Q4_K_M
matches the record
yes
checked
2026-10-31
onboarded on
x3
probemeasurementflag
endpointprovider openai · base url http://127.0.0.1:11434/v1 · model gemma4:31b—
digest found17ba34c06c801ba22f6047bab21ced9429c6d43e0e42ecf339b0f3e5a61e8d97—
defaults 14.7 · tokens out 193 · reasoned: yes · finish stop—
chosens 1.15 · tokens out 23 · reasoned: no · finish stop · settings (reasoning effort none)—
tokens per s20.1—
deterministicyes—
json validyes—
dgx · dgx · ollama 0.40.13 · gemma4:31b · onboarding pass 2026-10-31 · 23.5 tokens/s
where
dgx
server or device
ollama 0.40.13
build
gemma4:31b
17ba34c06c801ba22f6047bab21ced9429c6d43e0e42ecf339b0f3e5a61e8d97
matches the record
yes
checked
2026-10-31
onboarded on
dgx
probemeasurementflag
endpointprovider openai · base url http://100.69.49.126:11434/v1 · model gemma4:31b—
digest found17ba34c06c801ba22f6047bab21ced9429c6d43e0e42ecf339b0f3e5a61e8d97—
defaults 14.42 · tokens out 193 · reasoned: yes · finish stop—
chosens 0.98 · tokens out 23 · reasoned: no · finish stop · settings (reasoning effort none)—
tokens per s23.5—
deterministicyes—
json validyes—
mac · mac · ollama 0.40.13 · gemma4:31b-it-q4_K_M · onboarding pass 2026-10-31 · 11.5 tokens/s
where
mac
server or device
ollama 0.40.13
build
gemma4:31b-it-q4_K_M
17ba34c06c801ba22f6047bab21ced9429c6d43e0e42ecf339b0f3e5a61e8d97
Q4_K_M
matches the record
yes
checked
2026-10-31
onboarded on
mac
probemeasurementflag
endpointprovider openai · base url http://127.0.0.1:11435/v1 · model gemma4:31b-it-q4_K_M—
digest found17ba34c06c801ba22f6047bab21ced9429c6d43e0e42ecf339b0f3e5a61e8d97—
defaults 24.35 · tokens out 193 · reasoned: yes · finish stop—
chosens 2.0 · tokens out 23 · reasoned: no · finish stop · settings (reasoning effort none)—
tokens per s11.5—
deterministicyes—
json validyes—
hosted · homelab LiteLLM (the fleet's gateway) · eval-gemma-31b · onboarding pass 2026-10-31 · 17.9 tokens/s · 2 accepted
where
homelab LiteLLM (the fleet's gateway)
build
eval-gemma-31b
matches the record
yes
checked
2026-10-31
onboarded on
homelab LiteLLM (the fleet's gateway)
probemeasurementflag
endpointprovider litellm · base url http://homelab:4001/v1 · model eval-gemma-31b—
digest foundNone—
defaults 1.41 · tokens out 23 · reasoned: no · finish stop—
chosens 1.28 · tokens out 23 · reasoned: no · finish stop · settings (reasoning effort none)—
tokens per s17.9—
deterministicyes—
json validyes—

Accepted: where it ran: a hosted gateway serves the provider's current build; there is no digest, revision or hash to pin; determinism: a hosted provider batches requests, so temperature 0 does not repeat exactly; repeats measure it

What it can do, across its deployments: reasons: no · json mode: yes · reasoning off reasoning_effort: none

02settings history

whenwhat changedwhy
2026-10-31reasoning_effort: (server default) → noneit reasoned by default on the onboarding probe (2026-10-31); off unless a task needs it (RUNBOOK §16): left at the server's default, one model reasons at length and the next not at all

03arms

armdeploymentgoalits settingsarm smokeresult
ag_gemma_l_n200_v1 · adjudicated
ag_news_200
hosted—max_tokens 200, temperature 0.0nonecorrect 0.955
cnn_gemma_l_n200_v1 · judged/judge-summ-pick-v1
cnn_dailymail_200
hosted—max_tokens 1200, temperature 0.0nonejudge_won 0.001
cnn_gemma_l_n200_v1 · judged/judge-summ-v1
cnn_dailymail_200
hosted—max_tokens 1200, temperature 0.0nonejudge_score 4.715
cnn_gemma_l_n200_v1 · judged/judge-summ-v2
cnn_dailymail_200
hosted—max_tokens 1200, temperature 0.0nonejudge_score 4.603
se_gemma_l_pubtext_n100_v1 · pair
summeval_100
hostedA scorer that agrees with experts on summaries (SummEval)max_tokens 1500, temperature 0.0nonetau 0.551
se_gemma_l_pubtextcw_n100_v1 · pair
summeval_100
hostedA scorer that agrees with experts on summaries (SummEval)max_tokens 1500, temperature 0.0nonetau 0.540
triage_hosted_gemma4_31b_v1 · rescored 5f2c057c
orrery_triage_v1_labelled
hostedTriage decisions on Orrery's error signals—nonetriage_cost 0.723
triage_hosted_gemma4_31b_v2 · rescored 5f2c057c
orrery_triage_v1_labelled
hostedTriage decisions on Orrery's error signalsmax_tokens 1024pass 2026-10-31triage_cost 0.723
triage_local_dgx_gemma4_31b_v1 · rescored 5f2c057c
orrery_triage_v1_labelled
dgxTriage decisions on Orrery's error signalscall_timeout_s 600nonetriage_cost 0.677
triage_local_dgx_gemma4_31b_v2 · rescored 5f2c057c
orrery_triage_v1_labelled
dgxTriage decisions on Orrery's error signalsmax_tokens 1024, call_timeout_s 600pass 2026-10-31triage_cost 0.723
triage_local_mac_gemma4_31b_v1 · rescored 5f2c057c
orrery_triage_v1_labelled
macTriage decisions on Orrery's error signalscall_timeout_s 600nonetriage_cost 0.677
triage_local_mac_gemma4_31b_v2 · rescored 5f2c057c
orrery_triage_v1_labelled
macTriage decisions on Orrery's error signalsmax_tokens 1024, call_timeout_s 600pass 2026-10-31triage_cost 0.800
triage_local_x3_gemma4_31b_v1 · rescored 5f2c057c
orrery_triage_v1_labelled
x3Triage decisions on Orrery's error signalscall_timeout_s 600nonetriage_cost 0.692
triage_local_x3_gemma4_31b_v2 · rescored 5f2c057c
orrery_triage_v1_labelled
x3Triage decisions on Orrery's error signalsmax_tokens 1024, call_timeout_s 600check 2026-10-31triage_cost 0.708
cnn_gemma_l_n200_v1 · silver
cnn_dailymail_200
hosted—max_tokens 1200, temperature 0.0nonecoverage 0.342
ag_gemma_l_n200_v1
ag_news_200
hostedNews topic classification (AG News)max_tokens 200, temperature 0.0nonecorrect 0.870
cnn_gemma_l_n200_v1
cnn_dailymail_200
hostedShort news summaries (CNN)max_tokens 1200, temperature 0.0nonecoverage 0.320
db_gemma_l_n200_v1
dbpedia_280
hostedEncyclopedia topic classification (DBpedia)max_tokens 200, temperature 0.0nonecorrect 0.975
fn_gemma_l_n200_v1
few_nerd_280
hostedNamed-entity extraction (Few-NERD)max_tokens 600, temperature 0.0nonef1 0.648
sf_gemma_l_n200_v1
scifact_200
hostedEvidence retrieval for scientific claims (SciFact)max_tokens 700, temperature 0.0nonendcg_10 0.736
se_gemma_l_n100_pairwise_v1
summeval_100
hosted—max_tokens 1500, temperature 0.0nonepair_agreement 0.825
se_gemma_l_n100_pick_v1
summeval_100
hosted—max_tokens 1500, temperature 0.0nonetop1 0.364
se_gemma_l_n100_v1
summeval_100
hostedA scorer that agrees with experts on summaries (SummEval)max_tokens 1500, temperature 0.0nonetau 0.538