arm

cnn_llama_l_n200_v1 · judged/judge-summ-pick-v1

One config on cnn_dailymail_200, scored one way; its repeats averaged per item.

dataset: cnn_dailymail_200
scored against: judge judge-summ-pick-v1
goal: Short news summaries (CNN)
runs: 1
headline: judge_won 0.003
last run: 2026-11-02 16:17

Inspect it With the 29 arms scored the same way

01metrics

metric	mean	items	better
judge_won	0.003	198	higher
cost_usd	$0.000194	198	lower
judge_gemma_l	0.001	198	higher
judge_qwen_s	0.005	198	higher
latency_ms	3474.2	198	lower
reasoning_tokens	0.000	198	neutral
tokens_in	736.0	198	lower
tokens_out	64.2	198	lower
truncated	0.000	198	neutral

02cost and speed

cost per item: $0.000194 · 800 tok
speed per item: p50 3,646 · p95 5,538 ms

Per item, over its items: cost as dollars and tokens (a local model is free but still uses tokens), speed as the median and the slow tail (p95). A goal that names this arm weighs them in its decision.

03runs

run	made	items	cost	how
`cnn_llama_l_n200_v1_20260926T000443Z`	2026-11-02 16:17	198	$0.038365	judge

04config

config cnn_llama_l_n200_v1 · adapter examples/summarization-cnn-dailymail/adapter.py · credentials are never shown

family: llama
max_tokens: 1200
model: eval-llama-maverick
price_tier: large
prompt_file: prompt.txt
provider: litellm
reasoning: {"enabled": false}
temperature: 0.0
usd_per_mtok_in: 0.18
usd_per_mtok_out: 0.6

05conditions

data.items_sha256: 257feda902391b934b8cde940c15e91201096dbe96ec5a81337e9da007ce3ec7
deployment.model: eval-llama-maverick
deployment.source: adapter
deployment.upstream: openrouter/meta-llama/llama-4-maverick
fingerprint.version: 5
library.numpy: 2.4.6
library.openai: 3.19.2
library.rouge-score: 0.1.2
machine.cpu_count: 12
machine.platform: Darwin/x86_64
machine.processor: i386
machine.python: 3.11.8
machine.release: 24.6.0
reference.sha256: 5d0b02c3798017535572d24dd3ce0592332938a0c91e93684d163963a60a7cd9
scorer.id: judge-score:judge-summ-pick-v1
scorer.sha256: 7046021641ded3f501d1f940ca5cbf5cd06fa66131cbca684f2566f1a031844c

06the same config, scored other ways

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

arm	scored against	headline	mean
cnn_llama_l_n200_v1	gold/cnn_dailymail_200 gold	coverage	0.339
cnn_llama_l_n200_v1 · silver	silver/cnn_dailymail_200 silver	coverage	0.370
cnn_llama_l_n200_v1 · judged/judge-summ-v1	judge `judge-summ-v1`	judge_score	4.700
cnn_llama_l_n200_v1 · judged/judge-summ-v2	judge `judge-summ-v2`	judge_score	4.529

07rounds

Round r1 of Short news summaries (CNN), closed: one of its arms.