cnn_llama_s_n200_v1 · judged/judge-summ-v1

01metrics

02cost and speed

cost per item

$0.000111 · 812 tok

speed per item

p50 2,195 · p95 4,062 ms

Per item, over its items: cost as dollars and tokens (a local model is free but still uses tokens), speed as the median and the slow tail (p95). A goal that names this arm weighs them in its decision.

03runs

04config

config cnn_llama_s_n200_v1 · adapter examples/summarization-cnn-dailymail/adapter.py · credentials are never shown

family

llama

max_tokens

1200

model

eval-llama-scout

price_tier

small

prompt_file

prompt.txt

provider

litellm

reasoning

{"enabled": false}

temperature

0.0

usd_per_mtok_in

0.08

usd_per_mtok_out

0.3

05conditions

data.items_sha256

257feda902391b934b8cde940c15e91201096dbe96ec5a81337e9da007ce3ec7

deployment.model

eval-llama-scout

deployment.source

adapter

deployment.upstream

openrouter/meta-llama/llama-4-scout

fingerprint.version

5

library.numpy

2.4.6

library.openai

3.19.2

library.rouge-score

0.1.2

machine.cpu_count

12

machine.platform

Darwin/x86_64

machine.processor

i386

machine.python

3.11.8

machine.release

24.6.0

reference.sha256

5d0b02c3798017535572d24dd3ce0592332938a0c91e93684d163963a60a7cd9

scorer.id

judge-score:judge-summ-v1

scorer.sha256

4b5440bf0599fd12fd2d69a2697f1b0cdf54871486ef228b60be554750e82887

06the same config, scored other ways

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold