Measure your problem.
You do not need our benchmarks. You need a decision to make, a few dozen real inputs, and an afternoon. Write one adapter for your system; the harness brings frozen data, fingerprints, noise, significance and cost — the habits, enforced.
Eight things, one direction.
Every evaluation here — ours and yours — moves through the same eight artifacts. Pick one to see what it is, where it lives and the command that makes it.
Any files: tickets, transcripts, documents, prompts. Keep them under version control — an untracked input makes every run unattributable.
data/sources/ — or anywhereThe comparison contract: an item list with the hash of every item. Two runs on different datasets cannot be compared, and a changed input is a new version.
data/datasets/<id>.json$ make dataset-create DATASET_ID=tickets_v1In the guide →A working copy, re-verified against the frozen hashes every time — so a run can never measure bytes the dataset does not describe.
data/materialized/<id>/$ make dataset-materialize DATASET_ID=tickets_v1In the guide →What each item is scored against. Silver: a model wrote it — consistent, not correct. Gold: a person wrote or checked it. Gold always wins, and every run says which it used.
data/references/<tier>/<id>/$ make reference-create DATASET_ID=… CONFIG=…In the guide →One system under test: provider, model, prompt, settings, in a YAML file. Change one thing, bump the config_id — the fingerprint will prove what moved.
data/configs/*.yamlOne pass of an arm over the dataset: every output, every per-item score, billed cost, latency, and a fingerprint of data, code, model and build. Repeat it to measure noise.
data/runs/<run_id>/$ make experiment-run CONFIG=… REPEAT=3In the guide →The leaderboard with its significance block, paired tests, Holm-corrected families. A claim means separated, not scored higher.
your terminal$ make family-test DATASET_ID=… A=… AGAINST=…In the guide →The run future changes are measured against, recorded with why. Superseded, never edited — in six months, “why is this the baseline?” has an answer.
data/baselines/$ make run-promote RUN=… REASON="…"In the guide →Your question is probably one of these.
Each maps to an experiment in this repository you can copy — its adapter, its configs, its scorer — and change the data. The idea underneath all of them — build a private leaderboard for your own application instead of trusting a public one — is Chip Huyen's, from AI Engineering (influences).
Is the cheaper model good enough?
On DBpedia a 115× price difference bought nothing measurable. Run both on your items; if they tie, the cheap one wins.
Did my prompt change help?
Name the pair before you look, tune on a dev slice, confirm on the held-out rest.
Route or label tickets
A small model trained on your labels can beat every LLM — and part of its lead can be learning your labels' quirks. Read the disputed items.
Search and RAG retrieval
The answer is a ranked list over a corpus; the corpus is hashed into every fingerprint so two runs searched the same haystack.
Pull fields out of text
The answer is a set; matching and parsing are part of the instrument, and an unreadable answer must score zero, not be skipped.
Summaries and generated text
Score against references, clip for length so verbosity cannot buy recall, and check silver against a gold slice.
Catch regressions before release
Promote today's run as the baseline; compare every change against it with the noise floor you measured; validate in CI.
How many test items do I need?
Split your items into disjoint halves and see when two halves agree on the ranking — measured, not taken from a power table.
Seven steps, on a real example.
The tutorial kit ships six support tickets and summaries a person wrote. Every output below is what the command printed.
Freeze a dataset
Your inputs become a contract: an item list with a hash per item. Guide →
$ make dataset-create DATASET_ID=tickets_v1 ARGS="--source-dir tutorial/tickets" data/datasets/tickets_v1.json (6 item(s))
Describe your arms
One YAML per candidate. Change one thing; bump the id. Guide →
$ cp tutorial/configs/tickets_*.yaml data/configs/Draft silver references
Let your best system write first answers — and label them silver. Guide →
$ make reference-create DATASET_ID=tickets_v1 CONFIG=data/configs/tickets_author.yaml data/references/silver/tickets_v1 (6 item(s), tier=silver, $0.0000)
Commit, then measure — three times
The spread is how far an arm disagrees with itself. Guide →
$ make experiment-run CONFIG=data/configs/tickets_lead2_noisy.yaml REPEAT=3 overlap_f1 spread=0.044479 treat deltas below 0.044479 as noise
Compare
Read the significance block, not the row numbers. Guide →
$ make leaderboard DATASET_ID=tickets_v1 pairs distinguishable: 3 of 6
Make it golden
Add answers a person wrote; re-score every run against them for $0. Guide →
$ make rescore DATASET_ID=tickets_v1 REFERENCE=gold/tickets_v1 OUT=data/runs-gold NO detectable arm effect · pairs distinguishable: 0 of 6
Decide and record
Promote with a reason that says what the tests showed; validate in CI. Guide →
$ make run-promote RUN=… REASON="leads on gold but separates from no arm" VALIDATION PASSED
What you write. What you get.
You write
- An adapter —
call_system()sends one input to your system;score()compares an output with a reference. - Your inputs — real ones, sampled on purpose.
- The answers — gold, for a slice you can check; silver from a model you trust for the rest.
- One YAML per candidate — model, prompt, settings, price.
The harness gives you
- Datasets frozen by content hash, re-checked on every use.
- A fingerprint per run: data, prompt, code, model, build.
- Each arm's own noise, measured.
- Paired and Holm-corrected tests — separated or tied.
- Billed cost and latency beside quality.
- Re-scoring from stored outputs for $0; validation for CI; baselines with reasons.
Start on Monday
- Write the question as a decision: can ticket routing move to the cheaper model?
- Collect real inputs; freeze them.
- Write gold answers for a slice yourself.
- Write the adapter; one config per candidate.
- Dry-run; set a cost cap.
- Run each arm three times; note its noise.
- Test the comparison you named; promote with a reason; put
make validatein CI.
Rerun ours, or build yours.
Check every finding yourself.
Five levels, from recomputing every number for $0 to rerunning every model with one key — with a replication card that re-tests every separation.
Your first evaluation, end to end.
Freeze a dataset, build a golden reference set, measure, compare honestly, promote a baseline — offline, with the real output of every step.