The method · six habits · learned the hard way

How to know what you think you know.

The findings on this site took no special model and no big budget. They took a method — six habits, each learned by getting something wrong first. Here they are as the experiments taught them, with the step in the harness that keeps each one, and what it means for your own work.

01Experiment

An eval is an experiment.

The same 24 hosted models ran on five tasks with one prompt per task, at temperature 0 with reasoning off — so when a model's score differs between two tasks, the task is the only thing that changed.

Every run carries a fingerprint: hashes of the data, the prompt and the scoring code, the model behind the alias, library versions, the commit. Before the harness shows a single delta between two runs, it lists the fields that moved.

The habitChange one thing per arm, and record everything else — so “we only changed the prompt” is checkable rather than claimed.

In the harness

$ make run-compare BASE=… CAND=…
Fingerprints DIFFER. Fields that moved: arm.params.sentences

In your project Your arms are your candidates — prompts, models, retrieval settings, thresholds — one YAML file each.

02Data

The data is the instrument.

The summarisation sample was the first 200 articles of the CNN/DailyMail test set. The test set lists CNN's articles first, so every one was CNN's, while the benchmark is mostly DailyMail. Copying the first three sentences scored ROUGE-1 30.20 on that slice against 39.94 on the full test set.

No check in the harness could see it: every one compared the repository with itself. One calibration row — a known baseline against its published value — caught it. Now every experiment has one, and 7 of 7 match the literature.

The habitFreeze the data and hash it, so a number always means the same items. Then check your sample against the population it claims to represent.

In the harness

$ make dataset-create · make dataset-materialize
a changed input fails loudly instead of silently measuring different bytes

In your project Your dataset is a set of real inputs, frozen. Sample it on purpose — by date, by customer, by difficulty — and write down what it covers.

03Ground truth

Silver is not gold.

In the tutorial, the system that wrote the silver references scores a perfect 1.000 against them and ranks first. Re-scored against six summaries a person wrote, it falls to second — and no arm can be told apart from any other (p = 0.2124).

On AG News, nearly the whole lead of the three fine-tunes sits in 17 items whose gold labels most of the hosted field disputes. Read, many of those labels were wrong. Disputed gold is a finding, not noise.

The habitSilver gets you started; gold tells you the truth. Write a small gold set early, check silver against it, and read every item where strong systems agree against the gold.

In the harness

$ make rescore REFERENCE=gold/… · make silver-calibrate
re-score old runs against new gold for $0, then measure how far silver drifts

In your project A small set of answers written by someone who knows the domain is worth more than a large set a model generated: it is the only one that can tell you the model is wrong.

04Noise

Measure the noise before the difference.

BART's apparent lead at n=20 was 0.0737; across 200 articles it was 0.00008. Two independent 100-article evals of the same models agree at only ρ = 0.79.

A pilot correlates beautifully and picks the wrong winner — in three of five experiments here.

The habitRun every arm more than once and measure how far it disagrees with itself. Tune on a small dev slice, report on the rest, and measure how many items a ranking needs instead of assuming it.

In the harness

$ make experiment-run REPEAT=3 · make holdout · make rank-stability
treat deltas below 0.044479 as noise

In your project Before you believe the new prompt is a few points better, run the old one twice.

05Significance

Tested, not read.

A leaderboard sorts, so something is always first. In 4 of 5 experiments here the leader cannot be separated from a group of others — on summarisation, from 15 of them, the cheapest paid one 143× cheaper than the dearest model in the field.

Every claim on this site rests on a paired permutation test per item, with a Holm correction for the number of comparisons made at once.

The habitDecide the comparisons before reading the table, correct for how many you made, and report a tie as a tie.

In the harness

$ make family-test · make leaderboard
separated from 6 of 18 opponents   ← SciFact's leader, glm_s

In your project When the answer is “tied”, the cheapest member of the tie is the decision.

06The record

Decide on the record.

Three of these — the parser, the unparsed-as-empty rule, the reasoning flag — moved a single arm by more than the gap separating most of the field. The scoring code is a model-sized lever.

Every correction is kept: the lab notebook records the findings that were retracted along the way, several caused by this harness's own scorer.

The habitHash and test the scorer, re-score stored outputs when it changes, and promote a baseline only with a reason that says what the tests showed.

In the harness

$ make rescore · make run-promote REASON="…" · make validate
re-scoring is $0; validation exits non-zero, so it belongs in CI

In your project Keep a journal. The mistakes you write down are the checks you add next.

07Standing on

Built on Chip Huyen's framing.

The approach in this repository builds on Chip Huyen's books. Their framing — evaluation is engineering work on your data, a model is chosen on a private leaderboard rather than a public one, the dataset is a design decision, every experiment is tracked and versioned — is the foundation this harness was built on. What this project adds is a tool that enforces it, and five experiments that show what skipping it costs.

The idea-by-idea mapping, with chapters, is in Influences.

Chip Huyen — AI Engineering (O'Reilly, 2025). Ch. 3–4: why public benchmarks rarely pick the model for your application, private leaderboards, AI judges as subjective evaluation to be checked against exact and human evaluation. Ch. 8: dataset engineering — quality, coverage, quantity.
Chip Huyen — Designing Machine Learning Systems (O'Reilly, 2022). Ch. 4: sampling and labelling. Ch. 6: baselines, experiment tracking and versioning. Ch. 8: distribution shift.
Martin Zinkevich — Rules of Machine Learning (Google). Design and implement your metrics first.
Breck et al. — The ML Test Score (2017). Tests for data, models and infrastructure.
Bernardi et al. — 150 Successful Machine Learning Models (KDD 2019, Booking.com). Offline gains do not automatically become real ones.
More, curated by Chip Huyen: huyenchip.com/mlops

Use it on your problem →The step-by-step guide The experiments

Who built this

Built by Marko Dragoljevic.

I have led engineering teams for twenty years — at ServiceNow, EclecticIQ and Backbase — and through Human Copilot I now advise technology leaders. Which model, build or buy, own or rent: the questions this harness answers are the ones I help teams answer, on their own data.

Work with Human Copilot →DevWhateverOps: principles for building end to end ↗