Is the ranking real, or a lucky sample?
A small, drop-in harness for evaluating AI systems on your own data. Freeze a dataset, run your system against it, and get the answer most leaderboards skip: whether one system is actually better than another, or only ahead on this sample.
Four questions anyone building with AI ends up asking.
There are dozens of models, a leaderboard for each, and none of them measured on your problem. This harness exists to answer these on your own data — and the experiments are what it answered when pointed at 24 LLMs.
Agentic AI can do months of work in days. How do I trust what it builds?
The harnessFrozen data, a fingerprint on every run, refusals when a comparison is unsound — so an agent can iterate at full speed without a regression slipping through as progress.
What it foundThis project was built that way, with AI agents. A checker re-verifies 82 claims against the runs on every commit, and the lab notebook keeps every finding that had to be retracted along the way.
The lab notebook →Which model is best for what I need — not for someone else's benchmark?
The harnessRun five, ten or twenty-four models on your own frozen data and see quality, cost and latency side by side, with a test that says whether the ranking is real.
What it foundThe most expensive model was not the best in 5 of 5 experiments, and in 4 of 5 first place was a statistical tie.
The experiments →Do I need an LLM — or a GPU — at all?
The harnessPut a small classic model, or a plain baseline, in the same comparison as the LLMs: same data, same scoring, same test.
What it foundA free model that needs no GPU was first or tied for first in 5 of 5 — but only on text like the text it was trained on.
Per experiment →Own it or rent it?
The harnessEvery cost is read off the provider's bill and set beside quality, so renting an API can be weighed against running open weights on hardware you own.
What it foundA proprietary model never separated from the best open-weight model in any of the five experiments, and 64 GB of memory is enough to match the unlimited-hardware answer on four of five tasks.
The synthesis →Four ideas, one adapter file.
Most eval code answers “which scored highest”. That is the easy half; the hard half is whether the answer survives being asked again. The harness imports nothing domain-specific — everything about your system lives in one file, adapter.py, with four functions.
Try the whole loop in a minute.
The demo runs on bundled synthetic data — no API key, no network. It freezes a dataset, runs an arm three times and checks the record; make leaderboard then shows the ranking and whether it is real.
$ git clone https://github.com/chipi/eval-harness $ cd eval-harness/harness $ make install $ make demo $ make leaderboard # then, on your own system $ make dataset-create DATASET_ID=my_v1 $ make experiment-run CONFIG=data/configs/arm_a.yaml $ make leaderboard DATASET_ID=my_v1
The most expensive model never won.
To show the harness doing something real, it measured 24 frontier and open-weight LLMs, small fine-tuned models and baselines on five public benchmarks — 134 runs, $12.05 billed, every cost read off the provider's bill.
CNN/DailyMail
Summarise a news article in two or three sentences. A 1.6 GB fine-tune ties the best paid model — on CNN's articles.
AG News
Four-way topic labels. Three small fine-tunes take the top three places, largely by learning the dataset's labelling conventions.
DBpedia-14
Fourteen-way ontology labels on a saturated benchmark: the top is a group of nineteen, and the gold label is sometimes the error.
Few-NERD
Find and type entities in a sentence. The only clean winner in the repo: a fine-tuned tagger, separated from every rival.
SciFact
Retrieve the abstracts that support a scientific claim. LLM rerankers lead a twelve-way tie; a free dense retriever sits inside it.
The harness enforces a method.
Nothing here needed special access or a big budget. It took six habits — each learned by getting something wrong first — and a harness that enforces them. They work on any problem you can put into a dataset. The framing builds on Chip Huyen's AI Engineering and Designing Machine Learning Systems.
Nothing on this site is typed in by hand. Every figure is recomputed from the committed runs each time the site is built, and every number the prose quotes must appear in the report it comes from, or the build fails. Behind the reports, a checker verifies 82 claims against the runs on every commit.