Documentation · eval-harness

The harness, the method, and the evidence.

How to use eval-harness on your own system, the ideas it is built on, and the reference behind its five experiments: what each measured, every metric, model and test with its source, and how to rerun all of it yourself.

The harness

01Use the harnessYour first evaluation end to end, offline: freeze a dataset, build a golden reference set, measure, compare honestly, promote a baseline.

02ConceptsThe ideas it is built on: frozen datasets, fingerprints, gold and silver references, paired tests.

03IntegrationDrop the harness into another project: one adapter, four functions.

04RunbookEvery step and every verb: datasets, runs, the analysis scripts.

The experiments

05Reproduce itRerun any experiment and check its findings, from recomputing every number for $0 to rerunning every model with one key, and adding your own.

06ProtocolWhat each experiment measured, on which data and metric, and how that relates to the field's standard, with a calibration row each.

07ReferenceEvery metric, model, dataset and statistical test, with sources and licences.

08EvidenceCaptured external facts (model sizes, licences, calibration results), checkable offline.

Background

09Data & licencesWhat of each dataset is in the repository, what you download, each licence, and what it asks of you.

10InfluencesThe approach builds on Chip Huyen's books: idea by idea, with chapters, and the statistics it also stands on.

11PapersThe papers the work cites, and what each is cited for.

The experiment reports live in the repository's research/ directory, and the lab notebook — handovers, review rounds, the journal, every correction — in research/internal/.

← Back to the site