Architecture decisions (ADRs)¶
An ADR records a key decision in a few lines: the context, the decision, its consequences. The detail lives elsewhere and each ADR links it: what the user needs and why in a PRD, what they see and type in a UXS, and the technical design with its alternatives in an RFC. This list is kept small: only the decisions that shape the architecture.
- Once accepted, an ADR's decision is not rewritten. A later decision that changes it is a new ADR, and the old one is marked superseded, with a link. (On 2026-10-24 the four ADRs below were shortened when the PRD/UXS/RFC process was adopted: their detail moved, unchanged in substance, into the linked documents.)
- Each one names its status: Proposed, Accepted, Superseded or Rejected.
- The code column says whether the code reflects it: Yes, Partial (still landing) or No.
For how the pieces fit together, read the architecture overview.
Index¶
| ADR | Title | Status | Release | Decision | Detail | Code |
|---|---|---|---|---|---|---|
| ADR-0001 | Judge scorers | Accepted | 0.3.0 | Re-measure frozen outputs with LLM judges, refused until validated against human ratings, at least two vendors, blind, every verdict stored | PRD-003, RFC-005 | Yes |
| ADR-0002 | Catalog, run inspector and goals | Accepted | 0.4.0 | One paired test for the CLI and the inspector; the catalog and inspector as a generic core with a project adapter; goals first-class; this repository deployed as a full harness deployment | PRD-004, PRD-005, RFC-003, UXS-001, UXS-002 | Partial |
| ADR-0003 | Item fields and URL items | Accepted | 0.4.0 | Items fetched from a pinned URL; typed fields declared in a reusable field set, outside the dataset identity, values public only by opt-in; filter and per-group tests in the inspector | PRD-006, RFC-006 | Yes |
| ADR-0004 | Eval rounds, and a catalog that guides them | Accepted | 0.4.0 | Optional rounds per goal as files; decisions as local commands, the catalog read-only; suggestions computed from the evidence; the goal page leads with the answer; dataset lineage and the conditions check | PRD-005, RFC-004, UXS-001, UXS-003 | Yes |
| ADR-0005 | One provider layer; credentials per role, never identity | Accepted | 0.4.0 | Arms, judges and the judge panel call hosted models through one layer; a call goes where its provider: says; a judge reads EVAL_JUDGE_<NAME> first, then <NAME>; api_key_env is not part of any fingerprint |
PRD-007 | Yes |
| ADR-0006 | A goal weighs cost, quality and speed; a strategy picks | Accepted | 0.4.0 | Goals declare axes (quality, cost, speed, and the project's own metrics); a strategy decides who the contenders beside the quality leader are (balanced: not significantly different and within margin; quality_first: same quality; frugal: not significantly different); the tie-break picks among the contenders; guardrails are prose rules shown, never computed | PRD-005, UXS-001, UXS-002 | Yes |
| ADR-0018 | The catalog's period is counted from today, in the browser | Accepted | 0.4.2 | Pages carry per-day facts; the browser works the period out from the viewer's date; one builder per page feeds the page and its command (evh datasets --period P, ...); the period travels in the link, nothing remembered |
PRD-004, UXS-001 | Yes |
| ADR-0019 | Datasets from production: a capture lane beside the method | Accepted (references from outcomes: replaced by ADR-0022) | 0.6.0 | New commands (capture, dataset-from-capture, reference-from-outcomes, dataset-split, importer-check) end in ordinary datasets and references; stateless; four built-in importers the core never imports; outcomes become references through a recorded mapping; text: hidden on every public surface |
PRD-014, RFC-019 | No |
| ADR-0020 | A dataset's second reference is silver, with a caveat | Superseded by ADR-0022 | 0.6.0 | The reference a goal decides on is gold/<dataset>; every other one is silver/<name> with a caveat saying what it is, so the goal page shows one recommendation |
RFC-019, PRD-002 | No |
| ADR-0021 | A deployment, not a model, is what is vetted and fingerprinted | Accepted | 0.6.0 | Onboarding, smoke and the fingerprint's deployment part are per deployment; determinism and the build are a deployment's facts; the runner is outside the hash; in process only when the arm calls nothing elsewhere | ||
| ADR-0022 | A reference set is labels you promote; the tier is chosen then | Accepted | 0.6.0 | A label says what is right and who said it (person, model, rule); labels follow the item by its hash; a reference is made only by promoting labels, with a reason, at the tier you choose; reference-create and reference-from-outcomes removed; supersedes ADR-0020 |
PRD-002, RFC-023 | Yes |
| ADR-0023 | Picks wait in one list per project; a governed dataset grows only by a version you make | Accepted | 0.6.0 | Picks (targets, tags, note) in an append-only data/picks.jsonl; dataset-extend with the picks you choose, lineage per item; the pages select and give the command |
PRD-014, RFC-023 | Yes |
| ADR-0024 | Several projects run on one machine, and share nothing | Accepted | 0.6.0 | A stable port per project, derived from its path; a local catalog and catalog-serve name their project; every command notes a stray EVAL_HOME; locks keyed inside the project |
PRD-007, RFC-023 | Yes |
ADR-0002 is Partial until the project it came from runs on the harness's copy (its layer 4) and 0.4.0 is released. ADR-0004 is Yes for the decision as recorded; the parts of its proposal left out of 0.4.0 are listed in RFC-004.
Adding one¶
New work starts with a PRD (PRD index). Write an ADR only for a decision that
shapes the architecture: Status, Date, Detail (links to the PRD, UXS and RFC), then
Context, Decision and Consequences, a few lines each. Number it next (ADR-0005-short-slug.md)
and add its row here.