Designs (RFCs)¶
Purpose¶
An RFC holds the how: the technical design behind a requirement. It captures:
- the problem the design solves, with the evidence;
- the design: data formats, interfaces, algorithms, what each command reads and writes;
- the alternatives considered and why they were set aside;
- what changes for a project pinning the harness, and how it is tested.
An RFC implements a PRD and is referenced by the UX specification of any surface it drives. The key decision it settles is recorded in a few lines as an ADR, which links back here for the detail.
How RFCs work¶
- Reference the PRD: what the user needs comes first (PRD index).
- Design: the RFC is argued while it is Draft and may change freely.
- Build: In Progress while the code lands.
- Completed (vX.Y.Z): the design is built; a later change to it is a new RFC, or a dated note in this one that says what changed and when.
Other statuses: Rejected and Withdrawn, each with the reason kept.
Open RFCs¶
| RFC | Title | Related PRD | Status | Description |
|---|---|---|---|---|
| RFC-019 | Datasets from production: a lane beside the method | PRD-014 | Built (0.6.0) | New commands end in ordinary datasets and references: importers (jsonl, tsv, github, langfuse built in), captures, datasets from a capture, references from outcomes over flat fields, splits; second references silver (ADR-0020); text: hidden on every public surface; stateless; designed on a triage fleet |
| RFC-020 | Onboarding models, once, for every eval | — | Built (0.6.0) | Model records in data/models/ written by evh model-onboard (build per machine, capabilities, settings, a task-independent smoke); arms name a record; evh arm-smoke per arm on its task: runs, valid output, time and a full-run projection, build pinned, determinism, cost; an LLM layer (reasoning, slow, long, cut-off, no answer) when the arm calls an LLM; a catalog Models page and a page per model; advisory validation; designed on the triage and Few-NERD evals |
| RFC-021 | Fingerprint v5: the deployment, the settings, and the runner | — | Built (0.6.0) | The fingerprint split by what each part identifies: instrument, data, deployment (weights, build, server, hardware, host; from a model record, the adapter, or the params), settings (the params that change behaviour), each hashed; the runner recorded outside the hash; model records' builds become deployments, each onboarded; every committed run migrated to v5 |
| RFC-022 | Scoring that evolves: preview it, re-score with it, or adopt it from the next round | — | Built (0.6.0) | A round records the scorer that judged it; evh score-preview shows what a scoring change moves before anything is written; evh rescore --round and round-open --scoring re-score into one place the tool designates (runs-rescored/<scorer>/<source>/); rounds read what the goal reads; a limit may be the baseline, by the paired test |
| RFC-023 | Labels, and the reference sets promoted from them | PRD-002 | Accepted, built (0.6.0) | A label says what is right for one item and who said it (a person, a model, a rule); a reference is labels promoted on purpose, the tier chosen then; evaluating what production did scores against labels and never makes a reference; production's recorded decisions as an arm |
Completed RFCs¶
| RFC | Title | Related PRD | Version | Description |
|---|---|---|---|---|
| RFC-001 | The run fingerprint and code identity (v4) | PRD-001 | 0.4.0 | What a run records it was measured under; code hashed by its syntax tree, so documentation never moves a fingerprint |
| RFC-002 | Run records and derived runs | PRD-001, PRD-002 | 0.4.0 | One record format; derived: {kind, from, at, with} for rescored, judged and re-parsed runs |
| RFC-003 | The catalog, the inspector server and the adapter protocol | PRD-004 | 0.4.0 | One paired test; the questions an adapter answers; the build; the server's local and public modes |
| RFC-004 | Eval rounds | PRD-005 | 0.4.0 | Round records, membership by round:, the four commands, the evidence and its suggestions |
| RFC-005 | Judge scorers and the judging pass | PRD-003 | 0.3.0 | The scorer artifact, the validation gate, the listwise blind pass, its controls, judged runs |
| RFC-006 | Item fields and items at a URL | PRD-006 | 0.4.0 | Field sets and types, fields_sha256 outside the identity, V15, filters and breakdowns, pinned downloads |
| RFC-018 | A read-only MCP server | 0.7.0's PRD-012 (branch 070) |
0.5.0 | evh mcp on the official SDK, stdio and Streamable HTTP; thirteen read commands as tools, each a child process; --public, rate, Host check; --json for the views; the goals' tests on disk; this repository's server at /mcp; Python 3.10 |
Adding one¶
Name it RFC-NNN-short-slug.md (next number, three digits) with Status, Authors,
Related PRDs, Related RFCs and Code, then Abstract, Problem Statement, Design,
Alternatives Considered and Testing (and Open questions while it is a draft). Add its row under
Open RFCs.