Skip to content

RFC-019: Datasets from production: a lane beside the method

  • Status: Built for 0.6.0 (2026-10-31). Drafted third 2026-10-30: brought in line with the engineering plan after three independent reviews; accepted when the build lands. ADR-0020 (second references are silver) is the owner's call.
  • Replaced in part: references from outcomes (reference-from-outcomes, and second references as silver) by RFC-023: labels from a capture's outcomes are a rule label set, and a reference set is labels you promote.
  • Authors: Marko Dragoljevic
  • Related PRD: PRD-014; also PRD-006 (versions, item fields), PRD-002 (references)
  • Related ADRs: ADR-0019, ADR-0020
  • Related RFCs: RFC-006 (field sets), RFC-018 (the MCP server); on branch 070: 0.7.0's RFC-013 (declarations, consent), RFC-015 (validation and test, test looks), RFC-016 (MCP tools)

Abstract

A project improves a system by measuring it on what the system actually meets. This RFC adds a lane beside the method:

  • capture records from production systems with an importer;
  • optionally scrub them with the project's own scrubber;
  • select items from a capture into a new dataset version;
  • take a reference from outcomes, what production recorded afterwards;
  • split a dataset into validation and test.

The lane ends in the files the method already reads (a dataset file, a reference file), so materialize, run, compare and goals work as they are. Every step is a command that reads and writes files: no server, no state outside the project, no schedule. A dataset can also be labelled so that no text derived from it appears on any public surface.

Problem

  • A dataset is frozen from a directory of files or from pinned URLs (RFC-006). Records that live in a tracing system, a ledger file or an issue tracker reach a dataset only through a script each project writes. Nothing records what was read, which items were chosen and why, or links back to the record.
  • The case this is designed on: a triage fleet. An agent reads signals from observability (error-tracker issues, alerts), runs up to three probes against logs, metrics, traces and the source's state, and makes one decision per signal occurrence: dismiss, file an issue, escalate to a person, or clean up. Deterministic code before it (gates) and after it (filing and de-duplication) shapes what reaches GitHub. What it leaves behind:
  • every decision is a row in a ledger;
  • every decision is now captured with the probe answers the model saw;
  • every issue is closed by a person, with a comment that says what it was: fixed, or noise.
  • It has hand-labelled frozen fixtures and a scorer of its own. What it lacks is what the harness has: versions, references kept apart from items, paired tests, repeats, goals, a catalog.
  • And what neither has: a reference taken from what happened next.

Design

0. The lane and its boundary

  • New commands only. A project that never uses them sees no change.
  • The lane writes ordinary files: a dataset with lineage, and a reference. Runs, comparisons, goals and pages read them like any other.
  • What existing files gain is optional and outside identity:
  • a dataset's tags (the dataset-level form of items[].tags), made_from and text;
  • a reference manifest's from_outcomes.
  • Stateless. "Continuous" is running a capture again: --since-capture continues from an earlier one. A schedule and a server are later questions.

1. Importers

  • The interface. An importer is a class with records(query) -> Iterator[Record]. A record is {id, body, fields, created_at, source: {system, url}, labels, outcome}.
  • Ids are file-name safe, at most 200 characters. They may contain :, so captures are not supported on Windows.
  • Where an importer comes from:
  • a built-in name: jsonl, tsv, github or langfuse, in evaluation_harness/importers/, which the core never imports. They use the standard library and gh only;
  • <data>/importers/<name>.py:Importer;
  • or path.py:Class.
  • github reads issues by GraphQL through gh, and records each close and reopen event with its actor and the comment beside it. A fleet that writes with its owner's token can only be told from a person by its text, and a person's close comment carries the verdict.
  • langfuse reads traces. Its keys come from the environment only, and it records a path-only URL.
  • Reading a project's production stores is the project's own code (the fleet's build script). The lane reads what that code exports.
  • evh importer-check is the contract test: ids unique and well-formed, records well-formed, times parsed, hashes stable, and the window kept (every record inside the query's [since, until); with no since, a read from the middle record's time returns nothing older).

2. Captures

  • evh capture --importer NAME --query JSON --capture C writes two things:
  • data/captures/<C>/capture.json: the importer, its code's sha256, the query verbatim, the window, when, and per record its id, link, sha256 (the whole normalised record) and body_sha256;
  • records/<id>.json.
  • It never overwrites without --force.
  • A sample is the capture's: --sample-rate R --seed S keeps a record when the seeded hash of its id falls below R, as the records arrive, so a source far larger than the sample (a billion log lines a day) is never held at once. The importer receives the rule as its sample option and may sample at the source; the capture checks every record against the rule, records it (sample: rate, seed, read, kept), and a continued capture keeps it.
  • evh capture-scrub C --command "... {records}" runs the project's own scrubber, re-hashes, and records the command and its files' hashes.
  • evh captures lists the captures. V17 checks every record against its hash.
  • Commit before running. Untracked captures and sources mark runs dirty, so they are committed before any billed run.

3. A dataset version from a capture

  • The command: evh dataset-from-capture C --dataset-id D [--from D_prev] [--where F:V] [--dedup exact|text] [--one-per F] [--fieldset FS --field F] [--tag k=v]. It selects; it does not sample (the capture does).
  • Only the body becomes the item, written to data/sources/<D>/, so an outcome never enters an item's bytes.
  • The dataset is written through dataset-create's own functions and records made_from.
  • --one-per F keeps one record per value of a field, which gives independent items when many records share one fingerprint.

4. A dataset whose text is never shown publicly

  • text: hidden. No text that comes from such a dataset reaches a public surface: its items' sources, every arm's outputs on it, its references' texts and per-item labels, judge reasons, adjudication notes, source URLs, and its text and URL field values.
  • One function answers "may this text be shown publicly". These surfaces ask it:
  • the catalog;
  • the inspector's API;
  • the MCP tools;
  • evh dataset-items and evh inspect under EVH_PUBLIC;
  • and a deployment's data copy, which moves from render.sh into a Python module.
  • A marker test fails if any surface leaks.
  • It also closes two holes. The MCP inspect --where/--by hole on hidden fields was closed in 0.6.0's first part. A profile written --with-text is now checked again whenever the catalog builds.

5. Reference sets from outcomes

  • The command (removed in 0.6.0 by RFC-023: evh label-from-capture reads the same rules into a rule label set): reference-from-outcomes --dataset-id D --capture C --map outcomes.yaml [--name NAME].
  • The mapping is an ordered list of rules over the flat outcome fields the project's importer computes. Each rule has when, unless, exists, absent, range (signed), min_age_days, and a label or skip. The first match wins; no match leaves the item unlabelled. A real mapping's time arithmetic and list logic are the project's own semantics, computed into fields, so the language stays small.
  • The reference records from_outcomes: {capture, mapping, mapping_sha256, matched, skipped, unmatched}, and its own caveat: "OUTCOME: what production recorded afterwards, as of ; not a person's label".
  • It never overwrites a reference.
  • Outcomes live in the capture. Reading them again later makes a newer capture and a newer reference, never an edit.
  • Which reference decides (ADR-0020, accepted): a goal decides on gold/<D>. Every other reference on the dataset is silver/<name>, with its caveat, and is read through rescore --reference-id. The goal page then shows exactly one recommendation, and the others beside it.
  • Coverage is recorded as today. A scorer leaves the key out on an unlabelled item.

6. Splits

  • The command: evh dataset-split D --test 0.3 [--by hash|time] [--key F] --seed S writes D_validation and D_test, ordinary datasets with lineage.
  • By hash of the key (item id by default, or a field, so one fingerprint never lands on both sides): stable across versions.
  • By time: the newest share is test.

7. The catalog, the CLI and MCP

  • The datasets page groups and filters on tags, says where a dataset came from, and puts a version's profile beside its parent's.
  • evh captures, and the read-only MCP tool of the same name.
  • The writing commands are not MCP tools until 0.7.0's declarations and consent.

8. The triage fleet, end to end (the acceptance case)

The data, the references, the arms and the cost are in the application plan (research/internal/PLAN_0.6.0_FLEET.md).

  1. The decisions dataset. The fleet's 236 Orrery decisions, each a signal plus the probe answers as of its time, labelled by how much evidence survives (33 exact; 91 with only the signal).
  2. The adapter replays each decision through the fleet's own investigate(signal, probe_table), with the arm's model. It disables the fleet's own tracing and tags every call as an evaluation. The fleet's code comes from a pinned checkout.
  3. The references:
  4. the owner's verdicts decide (gold/): the 42 issues, fixed or noise, as their close comments say;
  5. the outcome mapping and the old fixture labels are shown as silver/.
  6. The goal. As first written, false_dismiss was the primary measure with no limits. As run (owner, 2026-10-30): triage_cost decides (a missed real problem costs 5 false alarms; _p3 and _p10 beside it), with a ceiling on false_alarm at production's 4-pass mean (21.25 of 31); cost and speed break ties.
  7. The de-duplication dataset: the fleet's 100 signals through its pure dedup functions, no model, a regression guard.
  8. Continuous: each month's decisions, captured exactly as the model saw them.

What round 1 showed, and did not (2026-10-31; the numbers in research/internal/TRIAGE_RESULTS_LIVE.md, its table generated from the harness). The lane works end to end: capture, dataset, gold, every arm replayed, scored and compared in the catalog. The comparison decides nothing yet: on 65 decisions the paired test finds no arm under the ceiling significantly different from the leader (evh goals names all 39 as contenders, the always-dismiss stub among them). The ceiling was set after the results and sits within one or two false alarms of 18 rows. 20 of the 65 gold labels are production's own decision, accepted by the owner after a model of the Anthropic family agreed with it. A no-decision replay scores as an escalation, a path production does not have. How large the gold must be, and how a no-decision is scored, are settled by the owner before round 2.

Settled 2026-10-31, and applied as round 2 of the goal, the scoring change RFC-022 makes a step of its own: a no-decision is wrong on both sides, and the false-alarm ceiling is no worse than production (the baseline) by the paired test, each arm on its own. Re-scored at $0, round 1's runs still show no significant difference (the three gpt-oss:20b rows fall from the top; two arms are ruled out on false alarms; 54 stay contenders). The next round grows the gold to the owner's 93 labelled decisions and runs the onboarded models.

Not in scope

A schedule, a server, state outside the project's files, a labelling interface, holding a project's private data, write tools for agents (0.7.0).

Alternatives considered

  • One flow, not a lane (the first draft). Rejected: it changes what exists for everyone.
  • The harness keeping records private (the first draft). Rejected: what to commit is the project's decision.
  • Vendor importers only as examples. Rejected for github and langfuse: one install does everything.
  • GitHub's close reason as the outcome. Rejected on the data: COMPLETED on 37 of 41 closes, whatever the close comment said.
  • A mapping language with paths, signed time windows and list quantifiers. Rejected: the project computes flat fields, and the language stays small.

Changes from the second draft

The changes, listed in the engineering plan's P10:

  • importers come from <data>/importers/ or path:Class, not config.yaml;
  • as_of was dropped, since outcomes live in the capture;
  • outcomes are read from the close comment, not the close reason;
  • the mapping runs over flat fields;
  • second references are silver/;
  • labels became tags;
  • the fleet's v1 is all 236 decisions, kept by evidence level, rather than 12 fixtures or "older decisions accepted as lost";
  • the deciding reference is the owner's verdicts.

Changes while building (0.6.0)

What the build changed or added beyond the third draft, each in the code and its tests:

  • The fleet's v1 is 233 decisions (the 236 with the 3 test signals removed), published on Hugging Face (chipi-nbgd/orrery-triage@v1, private until the owner reads it) and fetched by a sha256-pinned revision; git holds the ids and hashes. A capture's records may be kept out of git: V17 names such a capture as not fetched, and still fails one with some records missing.
  • How the owner's verdicts reach a decision: an issue's verdict labels the decision that filed it and earlier decisions on the same fingerprint (a missed problem). It does not label decisions made while the issue was open (finding duplicates is the filing stage's job) nor issues linked only by title (one generic title matched 37 fingerprints). A fixed issue gives a label accepting file and escalate: both were right.
  • The exact decisions no issue verdict reaches were labelled with a labeller model's help: it proposed a label with a reason, judging from what the agent could see and using hindsight only for what the signal was; the owner decided every row where it disagreed with production. verdicts.jsonl records the source of every label.
  • An outcome reference never recommends, with gold beside it or with no gold at all: the build had recommended from it when no gold existed, against ADR-0020; fixed, with a test.
  • text: hidden reaches the deployment's copy: _release.py copy-data replaces render.sh's copy list and refuses a release carrying untracked reference texts or run outputs.
  • Item ids may contain :, @, + and = (a record's id is often a fingerprint and a time).
  • Dataset tags on the views: the datasets page groups by tag key and names a capture-made dataset's origin; evh datasets --tag filters.
  • A capture keeps the window it records (found by the first live continuation, 2026-11-01: the triage importer ignored since, and a capture that said it began on 29 October held three test signals from August). evh capture names every record outside [since, until) and writes nothing; --late keeps records older than since that the earlier capture never saw, listed in capture.json as late. The triage importer now keeps to since/until and leaves the fleet's test signals out unless the query says "test_signals": true.
  • Newer outcomes are a fresh capture, not a new mechanism. A continued capture skips the records it holds, so their outcomes stay as captured. reference-from-outcomes --capture already reads any capture by item id: a fresh capture of the same query gives the same items today's outcomes, under a new reference name. On the fleet, three days after v1, it labelled 7 more of 233 decisions and changed none.
  • Sampling moved from the dataset to the capture (owner, 2026-11-01): dataset-from-capture --sample N --by F --seed S drew from a capture that already held everything, which cannot work at a billion records a day. evh capture --sample-rate R --seed S samples as the records are read, by a hash of each id, so the same rule draws the same records again; the importer may apply it at the source.
  • The catalog's Captures page (captures.html, under Data): each capture with its records, importer, record times, what it continues, and the datasets made from it; a dataset made from a capture links its origin there.
  • The triage example scores the model's own decision beside the fleet's (model_* metrics, before the fleet's gates): its dismiss gate turned 39 of production's 49 escalations from dismissals, so the fleet's score alone would measure the gate more than the model.

Testing (as built)

  • test_capture.py: importers, importer-check and capture (the contract, hashes, continuing), dataset-from-capture (selection, fields, tags), capture-scrub, V17 (an edited record fails; records not fetched are named; some missing fails).
  • test_importers.py: tsv, github (close events with their comments) and langfuse, on offline stand-ins.
  • test_outcome_reference.py: the rule language, never overwritten, exactly one recommendation per goal, the outcome group shown and not recommended, with gold and without; the datasets views' tags.
  • test_split.py: the share, stable across versions, groups kept whole, by time.
  • test_hidden_text.py: a shown and a hidden twin with their own marker words on every public surface; six mutations of the guards each fail it.
  • examples/triage-fleet/test_triage.py and examples/signal-fleet-dedup/test_dedup.py, in make ci: the scorers, the importer on the fleet's export, the adapters through the fleet's own code where it is present.
  • The triage example ran end to end with 10 hosted arms on the 65 owner-labelled decisions (the screen, 2026-10-30); the finals, the open-weight models' hosted twins and their self-hosted runs are added as they finish.