UXS-003: The command line and the first evaluation¶

Status: Active
Authors: Marko Dragoljevic
Related PRDs: PRD-001 (evals), PRD-007 (one install), PRD-004 (parity), PRD-005 (rounds)
Related RFC: RFC-004 (the round commands)
Related UX specs: UXS-001, UXS-002
Implementation paths: harness/scripts/cli.py (COMMANDS: every command, one line of help each), harness/Makefile (the same scripts as make targets), docs/GUIDE.md (the onboarding path, run as written by make guide-check)

Summary¶

The harness's main surface is the terminal. One command, evh, with one subcommand per verb; evh --help lists them and --help after any command gives its flags. In a checkout of this repository each is also a make target running the same script. This spec covers the order in which a newcomer meets the commands, how every command speaks (refusals, hints, exit codes), the parity table that maps every catalog view and action to its command, and the flow of an eval round.

Principles¶

The terminal can do everything the pages can. The pages are a comfort (UXS index, rule 2).
Refuse loudly, then say what to do. A refusal names what is wrong and, where there is one, the command that fixes it.
The next step is printed. A command that leads somewhere ends with the command to run next, in the form the user has: an evh command through evh, a make target in a checkout.
Nothing billed by surprise. Paid commands have a --dry-run, a cost cap, and a printed --resume.

Scope¶

In scope: the onboarding order, command conventions, the parity table, the round flow.

Non-goals: a TUI or interactive prompts; shell completion; colour in the output.

The first evaluation (onboarding)¶

The GUIDE is the onboarding path, offline and keyless, in the user's own project. Every command in it was run as written, and make guide-check runs it again in a fresh project on every change, comparing each quoted output. Its order is the order a newcomer meets the ideas:

step	what the user does	commands
0	set up a project with the tutorial kit	`pip install …`, `evh init --tutorial`
1	freeze a dataset	`evh dataset-create --dataset-id tickets_v1 --source-dir tutorial/tickets`
2	materialize	`evh dataset-materialize --dataset-id tickets_v1`
3	describe the arms, name the goal, commit	copy configs, `goal: tickets_summary`
4	run the best arm first (a dry run prices it)	`evh run --config data/configs/tickets_author.yaml [--dry-run]`
5	optional: make its outputs the reference (its outputs become a model's labels, which you promote)	`evh label-from-run --run <run_id> --set author_v1`, then `evh reference-promote --dataset-id tickets_v1 --labels author_v1 --tier silver --reason "…" --by <you>`
6	run the other arms, three times each	`evh run --config … --repeat 3`
7	compare	`evh leaderboard`, `evh pair-test`, `evh family-test`
8	choose: promote a baseline, then validate	`evh run-promote --run … --reason "…"`, `evh runs-list --baselines`, `evh validate`

The tutorials take it on from there: a local model and its own adapter (1), a hosted model with the user's own key (2), gold, silver-calibrate and an LLM judge (3, run by make guide-check), rounds and dataset versions (4, run), and the catalog, the inspector and fields (5, run).

Goals appear at step 3 and rounds only in tutorial 4: a newcomer can ignore both. Deeper material is tiered behind the GUIDE: Concepts (why), Runbook (every flag), Integration (your own system), Scenario (a newcomer's trial with hosted models).

How a command speaks¶

Refusals: ERROR: <why> on stderr and exit 1, e.g. ERROR: no goal 'x': one of … or "round r1 of g is still open: close it first (evh round-close …)". An unknown command or argument prints the usage and exits 2.
Exit codes (Runbook: exit codes): 0 done; 1 refused or nothing measured; 2 a usage error, or a run stopped by the cost cap with its partial results kept and resumable; 3 a judge scorer not validated.
Hints: a command that creates something ends with Next: <command>, e.g. after dataset-create: Next: evh dataset-materialize --dataset-id tickets_v1. Through evh the hint names an evh command; in a checkout, a make target.
Notes: caveats the user must read before the numbers are printed as NOTE: … lines (a filtered comparison, a spend against a budget, a rounding warning).
Tables: plain aligned columns with a dashed rule, no colour, so output can be pasted and diffed.
Help: evh --help lists every command with one line each (from cli.py's COMMANDS); --help after a command lists its flags. evh --version prints the version.

The parity table¶

Every view and action in the catalog and the inspector, and the command that gives the same answer. Where the page has a make target in a checkout, it runs the same script.

in the catalog or inspector	command	make target
home: the tiles with their trends, goals at a glance, activity and spend, coverage, what was recorded	`evh overview [--period P]`	`make overview`
the activity page: runs, spend and tokens by dataset, arm and kind	`evh activity [--period P] [--arms N]`	`make activity`
any page's period (the chips): its in-period columns, new rows, trends	`--period P` on `evh datasets`, `goals`, `references`, `metrics`, `arms`	`ARGS='--period 30d'`
the datasets page	`evh datasets`	`make datasets`
a dataset page: arms with their cost and speed, baseline, versions, references	`evh datasets --dataset-id <id>`	`make datasets DATASET_ID=<id>`
a dataset page's what's inside: items, words, the gold's shape, each item's row at its source	`evh dataset-items --dataset-id <id> [--item ID]` (written by `evh dataset-profile`)	`make dataset-items DATASET_ID=<id>`
the goals page	`evh goals`	`make goals`
a goal page: the answer so far, arms per dataset and group (a judge's group on its score), retired marks, one line per round	`evh goals --goal <id>`	`make goals GOAL=<id>`
a goal page: its rounds and suggestions (each suggestion's copy button copies its command)	`evh rounds --goal <id> [--round rN]`	`make rounds GOAL=<id>`
a goal page: a round's inspect button	`evh inspect --round rN [--goal <id>]`
the arms page: every arm, grouped, filtered (local / hosted, price tier)	`evh arms [--by dataset\\|config\\|model\\|provider\\|scored] [--where local\\|hosted] [--tier T]`	`make arms`
the arms page: the configs, with their rank on each goal	`evh arms --configs`	`make arms ARGS=--configs`
an arm page: metrics, cost and speed, runs, config, conditions, rounds	`evh arms --arm <arm id or config id>`	`make arms ARM=<id>`
the models page: every model record, its deployments, settings, onboarding, arms (0.6.0, RFC-020)	`evh models [--kind K] [--where self-hosted\\|hosted] [--status vetted\\|onboarded\\|check\\|none]`
a model page: its deployments and their onboarding, settings history, arms	`evh models --model <id>`
(no page action) onboarding a model's deployment, smoking an arm before its run	`evh model-onboard <id> --deployment <d>`, `evh arm-smoke --config <arm>`
reference set pages	`evh references [--reference-id <id>]`	`make references`
the Captures page	`evh captures [--capture ID] [--json]`
the Labels page; a dataset page's labels (0.6.0)	`evh labels [--dataset-id D]`
(no page action) labelling: a sheet to fill, then its import (0.6.0)	`evh label-sheet --dataset-id D --out sheet.csv [--items …] [--item ID …]`, `evh label-import sheet.csv --dataset-id D --set S --by NAME`
(no page action) a run's outputs, or outcome rules, as labels (0.6.0)	`evh label-from-run --run R --set S`, `evh label-from-capture --dataset-id D --capture C --map rules.yaml --set S`
a reference's "promoted from" (0.6.0)	`evh reference-promote --dataset-id D --labels S [--items …] --tier gold\\|silver --reason "…"`
scored against a label set (0.6.0)	`evh rescore --labels D/S`, `evh run --labels D/S`
a recorded arm's badge (0.6.0)	`evh arm-from-capture --dataset-id D --capture C --as NAME`
the inspector's sheet, its downloads; its selection's commands (0.6.0)	`evh inspect --sheet --dataset-id D --runs A,B … [--csv FILE [--fill label\\|pick\\|scores]]`; `evh pick --dataset-id D --item … [--for V] [--tag k=v] [--note "…"]`
(no page action) a sheet filled elsewhere, read back (0.6.0)	`evh sheet-import sheet.csv --dataset-id D --by WHO [--kind person\\|model] [--set S] [--scorer S]`	`make sheet-import ARGS="…"`
a dataset page's picks waiting; its selection's command (0.6.0)	`evh picks [--for V] [--tag k=v]`, `evh pick --dataset-id D --item ITEM --drop`; `evh dataset-extend V1 --dataset-id V2 --picks`, `evh dataset-extend V1 --dataset-id V2 --pick ITEM`
a promoted arm's deploy sheet (0.6.0)	`evh deploy-sheet --config ARM [--capture C]`
the metrics page	`evh metrics [--dataset-id <id>]`	`make metrics`
a metric page: summary, details, goals, per-dataset strip chart and arms	`evh metrics --metric <metric>`	`make metrics METRIC=<metric>`
inspector: a comparison	`evh inspect --runs a,b [--baseline a] [--metric m]`	`make inspect RUNS=a,b`
inspector: filter, breakdown	`evh inspect --runs … --where field:value --by field`
inspector: one item	`evh inspect --runs … --item <item_id>`
inspector: the per-item table	`evh inspect --sheet --dataset-id D --runs …` (the sheet; `--scores-only`)
inspector: the analysis charts, as numbers	`evh inspect --runs … --analysis [--x <metric>]`
the catalog's global search	`evh search <text>`
"promote its run" (a goal page with no decision)	`evh run-promote --run <run_id> --reason "…"`	`make run-promote`
a suggested retirement	`evh round-retire --goal g --round rN --arm <config> --reason "…"`	`make round-retire`
a suggested close	`evh round-close --goal g --round rN --conclusion "…" [--promote <run_id>]`	`make round-close`
(no page action) opening a round	`evh round-open --goal g --dataset d … [--adopt] [--keep all] [--scoring "…"]`	`make round-open`
a round's scorer and its scoring changes (the goal page's round section)	`evh rounds --goal g --round rN`
(no page action) what a scoring change would move	`evh score-preview --goal g [--round rN]`
(no page action) re-scoring a round	`evh rescore --goal g --round rN --why "…"`	`make rescore`
building and serving the pages	`evh catalog`, `evh catalog-serve`	`make catalog`, `make catalog-serve`

Every row has a command. Adding a view means adding its command in the same change.

For an agent, the same rows are MCP tools (0.5.0, evh mcp, RFC-018): every read command above is a tool with the command's own options as its arguments, answering what the command prints (with --json where the command has it). A new read command is either a tool or listed, with its reason, as not one (mcp_server.NOT_TOOLS); a test fails otherwise. Not tools in 0.5.0: the commands that write, spend or run a model, and, not yet, the report and statistics commands (leaderboard, pair-test, family-test, bootstrap-test, holdout, rank-stability, run-compare, cost-report).

The round flow¶

Rounds are optional (PRD-005). The commands, in the order a round goes:

evh round-open --goal tickets_summary --dataset tickets_v1 \
    --hypothesis "how many leading sentences summarise a ticket best" \
    --keep tickets_lead1_v1,tickets_lead2_v1 --budget-usd 1
# add `round: r1` beside `goal:` in each new arm's config, commit, then run them
evh rounds --goal tickets_summary
evh round-retire --goal tickets_summary --round r1 --arm tickets_lead1_v1 --reason "…"
evh round-close --goal tickets_summary --round r1 --conclusion "…" --promote <run_id>

round-open prints the round file it wrote, then, when it carried arms in: "Carried in as they are, their runs already count: (their configs need no change and they need not run again)", then "A NEW arm names the round in its config (round: r1, beside goal: tickets_summary) and runs; then: evh rounds --goal tickets_summary" and "Commit the round file before the runs: an uncommitted file makes them dirty."
evh rounds --goal G prints one line per round, then the open round in full: its hypothesis, "on : n arm(s) in play, leader ", each other arm's verdict against the leader with Δ and p (four decimals, as every command prints a p), NOTE: lines (the spend against the budget), and suggested next, each suggestion followed by its command, ready to paste. The close suggestion carries --promote <run_id> only when the leader is not already the baseline ("close the round with what it showed ( is already the baseline)").
A reason or a conclusion shorter than three words is refused: it must be a sentence.
--promote runs run-promote with the conclusion as the reason; if the promotion is refused, the round stays open.