UXS-002: The run inspector

• Status: Active
• Authors: Marko Dragoljevic
• Related PRDs: PRD-004 (catalog and inspector), PRD-006 (fields), PRD-002 (references)
• Related RFC: RFC-003 (the server, its API and limits)
• Related UX specs: UXS-001 (shell, tokens, tints, paging), UXS-003 (evh inspect)
• Implementation paths: harness/catalog_assets/inspect.js (the page), harness/scripts/catalog_serve.py (/api/compare, /api/item), harness/scripts/catalog_build.py (render_inspect: the shell and runs.js)

Summary

The inspector is the catalog's one live page, inspect.html, filled by the server that evh catalog-serve starts on 127.0.0.1. The user chooses arms on one dataset; the page tests each against a baseline with the harness's paired test and shows, in order: the verdicts, the conditions the arms ran under, any breakdown by a field, the analysis charts, the per-item table, and, on a click, one item's outputs side by side with the reference and the item itself. Its state lives in the URL, so every link from the catalog opens a ready comparison.

Principles

• The verdict first, then the evidence for it. Charts and items explain a verdict; they do not replace it.
• The same test as the command line. The page calls the server, the server calls evaluation_harness.stats; evh inspect calls the same function.
• Refuse and say why. Arms that cannot be tested against the baseline are set aside with a note, never tested anyway.
• Item text only here. The static pages never carry it, unless the repository already publishes it (a profile written --with-text, refused unless git has committed every item's file). A public deployment never returns the item itself, and shows an arm's output or a reference's text only when it publishes them (catalog-serve --publish outputs,references); otherwise the item view says which it withholds.

Scope

In scope: choosing arms, the controls, the verdict table, conditions, fields (filter and breakdown), the analysis charts, the per-item table, the item view.

Non-goals: writing anything; choosing a round (the page has no round picker: scope a round with evh inspect --round rN); comparing arms across datasets.

State in the URL

inspect.html?runs=<arm>,<arm>,...&baseline=<arm>&metric=<name>&where=<field>:<value>&by=<field>. Links from the catalog may also carry family and threshold, so a verdict reached from a goal page matches the one that page implied. Every control rewrites the URL and reloads. &item=<id> opens that item's view once the comparison has loaded (a dataset page's inspect link); an item the arms do not share says so instead.

Choosing arms

• Chips: each chosen arm as a chip with × to remove it.
• Add: a text box with a list of every arm, narrowed to arms on the chosen arms' dataset ("add a run on "), and an Add button.
• Nothing chosen: "Pick runs from an eval or dataset page in the catalog ("Inspect runs"), or add them above."
• No server (the file opened from disk, or the server stopped): "No inspector server. Start it with evh catalog-serve and open this page from the address it prints."
• At most 40 arms in one comparison (--max-arms); the catalog's group links stop at 40.

Controls

• metric: every metric the arms record, each with its format and "(archived reference)" when that applies, ↑ or ↓ for its direction, and how many arms have it. The default prefers a quality score over lengths, counts, cost and latency, then what the baseline recorded, then what most arms have.
• baseline: any chosen arm. Without one, a promoted baseline among the chosen arms, else the first arm.
• re-score what is not on record: a checkbox shown only when the adapter can re-score (the harness's own adapter cannot; a project's may).
• filter items by a field, then a value input fitted to its type (a list for a choice, yes/no for a boolean, min..max or one value for numbers and durations, an exact value otherwise) and Filter; each filter becomes a chip with ×. Shown when the dataset declares fields.
• break down by a choice or boolean field, or none.
• scored against (0.6.0: RFC-023): the reference (its tier pill) or a label set (its source chip and name) the scores are read against; arms scored against different ones are named and not tested, as for two references.
• only where they disagree (0.6.0): a checkbox keeping the items where the chosen arms decided differently (the adapter's same(a, b)), the baseline (production's recorded arm by default when it is chosen) against each other arm.

Notes above the results

Plain-language notes, one per line, for everything the comparison could not do: the baseline swapped because the chosen one lacks the metric; arms that read different input hashes; arms with no dataset_id on record; arms with no score on this metric; average-only arms set aside because they recorded exactly the same average as an arm already shown; arms left out because they are on another dataset; arms not tested because they were scored against another reference than the baseline's ("Pick a baseline scored against the same reference to compare them."); arms with only an average; outputs that did not follow the asked format; runs scored against references for only some items (reference_coverage); a filter ("a subset is a smaller sample, so fewer differences can be significant"); a breakdown ("a group is held to a stricter threshold than it would be alone") and items in no group.

Not scorable: when no arm has a per-item score, the page says so and offers a button per metric it can compare (output length, latency) instead.

01 verdicts

• Eyebrow "verdicts · ", the metric linked to its row on the metrics page.
• One row per arm against the baseline, smallest p first, tinted like the catalog's rows (UXS-001). Columns: run vs baseline, verdict (second, so a phone sees it), items, mean, Δ vs baseline, 95% CI of Δ (the confidence interval of the difference: exact up to 30 items, a t interval above), wins–losses (with ties), p (exact ≤30), Holm threshold. A p is printed as every command prints it: four decimals, <0.0001 for what would print as 0.0000. Arm labels keep their version (_v1) even when every arm shares it; only a shared " · " tail is cut.
• The verdict is a chip: ▲ significant: better (border --open), ▼ significant: worse (border --local), or plain not significant; on a neutral metric significant: higher / lower. When the number of items cannot reach p 0.05 at all, the row says "n items cannot reach 0.05".
• Under the table: "Baseline : mean m. Holm across k comparison(s)", whether the family or threshold came from the page the user came from, the direction, and where the scores come from.
• This table is never paged: its Holm thresholds are read together.

Conditions

From each arm's recorded conditions (fingerprint v5, RFC-021: the deployment, where the model ran, as deployment.*; the runner, where evh ran, as machine.* and library.*; harness, scorer, items, reference bytes):

• all equal: one line, "Same conditions: all n recorded fields match across these runs";
• otherwise "Conditions · k of n fields differ", the warnings, and a table of the differing fields, one column per arm, the cells that differ from the baseline marked. Warnings: different scorers ("these numbers are different measurements. Re-score them with one adapter (evh rescore) before reading the test."), different reference bytes, different item bytes, and for a speed measure either "compared across deployments (where the model ran)" when a deployment.* field differs, or "measured from different runners" when only a machine.* field does. A runner that differs is never a warning on a quality measure: the runner is outside the fingerprint's hash.

The baseline that wrote the reference

Among the comparison's warnings, above the verdicts: when the baseline is the arm that wrote the reference the arms are scored against, " wrote the reference these arms are scored against: it is graded against its own outputs, so every other arm reads as worse. Compare the others to each other with another baseline." evh inspect prints the same.

Average only

Arms that recorded only a run-level average: a table "Average only · n: no per-item scores, so no test", with each recorded average and its difference from the baseline's mean. Never tested.

Breakdown

With break down by set: "By · g group(s)", a table of group (with its item count), run vs baseline, verdict, items, Δ, wins–losses, p and Holm threshold, and the note that the test runs within each group, Holm-corrected over all tests together. Not paged.

02 analysis

Inline SVG charts in the catalog's tokens; each arm keeps one colour (--s1 to --s8, then muted; the baseline --baseline) on every chart, never a colour by rank. A legend names them.

• Spread per run: every item's score, one row per arm, baseline first, then best mean first; a bar and a number for each arm's mean, a dashed line for the baseline's mean.
• Difference from the baseline: each item's score minus the baseline's, one row per arm; the mean difference, wins–losses, and ✓ better / ✗ worse in words when significant after Holm (words, because an up-arrow beside a negative number reads wrong on a lower-is-better metric).
• against : a scatter of score against a per-item fact such as output length, latency, cost in USD, or tokens (in + out; a picker chooses which), one faded dot per arm per item and a cross at each arm's average: the quality–length, quality–speed, quality–cost or quality–tokens trade-off.

Hovering a dot shows its arm, item and value. Below 700px the charts switch to a narrow layout (labels above each row) rather than scroll.

03 per item

"per item · n": one row per item, one column per arm (the baseline's marked), each cell the score with ▲ or ▼ where it beats or trails the baseline on that item. Paged 10 at a time like every catalog table. A row opens on click, Enter or Space.

04 the item view

"item · ", opened under the table:

• the item's fields, when the dataset has them (a URL field as a link only for http(s));
• the item itself, "what every arm read", with its length; unfolded when 1,500 characters or fewer, folded otherwise. The local inspector only;
• a toggle to mark what each run adds and drops against the baseline (word-level), useful between near-identical outputs;
• columns: the reference (when the metric names one, else the baseline's), then each arm's output with its word count, the baseline's marked. Above each output, the run it comes from; for an arm with repeats, "the latest of N repeats; the score is the mean of all N". For a judged arm, the same line gives each judge's score for that item.
• In a catalog built by evh catalog, arm names (verdicts, the baseline line, average-only and breakdown rows) and the verdicts' metric link to their pages.
• (0.6.0) Every label the item has, each with its source chip, set name, who made it and note, above the outputs (the local inspector only); a recorded arm's output is headed "production, recorded" with its capture.

05 sheet (0.6.0: RFC-023)

Opened from a dataset ("Open its sheet", production's recorded arm chosen first) or with ?dataset=<id>&sheet=1[&runs=…][&item=<id>]: one dataset's items, not a comparison. The local inspector only (/api/sheet): it carries outputs and labels, so a public server refuses it. With no arm chosen, a row (or &item=, a pick's link) still opens the item, its fields and its labels. It is evh inspect --sheet in the browser, the same model.

• The items down, a column per arm across, in the terminal's order: item (short id, the full id in its title) and the start of the item (the adapter's title(item) when it has one: an alert's name, a ticket's subject); the signals, next to it because they are why a row matters: wrong (the first run's decision wrong against the chosen label set), disagree (another chosen arm decided differently), split (the candidates disagree among themselves), new (a field value the dataset chosen in "new against" has never had), unlabelled (no label in any set); then per arm its output (the adapter's decision(output), else the output, shortened; the item view has it whole; the first is headed "first: production") and a column per score it has on the item (its run's metrics, and a judging pass's where one covered the run); a column per label set with its source chip and per reference set; the winner columns (on the metric the sheet is read on, and per pick or pairwise judge); picked (for which datasets); and the item's fields, last. The best score per row on the chosen metric is marked.
• The select box and the item column stay in place as the arms scroll across.
• The footer, under each arm's columns: its mean on the chosen metric (and which way is better), its win rate per winner column (the share of the rows it was scored on that it wins, a shared win split, "—" for an arm scored on none; on a pick or pairwise verdict, each arm's share of what the judge gave over the rows it judged, a shared pick counting its share and a row it gave to an arm not shown being nobody's), and its verdict (the goal's decision when a goal names the dataset, else the paired test against the leader).
• Controls: the arms (chips, as above), "read on" (the metric), "wrong against" (a label set), "new against" (a dataset), one checkbox per signal to keep only those rows, "scores only" (hides the outputs; kept in the address as scores=only), the field filters and breakdown above, and a sort by any column.
• A row opens the item view (04) for that item.
• Downloads: "download (CSV)" for every row shown, with a box to add the empty label columns to fill (fill=label); built by the server as evh inspect --sheet … --csv builds it (/api/sheet?…&format=csv), a read: the page writes nothing (PRD-004).
• Selecting: a box per row and "select all shown"; with rows selected, a panel gives two commands to copy: evh pick --dataset-id <id> --item … --item … --for <dataset> --tag <key=value> --note "…" (the target, tags and note typed in the panel fill the command) and evh label-sheet --dataset-id <id> --item … --out sheet.csv, with "download these rows (CSV)" beside it.
• Paged 10 at a time like every catalog table; the counts above the table ("412 items · 9 wrong · 31 disagree · 4 new") and the metric it is read on.

Checklist for a change to the inspector

• evh inspect gives the same verdicts, conditions and breakdown, or gains the change too.
• The verdict and breakdown tables stay unpaged; the per-item table pages.
• No inline script or style (the pages run under a CSP without 'unsafe-inline').
• make browser-check passes (harness/tests/test_inspector_browser.py).

Parity with the terminal

on the page	in the terminal
choose arms, baseline, metric	evh inspect --runs a,b [--baseline a] [--metric m] (config ids are looked up under --from, default runs)
a round's arms (the goal page's inspect button on a round)	evh inspect --round rN [--goal g]: the goal's measure, against the round's leader; arms with no score on it are named and left out
filter, breakdown	--where field:value (repeatable), --by field
notes, verdicts, conditions, breakdown	printed in that order (notes as NOTE: lines)
the item view	--item <item_id>: fields, the item itself, the reference, each arm's output under (run <run_id>), or (run <run_id> · shown the latest of N repeats; the score is the mean of all N) for a repeated arm
the per-item table	evh inspect --sheet --dataset-id D --runs a,b: every item's score per arm (the sheet; --scores-only leaves the outputs out)
only where they disagree (0.6.0)	--disagree (an arm scored against a label set is chosen like any)
the sheet (0.6.0)	evh inspect --sheet --dataset-id D --runs A,B [--metric M\|SCORER/M] [--wrong-against SET] [--new-vs V] [--disagree] [--split] [--unlabelled] [--where field:value] [--by field] [--scores-only]
the sheet's download (0.6.0)	--csv FILE; --fill label [--choices a,b] adds the columns to fill; --fill pick\|scores [--seed N] [--rubric FILE] [--scale LO..HI] writes blind columns and a key, read back by evh sheet-import
selecting rows: the pick and sheet commands (0.6.0)	evh pick --dataset-id D --item …; evh label-sheet --dataset-id D --item …
the analysis charts	--analysis: the spread per arm and the difference from the baseline, per item, as numbers; --x <metric> adds each arm's mean on a second metric (the scatter)

For an agent, evh inspect is the MCP tool inspect, with the same options (runs, baseline, metric, where, by, disagree, analysis, x, round, goal, from, no-rescore, item, and the sheet's sheet, dataset-id, scores-only, wrong-against, new-vs, split, unlabelled); the options that write a file (csv, fill, choices, seed, rubric, scale) are not offered. A public server (evh mcp --public, and this repository's at /mcp) leaves out item and the sheet, as the public inspector serves no item text, output or label.