Regressions

WhatCopy Anchor Link

A number on the dashboard can move for reasons that are not the agent: the design system changed, the graders changed, the output cap changed, or the task's brief changed. And it moves when nothing changed, because a model answers differently each time.

packages/bench/src/regressions.ts separates those:

• Versions. A batch records dsHash (the design system), contextHash (what the agent received at its level: the prompt, the tools and the code behind them), graderHash (the graders and the page check, both from provenance.ts) and max_output_tokens. Runs with the same four are one version. bench:backfill-hashes fills the two newer hashes for older batches from their commit.
• Series. One model at one context. Its versions are compared with each other, never across models.
• Shared tasks. Two versions are compared on the tasks both ran with the same task hash, each task weighing the same; a task whose brief changed is named and left out.
• Noise. Each version's batches are its repeats. A metric is worse or better only when the two versions' ranges do not overlap and the means differ by more than a floor (0.5 design-system violations, 1 quality violation, 10 points of done, 10% of the cost). With one batch on each side there is no measure of noise and no verdict.

/regressions shows each series' last two versions compared, how much repeats of the same task vary, and every version over time. bench:regress compares two labels and exits with 1 on a regression, for CI.

WhyCopy Anchor Link

The plan's thesis is a benchmark over time: did the agents get worse after this change? That question needs to know what changed, and how big a change has to be to mean anything. The spread table answers the second: for some cells, repeats of one task differ by more than the difference between contexts.

Try itCopy Anchor Link

bun run bench:backfill-hashes    # once, for batches older than the grader and context hashes
bun run bench:regress -- --baseline free-pilot --candidate free-rerun --fail-on ds
infisical run -- bun run web     # /regressions

Extend itCopy Anchor Link

• In CI. Run a small free batch on a pull request with a label, then bench:regress -- --baseline <main's label> --candidate <the PR's label>. Needs the runs in CI's database.
• Another metric. Add it to METRICS and METRIC_RULES with its direction and floor.

Known gapsCopy Anchor Link

• A context hash is per level, from our files. Each level lists its sources in CONTEXT_SOURCES (provenance.ts); a new file a level depends on must be added there. Storybook's MCP server is not ours: its content is the design system, which dsHash covers, not the server's own code.

• Two repeats are few. The noise of a cell with two or three batches is a rough guess; a single run on one side cannot show its own noise, so a lone baseline run can read as a change.

• Batches run on a dirty tree get their commit's grader hash, which may not be what graded them; the backfill counts them (14 of 83).

• "Changed" lists causes that could be, not causes that were. The grader hash covers every grader; a change to axe is listed next to a design-system regression it cannot cause (anecdote 21). Read the rule deltas; a hash per group would say it outright.