The regression the graders did not cause
The first regression the new page found came right after a change to the graders. The graders could not have caused it.
Date: 2026-11-10. Context: graders v2 and a free mini-rerun (journal ch. 59).
What happened
Graders v2 turned on axe's best practices: a page without <main> or <h1> now counts as a
quality violation. A small rerun on two free models followed, so /regressions would have its
first real comparison. bench:regress found one regression and failed: Beam, with no context,
went from 32.3 to 47.2 design-system violations per run. "Changed: graders, output cap."
The graders that changed were axe's, which count in the quality group. No design-system rule
changed, so the graders could not have moved that number. The rules behind it said so:
raw-length from 16.7 to 24.8, unknown tokens, native elements, all design-system rules the model
writes more or less of from one run to the next. One of the nine baseline runs had ended on the
output cap with no page, and so with zero violations, pulling the baseline down; the new runs, under
a lower cap, all wrote pages. And the effect the graders did have, landmark-one-main and
page-has-heading-one appearing, was real but too small to lift the quality total above the noise
of the compiler's errors.
What we learned
- "What changed" is not "why it changed". The page listed two changes and one regression, which invites reading the first as the cause of the second. Neither was.
- Change one thing at a time. The candidate differed from the baseline in the graders and in the output cap; with three repeats a side, nothing can be pulled apart.
- The rules carry the explanation. The totals said "worse"; the rule deltas said which group, and so what could and could not be the cause.
What it means for Vektos
A regression gate in CI will meet this every time: a pull request changes one thing, and the runs move for another reason. The gate must say which rules moved and whether the change could reach them, or it will blame the wrong commit.
Follow-ups
- A grader hash per group (design system, quality, completeness), so a change to axe cannot be listed as a cause of a design-system regression.
- More repeats before a verdict counts as a failure in CI.
The cd that failed
A clone failed, so the cd after it failed, so a history rewrite meant for a scratch copy ran on the working repository. Nothing was lost, by luck and by a remote.
The task the paid models made easy
Counting every run, the KYC form looked like one of the easier tasks. It was the second hardest: the paid models had done it.