The comment copied from next door
The agent building the benchmark wrote, in the app, the one kind of comment the benchmark counts against agents.
Date: 2026-11-10. Context: showing a checker's answers on the run page (journal ch. 64).
What happened
The run page had to list a step's feedback, one paragraph per answer. A list needs a React key;
the answers have no id, so the agent (the assistant building Vektos) used the index, and silenced
the linter's no-array-index-key with // oxlint-disable-next-line. It had written the same line
twice that week in packages/genui, where the design system's own code may carry one.
In apps/web it may not. The dogfood test grades the app with the benchmark's own graders, and
the suppressions grader counts every disable comment: it is how an agent hides a rule it broke.
check:page failed on it. The fix was a key made of the tool and its answer, which are distinct
within a step, and no comment.
What we learned
- Agents copy what is next to them. The pattern was right in the package it came from and wrong one folder over. Nothing in the comment says where it is allowed.
- The benchmark's rules apply to the people writing it. Without the dogfood test the app would have shipped the exact violation it reports in other agents' code.
- A suppression is the cheapest way through a red check. That is why the graders count it: it is what an agent reaches for first.
What it means for Vektos
The same thing happens to the agents under test: an example in the design system's source, or in a Storybook story, is read as a pattern for app code. Where the design system's own code needs an exception, the exception should not look like an example.
Follow-ups
- Say in
packages/genuiwhy its two suppressions are allowed there and nowhere else. - Count suppressions copied from the design system's source in the runs: do agents that read the source (Storybook's MCP) write more of them?
The title the catalog only asked for
The catalog said a view's level-1 heading comes once. One model never wrote it, and nothing told it to until the page was graded.
The help that hurt
Letting views point at their data made the strong model perfect and the weak one worse, three reruns in a row. The new feature changed how the weak model wrote everything else.