The check that checked half

What happenedCopy Anchor Link

The first full set of real runs (5 tasks, 4 context levels, two models) had one result that looked too good. With ds-mcp-check, the context where the agent can call check_code on its own draft, design-system violations were zero in every run: 16 of 16 for Haiku, 5 of 5 for GPT-6 Luna. Without the checker (ds-mcp) the same models had about one per run (Haiku 0.9, Luna 2.1).

The quality column told a different story:

(Quality numbers in this anecdote were later corrected: anecdote 08 found that a compiler error restating a raw value was counted twice. Corrected: Haiku ds-mcp 1.6, Luna ds-mcp 16.1, Haiku ds-mcp-check-full 0.3. ds-mcp-check is unchanged.)

The checker removed the violations it can see and did not touch the rest. What was left was TS2769 (a prop value that does not exist), TS18048 (possibly undefined) and Panda's no-physical-properties (marginLeft where marginInlineStart is expected). Luna's 45 no-physical-properties findings are the same mistake repeated.

What we learnedCopy Anchor Link

check_code runs the benchmark's graders: raw values, token roles, suppressions, components. It does not run tsc or oxlint, which the benchmark's page checks add on top. So the agent was optimizing exactly the score the tool reports, and that score is a subset of what we measure.

This is the same shape as anecdote 05, seen from the agent's side: a clean grade on one check says nothing about another. It is also a reminder that an agent will fix what it is told about. The checker works; it is incomplete.

What it means for VektosCopy Anchor Link

• A context level is measured by the whole report, not by the number its own tool gives. Always read the design-system and quality columns together.
• ds-mcp-check is better but not free. It costs about 50% more than ds-mcp per run (Haiku $0.0101 against $0.0068) for the design-system gain, and adds nothing on quality.
• The fix is a product decision. Either check_code also runs tsc and lint on the snippet (more useful, slower, needs the real project to resolve types), or the docs say plainly that it covers the design system only. An agent that knew it had to pass the compiler would write marginInlineStart the first time.

Follow-upsCopy Anchor Link

• Done: ds-mcp-check-full. A check_code that also runs tsc and oxlint. Same tasks, same models:

  	design system	quality	cost per run
  Haiku, ds-mcp-check	0.0	2.3	$0.0101
  Haiku, ds-mcp-check-full	0.2	0.3	$0.0126
  Luna, ds-mcp-check	0.0	20.0	$0.0055
  Luna, ds-mcp-check-full	0.0	0.0	$0.0132

  Luna goes from 20 quality violations to none in every task. Haiku from 2.3 to 0.3. The tool had to be told about the compiler for the agent to fix what the compiler says.

• Add the share of quality violations that a design-system checker cannot see to the report, so the gap is visible without a query.