The compiler that counted twice

What happenedCopy Anchor Link

Asked how the quality column is computed, I looked at what it was made of. Of the 4,044 type errors in the real runs, 3,895 (96%) were one code: TS2769, "No overload matches this call". Panda runs with strictTokens, so css({ bg: "#fff" }) does not compile. The raw-value grader reports raw-color for that line, and the compiler reports TS2769 for the same call. One mistake, two violations, in two different columns.

In the runs without context the effect was large. Kimi wrote 294 raw values per run and was also charged 348 quality violations, almost all of them the same values again.

What we learnedCopy Anchor Link

• A grader that sees the same thing as another grader inflates a score. We knew this for lint: @pandacss/prefer-token is dropped for that reason (guide 11). The compiler was the same case and we missed it, because the error text ("No overload matches") does not say "raw color".
• The same line is not enough. Only 81% of the TS2769 sat on the line of a raw value. TypeScript reports an overload error at one property of the call, not always the bad one. The test that works is the call: a TS2769 inside a css() that also holds a raw value or a token used for the wrong job is that value seen by the compiler. That covers 91%.
• The other 9% are real, and invisible to our graders. Opening samples, they are token names that do not exist: fontSize: "3xl", fontWeight: "bold" (the scale has normal, medium, semibold), lineHeight: "relaxed", and breakpoints that do not exist ({ base: "column", md: "row" }: the design system defined none until anecdote 09's follow-up added them). The value is not raw, so raw-values is silent, and the compiler is the only one that knows.

What it means for VektosCopy Anchor Link

markCovered (graders/covered.ts) marks a TS2769 as covered when its css, cva or sva call holds a raw-values or token-roles violation. A covered violation is kept in the database with covered = true, shown in the report as "not counted", and left out of every count. The run is graded when it is saved, and bun run bench:recount fixed the 3,539 violations already stored.

Mean quality violations per run, before and after (design-system column unchanged):

The order of the contexts did not change. What changed is how bad "no context" looked on quality: it was mostly the design-system column read twice.

Follow-upsCopy Anchor Link

• Done: token-names (guide 03). fontWeight: "bold" and a breakpoint that does not exist are now design-system violations with a message that lists the choices, and the TS2769 on the same call is covered. Replaying the grader on the stored runs, against the token table of the time, covers 3,842 of the 3,895 TS2769 (98.6%, up from 91%).
• a11y is per rule, not per element. One color-contrast finding counts 1 whether it hits one element or fifty, while a raw value counts 1 each. Quality still mixes units.
• The check_code of ds-mcp-check-full still prints both lines to the agent. That is useful to it, and only the counting changed.