Graders

WhatCopy Anchor Link

A grader reads code produced by an agent and returns a list of violations. The benchmark adds them up into a score. Graders live in packages/bench/src/graders/:

Every grader returns the same Violation shape: grader, rule, file, line, column, property, value and a message a person can act on.

gradeSource(filename, source) runs the right graders for a file: the first three on .ts/.tsx/.js, the CSS versions of them on .css, nothing on other types. findProtectedEdits(changedPaths) takes the output of git diff --name-only.

WhyCopy Anchor Link

tsc and oxlint block bad code. A benchmark needs the opposite: let the agent write freely, then count what went wrong.

Only style values count. A value is graded when its key is a CSS property, a Panda shorthand (bg, px, rounded) or a condition (_hover, base, md) under one. Without that filter, fintech data like { change: "12%" } counts as a raw value (see anecdote 03). The graders read css() and recipe objects, style={{ … }}, JSX style props (<Box bg="…">), both branches of a ? "x" : "y", and template literals, with ${} parts read as 000.

The token name carries its job. bg.* is for backgrounds, fg.* for text and icons, border.* for borders. Status tokens split the same way: .default for fills, borders and icons, .subtle for backgrounds, .text for text. token-roles checks that the job of the property matches the job in the name. The table is data (VEKTOS_ROLES), so another design system can pass its own.

A name that is not a token is its own mistake. fontWeight: "bold" is not a raw value, so raw-values is silent, and the compiler only says "No overload matches this call". token-names says what it is and lists the choices. It compares against a table built from the design system's panda.config.ts (tokenTableFromConfig), so a token added there is a valid name at once and another design system passes its own table. It reads css(), cva() and sva() only, as Panda does: { width: "narrow" } in the page's data is not a style. Raw values are left to raw-values, so one mistake is one finding (anecdote 08: the compiler's error on the same call is covered).

Gaming the benchmark costs points. An agent can hide a violation instead of fixing it: a // vektos-ignore, a @ts-expect-error, a new "token" in panda.config.ts. The graders honor vektos-ignore, because a raw value can be intentional, but suppressions counts every one and protected-files counts every edit to the rules.

A page places a component; it does not restyle it. The components are governed (guide 06): outside-styling counts className, style and any style prop that is not a layout prop, on components imported from @vektos/ds. The rules are data (GovernedSystem, built in vektos-ds.ts from the design system's own layout-props.ts), so another system passes its own package and list. A component's own prop that shares a name with CSS (size, a Tooltip's content, a Pagination's page) is not counted: vektos-ds.ts reads every <Name>Props interface in the design system, so a new component's props are known without a line of grader code. The compiler rejects the same prop with TS2322 at the same attribute; that error is marked covered, so one mistake counts once. The escape hatch (UNSAFE_*) counts every time and vektos-ignore cannot silence it: a way out is never hidden by another.

Hand-written CSS is graded too. Panda's types stop at css(), so globals.css is the obvious way around them. gradeCss runs the same checks through PostCSS, including roles on var(--colors-fg-muted).

The graders use oxc-parser (an AST, not text search: a # in a comment or a URL is not a violation) because TypeScript 7 has no JavaScript API for reading code.

Try itCopy Anchor Link

cd packages/bench
bun test            # 33 pass, 2 todo

Or call it directly:

import { gradeSource } from "@vektos/bench";

gradeSource("page.tsx", `<p className={css({ bg: "fg.muted" })} style={{ marginTop: 12 }} />`);
// violations:
//   token-roles/wrong-role  bg = "fg.muted"
//     "`fg.muted` used as background on `bg`: `fg.*` is for text and icons."
//   raw-values/raw-number   marginTop = "12"
//     "`marginTop: 12` becomes 12px. Use a token through `css()`."

To mark a raw value as intentional, put the reason in a comment on the line above:

// vektos-ignore: brand color from the client, not in the theme yet
bg: "#ff6b00",

The violation moves to ignored, and the comment itself is a suppressions violation.

Extend itCopy Anchor Link

A grader is a function (filename, source) => Violation[]. To write a new one:

1. Create packages/bench/src/graders/<name>.ts and its <name>.test.ts. Add the grader's name to GraderName in types.ts.
2. Use parse and styleValues from ast.ts. styleValues already finds every literal written for a style property, with its position.
3. If the check applies to CSS too, export a check…(property, value) function and call it from css.ts, as raw-values and token-roles do.
4. Wire it into gradeSource in grade.ts.
5. Add a todo test for every gap you know about. The gap stays visible until fixed.

The test the Vektos design system and web app grade clean grades our own code like agent code. If your grader flags something there, either the code or the grader is wrong: fix one of them, or add a vektos-ignore with the reason.

Known gapsCopy Anchor Link

• token-names checks the categories the design system has tokens for. A property with no scale (opacity, transition) passes: the compiler lets any CSS value through there too. The property-to-category table is written out in token-names.ts; a property missing from it is not checked.
• token-names is not applied to old runs. The stored runs from before the grader have no token-names findings, and were graded against an earlier token table (see the results page). Run it on the stored files to see what it would have said: against the table of the time it explains 98.6% of the TS2769 errors, up from 91%.
• A key that is a CSS property and a variant name (variants: { color: { primary } }) is left alone, so a misspelled breakpoint is only found when its value is a plain value, not an object.
• Values passed through a variable (const c = "#fff"; css({ bg: c })) are not followed. Tracked as a todo test.
• Numbers for spacing in css() (p: 4) are not flagged: Panda may read them as a token. In style={{ … }} they are flagged, because React turns them into pixels.
• A data key that is also a CSS property ({ width: "Width 12px" }) is graded as a style.
• Shorthand strings are not split into tokens for token-roles: in border: "1px solid fg.muted" only the raw part is seen.
• Class-name styling (Tailwind) is not read yet. That arrives with the Tailwind adapter in phase 2.
• Each grader parses the file again. At benchmark sizes it does not matter.
• tsc and axe graders arrive with the runner in week 2.