Page graders

WhatCopy Anchor Link

The first graders read source code. A page can pass all of them and still be broken: it may not compile, crash on render, show nothing, or use the right tokens on a <button> that is not the design system's Button. This block adds the checks that need the page to exist as a running page.

They come in two kinds.

Static, in gradeSource (they read one file, so they run on every grade):

Dynamic, in packages/bench/src/page/ (they need real files and a browser):

Every violation belongs to a group, so a report can answer three questions apart (GRADER_GROUP in graders/types.ts):

A run is complete when it wrote page.tsx, the page mounted, every mustShow text is on screen, and, since graders v3 (ch. 68), it does what its task's behaviour.json asks. RunResult has rendered, missing and complete, plus one full-page screenshot per theme.

A violation can also be covered: a TS2769 ("No overload matches this call") inside a css, cva or sva call that holds a raw-values, token-roles or token-names violation. With strictTokens the compiler rejects css({ bg: "#fff" }), so it is the same mistake as the raw-value finding. It is stored with covered = true, shown in the report as "not counted", and left out of every count (markCovered in graders/covered.ts).

WhyCopy Anchor Link

• One mistake, one count. Covered violations exist for the same reason lint skips two rules, except that the compiler's error does not say what it is about, so it is matched by position (anecdote 08). Without it, 87% of the quality violations of runs without context were design-system violations again.
• Three groups, not one score. A page with a native <button> styled with perfect tokens is a design-system failure but not a quality failure. A page that crashes is the reverse. Mixed into one number you could not tell which context level helped with what.
• Lint is "quality", and it skips two rules. @pandacss/prefer-token and no-primitive-token report the same mistake as raw-values and token-roles. Counting both would double every raw color. The graders also stay stack-neutral: they do not need to know Panda exists.
• A real browser for mustShow. Server-rendered HTML is not what the user sees. The check reads innerText and the value of every field, so a form filled from the fixture passes and an empty page does not.
• Both themes. Anecdote 02: a token can pass contrast in light and fail in dark. axe runs twice, adding the dark class the second time. One violation per rule and theme, with the number of elements in the message, so a table with fifty bad cells is one finding, not fifty.
• React warnings count. The page is bundled in development mode so React reports a missing key or an invalid nesting through console.error, and that is a violation.
• native-element is a design-system rule. vektos-ignore can silence it, like a raw value. It cannot silence missing-use-client: a page that does not build is not an intentional choice.
• Only page.tsx needs "use client". A helper file may be imported by a client file, so flagging it would be wrong. Design-system components are already client components (Ark marks them), so only what the page itself calls counts.

How the page is checkedCopy Anchor Link

agent's files ──► apps/web/.bench/<id>/   (a scratch folder, deleted afterwards)
                    ├─ tsc --noEmit            → typecheck
                    ├─ oxlint -f json          → lint
                    ├─ panda cssgen            → the page's CSS
                    ├─ Bun.build               → one script
                    └─ Chrome + axe-core       → render, a11y, screenshots

• Why inside apps/web. That is where @vektos/ds, react and @types/react resolve, and where styled-system/* is mapped by the generated tsconfig.json. The folder is gitignored, and .bench/** is in Panda's include, so the lint rule file-not-included passes.
• Protected files never reach the folder. panda.config.ts, tsconfig.json and package.json are skipped, so an agent cannot loosen its own typecheck. The protected-files grader still reports the edit.
• One Chrome per batch. createPageChecker() starts it on the first check and close() shuts it down. The CLI calls ready() first, so a missing Chrome stops the run before any paid model call.
• Errors in our own sources are not the agent's. tsc also reads the design system; only errors in files the agent wrote become violations.

Try itCopy Anchor Link

bun run bench --model mock/ideal

task              page  done           ds   qual  steps  tokens in/out   cost
kyc-form          yes   yes            0    0     3      3917/1460       $0.0112
statement         yes   yes            0    0     3      3834/729        $0.0075

ds is the design-system group. qual is quality plus completeness. done is complete. Open results/<runId>/statement/light.png and dark.png.

bun run bench --model mock/lazy --task statement   # page NO, done NO (5 missing)
bun run bench --model mock/ideal --no-page         # the static graders only, no Chrome
bun test packages/bench/src/page                   # unit tests, no browser
bun run check:page                                 # the same plus the Chrome tests

check:page puts the reference solution of every task through these checks; each must come out with no findings. bun run check skips the Chrome tests so it stays fast. Run check:page before touching anything in packages/bench/src/page/.

Extend itCopy Anchor Link

• Another native element. Add a { tag, component } to VEKTOS_COMPONENTS (with types to match an <input> by its type; the first matching rule wins), or pass your own list to findNativeComponents.
• Another browser check. renderPage in page/render.ts has the page open in tab: add the check there and return a Violation with a render rule.
• Another axe tag set. Edit AXE_OPTIONS in page/render.ts. A test checks that axe knows every tag: an unknown one matches no rule and raises no error (anecdote 19). Batches before graders v2 (ch. 59) ran WCAG 2.0 and 2.1, A and AA, only; since v2 axe's best-practice rules run too (region stays off: landmarks around a page are the layout's). A change of tags changes graderHash, and /regressions names it (guide 23).
• Another browser. Set VEKTOS_BROWSER_PATH to any Chrome or Chromium executable.

Upgrading the browser: from system Chrome to a pinned ChromiumCopy Anchor Link

The checker uses the Google Chrome installed on the machine (channel: "chrome"): nothing to download and it works on a fresh clone. That is right for what the checks look at (does it mount, what does it say, does axe pass), which does not change between Chrome versions.

It is not enough for a visual diff (PLAN, phase 2). A screenshot comparison needs the same browser, fonts and rendering on every machine and in CI, or every Chrome update looks like a regression. When that block starts:

1. add playwright (the full package, which downloads browsers) next to playwright-core, and run bunx playwright install chromium in setup and in CI;
2. pin the version in the lockfile and launch with chromium.launch() instead of channel: "chrome": that is one line in launchBrowser() in page/render.ts;
3. keep VEKTOS_BROWSER_PATH as the override;
4. take the screenshots at a fixed viewport (VIEWPORT is already fixed at 1280×800) and disable animations.

Nothing else in the checker depends on which browser it is.

BehaviourCopy Anchor Link

Five tasks list what the page must do after the user acts, in tasks/<id>/behaviour.json (page/behaviour.ts): a few steps (fill a field by its label, check a box, submit) and one expectation (invalid: the field is marked invalid, by aria-invalid or the browser's own check; or shows: a text appears). Each runs on a fresh copy of the page, after axe. A failure is a behaviour violation in the completeness group, named after the behaviour, and the run is not done.

• Fair by construction. Labels come from the brief or the data (a test checks it); submit never names a button, it clicks the form's submit button or the page's last one, and leaves a disabled one alone, since keeping the user from submitting is an answer.
• Accessible by construction. An error counts when it is tied to its field: a message the screen reader never hears is not an error message.
• Graded, not briefed. behaviour.json is in the graders' hash, not the task's: the briefs did not change, the grading did.

Known gapsCopy Anchor Link

• Token names that do not exist are invisible to the design-system graders. fontWeight: "bold" is not a raw value; only an uncovered TS2769 reports it.
• Quality mixes units. a11y counts one per rule and theme, type and lint errors one per occurrence.
• One viewport, no mobile. 1280×800 only. A layout that breaks at 375 px passes.
• Little interaction. Five tasks have behaviours, and only forms: a dialog that never opens, a tab that does not switch, a select that cannot be opened are not tested yet.
• No screenshot comparison. Screenshots are saved, not compared. See above.
• axe finds about a third of accessibility problems. A page with no axe findings is not proven accessible.
• Fragment, not document. The region rule is off, because the layout around a page provides the landmarks.
• Pages only. The agent writes one page. Several routes, server actions or data fetching would need a real next build.
• Needs Chrome. On a machine without it, run with --no-page or set VEKTOS_BROWSER_PATH.