Running a benchmark
How to run Vektos tasks against a model with a chosen amount of context, and how to read the results.
Vektos hands a task to a model, lets it write a page, then counts how far the page drifted from the design system and what the run cost.
Before you start
You need Bun 1.4 or newer and a clone of the repo:
bun installReal models are reached through the Vercel AI Gateway.
Create a key and some credit there, and put the key in your environment as
AI_GATEWAY_API_KEY. Keep it in a secrets manager and inject it only for the command,
for example with Infisical:
infisical run -- bun run bench --model openai/gpt-6-luna --task statementOr, for a quick try: export AI_GATEWAY_API_KEY=....
The gateway also lists a few free models (nvidia/nemotron-3.5-lightning-free).
They cost nothing, so they suit a first test of the key and the tools. They are
rate-limited and the provider may keep prompts, so leave them out of real comparisons.
Mock models need no key. Start with them to check that everything works.
The page is opened in Google Chrome, so install it. To use another Chrome or Chromium,
set VEKTOS_BROWSER_PATH to its executable. To skip the browser, use --no-page.
Run it
bun run bench # all five tasks, mock/ideal, no context
bun run bench --task statement # one task
bun run bench --model anthropic/claude-sonnet-5.5 --context ds-mcp| Option | Meaning | Default |
|---|---|---|
--model | A gateway id such as anthropic/claude-sonnet-5.5, or a mock | mock/ideal |
--context | What the agent may learn about the design system | none |
--task | A task id. Repeat it for several | all tasks |
--max-steps | Most model calls per task | 20 |
--budget | Stop starting new tasks once the spend reaches this many dollars | none |
--out | Folder for results | results |
--label | Text that groups repeats of one experiment, saved with the batch | none |
--save, --no-save | Real runs are saved to the database; mocks are not. These override that | |
--no-page | Skip typecheck, lint and the browser: static graders only | off |
Context levels
| Level | What the agent gets |
|---|---|
none | The file tools only. It knows @vektos/ds exists |
ds-mcp | Our design-system MCP server: components, tokens, rules |
ds-mcp-check | The same, plus check_code: it can grade its own draft against the design-system rules |
ds-mcp-check-full | The same, but check_code also runs the TypeScript compiler and the linter on the draft |
storybook-mcp | Storybook's MCP. Run bun run storybook first |
Mock models
| Model | What it does |
|---|---|
mock/ideal | Reads the fixture, writes the task's reference solution |
mock/lazy | Says "Done." and writes nothing |
mock/rogue | Writes the solution, then edits panda.config.ts |
mock/careful | Writes the solution, then calls check_code on it (use with a check context) |
Read the result
task page done ds qual steps tokens in/out cost
kyc-form yes yes 0 0 3 3917/1460 $0.0112
statement NO NO (5 missing) 0 0 1 569/4 $0.0006- page: did the agent write
page.tsx? - done: the page mounted in the browser and shows every text the task asks for.
NO (5 missing)means five required texts are not on screen. A run with no violations can still beNO: an empty page breaks no rule. - ds: design-system violations: raw values, wrong token roles, token names or breakpoints that do not exist, native elements where a component exists, suppression comments, edits to protected files.
- qual: everything else that went wrong: type errors, lint, accessibility in light
and dark, console errors, a missing
"use client", a page that did not render. - steps: model calls.
- tokens in/out: input tokens (cached ones included) and output tokens.
- cost: dollars, priced per request with the table in
packages/pricing.
Under the table, total is our price and gateway is what the AI Gateway charged. If
they differ by more than 2% a warning names the model: its entry in packages/pricing
is out of date.
Real runs are also saved to Postgres, so you can compare them later. Start it once with
bun run db:up and bun run db:migrate, then:
bun run bench:history # one row per model and context
bun run bench:import # load runs from results/ that are not in the database
bun run bench:recount # decide again which stored violations are not counted twiceSee guides/12-database. A real run is backed up to ~/Backups/vektos when it ends (--no-backup skips that). A real run refuses to start if the
database is down (use --no-save to skip it).
To see the runs, not only count them, open the report:
bun run bench:report # results/report.html, opens in the browser
bun run bench:report --label haiku-contextsIt shows the latest run of each model and context side by side, with the screenshot in light and dark, the violations and the code (guides/13-report).
Every run is also saved in results/<runId>/<taskId>/: result.json has every
violation and the cost of every step, files/ has what the agent wrote, and
light.png and dark.png show the page. results/ is not committed.
How each check works, and what it does not see: guides/11-page-graders.