Benchmark tasks

WhatCopy Anchor Link

tasks/ holds the work we give to agents. Each task is a folder with four files:

The ten banking tasks (the five warehouse ones are in guide 24):

The first five need the eight components of the first design system; the last five, added in chapter 40, need the form, feedback and navigation components of chapters 36 to 38. No task says which component to use: components lists what our own answer uses, for the test.

loadTasks(dir) in @vektos/bench reads and validates the folders (zod schema, id must match the folder name, fixture must parse). @vektos/tasks exports tasksDir.

WhyCopy Anchor Link

• Plain files, not code. A team running Vektos on its own design system should add tasks without touching the runner.
• The prompt describes the product, not the design system. How much the agent knows about the design system is the variable we test. It comes from the context level (none, Storybook MCP, ds-mcp), never from the task.
• mustShow exists because an empty page is perfect. No code, no violations. Every task lists texts from the fixture or the prompt (a name, an IBAN, a field label) that the rendered page must show, as text or as a field value. A run that misses them is incomplete, whatever its violation count.
• Every task has a solution that grades clean. It proves the task can be done with the design system as it is, without raw values, without editing panda.config.ts. If our own answer needs a vektos-ignore, the task or the design system is wrong.

Try itCopy Anchor Link

cd tasks
bun test          # 46 tests: each solution grades clean, uses its components, renders mustShow

The test renders each solution with react-dom/server and searches the HTML for the mustShow texts. bun run check:page also puts every solution through the benchmark's own page checks, as if an agent had written it: typecheck, lint, a build, Chrome, axe in both themes and mustShow on screen. All ten have no findings. From code:

import { loadTasks } from "@vektos/bench";
import { tasksDir } from "@vektos/tasks";

const tasks = await loadTasks(tasksDir);
tasks.map((t) => t.id);
// ["kyc-form", "loan-contract", "payment-approval", "statement", "transfer-form"]
tasks[0].mustShow;
// ["Giulia Rossi", "giulia.rossi@example.com", "Date of birth", "Nationality", "Tax"]

Extend itCopy Anchor Link

To add a task, create a folder under tasks/ named like its id:

1. task.json with id, title, summary, components and at least one mustShow text. Pick texts the page cannot avoid showing: names and identifiers from the fixture, labels the prompt asks for. Avoid formatted numbers: €4,821.37 and 4.821,37 € are both right.
2. prompt.md: what to build, ending with "The data is in ./fixture.json."
3. fixture.json: realistic, synthetic, no real people or accounts.
4. solution.tsx: default-export a page component. Run bun run check: it must pass tsc, oxlint and the graders. Then bun run check:page: it must pass the page checks.
5. Put in quotes, in the prompt, every label mustShow expects ("Reason", "Money in"): a label the agent had to invent is not a fair test.

The tests pick it up automatically; update the lists in "there are ten banking tasks and five warehouse ones". A test checks that every mustShow text appears in the brief or the data.

Known gapsCopy Anchor Link

• Behaviour is not tested. The tests see the page as it loads. That an error appears after an empty submit, that a view goes back to page 1, that a toast confirms a copy: the prompt asks for it, our answer does it, and nothing checks it yet.
• Texts must be visible on load. The task test checks mustShow on server-rendered HTML and the runner checks it in Chrome (guide 11). Text inside a closed Dialog or Select list is in neither; pick texts visible on load.
• Fixture types are wide. JSON imports type "status": "pending" as string, so a lookup by status needs a type guard or a Record<string, …>. A cast (as) is a lint warning. Agents will hit the same thing (anecdote 05).
• The graders and lint disagree. Our solutions graded clean on the first try and still failed lint. Lint is now a grader in the runner's page checks (guide 11), under "quality".