Journal

This is the story of how Vektos was built, one block of work at a time, oldest first. Each chapter is written so that someone who was not there can follow along: what we were trying to do, what we built, how to try it on your machine, why we chose what we chose, what went wrong, and where it led next.

If you only want the big picture, read PLAN for the why and architecture for how the pieces fit. The guides go deeper on each piece. The anecdotes tell the longer stories behind the mistakes.

How to read this journalCopy Anchor Link

• To follow the story, read the chapters in order. Each one is self-contained: the goal, what was built, how to try it, why it was done that way, what went wrong, and where it led.
• To see why something is the way it is, use Decisions. It is the same project grouped by theme, with the alternatives that were rejected and a table of the decisions that were changed later.
• To learn from the mistakes, read the anecdotes. Thirteen of them, each a short story that taught the benchmark something.
• To use a piece, go to its guide: what it is, how to try it, how to extend it, and its known gaps.
• For the numbers, Results has the benchmark's findings and how far to trust them, and guides 15 and 16 have the retrieval evals.

Index of chaptersCopy Anchor Link

The project has three parts, in the order they were built. The benchmark (chapters 0 to 17): a design system, a way to count how well an agent follows it, and what each run costs. The dashboard and its repairs (chapters 18 to 22): looking at the data, and fixing the gaps it exposed. Retrieval (chapters 23 to 25): finding the text that explains a violation. The chat (chapters 27, 28 and 30) joins them; chapter 26 is this reading guide.

Before you startCopy Anchor Link

What Vektos is. A benchmark. You give an AI coding agent a small UI task ("show this bank statement as a table"), let it write the code, and then measure two things: did it follow the design system, and how much did the run cost. Running the same task with different models and different amounts of context tells you which setup gives compliant code for the least money.

Words used throughout.

• Design system: the shared kit a team builds UIs from. Here it is eight React components (Button, Input, Table…) plus a set of tokens.
• Token: a named design value. Instead of writing #2563eb you write accent.default. If the brand color changes, every screen follows.
• Semantic token: a token named after its job, not its look. fg.muted means "secondary text", whatever color that is in light or dark mode.
• Grader: a program that reads the agent's code and lists every rule it broke. Each broken rule is a violation.
• MCP (Model Context Protocol): a standard way to give an agent tools it can call, like "list the components" or "show me the Table props".
• Context level: how much help the agent gets about the design system: none, Storybook's MCP, our MCP, our MCP with a checker, or a checker that also compiles. The benchmark compares them.
• Batch and run: a run is one task done once by one model; a batch is one call of bun run bench, with one model, one context level and several tasks.
• Design-system hash: a fingerprint of the tokens and components a batch ran against. Two batches with different hashes are not comparable.
• Chunk: a piece of a document, cut at its headings, small enough to hand to a model and still meaningful alone.
• Embedding: a list of numbers that stands for the meaning of a text. Two texts about the same thing have close embeddings, even in other words.
• Reranker: a model that reads a question and a few candidate chunks together and sorts them, slower and sharper than the search that found them.
• recall@k and MRR: how often the right chunk is among the first k results, and how high it is on average (1 is always first). The two numbers the retrieval evals report.
• WCAG and the EAA: the web accessibility guidelines (the standard axe tests against) and the European Accessibility Act (the law that requires accessible products and services).
• Tenant: whose documents a piece of text belongs to. Retrieval never mixes them.

What you need. Bun 1.4 or newer and Git. Clone the repo, then:

bun install
bun run check     # typecheck, lint, format check and tests, all must pass

Chapter 0 · The spike: can this stack work at all?Copy Anchor Link

2026-11-08 · Week 0 · commits up to PR #24

The goalCopy Anchor Link

Before writing any benchmark code, prove the tools fit together. The plan called for some very new pieces: TypeScript 7 (the Go rewrite), Panda CSS for styling, Ark UI for accessible primitives, Next 16 and Storybook 10. If any of them refused to cooperate, better to find out on day one.

What we builtCopy Anchor Link

• A Bun workspace. One repo, several packages under packages/ and apps under apps/. Bun installs, runs scripts and runs tests.
• The design system skeleton in packages/ds: Panda config, one Button.
• The first grader, raw-values, in packages/bench. It parses a file with oxc-parser and flags hardcoded values like "#fff" or "12px" inside css().
• Storybook with the docs, accessibility and MCP addons, so the design system has live documentation that agents can also query.
• A docs site in apps/docs (the site you may be reading now), with search, an llms.txt file for agents and its own MCP server.
• Guides 01 to 05 and the architecture doc.

Try it yourselfCopy Anchor Link

bun run dev

This starts three things at once: the web app on localhost:3000, Storybook on localhost:6006 and the docs on localhost:3001. Then try the strict tokens. In any component, type:

css({ bg: "#1d4ed8" })   // TypeScript error: not assignable to ColorsValue

TypeScript itself refuses the raw color. That is Panda's strictTokens mode, and it is the first wall an agent hits. More in guide 01.

Decisions and whyCopy Anchor Link

• TypeScript 7 only, no fallback. TS 7 has no compiler API, so for reading code (the graders) we use oxc-parser, which is fast and has a stable AST.
• Panda's CSS reset is scoped to .vektos. Without that, the reset also wiped the styles of the docs pages. Now only elements inside a vektos class are reset.
• Branch, PR and rebase merge for every block. No ticket tracker: the roadmap in PLAN is enough for a one-person project.

What went wrongCopy Anchor Link

The token table on the docs page took five attempts. We kept styling it with the design system, and every attempt fought the scoped reset. The fix was to stop using the design system for a page about the design system. Full story in anecdote 01.

Where to go nextCopy Anchor Link

The stack works. Week 1 builds the real design system: theme, components, graders and an MCP server.

Chapter 1 · Tokens that say what they are forCopy Anchor Link

2026-11-08 · Week 1, block 1 · PR #25

The goalCopy Anchor Link

Give the design system colors with meaning, and a dark mode, so that "used the right kind of token for the job" becomes something a program can check.

What we builtCopy Anchor Link

• Semantic color tokens in packages/ds/panda.config.ts, grouped by job:

  • bg.* for backgrounds (bg.canvas, bg.surface, bg.subtle)
  • fg.* for text and icons (fg.default, fg.muted)
  • border.* for borders
  • accent.* for the brand color
  • danger.*, success.*, warning.* for status, each with .subtle (a tinted background) and .text (readable text on that background)

  Every token has a light value, a dark value and a one-line description.

• Dark mode through a .dark class on a parent element.

• A theme switcher in the Storybook toolbar.

• A lint rule (@pandacss/no-primitive-token) that complains when a component uses a raw palette shade like neutral.0 instead of a semantic token.

Try it yourselfCopy Anchor Link

bun run storybook   # http://localhost:6006

Open the Tokens page and flip the theme in the toolbar: every swatch changes. Then create a scratch file with some mistakes:

// packages/ds/src/scratch.tsx
import { css } from "styled-system/css";
export const bad = css({
  bg: "#ff0000",
  color: "neutral.0",
  px: "12px",
});

bun run lint

You should see prefer-token for the red and the 12px, and no-primitive-token for neutral.0. Delete the file afterwards. Guide 02 explains each rule.

Decisions and whyCopy Anchor Link

• The token name carries its job. Because backgrounds start with bg. and text with fg., a grader can spot bg: "fg.muted" (a text color painted as a background) from the name alone. No color math needed.
• Palette colors exist only to feed the semantic ones. brand.700 is valid, but a component that uses it ignores the theme. That is a mistake a benchmark should count.

Where to go nextCopy Anchor Link

Tokens alone do not make a design system. Next: the components.

Chapter 2 · Eight components, and the bugs no check could seeCopy Anchor Link

2026-11-08 · Week 1, block 2 · PR #26

The goalCopy Anchor Link

Build the eight components the fintech tasks will need, each with stories, so that both humans and agents can see how they are meant to be used.

What we builtCopy Anchor Link

• Components in packages/ds/src/: Button, Input, Select, FormField, Card, Table, Badge and Dialog. Each is an Ark UI primitive styled with a Panda recipe. Props are declared as an interface with a comment on each prop, because Storybook and our own MCP server read the docs from there.
• 25 stories with fintech content: transfers, invoices, statements.
• More tokens: spacing, sizes, radii, font sizes, shadows, z-index.
• One focus ring for everything, set globally in globalCss.
• Guide 06.

Try it yourselfCopy Anchor Link

bun run storybook

Open "FormField → With error" and switch to dark mode. Then use the components from code:

import { Button, FormField, Input, Select } from "@vektos/ds";

<FormField label="Amount" error="Enter an amount greater than zero.">
  <Input inputMode="decimal" />
</FormField>
<FormField label="Currency">
  <Select options={[{ label: "Euro (EUR)", value: "EUR" }]} defaultValue="EUR" />
</FormField>
<Button>Send transfer</Button>

Notice there is no styling in that snippet. That is the point: the components already look right, so an agent has no reason to add its own colors or spacing.

Decisions and whyCopy Anchor Link

• Components never draw their own focus ring. It lives in one global rule for .vektos :focus-visible. One less thing for an agent to forget.
• Select takes an options array, not a tree of child components. Fewer ways to hold it wrong.
• Recipes stay private. Agents get components, not the raw styles behind them.
• Popups wrap themselves in a vektos div. The Select list and the Dialog render outside the page tree (in a portal), so without this they would lose the theme.

What went wrongCopy Anchor Link

Two bugs passed TypeScript, lint and the accessibility checker, and showed up only when we looked at screenshots. Every empty required field was painted red (the _invalid condition also matched the browser's native :invalid). And the Select placeholder was styled on the wrong element. Full story in anecdote 02. The lesson for the benchmark: green checks are not enough, so every task result will get a screenshot.

A test script also reported the Dialog as broken. It had clicked a hidden button in Storybook's loading skeleton. Selectors now start from #storybook-root.

NumbersCopy Anchor Link

25 stories, 0 accessibility violations across 50 runs (25 stories × 2 themes).

Where to go nextCopy Anchor Link

The design system exists. Now we need the program that checks agent code against it.

Chapter 3 · The graders: counting mistakes fairlyCopy Anchor Link

2026-11-08 · Week 1, block 4 · PR #28 (built before block 3, because the MCP server's check_code tool needed it)

The goalCopy Anchor Link

Turn "did the agent follow the design system?" into a list of violations a computer can produce, with as few false alarms and as few misses as possible.

What we builtCopy Anchor Link

Four graders in packages/bench/src/graders/, all returning the same Violation shape (rule, file, line, message):

• raw-values: hardcoded colors, sizes and numbers. It now also catches the escape hatch ("[#fff]"), template strings, named colors like "red" and bare numbers like marginTop: 12.
• token-roles: the right kind of token used for the wrong job (bg: "fg.muted"), and palette shades used directly. The rules are a small table, VEKTOS_ROLES, so another design system can bring its own.
• suppressions: every @ts-expect-error, lint-disable comment or vektos-ignore.
• protected-files: edits to files an agent should not touch, like panda.config.ts, tsconfig.json or package.json.

Plus a CSS path (through PostCSS) for plain .css files, and one entry point, gradeSource(file, code), in packages/bench/src/grade.ts. Guide 03.

Try it yourselfCopy Anchor Link

cd packages/bench
bun test

Or from code:

import { gradeSource } from "@vektos/bench";

gradeSource("page.tsx", `<p className={css({ bg: "fg.muted" })} style={{ marginTop: 12 }} />`);
// token-roles/wrong-role  bg = "fg.muted"   fg.* is for text and icons
// raw-values/raw-number   marginTop = "12"  becomes 12px, use a token

To keep a raw value on purpose, explain it on the line above:

// vektos-ignore: brand color from the client, not in the theme yet
bg: "#ff6b00",

The violation moves to an ignored list, and the comment itself counts as a suppressions violation.

Decisions and whyCopy Anchor Link

• Only style properties are graded. The first grader flagged any string that looked like a value. In fintech data, rate: "12%" or width in a fixture is data, not style. Now a value counts only if its key is a real CSS property (from known-css-properties) or a Panda shorthand.
• Hiding a mistake is never free. vektos-ignore silences one violation but is counted itself, and so is every lint or TypeScript suppression. Otherwise an agent could "pass" by sprinkling disable comments.
• The design system grades itself. A test runs every grader on the design system and the web app. Our own code must be clean, or the graders are wrong.

What went wrongCopy Anchor Link

On a three-line fintech sample, the old grader had 4 false alarms and missed 3 real mistakes. The new one gets all of them right. That story is anecdote 03.

The reference run on our own 40 files found one real hit: minHeight: "100vh" in a Storybook decorator. It is now the repo's first vektos-ignore, with a reason.

The tokens guide claimed nothing blocked the escape hatch "[#fff]". Panda's prefer-token lint rule already does; the guide was corrected.

NumbersCopy Anchor Link

33 tests plus 2 written-down gaps (todo). 40 files graded, 1 real violation.

Known gapCopy Anchor Link

A data key that happens to share a CSS property name (width: 3 in a chart config) is still graded as a style.

Where to go nextCopy Anchor Link

The graders can judge code. Now the agent needs a clean way to learn the design system before writing it.

Chapter 4 · An MCP server that teaches the design systemCopy Anchor Link

2026-11-08 · Week 1, block 3 · PR #29

The goalCopy Anchor Link

Give agents one place to ask "what components exist, what props do they take, which tokens can I use?", with answers that are short, correct and built from the source, so they never drift from the code.

What we builtCopy Anchor Link

packages/ds-mcp, a small MCP server that talks over stdio. Its tools:

Where the answers come from:

• Props: react-docgen reads the interface and its comments in each component file (packages/ds-mcp/src/catalog/components.ts).
• Examples: the stories, rewritten with oxc into snippets that work on their own, with the props written out and the import line added (catalog/examples.ts).
• Tokens: imported straight from panda.config.ts (catalog/tokens.ts).
• Rules: the MDX pages, minus the sections meant for maintainers (catalog/guidelines.ts).

Guide 07.

Try it yourselfCopy Anchor Link

bun run ds-mcp              # starts the server, waits for an agent on stdin
cd packages/ds-mcp && bun test

To use it with Claude Code:

claude mcp add vektos-ds -- bun /path/to/vektos/packages/ds-mcp/src/cli.ts

Then ask "how do I show a list of invoices with a status?". The agent should call get_component with Table and read, among the rest:

## TableCell
| Prop | Type | Default | Description |
| numeric | `boolean` |  | Aligns to the end with tabular figures, so amounts line up digit by digit. |

Decisions and whyCopy Anchor Link

• Markdown answers, not JSON. Same information, fewer tokens, and models read tables well.
• Only semantic colors are listed. Palette shades are left out, because using one is a violation. Why show an agent the wrong answer?
• Maintainer sections are dropped. "How to add a token" is useful to us and harmful to an agent on a task: editing panda.config.ts is a protected-file violation.
• check_code is off by default. An agent that can run the grader before answering is a different experiment, so it becomes its own column in the results.
• We built our own instead of only using Storybook's MCP. We compared the two (below), and the difference was too big to ignore.

What went wrongCopy Anchor Link

• Storybook's MCP never sends the token names. Its tokens page arrives as raw MDX, so the live table is just the text <SemanticTokenTable />. Its tool list is also 17,852 characters against our 1,399. Full story in anecdote 04.
• Spacing came out as 0, 1, …, 8, 0.5, 1.5. JavaScript puts integer-like keys first in an object. Both our server and the Storybook token page now sort numeric scales.
• bun run --filter does not pass stdin through, so the server got no messages. The fix is a root script, bun run ds-mcp.
• The palette check missed shade 0 (neutral.0). Fixed in the graders.
• A colon in a docs description: broke the docs build. YAML needs quotes there.

NumbersCopy Anchor Link

17 tests in ds-mcp, 50 in the repo. Every example generated from the real stories grades with zero violations.

Where to go nextCopy Anchor Link

Week 2: five fintech tasks with sample data, a runner that hands a task to a model together with this MCP server, a price table to turn token counts into dollars, and a database for the results.

Chapter 5 · Five tasks, and why an empty page must failCopy Anchor Link

2026-11-08 · Week 2, block 1 · PR #31

The goalCopy Anchor Link

Write the work we will hand to agents: five realistic fintech screens, each with data to show and a hand-written answer that proves the job can be done with the design system as it is.

What we builtCopy Anchor Link

A new folder, tasks/, with one sub-folder per task. Every task has the same four files:

• task.json: the id, a title, a one-line summary, the components a good answer would use, and a mustShow list (more on that below).
• prompt.md: what to build, written like a ticket from a product manager.
• fixture.json: made-up but realistic data. Italian IBANs, a salary, a rent payment, a loan schedule that adds up.
• solution.tsx: our own answer.

The five screens are an account statement, a SEPA transfer form, a loan contract page, an identity check (KYC) form and a list of payments to approve through a dialog.

In packages/bench/src/tasks.ts, loadTasks(dir) reads the folders and checks them: the metadata must match a schema, the id must match the folder name, the fixture must be valid JSON. In tasks/tasks.test.ts, every solution is graded, checked for the components it should use, and rendered to HTML to look for its mustShow texts.

Try it yourselfCopy Anchor Link

cd tasks
bun test

You should see 21 passing tests, four per task plus one that counts the tasks. Open tasks/statement/prompt.md and then tasks/statement/solution.tsx side by side: the prompt never says "use the Table component", but the solution does. Whether an agent finds that out on its own is exactly what the benchmark measures.

To see what the runner will get, from any TypeScript file in the repo:

import { loadTasks } from "@vektos/bench";
import { tasksDir } from "@vektos/tasks";

const [first] = await loadTasks(tasksDir);
console.log(first.id, first.mustShow);
// kyc-form [ "Giulia Rossi", "giulia.rossi@example.com", "Date of birth", ... ]

Decisions and whyCopy Anchor Link

• The prompts never mention the design system. How much the agent knows about it is the thing we vary between runs (no context, Storybook's MCP, our ds-mcp). If the prompt said "use our Table", every context level would look the same.
• An empty page must fail. This was the surprise of the block. The graders count mistakes, so a page with no code has no mistakes: a perfect score for doing nothing. mustShow fixes that. Each task lists a few texts the finished page cannot avoid showing, like the account holder's name or the "Amount" label. A run that misses them is incomplete, whatever its violation count. We picked names and labels, not amounts, because €4,821.37 and 4.821,37 € are both right.
• Values inside form fields count as visible. The KYC form shows the customer's name in a read-only input. It is on screen, but it is an attribute, not text, so the first version of the check missed it.
• Tasks are plain files. A team using Vektos with its own design system should be able to add tasks without writing code. That is risk 1 in the plan.

What went wrongCopy Anchor Link

All five solutions graded clean on the first try. Then lint found twelve problems in them: the tasks/ folder was not in Panda's include (so the pages would have been unstyled), a role="status" where <output> is the native element, two physical properties (pt, borderTopWidth) where the design system wants logical ones, and four type casts. The graders and the linter measure different things, and the plan already said oxlint should be a grader; now we know why. Full story in anecdote 05.

The casts came from the fixtures: JSON turns "status": "pending" into a plain string, so looking up a badge by status needs either a cast (a lint warning) or a Record<string, …> with a fallback. The solutions now use the second. Agents will meet the same problem.

NumbersCopy Anchor Link

5 tasks, 21 task tests, 5 loader tests. 76 tests in the repo, all passing.

Where to go nextCopy Anchor Link

The pricing package, so every run can turn token counts into dollars. Then the runner: give a task and a context to a model, collect the files it writes, grade them.

Chapter 6 · What does a run cost?Copy Anchor Link

2026-11-08 · Week 2, block 2 · PR #32

The goalCopy Anchor Link

Turn "this run used 12,000 input tokens and 1,500 output tokens" into dollars, in a way that stays correct when prices change.

What we builtCopy Anchor Link

A small package, packages/pricing, with no dependencies:

• src/prices.ts holds PRICES, a table of ten current models from Anthropic, OpenAI, Google, DeepSeek, Moonshot and Mistral. Each entry has the model id, the day the price starts, four rates in dollars per million tokens (normal input, cache read, cache write, output), long-context rates where they exist, and where the numbers came from.
• src/cost.ts holds cost(usage, model), which prices one request, and sumCosts, which adds up the requests of a run.

The input is the usage object the AI SDK already returns, so the runner can pass it straight through.

Try it yourselfCopy Anchor Link

cd packages/pricing
bun test

Then, in any TypeScript file:

import { cost } from "@vektos/pricing";

const c = cost({ inputTokens: 12_000, outputTokens: 1_500 }, "anthropic/claude-sonnet-5.5");
console.log(c.usd.total); // 0.039: 12k × $2 + 1.5k × $10, per million

Try a model that is not in the table, like "acme/model". It throws instead of guessing.

Decisions and whyCopy Anchor Link

• Prices are never edited, only added. Model prices drop every few months. If we changed the table in place, last month's runs would quietly get cheaper and every "cost over time" chart would lie. A new price is a new entry with a later start date, and each run is priced with the entry valid on the day it ran.
• Price each request, not the whole run. Some models charge more for every token once a single prompt gets very long (over 100,000 tokens for Claude Haiku 5.5). An agent run is a chain of requests. Adding them up first and pricing the total would push a run of many small requests into the expensive tier by mistake.
• Cached tokens are their own line. A cache read costs a twentieth of a normal input token on the newest Claude models. A big but stable context, like an MCP server's tool list that is sent with every request, can be much cheaper than its token count suggests. The benchmark needs to see that difference, so the cost keeps every bucket.
• Model ids are gateway ids. anthropic/claude-sonnet-5.5 picks the model in the AI SDK and its price in our table. One string, no mapping.
• Where the numbers come from. The Vercel AI Gateway publishes a public list of every model it serves with per-token prices (/v1/models). We copied ten of them and wrote the date we checked next to each one.

What went wrongCopy Anchor Link

Nothing broke, but the price list was more complicated than the plan assumed. Besides the four basic rates, models have long-context tiers, regional prices, "fast" and batch variants and service tiers. We kept only what a benchmark run needs (basic rates and long-context tiers) and wrote the rest down as known gaps in guide 09.

NumbersCopy Anchor Link

10 models, 10 pricing tests, 86 tests in the repo.

Where to go nextCopy Anchor Link

The runner: give a model a task and a context (none, or our MCP server), let it write the page, then grade it and price it. First with a fake model, so the whole loop can be tested without an API key.

Chapter 7 · The runner: from a task to a graded, priced runCopy Anchor Link

2026-11-08 · Week 2, block 3 · PR #33

The goalCopy Anchor Link

Connect everything built so far. Give a model a task and a context, let it write a page, then grade the page and price the run. No API key needed to try it: the first version runs on fake models.

What we builtCopy Anchor Link

In packages/bench/src/runner/:

• workspace.ts: a tiny in-memory file system with three tools, list_files, read_file and write_file. It starts with only the task's fixture.json.
• context.ts: the four context levels (none, ds-mcp, ds-mcp-check, storybook-mcp). For the MCP ones it starts the server and hands its tools to the model.
• models.ts: three mock models. mock/ideal writes our reference solution, mock/lazy does nothing, mock/rogue writes the solution and then edits panda.config.ts.
• run.ts: runTask, the loop. It uses the AI SDK's generateText, prices every step with @vektos/pricing, grades every file written and flags protected-file edits.
• results.ts and cli.ts: save a run to results/ and print a table.

Plus docs/user/, the first end-user pages: running a benchmark and writing a task.

Try it yourselfCopy Anchor Link

bun run bench

Five rows, all with page yes, viol. 0 and a cost around a cent. Now the two misbehaving models:

bun run bench --model mock/lazy --task statement    # page NO, viol. 0
bun run bench --model mock/rogue --task statement   # page yes, viol. 1

The lazy one has no violations and still must not pass: that is what page NO and (next block) mustShow are for. The rogue one is caught by the protected-files grader. Compare context costs with the same fake model:

bun run bench --context ds-mcp --task statement     # more input tokens than --context none

Open results/ to see what was saved: a result.json with every step's cost and a files/ folder with the page.

Decisions and whyCopy Anchor Link

• Fake models first. The whole loop is testable now, in a second, for free. A real model only changes who answers.
• The agent cannot see the repo. Its files live in memory and contain only the fixture. It cannot read the reference solution or the design-system code, so the context level is the only way it learns about the design system.
• One system prompt for all runs. It describes the environment (Next, React 19, where to write the page) and names @vektos/ds, nothing more.
• Mock tokens are estimated from the request size, tool list included. Even with a fake model, a bigger context shows up as a bigger bill.
• The MCP server is started by file path. ds-mcp already depends on bench (for check_code), so bench importing ds-mcp would be a cycle.
• Cost per step, then summed, as the pricing chapter explained.

What went wrongCopy Anchor Link

• TypeScript refused to export the tool set: its inferred type pointed into the AI SDK's private folders. Annotating the return type as ToolSet fixed it.
• The first tests checked the write tool by calling it through the generic tool type, which has no input type. The logic moved to a plain writeFile function that both the tool and the test use.
• Lint was stricter than expected: a Promise.all over a mix of promises and undefined, and a switch that did not return in every branch. Both became plain loops and ifs.

NumbersCopy Anchor Link

97 tests in the repo (11 new, in the runner). A full mock run of five tasks costs about $0.05 at the mock rates. The ds-mcp context adds about 500 input tokens per request: its tool list is small, which is the point of building it.

Where to go nextCopy Anchor Link

Check the page itself: tsc, lint, accessibility and a real render in a browser that looks for the mustShow texts, plus a screenshot. Then run a real model, once there is a key.

Chapter 8 · Page graders: does the page work, not only does the code look right?Copy Anchor Link

2026-11-08 · Week 2, block 4 · PR #34

The goalCopy Anchor Link

Until now a run was graded by reading its source. A page that does not compile, crashes when it renders, or shows nothing could still score zero violations. This block opens the page for real: typecheck it, lint it, put it in a browser, check that it shows what the task asked for, and run an accessibility scan in light and dark. It still needs no API key.

What we builtCopy Anchor Link

In packages/bench/src/graders/ two new static graders:

• components.ts: flags <button>, <input>, <select>, <table> and <dialog> when the design system has Button, Input, Select, Table and Dialog. A page can use perfect tokens on a native button and still ignore the design system.
• use-client.ts: flags a page.tsx that calls a hook or passes onClick but has no "use client". Next.js will not build it and tsc does not notice.

In packages/bench/src/page/ the checks that need real files:

• workdir.ts: copies the agent's in-memory files to apps/web/.bench/<id>/ (a gitignored scratch folder where @vektos/ds and react resolve) and deletes it after. Files the agent was not allowed to write (tsconfig.json, panda.config.ts) are left out.
• typecheck.ts and lint.ts: tsc --noEmit and oxlint -f json with the repo's own config, parsed into the same Violation shape as everything else.
• render.ts: bundles the page with Bun.build, generates its CSS with panda cssgen, opens it in Chrome with Playwright, collects console errors, checks the task's mustShow texts against the screen, runs axe-core in the light and then the dark theme, and takes a screenshot of each.
• check.ts: createPageChecker(), one Chrome for a whole batch.

Then the wiring: runTask takes an optional pageChecker, RunResult gets rendered, missing, complete and screenshots, bun run bench shows done, ds and qual columns, and violations are split into three groups: design-system, quality and completeness (GRADER_GROUP).

Try it yourselfCopy Anchor Link

You need Google Chrome installed. Then:

bun run bench --model mock/ideal

task              page  done           ds   qual  steps  tokens in/out   cost
kyc-form          yes   yes            0    0     3      3917/1460       $0.0112
statement         yes   yes            0    0     3      3834/729        $0.0075
...

All five reference solutions pass everything: they compile, lint clean, render, show every mustShow text and have no axe findings in either theme. Open results/<runId>/payment-approval/dark.png to see the page the way the browser did.

bun run bench --model mock/lazy --task statement     # done NO (5 missing)
bun run bench --model mock/ideal --no-page           # the old, static output
bun run check:page                                   # the Chrome tests (the plain check skips them)

Decisions and whyCopy Anchor Link

• Three groups, not one score. A native <button> with perfect tokens is a design-system failure, a crash is a quality failure, an empty page is a completeness failure. One number would hide which context level helped with what.
• Lint reports as "quality" and drops two rules. prefer-token and no-primitive-token say the same thing as the raw-value and token-role graders, so keeping them would count every raw color twice.
• The scratch folder lives inside apps/web. That is the one place where the design system, React and the styled-system path all resolve without copying node_modules.
• System Chrome, not a downloaded browser. It is already installed, nothing to download, and the checks (does it mount, what does it say, does axe pass) do not change between Chrome versions. A future visual diff needs a pinned Chromium: guide 11 lists the exact steps and PLAN notes it.
• axe in both themes, because of anecdote 02: a token that passes contrast in light can fail in dark.
• Development bundle. React only reports a missing key or invalid nesting in development mode, so that is what the page runs in.
• vektos-ignore can silence a native element but not a missing "use client". The first is a design choice; the second is a build error.

What went wrongCopy Anchor Link

• oxlint linted nothing. The scratch folder is gitignored and starts with a dot, so oxlint answered "No files found to lint" with an empty list: a clean result for every possible page. Files are now passed by name and any output that is not the JSON report is an error (anecdote 06).
• The new grader flagged our own docs. Two native <table> in the Storybook token page. They are intentional, so they got a vektos-ignore with the reason.
• A syntax error has no lint code. oxlint's JSON leaves out code for a parse error and the schema rejected it. It now defaults to syntax-error.
• A hidden folder and --no-ignore did not help; only naming the files did.

NumbersCopy Anchor Link

128 tests in the repo, 9 of them start Chrome and run with bun run check:page. A full mock run of five tasks, with typecheck, lint, render, two axe scans and ten screenshots, takes about 6 seconds; the static graders alone take about 3. Per page: tsc about 0.2 s, panda cssgen about 0.1 s, the browser the rest.

Where to go nextCopy Anchor Link

Run a real model: the whole pipeline now works end to end on fake ones, and a real run is the first time the numbers mean something. After that, the database (packages/db), and then the RAG and chat weeks.

Chapter 9 · The first real runs, and checking the billCopy Anchor Link

2026-11-09 · Week 2, block 5 · PR #35

The goalCopy Anchor Link

Until now every run used fake models. This block runs two real ones through the Vercel AI Gateway, for a few cents, to answer three questions. Does the whole pipeline work with a real model? Is our cost the same as the bill? Can a batch be stopped before it spends too much?

What we builtCopy Anchor Link

• A gateway account with credit, and a key kept in Infisical. The key never lives in a file: infisical run -- bun run bench ... hands it to the one command that needs it. The secret has to be named exactly AI_GATEWAY_API_KEY; ours was first saved as AI_Gateway_API_KEY, and environment variables are case-sensitive.
• A free model in the price table. nvidia/nemotron-3.5-lightning-free costs 0, which makes it a free way to test the key and the tools. The runner refuses a model with no price, so it needed an entry.
• The gateway's own charge, next to ours. Each response carries providerMetadata.gateway.cost. The runner stores it per step (gatewayUsd), the CLI prints gateway $... in the total line and warns if our price is more than 2% off (costDrift).
• --budget <usd>. The CLI stops starting new tasks once the spend reaches it. --out <dir> moves the results, so tests do not fill results/.

Try it yourselfCopy Anchor Link

You need a gateway key (see running a benchmark).

infisical run -- bun run bench --model nvidia/nemotron-3.5-lightning-free --task statement
infisical run -- bun run bench --model openai/gpt-6-luna --task statement

task        page  done           ds   qual  steps  tokens in/out   cost
statement   yes   NO (5 missing) 20   44    3      5557/1461       $0.0000     (Nemotron)
statement   yes   yes            146  181   4      6881/3768       $0.0026     (Luna)

total $0.0049  ·  gateway $0.0049                                              (a later Luna run)

Then try the budget on fake models, where it is free:

bun run bench --budget 0.005 --no-page
# one task runs, then: budget $0.005 reached after $0.0112: 4 task(s) not run.

Decisions and whyCopy Anchor Link

• Compare with the bill in code, not by eye. The first check was a screenshot of the Vercel dashboard. It matched to four decimals, but four decimals cannot tell a $0.100 rate from a $0.125 one. The gateway's number in every response can, on every run.
• The budget stops tasks, not steps. Cutting a task in the middle would leave a half-written page with a partial cost. --max-steps already limits one task.
• Free models are for debugging. They are rate-limited, the provider may keep prompts and nobody else uses them. They stay out of any comparison.

What went wrongCopy Anchor Link

• The key did not arrive: the secret was called AI_Gateway_API_KEY. The CLI said key missing and the fix was one rename.
• The first doubt about pricing was real. Luna reported 1,158 tokens as cache writes, which OpenAI normally does not charge, and we priced them at $0.125 per million. If the gateway had billed them as normal input, every OpenAI run would be 5% too high. A run with the new check showed the two totals equal to the last digit: the table was right.

NumbersCopy Anchor Link

• Nemotron: 3 requests, $0, a page that crashed on render (style={css(...)}: css() returns a class name, not a style object) and 5 of 5 required texts missing.
• Luna, no context: a page that works and shows everything, with 146 design-system violations on the first run and 115 on the second: 99 raw lengths, 47 raw colors and no design-system component. The same model, task and context gave 146 and 115, so one run is not a measurement; repeats are needed before any ranking.
• Cost of the whole block: about $0.01 of the $10 credit.

Where to go nextCopy Anchor Link

One model across the four context levels on one task, to see whether ds-mcp moves the numbers at all. That costs about fifty cents and needs bun run storybook running for the Storybook level. If it does, the full pass over six models, with repeats.

Chapter 10 · A database for the historyCopy Anchor Link

2026-11-09 · Week 2, block 6 · PR #36

The goalCopy Anchor Link

Real runs had started, and they lived in results/: a gitignored folder, with no backup and no way to ask "how did this model do across all runs?". Before spending more credit, give the runs a proper home, with enough context saved beside each one to tell later whether two runs can be compared at all.

What we builtCopy Anchor Link

• Postgres 17 with pgvector in Docker (docker-compose.yml, bun run db:up), on port 5433 because a Postgres was already running on 5432. pgvector is not used yet; the RAG block will need it and changing the image later would mean moving the data.
• packages/db: the Drizzle schema, the first migration and a small connect(). Six tables: batches, runs, steps, violations, run_files, run_screenshots.
• Provenance (provenance.ts, tasks.ts): the git commit and whether the tree was dirty, a hash of the design system's tokens and components, and a hash of each task. They are saved with every batch and run.
• store.ts: saveBatch and saveRun, idempotent, one transaction per run.
• Three commands: bun run bench:import loads the old results/ folders, bun run bench:history prints one row per model and context, and bun run bench now saves real runs by itself (--save and --no-save override that, --label groups repeats).

Try it yourselfCopy Anchor Link

bun run db:up
bun run db:migrate
bun run bench:import
bun run bench:history

Imported 3 run(s) in 3 new batch(es); 0 already saved.

model                                 context        runs  ds      quality  done   cost
nvidia/nemotron-3.5-lightning-free    none           1     20.0    44.0     0%     $0.0000
openai/gpt-6-luna                     none           2     130.5   156.0    100%   $0.0075

Run the import again: 0 run(s) ... 3 already saved. Then check the tests that need the database: bun run check:db.

Decisions and whyCopy Anchor Link

• Drizzle 0.45, not the 1.0 release candidate. The plan said to revisit in week 2. 1.0 is at rc.5; the stable line is enough for these tables, and switching later is a codemod, not a rewrite.
• Provenance first, not later. A number is only meaningful with the code that produced it. If a token changes between two batches, a drop in violations may be the design system's doing, not the context's. The history flags rows built from more than one design-system version.
• Violations are rows, with their group, so the dashboard and the chat can ask which rules a context fixes. Silenced ones are kept, flagged ignored.
• Mocks do not save by default. They test the pipeline; in the history they would be noise. A real run refuses to start without the database, so a paid run is never lost.
• Tests make their own database, created from the migrations and dropped after, so they never touch the real data. They run with bun run check:db.
• The compose database is local. The password is in the file on purpose: it is not reachable from outside the machine.

What went wrongCopy Anchor Link

• I cd'd out of the repo. A chained command went up one directory too far and ran the formatter and the tests from the parent folder. Nothing was damaged (it only found our files), but the tests passed for the wrong reason until I reran them from the root.
• The importer crashed on a run with no files. A mock that writes nothing has no files/ folder, and the glob threw on the missing directory.
• Two lint warnings had been hiding in my own output. I had read only the last lines of bun run check and missed consistent-return warnings in the code of the previous block. Fixed in a separate commit.
• The three imported runs have no provenance. They ran before it existed, so their commit and design-system hash are null. They stay in the history, marked by what they lack.

NumbersCopy Anchor Link

152 tests in the plain check, 17 of them skipped (Chrome or database); check:db runs 104 with the database. Three real runs imported. Postgres 17.11 with pgvector 0.8.7.

Where to go nextCopy Anchor Link

Now the credit can be spent safely: one model across the four context levels on one task (about fifty cents), --labeled so the repeats group together. Then the full pass, and the dashboard on top of this history.

Chapter 11 · Backups: a copy of the historyCopy Anchor Link

2026-11-09 · Week 2, block 7 · PR #37

The goalCopy Anchor Link

The database lived in one Docker volume on one laptop. A wrong cleanup command or a dead disk would have erased every real run, and real runs cost money. Add a backup that works today, and that is the same tool the finished product will use.

What we builtCopy Anchor Link

In packages/db:

• backup.ts: backup() writes a compressed pg_dump (vektos-<timestamp>.dump) to ~/Backups/vektos and keeps the newest 14. restore() loads a dump into a new database and refuses to write over the one in use. listBackups() and prune().
• backup-cli.ts behind three commands: bun run db:backup, db:backups, db:restore.
• Automatic copies: after every real batch of bun run bench, and before db:migrate when the database already has tables.

Try it yourselfCopy Anchor Link

bun run db:backup
# Backup written: ~/Backups/vektos/vektos-2026-11-09T10-00-00.dump (0.55 MB)
bun run db:backups
bun run db:restore ~/Backups/vektos/vektos-2026-11-09T10-00-00.dump --into vektos_copy
docker exec vektos-db-1 psql -U vektos -d vektos_copy -c "select count(*) from runs"
docker exec vektos-db-1 psql -U vektos -d vektos -c "drop database vektos_copy"

Compare the counts of vektos and vektos_copy: they are equal. We did this with the real data (3 batches, 3 runs, 12 steps, 637 violations, 4 screenshots) before trusting it.

Decisions and whyCopy Anchor Link

• The tools run inside the container. The pg_dump on this Mac is version 14 and the server is 17; a dump tool older than its server refuses to run. For the compose database the code uses docker exec, for any other DATABASE_URL the pg_dump on the PATH.
• Restore never overwrites. It creates a database and prints its URL. Making it the live one is a deliberate step, so a wrong file cannot destroy good data.
• Outside the repo. The folder is ~/Backups/vektos, where iCloud or Time Machine can copy it. A backup inside the project would go with the project.
• After real runs only. A real batch is the data that cost money. Mock batches never trigger a backup, so tests do not fill the folder.
• One tool for local and production. The finished product will run the same commands against a managed Postgres, schedule them nightly and send the file to a bucket. The plan is in guide 12.

What went wrongCopy Anchor Link

• A script landed in the wrong file. I edited package.json from inside packages/db and changed that package's file instead of the root one; bun run db:backup said "script not found". I reverted it and applied the change at the root.
• A restored database name failed once. The default name contained a capital T from the timestamp, and the safe-name check (lowercase only) rejected it. The test caught it.

NumbersCopy Anchor Link

160 tests in the plain check, 112 with the database. A backup of the three real runs is 0.55 MB, mostly screenshots.

Where to go nextCopy Anchor Link

The spending can start: one model across the four context levels on one task, with --label. After that, the full pass and the dashboard.

Chapter 12 · A report you can look atCopy Anchor Link

2026-11-09 · Week 2, block 8 · PR #38

The goalCopy Anchor Link

The first real runs were in the database, but the only way to see them was a table of numbers or opening PNG files by hand. "39 violations without context, 0 with ds-mcp" is a claim. This block makes the evidence visible: the same task, written by the same model with different contexts, side by side.

What we builtCopy Anchor Link

• bun run bench:report, in packages/bench/src/report/:
  • data.ts reads the latest run of each task, model and context, with its violations, the tools it called, the page it wrote and its screenshots;
  • render.ts turns that into one self-contained HTML page (a pure function, tested without a database);
  • cli.ts writes results/report.html and opens it.
• The page: a summary table, then one section per task with a card per model and context (screenshot, counts, cost, tools, violations by rule, every violation, the code), then the commit and design-system hash of each batch.
• history() learned to filter by model, label and task, so the summary follows the filters.

Try it yourselfCopy Anchor Link

bun run bench:report

Click Dark: every screenshot switches. Click a screenshot: it fills the window, Escape closes it. Open Every violation on the Haiku card without context: each line has the file, the position, the offending value and a sentence on what to do. Now look at the ds-mcp card of the same model: no violations, and the Tools line shows it read the guidelines and the components first.

bun run bench:report --label haiku-contexts --no-open

Decisions and whyCopy Anchor Link

• The latest run per cell. Repeats are averaged in the summary. A card for each repeat would bury the comparison.
• One file with the screenshots inlined. It works offline, can be sent as an attachment and does not depend on results/ still existing.
• Everything a model wrote is escaped. A page, an error message or a violation value is untrusted text going into an HTML file. A test feeds it <script> and checks it stays text.
• Meaning is never colour alone. Every chip carries its number or word; every screenshot has alt text with model, context, task and theme. The report is held to what the benchmark measures.
• A static page, not the dashboard. The dashboard needs login, filters and live data. This is its most useful part, built first, and loadReport() is the data layer it will reuse.

What went wrongCopy Anchor Link

• A flaky test, caused by the page and not the test. Images were loading="lazy". A lazy image whose size is not yet known has zero height, so the browser test failed on one run in three. With inlined data lazy loading buys nothing, so it was removed, and eight runs in a row passed.
• A stale picture of my own work. I looked at the same screenshot file twice and read an old version for the "no screenshot" message; the HTML was right. I checked the file's text instead of trusting the image.

NumbersCopy Anchor Link

15 new tests for the report: 12 for the rendering (one of them in Chrome) and 3 against the database. A report of five runs is 1.2 MB, almost all screenshots.

Where to go nextCopy Anchor Link

Now the evidence can be reviewed as it arrives: the other three tasks and repeats for Haiku, so the table shows means, and then the Storybook MCP context once Storybook is running.

Chapter 13 · The first results: what context does to an agentCopy Anchor Link

2026-11-09 · Week 2, block 9 · PR #39

The goalCopy Anchor Link

Use the pipeline for what it was built for: the same five tasks, two models and the four context levels, repeated enough to see a pattern. About 80 real runs, for under a dollar.

What we ranCopy Anchor Link

• Claude Haiku 5.5, three repeats of every task at every context level (none, ds-mcp, ds-mcp-check, storybook-mcp), label haiku-5tasks.
• GPT-6 Luna, one pass of every task at every level, label luna-5tasks.
• Everything went through the AI Gateway, was saved to Postgres and backed up after each batch. Our price matched the gateway's charge on every run: no drift warning appeared.

Try it yourselfCopy Anchor Link

bun run bench:history
bun run bench:report --label haiku-5tasks

model                       context        runs  ds     quality  done   cost
claude-haiku-5.5            none           16    36.5   41.1     94%    $0.0479
claude-haiku-5.5            storybook-mcp  15    4.3    8.7      100%   $0.0841
claude-haiku-5.5            ds-mcp         16    0.9    2.6      100%   $0.1080
claude-haiku-5.5            ds-mcp-check   16    0.0    2.3      100%   $0.1622
gpt-6-luna                  none           7     141.7  166.7    86%    $0.0238
gpt-6-luna                  storybook-mcp  5     36.2   60.6     100%   $0.0234
gpt-6-luna                  ds-mcp         7     2.1    18.6     86%    $0.0198
gpt-6-luna                  ds-mcp-check   5     0.0    20.0     100%   $0.0276

Open the report and compare the none card of a task with its ds-mcp card.

What the numbers sayCopy Anchor Link

• Context is the biggest lever. From no context to ds-mcp, Haiku goes from 36.5 to 0.9 design-system violations per run, and Luna from 141.7 to 2.1. Both models write pages that work without it: 94% and 86% were complete. They just ignore the design system.
• Standard Storybook MCP helps, but far less than ours. 4.3 and 36.2 violations: it documents the components, and the agents still write raw lengths, raw colors and palette tokens, because nothing in it says tokens exist. Our server's list_tokens and guidelines are what close that gap.
• The weaker model gains more from context, and needs it more. Luna without context is four times as bad as Haiku. With ds-mcp they are within 1.2 of each other.
• More context costs more tokens but little money. ds-mcp sends about four times the input of none; per run, Haiku goes from $0.0030 to $0.0068. Cents on a task.
• ds-mcp-check removes the last violations and nothing else. See anecdote 07: zero design-system violations in every run, quality unchanged, cost up by half.

CaveatsCopy Anchor Link

• Three repeats for Haiku, one for Luna. Luna's own earlier runs without context were 146 and 115 on the same task. Read Luna's numbers as an order of magnitude.
• Five tasks, one design system. The pattern needs other tasks and other models before it is a claim.
• storybook-mcp for Luna ran after a restart, which does not change the comparison but explains a gap in the timestamps.

What went wrongCopy Anchor Link

• A batch stopped halfway. Docker Desktop quit during a Luna batch. The database became unreachable, the run hung instead of failing, and I hit my own tool's time limit. Two tasks of that batch were saved; the batch was run again from the start, so Luna's ds-mcp cell has 7 runs, not 5. A connection with no timeout should fail fast: it is a follow-up.
• Storybook was down when I needed it. The first time because I started a second copy on a port already in use (it exited at once, nothing damaged); the second time because the one running had stopped. I started one for the Luna batch and stopped it after.

NumbersCopy Anchor Link

About 80 real runs, around $0.45 of the $10 credit in this block and about $0.5 in total. Database: 8 context cells of 5 to 16 runs each.

Where to go nextCopy Anchor Link

Three open questions, each a small experiment: does check_code with type and lint checks (ds-mcp-check-full) fix what ds-mcp-check leaves? Do the other models of the plan (Sonnet, GPT-6 Sol, Gemini, Kimi, DeepSeek, Mistral) show the same ordering? And then the dashboard, which can now draw its charts from this history.

Chapter 14 · A database that goes quiet must not hang a paid runCopy Anchor Link

2026-11-09 · Week 2, block 10 · PR #40

The goalCopy Anchor Link

During the first results, Docker Desktop quit in the middle of a batch. The database stopped answering and bun run bench waited forever instead of failing. A paid run must stop and say what happened.

What we builtCopy Anchor Link

• withTimeout() in packages/db: it rejects when a call takes longer than a limit (30 seconds, or VEKTOS_DB_TIMEOUT_MS), saying which call. The connection also has a 10-second connect timeout.
• The CLI uses it for the check before the run, for saving the batch and for saving each run.
• If saving fails in the middle of a batch, the CLI stops, prints the table of what ran, says the runs are in results/ and that bun run bench:import loads them when the database is back, and exits with an error. It does not take the backup of a batch that failed to save.

Try it yourselfCopy Anchor Link

bun test packages/db/src/client packages/bench/src/cli

One of the tests starts a server that accepts the connection and never answers, like a Docker proxy that died, and checks that the CLI fails in under eight seconds with did not answer within 500 ms.

Decisions and whyCopy Anchor Link

• Stop, do not continue without saving. The runs would still be on disk, but each would cost money and be absent from the history and the report.
• The files are written before the save. results/ is the safety net: nothing is lost, only delayed.
• The call is waited on, not cancelled. A transaction stuck on a dead server cannot be cancelled from the client; the caller just stops waiting.

What went wrongCopy Anchor Link

The hang was found the expensive way: by waiting. The test that now covers it needs a fake server, which was not obvious until the failure had already happened once.

NumbersCopy Anchor Link

Three tests for withTimeout, one for the silent server. The timeout test takes under a second.

Where to go nextCopy Anchor Link

A context where check_code also runs the type checker and the linter.

Chapter 15 · A checker that also compilesCopy Anchor Link

2026-11-09 · Week 2, block 11 · PR #41

The goalCopy Anchor Link

Anecdote 07 found that check_code took design-system violations to zero and left type and lint errors where they were. The follow-up was one experiment: a check_code that also runs the compiler and the linter, and the same runs again.

What we builtCopy Anchor Link

• A new context level, ds-mcp-check-full: the ds-mcp tools plus a check_code that runs the design-system graders, tsc and oxlint on the agent's draft (runner/check-tool.ts). It has the same name and inputs as the old one, so the only difference between the two levels is what the tool checks.
• page/static.ts: typecheck and lint without a browser, reused from the page checks.
• A workspaceTools hook on Context, for tools that need the agent's files, and mock/careful, a mock that writes a page and calls check_code.

The tool lives in the runner, not in ds-mcp, because the draft imports ./fixture.json. The MCP server does not have the agent's other files, and without them the compiler would report an error that is not the agent's.

Try it yourselfCopy Anchor Link

bun run bench --model mock/careful --context ds-mcp-check-full --task statement
infisical run -- bun run bench --model openai/gpt-6-luna --context ds-mcp-check-full --label luna-5tasks
bun run bench:history

What the numbers sayCopy Anchor Link

• Luna fixes everything. 20 quality violations per run became zero, in all five tasks. It pays for it: twice the cost and about twice the steps, because it loops between writing and checking until the compiler is quiet.
• Haiku improves, less cleanly. 2.3 to 0.5, with two runs of fifteen left with something.
• The weaker model gains more again, as it did from context in the first place.
• The cost of correctness is small in dollars. Even Luna's heaviest task was 2.5 cents.

Decisions and whyCopy Anchor Link

• Same name, same inputs. If the tool had a different signature the comparison would mix "what it checks" with "how the agent calls it".
• In the runner, with the workspace. The check sees the draft and the project as the agent will leave them, so its verdict matches the final grade.
• No browser. Rendering takes a few seconds and Chrome; the compiler and linter take about a second and catch most of what is left.

What went wrongCopy Anchor Link

• The history table was too narrow for ds-mcp-check-full and glued the context to the run count. A column width, fixed.

NumbersCopy Anchor Link

Seven new tests (the tool and the level). Four more real batches: three for Haiku and one for Luna. About $0.26 spent in this block.

Where to go nextCopy Anchor Link

Two more models, DeepSeek and Kimi, on the five levels. If the ordering holds across four models from four makers, it is a result, not a coincidence.

Chapter 16 · Four models, five contextsCopy Anchor Link

2026-11-09 · Week 2, block 12 · PR #42

The goalCopy Anchor Link

Check whether the ordering found with Haiku and Luna holds for other makers: DeepSeek and Kimi, on all five context levels and five tasks. If it does, it is a result and not a coincidence of two models. The table now lives in Results.

What we ranCopy Anchor Link

• DeepSeek V4.1 Flash: one pass of every task at every level (label deepseek-5tasks).
• Kimi K3.1 Pro: one pass at every level with a budget of $0.80 per batch (label kimi-5tasks), because a smoke test had shown one run costs about $0.11.
• Storybook's MCP was started for the storybook-mcp level and stopped after.

Try it yourselfCopy Anchor Link

bun run bench:history
bun run bench:report --model moonshotai/kimi-k3.1-pro

Open the Kimi card without context and the one with ds-mcp for the same task, and read the cost line on each.

What the numbers sayCopy Anchor Link

• The ordering holds for all four models: none, then Storybook's MCP, then ds-mcp, then a checker, then a checker that compiles. The design-system violations go from 36 to 294 per run without context to about zero with it.
• Kimi without context is the worst of the four: 294 raw-value violations per run, more than twice Luna's. The most capable model writes the largest pages, and each is more chances to break the rules.
• Storybook's MCP works for two models and not for the other two. DeepSeek and Kimi reach 0.4; Haiku 4.3; Luna 36.2. A model that reads the component docs and then also looks around finds the tokens; one that does not, guesses.
• Quality needs a checker that compiles. ds-mcp-check leaves 2 to 20 quality violations per run; with ds-mcp-check-full, three models reach 0 and Haiku 0.5.
• Context can make a run cheaper. Kimi: $0.21 per run without context, $0.13 with ds-mcp. On a task without guidance it writes up to 24,000 tokens; with it, 6,000 to 14,000.

Decisions and whyCopy Anchor Link

• One pass, not three, for the new models. The budget goes first to learning whether the ordering holds; repeats come next for the cells that matter.
• A budget per batch for Kimi. One batch stopped after four tasks. It is better to know a cell is short than to find a $10 bill.
• Long runs in the background. Two foreground batches hit my own tool's time limit; the script now runs in the background and its output goes to a file.

What went wrongCopy Anchor Link

• A foreground command timed out twice. Not a hang: the batches simply took longer than the limit. The second time I split the work and ran the rest in the background.
• Storybook was down when the script reached it. I tried to start it detached with setsid, which macOS does not have, and the first batch that needed it failed at once, before spending anything. I started it as its own background job, then reran only the two tasks that had not run.
• Kimi's cost was underestimated. My estimate for the full six-model pass was $7; Kimi alone cost $4.30.

NumbersCopy Anchor Link

About 160 real runs in the history. $5.50 spent of $10: Haiku $0.59, Luna $0.16, DeepSeek $0.53, Kimi $4.27.

Where to go nextCopy Anchor Link

Repeats for the cells that decide the story, in particular none and storybook-mcp for the models with one run, and then the models left in the plan (Sonnet, GPT-6 Sol, Gemini, Mistral) only if the budget allows. The dashboard can start now: the history has enough to draw.

Chapter 17 · The compiler that counted twiceCopy Anchor Link

2026-11-09 · Week 2, block 13 · PR #43

The goalCopy Anchor Link

A simple question: how is the quality number computed? Answering it showed that a large part of it was the design-system number again. The goal became counting each mistake once. No new model runs were needed: everything came from the runs already stored.

What we builtCopy Anchor Link

• graders/covered.ts: markCovered(violations, files) splits violations into the ones that count and the ones already counted by another grader. A TS2769 is covered when the css, cva or sva call that holds it also holds a raw-values or token-roles violation. Outside a style call, the same line decides.
• A covered column on violations (migration 0001_covered.sql), set when a run is graded and again when it is saved, so runs imported from results/ get it too.
• bun run bench:recount (recount.ts): applies the rule to every stored run and fixes the flag where it differs. Safe to repeat.
• bench:history and the report leave covered violations out. The report still lists them, with a "not counted" chip.
• Anecdote 08, and the Results page with the corrected table.

Try it yourselfCopy Anchor Link

bun run db:migrate        # adds the column, after an automatic backup
bun run bench:recount     # Looked at 109 runs with compiler errors: 3539 violations are covered, 3539 changed.
bun run bench:history     # the quality column is now much lower in the `none` rows
bun test packages/bench/src/graders/covered

Decisions and whyCopy Anchor Link

• Keep the violation, flag it. Deleting would lose the evidence and make the rule impossible to change. A flag can be recomputed (bench:recount), like ignored.
• By call, not by line. Same-line matching covered 81% of the TS2769; by call, 91%. The compiler reports an overload error at one property of the call, not always the bad one.
• Only TS2769. The other compiler errors are real mistakes the design-system graders cannot see. Covering them would hide problems.
• quality still includes render problems. I had planned to make the history column quality-only. Render findings (runtime-error, build-failed) are 4 of 4,800, but dropping them would hide a page that crashes after mounting, which done does not show. So the column stays, and the guide says what is in it.
• No new runs. The data already had what was needed.

What went wrongCopy Anchor Link

• The error was there from the start. Lint had a list of skipped rules for this exact reason; the compiler was the same case and was not on it. It took a question from outside to see it.
• My first idea, same line, was too weak. I measured it before building it (81%) and then looked at the misses: the call was the right unit.
• Map.groupBy does not type-check with the project's lib, so the grouping is a loop.

NumbersCopy Anchor Link

3,895 TS2769 in the real runs; 3,539 covered (91%). Mean quality per run without context: Haiku 41.1 → 8.6, Luna 166.7 → 26.7, DeepSeek 107.8 → 15.6, Kimi 348.5 → 40.0. With ds-mcp-check and ds-mcp-check-full the numbers barely move. 11 new tests (8 for the rule, 2 for the database, 1 for the report). $0 spent.

Where to go nextCopy Anchor Link

The 9% that stays is mostly token names that do not exist (fontWeight: "bold", fontSize: "3xl"). A grader that checks style values against the token table would name that mistake. Older chapters (15 and 16) show the numbers as they were then; the Results page has the corrected ones.

Chapter 18 · The dashboardCopy Anchor Link

2026-11-09 · Week 2, block 14 · PR #44

The goalCopy Anchor Link

Until now the results lived in a terminal table and a static HTML report. The plan's "minimal dashboard" is the piece that makes them a product: an app you can open, see how the contexts compare, and click into any run. The rule for the block: build it with the design system, and run no new benchmark.

What we builtCopy Anchor Link

• Queries in packages/bench/src/dashboard.ts: overview, listRuns, filterOptions, getRun, getScreenshot. They count like bench:history (ignored and covered violations left out). history was split into the query and its CLI, and CONTEXT_ORDER moved to its own file, so the app imports no runner code.
• Subpath exports @vektos/db/client, @vektos/db/schema, @vektos/bench/dashboard, so Next bundles only what the pages use.
• Pages in apps/web/src/app: Overview (totals, the context × model table, the cost-versus-violations chart, the full table), Runs (links as filters), a run's page (cost, batch, screenshots, violations, steps, code), the screenshot route, an error page for a database that is down, and a not-found page.
• A dark-mode switch that remembers the choice and restores it before first paint.
• The chart is hand-written SVG on log scales: color is the context, shape is the model.
• dogfood.test.ts: the benchmark's graders run on apps/web/src and expect zero design-system violations.
• Two tokens (fontSizes.xl, sizes.page), a lint override for scrollable regions, and the PostCSS fix from anecdote 09.

Try it yourselfCopy Anchor Link

bun run db:up && bun run db:migrate
bun run web                       # http://localhost:3000

Look at the chart: the bottom-left is the goal. Kimi without context is the red diamond at the top right: expensive and non-compliant. Every ds-mcp-check-full point sits on the bottom line. Click a model name in the full table, then a run, then compare its two screenshots.

bun run build                     # production build
bun test packages/bench/src/dogfood
bun run check:db

Decisions and whyCopy Anchor Link

• Server components, filters as links. No client state to get wrong, and a filtered view is an address.
• One dependency, not a chart library. Twenty points do not need one, and a library would bring its own colors outside the tokens.
• The chart is hidden from screen readers, the table is not. The same numbers are in the table next to it, with a label for each cell.
• The app does not import the runner. Subpath exports keep the AI SDK and Playwright out of the Next bundle.
• New tokens, knowingly. The app needed a larger heading and a page width. Adding them makes later benchmark batches a different design-system version, and bench:history will say so.

What went wrongCopy Anchor Link

• Half the design system's CSS was missing and nothing failed. See anecdote 09: one line in the PostCSS config.
• The design system has holes agents also hit: no breakpoints, no monospace font, a spacing scale that stops at 8. I wrote a layout that way (flexWrap, flexBasis) because it was the only responsive option.
• A lint rule and axe disagreed about tabIndex on a scrollable table. The answer is in the config, not in a suppression comment.
• next dev kept stale CSS after the config changed; deleting .next fixed it.

NumbersCopy Anchor Link

Five routes, four of them dynamic. 7 tests for the queries, 1 for the design-system check. axe finds nothing on the three pages, light and dark. About 20 files in apps/web, none over 280 lines. $0 spent on models.

Where to go nextCopy Anchor Link

The grader for token names that do not exist (it would also tell the dashboard's author about fontFamily: "mono"), then repeats for the cells with one run, and the RAG chat the plan puts next.

Chapter 19 · Closing the design system's gapsCopy Anchor Link

2026-11-09 · Week 2, block 15 · PR #45

The goalCopy Anchor Link

Building the dashboard (chapter 18) found things the design system could not say: no breakpoints, no monospace font, a spacing scale that stopped at 8. They are the same walls the agents hit. This block fixes them, from data, and brings the dashboard up to date.

What we builtCopy Anchor Link

• Breakpoints sm 40rem, md 48rem, lg 64rem, xl 80rem, mobile first, exported as breakpoints from panda.config.ts and set as theme.breakpoints.
• Fonts: sans and mono. With a font category, strictTokens makes any other fontFamily a type error, so "monospace" and "Inter" no longer compile.
• Spacing 5, 10, 12, 16 and sizes 0, 6, 12, 16.
• The MCP server lists fonts and breakpoints through list_tokens, with a line on how breakpoints work; the Storybook "Tokens" page has a Fonts and a Breakpoints section.
• The dashboard uses them: a two-then-four column totals grid, padding that grows from md, the mono font, and minW: "0" on its columns.

How the values were chosenCopy Anchor Link

From the 159 pages agents wrote in the real runs:

So four breakpoints, the two families, and the spacing steps agents actually reached for. 7 and 14 are not added: one use is not a pattern.

Try it yourselfCopy Anchor Link

bun run --filter @vektos/ds prepare
bun test packages/ds-mcp
bun run storybook        # the Tokens page, Fonts and Breakpoints
bun run web

In an editor, write css({ fontFamily: "monospace" }) in apps/web and see the error; "mono" is accepted.

Decisions and whyCopy Anchor Link

• Close the gaps in the design system, not in the grader. An agent that writes md was right to expect a breakpoint; the system was missing a word. A grader for names that do not exist is next, and it should have fewer false alarms now.
• Evidence, not taste. Every step added has a use in the stored runs or in the dashboard.
• A font category makes fontFamily strict for free. No new rule was needed.
• This is a new design-system version. bench:history flags batches with a different dsHash. The results page now says its tables belong to the earlier version.

What went wrongCopy Anchor Link

• tsc kept passing on the old types. After prepare, apps/web still compiled fontFamily: "monospace" because of a stale tsconfig.tsbuildinfo. Deleting it showed the four errors. It is now a known gap in guide 01.

NumbersCopy Anchor Link

206 tests, none failing. Storybook builds. axe finds nothing on the dashboard at 1280, 800 and 390 px, and none of the three overflows sideways. No model run. $0.

Where to go nextCopy Anchor Link

The grader for token names that do not exist (fontWeight: "bold", fontSize: "3xl", a fontFamily that is not a token). It should also say when a breakpoint does not exist.

Chapter 20 · A grader for names that are not tokensCopy Anchor Link

2026-11-09 · Week 2, block 16 · PR #46

The goalCopy Anchor Link

Anecdote 08 left a class of mistake that no design-system grader could see: fontWeight: "bold", fontSize: "3xl", a breakpoint that does not exist. Not raw values, not tokens. The compiler rejects them with "No overload matches this call", and the benchmark counted that as a quality error. Now they are design-system violations with a message that names the choices.

What we builtCopy Anchor Link

• token-names (graders/token-names.ts), in the design-system group, with two rules: unknown-token and unknown-breakpoint.
• A token table built from the config (tokenTableFromConfig): scales by category, semantic colors, palette, breakpoints. VEKTOS_TOKENS reads packages/ds/panda.config.ts, so a token added there is valid here with no second edit, and another design system passes its own table. gradeSource takes it as an optional third argument.
• A written-out map from property to category (p, marginInlineStart to spacing; maxW to sizes; fontFamily to fonts…), and the keywords the compiler accepts besides tokens (auto, fit-content, inherit…), found by compiling a probe file and reading which lines failed.
• covered: a TS2769 in a call with a token-names finding is now covered too.
• check_code of the ds-mcp-check* contexts reports it with no change: it runs gradeSource.

Try it yourselfCopy Anchor Link

bun test packages/bench/src/graders/token-names
bun run bench --model mock/ideal --task statement --no-page

And ask the checker:

css({ fontWeight: "bold", p: { base: "4", tablet: "7" } })

It returns an unknown-token for bold (choices: normal, medium, semibold) and an unknown-breakpoint for tablet (choices: sm, md, lg, xl).

Checked against the stored runsCopy Anchor Link

I replayed the grader on the 121 stored runs with compiler errors, using the token table as it was when they ran (no breakpoints, no mono, the old spacing and sizes):

The commonest: fontWeight 700/600/bold/650, fontSize 2xl/3xl, lineHeight 1.2/1.5/relaxed, maxWidth 6xl/5xl, and the breakpoint md (before it existed).

Decisions and whyCopy Anchor Link

• Style calls only. A page's own data ({ width: "narrow" }) is not a style. Panda reads css, cva and sva; so do we. JSX style props and inline style are left out.
• Prefer a miss to a false alarm. Multi-part values, references (token(…)), numbers and categories with no tokens are skipped. A grader that cries wolf costs the benchmark its trust.
• Raw values stay with raw-values. 10px is one finding, not two.
• No backfill. Old runs were made against an older design system; grading them with today's table would make them look better than they were. They stay as they are, and the guide says so.
• A recipe variant is not a condition. variants: { color: { primary: … } } has a key that maps to an object; a breakpoint maps to a value.

What went wrongCopy Anchor Link

• The first version flagged a recipe. size: { sm: { fontSize: … } } made fontSize look like a breakpoint, because size is a CSS property. A test now pins it.
• 100dvh came out as an unknown token. Replaying the grader showed that raw-values did not know the dynamic and small viewport units (dvh, svh, lvh). It does now; a grader's test on real data found a gap in another grader.
• letterSpacing: "tight" is not caught. The design system has no letter-spacing scale, and the compiler lets any value through there. It is in the known gaps.

NumbersCopy Anchor Link

24 new tests for the grader and one for the units: 231 in bun run check, 182 with check:db. $0 spent: no model call.

Where to go nextCopy Anchor Link

Repeats for the cells with one run, now against the new design-system version and with the new grader, so the numbers can be compared among themselves. That costs money, so it waits for your go-ahead. A smaller step first: a letter-spacing scale, since agents reach for it.

Chapter 21 · A scale for letter spacingCopy Anchor Link

2026-11-09 · Week 2, block 17 · PR #47

The goalCopy Anchor Link

Chapter 20 ended with a known gap: letterSpacing: "tight" passed both the compiler and the grader, because the design system had no letter-spacing scale. This block closes it.

What we builtCopy Anchor Link

• letterSpacings in panda.config.ts: tight (-0.025em), normal (0em), wide (0.05em), wider (0.1em). With a scale, strictTokens rejects any other value, and 0.1em is now a type error.
• token-names maps letterSpacing to that scale, so loose is an unknown-token with the choices listed. A raw 0.1em stays with raw-values.
• A "Letter spacing" section on the Storybook Tokens page; guides 01 and 03 updated.

How the values were chosenCopy Anchor Link

From the stored pages: agents wrote tight spacing between -0.01em and -0.045em (headings) and wide spacing between 0.08em and 0.12em (small uppercase labels). So one step for each (tight, wider), a half step (wide) and the neutral one (normal). Raw values were already violations: 0.1em alone appeared 9 times.

Try it yourselfCopy Anchor Link

bun run --filter @vektos/ds prepare
bun test packages/bench/src/graders/token-names

In apps/web, css({ letterSpacing: "wider" }) compiles; "loose" and "0.1em" give TS2769.

Decisions and whyCopy Anchor Link

• Four steps, not eight. Agents cluster in two places; a scale with a name for each cluster is enough.
• Same design-system version bump as chapter 19. Nothing has been run since, so no batch is split in two by this change.

What went wrongCopy Anchor Link

Nothing this time. Compiler and grader were checked on the same three values and agree.

NumbersCopy Anchor Link

232 tests, none failing. $0.

Where to go nextCopy Anchor Link

The dashboard gaps: a trend per batch, a label filter, pagination.

Chapter 22 · Batches, labels and pagesCopy Anchor Link

2026-11-09 · Week 2, block 18 · PR #48

The goalCopy Anchor Link

Three gaps from the dashboard guide: no way to tell which design-system version a batch used, no way to hide the smoke-test runs, and a run list that stopped at 100. The design system changed in chapters 19 and 21, so "which version?" is now a real question.

What we builtCopy Anchor Link

• /batches: one row per benchmark call with its label, model, context, design-system hash, runs, mean violations and cost. listBatches compares each batch with the one before it in time and marks a "new version" when the hash changed. A batch with no hash (the early ones) does not count as a change.
• Batch filter on /runs and on the Overview (?label=): /?label=haiku-5tasks is the whole overview for one experiment, and the smoke batches are one click away from gone.
• Pagination on /runs: 50 to a page, ?page=, Previous and Next links, and a way back from a page that does not exist.
• A Pager component, a "Batches" link in the header, and six database tests.

Try it yourselfCopy Anchor Link

bun run db:up
bun run web                      # http://localhost:3000/batches
bun run check:db

Open /runs?label=kimi-5tasks, then /?label=kimi-5tasks. Open /runs?page=99 for the way back.

Decisions and whyCopy Anchor Link

• No trend chart yet. All 40 stored batches have the same hash (e02ef397), so a line would be flat and say nothing. The table is the honest first step, and it will show the change as soon as a batch runs on the new design system.
• Offset pagination. Simple, and the list is a few hundred rows. A stable order (started_at desc, id) keeps pages from overlapping.
• Plain links for filters and pages, as before: the address is the state.

What went wrongCopy Anchor Link

• The header overflowed on a phone. A fourth link ("Batches") pushed the dark-mode button off a 390 px screen, on every page. axe does not see this; the overflow check did. The bar and the nav now wrap, and the links are tighter on small screens.
• A branch that could never run. The first "no page" message tested total, but total comes from the rows, and a page past the end has none. Found by reading it back.

NumbersCopy Anchor Link

186 tests with check:db, none failing. axe finds nothing on the new pages at 1280, 800 and 390 px; no sideways scroll at 390 or 320 px. $0.

Where to go nextCopy Anchor Link

The RAG package: ingest the docs, chunk them by heading, search, and measure retrieval.

Chapter 23 · Retrieval, part one: find the page that answersCopy Anchor Link

2026-11-09 · Week 3, block 19 · PR #49

The goalCopy Anchor Link

The plan has a chat that explains every violation and cites its sources. A chat needs to find the right text first. This block builds that, by hand and without a framework: cut the project's Markdown into chunks, keep them in Postgres, search them, and measure how often the search finds the answer. Vectors come in the next block, so there is a plain baseline to beat.

What we builtCopy Anchor Link

• packages/rag, a new package with four parts:
  • chunk.ts: Markdown to chunks. It cuts at headings (not inside code fences), packs paragraphs up to 220 words, cuts a long list between items, never cuts a code block or a table, and joins a tiny section to the one before it. Each chunk keeps its heading path: Graders › Known gaps.
  • ingest.ts: stores documents and chunks for a tenant. A document is skipped when its hash has not changed; a changed one is replaced in one transaction; a document that is no longer given is removed.
  • search.ts: full-text search in Postgres. A question becomes an OR of its words.
  • eval.ts: recall@k and MRR over a dataset of questions.
• Two tables, rag_sources and rag_chunks (migration 0002_rag_lexical.sql). The chunk has a tsvector column that Postgres computes from its heading (weight A) and body (weight B), and a GIN index.
• 40 questions in packages/rag/evals/vektos-docs.json, about the guides, the anecdotes, the plan and the journal, each with the document that answers it and a text the chunk must contain. A test checks that every question still points at something.
• bun run rag:ingest, rag:search "…", rag:eval, and guide 15.

Try it yourselfCopy Anchor Link

bun run db:up && bun run db:migrate
bun run rag:ingest
# 33 documents for "vektos": 33 added, ... ; 326 chunks written
bun run rag:search "why does the grader count the compiler error twice"
bun run rag:eval

The search returns the anecdote "The compiler that counted twice" first. The eval prints recall and MRR, then every question it did not answer at position 1, with what it found instead.

The first measurementCopy Anchor Link

The second one is what the search uses. The misses are mostly questions that use other words than the text: "compile errors" where the docs say "type errors", "ensure the dashboard follows the design system" where they say "dogfood". That is what vectors are for.

Decisions and whyCopy Anchor Link

• No framework. The plan says chunking, retrieval and reranking by hand: they are the parts worth being able to explain.
• Questions are not worded like the headings. A question that copies a heading would measure nothing. They are written the way a person would ask.
• A hit needs the right document and the right text. The right file but the wrong section is a miss; otherwise a long guide always looks found.
• Tenant on every row. Multi-tenant isolation is a risk in the plan (PLAN §8). It is cheaper to put the column in the first table than to add it later, and a test checks it.
• A floor in the tests, not a target. check:db fails if recall@10 drops under 0.5 or MRR under 0.3. The measured values are higher, so a new document does not break the test, and a broken ranking does.
• The database checks the question's characters too. A test sends quotes, backslashes, |, & and a drop table as a question; the words are made of letters and digits only, so none of it reaches the query syntax.

What went wrongCopy Anchor Link

• The ingest said "unchanged" after the chunker changed. I made the chunker cut long lists, ran ingest, and it reported 33 documents unchanged: the hash covered only the text. A better chunker would have left the old cuts in the database. The hash now includes the chunker's version and sizes, and a test pins it. An ingest that is idempotent on the input must be idempotent on the function too.
• My own test was wrong. A section of 219 words plus one of 2 fits in 220; I had written 215 and expected them to stay apart. The chunker was right.
• check:db did not run the new package. The script listed packages by name, so the first green run said nothing about rag. It lists it now.
• The dataset is easy to break. Any edit to a sentence a question quotes fails a test. That is the point, and also a chore; the questions quote old, stable text where they can.

NumbersCopy Anchor Link

244 tests with check:db, none failing. 326 chunks from 33 documents, 110 words each on average, 233 at most. Ingest takes about a second. $0, no model call.

Where to go nextCopy Anchor Link

Embeddings. Vectors for the chunks (the AI Gateway reaches an embedding model; the whole corpus costs less than a cent), a vector search next to the lexical one, a hybrid ranking that merges both, and reranking. Then the same 40 questions say whether it helped.

Chapter 24 · Retrieval, part two: meaning, merging, rerankingCopy Anchor Link

2026-11-09 · Week 3, block 20 · PR #50

The goalCopy Anchor Link

Chapter 23 ended with a baseline: full-text search finds the right text first for 25% of questions. This block adds the other halves of the plan (vectors, a hybrid and a reranker) and measures each one, so we know which stages earn their place.

What we builtCopy Anchor Link

• Embeddings: rag_embeddings holds one vector per chunk and model, in a vector column with no fixed size (models give 1,024, 1,536 or 2,560 numbers). Migration 0003 creates the extension and the table. embedMissing makes only the vectors that do not exist yet.
• searchVector: cosine distance, one model at a time.
• reciprocalRankFusion and searchHybrid: merge the full-text and vector lists by position, not by score (the scores are not on the same scale).
• rerankHits and searchReranked: the hybrid's best 30, sorted again by a reranking model that reads the question and each candidate together.
• Embedder and Reranker are plain functions. The real ones use the AI SDK's embedMany and rerank through the gateway; fake.ts has deterministic ones, so every test runs offline.
• rag:embed, rag:search --mode, rag:eval (four modes side by side, --kind to split questions from keywords), and eight keyword searches added to the dataset.
• Guide 15 rewritten, and anecdote 10.

Try it yourselfCopy Anchor Link

bun run db:migrate
bun run rag:ingest
infisical run -- bun run rag:embed        # 339 chunks, about $0.001
infisical run -- bun run rag:eval

Expected, on the questions: lexical MRR 0.39, vector 0.62, hybrid 0.54, rerank 0.76.

What the numbers saidCopy Anchor Link

The surprise is the third row: the hybrid is worse than the best single search, with all four embedding models I tried. Vectors win on sentences, words win on identifiers, and merging them with equal votes lets the weaker list pull the stronger one down. The reranker is what fixes the order. Anecdote 10 has the experiments (list weights, candidate depth, different models).

I tried four embedding models on the same chunks (OpenAI small, Cohere v5 fast, Voyage 4 lite, Qwen3 4B) and three rerankers (Voyage 2.5 lite, Cohere v5 pro and fast). OpenAI's small embedding was best and among the cheapest. The rerankers are within noise, so the cheapest is the default.

Decisions and whyCopy Anchor Link

• One vector per chunk and model. It cost one more column in the key and made the model comparison a loop over rag:embed.
• No vector index yet. 339 vectors compare instantly. An index per model needs a fixed size and is a decision for when the corpus is bigger.
• Embedding is a separate command from ingest. Ingest is free and fast, and runs often; embedding calls a paid model, so it is explicit.
• Rerank is the default search. It is the best on both kinds. lexical stays for when there is no key.
• Tests do not call models. The fake embedder hashes words into slots: close when words are shared, far when not. It is enough to test tenants, models, fusion order and the rerank contract; the real quality is measured by the evals, which cost a few cents and run by hand.
• Check the hypothesis, then write the dataset. I suspected the hybrid lost because the dataset had only sentences. I added eight keyword searches to see; the order flipped, which turned a vague worry into a result.

What went wrongCopy Anchor Link

• A getter that was a number. The first cached() embedder used Object.assign(fn, { get tokens() {…} }). Object.assign reads the getter once and copies the value, so tokens stayed at 0 and the cost line would have said "0 tokens". A test caught it; the function now returns a plain object with a real getter, which is also simpler.
• A threshold I made up. My test expected a cosine of at least 0.3 for the best chunk. The fixture chunks are full of filler words, so the true value is lower. The test now checks that it is positive and sorted.
• Most "misses" were not. Of the questions not answered first, many returned the journal chapter that restates the guide's answer, or the right document but another section. The dataset names one chunk, so they count against the search. It is a known gap in the guide.
• The spend is an estimate. I counted tokens, not what the gateway billed: about a cent for embedding the corpus with four models, and a few cents for the rerank evals (each runs 30 candidates per question). Under ten cents in all.

NumbersCopy Anchor Link

334 tests in bun run check, 285 with check:db, none failing. 339 chunks, 4 embedding models, 3 rerankers. Under $0.10 on the gateway, no benchmark run.

Where to go nextCopy Anchor Link

The regulatory documents (EAA, WCAG 2.2) as a second corpus, so the chat can cite a rule behind a violation. After that, the chat itself: SQL tools for the numbers, retrieval for the text, with citations.

Chapter 25 · Two laws as a corpus, and the link from a violation to its ruleCopy Anchor Link

2026-11-09 · Week 3, block 21 · PR #51

The goalCopy Anchor Link

Vektos finds violations. The chat in the plan has to say why they matter, and the answer is not in the project's own docs: it is in WCAG (the standard the accessibility checker tests) and in the European Accessibility Act (the law that makes WCAG-like requirements binding for banks and shops in the EU). This block puts both texts into retrieval, and builds the step the plan calls "the bridge": from a violation the benchmark reported to the rule behind it, with a citation that can be checked.

What we builtCopy Anchor Link

• corpus/wcag-2.2.md (87 success criteria and the glossary) and corpus/eaa-directive-2019-882.md (recitals, 35 articles, 6 annexes), committed, each with its source and licence in the frontmatter.
• Two converters in packages/rag/src/corpus/, with tests on small examples, and bun run rag:corpus to download and regenerate both:
  • cleanWcag takes what pandoc makes from the W3C page and removes the furniture (contents, editors, glossary back-references, bibliography), turns in-page links into text, and makes each glossary term a heading.
  • eaaToMarkdown reads the Directive's XHTML with Bun's HTMLRewriter. The text has no headings, so it recognises the paragraph classes: a chapter, an article and its title, an annex and its sections, list items that are table rows.
• min-words: 0 in a document's frontmatter, so its short sections stay apart; the chunker version went to 3.
• getCriterion (a criterion by number) and criterionFromAxeTag (wcag143 → 1.4.3), in criterion.ts, and bun run rag:criterion 1.4.3.
• 24 more eval cases: 17 questions about the two texts, and 7 "violation messages" (axe's own wording of a rule). A case can now name the heading its answer must be under.
• rag:eval --scope docs|corpus --kind question|keyword|violation, guide 16, and anecdote 11.

Try it yourselfCopy Anchor Link

brew install pandoc                       # only to regenerate WCAG
bun run rag:corpus                        # wcag-2.2.md: 20,562 words; the Directive: 23,541
bun run rag:ingest                        # 38 documents, 772 chunks
infisical run -- bun run rag:embed        # about $0.003
bun run rag:criterion 2.4.7               # Focus Visible, on its own
infisical run -- bun run rag:eval --scope corpus

Decisions and whyCopy Anchor Link

• HTML, not PDF. The plan said "2 regulatory PDFs". Both texts exist as structured HTML, with the numbering in the markup. A PDF is a layout, and rebuilding headings from a layout is a project of its own.
• EUR-Lex refused, the Publications Office answered. A script that asks EUR-Lex for the Directive gets an empty "202" (a bot check). The same text is served from publications.europa.eu/resource/celex/32019L0882 for Accept: application/xhtml+xml. I tried it after the Cellar's HTML answer was a 404 ("no content of that type"). It is the official text.
• Generated Markdown is committed. The evals must read the same text on every machine, and a network fetch in a test is a test that fails when a site is down. The cost is 300 KB in the repo. The licences allow it (W3C: copying with attribution; EU: reuse with acknowledgement), and the repository is private. Regenerating is one command.
• Custom conversion for the Directive, pandoc for WCAG. WCAG's HTML has real headings, so pandoc is right and the rest is cleaning. The Directive's has none, and pandoc would have made its list items into tables. Reading the paragraph classes was less code than undoing that.
• Bun's HTMLRewriter over a Python script. The project is Bun and TypeScript. Streaming events are enough to track the nesting of lists, and the converter is a function with tests.
• One file per criterion, even a tiny one. See what went wrong.
• Look up, then search. For an accessibility violation the checker already knows the rule's criteria. Following the tag gets 7 of 7; searching for the message gets 3 of 7. Search stays for questions in a person's words and for rules with no tag.
• The corpus lives in the vektos tenant for now. A chat for a client would need to search that client's documents and the shared ones. That is a decision for when the chat exists, and it is in the guide's known gaps.
• Not done: a link from a design-system violation to the Directive. A raw colour or a native <button> has no WCAG number. The honest link is through the harmonised standard (EN 301 549), which is not in the corpus. It is left as a gap, not faked.

What went wrongCopy Anchor Link

• The chunker merged 19 criteria into their neighbours (see the anecdote). The tiny-section rule was right for guides and wrong for a standard. Found by writing getCriterion, not by the evals, which kept passing because they checked words and not headings. The fix is a frontmatter flag, and the evals now check headings where the citation matters (22 of the 72 cases); four of them would have failed on the old chunker.
• My first dataset test had no way to see it. The cure was a field, heading, in the case.
• A () after every footnote. The Directive's footnote marker is ( + a superscript + ). Dropping the superscript left () all over the recitals. Cleaned in the converter, and a test pins it.
• ts_rank and the new chunks. Nothing broke, but all four embedding models' vectors became stale: the chunker version is in the hash, so every document was re-ingested and every chunk re-embedded (772). Only the default model was re-embedded; the other three would need rag:embed --embedding-model ….
• A shell quirk. for sc in "--scope corpus" does not split in zsh (${=sc} does), so four evals failed with "unknown option" before I saw why. No harm, but it looked like a code bug.

NumbersCopy Anchor Link

72 cases over 772 chunks from 38 documents. MRR on the 17 questions about the two texts: lexical 0.50, vector 0.77, hybrid 0.58, rerank 0.87 (recall@5 100%). Axe-tag lookup: 7 of 7. 324 tests with check:db (373 in bun run check), none failing. Gateway spend for this block: about half a cent for embeddings and a few cents for the evals (under $0.10 in all, estimated from tokens). No benchmark run.

Where to go nextCopy Anchor Link

The chat. It has what it needs now: SQL over the benchmark's runs for numbers, retrieval for text, and a way to turn "this run had a contrast violation" into "WCAG 1.4.3, here is its text". Before that, a pass over this journal: an index of chapters and a list of the decisions, so the whole project can be read from the top.

Chapter 26 · A journal you can traceCopy Anchor Link

2026-11-09 · Week 3, block 22 · PR #52

The goalCopy Anchor Link

By this point the journal has 26 chapters and more than 16,000 words. Each chapter is clear on its own, but the whole is hard to use for the question that matters when the project is done: why is it like this, and what did we get wrong on the way? This block makes the project traceable from the top, without adding any code.

What we builtCopy Anchor Link

• A reading guide and an index at the top of the journal: one line for each of the 27 chapters (0 to 26), with its date and pull request, a link to the chapter, and the guides and anecdotes that belong with it.
• decisions.md, a new page. The same project grouped by theme (how we work, stack, design system, graders, runner, cost, data, retrieval) instead of by date. It has 71 decisions, each with its reason, what was rejected, and where the story is told.
• A table of what changed our mind: ten decisions that were reversed or corrected, each with the evidence that changed it. It is the most useful part: it is where the project learned.
• A list of open decisions: eight things that are known gaps and not yet decided, such as whether the regulatory corpus is shared between tenants.
• Ten more words in the glossary (context level, batch, design-system hash, chunk, embedding, reranker, recall and MRR, WCAG and the EAA, tenant).

Try it yourselfCopy Anchor Link

bun run docs            # http://localhost:3001, then open Journal and Decisions

Read the index, pick a chapter, then open its row in Decisions' "What changed our mind" table if it has one. Or start from a theme in Decisions and follow the chapter number back.

Decisions and whyCopy Anchor Link

• By theme, in a separate page. The journal is a story and should stay one. A reader who wants "everything about cost" should not have to scan 26 chapters. The two pages point at each other.
• A table for what changed. A decision that was right the first time teaches little; one that was corrected shows how the project reasoned. They deserve their own table, with the evidence.
• Open decisions are listed, not hidden. A finished project with honest loose ends is more useful to its next reader than one that pretends to have none.
• Links are checked, not trusted. The chapter anchors in the index are the ones the docs site generates; I compared all 27 with the built HTML. Nothing checks them automatically: a new chapter's row is checked by hand against the built site.
• No new numbers. Everything in Decisions comes from a chapter, a guide or an anecdote. Where I could not support a sentence, I changed it.

What went wrongCopy Anchor Link

• My first draft claimed three things I could not support. "51 pull requests so far", "four anecdotes changed the design" and a rule about spending "set after the first $5". I had written them from the feeling of the project, not from a source. I reread the draft against the chapters and rewrote each one to say only what the record shows.
• The index generator tripped on the chapter 3 date. That chapter's date line wraps onto two lines, which my extraction did not expect. A reminder that the journal is now read by tools as well as by people: its structure is worth keeping regular.
• The index is not self-updating. I generated it from the headings once. A new chapter needs a row by hand, and a line at the end of this chapter says so.

NumbersCopy Anchor Link

27 chapters, 71 decisions, 10 that were reversed or corrected, 8 still open, 11 anecdotes. 27 of 27 in-page links checked against the built site. $0.

Where to go nextCopy Anchor Link

The chat: SQL over the benchmark's runs for the numbers, retrieval for the text, and the lookup from a violation to its WCAG criterion for the citation. When you add a chapter, add its row to the index at the top of this file.

Chapter 27 · The chatCopy Anchor Link

2026-11-09 · Week 4, block 23 · PR #53

The goalCopy Anchor Link

The project can find violations, count what they cost, and look up the rule behind one. Nothing yet lets a person ask. This block is the chat the plan describes: numbers from the benchmark's database, explanations from the documents and the two legal texts, citations that can be checked, and results drawn with the design system the benchmark is about. It also has a hard limit: a total of one dollar for everything the chat spends while it is built and tried.

What we builtCopy Anchor Link

• packages/chat:
  • six read-only tools: compare_contexts, list_runs, get_run, top_rules (typed, parameterised queries on the dashboard's own), search_docs (hybrid search and rerank) and explain_rule (axe's tag to the WCAG criterion, from chapter 25);
  • the endpoint as a function (createChatHandler), so the route passes the real model and a test passes a scripted one;
  • citations that are checked: each excerpt has a ref, the model cites [3f9a1b2c], and an answer's refs are compared with the sources in its tool results;
  • the spending cap, in the database (chat_usage, migration 0004).
• In apps/web: POST /api/chat, GET /api/chat/budget, the /chat page and a client component that draws each tool result with the dashboard's tables, badges and cards, the model's text with numbered citation marks, and the budget left.
• bun run chat:try "question" to ask from the terminal, a new query (ruleCounts) in the dashboard, guide 17 and anecdote 12.
• Decisions gained a chat section: 8 decisions and three open ones. It now has 79 decisions, 10 reversals and 11 open ones.

Try it yourselfCopy Anchor Link

bun run db:up && bun run db:migrate
infisical run -- bun run web                       # open http://localhost:3000/chat
infisical run -- bun run chat:try "Which context level has the fewest design-system violations?"
curl localhost:3000/api/chat/budget                # {"spent":0.0143,"cap":1,...}

What the real trials showedCopy Anchor Link

About eleven questions with Claude Haiku 5.5, $0.0143 in all (29 billed calls: 24 answer steps, $0.0128, and 5 searches, $0.0015). Our price table and the gateway's own figure were identical on all 24 steps.

Decisions and whyCopy Anchor Link

• Typed tools, no free SQL. The plan says "SQL tools" with named examples (compare_runs, get_violations) and leaves open who writes the SQL. A model that writes it needs a sandbox, a read-only role and a timeout, and still might read what it should not. A model that fills the parameters of queries that already exist can do none of that. The cost is that a question no tool covers is not answerable; the tools are cheap to add.
• The cap is a table, checked before the call. A promise to spend little is not a limit. Each step and search is a row; a request is refused with a 429 if the sum plus a reserve of $0.05 could pass the cap, before the model is called; a step that takes the total to the cap is the last. It survives a restart and covers every tab.
• Never round the cost down. A step is priced by the price table, and the gateway's figure is stored beside it; the dearer counts. A search is charged an embedding plus a flat $0.0003, more than twice the most its rerank can cost.
• Drop tool results sent by the client. In the conversation history, a browser could send a made-up tool result (a source that does not exist, a table with other numbers) and the model would take it for real. Only text is kept. A test sends a forged one and checks the model never sees it.
• A citation is a ref the tools returned. The check is provenance, not truth: it proves the source was given, not that it says what the sentence says. That limit is written in the guide.
• Plain text from the model. The first answers used markdown; the page does not render it, so the prompt says no asterisks and no headings.
• Short labels on the starter buttons. See what went wrong.
• The endpoint is a factory. One function takes a database, a model and a search; production and tests differ only in what they pass.
• The model is Haiku 5.5, as you chose. It is in the price table already, costs fractions of a cent a question, and was good enough that no question needed a larger one.

What went wrongCopy Anchor Link

• A class that did not win (anecdote 12). The starter buttons ran off a phone screen. I passed a class to Button to let it wrap; the compiler, the linter and the graders were satisfied, the page did not change. Panda's cx does not merge atomic classes, so nowrap and normal were both on the button and nowrap won. A sibling of the problem in anecdote 09, and the sort of override an agent will try.
• The types could not be named. createTools returned types that live inside the AI SDK's own dependencies, and TypeScript refused to write them into a declaration file. Adding the SDK's packages as dependencies did not help. The fix was to write each tool's input and output types out, which the page needs anyway to draw each result.
• The web app pulled in the backup tools. Importing @vektos/db from the chat made the web app compile backup.ts, which uses Bun. The same fix as the dashboard's in chapter 18: import from the client and schema subpaths. It had to be applied to the retrieval package too.
• A test of mine was wrong twice. I expected a million tokens of Haiku to cost $0.10; above 100,000 tokens it uses the long-context rate, so it was $0.50. And a cap test could not reach the "stop in the middle" branch with cheap steps, because the reserve already blocks a request first; it needed a 600,000-token step.
• Markdown nobody renders, and a model that wrote "Haiku 4.5". Both small. The first is in the prompt now. The second is a limit: the numbers were right and the name beside them was not.
• The first measurement after sed quoting. Twice a screenshot was "not found" when I read it a moment after writing it. A race of mine, not a bug.

NumbersCopy Anchor Link

409 tests in bun run check, 360 with check:db, none failing. The web build has the two new routes and the page. axe finds nothing on /chat at 1280 and 390 px, with and without an answer, in light and dark; no sideways scroll at 390 and 320. Spent on the chat: $0.0143 of the $1 cap. No benchmark run.

Where to go nextCopy Anchor Link

Things that matter before anyone else uses this: sign-in (anyone who can reach the port can spend the cap), saving conversations, and a check of the model's sentences against the tool's table, not only of its citations. Then the demo: a README that shows the dashboard and the chat, and a short video. When you add a chapter, add its row to the index at the top of this file.

Chapter 28 · A chat that says why it cannot answerCopy Anchor Link

2026-11-09 · Week 4, block 24 · PR #54

The goalCopy Anchor Link

Right after the chat was merged, it failed for the person who opened it: they asked a question and got "The assistant could not answer. Try again." Trying again could never work. The server had been started with bun run web and not through infisical run, so it had no AI Gateway key. This block makes the chat say so.

What we builtCopy Anchor Link

• A check at the door. createChatHandler looks for AI_GATEWAY_API_KEY first, when it would use the gateway. Without it the answer is 503 with { error: "no-key", message }: "The chat needs AI_GATEWAY_API_KEY. Start the app with: infisical run -- bun run web". Nothing is called, nothing is spent. A model passed in (a test's) needs no key.
• A notice on the page. /chat checks on the server and, when the key is missing, shows the same sentence above the form and disables the form and the starter buttons, as it already did when the cap was reached.
• The real error in the log. Any other failure still tells the reader "try again", but writes chat: the model call failed: … with the actual error to the server's console.
• Four tests, a line in guide 17 and in the known gaps.

Try it yourselfCopy Anchor Link

bun run web                                    # no key: open /chat, read the notice
curl -i -X POST localhost:3000/api/chat -d '{"messages":[{"id":"u","role":"user","parts":[{"type":"text","text":"hi"}]}]}'
# HTTP/1.1 503 ... {"error":"no-key","message":"The chat needs AI_GATEWAY_API_KEY. ..."}
infisical run -- bun run web                   # with the key: it answers

Decisions and whyCopy Anchor Link

• Say the cause. A generic message is right for a failure the reader cannot fix. This one they can: start the app another way. It is the same lesson as the cap: when the system refuses, the refusal should say what to do.
• Check before anything else, before the budget and the body. It is the cheapest check and the one that makes every other answer pointless if it fails.
• A 503, not a 401. The caller did nothing wrong; the server is not set up.
• The page checks too. An error after a question is a bad place to learn that the form could never have worked.
• Do not leak the real error to the browser. The reader sees one sentence; the log has the detail. A key that is set but wrong still shows the generic one, and the log says why.

What went wrongCopy Anchor Link

• I only ever tried it the way that worked. All my trials, and the screenshots, ran through infisical run. The first time someone started the app the plain way it failed, with a message that explained nothing. A path I never tried is the path someone else takes first. The new tests start a handler with no key.
• The notice did not appear at first. My edit that inserted it into the page found nothing to replace, because the formatter had put the <form> on several lines; the form was disabled and there was no text to say why. A screenshot showed it. A text edit that does not check it matched is an edit that may have done nothing.
• The linter caught what the missing edit hid. Once the notice was really in the page, the lint asked for the <output> element, which already means "status", instead of a paragraph with a role. Used.
• A reading of the screenshot, not the log, found the cause. The log in /tmp/vk-web.log was empty; the process listing showed next dev started without Infisical, and a request to the API showed the error stream. That is the trail the new log line would have left in one step.

NumbersCopy Anchor Link

413 tests in bun run check, 364 with check:db, none failing. axe finds nothing on the page with the notice at 1280 and 390 px; no sideways scroll. No spend on the chat: the cap stays at $0.0143 of $1. No benchmark run.

Where to go nextCopy Anchor Link

The demo: a README with screenshots of the dashboard and the chat, a "how to use it" guide, and a script for a short video. When you add a chapter, add its row to the index at the top of this file.

Chapter 29 · Every name is a linkCopy Anchor Link

2026-11-09 · Week 4, block 25 · PR #55

The goalCopy Anchor Link

You looked at a table the chat had drawn and asked for the obvious: a model and a context should be clickable, and lead where they belong. The same was true of most of the dashboard. A model's name, a context, a task, a batch, a rule: they were text, and to see the runs behind one you had to go to /runs and pick filters by hand.

What we builtCopy Anchor Link

• One helper for every link to runs, runsHref({ model, context, task, label, batch, rule }), and three small components (ModelLink, ContextLink, RuleLink). A name always leads to the run list filtered by it.
• Two new filters on /runs, ?batch= and ?rule= (a11y/color-contrast), shown as a sentence with a way to remove them. listRuns gained both; a rule matches a run that broke it and did not have it ignored or covered.
• Links everywhere a name appears: the Overview (model headers, contexts, and each count, which opens the runs behind it), /runs, /batches (model, context, and the date, which opens the batch), the run page (model, context, "all runs of this task", batch, label, and each rule), and the chat's tables and run card. A rule in the chat's "top rules" keeps the filter the tool used.
• Links that look like links. A global rule in the design system: .vektos a:not([class]) is underlined in a new token, accent.text.
• Costs the chat cannot round to zero. Each cost in a tool result has a text twin ("$0.0126") and the prompt says to quote it.

Try it yourselfCopy Anchor Link

infisical run -- bun run web

Open the Overview, click a count in the table, then a run, then a rule in "Violations by rule". Or ask the chat "which context level gives the fewest violations?" and click a model in its table.

Decisions and whyCopy Anchor Link

• The run list is the destination. It already has filters, pages and a clear empty state. A page per model or per rule would be four new pages saying less. A link is a filter that someone set for you.
• One helper, one order. The same filters give the same address, so the browser's history and shared links are stable, and there is one place to change if the run list moves.
• Links keep the filters you already chose. In the Overview filtered by batch, a model link keeps the batch. A rule from the chat's "top rules without context" opens runs without context.
• Underlined, not only colored. A link told apart by color alone fails WCAG 1.4.1, and an underline is what everyone reads as "click me".
• Only plain links are styled. A filter pill or a header item has its own class and look; a global rule that underlined them would fight every component.
• A text twin for costs, not a smarter model. The model had the right number and rounded it. Giving it a string to copy is cheaper than asking it to be careful, and testable.

What went wrongCopy Anchor Link

• The links were invisible. The first screenshot after adding them looked the same as before: Panda's reset removes the browser's color and underline from <a>. Some names had been links for weeks (a task in /runs, the contexts in the Overview) and nobody could tell, which is likely why the dashboard felt like it had no links at all.
• The fix failed contrast in dark mode. I colored links with accent.default, whose description says "readable as text on a surface". In the table headers (bg.subtle) in dark mode it is 3.97:1. axe found it on the first pass. A new token, accent.text, picked by computing the contrast on all three backgrounds in both themes (at least 6.1:1 light, 5.7:1 dark). It is anecdote 02 again: a token right for its description, wrong one background over. Its follow-ups now say so.
• TypeScript 7 does not load @types on its own. A test in the web app could not find bun:test even with @types/bun installed: the default list of types is now empty. A reference at the top of the test file loads Bun's types for that file's program without making them part of the app's code.
• A prompt in a template string. I wrote a field name in backticks inside the system prompt, which is itself a template literal; the backtick closed it. Spelled out instead.
• The filter pills can lead nowhere. On /runs?rule=… they list every model and context, also ones that never broke the rule. Four of 204 links crawled lead to an empty list for that reason. Written in the guide's known gaps.

NumbersCopy Anchor Link

A crawl of every internal link on six pages: 204 distinct links, all answering 200, 4 leading to an empty list (the pills above). axe finds nothing on five pages at 1280 and 390 px, light and dark (20 checks). 419 tests in bun run check, 367 with check:db, none failing. The chat, asked the same question again, quoted $0.0126 and $0.00552; that answer cost $0.0011. With your own questions in the meantime, the chat has spent $0.0174 of its $1 cap. The design system has a new token and a global rule, so its hash changed; no batch has run since. No benchmark run.

Where to go nextCopy Anchor Link

Sign-in with GitHub and saved conversations (block B of the plan), now that the OAuth app exists. When you add a chapter, add its row to the index at the top of this file.

Chapter 30 · Sign-in and saved conversationsCopy Anchor Link

2026-11-09 · Week 4, block 26 · PR #56

The goalCopy Anchor Link

The chat had one safeguard: a spending cap. Anyone who could reach the port could ask, and a reload lost the conversation. You chose what to build: sign-in for the chat only (the dashboard stays the public demo), with GitHub, for a list of usernames; and conversations saved on the server.

What we builtCopy Anchor Link

• GitHub sign-in with Better Auth, in apps/web/src/lib/auth.ts and /api/auth/*. Sessions, accounts and the OAuth state live in four auth_* tables, generated by Better Auth's CLI, renamed with a prefix (user is reserved in Postgres) and kept as our own migration (0005).
• An allowlist, VEKTOS_ALLOWED_GITHUB, checked when an account would be created (a stranger leaves no row), when a session would start, and on every chat request.
• Saved conversations: chat_conversations and chat_messages, a list beside the chat, an address per conversation (/chat/<id>), and "Delete conversation".
• The history from the database. The browser now sends one thing: { id, message }. The server checks the conversation is the person's, reads the earlier messages with their tool results, saves the question, streams the answer and saves it when it ends.
• A cap per person, $0.50 inside the $1, with both shown under the question; and each cost now records who asked and in which conversation.
• Guide 18, with a table of what is stored and how to remove it.

Try it yourselfCopy Anchor Link

bun run db:migrate
infisical run -- bun run web       # /chat → Sign in with GitHub, ask, reload
bun run check:db                   # the endpoint's checks, conversations, the allowlist

Decisions and whyCopy Anchor Link

• Two checks of the list, for two reasons. Refusing the account means a stranger's name and email are never stored. Checking every request means removing a name works at once, without waiting for a session to expire.
• Better Auth. OAuth state, sessions and cookies are where a homemade version fails quietly. It supports Next 16 and Drizzle 0.45, stores everything in our database, and lets me switch off what we do not use: telemetry, email and password, changing one's own profile.
• Encrypt a token we never use. GitHub returns one; Better Auth keeps it. Encrypting it costs nothing and makes a database copy less valuable.
• The id is made in the browser. The page can show a new conversation before the server answers, and the address can become /chat/<id> at once. Ownership is decided by the first request, and someone else's id is a plain 404.
• Read the history, do not receive it. This replaced chapter 27's rule (keep only the text of what the browser sends). With the server owning the history, it can trust the tool results in it, and a follow-up no longer pays for the same tool call twice.
• The sidebar is updated by the page, not by the server. After the first answer of a new conversation, reloading from the server could arrive before the answer is saved. The page adds the conversation to its own list and changes the address with replaceState.
• Finish the answer even if the reader leaves (consumeStream). Its cost is counted and the answer is saved; otherwise a closed tab would spend money and keep nothing.

What went wrongCopy Anchor Link

• A test that waited for a status, not for the answer. A request that returned 200 kept running after the test moved on, and wrote its cost into the next test's empty table, which then saw two rows where it expected one. I first suspected the order of the SDK's callbacks and tested it alone (it was fine). The cause was the new behaviour working as designed: an answer finishes even when nobody reads it. The test now reads the response to the end.
• Counting before writing. In the same hunt I moved the cost update before the database write, so the stop decision never depends on an await. Not the cause of the failure, but a race waiting to happen.
• user is a reserved word in Postgres. Better Auth's CLI names its table user. Every hand-written query would need quotes; the tables are auth_user and so on, mapped in the adapter.
• A literal backslash-n. A test meant to check that a title is one line had \n in it as two characters, because of how I wrote the file. The test failed for the wrong reason.
• A message that claimed too much. The sign-in page said "That account is not on the list" for any failed sign-in. It now says "Sign-in did not complete", and that a refused account leaves nothing stored.
• I cannot sign in for you. The OAuth flow needs your browser and your GitHub. I checked everything up to the GitHub authorisation page (the redirect address, the scopes) and then asked you to sign in, ask and reload; then I read the database to see what had been stored.

NumbersCopy Anchor Link

428 tests in bun run check, 376 with check:db, none failing. axe finds nothing on the sign-in page and the refused page, 1280 px light and 390 px dark. In the database after your first sign-in: one user, one session for seven days, a token stored encrypted, one conversation with two messages, three costs with their user and conversation. The chat has spent $0.0187 of its $1 cap in all. No benchmark run.

Where to go nextCopy Anchor Link

The demo: a README with screenshots of the dashboard and the chat, a "how to use it" guide, and a script for a short video. Then deploying it, which now has what it needs: sign-in and caps. When you add a chapter, add its row to the index at the top of this file.

Chapter 31 · A new palette, the same namesCopy Anchor Link

2026-11-09 · Week 4, block 27 · PR #57

The goalCopy Anchor Link

You wanted the design system to look like Adham Dannaway's Practical UI, with the blues a little warmer. Practical UI also renames the tokens (text.strong, fill.*, stroke.*). You liked those names, and asked whether it made sense to break the comparison with the saved runs now.

What we builtCopy Anchor Link

• A new palette in packages/ds/panda.config.ts.
  • Light theme: navy-tinted neutral shades: #0b1220 text, #3e4a5f muted text, #cfd6e1 card borders, #eef2f7 page, white cards.
  • Dark theme: a new gray family: #07090e page, #121821 cards, #2e3847 borders, #f5f7fa text.
  • One accent for both themes: the light cobalt of a paint swatch you sent, #3376bc, with white text on it. accent.text is a darker cobalt for links in light mode (#1f5a96) and a lighter one in dark mode (#8fc0f2).
  • success.default and warning.default moved one step darker in light mode (green 700, yellow 700), so icons in them reach 4:1 on every background.
  • Status text in dark mode moved one step lighter (red, green and yellow 200 instead of 300): the badges in the overview table read more easily, at least 10.8:1.
  • Warnings are yellow (hue 44° to 55°), not amber: you found amber too orange. The palette family is now yellow; amber is gone. Light: #715400 text on #fef9c3. Dark: #fef08a text on #3a2f06.
• Radii in proportion. Filter pills, the pager and Badge were fully round (full). Pills and the pager now use md, like Button and Input; Badge, a smaller element, uses sm.
• accent.default is no longer text. Its description says so, and the two places that used it as text now use accent.text: the outline Button label and the "back to runs" link.
• A selected filter pill keeps its colour on hover (apps/web/src/components/filter-group.tsx). Before, hovering it applied the grey hover of an unselected pill under its white text.
• The same semantic names. bg.canvas, fg.muted, accent.default and the rest keep their jobs; only what they point at changed.
• A roadmap entry in docs/PLAN.md: Practical UI's names as an experiment of their own.

Try it yourselfCopy Anchor Link

bun run --filter @vektos/ds prepare   # regenerate styled-system
bun run check                          # 428 tests, none failing
bun run check:page                     # the page graders in Chrome
bun run web                            # http://localhost:3000, then "Dark mode"

The selected pills and the buttons are the same cobalt in both themes, with corners like the buttons'. In light mode white cards sit on a pale blue-grey page; in dark mode dark navy cards sit on a near-black page.

Decisions and whyCopy Anchor Link

• Colours now, names later. Graders (token-roles), tasks, solutions and 159 saved runs all use today's names. Changing names and colours together would make any difference in the results impossible to explain. Renaming is a good question on its own: does the vocabulary of a design system change how well an agent follows it?
• One accent, two link colours. You wanted the same cobalt in both themes. As a fill that works: white text on it is 4.7:1. As text it cannot: a colour readable on white (at most about 0.18 luminance) is too dark to read on near-black (it needs about 0.2). So the accent is shared, and links get their own token per theme.
• Contrast from the neutrals, not the accent. You asked for more contrast through background, borders and text. Body text is at least 15.6:1 in light and 14.5:1 in dark, muted text 7.5:1 and 8.5:1, control borders 6.2:1 and 8.5:1. Card borders are darker, and the page is a step away from the cards in both themes.
• Two grey families. Light text and dark cards are colours of about the same lightness but a different tint, so one scale could not serve both themes. The dark theme got gray.
• Every value picked by its contrast. A small script computed every text and background pair in both themes before the values went in, and axe checked the pages after.

What went wrongCopy Anchor Link

• Five tries to find the blue. First I read "warmer" as "towards violet": an indigo at 235° and a dark theme tinted the same way (#11132a). You found it cold and murky. Second, plain grey backgrounds and a blue at 228°: it looked the same as before. Third, a page with six blues side by side; your answer was "more navy, less violet", so a fourth version at 224°. Then you sent a photo of the colour you wanted, a light cobalt, and asked for it in both themes. Lesson: a colour word means different things to different people; a sample beats a description, so ask for one first.
• accent.default was used as text. Its old description said it was "also readable as text on a surface". With the swatch colour, the "back to runs" link was 4.42:1 on the page, and axe caught it. The rule is now in the description, and the two uses moved to accent.text.
• A palette change is a new design-system version. The batch hash covers panda.config.ts, so the next batches get a new hash and the history will warn when old and new runs share a row. That is right for once: the DS MCP shows the hex values and descriptions to the agent, and axe measures contrast on the rendered page. An agent that colours text with accent.default now fails contrast on the page background.

NumbersCopy Anchor Link

428 tests in bun run check, 174 in check:page, none failing. axe (WCAG 2.2 AA) finds nothing on the overview, runs, one run, batches and chat, light and dark, at 1280 px. No benchmark run.

Where to go nextCopy Anchor Link

A batch on the new palette, to see whether contrast findings move. Then the naming experiment. When you add a chapter, add its row to the index at the top of this file.

Chapter 32 · Does the AI SDK change what the agent reads?Copy Anchor Link

2026-11-09 · Week 4, block 28 · PR #58

The goalCopy Anchor Link

You were comparing gateways: Requesty against Vercel's AI Gateway. Requesty did not win, since it adds 5% to the price, has no rerank and its strength is keeping data in the EU, which we do not need. Then you asked for alternatives to the AI SDK itself, and that raised a question about the benchmark. Every real run goes runner → AI SDK → AI Gateway → provider. If the AI SDK rewrote the prompt or the tools on the way, part of what Vektos measures would be the library, not the model.

Two plans came out of it. Plan B (this chapter) checks the AI SDK first, offline. Plan A (later, in the PLAN) puts the gateway behind a small interface, so another provider becomes configuration. B went first because it is free and quick, and because its answer decides whether the runner should leave the AI SDK at all.

What we builtCopy Anchor Link

• A fake model server (packages/bench/src/payload/replay.ts). It records every request and answers from a script: first a call to a file tool and to an MCP tool, then a short reply. It has one set of answers per wire format: Anthropic Messages, OpenAI Responses and the AI Gateway's own protocol.
• The runner, pointed at it. runTask takes an optional languageModel. model still picks the price, but the call goes to this model. The AI SDK side of the check is runTask itself, so its requests are the ones a real run sends.
• The same loop written by hand with @anthropic-ai/sdk and openai (packages/bench/src/payload/official.ts). It sends the same prompts and tools. Each tool's schema comes from its source: zod's conversion for the file tools, the MCP server's own list (through the MCP SDK's client) for the rest. It sends only the fields each API requires.
• A field-by-field diff (diff.ts). It sorts keys and lines lists up by meaning: tools by name, messages by role, tool calls and results by call id. A message added at the top of the list then shows as one difference, not as every message after it.
• bun run bench:payload writes results/payload-diff/<time>/: the raw bodies and diff.md.
• Two tests in bun run check (payload.test.ts). The first checks that both sides send the model the same words: system prompt, task prompt, tool descriptions and tool results. The second keeps snapshots of the AI SDK's requests, so an upgrade that changes the wire fails until someone reads it.
• Guide 19 and anecdote 13.

Try it yourselfCopy Anchor Link

bun run bench:payload                              # statement, ds-mcp-check
bun run bench:payload --task kyc-form --context none
cd packages/bench && bun test src/payload          # 14 tests, 3 snapshots

Expected output:

Wrote results/payload-diff/2026-11-09T15-42-28/diff.md

Open diff.md. The texts are not in any table, because they are equal. What is left is format and defaults: max_tokens 128000 against 16000, tool_choice auto, the $schema dialect, and additionalProperties: false on the MCP tools.

Decisions and whyCopy Anchor Link

• The runner stays on the AI SDK. The model reads the same words. The differences are in format (text as a list of blocks, a developer message instead of instructions) and in defaults. None of them is worth a second code path for every provider.
• Offline, no key. Both SDKs accept a fetch, so a fake server sees exactly what would go out. It cost $0, well under the $0.50 you allowed.
• The runner itself, not a copy of its call. Adding languageModel to runTask is one line. A copy of the generateText call would drift the day someone adds an option to the runner.
• A test task, not a real one, in the snapshot. Editing statement or the design system's docs should not break a test about the wire.
• The provider seam waits. It goes in packages/models, with the gateway and a fake provider in tests, so no new keys. The gateway_usd columns keep their name in SQL and are mapped to reportedUsd in Drizzle, so no migration. A real rename would risk a DROP COLUMN if drizzle-kit misread it, for a cosmetic gain.

What went wrongCopy Anchor Link

• The first diff was unreadable. OpenAI's input list had a developer message at the top on one side and not on the other. Compared by position, every later message was "different". Lining lists up by role and call id fixed it, and a separate row reports a change of order, so it cannot hide one.
• We almost blamed the wrong package. additionalProperties: false on the MCP tools looked like the MCP server's doing. The MCP SDK's own client showed the server never sends it. It is added by @ai-sdk/mcp.

NumbersCopy Anchor Link

$0 spent. 14 new tests and 3 snapshots. bun run check: 346 tests passing, none failing. On statement with ds-mcp-check, 12 and 13 differing fields in the two Anthropic requests, 12 and 17 in the OpenAI ones. None of them changes the text the model reads.

Where to go nextCopy Anchor Link

Decide whether the runner sets maxOutputTokens itself. One real call, with a key and for less than a cent, would show what the gateway does with the request. Then Plan A, the provider seam. When you add a chapter, add its row to the index at the top of this file.

Chapter 33 · Mint and inkCopy Anchor Link

2026-11-09 · Week 4, block 29 · PR #59

The goalCopy Anchor Link

The cobalt palette of chapter 31 still felt flat to you, in both themes: you wanted it warmer and brighter. Words had already failed five times, so this time the choice was made by looking.

What we builtCopy Anchor Link

• Previews before code. One HTML file per round, outside the repo, with the same mini dashboard (navigation, filters, stat cards, a table with badges, buttons, an input) in light and dark, and the contrast of every text pair printed under each panel. Three rounds of three: warm neutrals with cobalt (rejected), three palettes around a mint you named, #85f4b1, then three hybrids of the one you liked. You picked "mint + ink".
• The palette in packages/ds/panda.config.ts: green-tinted neutral and gray, a brand family for mint (400), its hover (500), emerald (700) and the text on mint (900), an ink family, and cyan instead of green for success.
• Ink regions. A new token, bg.inverse, and the dark class on the same element. The app header (apps/web/src/app/layout.tsx) and TableHead are ink in both themes.
• Mint where it shows, emerald where it must. accent.default is the mint fill; the focus ring, the outline Button, the Select check mark and the chart marker for ds-mcp moved to accent.text, which is emerald in light mode and mint in dark mode.
• Card has accent, a mint line along the top edge. The run count on the overview uses it.
• The current navigation item is mint with dark text, like a selected filter.

Try it yourselfCopy Anchor Link

bun run --filter @vektos/ds prepare   # regenerate styled-system
bun run check                          # 442 tests, none failing
bun run web                            # http://localhost:3000
bun run storybook                      # Card → Accent, Card → With table

A near-black header with the current page in mint; table heads in the same ink; mint buttons and filters with dark text. Switch to dark mode: the header and table heads stay, the page around them goes dark.

Decisions and whyCopy Anchor Link

• Look, then build. A preview costs a few minutes and no commit; a wrong palette in the design system costs a branch, a review and a new design-system version for the benchmark.
• One accent, two jobs split. Mint cannot be text or an outline on white (1.3:1). Rather than a darker mint that would no longer be the colour you chose, the fill stays mint and everything that must be seen on its own uses accent.text.
• A scoped dark theme instead of inverse tokens. Panda's dark condition is .dark &, and the CSS variables for dark mode are declared on .dark, so the class works on any element. One token (bg.inverse) instead of five or six keeps what agents read small, and anything placed in an ink region is right by construction.
• Cyan for "ok". Green next to a mint accent reads as "selected", not "succeeded".

What went wrongCopy Anchor Link

• Text in an ink region was the light colour. color is inherited as a computed value, so a header with dark still showed the near-black text set on <html>. The fix is to set color: "fg.default" on the region itself; guide 01 says so.
• A missed default. Card got accent?: boolean, and Stat passed accent through as boolean | undefined; with exactOptionalPropertyTypes that is a type error. Stat now defaults it to false.

NumbersCopy Anchor Link

442 tests in bun run check, none failing; check:page green. axe (WCAG 2.2 AA) finds nothing on the overview, runs, one run, batches and chat, light and dark. Lowest contrasts: body text 14.1:1, muted text 6.3:1, accent.text 5.7:1, text on mint 9.6:1 (hover) to 11.6:1. No benchmark run; the design system has a new hash.

Where to go nextCopy Anchor Link

A batch on the new design system, to see whether agents' pages change under it. Then the naming experiment. When you add a chapter, add its row to the index at the top of this file.

Chapter 34 · The grader learns the mintCopy Anchor Link

2026-11-09 · Week 4, block 30 · PR #60

The goalCopy Anchor Link

The first step of expanding the design system, before any new component: make the benchmark count what the design system now says. Chapters 31 and 33 changed the accent. accent.default became a mint fill, too light to read on a page, and accent.text took over text, links, outline buttons and focus rings. The token-roles grader had not followed: it accepted any accent.* for any job. An agent writing color: "accent.default" broke a rule of the design system and the benchmark did not count it; only axe, on the rendered page, would notice the contrast, and call it a quality problem.

What we builtCopy Anchor Link

• The accent's jobs in token-roles:
  • accent.default and accent.emphasized: fills, a border that continues the fill, shadows. Not text, not icons, not outlines.
  • accent.fg: text and icons on the fill (unchanged).
  • accent.text: text, icons, borders and outlines. Never a fill.
• A new colour role, outline. outline-color and ring-color (focus rings) used to count as borders. A selected pill whose border is the same mint as its fill is right; a focus ring in that mint is invisible on white. So the two are told apart, and border.*, accent.text and the status .default tokens may be outlines.
• The words agents read. The MCP server's colour guide and guide 01's table say the same rule.

Try it yourselfCopy Anchor Link

bun test packages/bench/src/graders/token-roles

css({ color: "accent.default" }) now gives wrong-role: "accent.default is a fill; text, icons and outlines use accent.text".

Decisions and whyCopy Anchor Link

• Grader first, components second. Every component added from now on is graded by its own tests and by the dogfood test; the rule must be right before there is more code to check.
• A border may continue a fill. The selected filter pill, the hovered outline button and the Card accent line use accent.default as a border, and they are right. A rule that flagged them would be a grader crying wolf.
• A new role instead of a special case. outline is a different job: it must stand out from what is around it. A role says that once, for every token family.

What went wrongCopy Anchor Link

• A test that approved the mistake. token-roles.test.ts listed outlineColor: "accent.default" among the correct uses: it was written when the accent was a dark cobalt that worked as a ring. The palette changed under the test, and the test kept saying yes. It now uses accent.text, and a new test lists the wrong uses.
• Nothing else had to change. The design system, the dashboard, the chat and the five reference solutions already used accent.text where the rule asks for it: chapter 33 had moved them by hand. The grader now makes sure the next ones do too.

NumbersCopy Anchor Link

443 tests in bun run check, none failing; the dogfood test still finds nothing in apps/web. $0. No benchmark run.

Where to go nextCopy Anchor Link

The governed component model (decisions): a typed layout prop instead of className, UNSAFE_className as a counted way out, Stack and Grid, and recipes without layout. When you add a chapter, add its row to the index at the top of this file.

Chapter 35 · The page places, the component looksCopy Anchor Link

2026-11-09 · Week 4, block 31 · PR #61

The goalCopy Anchor Link

Anecdote 12 left a question open: what should a design-system component do with a className that sets something its recipe already sets? <Button className={css({ whiteSpace: "normal" })}> had compiled, linted, graded clean, and done nothing, because Panda's atomic classes win by stylesheet order, not by who came last. You chose the answer used by Adobe's Spectrum 2: the component refuses. It owns how it looks; the page only says where it sits.

What we builtCopy Anchor Link

• One list of layout props, packages/ds/src/layout-props.ts: margins, width, minWidth, maxWidth, a place in a flex or grid container (flexGrow, alignSelf, gridColumn…) and position. Long logical names only, no height. STACK_PROPS and GRID_PROPS add what the two layout primitives arrange.
• Types in layout.ts: Governed<P> removes className and style from a native element's props, LayoutProps adds the list, EscapeHatch adds UNSAFE_className and UNSAFE_style. splitLayout and governedClass build the class: the recipe's, then the layout's in one css() call, then the escape hatch.
• All eight components migrated. Roots take layout props; parts (CardBody, TableCell) take none. Input and Table keep "full width" as a layout default outside the recipe, so a page's width really replaces it. CardHeader now keeps its own gap below, which removed the most common override in the dashboard.
• Stack and Grid, with stories, for the other common reason to put a class on a component: arranging what is inside a CardBody.
• Panda's jsxMatchTag in panda.config.ts: from @vektos/ds, Panda reads only the layout props (and the arrangement props for Stack and Grid). A colour forced past the types generates no CSS. Checked with a probe before writing any component.
• A grader, outside-styling (packages/bench/src/graders/outside-styling.ts): rule styled-from-outside for className, style and non-layout style props on @vektos/ds components, rule unsafe-escape for every use of the way out, which vektos-ignore cannot silence. Its rules are data built from the same list (vektos-ds.ts). The compiler's TS2322 on the same line is marked covered, so one mistake counts once.
• The migration: 20 overrides in the dashboard, the chat and two reference solutions became layout props, eight Stacks, or nothing at all (the CardHeader padding). A failed run's error became a plain <p role="alert"> inside the card: the page's own element, which it may style.
• What agents read. The Storybook introduction (served by the MCP as guidelines) has a fifth rule and Stack/Grid in the "which component" table; get_component lists the layout props of a governed component and hides UNSAFE_*.

Try it yourselfCopy Anchor Link

bun test packages/bench/src/graders/outside-styling
bun run storybook   # Stack and Grid, under their own names

In a page, <Card className={css({ p: "4" })}> is a type error, and graded:

outside-styling/styled-from-outside  <Card className>
  `className` on `Card`: a design-system component takes layout props (`marginInlineStart`,
  `flexGrow`, `width`…) and its own variants, nothing else. Arrange content with `Stack` or `Grid`.

<Card maxWidth="lg"> and <Stack gap="4"> are clean.

Decisions and whyCopy Anchor Link

• Refuse, do not merge. Merging the caller's class into the recipe would make the override work, and make every component restylable by any agent: the design system becomes a suggestion. Refusing makes the rule visible in three places at once: the compiler, the stylesheet and the benchmark.
• One list, three readers. The types, Panda and the grader read layout-props.ts. Three copies would drift; one cannot.
• The escape hatch exists and costs. Real design systems need one. Named UNSAFE_, hidden from the MCP's help and counted every time, it is there for a person who knows why, not for an agent looking for a shortcut.
• Reported at the attribute. The first version reported at the tag name; the compiler reports TS2322 at the attribute, so a multi-line tag would have counted twice. Both now point at the same line.

What went wrongCopy Anchor Link

• ReferenceError: LAYOUT_PROPS is not defined in the Next dev server, the first time it compiled globals.css after panda.config.ts started importing the list. Loading the config with Node directly worked, and a clean restart (rm -rf apps/web/.next packages/ds/node_modules/.panda) never showed it again. A stale bundle of the config is the best guess; it was not reproduced.
• A stray src/layout-props.d.ts appeared after a failed tsc run and broke format:check. Deleted; it has not come back.
• react-docgen's composes is not what it seems. It lists only the types it could not resolve, so LayoutProps showed up for one file and not for another. The MCP reads interface XProps extends … LayoutProps from the source instead.
• Two tests approved the old way. The MCP's example fixture and the "careless agent page" test had a className on a component. The first was fixed, the second gained the two new findings it should have had all along.
• Storybook args generate no CSS (anecdote 14). The first Stack and Grid stories passed their layout through args; the responsive row and the grid columns did not exist in the stylesheet. They now write the values in the JSX.

NumbersCopy Anchor Link

bun run check 456 tests, check:db 402, check:page 200, none failing. The dashboard: five pages at 1280 and 390 pixels, light and dark, 20 axe checks, no problems. Storybook: the new stories and the migrated ones, at two widths, no axe problems. 35 files changed and 7 new. $0. No benchmark run.

Where to go nextCopy Anchor Link

DS-2, the form controls: Checkbox, RadioGroup, Switch and Textarea, governed from the start. When you add a chapter, add its row to the index at the top of this file.

Chapter 36 · Form controlsCopy Anchor Link

2026-11-09 · Week 4, block 32 · PR #62

The goalCopy Anchor Link

DS-2 of the expansion: the form controls a fintech page asks for and the design system did not have. Until now an agent asked for a consent, a payout frequency or a note had two choices: a native <input type="checkbox"> styled by hand, or nothing. The four new components are the first built governed from the start (chapter 35), so they are also the first test of that model on new code.

What we builtCopy Anchor Link

• Checkbox (packages/ds/src/checkbox.tsx): label as children, helperText, error, required, indeterminate for "select all".
• RadioGroup (radio-group.tsx): Ark's Fieldset and RadioGroup together, so the question is a <legend> and the group's name. options like Select, each with an optional description; orientation, helperText, error, required.
• Switch (switch.tsx): a setting that applies at once, with a helperText.
• Textarea (textarea.tsx): Input's look on several lines, rows 4, resizable in height, inside a FormField.
• Stories for each, 21 in all, light and dark.
• One focus ring, still. The global rule now also draws on [data-focus-visible] parts named control and item-control: a checkbox focuses a hidden input, and Ark marks the box instead.
• The components grader matches an <input> by its type. type="checkbox" points to Checkbox or Switch, type="radio" to RadioGroup, <textarea> to Textarea, and a hidden input passes: it draws nothing.
• What agents read. Five new rows in the "Choosing a component" table, and the MCP now tells a component with its own props ("No other props") from one that passes an element's through.

Try it yourselfCopy Anchor Link

bun run storybook   # Checkbox, RadioGroup, Switch, Textarea
bun test packages/bench/src/graders/components

<RadioGroup
  label="Transfer speed"
  defaultValue="standard"
  options={[
    { label: "Standard", value: "standard", description: "Free. Arrives the next working day." },
    { label: "Instant", value: "instant", description: "EUR 1.00. Arrives in seconds." },
  ]}
/>
<Checkbox required error={agreed ? undefined : "Confirm the details to continue."}>
  I confirm the details above are correct
</Checkbox>

Decisions and whyCopy Anchor Link

• Small, explicit APIs. Ark calls onCheckedChange with a details object and allows "indeterminate" as a checked value. Our Checkbox calls back with a boolean and makes indeterminate a separate prop. Two fewer ways to hold it wrong, the reason Select takes options.
• Each control brings its label. A checkbox inside a FormField would have two labels; the rule "never wrap it in a FormField" is in each component's description.
• Checked is a mint fill with an emerald edge. The mint alone is about 1.3:1 against white, and a control's boundary needs 3:1 (WCAG 1.4.11). An accent.text border gives it, and keeps the roles of chapter 34: accent.text is a border here, never a fill.
• The thumb of the switch is an icon. A filled circle in currentColor, coloured fg.muted off and accent.fg on. As a background it would have needed a bg.* token dark enough to see on bg.subtle, and there is none: no new token for one part.

What went wrongCopy Anchor Link

• The MCP had been saying something false. Every component got "Also accepts the props of the underlying HTML element", including Select, which accepts none. With three more components like it, the sentence would have told agents to pass onClick or aria-describedby that go nowhere. It now reads the props interface: if it does not extend an element's props, it says "No other props".
• A retrieval question changed its answer. The RAG evaluation set (packages/rag/evals/ vektos-docs.json) finds "How many components…" by the sentence "has eight components" in guide 06. The guide now says twelve, so the expected text moved with it. The docs indexed in the database are still the old ones until the next ingest, which costs a few cents of embeddings and waits for a go-ahead.
• Otherwise, nothing. The governed model held: no new component needed a className, the stories wrote layout in the JSX from the start (anecdote 14), and the checks passed first time.

NumbersCopy Anchor Link

bun run check 457 tests, check:db 403, check:page 201, none failing. Storybook: 21 new stories in light and dark, 42 axe checks, no problems; keyboard checked by hand on each control (Tab shows the ring on the drawn control, Space checks, arrows move the radio). $0. No benchmark run.

Where to go nextCopy Anchor Link

DS-3, feedback: Alert, Tooltip, Skeleton and Spinner, Toast. When you add a chapter, add its row to the index at the top of this file.

Chapter 37 · FeedbackCopy Anchor Link

2026-11-09 · Week 4, block 33 · PR #63

The goalCopy Anchor Link

DS-3: the components that tell the user what is happening. A failed run, a saved change, a hint on an icon button, something loading. The dashboard had written each of these by hand: a red <p role="alert">, a grey "Thinking…", an <output> in danger red. An agent asked for the same would do the same, and nothing in the design system would say otherwise.

What we builtCopy Anchor Link

• Alert (packages/ds/src/alert.tsx): tone, title, the message as children. danger and warning are role="alert", the others role="status". Each tone has its own icon shape (an i, a tick, an exclamation, a cross), so it is not told by colour alone.
• toast() and <Toaster /> (toast.tsx): one toaster at the root, toast({ title, description, tone }) from anywhere, toast.dismiss(id). The tones of Badge, mapped to Zag's types inside. A danger toast stays until closed.
• Tooltip (tooltip.tsx): content and one trigger. An ink region like the header, with a border so it shows on a dark page. It describes the trigger; an icon button still needs its aria-label.
• Spinner (spinner.tsx): an <output> (a live region by itself) with a label, hidden unless showLabel.
• Skeleton (skeleton.tsx): lines of text, or a block in two sizes. Hidden from screen readers; the region that loads is the one marked aria-busy.
• Tokens: two animations with their keyframes, paused under reduced motion, and a tooltip layer. Zag's toast variables, raw by nature, sit in the config's globalCss.
• The dashboard adopts them. A failed run shows an Alert; the chat's error is an Alert, its "no gateway key" notice a warning Alert, and "Thinking…" a Spinner with its label.
• What agents read. Four new rows in "Choosing a component". In the MCP, get_component toast finds Toaster (a family's other exports are aliases), Toaster "takes no props", and the internal ToneIcon moved to src/internal/, out of the component list.

Try it yourselfCopy Anchor Link

bun run storybook   # Alert, Toast, Tooltip, Spinner, Skeleton

<Alert tone="danger" title="The transfer failed">
  The recipient's bank rejected the IBAN. Check it and send the transfer again.
</Alert>

<Button onClick={() => toast({ title: "Run saved", tone: "success" })}>Save</Button>
<Toaster />

Open a failed run in the dashboard (bun run web, then a run with an error): the error is an Alert.

Decisions and whyCopy Anchor Link

• Toasts through a function, not a component per message. A toast is the result of an action, so it is called where the action ends. One <Toaster /> owns where they appear. The function takes four fields, not Zag's twenty.
• Only waiting moves. The design system had no motion at all; it now has two animations, both meaning "wait", both paused for users who ask for less motion. Toasts and tooltips appear without animation.
• A library's raw values live in the config. Zag positions toasts with CSS variables the stylesheet must apply. In toast.tsx they would need escape hatches, and the dogfood test would count them. In panda.config.ts, where raw values belong, they are what they are: mechanics, not style.

What went wrongCopy Anchor Link

• A toast took every page down (anecdote 15). toast.tsx created its toaster when the module loaded; Ark's createToaster is client-only; @vektos/ds is one barrel that server components import. Every page of the dashboard crashed, and every check had passed. "use client" fixed it. next build would have caught it: one more reason for CI.
• Toasts off the edge of the screen. Zag gives each toast position: absolute and its place in CSS variables; without the lines that apply them, they hung half outside the window. Found on the first screenshot.
• The grader counted a tooltip's content. content is a CSS property, so <Tooltip content="…"> was "styled from outside". It joins size in the list of a component's own props named like CSS. That list is written by hand; it is a known gap.
• The governed model caught me. A story put paddingBlock on a Stack. It is not a layout prop, and the compiler said so. marginBlock it is.
• Two more claims the MCP got wrong: Toaster "accepts the props of the underlying HTML element" (it takes none), and ToneIcon listed as a part of Alert.
• The retrieval question moved again: guide 06 now says "seventeen components".

NumbersCopy Anchor Link

bun run check 457 tests, check:db 403, check:page 201, none failing. Storybook: 16 new stories, 32 axe checks in light and dark (tooltips opened, toasts shown), none failing; toasts checked by hand (a neutral one goes after five seconds, a danger one stays, Close removes it). Dashboard: six pages, including a failed run, at two widths and two themes, 24 axe checks, none failing. $0. No benchmark run.

Where to go nextCopy Anchor Link

DS-4, navigation: Tabs, Pagination, Breadcrumb and Link. The dashboard adopts them, and Link replaces the global a:not([class]) rule. When you add a chapter, add its row to the index at the top of this file.

Chapter 38 · NavigationCopy Anchor Link

2026-11-09 · Week 4, block 34 · PR #64

The goalCopy Anchor Link

DS-4, the last block of the design-system expansion: the pieces that move between pages. Until now the design system had no link at all. A global rule, .vektos a:not([class]), underlined every bare <a>, and the dashboard drew its own pager and its own "← All runs". A rule that styles bare elements makes the component optional; the plan was to replace it with a Link, and to give the dashboard the navigation components it had written by hand.

What we builtCopy Anchor Link

• Link and RouterProvider (packages/ds/src/link.tsx): an underlined link in accent.text, with external for another site (new tab, and "(opens in a new tab)" for screen readers). The design system does not import Next. The app gives it a navigate and a prefetch function once, through RouterProvider; a plain click on an internal path then goes through the router, and a click with a modifier key is left to the browser. The link stays a real <a href>.
• Breadcrumb (breadcrumb.tsx): items, the current page last and marked aria-current="page".
• Pagination (pagination.tsx): page, count, and href(page) for links or onPageChange for buttons. First, last and the pages around the current one, with gaps; on a phone the numbers give way to "Page 5 of 12". The choice of pages is a tested pure function (internal/page-range.ts).
• Tabs (tabs.tsx): items with value, label and content, on Ark's tabs; the selected tab's line sits on the panel's top edge.
• The global link rule is gone, and the components grader counts a native <a>, pointing to Link.
• The dashboard adopts them. components/router.tsx wraps the app in RouterProvider with Next's router. Every text link is Link; the pager is Pagination (the old pager.tsx is deleted); a run's page starts with a Breadcrumb and shows its screenshots in Tabs, light first. The filter pills, the header and the Overview's badge links keep next/link with their own class.
• The grader reads a component's own props from its interface. vektos-ds.ts collects every <Name>Props in the design system, so page on Pagination is the component's prop, not CSS.

Try it yourselfCopy Anchor Link

bun run storybook   # Link, Breadcrumb, Pagination, Tabs
bun run web         # open a run: Breadcrumb, Tabs; /runs: Pagination

// Once, in a client component around the app:
<RouterProvider navigate={(href) => router.push(href)} prefetch={(href) => router.prefetch(href)}>
  {children}
</RouterProvider>

// Anywhere, server components included:
<Breadcrumb items={[{ label: "Runs", href: "/runs" }, { label: "loan-contract" }]} />
<Pagination page={2} count={9} href={(page) => `/runs?page=${page}`} />

Decisions and whyCopy Anchor Link

• A provider, not next/link. The design system serves any React app, and the benchmark's pages are bundled without Next. asChild on every link would work too, but asks the page to write two components for one link. Spectrum's RouterProvider showed the shape.
• No bare-element styles. With the global rule, an agent's <a> looked right, so Link was optional. Now a bare <a> looks like text and is counted, as <button> always was.
• Our own page range, not Ark's pagination. Ark's link mode renders buttons with an href. Ten lines of a pure function, with tests, gave links that are links.

What went wrongCopy Anchor Link

• A hook where a server component renders. The first Pagination called the router hook in each item, and took href as a function. Server pages cannot run that hook, nor pass a function to a client component. It now computes every address on the server and hands a small client anchor (internal/routed-anchor.tsx) a string.
• Slots named like CSS. Pagination's recipe had slots called position and gap. The token-names grader read the slot's object as a condition map of the CSS property, and took hideFrom and hideBelow for unknown breakpoints. The slots are renamed, and the grader now knows Panda's hideFrom, hideBelow and srOnly as style properties.
• The third CSS-named prop. After size and content, page (a CSS property for print) was "styled from outside". Instead of a third line in a hand-written list, the grader reads the components' interfaces.
• A test tied to the alphabet. The MCP test for an unknown name checked that the list started with "Alert, Badge, Button"; Breadcrumb broke it. It now checks the shape of the list.
• A sort that was not one. toSorted((theme) => …) put light first by luck: a comparator takes two arguments. Fixed before the commit.
• The ds package had no Bun types, so its first test did not typecheck (TypeScript 7 includes no @types unless listed). It has @types/bun and "types": ["bun"] now.

NumbersCopy Anchor Link

bun run check 462 tests, check:db 404, check:page 202, none failing; next build passes. Storybook: 12 new stories in light, dark and at 390 pixels, 36 axe checks, none failing; arrows move the tabs, buttons move the pagination. Dashboard: six pages (the run list's second page included) at two widths and two themes, 24 axe checks, none failing; a breadcrumb and a pagination link move without a page load. $0. No benchmark run.

Where to go nextCopy Anchor Link

The design system has grown from eight components to twenty-one, governed. Next on the plan: CI on GitHub Actions (bun run check and next build, after anecdote 15), then more tasks that use the new components, then the rerun on free models. When you add a chapter, add its row to the index at the top of this file.

Chapter 39 · CICopy Anchor Link

2026-11-09 · Week 4, block 35 · PR #65

The goalCopy Anchor Link

Until now every check of every block ran on one Mac, and "green" meant green there. Two things made that worth fixing now. Anecdote 15: a design-system module broke every page of the app and passed bun run check, because check never builds the app. And the design system had just grown from eight components to twenty-one, so the next blocks (more tasks, the rerun) would lean on it.

You asked first whether CI costs anything. It does not: GitHub's free plan includes 2,000 Linux minutes a month for a private repository, with a $0 spending limit, and counts no minutes for a public one. A run takes about two and a half minutes.

What we builtCopy Anchor Link

• .github/workflows/ci.yml: one job on ubuntu-latest, for pull requests to main and pushes to main. Postgres 17 with pgvector from the same docker-compose.yml as on a laptop, Bun 1.4.4, bun install --frozen-lockfile, then bun run check, bun run check:db, bun run build and bun run docs:build. Caches for Bun's downloads and Next's build cache. A new push cancels the previous run of the same pull request.
• No secrets. No AI Gateway key, no auth secrets: nothing in CI can call a model.
• The rule in AGENTS.md now says a block is merged when its CI run is green (gh pr checks --watch), not when check passed locally.
• Guide 20: what runs, why, how to reproduce a run from a fresh clone, and what is left out.

Try it yourselfCopy Anchor Link

gh pr checks --watch       # on a branch with a pull request
gh run list --limit 5

A fresh clone reproduces it on a laptop:

git clone <repo> /tmp/vektos && cd /tmp/vektos && bun install --frozen-lockfile
bun run check && bun run db:up && bun run check:db && bun run build && bun run docs:build

Decisions and whyCopy Anchor Link

• Rehearse on a fresh clone first. Before writing the workflow, the steps ran in a clean clone in a temporary folder: no node_modules, no .next, no styled-system. That is where a missing generated file would show. None did: bun install runs the design system's Panda codegen, and tsc -b needs neither next-env.d.ts nor Fumadocs' .source.
• The database through docker compose, not a service container. The backup test reaches the compose database with docker exec vektos-db-1. A GitHub service container has a generated name, and pg_dump on the runner is older than Postgres 17. Compose gives CI the same container, by the same name, as a laptop.
• check:db in, check:page out. The database tests cost about twenty seconds and cover 404 tests that only ran locally. The browser tests need Chrome installed; they can come later.
• One job, cheap steps first. A lint error stops the run before the builds. Two parallel jobs would finish sooner and use twice the setup minutes.

What went wrongCopy Anchor Link

• Nothing in the run. It passed on the first push: 462 tests in check, 392 database tests passed and 10 skipped (the same ones as on the Mac), both builds.
• A commit type that the rules do not have. The workflow was first committed as ci: …; AGENTS.md lists feat, fix, docs, style, refactor, test and chore. It became chore(ci): … before the merge.
• GitHub cannot enforce it. Required checks need a paid plan for a private repository, so a red pull request can still be merged by hand. The rule lives in AGENTS.md until the repository is public.

NumbersCopy Anchor Link

First run, with empty caches: 2 min 34 s. Install 6 s, check 39 s, check:db 22 s, web build 29 s, docs build 26 s. At the pace of these blocks, about 400 to 500 minutes a month, a quarter of the free allowance. $0.

Where to go nextCopy Anchor Link

More tasks that use the new components (the plan's tasks 5 to about 10), then evaluations for the chat, then the rerun on free models. When you add a chapter, add its row to the index at the top of this file.

Chapter 40 · Ten tasksCopy Anchor Link

2026-11-09 · Week 4, block 36 · PR #66

The goalCopy Anchor Link

The first block of the plan written after the design-system expansion (docs/PLAN.md, "Next steps"). The benchmark had five tasks, all built from the first eight components: cards, tables, badges, forms, a dialog. The design system now has twenty-one. A context that teaches an agent Switch, Tabs or toast() could not show it in the results, because no task asked for them.

Before starting you asked whether more tasks means more money. Not by themselves: writing tasks and their answers costs nothing. A batch costs tasks × contexts × models × repeats, so ten tasks double a batch on a paid model; the next batches run on free models first.

What we builtCopy Anchor Link

Five tasks, each a folder of four files (guide 08):

• Synthetic data, made-up companies, Italian IBANs in the right shape; the 32 transactions are generated by a small script so their order is fixed.
• Labels in quotes. Every label that mustShow looks for ("Reason", "What happened", "Money in", "Cancel mandate") is written in the prompt. An agent should not have to guess a label to pass.
• The page checks run on every answer. check:page used to put only the statement solution through the benchmark's page checks. It now runs all ten: typecheck, lint, a build, Chrome, axe in both themes, mustShow on screen. All ten come out clean, in about a second each.
• docs/PLAN.md has the order decided before this block: tasks, DS-5, the generative UI spike and package, DS-6 and DS-7, the rerun on free models, the "spec" experiment.

Try it yourselfCopy Anchor Link

bun test tasks             # 46 tests: grade clean, use their components, render mustShow
bun run check:page         # every answer through the page checks, in Chrome
bun run bench --model mock/ideal --no-page --task transactions

Decisions and whyCopy Anchor Link

• One new family per task. A task that needs everything tests nothing in particular. Each of the five leans on one family the old tasks never used, so a difference between contexts has a likely cause.
• No loading states. A task built around Skeleton or Spinner would show different texts while it loads and after, and mustShow reads the page once. They wait for a check that can follow a page over time.
• Our answers are written the way we ask agents to write. Stack and Grid for layout, components for everything they cover, css() only on the page's own elements.

What went wrongCopy Anchor Link

• A test that knew the five tasks by name. The CLI's --budget test listed the five ids and expected "4 task(s) not run". It now reads the tasks from the folder.
• Behaviour is still untested. The tests see a page as it loads. That a dispute shows its errors after an empty submit, or that a cancelled mandate gets its badge, is asked by the prompt and done by our answer, and checked by nobody. Written down as a known gap in guide 08.

NumbersCopy Anchor Link

Ten tasks. bun run check 491 tests, check:db 413, check:page 211, none failing; the page checks of the five new answers take 1.0 to 1.4 seconds each. $0. No benchmark run.

Where to go nextCopy Anchor Link

DS-5, content: Heading, Text, Stat, DataList, EmptyState, Progress, Avatar, IconButton, Format. When you add a chapter, add its row to the index at the top of this file.

Chapter 41 · ContentCopy Anchor Link

2026-11-09 · Week 4, block 37 · PR #67

The goalCopy Anchor Link

DS-5 of the plan: the components that carry content rather than controls. Two reasons at once. Pages kept writing the same things by hand: a title as <h1 className={css(…)}>, a list of facts as a <dl> with a grid, an amount through Intl.NumberFormat. And the generative UI blocks that come next need them: a JSON spec cannot write an <h1> or call Intl, so whatever a model composes must come from components. Names and parts were looked up in @pyck/react through its MCP server; the APIs and the code are ours.

What we builtCopy Anchor Link

• Heading: level 1 to 4 for the outline, size for the look, defaulting by level.
• Text: size, tone (muted and the status tones), weight, numeric, mono. As a span it keeps the size and colour around it unless told otherwise.
• FormatNumber and FormatDate, with LocaleProvider: a raw value in, the locale's text out. Dates are read in UTC, so 2026-11-08 is the 8th everywhere.
• Stat and DataList: description lists, so a screen reader reads each value with its label. Stat has a change and a changeTone (good news, bad news), not an arrow.
• EmptyState: what is empty, why, what to do.
• Progress and ProgressCircle on Ark: a label, a value or null, and the value in words when a percentage is not the point ("EUR 920 of 1,000").
• Avatar on Ark (initials from the name), IconButton (its aria-label required by the type) and CloseButton.
• A sizes.2 token for the progress bar, and the ring's --size and --thickness in the config with the other library mechanics.
• The dashboard adopts them. PageHeader is a Heading and a Text; the overview's stats are Stats in cards; a run's facts are two DataLists; the empty overview is an EmptyState; the chat's two budgets are Progress bars with the spend in words, red once a cap is reached.
• 20 stories, 52 axe checks in light and dark; the guidelines served to agents have nine new rows in "Choosing a component".

Try it yourselfCopy Anchor Link

bun run storybook   # Heading, Text, Stat, DataList, EmptyState, Progress, Avatar, IconButton, Format