Results

Two rounds so far. Four paid models wrote five fintech tasks at five context levels (below). Then two free models wrote all ten tasks at the same five levels, three times each: 300 runs at $0 (the free rerun). Every number is the mean per run, from the history in Postgres (bun run bench:history); bun run bench:report and the web app show the pages behind them.

Four paid models, five tasksCopy Anchor Link

Design system counts violations of the design-system graders (raw values, token roles, native elements, suppressions, protected files). Quality counts type errors, lint findings, accessibility findings in light and dark, and render problems.

Each cell is design system / quality. A compiler error that only restates a design-system violation (TS2769 inside a css() call with a raw value) is counted in the first number, not the second. Before that was fixed, quality in the none row read 41, 167, 108 and 349: see anecdote 08.

What it saysCopy Anchor Link

• Context is the lever, and the order holds across four makers. From no context to our MCP, every model drops from tens or hundreds of design-system violations to about zero.
• Writing without context is not a weak-model problem. The most capable and most expensive model, Kimi, wrote the most raw values (294 per run): it writes longer, more elaborate pages, and each one is more chances to break the rules.
• Our MCP beats Storybook's on tokens. With Storybook's MCP, DeepSeek and Kimi reach 0.4 design-system violations, but Haiku stays at 4.3 and Luna at 36.2. Storybook documents the components; nothing in it says tokens exist, and agents that skip that guess raw values.
• A checker takes the design-system column to zero and leaves quality alone, unless it compiles too. ds-mcp-check leaves 2 to 20 quality violations per run. With type and lint checks in the tool (ds-mcp-check-full) three models reach zero and Haiku 0.3 (anecdote 07).
• Context can lower the bill. Kimi without context costs $0.21 per run, with ds-mcp $0.13: the guided agent writes less. Cheap models pay a little more for context (Haiku $0.003 to $0.007), still cents.

The free rerun: ten tasks, three repeatsCopy Anchor Link

Nemotron 3.5 Lightning and Beam, both free, label free-rerun (bun run bench:history -- --label free-rerun). 30 runs a cell: ten tasks, three repeats. Done is the share of runs whose page showed everything the task asks for.

• The order holds on free models. Context takes the design-system column from 10.5 (Nemotron) and 38.0 (Beam) to about zero, and the full checker gives the best quality and the most pages done.
• Following the rules is not finishing the page. Nemotron at ds-mcp-check breaks almost no design-system rule and finishes 17% of its pages. Done is a column of its own for that reason.
• Storybook's MCP is close to ours for a strong free model (Beam 0.8 against 0.3) and far from it for a weak one (Nemotron 6.3 against 1.7).

Which tasks are hardestCopy Anchor Link

On /tasks, read only on the series that ran every task (here the ten free ones), each weighing the same: otherwise the first five tasks, also run by the paid models, would look easier. Hardest: Account details, 43% done; easiest: Account statement, 83%. Eight of the ten sit between 43% and 57%, inside the ± 18 points that 30 runs allow; payment approval (67%) and the statement stand apart from the hardest. The forms (KYC, card dispute) leave the most quality violations, about 13 a run, against 4 to 7 for the rest. Counting every run instead would have put KYC at 67% done, not 47%: the paid models did it.

Views instead of codeCopy Anchor Link

The same free models, answering with a view composed from the catalog of @vektos/genui instead of code (--context genui-spec, label genui-forms, two repeats). The catalog now has forms, so six tasks of ten can be answered this way. On the four form tasks:

• A view cannot break the design system, by construction: zero design-system violations on all 24 runs. The only quality findings were a missing level-1 heading in seven of Nemotron's views (anecdote 23).
• The weak model gains most. Nemotron finishes more pages composing a view than writing code with any help, at about a third of the output tokens and a quarter of the time of its best code context.
• The validator can carry a grader's rule. With a page profile in submit_view (one level-1 heading, headings in order), a second rerun (genui-page) had no quality finding at all: the validator asked Nemotron for a title three times, and it added one each time. Nemotron 11 of 12 done, Beam 12 of 12.
• Binding the data is not free. With views that point at the task's data instead of copying it (genui-spec-data), Beam did 12 of 12 and Nemotron 7 to 8 of 12, against 11 copying: the weak model started writing its views as broken JSON text (anecdote 25).
• It answers a smaller question. A view shows the form; it does not validate after a submit or confirm. The page check grades what a page shows when it loads, for code runs too, so the numbers are comparable; the work behind them is not the same.

A second product: the warehouseCopy Anchor Link

Five warehouse tasks (stock levels, a pick list, a shipment, a goods receipt, a stock adjustment) on the same design system, run once at four contexts by the same free models (label warehouse-free; none twice). Each cell is design system / quality / done, against the banking cells of the same contexts:

• The order of the contexts carries over. No help, tens of design-system violations; with our MCP, a checker or a view, about none. A second product, the same shape.
• The warehouse is harder to finish without help. Beam finished 30% of warehouse pages with no context against 73% of banking ones, most failing to compile at all (unterminated strings, missing semicolons). With the full checker both models finished more warehouse pages than banking ones.
• The warehouse's own components (genui-spec-domain: StockTable, ReceiptLines…) cut what both models wrote by 37 to 46%. Beam used the right one every time and finished 5 of 5; Nemotron used them in two tasks of five and finished 2, against 3 with the shared components (anecdote 26).
• Five runs a cell is little. A share done over five runs is uncertain by about ± 45 points; read the warehouse column as a direction. The banking quality counts come from earlier graders (before chapter 59) except at genui-spec.

What the pages doCopy Anchor Link

Since graders v3 a page is done when it also does what its brief asks after the user acts: an error tied to the field it is about, a confirmation after a valid submit, a difference shown as it is typed (five form tasks; guide 11). One short run, two free models, three contexts (label behaviour-v3):

• "Shows the right text" overstated "done". 20 of 30 pages showed every text; 8 did what the brief asked. The most common miss: an error shown as red text but not tied to its field, which a screen reader never announces.
• Views answer less. A generated form has no confirmation after a submit and cannot show a difference as it is typed; its required fields do get the browser's own check.
• One miss was the design system's. Its date picker had no label inside a form field (anecdote 27); fixed.

How far to trust itCopy Anchor Link

• One design system, ten tasks, one prompt per task. The pattern is robust across models; the exact numbers belong to this setup.
• The design system changed after these runs. Later work added breakpoints, font families and larger spacing and size steps, so batches run from now on are a different design-system version in bench:history (dsHash). The paid table comes from the earlier version; the free rerun from the later one. Compare within a table, not across. /regressions keeps versions apart (guide 23).
• Few repeats. Haiku has three per cell, the other paid models one; the free rerun three. The same Luna task without context gave 146 and 115 violations on two runs, and on /regressions Beam without context spreads by tens between repeats of one task: read single-run cells as an order of magnitude.
• The graders changed after the free rerun. Graders v2 (chapter 59) added axe's best practices, so quality counts from then on are not comparable with these.
• A few cells are short. Kimi without context ran four tasks, not five: its batch hit the $0.80 budget before the last one.
• Quality is a count of mixed things. Type errors and lint findings count one per occurrence; an accessibility rule counts one per theme however many elements it hits. Read it as "how much is left to fix", not as a score out of 100. A token name that does not exist (fontWeight: "bold") is counted here, not in the design-system column.
• Graders measure rule-following, not taste. A page with zero violations can still look poor; the report shows the screenshot next to the number for that reason.

Reproduce itCopy Anchor Link

bun run db:up && bun run db:migrate
infisical run -- bun run bench --model anthropic/claude-haiku-5.5 --context ds-mcp --label try
bun run bench:history
bun run bench:report

Cost of the runs behind this page: about $5.50 in total, of which Kimi $4.30; the free rerun cost nothing. The cost column of bench:history is each cell's total, not the mean per run.