Runner

WhatCopy Anchor Link

The runner lives in packages/bench/src/runner/. runTask({ task, model, context }):

1. gives the model a system prompt, the task prompt and a set of tools;
2. lets it call the tools until it stops (or hits maxSteps);
3. grades every file it wrote and flags edits to protected files;
4. with a page checker, typechecks, lints and renders the page (guide 11);
5. prices each model call and adds them up.

WhyCopy Anchor Link

• In-memory files. Runs are repeatable, can run in parallel, and an agent cannot damage the repo. The agent sees only fixture.json, so it cannot read the reference solution or the design-system source: the context level is the only way in.
• One system prompt for everyone. It says Next, React 19, @vektos/ds, Panda and where to write. It says nothing about components or tokens.
• Mock models. A mock plays a fixed script, so the loop (tools, MCP, grading, pricing, results) is tested without a key and without cost. Mock token counts are estimated from the request size, tool list included, so a bigger context costs more even in a mock.
• Cost per step. onStepEnd gives the usage of each request; long-context rates depend on one request, not the run (guide 09).
• A path must be a path. write_file refuses one with control characters, quotes or brackets, longer than 200 characters or with a part longer than 100, and says so to the agent. Beam once sent a page's whole content as its path; the results could not be written and the batch stopped (ch. 68). Writing a result now skips a file the disk refuses instead of stopping.
• The feedback is kept. Each step records what the tools that judge the agent's work answered (FEEDBACK_TOOLS in run.ts: check_code and the spec mode's submit_view), at most 4,000 characters each; the run's page shows it under the call. Other tools' answers (files read, documentation) are left out: large, and they say nothing about the run.
• The gateway's charge is kept beside ours. A real run stores what the AI Gateway billed for each request (gatewayUsd) and the CLI warns if our price is off by more than 2%. A wrong price table cannot go unnoticed for long.
• A budget stops the batch, not the task. --budget 1 stops starting tasks once the spend reaches $1. A task already running is not cut short (--max-steps limits it), so the total can pass the budget by one task.
• MCP by path. ds-mcp is started as bun packages/ds-mcp/src/cli.ts, not imported: ds-mcp depends on bench for check_code, and an import would be a cycle.

Try itCopy Anchor Link

bun run bench

A whole sweep on the free models, as in chapter 53 (Storybook must be running for storybook-mcp; about three hours a model; $0):

for context in none storybook-mcp ds-mcp ds-mcp-check ds-mcp-check-full; do
  infisical run -- bun run bench --model reflection/beam-free --context $context \
    --label free-rerun --budget 0.05 --no-backup
done
bun run bench:history -- --label free-rerun

task              page  done           ds   qual  steps  tokens in/out   cost
kyc-form          yes   yes            0    0     3      3917/1460       $0.0112
statement         yes   yes            0    0     3      3834/729        $0.0075
...

bun run bench --model mock/rogue --task statement   # ds 1: panda.config.ts edited
bun run bench --model mock/lazy                     # page NO: did nothing
cd packages/bench && bun test src/runner

Full options in the user guide.

Extend itCopy Anchor Link

• A context level. Add it to CONTEXT_LEVELS and openContext in context.ts.
• A mock behaviour. Add a case to plan() in models.ts: a list of tool calls and a final text.
• A real model. Use its gateway id. Add it to PRICES first, or the run refuses to start.
• A local model (the provider seam). --model local/<name> asks an OpenAI-compatible server for <name>: VEKTOS_LOCAL_BASE_URL (Ollama's http://localhost:11434/v1 by default), VEKTOS_LOCAL_API_KEY if it needs one. It costs nothing and needs no gateway key: bun run bench --model local/qwen2.5-coder:7b --context ds-mcp --task statement --no-save. Another provider is one more branch in resolveModel (models.ts) and its prices in pricesFor.
• The output cap. --max-output-tokens (default MAX_OUTPUT_TOKENS, 20,000) is what one model call may write; the batch stores it.
• Another model client. runTask({ languageModel }) calls that model instead of the one model names; model still picks the price. The payload check uses it to see what the runner sends (guide 19).
• Another runner (Claude Code, Codex). RunResult is the contract; produce the same shape and the rest of the pipeline works.

Known gapsCopy Anchor Link

• Batches before chapter 55 ran without a cap: each provider's default applied (Anthropic's SDK sent 128,000, the others nothing). Their max_output_tokens is null, and the run page says so.
• Results are files first. The database keeps them as history (guide 12); results/ stays as the readable copy.
• Repeats are separate batches. Three runs of a cell are three batches with one label; no variance is computed yet.