The words were the same
We feared the AI SDK was rewriting what the agent reads. A diff of its requests against the official SDKs' showed the same words, a different format and a few defaults that nobody had chosen.
Date: 2026-11-09. Context: deciding whether the runner should leave the AI SDK
(feat/bench-payload-diff, guide 19).
What happened
Comparing gateways (Vercel's AI Gateway, Requesty) raised a question about the benchmark itself. Every real run goes runner → AI SDK → AI Gateway → provider. If the AI SDK rewrote the system prompt, the tool schemas or the tool results, part of what Vektos measures would be the AI SDK, not the model.
Instead of guessing, we captured the requests. A fake server answered the runner and a
hand-written loop with the official SDKs, with the same script: one file tool and one MCP tool
called, then a short reply. The task was statement, the context ds-mcp-check. Then we
compared the bodies field by field.
The words were identical. The system prompt, the task prompt, every tool description and both tool results were the same, character for character. What differed:
| Difference | Anthropic | OpenAI | Does the model see it? |
|---|---|---|---|
| Text sent as a list of blocks, not a plain string | system, prompt, MCP result | prompt, MCP result | No: same text |
System prompt as a developer message, not instructions | yes | Unlikely: instructions also becomes a message at the top | |
tool_choice: auto sent explicitly | yes | yes | No: it is the default |
max_tokens: 128000, the model's maximum | yes | Only as a ceiling | |
$schema draft-07 instead of zod's 2020-12, on the file tools | yes | yes | Unlikely: it names the schema's dialect, the fields are the same |
additionalProperties: false added to every MCP tool | yes | yes | Yes, slightly: "no other arguments" |
Replayed tool calls without their item id and status | yes | No: the conversation is sent whole |
What we learned
- The layer keeps the meaning. Nothing the model reads as language changed. The run-to-run
noise of a model is far larger than a
developerrole instead ofinstructions. - Two defaults were chosen by nobody.
max_tokensis 128,000 because the AI SDK uses the model's maximum when the caller sets none. Through the gateway we do not set one either, so the gateway decides, and we cannot see what it picks.additionalProperties: falsecomes from@ai-sdk/mcp, which adds it to every MCP tool's schema. The MCP server never said it. - Measure the layer, do not argue about it. "Abstractions change the prompt" sounds right and is easy to believe. It took one offline script and no key to find out.
What it means for Vektos
The runner stays on the AI SDK. The question "is it the model or the library?" now has an answer
with evidence, and the snapshots in payload.test.ts keep it true. An upgrade that changes what
is sent fails bun run check until someone reads the change.
The gateway hop is still unseen. The AI SDK sends the gateway its own call options, and the provider request is built on Vercel's side.
Follow-ups
- Decide whether the runner sets
maxOutputTokensitself, so that the ceiling is the same for every model and does not depend on the AI SDK's or the gateway's defaults. - Check the gateway hop with one real call: the same first step through the gateway and direct,
then compare
inputTokens. It needs a provider key and costs well under a cent. - Extend the check to Gemini and the OpenAI-compatible APIs before running those models directly.
The class that did not win
A design-system button could not be changed with a class. Two Panda classes for the same property both stayed on the element, and the wrong one won. Nothing in the toolchain can see it.
The args Panda never saw
The first Stack and Grid stories passed their layout through Storybook args. They rendered with no gap, no responsive row and no columns, and no tool said a word.