Payload check

WhatCopy Anchor Link

bun run bench:payload runs one task through two clients and keeps every request they send.

• The AI SDK side is the runner itself. runTask gets a languageModel pointed at a fake server, so the requests are the ones a real run sends.
• The official side is a loop written by hand with @anthropic-ai/sdk (Messages API) and openai (Responses API). It gets the same system prompt, task prompt and tools. Each tool's schema comes from its source: zod's own conversion for the file tools, the MCP server's own list for the rest.
• A fake server answers both. The script is the same for both: the first answer calls a file tool and an MCP tool, the second answers with text. Nothing leaves the machine.
• The same run through the AI Gateway shows what the gateway receives from the AI SDK.

WhyCopy Anchor Link

Vektos measures how agents behave. If a library between the runner and the model rewrote the prompt or the tools, part of what we measure would be the library. This check shows the layer's effect directly instead of assuming it. The first result is in anecdote 13. The model reads the same words. The differences are in format and defaults, and none of them is a reason to leave the AI SDK.

Two tests keep it true:

• Same words. Both sides must carry the system prompt, the task prompt, every tool description and every tool result, word for word.
• Snapshots. The AI SDK's requests (Anthropic, OpenAI, gateway) are kept in __snapshots__/payload.test.ts.snap. An upgrade of ai or a provider package that changes the wire fails bun run check, so a change cannot slip in with a version bump.

Try itCopy Anchor Link

bun run bench:payload
bun run bench:payload --task kyc-form --context none

Wrote results/payload-diff/2026-11-09T15-42-28/diff.md

diff.md has one table per API and request. A few rows from the Anthropic table:

| Field | AI SDK | Official SDK |
|---|---|---|
| `max_tokens` | `20000` | `16000` |
| `tool_choice` | `{"type":"auto"}` | — |
| `tools[get_component].input_schema.additionalProperties` | `false` | — |
| `tools[list_files].input_schema.$schema` | `"http://json-schema.org/draft-07/schema#"` | `"https://json-schema.org/draft/2020-12/schema"` |

The folder also has the raw bodies (anthropic-ai-sdk.json, openai-official.json, gateway.json…), to read whole.

When a snapshot fails after an upgrade:

cd packages/bench && bun test src/payload     # read the diff
bun test src/payload -u                       # accept it, once understood

Extend itCopy Anchor Link

• Another API. Add its answers to replay.ts, a loop to official.ts and a family to MODELS in capture.ts. For Gemini, use @ai-sdk/google and @google/genai.
• Another context. storybook-mcp needs an HTTP transport on the official side, and ds-mcp-check-full needs the raw form of its workspace tool. Both are refused for now.
• A longer script. ScriptedCall[] in the CLI sets the first answer's tool calls.

Until chapter 55 the AI SDK's row read 128000: with no cap from the runner, the Anthropic provider sends the model's maximum. The runner now sends MAX_OUTPUT_TOKENS to every provider, and the snapshots changed by exactly that.

Known gapsCopy Anchor Link

• The gateway is a black box. We see what the AI SDK sends to the gateway, not what the gateway sends to the provider. Checking it needs a real call with a key: the same first step through the gateway and direct, then compare inputTokens.
• Two APIs. Anthropic Messages and OpenAI Responses. Not Gemini, not the OpenAI-compatible APIs (DeepSeek, Moonshot, Mistral).
• No streaming. The runner calls generateText, so only plain requests are compared.
• The official side is one way to write it. Another engineer could choose another max_tokens or strict tools. The table shows the choices, not a single right answer.