The help that hurt
Letting views point at their data made the strong model perfect and the weak one worse, three reruns in a row. The new feature changed how the weak model wrote everything else.
Date: 2026-11-10. Context: data binding in the generative UI catalog (journal ch. 65).
What happened
Views could now point at the task's data, {"$data": "/transactions"}, instead of copying it. It
looked like pure gain: no mistyped numbers, shorter specs. The first free rerun said otherwise.
Beam bound data in all twelve views and dropped from 12 to 10 pages done; Nemotron used it in
two of twelve and dropped from 11 to 8.
The stored feedback (chapter 64) gave the causes. Beam showed "Giulia" and "Rossi" apart, because
the data had them apart and a pointer could not join them, while the brief wanted the full name.
A bound number in a DataList, which took only text, was refused. Both were the catalog's fault,
and both were fixed: Beam went back to 12 of 12. Nemotron stayed at 8, then 7. It was sending a
view's children as one JSON string, with a brace missing or a Python True. Its views were refused for it 3 times in twelve runs before the prompt showed {"$data": …} examples, and 9 to 15 times in twelve after. A clearer refusal ("this is JSON written as
text, and it does not parse") helped twice in nine.
What we learned
- A new capability changes the whole answer, not only the part that uses it. Nemotron hardly used binding, yet it wrote everything worse once the prompt explained it.
- Help is per model. The same addition made one model perfect and cost the other a quarter of its pages. A default that suits the strong model taxes the weak one.
- Feedback stored is diagnosis possible. Without the tool's answers in the database, three reruns would have shown only the totals.
What it means for Vektos
That is the benchmark's thesis turned on its own tools: what an agent is given is the variable,
and it has to be measured per model. Binding became its own context level, genui-spec-data,
next to genui-spec, rather than a change to it.
Update: not only the binding
The warehouse rerun (chapter 66) ran Nemotron at genui-spec, without binding, on five new
tasks. Two of its five views failed the same way: children sent as one JSON string that does not
parse, on the two longest forms. So the habit is not caused by binding alone; long views bring
it too, and binding made the views longer and the prompt richer in JSON examples. The lesson
holds, its cause is wider.
Follow-ups
- Try binding with fewer words in the prompt: one example instead of three.
- Measure the string-children failures against the size of the view, with and without binding.
- Check whether the string-children habit appears in the chat with the same prompt.
- Add the paid models to the panel before calling binding good or bad in general.
The comment copied from next door
The agent building the benchmark wrote, in the app, the one kind of comment the benchmark counts against agents.
The shortcut only one model took
Given a StockTable that draws the whole stock page in one node, the strong model used it every time. The weak one mostly built the page by hand, or picked the wrong shortcut.