The task the paid models made easy
Counting every run, the KYC form looked like one of the easier tasks. It was the second hardest: the paid models had done it.
Date: 2026-11-10. Context: task difficulty on /tasks (journal ch. 61).
What happened
The first query for task difficulty was the obvious one: for each task, the share of its runs that were done. The KYC form came out at 67%, in the easier half. The five tasks added later all came out below 60%.
The five original tasks had been run by the four paid models as well as the two free ones; the five later ones by the free models only. Stronger agents had lifted the first five. Read only on the agents that ran every task (the ten free series), each weighing the same, the KYC form fell to 47%, second hardest of ten.
What we learned
- A mean over runs is a mean over whoever ran them. When the mix of agents differs between tasks, the ranking measures the agents as much as the tasks.
- Compare on a panel. The same agents, measured the same way, on every item compared. Fewer runs count, but they count for the same thing.
- The same trap waits everywhere in the app. A model with more runs at easy contexts, a context tried only on easy tasks: any average that pools unequal groups can do this.
What it means for Vektos
Every new comparison in the app (tasks, contexts, models) should say which runs it is read on,
and leave out the ones that would tilt it. /regressions already compares on shared tasks only;
/tasks now reads difficulty on a panel and says how many series it left out.
Follow-ups
- Check the overview's per-context means for the same imbalance: the paid models ran fewer tasks and fewer repeats than the free ones.
- Rerun the paid models on the five later tasks, if the budget allows, to widen the panel.
The regression the graders did not cause
The first regression the new page found came right after a change to the graders. The graders could not have caused it.
The title the catalog only asked for
The catalog said a view's level-1 heading comes once. One model never wrote it, and nothing told it to until the page was graded.