Zero rules, no error
The benchmark asked axe for the tag "best-practices". axe has "best-practice". It ran no rule for that tag, for weeks, and said nothing.
Date: 2026-11-10. Context: APP-5, quality (feat/app-quality, journal ch. 52).
What happened
APP-5 added a check of the app's own pages with axe, including the heading order, a
"best-practice" rule. Looking at how the benchmark called axe, to reuse it, showed its options:
runOnly with the tags wcag2a, wcag2aa, wcag21a, wcag21aa and best-practices. axe's tag is
best-practice, singular. Asked for the plural, axe ran zero rules for it, returned no error, and
the page check reported its findings as if the tag had been honoured. The same options even turned
off one best-practice rule, region, by name: the intent was clear, and it had never happened.
What we learned
- A filter that matches nothing looks like a clean result. axe validates rule ids, not tags. An unknown tag is not an error; it is an empty set.
- The third time is a pattern. A resize that did nothing (anecdote 18), story ids that drew "Couldn't find story", and now a tag that selected nothing: three checks of nothing, each passing. A check needs a test that it can fail.
- Fixing it is not free. Turning on the rules that were meant to run would raise the quality count of every new run, and the stored runs, paid ones included, cannot be graded again. The honest fix keeps the behaviour the data has and removes the dead tag.
A day later it happened a fourth time. The new grader hash globbed its files with a pattern Bun did not expand, found none, and returned the hash of nothing; its test passed because it skipped itself whenever a file under the folder was being edited, which it always was while the hash was written. The hash now refuses an empty set, and the test checks it is not the hash of nothing before anything else.
What it means for Vektos
The benchmark's numbers are only as good as the checks behind them, and a misspelt option is as
invisible to a reviewer as it was to the agent that wrote it. Every check that filters by a name
should prove the name exists. page.test.ts now asks axe whether it knows each tag.
Follow-ups
- Decide axe's
best-practicerules with the next grader version, and mark runs with that version. - Look for other name-based filters in the graders (rule ids, grader names) and test that each one matches something.
The 390 that was 1440
Eighty accessibility checks reported clean at 390 pixels. They had run at 1440: the resize did nothing to a tab in the background, and nothing said so.
The cd that failed
A clone failed, so the cd after it failed, so a history rewrite meant for a scratch copy ran on the working repository. Nothing was lost, by luck and by a remote.