A citation that pointed at the neighbour
A chunker tuned on our own guides merged 19 short WCAG criteria into the criterion before them. The evals still passed, because they checked the words and not the citation.
Date: 2026-11-09. Context: adding WCAG 2.2 and the European Accessibility Act to retrieval
(feat/rag-regulations).
What happened
The chunker had a rule from the first day: a section of fewer than 30 words joins the one before
it, so no chunk is only a heading. On the project's guides it was right, because ### Numbers
with one line says nothing alone.
Then WCAG 2.2 went in, and I wrote a function to fetch a criterion by its number
(getCriterion("2.4.7")). It found nothing for 19 of the 87 criteria, among them 1.3.1, 2.4.2,
2.4.7 and 3.1.1. They were short ("Web pages have titles that describe topic or purpose") and the
rule had merged each into the criterion before it. The text was still there, under the wrong
heading: a chunk headed Success Criterion 2.4.6 Headings and Labels also held 2.4.7, Focus
Visible.
The evals had been green the whole time. A case passed when the right document held a chunk with the right words, and it did. Nobody would have noticed until the chat cited "WCAG 2.4.6" for a keyboard focus problem.
What we learned
- A check on the words is not a check on the citation. For a recipe or a guide, finding the paragraph is the job. For a law or a standard the answer is the paragraph and where it comes from: the right words under the wrong article is a wrong answer. The dataset had no way to say so.
- A heuristic tuned on one corpus is a bet on that corpus. The 30-word rule came from our own docs. It was never tested on text where every section, however short, is a unit.
- It was found by building the next thing. Not by a test, not by the evals: by writing a function that depended on the headings being right. Using the output in a second way is a check the first way did not have.
- The chunker's rule belongs to the document, not to the program. What counts as a unit is a
property of the text. A document can now say
min-words: 0in its frontmatter.
Fixed, then measured: with the old chunker, 4 of the 22 cases that name a heading fail (2.4.7, 3.1.1, 2.4.2 and "Usage without vision" of the Directive's Annex I). With the new one, none.
What it means for Vektos
The chat will print citations. A citation is a claim that can be checked, so the evals check it: a case can name the heading the answer must be under, and the dataset test fails if no chunk has that heading and those words. The same lesson as anecdote 10 from the other side: a score is only as strong as what the dataset asks.
The second finding came from the same work. For an accessibility violation, searching for its
message found the right criterion first for 3 of 7 cases. But the checker itself, axe, tags every
rule with the criteria it tests (wcag143 is 1.4.3), and following that tag is right for 7 of 7.
When the producer of the data already knows the answer, reading it beats retrieving it.
Follow-ups
- Cases with a
headingexist for 22 of the 72. The rest check the words only; every case about a law or a standard should name its article or criterion. - Other documents with units to cite (a client's policy, a contract) need
min-words: 0too. A warning when a document has many short sections and the flag is off would catch it earlier. - The 30 axe rules with no criterion (best practices) still need search. A hand-made list from rule to the nearest criterion, checked against the corpus, would cover the most common ones.
The hybrid that was worse than its best half
Merging full-text search with vector search scored below vector search alone, with four different embedding models. A reranker fixed it, and the reason says where each half helps.
The class that did not win
A design-system button could not be changed with a class. Two Panda classes for the same property both stayed on the element, and the wrong one won. Nothing in the toolchain can see it.