Regulatory corpus

WhatCopy Anchor Link

Two public texts are kept in corpus/ and searched together with the project's own docs, so a question like "why is low contrast a problem?" can be answered with a rule and a citation:

brew install pandoc                     # only to regenerate WCAG
bun run rag:corpus                      # download and convert both texts into corpus/
bun run rag:ingest                      # ingest docs/ and corpus/
infisical run -- bun run rag:embed      # vectors for the new chunks
bun run rag:criterion 1.4.3             # a WCAG criterion by number, no search

The Markdown is committed, so the evals read the same text on every machine. Each file starts with its source and license in the frontmatter, and with min-words: 0 (see Why).

Two ways to reach the rule behind a violation:

The lookup is for violations that axe tags (75 of its 105 rules). Search is for everything else: a rule with no tag, a question in a person's words, a design-system violation that has no WCAG number.

WhyCopy Anchor Link

• HTML, not PDF. The plan said "2 regulatory PDFs". Both texts are published as structured HTML (the W3C page, the Publications Office's XHTML), so the headings, the lists and the numbering are in the markup. A PDF would have meant guessing them back from a layout.
• The Publications Office, not EUR-Lex. EUR-Lex's own pages answer a script with an empty "202" (a bot check). The Publications Office serves the same text at publications.europa.eu/resource/celex/32019L0882 when asked for XHTML in English.
• Converted by code that is tested. The Directive's XHTML has no headings: a chapter is a paragraph with a class. eaaToMarkdown reads it with Bun's streaming HTMLRewriter and writes ## CHAPTER II — … and ### Article 4 — …; two-cell tables become (a) text lines; footnotes and the signature are dropped; everything else stays word for word. WCAG goes through pandoc and cleanWcag removes the page furniture (table of contents, editors, the glossary's back-references, the bibliography) and turns each glossary term into a heading.
• A criterion or an article is a unit to cite. The chunker joins a section of fewer than 30 words to the one before it, which is right for a guide (### Numbers with one line) and wrong for a law: 19 of WCAG's 87 criteria, among them 1.3.1, 2.4.2, 2.4.7 and 3.1.1, were merged into their neighbour, so a citation of 2.4.7 would have pointed at 2.4.6. A document can say min-words: 0 in its frontmatter and its short sections stay apart. (Anecdote 11.)
• A deterministic link where there is one. axe-core tags each rule with the criteria it tests. That is exact, and free. Search, which can put 1.4.11 above 1.4.3 for a contrast message, is the fallback.
• The headings are part of the eval. A case can name the heading the answer must be under. Finding the right words under the wrong article is a miss.

Try itCopy Anchor Link

bun test packages/rag/src/corpus         # the two converters, on small examples
bun run check:db                         # includes: every axe rule leads to its criterion
bun run rag:criterion 2.4.7              # "Focus Visible", on its own
infisical run -- bun run rag:eval --scope corpus
infisical run -- bun run rag:eval --kind violation
infisical run -- bun run rag:search "do very small companies have to comply" --limit 3

What the evals sayCopy Anchor Link

72 cases over 772 chunks from 38 documents (the 48 about the project, 17 questions about the two texts, 7 violation messages). Embeddings openai/text-embedding-3-small, reranker voyage/rerank-2.5-lite.

• The pattern of guide 15 holds on a different kind of text: vectors beat words, the hybrid is worse than vectors, and the reranker is best (recall@5 of 100% on the questions about the texts).
• Searching for a violation by its message is the weakest of the three kinds. Seven cases are too few to rank the searches, but not to see that the lookup is the right tool.
• The four questions the reranker did not put first: colour alone for errors (rank 4), very small companies (rank 2), users who cannot see (rank 2) and online shops (rank 2). Each uses other words than the text: "colour" for "Color", "very small companies" for "Microenterprises".

Cost. Embedding the two texts costs about $0.002 with OpenAI small (about 105,000 tokens).

Extend itCopy Anchor Link

• Another text (PSD2, GDPR, a client's policy). Convert it to Markdown with headings that are the units to cite, put it in corpus/ with source, license and min-words: 0 in the frontmatter, then rag:ingest and rag:embed. Add cases with a heading to the dataset.
• Another converter. corpus/fetch.ts has one entry per text. A text that is HTML with real headings needs pandoc and a cleaner like cleanWcag; one that is not needs a converter like eaaToMarkdown, with a test on a small example.
• A rule from another checker. criterionFromAxeTag reads axe's tag format. Another tool that names a WCAG number needs its own small reader, then getCriterion works unchanged.

Known gapsCopy Anchor Link

• English only. The Directive exists in every EU language and only the Official Journal text is authentic; this is a reading copy.
• The original text of the Directive, not a consolidated version. Later amendments are not applied.
• No harmonised standard. The technical requirements that make the Directive concrete (EN 301 549, which points at WCAG) are not in the corpus. The link from a design-system violation to an article of the Directive is not made: only axe's violations have a number.
• A violation with no tag has no deterministic link. 30 of axe's 105 rules (best practices) carry no criterion; for those the search is all there is, and it is the weaker half.
• Only 17 questions about the texts and 7 violation messages. Enough to see a pattern, not to rank models. A person who did not write the dataset should write more.
• The corpus belongs to one tenant. The regulatory text sits in the vektos tenant. A chat for a client's tenant would need to search its own documents and the shared ones; that decision is made when the chat is built.
• The licences are summarised, not reviewed by a lawyer. W3C allows copying with attribution; EU legal texts may be reused with acknowledgement of the source. The repository is private.