Retrieval (RAG)

WhatCopy Anchor Link

packages/rag finds the text that answers a question. There is no RAG framework: chunking, ingest, the three searches, the fusion, the reranking and the evals are written by hand, so each step can be read, tested and explained.

bun run db:up && bun run db:migrate
bun run rag:ingest                                    # docs/, the design system's MDX, the README, corpus/
infisical run -- bun run rag:embed                    # vectors for the chunks (about $0.001)
infisical run -- bun run rag:search "how do I restore a backup"
infisical run -- bun run rag:eval                     # all four searches, side by side

The searches, from simple to best (--mode):

The tables are rag_sources, rag_chunks (with a tsvector column the database computes) and rag_embeddings (one row per chunk and model, in a vector column with no fixed size, because models differ). Every row has a tenant.

WhyCopy Anchor Link

• The chunk carries its place. Its heading (Graders › Known gaps) says which guide and which part. The full-text search weights it above the body, and the embedding is made from heading plus text.
• Cut at headings, not at a length. A fence, a table or a paragraph is never cut. A long list is cut between items. A tiny section joins the one before it.
• Ingest is idempotent on the function too. A document is stored with the hash of its text and of how it is cut (CHUNKER_VERSION, sizes). An unchanged document is skipped; a better chunker re-ingests everything.
• One vector per chunk and model. Models give vectors of 1,024, 1,536 or 2,560 numbers, and a vector is only compared with its own model's. Keeping them apart made it cheap to compare four models on the same chunks.
• Both searches, then a reranker. Each search is good at something the other misses (see below). The reranker reads the question and each candidate together, which is slower and sharper, so it only sees the 30 candidates the hybrid found.
• Tenants are separate from the first table. Every query filters on tenant, and a test checks that one tenant never sees another's chunks, text or vectors.
• Models are arguments, and tests use a fake. Embedder and Reranker are functions. The real ones call the AI Gateway; fake.ts has a deterministic pair, so every test runs without network or money.

Try itCopy Anchor Link

bun test packages/rag            # chunker, fusion, rerank, evals, dataset check
bun run check:db                 # ingest, tenants, vectors and hybrid against a real database
infisical run -- bun run rag:eval
infisical run -- bun run rag:eval --kind keyword --show lexical
infisical run -- bun run rag:search "TS2769" --mode lexical

What the evals sayCopy Anchor Link

These numbers are the first measurement, on the project's own documents only: 48 cases over 339 chunks from 34 documents. Guide 16 adds two legal texts (772 chunks) and 24 more cases, and the pattern holds.

The cases are 40 questions written as a person would ask, and 8 keyword searches (TS2769, bench:recount) typed as they are. Embeddings are openai/text-embedding-3-small, the reranker voyage/rerank-2.5-lite.

Reading it (anecdote 10):

• Vectors win on sentences, words win on identifiers.
• The hybrid is worse than the better single search on both kinds. Its value is candidates (recall@10 of 100% on keywords), not order.
• The reranker is what puts the answer first, and it is best on both kinds.

Other things measured, all on the questions:

The rerankers are within noise of each other, so the cheapest is the default. With 40 questions a gap of two or three points is not a finding.

Cost. Embedding the corpus costs about $0.0013 with OpenAI small, $0.0014 with Voyage or Qwen and $0.0055 with Cohere (rag:embed prints it, from the gateway's price list). A search costs a fraction of a cent: one embedding of the question plus a rerank of 30 chunks.

Extend itCopy Anchor Link

• A new question. Add { id, question, path, contains, kind } to evals/vektos-docs.json. A test fails if no chunk of path contains the text, which also tells you when a document changed under a question. Write it the way a person would ask, not the way the heading is worded.
• Another embedding model. rag:embed --embedding-model <gateway id> adds a vector per chunk for that model; rag:eval --embedding-model <id> measures it. Add its price to EMBEDDING_USD_PER_TOKEN to see the cost.
• Another reranker. --rerank-model <gateway id>.
• Another corpus. Pass folders to collectFiles(base, roots) and a tenant to ingest, then embedMissing. The same functions serve a client's documents.
• A document whose sections are units to cite (a law, a standard): put min-words: 0 in its frontmatter, so a short section is not merged into the one before it. Cases of the dataset can then name the heading the answer must be under.
• A slice of the eval. rag:eval --kind question|keyword|violation and --scope docs|corpus measure one kind of case or one corpus.
• A change to the chunker. Raise CHUNKER_VERSION, or ingest keeps the old chunks. Embeddings are made for the new chunks only (rag:embed after rag:ingest).

Known gapsCopy Anchor Link

• 48 cases is small, and the 8 keyword searches are a hint. A difference of one or two points is noise. The numbers show the shape (vectors for sentences, words for identifiers, reranking over both), not a ranking to the second decimal.
• The dataset is written by the author of the docs. It tests questions the author can think of, not the ones a stranger asks.
• The corpus overlaps itself. The journal restates what the guides say, so a question can have two good answers and the dataset names one. Several "misses" return the right text from the journal; a few more return the right document but another section of it.
• No index on the vectors. Search compares the question with every vector of the model: 339 of them, instantly. At tens of thousands, an hnsw index per model is needed, and it needs a fixed vector size.
• The reranker's cost is not measured. The gateway charges a small amount per call; the CLI prints what embeddings cost but not what reranking did.
• The numbers on this page are on the docs alone. The corpus of guide 16 changes them a little; the table there is the current one.
• Re-embedding after a document changes is a separate command (rag:embed), not part of rag:ingest, because it costs money.