Ground-truth retrieval benchmark

The production pipeline, measured on public repositories anyone can inspect. Every number below was produced on hardware we own with local models, and no code or questions left it. The full method, every table since the first run, and the noise-floor analysis live in docs/BENCHMARK.md, which ships with every install.

Method

Corpora: five public repositories pinned to a tag and indexed exactly like a customer repository — express v5.1.0 (JavaScript), zod v3.24.1 (TypeScript, one monolithic core file), flask 3.1.0 (Python), ripgrep 14.1.1 (Rust workspace), and kubernetes v1.36.2 (Go, ~29k files).

Questions: 472 authored questions (96 per corpus, 88 for kubernetes), each declaring the file a correct answer must cite and the facts the answer text must contain, drafted from the source with per-question evidence and adversarially adjudicated. They are the kind of questions a developer actually asks:

• "How does the trust proxy setting affect how the client IP address is determined?"
• "Where is the etag setting compiled into the function that generates ETags?"
• "What handles a request when no route matches it?"

Run: every question goes fresh through the full production pipeline (retrieval, context assembly, local model) with no cache and no cherry-picking, one corpus per clean install. Reports are machine-generated by the same eval harness that ships in the product, and a nightly gate replays the retrieval half against committed baselines so a regression cannot ship unnoticed.

Results: full sweep, 2026-08-20

qwen3.5:9b + nomic-embed-text, one RTX 5080 workstation, fully local. 472 questions, zero errors.

That is the citation guarantee, measured: 471 of 472 answers carry fully resolved file-and-line citations, from an engine that declines rather than guesses when retrieval comes up empty. The numbers are not perfect on purpose — the question sets were tripled and made adversarially harder in August 2026 precisely so a table like this reads as a measurement rather than a brochure. zod is the deliberately hard shape (most answers live inside one 5k-line file), and kubernetes is the scale test.

Earlier tables — the first 30-question express run from June (100% cited, on qwen3-coder:30b) and the August 13 sweeps on the smaller sets — are preserved in the repository's BENCHMARK.md, labelled by date and set size. They are not comparable row-by-row to the current sets, and we do not quote them.

What actually moved the numbers

Credit belongs to the right change, so every retrieval change ships only after a paired replay against the previous baseline on identical embeddings. The largest single gain was a retrieval fix (folding exact-name matches into the hybrid ranking instead of letting a literal symbol short-circuit it). The cross-encoder reranker's contribution was re-measured on the current sets on 2026-08-25 as the only variable:

Consistent across four languages, so the reranker stays on by default. Its cost used to be memory rather than time: scoring all candidates in one padded batch held the search process at ~2.5 GB. Scoring one pair per pass takes that to 343 MB and is faster (median 700 → 558 ms on express) with identical rankings.

Large-repo benchmark: kubernetes

Express is mid-size and recognizable, so kubernetes v1.36.2 is the credibility counterweight: ~5M lines of Go across 29k files, 88 authored questions spanning the scheduler, kubelet, kube-proxy, controllers, client-go, and the apiserver. It indexes to 135,488 chunks and scores 0.727 file-hit / 0.716 citation file-hit — mid-pack among the five corpora rather than the weakest, with latency flat at 17 s. The ranking holds up at scale rather than degrading with corpus size.

Monorepo scale on modest hardware

Our first-class hardware floor is a 16 GB machine with no GPU. To measure it honestly we built a fixture that freezes all five corpora as subdirectories of one repository — 20,616 files, 118,952 chunks, the single-partition worst case — and ran it on a 26 GB Apple M4 Pro as the closest available proxy for the 16 GB floor.

Small corpora pay a measurable price when a giant shares their partition (express reads 0.906 alone and 0.677 inside the monorepo); we publish that number rather than hide it, and the attribution tool that ships in the product names the pipeline stage that loses each question.

The same fixture settled the vector store. Code vectors now live in a local SQLite file (sqlite-vec, binary-quantized with an exact rescore) instead of a ChromaDB service, at measured parity: express, zod and flask reproduce ChromaDB's rankings with zero flipped questions, ripgrep and the monorepo move within noise. The bare vector lookup is slower (9.5 ms vs 4.5 ms at 119k chunks) — invisible inside the pipeline, and the trade buys a install with no vector database to run.

Generation is the honest limit on CPU-only machines: qwen3.5:4b decodes at 20–34 tokens/s and qwen3.5:9b at 11–13, but the first token arrives 5–10 seconds later per thousand tokens of retrieved context. That is why the product sizes its context window by RAM and defaults to the model that fits.

History at scale

Git-history search has its own scale measurement. We indexed the complete kubernetes commit history (82,616 non-merge commits) and measured filtered history queries (by repository, author, and time window) on the legacy ChromaDB backend and the optional Qdrant engine, same machine, same embeddings, same query set. The SQLite default has not been measured at this scale yet.

On the ChromaDB history backend, those queries took roughly 150 milliseconds at that scale, and the cost grew with corpus size. On the optional Qdrant backend they ran in under 2 milliseconds and stayed flat across every filter type — roughly seventy times faster. Absolute times are specific to our test workstation; the ratio and the shape of the curves are what transfer. This measurement is why the Qdrant option exists: it's the engine we recommend switching on when full git history takes a repository into tens of thousands of commits, and SourceVault downloads, verifies, and runs it for you when you select it in Settings.

Against cloud assistants

Cursor, Copilot, and Cody can't be driven headlessly, so a scored head-to-head has to be produced by hand. When we publish one it will include both sides' full transcripts, on this same question set. What can be compared today are the structural properties, the ones that don't depend on who runs the benchmark:

The second row is the one a cloud vendor can't match by shipping a feature: answering "why was this changed?" requires your commit history, and their indexers never see it.

Verify it on your own code

The benchmark harness ships inside every SourceVault install; the same machinery produced this page. Run it against your repositories during the free 7-day trial and get the same retrieval-quality report on your own code. If the answers don't cite your code with file-and-line proof, don't buy it.