Ground-truth retrieval benchmark

The production pipeline, measured on a public repository anyone can inspect. Every number below was produced on one workstation with local models, and no code or questions left it.

Method

Corpus: express v5.1.0 (commit cd7d439), indexed exactly like a customer repository.

Questions: 30 authored questions with known answers, each declaring the file a correct answer must cite and the facts the answer text must contain. They are the kind of questions a developer actually asks:

• "How does the trust proxy setting affect how the client IP address is determined?"
• "Where is the etag setting compiled into the function that generates ETags?"
• "What handles a request when no route matches it?"

Run: every question goes fresh through the full production pipeline (retrieval, context assembly, local model) with no cache and no cherry-picking. Reports are machine-generated by the same eval harness that ships in the product.

Results: production pipeline, 2026-06-18

qwen3-coder:30b + nomic-embed-text, one workstation, fully local.

That is the citation guarantee, measured: answers about your code, with file-and-line proof, from an engine that declines rather than guesses when retrieval comes up empty.

One stricter metric, for completeness: in 83% of questions the answer's citations included the exact file our ground truth named (up from 60% after a retrieval fix; see the changelog). The remainder answered correctly while citing related code, for example the place a function is used rather than the line it's defined on. We publish the strict number alongside the flattering one.

What actually moved the numbers

Credit belongs to the right change. The jump in exact file-hit came from a retrieval fix (folding exact-name matches into the hybrid ranking instead of letting a literal symbol short-circuit it), and it did not come from the cross-encoder reranker. Here is the isolated effect of each, on the same 30-question set:

The reranker's own contribution here is a single question (56.7% to 60.0%), which is within noise on a set this size, so we won't present it as a proven file-hit or precision win. We keep the cross-encoder on as a default anyway: it re-reads the top candidates against your actual question at sub-second cost, and on larger, noisier corpora a reranker typically earns more than it does on 30 clean questions. Its isolated benefit after the recall fix still needs re-measuring on a bigger question set, and we'll publish that number when a real run produces it.

Large-repo benchmark (in progress)

Thirty questions on Express is a clean, inspectable start, but it is small. A larger set is staged: 28 verified questions against kubernetes v1.36.2, a repository big enough to stress retrieval where it actually strains. The questions exist and are checked; the run does not yet. We won't print a large-repo accuracy number until it comes from a real local run. When it does, this section gets the figures and the method, exactly as above.

History at scale

One large-repo measurement already exists: git-history search. We indexed the complete kubernetes commit history (82,616 non-merge commits) and measured filtered history queries (by repository, author, and time window) on both supported backends, same machine, same embeddings, same query set.

On the default backend (ChromaDB), those queries took roughly 150 milliseconds at that scale, and the cost grew with corpus size. On the optional Qdrant backend they ran in under 2 milliseconds and stayed flat across every filter type — roughly seventy times faster. Absolute times are specific to our test workstation; the ratio and the shape of the curves are what transfer. This measurement is why the Qdrant option exists: it's the engine we recommend switching on when full git history takes a repository into tens of thousands of commits, and SourceVault downloads, verifies, and runs it for you when you select it in Settings.

Against cloud assistants

Cursor, Copilot, and Cody can't be driven headlessly, so a scored head-to-head has to be produced by hand. When we publish one it will include both sides' full transcripts, on this same question set. What can be compared today are the structural properties, the ones that don't depend on who runs the benchmark:

The second row is the one a cloud vendor can't match by shipping a feature: answering "why was this changed?" requires your commit history, and their indexers never see it.

Verify it on your own code

The benchmark harness ships inside every SourceVault install; the same machinery produced this page. Run it against your repositories during the free 7-day trial and get the same retrieval-quality report on your own code. If the answers don't cite your code with file-and-line proof, don't buy it.