Retrieval gate: falsifiable vectors-only-if-earned benchmark

↗ view on GitHub · Eli Ziff · 2026-07-28 · 18cba572

Makes the vector-boundary rule executable: a versioned retrieval set
(48 items, 4 jurisdictions, gold locators + verbatim gold quotes), a
frozen corpus slice (48 docs / 2,162 sections pinned to the corpus
revision with parquet hashes), a deterministic scorer (Recall@5/10,
top-1 locator accuracy, unsupported-claim rate over claims), and
applyAsymmetricGate encoding the burden of proof: a candidate must gain
>= 5 points absolute Recall@10 with no regression elsewhere; ties fail.

Document identity is citationLookupKey, a Beaver-owned port of the
reference implementation's corpus-index normalizer, proven equivalent
by a differential test against an oracle dump (the probe imports the
read-only reference only at test time). The differential caught a real
divergence: Python casefold() folds sharp-s to "ss"; the port now does
so explicitly.

Baseline lexical-bm25-v1 measured: Recall@5 85.4%, Recall@10 89.6%,
locator accuracy 81.3%, unsupported-claim rate 0.0% (0/43), ~1 ms mean
per query. Numbers independently reproduced before commit. Set notes
carry the honest limitation: heading-derived queries are lexically
biased; adversarial set-v2 is required before the gate is
decision-grade.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents 59cf82e0
Stats 11 files changed , +15252
Part of Deterministic legal-text tools ("SLA audit organs")

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-18cba572.md from inside the repo you want the change in.

⬇ Download capture-commit-18cba572.md