Retrieval gate: falsifiable vectors-only-if-earned benchmark
Makes the vector-boundary rule executable: a versioned retrieval set (48 items, 4 jurisdictions, gold locators + verbatim gold quotes), a frozen corpus slice (48 docs / 2,162 sections pinned to the corpus revision with parquet hashes), a deterministic scorer (Recall@5/10, top-1 locator accuracy, unsupported-claim rate over claims), and applyAsymmetricGate encoding the burden of proof: a candidate must gain >= 5 points absolute Recall@10 with no regression elsewhere; ties fail. Document identity is citationLookupKey, a Beaver-owned port of the reference implementation's corpus-index normalizer, proven equivalent by a differential test against an oracle dump (the probe imports the read-only reference only at test time). The differential caught a real divergence: Python casefold() folds sharp-s to "ss"; the port now does so explicitly. Baseline lexical-bm25-v1 measured: Recall@5 85.4%, Recall@10 89.6%, locator accuracy 81.3%, unsupported-claim rate 0.0% (0/43), ~1 ms mean per query. Numbers independently reproduced before commit. Set notes carry the honest limitation: heading-derived queries are lexically biased; adversarial set-v2 is required before the gate is decision-grade. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | 59cf82e0 |
| Stats | 11 files changed , +15252 |
| Part of | Deterministic legal-text tools ("SLA audit organs") |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-18cba572.md
from inside the repo you want the change in.