F1: one explicit per-doc cap for every arm, recorded on the row
`legalbench-rag-grounding.ts` set `perDocCap: 24` ONLY when a rerank model was configured. Without one the lexical path fell through to searchPassages' default of 2, and an injected pool (--pool-jsonl) took `slice(0, k)` with no cap at all. The registered Tier C comparison runs plain lexical / ctx / ctx+rerank / injected fused pool, so three of the four arms would have run at three different caps, none of them declared. Now one flag, `--per-doc-cap`, applied to the lexical path and the injected-pool path alike (`capHitsPerDoc` in passageRetrieval applies the same rule searchPassages applies internally - unit-tested for equivalence), printed at startup and written to every row as `per_doc_cap` (null on the product path, which has no cap). SHIPPED DEFAULT 24, deliberately: - it is the crowned config's value everywhere else - the ablation sweeps, legalbench-rerank-eval, and legalbench-dense-dump, which BUILT the injected pool sidecars, all use 24; a2ajPassageSearch uses it too; - any cap >= k is equivalent to uncapped, so at the composed k (4 or 6) it is INERT: it changes nothing for the injected-pool arm, which was uncapped, and reproduces the uncapped behaviour for the lexical arms; - at the k=48 rerank pool, where a cap does bind, 24 is exactly the value the crowned rerank config was measured on. So the cap is now identical across arms while both registered behaviours are preserved in the place each was registered. Measured, deterministic, no receipt written (96 cells, t1600/o120/w16, LF instrument, lexical char recall): k=6 cap 2 R=0.2123 cap 24 R=0.4524 cap 48 R=0.4524 k=4 cap 2 R=0.2019 cap 24 R=0.3330 cap 48 R=0.3330 k=48 cap 2 R=0.2332 cap 24 R=0.8332 cap 48 R=0.8937 maud alone moves 0.0210 -> 0.1672 at k=6 (8x): exactly the "gold concentrates inside one 300 KB agreement" mechanism, and ~16x the +-0.015 composer band. This is the one fix in P0 that MOVES a number, and it moves it in the arms that were silently mis-capped. Honest disclosure: an injected pool drawn at k=48 for reranking is now capped at 24/doc where it was uncapped, so a fused+rerank arm is not byte-identical to Stage 18's; at k<=24 the injected path is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | 9d4eafe6 |
| Stats | 3 files changed , +115 , -4 |
| Part of | Evaluation harness: Beaver-CAN and LegalBench-RAG adapters |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-a40f46ee.md
from inside the repo you want the change in.