Stage 18 instrument fix: runners load the LF-normalized corpus; D3/D4/D8

↗ view on GitHub · Eli Ziff · 2026-07-31 · 328eb32b

Corpus text is normalized (CRLF -> LF) at file read in legalbench-rag-run.ts,
which is now the one builder of the source db every LegalBench script reads
(`--build-only`). The normalized build is a SEPARATE file (a2aj-mini-lf.sqlite):
passage sidecars are keyed by source-db path + chunk params and NOT by source
content, so sharing one path would have silently reused CRLF-built indexes over
LF text. The corpus files on disk are untouched.

Every runner now points at it; the ablate sweep keeps a `--raw-coords` escape
onto the historical db so before/after is one code path over two corpora, and
tags arm labels/receipt filenames/rows with the coordinate space. Grounding rows
carry `coords: "lf"` and the resume key includes it, so a pre-fix (raw) receipt
file can never be resumed into an LF run.

D3: rerankPassages' `fallback` was dropped on the floor - now recorded per cell
(`rerank_fallback`) and printed as a rate; a fallback cell ran the lexical
order, not the reranker.
D4: the flat "overall" mean hides source-mix artifacts on partial runs - the
summary now prints the per-source cell mix (with an UNBALANCED warning) and a
source-balanced macro mean beside the flat one.
D8: doc-hit rates were printed on inconsistent denominators (grounded over
answered, baseline over all ok rows) - both denominators are now explicit and
the baseline doc rate is also printed on the answered subset.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents fab8e767
Stats 5 files changed , +184 , -44
Part of Legal grounding and retrieval research

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-328eb32b.md from inside the repo you want the change in.

⬇ Download capture-commit-328eb32b.md