eliziff turns legal AI retrieval into a reliability discipline

This fork is less interested in declaring a better answer engine than proving when its evidence can be trusted.

searchcompliance

eliziff has built a structured research programme around a simple legal-AI question: can the system support what it says with the right source, quote and context? The work tests several ways to improve retrieval, ranking and evidence checks, while recording failures as seriously as successes.

  • Claim checks to catch unsupported statements.
  • Quote-first answers that put the source material ahead of the summary.
  • Better document retrieval and ranking to surface more relevant evidence.
  • Context headers and evidence checks to preserve where a passage came from.

Most tellingly, the team found that an earlier scoring method was flawed, corrected it, and reran the work against a cleaner benchmark. That is a valuable instinct in legal technology: retire a promising approach if its measurement cannot withstand scrutiny.

So what Legal teams evaluating AI research tools should care because defensible answers depend as much on reliable testing and traceability as on fluent output.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

113 commits from eliziff/Beaver, oldest first. Source extracted verbatim from the harvested git log.

SHA Subject Author Date
b0338f65 legal grounding: experiment apparatus, stages 1-3 record, stage 4 pre-registration Eli Ziff 2026-07-30 ↗ GitHub
commit body
Commits the until-now-untracked grounding experiment: the provisional
legalEvidenceExperiment lib (evidence receipts, structured claims,
compose_check / evidence_first / holistic_check finalizers, fail-closed
rendering), its unit suite, the two runner scripts (held-constant
benchmark matrix; fixed verifier probe suite), and the experiment log
recording stages 1-3 (all falsified under the strict rule; no winner).

Stage 4 lands pre-registered under the REVISED decision rule
(direction 2026-07-30: reasonably grounded without unreasonable token
spend, instead of a strict win against the unverified control on every
gate - a gate no verification arm can pass by construction):

- new tiered_check mode: a deterministic verbatim-quote tier renders
  fully-quoted answers with ZERO checker calls (balanced citation-tail
  stripping; typography-only normalization - curly quotes, dash widths,
  NBSP, ellipsis, whitespace - never case or words; >= 25 chars;
  contiguous substring of ONE cited passage). Everything else escalates
  to Stage 3's single fail-closed holistic check, hardened with at most
  three attempts on transport-class failures only; semantic verdicts
  are never retried. Coverage on the deterministic path is honestly
  not_run in the receipt; composition is instructed to quote exactly
  where the passage answers.
- probe suite gains three deterministic-tier probes (pure quote,
  mutated quote, spliced prose) and per-probe modes; receipts record
  passed + per-claim deterministic_support (schema 4, optional field).

Frozen-hypothesis probe run (receipts stage4-verifier-probes.jsonl):
Claude 13/14 exact labels, 14/14 allow/deny (sole disagreement the
Stage-3 lost-attribution partially_supported-vs-unsupported, fail-closed
either way); pure-quote passed with 0 model calls, mutated and spliced
probes escalated and failed closed on BOTH providers; ALCE leave-one-out
necessity matched on both. Codex: 11/14 with three transport nulls on
the retry-less holistic-mode probes; its tiered-mode probes (retry
harness) all completed. The held-constant 3-arm matrix is running;
results land in the log by amendment, not by editing the frozen section.

Unit tests 12 passed; tsc clean. Experiment remains off by default.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
0b581df1 grounding runners: bounded concurrency with per-model lane caps Eli Ziff 2026-07-30 ↗ GitHub
commit body
Cells are independent (model, arm, case) triples; both runners now pump
them through a bounded pool (--concurrency, default 4) with a per-model
cap (--per-model-concurrency, default 2) so parallel Claude lanes buy
wall-clock while the overload-prone provider stays throttled and
concurrency cannot manufacture transport errors. Receipts append in
completion order; per-cell latency is measured inside the cell, and the
pool changes wall-clock only, never inputs. Stage 4 matrix relaunched
under 4/2 after 6 sequential control cells; noted for the log.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
413f7032 grounding: Stage 4 results - tiered_check clears every quality gate on Claude Eli Ziff 2026-07-30 ↗ GitHub
commit body
H4 falsified as frozen (codex tiered lost one answerable cell its
holistic lane kept; the zero-call deterministic path fired only in the
probe because composition never produced a fully-quoted answer), and
recorded plainly. But the Claude lane of tiered_check is the first
configuration in four stages to clear every revised-rule quality gate
in one run: both housing support gates (weak Alabama rejected,
sufficient row rendered), all answerable cells rendered, 14/14 probe
allow/deny, no deterministic-tier false pass anywhere (both cleared
claims verified verbatim), fail-closed throughout - at one checker call
per cell, +8% output tokens and ~2.4x cache-write vs control. The
same-run holistic arm flipped BOTH housing decisions wrong, the third
weak-reference false accept across runs: stochastic same-model judging
confirmed, and the tiered arm's quote-preferring composition is the
only upstream delta that fixed it. Codex completed 18/18 non-control
cells (prior overloads were provider weather) but stays disqualified.

Promotion: claude-p + tiered_check recommended experimental lane, flag
off by default. Next stage's independent variables named: firmer
quote-anchored composition so the zero-call path carries real answers;
authority/current-law validation still untested. Receipts archived
under OpenLegalData with hashes (stage4-verifier-probes, stage4-h4);
pre-registration is b0338f65.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
29bc7e6f grounding: Stage 5 pre-registration - scale 4x, cross the checker model Eli Ziff 2026-07-30 ↗ GitHub
commit body
Controls for the composing model being inherently more trustworthy at
legal: verification passes can now run on a different model than the
composer (finalize gains checkerModel; composition and the repair pass
stay on the composer), and the runner gains a --checker-models factor
(same | cross | explicit id) recorded per receipt, with control cells
running once. Matrix scales to 24 held-constant items: 12 CSLB (4
ordinary case, 4 ordinary legislation, 4 adversarial), 4 CLERC
continuations, 8 HousingQA rows (audited 163/0 sufficiency pair plus
six unaudited yes/no-balanced rows whose benchmark labels are NOT
treated as sufficiency gold). Frozen Hypothesis 5, predictions, and
falsification conditions recorded in the log before the run; unit
tests 12 passed, tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
de59524e grounding: Stage 5 results - the harness carries the decision, not the model Eli Ziff 2026-07-30 ↗ GitHub
commit body
144 cells (24 items x 2 composers x {control, tiered-same, tiered-cross}),
3 errors. H5 substantially supported; no falsification clause tripped.

The model-trust control answers cleanly: quote-anchored Claude
compositions were judged correctly on the audited sufficiency pair by
BOTH checker families (2/2 same-model, 2/2 codex-checked), Claude
checking rescued Codex compositions to 2/2, and the run''s only false
accept required Codex in BOTH roles (0/2 all-codex lane, which also
falsely rejected the sufficient row). Crossing the checker never
introduced a false accept; 11 of 12 same-vs-cross disagreements were
fail-closed and the twelfth fixed a false rejection. Codex control
target-F1 (0.40) beats Claude (0.29) - the crossing isolates
grounding-decision reliability from raw legal capability.

Deterministic tier at scale: 17 clears, all 17 re-audited genuine
verbatim under the lib''s real logic (the analysis script''s 2 crude-audit
alarms did not survive); the zero-checker-call path fired on three real
cells for the first time, all Codex compositions. Claude tiered''s four
non-audited rejections are all fail-closed partially_supported, two
repeating the audited weak-reference overreach shape on inspection -
recorded as fail-closed conservatism pending human audit (the one
conditional element of gate (a)).

Recommended experimental lane after five stages: quote-anchored tiered
composition with a Claude-family checker, composer either family; flag
stays off by default. Receipts archived with hash (stage5-h5.jsonl);
pre-registration 29bc7e6f.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
25c4b6b5 grounding: Stage 6 pre-registration - quote_first with deterministic contract enforcement Eli Ziff 2026-07-30 ↗ GitHub
commit body
New quote_first mode: verbatim quotation claims plus at most one
conclusion claim, enforced DETERMINISTICALLY at submission - the tool
rejects any answer with more than one non-verbatim claim (checked by
the same tier that renders quotes checker-free), naming offenders so
the model requotes within one extra iteration. Verifier machinery
unchanged from tiered_check; claims carry no citation text per the
production citation-placement wiring. Frozen H6, 41-item matrix (246
cells), predictions and falsification in the log before the run. Unit
tests 15 passed (contract rejection + acceptance pinned); tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
e8074475 docs: legal grounding research plan - framing lint + citation-web priors Eli Ziff 2026-07-30 ↗ GitHub
commit body
Owner doc for the post-Stage-6 program, grounded in measured evidence:
settled findings (harness > model family; quote anchoring; overreach as
THE failure shape; 0.69-AUC novel-content lint signal with source-class
conditioning requirement; citation-web probes) and five workstreams -
A: Stage 6 verdict; B: breadth-first misgrounding corpus (external
datasets + our receipts + court-database mischaracterizations);
C: deterministic legalClaimLint + Stage 7 lint-cascade ablation;
D: stands-for profiles from the citator + journals commentary graph
with hierarchy/temporal priors, Stage 8 ablations; E: small-model
flagger cascade last. Each stage pre-registers in the experiment log.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
a8192c34 grounding: Stage 6 results - the contract can quote; the cap cannot conclude Eli Ziff 2026-07-30 ↗ GitHub
commit body
H6 falsified on coverage gates while decisively proving its mechanism.
246 cells, 1 transport error. Deterministic-clear rates jumped 0% -> 57%
(Claude) / 62% (Codex) of claims; Codex produced the first three real
zero-checker-call cells; all 110 deterministic clears re-audit verbatim
(cumulative tier soundness 127/127 across stages 4-6, zero false
passes). The hard one-conclusion cap broke coverage: both lanes lost
the sufficient housing row (Codex conclusion wrongly refused beside a
correct quote; Claude drew a stochastic partially_supported on a
minimal quote+conclusion answer), and Claude could not express
adversarial premise corrections or CLERC continuations under the cap
(8 non-submissions). tiered_check under the Claude checker went 4/4 on
the audited pair at 4x Stage 5 n, reconfirming the recommended lane.
Refined conclusion: hard enforcement stays at the CLAIM level; Stage 7
ablates the soft contract + deterministic lint (H13 corpus-alienness
AUC 0.834, H14 prompt-gravitation, H7 features); the free conclusion
claim is the residual H12 attested characterizations target. Receipts
archived with hash; pre-registration 25c4b6b5.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
379cc12e grounding: legalClaimLint (H7/H10/H13/H14) + interim alienness index Eli Ziff 2026-07-30 ↗ GitHub
commit body
Deterministic claim lint, zero model calls, every feature a receipt
(feature, value, threshold, fired): QUIP-family novel-content fraction,
abstraction/absolutes/modality-upgrade lexicon deltas, entity-poverty
profile, prompt-gravitation (H14: prompt-only vocabulary share + the
prompt-x-alien co-occurrence), temporal inversion (H10: follow/apply
asserted against date order - hard flag), and corpus-alienness (H13,
probed AUC 0.834): trigram attestation spectrum against a reference
index of the local corpus. Uncalibrated features never fire; a missing
index reports null, never clean.

Index: scripts/build_alienness_index.py - stratified-sample sqlite of
FNV-1a-hashed trigram counts with a build manifest, explicitly marked
INTERIM per the standing base-repos directive (infini-gram vs Data
Portraits evaluation in flight); the TS reader isolates the store
behind corpusAlienness() so the backing can swap without touching
consumers. Cross-runtime hash parity pinned by test vectors computed
by the Python builder. 8 unit tests green (overreach shape flags,
faithful paraphrase clean, temporal hard-fire, uncalibrated-never-
fires); tsc clean.

Plan doc: footnote-pair prong added to stands-for profiles (journal
footnote citations as editor-verified proposition<->authority pairs,
using existing journals footnote blocks + engine note-crossrefs + ALR
pairing reference); US external complements recorded (CourtListener
Parentheticals, LePaRD, delta-Stance, courts-db); base-repos directive
recorded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
5f65432b grounding: adopt tokengrams as the H13 base; lint calibration harness Eli Ziff 2026-07-30 ↗ GitHub
commit body
Base-repo decision per the standing directive, scout-verified:
EleutherAI tokengrams (MIT, Windows-native cp311 wheel, exact
suffix-array counts, memory-mapped, proven at 16x our corpus) over
infini-gram (WSL-only here) and Data Portraits (unlicensed,
membership-only). Installed and smoke-tested (exact counts verified).
build_tokengrams_index.py: stratified sample -> GPT-2 u16 tokens with
EOS document separators -> MemmapIndex.build, with a manifest;
launched in background alongside the full citator build. The word-
trigram sqlite interim stays behind the corpusAlienness() seam until
Stage 7 calibration compares both signals.

calibrate-claim-lint.ts (workstream C2): runs the SHIPPED lint over
archived receipts against the installed index. Pipeline-grade numbers
on 557 claims: unattested_trigram_share AUC 0.705 (top feature; the
probe-grade 0.834 used a smaller cases-only reference - both recorded),
novel_content_fraction 0.666; operating point unattested > 0.70 flags
20.6% of rejected at 4.3% accepted (~5:1 enrichment). Labels remain
checker-derived, not gold.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
85b7601f grounding: char-stream index mode + three-way attestation calibration Eli Ziff 2026-07-30 ↗ GitHub
commit body
build_tokengrams_index.py gains --stream chars: character codes as u16
tokens (BMP passthrough, 0xFFFE overflow, 0xFFFF doc separator) - the
QUIP-canonical attestation signal on the same tokengrams base, no
tokenizer dependency, no BPE boundary artifacts. Meta records the
stream. calibrate_attestation_signals.py decides the H13 configuration
empirically: word-trigram (interim sqlite) vs GPT-2 token n-grams
(n=5,8; leading-space boundary mitigation) vs char n-grams (n=30,50)
over the same labeled claims, AUC table + threshold sweeps; decision
rule recorded (best AUC at matched operating points, tie-broken by
boundary robustness and base-repo preference). Chars build + chained
calibration launched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
140f0bd2 grounding: C2 attestation configuration decided - word-trigrams win Eli Ziff 2026-07-30 ↗ GitHub
commit body
Three-way calibration over 553 labeled claims under the pre-stated
rule: word-trigram unattested share (AUC 0.694) beats every tokengrams
stream configuration at the current reference scale - char n=12..24
plateau at 0.677-0.640, GPT-2 token n=3..8 at 0.646-0.588, and the
QUIP-canonical char n=30/50 SATURATE (~100% unattested for accepted
and rejected alike): our 230M-char reference sits ~4 orders below the
web-scale corpora that configuration presumes, so contrast lives at
coarse word granularity. Engine (tokengrams) settled regardless; queued
follow-ups: u32 word-id stream so the winning signal serves from the
base engine (retiring the hand-rolled sqlite), and a full-corpus
reference rebuild + re-test, since reference size binds the finer
signals. Breadth-corpus re-test still gates production thresholds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
1d322317 grounding: misgrounding corpus v1 builder (workstream B2) - 6,283 rows Eli Ziff 2026-07-30 ↗ GitHub
commit body
Unifies four provenance classes without label promotion:
expert_annotated (RegLab legal_rag_hallucinations, CC-BY: 176 grounded
/ 14 misgrounded / 18 ungrounded full responses), court_described
(Charlotin AI Hallucination Cases: 2,964 fabricated, 1,426
misrepresented, 875 false quotes, 34 outdated - text flagged as court/
curator PARAPHRASE, never model text), checker_derived (our receipts:
494 grounded / 63 misgrounded claims WITH exact spans), and
benchmark_adversarial (100 CSLB false-premise rows). Breadth achieved:
42 jurisdictions (USA/Canada/Australia/UK+), 9 source classes (case
law, legislation, doctrinal work...), 91 model families, 29 areas of
law. Rows live outside git under OpenLegalData
(misgrounding-corpus/corpus-v1.jsonl, sha256 a86680...23ba); the repo
keeps the builder and aggregates. This corpus gates lint thresholds
(C2) and supplies the Stage 8 adversarial probes (the contract should
be unable to express the misrepresented items).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
cc5c13b7 grounding: H13 external validation - negative result, scope narrowed Eli Ziff 2026-07-30 ↗ GitHub
commit body
RegLab expert-labeled slice (174 grounded vs 32 misgrounded/ungrounded
real tool responses): word-trigram alienness INVERTED (AUC 0.372),
prompt-share chance (0.550). Diagnosis recorded before any threshold
ships: reference mismatch (grounded US responses quote US statutes -
maximally alien to the Canadian reference; jurisdiction-matched
references required), unit mismatch (claim-calibrated signal scored on
full responses; segment first), and a scope boundary - RegLab
misgrounding is largely RELATIONAL (real citation, plausible register,
wrong support), invisible to lexical lint and precisely the H12
stands-for machinery target. H13 stands as claim-level,
same-jurisdiction composed-overreach screening only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
2919b8c1 grounding: US lane - RegLab source retrieval + claim segmentation Eli Ziff 2026-07-30 ↗ GitHub
commit body
Sets up RegLab's expert labels properly: eyecite extraction (442
distinct case citations across 208 labeled responses), CourtListener
citation-lookup + opinion cache, and claim-level segmentation via
full_span citation masking (1,704 claims, 766 with attached
citations). Records the US-materials directive in the research plan.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
880d4fdb grounding: local CL bulk as tier-1 citation resolution Eli Ziff 2026-07-30 ↗ GitHub
commit body
The 5.5 GB local CourtListener bulk sqlite resolves 346/442 RegLab
citations offline (vs CAP's 289), including LEXIS and post-2020 cites;
its opinion table is a head sample only, so CAP static remains the
text tier. Either-tier resolution: 349/442 (79%).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
43b68009 grounding: widened deterministic tier (H12) + US reference builder Eli Ziff 2026-07-30 ↗ GitHub
commit body
attestedCharacterizationReceipt: citator-provider evidence whose span
is another court's classified prose about the cited case - claims
clear the verbatim tier only by quoting the citing court's own words,
with attribution recorded in the receipt (never parsed from prose).
Tests: verbatim clears; one-word citance mutation fails; splice across
characterization+passage evidence fails. Plus build_us_reference.py
(stratified CAP static sample) and --jsonl/--suffix on the alienness
index builder for the jurisdiction-matched US index.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
23f33745 grounding: US reference fetch switched to CAP bulk volume zips Eli Ziff 2026-07-30 ↗ GitHub
commit body
Bulk-first: one <slug>/<vol>.zip request per volume (~108 total)
replaces ~6k per-case JSON calls; zips cache on disk so re-runs and
future scale-ups are offline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
256e5382 H13 gate closed: US-reference index recovers alienness on expert labels Eli Ziff 2026-07-30 ↗ GitHub
commit body
Point the source-anchored scorer at a jurisdiction-matched index via
ALIENNESS_INDEX_PATH; with the CAP-bulk US reference (17,547 docs /
229M chars, seed 47) unattested_trigram_share max-pool AUC vs
misgrounded rises 0.689 -> 0.781. Index-free features unchanged,
isolating the reference as the moving part. n=8 positives gate stands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
64ac9abd standsForProfile: journal commentary joins as an attested source Eli Ziff 2026-07-30 ↗ GitHub
commit body
Commentary candidates from journal_commentary.sqlite (rank-1 citations
of paired notes; the author's proposition sentence is the attested
characterization). The excerpt classifier gates propositions but never
rewrites them - they are sentence-exact and hash verbatim. Court prose
outranks commentary; null when no commentary DB is installed, so
"absent" never reads as "empty". Fixture-backed tests cover rejection,
rank-1-only attribution, and ordering.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
9442ad3f Stage 8 coverage gate measured; plan doc records footnote-pair prong Eli Ziff 2026-07-30 ↗ GitHub
commit body
CSLB case cells: 4 rich / 1 thin / 5 none profiles (small-claims rows
uncited - the few-citers prior computable); CLERC is US and profiles
none until the CourtListener Parentheticals lane; 121 Canadian
Charlotin Misrepresented rows available as live-citator probes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
f9f71f3f attested_framing arm: Stage 8 stands-for gate + feed-forward Eli Ziff 2026-07-30 ↗ GitHub
commit body
New experiment mode: quote_first contract plus a deterministic
stands-for gate - claims with characterization language citing case
evidence must clear the widened verbatim tier (cited passage or an
attested characterization named in evidence_ids); the one-paraphrase
conclusion allowance never covers them. Runner feeds forward ranked
standsForProfile candidates (court prose + journal commentary) as
citator receipts; commentary receipts label the journal, not a court.
Statute cells are a no-op by construction (gate requires case-class
evidence). 23/23 lib tests incl. composed-framing rejection, verbatim
attested acceptance, HousingQA no-op vector.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
d6c29aaf Stage 8 runner: neutral-citation lookup + attested coverage audit Eli Ziff 2026-07-30 ↗ GitHub
commit body
The attested_framing feed-forward passed full benchmark citation
strings (style-of-cause + pinpoint) to standsForProfile, whose lookup
keys whole strings by contract - every CSLB cell silently degenerated
to a none-profile. Extract neutral citations with the citator
builder's own NEUTRAL_RE first (2007 BCCA 40 -> 3 candidates,
2012 SCC 57 -> 8, both verified live), record per-citation candidate
counts in a new attested_characterizations receipt field, and add a
--cases id filter for targeted cells (Charlotin probes will need it).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
95071ce7 Stage 7 lint_gated arm: frozen soft gate + one-bounce revision Eli Ziff 2026-07-30 ↗ GitHub
commit body
Freeze the pre-registered operating points on RegLab expert labels
(scripts/freeze-stage7-thresholds.ts, zero grounded max-pooled
responses flagged at >=12 lint content words: novel_content_fraction
0.666667, unattested_trigram_share 0.823529, prompt_only_share
0.333333) and ship them as STAGE7_LINT_THRESHOLDS. The lint gains the
min-content applicability gate (the saturating grounded claims were
citation fragments) and an exported contentWordCount so calibration
gates on the lint's own definition. lint_gated mode: verbatim-tier
claims skip lint; a flagged composed claim costs exactly one typed
revision bounce naming feature values; after the bounce fired receipts
ride into the holistic checker prompt and the receipt event. Runner
wires the question and jurisdiction-matched alienness index per cell.
27/27 experiment vectors, 8/8 lint vectors, typecheck clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
3df9c89d Experiment log: Stage 7 threshold-freeze record (pre-run) Eli Ziff 2026-07-30 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
90860bd0 docs: Stage 8 results by amendment (suppression confirmed, affordance declined) Eli Ziff 2026-07-30 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
c320dd13 docs: pre-register evidence-layer ensemble & redundancy protocol (C4) Eli Ziff 2026-07-30 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
3de157c5 docs: pre-register Stage 8b (H15 slot contract, H16 fallback rung, H17 prompt factorization) + H18 transparent-witness direction Eli Ziff 2026-07-30 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
2cb81b19 docs: Stage 8b amendments - H16 premise corrected pre-run, model-diversity axis added Eli Ziff 2026-07-30 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
3e3d134c docs: Stage 7 verdict (economic claim falsified, safety held), typed claim roles, diversity roster amendments Eli Ziff 2026-07-30 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
0c2e4adf Stage 8b contract: typed claim roles, required slot, diff-carrying bounces Eli Ziff 2026-07-30 ↗ GitHub
commit body
Typed claim roles (schema-shared, all structured arms): claims gain
kind = quotation | conclusion | premise_correction; premise corrections
carry premise_source (question | prior_answer) + premise_text, enforced
as a verbatim substring of the named source - the schema-level premise
distinction. Verified corrections are exempt from the conclusion
allowance and the lint bounce (novel by design), but never from the
stands-for bar (no laundering).

H15 required_slot arm: every cited case's characterization slot must be
filled by an attested-verbatim quote (citator receipt) or the exact
typed refusal sentence; free paraphrase is not a slot value.

H16' diff-carrying rejections: quoteRepair.ts ports the ALR-Quote-
Verifier alignment approach (token-level longest common run) narrowed to
Beaver's contiguous-substring tier - failed quotations bounce WITH the
cited span's closest verbatim excerpt, which clears the tier by
construction when requoted (round-trip proven in tests).

H17 prompt factorization: one base composition prompt shared verbatim by
every structured arm; mechanisms ride as named modules injected only on
applicable cells (attested/slot text only on case-law cells), recorded
per-cell as prompt_modules (runner receipt schema v2).

Instrumentation: every typed rejection archives pre-bounce claims +
rejection text into the receipt event (schema v6), closing the Stage 7
gap. 13 new unit vectors; 40/40 pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
8ee39ef3 docs: Stage 8b H16' diff-carrying upgrade (ALR port) + launch record Eli Ziff 2026-07-30 ↗ GitHub
commit body
Registered pre-run: bounces now hand the repair (cited span's closest
contiguous excerpt, ALR-Quote-Verifier machinery ported per Eli's
directive), with the excerpt-parroting falsifier; implementation commit
0c2e4adf; smoke evidence for the 180s timeout; 656-cell factored run
launched on the 41-item matrix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
c9d7d60e docs: Stage 8b verdicts - H15 slot binds but costs coverage, H16' repair supported, H17 confirmed Eli Ziff 2026-07-30 ↗ GitHub
commit body
Analysis over 859-row receipts (haiku lane stopped early per Eli; its
latency/transport-attrition profile registered as the small-tier result).
Falsifiers audited: no degenerate slot-filling, no excerpt-parroting;
coverage-collapse and below-tiered-pass-rate falsifiers DID fire on H15.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
db011814 docs: Stage 9 pre-registration - rank-policy ablation (H19), cheap selection (H20), temporal flag, alienness advisory Eli Ziff 2026-07-30 ↗ GitHub
commit body
Frozen before implementation. Per Eli: courts-over-journals ranking is
an assumption to test, not assume; three policies (authority / banded
recency with commentary in the top band / flat recency).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
a5968bb9 Stage 9: rank-policy ablation (H19), top-3 offer + predeclare (H20), temporal flag, alienness advisory Eli Ziff 2026-07-30 ↗ GitHub
commit body
standsForProfile gains authority/banded_recency/flat_recency ordering
(same candidate set, same gates - only rank moves); runner crosses
policies over case cells of attested arms with per-cell receipts
(rank_policy, attested_offer, quoted_attested_rank,
conclusion_alienness, schema v3) and resume keys extended with policy.
temporalOrderInversion rejects active-voice impossible orders from
neutral-citation years; bounce-time alienness advisory names
corpus-alien phrasing on rejections that already happened (advisory
only, per registration). 12 new unit vectors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
9bdcc352 docs: Stage 9 amendment - flat_recency only live; A/B policies postponed, not retired Eli Ziff 2026-07-30 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
5164aded docs: Stage 9 verdicts - refusal escape hatch dead (15/15 -> 0), no-sub half-fixed, ranks 1-3 all used, temporal flag clean Eli Ziff 2026-07-30 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
8d7434c7 docs: Stage 10 pre-registration - H18 witness panel + persistent transport adoption Eli Ziff 2026-07-30 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
ab22fc81 Stage 10: witness_panel arm - required_slot contract + pre-composition facts panel (H18) Eli Ziff 2026-07-30 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
faabcc04 docs: Stage 10 verdict - H18 witness panel falsified; persistence is a latency lever, not reliability Eli Ziff 2026-07-30 ↗ GitHub
commit body
P1 falsified (no first-sub acceptance gain any model); P2 partial (no-sub
direction holds, mini misses <=5); P3 perfect (zero parroting, accepted or
attempted); P4 controls within floor; P5 split (claude median 29.5s->20.2s,
errors 4->4 + 8 new; content-shaped JSON failures recur on identical cells
across arms, plus 90s-cap stream aborts under sustained persist load).
Amendment: Stage 9 escape-hatch zero was behavioral, not gate-enforced;
2/246 recurrence recorded, hardening candidate noted. Receipts hashed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
3fc6dcd2 docs: roster amendment (luna-medium baseline, mini retired), H19 banked-slice null, Stage 11 registration Eli Ziff 2026-07-30 ↗ GitHub
commit body
H19 three-way on the 22 banked sol/attested_framing cells vs same-cell
flat_recency rows: no policy effect beyond the checker noise floor.
Stage 11 pre-registers luna-medium on the 41-item matrix, required_slot,
A/B/C policies on case cells (77 cells) - H19 where the slot binds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
4b43e338 docs: Stage 12 registration (sonnet-5 full matrix + opus-5 case cells); restore Durable receipts heading Eli Ziff 2026-07-30 ↗ GitHub
commit body
Capacity-vs-harness discriminator on the claude lane: model upgrade is
the only change vs the Stage 10 sonnet-4-6 required_slot lane, except a
labeled 90s->180s timeout fix. Also restores the Durable receipts
heading dropped by the Stage 11 insertion edit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
31e1fed0 docs: Stage 11 verdict - first H19 policy effect (flat_recency costs coverage under the slot); luna-medium baseline bails more than sol Eli Ziff 2026-07-30 ↗ GitHub
commit body
P1 fires: 7/18 vs 3/18 no-sub separates flat_recency from authority and
banded beyond the 13% gate; authority vs banded indistinguishable, so
the banded proposal survives at no measured cost. P2 falsified: the new
luna-medium baseline no-subs more than sol-low. Zero errors, zero false
passes, zero refusal-despite-candidates. Receipts hashed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
27401f0b docs: Stage 12 cut verdict (Claude 5 breaks claude-p JSON protocol, both lanes transport-falsified) + Stage 13 registration (codex tier/effort ladder) Eli Ziff 2026-07-30 ↗ GitHub
commit body
Stage 13 per Eli corrected directive: luna max + terra max + sol medium
on the full matrix, sol max on the 18 case cells; authority ordering
held constant per the Stage 11 recommendation. P1 discriminates
capacity-bound vs contract-bound residual no-submission via within-model
effort ladders.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
151a68e7 scripts: cache-aware token telemetry per model/effort over grounding receipts Eli Ziff 2026-07-30 ↗ GitHub
commit body
Sweeps all receipt JSONLs, dedupes cells, reports uncached input, cache
read/write, output, reasoning, and per-cell means. Codex lane carries
cached_tokens + reasoning_tokens from the Responses API; claude-p lane
carries cache_read/write_input_tokens (its inputTokens is uncached-only,
so effective input = uncached + cache read).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
cdb7afca Stage 12b amendment + Stage 14 registration; LegalBench-RAG grounding adapter Eli Ziff 2026-07-30 ↗ GitHub
commit body
Stage 12b: transport probe reran the 14 failed Claude-5 cells over the
hardened claude-p transport - attrition 14/14 to 0/14, repair layer
absorbed every failure, Claude-5 lanes unblocked.

Stage 14: LegalBench-RAG-mini over the grounding harness, pre-registered
with frozen P1-P4 before any scored run. Adapter pipelines product bm25
retrieval into the required_slot contract and scores located verbatim
quote spans against human gold char spans deterministically, paired
per-test against the raw-retrieval baseline, with typed
answered/declined/rejected/abstained outcomes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
880287eb Stage 13 + Stage 14 pilot results; harden LegalBench quote locator Eli Ziff 2026-07-30 ↗ GitHub
commit body
Stage 13 ladder verdict: sol no-sub falls monotonically with effort
(7/18 to 2/18) but effort buys typed honesty, not passes; luna
saturates at medium; every residual bail in every strong lane is a
CLERC continuation cell. sol@medium crowned; max effort retired for
grounding runs. Attrition 1/141.

Stage 14 pilot (32 real-contract tests, luna@medium): answered-only
char precision 0.44-0.66 vs 0.0077 retrieval baseline, zero
wrong-document answers, all 30 retrieval-failed cells declined or
abstained typed. P1 alarm audited: gate-accepted quote wrapped in
curly quote glyphs defeated the exact-indexOf locator; locator now
strips wrapping glyphs and falls back to a whitespace/quote-tolerant
regex with exact offsets. P4 volume prediction falsified (quotes
cover the gold span; that is the point).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
83dcc467 Stage 15 registration: retrieval overhaul + passage-retriever wiring Eli Ziff 2026-07-30 ↗ GitHub
commit body
Adapter takes --retriever passage with chunk/weight config recorded per
row and in the resume key. Offline ablation over 776 tests picked
t1600/o120/w16: doc recall 0.50 to 0.99, char recall 0.021 to 0.29 vs
the doc-level baseline. Confirm run pre-registered: sol@medium on the
Stage 14 cells, frozen invariants that precision survives coverage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
128e4c0f Stage 15 confirm verdict: 7x coverage, invariants held; pool-recall probe Eli Ziff 2026-07-30 ↗ GitHub
commit body
Answered 21/32 (pilot 3/32), unlocated 0, wrong-doc 0, answered-only
precision 0.44 (57x original baseline). Pool recall 0.52@12 / 0.68@24 /
0.85@48: gold is in the lexical pool below the cut, so the flat-rate
model lever is deep-pool reranking, not query expansion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
d968cacb Stage 16 registration + whole-pool LLM reranker (W2) with gold eval Eli Ziff 2026-07-30 ↗ GitHub
commit body
Structure-native retrieval program registered (W1 skeleton chunking,
W2 whole-pool listwise reranker, W3 citation short-circuit, W4
proposition surrogates, W5 dense gate). W2 shipped: one flat-rate
listwise call ranks the k=48 lexical pool to top-4; the model returns
indices only so evidence text is untouchable; parse failure falls
back to lexical order, typed and counted. Smoke (8 tests): R@4 0.42
to 0.66 against a 0.80 pool bound, zero fallbacks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
e8b9a67a Stage 16 W2 verdict (all predictions held) + rerank wiring + confirm registration Eli Ziff 2026-07-30 ↗ GitHub
commit body
Full 776-test eval: reranked R@4 0.6697 vs paired lexical 0.4012,
P@4 doubled, doc@4 0.9987, fallbacks 0/776 - 79% of the pool bound
captured at 5.1s/call. W5 dense-lane gate stays closed (0.67 >= 0.5).
Adapter takes --rerank <model> (k=48 pool, listwise to top-k, recorded
in the retriever string and resume key). Stage 16 confirm run
pre-registered.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
049c1226 Stage 16 confirm verdict: answered 26/32, precision rose with coverage Eli Ziff 2026-07-30 ↗ GitHub
commit body
All five frozen predictions held: answered 26/32 (pilot 3/32, Stage 15
21/32), answered-only precision 0.58 (above Stage 15 0.44), zero
abstentions, zero wrong-doc answers, zero unlocated quotes, zero
errors. Session arc on the same 32 real-contract cells: retrieval was
the bottleneck; passage indexing then listwise reranking removed most
of it with every soundness invariant intact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
799cba54 Stage 17 registration: full 776-test grounding verdict, crowned stack Eli Ziff 2026-07-30 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
3393a876 Reranker default preview 500 -> 1600 (full passage), measured +0.20 R@4 Eli Ziff 2026-07-30 ↗ GitHub
commit body
Paired A/B on 120 gold tests: R@4 0.562 -> 0.762, P@4 0.090 -> 0.111,
0 fallbacks, latency down. Ranking on the full passage instead of its
first 31% captures ~90% of the pool bound. Receipts:
stage16b-preview1600.jsonl.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
337b19f4 docs: Stage 16b reranker preview ablation (0.5624 -> 0.7615 paired) + receipt hash Eli Ziff 2026-07-30 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
25ef6ec5 Stage 17 verdict: crowned stack holds on full LegalBench-RAG mini (776 cells) Eli Ziff 2026-07-30 ↗ GitHub
commit body
651 answered (83.9%), P=0.505 answered-only grounded char precision vs
0.087 baseline at 1/7th the quoted chars; 1,358/1,358 quoted spans
verbatim-audited, zero false passes; doc-miss audit resolves 6/7 to a
byte-identical CUAD duplicate file, true rate 0.15%. Runner: fix
usage-undefined crash in the finalizer merge; add --rerank-preview so a
resume never mixes rerank configs in one receipt file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
8abce3be Stage 18 registration: maud pool-recall arms + grounded composition Eli Ziff 2026-07-30 ↗ GitHub
commit body
Published-score calibration section (arXiv:2408.10343 baselines; maud
P@1 2.65%, R@64 28%) anchors what these char metrics can reach. Frozen
predictions: R1 clause-mode (maud pool R@48 0.35->0.42+), R2 phrase
bigrams, rerank confirm (maud rerank R@4 0.23->0.30+), G full-bed
grounded confirm (overall P 0.505->0.55+, maud 0.14->0.25+, strict
no-regression falsifier), D dense lane registered-deferred (no
embedding backend reachable today).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
1b002e3b Stage 18 R2: phrase-bigram FTS terms + four-arm retrieval ablation Eli Ziff 2026-07-30 ↗ GitHub
commit body
passageQueryPhrases pairs adjacent content words from the ORIGINAL
query (pairing the stopword-filtered stream would fabricate phrases);
searchPassages gains phrases option, additive OR phrase terms. Ablate
script gains --stage18: {chars,clause} x {plain,phrases} at
t1600/o120/w16 over all 776 gold tests, per-source lexR4 + poolR48,
JSONL receipts. 16/16 unit tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
71aea944 Stage 18: R1/R2 falsified by frozen gates; fair-comparison protocol; G wiring Eli Ziff 2026-07-30 ↗ GitHub
commit body
Clause mode moved maud pool +0.023 (needed +0.05) and cost contractnli
-0.072; phrase bigrams strongly negative (lexR4 0.406->0.249) - both
recorded falsified. Fair500 chunk-matched row shows raw BM25 at/below
the paper's embedding baselines: the stack's edge is pool+rerank+
contract, not the retriever. Locked three-mode scoring protocol
(answered-only 0.505 / forced 0.4375 / zero-credit 0.4236) for all
future headline reports incl. the one-shot Stage 19 holdout. Runner
gains --stitch (post-rerank same-doc span merge, baseline unstitched).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
d142c1b0 Stage 18 R4 registration: rerank compute ablation + effort plumbing Eli Ziff 2026-07-30 ↗ GitHub
commit body
Plumb reasoningEffort through completeText -> codex rerank path;
--rerank-effort runner flag recorded in the retriever id; --rerank-arms
mode in the ablation harness (5 arms, resumable, fallback-counted).
Registered with frozen predictions before any cell ran.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
47ab92ab Stage 18 R5 registration: contextual enrichment lane (headers-only, hash-keyed sidecar) Eli Ziff 2026-07-30 ↗ GitHub
commit body
Optional contextJsonl on the passage index: LLM situating headers in
the FTS context column of a content-hash-keyed sidecar variant; plain
path byte-unchanged (unit vector). Generator script (resumable, flat
rate). Registered with frozen predictions before any header ran.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
07213e7d Stage 18 G verdict: not falsified, new champion; P1 clean under locator-mirrored audit Eli Ziff 2026-07-30 ↗ GitHub
commit body
681/776 answered (87.8%), answered-only P 0.5377 / R 0.5708 vs S17
0.5050/0.5114. Two predictions missed honestly (P<0.55, maud<0.25);
falsifiers all clear. Audit extended to whitespace-class equivalence
(mirrors locateQuote's documented tolerance; 20/20 artifacts resolve).
Receipts hashed; holdout stays unburned pending R4/R5/dense.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
b78a0296 Stage 18 R4 verdict: rerank seat frozen at luna@default; effort deviation breaks format compliance Eli Ziff 2026-07-30 ↗ GitHub
commit body
No displacement (sol -0.007, terra -0.023). Prediction 1 falsified
instructively: low/high effort cost 21%/29% fallback vs 0 at default.
All arms hit maud's ~0.29 pool ceiling - maud is a pool problem.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
a24cde12 Stage 18 R5 verdict: middle band (+0.040 maud pool, bar +0.05); lint no-op; R5b registered Eli Ziff 2026-07-30 ↗ GitHub
commit body
Headers lift maud pool R@48 0.3619 -> 0.4019 (raw w4) - clears the
falsifier (+0.03) but not the adoption bar (+0.05). Lint arm within
±0.01 of raw everywhere: measured no-op, kept as optional guard.
Off-target: cuad pool +0.0677, overall lexical R@4 +0.13 (recorded,
not adopted). R5b (luna@medium on maud only) registered with frozen
gates before generation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
30bf6dd8 Stage 18 R5b verdict: ADOPT medium-effort maud headers (pool 0.4310, w4) Eli Ziff 2026-07-30 ↗ GitHub
commit body
Frozen adopt bar (maud pool R@48 >= 0.4119 with contractnli guard) met
at every weight; best w4 0.4310 vs champion 0.3619. Retirement not
triggered. Frozen prediction (< +0.03 over low best) hit narrowly but
compounding cleared the bar the low tier missed. Adopted delta:
combined-medium context sidecar at weight 4, riding one grounded
confirm before Stage 19 config freeze. Receipt hashes recorded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
c4c53725 Stage 18 R5b confirm registration: context sidecar flags in grounded runner Eli Ziff 2026-07-30 ↗ GitHub
commit body
--context-jsonl/--context-weight plumb the adopted combined-medium
sidecar into searchPassages; retriever label carries +ctx(w4@sha12) so
receipt files cannot mix header versions. Frozen KEEP/DROP gates
registered before the confirm run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
34a429de Stage 18 arm D registration: dense lane run (bge-base, 6 pool arms) Eli Ziff 2026-07-31 ↗ GitHub
commit body
Fully local fastembed eval over the crowned index: dense-only and
RRF-fused pools vs the plain and R5b-adopted header pools, frozen
OPEN/CLOSED gates (maud fused >= 0.50 opens; <= ctx+0.01 retires).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
c2a99289 Stage 18 R5b grounded confirm: KEEP; config frozen as G+ctx Eli Ziff 2026-07-31 ↗ GitHub
commit body
776/776 cells, 0 errors, P1 clean (1,564 spans, 0 mismatches, 0 true
doc-miss). All frozen KEEP gates pass: maud answered-only R 0.1385
(bar 0.1170), overall P 0.5535 / R 0.5832 (bars 0.5277/0.5608). All
four fair-comparison modes improved over G. Retrieval config frozen
as G+ctx(w4@36d041f7d66b); only the dense lane (in flight) remains
before the Stage 19 holdout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
68d0013d Stage 18 arm D amendment: dense backend to 3080 Ti, Qwen3-Embedding-4B Eli Ziff 2026-07-31 ↗ GitHub
commit body
CPU bge-base run killed twice for machine load; per Eli, the dense
eval moves to the desktop GPU over Tailscale with the strongest
embedder that fits 12 GB. Gates, arms, and pool machinery unchanged;
no results were observed under the old backend.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
b8502f63 Stage 18 arm D verdict: OPEN - fused dense+ctx pool clears every gate Eli Ziff 2026-07-31 ↗ GitHub
commit body
Qwen3-Embedding-4B on the 3080 Ti: fused maud pool R@48 0.6351 vs
the 0.50 OPEN bar; every source improves over the ctx pool; ALL
0.8189 -> 0.9015. Both held-over bge-base prediction bands overshot
in the flagged model-upgrade direction. Hybrid must ride its own
registered grounded confirm before joining the frozen config.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
37cb725d Stage 18 arm D confirm registration + --pool-jsonl runner injection Eli Ziff 2026-07-31 ↗ GitHub
commit body
Registers the fused-hybrid grounded confirm (frozen KEEP/DROP gates
vs the R5b ctx confirm baselines) and adds precomputed-pool
injection to the grounding runner: per-test spans from a hashed
sidecar replace the lexical pool stage; rerank/stitch/contract
unchanged. Eli's laptop-inference constraint recorded: this lane is
the quality ceiling, not a production dependency.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
89a346b0 Stage 18 fused confirm: sidecar landed, sanity 0.6351 reproduced, sha pinned Eli Ziff 2026-07-31 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
22e24805 Stage 18 fused confirm verdict: MIDDLE -> default DROP; G+ctx stays frozen Eli Ziff 2026-07-31 ↗ GitHub
commit body
maud ansR 0.1449 (ctx 0.1385) missed the +0.02 KEEP bar; overall
P/R gates passed (0.5525/0.6002). P1 clean. Pool lift dies at
rerank/composition, not retrieval. Receipts hashed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
9718b94b Stage 18 C1 registration: coverage + spec composition arms, runner flags Eli Ziff 2026-07-31 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
fa59e29b Stage 18 F registration: name-strip audit, negative-control, plain-prompt arms Eli Ziff 2026-07-31 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
bc705f07 Stage 18 audit results: strict-answered, byte-exact, R1 falsification overturned Eli Ziff 2026-07-31 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
edabce75 Stage 18 registration-discipline audit: R5b/R1/gate findings + standing corrections Eli Ziff 2026-07-31 ↗ GitHub
commit body
Independent adversarial audit of Stages 14-18 (git forensics + full
receipt recomputation). Data integrity clean; decision-structure
findings recorded: R5b contingency was post-hoc, R1 overturn is
half-supported (maud prong still fires at +0.0319 < +0.05), fused
DROP gate coverage-confounded (paired maud +0.0181) and 2x the
incumbent's bar, double-count inflation extends to retrieval-only
fair mode and the paper calibration, Stage 16b never registered.
Adopted: clip-at-1.0 everywhere, paired-subset reporting, bar
rationale requirement, Stage 19 disclosure list, tracked scorers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
49107804 Stage 18 CRITICAL: CRLF gold-coordinate defect - all maud scores wrong; fused flips to KEEP Eli Ziff 2026-07-31 ↗ GitHub
commit body
Two independent adversarial audits converged on the same defect and a
first-hand minimal repro confirms it: the 17 maud corpus files are
CRLF, upstream gold spans are LF coordinates (334/334 match the
answer oracle under LF, 0/334 raw). Corrected: ctx maud ansR
0.1385->0.4985, fused 0.1449->0.6057, fused pool R@48 0.635->0.863.
Fused grounded confirm flips DROP->KEEP (+0.107 vs +0.02 bar); R4
invalidated; maud-bottleneck narrative was ~65% artifact. Companion
findings recorded: composer noise floor (paired 1-sigma 0.0065,
gates of 0.01-0.02 inside noise), holdout task-staleness (98-100%
template reuse; privacy_qa has NO holdout), bed easiness (doc named
in query, doc_hit ~100%), D2-D8 scorer defects. Standing corrections
adopted: coordinate-corrected scoring everywhere, C1 judged both
ways, fused reinstated as ceiling arm, Stage 19 protocol additions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
62d614ab Stage 18 F1 result: name-stripped retrieval collapses (docR 1.00->0.71, poolR48 0.89->0.37) Eli Ziff 2026-07-31 ↗ GitHub
commit body
Registered prediction confirmed ~3x over: the crowned retrieval numbers
are substantially a query-format artifact (queries name their gold
document). Reported, no gate, per registration.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
13359499 Stage 18 C1/F2/F3 verdict: both coverage arms DROP on precision; F2 fails; F3 prices the contract Eli Ziff 2026-07-31 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
1b115b36 Stage 18 instrument fix: LF corpus normalization helper, gold oracle, D2 union-merge Eli Ziff 2026-07-31 ↗ GitHub
commit body
The 17 maud corpus files are CRLF(+BOM); upstream gold spans are LF
coordinates (decidable from the shipped `answer` string). Adds
normalizeCorpusText (CRLF->LF, BOM kept - gold counts the BOM: 334/334
maud snippets slice correctly with it, 0/334 without) plus the source-db
path constants that keep the normalized and raw builds in separate
coordinate spaces.

legalbench-gold-oracle-check.ts asserts text[start:end]==answer for every
snippet of every source; 1362/1362 pass normalized, 334 maud fail --raw.

D2: charPrecisionRecall summed every pairwise retrieved x gold overlap, so
overlapping chunks/pool spans credited the same gold characters more than
once and recall could exceed 1.0. Credited intersections are now
union-merged per document and both ratios clipped at 1.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
ede4464a Stage 19 literature re-check: CRLF artifact unreported upstream; name-leakage prior (issue #3); replicate/holdout practice Eli Ziff 2026-07-31 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
328eb32b Stage 18 instrument fix: runners load the LF-normalized corpus; D3/D4/D8 Eli Ziff 2026-07-31 ↗ GitHub
commit body
Corpus text is normalized (CRLF -> LF) at file read in legalbench-rag-run.ts,
which is now the one builder of the source db every LegalBench script reads
(`--build-only`). The normalized build is a SEPARATE file (a2aj-mini-lf.sqlite):
passage sidecars are keyed by source-db path + chunk params and NOT by source
content, so sharing one path would have silently reused CRLF-built indexes over
LF text. The corpus files on disk are untouched.

Every runner now points at it; the ablate sweep keeps a `--raw-coords` escape
onto the historical db so before/after is one code path over two corpora, and
tags arm labels/receipt filenames/rows with the coordinate space. Grounding rows
carry `coords: "lf"` and the resume key includes it, so a pre-fix (raw) receipt
file can never be resumed into an LF run.

D3: rerankPassages' `fallback` was dropped on the floor - now recorded per cell
(`rerank_fallback`) and printed as a rate; a fallback cell ran the lexical
order, not the reranker.
D4: the flat "overall" mean hides source-mix artifacts on partial runs - the
summary now prints the per-source cell mix (with an UNBALANCED warning) and a
source-balanced macro mean beside the flat one.
D8: doc-hit rates were printed on inconsistent denominators (grounded over
answered, baseline over all ok rows) - both denominators are now explicit and
the baseline doc rate is also printed on the answered subset.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
a7c9cc3c Stage 18 instrument-fix log: oracle 1362/1362, LF db, D2-D8; maud ctx headers invalidated (Stage 19 prerequisite) Eli Ziff 2026-07-31 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
89ab422b Stage 19 registration: two-arm hold-out burn, gates and disclosures frozen before any hold-out label Eli Ziff 2026-07-31 ↗ GitHub
commit body
Bed derived deterministically (mini rule continued): 3 sources, 55 docs,
582 tests; privacy_qa has no hold-out. Prerequisites gated on a 100%
gold-span oracle and a CR-free corpus assert. A1 = frozen G+ctx x3
replicates (laptop); A2 = fused ceiling, diagnostic only. Adoption rule:
none - Stage 19 selects nothing. Ten mandatory disclosures registered
beside the headline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
2d0e8bbb Stage 19 prerequisite: derive and pin the hold-out split; gold oracle passes 905/905 Eli Ziff 2026-07-31 ↗ GitHub
commit body
deriveSplitTests continues the mini walk past every document the mini
derivation touched (including one truncated at the cap), so the split is
document-blocked by construction; setup asserts zero overlap against the
pinned mini corpus rather than trusting it. Hold-out: 3 sources, 55 docs,
582 tests - privacy_qa mini IS the complete upstream benchmark (194/194)
and yields no hold-out, recorded in derivation.sources_without_split.

Gates, run before any model call: gold oracle 905/905 on the normalized
load (contractnli 276, cuad 311, maud 318) and 587/905 with --raw (maud
0/318), so normalization is load-bearing here too; db-level re-check on
a2aj-holdout-lf.sqlite also 905/905. CR census reproduces the upstream
signature on unseen documents: 18/18 maud files carry \r, the other two
sources clean.

Mini re-derives byte-identical to its pinned manifest (73 files).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
a9aa7efb Stage 18R registration: re-trace the crowning chain on the corrected instrument; Stage 19 blocked behind it Eli Ziff 2026-07-31 ↗ GitHub
commit body
Re-scoring repairs a broken measurement, not a broken selection: chunk
boundaries, the ctx header sidecar and the dense embeddings are INPUTS
the CRLF fix changes, and the G+ctx composed effect (+0.0158) is smaller
than its own null control (+0.0277). Four tiers registered (deterministic
retrieval, rerank with fallback recorded, replicated grounded confirms,
dense re-embed). Verdicts decided below 0.015 are re-labelled
UNDETERMINED; load-bearing ones are re-tested with replicates. Stop rule:
if Tier C cannot separate G+ctx from plain G, the crowning unwinds and
the hold-out sees the simpler config.

Also amends Stage 19: staged replicates 1 -> 3 -> 5 per Eli, with a
pre-committed escalation rule keyed on spread rather than on the result.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
b1c1f048 Stage 18R Tier D: commit the dense-lane dumper, repointed at the LF instrument Eli Ziff 2026-07-31 ↗ GitHub
commit body
Arm D's dumper lived only in a scratchpad file and pointed at the raw
CRLF db, so the dense lane could not be re-derived from the repo at all.
Committed here, split-aware, with the header sidecar as a CLI argument
(the CRLF-era sidecar keys nothing on the normalized index).

Measured consequence of the fix: the raw index carried 5,984 passages,
the normalized one carries 5,966 - the dense lane was embedded over a
chunk set that no longer exists, which is why Tier D re-embeds rather
than re-scores.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
3fb6fd34 Stage 18R Tier D: commit the dense lane so arm D can be re-derived Eli Ziff 2026-07-31 ↗ GitHub
commit body
The original arm D ran from scratchpad files (fastembed CPU, then
Qwen3-Embedding-4B on the 3080 Ti) with the work dir and header sidecar
hardcoded, over the raw (CRLF) 5,984-chunk index. That index no longer
exists, so the arm cannot be re-scored -- only re-derived. An arm that
cannot be re-derived from the repo cannot be re-traced.

legalbench_dense_pools.py ports the GPU lane verbatim in its arm
definitions (lex/ctx/dense/densectx/fused/fusedlex), pooling (top-k with
per-doc cap) and RRF rule (1/(60+rank+1)), with paths parameterized. It
builds pools only and never scores: scoring stays in the committed
instrument so the conventions live in one place. The ctx embedding cache
is keyed by the header sidecar's own hash, so a regenerated sidecar can
never be served a matrix embedded over the previous one.

legalbench_pool_rescore.py gains --coords lf (a sidecar built on the
normalized db is already in gold coordinates; mapping it again would
shift every maud offset a second time) and --arm for multi-arm sidecars.
Verified unchanged on the recorded fused sidecar: maud 0.8625, ALL
0.9584.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
17884220 perf(passageRetrieval): fetch each document body once per search, not once per hit Eli Ziff 2026-07-31 ↗ GitHub
commit body
searchPassages pulled the whole document row for every hit and then sliced
~1.6 KB out of it. With perDocCap 24 over a 330 KB merger agreement that is
the same body fetched 24 times - roughly 16 MB allocated per query to keep
77 KB. Measured: a 776-query profile exhausted a 2 GB heap partway through.

Memoized per call (pure; the cache dies with the call, so nothing goes
stale). Measured on the Stage 18 four-arm sweep over 776 tests:
13m -> 4m06s, a 3.2x speedup, with the receipt byte-identical
(sha b5ab5a94e25df7cab7d80502bf750475b1ff443b21ea174608a2e8d0e19fd8cb
before and after) - no retrieved span, ordering or score changed.

This is the product retrieval path, not just the experiment harness, so it
lands on the laptop-CPU inference target too. 17 passageRetrieval tests pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
1b0462cf Stage 18R Tier D verdict: dense/fused pool advantage reproduces on LF Eli Ziff 2026-07-31 ↗ GitHub
commit body
Arm D re-derived from scratch on the normalized 5,966-chunk index with
the same Qwen3-Embedding-4B fp16 lane on the 3080 Ti (plain matrix
409.6s vs the original run's 409.6s; ctx 497.2s vs 507.9s).

Every non-maud cell of the fresh derivation reproduces the CRLF-era arm
table to four decimals across all six arms -- the only non-maud move is
cuad/ctx at -0.0027, caused by regenerated maud headers perturbing
global FTS statistics, not by the coordinate fix. maud is the only
column that moves, on every arm.

Fused maud pool R@48 0.8392 (ctx 0.6442) clears the frozen 0.50 OPEN bar
with no source dropping vs ctx: the verdict does not flip. Fused and
densectx keep their advantage nearly intact; dense-alone weakens by two
thirds on maud, because the lexical baseline it beat was the corrupted
one (lex maud recorded 0.3489, truly 0.6028).

Re-scoring the old sidecar gave maud 0.8625 / ALL 0.9584; re-deriving
gives 0.8392 / 0.9525. Re-scoring recovers offsets but not chunk
boundaries and keeps the old headers, so it flatters the arm by ~0.02 on
maud -- the measured size of "re-scoring is not enough".

Receipts (receipt store, not git): stage18r-dense-pools-lf.jsonl
8EC2FB7C, stage18r-fused-pools-lf.jsonl DCF8137F, combined LF header
sidecar C484C006, manifest stage18r-tierD-receipts.json 43759F0F.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
eaf19f89 Stage 18R Tier A verdict + Tier C design: stop condition cleared, w4 retained, rerank dropped from default Eli Ziff 2026-07-31 ↗ GitHub
commit body
Tier A (deterministic, LF): maud pool recall now RISES with context
weight (0.6028 w0 -> 0.6544 w1 -> 0.6442 w4) after the header
regeneration, against 0.6028 -> 0.5788 before. Enrichment is worth ~2x
what was recorded (maud lexR4 +0.193 vs +0.092). w1 is now maud-optimal
but w4 wins every general metric, so w4 is RETAINED and w1 recorded -
re-picking on the hardest single source is benchmark tuning.

R1 clause: beats chars on every source on LF (contractnli pool rises
where the original recorded a drop); maud prong still misses, so R1
stays falsified on half its recorded basis. R2: re-falsified decisively.

Tier C staged for a fast first read: pools injected via --pool-jsonl so
only composition is paid, four arms x1 replicate, spread-keyed
escalation to 3 only for pairs inside the band. Rerank dropped from the
default candidate per Eli (76.8k chars/query at p1600, gain not
separable from filename matching per F1); retained as a control arm that
prices it. Preview-500 isolation retired as moot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
1e73f86b perf(passageRetrieval): rank without fts5 content columns; fetch labels per kept hit Eli Ziff 2026-07-31 ↗ GitHub
commit body
The ranked query selected passage_search.citation and passage_search.name.
An fts5 column read materializes the WHOLE stored row - the passage text
plus its context header - so those two short strings cost ~1.6 KB per
candidate over a candidate pool of k*perDocCap*4. At the crowned config
that is 4,608 candidates against a 5,966-row index: essentially the entire
index body pulled through, to label the <=48 rows that survive the
perDocCap filter. Same class as 17884220: a large payload fetched
repeatedly for a small piece of it.

Labels now come from the same table, per surviving row, by rowid.
Identical strings from identical rows, so nothing about ranking or
selection moves.

Measured on the LegalBench mini bed (776 queries x crowned config, plus
five further configs on a private copy of the db to avoid contention):
  ranked query alone   50.4 -> 23.5 ms/query
  searchPassages       81.5 -> 32.5 ms/query (2.5x)
  6-config digest run  197.5s -> 74.5s (2.65x)

Unchanged-output proof: sha256 of every returned PassageHit array over
2,352 query-config cells - chars/clause modes, plain/phrases, contextWeight
0 and 2, four distinct sidecars, k in {4,8,48}, perDocCap in {2,24} - is
byte-identical before and after (digest of digests
07f50c02e7e111f9308c1c6da00718f298c564421c812753dbfc510d04716206). 76 tests
pass across passageRetrieval, a2ajPassageSearch, a2ajPassageWiring,
legalbenchRag, legalbench.

searchPassages is the product retriever, so this lands on the laptop-CPU
inference target, not just the harness.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
0cadfd44 perf(passageRetrieval): cache the context-sidecar digest instead of re-hashing per query Eli Ziff 2026-07-31 ↗ GitHub
commit body
`paramsKey` re-read and re-hashed the whole contextJsonl enrichment file
every time it ran, and it runs twice per searchPassages call - once to
derive the sidecar path, once to compare against the stored params. On
the 2.5 MB Stage 18R headers file that is 3.12 ms per pass, 6.2 ms per
query, to recompute a value that is constant for the entire run.

Cached per (path, mtimeMs, size). The key stays content-addressed: a
headers file rewritten between calls changes size or mtime and is
re-hashed, and the cache only ever short-circuits a file that has not
moved.

Measured on 200 LegalBench mini queries at the crowned config with
contextWeight 2: 70.80 -> 62.82 ms/query (11%). Unchanged-output proof:
sha256 of every returned PassageHit array is byte-identical over all 200
(digest of digests
48cfdcb3367f7091329a2702bf715c84526ce0520abe07f0baf7677613d651a1), and
the sidecar selected is the same file. Plain (no-context) queries are
unaffected - that branch never hashed anything.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
74831c08 Decouple the per-document cap from reranking in the product passage lane Eli Ziff 2026-07-31 ↗ GitHub
commit body
perDocCap was spelled `rerankModel ? 24 : undefined`, so you could not change
reranking without also changing document diversity - a confound sitting under
every rerank measurement this codebase has taken. Now an explicit `perDocCap`
argument with the defaults spelled out.

Behaviour preserved, measured not assumed: over 160 real queries on the
LegalBench LF index, `undefined` and explicit `2` produce byte-identical hit
sets at k=8 (sha 7ffcbdef5cd13bf4) and k=48 (bb4d61780de876b8). The existing
tests never exercised perDocCap at all, so they were no evidence either way.

Recorded honestly in the comment: the reranked 24 is deliberate, but the
un-reranked 2 is searchPassages' own `?? 2`, which entered in 3997cf12 with
the benchmark ablation harness and carries no rationale - unlike every other
default in that function. The cap binds hard (k=8: cap 2 -> 4.54 documents per
query, cap 24 -> 1.49), so it is a real search-policy question, and the
benchmark's answer must NOT be copied: uncapping wins there only because every
LegalBench query names its one gold document. Owed: a measurement on a
text-bearing product corpus; the local A2AJ store is metadata_only today.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
fc820b65 Stage 20: checker-family crossing harness on banked compositions Eli Ziff 2026-07-31 ↗ GitHub
commit body
The program's founding premise -- "the harness carries grounding
decisions, not the model family" (H5) -- rests on 8 cells over one item
pair, with a single false accept as the decisive datum. A replication was
registered twice (experiment log ~1299-1302 and ~1573) and never run.

This runs it on BANKED compositions: no composer call, only the checker
varies. Three conditions per sampled row -- the archived verdict, a
same-model re-check (the run-to-run stochasticity floor at THIS n), and
the other family's anchor model.

- legalGroundingBenchmarks.ts: the CSLB/CLERC/HousingQA case builders
  lifted out of the experiment script unchanged. Run receipts store the
  answer and evidence but not the question, so a re-check has to rebuild
  the item; a second copy of the builders would drift from the runs it
  re-checks.
- holisticVerificationPass exported so the crossing reuses the exact
  system prompt, tool and payload the original runs used.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
28284f04 C4: committed builder for the evidence-layer claim feature matrix Eli Ziff 2026-07-31 ↗ GitHub
commit body
Every pre-2026-07-31 aggregate in this program came from ephemeral
scratchpad code and is unreproducible from the repo. This is the
reproducible builder for the C4 matrix: one row per archived composed
claim over the CA/US grounding lane (stages 1-13), every deterministic
lint witness a column, citator columns (citer count, cited court level,
stands-for tier) joined per cited key from the local noteup graph.

Zero model calls. The lint is recomputed offline against the same
jurisdiction-matched alienness index and the same benchmark question
text the runs used. Instrument oracle: the 121 Stage 7 claims that
archived their own lint receipts reproduce bit-exactly (max abs delta
0.0 on all nine features).

Labels are checker-derived and carry the measured 0-13% flip rate; the
cell-level and C2-propagated claim-level labels are emitted as separate
columns, and Stage 7's lint_gated arm is tagged lint_in_loop because its
archived claim text is post-revision.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
f0f675ee Stage 20 verdict: premise #1 falsified - checker family carries ~36% of grounding decisions Eli Ziff 2026-07-31 ↗ GitHub
commit body
The program's founding premise ("the harness carries grounding decisions,
not the model family") rested on 8 cells over one item pair. Replicated on
289 banked compositions, 1,156 checker calls, no composer call:

  within-family replicate   14/222 =  6.3% [3.8, 10.3]
  archived vs fresh same    48/444 = 10.8% [8.3, 14.0]
  cross-family             159/444 = 35.8% [31.5, 40.4]
  paired difference               +29.5 pp [23.2, 36.0]

Both same-family figures land inside the registered H17 0-13% floor, so
the instrument is behaving; the cross-family effect is ~5x that floor.

Direction is symmetric (73 fail-closed / 86 fail-open), not Stage 5's
11/12 fail-closed. On housing:0 -- the audited-insufficient reference --
the claude checker false-accepts 6/16 and codex catches it, so "the only
false accept required Codex in both roles" does not hold.

Aggregate accept rates over identical compositions match within 3 pp
(claude 61.9%, codex 59.0%). That is exactly how an 8-cell table reads as
family-invariant while 36% of individual decisions flip.

The deterministic tier is invariant by construction: 0 mismatches over
4,951 banked claims. But shadow-crossing its 67 all-clear rows gives
codex 83.6% vs claude 52.2% acceptance, and 8 claude calls on 4 rows
returned `unsupported` on answers whose every claim is a verified
verbatim quote. Logged as an open contradiction, not explained away.

Headline counts recomputed independently from the receipt bytes before
logging. Nothing retired on this run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
ec801e9a C4 matrix: carry Stage 9's banked alienness spectrum and the gold columns Eli Ziff 2026-07-31 ↗ GitHub
commit body
Two additions the study needs and the matrix was missing:

conclusion_alienness - Stage 9 recorded conclusion-claim alienness
spectra on 424/515 rows and logged it as "C4 matrix growth". It was
never analysed. Carrying it onto the row makes boilerplate and
attestedRare separable witnesses, which no run has ever scored.

target_token_f1 / expected_answer_match / inline_citation_rate - gold-
derived cell quality, the only quality axis in the archive that does
not inherit the checker's measured 0-13% flip rate. H7's decision-
quality disjunct needs a label the checker cannot circle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
1d09f681 C4 + H7: the in-concert layer, and the disjunct Stage 7 never measured Eli Ziff 2026-07-31 ↗ GitHub
commit body
Workstream C4 was registered six times in the experiment log and never
once produced a result; the two existing harnesses report solo AUC only.
This is the missing layer, and it is committed rather than run from a
scratchpad so the numbers are reproducible.

Method: rank-AUC with cluster-bootstrap bands computed from THAT
comparison's n; frozen (unfitted) rank-sum as the in-concert combiner,
so the ensemble leaks no labels; marginal value as a PAIRED bootstrap on
the delta, both leave-one-out and best-subset-add; a redundancy matrix;
and an L2 logistic combiner under leave-cluster-group-out CV used only
as a measuring instrument. Everything is EXPLORATORY per the C4 sample-
size gate: the expert lane has 8 misgrounded responses.

It also carries the H7 half that was dropped. C3's gate was disjunctive
("checker-call reduction OR decision-quality gain") and Stage 7's frozen
falsifier list kept only the economic disjunct. Decision quality is
measured here on receipts where the lint was never in the loop, against
the checker verdict and against gold token-F1, stratified by the two
populations the lint was built for.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
2da7cf16 feat: benchmark legal retrieval tool surfaces Eli Ziff 2026-08-01 ↗ GitHub
5c376852 Add deterministic quote span control Eli Ziff 2026-08-01 ↗ GitHub
58682472 feat: add accretive legal evidence union Eli Ziff 2026-08-01 ↗ GitHub
2a1c36ee feat: integrate lean legal retrieval and review Eli Ziff 2026-08-01 ↗ GitHub
33189ea5 Organize legal grounding framing experiment Eli Ziff 2026-08-02 ↗ GitHub
7e3b5c4c Checkpoint adaptive legal retrieval ablations Eli Ziff 2026-08-02 ↗ GitHub
b1161ea0 bench: add native-work-product grounding arm Eli Ziff 2026-08-03 ↗ GitHub
cc144369 docs: preregister native-grounding one-shot arm Eli Ziff 2026-08-03 ↗ GitHub
d4a040f4 bench: add private quote-first grounding arm Eli Ziff 2026-08-03 ↗ GitHub
9a36c283 bench: link selective grounding to draft claims Eli Ziff 2026-08-03 ↗ GitHub
06418894 feat(evidence): agent-consultable attested-characterization tool + citeable journal articles Eli Ziff 2026-08-04 ↗ GitHub
commit body
- consult_attested_characterization (MIKE_CONSULT_ATTESTATIONS=1): a read-only
  citator tool returning up to 3 attested characterizations of a cited case,
  ranked scc_journal_first (court-level band, occurrence count within band,
  then recency; two case slots then journal), each with a registered
  evidence_id and an actionable follow_up pulling the citing case (a2aj_fetch)
  or journal article (public_legal_source_fetch, with a page-lookup hint).
- StandsForCandidate now carries sourceArticleId + citingUrl so follow-ups and
  rendered deep links resolve.
- public_legal_source_fetch (provider journal) registers a passage-scope
  public-journal-v1 receipt: the pulled article becomes citeable by
  submit_grounded_answer (the attestation receipts already were).
- Render surfaces the "as characterized by/in ..." attribution and deep-links
  case attestations to the exact prose in the citing case via text-fragment
  pinpoint.
- Tests cover rank ordering, the tool surface, dispatcher registration, and
  the render (93 pass).

Also in this commit (same tree): the concurrent silo'd derived-section-index
integration (localAssistantTools / lab-beaver-arm / upstreamMikeBenchmarkSurface)
and the Harvey LAB launchpad generator (tools/generate_launchpad.py + smoke
test). The generated launchpad.html/materials.json are gitignored - regenerable,
and the 86 MB materials JSON embeds already-tracked corpus text.

Co-Authored-By: Claude <noreply@anthropic.com>
810a7767 docs(lab): treatment v2 design from tri-agent forensics - floor returns, grounding v2, watchdog+cap fixes Eli Ziff 2026-08-06 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011pfUVhNFTRvhYGXBwoKNj6
ddb08827 docs(lab): v3 quote-grounding organ assessed DON'T-BUILD; DPA attribution corrected to skipped document; read-coverage typed error is the real candidate Eli Ziff 2026-08-06 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011pfUVhNFTRvhYGXBwoKNj6

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-1010.md from inside the repo you want the changes in.

⬇ Download capture-thread-1010.md