eliziff puts legal AI on a tighter leash

This fork is testing how an AI legal assistant can ask for less context, stop digging sooner, and leave a clearer trail when it cannot finish the job.

workflowknowledge-management

The work is experimental, but the direction is sound: disclose document structure only when needed, keep searches bounded, and make the assistant say when it has run out of useful options. A claimed navigation approach was later withdrawn after the underlying setup changed, which is a useful sign of caution rather than a finished feature set.

  • Controlled tool use: the assistant is designed to stop its research loop instead of chasing indefinitely.
  • On-demand document structure: it can request more detail as a matter develops, rather than loading a fixed map upfront.
  • Durable evidence handoffs: findings can be passed on with supporting context.
  • Failure recovery: a path for cases where no suitable tool is available.
  • Legal-focused compaction: long working context is condensed to keep the matter manageable.

The implementation is not ready to lift wholesale, but legal-product teams building research or review workflows should inspect the prompts and measurement approach.

So what Worth watching for teams that need AI assistance to be bounded, auditable, and candid about its limits.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

84 commits from eliziff/Beaver, oldest first. Source extracted verbatim from the harvested git log.

SHA Subject Author Date
2c3cf7f9 Add local-qwen verifier-loop experiment script Eli Ziff 2026-07-28 ↗ GitHub
commit body
Measures whether a local ollama Qwen can translate the amendment
instructions the deterministic grammar refuses (definition-scoped,
portion-scoped) into typed AmendOps, with applyAmendOps as the
rejection sampler and exact-match gold from the current consolidation.
Fixture soundness proven: ideal ops score exact_match on all three
cases (two real SC 2021 c 11 refusals + one synthetic).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns
c73ae62c shrink: share workflow loading skeleton Eli Ziff 2026-07-28 ↗ GitHub
58df4f31 shrink: inline one-off skeleton Eli Ziff 2026-07-28 ↗ GitHub
115e73bf Keep qwen thinking on; retry transient ollama network failures Eli Ziff 2026-07-28 ↗ GitHub
commit body
The verifier-loop caller now retries fetch failures (3 attempts,
linear backoff) and reports error.cause, and no longer sets
think:false - the models are measured with their reasoning intact
per the experiment's fairness rule; structured output rides the
format channel.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns
109dde4f System prompt carries the contract; structure and reads go on-demand Eli Ziff 2026-07-28 ↗ GitHub
commit body
SLA prompt section no longer embeds skeleton outlines (up to 8 x 4k
chars of per-document state as standing prompt weight): it now carries
only the always-on contract plus a one-line-per-document inventory
(name, ~tokens, handle count), and the model fetches outlines when it
reaches for a document via the existing library_outline tool.

Reading gains the compositional glue coding agents get from grep's
file:line: library_find hits now carry their original-text offset
(`at`) and deepest enclosing structural handle (`section`), and
library_read accepts offset/max_chars windows for text without numbered
structure - so the follow-up to a hit is a section read or a bounded
window, not the 300k-char whole-document default. That default is
unchanged pending an A/B (pilot-04 showed breadth-extraction tasks can
lose recall under forced-lean reading).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns
dc5d5ac3 Full-native coding surface: Edit joins Glob/Grep/Read, id bridge removed Eli Ziff 2026-07-29 ↗ GitHub
commit body
The hybrid arm measured duplicate discovery (library_list and Glob both
called twice); files are now path-addressed end to end. Edit performs
exact-string replacement as a tracked change through the revise
machinery - the active version resolves server-side, unique-match
failures reply in Edit's own grammar so the trained expand-and-retry
reflex has signal, and replace_all is refused typed. lean-smoke grades
whichever edit tool the shape serves.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
10230c53 Vendor cslb/repo as files, not a gitlink Eli Ziff 2026-07-29 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
5e188b38 A pathological document announces itself on first read Eli Ziff 2026-07-29 ↗ GitHub
commit body
The sniffer now runs at the read choke points: extractLocalDocument
(library_read/library_find/library_outline) and readDocumentContent
(read_document/fetch_documents) surface scanDocxPathology's
notes_of_caution - manual redlines, unresolved tracked changes, text
boxes, unicode traps - as additive metadata. The extracted text stays
byte-identical, held by test against extractDocxBodyText on both paths;
find_in_document opts out so hit offsets keep anchoring into pure text.

The report caches beside the text parse under its own parseCache
identity (docx-pathology v1, JSON-in-string); an unparseable entry
degrades to no notes, never a failed read. Note wording comes from the
report verbatim, capped at 8 lines per result.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
6c42c881 \s expands to the source's whitespace class in both runtimes Eli Ziff 2026-07-29 ↗ GitHub
commit body
The dialect contract claimed \d \w \s \b were pinned ASCII on both
sides; measured, \s was the exception - JS \s is Unicode-ish in every
mode (NBSP yes, FEFF yes, \x1c no) while Python re.ASCII \s is
ASCII-only, so tables diverged between runtimes on NBSP-bearing legal
text and no vector caught it. Both loaders now expand \s/\S at compile
time to the exact whitespace set Python's Unicode \s gave the source
grammars when they were battle-tested. cite.neutral gains the
NBSP-separated CanLII-render vector; the engine differential dropped
from 62 unicode findings to zero across 49 entries and 2,114 inputs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
044ee14a \w and \b expand to the frozen source word class in both runtimes Eli Ziff 2026-07-29 ↗ GitHub
commit body
Same contract amendment as the engine side: the loaders expand
\w/\W/\b/\B against the generated BMP word class so accented text
behaves as the battle-tested sources did - "les traités 12 pays" no
longer grows a phantom "s 12" section pinpoint under ASCII \b, pinned
as vectors on both section-pinpoint dialects. \d stays ASCII by
decision. 16 tests green including both live Python round-trips;
differential with the French leg: 0 drift, 0 findings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
dc1bde91 V3 schema encoding behind a flag, repriced against today's roster Eli Ziff 2026-07-29 ↗ GitHub
commit body
The survey priced signature-form tool schemas at -58%; re-measured on
the current 42-tool surface it is -35% (~3,176 tokens) - the
single-homing pass already took much of the per-parameter prose the
original number counted, and the smallest tools get slightly heavier
under the signature boilerplate. MIKE_SCHEMA_ENCODING=v3 swaps the
Responses-route serialization to signature + verbatim tool description
+ permissive schema; dispatch-side typed validation is unchanged. The
A/B this exists for is argument correctness: malformed-call rounds vs
score, encoding the only variable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
c6cc4583 Structure probes: the evidence that killed two naive detectors Eli Ziff 2026-07-29 ↗ GitHub
commit body
Cases: ladder_broken was 99.85% detector artifact (1,988 of 1,994 -
A2AJ text is left-flushed, quoted paragraphs are byte-identical to
host paragraphs, adjacent-break counting cannot work in principle);
the real hidden failure is endnote/authority lists scoring as perfect
ladders (10% of a 1,127-doc cross-court sample, near-total for
pre-1970 SCC). ALR paragraph_index vs naive v1 on 60 hand-verified
docs across 16 courts: presence 95% vs 43%, flag precision 93% vs 31%.
gt_truth.json is that hand-verified gold. Laws: the weak sets are four
rendering conventions, not dialects; ALR SECTION_MARK_RE + three
catalog-traced lookahead extensions + range expansion + named-heading
recovery takes them from .54-.65 to .965-1.000 - and found ALR's own
SK hyphen-gate under-fire (total refusal where .85 recall exists).
Journals harvester feeds the once-vacuous grammar entries real
extraction text. Four A2AJ corpus defects recorded (Sept-Oct 2003 BCCA
line-collapsed rows). Verdict, both fronts: port ALR, delete v1,
record precision not just recall.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
247be0ed Structure cascade: absence of paragraphs is a question, not an answer Eli Ziff 2026-07-29 ↗ GitHub
commit body
structure_ref.py carries the faithful ALR ports (paragraph_index
proven 0-mismatch over 298 corpus docs; page_structure with
reporter-anchored starts) plus marked Beaver extensions: the
section-lookahead widenings, range expansion, named-heading recovery,
and an endnote-ladder detector built from the taxonomy's measured
separation. sweep.py deletes its naive detectors and runs the cascade
- paragraphs, then pages, then endnote ladders, then heading hints -
so only a document where everything comes up empty lands in
no_addressable_structure, a close-inspection queue. Laws score
recovery AND precision per detector. New prefilter gates on the
heavy unprefiltered entries. probes/vet_misses.py re-examines every
previously-flagged miss under the cascade (first 40: all prior
ladder_broken flags resolve to clean paragraph scopes). Also lands
docs/reinvention-ledger-2026-07-29.md - the audit: 63 capability rows,
the hidden-port finding on sourceDocA2AJ, the missing cited-scope
quote resolution as top gap, and the citator confirmed genuinely ours.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
5e9d5c13 Ledger: the endnote detector repeated the pattern the ledger exists to catch Eli Ziff 2026-07-29 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
a17781a3 FP-scan the structure claims; rebuild endnote detection on the fidelity engine's contract Eli Ziff 2026-07-29 ↗ GitHub
commit body
fp_scan.py screens every cascade claim (vetted + fresh reservoir
populations) and scores law detectors against the sections oracle.
It caught three FP classes: room lists (405..), year tables (1985..),
and appendix [55]+ labels claimed as endnotes; and 'Form s'/'Order to'
garbage from the named-heading canonicalizer (precision 0.293).

endnote_index is no longer hand-authored: labels are the engine's
label.line-start table entry copied byte-equal (provenance comment,
--parity drift assertion, no runtime import), and ladders open only
at expected number 1 or under a Notes/Endnotes heading, per
core._infer_note_region_modes. Pairing stays geometric and
engine-only; claims carry a typed anchor field instead.

Measured after: endnote FP screens 0, 150 false endnote claims
corrected to none, 15 new true ladders from label forms the old
grammar missed; named precision 0.579 (residue is oracle-scope
mismatch on real Schedule/Preamble units), combined law precision
0.955 -> 0.976 at unchanged recall.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
d01ddbc2 Act on reinvention-ledger items 5 and the sync gate; pin engine faef914 Eli Ziff 2026-07-29 ↗ GitHub
commit body
sweep.py resolves public_endpoint-*.db by newest glob instead of a
pinned content-hash filename; engine --check now fails on grammar-table
drift (verified both directions).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
07705b4e docs: B4 recon - Charlotin rows are episode paraphrases, not probes Eli Ziff 2026-07-30 ↗ GitHub
commit body
All Charlotin-origin corpus rows lack spans/questions/verbatim text;
revised plan mines the sanctioning judgments (A2AJ-fetchable) for the
verbatim offending propositions - claim-level positives with judicial
provenance.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
c81627a4 runner: --resume 1 skips completed cells in the existing output Eli Ziff 2026-07-30 ↗ GitHub
commit body
Non-error rows in the output file mark their cell done; errored cells
re-run (analysis dedupes keeping the last row per cell). Lets a live
run be stopped and relaunched at higher concurrency without losing
finished cells.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
799948f9 Render premise corrections marked; salvage typed content on failed cells Eli Ziff 2026-07-30 ↗ GitHub
commit body
Per Eli: verified premise corrections should be shown to the user, not
buried in an abstention. renderLegalEvidenceAnswer now (a) renders any
premise_correction whose anchor re-verifies deterministically at render
time as a visible **Premise correction** with the contested words, and
(b) on failed cells salvages receipts-backed typed content - verified
corrections and exact no-attestation statements from the rejected
answer or last bounce, excluding claims the bounce errors name - above
the typed abstention line. Never parsed from prose; unverified anchors
never surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
45e70fc1 W1 context column: heading-path prefixes in dedicated FTS column, default off Eli Ziff 2026-07-30 ↗ GitHub
commit body
Measured on LegalBench gold: weight 2 is slightly negative (R@4 0.2865
to 0.2763) because headings are regex-guessed on plain text and queries
do not use heading vocabulary. Column ships default-off for the product
skeleton lane, where true heading paths exist.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
09767629 Clause-skeleton chunk mode: spans snap to clause/section starts Eli Ziff 2026-07-30 ↗ GitHub
commit body
Per Eli: chunk contracts into a skeleton prior to retrieval like A2AJ
paragraphs - retrieval unit = citable unit. clauseChunkText packs
whole clauses to target size, subdividing only clauses over 2x target;
auto-falls back to character mode under 3 clause starts. Gold
ablation, lexical @4: maud R 0.049 to 0.089 (+82% rel), cuad 0.184 to
0.262, contractnli flat, privacy_qa -0.02 (weak numbering; density-
gated auto-mode is the follow-up). Sidecar params v3.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
1470f4f4 tests: pin clauseChunkText contract (tiling, packing, oversize split, joints, fallback, index identity) Eli Ziff 2026-07-30 ↗ GitHub
commit body
The clause-skeleton chunker shipped with the Stage 15/16 retrieval
overhaul measured (+82% maud rel lexical R@4) but untested. Pins:
spans tile with starts only at clause boundaries; whole clauses pack
to target; a >2x-target clause subdivides without overlap; Section /
ARTICLE / dotted / (a) / (iv) joints recognized; <3 joints falls back
to chunkText; mode is part of the sidecar index identity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
bb365d32 R5b: --citation-prefix flag scopes header generation to one source Eli Ziff 2026-07-30 ↗ GitHub
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
e347b85c Ground-up restart plan: four audits, four failure classes, and the ideas that were never fairly tested Eli Ziff 2026-07-31 ↗ GitHub
commit body
Four parallel audits (assets+perf, case-law discards, retrieval discards, bed
design) found four distinct failure classes, each of which produced confident
verdicts that favoured the incumbent: instrument defects (CRLF, D2 double-count,
D3 fallback blending), statistics borrowed from the wrong n (premise #1 is 8
cells over one item pair; flat_recency retired on p=0.264; H18 falsified on
<=1-cell deltas), detector blindness read as a negative result (zero subsections
across all 69 documents, so R1 never tested clause chunking on the sources where
it failed), and a bookkeeping error that crowned the current composer lane.

Plan phases: P0 stop the bleeding (perDocCap confound blocks every composed run;
ensurePassageIndex rebuilds on transient SQLITE_BUSY; repair the skeleton
detector; withdraw R1). P1 free re-tests on banked data (checker-family crossing
~150 calls decides premise #1; C4 marginal-value study zero calls). P2 build the
case-law bed - gated on the A2AJ full-text import, since the provider sqlite is
metadata_only with 0 of 248,536 documents carrying text. P3 the arms never run,
each shipping the control that makes it falsifiable. P4 close out LegalBench
cheaply and stop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
e15d2d0d Fix the gold-corpus path in rule 4 Eli Ziff 2026-07-31 ↗ GitHub
commit body
Desktop/legal-generalization-corpus does not exist; the gold is in-repo at
benchmarks/legal-generalization-corpus/ (138 MB, 220 files: raw/, text/,
amending-acts/, canadian/structure-gold/, cslb/, and gold/ carrying CUAD v1,
MAUD v1 and 79 USLM public laws). The wrong path has been steering local-data
lookups away from data we already have.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
1ca6a238 F2: never reindex on a transient sqlite lock Eli Ziff 2026-07-31 ↗ GitHub
commit body
`ensurePassageIndex`'s meta probe was wrapped in `catch {}`, so a
SQLITE_BUSY raised by a concurrent reader fell through to
DROP TABLE + full reindex of a HEALTHY sidecar. Observed live twice on
2026-07-31; on the 5.5 GB CourtListener corpus it is hours of work
destroyed by a race.

Verified shape (node 22.20, node:sqlite): the read-only OPEN succeeds and
the `SELECT ... FROM meta` throws `Error { code: "ERR_SQLITE_ERROR",
errcode: 5, errstr: "database is locked" }` - i.e. the failure lands
exactly in the swallowed catch, not at open.

`isSqliteLockError` classifies on the primary result code (low byte of
the extended `errcode`): SQLITE_BUSY 5 and SQLITE_LOCKED 6, so 261/517/773
classify as transient too; message fallback for errors that lost the
numeric field. Transient -> rethrow (caller retries); corrupt/foreign ->
rebuild, unchanged.

Tests (backend/src/lib/__tests__/passageRetrieval.test.ts): a real
exclusive lock produces a real errcode-5 error and is classified
transient; under that lock `ensurePassageIndex` rethrows "database is
locked" and the sidecar survives (built:false, identical counts
afterwards); a foreign-but-readable sidecar still rebuilds. The
integration test discriminates the two code paths by pointing sourceDb at
a missing file - the rebuild path opens the source FIRST, so pre-fix it
reports "unable to open database file". Mutation-checked: reverting the
rethrow fails 2 of the 3 tests.

No scoring, chunking or retrieval semantics touched; no measured number
moves.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
1d220fce F3: resume keys carry every dimension that changes what a cell means Eli Ziff 2026-07-31 ↗ GitHub
commit body
legalbench-rag-grounding keyed cells on
coords|model|effort|k|retriever|test_id. `arm` was written to the row but
LEFT OUT of the key, and --coverage, --spec, --plain and --exclude-gold
each change the arm - the prompt, and for --exclude-gold the evidence -
while changing nothing the key could see. Those are exactly the flags the
C1a/C1b/F2/F3 experiments ran under, so a resume could have declared one
arm's cells done for another. Measured occurrences: zero (every existing
receipt is raw-CRLF, and `coords` already separates those), but live.

The sibling legal-grounding-experiment had the mirror-image hole: `arm`
was in its key, `effort` was not, which already forced a Stage 13 ladder
lane onto its own output file to stay separable.

Both keys now live in src/lib/experimentReceipts beside the receipt guard:
  legalbenchRagCellKey  coords|model|effort|arm|k|retriever|per_doc_cap|test_id
  legalGroundingCellKey model|effort|arm|checker|case_id|rank_policy

Backwards compatible, deliberately: a missing field falls back to what the
row's own labels prove it ran under, so pre-fix receipts stay resumable
instead of being re-run wholesale. `arm` absent = required_slot;
`per_doc_cap` absent is derived from the retriever label (product = n/a,
passage:pool(...) = uncapped, +rerank = 24, otherwise 2 - the exact pre-fix
semantics); `coords` absent = the raw-CRLF instrument, which still can
never satisfy an LF cell.

Verified on the real receipt store (read-only, nothing written): stage14
keys as ...|product|n/a|..., stage17/18 as ...|24|..., the fused-pool file
as ...|uncapped|..., the plain-arm file as ...|plain|..., and no file has
fewer distinct keys than non-error rows.

Tests: 11 in src/lib/__tests__/experimentReceipts.test.ts, including all
six arm labels distinct, every other dimension distinct, identical cells
identical, and the legacy backfills. Full backend suite: 1427 passed
(2 pre-existing docx/compare timeouts under parallel load, both pass in
isolation, neither touches this code).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
4c90e0ac Register the structure-graph round; record Phase 0, the CL widening, and two corrections Eli Ziff 2026-07-31 ↗ GitHub
commit body
New doc legal-structure-graph-round-2026-07-31.md registers Eli's design before
any of it is built: layers ordered by VERIFIABILITY (hard refs have a free
oracle; model relational edges over a given skeleton are endpoint-verifiable;
similarity edges have no oracle and rank last), ablation ladder in build order
led by deterministic edge-following at retrieval time (no model, no tokens, no
round trips), the maud-vs-contractnli/privacy_qa density gap (13.1 vs 0.6-0.8
per 10k chars) as the built-in negative control, and an explicit kill criterion
on gold-graph alignment. Multi-doc comparator deferred: LegalBench-RAG is
single-document by construction and has no bed for it.

Restart plan gains a status section: Phase 0's four fixes, the perDocCap
confound measured at ~16x the composer band, the product-lane decoupling, the
CourtListener widening to 55,504 opinion rows, and two corrections of record -
the "skeleton is blind to subsections" claim was an artifact of the probe that
produced it (toBlock flattens every kind to "section"; nodes carry 2,347
subsections), and the real defect was page-number runs winning the spine
competition and suppressing real headings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
0d5e4568 Correct the cross-reference verdict: unproven on this bed, not refuted Eli Ziff 2026-07-31 ↗ GitHub
commit body
The earlier "retrieval-time edge following is DEAD" conclusion was an overreach
and is withdrawn. Four reasons the measurement cannot carry it:

1. The metric structurally cannot reward the mechanism. LegalBench gold is
   where the answer text is, not what must be read to understand it, so
   resolving "as defined in Section 1.1" to fetch a deal-term definition -
   correct legal behaviour - scores as precision loss by construction.
2. Small n: 15/168 contractnli and 14/153 cuad gold fragments had an edge at
   all. The "cuad loses 6x" headline is fourteen fragments.
3. Half instrument strength: 24/69 documents refused by the integrity gate,
   including 8/17 maud, and the resolver's ceiling (the skeleton section
   inventory) was only repaired mid-run.
4. It measured a proxy - pool expansion scored by character overlap - not the
   registered proposal, which is structure as orientation for a composer.

Edge-following is therefore UNPROVEN here: it earns no arm yet, and no positive
claim either. The similarity-edge retirement stands, since those lost to a
deterministic control and do not depend on the gold-span objection. The
resolver stands on its own as a working legal primitive.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
d2c5edbb Withdraw the similarity/defined-term retirement too; adopt a retirement standard Eli Ziff 2026-07-31 ↗ GitHub
commit body
The gold-span objection applies to the weak layers as well, and letting their
retirement stand on "they lost to a deterministic control" was wrong: a clean
control does not repair a metric that cannot express the mechanism. A
defined-term edge exists precisely to reach a definition, and a definition is
by construction never the answer span. One untuned implementation each, 14-31
fragments, 24/69 documents refused.

Nothing in this round is retired. Adopted standard, since the existing ensemble
doctrine already required it (never discard a witness on weak solo performance;
retire only on demonstrated redundancy): retiring any layer needs a metric that
can express the mechanism, adequate n with the band computed from that n, full
instrument strength with the refusal rate reported, and testing in concert
rather than solo.

Honest status of every layer in this round: built and unproven.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
c3ae6b9a Adopt Eli's retirement standard; the registered kill criterion was a design error Eli Ziff 2026-07-31 ↗ GitHub
commit body
"none of this is remotely 'retired' until the most robust deterministic first
layer is built and we test it with real model runs across several ablations."

That standard governs. It also invalidates the premise of the kill criterion I
registered this morning: it made a deterministic proxy sufficient to close a
hypothesis, so a cheap gate could retire ideas without a single model run. A
proxy says where to look; it cannot retire anything. The gold-graph measurement
is downgraded to a diagnostic and the section heading now says so.

Nothing in this round is retired and nothing can be until the deterministic
layer is genuinely robust - today it is 24/69 documents refused, 8/17 on maud,
skeleton repaired mid-run, one untuned implementation per edge class, no
variants tried - and until real composed runs across several ablations test it,
solo and in concert, retiring only on demonstrated redundancy.

Honest status of every layer: built, unproven, and never once put in front of a
composer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
285b86f7 Rewrite both docs tight: 259 -> 106 and 256 -> 133 lines Eli Ziff 2026-07-31 ↗ GitHub
commit body
Same content, half the words. Collapsed the amendment-on-amendment layering in
the structure doc into one current-state read, and cut the restart plan's
narrative padding to tables and short claims. Nothing dropped: the four defect
classes, the eight principles (now including "nothing retired on a
deterministic proxy"), Phase 0's seven commits, the Phase 1-4 queues and the
do-not list all survive.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
6db89765 Drop the upstream filing: internal finding, we are not contacting anyone Eli Ziff 2026-07-31 ↗ GitHub
commit body
Eli never asked for this - it came from a literature-check recommendation and I
carried it into the plan unprompted. The CRLF/LF portability defect stays an
internal finding. "Not filing or contacting the maintainers" is now stated where
the finding is recorded, and "contact or file anything upstream" is first on the
do-not list so it cannot creep back.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
cfc15494 Record the structure round in the registered doc, and what is left Eli Ziff 2026-07-31 ↗ GitHub
commit body
Refusals 24 -> 16 on mini and 14 -> 9 on the disjoint holdout, with the
residue characterized rather than rounded off: six privacy policies carrying
~11 internal references between them (refusing there is correct), two NDAs
with zero references, and four maud documents holding 2,016 references whose
body headings survive extraction nowhere.

Also records the split the round exposed but did not build: a contents page
is a legitimate OUTLINE and an illegitimate span index. The reach gate is
right to refuse resolving into one, and the outline it throws away is richer
than the reading kept (Boingo 13 titled heads vs 80; holdout ASPIRITY 11 vs
102). And the negative result, with its number, so nobody re-ports it blind:
Text-Fidelity's later-recurrence contents proof costs 19 resolved on mini and
110 on the holdout here, because uniqueLabel has already handed the bare
label to the contents entry.

Nothing about any hypothesis or arm is judged; the instrument got stronger.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
0fcc4aeb Grep across the graph, and name the page states for what they are Eli Ziff 2026-07-31 ↗ GitHub
commit body
SCOPE COMPOSITION. library_find gains `section`, `follow` and `depth`: search
one provision and, optionally, the provisions it references, the provisions
that reference it, or both, out to three hops. Only RESOLVED edges expand the
scope -- an external or unresolved reference names no span in this document,
so following it could only widen the search to nothing.

Page scope and structural scope compose by INTERSECTION. Two filters both
supplied read as "in these pages AND in this part of the document", the only
reading under which asking for more narrowing cannot widen the result. The
graph is compiled only when it will actually be walked, so a plain section
scope still costs a skeleton and nothing more.

PAGE STATES. Eli: "for where we can't detect a page number, that's not
'unavailable', it's 'pdf page numbers only'." Printed labels are DETECTED --
from header and footer furniture, left unresolved when they conflict -- so
their absence is a detection outcome, never a missing document feature. The
states are now `artifact`, `markers`, `unpaginated` (no fixed pages at all)
and `unindexed` (paged, index not built), and `pageSchemes` reports which of
the two addressing schemes actually work so a refusal can say "PDF page
numbers only" instead of implying the document has no pages.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
bae8ce6f Stage 21 report: replicate floor covers every headline metric Eli Ziff 2026-07-31 ↗ GitHub
commit body
The floor table only carried f1_best, reached_any and n_tool_calls, so
the two metrics the arms actually separate on - reached_by_read and
chars_exposed - had no noise band to be judged against. It now reports
the same metric set as the between-arm table, line for line, and says
in the header that a between-arm difference smaller than its own floor
is undecided at this n rather than null.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
538072b0 Stage 21 result: address nav surface undecided; its trigger fired; three of four extra affordances never called Eli Ziff 2026-07-31 ↗ GitHub
commit body
881 scored agentic cells on LegalBench-RAG mini. Paired n=215 (asis,
rep 1 at the full pre-registered n=160 per arm) and n=127 (stripped).
Both bed oracles passed before the first model call.

Outcome metrics are undecided: f1_best +0.0105 [-0.0028, 0.0242]
against a within-arm replicate floor of 0.028-0.045, tool calls -0.20
against a floor of 0.81-1.08, reached_by_read +0.0419 against a floor
of 0.13-0.14. The registered disqualification trigger DID fire -
reached_any -0.0977 [-0.1404, -0.0591] - and is recorded as fired; the
same metric's within-arm floor (0.162) is larger than the gap, so both
registered rules fired at once and both are reported.

The one difference that clears its own floor and reproduces in both
query forms: the address arm exposes 48% less document for the same
answer quality (5,818 vs 11,130 chars), which also makes it cheaper in
total input tokens despite carrying 2,197 more schema chars.

Affordance uptake, out of 2,085 address-arm tool calls: library_links
0.0%, follow actually walked 0.2%, find scoping 0.4%, tail reads 0.3%,
page addressing 0.0% (structurally inert on this unpaginated bed, as
pre-registered). The model filled in follow="none", depth=1 on every
find call - paying for the graph walk and declining it 1,347 times out
of 1,350. Document length does not predict an arm advantage: in the
navigation-mandatory band the arms are dead level.

Contamination instrument clean: zero off-schema parameters in 4,117
tool calls. 27.4% of asis cells were never solved by either arm, and
the read-it-whole shortcut, available on 67.5% of cells, was taken in
5 of 430. 354 cells lost to Codex rate limiting are excluded from
scoring rather than scored as zero. Nothing retired.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
31ae135b Prose travels with its domain: research instructions defer with research tools Eli Ziff 2026-07-31 ↗ GitHub
commit body
Instructions for tools that are not loaded are pure cost. The research prose
-- CourtListener, A2AJ, public sources -- explains tools the address arm
defers, so it now defers with them and arrives in the describe_tools reply
that opens the domain.

This is not a new mechanism; SPREADSHEET_CITATION_PROMPT already worked this
way, spliced in only when a spreadsheet is actually in context. It was simply
never applied to the research half.

  prose    A 1,958   B 1,227
  schemas  A 10,130  B 4,004
  BASE     A 12,088  B 5,231    -57%

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
8ddf101f Stage 21 addendum: the second frozen pairing disagrees with the first Eli Ziff 2026-07-31 ↗ GitHub
commit body
Receipts finished filling before the chains stopped (1,093 scored cells,
up from 881). The extra rows are all post-drift, so the first pinned
subset is unchanged, but the later schemas now form a second and larger
internally-consistent pairing - legacy 2671ch x address 5031ch, the
CURRENT definitions on the tree, paired n=186 asis.

It disagrees with the first. f1_best +0.0231 [0.0070, 0.0426] becomes
+0.0013 [-0.0211, 0.0269]; reached_by_read +0.1000 [0.0200, 0.1827]
becomes -0.0269 [-0.0867, 0.0391]. Neither pairing's f1_best clears its
within-arm replicate floor, so the honest reading of both together is
no demonstrated quality effect, and the early positive is a small-n
fluctuation rather than something a later revision erased.

Corrects the correction: it said the disqualification trigger does not
fire on the registered arms. That holds for the EARLY pairing only. On
the current arms the trigger fires robustly on the largest frozen
subset - reached_any -0.1774 [-0.2251, -0.1315], reached_all -0.2849 -
and is recorded as fired. As before the metric is unresolvable at this
n: arm B's own replicate floor on reached_any is 0.2391, larger than
the gap.

The one effect that is real on the current arms is exposure:
chars_exposed -9,789 [-12727, -7075] against floors of 2,556 and 5,638.
The current arm B reads about a quarter as much document (3,344 vs
13,133 chars) for statistically identical answer quality, which is
exactly why reached_any falls while reached_by_read stays flat. Arm B
is also slower (+1,383 ms).

Unchanged across both pairings and now 2,941 tool calls: library_links
0 calls, follow walked the graph 0 times in 597 find calls, page
addressing 0, off-schema usage 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
9572d3ca Cut the rest of the deferred prose, and open the seam that blocks disclosure Eli Ziff 2026-07-31 ↗ GitHub
commit body
PROSE. Three cuts, all the same defect in different places: instructions for
tools the address arm does not load.

- The DOCX generation routing block explained which of generate_docx /
  generate_excel / generate_ppt to call. All deferred under "drafting", so
  the routing now travels with them.
- "doc_id" claimed to be the chat-local label without saying it is NOT the
  Library document_id the library_* tools take. Two id vocabularies, one
  rule, no statement of which applies where -- in the highest-stakes output
  the model produces. Both arms get the disambiguation; it is a correctness
  fix, not an arm difference.
- The generic conduct rules ("be precise", "do not fabricate", "batch
  independent tool calls") are dropped in the address arm as an explicit
  bet: whether a frontier model needs telling, or whether these restate its
  defaults. Never measured here, so it is a hypothesis, not a cleanup.

  prose    A  1,970   B  1,077   -893
  schemas  A 10,130   B  4,004   -6,126
  BASE     A 12,100   B  5,081   -58%

THE SEAM. streamResponsesApi snapshotted toResponseTools(tools) before the
tool loop and sent that snapshot every iteration, so a tool revealed mid-
conversation could never be called -- progressive disclosure was unbuildable
without duplicating the provider loop, which the one-implementation rule
forbids. `resolveTools?: () => OpenAIToolSchema[]` is re-read per iteration;
absent, behaviour is byte-identical to before. Codex routes through the same
function, so one change serves both lanes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
fc75e69d Stage 21: re-score the addendum on the final receipts (n=186 -> n=191) Eli Ziff 2026-07-31 ↗ GitHub
commit body
The last chains drained after the addendum was written (1,093 -> 1,100
scored cells). Re-scored on the final set, every figure in the later
frozen pairing is unchanged within rounding. The only movement is
f1_best, +0.0013 [-0.0211, 0.0269] -> -0.0011 [-0.0224, 0.0245]: the
point estimate crosses zero, which strengthens the "no demonstrated
quality effect" reading rather than weakening it. Recorded because the
committed table quoted the earlier n.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
d31d5442 Add legal compaction on-demand experiment Eli Ziff 2026-08-01 ↗ GitHub
c31bb94a fix: let agent tool loops stop naturally Eli Ziff 2026-08-01 ↗ GitHub
ad94a840 fix: bound hybrid evidence review Eli Ziff 2026-08-01 ↗ GitHub
77875f43 Bound research context with durable evidence Eli Ziff 2026-08-02 ↗ GitHub
e06d28b2 Tighten context handoff transitions Eli Ziff 2026-08-02 ↗ GitHub
10ff1279 Clarify drafting context role Eli Ziff 2026-08-02 ↗ GitHub
cbfa8f90 Register context refresh live run Eli Ziff 2026-08-02 ↗ GitHub
f1685962 Preserve compaction card repair state Eli Ziff 2026-08-02 ↗ GitHub
171cd82d Checkpoint upstream comparator and tool-loop semantics Eli Ziff 2026-08-02 ↗ GitHub
6481e9ae Compact Qwen discovery protocol Eli Ziff 2026-08-02 ↗ GitHub
72fe9ed9 Fix compacted no-tool recovery loop Eli Ziff 2026-08-02 ↗ GitHub
107f328b Add bounded discovery read ablation Eli Ziff 2026-08-02 ↗ GitHub
d9f192d0 Restore native-style content compaction Eli Ziff 2026-08-02 ↗ GitHub
94a90392 Preregister Luna coding-context ablation Eli Ziff 2026-08-02 ↗ GitHub
70bd7b9c Compact Qwen synthesis vocabulary and tools Eli Ziff 2026-08-02 ↗ GitHub
804799bd Limit 1k discovery hits Eli Ziff 2026-08-02 ↗ GitHub
8dff6cc6 Allow custom held-out legal tasks Eli Ziff 2026-08-02 ↗ GitHub
588d2c23 Preregister minimal Mike retrieval ablations Eli Ziff 2026-08-02 ↗ GitHub
dbcf2d62 Add exact upstream terminal ablation Eli Ziff 2026-08-02 ↗ GitHub
7e8ad9dc Correct upstream comparator provenance Eli Ziff 2026-08-02 ↗ GitHub
2f3d1036 Preregister minimal Mike architecture matrix Eli Ziff 2026-08-02 ↗ GitHub
9b90bc8a Correct registered launcher fingerprint Eli Ziff 2026-08-02 ↗ GitHub
c4cdccc5 Preregister grounded cache twelve-task matrix Eli Ziff 2026-08-02 ↗ GitHub
9a77c5ca Exclude mega-suite from grounded cache pass Eli Ziff 2026-08-02 ↗ GitHub
c7e4b741 docs: bind lean batch preregistration commits Eli Ziff 2026-08-03 ↗ GitHub
b79a91e0 docs: bind compaction correction preregistration Eli Ziff 2026-08-03 ↗ GitHub
87eb0fef docs: preregister Mike one-shot experiment Eli Ziff 2026-08-03 ↗ GitHub
9c1c8855 docs: preregister bounded fact index rerun Eli Ziff 2026-08-03 ↗ GitHub
df55f399 docs: preregister conflict-first one-shot follow-up Eli Ziff 2026-08-03 ↗ GitHub
124cd7c4 docs: preregister interrupted Indenture replacement Eli Ziff 2026-08-03 ↗ GitHub
d0422433 docs: preregister xhigh Mike control Eli Ziff 2026-08-03 ↗ GitHub
5762ebf0 docs: preregister quote-first replication Eli Ziff 2026-08-03 ↗ GitHub
633c6ba2 docs: preregister xhigh Banking replacement Eli Ziff 2026-08-03 ↗ GitHub
0f2b1f90 docs: preregister minimal Sol max candidate Eli Ziff 2026-08-03 ↗ GitHub
43b795c0 docs: preregister adaptive review ablation Eli Ziff 2026-08-03 ↗ GitHub
a5ac5b25 docs: preregister fresh omissions scout Eli Ziff 2026-08-03 ↗ GitHub
8309a1c4 docs: record one-shot context verdict Eli Ziff 2026-08-03 ↗ GitHub
83e27b83 docs: include effort-matched Mike controls Eli Ziff 2026-08-03 ↗ GitHub
85b4837e docs: preregister linked and final-check ablations Eli Ziff 2026-08-03 ↗ GitHub
d24da27a feat(assistant): offer low effort on DeepSeek V4 models Eli Ziff 2026-08-04 ↗ GitHub
commit body
The reasoning-effort dropdown for deepseek-* models only surfaced high and
max, so the cheap lane could never be pushed cheaper. Add low to the option
list (default stays high) and lock the adapter passthrough with a test that
reasoning_effort: "low" reaches the request body.

Co-Authored-By: Claude <noreply@anthropic.com>
2c1eb539 Score the paragraph spine as a global monotone chain Eli Ziff 2026-08-06 ↗ GitHub
commit body
The spine was picked by walking line-start markers and accepting the first
monotone run, so a marker joined onto the end of a heading line
("II. Analysis [12] The appellant argues...") was invisible: the run
stopped one short, or started partway down at whatever number appeared
first.

Replaces the walk with the mechanism the footnote pairer already uses:
enumerate every candidate for every number, price each by the evidence
behind it (line start, heading-joined, adjacent to a confirmed link), and
take the highest-scoring increase-by-one chain over the document as a
whole. Evidence is priced, not gated. The chain must root at 1 and have no
gaps, and endnote ladders are detected and declined - a numbered reference
list in the last quarter of a judgment reads as a perfect 1..N ladder and
is not paragraphs.

The heading grammar stops rejecting enumerated levels on length alone: an
author who numbered a level has already said it is a heading, so it keeps
the long word cap rather than the short one meant for levels with nothing
but brevity to recommend them.

Over all 224,972 A2AJ case documents: 2,090 gain a spine, 2,709 lose one,
29,899 relabelled, net +562,044 paragraphs. Of the losses, 2,103 were
never rooted at 1 and 563 were endnote ladders. Legislation is
byte-identical; CourtListener over its own 55,504 opinions is 55,489
identical, 8 gained, 2 lost. Heading recall, scored against labels derived
from document position rather than from the grammar, rises 73.70% to
90.94% with the strict negative set still at exactly zero false accepts.

Four sourcedoc fixtures were mid-decision excerpts - one opening 44,435
characters into Sattva - that no chain rooted at paragraph 1 could match;
they are replaced with the full A2AJ records, so the beaver-can manifests
pinning their hashes are refreshed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VyTUeXhYSK9awyArAtm83c
d972f3e3 fix(llm): retry deepseek rounds on zero-progress transport failures Eli Ziff 2026-08-06 ↗ GitHub
commit body
A single undici "TypeError: terminated" (mid-stream TLS reset) killed a
40-minute flash lab run at top level - the deepseek adapter had no retry
at all, unlike the openai/claude-p lanes. Wrap each round's request +
SSE consumption in a classified retry (3 attempts, 2s/4s backoff):
transport-class errors only (socket/DNS errnos, undici terminated/fetch
failed, HTTP 429/5xx), and only while zero deltas have been forwarded -
consumers accumulate deltas into the persisted turn, so post-emission
retries would duplicate text and resampled output cannot be stitched.
The exposed window is dominated by silent prefill on large contexts,
which is exactly the zero-progress case. Retries are trace-recorded as
"transport_retry". Semantic errors (4xx, provider chunk.error) stay
fatal; classifier probed against the captured error shapes (10/10).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011pfUVhNFTRvhYGXBwoKNj6
0a66c8e6 Add legal primitives experiment suite Eli Ziff 2026-08-08 ↗ GitHub

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-1009.md from inside the repo you want the changes in.

⬇ Download capture-thread-1009.md