2c3cf7f9 | Add local-qwen verifier-loop experiment script | Eli Ziff | 2026-07-28 | ↗ GitHub |
commit body Measures whether a local ollama Qwen can translate the amendment
instructions the deterministic grammar refuses (definition-scoped,
portion-scoped) into typed AmendOps, with applyAmendOps as the
rejection sampler and exact-match gold from the current consolidation.
Fixture soundness proven: ideal ops score exact_match on all three
cases (two real SC 2021 c 11 refusals + one synthetic).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns
|
c73ae62c | shrink: share workflow loading skeleton | Eli Ziff | 2026-07-28 | ↗ GitHub |
58df4f31 | shrink: inline one-off skeleton | Eli Ziff | 2026-07-28 | ↗ GitHub |
115e73bf | Keep qwen thinking on; retry transient ollama network failures | Eli Ziff | 2026-07-28 | ↗ GitHub |
commit body The verifier-loop caller now retries fetch failures (3 attempts,
linear backoff) and reports error.cause, and no longer sets
think:false - the models are measured with their reasoning intact
per the experiment's fairness rule; structured output rides the
format channel.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns
|
109dde4f | System prompt carries the contract; structure and reads go on-demand | Eli Ziff | 2026-07-28 | ↗ GitHub |
commit body SLA prompt section no longer embeds skeleton outlines (up to 8 x 4k
chars of per-document state as standing prompt weight): it now carries
only the always-on contract plus a one-line-per-document inventory
(name, ~tokens, handle count), and the model fetches outlines when it
reaches for a document via the existing library_outline tool.
Reading gains the compositional glue coding agents get from grep's
file:line: library_find hits now carry their original-text offset
(`at`) and deepest enclosing structural handle (`section`), and
library_read accepts offset/max_chars windows for text without numbered
structure - so the follow-up to a hit is a section read or a bounded
window, not the 300k-char whole-document default. That default is
unchanged pending an A/B (pilot-04 showed breadth-extraction tasks can
lose recall under forced-lean reading).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018bhP89cYSdEEjHNPnxacns
|
dc5d5ac3 | Full-native coding surface: Edit joins Glob/Grep/Read, id bridge removed | Eli Ziff | 2026-07-29 | ↗ GitHub |
commit body The hybrid arm measured duplicate discovery (library_list and Glob both
called twice); files are now path-addressed end to end. Edit performs
exact-string replacement as a tracked change through the revise
machinery - the active version resolves server-side, unique-match
failures reply in Edit's own grammar so the trained expand-and-retry
reflex has signal, and replace_all is refused typed. lean-smoke grades
whichever edit tool the shape serves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
10230c53 | Vendor cslb/repo as files, not a gitlink | Eli Ziff | 2026-07-29 | ↗ GitHub |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
5e188b38 | A pathological document announces itself on first read | Eli Ziff | 2026-07-29 | ↗ GitHub |
commit body The sniffer now runs at the read choke points: extractLocalDocument
(library_read/library_find/library_outline) and readDocumentContent
(read_document/fetch_documents) surface scanDocxPathology's
notes_of_caution - manual redlines, unresolved tracked changes, text
boxes, unicode traps - as additive metadata. The extracted text stays
byte-identical, held by test against extractDocxBodyText on both paths;
find_in_document opts out so hit offsets keep anchoring into pure text.
The report caches beside the text parse under its own parseCache
identity (docx-pathology v1, JSON-in-string); an unparseable entry
degrades to no notes, never a failed read. Note wording comes from the
report verbatim, capped at 8 lines per result.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
6c42c881 | \s expands to the source's whitespace class in both runtimes | Eli Ziff | 2026-07-29 | ↗ GitHub |
commit body The dialect contract claimed \d \w \s \b were pinned ASCII on both
sides; measured, \s was the exception - JS \s is Unicode-ish in every
mode (NBSP yes, FEFF yes, \x1c no) while Python re.ASCII \s is
ASCII-only, so tables diverged between runtimes on NBSP-bearing legal
text and no vector caught it. Both loaders now expand \s/\S at compile
time to the exact whitespace set Python's Unicode \s gave the source
grammars when they were battle-tested. cite.neutral gains the
NBSP-separated CanLII-render vector; the engine differential dropped
from 62 unicode findings to zero across 49 entries and 2,114 inputs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
044ee14a | \w and \b expand to the frozen source word class in both runtimes | Eli Ziff | 2026-07-29 | ↗ GitHub |
commit body Same contract amendment as the engine side: the loaders expand
\w/\W/\b/\B against the generated BMP word class so accented text
behaves as the battle-tested sources did - "les traités 12 pays" no
longer grows a phantom "s 12" section pinpoint under ASCII \b, pinned
as vectors on both section-pinpoint dialects. \d stays ASCII by
decision. 16 tests green including both live Python round-trips;
differential with the French leg: 0 drift, 0 findings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
dc1bde91 | V3 schema encoding behind a flag, repriced against today's roster | Eli Ziff | 2026-07-29 | ↗ GitHub |
commit body The survey priced signature-form tool schemas at -58%; re-measured on
the current 42-tool surface it is -35% (~3,176 tokens) - the
single-homing pass already took much of the per-parameter prose the
original number counted, and the smallest tools get slightly heavier
under the signature boilerplate. MIKE_SCHEMA_ENCODING=v3 swaps the
Responses-route serialization to signature + verbatim tool description
+ permissive schema; dispatch-side typed validation is unchanged. The
A/B this exists for is argument correctness: malformed-call rounds vs
score, encoding the only variable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
c6cc4583 | Structure probes: the evidence that killed two naive detectors | Eli Ziff | 2026-07-29 | ↗ GitHub |
commit body Cases: ladder_broken was 99.85% detector artifact (1,988 of 1,994 -
A2AJ text is left-flushed, quoted paragraphs are byte-identical to
host paragraphs, adjacent-break counting cannot work in principle);
the real hidden failure is endnote/authority lists scoring as perfect
ladders (10% of a 1,127-doc cross-court sample, near-total for
pre-1970 SCC). ALR paragraph_index vs naive v1 on 60 hand-verified
docs across 16 courts: presence 95% vs 43%, flag precision 93% vs 31%.
gt_truth.json is that hand-verified gold. Laws: the weak sets are four
rendering conventions, not dialects; ALR SECTION_MARK_RE + three
catalog-traced lookahead extensions + range expansion + named-heading
recovery takes them from .54-.65 to .965-1.000 - and found ALR's own
SK hyphen-gate under-fire (total refusal where .85 recall exists).
Journals harvester feeds the once-vacuous grammar entries real
extraction text. Four A2AJ corpus defects recorded (Sept-Oct 2003 BCCA
line-collapsed rows). Verdict, both fronts: port ALR, delete v1,
record precision not just recall.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
247be0ed | Structure cascade: absence of paragraphs is a question, not an answer | Eli Ziff | 2026-07-29 | ↗ GitHub |
commit body structure_ref.py carries the faithful ALR ports (paragraph_index
proven 0-mismatch over 298 corpus docs; page_structure with
reporter-anchored starts) plus marked Beaver extensions: the
section-lookahead widenings, range expansion, named-heading recovery,
and an endnote-ladder detector built from the taxonomy's measured
separation. sweep.py deletes its naive detectors and runs the cascade
- paragraphs, then pages, then endnote ladders, then heading hints -
so only a document where everything comes up empty lands in
no_addressable_structure, a close-inspection queue. Laws score
recovery AND precision per detector. New prefilter gates on the
heavy unprefiltered entries. probes/vet_misses.py re-examines every
previously-flagged miss under the cascade (first 40: all prior
ladder_broken flags resolve to clean paragraph scopes). Also lands
docs/reinvention-ledger-2026-07-29.md - the audit: 63 capability rows,
the hidden-port finding on sourceDocA2AJ, the missing cited-scope
quote resolution as top gap, and the citator confirmed genuinely ours.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
5e9d5c13 | Ledger: the endnote detector repeated the pattern the ledger exists to catch | Eli Ziff | 2026-07-29 | ↗ GitHub |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
a17781a3 | FP-scan the structure claims; rebuild endnote detection on the fidelity engine's contract | Eli Ziff | 2026-07-29 | ↗ GitHub |
commit body fp_scan.py screens every cascade claim (vetted + fresh reservoir
populations) and scores law detectors against the sections oracle.
It caught three FP classes: room lists (405..), year tables (1985..),
and appendix [55]+ labels claimed as endnotes; and 'Form s'/'Order to'
garbage from the named-heading canonicalizer (precision 0.293).
endnote_index is no longer hand-authored: labels are the engine's
label.line-start table entry copied byte-equal (provenance comment,
--parity drift assertion, no runtime import), and ladders open only
at expected number 1 or under a Notes/Endnotes heading, per
core._infer_note_region_modes. Pairing stays geometric and
engine-only; claims carry a typed anchor field instead.
Measured after: endnote FP screens 0, 150 false endnote claims
corrected to none, 15 new true ladders from label forms the old
grammar missed; named precision 0.579 (residue is oracle-scope
mismatch on real Schedule/Preamble units), combined law precision
0.955 -> 0.976 at unchanged recall.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
d01ddbc2 | Act on reinvention-ledger items 5 and the sync gate; pin engine faef914 | Eli Ziff | 2026-07-29 | ↗ GitHub |
commit body sweep.py resolves public_endpoint-*.db by newest glob instead of a
pinned content-hash filename; engine --check now fails on grammar-table
drift (verified both directions).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
07705b4e | docs: B4 recon - Charlotin rows are episode paraphrases, not probes | Eli Ziff | 2026-07-30 | ↗ GitHub |
commit body All Charlotin-origin corpus rows lack spans/questions/verbatim text;
revised plan mines the sanctioning judgments (A2AJ-fetchable) for the
verbatim offending propositions - claim-level positives with judicial
provenance.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
c81627a4 | runner: --resume 1 skips completed cells in the existing output | Eli Ziff | 2026-07-30 | ↗ GitHub |
commit body Non-error rows in the output file mark their cell done; errored cells
re-run (analysis dedupes keeping the last row per cell). Lets a live
run be stopped and relaunched at higher concurrency without losing
finished cells.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
799948f9 | Render premise corrections marked; salvage typed content on failed cells | Eli Ziff | 2026-07-30 | ↗ GitHub |
commit body Per Eli: verified premise corrections should be shown to the user, not
buried in an abstention. renderLegalEvidenceAnswer now (a) renders any
premise_correction whose anchor re-verifies deterministically at render
time as a visible **Premise correction** with the contested words, and
(b) on failed cells salvages receipts-backed typed content - verified
corrections and exact no-attestation statements from the rejected
answer or last bounce, excluding claims the bounce errors name - above
the typed abstention line. Never parsed from prose; unverified anchors
never surface.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
45e70fc1 | W1 context column: heading-path prefixes in dedicated FTS column, default off | Eli Ziff | 2026-07-30 | ↗ GitHub |
commit body Measured on LegalBench gold: weight 2 is slightly negative (R@4 0.2865
to 0.2763) because headings are regex-guessed on plain text and queries
do not use heading vocabulary. Column ships default-off for the product
skeleton lane, where true heading paths exist.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
09767629 | Clause-skeleton chunk mode: spans snap to clause/section starts | Eli Ziff | 2026-07-30 | ↗ GitHub |
commit body Per Eli: chunk contracts into a skeleton prior to retrieval like A2AJ
paragraphs - retrieval unit = citable unit. clauseChunkText packs
whole clauses to target size, subdividing only clauses over 2x target;
auto-falls back to character mode under 3 clause starts. Gold
ablation, lexical @4: maud R 0.049 to 0.089 (+82% rel), cuad 0.184 to
0.262, contractnli flat, privacy_qa -0.02 (weak numbering; density-
gated auto-mode is the follow-up). Sidecar params v3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
1470f4f4 | tests: pin clauseChunkText contract (tiling, packing, oversize split, joints, fallback, index identity) | Eli Ziff | 2026-07-30 | ↗ GitHub |
commit body The clause-skeleton chunker shipped with the Stage 15/16 retrieval
overhaul measured (+82% maud rel lexical R@4) but untested. Pins:
spans tile with starts only at clause boundaries; whole clauses pack
to target; a >2x-target clause subdivides without overlap; Section /
ARTICLE / dotted / (a) / (iv) joints recognized; <3 joints falls back
to chunkText; mode is part of the sidecar index identity.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
bb365d32 | R5b: --citation-prefix flag scopes header generation to one source | Eli Ziff | 2026-07-30 | ↗ GitHub |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
|
e347b85c | Ground-up restart plan: four audits, four failure classes, and the ideas that were never fairly tested | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body Four parallel audits (assets+perf, case-law discards, retrieval discards, bed
design) found four distinct failure classes, each of which produced confident
verdicts that favoured the incumbent: instrument defects (CRLF, D2 double-count,
D3 fallback blending), statistics borrowed from the wrong n (premise #1 is 8
cells over one item pair; flat_recency retired on p=0.264; H18 falsified on
<=1-cell deltas), detector blindness read as a negative result (zero subsections
across all 69 documents, so R1 never tested clause chunking on the sources where
it failed), and a bookkeeping error that crowned the current composer lane.
Plan phases: P0 stop the bleeding (perDocCap confound blocks every composed run;
ensurePassageIndex rebuilds on transient SQLITE_BUSY; repair the skeleton
detector; withdraw R1). P1 free re-tests on banked data (checker-family crossing
~150 calls decides premise #1; C4 marginal-value study zero calls). P2 build the
case-law bed - gated on the A2AJ full-text import, since the provider sqlite is
metadata_only with 0 of 248,536 documents carrying text. P3 the arms never run,
each shipping the control that makes it falsifiable. P4 close out LegalBench
cheaply and stop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
e15d2d0d | Fix the gold-corpus path in rule 4 | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body Desktop/legal-generalization-corpus does not exist; the gold is in-repo at
benchmarks/legal-generalization-corpus/ (138 MB, 220 files: raw/, text/,
amending-acts/, canadian/structure-gold/, cslb/, and gold/ carrying CUAD v1,
MAUD v1 and 79 USLM public laws). The wrong path has been steering local-data
lookups away from data we already have.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
1ca6a238 | F2: never reindex on a transient sqlite lock | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body `ensurePassageIndex`'s meta probe was wrapped in `catch {}`, so a
SQLITE_BUSY raised by a concurrent reader fell through to
DROP TABLE + full reindex of a HEALTHY sidecar. Observed live twice on
2026-07-31; on the 5.5 GB CourtListener corpus it is hours of work
destroyed by a race.
Verified shape (node 22.20, node:sqlite): the read-only OPEN succeeds and
the `SELECT ... FROM meta` throws `Error { code: "ERR_SQLITE_ERROR",
errcode: 5, errstr: "database is locked" }` - i.e. the failure lands
exactly in the swallowed catch, not at open.
`isSqliteLockError` classifies on the primary result code (low byte of
the extended `errcode`): SQLITE_BUSY 5 and SQLITE_LOCKED 6, so 261/517/773
classify as transient too; message fallback for errors that lost the
numeric field. Transient -> rethrow (caller retries); corrupt/foreign ->
rebuild, unchanged.
Tests (backend/src/lib/__tests__/passageRetrieval.test.ts): a real
exclusive lock produces a real errcode-5 error and is classified
transient; under that lock `ensurePassageIndex` rethrows "database is
locked" and the sidecar survives (built:false, identical counts
afterwards); a foreign-but-readable sidecar still rebuilds. The
integration test discriminates the two code paths by pointing sourceDb at
a missing file - the rebuild path opens the source FIRST, so pre-fix it
reports "unable to open database file". Mutation-checked: reverting the
rethrow fails 2 of the 3 tests.
No scoring, chunking or retrieval semantics touched; no measured number
moves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
1d220fce | F3: resume keys carry every dimension that changes what a cell means | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body legalbench-rag-grounding keyed cells on
coords|model|effort|k|retriever|test_id. `arm` was written to the row but
LEFT OUT of the key, and --coverage, --spec, --plain and --exclude-gold
each change the arm - the prompt, and for --exclude-gold the evidence -
while changing nothing the key could see. Those are exactly the flags the
C1a/C1b/F2/F3 experiments ran under, so a resume could have declared one
arm's cells done for another. Measured occurrences: zero (every existing
receipt is raw-CRLF, and `coords` already separates those), but live.
The sibling legal-grounding-experiment had the mirror-image hole: `arm`
was in its key, `effort` was not, which already forced a Stage 13 ladder
lane onto its own output file to stay separable.
Both keys now live in src/lib/experimentReceipts beside the receipt guard:
legalbenchRagCellKey coords|model|effort|arm|k|retriever|per_doc_cap|test_id
legalGroundingCellKey model|effort|arm|checker|case_id|rank_policy
Backwards compatible, deliberately: a missing field falls back to what the
row's own labels prove it ran under, so pre-fix receipts stay resumable
instead of being re-run wholesale. `arm` absent = required_slot;
`per_doc_cap` absent is derived from the retriever label (product = n/a,
passage:pool(...) = uncapped, +rerank = 24, otherwise 2 - the exact pre-fix
semantics); `coords` absent = the raw-CRLF instrument, which still can
never satisfy an LF cell.
Verified on the real receipt store (read-only, nothing written): stage14
keys as ...|product|n/a|..., stage17/18 as ...|24|..., the fused-pool file
as ...|uncapped|..., the plain-arm file as ...|plain|..., and no file has
fewer distinct keys than non-error rows.
Tests: 11 in src/lib/__tests__/experimentReceipts.test.ts, including all
six arm labels distinct, every other dimension distinct, identical cells
identical, and the legacy backfills. Full backend suite: 1427 passed
(2 pre-existing docx/compare timeouts under parallel load, both pass in
isolation, neither touches this code).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
4c90e0ac | Register the structure-graph round; record Phase 0, the CL widening, and two corrections | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body New doc legal-structure-graph-round-2026-07-31.md registers Eli's design before
any of it is built: layers ordered by VERIFIABILITY (hard refs have a free
oracle; model relational edges over a given skeleton are endpoint-verifiable;
similarity edges have no oracle and rank last), ablation ladder in build order
led by deterministic edge-following at retrieval time (no model, no tokens, no
round trips), the maud-vs-contractnli/privacy_qa density gap (13.1 vs 0.6-0.8
per 10k chars) as the built-in negative control, and an explicit kill criterion
on gold-graph alignment. Multi-doc comparator deferred: LegalBench-RAG is
single-document by construction and has no bed for it.
Restart plan gains a status section: Phase 0's four fixes, the perDocCap
confound measured at ~16x the composer band, the product-lane decoupling, the
CourtListener widening to 55,504 opinion rows, and two corrections of record -
the "skeleton is blind to subsections" claim was an artifact of the probe that
produced it (toBlock flattens every kind to "section"; nodes carry 2,347
subsections), and the real defect was page-number runs winning the spine
competition and suppressing real headings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
0d5e4568 | Correct the cross-reference verdict: unproven on this bed, not refuted | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body The earlier "retrieval-time edge following is DEAD" conclusion was an overreach
and is withdrawn. Four reasons the measurement cannot carry it:
1. The metric structurally cannot reward the mechanism. LegalBench gold is
where the answer text is, not what must be read to understand it, so
resolving "as defined in Section 1.1" to fetch a deal-term definition -
correct legal behaviour - scores as precision loss by construction.
2. Small n: 15/168 contractnli and 14/153 cuad gold fragments had an edge at
all. The "cuad loses 6x" headline is fourteen fragments.
3. Half instrument strength: 24/69 documents refused by the integrity gate,
including 8/17 maud, and the resolver's ceiling (the skeleton section
inventory) was only repaired mid-run.
4. It measured a proxy - pool expansion scored by character overlap - not the
registered proposal, which is structure as orientation for a composer.
Edge-following is therefore UNPROVEN here: it earns no arm yet, and no positive
claim either. The similarity-edge retirement stands, since those lost to a
deterministic control and do not depend on the gold-span objection. The
resolver stands on its own as a working legal primitive.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
d2c5edbb | Withdraw the similarity/defined-term retirement too; adopt a retirement standard | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body The gold-span objection applies to the weak layers as well, and letting their
retirement stand on "they lost to a deterministic control" was wrong: a clean
control does not repair a metric that cannot express the mechanism. A
defined-term edge exists precisely to reach a definition, and a definition is
by construction never the answer span. One untuned implementation each, 14-31
fragments, 24/69 documents refused.
Nothing in this round is retired. Adopted standard, since the existing ensemble
doctrine already required it (never discard a witness on weak solo performance;
retire only on demonstrated redundancy): retiring any layer needs a metric that
can express the mechanism, adequate n with the band computed from that n, full
instrument strength with the refusal rate reported, and testing in concert
rather than solo.
Honest status of every layer in this round: built and unproven.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
c3ae6b9a | Adopt Eli's retirement standard; the registered kill criterion was a design error | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body "none of this is remotely 'retired' until the most robust deterministic first
layer is built and we test it with real model runs across several ablations."
That standard governs. It also invalidates the premise of the kill criterion I
registered this morning: it made a deterministic proxy sufficient to close a
hypothesis, so a cheap gate could retire ideas without a single model run. A
proxy says where to look; it cannot retire anything. The gold-graph measurement
is downgraded to a diagnostic and the section heading now says so.
Nothing in this round is retired and nothing can be until the deterministic
layer is genuinely robust - today it is 24/69 documents refused, 8/17 on maud,
skeleton repaired mid-run, one untuned implementation per edge class, no
variants tried - and until real composed runs across several ablations test it,
solo and in concert, retiring only on demonstrated redundancy.
Honest status of every layer: built, unproven, and never once put in front of a
composer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
285b86f7 | Rewrite both docs tight: 259 -> 106 and 256 -> 133 lines | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body Same content, half the words. Collapsed the amendment-on-amendment layering in
the structure doc into one current-state read, and cut the restart plan's
narrative padding to tables and short claims. Nothing dropped: the four defect
classes, the eight principles (now including "nothing retired on a
deterministic proxy"), Phase 0's seven commits, the Phase 1-4 queues and the
do-not list all survive.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
6db89765 | Drop the upstream filing: internal finding, we are not contacting anyone | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body Eli never asked for this - it came from a literature-check recommendation and I
carried it into the plan unprompted. The CRLF/LF portability defect stays an
internal finding. "Not filing or contacting the maintainers" is now stated where
the finding is recorded, and "contact or file anything upstream" is first on the
do-not list so it cannot creep back.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
cfc15494 | Record the structure round in the registered doc, and what is left | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body Refusals 24 -> 16 on mini and 14 -> 9 on the disjoint holdout, with the
residue characterized rather than rounded off: six privacy policies carrying
~11 internal references between them (refusing there is correct), two NDAs
with zero references, and four maud documents holding 2,016 references whose
body headings survive extraction nowhere.
Also records the split the round exposed but did not build: a contents page
is a legitimate OUTLINE and an illegitimate span index. The reach gate is
right to refuse resolving into one, and the outline it throws away is richer
than the reading kept (Boingo 13 titled heads vs 80; holdout ASPIRITY 11 vs
102). And the negative result, with its number, so nobody re-ports it blind:
Text-Fidelity's later-recurrence contents proof costs 19 resolved on mini and
110 on the holdout here, because uniqueLabel has already handed the bare
label to the contents entry.
Nothing about any hypothesis or arm is judged; the instrument got stronger.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
0fcc4aeb | Grep across the graph, and name the page states for what they are | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body SCOPE COMPOSITION. library_find gains `section`, `follow` and `depth`: search
one provision and, optionally, the provisions it references, the provisions
that reference it, or both, out to three hops. Only RESOLVED edges expand the
scope -- an external or unresolved reference names no span in this document,
so following it could only widen the search to nothing.
Page scope and structural scope compose by INTERSECTION. Two filters both
supplied read as "in these pages AND in this part of the document", the only
reading under which asking for more narrowing cannot widen the result. The
graph is compiled only when it will actually be walked, so a plain section
scope still costs a skeleton and nothing more.
PAGE STATES. Eli: "for where we can't detect a page number, that's not
'unavailable', it's 'pdf page numbers only'." Printed labels are DETECTED --
from header and footer furniture, left unresolved when they conflict -- so
their absence is a detection outcome, never a missing document feature. The
states are now `artifact`, `markers`, `unpaginated` (no fixed pages at all)
and `unindexed` (paged, index not built), and `pageSchemes` reports which of
the two addressing schemes actually work so a refusal can say "PDF page
numbers only" instead of implying the document has no pages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
bae8ce6f | Stage 21 report: replicate floor covers every headline metric | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body The floor table only carried f1_best, reached_any and n_tool_calls, so
the two metrics the arms actually separate on - reached_by_read and
chars_exposed - had no noise band to be judged against. It now reports
the same metric set as the between-arm table, line for line, and says
in the header that a between-arm difference smaller than its own floor
is undecided at this n rather than null.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
538072b0 | Stage 21 result: address nav surface undecided; its trigger fired; three of four extra affordances never called | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body 881 scored agentic cells on LegalBench-RAG mini. Paired n=215 (asis,
rep 1 at the full pre-registered n=160 per arm) and n=127 (stripped).
Both bed oracles passed before the first model call.
Outcome metrics are undecided: f1_best +0.0105 [-0.0028, 0.0242]
against a within-arm replicate floor of 0.028-0.045, tool calls -0.20
against a floor of 0.81-1.08, reached_by_read +0.0419 against a floor
of 0.13-0.14. The registered disqualification trigger DID fire -
reached_any -0.0977 [-0.1404, -0.0591] - and is recorded as fired; the
same metric's within-arm floor (0.162) is larger than the gap, so both
registered rules fired at once and both are reported.
The one difference that clears its own floor and reproduces in both
query forms: the address arm exposes 48% less document for the same
answer quality (5,818 vs 11,130 chars), which also makes it cheaper in
total input tokens despite carrying 2,197 more schema chars.
Affordance uptake, out of 2,085 address-arm tool calls: library_links
0.0%, follow actually walked 0.2%, find scoping 0.4%, tail reads 0.3%,
page addressing 0.0% (structurally inert on this unpaginated bed, as
pre-registered). The model filled in follow="none", depth=1 on every
find call - paying for the graph walk and declining it 1,347 times out
of 1,350. Document length does not predict an arm advantage: in the
navigation-mandatory band the arms are dead level.
Contamination instrument clean: zero off-schema parameters in 4,117
tool calls. 27.4% of asis cells were never solved by either arm, and
the read-it-whole shortcut, available on 67.5% of cells, was taken in
5 of 430. 354 cells lost to Codex rate limiting are excluded from
scoring rather than scored as zero. Nothing retired.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
31ae135b | Prose travels with its domain: research instructions defer with research tools | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body Instructions for tools that are not loaded are pure cost. The research prose
-- CourtListener, A2AJ, public sources -- explains tools the address arm
defers, so it now defers with them and arrives in the describe_tools reply
that opens the domain.
This is not a new mechanism; SPREADSHEET_CITATION_PROMPT already worked this
way, spliced in only when a spreadsheet is actually in context. It was simply
never applied to the research half.
prose A 1,958 B 1,227
schemas A 10,130 B 4,004
BASE A 12,088 B 5,231 -57%
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
8ddf101f | Stage 21 addendum: the second frozen pairing disagrees with the first | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body Receipts finished filling before the chains stopped (1,093 scored cells,
up from 881). The extra rows are all post-drift, so the first pinned
subset is unchanged, but the later schemas now form a second and larger
internally-consistent pairing - legacy 2671ch x address 5031ch, the
CURRENT definitions on the tree, paired n=186 asis.
It disagrees with the first. f1_best +0.0231 [0.0070, 0.0426] becomes
+0.0013 [-0.0211, 0.0269]; reached_by_read +0.1000 [0.0200, 0.1827]
becomes -0.0269 [-0.0867, 0.0391]. Neither pairing's f1_best clears its
within-arm replicate floor, so the honest reading of both together is
no demonstrated quality effect, and the early positive is a small-n
fluctuation rather than something a later revision erased.
Corrects the correction: it said the disqualification trigger does not
fire on the registered arms. That holds for the EARLY pairing only. On
the current arms the trigger fires robustly on the largest frozen
subset - reached_any -0.1774 [-0.2251, -0.1315], reached_all -0.2849 -
and is recorded as fired. As before the metric is unresolvable at this
n: arm B's own replicate floor on reached_any is 0.2391, larger than
the gap.
The one effect that is real on the current arms is exposure:
chars_exposed -9,789 [-12727, -7075] against floors of 2,556 and 5,638.
The current arm B reads about a quarter as much document (3,344 vs
13,133 chars) for statistically identical answer quality, which is
exactly why reached_any falls while reached_by_read stays flat. Arm B
is also slower (+1,383 ms).
Unchanged across both pairings and now 2,941 tool calls: library_links
0 calls, follow walked the graph 0 times in 597 find calls, page
addressing 0, off-schema usage 0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
9572d3ca | Cut the rest of the deferred prose, and open the seam that blocks disclosure | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body PROSE. Three cuts, all the same defect in different places: instructions for
tools the address arm does not load.
- The DOCX generation routing block explained which of generate_docx /
generate_excel / generate_ppt to call. All deferred under "drafting", so
the routing now travels with them.
- "doc_id" claimed to be the chat-local label without saying it is NOT the
Library document_id the library_* tools take. Two id vocabularies, one
rule, no statement of which applies where -- in the highest-stakes output
the model produces. Both arms get the disambiguation; it is a correctness
fix, not an arm difference.
- The generic conduct rules ("be precise", "do not fabricate", "batch
independent tool calls") are dropped in the address arm as an explicit
bet: whether a frontier model needs telling, or whether these restate its
defaults. Never measured here, so it is a hypothesis, not a cleanup.
prose A 1,970 B 1,077 -893
schemas A 10,130 B 4,004 -6,126
BASE A 12,100 B 5,081 -58%
THE SEAM. streamResponsesApi snapshotted toResponseTools(tools) before the
tool loop and sent that snapshot every iteration, so a tool revealed mid-
conversation could never be called -- progressive disclosure was unbuildable
without duplicating the provider loop, which the one-implementation rule
forbids. `resolveTools?: () => OpenAIToolSchema[]` is re-read per iteration;
absent, behaviour is byte-identical to before. Codex routes through the same
function, so one change serves both lanes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
fc75e69d | Stage 21: re-score the addendum on the final receipts (n=186 -> n=191) | Eli Ziff | 2026-07-31 | ↗ GitHub |
commit body The last chains drained after the addendum was written (1,093 -> 1,100
scored cells). Re-scored on the final set, every figure in the later
frozen pairing is unchanged within rounding. The only movement is
f1_best, +0.0013 [-0.0211, 0.0269] -> -0.0011 [-0.0224, 0.0245]: the
point estimate crosses zero, which strengthens the "no demonstrated
quality effect" reading rather than weakening it. Recorded because the
committed table quoted the earlier n.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
|
d31d5442 | Add legal compaction on-demand experiment | Eli Ziff | 2026-08-01 | ↗ GitHub |
c31bb94a | fix: let agent tool loops stop naturally | Eli Ziff | 2026-08-01 | ↗ GitHub |
ad94a840 | fix: bound hybrid evidence review | Eli Ziff | 2026-08-01 | ↗ GitHub |
77875f43 | Bound research context with durable evidence | Eli Ziff | 2026-08-02 | ↗ GitHub |
e06d28b2 | Tighten context handoff transitions | Eli Ziff | 2026-08-02 | ↗ GitHub |
10ff1279 | Clarify drafting context role | Eli Ziff | 2026-08-02 | ↗ GitHub |
cbfa8f90 | Register context refresh live run | Eli Ziff | 2026-08-02 | ↗ GitHub |
f1685962 | Preserve compaction card repair state | Eli Ziff | 2026-08-02 | ↗ GitHub |
171cd82d | Checkpoint upstream comparator and tool-loop semantics | Eli Ziff | 2026-08-02 | ↗ GitHub |
6481e9ae | Compact Qwen discovery protocol | Eli Ziff | 2026-08-02 | ↗ GitHub |
72fe9ed9 | Fix compacted no-tool recovery loop | Eli Ziff | 2026-08-02 | ↗ GitHub |
107f328b | Add bounded discovery read ablation | Eli Ziff | 2026-08-02 | ↗ GitHub |
d9f192d0 | Restore native-style content compaction | Eli Ziff | 2026-08-02 | ↗ GitHub |
94a90392 | Preregister Luna coding-context ablation | Eli Ziff | 2026-08-02 | ↗ GitHub |
70bd7b9c | Compact Qwen synthesis vocabulary and tools | Eli Ziff | 2026-08-02 | ↗ GitHub |
804799bd | Limit 1k discovery hits | Eli Ziff | 2026-08-02 | ↗ GitHub |
8dff6cc6 | Allow custom held-out legal tasks | Eli Ziff | 2026-08-02 | ↗ GitHub |
588d2c23 | Preregister minimal Mike retrieval ablations | Eli Ziff | 2026-08-02 | ↗ GitHub |
dbcf2d62 | Add exact upstream terminal ablation | Eli Ziff | 2026-08-02 | ↗ GitHub |
7e8ad9dc | Correct upstream comparator provenance | Eli Ziff | 2026-08-02 | ↗ GitHub |
2f3d1036 | Preregister minimal Mike architecture matrix | Eli Ziff | 2026-08-02 | ↗ GitHub |
9b90bc8a | Correct registered launcher fingerprint | Eli Ziff | 2026-08-02 | ↗ GitHub |
c4cdccc5 | Preregister grounded cache twelve-task matrix | Eli Ziff | 2026-08-02 | ↗ GitHub |
9a77c5ca | Exclude mega-suite from grounded cache pass | Eli Ziff | 2026-08-02 | ↗ GitHub |
c7e4b741 | docs: bind lean batch preregistration commits | Eli Ziff | 2026-08-03 | ↗ GitHub |
b79a91e0 | docs: bind compaction correction preregistration | Eli Ziff | 2026-08-03 | ↗ GitHub |
87eb0fef | docs: preregister Mike one-shot experiment | Eli Ziff | 2026-08-03 | ↗ GitHub |
9c1c8855 | docs: preregister bounded fact index rerun | Eli Ziff | 2026-08-03 | ↗ GitHub |
df55f399 | docs: preregister conflict-first one-shot follow-up | Eli Ziff | 2026-08-03 | ↗ GitHub |
124cd7c4 | docs: preregister interrupted Indenture replacement | Eli Ziff | 2026-08-03 | ↗ GitHub |
d0422433 | docs: preregister xhigh Mike control | Eli Ziff | 2026-08-03 | ↗ GitHub |
5762ebf0 | docs: preregister quote-first replication | Eli Ziff | 2026-08-03 | ↗ GitHub |
633c6ba2 | docs: preregister xhigh Banking replacement | Eli Ziff | 2026-08-03 | ↗ GitHub |
0f2b1f90 | docs: preregister minimal Sol max candidate | Eli Ziff | 2026-08-03 | ↗ GitHub |
43b795c0 | docs: preregister adaptive review ablation | Eli Ziff | 2026-08-03 | ↗ GitHub |
a5ac5b25 | docs: preregister fresh omissions scout | Eli Ziff | 2026-08-03 | ↗ GitHub |
8309a1c4 | docs: record one-shot context verdict | Eli Ziff | 2026-08-03 | ↗ GitHub |
83e27b83 | docs: include effort-matched Mike controls | Eli Ziff | 2026-08-03 | ↗ GitHub |
85b4837e | docs: preregister linked and final-check ablations | Eli Ziff | 2026-08-03 | ↗ GitHub |
d24da27a | feat(assistant): offer low effort on DeepSeek V4 models | Eli Ziff | 2026-08-04 | ↗ GitHub |
commit body The reasoning-effort dropdown for deepseek-* models only surfaced high and
max, so the cheap lane could never be pushed cheaper. Add low to the option
list (default stays high) and lock the adapter passthrough with a test that
reasoning_effort: "low" reaches the request body.
Co-Authored-By: Claude <noreply@anthropic.com>
|
2c1eb539 | Score the paragraph spine as a global monotone chain | Eli Ziff | 2026-08-06 | ↗ GitHub |
commit body The spine was picked by walking line-start markers and accepting the first
monotone run, so a marker joined onto the end of a heading line
("II. Analysis [12] The appellant argues...") was invisible: the run
stopped one short, or started partway down at whatever number appeared
first.
Replaces the walk with the mechanism the footnote pairer already uses:
enumerate every candidate for every number, price each by the evidence
behind it (line start, heading-joined, adjacent to a confirmed link), and
take the highest-scoring increase-by-one chain over the document as a
whole. Evidence is priced, not gated. The chain must root at 1 and have no
gaps, and endnote ladders are detected and declined - a numbered reference
list in the last quarter of a judgment reads as a perfect 1..N ladder and
is not paragraphs.
The heading grammar stops rejecting enumerated levels on length alone: an
author who numbered a level has already said it is a heading, so it keeps
the long word cap rather than the short one meant for levels with nothing
but brevity to recommend them.
Over all 224,972 A2AJ case documents: 2,090 gain a spine, 2,709 lose one,
29,899 relabelled, net +562,044 paragraphs. Of the losses, 2,103 were
never rooted at 1 and 563 were endnote ladders. Legislation is
byte-identical; CourtListener over its own 55,504 opinions is 55,489
identical, 8 gained, 2 lost. Heading recall, scored against labels derived
from document position rather than from the grammar, rises 73.70% to
90.94% with the strict negative set still at exactly zero false accepts.
Four sourcedoc fixtures were mid-decision excerpts - one opening 44,435
characters into Sattva - that no chain rooted at paragraph 1 could match;
they are replaced with the full A2AJ records, so the beaver-can manifests
pinning their hashes are refreshed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VyTUeXhYSK9awyArAtm83c
|
d972f3e3 | fix(llm): retry deepseek rounds on zero-progress transport failures | Eli Ziff | 2026-08-06 | ↗ GitHub |
commit body A single undici "TypeError: terminated" (mid-stream TLS reset) killed a
40-minute flash lab run at top level - the deepseek adapter had no retry
at all, unlike the openai/claude-p lanes. Wrap each round's request +
SSE consumption in a classified retry (3 attempts, 2s/4s backoff):
transport-class errors only (socket/DNS errnos, undici terminated/fetch
failed, HTTP 429/5xx), and only while zero deltas have been forwarded -
consumers accumulate deltas into the persisted turn, so post-emission
retries would duplicate text and resampled output cannot be stitched.
The exposed window is dominated by silent prefill on large contexts,
which is exactly the zero-progress case. Retries are trace-recorded as
"transport_retry". Semantic errors (4xx, provider chunk.error) stay
fatal; classifier probed against the captured error shapes (10/10).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011pfUVhNFTRvhYGXBwoKNj6
|
0a66c8e6 | Add legal primitives experiment suite | Eli Ziff | 2026-08-08 | ↗ GitHub |