Structure detection: segmentations compete, and a contents page is not one
The cross-reference resolver's ceiling was never its grammar; it was the skeleton's section inventory. Every structural grammar in legalTextSkeleton keys on a LINE START, and seven of the seventeen maud merger agreements have no line starts to key on: their extraction joined each page into one line, mean line length 226-310 characters against 98-183 for their well-lineated siblings. Those seven detect 13-33 sections where the others detect 78-111, and all seven are refused. compileAgreementSkeleton now compiles the document under more than one segmentation and lets the document choose. Recovery is offset-exact by construction - the first space of an internal run becomes a newline and the rest of the run stays, so the text is the same length character for character and differs only in whitespace; the SourceDoc, the defined terms and every span are still built from the ORIGINAL text whichever reading wins. Nothing about any grammar or guard changes: discoverNodes applies headingLike, the integer ceiling, spine ownership and the ladder to a recovered line exactly as to a real one. Two guards, because the first cut of this was wrong in two ways and the corpus said so: A reference may not endorse a provision minted out of ITSELF. Offer a line start before "... See also Section 9.99 for notices." and that prose becomes a head, the reference resolves onto it, and the reading that invented it scores higher - a selector rewarding a detector for hallucinating exactly the provision the document lacks. Targets beginning inside the reference span do not count. A hypothesis must produce heads that span the document. Space-run recovery reveals a table of contents preferentially (a contents line is padded with runs, a body heading is not), and 80-99 contents entries resolve to each other beautifully. Head span over the 69 documents: 0.0077, 0.0079, 0.0083, 0.0122 for contents-only inventories, then 0.1237, then 0.30-1.00. Gate at 0.05, in the 10x gap. MEASURED AND REJECTED, recorded in the module so it is not retried blind: offering a line start wherever a head grammar could begin recovers the single-space extraction dialect and four more documents, and is unsound. A merger agreement's definitions index is a list of "'Balance Sheet Date' has the meaning set forth in Section 6.16(a)" - each entry becomes a head that real references elsewhere resolve onto. CAI International compiles 272 heads against roughly 100 real ones and its endorsement score RISES. Endorsement cannot police a hypothesis that mints the provisions being sought. legalCrossReference gains the reach gate the same measurement exposed, and it catches a defect that predates any of this: Acacia Communications compiles 85 contents heads with NO recovery at all, resolves 89 references onto them, and reports integrity 1.00. Every target sits at 0.01-0.02 of the document; every document whose targets are real provisions reaches 0.16 or beyond. A graph with no reach has resolved into a contents page and now says so. integrityThreshold 0 still yields the raw census. CENSUS, 69 LegalBench-RAG mini documents, zero model calls, question-blind (before -> after): documents refused 24 -> 19, references resolved 2,428 -> 3,891 (+60%), accepted references that miss 11.2% -> 10.6%. Detected (9,234) and external (1,720) are unchanged, which is the control: no reference grammar moved. Per source, refused: maud 8 -> 5, cuad 7 -> 5, contractnli 3 -> 3, privacy_qa 6 -> 6; maud resolved 2,085 -> 3,494, cuad 320 -> 374. Of the five maud documents still refused, three are refused BECAUSE of this work rather than in spite of it: Acacia, Boingo and Anworth have no visible body structure at all, and the reading that made them look healthy was their contents page. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Committed | |
| Parents | 28284f04 |
| Stats | 7 files changed , +562 , -10 |
| Part of | SourceDoc / DOCX engine: redlines, tracked changes, editing benchmark |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-bdbab160.md
from inside the repo you want the change in.