Structure detection: segmentations compete, and a contents page is not one

↗ view on GitHub · Eli Ziff · 2026-07-31 · bdbab160

The cross-reference resolver's ceiling was never its grammar; it was the
skeleton's section inventory. Every structural grammar in legalTextSkeleton
keys on a LINE START, and seven of the seventeen maud merger agreements
have no line starts to key on: their extraction joined each page into one
line, mean line length 226-310 characters against 98-183 for their
well-lineated siblings. Those seven detect 13-33 sections where the others
detect 78-111, and all seven are refused.

compileAgreementSkeleton now compiles the document under more than one
segmentation and lets the document choose. Recovery is offset-exact by
construction - the first space of an internal run becomes a newline and the
rest of the run stays, so the text is the same length character for
character and differs only in whitespace; the SourceDoc, the defined terms
and every span are still built from the ORIGINAL text whichever reading
wins. Nothing about any grammar or guard changes: discoverNodes applies
headingLike, the integer ceiling, spine ownership and the ladder to a
recovered line exactly as to a real one.

Two guards, because the first cut of this was wrong in two ways and the
corpus said so:

  A reference may not endorse a provision minted out of ITSELF. Offer a line
  start before "... See also Section 9.99 for notices." and that prose
  becomes a head, the reference resolves onto it, and the reading that
  invented it scores higher - a selector rewarding a detector for
  hallucinating exactly the provision the document lacks. Targets beginning
  inside the reference span do not count.

  A hypothesis must produce heads that span the document. Space-run recovery
  reveals a table of contents preferentially (a contents line is padded with
  runs, a body heading is not), and 80-99 contents entries resolve to each
  other beautifully. Head span over the 69 documents: 0.0077, 0.0079,
  0.0083, 0.0122 for contents-only inventories, then 0.1237, then 0.30-1.00.
  Gate at 0.05, in the 10x gap.

MEASURED AND REJECTED, recorded in the module so it is not retried blind:
offering a line start wherever a head grammar could begin recovers the
single-space extraction dialect and four more documents, and is unsound. A
merger agreement's definitions index is a list of "'Balance Sheet Date' has
the meaning set forth in Section 6.16(a)" - each entry becomes a head that
real references elsewhere resolve onto. CAI International compiles 272 heads
against roughly 100 real ones and its endorsement score RISES. Endorsement
cannot police a hypothesis that mints the provisions being sought.

legalCrossReference gains the reach gate the same measurement exposed, and
it catches a defect that predates any of this: Acacia Communications
compiles 85 contents heads with NO recovery at all, resolves 89 references
onto them, and reports integrity 1.00. Every target sits at 0.01-0.02 of the
document; every document whose targets are real provisions reaches 0.16 or
beyond. A graph with no reach has resolved into a contents page and now says
so. integrityThreshold 0 still yields the raw census.

CENSUS, 69 LegalBench-RAG mini documents, zero model calls, question-blind
(before -> after): documents refused 24 -> 19, references resolved 2,428 ->
3,891 (+60%), accepted references that miss 11.2% -> 10.6%. Detected (9,234)
and external (1,720) are unchanged, which is the control: no reference
grammar moved. Per source, refused: maud 8 -> 5, cuad 7 -> 5, contractnli
3 -> 3, privacy_qa 6 -> 6; maud resolved 2,085 -> 3,494, cuad 320 -> 374.

Of the five maud documents still refused, three are refused BECAUSE of this
work rather than in spite of it: Acacia, Boingo and Anworth have no visible
body structure at all, and the reading that made them look healthy was
their contents page.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Committed
Parents 28284f04
Stats 7 files changed , +562 , -10
Part of SourceDoc / DOCX engine: redlines, tracked changes, editing benchmark

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-bdbab160.md from inside the repo you want the change in.

⬇ Download capture-commit-bdbab160.md