Intra-document cross-reference graph, and the two measurements that judge it

↗ view on GitHub · Eli Ziff · 2026-07-31 · 2d903c82

legalCrossReference compiles a contract's own pointers into a typed graph:
nodes are skeleton provisions, edges carry the reference span, the
normalized locator, the target span, and a status of resolved / external /
unresolved / abstained. Zero model calls. Nothing is wired into retrieval.

Refusals, each with the corpus number that motivated it:
  ambiguous_label      a table of contents repeats every heading, so the
                       label claims two spans and createSourceDoc dropped
                       it; refuse rather than pick (161 references)
  depth_not_numbered   the document does not number this FAMILY - a merger
                       agreement anchors sec4.2 and art6 but never a bare
                       "sec16", so its 1,357 bare-integer references (all
                       external statute cites) must abstain (2,081)
  no_containing_section  "clause (ii)" outside any numbered section (98)
  document_abstained   the integrity gate below

Integrity gate: per-document resolution is sharply bimodal on maud (0.24
-0.40 for eight documents whose sections the skeleton largely misses, then
0.58-1.00 for nine), so a document whose accepted references mostly miss
refuses wholesale - what is unreliable there is our view of the numbering,
not the document. Gate at 0.5, in the widest gap; a calibration, re-derive
it when the structural detector changes.

MEASUREMENT 1 (resolver, no gold needed), 9,234 references over 69 docs:
gated, 2,428 resolved / 307 unresolved / 1,720 external / 4,779 refused -
3.3% of accepted references miss. Ungated the same census is 3,341 / 1,812
/ 1,720 / 2,361. The instrument is sound on the documents it accepts, and
its ceiling is the skeleton's section inventory: maud documents with 97-111
detected sections resolve 96-97% of accepted references, those with 13-33
resolve 24-40%.

MEASUREMENT 2 (gold n graph alignment) is the negative half and is why no
arm is registered here. Standing on a gold fragment and following outgoing
literal edges reaches OTHER gold fragments at 2.28% precision on maud vs
1.44% for the same budget spent contiguously (1.6x better), but 1.15% vs
6.99% on cuad and 4.06% vs 5.06% on contractnli. Coverage is the harder
limit: 31 of 227 fragmented maud gold fragments have any outgoing edge,
14/153 on cuad, 0/384 on privacy_qa. Both weak layers lose to contiguous
context everywhere (defined-term 0.12-1.29%, lexical 0.81-4.57%).

Also in legalReferenceGrammar: isExternalReferenceInContext, a both-flanks
external test the one-flank isExternalReference is structurally blind to
("Code Section 59A", "Treasury Regulation Section 1.482", "Sections 302 and
906 of the Sarbanes-Oxley Act", "Section 262 thereof"). +313 external
references on the mini corpus at a cost of 2, both verified genuinely
external by hand. isExternalReference itself is untouched, so the three
shipping consumers keep their behaviour.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents 4c90e0ac
Stats 3 files changed , +892 , -1
Part of Evaluation harness: Beaver-CAN and LegalBench-RAG adapters

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-2d903c82.md from inside the repo you want the change in.

⬇ Download capture-commit-2d903c82.md