Intra-document cross-reference graph, and the two measurements that judge it
legalCrossReference compiles a contract's own pointers into a typed graph:
nodes are skeleton provisions, edges carry the reference span, the
normalized locator, the target span, and a status of resolved / external /
unresolved / abstained. Zero model calls. Nothing is wired into retrieval.
Refusals, each with the corpus number that motivated it:
ambiguous_label a table of contents repeats every heading, so the
label claims two spans and createSourceDoc dropped
it; refuse rather than pick (161 references)
depth_not_numbered the document does not number this FAMILY - a merger
agreement anchors sec4.2 and art6 but never a bare
"sec16", so its 1,357 bare-integer references (all
external statute cites) must abstain (2,081)
no_containing_section "clause (ii)" outside any numbered section (98)
document_abstained the integrity gate below
Integrity gate: per-document resolution is sharply bimodal on maud (0.24
-0.40 for eight documents whose sections the skeleton largely misses, then
0.58-1.00 for nine), so a document whose accepted references mostly miss
refuses wholesale - what is unreliable there is our view of the numbering,
not the document. Gate at 0.5, in the widest gap; a calibration, re-derive
it when the structural detector changes.
MEASUREMENT 1 (resolver, no gold needed), 9,234 references over 69 docs:
gated, 2,428 resolved / 307 unresolved / 1,720 external / 4,779 refused -
3.3% of accepted references miss. Ungated the same census is 3,341 / 1,812
/ 1,720 / 2,361. The instrument is sound on the documents it accepts, and
its ceiling is the skeleton's section inventory: maud documents with 97-111
detected sections resolve 96-97% of accepted references, those with 13-33
resolve 24-40%.
MEASUREMENT 2 (gold n graph alignment) is the negative half and is why no
arm is registered here. Standing on a gold fragment and following outgoing
literal edges reaches OTHER gold fragments at 2.28% precision on maud vs
1.44% for the same budget spent contiguously (1.6x better), but 1.15% vs
6.99% on cuad and 4.06% vs 5.06% on contractnli. Coverage is the harder
limit: 31 of 227 fragmented maud gold fragments have any outgoing edge,
14/153 on cuad, 0/384 on privacy_qa. Both weak layers lose to contiguous
context everywhere (defined-term 0.12-1.29%, lexical 0.81-4.57%).
Also in legalReferenceGrammar: isExternalReferenceInContext, a both-flanks
external test the one-flank isExternalReference is structurally blind to
("Code Section 59A", "Treasury Regulation Section 1.482", "Sections 302 and
906 of the Sarbanes-Oxley Act", "Section 262 thereof"). +313 external
references on the mini corpus at a cost of 2, both verified genuinely
external by hand. isExternalReference itself is untouched, so the three
shipping consumers keep their behaviour.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | 4c90e0ac |
| Stats | 3 files changed , +892 , -1 |
| Part of | Evaluation harness: Beaver-CAN and LegalBench-RAG adapters |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-2d903c82.md
from inside the repo you want the change in.