Stage 21 result: address nav surface undecided; its trigger fired; three of four extra affordances never called

↗ view on GitHub · Eli Ziff · 2026-07-31 · 538072b0

881 scored agentic cells on LegalBench-RAG mini. Paired n=215 (asis,
rep 1 at the full pre-registered n=160 per arm) and n=127 (stripped).
Both bed oracles passed before the first model call.

Outcome metrics are undecided: f1_best +0.0105 [-0.0028, 0.0242]
against a within-arm replicate floor of 0.028-0.045, tool calls -0.20
against a floor of 0.81-1.08, reached_by_read +0.0419 against a floor
of 0.13-0.14. The registered disqualification trigger DID fire -
reached_any -0.0977 [-0.1404, -0.0591] - and is recorded as fired; the
same metric's within-arm floor (0.162) is larger than the gap, so both
registered rules fired at once and both are reported.

The one difference that clears its own floor and reproduces in both
query forms: the address arm exposes 48% less document for the same
answer quality (5,818 vs 11,130 chars), which also makes it cheaper in
total input tokens despite carrying 2,197 more schema chars.

Affordance uptake, out of 2,085 address-arm tool calls: library_links
0.0%, follow actually walked 0.2%, find scoping 0.4%, tail reads 0.3%,
page addressing 0.0% (structurally inert on this unpaginated bed, as
pre-registered). The model filled in follow="none", depth=1 on every
find call - paying for the graph walk and declining it 1,347 times out
of 1,350. Document length does not predict an arm advantage: in the
navigation-mandatory band the arms are dead level.

Contamination instrument clean: zero off-schema parameters in 4,117
tool calls. 27.4% of asis cells were never solved by either arm, and
the read-it-whole shortcut, available on 67.5% of cells, was taken in
5 of 430. 354 cells lost to Codex rate limiting are excluded from
scoring rather than scored as zero. Nothing retired.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents e0820eac
Stats 1 file changed , +138
Part of Legal agent-surface and context-compaction experiments

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-538072b0.md from inside the repo you want the change in.

⬇ Download capture-commit-538072b0.md