Stage 21 result: address nav surface undecided; its trigger fired; three of four extra affordances never called
881 scored agentic cells on LegalBench-RAG mini. Paired n=215 (asis, rep 1 at the full pre-registered n=160 per arm) and n=127 (stripped). Both bed oracles passed before the first model call. Outcome metrics are undecided: f1_best +0.0105 [-0.0028, 0.0242] against a within-arm replicate floor of 0.028-0.045, tool calls -0.20 against a floor of 0.81-1.08, reached_by_read +0.0419 against a floor of 0.13-0.14. The registered disqualification trigger DID fire - reached_any -0.0977 [-0.1404, -0.0591] - and is recorded as fired; the same metric's within-arm floor (0.162) is larger than the gap, so both registered rules fired at once and both are reported. The one difference that clears its own floor and reproduces in both query forms: the address arm exposes 48% less document for the same answer quality (5,818 vs 11,130 chars), which also makes it cheaper in total input tokens despite carrying 2,197 more schema chars. Affordance uptake, out of 2,085 address-arm tool calls: library_links 0.0%, follow actually walked 0.2%, find scoping 0.4%, tail reads 0.3%, page addressing 0.0% (structurally inert on this unpaginated bed, as pre-registered). The model filled in follow="none", depth=1 on every find call - paying for the graph walk and declining it 1,347 times out of 1,350. Document length does not predict an arm advantage: in the navigation-mandatory band the arms are dead level. Contamination instrument clean: zero off-schema parameters in 4,117 tool calls. 27.4% of asis cells were never solved by either arm, and the read-it-whole shortcut, available on 67.5% of cells, was taken in 5 of 430. 354 cells lost to Codex rate limiting are excluded from scoring rather than scored as zero. Nothing retired. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | e0820eac |
| Stats | 1 file changed , +138 |
| Part of | Legal agent-surface and context-compaction experiments |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-538072b0.md
from inside the repo you want the change in.