Ablation: the address grammar costs tokens; disclosure pays for it

↗ view on GitHub · Eli Ziff · 2026-07-31 · ddcb05cf

162 receipts, 27 tasks x 3 surfaces x 2 replicates, matched on every task.
beaver-address-no-disclosure holds the address grammar constant and removes
only progressive disclosure, which attributes the 19% the v1 headline
reported.

The finding inverts the natural reading. Served whole, the address grammar is
14.6% MORE expensive than the legacy surface it replaces (79,425 vs 69,281
tokens per run): library_links, describe_tools and the extra read/find
parameters make its catalogue 28,802 bytes against legacy's 25,809, and every
request carries the difference. Deferring 19 of 29 tools within the same
grammar takes it to 56,149 - a 29.3% saving - which nets out to the reported
19%. The lever is what ships in the first request, not the grammar.

A second-order effect is worth more than the tokens. Retyped document
characters on identical tasks: 327 with the full catalogue resident, 126 with
it deferred, and runs containing a misquote fall from 14/54 to 9/54. When
library_revise_docx is on the table the model reaches for it and retypes;
when it sits behind a domain the model uses the addressed tool instead. Every
misquote in this campaign came from a retyping-based argument, so deferral is
acting as a nudge toward the safer edit route, not merely as a saving.

Correctness is unmoved: 81/79/79 percent excluding floor tasks, one run of
forty-eight between best and worst, inside the replicate floor. No surface
made the model better at editing; they differ in cost and in route.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents b53c1353
Stats 1 file changed , +80
Part of SourceDoc / DOCX engine: redlines, tracked changes, editing benchmark

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-ddcb05cf.md from inside the repo you want the change in.

⬇ Download capture-commit-ddcb05cf.md