Ablation: the address grammar costs tokens; disclosure pays for it
162 receipts, 27 tasks x 3 surfaces x 2 replicates, matched on every task. beaver-address-no-disclosure holds the address grammar constant and removes only progressive disclosure, which attributes the 19% the v1 headline reported. The finding inverts the natural reading. Served whole, the address grammar is 14.6% MORE expensive than the legacy surface it replaces (79,425 vs 69,281 tokens per run): library_links, describe_tools and the extra read/find parameters make its catalogue 28,802 bytes against legacy's 25,809, and every request carries the difference. Deferring 19 of 29 tools within the same grammar takes it to 56,149 - a 29.3% saving - which nets out to the reported 19%. The lever is what ships in the first request, not the grammar. A second-order effect is worth more than the tokens. Retyped document characters on identical tasks: 327 with the full catalogue resident, 126 with it deferred, and runs containing a misquote fall from 14/54 to 9/54. When library_revise_docx is on the table the model reaches for it and retypes; when it sits behind a domain the model uses the addressed tool instead. Every misquote in this campaign came from a retyping-based argument, so deferral is acting as a nudge toward the safer edit route, not merely as a saving. Correctness is unmoved: 81/79/79 percent excluding floor tasks, one run of forty-eight between best and worst, inside the replicate floor. No surface made the model better at editing; they differ in cost and in route. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | b53c1353 |
| Stats | 1 file changed , +80 |
| Part of | SourceDoc / DOCX engine: redlines, tracked changes, editing benchmark |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-ddcb05cf.md
from inside the repo you want the change in.