docx-edit-bench: drive progressive tool disclosure from the runner

↗ view on GitHub · Eli Ziff · 2026-07-31 · 08389cf2

Arm B now ships 10 resident tools and defers 19 behind describe_tools.
The provider adapter freezes its tool list when a call starts, so a domain
opened mid-call cannot become callable inside it; the runner ends the call
at the disclosure and continues in a new one, replaying the exchange. The
replay cost is counted. A surface that defers nothing never restarts, so
one path serves both arms.

A call to a tool the surface has not served is refused by the runner rather
than executed - the handlers dispatch on name alone, so without that guard
a guessed name would make the deferral measure as free.

Two arm-agnostic task fields: resident_route_exists (true for all 27 - every
reference solution is a literal substitution the resident text-op tool can
execute, which bounds what this bed can say about deferral) and
alternative_route_domains. Task version 1 -> 2; checks unchanged, self-test
still 27/27 solvable and 27/27 rejecting every wrong result.

Registration amended (2026-07-31a) with the redefined arm, the measured
48.5% first-request schema saving under the sealed flag rather than the
60.5% of the unsealed catalogue, and the seven superseded receipts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents 9572d3ca
Stats 9 files changed , +441 , -117
Part of SourceDoc / DOCX engine: redlines, tracked changes, editing benchmark

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-08389cf2.md from inside the repo you want the change in.

⬇ Download capture-commit-08389cf2.md