Stage 22 registration: arm B as the whole product bet; token census done, model runs BLOCKED on a provider-loop seam

↗ view on GitHub · Eli Ziff · 2026-07-31 · fd9ea3a3

Registered before any Stage 22 model call. Arm B is no longer a naming
change: it adds progressive tool disclosure and capability-on-contact,
so Stage 21 measured a strictly smaller thing (four nav tools sent
alone) and its result does not carry over.

Deterministic half is done and verified by probe rather than taken on
trust. Provider-measured input tokens on codex:gpt-5.6-sol, same system
prompt, tools attached vs not: arm A 43 tools / 38,936 chars / 6,709
tokens; arm B 11 resident tools / 15,290 chars / 2,888 tokens, with 34
tools deferred behind describe_tools. That is -3,821 tokens, -57.0% on
the first request. The brief quoted 10,130 -> 4,004 (-60.5%); both are
recorded, and the measured pair is the one this stage will use because
it is what this model is actually billed.

Model runs are blocked. Progressive disclosure needs the tool list to
grow after describe_tools fires, and it cannot through the public API:
openai.ts:268 snapshots toResponseTools(tools) before the loop at 298
and sends that snapshot every iteration, and LlmMessage has no
tool-call or tool-result role, so a harness cannot drive its own
turn-by-turn loop either. The fix is one line in backend/src/lib/llm -
move toResponseTools inside the iteration and read a mutable ref - or a
duplicate provider loop in scripts, which the one-implementation rule
forbids. Reported, not fixed: this stage does not touch backend/src.

Also enforces the Stage 21 lesson. navSchemas() now hashes the exact
schemas about to be sent, --expect-hash refuses to start when they
differ (verified: it aborts with ARM DRIFT), and every row records
schema_hash.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents eae36b2c
Stats 2 files changed , +138 , -2
Part of Evaluation harness: Beaver-CAN and LegalBench-RAG adapters

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-fd9ea3a3.md from inside the repo you want the change in.

⬇ Download capture-commit-fd9ea3a3.md