Stage 22 registration: arm B as the whole product bet; token census done, model runs BLOCKED on a provider-loop seam
Registered before any Stage 22 model call. Arm B is no longer a naming change: it adds progressive tool disclosure and capability-on-contact, so Stage 21 measured a strictly smaller thing (four nav tools sent alone) and its result does not carry over. Deterministic half is done and verified by probe rather than taken on trust. Provider-measured input tokens on codex:gpt-5.6-sol, same system prompt, tools attached vs not: arm A 43 tools / 38,936 chars / 6,709 tokens; arm B 11 resident tools / 15,290 chars / 2,888 tokens, with 34 tools deferred behind describe_tools. That is -3,821 tokens, -57.0% on the first request. The brief quoted 10,130 -> 4,004 (-60.5%); both are recorded, and the measured pair is the one this stage will use because it is what this model is actually billed. Model runs are blocked. Progressive disclosure needs the tool list to grow after describe_tools fires, and it cannot through the public API: openai.ts:268 snapshots toResponseTools(tools) before the loop at 298 and sends that snapshot every iteration, and LlmMessage has no tool-call or tool-result role, so a harness cannot drive its own turn-by-turn loop either. The fix is one line in backend/src/lib/llm - move toResponseTools inside the iteration and read a mutable ref - or a duplicate provider loop in scripts, which the one-implementation rule forbids. Reported, not fixed: this stage does not touch backend/src. Also enforces the Stage 21 lesson. navSchemas() now hashes the exact schemas about to be sent, --expect-hash refuses to start when they differ (verified: it aborts with ARM DRIFT), and every row records schema_hash. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | eae36b2c |
| Stats | 2 files changed , +138 , -2 |
| Part of | Evaluation harness: Beaver-CAN and LegalBench-RAG adapters |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-fd9ea3a3.md
from inside the repo you want the change in.