fix(lab): isolate claude -p auth env at a single choke point
Judge/scoring/adapter spawned `claude -p` with the parent env intact, so a run launched from a `deepclaude` shell inherited ANTHROPIC_BASE_URL + ANTHROPIC_AUTH_TOKEN and left the flat-rate subscription entirely: DeepSeek mapped the unknown id `claude-opus-5` onto its flagship and billed deepseek-v4-pro. Measured on the Aug 3-4 rounds: ~3,074 calls, ~110M input tokens, cache_read=0 (every `claude -p` is a fresh process, so nothing cached). The deepclaude flash pins never applied - ANTHROPIC_DEFAULT_*_MODEL remaps aliases, and these call sites pass explicit full model ids. - utils/claude_cli.py: new. auth_isolated_env() (mirrors claudeP.ts), resolve_cli(), run()/run_json(), and verify_served_model() - which fails loudly when a non-Anthropic model appears in the result envelope's modelUsage, catching the next leak whatever its mechanism. - judge.py, scoring.py: spawn via claude_cli; verify every envelope. ForeignModelError is a RuntimeError so the ValueError retry loops cannot swallow it and retry at the proxy's expense; scoring.py re-raises it past its broad except rather than degrading it to "matching failed". - adapters/claude_code.py: was stripping only ANTHROPIC_API_KEY, which left the routing vars intact; now uses the shared isolated env and verifies the streamed result envelope. Verified: with a full deepclaude-style env poisoned in the parent, both claude_cli.run_json and Judge(claude-code/...) are served by claude-sonnet-4-6 / provider=firstParty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EvZhii5uiXxeUbJHxcdfd8
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | 76e54625 |
| Stats | 4 files changed , +199 , -43 |
| Part of | Evaluation harness: Beaver-CAN and LegalBench-RAG adapters |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-f162758f.md
from inside the repo you want the change in.