fix(lab): isolate claude -p auth env at a single choke point

↗ view on GitHub · Eli Ziff · 2026-08-05 · f162758f

Judge/scoring/adapter spawned `claude -p` with the parent env intact, so a
run launched from a `deepclaude` shell inherited ANTHROPIC_BASE_URL +
ANTHROPIC_AUTH_TOKEN and left the flat-rate subscription entirely: DeepSeek
mapped the unknown id `claude-opus-5` onto its flagship and billed
deepseek-v4-pro. Measured on the Aug 3-4 rounds: ~3,074 calls, ~110M input
tokens, cache_read=0 (every `claude -p` is a fresh process, so nothing
cached). The deepclaude flash pins never applied - ANTHROPIC_DEFAULT_*_MODEL
remaps aliases, and these call sites pass explicit full model ids.

- utils/claude_cli.py: new. auth_isolated_env() (mirrors claudeP.ts),
  resolve_cli(), run()/run_json(), and verify_served_model() - which fails
  loudly when a non-Anthropic model appears in the result envelope's
  modelUsage, catching the next leak whatever its mechanism.
- judge.py, scoring.py: spawn via claude_cli; verify every envelope.
  ForeignModelError is a RuntimeError so the ValueError retry loops cannot
  swallow it and retry at the proxy's expense; scoring.py re-raises it past
  its broad except rather than degrading it to "matching failed".
- adapters/claude_code.py: was stripping only ANTHROPIC_API_KEY, which left
  the routing vars intact; now uses the shared isolated env and verifies the
  streamed result envelope.

Verified: with a full deepclaude-style env poisoned in the parent, both
claude_cli.run_json and Judge(claude-code/...) are served by
claude-sonnet-4-6 / provider=firstParty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EvZhii5uiXxeUbJHxcdfd8
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents 76e54625
Stats 4 files changed , +199 , -43
Part of Evaluation harness: Beaver-CAN and LegalBench-RAG adapters

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-f162758f.md from inside the repo you want the change in.

⬇ Download capture-commit-f162758f.md