Stage 21 CORRECTION: arms drifted mid-run; the fired trigger was an artifact

↗ view on GitHub · Eli Ziff · 2026-07-31 · eae36b2c

A concurrent session edited backend/src/lib/chat/localAssistantTools.ts
three times between my registration commit (19:30:42) and my result
(19:57:34). The harness starts a fresh process per condition, so each
condition picked up whatever the file said at its start: the receipts
hold FOUR distinct arm B schemas and TWO distinct arm A schemas, and
the published n=215 table averaged all of them. Arm A was supposed to
be frozen and was not.

schema_chars was recorded on every row, which is the only reason this
was detectable rather than silently averaged. The report now prints a
schema-identity census that shouts when an arm drifted, and takes
--pin-schema to score one frozen pairing.

Scored on the registered arms alone (legacy=2809ch, address=5006ch,
paired n=100 asis / n=34 stripped), two headline claims are withdrawn:
the disqualification trigger does NOT fire (reached_any is 1.0000 in
both arms, difference exactly 0, against -0.0977 [-0.1404, -0.0591] in
the mixed set), and the 48% document-exposure reduction disappears
(-352 [-1122, 461] against -5,312). Every remaining point estimate
favours arm B and every one sits inside its own within-arm replicate
floor: undecided, leaning positive, resolvable only with more
replicates.

Unchanged because they are counts, not contrasts: on the registered
arm B, library_links was called 0 times in 849 tool calls, follow
walked the graph 0 times in 534 find calls, page addressing 0 times,
and off-schema parameter usage was 0 across 1,560 calls.

Lesson: verifying the arms once before a run is not enough on a shared
tree. Pin the arm by content hash at process start and refuse to append
a row whose hash differs from the registered one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents fae74a2f
Stats 2 files changed , +128 , -1
Part of Evaluation harness: Beaver-CAN and LegalBench-RAG adapters

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-eae36b2c.md from inside the repo you want the change in.

⬇ Download capture-commit-eae36b2c.md