Stage 21 CORRECTION: arms drifted mid-run; the fired trigger was an artifact
A concurrent session edited backend/src/lib/chat/localAssistantTools.ts three times between my registration commit (19:30:42) and my result (19:57:34). The harness starts a fresh process per condition, so each condition picked up whatever the file said at its start: the receipts hold FOUR distinct arm B schemas and TWO distinct arm A schemas, and the published n=215 table averaged all of them. Arm A was supposed to be frozen and was not. schema_chars was recorded on every row, which is the only reason this was detectable rather than silently averaged. The report now prints a schema-identity census that shouts when an arm drifted, and takes --pin-schema to score one frozen pairing. Scored on the registered arms alone (legacy=2809ch, address=5006ch, paired n=100 asis / n=34 stripped), two headline claims are withdrawn: the disqualification trigger does NOT fire (reached_any is 1.0000 in both arms, difference exactly 0, against -0.0977 [-0.1404, -0.0591] in the mixed set), and the 48% document-exposure reduction disappears (-352 [-1122, 461] against -5,312). Every remaining point estimate favours arm B and every one sits inside its own within-arm replicate floor: undecided, leaning positive, resolvable only with more replicates. Unchanged because they are counts, not contrasts: on the registered arm B, library_links was called 0 times in 849 tool calls, follow walked the graph 0 times in 534 find calls, page addressing 0 times, and off-schema parameter usage was 0 across 1,560 calls. Lesson: verifying the arms once before a run is not enough on a shared tree. Pin the arm by content hash at process start and refuse to append a row whose hash differs from the registered one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | fae74a2f |
| Stats | 2 files changed , +128 , -1 |
| Part of | Evaluation harness: Beaver-CAN and LegalBench-RAG adapters |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-eae36b2c.md
from inside the repo you want the change in.