Stage 21 amendment: unsolvable-cell rate, shortcut census, undecided label

↗ view on GitHub · Eli Ziff · 2026-07-31 · d0a311df

Registered after the asis rep-1 arms started but before any aggregate
score was read (disclosed in the doc). Prompted by an agentic-benchmark
audit; three reporting additions, no change to arms, prompt, sample,
model or scorer.

A cell counts as solved at f1_best >= 0.5. Cells no arm ever solves are
counted and printed as their own number, and every paired difference is
reported twice: over all sampled cells, and over the subset at least
one arm solved. Zero-in-both cells pull a paired difference toward zero
and would otherwise read as arm equality when they are signal absence.

The read-it-whole shortcut is measured, and it is large and confounded
with source: against library_read's 24,000-char default and the 64,000
tool-result ceiling, 39.4% of the n=160 sample fits one default read
and another 28.1% fits one deliberate read, so 67.5% of cells have a
whole-document shortcut available. contractnli is 38/40 in the smallest
band; maud is 40/40 above 200k. Paired differences are therefore
reported stratified by document-size band, with the shortcut-TAKEN rate
(one read covering >=95% of the document) beside it.

Differences smaller than the measured within-arm replicate floor are
labelled undecided rather than null.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents fdb7fb60
Stats 2 files changed , +134
Part of Evaluation harness: Beaver-CAN and LegalBench-RAG adapters

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-d0a311df.md from inside the repo you want the change in.

⬇ Download capture-commit-d0a311df.md