Stage 21 amendment: unsolvable-cell rate, shortcut census, undecided label
Registered after the asis rep-1 arms started but before any aggregate score was read (disclosed in the doc). Prompted by an agentic-benchmark audit; three reporting additions, no change to arms, prompt, sample, model or scorer. A cell counts as solved at f1_best >= 0.5. Cells no arm ever solves are counted and printed as their own number, and every paired difference is reported twice: over all sampled cells, and over the subset at least one arm solved. Zero-in-both cells pull a paired difference toward zero and would otherwise read as arm equality when they are signal absence. The read-it-whole shortcut is measured, and it is large and confounded with source: against library_read's 24,000-char default and the 64,000 tool-result ceiling, 39.4% of the n=160 sample fits one default read and another 28.1% fits one deliberate read, so 67.5% of cells have a whole-document shortcut available. contractnli is 38/40 in the smallest band; maud is 40/40 above 200k. Paired differences are therefore reported stratified by document-size band, with the shortcut-TAKEN rate (one read covering >=95% of the document) beside it. Differences smaller than the measured within-arm replicate floor are labelled undecided rather than null. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | fdb7fb60 |
| Stats | 2 files changed , +134 |
| Part of | Evaluation harness: Beaver-CAN and LegalBench-RAG adapters |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-d0a311df.md
from inside the repo you want the change in.