Stage 21 harness: exclude unrecovered transport failures from scoring
A cell whose only receipt row is a 429 was being scored as f1=0, so a rate limit was masquerading as the arm failing the cell and was moving the arm mean by however many cells the network happened to drop. Error rows now leave the scored set entirely and are reported as a count. Adds two paired controls for the navigation claim: chars_exposed (union of every document character a cell put in front of the model) and mean_read_width. reached_by_read would otherwise be cheap to win by reading wider windows rather than by targeting better. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | c6996918 |
| Stats | 1 file changed , +55 , -5 |
| Part of | Evaluation harness: Beaver-CAN and LegalBench-RAG adapters |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-7e5b9037.md
from inside the repo you want the change in.