Stage 21 harness: retryable transport failures, stricter nav metric

↗ view on GitHub · Eli Ziff · 2026-07-31 · 6a6edacd

The Codex backend 429s under six concurrent harness processes, and the
per-cell catch wrote an error row that resume then treated as done -
baking a scored zero into the arm mean for a cell the model never got
to answer. Resume now keys on completed rows only, and since receipts
are append-only the retry appends a second row: the report keeps the
completed one, counts the error rows separately as an instrument
failure rate, and reports how many cells were de-duplicated.

Adds reached_by_read, derived from stored traces with no re-run.
reached_any counts library_find's context window either side of every
hit, which a couple of finds can blanket on a 10k document - it
saturates at 1.000 and stops discriminating. reached_by_read asks
whether a library_read, the model committing to a location, landed on
gold. On the partial asis pairing it separates the arms where
reached_any cannot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents 1e810f46
Stats 1 file changed , +82 , -8
Part of Evaluation harness: Beaver-CAN and LegalBench-RAG adapters

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-6a6edacd.md from inside the repo you want the change in.

⬇ Download capture-commit-6a6edacd.md