docs(lab): requirements echo on tax = NULL - 47/77 vs no-echo v5 mean 45.9 (judge floor ~10)
T2 reqecho contrast arm (coding_markdown_v5_reqecho_v1 @ 20-01-25)
judged deepseek-v4-flash. Mechanism verified: fetch_requirements
re-served the verbatim 202-char task prompt once (echo_call_count 1),
coverage gate fired (9 unread). Score 47/77 = +1.1 vs no-echo v5
deepseek {45,42,53,48,51,38,48,42} - under the ~10-criterion tax
judge-spread floor (d153d596), so a draw. Criteria profile is a
different 47 (+7 wins incl C-001 risk table, -6 losses incl C-030
8/8->fail): run-to-run variance, not a mechanism signature, since
the echoed prompt carries no structural content. Cost +10% cacheadj.
Echo only testable on a requirements-rich prompt; draft-mode stays
parked.
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | 496edf4f |
| Stats | 1 file changed , +41 |
| Part of | Evaluation harness: Beaver-CAN and LegalBench-RAG adapters |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-1b6a8900.md
from inside the repo you want the change in.