feat(eval): k-fold judging, no-clobber scores, majority aggregation, match telemetry
Audit fix wave A6:
- run_eval gains --judge-samples K (writes scores.k1..kK.json, skips
existing samples), --out-suffix, and --force; an existing scores file is
never overwritten without --force. Re-judging used to clobber scores.json
in place, which destroyed the prior sample and made judge-only variance
unmeasurable.
- New evaluation/aggregate_judgments.py: per-criterion majority verdicts
across samples + flip telemetry (n_flipping_criteria, criterion_flip_rate
- the judge noise floor). Refuses cross-judge aggregation.
- scoring.py records deliverable resolution method per deliverable
("exact" / "sole_extension" / "fuzzy:<overlap>" / "llm" / "unmatched")
into scores.json - an unmatched deliverable mass-fails every criterion
tied to it and was previously invisible.
- Env hygiene was already handled by utils/claude_cli.py (auth-isolated
spawns + served-model verification); no change needed there.
tests/test_scoring.py updated for the tuple return; 29 pass (6 prior errors
are a machine-local pytest basetemp permission issue, green with --basetemp).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011pfUVhNFTRvhYGXBwoKNj6
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | e14b83b8 |
| Stats | 4 files changed , +225 , -19 |
| Part of | Evaluation harness: Beaver-CAN and LegalBench-RAG adapters |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-207ab5a7.md
from inside the repo you want the change in.