feat(eval): k-fold judging, no-clobber scores, majority aggregation, match telemetry

↗ view on GitHub · Eli Ziff · 2026-08-05 · 207ab5a7

Audit fix wave A6:
- run_eval gains --judge-samples K (writes scores.k1..kK.json, skips
  existing samples), --out-suffix, and --force; an existing scores file is
  never overwritten without --force. Re-judging used to clobber scores.json
  in place, which destroyed the prior sample and made judge-only variance
  unmeasurable.
- New evaluation/aggregate_judgments.py: per-criterion majority verdicts
  across samples + flip telemetry (n_flipping_criteria, criterion_flip_rate
  - the judge noise floor). Refuses cross-judge aggregation.
- scoring.py records deliverable resolution method per deliverable
  ("exact" / "sole_extension" / "fuzzy:<overlap>" / "llm" / "unmatched")
  into scores.json - an unmatched deliverable mass-fails every criterion
  tied to it and was previously invisible.
- Env hygiene was already handled by utils/claude_cli.py (auth-isolated
  spawns + served-model verification); no change needed there.

tests/test_scoring.py updated for the tuple return; 29 pass (6 prior errors
are a machine-local pytest basetemp permission issue, green with --basetemp).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011pfUVhNFTRvhYGXBwoKNj6
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents e14b83b8
Stats 4 files changed , +225 , -19
Part of Evaluation harness: Beaver-CAN and LegalBench-RAG adapters

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-207ab5a7.md from inside the repo you want the change in.

⬇ Download capture-commit-207ab5a7.md