F3: resume keys carry every dimension that changes what a cell means

↗ view on GitHub · Eli Ziff · 2026-07-31 · 1d220fce

legalbench-rag-grounding keyed cells on
coords|model|effort|k|retriever|test_id. `arm` was written to the row but
LEFT OUT of the key, and --coverage, --spec, --plain and --exclude-gold
each change the arm - the prompt, and for --exclude-gold the evidence -
while changing nothing the key could see. Those are exactly the flags the
C1a/C1b/F2/F3 experiments ran under, so a resume could have declared one
arm's cells done for another. Measured occurrences: zero (every existing
receipt is raw-CRLF, and `coords` already separates those), but live.

The sibling legal-grounding-experiment had the mirror-image hole: `arm`
was in its key, `effort` was not, which already forced a Stage 13 ladder
lane onto its own output file to stay separable.

Both keys now live in src/lib/experimentReceipts beside the receipt guard:
  legalbenchRagCellKey  coords|model|effort|arm|k|retriever|per_doc_cap|test_id
  legalGroundingCellKey model|effort|arm|checker|case_id|rank_policy

Backwards compatible, deliberately: a missing field falls back to what the
row's own labels prove it ran under, so pre-fix receipts stay resumable
instead of being re-run wholesale. `arm` absent = required_slot;
`per_doc_cap` absent is derived from the retriever label (product = n/a,
passage:pool(...) = uncapped, +rerank = 24, otherwise 2 - the exact pre-fix
semantics); `coords` absent = the raw-CRLF instrument, which still can
never satisfy an LF cell.

Verified on the real receipt store (read-only, nothing written): stage14
keys as ...|product|n/a|..., stage17/18 as ...|24|..., the fused-pool file
as ...|uncapped|..., the plain-arm file as ...|plain|..., and no file has
fewer distinct keys than non-error rows.

Tests: 11 in src/lib/__tests__/experimentReceipts.test.ts, including all
six arm labels distinct, every other dimension distinct, identical cells
identical, and the legacy backfills. Full backend suite: 1427 passed
(2 pre-existing docx/compare timeouts under parallel load, both pass in
isolation, neither touches this code).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents a8cdd08c
Stats 4 files changed , +246 , -18
Part of Legal agent-surface and context-compaction experiments

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-1d220fce.md from inside the repo you want the change in.

⬇ Download capture-commit-1d220fce.md