docx-edit-bench v1: reusable document-editing benchmark + A/B registration
A surface-agnostic benchmark for "can a model edit an arbitrary span of a real document": 27 tasks over 11 fixtures, tasks as data, checkers as a library, tool surfaces as configuration. Nothing in src/ knows about MIKE_NAV_SHAPE, so the next surface is measurable without editing it. Fixtures are built, not committed: six prose documents rendered through the product's own markdown-to-DOCX path (a 25k-char agreement down to an unnumbered letter, two of them page-marked), and five pathology documents built with the DOCX packager reusing the shapes in the existing docx-pathologies fixtures - auto-numbering that lives only in numbering.xml, a signing-limit table, a real tracked deletion plus a manual strike/colour redline that still reads as operative, comments, parallel EN/FR with a bilingual table, OCR quote/dash/spacing damage, and footnotes off the body plane. Identity is the sha256 of the extracted body text; DOCX bytes are not reproducible across builds. Validity is established before any model call, and both halves separately. Task validity: 27/27 tasks carry a reference solution that applies cleanly and passes the checker. Outcome validity: 27/27 reject every wrong result and 0 site checks have never been observed failing - wrong results come from hand-written near misses, two automatic probes (the untouched document; a partial regeneration that satisfies the positive half while destroying the tail), and one synthetic sensitivity probe per guard that damages exactly what that guard protects. The whole-document-rewrite shortcut is blocked by a zero budget for destroying original lines the reference solution keeps. Registration of the first consumer - the legacy/address navigation A/B - is appended to docs/legal-grounding-experiments-2026-07-30.md and is frozen by this commit. It records the served schema of both arms, the edit-scope leak found in the legacy arm during setup and since fixed by a concurrent session, and the description leaks left unpatched because they are part of the surface under test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | bae8ce6f |
| Stats | 21 files changed , +3367 |
| Part of | SourceDoc / DOCX engine: redlines, tracked changes, editing benchmark |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-146ceb3f.md
from inside the repo you want the change in.