docx-edit-bench v1: reusable document-editing benchmark + A/B registration

↗ view on GitHub · Eli Ziff · 2026-07-31 · 146ceb3f

A surface-agnostic benchmark for "can a model edit an arbitrary span of a
real document": 27 tasks over 11 fixtures, tasks as data, checkers as a
library, tool surfaces as configuration. Nothing in src/ knows about
MIKE_NAV_SHAPE, so the next surface is measurable without editing it.

Fixtures are built, not committed: six prose documents rendered through
the product's own markdown-to-DOCX path (a 25k-char agreement down to an
unnumbered letter, two of them page-marked), and five pathology documents
built with the DOCX packager reusing the shapes in the existing
docx-pathologies fixtures - auto-numbering that lives only in
numbering.xml, a signing-limit table, a real tracked deletion plus a
manual strike/colour redline that still reads as operative, comments,
parallel EN/FR with a bilingual table, OCR quote/dash/spacing damage, and
footnotes off the body plane. Identity is the sha256 of the extracted
body text; DOCX bytes are not reproducible across builds.

Validity is established before any model call, and both halves
separately. Task validity: 27/27 tasks carry a reference solution that
applies cleanly and passes the checker. Outcome validity: 27/27 reject
every wrong result and 0 site checks have never been observed failing -
wrong results come from hand-written near misses, two automatic probes
(the untouched document; a partial regeneration that satisfies the
positive half while destroying the tail), and one synthetic sensitivity
probe per guard that damages exactly what that guard protects. The
whole-document-rewrite shortcut is blocked by a zero budget for
destroying original lines the reference solution keeps.

Registration of the first consumer - the legacy/address navigation A/B -
is appended to docs/legal-grounding-experiments-2026-07-30.md and is
frozen by this commit. It records the served schema of both arms, the
edit-scope leak found in the legacy arm during setup and since fixed by a
concurrent session, and the description leaks left unpatched because they
are part of the surface under test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents bae8ce6f
Stats 21 files changed , +3367
Part of SourceDoc / DOCX engine: redlines, tracked changes, editing benchmark

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-146ceb3f.md from inside the repo you want the change in.

⬇ Download capture-commit-146ceb3f.md