Retract amendment 2026-07-31c; reinstate the bilingual family

↗ view on GitHub · Eli Ziff · 2026-07-31 · d955b15f

The previous amendment gated fixture selection on a licensing analysis this
benchmark had no business making. Scope and permissions for this work are
settled outside it. Retracted in full: the withdrawal of the bilingual and
Quebec family, every redistributable/reference-only/do-not-use verdict, and
the CourtListener attribution observations, which were not findings of this
experiment.

Fixture provenance is now a reproducibility record - source, citation,
modifications, obscurity, retrieval date - not a legal one. licence and
licence_verdict are gone from the type, and nothing is gated on them.

Quebec is restated as what it is: an availability fact. The A2AJ coverage
tool reports 24 legislation and regulation datasets and none of them is
Quebec, so Quebec statutory material is not obtainable from the corpus this
benchmark draws on. The family does not depend on one - a bilingual pair
where an edit to one version and not the other is a defect works on any
equally-authoritative instrument.

Also records what coverage actually reports rather than what spelunking
guessed: 962 federal statutes and 4,876 federal regulations, bilingual
throughout, and fetch_document takes output_language=both, which returns the
English and French versions together - the bilingual fixture with no
assembly. The corpus is queried through coverage, search_legal_documents and
fetch_document, not by reverse-engineering the on-disk layout.

Carries forward the one real method note: assert text[start:end] == expected
on the scorer's own bytes before quoting any v2 number.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents ddcb05cf
Stats 2 files changed , +73 , -12
Part of Deterministic legal-text tools ("SLA audit organs")

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-d955b15f.md from inside the repo you want the change in.

⬇ Download capture-commit-d955b15f.md