Retract amendment 2026-07-31c; reinstate the bilingual family
The previous amendment gated fixture selection on a licensing analysis this benchmark had no business making. Scope and permissions for this work are settled outside it. Retracted in full: the withdrawal of the bilingual and Quebec family, every redistributable/reference-only/do-not-use verdict, and the CourtListener attribution observations, which were not findings of this experiment. Fixture provenance is now a reproducibility record - source, citation, modifications, obscurity, retrieval date - not a legal one. licence and licence_verdict are gone from the type, and nothing is gated on them. Quebec is restated as what it is: an availability fact. The A2AJ coverage tool reports 24 legislation and regulation datasets and none of them is Quebec, so Quebec statutory material is not obtainable from the corpus this benchmark draws on. The family does not depend on one - a bilingual pair where an edit to one version and not the other is a defect works on any equally-authoritative instrument. Also records what coverage actually reports rather than what spelunking guessed: 962 federal statutes and 4,876 federal regulations, bilingual throughout, and fetch_document takes output_language=both, which returns the English and French versions together - the bilingual fixture with no assembly. The corpus is queried through coverage, search_legal_documents and fetch_document, not by reverse-engineering the on-disk layout. Carries forward the one real method note: assert text[start:end] == expected on the scorer's own bytes before quoting any v2 number. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | ddcb05cf |
| Stats | 2 files changed , +73 , -12 |
| Part of | Deterministic legal-text tools ("SLA audit organs") |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-d955b15f.md
from inside the repo you want the change in.