legal grounding: experiment apparatus, stages 1-3 record, stage 4 pre-registration
Commits the until-now-untracked grounding experiment: the provisional legalEvidenceExperiment lib (evidence receipts, structured claims, compose_check / evidence_first / holistic_check finalizers, fail-closed rendering), its unit suite, the two runner scripts (held-constant benchmark matrix; fixed verifier probe suite), and the experiment log recording stages 1-3 (all falsified under the strict rule; no winner). Stage 4 lands pre-registered under the REVISED decision rule (direction 2026-07-30: reasonably grounded without unreasonable token spend, instead of a strict win against the unverified control on every gate - a gate no verification arm can pass by construction): - new tiered_check mode: a deterministic verbatim-quote tier renders fully-quoted answers with ZERO checker calls (balanced citation-tail stripping; typography-only normalization - curly quotes, dash widths, NBSP, ellipsis, whitespace - never case or words; >= 25 chars; contiguous substring of ONE cited passage). Everything else escalates to Stage 3's single fail-closed holistic check, hardened with at most three attempts on transport-class failures only; semantic verdicts are never retried. Coverage on the deterministic path is honestly not_run in the receipt; composition is instructed to quote exactly where the passage answers. - probe suite gains three deterministic-tier probes (pure quote, mutated quote, spliced prose) and per-probe modes; receipts record passed + per-claim deterministic_support (schema 4, optional field). Frozen-hypothesis probe run (receipts stage4-verifier-probes.jsonl): Claude 13/14 exact labels, 14/14 allow/deny (sole disagreement the Stage-3 lost-attribution partially_supported-vs-unsupported, fail-closed either way); pure-quote passed with 0 model calls, mutated and spliced probes escalated and failed closed on BOTH providers; ALCE leave-one-out necessity matched on both. Codex: 11/14 with three transport nulls on the retry-less holistic-mode probes; its tiered-mode probes (retry harness) all completed. The held-constant 3-arm matrix is running; results land in the log by amendment, not by editing the frozen section. Unit tests 12 passed; tsc clean. Experiment remains off by default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | ed8f6f0c |
| Stats | 5 files changed , +3519 |
| Part of | Legal grounding and retrieval research |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-b0338f65.md
from inside the repo you want the change in.