legal grounding: experiment apparatus, stages 1-3 record, stage 4 pre-registration

↗ view on GitHub · Eli Ziff · 2026-07-30 · b0338f65

Commits the until-now-untracked grounding experiment: the provisional
legalEvidenceExperiment lib (evidence receipts, structured claims,
compose_check / evidence_first / holistic_check finalizers, fail-closed
rendering), its unit suite, the two runner scripts (held-constant
benchmark matrix; fixed verifier probe suite), and the experiment log
recording stages 1-3 (all falsified under the strict rule; no winner).

Stage 4 lands pre-registered under the REVISED decision rule
(direction 2026-07-30: reasonably grounded without unreasonable token
spend, instead of a strict win against the unverified control on every
gate - a gate no verification arm can pass by construction):

- new tiered_check mode: a deterministic verbatim-quote tier renders
  fully-quoted answers with ZERO checker calls (balanced citation-tail
  stripping; typography-only normalization - curly quotes, dash widths,
  NBSP, ellipsis, whitespace - never case or words; >= 25 chars;
  contiguous substring of ONE cited passage). Everything else escalates
  to Stage 3's single fail-closed holistic check, hardened with at most
  three attempts on transport-class failures only; semantic verdicts
  are never retried. Coverage on the deterministic path is honestly
  not_run in the receipt; composition is instructed to quote exactly
  where the passage answers.
- probe suite gains three deterministic-tier probes (pure quote,
  mutated quote, spliced prose) and per-probe modes; receipts record
  passed + per-claim deterministic_support (schema 4, optional field).

Frozen-hypothesis probe run (receipts stage4-verifier-probes.jsonl):
Claude 13/14 exact labels, 14/14 allow/deny (sole disagreement the
Stage-3 lost-attribution partially_supported-vs-unsupported, fail-closed
either way); pure-quote passed with 0 model calls, mutated and spliced
probes escalated and failed closed on BOTH providers; ALCE leave-one-out
necessity matched on both. Codex: 11/14 with three transport nulls on
the retry-less holistic-mode probes; its tiered-mode probes (retry
harness) all completed. The held-constant 3-arm matrix is running;
results land in the log by amendment, not by editing the frozen section.

Unit tests 12 passed; tsc clean. Experiment remains off by default.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents ed8f6f0c
Stats 5 files changed , +3519
Part of Legal grounding and retrieval research

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-b0338f65.md from inside the repo you want the change in.

⬇ Download capture-commit-b0338f65.md