Stage 20 verdict: premise #1 falsified - checker family carries ~36% of grounding decisions

↗ view on GitHub · Eli Ziff · 2026-07-31 · f0f675ee

The program's founding premise ("the harness carries grounding decisions,
not the model family") rested on 8 cells over one item pair. Replicated on
289 banked compositions, 1,156 checker calls, no composer call:

  within-family replicate   14/222 =  6.3% [3.8, 10.3]
  archived vs fresh same    48/444 = 10.8% [8.3, 14.0]
  cross-family             159/444 = 35.8% [31.5, 40.4]
  paired difference               +29.5 pp [23.2, 36.0]

Both same-family figures land inside the registered H17 0-13% floor, so
the instrument is behaving; the cross-family effect is ~5x that floor.

Direction is symmetric (73 fail-closed / 86 fail-open), not Stage 5's
11/12 fail-closed. On housing:0 -- the audited-insufficient reference --
the claude checker false-accepts 6/16 and codex catches it, so "the only
false accept required Codex in both roles" does not hold.

Aggregate accept rates over identical compositions match within 3 pp
(claude 61.9%, codex 59.0%). That is exactly how an 8-cell table reads as
family-invariant while 36% of individual decisions flip.

The deterministic tier is invariant by construction: 0 mismatches over
4,951 banked claims. But shadow-crossing its 67 all-clear rows gives
codex 83.6% vs claude 52.2% acceptance, and 8 claude calls on 4 rows
returned `unsupported` on answers whose every claim is a verified
verbatim quote. Logged as an open contradiction, not explained away.

Headline counts recomputed independently from the receipt bytes before
logging. Nothing retired on this run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9ToHYJVDxfeJcJwdzrP2H
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents bdbab160
Stats 2 files changed , +88 , -5
Part of Legal grounding and retrieval research

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-f0f675ee.md from inside the repo you want the change in.

⬇ Download capture-commit-f0f675ee.md