grounding: adopt tokengrams as the H13 base; lint calibration harness

↗ view on GitHub · Eli Ziff · 2026-07-30 · 5f65432b

Base-repo decision per the standing directive, scout-verified:
EleutherAI tokengrams (MIT, Windows-native cp311 wheel, exact
suffix-array counts, memory-mapped, proven at 16x our corpus) over
infini-gram (WSL-only here) and Data Portraits (unlicensed,
membership-only). Installed and smoke-tested (exact counts verified).
build_tokengrams_index.py: stratified sample -> GPT-2 u16 tokens with
EOS document separators -> MemmapIndex.build, with a manifest;
launched in background alongside the full citator build. The word-
trigram sqlite interim stays behind the corpusAlienness() seam until
Stage 7 calibration compares both signals.

calibrate-claim-lint.ts (workstream C2): runs the SHIPPED lint over
archived receipts against the installed index. Pipeline-grade numbers
on 557 claims: unattested_trigram_share AUC 0.705 (top feature; the
probe-grade 0.834 used a smaller cases-only reference - both recorded),
novel_content_fraction 0.666; operating point unattested > 0.70 flags
20.6% of rejected at 4.3% accepted (~5:1 enrichment). Labels remain
checker-derived, not gold.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents 379cc12e
Stats 2 files changed , +273
Part of Legal grounding and retrieval research

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-5f65432b.md from inside the repo you want the change in.

⬇ Download capture-commit-5f65432b.md