grounding: adopt tokengrams as the H13 base; lint calibration harness
Base-repo decision per the standing directive, scout-verified: EleutherAI tokengrams (MIT, Windows-native cp311 wheel, exact suffix-array counts, memory-mapped, proven at 16x our corpus) over infini-gram (WSL-only here) and Data Portraits (unlicensed, membership-only). Installed and smoke-tested (exact counts verified). build_tokengrams_index.py: stratified sample -> GPT-2 u16 tokens with EOS document separators -> MemmapIndex.build, with a manifest; launched in background alongside the full citator build. The word- trigram sqlite interim stays behind the corpusAlienness() seam until Stage 7 calibration compares both signals. calibrate-claim-lint.ts (workstream C2): runs the SHIPPED lint over archived receipts against the installed index. Pipeline-grade numbers on 557 claims: unattested_trigram_share AUC 0.705 (top feature; the probe-grade 0.834 used a smaller cases-only reference - both recorded), novel_content_fraction 0.666; operating point unattested > 0.70 flags 20.6% of rejected at 4.3% accepted (~5:1 enrichment). Labels remain checker-derived, not gold. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | 379cc12e |
| Stats | 2 files changed , +273 |
| Part of | Legal grounding and retrieval research |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-5f65432b.md
from inside the repo you want the change in.