sweep: worker-side parquet shards + derived prefilter gates
The first full run starved: a single-threaded parent fed whole documents from duckdb through IPC and 4 workers idled at a 4.6% duty cycle (~26 h trajectory vs the 1.7 h projection - the perf gate had measured the scorer, never the feeder). Full tier now ships tiny (parquet, set, lang, row-slice) jobs; workers decode their own rows with pyarrow (measured 490 MB/s on FC) and return per-shard aggregates, so the parent never holds document text. Sampled tiers keep the parent-fed path; both paths share _fold_record/_summarize so they cannot drift. Proven: probes/shard_gate_check.py reproduces the legacy aggregation EXACTLY on CMAC and CHRT (docs, chars, every counter). Prefilter gates are now also derived from the pattern AST where no hand gate exists (mandatory-literal OR-sets - sound by construction; 3 new entries gated, 1.25x measured scan speedup), and the probe asserts zero gate misses over a 1,862-doc reservoir sample. That assertion caught a real pre-existing bug: cite.statute.judgment.fr's hand gate listed only dotted heads (l.r/l.c) and silently undercounted dotless LRC/LC/CQLR citations in every earlier run; the gate now carries the dotless heads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | 113bd450 |
| Stats | 2 files changed , +476 , -53 |
| Part of | Deterministic legal-text tools ("SLA audit organs") |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-8c183b29.md
from inside the repo you want the change in.