sweep: worker-side parquet shards + derived prefilter gates

↗ view on GitHub · Eli Ziff · 2026-07-29 · 8c183b29

The first full run starved: a single-threaded parent fed whole
documents from duckdb through IPC and 4 workers idled at a 4.6% duty
cycle (~26 h trajectory vs the 1.7 h projection - the perf gate had
measured the scorer, never the feeder). Full tier now ships tiny
(parquet, set, lang, row-slice) jobs; workers decode their own rows
with pyarrow (measured 490 MB/s on FC) and return per-shard aggregates,
so the parent never holds document text. Sampled tiers keep the
parent-fed path; both paths share _fold_record/_summarize so they
cannot drift. Proven: probes/shard_gate_check.py reproduces the legacy
aggregation EXACTLY on CMAC and CHRT (docs, chars, every counter).

Prefilter gates are now also derived from the pattern AST where no hand
gate exists (mandatory-literal OR-sets - sound by construction; 3 new
entries gated, 1.25x measured scan speedup), and the probe asserts zero
gate misses over a 1,862-doc reservoir sample. That assertion caught a
real pre-existing bug: cite.statute.judgment.fr's hand gate listed only
dotted heads (l.r/l.c) and silently undercounted dotless LRC/LC/CQLR
citations in every earlier run; the gate now carries the dotless heads.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents 113bd450
Stats 2 files changed , +476 , -53
Part of Deterministic legal-text tools ("SLA audit organs")

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-8c183b29.md from inside the repo you want the change in.

⬇ Download capture-commit-8c183b29.md