sweep: duckdb slice reads + anchor-windowed scanning (probe-proven)
Feeder fix: scan_shard reads its own LIMIT/OFFSET slice via a worker-local duckdb connection (threads=1) - pyarrow iter_batches held each single-row- group column chunk (~2.5 GB BCSC) per worker and OOMed at 6 workers. Scan fix: anchor-windowed match counting. Every entry gets a mandatory- literal OR-set (AST-derived where possible, else hand PREFILTERS/_HAND_ ANCHORS) plus an AST max-width pad (bounded repeats exact; unbounded assumed <=64-span, capped). Matches only ever live inside merged windows around anchor hits; real-coverage bailout (>60% doc) and an edge clip- guard fall back to full scans. shard_gate_check now asserts windowed==full per entry per doc, zero tolerance: 0 gate misses, 0 window mismatches over 1862 docs x 39 entries; 1.98x reservoir scan speedup; CMAC+CHRT sharded aggregation byte-identical to straight reads (SLICE_ROWS=1000 forced multi-slice). The differential caught two real bugs en route: a bounded- repeat width cap clipping signal.source tails (133 mismatches) and a hand anchor set missing signal.source inline branch. Whole scan_doc: 0.68 MB/s/core on the 172-doc spread sample -> ~35-40 min projected for the 9.6 GB full tier at 6-7 workers (vs ~26 h observed trajectory of the parent-fed run). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | 8c183b29 |
| Stats | 3 files changed , +308 , -63 |
| Part of | Deterministic legal-text tools ("SLA audit organs") |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-742d940b.md
from inside the repo you want the change in.