sweep: duckdb slice reads + anchor-windowed scanning (probe-proven)

↗ view on GitHub · Eli Ziff · 2026-07-29 · 742d940b

Feeder fix: scan_shard reads its own LIMIT/OFFSET slice via a worker-local
duckdb connection (threads=1) - pyarrow iter_batches held each single-row-
group column chunk (~2.5 GB BCSC) per worker and OOMed at 6 workers.

Scan fix: anchor-windowed match counting. Every entry gets a mandatory-
literal OR-set (AST-derived where possible, else hand PREFILTERS/_HAND_
ANCHORS) plus an AST max-width pad (bounded repeats exact; unbounded
assumed <=64-span, capped). Matches only ever live inside merged windows
around anchor hits; real-coverage bailout (>60% doc) and an edge clip-
guard fall back to full scans. shard_gate_check now asserts windowed==full
per entry per doc, zero tolerance: 0 gate misses, 0 window mismatches over
1862 docs x 39 entries; 1.98x reservoir scan speedup; CMAC+CHRT sharded
aggregation byte-identical to straight reads (SLICE_ROWS=1000 forced
multi-slice). The differential caught two real bugs en route: a bounded-
repeat width cap clipping signal.source tails (133 mismatches) and a hand
anchor set missing signal.source inline branch.

Whole scan_doc: 0.68 MB/s/core on the 172-doc spread sample -> ~35-40 min
projected for the 9.6 GB full tier at 6-7 workers (vs ~26 h observed
trajectory of the parent-fed run).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
Repository eliziff/Beaver
Author Eli Ziff <eliasziff@gmail.com>
Authored
Parents 8c183b29
Stats 3 files changed , +308 , -63
Part of Deterministic legal-text tools ("SLA audit organs")

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-742d940b.md from inside the repo you want the change in.

⬇ Download capture-commit-742d940b.md