Sweep survives its own corpus: no full-tier sort, no giant-doc OOM
The first full run died silently mid-transition from BCCA to BCSC: ORDER BY at full tier forces duckdb to sort whole parquets (BCSC is 916 MB), and the pool deadlocked with the main process holding the failures file open - visible only as a stale file handle hours later. Full tier now streams unordered (sampled tiers keep deterministic order), every court/set prints a progress line unbuffered, documents over 8M chars scan head-only and record oversize_doc, and workers recycle every 200 tasks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012AgUm5EcRKbT3duVFMbrQD
| Repository | eliziff/Beaver |
|---|---|
| Author | Eli Ziff <eliasziff@gmail.com> |
| Authored | |
| Parents | 99c9e300 |
| Stats | 1 file changed , +25 , -3 |
| Part of | Deterministic legal-text tools ("SLA audit organs") |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-bcbc73f9.md
from inside the repo you want the change in.