Harden text extraction: strip NUL bytes, add legacy .doc fallback
Two embed-pipeline gaps found while backfilling the Vault:
- A stray NUL byte (\u0000) in extracted PDF text aborts the whole chunk
insert ('unsupported Unicode escape sequence'). Add sanitizeText() to strip
NUL + other C0 control chars (keeping tab/newline/CR) on both the text-layer
and OCR paths.
- Legacy .doc (OLE2) yields no text via mammoth. Fall back to the
LibreOffice-converted PDF rendition the upload pipeline already produced.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| Repository | emileriksenkeev/sydOS |
|---|---|
| Author | emileriksenkeev <emil@keev.co> |
| Authored | |
| Parents | 15dd519c |
| Stats | 1 file changed , +42 , -3 |
| Part of | Text extraction and OCR hardening |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-8e761a0f.md
from inside the repo you want the change in.