emileriksenkeev makes scanned documents searchable again

sydOS is hardening document intake so image-only and awkwardly formatted files do not disappear from search.

searchinfrastructure

The important fix catches a subtle failure: scanned PDFs could be mistaken for text-bearing documents because page placeholders looked like extracted content. They were indexed as empty material instead of sent through OCR, leaving 278 workspace documents effectively invisible to search.

The fork also broadens the safety net:

OCR for scanned PDFs and standalone images, making their text searchable.
Smarter handling of very large scans, reducing the risk of processing failures.
Support for older Word documents and mislabeled image files.
Cleanup of problematic hidden characters before extracted text is stored.

This is unglamorous work with real practical value: reliable search depends on recognising when a document contains no usable text at all.

So what Litigation, investigations, and legal-ops teams handling mixed document collections should care because a silent extraction failure can turn relevant evidence into a search blind spot.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

6 commits from emileriksenkeev/sydOS, oldest first. Source extracted verbatim from the harvested git log.

SHA	Subject	Author	Date
`1f74871f`	Syd OS: OCR pipeline for scanned/image-only PDFs	emileriksenkeev	2026-07-11	↗ GitHub
commit body - ocrDocument.ts: rasterize with pdftoppm (150dpi, capped pages), transcribe with Claude vision (OCR_MODEL, default claude-haiku-4-5) - preserves chat screenshot structure (speaker/timestamp/message); persists documents.ocr_text and re-embeds so the doc becomes semantically searchable - read_document + embedDocument fall back to ocr_text when a PDF has no text layer; upload hook auto-OCRs new scanned uploads after embed finds no text - ocrBackfill.ts: batch-OCR existing docs with no chunks and no ocr_text Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`8e761a0f`	Harden text extraction: strip NUL bytes, add legacy .doc fallback	emileriksenkeev	2026-07-12	↗ GitHub
commit body Two embed-pipeline gaps found while backfilling the Vault: - A stray NUL byte (\u0000) in extracted PDF text aborts the whole chunk insert ('unsupported Unicode escape sequence'). Add sanitizeText() to strip NUL + other C0 control chars (keeping tab/newline/CR) on both the text-layer and OCR paths. - Legacy .doc (OLE2) yields no text via mammoth. Fall back to the LibreOffice-converted PDF rendition the upload pipeline already produced. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`b61d3d10`	fix(ocr): adaptive dpi + size-aware batching for oversized scans	emileriksenkeev	2026-07-14	↗ GitHub
commit body ~40 scanned permits/receipts failed OCR: high-res scans rasterised at 150 dpi produced PNGs over the API's 10 MB per-image cap, and six fat pages in one request tripped the request-size ceiling (413). Re-rasterise at 96/72 dpi when a page exceeds 4.5 MB, and cut batches by cumulative bytes as well as page count. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`d1d7cd55`	Image-document OCR: jpg/jpeg/png become searchable	emileriksenkeev	2026-07-16	↗ GitHub
commit body Completes the image-OCR change set whose lib half (ocrDocument.ts imagePages/media-type support + convert.ts imageToPdf) was swept into 98f01cf alongside the bridge work: - routes/documents.ts + routes/projects.ts: allow jpg/jpeg/png uploads, stamp correct image content types, and serve images inline from the display route. The project upload handler also gains the same fire-and-forget embed->OCR hook single-document uploads already had, so project uploads no longer depend on manual backfill runs. - lib/chatTools.ts: read_document defers images straight to ocr_text instead of throwing inside the mammoth fallback before the OCR fallback could run. - scripts/ocrBackfill.ts: covers image documents; run `node dist/scripts/ocrBackfill.js <projectId>` to heal the HR backlog. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`71413433`	OCR: sniff image magic bytes; route mislabeled formats via LibreOffice	emileriksenkeev	2026-07-16	↗ GitHub
commit body The HR backfill surfaced an iPhone HEIC stored as ".JPEG": we sent its raw bytes as image/jpeg and the API replied 400 "Could not process image". Trust the magic bytes over the extension, and when the bytes aren't real JPEG/PNG, force the LibreOffice->PDF->pdftoppm shrink path, which re-encodes to PNG the API accepts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`39c86e4f`	Fix scanned-PDF blindness: marker-poisoned text checks + grounded tabular	emileriksenkeev	2026-07-17	↗ GitHub
commit body Root cause of wrong tabular extractions (and 278 workspace docs being invisible to search): extractPdfText emits a "[Page N]" marker for every page even when the page is image-only, so scanned PDFs produce marker-only text whose raw length passes every "does this document have a text layer?" threshold. Those docs were embedded as junk markers, never OCR'd, and fed to tabular reviews as blank - where the model invented plausible values (dates, reference numbers) from the filename and flagged them green. - chatTools: meaningfulPdfTextLength() measures marker-stripped text; read_document's OCR fallback now uses it. - embedDocument: marker-aware fallback to ocr_text, and marker-only text no longer counts as extractable. - ocrDocument: "has text layer" check is marker-aware, so scanned PDFs actually get OCR'd. - tabular: falls back to documents.ocr_text when the text layer is empty (generate + single-cell paths); when a document has no readable text at all the model is not called - cells get an explicit "Not Found / no readable text" result instead of hallucinations; the single-cell path no longer nulls content before regenerating. - tabular grounding: every [[quote]] citation is verified against the supplied document text server-side - values whose quotes don't appear are flagged red as unverified. Prompts now forbid inferring values from filenames, define flags as confidence signals, and instruct the model to distinguish operative dates from form-template/print dates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-912.md from inside the repo you want the changes in.

⬇ Download capture-thread-912.md