Fix scanned-PDF blindness: marker-poisoned text checks + grounded tabular

↗ view on GitHub · emileriksenkeev · 2026-07-17 · 39c86e4f

Root cause of wrong tabular extractions (and 278 workspace docs being
invisible to search): extractPdfText emits a "[Page N]" marker for every
page even when the page is image-only, so scanned PDFs produce
marker-only text whose raw length passes every "does this document have
a text layer?" threshold. Those docs were embedded as junk markers,
never OCR'd, and fed to tabular reviews as blank - where the model
invented plausible values (dates, reference numbers) from the filename
and flagged them green.

- chatTools: meaningfulPdfTextLength() measures marker-stripped text;
  read_document's OCR fallback now uses it.
- embedDocument: marker-aware fallback to ocr_text, and marker-only
  text no longer counts as extractable.
- ocrDocument: "has text layer" check is marker-aware, so scanned PDFs
  actually get OCR'd.
- tabular: falls back to documents.ocr_text when the text layer is
  empty (generate + single-cell paths); when a document has no readable
  text at all the model is not called - cells get an explicit
  "Not Found / no readable text" result instead of hallucinations; the
  single-cell path no longer nulls content before regenerating.
- tabular grounding: every [[quote]] citation is verified against the
  supplied document text server-side - values whose quotes don't appear
  are flagged red as unverified. Prompts now forbid inferring values
  from filenames, define flags as confidence signals, and instruct the
  model to distinguish operative dates from form-template/print dates.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Repository emileriksenkeev/sydOS
Author emileriksenkeev <emil@keev.co>
Authored
Parents 6c7f015d
Stats 4 files changed , +155 , -24
Part of Text extraction and OCR hardening

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-39c86e4f.md from inside the repo you want the change in.

⬇ Download capture-commit-39c86e4f.md