fix(pdf): extract AcroForm field values, not just drawn page text

🟢 open · #387 · open-legal-products/mike ← terpstra-209/mike · opened 8d ago by terpstra-209 · +72-3 across 2 files · ↗ on GitHub

From the PR description

Summary

extractPdfText only reads page.getTextContent(), which returns page content-stream text. It never sees a filled-but-unflattened PDF form, because those values live in annotation field data (fieldName/fieldValue), not on the page itself.

Why

A PDF form filled out in a viewer that doesn't flatten the form (e.g. macOS Preview's plain Save/Export) writes typed values into AcroForm field annotations rather than drawing them onto the page. page.getTextContent() is blind to that - it only sees what's actually drawn. Command-line pdftotext reads the field values directly and correctly shows the form as filled; this app's PDF extraction returned the page as if it were blank, and since extractPdfText is the only source of PDF text for read_document (documentOps.ts), this happened regardless of which LLM read it downstream.

Changes

backend/src/lib/chat/tools/documentOps.ts - extractPdfText now also calls page.getAnnotations() per page and appends any non-empty field values as a [Page N form fields] block after that page's normal text. The annotations read is wrapped in its own try/catch so a failure on one page falls back to that page's already-extracted text instead of losing it to the function's outer catch-all.

No other call site needed changes - readDocumentContent and both callers of extractPdfText just forward the returned string as-is.

Testing

New backend/src/lib/chat/tools/documentOps.test.ts - mocks pdfjs-dist's getDocument, and covers:

  • filled field values are merged into the extracted text
  • empty and unnamed fields are skipped
  • an annotation-read failure falls back to page text instead of losing it
vitest run src/lib/chat/tools/documentOps.test.ts   2 passed
npm test --prefix backend                           841 passed | 25 skipped (1 unrelated pre-existing flake, passes in isolation)
tsc --noEmit --prefix backend                       clean
npm run build --prefix backend                      clean

The investigation and code in this PR were done by Claude (Sonnet 5) via Claude Code; I reviewed and am submitting it.

Our analysis

Extract unflattened PDF form values — read the full analysis →

Think the analysis missed something the PR description covers?

Capture this PR into my fork

Download a Markdown prompt that tells Claude how to port every commit in this PR into your working tree. Run it via claude -p < capture-pull-387.md from inside the repo you want the changes in.

⬇ Download capture-pull-387.md