feat: store document passages for search across a matter (stage 1)

↗ view on GitHub · Ethan Ward · 2026-08-18 · 6f4d2edf

Groundwork for asking a question of a whole matter rather than of documents
handed over one at a time. Nothing user-facing yet.

Every document is now cut into passages when it is stored - after OCR for a
scan, since before that there is no text - and the passages are saved with the
page each came from, so a later search can cite by page. Storing happens after
the upload is answered and never fails it.

- lib/passages.ts    cuts page-labelled text into overlapping passages that
                     never span a page. Pure and unit-tested.
- lib/passageIndex.ts reads a stored version from whichever copy has the text
                     (the searchable PDF for scans, the file for a plain PDF,
                     the grid for a spreadsheet) and stores its passages,
                     replacing any held for that version so re-indexing is safe.
- migrations/20260818_01_document_passages.sql  the passages table, with
                     word-search and trigram indexes and user/project columns
                     so a search can be scoped to what the reader may see.
- scripts/backfillPassages.ts  indexes documents uploaded before this existed.

Word search over the stored passages is stage 3; the meaning fingerprint is a
later migration once the local model is chosen. Verified on the live server:
the existing scan indexed to page-numbered passages and word search found real
terms with correct pages.

New logic lives in new files; the edits to documents.ts and projects.ts only
add the call that stores passages after an upload.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PoCRDAPyEq4wwBVFFbDzck
Repository wardethan2000-eng/mike
Author Ethan Ward <ethanward321@hotmail.com>
Authored
Parents e08bf61d
Stats 8 files changed , +681 , -1
Part of Matter-wide document search and grounded answers

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-6f4d2edf.md from inside the repo you want the change in.

⬇ Download capture-commit-6f4d2edf.md