feat: store document passages for search across a matter (stage 1)
Groundwork for asking a question of a whole matter rather than of documents
handed over one at a time. Nothing user-facing yet.
Every document is now cut into passages when it is stored - after OCR for a
scan, since before that there is no text - and the passages are saved with the
page each came from, so a later search can cite by page. Storing happens after
the upload is answered and never fails it.
- lib/passages.ts cuts page-labelled text into overlapping passages that
never span a page. Pure and unit-tested.
- lib/passageIndex.ts reads a stored version from whichever copy has the text
(the searchable PDF for scans, the file for a plain PDF,
the grid for a spreadsheet) and stores its passages,
replacing any held for that version so re-indexing is safe.
- migrations/20260818_01_document_passages.sql the passages table, with
word-search and trigram indexes and user/project columns
so a search can be scoped to what the reader may see.
- scripts/backfillPassages.ts indexes documents uploaded before this existed.
Word search over the stored passages is stage 3; the meaning fingerprint is a
later migration once the local model is chosen. Verified on the live server:
the existing scan indexed to page-numbered passages and word search found real
terms with correct pages.
New logic lives in new files; the edits to documents.ts and projects.ts only
add the call that stores passages after an upload.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PoCRDAPyEq4wwBVFFbDzck
| Repository | wardethan2000-eng/mike |
|---|---|
| Author | Ethan Ward <ethanward321@hotmail.com> |
| Authored | |
| Parents | e08bf61d |
| Stats | 8 files changed , +681 , -1 |
| Part of | Matter-wide document search and grounded answers |
Capture this commit into my fork
Download a Markdown prompt that tells Claude how to port this
exact commit into your working tree. Run it via
claude -p < capture-commit-6f4d2edf.md
from inside the repo you want the change in.