nwhitehouse teaches Mike's table review to read the whole document

The grid-style review tool can now answer questions that no column was built to capture.

searchdiscovery

Mike's tabular review pulls answers out of documents into a spreadsheet of columns. The catch: if a question doesn't map to a column, the assistant had nothing to work with and would simply decline. nwhitehouse closes that gap by indexing the full text of every uploaded document and letting the chat reach for relevant passages when no column fits. Under the hood it leans on semantic search - matching by meaning rather than keywords - over the documents in a review, so a question that spans a whole file gets a real answer instead of a shrug.

The same run of work hardens the plumbing: large reviews no longer trip over their own backlog when catching up old documents, malformed text from scanned PDFs gets cleaned before it can crash anything, and results that slipped through a timing gap in the original version are now refetched.

So what Worth a look for anyone whose document review throws questions that live across a file rather than in a tidy column - the trade-off is an OpenAI key to run the embeddings.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

4 commits from nwhitehouse/mike, oldest first. Source extracted verbatim from the harvested git log.

SHA	Subject	Author	Date
`a187e3a0`	[feat-024] RAG chat over tabular-review docs	Nick Whitehouse	2026-05-07	↗ GitHub
commit body Embeds every doc on upload via a new embed_document job type on the bug-007 worker pool, stores chunks + embeddings in document_chunks (pgvector, HNSW), and injects top-K passages into the TR chat system prompt before the LLM call. - migrations 007 (vector ext, table, RPC, RLS) + 008 (job_type column) - text-embedding-3-small (1536 dim) via direct fetch with batched retry - 800-tok chunks / 150-tok overlap, page-aware via "## Page N" markers - POST /single-documents/embed-backfill, gated by ENABLE_EMBED_BACKFILL - TR chat falls back to cell-only context when OPENAI_API_KEY unset Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`dbf18a87`	[feat-024] Strip NUL bytes from chunk input	Nick Whitehouse	2026-05-07	↗ GitHub
commit body Postgres text/jsonb encoding rejects U+0000 with "unsupported Unicode escape sequence" - pdfjs occasionally emits NUL in extracted text and the embed insert fails for the whole doc. Drop them upfront in the chunker so both worker + script paths are covered. Also adds a one-shot backfill script that bypasses the HTTP endpoint + worker pool: chunks and embeds every doc that has no rows in document_chunks. Useful for backfilling docs that pre-date this feature without restarting the backend or wiring auth. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`9d98a18e`	[feat-024] Backfill script: wipe-and-reinsert for idempotency	Nick Whitehouse	2026-05-07	↗ GitHub
commit body Without the wipe, re-running the script crashes on (document_id, chunk_index) unique-constraint violations whenever a prior run partially landed chunks for a doc. Match the worker's processEmbedDocumentItem semantics so the script is safe to re-run on any state. Also bumps the chunkedSet query past supabase-js's 1000-row default since document_chunks easily exceeds that on a real review. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`4f7e58ca`	[feat-024] Refetch review on job terminate + use passages over column-refusal	Nick Whitehouse	2026-05-07	↗ GitHub
commit body Two fixes for symptoms reported on prod: 1. After a tabular_generate job finishes, the poll loop's prev.map delta only updates cells that already exist in state - anything completed in the gap between the last delta poll and the terminal-status flip stays blank until the user hits refresh. Refetch the canonical review state once polling exits so the UI matches the DB without manual reload. 2. The TR chat system prompt framed the world entirely around columns ("call read_table_cells before answering"), so the LLM refused questions whose subject wasn't a column even when RETRIEVED PASSAGES contained the answer. Tighten the framing to tell the model both sources are valid grounding and to consult passages before declining. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-155.md from inside the repo you want the changes in.

⬇ Download capture-thread-155.md