Document extraction now preserves tracked changes and comments instead of flattening them
JonasBoury landed a change that stops the ingest pipeline from silently discarding review markup. DOCX tracked changes and comment bubbles now come through as readable inline markers, and PDFs with color-coded annotations get an optional PyMuPDF extractor backed by pdfjs fallback.
Most legal document AI pipelines throw away tracked changes on ingest - the model sees the final merged text and has no idea what was added, deleted, or flagged. This commit changes that for both DOCX and PDF paths.
For DOCX, backend/src/lib/docxTrackedChanges.ts now walks the OOXML tracked-changes nodes and emits {++inserted++}, {--deleted--}, and {>>by AUTHOR: comment text<<} markers inline. Author names and comment bodies come from word/comments.xml. Files with no tracked changes continue through Mammoth unchanged.
For PDF, a new script at backend/scripts/redline_extract.py uses PyMuPDF to classify span colors against red/blue/green targets with a 30-unit RGB tolerance - the same algorithm as Diff Master's Pyodide-based processor, now running as a plain Python subprocess. backend/src/lib/pdfRedlineExtract.ts spawns it with a 30-second timeout and falls back to the existing pdfjs path on any error. The extractor requires PyMuPDF (pip install pymupdf); the PYTHON_BIN env var lets you point at a specific interpreter.
The prompt layer is updated to match. Chat and tabular extraction prompts now include instructions for interpreting the markers: current text = insertions accepted + deletions dropped; comments are marginalia; strip markers from extracted field values unless explicitly asked.
Verification covers the backend build plus synthetic smoke tests on both formats - a DOCX with insertion, deletion, and comment markup, and a PDF with red/blue/green spans generated through PyMuPDF.
Spotted something wrong? Or know the PR text has fresher detail than the writeup above?