feat: server-side citation verification against source text
From the PR description
Summary
Hallucinated citations are the failure mode that gets a small firm sanctioned. This PR adds server-side verification of assistant citations: after the model's <CITATIONS> block is parsed, every document quote is checked against the actual extracted text of the cited document before the citation is streamed to the client or persisted.
Each quote is located in the source with progressively more tolerant matching - exact substring, then whitespace/case-normalized, then punctuation-tolerant - and gets a per-quote verification record plus char offsets into the extracted text. Each citation gets an aggregate verification_status:
verified- quote is character-identical to the sourcerepaired- quote was found under tolerant matching but drifted; the exact source excerpt is swapped into the displayed quote so drifted text is never presented as the source's wordsunverified- no match; the model's text is preserved but must be treated as untrusted
Cross-page quotes ([[PAGE_BREAK]] sentinel) are split and each segment verified independently. Case-law citations pass through untouched - their existence is already verified via CourtListener and is never re-marked here.
Changes
backend/src/lib/chat/verifyCitations.ts(new) - quote location + verification + annotation/batch verification logicbackend/src/lib/chat/verifyCitations.test.ts(new) - 16 focused unit tests (exact/drifted/fabricated quotes, unreadable sources, cross-page quotes, case-citation passthrough, aggregation)backend/src/lib/chat/streaming.ts- verify final citations before thecitationsSSE frame; source text is fetched at most once per document per turn (memoized), viareadDocumentContent(..., { emitEvents: false })so no extra events firebackend/src/lib/chat/tools/documentOps.ts-normalizeWithMapgains an optionalstripPunctuationmode and is exported (used by tier-3 tolerant matching)backend/src/lib/chat/types.ts-CitationVerificationStatus/QuoteVerificationtypesbackend/tsconfig.json- exclude*.test.ts/__tests__from thetscbuild so tests don't require test deps at build time
No new runtime dependencies. No frontend changes: the new fields ride along on the existing citation annotations; a missing verification_status is documented as "treat as untrusted", so existing clients are unaffected.
Why
Citation trust is mission-core: the assistant's answer is only usable in legal work if every quote it attributes to a document is actually in that document. This closes the gap server-side, where the source of truth (document bytes in storage) lives - the client never has to be trusted to check.
No new vendors. Negligible compute cost: verification is pure string matching over document text the server already extracts, with source text memoized per document per turn - no extra LLM calls and no network egress (the tests explicitly prove verification only reads bytes handed to it).
Testing
npm install && npm run buildinbackend/- green on this branch as committednpx vitest run src/lib/chat/verifyCitations.test.ts- 16/16 passed (vitest installed locally with--no-saveand not committed; upstream has no test harness yet, so tests run atop the test-harness PR)frontend/untouched
Provenance
All changes are mechanical ports of code in amal66/mike@origin/main (commit b3166dd); exceptions: none. Sources: apps/api/src/lib/tools/verifyCitations.ts + .test.ts (logic and tests, verbatim modulo import paths and mock/comment path rewrites), apps/api/src/lib/tools/stream.ts (streaming hook, verbatim modulo createCitationAnnotation → upstream's createCitation), apps/api/src/lib/tools/docRead.ts (normalizeWithMap extension, verbatim), packages/core/src/types.ts (the two verification types, inlined verbatim), apps/api/tsconfig.json (test-file build exclusion).
Credits & prior art
- @Gadoes (Gadoes/dispumike) - independently parallels this work: their fork built citation verification for case-law sources (via legal-database integrations such as CourtListener/EUR-Lex). This PR verifies document quotes against source text server-side - same goal (no hallucinated citations reach a lawyer), different mechanism.
🤖 Generated with Claude Code
Our analysis
Verify assistant citations against source documents — read the full analysis →
Think the analysis missed something the PR description covers?
Capture this PR into my fork
Download a Markdown prompt that tells Claude how to port every
commit in this PR into your working tree. Run it via
claude -p < capture-pull-224.md from
inside the repo you want the changes in.