Hallucination-probe scores wired into chat and tabular review
mglynnhenley added per-token hallucination scoring to both chat turns and tabular cells, routing completed assistant output through an external Modal-hosted probe service and streaming the scores back over the existing SSE channel. The whole integration sits behind a single env var and degrades gracefully when it's absent.
The probe client lives in backend/src/lib/probe/scorer.ts. When PROBE_API_URL is set, it takes the completed assistant message, constructs a prefilled conversation, and sends it to an OpenAI-chat-completions-compatible endpoint with include_scores: true. The response comes back as a plain non-streaming call that returns response.scores - a { probeName: number[] } object, one float per token. An earlier iteration streamed with max_tokens: 1 and read scores off each chunk; that was replaced with a single non-streaming call and the timeout bumped from 60s to 240s. The resulting arrays fan out token by token through an onScore callback, so the UI animation still works.
On the storage side, one migration (001_probe_scores.sql) adds probe_scores jsonb and probe_status text to tabular_cells and probe_scores jsonb to chat_messages. The base schema (000_one_shot_schema.sql) was kept vanilla after PR review reverted an early attempt to put the columns there.
A content_done SSE event was added to split the typing indicator from the probe animation. The browser drops the spinner as soon as Claude's text finishes; probe tints then fade in token by token as score events arrive. The UI switched from a separate heat-strip and badge (ProbeBadge.tsx, HighlightedSummary.tsx) to inline background tints over the assistant text, with a localStorage-persisted threshold slider (useProbeThreshold.ts, default 0.3) to hide low-confidence tokens. There is also a bundled /mock-probe route behind ENABLE_PROBE_MOCK=true so you can work locally without the Modal service.
One loose end: the scorer uses as unknown as type assertions at two call sites to work around the fact that include_scores and the top-level scores field are not in the standard OpenAI schema. Anyone porting this needs to reproduce the same custom server contract or those casts will hide a silent mismatch.
Spotted something wrong? Or know the PR text has fresher detail than the writeup above?