External hallucination-probe service wired in, with heat-strip UI and mock endpoint

mglynnhenley forked Mike to add post-hoc hallucination scoring: after each Claude or Gemini turn, the completed assistant text is sent to a Modal-hosted probe service which returns per-token confidence scores. Those scores stream back over the same SSE channel and light up inline tints over the assistant text.

chat-uicompliance

The integration is behind PROBE_API_URL in .env. Leave it unset and the scorer returns null without touching the request path. Set it and the backend replays the completed assistant message as a prefilled turn to an OpenAI-chat-completions-compatible endpoint, reading back a { probeName: number[] } payload. Scores persist on tabular_cells.probe_scores and chat_messages.probe_scores (both jsonb) so they survive page reloads.

Text appears immediately. Probe tints arrive asynchronously, keeping perceived latency tied to the model rather than the scoring service. A content_done SSE event signals the frontend to drop the typing indicator; probe score events can keep arriving after that. The UI renders background tints inline over assistant text, and a threshold slider stored in localStorage (default 0.3) lets users suppress low-confidence highlights.

For local development there is a regex-based mock endpoint at /mock-probe/v1/chat/completions, enabled with ENABLE_PROBE_MOCK=true. The live scorer has a circuit breaker - an unreachable probe degrades to unhighlighted output rather than breaking the chat response.

A separate "Think" toggle was added to the chat input to make adaptive thinking opt-in per turn. The commit notes that Sonnet 4.6 rejects the thinking flag when sent unconditionally, so the default is off.

So what Worth a look if you want to surface confidence signals on assistant output and already have or plan to run a compatible probe endpoint. The env-gate pattern, the `content_done` SSE split, and the async tint animation are all worth borrowing independently. The Sonnet 4.6 thinking-flag note is a useful standalone gotcha. Skip if you have no probe service - the feature leans on a custom server contract that isn't part of the standard OpenAI API.

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

3 commits from mglynnhenley/mikehasprobes, oldest first. Source extracted verbatim from the harvested git log.

SHA	Subject	Author	Date
`e7549126`	Add hallucination-probe scoring across chat + tabular review	Matilda	2026-05-06	↗ GitHub
commit body Wire Mike to a Modal-hosted, OpenAI-compatible probe service. After each Claude/Gemini response, send the completion as a prefilled assistant turn to the probe and stream per-token scores onto the existing SSE channel. Persist scores on `chat_messages.probe_scores` and `tabular_cells.probe_scores`. UI fades a heat-strip + risk badge under cells/messages as scores arrive. Also: local mock probe at /mock-probe for development without the Modal service, and a "Think" toggle on the chat input so users can opt into adaptive thinking per turn (off by default - Sonnet 4.6 was rejecting the unconditional flag). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`ef3d78ce`	Keep 000 base schema clean; probe columns live in 001/002	Matilda	2026-05-06	↗ GitHub
commit body Per review: the one-shot base schema should stay vanilla. Probe score columns are additive and belong only in 001_probe_scores.sql (tabular_cells) and 002_chat_probe_scores.sql (chat_messages), which already exist as incremental migrations. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`4dcbc056`	Consolidate probe migrations into single 001	Matilda	2026-05-06	↗ GitHub
Merge 002's chat_messages.probe_scores into 001 alongside the tabular_cells columns. One migration covers the entire probe schema extension; 002 deleted. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

SHA

Subject

Author

Date

e7549126

Add hallucination-probe scoring across chat + tabular review

Matilda

2026-05-06

↗ GitHub

commit body

Wire Mike to a Modal-hosted, OpenAI-compatible probe service. After
each Claude/Gemini response, send the completion as a prefilled
assistant turn to the probe and stream per-token scores onto the
existing SSE channel. Persist scores on `chat_messages.probe_scores`
and `tabular_cells.probe_scores`. UI fades a heat-strip + risk badge
under cells/messages as scores arrive.

Also: local mock probe at /mock-probe for development without the
Modal service, and a "Think" toggle on the chat input so users can
opt into adaptive thinking per turn (off by default - Sonnet 4.6 was
rejecting the unconditional flag).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

ef3d78ce

Keep 000 base schema clean; probe columns live in 001/002

Matilda

2026-05-06

↗ GitHub

commit body

Per review: the one-shot base schema should stay vanilla. Probe
score columns are additive and belong only in 001_probe_scores.sql
(tabular_cells) and 002_chat_probe_scores.sql (chat_messages),
which already exist as incremental migrations.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

4dcbc056

Consolidate probe migrations into single 001

Matilda

2026-05-06

↗ GitHub

Merge 002's chat_messages.probe_scores into 001 alongside the
tabular_cells columns. One migration covers the entire probe schema
extension; 002 deleted.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-279.md from inside the repo you want the changes in.