mglynnhenley wires a hallucination probe into every AI answer

This fork scores the AI's confidence token by token, then shows users where to squint.

chat-uicontract-review

After Claude or Gemini drafts a response, mglynnhenley's fork quietly ships the draft to a separate scoring service that rates how likely each word is to be made up. Those scores get stored alongside the answer and streamed back to the screen, so cells in a review grid and replies in the chat light up as the verdicts come in.

The interface went through a quick iteration: an early version showed a separate risk badge and heat strip, but the team replaced that with subtle background tints painted directly over the AI's words, plus a slider so users can hide anything below a confidence threshold they choose. There's also a new opt-in 'Think' toggle that lets users ask the model to reason harder on a given turn.

So what Anyone betting on AI for legal review should watch this pattern - surfacing per-sentence confidence inline is a more honest UX than a single 'trust me' answer.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

5 commits from mglynnhenley/mikehasprobes, oldest first. Source extracted verbatim from the harvested git log.

SHA	Subject	Author	Date
`e7549126`	Add hallucination-probe scoring across chat + tabular review	Matilda	2026-05-06	↗ GitHub
commit body Wire Mike to a Modal-hosted, OpenAI-compatible probe service. After each Claude/Gemini response, send the completion as a prefilled assistant turn to the probe and stream per-token scores onto the existing SSE channel. Persist scores on `chat_messages.probe_scores` and `tabular_cells.probe_scores`. UI fades a heat-strip + risk badge under cells/messages as scores arrive. Also: local mock probe at /mock-probe for development without the Modal service, and a "Think" toggle on the chat input so users can opt into adaptive thinking per turn (off by default - Sonnet 4.6 was rejecting the unconditional flag). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`ef3d78ce`	Keep 000 base schema clean; probe columns live in 001/002	Matilda	2026-05-06	↗ GitHub
commit body Per review: the one-shot base schema should stay vanilla. Probe score columns are additive and belong only in 001_probe_scores.sql (tabular_cells) and 002_chat_probe_scores.sql (chat_messages), which already exist as incremental migrations. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`4dcbc056`	Consolidate probe migrations into single 001	Matilda	2026-05-06	↗ GitHub
Merge 002's chat_messages.probe_scores into 001 alongside the tabular_cells columns. One migration covers the entire probe schema extension; 002 deleted. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`d0df46d4`	Merge pull request #1 from mglynnhenley/probe-scorer	Matilda	2026-05-06	↗ GitHub
Hallucination-probe scoring for chat + tabular review
`9206a2a0`	Add inline probe highlighting and threshold slider	Matilda	2026-05-07	↗ GitHub
commit body - Render probe scores as inline background tints over the assistant text instead of a separate heat strip - Add per-user highlight threshold slider (localStorage-persisted) so low-confidence tokens can be hidden from the chat view - Switch probe scorer to non-streaming chat completions; reads scores from top-level response.scores, drops the -analyze model variant - Emit content_done SSE event so the typing indicator drops as soon as Claude finishes, while probe scores keep streaming in Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-96.md from inside the repo you want the changes in.

⬇ Download capture-thread-96.md