⚖MikeWatch Forks of the Mike legal-AI codebase, distilled for legal-tech readers
Home · mglynnhenley/mikehasprobes · PR #1

Hallucination-probe scoring for chat + tabular review

✅ merged · #1 · mglynnhenley/mikehasprobes ← mglynnhenley/mikehasprobes · opened 20d ago by mglynnhenley · merged 20d ago by mglynnhenley · self · +605-16 across 23 files · ↗ on GitHub

From the PR description

Summary

  • Wire Mike to an external OpenAI-compatible probe service (Modal-hosted, see PROBE_API_URL). After each Claude/Gemini turn, the assistant text is sent as a prefilled message + extra_body: { include_scores: true }; per-token scores stream back over the existing SSE channel and persist on chat_messages.probe_scores / tabular_cells.probe_scores.
  • Frontend renders a fading heat strip (HighlightedSummary) + risk badge (ProbeBadge) under assistant messages and tabular cells as scores arrive.
  • Adds a local /mock-probe endpoint (regex-based, no model) so dev works without the real probe service. Toggled by ENABLE_PROBE_MOCK=true.
  • Adds a "Think" toggle next to the model selector so users can opt into adaptive thinking per turn - defaults off because Sonnet 4.6 was rejecting the unconditional flag.

Tickets: MAT-37, MAT-38, MAT-39.

Test plan

  • Backend boots with mock enabled (ENABLE_PROBE_MOCK=true); /mock-probe/v1/chat/completions returns SSE chunks with scores.
  • Send chat with Claude, includes a known hallucination ("Berlin", "1989") - heat strip fades in under the assistant reply.
  • Run a tabular review against a doc; cells render text immediately and probe-derived badge fades in after.
  • Probe service unreachable → cells/messages render normally with no highlight (circuit breaker in scorer.ts).
  • "Think" toggle off → Claude requests succeed on Sonnet 4.6; on → adaptive thinking enabled (Opus 4.7).

🤖 Generated with Claude Code

Our analysis

Wire Mike to an external hallucination-probe service with heat-strip UI — read the full analysis →

Think the analysis missed something the PR description covers?

Commits in this PR (3)

SHA Subject Author Date
e7549126 Add hallucination-probe scoring across chat + tabular review Matilda 2026-05-06 ↗ GitHub
commit body 
Wire Mike to a Modal-hosted, OpenAI-compatible probe service. After
each Claude/Gemini response, send the completion as a prefilled
assistant turn to the probe and stream per-token scores onto the
existing SSE channel. Persist scores on `chat_messages.probe_scores`
and `tabular_cells.probe_scores`. UI fades a heat-strip + risk badge
under cells/messages as scores arrive.

Also: local mock probe at /mock-probe for development without the
Modal service, and a "Think" toggle on the chat input so users can
opt into adaptive thinking per turn (off by default - Sonnet 4.6 was
rejecting the unconditional flag).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ef3d78ce Keep 000 base schema clean; probe columns live in 001/002 Matilda 2026-05-06 ↗ GitHub
commit body 
Per review: the one-shot base schema should stay vanilla. Probe
score columns are additive and belong only in 001_probe_scores.sql
(tabular_cells) and 002_chat_probe_scores.sql (chat_messages),
which already exist as incremental migrations.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
4dcbc056 Consolidate probe migrations into single 001 Matilda 2026-05-06 ↗ GitHub 
Merge 002's chat_messages.probe_scores into 001 alongside the
tabular_cells columns. One migration covers the entire probe schema
extension; 002 deleted.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Capture this PR into my fork

Download a Markdown prompt that tells Claude how to port every commit in this PR into your working tree. Run it via claude -p < capture-pull-1.md from inside the repo you want the changes in.

⬇ Download capture-pull-1.md