Every answer Mike writes now gets scored word by word for how likely it is to be made up.
mglynnhenley/mikehasprobes
Mike fork adding per-token hallucination scoring over an external probe service
mglynnhenley/mikehasprobes has one defining addition: after each Claude or Gemini completion, the assistant text is scored by a Modal-hosted, OpenAI-compatible probe service. Per-token confidence scores stream back over the existing SSE channel and render as inline background tints on the assistant message. The feature is fully opt-in via PROBE_API_URL; absent that env var the rest of the app runs unchanged.
The fork also ships a local mock probe endpoint for development, a content_done SSE event to decouple the typing indicator from the scoring animation, and a localStorage-persisted threshold slider so users can tune which tokens get highlighted. A "Think" toggle for Anthropic adaptive thinking was added alongside, made opt-in after the author found Sonnet 4.6 rejected the flag when sent unconditionally.
What's in it
- Hallucination heat strip A fading visual band beneath each AI reply showing where the model is steadier and where it's reaching.
- Token-level confidence scoring Doubt is computed across the answer rather than as a single overall verdict, so users can localize where to squint.
- External probe service Scoring is handled by a dedicated hallucination-probe service rather than being inferred from the assistant itself.
- Coverage across chat and tabular review The same confidence signal follows users from conversational Q&A into structured row-by-row document work.
Direction
chat-uicompliance
Activity
📝 mglynnhenley makes Mike show you which words it might be making up 3 commits 2mo ago chat-uicontract-review draft
A new colour overlay tints the parts of an answer the AI is least sure about - so you can see the risky bits before you trust them.
✅ #1 Hallucination-probe scoring for chat + tabular review +605 -16 2mo ago self by
mglynnhenley ↗ analysis ↗ GitHub ## Summary
Threads of work (detailed view)
mglynnhenley gives Mike a built-in lie detector
Every answer Mike writes now gets scored word by word for how likely it is to be made up.
Hallucination-probe scores wired into chat and tabular review
mglynnhenley added per-token hallucination scoring to both chat turns and tabular cells, routing completed assistant output through an external Modal-hosted probe service and streaming the scores back over the existing SSE channel. The whole integration sits behind a single env var and degrades gracefully when it's absent.
mglynnhenley makes Mike show you which words it might be making up
A new colour overlay tints the parts of an answer the AI is least sure about - so you can see the risky bits before you trust them.
External hallucination-probe service wired in, with heat-strip UI and mock endpoint
mglynnhenley forked Mike to add post-hoc hallucination scoring: after each Claude or Gemini turn, the completed assistant text is sent to a Modal-hosted probe service which returns per-token confidence scores. Those scores stream back over the same SSE channel and light up inline tints over the assistant text.
Pull requests (detailed view)
✅ Merged (1)
by
mglynnhenley · opened 2mo ago · merged 2mo ago by mglynnhenley