nwhitehouse keeps the assistant streaming while it reaches for tools

Tool-using turns used to go silent and slow; now the answer keeps flowing in real time, with a safety net for when the model misbehaves.

chat-uiinfrastructure

nwhitehouse runs a custom, fine-tuned version of the AI model behind this fork. The snag: whenever the assistant needed to use a tool - look something up, trigger an action - it had to abandon live, word-by-word replies and wait for the whole answer to arrive at once. Slower, and it felt like the chat had stalled. This thread fixes that. The assistant now streams in real time even on tool-using turns, quietly hiding the model's internal scaffolding from what the user actually sees.

There's a second fix for a failure that showed up fast: sometimes the model signals it wants to use a tool but sends nothing usable. nwhitehouse added a fallback that catches exactly that case and re-asks once, so the action still fires instead of failing silently - at the cost of one extra round-trip on the affected turns only.

So what Worth a look for anyone running their own fine-tuned model behind a legal assistant: the responsiveness fix and the silent-failure fallback are both reusable.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

2 commits from nwhitehouse/mike, oldest first. Source extracted verbatim from the harvested git log.

SHA	Subject	Author	Date
`6d2aac9e`	[feat-001] Stream tokens during tool-using turns	Nick Whitehouse	2026-05-03	↗ GitHub
commit body Olava previously fell back to a non-streaming request whenever tools were forwarded, because vLLM's tool-call streaming is broken for the LoRA's custom <tool_call><function=...><parameter=...> markup - `delta.tool_calls` arrives empty even when finish_reason is "tool_calls". Fix it client-side: keep streaming on, accumulate raw delta.content, but filter the user-visible stream through a small state machine that hides <think>...</think> blocks and everything after a <tool_call> open tag. Held-back tail handles markup that spans chunk boundaries. After the stream ends, run the existing parseCustomToolCall() on the raw buffer to extract the call and dispatch via runTools - same path the non-streaming branch already used. Also fixes a related bug: the no-tools "streaming" path was buffering the entire response and emitting one giant onContentDelta at the end. Now genuinely per-token in both paths. Emergency rollback available via OLAVA_FORCE_NONSTREAM_TOOLS=true. Adds backlog.md to track the sprint. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`e772ac55`	[bug-002] Recover tool calls when vLLM streaming drops the payload	Nick Whitehouse	2026-05-03	↗ GitHub
commit body feat-001's premise was that the Olava LoRA's custom tool-call markup (<tool_call><function=...><parameter=...>) would arrive in delta.content during streaming, where parseCustomToolCall could extract it. Verified against the live RunPod endpoint with a "What's the latest court opinion involving AI" query: vLLM finishes with finish_reason="tool_calls" but neither populates delta.tool_calls (accCalls=0) nor includes the markup in delta.content (raw text comes through as just "\n\n"). The tool-call info just disappears in streaming mode for this LoRA. Fix: when streaming finishes with finish_reason="tool_calls" but no tool call extracted from either channel, re-issue the iter as a single non-streaming request and parse the markup from message.content. One extra request per tool-using iter, only on iter 0. Iter 1+ (the prose answer iters that come after the tool runs) stream normally - that's where the streaming win actually lives. Net behaviour: - Tool-free turns: stream tokens (feat-001 win preserved). - Tool-using turns iter 0: ~2s of dead air to detect + recover the call. Same as the original always-non-stream behaviour. - Tool-using turns iter 1+: stream prose tokens to the user. OLAVA_FORCE_NONSTREAM_TOOLS=true escape hatch from feat-001 still works if the recovery itself misbehaves. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-120.md from inside the repo you want the changes in.

⬇ Download capture-thread-120.md