OpenAI-compatible local inference: Ollama and Qwen via env-configured base URL
punyaslokdutta makes Mike's OpenAI adapter point at any OpenAI-compatible endpoint - Ollama, vLLM, or the hosted service. Set `OPENAI_BASE_URL`, a model name, and an endpoint mode flag, and the same adapter talks to Ollama, vLLM, or any chat-completions-compatible runtime. A local runtime with no API key is treated as a valid provider, unblocking fully offline use.
The change is driven by three environment variables: a base URL, a model name, and an endpoint mode. When those are configured and no hosted OpenAI key is present, the local runtime counts as an available provider - the provider-availability check is adjusted to recognize this. That's the piece that makes offline mode actually work rather than just silently failing to load models.
Two Qwen 3 sizes (8B and 14B) are added to the model picker as concrete local options. The streaming path for the OpenAI-compatible route is also changed to deliver output incrementally. Without streaming, Mike would render a blank response while Ollama spent seconds generating; punyaslokdutta explicitly calls this out as a UX fix rather than a background optimization.
The branch documentation covers a Docker-plus-native-Ollama workflow for running the full stack locally end to end. Validation was done against a backend build and a manual rebuild of the local Docker stack with Supabase keys wired through the env files. The scope is deliberately narrow: local inference enablement only, with benchmark work left for a follow-on.
The PR is open against willchen96/mike and has not landed. The diff snapshot in this fork primarily shows the storage backend swap commit rather than the inference changes, so the local inference code may be on a separate branch from what's visible here.
Spotted something wrong? Or know the PR text has fresher detail than the writeup above?