Add local Ollama/Qwen support through the OpenAI-compatible provider

⛔ closed · #122 · Open-Legal-Products/mike ← punyaslokdutta/HarveyOss · opened 2mo ago by punyaslokdutta · closed 2mo ago · +2,163-49 across 25 files · ↗ on GitHub

From the PR description

Summary

  • add OpenAI-compatible local runtime support for self-hosted endpoints like Ollama
  • add Qwen model options in Mike's model picker and document the Docker-plus-native-Ollama setup
  • make the OpenAI-compatible path stream local model output incrementally so Mike surfaces answers instead of appearing blank while Ollama is still generating

What changed

  • made the OpenAI adapter configurable via OPENAI_COMPAT_BASE_URL, OPENAI_COMPAT_MODEL, and OPENAI_COMPAT_ENDPOINT_MODE
  • added support for chat-completions style OpenAI-compatible backends used by Ollama and vLLM
  • treated env-configured OpenAI-compatible runtimes as available even without a hosted OpenAI API key
  • added Qwen 3 8B and 14B options to the model picker
  • documented the local Docker app stack plus native Ollama workflow for testing Mike locally

Validation

  • npm run build --prefix backend
  • local Docker stack rebuilt with the Ollama-compatible env path
  • verified local Supabase keys were wired into the Docker env files for the local dev stack

Notes

  • frontend lint still reports existing repo-wide issues unrelated to this branch
  • this branch is intentionally scoped to local inference enablement first; broader benchmark work can build on top

Screenshot

Mike with Qwen on Ollama

Our analysis

Add OpenAI-compatible local inference with Qwen on Ollama — read the full analysis →

Think the analysis missed something the PR description covers?

Capture this PR into my fork

Download a Markdown prompt that tells Claude how to port every commit in this PR into your working tree. Run it via claude -p < capture-pull-122.md from inside the repo you want the changes in.

⬇ Download capture-pull-122.md