Add LocalLLM/vLLM provider support

⛔ closed · #20 · willchen96/mike ← jpbreda/mike · opened 24d ago by jpbreda · closed 16d ago · +515-52 across 17 files · ↗ on GitHub

From the PR description

Overview

Adds support for OpenAI-compatible local LLM endpoints (vLLM) as a third provider alongside Claude and Gemini. This enables users to connect Mike to self-hosted LLM instances while maintaining compatibility with commercial APIs.

Key Changes

Backend:

  • New openai.ts provider adapter for OpenAI-compatible endpoints
  • Configurable model names via environment variables (VLLM_MAIN_MODEL, VLLM_LIGHT_MODEL)
  • Server-side configuration (no user API key management required)
  • Added fallback for missing title parameter in generate_docx tool

Frontend:

  • Added "LocalLLM" model group to model selector
  • Default models: LocalLLM Main and LocalLLM Lite
  • Updated model availability checks to treat LocalLLM as always available
  • Updated account settings page to reflect LocalLLM configuration

Configuration:

  • New environment variables:
    • VLLM_BASE_URL - Endpoint URL for OpenAI-compatible API
    • VLLM_API_KEY - Optional API key for authentication
    • VLLM_MAIN_MODEL - Model name for main chat tasks
    • VLLM_LIGHT_MODEL - Model name for lightweight tasks (title generation)

Testing

  • Tested with vLLM endpoint at https://bredaai.com/api/v1
  • Verified document generation workflow
  • Confirmed model selection and routing works correctly

Notes

  • LibreOffice is required for document conversion (DOCX ↔ PDF)
  • LocalLLM models are server-configured and available to all users
  • Maintains backward compatibility with existing Claude/Gemini providers

Our analysis

Add vLLM as a third LLM provider — read the full analysis →

Think the analysis missed something the PR description covers?

Commits in this PR (3)

SHA Subject Author Date
0c84ef49 feat: Add LocalLLM (vLLM) provider support Joseph Breda 2026-05-02 ↗ GitHub
commit body
- Add OpenAI-compatible LLM provider for local vLLM endpoints
- Support for configurable model names via environment variables
- Add LocalLLM Main and LocalLLM Lite as default models
- Update model selector to include LocalLLM options
- Fix generate_docx title fallback for missing parameters
- Add LibreOffice dependency note for document conversion
cc951105 feat: Replace unsloth model with placeholder in .env.example Joseph Breda 2026-05-04 ↗ GitHub
fe3fd823 docs: Add LLM configuration options to README Joseph Breda 2026-05-04 ↗ GitHub

Capture this PR into my fork

Download a Markdown prompt that tells Claude how to port every commit in this PR into your working tree. Run it via claude -p < capture-pull-20.md from inside the repo you want the changes in.

⬇ Download capture-pull-20.md