jpbreda adds vLLM as a third LLM provider alongside Claude and Gemini
This fork wires a self-hosted vLLM endpoint into Mike's provider system, letting operators swap the model layer to a local inference server instead of routing all requests through Anthropic or Google. The PR closed without merging upstream on May 10.
The implementation uses the openai npm package (v4.87.3) to talk to any OpenAI-compatible endpoint - vLLM exposes this interface by default. Three env vars control the integration: VLLM_BASE_URL, VLLM_API_KEY, and two model name vars (VLLM_MAIN_MODEL, VLLM_LIGHT_MODEL) for the main and lightweight task variants. Adding a new model to the picker is then a config change, not a code change.
The frontend model selector gains a "LocalLLM" group with main and lite options. The availability check for this provider returns true unconditionally - the server-side config determines whether it actually works, not a per-user credential. That's a reasonable design for a self-hosted deployment, but it means the LocalLLM options appear in the picker even on instances where VLLM_BASE_URL is unset. Users on those instances will see the options and get backend failures if they select them. You'd want to guard on the env var at startup or return availability false when the base URL is missing.
jpbreda noted testing against a personal vLLM endpoint with document generation verified end-to-end. The diff is large (+498/-52 across 16 files) but most of that is the lockfile entries for the openai SDK's transitive dependencies.
Spotted something wrong? Or know the PR text has fresher detail than the writeup above?