Local LLM provider added via server-configured OpenAI-compatible endpoint

manueljpconde wired a fourth inference path into mikeEU: a backend-only `local:server` provider that routes to any OpenAI-compatible `/v1/chat/completions` endpoint. Ollama, LM Studio, and similar runtimes work out of the box. The browser never sees the URL or credentials.

infrastructuresecurity

The implementation lives in backend/src/lib/llm/local.ts (206 lines) and reads configuration from six env vars: LOCAL_LLM_BASE_URL, LOCAL_LLM_MODEL, LOCAL_LLM_API_KEY, LOCAL_LLM_LABEL, LOCAL_LLM_SUPPORTS_TOOLS, and LOCAL_LLM_TIMEOUT_MS (default 120 s). ENABLE_LOCAL_LLM=false keeps the provider dark unless an operator explicitly opts in.

Tool gating is the more interesting design decision. When LOCAL_LLM_SUPPORTS_TOOLS is false (the default), the provider is removed from the tabular-model preferences surface entirely, and the active tool list passed to the LLM is set to empty. A system-prompt injection appends a plain-text notice that document editing, DOCX generation, and tabular extraction are unavailable on that turn. The logic sits in runLLMStream in chatTools.ts around providerForModel(selectedModel).

Tests cover provider routing, config parsing, SSE frame splitting, and the no-tool streaming path. The author noted that pre-existing lint failures in the frontend are outside this change; touched files passed lint cleanly.

So what Worth a look if you want to let operators route inference to on-prem hardware without exposing endpoint details to the browser. The tool-gating logic is conservative by default and saves you from a class of broken-state bugs where local models silently fail at function calling. Skip it if you only run hosted providers - it adds env complexity for no benefit in that case.

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

SHA	Subject	Author	Date
`c0361944`	feat: add self-hosted local llm provider	Manuel Conde	2026-05-09	↗ GitHub

SHA

Subject

Author

Date

c0361944

feat: add self-hosted local llm provider

Manuel Conde

2026-05-09

↗ GitHub

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-255.md from inside the repo you want the changes in.

Local LLM provider added via server-configured OpenAI-compatible endpoint

Commits in this thread

Capture this thread into my fork