WS3 backend: local/on-premises OpenAI-compatible model provider

↗ view on GitHub · sloth-ninja · 2026-07-08 · 510eda76

New "local" LLM provider for any OpenAI-compatible chat-completions server
(Ollama, LM Studio, vLLM) alongside Claude/Gemini/OpenAI, per
docs/MIGRATION_SPEC.md §5 + §6 decisions #2/#5. Positioning is data
sovereignty, not cost.

- lib/llm/localConfig.ts (new): resolves LOCAL_LLM_BASE_URL (primary),
  OPENAI_BASE_URL (documented alias only when the former is unset, never
  affects the cloud OpenAI client), LOCAL_LLM_MODELS, LOCAL_LLM_API_KEY
  (defaults to bearer "ollama").
- lib/llm/localOpenAI.ts (new): streamLocal/completeLocalText, a new
  chat-completions streaming client (not a base-URL override of the
  existing Responses-API openai.ts) mirroring its callback contract and
  tool loop. Malformed tool-call JSON is never executed - it's fed back as
  a tool error so the model can retry, never crashing the stream.
- types.ts/models.ts/index.ts: "local" provider registration, local:
  prefix dispatch, registry bypass for local: ids once configured.
- routes/user.ts: local status merged additively into all apiKeyStatus
  responses; routes/tabular.ts: local models bypass the per-user-key gate
  (server-env configured, not key-gated).
- .env.example + CLAUDE.md env registry updated.
- vitest added (first backend test suite): 25 tests covering config
  resolution, provider/model dispatch, and the streaming client against
  canned SSE / mocked fetch (content deltas, tool-call fragment
  accumulation, malformed-JSON handling, second-turn message shape,
  default bearer token) - no live server required.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Repository Sloth-ninja/JessicaOSS
Author sloth-ninja <ezanahaddis@gmail.com>
Authored
Committed
Parents b02b2dd3
Stats 15 files changed , +2174 , -13
Part of WS3: local/on-premises OpenAI-compatible model provider

Capture this commit into my fork

Download a Markdown prompt that tells Claude how to port this exact commit into your working tree. Run it via claude -p < capture-commit-510eda76.md from inside the repo you want the change in.

⬇ Download capture-commit-510eda76.md