Sloth-ninja puts Mike's AI work behind the firm's firewall

This fork adds an on-premises model option for firms that need AI traffic to stay on their own hardware.

infrastructuresecurity

The new option is aimed at data sovereignty. A firm can connect Mike to a locally hosted, OpenAI-compatible model service while keeping the chat experience, streamed replies and tool use intact. Sloth-ninja's guidance is candid: the tested local model works, but cloud models may still deliver stronger results.

  • Local model choices appear in chat and account settings only after an administrator configures them.
  • Existing saved choices fall back safely if the local service later becomes unavailable.
  • Staff cannot enter their own model-server addresses, which keeps that connection under server-side control.

Cloud OpenAI traffic remains separate from this setup.

So what Managing partners, GCs and legal-ops teams with strict hosting requirements should look here if on-premises AI is a live procurement condition.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

5 commits from Sloth-ninja/JessicaOSS, oldest first. Source extracted verbatim from the harvested git log.

SHA Subject Author Date
510eda76 WS3 backend: local/on-premises OpenAI-compatible model provider sloth-ninja 2026-07-08 ↗ GitHub
commit body
New "local" LLM provider for any OpenAI-compatible chat-completions server
(Ollama, LM Studio, vLLM) alongside Claude/Gemini/OpenAI, per
docs/MIGRATION_SPEC.md §5 + §6 decisions #2/#5. Positioning is data
sovereignty, not cost.

- lib/llm/localConfig.ts (new): resolves LOCAL_LLM_BASE_URL (primary),
  OPENAI_BASE_URL (documented alias only when the former is unset, never
  affects the cloud OpenAI client), LOCAL_LLM_MODELS, LOCAL_LLM_API_KEY
  (defaults to bearer "ollama").
- lib/llm/localOpenAI.ts (new): streamLocal/completeLocalText, a new
  chat-completions streaming client (not a base-URL override of the
  existing Responses-API openai.ts) mirroring its callback contract and
  tool loop. Malformed tool-call JSON is never executed - it's fed back as
  a tool error so the model can retry, never crashing the stream.
- types.ts/models.ts/index.ts: "local" provider registration, local:
  prefix dispatch, registry bypass for local: ids once configured.
- routes/user.ts: local status merged additively into all apiKeyStatus
  responses; routes/tabular.ts: local models bypass the per-user-key gate
  (server-env configured, not key-gated).
- .env.example + CLAUDE.md env registry updated.
- vitest added (first backend test suite): 25 tests covering config
  resolution, provider/model dispatch, and the streaming client against
  canned SSE / mocked fetch (content deltas, tool-call fragment
  accumulation, malformed-JSON handling, second-turn message shape,
  default bearer token) - no live server required.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
db46dd58 WS3 frontend: local model picker, availability, and preferences sloth-ninja 2026-07-08 ↗ GitHub
commit body
Surfaces the new "local" provider in the model picker, gated purely on
server-reported configuration (never a per-user key):

- mikeApi.ts: ApiKeyStatus gains optional local?: {configured, models}
  (additive; ApiKeyProvider deliberately untouched - no BYO local key).
- UserProfileContext.tsx: UserProfile gains localModels: string[].
- ModelToggle.tsx: "Local (on-premises)" group merged in at runtime from
  server-reported models, listed last, hidden entirely when unconfigured;
  group header links to docs/local-models.md, each row hints "runs on your
  own hardware".
- modelAvailability.ts: ModelProvider gains "local"; availability is
  server-reported (localModels), never API-key-gated.
- useSelectedModel.ts: accepts localModels; a persisted local: selection
  is re-validated against the reported list and falls back to
  DEFAULT_MODEL_ID gracefully when no longer valid.
- ChatInput.tsx, TabularReviewView.tsx, TRChatPanel.tsx,
  account/models/page.tsx: localModels threaded from useUserProfile()
  through every isModelAvailable/ModelToggle call site (chat, tabular
  chat, tabular generation gating, title/tabular account preferences).

ApiKeyMissingModal needed no changes: it only fires when isModelAvailable
returns false, which a hidden (unconfigured) local model never triggers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ad62ab48 docs: local-models.md setup guide + WS3 build log entry sloth-ninja 2026-07-08 ↗ GitHub
commit body
Ollama/LM Studio setup, Qwen 2.5 14B Instruct (Apache-2.0) recommendation,
data-sovereignty positioning, honest quality caveat pointing at the
README eval table (Day-3 model-comparison run), and a tool-calling
reliability note (malformed calls surface as tool errors, never crashes).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
16d31f9d WS3 design polish: truncate long local model names in dropdown rows sloth-ninja 2026-07-08 ↗ GitHub
commit body
Local model ids/labels are server-reported and can be much longer than
the static catalog entries (e.g. Ollama/vLLM model names). The row label
span had no min-w-0/truncate, so a long name would wrap onto multiple
lines and misalign the trailing check/alert icon. Matches the
min-w-0 flex-1 truncate convention already used elsewhere (e.g.
AssistantSidePanel tab labels, DocumentSidePanel rows).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ad3a768b WS3: tolerate CRLF-delimited SSE frames in the local chat-completions client sloth-ninja 2026-07-08 ↗ GitHub
Review finding: SSE permits \r\n line endings and the local provider
promises compatibility with any OpenAI-compatible server, not just
Ollama/LM Studio/vLLM (which all emit \n\n).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-1222.md from inside the repo you want the changes in.

⬇ Download capture-thread-1222.md