feat(llm): add model registry and OpenAI-compatible provider foundation

🟢 open · #339 · open-legal-products/mike ← duncanmcqueen/mike · opened 24d ago by duncanmcqueen · +1,818 across 13 files · ↗ on GitHub

From the PR description

Summary

Lets a deployment declare a self-hosted or third-party OpenAI-compatible model through MIKE_MODEL_CONFIG_JSON, without a code change, and makes those models usable even when the endpoint behind them cannot be trusted to emit structured tool calls.

Declared models are served by the existing AI SDK provider layer via @ai-sdk/openai-compatible, so they inherit its transport, retries and streaming rather than introducing a second HTTP client.

What changed

Registry

  • lib/llm/registry.ts - parses MIKE_MODEL_CONFIG_JSON once per process, validating entries and dropping malformed ones. Invalid JSON fails loudly at startup.
  • lib/llm/models.ts - providerForModel and resolveModel consult the registry, so a declared id resolves and persists like any catalog model.
  • lib/llm/providers.ts - an openai-compatible branch builds the adapter from the declared baseUrl, key and apiModel.

Only openai-compatible endpoints are declarable. The hosted providers are already covered by the static catalog and the openrouter/, vercel/ and opencode-go/ prefixes, so a configured entry for one of them would be a second, subtly different way to say the same thing.

Tool-call tolerance

Self-hosted Qwen, DeepSeek and GLM builds routinely describe tool calls in prose and wrap reasoning in <think> tags. Rather than a bespoke client, this is a language-model middleware (wrapLanguageModel):

  • lib/llm/toolCallParsing.ts - pure text handling: <think> routing, tool markup suppression across chunk boundaries, and recovery of tool calls from JSON in <tool_call> markers, XML-ish <function=name> blocks, DeepSeek DSML invocations and single-key maps, with jsonrepair for malformed arguments.
  • lib/llm/localModelMiddleware.ts - applies that to the AI SDK stream: reasoning parts instead of visible text, real tool-call parts instead of prose, and a corrected finish reason.

A tolerant model answering a tool-declaring turn is served through the endpoint's non-streaming path, because these models interleave markup with prose in a way a partial stream cannot be reassembled from. Tolerance defaults on for location: "local" and is overridable per model.

Other

  • routes/tabular.ts - availability messages understand declared models: a local endpoint needs no key, a cloud one is checked against its own declaration.
  • docs/configured-models.md - configuration reference and the tolerance rules.

Why

Mike could only talk to the providers it was compiled to know about. Operators running a model on their own hardware had no way in, and the endpoints they actually run are exactly the ones that need the tolerance layer.

Testing

  • backend: tsc --noEmit clean; vitest run - 764 passed, 25 skipped, including 39 new tests across toolCallParsing, localModelMiddleware and registry. The middleware tests drive it with fake doStream/doGenerate results, so they exercise the parsing and stream rewriting without mocking the network.

13 files changed, 1818 insertions.

Our analysis

Add configurable OpenAI-compatible model registry — read the full analysis →

Think the analysis missed something the PR description covers?

Capture this PR into my fork

Download a Markdown prompt that tells Claude how to port every commit in this PR into your working tree. Run it via claude -p < capture-pull-339.md from inside the repo you want the changes in.

⬇ Download capture-pull-339.md