Pre-Phase-7: model routing seam added before connector work begins

Archibald312 deferred local inference (Ollama, vLLM) to post-launch and instead landed the routing surface that connectors and the eventual local-inference adapter will plug into. The behavioral change today is zero - every chat still resolves to the user's requested model - but the decision point now exists as a tested, audited abstraction.

infrastructurecompliance

The schema change is minimal: nullable model_preference text columns on projects and documents. Existing rows are unaffected.

backend/src/lib/llm/routing.ts (156 lines) implements resolveModelRouting(). Precedence runs document → project → request: any document in the request with a non-null preference wins first; if none, the project preference; if none, the caller's requested model. When two documents in the same request carry different preferences, the resolver captures the conflict with a structured reason and takes the first non-null value rather than blocking. Unknown model IDs are rejected with a recorded reason. All decisions write into the routing_policy_applied jsonb column in audit_log - that column was provisioned in Phase 6 specifically for this.

streamChatWithTools now accepts a routing context, dispatches with the resolved model, and records the policy decision. The main chat path passes the chat's project ID and the document IDs already tracked in the in-memory document index.

Eight Vitest tests cover: document-level precedence over project-level, project-level over request, conflict capture, unknown-model rejection at both layers, DB error tolerance when the preference lookup fails, and the early-exit path when no document IDs are supplied. Two manual verification steps - confirming a document-level override actually routes and audits correctly end-to-end, and confirming a no-document chat records a request-sourced resolution - were left unchecked in the test plan at merge time.

Phase 7 connectors will populate documents.model_preference at ingest from their declared preference. Phase 14 wires in a local-inference adapter as another model option. Neither requires touching dispatch sites.

So what Worth importing if your fork will eventually need per-source model routing, data-privacy gating by document sensitivity, or local-inference as an option. The seam is low-risk to add (nullable columns, behavior-identical today) and the test coverage makes the precedence rules explicit. The cost to adopt is moderate: you need to wire the chat dispatch path to pass project and document IDs to the resolver, which touches the request handlers but is straightforward. Skip if your model selection will never need per-document or per-project overrides.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?