feat: user-defined LLM committees

🟢 open · #341 · open-legal-products/mike ← duncanmcqueen/mike · opened 24d ago by duncanmcqueen · +3,498-25 across 37 files · ↗ on GitHub

From the PR description

Summary

Adds user-defined model committees: a committee answers one prompt with several models at once and has a chair model synthesize their replies into the single answer the user sees. Committees are built under Settings > Models, stored on the profile (user_profiles.model_committees), and selected from a Committee group in the model picker like any other model.

Stacked on #339 - the first commit here is that PR's foundation. Retarget or rebase once it merges.

What changed

Backend

  • lib/llm/committee.ts - members run concurrently against the ordinary provider layer, then the chair synthesizes. Guards against self-chairing, self-membership and cycles.
  • lib/userCommittees.ts - normalization for reads (malformed rows are dropped, never thrown, so a bad row cannot lock a user out of Settings) and strict validation for writes (max 8 committees, 2-8 members).
  • lib/llm/models.ts - missingCommitteeApiKeyModels / modelHasApiKey / resolveUsableModel: availability resolves recursively through a committee's members and chair.
  • lib/routerModels.ts - resolveRequestedModel recognises committee ids and gates them on availability, taking committees the caller already loaded so no request reads the profile twice.
  • lib/userSettings.ts, lib/chat/streaming.ts, routes/chat.ts, routes/tabular.ts - committees are loaded once with the rest of the profile and threaded to dispatch.
  • routes/user.ts - modelCommittees on the profile (select / serialize / PATCH) plus GET /user/models. Profile reads get their own retry tier, so a database without the column keeps every other live column.
  • migrations/20260823_02_user_model_committees.sql + schema.sql.

Frontend

  • components/settings/CommitteeSettingsSection.tsx - create, edit and delete committees. Only models the user can actually run are offered as members.
  • ModelToggle - a Committee group, shown only when every member and the chair is usable.
  • mikeApi / UserProfileContext - committee types, profile state, save.

Design notes

Availability is all-or-nothing. A committee is hidden from the picker and refused at request time unless every member and its chair is usable. Half a committee is not a usable answer.

A deleted committee is an explicit error, not a silent fallback to another model - quietly answering with a different model misrepresents which model wrote the response.

No tool-calling loop. When the caller supplies tools (the main chat path always does) they are dropped and the members are told so, rather than being left to claim document or case-law work they could not do.

Cost. Members are separate model calls, so a committee costs roughly the sum of its members plus the chair. Running them concurrently keeps the latency close to the slowest member instead of the sum.

Why

Different models disagree in useful ways on legal questions. A committee makes that disagreement visible to a synthesizing model rather than making the user run the same prompt several times by hand.

Testing

  • backend: tsc --noEmit clean; vitest run - 789 passed, 25 skipped (+25 over #339's foundation, covering committee dispatch, concurrency, cycle/self-reference guards, and committee validation).
  • frontend: tsc --noEmit clean; npm run lint 0 errors; vitest run - 564 passed, 14 failed. All 14 failures are pre-existing on main (mikeApi.test.ts Blob handling, useSelectedModel, ChatInput.modelSelection) and were confirmed against the baseline before these changes.
  • Migration follows the dated convention with schema.sql updated to match.

28 files changed in this commit (1681 insertions); 37 files and 3498 insertions including #339's foundation.

Our analysis

Add user-defined model committees — read the full analysis →

Think the analysis missed something the PR description covers?

Capture this PR into my fork

Download a Markdown prompt that tells Claude how to port every commit in this PR into your working tree. Run it via claude -p < capture-pull-341.md from inside the repo you want the changes in.

⬇ Download capture-pull-341.md