Provider SDKs deleted; all LLM traffic routes through Vercel AI Gateway

hosman20 swaps three provider SDKs for a single gateway client and removes the customer-facing API key UI entirely. The platform now pays for inference; customers don't see provider tokens.

infrastructureintegration

Commit 07a46ce is a +708/-1353 change across 19 files. The deleted side includes backend/src/lib/llm/{claude,gemini,openai,tools}.ts, backend/src/lib/userApiKeys.ts, the GET/PUT /user/api-keys routes, and frontend/src/app/(pages)/account/models/page.tsx (341 lines). Four env vars go away: GEMINI_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY, and USER_API_KEYS_ENCRYPTION_SECRET.

The replacement is backend/src/lib/llm/gateway.ts, 343 lines, using ai@^6.0.182 and @ai-sdk/gateway. The adapter accepts "provider/model" strings and auto-routes based on the prefix. On Vercel, VERCEL_OIDC_TOKEN is preferred over the static AI_GATEWAY_API_KEY. Per-request attribution moves from apiKeys params to a StreamChatAttribution shape forwarded as x-user-id and x-org-id headers, which the gateway uses to slice cost reporting.

runLLMStream gains totalTokens?: number on its return type, pulled from streamResult.usage?.totalTokens. That's what the billing middleware reads to increment tokens_used_this_period. The gateway test suite (gateway.test.ts, 266 lines) mocks the entire ai package so tests don't hit providers; it drives onChunk manually and inspects the headers and model slug passed to streamText.

Two leftovers: mikeApi.ts still exports stale getApiKeyStatus/saveApiKey functions pointing to 404 routes (cleaned in the Batch 7 follow-up), and public.user_api_keys in Postgres is left orphaned.

So what Worth importing as a unit if you want to flip to a platform-pays model and already use Vercel. The pieces move together - routes, types, env, and frontend tabs are all coupled. Pulling only the gateway adapter without the rest of the removal leaves broken routes and dead UI in place.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

1 commit from hosman20/mike-2.0, oldest first. Source extracted verbatim from the harvested git log.

SHA Subject Author Date

07a46ceb feat(llm): replace direct provider SDKs with Vercel AI Gateway z 2026-05-13 ↗ GitHub

SHA	Subject	Author	Date
`07a46ceb`	feat(llm): replace direct provider SDKs with Vercel AI Gateway	z	2026-05-13	↗ GitHub
commit body Batch 4 - route all LLM traffic through the Vercel AI Gateway via AI SDK v6. Tokens are now paid by the platform, not the customer. Backend: - New gateway adapter at backend/src/lib/llm/gateway.ts (uses ai@^6.0.182, "provider/model" strings auto-route via the bundled gateway provider). - 12 unit tests in backend/src/lib/llm/__tests__/gateway.test.ts. - Deleted backend/src/lib/llm/{claude,gemini,openai,tools}.ts and backend/src/lib/userApiKeys.ts - the per-provider SDKs and the customer-key vault are gone. - backend/src/lib/llm/index.ts now re-exports from gateway and routes streamChatWithTools / completeText through it. - backend/src/lib/llm/types.ts: added StreamChatAttribution ({ userId?, orgId? }) for gateway cost-slicing; UserApiKeys kept as @deprecated alias so existing call-sites still typecheck. - backend/src/lib/userSettings.ts: drop per-user api_keys lookup. - backend/src/lib/chatTools.ts: runLLMStream now forwards attribution instead of apiKeys and surfaces aggregate usage.totalTokens. - backend/src/routes/{chat,projectChat,tabular}.ts: attribution ({ userId }) replaces apiKeys; legacy queryGemini/title helpers refactored to take userId. - backend/src/routes/user.ts: deleted GET/PUT /user/api-keys. - backend/.env.example: replaced GEMINI_/ANTHROPIC_/OPENAI_API_KEY and USER_API_KEYS_ENCRYPTION_SECRET with AI_GATEWAY_API_KEY. Frontend: - Deleted frontend/src/app/(pages)/account/models/page.tsx - the user-facing API key UI no longer applies. - frontend/src/app/(pages)/account/layout.tsx: removed the "Models & API Keys" tab (Billing tab arrives in batch 6). Intermediate state notes: - frontend mikeApi.ts still exports getApiKeyStatus + saveApiKey against now-404 routes; UserProfileContext still imports them. Both are cleaned up in batches 6/8. - public.user_api_keys table is not dropped here - orphaned but harmless; can be removed in a follow-up migration. Agent ID: a5511de9cafb2355a (vercel:ai-architect).

commit body

Batch 4 - route all LLM traffic through the Vercel AI Gateway via
AI SDK v6. Tokens are now paid by the platform, not the customer.

Backend:
- New gateway adapter at backend/src/lib/llm/gateway.ts (uses
  ai@^6.0.182, "provider/model" strings auto-route via the bundled
  gateway provider).
- 12 unit tests in backend/src/lib/llm/__tests__/gateway.test.ts.
- Deleted backend/src/lib/llm/{claude,gemini,openai,tools}.ts and
  backend/src/lib/userApiKeys.ts - the per-provider SDKs and the
  customer-key vault are gone.
- backend/src/lib/llm/index.ts now re-exports from gateway and routes
  streamChatWithTools / completeText through it.
- backend/src/lib/llm/types.ts: added StreamChatAttribution
  ({ userId?, orgId? }) for gateway cost-slicing; UserApiKeys kept
  as @deprecated alias so existing call-sites still typecheck.
- backend/src/lib/userSettings.ts: drop per-user api_keys lookup.
- backend/src/lib/chatTools.ts: runLLMStream now forwards attribution
  instead of apiKeys and surfaces aggregate usage.totalTokens.
- backend/src/routes/{chat,projectChat,tabular}.ts: attribution
  ({ userId }) replaces apiKeys; legacy queryGemini/title helpers
  refactored to take userId.
- backend/src/routes/user.ts: deleted GET/PUT /user/api-keys.
- backend/.env.example: replaced GEMINI_/ANTHROPIC_/OPENAI_API_KEY
  and USER_API_KEYS_ENCRYPTION_SECRET with AI_GATEWAY_API_KEY.

Frontend:
- Deleted frontend/src/app/(pages)/account/models/page.tsx - the
  user-facing API key UI no longer applies.
- frontend/src/app/(pages)/account/layout.tsx: removed the
  "Models & API Keys" tab (Billing tab arrives in batch 6).

Intermediate state notes:
- frontend mikeApi.ts still exports getApiKeyStatus + saveApiKey
  against now-404 routes; UserProfileContext still imports them.
  Both are cleaned up in batches 6/8.
- public.user_api_keys table is not dropped here - orphaned but
  harmless; can be removed in a follow-up migration.

Agent ID: a5511de9cafb2355a (vercel:ai-architect).

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-406.md from inside the repo you want the changes in.

⬇ Download capture-thread-406.md