[Fix 01] Batch the directory modal's project fetch (N+1 getProject storm)

✅ merged · #238 · open-legal-products/mike ← amal66/mike · opened 3mo ago by amal66 · merged 3mo ago by willchen96 · +58-17 across 3 files · ↗ on GitHub

From the PR description

Summary

Every directory picker (Add Documents modal, Use Workflow modal, the assistant project selector) loads its Projects tab through useDirectoryData, which fired one GET /projects/:id for every existing project the moment the modal opened. Each of those requests costs a GoTrue auth verification plus ~6 PostgREST queries, so an account with N projects hit the Supabase gateway with an ~8xN-request burst per modal open. This PR replaces the fan-out with a single batched request.

Changes

  • backend/src/routes/projects.ts: GET /projects accepts ?include=documents and returns each project's documents from one batched documents query (.in("project_id", ...)), running the same attach helpers as GET /projects/:id once across all documents. Response shape without the flag is unchanged.
  • frontend/src/app/lib/mikeApi.ts: listProjects() takes an optional { includeDocuments }.
  • frontend/src/app/components/shared/useDirectoryData.ts: loadProjects() makes one listProjects({ includeDocuments: true }) call instead of listProjects() + Promise.all(projects.map(getProject)).

Why

The burst is not just slow - it takes the gateway down. Reproduced against the Supabase CLI stack with a 43-project account: overlapping modal-open storms (6 concurrent opens, 8 rounds, a project-create POST fired mid-burst each time, i.e. exactly what the e2e suite does) drove GoTrue into Postgres connection exhaustion - failed to connect to host=supabase_db_mike user=supabase_auth_admin ... context deadline exceeded - producing 467x 500 + 641x 504 on /auth/v1/user in a single run; 38/48 project creates and 1458/2064 project GETs failed. On slower CI runners Kong surfaces the same collapse as the intermittent 502 ("An invalid response was received from the upstream server") that the e2e suite (#220) had to retry around. Real users with accumulated projects hit the same wall.

Testing

  • Same storm harness after the fix (one batched request per modal open, same overlap and mid-burst POSTs): 0 failures - 48/48 GETs and 48/48 creates OK, zero 5xx anywhere in the gateway logs.
  • Backend tsc --noEmit clean; frontend production build clean.
  • Full Playwright e2e suite run 5x consecutively against a production build with this fix (including 3 runs after the retry scaffolding was stripped from the specs in #220): 23/23 non-LLM specs green every run, zero Playwright retries, with ~40 accumulated projects in the test account.

🤖 Generated with Claude Code

Our analysis

Batch project document loading for directory pickers — read the full analysis →

Think the analysis missed something the PR description covers?

Commits in this PR (1)

SHA Subject Author Date
36cddb25 fix: batch directory-modal project fetch instead of N+1 getProject burst QA Runner 2026-07-20 ↗ GitHub
commit body
Every directory picker (AddDocumentsModal, UseWorkflowModal, the assistant
project selector) loads its "Projects" tab through useDirectoryData, which
fired one GET /projects/:id for EVERY existing project the moment the modal
opened. Each of those requests costs an auth verification against GoTrue
plus ~6 PostgREST queries, so an account with N projects produced an
~8xN-request burst on the Supabase gateway per modal open.

Under that burst the gateway genuinely falls over: measured locally against
the Supabase CLI stack, overlapping modal-open storms drove GoTrue into
Postgres connection exhaustion ("failed to connect ... context deadline
exceeded") and produced 467x 500 + 641x 504 on /auth/v1/user in a single
run - surfacing to users as failed project creates/loads, and to the e2e
suite as the intermittent Kong 502s its specs currently retry around.

Fix: GET /projects now accepts ?include=documents and returns each
project's documents from one batched query (same attach helpers as
GET /projects/:id, run once across all documents), and useDirectoryData
uses it. A modal open is now 1 API request and a fixed number of DB
queries regardless of project count. After the change the same storm
harness produced zero 5xx and zero auth failures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Capture this PR into my fork

Download a Markdown prompt that tells Claude how to port every commit in this PR into your working tree. Run it via claude -p < capture-pull-238.md from inside the repo you want the changes in.

⬇ Download capture-pull-238.md