[Testing 12] test: Playwright e2e suite + CI merge gate (auth, chat, projects, tabular, workflows)
From the PR description
Merge-order note: please merge #212 (the missing
chat_messages.workflowcolumn fix) before this. The cold-load chat spec currently has to test around that bug - messages silently fail to persist on a fresh database - by asserting the empty-chat redirect instead of rendered messages. Once #212 lands, that spec will be strengthened to assert real message rendering.
Summary
Adds a Playwright end-to-end suite plus the CI workflow that runs it on every PR, so changes can be gated on a real browser-driven pass. 27 specs across 7 files cover auth flows, the critical path (create project → upload PDF → ask a question → streamed answer), chat management, project management, tabular reviews, and workflows/account settings.
Verified green against this repo's own stack: the workflow builds a full local environment from backend/schema.sql + backend/migrations/ and runs the suite - latest run 23 passed / 4 skipped / 0 failed in ~7 min.
Changes
e2e/- the specs plusauth.setup.ts(idempotently bootstraps a confirmede2e@mike.localvia the Supabase admin API and saves PlaywrightstorageState) andllm.ts(skip-guard helper).playwright.config.ts- single worker (shared backend + one test user), asetupproject feeding achromiumproject; localwebServerbootsbackend/andfrontend/, disabled under CI (the workflow owns the servers)..github/workflows/e2e.yml- on everypull_request, boots MinIO + local Supabase (loadsschema.sql, applies migrations, grantsservice_role, reloads PostgREST), builds and serves a production web build, then runs Playwright and uploads the report/traces.- Root
package.json/package-lock.json/tsconfig.json- Playwright dev-deps,test:e2e*scripts, and typechecking scoped toe2e/.docs/e2e-ci.mddocuments setup and the branch-protection step..gitignoreentries for Playwright artifacts.
Why
The repo has no end-to-end coverage today; a regression in auth, chat, projects, tabular, or workflows ships silently. This makes those flows a merge gate. Two decisions worth calling out:
- Production build, not
next dev- dev's on-demand compilation and hydration-error overlay make the suite flaky; a production build is deterministic. - Schema + migrations +
service_rolegrant -schema.sqlalone lags the migrations (e.g.GET /workflows/:idqueriesworkflow_open_source_submissions, only a migration creates it) and, loaded via rawpsql, leavesservice_rolewithout table grants; the workflow reconciles both so CI matches production behavior.
Testing
- CI:
e2e / playwright- 23 passed, 4 skipped, 0 failed (the 4 skips are the LLM-dependent specs; see below). npx tsc --noEmittypechecks the suite.- Locally:
npm ci && npx playwright install --with-deps chromium && npm run test:e2eagainst a running stack.
Optional secret - the 4 LLM-dependent specs (chat send / critical-path) test.skip when ANTHROPIC_API_KEY is unset, so the suite is green without any secret; set the key to also run and enforce them.
To make it merge-blocking, mark e2e / playwright a required status check in branch protection - steps and a gh api one-liner are in docs/e2e-ci.md. The workflow reports pass/fail on its own; only branch protection turns a red run into a blocked merge.
Flakiness root-cause fix (update)
An earlier revision of these specs carried retry loops and settle-waits around intermittent local-Supabase gateway 5xxs. That flakiness has now been root-caused and fixed, and the scaffolding removed - the suite passes with none of it.
Diagnosis. useDirectoryData fired one GET /projects/:id per existing project every time a directory modal opened (Add Documents / New Project / New Review / Use Workflow). Each request costs a GoTrue auth check plus ~6 PostgREST queries, so the burst scaled ~8x with project count. Reproduced against the Supabase CLI stack with a 43-project account: overlapping modal-open storms drove GoTrue into Postgres connection exhaustion (failed to connect ... user=supabase_auth_admin ... context deadline exceeded), yielding 467x 500 + 641x 504 on /auth/v1/user in one run - 38/48 creates and ~70% of the GETs failed. On CI-class hardware Kong surfaces the same collapse as the 502s the retries papered over.
Fix. #238 (olp-pr/directory-fetch-storm) - GET /projects?include=documents returns each project's documents from one batched query, and useDirectoryData uses it: one request per modal open regardless of project count. After the fix the identical storm harness produced zero failures (96/96 OK). The same commit is cherry-picked onto this branch so the suite passes here; git dedupes it when both merge.
Retry scaffolding removed (test(e2e): drop transient-5xx retry scaffolding...): create-with-retry loops (project x2, chat x2, workflow, tabular-review, review add-document), reload-guards for transient "Project not found", the networkidle settle-waits that let the fetch burst drain before submitting, and the storm-sized 180s test timeouts. Kept: the title-generation settle logic, two bounded retries that guard documented client-side hydration races unrelated to the gateway (chat submit model seeding; account display-name), and Playwright's own retries: 2 on CI.
Evidence. With the fix and the scaffolding stripped, 3 consecutive full-suite runs (after 2 pre-de-retry runs with the fix) were green - 23 passed / 4 LLM-skipped each time, zero Playwright retries - against a production build and an account seeded with ~40 projects, a larger directory fan-out than CI's fresh database ever produces.
LLM specs verified (update)
The 4 LLM-gated specs previously selected a "Demo (no key needed)" model that exists only in a fork's demo provider - not in this repo's ModelToggle.MODELS - so setting the ANTHROPIC_API_KEY secret unskipped them and they then failed at model selection. Fixed in test(e2e): select a real Claude model in LLM-gated specs: the helper now picks Claude Sonnet 4.6 (the cheapest Anthropic entry in the toggle), and critical-path Step 8 asserts a nonempty streamed assistant answer (presence-based, deterministic against real LLM output) instead of the fork's canned "Demo mode" reply.
Verified locally against a production build + Supabase CLI stack + MinIO with only ANTHROPIC_API_KEY configured:
- keyless Playwright env: 23 passed / 4 skipped (skip guards intact)
ANTHROPIC_API_KEYexported: 27 passed / 0 skipped (1.2m, zero retries)npx tsc --noEmitclean
With the secret set, a full run makes ~4 Sonnet 4.6 chat turns plus up to 4 Haiku 4.5 title generations - a few cents per run.
Our analysis
Add Playwright coverage for critical legal-AI flows — read the full analysis →
Think the analysis missed something the PR description covers?
Commits in this PR (17)
| SHA | Subject | Author | Date | |
|---|---|---|---|---|
94987e6b | test: Playwright e2e suite (auth, chat, projects, tabular reviews, workflows) | Amal | 2026-07-17 | ↗ GitHub |
commit bodyPort of the amal66/mike fork's Playwright end-to-end suite onto the upstream backend/ + frontend/ layout: - e2e/: auth.setup (bootstraps a confirmed e2e@mike.local user via the Supabase admin API and saves storageState), auth-flows, critical-path (create project -> upload PDF -> ask a question -> streamed response), chat-management, project-management, tabular-reviews, workflows-account; fixtures/test.pdf - playwright.config.ts: single-worker (shared test user), setup project + chromium project with saved auth state; webServer adapted from the fork's monorepo command (npm run dev --workspace apps/web) to upstream's layouts: backend `npm run dev` (health-checked on :3001) and frontend `npm run dev` (:3000) - root package.json (upstream has none): @playwright/test, typescript dev-deps and test:e2e scripts, trimmed from the fork's root manifest - root tsconfig.json scoped to e2e/ + playwright.config.ts so `npx tsc --noEmit` typechecks the suite - .gitignore: playwright artifacts + e2e/.auth (session tokens) Specs select by ARIA role/name and placeholder text only - no data-testid attributes, so no app-code changes are required. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CEguyEgXa9JjCciXCcVemC | ||||
ffad860e | fix(e2e): realign selectors with the olp UI and upstream route shape | Amal | 2026-07-17 | ↗ GitHub |
commit bodyupstream-main (olp's tip) ships the #215/#216 liquid-surfaces UI, so the same
selector drifts fixed on the fork's suite (commit 3d46814 on main) also apply
here. Ports those realigned hunks, plus one upstream-specific route fix.
* Chat input placeholder is "How can I help?", not
"Ask a question about your documents..." (ChatInput/TRChatInput). Updated in
chat-management (rename/delete/project-assistant) and critical-path.
* The project-assistant empty state replaced the "+ Create New" text link with
a PillButton reading "Create" (ProjectAssistantTable) - critical-path and
chat-management now use getByRole("button", { name: "Create" }).
* The sidebar chat row's active marker is APP_SURFACE_ACTIVE_CLASS
("bg-app-surface-active"), not "bg-gray-200/60"; the row wrapper is now h-8,
not h-9 (SidebarChatItem). The rename/delete tests locate the row accordingly.
* The documents-toolbar folder button is "Folder" (TabPillButton wired to the
root createFolderAction in ProjectDocumentsView), not "Add Subfolder"; it
still opens the autofocused "Folder name" root input.
* NewTRModal's footer submit and the tabular page CTA both read "Create", so
the create-review helper scopes to the modal submit
(button[name="modalAction"][value="create-review"]).
* The built-in workflow detail test navigated to the flat /workflows/[id],
which upstream does not route - only the typed /workflows/assistant/[id] and
/workflows/tabular-review/[id] exist. builtin-cp-checklist is assistant-type,
so it now navigates to /workflows/assistant/builtin-cp-checklist (the path
the app itself links to via workflowDetailPath; works on the fork too).
Verified: full 27-test suite green against upstream-main + test-harness +
demo-mode(+provider-registry), backend + local Supabase (upstream schema.sql +
backend/migrations) + MinIO, demo model, e2e@mike.local.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CEguyEgXa9JjCciXCcVemC
| ||||
d8d38174 | ci(e2e): run Playwright suite on PRs as a merge gate | QA Runner | 2026-07-17 | ↗ GitHub |
commit bodyAdds .github/workflows/e2e.yml adapted to this repo's backend/ + frontend/ layout: on every pull_request into main (or upstream-main), boot MinIO + local Supabase (loading backend/schema.sql), start the API and web dev servers, and run the Playwright suite, uploading the HTML report/traces on pass or fail. playwright.config.ts already disables its local webServer when CI=true, so the job owns the stack. docs/e2e-ci.md documents the one required secret (ANTHROPIC_API_KEY) and the branch-protection steps that make the 'e2e / playwright' check required, so a red run blocks the merge button. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> | ||||
d427cc26 | ci(e2e): upload the Playwright report even on timeout/cancel | QA Runner | 2026-07-17 | ↗ GitHub |
commit bodySwitch the artifact step from !cancelled() to always() so a run that hits the 30-minute job timeout (the failure mode when many specs retry) still uploads the HTML report and traces instead of skipping the upload - that partial report is exactly what's needed to diagnose the failing specs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> | ||||
9daee405 | ci(e2e): grant service_role table access after loading schema.sql | QA Runner | 2026-07-17 | ↗ GitHub |
commit bodyThe backend queries exclusively as service_role (lib/supabase.ts, service key, 'bypasses RLS'). schema.sql revokes client grants (anon/authenticated) but assumes a hosted Supabase where service_role already has full table access - on a fresh CLI stack loaded via psql it gets none, so the first backend write 500s with 'permission denied for table user_profiles' and every project/chat spec fails. Granting service_role after the schema load reproduces the production grant posture. Verified in CI: POST /projects 500 -> 201, GET /chat 500 -> 200. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> | ||||
42346d69 | ci(e2e): apply migrations, serve prod build, skip LLM specs without a key | QA Runner | 2026-07-17 | ↗ GitHub |
commit bodyThree fixes that take the suite from ~everything-failing to green (verified end
to end against a local stack):
1. schema.sql lags the migrations (missing e.g. workflow_open_source_submissions,
which GET /workflows/:id queries → 500). Apply every dated migration on top of
schema.sql (idempotent; 0 errors on a fresh DB), grant service_role AFTER so
new tables are covered, then reload PostgREST. Fixes the workflow specs.
2. Serve a production build (next build + next start) instead of next dev. The dev
server's on-demand compilation (slow first hit → waitForResponse timeouts) and
hydration-error overlay (injects nextjs__container_errors DOM that pollutes text
locators, e.g. getByText('All') matching 'Call Stack') made the suite flaky.
Fixes tabular-reviews list render + the timing flakes.
3. LLM-dependent specs (chat rename/delete/submit, critical-path) require a model
key to send a message. Guard them with test.skip(!hasLlmKey) (e2e/llm.ts) and
expose ANTHROPIC_API_KEY to the Playwright process, so a keyless run is green on
the ~23 other specs and the LLM specs run + enforce only when the secret is set.
Local result (no key, with MinIO+LibreOffice as on ubuntu-latest): 23 passed,
4 skipped, 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| ||||
094fe565 | docs(e2e): document keyless-green behavior, migrations, and prod build | QA Runner | 2026-07-17 | ↗ GitHub |
commit bodyThe suite is green with no secret (LLM specs skip). Update docs/e2e-ci.md and the workflow header to reflect the migrations step, the production build, and the optional (not required) ANTHROPIC_API_KEY. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> | ||||
8bb44e9c | test(e2e): update built-in workflow id after upstream rename | QA Runner | 2026-07-20 | ↗ GitHub |
commit bodyThe workflows UI refresh (olp/main fa21ac8) renamed the built-in workflow id builtin-cp-checklist -> builtin-draft-cp-checklist (title 'Draft CP Checklist' unchanged). The read-only-detail spec navigated to the old id URL and got a 'workflow not found' page. Point it at the new id. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> | ||||
5f8e4ab4 | fix: batch directory-modal project fetch instead of N+1 getProject burst | QA Runner | 2026-07-20 | ↗ GitHub |
commit bodyEvery directory picker (AddDocumentsModal, UseWorkflowModal, the assistant
project selector) loads its "Projects" tab through useDirectoryData, which
fired one GET /projects/:id for EVERY existing project the moment the modal
opened. Each of those requests costs an auth verification against GoTrue
plus ~6 PostgREST queries, so an account with N projects produced an
~8xN-request burst on the Supabase gateway per modal open.
Under that burst the gateway genuinely falls over: measured locally against
the Supabase CLI stack, overlapping modal-open storms drove GoTrue into
Postgres connection exhaustion ("failed to connect ... context deadline
exceeded") and produced 467x 500 + 641x 504 on /auth/v1/user in a single
run - surfacing to users as failed project creates/loads, and to the e2e
suite as the intermittent Kong 502s its specs currently retry around.
Fix: GET /projects now accepts ?include=documents and returns each
project's documents from one batched query (same attach helpers as
GET /projects/:id, run once across all documents), and useDirectoryData
uses it. A modal open is now 1 API request and a fixed number of DB
queries regardless of project count. After the change the same storm
harness produced zero 5xx and zero auth failures.
[Cherry-pick overlap note: this same commit is submitted separately as the
olp-pr/directory-fetch-storm PR, where it belongs (production fix). It is
carried on this e2e branch only so the de-retried suite passes here; on
merge of both branches git resolves the duplicate cleanly.]
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| ||||
af38a450 | test(e2e): drop transient-5xx retry scaffolding now that the fetch storm is fixed | QA Runner | 2026-07-20 | ↗ GitHub |
commit bodyWith the directory fetch storm eliminated (previous commit: batched listProjects?include=documents instead of a getProject burst per modal open), the local Supabase gateway no longer collapses under modal-open load, so the specs' 5xx-oriented scaffolding is removed: - create-with-retry loops (project create x2, chat create x2, workflow create, tabular-review create x10-attempt, review add-document x6) - reload-guards for transient "Project not found" (waitForProjectLoaded, gotoProjectRow, project-assistant tab loop) - networkidle settle-waits whose purpose was to let the per-project fetch burst drain before submitting (NewProjectModal / NewTRModal) - oversized per-test timeouts justified by the storm (180s -> 60-120s) Kept, deliberately: - the title-generation settle logic in chat rename/delete - chat submit's model-seeding retry loop (guards a documented useSelectedModel localStorage hydration race, unrelated to the gateway) - account display-name toPass block (guards a documented profile hydration race, unrelated to the gateway) - generic short networkidle settles after login/logout navigation - Playwright's own retries: 2 on CI Verified: with the fix, 3 consecutive full-suite runs (plus 2 more before de-retry) were green with zero Playwright retries, running against an account seeded with ~40 projects - a larger directory fan-out than CI's fresh database ever produces. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | ||||
d4448fa8 | test(e2e): select a real Claude model in LLM-gated specs (demo model is fork-only) | QA Runner | 2026-07-20 | ↗ GitHub |
commit bodyThe 4 LLM-gated specs (chat-management rename/delete/project-assistant and
the critical-path project flow) selected a "Demo (no key needed)" model in
the ModelToggle. That model exists only in a fork's demo provider - upstream
ModelToggle.MODELS has no such entry - so on this repo, setting the
ANTHROPIC_API_KEY secret unskipped the specs and they then failed at the
model-selection step.
Fix:
- Replace the selectDemoModel helpers with selectClaudeModel, which picks
"Claude Sonnet 4.6" (the cheapest Anthropic entry in ModelToggle.MODELS).
With ANTHROPIC_API_KEY exported to the backend, userApiKeys.envApiKey()
reports the claude provider as configured, so the model is available and
the submit is not blocked by the ApiKeyMissingModal.
- critical-path Step 8: the canned-reply assertion getByText("Demo mode")
becomes a presence + nonempty assertion on the assistant answer container
(MarkdownContent's "div.prose.font-serif.text-gray-900", unique to
assistant answer content) - deterministic against nondeterministic real
LLM output, same 60s budget.
- Comments rewritten to describe the upstream reality: specs run only when
ANTHROPIC_API_KEY is set (e2e/llm.ts), the backend uses the env key, and
title generation resolves to claude-haiku-4-5 via resolveTitleModel.
Verified against a local prod-build stack (backend :3001, frontend :3002,
local Supabase + MinIO), only ANTHROPIC_API_KEY configured:
- keyless Playwright env: 23 passed / 4 skipped
- ANTHROPIC_API_KEY exported: 27 passed / 0 skipped (1.2m)
- npx tsc --noEmit clean
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| ||||
62da94be | test(e2e): self-bootstrapping local Supabase stack for local runs | willchen96 | 2026-07-25 | ↗ GitHub |
commit bodynpm run test:e2e:local (or plain test:e2e via the webServer hook) boots Docker-checked local Supabase, loads schema + migrations + grants, wires backend/.env and frontend/.env.local, and raises the backend rate-limit caps the same way CI does - a full suite run exceeds the default 300-requests/15-min general cap, after which every call 429s and the profile/list waits time out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | ||||
cb2306c5 | fix(frontend): unify full-screen loaders to remove hydration mismatch | willchen96 | 2026-07-25 | ↗ GitHub |
commit bodyThe provider Suspense fallback, the MFA gate loader, and the (pages) layout auth loader were three near-identical spinners with different Tailwind classes. Server and client can resolve those gates differently on first paint (auth settles from localStorage before the boundary hydrates), so the differing attributes surfaced as a hydration mismatch on nearly every dev page load. One shared FullScreenLoader makes every branch render byte-identical DOM. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | ||||
4728fd19 | chore(frontend): pin turbopack.root to silence multi-lockfile warning | willchen96 | 2026-07-25 | ↗ GitHub |
The repo has lockfiles at the root (Playwright harness) and in frontend/, so Next.js guesses the workspace root and warns on every dev-server start. Pin it explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | ||||
c196f9b2 | Merge remote-tracking branch 'origin/main' into olp-pr/e2e-playwright | willchen96 | 2026-07-25 | ↗ GitHub |
f3173a9d | ci(e2e): narrow post-migration service_role grants to match schema.sql | willchen96 | 2026-07-26 | ↗ GitHub |
commit bodyschema.sql now grants service_role its own narrowed privileges (upstream "narrow service role schema grants"), so the broad GRANT ALL workaround is obsolete - and worse, it widened the test database beyond what prod gets, letting e2e pass on privileges a real deployment lacks. Keep a post-migration re-grant (schema.sql's ON ALL statements can't cover tables the migrations create) but with the identical narrowed set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | ||||
fb3ec2d6 | chore: gitignore the local Supabase scaffold | willchen96 | 2026-07-26 | ↗ GitHub |
commit bodybackend/supabase/ is supabase-init output used only by the local e2e stack; CI scaffolds its own fresh copy each run, so it must not be committed - hide it from status/source-control noise. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | ||||
Capture this PR into my fork
Download a Markdown prompt that tells Claude how to port every
commit in this PR into your working tree. Run it via
claude -p < capture-pull-220.md from
inside the repo you want the changes in.