[Docs 13] docs(e2e): exact ANTHROPIC_API_KEY setup to enable the 4 LLM-gated specs
From the PR description
Summary
Follow-up to #220: everything a maintainer needs to switch on the four LLM-gated e2e specs (chat rename, chat delete, project-assistant create+submit, and the critical-path "ask a question" flow) - exact ANTHROPIC_API_KEY secret setup (UI + CLI), the fork-PR secrets caveat, expected cost, and how to verify in the Actions log that the specs ran instead of skipped.
Stacked on #220's content. Because #220 is not merged yet, this branch carries its e2e commits (10 commits through 039e433) so the docs can be reviewed against the specs they describe. Review the top 3 commits only:
282b2dd- docs(e2e): exact ANTHROPIC_API_KEY setup to enable the 4 LLM specs; correct the gate rationale0cc635b- docs(e2e): mark the model-selection gap fixed - specs pick a Claude model when the key is set72242ff- docs(e2e): reconcile the docs with what this branch actually contains
Once #220 merges, this rebases down to just those three.
Changes
docs/e2e-ci.md- "Enable the LLM specs" section: step-by-step UI path (Settings → Secrets and variables → Actions → New repository secret →ANTHROPIC_API_KEY), CLI (gh secret set ANTHROPIC_API_KEY --repo Open-Legal-Products/mike), the fork-PR caveat (secrets are withheld from forkpull_requestruns; use a maintainer branch orworkflow_dispatch), per-run cost (a handful of short completions - cents), and how to read the "Run Playwright" log. Plus a "Model selection" section describing theselectClaudeModelfix this branch carries.e2e/llm.ts- doc comment rewritten with the precise gate rationale: this codebase ships no keyless model (everyModelToggleentry needs an Anthropic/Google/OpenAI key beforeChatInputwill send; the backend model set has no demo id either), so keyless the four specs cannot submit at all - and the auto title-generation 500 is not why the gate exists. Comment-only..github/workflows/e2e.yml- comment-only alignment with the docs.
No behavioral change in the doc commits; the four test.skip(!hasLlmKey, LLM_SKIP_REASON) lines are untouched.
Docs accuracy fix (top commit)
Earlier revisions of these docs told inconsistent stories about model selection (an open "known gap" in one file, "fixed on another branch" in another, plus pass/skip counts with no provenance). The top commit reconciles them so the docs describe exactly what this branch contains:
- The specs in this branch already use
selectClaudeModel(Claude Sonnet 4.6) when the key is set; the staleselectDemoModel"known gap" paragraph ine2e/llm.tsis gone. - The 23 passed / 4 skipped keyless and 27 passed / 0 skipped with-key figures are now attributed to where they were measured - the spec-fix commit's own verification against a full local stack (see the
test(e2e): select a real Claude model...commit message) - and the docs state they have not been re-measured since this branch was rebased onto currentmain. The unattributed "~7 minutes typical run" figure is dropped in favor of the structural 27-spec / 4-gated breakdown.
Testing
npx tsc --noEmitat the repo root (coverse2e/): clean.npx playwright test --list: 27 tests in 7 files - the specs parse and every referenced helper resolves (selectClaudeModelexists ine2e/chat-management.spec.tsande2e/critical-path.spec.ts).- Keyless-skip logic verified by inspection:
e2e/llm.tsgates onBoolean(process.env.ANTHROPIC_API_KEY)and exactly 4 specs calltest.skip(!hasLlmKey, LLM_SKIP_REASON). A full e2e run needs the local Supabase stack and was not attempted for this docs PR.
🤖 Generated with Claude Code
Our analysis
Document LLM-gated e2e checks — read the full analysis →
Think the analysis missed something the PR description covers?
Commits in this PR (14)
| SHA | Subject | Author | Date | |
|---|---|---|---|---|
916f2600 | test: Playwright e2e suite (auth, chat, projects, tabular reviews, workflows) | Amal | 2026-07-17 | ↗ GitHub |
commit bodyPort of the amal66/mike fork's Playwright end-to-end suite onto the upstream backend/ + frontend/ layout: - e2e/: auth.setup (bootstraps a confirmed e2e@mike.local user via the Supabase admin API and saves storageState), auth-flows, critical-path (create project -> upload PDF -> ask a question -> streamed response), chat-management, project-management, tabular-reviews, workflows-account; fixtures/test.pdf - playwright.config.ts: single-worker (shared test user), setup project + chromium project with saved auth state; webServer adapted from the fork's monorepo command (npm run dev --workspace apps/web) to upstream's layouts: backend `npm run dev` (health-checked on :3001) and frontend `npm run dev` (:3000) - root package.json (upstream has none): @playwright/test, typescript dev-deps and test:e2e scripts, trimmed from the fork's root manifest - root tsconfig.json scoped to e2e/ + playwright.config.ts so `npx tsc --noEmit` typechecks the suite - .gitignore: playwright artifacts + e2e/.auth (session tokens) Specs select by ARIA role/name and placeholder text only - no data-testid attributes, so no app-code changes are required. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CEguyEgXa9JjCciXCcVemC | ||||
e5510287 | fix(e2e): realign selectors with the olp UI and upstream route shape | Amal | 2026-07-17 | ↗ GitHub |
commit bodyupstream-main (olp's tip) ships the #215/#216 liquid-surfaces UI, so the same
selector drifts fixed on the fork's suite (commit 3d46814 on main) also apply
here. Ports those realigned hunks, plus one upstream-specific route fix.
* Chat input placeholder is "How can I help?", not
"Ask a question about your documents..." (ChatInput/TRChatInput). Updated in
chat-management (rename/delete/project-assistant) and critical-path.
* The project-assistant empty state replaced the "+ Create New" text link with
a PillButton reading "Create" (ProjectAssistantTable) - critical-path and
chat-management now use getByRole("button", { name: "Create" }).
* The sidebar chat row's active marker is APP_SURFACE_ACTIVE_CLASS
("bg-app-surface-active"), not "bg-gray-200/60"; the row wrapper is now h-8,
not h-9 (SidebarChatItem). The rename/delete tests locate the row accordingly.
* The documents-toolbar folder button is "Folder" (TabPillButton wired to the
root createFolderAction in ProjectDocumentsView), not "Add Subfolder"; it
still opens the autofocused "Folder name" root input.
* NewTRModal's footer submit and the tabular page CTA both read "Create", so
the create-review helper scopes to the modal submit
(button[name="modalAction"][value="create-review"]).
* The built-in workflow detail test navigated to the flat /workflows/[id],
which upstream does not route - only the typed /workflows/assistant/[id] and
/workflows/tabular-review/[id] exist. builtin-cp-checklist is assistant-type,
so it now navigates to /workflows/assistant/builtin-cp-checklist (the path
the app itself links to via workflowDetailPath; works on the fork too).
Verified: full 27-test suite green against upstream-main + test-harness +
demo-mode(+provider-registry), backend + local Supabase (upstream schema.sql +
backend/migrations) + MinIO, demo model, e2e@mike.local.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CEguyEgXa9JjCciXCcVemC
| ||||
82fd540e | ci(e2e): run Playwright suite on PRs as a merge gate | QA Runner | 2026-07-17 | ↗ GitHub |
commit bodyAdds .github/workflows/e2e.yml adapted to this repo's backend/ + frontend/ layout: on every pull_request into main (or upstream-main), boot MinIO + local Supabase (loading backend/schema.sql), start the API and web dev servers, and run the Playwright suite, uploading the HTML report/traces on pass or fail. playwright.config.ts already disables its local webServer when CI=true, so the job owns the stack. docs/e2e-ci.md documents the one required secret (ANTHROPIC_API_KEY) and the branch-protection steps that make the 'e2e / playwright' check required, so a red run blocks the merge button. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> | ||||
82cbe4d7 | ci(e2e): upload the Playwright report even on timeout/cancel | QA Runner | 2026-07-17 | ↗ GitHub |
commit bodySwitch the artifact step from !cancelled() to always() so a run that hits the 30-minute job timeout (the failure mode when many specs retry) still uploads the HTML report and traces instead of skipping the upload - that partial report is exactly what's needed to diagnose the failing specs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> | ||||
28a6cb64 | ci(e2e): grant service_role table access after loading schema.sql | QA Runner | 2026-07-17 | ↗ GitHub |
commit bodyThe backend queries exclusively as service_role (lib/supabase.ts, service key, 'bypasses RLS'). schema.sql revokes client grants (anon/authenticated) but assumes a hosted Supabase where service_role already has full table access - on a fresh CLI stack loaded via psql it gets none, so the first backend write 500s with 'permission denied for table user_profiles' and every project/chat spec fails. Granting service_role after the schema load reproduces the production grant posture. Verified in CI: POST /projects 500 -> 201, GET /chat 500 -> 200. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> | ||||
7039461c | ci(e2e): apply migrations, serve prod build, skip LLM specs without a key | QA Runner | 2026-07-17 | ↗ GitHub |
commit bodyThree fixes that take the suite from ~everything-failing to green (verified end
to end against a local stack):
1. schema.sql lags the migrations (missing e.g. workflow_open_source_submissions,
which GET /workflows/:id queries → 500). Apply every dated migration on top of
schema.sql (idempotent; 0 errors on a fresh DB), grant service_role AFTER so
new tables are covered, then reload PostgREST. Fixes the workflow specs.
2. Serve a production build (next build + next start) instead of next dev. The dev
server's on-demand compilation (slow first hit → waitForResponse timeouts) and
hydration-error overlay (injects nextjs__container_errors DOM that pollutes text
locators, e.g. getByText('All') matching 'Call Stack') made the suite flaky.
Fixes tabular-reviews list render + the timing flakes.
3. LLM-dependent specs (chat rename/delete/submit, critical-path) require a model
key to send a message. Guard them with test.skip(!hasLlmKey) (e2e/llm.ts) and
expose ANTHROPIC_API_KEY to the Playwright process, so a keyless run is green on
the ~23 other specs and the LLM specs run + enforce only when the secret is set.
Local result (no key, with MinIO+LibreOffice as on ubuntu-latest): 23 passed,
4 skipped, 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| ||||
e6d53c4f | docs(e2e): document keyless-green behavior, migrations, and prod build | QA Runner | 2026-07-17 | ↗ GitHub |
commit bodyThe suite is green with no secret (LLM specs skip). Update docs/e2e-ci.md and the workflow header to reflect the migrations step, the production build, and the optional (not required) ANTHROPIC_API_KEY. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> | ||||
9f9a280f | test(e2e): update built-in workflow id after upstream rename | QA Runner | 2026-07-20 | ↗ GitHub |
commit bodyThe workflows UI refresh (olp/main fa21ac8) renamed the built-in workflow id builtin-cp-checklist -> builtin-draft-cp-checklist (title 'Draft CP Checklist' unchanged). The read-only-detail spec navigated to the old id URL and got a 'workflow not found' page. Point it at the new id. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> | ||||
86a0d781 | test(e2e): drop transient-5xx retry scaffolding now that the fetch storm is fixed | QA Runner | 2026-07-20 | ↗ GitHub |
commit bodyWith the directory fetch storm eliminated (previous commit: batched listProjects?include=documents instead of a getProject burst per modal open), the local Supabase gateway no longer collapses under modal-open load, so the specs' 5xx-oriented scaffolding is removed: - create-with-retry loops (project create x2, chat create x2, workflow create, tabular-review create x10-attempt, review add-document x6) - reload-guards for transient "Project not found" (waitForProjectLoaded, gotoProjectRow, project-assistant tab loop) - networkidle settle-waits whose purpose was to let the per-project fetch burst drain before submitting (NewProjectModal / NewTRModal) - oversized per-test timeouts justified by the storm (180s -> 60-120s) Kept, deliberately: - the title-generation settle logic in chat rename/delete - chat submit's model-seeding retry loop (guards a documented useSelectedModel localStorage hydration race, unrelated to the gateway) - account display-name toPass block (guards a documented profile hydration race, unrelated to the gateway) - generic short networkidle settles after login/logout navigation - Playwright's own retries: 2 on CI Verified: with the fix, 3 consecutive full-suite runs (plus 2 more before de-retry) were green with zero Playwright retries, running against an account seeded with ~40 projects - a larger directory fan-out than CI's fresh database ever produces. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | ||||
039e433a | test(e2e): select a real Claude model in LLM-gated specs (demo model is fork-only) | QA Runner | 2026-07-20 | ↗ GitHub |
commit bodyThe 4 LLM-gated specs (chat-management rename/delete/project-assistant and
the critical-path project flow) selected a "Demo (no key needed)" model in
the ModelToggle. That model exists only in a fork's demo provider - upstream
ModelToggle.MODELS has no such entry - so on this repo, setting the
ANTHROPIC_API_KEY secret unskipped the specs and they then failed at the
model-selection step.
Fix:
- Replace the selectDemoModel helpers with selectClaudeModel, which picks
"Claude Sonnet 4.6" (the cheapest Anthropic entry in ModelToggle.MODELS).
With ANTHROPIC_API_KEY exported to the backend, userApiKeys.envApiKey()
reports the claude provider as configured, so the model is available and
the submit is not blocked by the ApiKeyMissingModal.
- critical-path Step 8: the canned-reply assertion getByText("Demo mode")
becomes a presence + nonempty assertion on the assistant answer container
(MarkdownContent's "div.prose.font-serif.text-gray-900", unique to
assistant answer content) - deterministic against nondeterministic real
LLM output, same 60s budget.
- Comments rewritten to describe the upstream reality: specs run only when
ANTHROPIC_API_KEY is set (e2e/llm.ts), the backend uses the env key, and
title generation resolves to claude-haiku-4-5 via resolveTitleModel.
Verified against a local prod-build stack (backend :3001, frontend :3002,
local Supabase + MinIO), only ANTHROPIC_API_KEY configured:
- keyless Playwright env: 23 passed / 4 skipped
- ANTHROPIC_API_KEY exported: 27 passed / 0 skipped (1.2m)
- npx tsc --noEmit clean
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| ||||
282b2dd2 | docs(e2e): exact ANTHROPIC_API_KEY setup to enable the 4 LLM specs; correct the gate rationale | QA Runner | 2026-07-20 | ↗ GitHub |
commit bodyInvestigated whether the four LLM-gated specs (chat rename, chat delete, project-assistant create+submit, critical path) could run keyless with the demo model they select. They cannot on this codebase: no keyless model exists here - ModelToggle's MODELS are all Anthropic/Google/OpenAI entries gated on a configured provider key (env or user-stored, backend/src/lib/userApiKeys.ts), ChatInput.handleSubmit hard-blocks a send for an unavailable model, and the backend model set (backend/src/lib/llm/models.ts ALL_MODELS) has no demo id either. The 'Demo (no key needed)' model the specs target lives only in the amal66 fork. So the blanket test.skip(!hasLlmKey, ...) guard stays as-is - the evidence does not support narrowing it. What this change does instead: - docs/e2e-ci.md: new 'Enable the LLM specs' section with the exact UI path and gh CLI command for the ANTHROPIC_API_KEY repository secret, the fork-PR secrets caveat (use a maintainer branch or workflow_dispatch to exercise the specs), expected per-run cost, and how to confirm in the Actions log that the specs ran (27 passed, no skipped line) rather than skipped (4 skipped / 23 passed). Also documents the known gap: until the specs' selectDemoModel helper is pointed at a Claude model (or the demo provider is upstreamed), the secret unskips the specs but they fail at model selection. - e2e/llm.ts: rewrite the doc comment with the precise gate rationale (no keyless model in this repo; title-generation 500s are best-effort and not the reason) and pointers to the setup docs. - .github/workflows/e2e.yml: comment-only updates aligning the skip narrative (23 keyless specs, not ~20/~23) and pointing at the new docs section. No behavioral change anywhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | ||||
0cc635bd | docs(e2e): mark the model-selection gap fixed - specs pick a Claude model when the key is set | QA Runner | 2026-07-20 | ↗ GitHub |
commit bodyThe "Known gap" paragraph predicted the 4 LLM specs would fail with the secret set because they selected the fork-only "Demo (no key needed)" model. That is now fixed on the e2e suite branch (#220): the helper selects "Claude Sonnet 4.6" and the critical-path assertion checks for a nonempty streamed assistant answer. Update the paragraph accordingly: setting ANTHROPIC_API_KEY now yields 27 passed / 0 skipped (verified locally; keyless stays 23 passed / 4 skipped). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | ||||
72242ff0 | docs(e2e): reconcile the docs with what this branch actually contains | QA Runner | 2026-07-22 | ↗ GitHub |
commit bodyThe two doc commits were written against a lineage where the
selectClaudeModel spec fix lived on a separate branch, which left three
stale claims after the rebase:
- e2e/llm.ts still called selectDemoModel a known gap and pointed at a
docs section title that no longer exists; the specs in this very
branch already use selectClaudeModel. Replace the paragraph with the
current behavior.
- docs/e2e-ci.md attributed the fix to "the e2e suite branch (#220)";
this branch carries that commit itself, so say so and name the commit.
- The 23/4 keyless and 27/0 with-key figures were presented as bare
facts ("verified locally"). Attribute them to where they were actually
measured - the spec-fix commit's own verification against a full local
stack - and state they have not been re-measured since the rebase onto
current main. Drop the unattributed "~7 minutes" typical-run figure in
favor of the structural 27-spec / 4-gated breakdown (confirmed by
`npx playwright test --list`: 27 tests in 7 files).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| ||||
5dbb45e9 | Merge remote-tracking branch 'origin/main' into olp-pr/e2e-llm-unskip | willchen96 | 2026-07-26 | ↗ GitHub |
# Conflicts: # .github/workflows/e2e.yml # .gitignore # docs/e2e-ci.md # e2e/llm.ts # package.json # playwright.config.ts | ||||
Capture this PR into my fork
Download a Markdown prompt that tells Claude how to port every
commit in this PR into your working tree. Run it via
claude -p < capture-pull-236.md from
inside the repo you want the changes in.