[Docs 13] docs(e2e): exact ANTHROPIC_API_KEY setup to enable the 4 LLM-gated specs

✅ merged · #236 · open-legal-products/mike ← amal66/mike · opened 3mo ago by amal66 · merged 3mo ago by willchen96 · +117-20 across 3 files · ↗ on GitHub

From the PR description

Summary

Follow-up to #220: everything a maintainer needs to switch on the four LLM-gated e2e specs (chat rename, chat delete, project-assistant create+submit, and the critical-path "ask a question" flow) - exact ANTHROPIC_API_KEY secret setup (UI + CLI), the fork-PR secrets caveat, expected cost, and how to verify in the Actions log that the specs ran instead of skipped.

Stacked on #220's content. Because #220 is not merged yet, this branch carries its e2e commits (10 commits through 039e433) so the docs can be reviewed against the specs they describe. Review the top 3 commits only:

  • 282b2dd - docs(e2e): exact ANTHROPIC_API_KEY setup to enable the 4 LLM specs; correct the gate rationale
  • 0cc635b - docs(e2e): mark the model-selection gap fixed - specs pick a Claude model when the key is set
  • 72242ff - docs(e2e): reconcile the docs with what this branch actually contains

Once #220 merges, this rebases down to just those three.

Changes

  • docs/e2e-ci.md - "Enable the LLM specs" section: step-by-step UI path (Settings → Secrets and variables → Actions → New repository secret → ANTHROPIC_API_KEY), CLI (gh secret set ANTHROPIC_API_KEY --repo Open-Legal-Products/mike), the fork-PR caveat (secrets are withheld from fork pull_request runs; use a maintainer branch or workflow_dispatch), per-run cost (a handful of short completions - cents), and how to read the "Run Playwright" log. Plus a "Model selection" section describing the selectClaudeModel fix this branch carries.
  • e2e/llm.ts - doc comment rewritten with the precise gate rationale: this codebase ships no keyless model (every ModelToggle entry needs an Anthropic/Google/OpenAI key before ChatInput will send; the backend model set has no demo id either), so keyless the four specs cannot submit at all - and the auto title-generation 500 is not why the gate exists. Comment-only.
  • .github/workflows/e2e.yml - comment-only alignment with the docs.

No behavioral change in the doc commits; the four test.skip(!hasLlmKey, LLM_SKIP_REASON) lines are untouched.

Docs accuracy fix (top commit)

Earlier revisions of these docs told inconsistent stories about model selection (an open "known gap" in one file, "fixed on another branch" in another, plus pass/skip counts with no provenance). The top commit reconciles them so the docs describe exactly what this branch contains:

  • The specs in this branch already use selectClaudeModel (Claude Sonnet 4.6) when the key is set; the stale selectDemoModel "known gap" paragraph in e2e/llm.ts is gone.
  • The 23 passed / 4 skipped keyless and 27 passed / 0 skipped with-key figures are now attributed to where they were measured - the spec-fix commit's own verification against a full local stack (see the test(e2e): select a real Claude model... commit message) - and the docs state they have not been re-measured since this branch was rebased onto current main. The unattributed "~7 minutes typical run" figure is dropped in favor of the structural 27-spec / 4-gated breakdown.

Testing

  • npx tsc --noEmit at the repo root (covers e2e/): clean.
  • npx playwright test --list: 27 tests in 7 files - the specs parse and every referenced helper resolves (selectClaudeModel exists in e2e/chat-management.spec.ts and e2e/critical-path.spec.ts).
  • Keyless-skip logic verified by inspection: e2e/llm.ts gates on Boolean(process.env.ANTHROPIC_API_KEY) and exactly 4 specs call test.skip(!hasLlmKey, LLM_SKIP_REASON). A full e2e run needs the local Supabase stack and was not attempted for this docs PR.

🤖 Generated with Claude Code

Our analysis

Document LLM-gated e2e checks — read the full analysis →

Think the analysis missed something the PR description covers?

Commits in this PR (14)

SHA Subject Author Date
916f2600 test: Playwright e2e suite (auth, chat, projects, tabular reviews, workflows) Amal 2026-07-17 ↗ GitHub
commit body
Port of the amal66/mike fork's Playwright end-to-end suite onto the
upstream backend/ + frontend/ layout:

- e2e/: auth.setup (bootstraps a confirmed e2e@mike.local user via the
  Supabase admin API and saves storageState), auth-flows, critical-path
  (create project -> upload PDF -> ask a question -> streamed response),
  chat-management, project-management, tabular-reviews,
  workflows-account; fixtures/test.pdf
- playwright.config.ts: single-worker (shared test user), setup project
  + chromium project with saved auth state; webServer adapted from the
  fork's monorepo command (npm run dev --workspace apps/web) to
  upstream's layouts: backend `npm run dev` (health-checked on :3001)
  and frontend `npm run dev` (:3000)
- root package.json (upstream has none): @playwright/test, typescript
  dev-deps and test:e2e scripts, trimmed from the fork's root manifest
- root tsconfig.json scoped to e2e/ + playwright.config.ts so
  `npx tsc --noEmit` typechecks the suite
- .gitignore: playwright artifacts + e2e/.auth (session tokens)

Specs select by ARIA role/name and placeholder text only - no
data-testid attributes, so no app-code changes are required.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CEguyEgXa9JjCciXCcVemC
e5510287 fix(e2e): realign selectors with the olp UI and upstream route shape Amal 2026-07-17 ↗ GitHub
commit body
upstream-main (olp's tip) ships the #215/#216 liquid-surfaces UI, so the same
selector drifts fixed on the fork's suite (commit 3d46814 on main) also apply
here. Ports those realigned hunks, plus one upstream-specific route fix.

  * Chat input placeholder is "How can I help?", not
    "Ask a question about your documents..." (ChatInput/TRChatInput). Updated in
    chat-management (rename/delete/project-assistant) and critical-path.
  * The project-assistant empty state replaced the "+ Create New" text link with
    a PillButton reading "Create" (ProjectAssistantTable) - critical-path and
    chat-management now use getByRole("button", { name: "Create" }).
  * The sidebar chat row's active marker is APP_SURFACE_ACTIVE_CLASS
    ("bg-app-surface-active"), not "bg-gray-200/60"; the row wrapper is now h-8,
    not h-9 (SidebarChatItem). The rename/delete tests locate the row accordingly.
  * The documents-toolbar folder button is "Folder" (TabPillButton wired to the
    root createFolderAction in ProjectDocumentsView), not "Add Subfolder"; it
    still opens the autofocused "Folder name" root input.
  * NewTRModal's footer submit and the tabular page CTA both read "Create", so
    the create-review helper scopes to the modal submit
    (button[name="modalAction"][value="create-review"]).
  * The built-in workflow detail test navigated to the flat /workflows/[id],
    which upstream does not route - only the typed /workflows/assistant/[id] and
    /workflows/tabular-review/[id] exist. builtin-cp-checklist is assistant-type,
    so it now navigates to /workflows/assistant/builtin-cp-checklist (the path
    the app itself links to via workflowDetailPath; works on the fork too).

Verified: full 27-test suite green against upstream-main + test-harness +
demo-mode(+provider-registry), backend + local Supabase (upstream schema.sql +
backend/migrations) + MinIO, demo model, e2e@mike.local.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CEguyEgXa9JjCciXCcVemC
82fd540e ci(e2e): run Playwright suite on PRs as a merge gate QA Runner 2026-07-17 ↗ GitHub
commit body
Adds .github/workflows/e2e.yml adapted to this repo's backend/ + frontend/
layout: on every pull_request into main (or upstream-main), boot MinIO +
local Supabase (loading backend/schema.sql), start the API and web dev
servers, and run the Playwright suite, uploading the HTML report/traces on
pass or fail. playwright.config.ts already disables its local webServer when
CI=true, so the job owns the stack.

docs/e2e-ci.md documents the one required secret (ANTHROPIC_API_KEY) and the
branch-protection steps that make the 'e2e / playwright' check required, so a
red run blocks the merge button.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
82cbe4d7 ci(e2e): upload the Playwright report even on timeout/cancel QA Runner 2026-07-17 ↗ GitHub
commit body
Switch the artifact step from !cancelled() to always() so a run that hits the
30-minute job timeout (the failure mode when many specs retry) still uploads
the HTML report and traces instead of skipping the upload - that partial report
is exactly what's needed to diagnose the failing specs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
28a6cb64 ci(e2e): grant service_role table access after loading schema.sql QA Runner 2026-07-17 ↗ GitHub
commit body
The backend queries exclusively as service_role (lib/supabase.ts, service key,
'bypasses RLS'). schema.sql revokes client grants (anon/authenticated) but
assumes a hosted Supabase where service_role already has full table access - on
a fresh CLI stack loaded via psql it gets none, so the first backend write 500s
with 'permission denied for table user_profiles' and every project/chat spec
fails. Granting service_role after the schema load reproduces the production
grant posture.

Verified in CI: POST /projects 500 -> 201, GET /chat 500 -> 200.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
7039461c ci(e2e): apply migrations, serve prod build, skip LLM specs without a key QA Runner 2026-07-17 ↗ GitHub
commit body
Three fixes that take the suite from ~everything-failing to green (verified end
to end against a local stack):

1. schema.sql lags the migrations (missing e.g. workflow_open_source_submissions,
   which GET /workflows/:id queries → 500). Apply every dated migration on top of
   schema.sql (idempotent; 0 errors on a fresh DB), grant service_role AFTER so
   new tables are covered, then reload PostgREST. Fixes the workflow specs.

2. Serve a production build (next build + next start) instead of next dev. The dev
   server's on-demand compilation (slow first hit → waitForResponse timeouts) and
   hydration-error overlay (injects nextjs__container_errors DOM that pollutes text
   locators, e.g. getByText('All') matching 'Call Stack') made the suite flaky.
   Fixes tabular-reviews list render + the timing flakes.

3. LLM-dependent specs (chat rename/delete/submit, critical-path) require a model
   key to send a message. Guard them with test.skip(!hasLlmKey) (e2e/llm.ts) and
   expose ANTHROPIC_API_KEY to the Playwright process, so a keyless run is green on
   the ~23 other specs and the LLM specs run + enforce only when the secret is set.

Local result (no key, with MinIO+LibreOffice as on ubuntu-latest): 23 passed,
4 skipped, 0 failed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
e6d53c4f docs(e2e): document keyless-green behavior, migrations, and prod build QA Runner 2026-07-17 ↗ GitHub
commit body
The suite is green with no secret (LLM specs skip). Update docs/e2e-ci.md and the
workflow header to reflect the migrations step, the production build, and the
optional (not required) ANTHROPIC_API_KEY.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
9f9a280f test(e2e): update built-in workflow id after upstream rename QA Runner 2026-07-20 ↗ GitHub
commit body
The workflows UI refresh (olp/main fa21ac8) renamed the built-in workflow id
builtin-cp-checklist -> builtin-draft-cp-checklist (title 'Draft CP Checklist'
unchanged). The read-only-detail spec navigated to the old id URL and got a
'workflow not found' page. Point it at the new id.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
86a0d781 test(e2e): drop transient-5xx retry scaffolding now that the fetch storm is fixed QA Runner 2026-07-20 ↗ GitHub
commit body
With the directory fetch storm eliminated (previous commit: batched
listProjects?include=documents instead of a getProject burst per modal
open), the local Supabase gateway no longer collapses under modal-open
load, so the specs' 5xx-oriented scaffolding is removed:

- create-with-retry loops (project create x2, chat create x2, workflow
  create, tabular-review create x10-attempt, review add-document x6)
- reload-guards for transient "Project not found" (waitForProjectLoaded,
  gotoProjectRow, project-assistant tab loop)
- networkidle settle-waits whose purpose was to let the per-project fetch
  burst drain before submitting (NewProjectModal / NewTRModal)
- oversized per-test timeouts justified by the storm (180s -> 60-120s)

Kept, deliberately:
- the title-generation settle logic in chat rename/delete
- chat submit's model-seeding retry loop (guards a documented
  useSelectedModel localStorage hydration race, unrelated to the gateway)
- account display-name toPass block (guards a documented profile
  hydration race, unrelated to the gateway)
- generic short networkidle settles after login/logout navigation
- Playwright's own retries: 2 on CI

Verified: with the fix, 3 consecutive full-suite runs (plus 2 more before
de-retry) were green with zero Playwright retries, running against an
account seeded with ~40 projects - a larger directory fan-out than CI's
fresh database ever produces.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
039e433a test(e2e): select a real Claude model in LLM-gated specs (demo model is fork-only) QA Runner 2026-07-20 ↗ GitHub
commit body
The 4 LLM-gated specs (chat-management rename/delete/project-assistant and
the critical-path project flow) selected a "Demo (no key needed)" model in
the ModelToggle. That model exists only in a fork's demo provider - upstream
ModelToggle.MODELS has no such entry - so on this repo, setting the
ANTHROPIC_API_KEY secret unskipped the specs and they then failed at the
model-selection step.

Fix:
- Replace the selectDemoModel helpers with selectClaudeModel, which picks
  "Claude Sonnet 4.6" (the cheapest Anthropic entry in ModelToggle.MODELS).
  With ANTHROPIC_API_KEY exported to the backend, userApiKeys.envApiKey()
  reports the claude provider as configured, so the model is available and
  the submit is not blocked by the ApiKeyMissingModal.
- critical-path Step 8: the canned-reply assertion getByText("Demo mode")
  becomes a presence + nonempty assertion on the assistant answer container
  (MarkdownContent's "div.prose.font-serif.text-gray-900", unique to
  assistant answer content) - deterministic against nondeterministic real
  LLM output, same 60s budget.
- Comments rewritten to describe the upstream reality: specs run only when
  ANTHROPIC_API_KEY is set (e2e/llm.ts), the backend uses the env key, and
  title generation resolves to claude-haiku-4-5 via resolveTitleModel.

Verified against a local prod-build stack (backend :3001, frontend :3002,
local Supabase + MinIO), only ANTHROPIC_API_KEY configured:
- keyless Playwright env: 23 passed / 4 skipped
- ANTHROPIC_API_KEY exported: 27 passed / 0 skipped (1.2m)
- npx tsc --noEmit clean

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
282b2dd2 docs(e2e): exact ANTHROPIC_API_KEY setup to enable the 4 LLM specs; correct the gate rationale QA Runner 2026-07-20 ↗ GitHub
commit body
Investigated whether the four LLM-gated specs (chat rename, chat delete,
project-assistant create+submit, critical path) could run keyless with the
demo model they select. They cannot on this codebase: no keyless model
exists here - ModelToggle's MODELS are all Anthropic/Google/OpenAI entries
gated on a configured provider key (env or user-stored,
backend/src/lib/userApiKeys.ts), ChatInput.handleSubmit hard-blocks a send
for an unavailable model, and the backend model set
(backend/src/lib/llm/models.ts ALL_MODELS) has no demo id either. The
'Demo (no key needed)' model the specs target lives only in the amal66
fork. So the blanket test.skip(!hasLlmKey, ...) guard stays as-is - the
evidence does not support narrowing it.

What this change does instead:
- docs/e2e-ci.md: new 'Enable the LLM specs' section with the exact UI
  path and gh CLI command for the ANTHROPIC_API_KEY repository secret, the
  fork-PR secrets caveat (use a maintainer branch or workflow_dispatch to
  exercise the specs), expected per-run cost, and how to confirm in the
  Actions log that the specs ran (27 passed, no skipped line) rather than
  skipped (4 skipped / 23 passed). Also documents the known gap: until the
  specs' selectDemoModel helper is pointed at a Claude model (or the demo
  provider is upstreamed), the secret unskips the specs but they fail at
  model selection.
- e2e/llm.ts: rewrite the doc comment with the precise gate rationale
  (no keyless model in this repo; title-generation 500s are best-effort
  and not the reason) and pointers to the setup docs.
- .github/workflows/e2e.yml: comment-only updates aligning the skip
  narrative (23 keyless specs, not ~20/~23) and pointing at the new docs
  section. No behavioral change anywhere.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
0cc635bd docs(e2e): mark the model-selection gap fixed - specs pick a Claude model when the key is set QA Runner 2026-07-20 ↗ GitHub
commit body
The "Known gap" paragraph predicted the 4 LLM specs would fail with the
secret set because they selected the fork-only "Demo (no key needed)"
model. That is now fixed on the e2e suite branch (#220): the helper
selects "Claude Sonnet 4.6" and the critical-path assertion checks for a
nonempty streamed assistant answer. Update the paragraph accordingly:
setting ANTHROPIC_API_KEY now yields 27 passed / 0 skipped (verified
locally; keyless stays 23 passed / 4 skipped).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
72242ff0 docs(e2e): reconcile the docs with what this branch actually contains QA Runner 2026-07-22 ↗ GitHub
commit body
The two doc commits were written against a lineage where the
selectClaudeModel spec fix lived on a separate branch, which left three
stale claims after the rebase:

- e2e/llm.ts still called selectDemoModel a known gap and pointed at a
  docs section title that no longer exists; the specs in this very
  branch already use selectClaudeModel. Replace the paragraph with the
  current behavior.
- docs/e2e-ci.md attributed the fix to "the e2e suite branch (#220)";
  this branch carries that commit itself, so say so and name the commit.
- The 23/4 keyless and 27/0 with-key figures were presented as bare
  facts ("verified locally"). Attribute them to where they were actually
  measured - the spec-fix commit's own verification against a full local
  stack - and state they have not been re-measured since the rebase onto
  current main. Drop the unattributed "~7 minutes" typical-run figure in
  favor of the structural 27-spec / 4-gated breakdown (confirmed by
  `npx playwright test --list`: 27 tests in 7 files).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5dbb45e9 Merge remote-tracking branch 'origin/main' into olp-pr/e2e-llm-unskip willchen96 2026-07-26 ↗ GitHub
# Conflicts:
#	.github/workflows/e2e.yml
#	.gitignore
#	docs/e2e-ci.md
#	e2e/llm.ts
#	package.json
#	playwright.config.ts

Capture this PR into my fork

Download a Markdown prompt that tells Claude how to port every commit in this PR into your working tree. Run it via claude -p < capture-pull-236.md from inside the repo you want the changes in.

⬇ Download capture-pull-236.md