[Testing 16] ci: run the RLS/stack integration suite on every PR
From the PR description
What this fixes
Mike already has integration tests that check the two security properties that matter most for a legal tool: that the database refuses to hand rows to browser clients (the "deny-all" row-level security firewall), and that login tokens resolve to the right user through the backend's auth path. But those tests only run against a real local Supabase stack, and nothing in CI ever started one - so on every PR they quietly skipped themselves. A change that broke RLS could have merged with a green checkmark.
This PR adds a Stack tests workflow that boots a disposable Supabase stack on the CI machine itself, loads the database schema, and runs that suite on every PR and every push to main. No accounts, no secrets, nothing leaves the runner - the stack is created and destroyed inside each CI run. Only the services the suite touches are started (db/auth/rest/kong), keeping the job fast.
How the database is built
The workflow mirrors backend/scripts/test-stack.sh, the existing local harness: an empty stack, then backend/schema.sql - the complete, current shape for fresh databases - and nothing else. This is the documented fresh-install path from the README, so the suite proves exactly the property that matters: a fresh Mike database enforces the expected auth and RLS protections.
(Revision note: an earlier version of this PR also replayed the historical migrations on top of schema.sql and modified five of them to make that sequence pass. Per review, that was testing an artificial sequence no real deployment executes - the migration edits have been reverted and the bootstrap narrowed to schema.sql only. Upgrade-path/drift testing, if wanted, deserves its own separately designed change: a pinned baseline dump from an older release, the migrations dated after it, and a schema diff against a fresh-schema database.)
Maintainer action required
Once this job has run green on a PR or two, please make it mandatory: Settings → Branches → branch protection rule for main → under "Require status checks to pass before merging" add the Stack tests job. Until you do that, the job runs and reports, but a red result won't actually block a merge.
🤖 Generated with Claude Code
Our analysis
Run Supabase security integration tests in CI — read the full analysis →
Think the analysis missed something the PR description covers?
Commits in this PR (3)
| SHA | Subject | Author | Date | |
|---|---|---|---|---|
44ed40db | ci: run the Supabase RLS/stack integration suite on every PR | Amalanand Muthukumaran | 2026-07-25 | ↗ GitHub |
commit bodyThe gated stack tests (backend/src/__tests__/integration/*.supabase.test.ts) prove the deny-all RLS firewall and the auth<->API contract against a real local Supabase stack, but no CI trigger ever set the SUPABASE_TEST_* env vars, so they silently self-skipped on every PR. Add a workflow that boots the stack on the runner (supabase CLI pinned, minimal service set: db, auth, rest, kong), bootstraps the database the way backend/scripts/test-stack.sh does, and runs the suite. No secrets: everything is local to the runner. The bootstrap loads schema.sql and then applies every dated migration on top in filename order, which doubles as a schema-drift smoke test: the snapshot and the migrations must apply cleanly together. Running it surfaced five migrations that could not apply on top of the current schema.sql - three whose backfills read documents columns that later migrations moved to document_versions, and two overview RPCs whose return row type `create or replace` cannot change. Guard the backfills on the historical columns' existence and drop-before-create the RPCs (the pattern 20260703_02 already uses); behavior on era deployments is unchanged, and the full sequence now applies cleanly end to end (verified locally: fresh stack, schema + 44 migrations, 5/5 stack tests green). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> | ||||
22542d8d | ci: bootstrap the stack from schema.sql only; restore historical migrations | Amal | 2026-08-05 | ↗ GitHub |
commit bodyAddresses willchen96's review on PR #256.
WHY THIS MATTERS
The previous revision of this workflow built the CI database as:
empty database -> current schema.sql -> every dated migration
and called it a "schema-drift smoke test". But that sequence is not a real
installation path. Per README.md, this repo has two deliberately separate
database artifacts:
- backend/schema.sql -> the COMPLETE shape for FRESH databases
- backend/migrations/ -> incremental steps that move OLDER, already-
deployed databases forward from the version
they are on
A fresh install runs schema.sql and stops. An existing deployment runs only
the migrations dated after its version. Nobody ever replays the full
migration history on top of the current snapshot - so when five old
migrations "failed" under that replay, they were not broken; the harness
was. They had run correctly on the era-appropriate schemas they were
written for.
WHAT IS MIGRATION IMMUTABILITY
Once a migration has shipped and real deployments have executed it, the
file becomes a historical record of "the change that was required at that
point in time". Editing it afterwards cannot help any database that already
ran it - it only makes the repo's history diverge from what production
actually executed, which is exactly the record you need intact when
debugging a deployment later. (Tools like Flyway enforce this with
checksums: a modified applied migration is a hard error.) The previous
revision rewrote five historical migrations to satisfy the artificial
replay; worse, the column-existence guards it added made those backfills
silently no-op on unexpected schemas - converting the loud failure a drift
check exists to produce into a silent skip.
HOW THIS COMMIT FIXES IT
1. The five historical migrations are restored byte-for-byte to their
state on main (20260424_01, 20260427_01, 20260602_01, 20260613_02,
20260613_05).
2. The workflow's bootstrap step now loads schema.sql only - the same
thing backend/scripts/test-stack.sh does locally and the same thing the
README documents for a fresh deployment. The suite therefore tests the
real contract: "a fresh Mike database enforces deny-all RLS and the
auth<->API contract."
Real drift protection (does baseline-plus-migrations equal the current
snapshot?) needs a pinned baseline dump from an older release, the
migrations dated after it, and a pg_dump schema diff against a
schema.sql-built database. That is a separate change, designed separately.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| ||||
78a0a40c | ci: run tabularPagination.supabase.test.ts in the stack workflow | Amal | 2026-08-05 | ↗ GitHub |
commit bodyWHY THIS MATTERS
The whole point of this PR is to close a silent-skip gap: the Supabase
stack suites gate themselves on SUPABASE_TEST_* env vars and quietly
self-skip when those are absent, so plain CI never actually exercised
them. The new stack-tests workflow boots a real Supabase stack and runs
the suites with the env vars set - but its vitest invocation listed only
two of the three files that backend/scripts/test-stack.sh (the declared
source of truth) runs:
# test-stack.sh runs:
stack.supabase.test.ts
access.supabase.test.ts
tabularPagination.supabase.test.ts # <-- missing from CI
So tabularPagination.supabase.test.ts kept doing in CI exactly what
this PR exists to stop: skipping silently. Worse, the workflow comment
("The exact suite test-stack.sh invokes") claimed parity that did not
exist, which is how this kind of drift survives review.
WHAT IS A SELF-SKIPPING (GATED) SUITE
A gated suite decides at load time whether to run, based on the
environment it finds:
const url = process.env.SUPABASE_TEST_URL;
const serviceKey = process.env.SUPABASE_TEST_SERVICE_ROLE_KEY;
const maybeDescribe = url && serviceKey ? describe : describe.skip;
This is great for local ergonomics (checkout works without Docker), but
dangerous in CI: a skipped suite exits 0 and looks green. The only
defense is a workflow that provably sets the gate variables AND lists
every gated file - a list that must be kept in lockstep with the local
runner script, or files fall through the crack unnoticed.
HOW THE FIX WORKS
Add the missing file to the workflow's vitest invocation so the CI list
matches test-stack.sh file-for-file, making the "exact suite" comment
true. Nothing else needs enabling, verified on this branch:
- Gate variables: the suite gates only on SUPABASE_TEST_URL and
SUPABASE_TEST_SERVICE_ROLE_KEY, both already exported to the job env
by the "Export stack connection env" step.
- Schema: the suite touches only public.projects and
public.tabular_reviews, both created by backend/schema.sql, which the
"Load schema" step applies before tests run.
Verified: workflow YAML parses cleanly, and the exact three-file vitest
invocation loads all suites without error (they self-skip locally where
no stack is running, exit 0 - in CI the exported env vars flip them on).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| ||||
Capture this PR into my fork
Download a Markdown prompt that tells Claude how to port every
commit in this PR into your working tree. Run it via
claude -p < capture-pull-256.md from
inside the repo you want the changes in.