amal66 makes chat reliability a deliberate test discipline

This fork adds deeper checks for the security controls and chat streams that can fail quietly in ordinary testing.

securityinfrastructure

amal66 has added two on-demand test tools aimed at the places where a green routine test suite can still leave uncomfortable gaps. They are deliberately kept out of everyday merge checks, avoiding slower reviews and accidental load on live systems.

  • Security stress tests: introduce small, deliberate flaws into access controls, signed downloads, secret masking and legal citations, then identify which changes existing tests fail to catch.
  • Chat-stream resilience test: simulates concurrent live chat sessions and checks that each starts promptly and finishes, focused on past cases where long-running work left users waiting on a stalled response.

The second tool requires an authenticated staging environment and creates real chats, so it is explicitly not for production use.

So what Legal teams relying on Mike's shared matter access or chat workflow should care because this is practical insurance against subtle security gaps and silent conversational dead ends.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?

Commits in this thread

2 commits from amal66/mike, oldest first. Source extracted verbatim from the harvested git log.

SHA Subject Author Date
f80722a7 test: mutation testing (security libs) + SSE load harness, on-demand Amalanand Muthukumaran 2026-07-25 ↗ GitHub
commit body
Two on-demand depth tools, neither a merge gate:

- Stryker mutation testing scoped to the security-critical backend libs
  (access.ts, downloadTokens.ts, safeError.ts, chat/citations.ts).
  Measured 74.0-76.4% mutation score across runs; thresholds.break=69
  fails only on real regressions. `npm run test:mutation` locally (~3
  min), .github/workflows/mutation.yml on demand + monthly cron, HTML
  report uploaded as artifact.

- k6 load harness for the SSE chat stream (loadtest/sse-stream.js):
  ramps to N concurrent POST /chat streams, checks TTFB and that every
  stream delivers events through to the [DONE] sentinel - the past
  incident class (streams timing out on long tool calls). Lenient,
  documented thresholds. .github/workflows/loadtest.yml is
  workflow_dispatch-only and boots nothing: point it at a staging
  stack (PR #210).

docs/test-depth.md explains how to read the mutation report, how to run
the k6 harness against the local stack, and why neither gates merges.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
64ddcf4c docs: replace dead PR #210 staging-stack references with self-contained target guidance Amal 2026-08-05 ↗ GitHub
commit body
WHY THIS MATTERS
The load-test harness (workflow, doc, and k6 script) all told operators
to point it at "the staging stack from PR #210". PR #210 was CLOSED
unmerged - it was rejected for leaking admin credentials - so the one
piece of infrastructure the harness claims to require does not exist.
A newcomer following the docs would hit a dead end: the reference reads
as "there is a blessed staging stack somewhere", when in reality there
is no such stack and never will be from that PR. Worse, linking to a
credential-leaking PR as recommended reading is itself a small security
smell. Docs that point at dead artifacts erode trust in all the other
docs around them.

WHAT IS A SELF-CONTAINED REFERENCE
Documentation can either point at an artifact ("use the stack from PR
#210") or describe a contract ("use any stack that serves X behind Y").
Pointers are fragile: PRs get closed, branches get deleted, staging
environments get torn down, and the doc silently rots. A contract-style
reference survives all of that because it tells the operator what the
target must PROVIDE rather than where one specific instance lived. For
an on-demand load harness - which by design boots nothing itself - the
contract is the only stable thing to document.

HOW THE FIX WORKS
Every "#210" reference in the three harness files is replaced with the
actual contract the target stack must satisfy:

  - a deployed, NON-production backend the operator owns,
  - serving the backend API's streaming endpoint:
      POST {BASE_URL}/chat
      Authorization: Bearer <supabase access token>
  - with real LLM provider keys configured (each iteration performs a
    real chat completion).

Concretely:
  - .github/workflows/loadtest.yml: the header comment and the
    `target_url` input description now describe that contract, e.g.
      description: "Backend base URL of a non-production stack you
      deployed (serving the backend API with auth), ..."
  - docs/test-depth.md: the "Running from GitHub Actions" section
    describes the same contract instead of naming PR #210.
  - loadtest/sse-stream.js: the header comment does likewise.

No behavior changes - the workflow inputs, k6 scenario, thresholds, and
safety warnings ("never point at production") are untouched. Verified by
grepping the three files for "#210" (no matches), YAML-parsing the
workflow, and `node --check` on the k6 script.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Capture this thread into my fork

Download a single Markdown prompt that tells Claude how to port every commit above into your working tree — adapting paths and structure to match your repo. Run it via claude -p < capture-thread-1300.md from inside the repo you want the changes in.

⬇ Download capture-thread-1300.md