amal66 tries to put Mike's safety checks on a hard fail

The proposal would have made it much harder for a seemingly harmless change to quietly weaken safeguards around legal AI responses.

securitycompliance

amal66 built an offline test harness using recorded assistant answers and legal documents, rather than live AI calls. That matters because every contributor change could be checked without API keys, network access, variable results, or added cost.

  • Citation accuracy: catches invented authorities, altered quotations, and mismatches between citations and source documents.
  • Prompt injection: checks that hostile instructions hidden in retrieved material cannot steer the assistant or its tools.
  • Privilege and sensitive-data leakage: looks for information crossing matter boundaries or escaping in responses.

It was designed to block changes that weakened these checks, while avoiding false alarms on harmless edge cases. The pull request closed without being merged, so none of this protection is active in Mike.

So what Legal teams assessing AI controls should care because this was a credible route to testing guardrails before changes reach users.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?