Shawnaldinho spots a filename that could hijack the assistant, then lets the fix lapse

A document named the right way could smuggle instructions past the AI and make it ignore its own rules.

securitychat-ui

Shawnaldinho flagged a real weakness in the chat backend: names the user controls - filenames, folder names, workflow titles - were being dropped straight into the instructions the AI reads. Because the model treats those lists as structure, a carefully crafted filename could break out of the list and pose as a fresh system-level command. The worked example was a file whose name told the assistant to ignore its prior instructions and dump its own hidden setup prompt.

The proposed fix cleaned every spot where user text meets the prompt - stripping control characters, neutralising the brackets an attacker would need, and capping length - and explicitly told the model to treat anything inside the document list, filenames included, as untrusted data rather than orders. Legitimate names would render exactly as before. But it was closed the next morning without merging, so the guard never actually landed here.

So what Anyone running a legal AI where clients or staff upload documents should read this as a live risk, not a solved one.

View this fork on GitHub →

Spotted something wrong? Or know the PR text has fresher detail than the writeup above?