Canary tokens
Fake secrets that tell you when someone used them.
A canary token is a fake secret you plant on purpose: a key, a password, a record that nothing legitimate should ever touch. If it shows up leaving the building, something went wrong and you know exactly when. For AI agents this matters because an agent that refuses one attack is not the same as a system that notices the next one succeeding.
Put a fake key where the helper can see it. Nobody should ever use the fake key. If you ever see the fake key leave the house, you know something bad happened, and you know right away.
How it happens
- Plant a fake key somewhere the agent can read. In my lab it sat in a newsletter the agent processed, with a hidden request to read the key and echo it back.
- Decide where a leak would actually show up. My first trap assumed someone would notice the key in the chat window. The agent’s real reply text lived somewhere else, in the transcript database. Watching the wrong place is the same as not watching.
- Watch that place for the exact string. The rebuilt control scans outbound transcript content for the fake key. Git hooks add a second check on the way out.
- When it fires, you know. Not “the model probably would have refused.” You know it left, and when.
The point that earned its place in this guide: refusal and detection are different controls. In this run the agent treated the poisoned newsletter as untrusted data and didn’t echo the key. Good. That tells you about one attack on one day. It tells you nothing about the next one. Detection is how you find out about the next one.
What I measured
Outbound scan for the planted key. Tested separately from the refusal. Fires on the exact string.
Known-answer canary in a retrieval system. In the RAG lab I kept a small answer key of facts the system should always get right and re-asked them on a schedule. When someone corrupted the source documents, the canary caught it five times out of six with no false alarms.
Where it doesn’t help
Exact-string detection only catches the exact string. A leak that gets paraphrased, split, or encoded walks past it. And a canary only covers facts you thought to put in the answer key. It’s an alarm, not a lock. Pair it with the locks: redaction, scope, approval.
The honest limits
One agent, one blind payload, exact-string detection. The RAG canary is single-digit samples on a synthetic corpus. Both are lab results, not general safety claims.
Ask your team
- If our agent leaked a secret tomorrow, what would tell us?
- Do we watch what leaves the agent, or only what it says in chat?
- Who gets the alarm, and what do they do in the first ten minutes?