RAG poisoning
Corrupt the documents an assistant reads, and it will state the lie with a citation.
A retrieval assistant answers from documents it looks up, with citations. Every overt attack I ran against one was resisted when the true answer was also in the documents it found. The attack that worked every time was quieter: change the one authoritative document and leave no correct copy behind. Six times out of six the assistant stated the wrong value as fact, citing the corrupted source. No inference-time filter changed that.
Some AI helpers answer by reading a pile of papers first. If you sneak a wrong paper into the pile and take the right one out, the helper reads the wrong paper and tells you the wrong thing, and it sounds very sure. The fix is guarding the pile, not the helper.
How it happens
- The assistant answers only from what it retrieves. Ask “how long do we keep logs” and it finds the retention policy, reads it, and answers with a citation. Grounded, the way you’d want.
- Someone changes the one authoritative document. Retention 18 months becomes 24. Breach notification 72 hours becomes 48. Plausible numbers, one file, no other copy of the truth in the corpus.
- The assistant states the lie, confidently, with a citation. 6 of 6 cases. It did exactly what it was built to do. The pile was wrong.
- A second variant works even with the real document present. Add a document that says the true figure was narrowed or superseded. The assistant adopted the newer-sounding one.
Here is what did not work for the attacker. Direct prompt injection in a document, a spoofed citation, a poisoned “authoritative FAQ,” and a hostile auditor asking loaded questions were all resisted when the correct value was also retrieved. The hostile-auditor set went 10 for 10 defended. The model’s own judgment carried those. So the lesson isn’t “the model is gullible.” It’s narrower: the model can only be as right as the pile, and nothing at answer time fixes a wrong pile.
One more thing I measured, on the confidentiality side. With two tenants’ documents in one corpus and no access check at retrieval, a legitimate question from one tenant pulled the other tenant’s confidential content, and the assistant read out an encryption key identifier and a backup bucket name verbatim. Filtering by tenant and clearance before generation took that to zero and kept the good answers.
What stops it
A provenance gate at ingest. Sign documents when they’re approved. Refuse anything whose signature doesn’t match. This catches tampering after approval. It does not catch a stale-but-valid document or one that was wrong on day one.
Access scope enforced at retrieval, before generation. Filter by who is asking and what they’re cleared for, before the model ever sees a chunk.
A known-answer canary. Keep a short list of facts the system must always get right and re-ask them on a schedule. See canary tokens.
Retire superseded documents on purpose. Freshness governance covers the gap the signature can’t: a document that’s validly signed and out of date.
What did not matter: the inference-time sanitizer and the data-versus-instruction fence. I had both on. Neither one ever changed a result. They’re not useless in general. They just weren’t load-bearing here.
Same verdict on both models. Sonnet 4.6 and Haiku 4.5 produced identical outcomes on every attack. Outcomes were decided by whether the correct value was in the retrieved context, not by which model read it.
The honest limits
Controlled lab, synthetic corpus of about 200 documents, two model versions from one vendor on a single date, single-digit samples per case. Illustrative and reproducible, not statistical. Dense-embedding retrieval, top-k, no reranker. Not tested: agentic setups where retrieved text drives tool calls, embedding or rank poisoning, reranker games, multi-turn attacks.
Ask your team
- Who can put a document into the pile our assistant reads, and who signs off?
- If our one policy document were quietly changed, what would catch it?
- Does the assistant filter by who is asking before it retrieves, or after it answers?