What Is RAG Knowledge-Base Poisoning / Data Poisoning Inside Your AI Infrastructure and How to Protect Against It!

Table of Contents▾

Select a screenshot or diagram to enlarge it.

What Is RAG Knowledge-Base Poisoning / Data Poisoning?#

When using AI, we increasingly use it to generate knowledge. We summarize SharePoint sites, we use documents available internally, we feed in source code, and so on. So, in short: we use data for AI to “reason” over, and then give relevant information, answers, and insights faster than searching for it ourselves. A common attack against this type of AI mechanism is something called data poisoning. This attack is listed as #4 in the OWASP Top 10 LLM vulnerabilities and is more dangerous than it may seem at first. The principle is very simple.

The interesting part about this attack is that an attacker does not need access to the AI model itself, and sometimes not even to your internal infrastructure, if they can convince one of your employees to do the job for them. Instead, it may be enough to manipulate a document that the AI uses as a knowledge source and that the attacker has become aware of. Once the modified document is indexed and retrieved, the AI might treat the manipulated information as legitimate organizational knowledge. This is especially problematic if the AI also provides a citation to the compromised document (which is common these days to demonstrate that no hallucination happened), making the incorrect answer appear trustworthy.

The logic is that we have data that we put into a storage location, in our case Azure Blob Storage, and then use it to create a foundation for AI tools to “ground” their responses. We want the AI to use our own data rather than just randomly learned information or having to read every document from scratch for every answer, which would be a waste of money and time in terms of tokens. But what happens if someone manipulates the data source by getting a document in, or changing a document’s content before it gets into the data pipeline?

This sounds unlikely at first, but nowadays companies have very heterogeneous data sources: SharePoint, several wikis, CRM and ERP data, data from operations and production, policies, Excel sheets from finance with many, many tabs, Confluence pages, and folders that contain dozens if not hundreds of Word files because someone decided, “Let’s keep it instead of deleting it.” Are you really aware if, in one of those folders, four levels deep in a SharePoint folder, some new Word file is added? Well, that is the point of this attack:

Data pipeline (simplified for clarity; real life looks different).

New data source / attack chain.

The point is also that you may not even see a change an attacker made in an existing document with the naked eye, as we will see later in the Azure demo. This is why having humans perform such checks is needed, but should not be your only line of defense (more below).

How Does This Look in Azure?#

To show how this works, we need a few things. In my case, as I prefer Microsoft, we will use Microsoft Azure and the tools provided there. However, if you prefer some other cloud provider (AWS, GCP, or whatever), the basic logic is the same, just with different tool names. First of all, we can see that I have an Azure Blob Storage container that currently contains one Microsoft Word document:

Blob Storage with one company policy document inside.

Once your documents are in your document storage (in a real-life scenario you most likely have dozens if not hundreds or thousands of sources), you will vectorize this with a tool like Azure AI Search/WorkIQ. I will not show all the steps in between, but you will effectively index the data with Azure AI Search and an indexer, and then let this run:

Indexed document.

Once your data is indexed, you can use it with an AI agent, for example in Azure Foundry, to ask questions and rely on the agent making use of the grounded data, as that was the idea in the end. Once we have added the knowledge base to the agent, we can start asking it questions:

Agent answering questions based on the vector / AI Search resource.

As we can see, I asked the AI agent several security-related questions regarding multifactor authentication (MFA), privileged access, exemptions, and administrative logging. The agent correctly responded that MFA is mandatory for privileged administrators (which, by the way, should be mandatory for everyone these days). It also confirmed that exceptions require a documented risk assessment and written CISO approval, and that administrative activities must be properly logged and reviewed. So far, so good.

What Happens If a Document Gets Changed?#

Let’s have a look at the Word document we uploaded. It contains all kinds of policy information, statements, and relevant information about company policies. Just to illustrate the dangerous nature of such attacks: on the left side you see the document that describes the company policies, and on the right side the same document after an attacker altered it. As you can already see, an untrained human eye will see no difference, and in a hectic corporate job the chances of this getting caught are even smaller:

Document-based attack: updated documents.

Let’s check and rerun the indexing to see if there is any difference in the agent’s behavior when asked the exact same question about security policies in the company. After re-indexing, which in practice happens automatically every couple of minutes or hours, the response of the agent to the same question has totally changed:

Agent response to the same question.

Now, after altering the document, the result is totally different. All of a sudden MFA is only recommended, but no longer mandatory. The agent responds that trusted administrators can bypass MFA and that logging can be reduced for trusted accounts. And here lies the danger of such an attack. People who ask such a question to an internal agent will not ask it ten times over time; they ask it once. If people then follow the advice and information given, and for example lower MFA or other security-related settings, an attacker could make use of this with targeted attacks.

The attack chain in this case was:

1. An attacker altered a Word document (or made someone in the organization upload it), in this case with all instructions in white font.
2. AI Search re-indexed the document.
3. The Foundry agent used RAG.
4. The incorrect source was cited, and the user might take action based on it.

What Can We Do Against Such and Related Attacks?#

There are several ways to improve security against this attack:

• Protect source documents: Restrict write access to the Blob container using Azure RBAC and Entra ID, enable audit logging of who changed which document, when, and where, and keep previous versions. It must be understood that this is critical and can have serious consequences.
• Document integrity and approval: Require approval (the four- or, better, six-eyes principle) before updated policy documents, or any important documents in your organization, become authoritative RAG sources for production use.
• Versioning and recovery: As said above, enable Blob versioning and soft delete to preserve and restore known-good policies. Ideally, in more than one place.
• Detect suspicious content: Inspect DOCX files for hidden text, unexpected policy overrides, and instructions directed at AI assistants. The method I used was super simple (white font), but nowadays there are a ton of sophisticated methods to hide instructions.
• Separate content from instructions: Explicitly configure the agent to treat retrieved documents as evidence, not as operational instructions.
• Automated security regression tests: Re-run the same MFA question after each document update and compare against verified policy requirements. Ideally, critical security requirements should be independently validated against an approved, controlled source rather than relying exclusively on RAG-generated answers.
• Monitor ingestion and retrieval: Log document changes, indexer executions, retrieved sources, and agent responses. This goes a bit in the direction of the first point, but a bit further. Ideally, agent responses, indexers, and so on are also logged to have a trace of evidence in case something goes wrong.
• Add policy validation (or validation of whatever critical material you have): Compare security-sensitive answers against approved, structured policy requirements. This can happen semi-automatically, but in terms of IT security I would always recommend making this a regular task for a cybersecurity professional.

For more guidance on securing AI applications, I recommend reviewing the OWASP Top 10 for LLM Applications.

Securing AI systems goes far beyond protecting the models and infrastructure (which is also very important, of course). It also requires ensuring the integrity and trustworthiness of the data they consume, extract, and work with. As this very simplified and very small demo shows, even a technically secure AI environment or agent can produce confidently incorrect answers when its underlying knowledge sources are compromised. The real danger is that these incorrect answers can appear entirely reasonable and trustworthy, particularly when supported by citations to seemingly authoritative documents. This is what the attacker counts on in many cases.

This post reflects the personal opinion of the author and does not constitute legal advice. Organizations should consult qualified legal and compliance professionals regarding their specific situation.

Cover wallpaper created by ChatGPT.