Building an Azure AI Foundry Agent based on your own RAG in Azure AI Search to get better answers and minimize hallucinations

Table of Contents▾

When working with AI Agents in the compliance and legal sectors, precision is key. Also, in highly regulated sectors like the pharmaceutical industry, legal, and financial institutions, precision is very important, and hallucinations often cannot be accepted (sometimes even for legal or compliance reasons).

However, LLMs are non-deterministic “by design.” So, here we have a problem that we cannot overcome “by design,” as LLMs literally guess the next token (or word, to keep it simple) that should follow. This happens as LLMs are based on statistical probability.

For this article, we will build a small AI Agent for the pharmaceutical industry, generate the knowledge sources, generate the RAG, and then check how accurately the Agent answers. What we need for this is the following:

• Azure Subscription (turn cost management on to avoid bad surprises)
• Azure Foundry resource & project
• Training/raw data from the chosen industry & a way to make it more accessible for AI.
• Azure AI Search / knowledge base
• A deployed base model from Azure Foundry
• An Agent to test this out with a LLM behind

---

Data Preparation#

Every AI project needs clean data. This is also the case for our small prototype. For now I used the following regulations:

• AI Act
• Cyber Resilience Act (CRA)
• Data Act
• Good Clinical Practice (GCP)
• General Data Protection Regulation (GDPR)
• Good Manufacturing Practice (GMP)
• In Vitro Diagnostic Regulation (IVDR)
• Medical Device Regulation (MDR)
• Network Information Security Directive 2 (NIS2)
• Product Liability Directive (PLD) - the updated version that treats bugs as defective products and makes the vendor legally liable

As I described in a recent post, we will transform those regulations into AI optimized JSON files first. This can be done locally with the Python script I provided. It’s not strictly needed, as Azure can also read PDF files, but I would recommend giving clean, structured text as input rather than PDF files.

Training JSON file for GDPR

After this is done, we upload all the data to a Storage Account in Azure to have it in the cloud. This is needed as we want to train the AI in the cloud and not locally. Local RAG is also possible, but might be the subject of another article.

Azure Storage Account for Data

---

Base Agent Creation in Azure Foundry#

Next we need a base agent in Azure. For this I created a foundry resource in my resource group and a project.

Foundry Agent Resource in Azure

After that is done we go to ai.azure.com and create a base agent with a random LLM behind. It doesn’t need to be the best and biggest model for now; just use one you feel comfortable with:

Base Agent Foundry

This agent is now still pretty broad, has no specific pharmaceutical / legal knowledge and will answer specific pharma- & legal questions based on the normal LLM that powers this agent. That is of course not what we want, so we need to give this agent a bit more information. Also, we will add a System instruction like the following:

Instructions to Agent

This is a first safety layer you might take for your agents. It is clearly not enough for a highly regulated industry, but it is meant to illustrate the direction you should take when working with Agents in such industries.

---

Adding the knowledge base / AI Search#

Next, we need to add an “AI Search” resource to our resource group. For this you have to just add another resource with type “AI Search”. When you do that for a prototype, pay attention to the pricing tiers of Azure:

Pricing Tier

Azure has a default SKU that will cost you around 300 USD per month. So if you do not need this there is even a free version. After this is done you can add your files and create the knowledge base for your agent.

AI Search Importer for Data in your storage account

After that one is done, we need to import our JSON files from the storage account into the AI Search. For this I use the special JSON Lines import mechanism, but this depends on the fact which type of files you are effectively using. I have optimized this before to improve quality, but this is up to you.

Parsing Mode of Files Azure AI Search

After this is done we will need to decide how to vectorize the data. There are multiple ways in Azure how to do that. Those depend on your technical needs, maturity of your project and other specifics. The one I chose here just relies on the fact that I want to quickly prototype something.

Vector of Data

Once this is run we are, from a data source perspective, done here. In my specific case, I will assign Managed Identities for access rights, but this is specific to this use case. You can also use API keys for example. Once the vector index is created and trained, we can click into it and see the actual vectors.

Index of pharmaceutical laws and general EU regulations

This index is the actual base that we connect to the Agent later to ensure better replies connected to the respective legal regulations.

---

Connecting the Agent to the new Vector#

So, let’s recap at this point. We have a semi-modified AI Agent in Foundry that has nothing more than instructions on what to do at the moment. Secondly, we have a knowledge vector in Azure AI Search which should serve as a foundation for our AI Agent. What we need to do now is to bring the two components together.

To do this we go back to our Agent in Foundry and connect the Azure AI Search as knowledge to our Agent.

Connecting AI Search as knowledge to the agent

Now, after this is done, we have added a connection to our RAG, which contains EU laws and pharmaceutical-specific knowledge. So after connecting the RAG we can ask our Agent EU specific legal questions with a focus on a specific highly-regulated industry… in our case the pharmaceutical industry.

High Risk AI Check of Agent

In our example I asked the Agent a very simple question: “What is high risk AI in the EU”. The response I got mentions a couple of sources. This is where past general-purpose AI Agents fell short, as they either hallucinated answers or misread the law. Not in all cases, but often enough to cause me some headache. So let’s check the answers. Recital 131 of the AI Act states:

In order to facilitate the work of the Commission and the Member States in the AI field as well as to increase the transparency towards the public, providers of high-risk AI systems other than those related to products falling within the scope of relevant existing Union harmonisation legislation, (…)

So, this is correct in my opinion. Let’s check a second statement about Article 49 of the AI Act, where the Agent claims regulation for placing high-risk AI products on the market:

Registration 1. Before placing on the market or putting into service a high-risk AI system listed in Annex III, with the exception of high-risk AI systems referred to in point 2 of Annex III, the provider or, where applicable, the authorised representative shall register themselves and their system in the EU database referred to in Article 71. (…)

So, again, the Agent answered correctly and provided precise information. After that, let’s ask a pharma-specific question to check if the Agent answers correctly in the specific field it has knowledge of: Prompt was…

“How is a medical device classified if it incorporates a medicinal product as an ancillary substance?”

to which the Agent replied:

“A medical device that incorporates a medicinal product as an integral part, where the substance is considered a medicinal product if used separately and has an action ancillary to that of the device, is classified as a Class III medical device in the European Union. This is based on Rule 14 in Annex VIII of the Medical Device Regulation, which states that devices incorporating such a substance (including medicinal products derived from human blood or plasma) and where the substance’s action is ancillary to the device, are Class III devices. Additionally, the quality, safety, and usefulness of the medicinal substance incorporated in the device must be verified following the relevant medicinal product directives by consulting the competent medicinal products authority. Before issuing technical documentation assessment, a notified body must seek a scientific opinion from the designated competent authority on the quality and safety of the substance incorporated into the device.

In summary:

• If the medicinal substance is ancillary to the device’s action, the device is classified as Class III.
• The device undergoes a conformity assessment considering both medical device and medicinal substance regulations.
• The medicinal product authority is involved in evaluating the safety and usefulness of the incorporated substance.

Let me know if you need more details on the specific regulatory articles or procedures.”

Without being a pharmaceutical expert by any stretch of the imagination, when I open the regulation and read Rule 14 of Annex VIII of the MDR, it says this:

All devices incorporating, as an integral part, a substance which, if used separately, can be considered to be a medicinal product, as defined in point 2 of Article 1 of Directive 2001/83/EC, including a medicinal product derived from human blood or human plasma, as defined in point 10 of Article 1 of that Directive, and that has an action ancillary to that of the devices, are classified as class III.

it at least sounds kinda correct. This prototype does not claim that all information provided is correct, but rather demonstrates how to improve the quality of your AI responses in general. Besides this, overlaps between regulations can also be explained a bit more clearly, offering better points for further investigation:

Random Question with pharmaceutical and AI related information

---

Summary#

This article was meant as a short inspiration for people working in specific industries and wanting to improve the AI experience. Of course this is a demo, and in practice you would not use a “mini” model but rather the best available one like Gemini3 or GPT5.2.

Sources for this article:

• Reasoning Retrieval Engine: youtube.com/watch?v=PeTmOidqHM8
• RAG process: youtube.com/watch?v=xXTuxKdzZrI