Skip to content
d.uhlmann (home)Nederlands
Back to Blueprints
Compliance10 min read

What is "high-risk" AI under the EU AI Act & Why you should be careful!

What qualifies as high-risk AI under Article 6 and Annexes I and III of the EU AI Act, which requirements follow from it, and why skipping the classification can be costly.

Table of Contents

I observed a recent industry trend towards AI implementation that might have some hidden regulatory risks that, if uncovered too late, can cause reputational and financial damage. So, the goal here is to quickly give an overview about what “high-risk AI” is and what potential impacts this might have. To avoid non-compliant AI implementation I would advise to always do an EU AI Act & IT Security assessment (a part of the so-called “compliance layer” or “due diligence layer”) to avoid the implementation of systems that have underlying compliance risks. To clarify this I want to highlight with this Article:

  1. What does classify as “high-risk” AI

  2. What kind of requirements need to be fulfilled when dealing with such systems

  3. Why shortcuts may lead to negative consequences

Which Systems do classify as “high-risk” AI systems under the AI Act & why is this important?

When working with AI in the EU organizations need to pay attention to the EU AI Act - this is not the only case but the most common one. A regulation aimed at compliant, safe and fair use of AI technology in accordance with the Charter of Fundamental Rights of the European Union (https://www.europarl.europa.eu/charter/pdf/text_en.pdf). The AI Act is a risk based regulation aimed at reducing the risk of human rights violations by AI systems to people in the EU. Sounds a bit technocratic, so long story short: You have to follow this regulation if you put AI systems into use in the EU. A special focus lays on “high-risk” AI systems as those systems can potentially harm the human rights of EU citizens and are therefore regulated very strictly.

What “high-risk” AI is is defined in Art. 6 (“Classification rules for high-risk AI systems”) of the AI Act. When starting to read this legislation we can already get a lot of know-how out of the plain English legal text. For example Art. 6(1) states the following:

Irrespective of whether an AI system is placed on the market or put into service independently of the products referred to in points (a) and (b), that AI system shall be considered to be high-risk where both of the following conditions are fulfilled:

where (a) states the following:

the AI system is intended to be used as a safety component of a product, or the AI system is itself a product, covered by the Union harmonisation legislation listed in Annex I;

and (b) states the following:

the product whose safety component pursuant to point (a) is the AI system, or the AI system itself as a product, is required to undergo a third-party conformity assessment, with a view to the placing on the market or the putting into service of that product pursuant to the Union harmonisation legislation listed in Annex I.

Besides this, we also have an important hint in Art. 6(2):

In addition to the high-risk AI systems referred to in paragraph 1, AI systems referred to in Annex III shall be considered to be high-risk.

Now, this itself does not tell us too much of what high-risk AI actually is, so we need to check those Annex I and Annex III of the AI Act to see what is classified as “high-risk” AI. To avoid that this becomes a 20 page legal deep dive I will include a list here of systems that might be classified as “high-risk” if AI is involved (for precise concrete judgement please read the law in detail):

These regulations target special areas that are already covered by specialized EU laws. Despite those Annex III defines other areas where AI applications might be considered as “high-risk”:

  • Biometrics
  • Education and vocational training
  • Employment, workers’ management and access to self-employment
  • Access to and enjoyment of essential private services and essential public services and benefits
  • Law enforcement, in so far as their use is permitted under relevant Union or national law (but there are exceptions for this one)
  • Migration, asylum and border control management, in so far as their use is permitted under relevant Union or national law
  • Administration of justice and democratic processes

Decision Graph to determine high-risk AI systems

Decision Graph to determine high-risk AI systems

There are certain exceptions that Art. 6(3) makes for certain type of systems like the ones below:

  • performing a very narrow procedural task: Art. 6(3)(a)
  • Improves the result of a previously completed task by a human Art. 6(3)(b)
  • Detects decision-making patterns without replacing human judgement Art. 6(3)(c)
  • Performs only a preparatory task for an assessment Art. 6(3)(d)

Those kind of systems might not be considered high risk. One important note however. Any AI system that performs some sort of profiling like automated processing to evaluate personal aspects of a natural person is always classified as high risk, regardless of the exceptions we just saw. Also all those criteria need to be crosschecked with the concrete legal text. Also, in case of any doubts a full legal estimation is often a good idea, as court cases are rare, and the AI Act is new to most organizations.

Let’s close this chapter with a practical example that might be classified as high-risk AI. An IT provider wants to implement an LLM based AI system for end-user chats for an organization that helps people with mental issues like sexual-orientation, potential depression and other related areas. The chat will be also open to teenagers as nowadays it’s not really possible anymore to block minors from entering websites. Such a system would be (very likely) classified as high-risk under Art. 6(2) / Annex III or Art. 6(1) of the AI Act if this bot is classified as a medical device. Exceptions under Art. 6(3) might be hard to justify as this system might be always profiling natural persons, regardless of the concrete technical implementation. There might be even an argument for Art. 5 which might ban the implementation all together.. Long story short: Do a check first.

What kind of requirements need to be fulfilled when dealing with such systems

Now, as we know what “high-risk” AI actually means we have to understand what kind of requirements need to be met to put those kinds of systems into service. To be fair, if you have a high-risk AI system, you have to follow a ton of requirements. There is nothing to sugar-coat here. It will be a lot of work, so first recommendation should be to be very careful and avoid such AI systems in the first place, as they come with a long list of requirements. The concrete requirements you will need to follow as a absolute base are mostly described in Chapter III - section 2 (“Requirements for high-risk AI systems”) and section 3 (“Obligations of providers and deployers of high-risk AI systems and other parties”) of the AI Act. There are more nuances then that but those sections serve as a solid starting point.

  • Risk Management (Art. 9): A continuous risk management has to be established and documented. This has to be done before the launch of the product. To be clear here, we are not talking about “pure documentation” but about an actual running system. This system has to mitigate all known and foreseeable risks throughout the entire lifecycle of the application.
  • Data Governance (Art. 10): For training, validation and testing datasets need to be chosen that are relevant, representative and free of errors (as much as possible). An important link also refers to GDPR. If you use data that contains GDPR relevant information you need to anonymize it or you need explicit approval of each person (data subject). Otherwise you are in GDPR & AI Act violation.
  • Technical Documentation (Art. 11): This article which links to Annex IV of the AI Act demands detailed documentation before launch of the product to demonstrate compliance to the respective authorities in your country. A word of caution here. “Technical Documentation” sounds relatively straightforward. But please check out Annex IV as this contains a long list of very concrete requirements
  • Record-keeping (Art. 12): A high-risk AI system must keep logs of events to be able to trace and monitor the system and (more importantly) identify potential risks (see Art. 9 for that)
  • Transparency (Art. 13): This requirement demands that users can interpret outputs. I will not open the whole Explainable AI box of pandora here, but this can be tricky. With such requirements a best-effort approach is sometimes a good idea. Go as far as you can and document everything.
  • Human Oversight (Art. 14): Build-in measures (like a “stop button” and/or “approval layers”) that allow natural persons to effectively oversee and intervene in the systems operation. This requirement is very important. For high risk AI a Human-in-the-Loop approach is key to avoid mistakes and also resolve accountability issues. In case of disaster the AI will not be held accountable but rather the person who put it into the market and/or deployed it.
  • Accuracy, Robustness & Cybersecurity (Art. 15): The AI system must “be designed and developed in such a way that they achieve an appropriate level of accuracy, robustness and cybersecurity (…)” (Art. 15(1). Especially the Cybersecurity part can become tricky as most modern AI systems are LLM based and those system still show vulnerabilities left and right. Also the amount of people who can professionally pen-test those systems is globally very scarce.
  • Quality Management System (Art. 17): If you are a provider of such a system you must implement and document a robust QMS for policies and procedures to ensure regulatory compliance at all times.

Now, next to those points I want to mention two more requirements from section 5 (“Standards, conformity assessment, certificates, registration”)

  • Conformity Assessment (Art. 43): Before putting the system into the market it must undergo a formal procedure to verify that all requirements are met (conformity check)
  • Registration (Art. 49): If you plan to deploy such a system you need to do a registration in the provided database by the European Union. This is the formal registration process.

As we can see high-risk AI demands a lot of requirements, and we have only scratched the surface here.

Why shortcuts may lead to catastrophic consequences (for the implementer, deployer and user)

The trend I mentioned in the beginning - avoiding or forgetting the compliance check - can have very bad impacts for the organization. Especially IT implementation companies must ensure that their customers have systems implemented in a legally compliant way. Even when doing a “proof of concept” or any other type of prototype I would always recommend to put the “compliance glasses” on to in a best case scenario avoid “embarrassing” questions by the customers compliance team and worst case putting a high-risk AI system into the market without knowing that its actually high-risk. I will not discuss fines here as I see this as a spreading of fear, but organizations that violate the AI Act might face penalties, but even more might loose reputation of their stakeholders and customers.

Closing Word & Limitations

This short piece should give a first impression of high-risk AI under the AI Act. Of course, it is highly condensed, and does by far not cover all relevant requirements & action points. For this, a comprehensive legal analysis is needed with a concrete use-case. Ultimately, treating AI Act compliance as a ‘design requirement’ rather than a hurdle will allow your organization to build a foundation of trust with users / customers. IT implementation companies in particular should be committed to help customers to transform these legal obligations into a strategic advantage for sustainable AI growth.

Disclaimer

This post is for informational purposes only & only constitutes the opinion of the author - not the companies the author might be employed and/or connected with in any manner! Also this does not constitute legal advice. For specific guidance related to the topics discussed or other legal matters, please consult a qualified professional / lawyer and/or audit firms tech consultancy advisory for qualified and legally binding answers.