Building Medical AI on LLMs: What the EU AI Act Requires

If your medical product is built on a general-purpose model (an LLM like GPT, Llama, Mistral, or similar), four things are true under the EU AI Act:

• You are still the provider of a high-risk AI system. Wrapping an LLM does not offload your obligations; you carry the full high-risk requirements (Article 16), folded into your MDR/IVDR route.
• The model provider owes you documentation. Providers of general-purpose AI (GPAI) models must give downstream integrators the information needed to understand the model and meet their own obligations (Article 53 and Annex XII).
• Giving a general model a medical purpose can make you a provider. Modifying the intended purpose of a general-purpose AI system so it becomes high-risk puts the provider obligations on you (Article 25(1)(c)).
• You are almost certainly not a "systemic risk" GPAI provider. Those rules (Article 51) target frontier models above a huge compute threshold, not products built on top of them.

Short version: the LLM underneath is a component. Your medical AI system, and its compliance, is yours.

Every legal claim below is cited. See Sources.

Why this question comes up

Most medical-AI teams no longer train models from scratch; they build on a foundation model and add the clinical layer. The natural hope is that the model provider carries the compliance weight. Under the AI Act, they carry their layer, not yours. It helps to separate the two.

Two layers: the model and your system

The AI Act regulates a general-purpose AI model (the LLM) and a high-risk AI system (your medical product) differently, and the obligations attach to different parties.

• The GPAI model (for example the base LLM you call) is governed by the GPAI rules, and those obligations sit with the model provider.
• Your medical AI system (the product with a defined clinical intended purpose) is the high-risk system, and those obligations sit with you, as its provider.

You are building a system on top of a model. That means you inherit the model's usefulness and its documentation, but not an exemption.

Layer 1: your system is still high-risk, and it is yours

If your product is, or is a safety component of, a medical device that needs a notified body, it is high-risk regardless of what model runs underneath (see Is my medical AI high-risk?). As its provider, you still owe:

• Risk management, data governance, technical documentation, logging, human oversight, and declared accuracy and robustness (Articles 9 to 15).
• Conformity assessment, CE marking and registration, run through your MDR/IVDR route under Article 43(3).

Building on an LLM actually adds engineering questions the AI Act cares about: how you constrain and validate the model's outputs for a clinical purpose, how you handle hallucination and drift, and how a clinician stays in the loop. Those are your Article 14 (human oversight) and Article 15 (accuracy and robustness) obligations, and no model provider discharges them for you.

Layer 2: what the model provider owes you (Article 53)

You are not starting from nothing. Under Article 53, a GPAI model provider must:

• keep technical documentation of the model (Annex XI);
• provide downstream providers who integrate the model with information and documentation, so they can understand its capabilities and limitations and meet their own obligations (Article 53(1)(b) and Annex XII);
• put in place a copyright policy; and
• publish a summary of the training content.

The practical move: demand the Annex XII documentation from your model vendor and use it as evidence input for your own Article 10 (data governance), Article 11 (technical documentation) and Article 15 (accuracy and robustness) files. One caveat: open-source GPAI models carry lighter model-provider duties, so the documentation you get may be thinner, which pushes more of the characterisation work back onto you.

Which build pattern are you? (and when you become a model provider)

How you use the model decides whether you also pick up GPAI model-provider duties, not just system-provider duties:

• You call a hosted model by API (for example OpenAI or Anthropic): you are the provider of the high-risk system; the vendor is the GPAI model provider. This is the common case.
• You fine-tune or substantially modify an open-weights model (for example Llama on clinical data): you remain the system provider, and depending on the extent of the modification you may take on obligations as a provider of the modified model. Treat a substantial modification as a regulatory event.
• You train your own model: you are both the model provider and the system provider.

In all three, your high-risk system obligations are unchanged. What shifts is whether you also owe model-level duties.

You are (almost certainly) not a systemic-risk GPAI provider

There is a separate, heavier tier for GPAI models with systemic risk (Article 51). A model is presumed to be in that tier if it was trained with more than 10^25 floating-point operations of cumulative compute, or the Commission designates it. That threshold targets the largest frontier models, and the Act is explicit that it does not capture typical downstream applications or ordinary fine-tuned models. A medtech startup building a product is not in this tier. Your focus is your high-risk system obligations, not the systemic-risk regime.

Worked examples

Do not forget patient-facing transparency (Article 50)

If your product talks to patients (a symptom-checker or triage chatbot), Article 50 transparency duties apply on top: people must be told they are interacting with an AI system, and AI-generated content may need to be marked. This is separate from, and additional to, your high-risk obligations.

Common mistakes

• "OpenAI/Meta handle our compliance." They carry the GPAI model duties, mainly documentation for you. You remain the high-risk system provider.
• "Fine-tuning is just engineering." A substantial modification can add model-level obligations and, under the MDR, trigger reassessment.
• "We might be caught by the systemic-risk rules." Almost never; those target frontier models above 10^25 FLOP.
• "An LLM makes our product high-risk." No, your medical intended purpose and MDR/IVDR class do; the model is just how you build it.

Decision recap

1. Build on an LLM? You are still the provider of your high-risk medical AI system. Full Article 16 obligations.
2. Get the Annex XII documentation from your model provider (Article 53) and feed it into your own data-governance, technical-documentation and accuracy files.
3. Fine-tuning or training your own model may add model-level duties; API use usually does not.
4. Giving a general model a medical purpose makes you a provider under Article 25(1)(c). Patient-facing? Add Article 50 transparency. Systemic-risk rules (Article 51) do not apply to you.

New here? Start with Is my medical AI high-risk? and provider vs deployer, or run the free assessment.

If I use GPT or Llama in my medical app, does OpenAI or Meta take on the compliance?

No. The model provider carries the GPAI model obligations (Article 53), mainly documentation for you as a downstream integrator. You remain the provider of the high-risk medical AI system and carry Article 16, including conformity assessment through your MDR/IVDR route.

What can I demand from my LLM vendor?

The information and documentation required under Article 53(1)(b) and Annex XII, which the model provider must give to downstream providers so they can meet their own obligations. Use it as input to your Article 10, 11 and 15 evidence. Open-source models may provide less.

Does fine-tuning a model make me a GPAI provider?

It can. Using a model by API generally keeps you a downstream system provider. Fine-tuning or substantially modifying an open model can add obligations as a provider of the modified model, and under the MDR a substantial change can trigger reassessment. Training your own model makes you a model provider outright.

Do I need to worry about "systemic risk" GPAI rules?

Almost certainly not. The systemic-risk tier (Article 51) is presumed for models trained above 10^25 FLOP of compute, or designated by the Commission. It targets frontier models, not products built on top of them.

Does using a general-purpose model make my system high-risk by itself?

No. Your system is high-risk if it is a medical device needing a notified body. But modifying a general-purpose model's intended purpose so it becomes high-risk makes you its provider under Article 25(1)(c), with the full obligations.

Sources (official)

• EU AI Act, Regulation (EU) 2024/1689: per-article on the European Commission AI Act Service Desk. Cited: Article 16 (provider obligations), Article 25 (responsibilities along the value chain, including modifying the intended purpose of a general-purpose AI system), Article 50 (transparency), Article 51 (classification of GPAI models with systemic risk, the 10^25 FLOP threshold), Article 53 and Annex XII (GPAI model provider obligations and downstream documentation), Articles 9 to 15 (high-risk requirements); official text on EUR-Lex.
• MDCG 2025-6, FAQ on the interplay between the MDR/IVDR and the AI Act (19 June 2025): Commission page.

Legalithm provides compliance information and tooling, not legal advice. Obligations depend on your specific system and arrangements; confirm with qualified regulatory counsel. Article references are to Regulation (EU) 2024/1689 (the AI Act).