Ethics and Regulation · Related to Chapter 15

Who Is Responsible When Artificial Intelligence Makes a Mistake in Medicine?

By André Leite and Vinícius Lain, authors of AI in Healthcare.

The question comes up sooner or later in any serious discussion about artificial intelligence in medicine: if the AI gets it wrong and the patient is harmed, who answers for it?

The more mature answer is not a single name. It is a design of responsibility. The physician, the institution and the vendor each hold a different position in the chain of decisions, and how a technology was chosen, validated, deployed, supervised and monitored becomes as important as the technology itself.

This text discusses governance principles and does not replace legal analysis of a specific case, which depends on the law, the jurisdiction, the type of technology and the circumstances of the event.

An algorithm has no CRM

An AI can suggest, prioritize, classify, summarize or alert. But on its own it does not carry the same professional and ethical relationship that exists between a doctor and a patient. (In Brazil, the CRM is the physician's license number, the registration that ties a doctor to professional accountability.) When a clinical decision is taken on and signed by a professional, the presence of an algorithm in the workflow does not automatically make the machine responsible for the act.

That does not mean all the responsibility falls on the physician. It means automation does not create a vacuum.

The physician's responsibility

In general terms, the physician remains responsible for the clinical judgment they exercise, especially when the tool works as support and not as an autonomous system. That includes interpreting the recommendation in the context of the patient, recognizing situations where the tool should not be followed and reviewing outputs that do not fit the clinical reality.

One of the most important risks is automation bias: the tendency to trust a recommendation too much because it came from a system seen as objective or sophisticated. Good governance has to build a culture in which disagreeing with the AI is possible, expected and documented when necessary.

The institution's responsibility

Hospitals, clinics and healthcare organizations make choices too. They choose the tool, the vendor, the context of use and the level of autonomy. They define training, integration, supervision, response protocols and monitoring.

So it is not enough to install a solution and hand all the risk to the end user. A mature institution has to be able to show why it adopted that technology, what evidence it reviewed, whether it ran local validation when relevant, how it trained the team and how it tracks performance and incidents.

The legal and ethical question stops being only "who clicked?" and starts to include "how was the system designed?"

The vendor's responsibility

The developer or vendor answers for its own dimensions of the product: promised performance, defects, technical failures, controls, documentation, updates and the information given to the customer.

An institution should not accept a tool whose operating logic, limitations, versioning and behavior after updates are completely opaque. The higher the clinical risk, the greater the need for a contract, traceability and technical accountability to match.

Governance is what exists before the problem

When an adverse event happens, it is too late to invent governance. The organization needs to have built a body of evidence and processes beforehand.

Among the most important elements:

a clear definition of what the tool is for;
validation and performance criteria;
a defined population and context of use;
training for the professionals;
human supervision proportional to the risk;
a record of the system's version and changes;
logs and traceability of recommendations;
continuous performance monitoring;
a protocol for incidents and failures;
a real possibility of review and disagreement;
transparent communication with patients when relevant.

This design does not exist only to protect the institution legally. It exists, above all, to protect the patient.

Transparency without alarmism

Another hard question is whether the patient needs to know that an AI tool is taking part in their care. There is no single answer for every use and context, but there is an important principle: trust requires honesty, communicated well.

The communication does not have to turn the visit into an engineering lecture. It can explain, in plain language, that support tools are used to increase safety, organize information or reduce delays, and that the clinical decision remains under human supervision.

The biggest risk is not the machine getting it wrong

Every technology can fail. The more serious problem is using a system without a method: without knowing its limits, without defining who responds to the alerts, without recording versions, without monitoring performance and without knowing what to do when it fails.

When that happens, the organization is not just taking on technological risk. It is turning innovation into improvisation.

Distributed responsibility is not diluted responsibility

Saying that the physician, the institution and the vendor have different roles does not mean "nobody is responsible." It means exactly the opposite: the chain has to make clear who answers for each component.

The AI can suggest the path. The institution must create a safe environment to use it. The vendor has to deliver a product that matches what it promises. And the professional has to exercise judgment within their own scope of practice.

There is no responsibility vacuum in medicine. Technology expands capability, and with it the need for method.

André Leite and Vinícius Lain are the authors of AI in Healthcare: How Technology Is Transforming the Future of Human Care.

Keep reading

Ethics and Regulation