Ellen Svensson

AI-driven diagnostic and therapeutic systems are being rapidly integrated into clinical practice. A growing literature contends that this integration threatens patient autonomy by exposing patients to a novel form of AI paternalism. I argue that this framing is misguided on both theoretical and practical grounds. Theoretically, AI systems cannot be paternalists: their outputs lack the normative and agential standing that paternalism requires – a limitation I articulate as the is–ought gap for AI-generated output. Any paternalistic force in AI-integrated care must therefore reside in human agents, not in the systems themselves. Hence, we should not blame the code for any paternalism that patients may be subjected to in AI-integrated healthcare, but rather the human agents who purpose and develop these systems. In addition to these theoretical and conceptual concerns, there are practical worries. Casting AI integration as a case of paternalism risks framing the use of AI as prima facie objectionable and channels the debate toward a narrow, binary remedy that leans heavily on algorithmic transparency as a prerequisite for informed consent and shared decision-making. These discussions, I argue, are better served by revising our underlying conception of autonomy itself. I therefore conclude this article by arguing that replacing traditional, individualist conceptions of autonomy with a relational account better captures the distributed, socially embedded character of decision-making in AI-integrated care. This also redirects normative attention from the prevention of paternalism toward the conditions under which patient autonomy can be actively enabled and promoted.

The use of AI in digital mental healthcare promises to make treatments more effective, accessible, and scalable than ever before. At the same time, the use of AI opens a myriad of ethical concerns, including the lack of transparency, the risk of bias leading to increasing social inequalities, and the risk of responsibility gaps. This raises a crucial question: Can we rely on these systems to deliver care that is both ethical and effective? In attempts to regulate and ensure the safe usage of AI-powered tools, calls to trustworthy AI systems have become central. However, the use of terms such as “trust” and “trustworthiness” risks increasing anthropomorphization of AI systems, attaching human moral activities, such as trust, to artificial systems. In this article, we propose that terms such as “trustworthiness” be used with caution regarding AI and that when used, they should reflect an AI system's ability to consistently demonstrate measurable adherence to ethical principles, such as respect for human autonomy, nonmaleficence, fairness, and transparency. On this approach, trustworthy and ethical AI has the possibility of becoming a tangible goal rather than wishful thinking.

AI is entering clinical care and the healthcare sector in a big way, at the same time, a growing number of scholars are concerned that this technology cannot adhere to current bioethical principles. In particular, there are increasing concerns that AI poses a threat to the autonomy of patients by being irreconcilable with the practice of informed consent. In this essay, I shall defend the thesis that some applications of AI can be reconciled with a revised version of informed consent – what I call AI Adapted Informed Consent. This solution shall not rest on the idea of making black box AI more transparent or explicable. Instead, I shall argue that black box AI does not necessarily withhold the kind of information necessary for informed consent. Rather, patients can be given epistemic access to the kind of information necessary to make an informed decision, as well as being informed as to how the AI is used in the medical decision-making and in the assessment of their medical situation. Hence, this solution offers a re-interpretation of informed consent as information about contextual functioning and role of AI in medical decision-making. Drawing on republican interpretations of freedom as nondomination, I argue that demands for informed consent can only be restrained if it preserves the voluntariness of our decisions. Hence, I shall conclude that my adapted informed consent thesis allows for the possibility that some applications of black box AI in clinical care can be reconciled with informed consent and due respect for patient autonomy – if three specific conditions can be met.

The Covid-19 pandemic and the Swedish strategy when dealing with the crisis, have shown that a status quo bias can be highly damaging for bioethical decision-making. The pandemic presented a situation of high levels of uncertainties, where incomparable values such as health and freedom, were being put against each other. Because of this, mathematically oriented utility calculations or maximin principles such as the Precautionary Principle, recommended by the Swedish Corona-Commission, were not applicable. This I conclude since the Precautionary Principle cannot be applied when the outcomes of different alternatives are not comparable in value-terms. To answer the question of what signifies good bioethical decisionmaking during a crisis, risk and uncertainties I therefore present a five-step Crisis Checklist intended to give an overview of what a bioethical decision should entail.