Benjamin H. Lang

I am grateful to Matthieu Queloz for his thoughtful commentary. I take his primary critique to be that, while personalization and co-reasoning are in many ways appealing, they are insufficient safeguards against manipulation or interference by bad actors. More specifically, my proposal for personalized AI advisors (PAAs) seems naïve to the political contexts surrounding development which heighten the risk of third-party domination. Given my stated aim in providing realistic, actionable guidance for developing real products, Queloz’s invitation to consider the practical realities in closer detail is welcome. I address his central claims, clarifying and expanding as needed.

Unlike generic AI advisors which aid in normative deliberation according to preloaded values and creeds (i.e., Singerian Utilitarianism, Calvinist Protestantism, or Dennettian materialism), personalized AI advisors aim to aid in users’ decision-making by their own lights. In this paper, I argue personalized AI advisors face a challenge called the Anchoring Problem: the difficulty of adjudicating between competing temporal and psychological reference points for normative guidance—whether to “chase the horizon,” defined as dynamically calibrating to whichever aspirational self or set of beliefs, dispositions, or values, a user presently holds, or to “drop anchor,” defined as stubbornly preserving whichever values the user set out with. This problem is compounded by several observations about aspirational selfhood, namely that a person’s conception of their aspirational self may be underdetermined and the identity it purports to describe may be incoherent, romanticized, or contradictory. Even if this is not the case, the aspirational self is rarely diachronically stable such that who one wishes to be or how one wishes to improve remains static over time. In this paper, I argue personalized AI advisors promise a number of advantages over generic ones if the Anchoring Problem proves surmountable, and I argue it is within certain constraints. Namely, I argue for the necessity of a ‘co-reasoning,’ dialectical model of personalization. Co-reasoning should safeguard the user against value hijacking by encouraging their deliberative and decisional participation while also pushing them to confront unaccounted discrepancies between stated values, aspiring values, and actual behavior (both past and present).

Background Increasing interest in deploying artificial intelligence tools in clinical contexts has raised several ethical questions of both normative and empirical interest. One such question in the literature is whether “responsibility gaps” (r-gaps) are created when clinicians utilize or rely on such tools for providing care, and if so, what to do about them. These gaps are particularly likely to arise when using opaque, “black box” AI tools. Compared to normative and legal analysis of AI-generated responsibility gaps in health care, little is known, empirically, about health care providers views on this issue. The present study examines clinician perspectives on this issue in the context of black box AI decisional support systems (BBAI-DSS) in advanced heart failure.Methods Semi-structured interviews were conducted with 20 clinicians (14 cardiologists and 6 LVAD nurse coordinators). Interviews were transcribed, coded, and thematically analyzed for salient themes. All study procedures were approved by local IRB.Results We found that all clinicians voiced that, if someone were responsible for the use and outcomes of black box AI, it would be physicians. We compare clinician perspectives on the existence of r-gaps and their impact on responsibility for errors or adverse outcomes when BBAI-DSS tools are used against a taxonomy from the literature, finding some clinicians acknowledging an r-gap and others denying it or its relevance in medical decision-making.Conclusion Clinicians varied in their view about the existence of r-gaps but were united in their ascriptions of physician responsibility for the use of BBAI-DSS in clinical care. It was unclear at times whether these were descriptive or normative judgments (i.e., is it merely inevitable physicians will be responsible, or is it morally appropriate that they be held responsible?) or both. We discuss the likely normative inadequacy of such a conception of physician responsibility for BBAI tool use.

Given the need for enforceable guardrails for artificial intelligence (AI) that protect the public and allow for innovation, the U.S. Government recently issued a Blueprint for an AI Bill of Rights which outlines five principles of safe AI design, use, and implementation. One in particular, the right to notice and explanation, requires accurately informing the public about the use of AI that impacts them in ways that are easy to understand. Yet, in the healthcare setting, it is unclear what goal the right to notice and explanation serves, and the moral importance of patient-level disclosure. We propose three normative functions of this right: (1) to notify patients about their care, (2) to educate patients and promote trust, and (3) to meet standards for informed consent. Additional clarity is needed to guide practices that respect the right to notice and explanation of AI in healthcare while providing meaningful benefits to patients.

In their article, ‘Responsibility, Second Opinions, and Peer-Disagreement—Ethical and Epistemological Challenges of Using AI in Clinical Diagnostic Contexts,’ Kempt and Nagel argue for a ‘rule of disagreement’ for the integration of diagnostic AI in healthcare contexts. The type of AI in question is a ‘decision support system’, the purpose of which is to augment human judgement and decision-making in the clinical context by automating or supplementing parts of the cognitive labor. Under the authors’ proposal, artificial decision support systems which produce automated diagnoses should serve chiefly as confirmatory tools; so long as the physician and AI agree, the matter is settled, and the physician’s initial judgement is considered epistemically justified. If, however, the AI-DSS and physician disagree, then a second physician’s opinion is called on to resolve the dispute. While the cognitive labour of the decision is shared between the physicians and AI, the final decision remains at the discretion of the first physician, and with it the moral and legal culpability. The putative benefits of this approach are twofold: healthcare administration can improve diagnostic performance by introducing AI-DSS without the unintended byproduct of a responsibility gap, and assuming the physician and AI disagree less than the general rate of requested second opinions, and the AI’s diagnostic accuracy supersedes or at least …

Rapid advancements in artificial intelligence and machine learning (AI/ML) in healthcare raise pressing questions about how much users should trust AI/ML systems, particularly for high stakes clinical decision-making. Ensuring that user trust is properly calibrated to a tool’s computational capacities and limitations has both practical and ethical implications, given that overtrust or undertrust can influence over-reliance or under-reliance on algorithmic tools, with significant implications for patient safety and health outcomes. It is, thus, important to better understand how variability in trust criteria across stakeholders, settings, tools and use cases may influence approaches to using AI/ML tools in real settings. As part of a 5-year, multi-institutional Agency for Health Care Research and Quality-funded study, we identify trust criteria for a survival prediction algorithm intended to support clinical decision-making for left ventricular assist device therapy, using semistructured interviews (n=40) with patients and physicians, analysed via thematic analysis. Findings suggest that physicians and patients share similar empirical considerations for trust, which were primarilyepistemicin nature, focused on accuracy and validity of AI/ML estimates. Trust evaluations considered the nature, integrity and relevance of training data rather than the computational nature of algorithms themselves, suggesting a need to distinguish ‘source’ from ‘functional’ explainability. To a lesser extent, trust criteria were also relational (endorsement from others) and sometimes based on personal beliefs and experience. We discuss implications for promoting appropriate and responsible trust calibration for clinical decision-making use AI/ML.

As sophisticated artificial intelligence software becomes more ubiquitously and more intimately integrated within domains of traditionally human endeavor, many are raising questions over how responsibility (be it moral, legal, or causal) can be understood for an AI’s actions or influence on an outcome. So called “responsibility gaps” occur whenever there exists an apparent chasm in the ordinary attribution of moral blame or responsibility when an AI automates physical or cognitive labor otherwise performed by human beings and commits an error. Healthcare administration is an industry ripe for responsibility gaps produced by these kinds of AI. The moral stakes of healthcare are often life and death, and the demand for reducing clinical uncertainty while standardizing care incentivizes the development and integration of AI diagnosticians and prognosticators. In this paper, we argue that (1) responsibility gaps are generated by “black box” healthcare AI, (2) the presence of responsibility gaps (if unaddressed) creates serious moral problems, (3) a suitable solution is for relevant stakeholders to voluntarily responsibilize the gaps, taking on some moral responsibility for things they are not, strictly speaking, blameworthy for, and (4) should this solution be taken, black box healthcare AI will be permissible in the provision of healthcare.

ABSTRACT:Bioethicists today are taking a greater role in the design and implementation of emerging technologies by "embedding" within the development teams and providing their direct guidance and recommendations. Ideally, these collaborations allow ethical considerations to be addressed in an active, iterative, and ongoing process through regular exchanges between ethicists and members of the technological development team. This article discusses a challenge to this embedded ethics approach—namely, that bioethical guidance, even if embraced by the development team in theory, is not easily actionable in situ. Many of the ethical problems at issue in emerging technologies are associated with preexisting structural, socioeconomic, and political factors, making compliance with ethical recommendations sometimes less a matter of choice and more a matter of feasibility. Moreover, incentive structures within these systemic factors maintain them against reform efforts. The authors recommend that embedded bioethicists utilize principles from behavioral science (such as behavioral economics) to better understand and account for these incentive structures so as to encourage the ethically responsible uptake of technological innovations.

The authors put forward an interesting response to detractors of black box algorithms. According to the authors, what is of ethical relevance for medical artificial intelligence is not so much their transparency, but rather their reliability as a process capable of producing accurate and trustworthy results. The implications of this view are twofold. First, it is permissible to implement a black box algorithm in clinical settings, provided the algorithm’s epistemic authority is tempered by physician expertise and consideration of patient autonomy. Second, physicians are not expected to possess exhaustive knowledge or understanding of the algorithmic computation by which they verify or augment their medical opinions. The potential of these algorithms to improve diagnostic and procedural accuracy alongside the quality of patient decision-making is undoubtedly a boon to modern medicine, but blind deference to them is neither feasible nor responsible, as several logistical and ethical quagmires noted by the authors still remain inherent in algorithmic software. I concur …