Inbar Cohen

As artificial intelligence and machine learning (AI/ML) systems become increasingly pervasive in society, their opacity—i.e., the difficulty, and sometimes impossibility, of understanding why they make the decisions they make—has become a serious problem. This is especially true in sensitive decision-making contexts, such as criminal justice, health care, and finance, or in choices requiring allocation of scarce resources. One attempt to “open up” the AI/ML black box has been the emergence of post hoc explainability algorithms—algorithms which generate post hoc approximations to black box models. However, such algorithms have been criticized as merely providing after the fact rationalizations for the decisions these systems make. In this paper, we defend and articulate a different concept—AI/ML justifiability. We explore several ways in which an algorithm could be justifiable, and we argue that pursuing justifiability is a worthwhile goal. A key to our argument is a distinction from the philosophy of action between motivating and normative reasons: effective explanations require (but are unable to provide) motivating reasons, while effective justifications require (and can indeed provide) normative reasons alone. We conclude that as long as a model is justifiable, it can be trusted even if it cannot be explained.

Direct-to-consumer medical artificial intelligence/machine learning applications are increasingly used for a variety of diagnostic assessments, and the emphasis on telemedicine and home healthcare during the COVID-19 pandemic may further stimulate their adoption. In this Perspective, we argue that the artificial intelligence/machine learning regulatory landscape should operate differently when a system is designed for clinicians/doctors as opposed to when it is designed for personal use. Direct-to-consumer applications raise unique concerns due to the nature of consumer users, who tend to be limited in their statistical and medical literacy and risk averse about their health outcomes. This creates an environment where false alarms can proliferate and burden public healthcare systems and medical insurers. While similar situations exist elsewhere in medicine, the ease and frequency with which artificial intelligence/machine learning apps can be used, and their increasing prevalence in the consumer market, calls for careful reflection on how to effectively regulate them. We suggest regulators should strive to better understand how consumers interact with direct-to-consumer medical artificial intelligence/machine learning apps, particularly diagnostic ones, and this requires more than a focus on the system’s technical specifications. We further argue that the best regulatory review would also consider such technologies’ social costs under widespread use.

Artificial intelligence and machine learning (AI/ML) algorithms are increasingly developed in health care for diagnosis and treatment of a variety of medical conditions (1). However, despite the technical prowess of such systems, their adoption has been challenging, and whether and how much they will actually improve health care remains to be seen. A central reason for this is that the effectiveness of AI/ML-based medical devices depends largely on the behavioral characteristics of its users, who, for example, are often vulnerable to well-documented biases or algorithmic aversion (2). Many stakeholders increasingly identify the so-called black-box nature of predictive algorithms as the core source of users' skepticism, lack of trust, and slow uptake (3, 4). As a result, lawmakers have been moving in the direction of requiring the availability of explanations for black-box algorithmic decisions (5). Indeed, a near-consensus is emerging in favor of explainable AI/ML among academics, governments, and civil society groups. Many are drawn to this approach to harness the accuracy benefits of noninterpretable AI/ML such as deep learning or neural nets while also supporting transparency, trust, and adoption. We argue that this consensus, at least as applied to health care, both overstates the benefits and undercounts the drawbacks of requiring black-box algorithms to be explainable.

While the literature on putting a “human in the loop” in artificial intelligence (AI) and machine learning (ML) has grown significantly, limited attention has been paid to how human expertise ought to be combined with AI/ML judgments. This design question arises because of the ubiquity and quantity of algorithmic decisions being made today in the face of widespread public reluctance to forgo human expert judgment. To resolve this conflict, we propose that human expert judges be included via appeals processes for review of algorithmic decisions. Thus, the human intervenes only in a limited number of cases and only after an initial AI/ML judgment has been made. Based on an analogy with appellate processes in judiciary decision-making, we argue that this is, in many respects, a more efficient way to divide the labor between a human and a machine. Human reviewers can add more nuanced clinical, moral, or legal reasoning, and they can consider case-specific information that is not easily quantified and, as such, not available to the AI/ML at an initial stage. In doing so, the human can serve as a crucial error correction check on the AI/ML, while retaining much of the efficiency of AI/ML’s use in the decision-making process. In this paper we develop these widely applicable arguments while focusing primarily on examples from the use of AI/ML in medicine, including organ allocation, fertility care, and hospital readmission.

Explainability in artificial intelligence and machine learning (“AI/ML”) is emerging as a leading area of academic research and a topic of significant regulatory concern. Indeed, a near-consensus exists in favor of explainable AI/ML among academics, governments, and civil society groups. In this project, we challenge this prevailing trend. We argue that for explainability to be a moral requirement– and even more so for it to be a legal requirement– it should satisfy certain desiderata which it currently does not, and possibly cannot. In particular, we will argue that the currently prevailing approaches to explainable AI/ML are (1) incapable of guiding our action and planning, (2) incapable of making transparent the actual reasons underlying an automated decision, and (3) incapable of underwriting normative (moral/legal) judgments, such as blame and resentment. This stems from the post hoc nature of the explanations offered by prevailing explainability algorithms. We will explain that these algorithms are “insincere-by-design,” so to speak. And this renders them of very little value to legislators or policymakers who are interested in (the laudable goal of) transparency in automated decision making. There is, however, an alternative– interpretable AI/ML which we will distinguish from explainable AI/ML. Interpretable AI/ML can be useful where it is appropriate, but represents real tradeoffs and in some instances (in medicine and elsewhere) adopting an interpretable AI/ML may mean adopting a less accurate AI/ML. We argue that it is better to face those trade-offs head on, rather than embrace the fool’s gold of explainable AI/ML.

This project represents the first systematic assessment of the US Food and Drug Administration’s postmarket surveillance of legally marketed artificial intelligence and machine learning based medical devices. We focus on the Manufacturer and User Facility Device Experience database—the FDA’s central tool for tracking the safety of marketed AI/ML devices. In particular, we evaluate the data pertaining to adverse events associated with approximately 950 medical devices incorporating AI/ML functions for devices approved between 2010 through 2023, and we find that the existing system is insufficient for properly assessing the safety and effectiveness of AI/ML devices. Inparticular,we make three contributions: (1) characterize the adverse event reports for such devices, (2) examine the ways in which the existing FDA adverse reporting system for medical devices falls short, and (3) suggest changes FDA might consider in its approach to adverse event reporting for devices incorporating AI/ML functions.

Recently in Israel, a woman was mistakenly implanted with an embryo that is genetically related to another couple. Unfortunately, this case is not an isolated occurrence, as other cases of embryo mix-ups have been reported in several countries, including the USA, China, the UK and various other countries within the European Union. Cases of mixed-up embryos are ethically and legally complex: the woman who carried the pregnancy and the woman who is genetically related to the resulting child—both of whom endured emotionally and physically demanding infertility treatments—along with their partners, may be unwilling to relinquish parental rights over the child. This article explores four possible approaches, found in numerous common law jurisdictions, which can be used to address cases involving embryo mix-ups. Our analysis reveals several avenues through which legal parentage can be established. It can be done through gestation and the marital presumption, genetic connections, by adhering to the principle of the best interests of the child, or by recognising multiple individuals as legal parents. We review the advantages and disadvantages of each approach, but we have one clear recommendation: resolving embryo mix-up cases should be done proactively through the establishment of legislation and guidelines, rather than relying on post hoc individual court decisions. Such legislation and guidelines should guarantee the consistency of values throughout diverse reproductive contexts and mandate that fertility clinics and medical professionals provide individuals with comprehensive information regarding the potential risks associated with assisted reproductive treatments.

On August 5, 2010, a cave-in left thirty-three Chilean miners trapped underground. The Chilean government embarked on a massive rescue effort, with assistance from multiple international teams, experts, and donors that cost an estimated USD $10-20 million. There is a puzzle here. Many mine safety measures that would have been more cost effective had not been taken in Chile earlier by any of the parties either by the Chilean government or by international donors. The Chilean story is a vivid, real illustration of a persistent problem, the identified lives effect. Human beings show a greater inclination to assist (and avoid harming) persons and groups identified as those at high risk of great harm than to assist (and avoid harming) persons and groups who will suffer (or already suffer) similar harm but are not identified (as yet). The problem touches almost every aspect of human life and politics: health, the environment, the law. This volume is the first book to tackle the effect from all necessary perspectives. What can social and cognitive sciences teach us about the origin and triggers of the effect? On the philosophical and bioethical level, is the effect a "bias" to be eliminated or is it morally justified? What implications does the law have for health care, law, the environment, and other domains?

Research on psychedelic medicines is experiencing a revival. Some clinicians, scientists, and ethicists believe that psychedelics are so different from other treatments that they warrant special consideration in how they are researched, regulated, commercialized, and administered. Others argue that psychedelic medicines show clinical potential, but they should be treated like other medical interventions. In other words, identical standards should apply. This article analyzes whether psychedelic medicines warrant special consideration from a regulatory and ethical perspective.