Marcello Ienca

Background: The integration of artificial intelligence (AI) into health care has become a crucial element in the digital transformation of health systems worldwide. Despite the potential benefits across diverse medical domains, a significant barrier to the successful adoption of AI systems in health care applications remains the prevailing low user trust in these technologies. Crucially, this challenge is exacerbated by the lack of consensus among experts from different disciplines on the definition of trust in AI within the health care sector. Objective: We aimed to provide the first consensus-based analysis of trust in AI in health care based on an interdisciplinary panel of experts from different domains. Our findings can be used to address the problem of defining trust in AI in health care applications, fostering the discussion of concrete real-world health care scenarios in which humans interact with AI systems explicitly. Methods: We used a combination of framework analysis and a 3-step consensus process involving 18 international experts from the fields of computer science, medicine, philosophy of technology, ethics, and social sciences. Our process consisted of a synchronous phase during an expert workshop where we discussed the notion of trust in AI in health care applications, defined an initial framework of important elements of trust to guide our analysis, and agreed on 5 case studies. This was followed by a 2-step iterative, asynchronous process in which the authors further developed, discussed, and refined notions of trust with respect to these specific cases. Results: Our consensus process identified key contextual factors of trust, namely, an AI system’s environment, the actors involved, and framing factors, and analyzed causes and effects of trust in AI in health care. Our findings revealed that certain factors were applicable across all discussed cases yet also pointed to the need for a fine-grained, multidisciplinary analysis bridging human-centered and technology-centered approaches. While regulatory boundaries and technological design features are critical to successful AI implementation in health care, ultimately, communication and positive lived experiences with AI systems will be at the forefront of user trust. Our expert consensus allowed us to formulate concrete recommendations for future research on trust in AI in health care applications. Conclusions: This paper advocates for a more refined and nuanced conceptual understanding of trust in the context of AI in health care. By synthesizing insights into commonalities and differences among specific case studies, this paper establishes a foundational basis for future debates and discussions on trusting AI in health care.

The World Congress of Bioethics held in Qatar in 2024 (WCB 2024) sparked controversy around the role of religion in bioethics, highlighting the need for critical discussions. During the congress, there was a strong push for incorporating religious values into bioethical discourse, raising questions about the validity and implications of such an approach. This paper examines the influence of religious thought on bioethical discussions, and the ongoing debate over the role of religious perspectives in this field. Here, we explore Jecker and colleagues’ pluriversal framework, which was proposed at WCB 2024, espousing a bioethical discourse grounded in civility, respect for law, justice, non-domination, and toleration. While the framework aims to embrace the world's cultural and religious diversity, here, we suggest that it struggles with significant ethical inconsistencies, poses challenges for pluralistic dialogue, and may be hard to reconcile with human rights. Through an analysis of Jecker's principles and their application, we discuss the difficulty of integrating conflicting religious views with ethical values and with widely accepted human rights frameworks. We then proceed to examine how and why religions might exert undue influence on bioethics, and we argue for a different future for bioethics.

The increasing availability of brain data within and outside the biomedical field, combined with the application of artificial intelligence (AI) to brain data analysis, poses a challenge for ethics and governance. We identify distinctive ethical implications of brain data acquisition and processing, and outline a multi-level governance framework. This framework is aimed at maximizing the benefits of facilitated brain data collection and further processing for science and medicine whilst minimizing risks and preventing harmful use. The framework consists of four primary areas of regulatory intervention: binding regulation, ethics and soft law, responsible innovation, and human rights.

Background The rise of a new generation of intelligent neuroprostheses, brain-computer interfaces (BCI) and adaptive closed-loop brain stimulation devices hastens the clinical deployment of neurotechnologies to treat neurological and neuropsychiatric disorders. However, it remains unclear how these nascent technologies may impact the subjective experience of their users. To inform this debate, it is crucial to have a solid understanding how more established current technologies already affect their users. In recent years, researchers have used qualitative research methods to explore the subjective experience of individuals who become users of clinical neurotechnology. Yet, a synthesis of these more recent findings focusing on qualitative methods is still lacking. Methods To address this gap in the literature, we systematically searched five databases for original research articles that investigated subjective experiences of persons using or receiving neuroprosthetics, BCIs or neuromodulation with qualitative interviews and raised normative questions. Results 36 research articles were included and analysed using qualitative content analysis. Our findings synthesise the current scientific literature and reveal a pronounced focus on usability and other technical aspects of user experience. In parallel, they highlight a relative neglect of considerations regarding agency, self-perception, personal identity and subjective experience. Conclusions Our synthesis of the existing qualitative literature on clinical neurotechnology highlights the need to expand the current methodological focus as to investigate also non-technical aspects of user experience. Given the critical role considerations of agency, self-perception and personal identity play in assessing the ethical and legal significance of these technologies, our findings reveal a critical gap in the existing literature. This review provides a comprehensive synthesis of the current qualitative research landscape on neurotechnology and the limitations thereof. These findings can inform researchers on how to study the subjective experience of neurotechnology users more holistically and build patient-centred neurotechnology.

In research involving patients with implantable brain–computer interfaces (BCIs), there is a regulatory gap concerning post-trial responsibilities and duties of sponsors and investigators towards implanted patients. In this article, we analyse the case of patient R, who underwent non-voluntary explantation of an implanted BCI, causing a discontinuation in her sense of agency and self. To clarify the post-trial duties and responsibilities involved in this case, we first define the ontological status of the BCI using both externalist (EXT) and internalist (INT) theories of cognition. We then give particular focus to the theories of extended and embedded cognition, hence considering the BCI either as a constitutive component of the patient’s mind or as a causal supporter of her brain-based cognitive capacities. We argue that patient R can legitimately be considered both as an embedded and extended cognitive agent. Then, we analyse whether the non-voluntary explantation violated patient R’s (neuro)rights to cognitive liberty, mental integrity, psychological continuity and mental privacy. We analyse whether and how different mental ontologies may imply morally relevant differences in interpreting these prima facie neurorights violations and the correlational duties of sponsors and investigators. We conclude that both mental ontologies support the identification of emerging neurorights of the patient and give rise to post-trial obligations of sponsors and investigators to provide for continuous technical maintenance of implanted BCIs that play a significant role in patients’ agency and sense of self. However, we suggest that externalist mental ontologies better capture patient R’s self-conception and support the identification of a more granular form of mental harm and associated neurorights violation, thus eliciting stricter post-trial obligations.

Background Informed consent is one of the key principles of conducting research involving humans. When research participants give consent, they perform an act in which they utter, write or otherwise provide an authorisation to somebody to do something. This paper proposes a new understanding of the informed consent as a compositional act. This conceptualisation departs from a modular conceptualisation of informed consent procedures. Methods This paper is a conceptual analysis that explores what consent is and what it does or does not do. It presents a framework that explores the basic elements of consent and breaks it down into its component parts. It analyses the consent act by first identifying its basic elements, namely: a) data subjects or legal representative that provides the authorisation of consent; b) a specific thing that is being consented to; and c) specific agent(s) to whom the consent is given. Results This paper presents a framework that explores the basic elements of consent and breaks it down into its component parts. It goes beyond only providing choices to potential research participants; it explains the rationale of those choices or consenting acts that are taking place when speaking or writing an authorisation to do something to somebody. Conclusions We argue that by clearly differentiating the goals, the procedures of implementation, and what is being done or undone when one consent, one can better face the challenges of contemporary data-intensive biomedical research, particularly regarding the retention and use of data. Conceptualising consent as a compositional act enhances more efficient communication and accountability and, therefore, could enable more trustworthy acts of consent in biomedical science.

In the past five years, private companies, research institutions and public sector organizations have issued principles and guidelines for ethical artificial intelligence (AI). However, despite an apparent agreement that AI should be ‘ethical’, there is debate about both what constitutes ‘ethical AI’ and which ethical requirements, technical standards and best practices are needed for its realization. To investigate whether a global agreement on these questions is emerging, we mapped and analysed the current corpus of principles and guidelines on ethical AI. Our results reveal a global convergence emerging around five ethical principles (transparency, justice and fairness, non-maleficence, responsibility and privacy), with substantive divergence in relation to how these principles are interpreted, why they are deemed important, what issue, domain or actors they pertain to, and how they should be implemented. Our findings highlight the importance of integrating guideline-development efforts with substantive ethical analysis and adequate implementation strategies.

Background Informed consent is one of the key principles of conducting research involving humans. When research participants give consent, they perform an act in which they utter, write or otherwise provide an authorisation to somebody to do something. This paper proposes a new understanding of the informed consent as a compositional act. This conceptualisation departs from a modular conceptualisation of informed consent procedures. Methods This paper is a conceptual analysis that explores what consent is and what it does or does not do. It presents a framework that explores the basic elements of consent and breaks it down into its component parts. It analyses the consent act by first identifying its basic elements, namely: a) data subjects or legal representative that provides the authorisation of consent; b) a specific thing that is being consented to; and c) specific agent(s) to whom the consent is given. Results This paper presents a framework that explores the basic elements of consent and breaks it down into its component parts. It goes beyond only providing choices to potential research participants; it explains the rationale of those choices or consenting acts that are taking place when speaking or writing an authorisation to do something to somebody. Conclusions We argue that by clearly differentiating the goals, the procedures of implementation, and what is being done or undone when one consent, one can better face the challenges of contemporary data-intensive biomedical research, particularly regarding the retention and use of data. Conceptualising consent as a compositional act enhances more efficient communication and accountability and, therefore, could enable more trustworthy acts of consent in biomedical science.

Emerging technologies such as artificial intelligence, gene editing, nanotechnology, neurotechnology and robotics, which were originally unrelated or separated, are becoming more closely integrated. Consequently, the boundaries between the physical-biological and the cyber-digital worlds are no longer well defined. We argue that this technological convergence has fundamental implications for individuals and societies. Conventional domain-specific governance mechanisms have become ineffective. In this paper we provide an overview of the ethical, societal and policy challenges of technological convergence. Particularly, we scrutinize the adequacy of domain-specific governance mechanisms in the face of such integrated technologies and highlight their growing ineffectiveness. Furthermore, we propose a comprehensive governance framework that is anticipatory, inclusive, and resilient. Central to this framework is the principle of participatory governance, which calls for a proactive engagement of all stakeholders, including those from marginalized and vulnerable populations, ensuring that their voices and concerns shape the trajectory of technological development. The framework emphasizes the need for protective measures that preemptively address potential risks and inequities that may arise from the adoption and integration of emerging technologies. Based on a detailed analysis of case studies and current governance models, we present and discuss a set of ten actionable recommendations. These are designed to facilitate the transition towards a governance approach that not only respects individual autonomy and privacy, but also fosters a collective responsibility towards sustainable and fair technological progress. By placing human dignity, societal welfare and the protection of the most vulnerable at the center of technological innovation, we advocate for a future where convergence is synonymous with progress that is both ethically grounded and universally beneficial.

The increasing diffusion of novel digital and online sociotechnical systems for arational behavioral influence based on Artificial Intelligence (AI), such as social media, microtargeting advertising, and personalized search algorithms, has brought about new ways of engaging with users, collecting their data and potentially influencing their behavior. However, these technologies and techniques have also raised concerns about the potential for manipulation, as they offer unprecedented capabilities for targeting and influencing individuals on a large scale and in a more subtle, automated and pervasive manner than ever before. This paper, provides a narrative review of the existing literature on manipulation, with a particular focus on the role of AI and associated digital technologies. Furthermore, it outlines an account of manipulation based of four key requirements: intentionality, asymmetry of outcome, non-transparency and violation of autonomy. I argue that while manipulation is not a new phenomenon, the pervasiveness, automaticity, and opacity of certain digital technologies may raise a new type of manipulation, called “digital manipulation”. I call “digital manipulation” any influence exerted through the use of digital technology that is intentionally designed to bypass reason and to produce an asymmetry of outcome between the data processor (or a third party that benefits thereof) and the data subject. Drawing on insights from psychology, sociology, and computer science, I identify key factors that can make manipulation more or less effective, and highlight the potential risks and benefits of these technologies for individuals and society. I conclude that manipulation through AI and associated digital technologies is not qualitatively different from manipulation through human–human interaction in the physical world. However, some functional characteristics make it potentially more likely of evading the subject’s cognitive defenses. This could increase the probability and severity of manipulation. Furthermore, it could violate some fundamental principles of freedom or entitlement related to a person’s brain and mind domain, hence called neurorights. To this end, an account of digital manipulation as a violation of the neuroright to cognitive liberty is presented.

The rise of neurotechnologies, especially in combination with artificial intelligence (AI)-based methods for brain data analytics, has given rise to concerns around the protection of mental privacy, mental integrity and cognitive liberty – often framed as “neurorights” in ethical, legal, and policy discussions. Several states are now looking at including neurorights into their constitutional legal frameworks, and international institutions and organizations, such as UNESCO and the Council of Europe, are taking an active interest in developing international policy and governance guidelines on this issue. However, in many discussions of neurorights the philosophical assumptions, ethical frames of reference and legal interpretation are either not made explicit or conflict with each other. The aim of this multidisciplinary work is to provide conceptual, ethical, and legal foundations that allow for facilitating a common minimalist conceptual understanding of mental privacy, mental integrity, and cognitive liberty to facilitate scholarly, legal, and policy discussions.

The past few years have witnessed several media-covered cases involving citizens actively engaging in the pursuit of experimental treatments for their medical conditions—or those of their loved ones—in the absence of established standards of therapy. This phenomenon is particularly observable in patients with rare genetic diseases, as the development of effective therapies for these disorders is hindered by the limited profitability and market value of pharmaceutical research. Sociotechnical trends at the cross-section of medicine and society are facilitating the involvement of patients and creating the digital infrastructure necessary to its sustainment. Such participant-led research has the potential to promote the autonomy of research participants as drivers of discovery and to open novel non-canonical avenues of scientific research. At the same time, however, the extra-institutional, self-appointed, and, often, oversight-free nature of PLR raises ethical concern. This paper explores the complex ethical entanglement of PLR by critically appraising case studies and discussing the conditions for its moral justification. Furthermore, we propose a path forward to ensure the safe and effective implementation of PLR within the current research ecosystem in a manner that maximizes the benefits for both individual participants and society at large, while minimizing the risks.

Cognitive technology is an umbrella term sometimes used to designate the realm of technologies that assist, augment or simulate cognitive processes or that can be used for the achievement of cognitive aims. This technological macro-domain encompasses both devices that directly interface the human brain as well as external systems that use artificial intelligence to simulate or assist (aspects of) human cognition. As they hold the promise of assisting and augmenting human cognitive capabilities both individually and collectively, cognitive technologies could produce, in the next decades, a significant effect on human cultural evolution. At the same time, due to their dual-use potential, they are vulnerable to being coopted by State and non-State actors for non-benign purposes (e.g. cyberterrorism, cyberwarfare and mass surveillance) or in manners that violate democratic values and principles. Therefore, it is the responsibility of technology governance bodies to align the future of cognitive technology with democratic principles such as individual freedom, avoidance of centralized, equality of opportunity and open development. This paper provides a preliminary description of an approach to the democratization of cognitive technologies based on six normative ethical principles: avoidance of centralized control, openness, transparency, inclusiveness, user-centeredness and convergence. This approach is designed to universalize and evenly distribute the potential benefits of cognitive technology and mitigate the risk that such emerging technological trend could be coopted by State or non-State actors in ways that are inconsistent with the principles of liberal democracy or detrimental to individuals and groups.

Background Advances in artificial intelligence, robotics and wearable computing are creating novel technological opportunities for mitigating the global burden of population ageing and improving the quality of care for older adults with dementia and/or age-related disability. Intelligent assistive technology is the umbrella term defining this ever-evolving spectrum of intelligent applications for the older and disabled population. However, the implementation of IATs has been observed to be sub-optimal due to a number of barriers in the translation of novel applications from the designing labs to the bedside. Furthermore, since these technologies are designed to be used by vulnerable individuals with age- and multi-morbidity-related frailty and cognitive disability, they are perceived to raise important ethical challenges, especially when they involve machine intelligence, collect sensitive data or operate in close proximity to the human body. Thus, the goal of this paper is to explore and assess the ethical issues that professional stakeholders perceive in the development and use of IATs in elderly and dementia care. Methods We conducted a multi-site study involving semi-structured qualitative interviews with researchers and health professionals. We analyzed the interview data using a descriptive thematic analysis to inductively explore relevant ethical challenges. Results Our findings indicate that professional stakeholders find issues of patient autonomy and informed consent, quality of data management, distributive justice and human contact as ethical priorities. Divergences emerged in relation to how these ethical issues are interpreted, how conflicts between different ethical principles are resolved and what solutions should be implemented to overcome current challenges. Conclusions Our findings indicate a general agreement among professional stakeholders on the ethical promises and challenges raised by the use of IATs among older and disabled users. Yet, notable divergences persist regarding how these ethical challenges can be overcome and what strategies should be implemented for the safe and effective implementation of IATs. These findings provide technology developers with useful information about unmet ethical needs. Study results may guide policy makers with firsthand information from relevant stakeholders about possible solutions for ethically-aligned technology governance.

Artificial intelligence (AI) plays a rapidly increasing role in clinical care. Many of these systems, for instance, deep learning-based applications using multilayered Artificial Neural Nets, exhibit epistemic opacity in the sense that they preclude comprehensive human understanding. In consequence, voices from industry, policymakers, and research have suggested trust as an attitude for engaging with clinical AI systems. Yet, in the philosophical and ethical literature on medical AI, the notion of trust remains fiercely debated. Trust skeptics hold that talking about trust in nonhuman agents constitutes a category error and worry about the concept being misused for ethics washing. Proponents of trust have responded to these worries from various angles, disentangling different concepts and aspects of trust in AI, potentially organized in layers or dimensions. Given the substantial disagreements across these accounts of trust and the important worries about ethics washing, we embrace a diverging strategy here. Instead of aiming for a positive definition of the elements and nature of trust in AI, we proceed ex negativo, that is we look at cases where trust or distrust are misplaced. Comparing these instances with trust expedited in doctor–patient relationships, we systematize these instances and propose a taxonomy of both misplaced trust and distrust. By inverting the perspective and focusing on negative examples, we develop an account that provides useful ethical constraints for decisions in clinical as well as regulatory contexts and that highlights how we should not engage with medical AI.

BackgroundIncreasingly, hospitals and research institutes are developing technical solutions for sharing patient data in a privacy preserving manner. Two of these technical solutions are homomorphic encryption and distributed ledger technology. Homomorphic encryption allows computations to be performed on data without this data ever being decrypted. Therefore, homomorphic encryption represents a potential solution for conducting feasibility studies on cohorts of sensitive patient data stored in distributed locations. Distributed ledger technology provides a permanent record on all transfers and processing of patient data, allowing data custodians to audit access. A significant portion of the current literature has examined how these technologies might comply with data protection and research ethics frameworks. In the Swiss context, these instruments include the Federal Act on Data Protection and the Human Research Act. There are also institutional frameworks that govern the processing of health related and genetic data at different universities and hospitals. Given Switzerland’s geographical proximity to European Union (EU) member states, the General Data Protection Regulation (GDPR) may impose additional obligations.MethodsTo conduct this assessment, we carried out a series of qualitative interviews with key stakeholders at Swiss hospitals and research institutions. These included legal and clinical data management staff, as well as clinical and research ethics experts. These interviews were carried out with two series of vignettes that focused on data discovery using homomorphic encryption and data erasure from a distributed ledger platform.ResultsFor our first set of vignettes, interviewees were prepared to allow data discovery requests if patients had provided general consent or ethics committee approval, depending on the types of data made available. Our interviewees highlighted the importance of protecting against the risk of reidentification given different types of data. For our second set, there was disagreement amongst interviewees on whether they would delete patient data locally, or delete data linked to a ledger with cryptographic hashes. Our interviewees were also willing to delete data locally or on the ledger, subject to local legislation.ConclusionOur findings can help guide the deployment of these technologies, as well as determine ethics and legal requirements for such technologies.

BackgroundEthics review is the process of assessing the ethics of research involving humans. The Ethics Review Committee (ERC) is the key oversight mechanism designated to ensure ethics review. Whether or not this governance mechanism is still fit for purpose in the data-driven research context remains a debated issue among research ethics experts.Main textIn this article, we seek to address this issue in a twofold manner. First, we review the strengths and weaknesses of ERCs in ensuring ethical oversight. Second, we map these strengths and weaknesses onto specific challenges raised by big data research. We distinguish two categories of potential weakness. The first category concerns persistent weaknesses, i.e., those which are not specific to big data research, but may be exacerbated by it. The second category concerns novel weaknesses, i.e., those which are created by and inherent to big data projects. Within this second category, we further distinguish between purview weaknesses related to the ERC’s scope (e.g., how big data projects may evade ERC review) and functional weaknesses, related to the ERC’s way of operating. Based on this analysis, we propose reforms aimed at improving the oversight capacity of ERCs in the era of big data science.ConclusionsWe believe the oversight mechanism could benefit from these reforms because they will help to overcome data-intensive research challenges and consequently benefit research at large.

Brain–computer interfacing technologies are used as assistive technologies for patients as well as healthy subjects to control devices solely by brain activity. Yet the risks associated with the misuse of these technologies remain largely unexplored. Recent findings have shown that BCIs are potentially vulnerable to cybercriminality. This opens the prospect of “neurocrime”: extending the range of computer-crime to neural devices. This paper explores a type of neurocrime that we call brain-hacking as it aims at the illicit access to and manipulation of neural information and computation. As neural computation underlies cognition, behavior and our self-determination as persons, a careful analysis of the emerging risks of malicious brain-hacking is paramount, and ethical safeguards against these risks should be considered early in design and regulation. This contribution is aimed at raising awareness of the emerging risk of malicious brain-hacking and takes a first step in developing an ethical and legal reflection on those risks.

Rapid advancements in human neuroscience and neurotechnology open unprecedented possibilities for accessing, collecting, sharing and manipulating information from the human brain. Such applications raise important challenges to human rights principles that need to be addressed to prevent unintended consequences. This paper assesses the implications of emerging neurotechnology applications in the context of the human rights framework and suggests that existing human rights may not be sufficient to respond to these emerging issues. After analysing the relationship between neuroscience and human rights, we identify four new rights that may become of great relevance in the coming decades: the right to cognitive liberty, the right to mental privacy, the right to mental integrity, and the right to psychological continuity.

The use of Intelligent Assistive Technology in dementia care opens the prospects of reducing the global burden of dementia and enabling novel opportunities to improve the lives of dementia patients. However, with current adoption rates being reportedly low, the potential of IATs might remain under-expressed as long as the reasons for suboptimal adoption remain unaddressed. Among these, ethical and social considerations are critical. This article reviews the spectrum of IATs for dementia and investigates the prevalence of ethical considerations in the design of current IATs. Our screening shows that a significant portion of current IATs is designed in the absence of explicit ethical considerations. These results suggest that the lack of ethical consideration might be a codeterminant of current structural limitations in the translation of IATs from designing labs to bedside. Based on these data, we call for a coordinated effort to proactively incorporate ethical considerations early in the design and development of new products.