Beatrice Marchegiani

The Personalized Patient Preference Predictor (P4) has been proposed as an AI tool to aid surrogate decision-making when incapacitated patients lack advance directives. Unlike population-level Patient Preference Predictors (PPPs), which infer preferences from demographic correlations, P4s fine-tune large language models (LLMs) on a patient's digital footprint to simulate their likely treatment preferences. The goal is to preserve autonomy by grounding predictions in individualized data rather than broad statistical trends. This paper argues that P4s face a fundamental dilemma: they are either unnecessary or unhelpful. When relevant, individualized evidence of preferences exists in a patient's digital footprint, the P4 is unnecessary and suboptimal, since the original data can be examined directly, with richer context than a generative model can preserve. When no such evidence exists, the P4 is unhelpful or misleading, producing plausible-sounding outputs that in the best case rely on population-level correlations rather than the patient's own values. To address these limitations, I propose a better AI-powered alternative: the Patient Preference Retriever (PPR). Rather than generating new text, the PPR uses vector search techniques to retrieve relevant statements from a patient's digital footprint, presenting them verbatim alongside metadata such as date, context, and source. This approach offers greater transparency, respects autonomy more reliably, and supports surrogate decision-makers in weighing authentic evidence. I conclude that while advance directives remain the gold standard, retrieval-based approaches like the PPR provide a more reliable and ethically defensible use of AI in surrogate decision-making than generative approaches like P4s.

Can an AI truly be your digital twin? Technology companies, startups, and even academic researchers increasingly claim so. From grief-bots that promise to let you talk with deceased loved ones to clinical tools designed to predict patients' treatment preferences, personalized Large Language Models are being marketed as faithful replications of individual identity, personality, and values. The digital twin label—borrowed from industrial engineering, where it describes computational models precisely mirroring physical systems—lends these claims an aura of scientific credibility. But is this metaphor appropriate, or is it dangerously misleading? This paper argues that applying the digital twin metaphor to personalized LLMs constitutes a systematic mischaracterization with serious ethical consequences. To this end, we first outline three plausible interpretations of the PDT metaphor—behaviorist, representational, and phenomenal. We then show that current AI systems do not satisfy the conditions of any of these interpretations, rendering the PDT metaphor ultimately inappropriate. The consequences of this metaphorical overreach are far from abstract. When vulnerable individuals—grieving families, patients facing incapacity, people seeking psychological support—interact with systems marketed as preserving human essence, they risk forming attachments to sophisticated illusions. The metaphor fosters misplaced trust, distorts public understanding of AI capabilities, and shapes policy debates on false premises. As these technologies proliferate into healthcare, legal proceedings, and intimate relationships, metaphorical precision becomes an ethical imperative. We conclude by proposing alternative frameworks that honestly represent what these systems can and cannot do.

Conversational AIs (CAIs) are autonomous systems capable of engaging in natural language interactions with users. Recent advancements have enabled CAIs to engage in conversations with users that are virtually indistinguishable from human interactions. The proliferation of advanced CAIs creates a significant risk of users misattributing human‐like traits to CAIs. Such false beliefs can occur when the CAI's nature is not disclosed and users mistakenly believe they are interacting with a human, or even if the CAI is disclosed, through subconscious anthropomorphism. Existing literature on anthropomorphism and AI addresses the instrumental harms associated with anthropomorphism. I argue that anthropomorphizing CAIs might be bad in itself as it undermines user autonomy. My analysis focuses on how anthropomorphic false beliefs lead users to misapply behavioural norms, thereby affecting their autonomy. My argument will proceed as follows. I discuss the prevalence of CAIs, and establish the empirical claim that users are likely to form anthropomorphic false beliefs about CAIs. I explore existing accounts of how some false beliefs undermine autonomy and outline a plausible characterization to identify which false beliefs do so. I apply this characterization to CAIs and argue that misattributing human‐like capabilities to CAIs is the kind of false belief that undermines autonomy.