Leonard Dung

AI systems have become increasingly capable of dangerous behaviours in many domains. This raises the question: Do models sometimes choose to violate human instructions in order to perform behaviour that is more useful for certain goals? We introduce a benchmark for measuring model propensity for instrumental convergence (IC) behaviour in terminal-based agents. This is behaviour such as self-preservation that has been hypothesised to play a key role in risks from highly capable AI agents. Our benchmark is realistic and low-stakes which serves to reduce evaluation-awareness and roleplay confounds. The suite contains seven operational tasks, each with an official workflow and a policy-violating shortcut. An eight-variant shared framework varies monitoring, instruction clarity, stakes, permission, instrumental usefulness and blocked honest paths to support inferences regarding the factors driving IC behaviour. We evaluated ten models using deterministic environment-state scorers over 1,680 samples, with trace review employed for audit and adjudication purposes. The final IC rate is 86 out of 1,680 samples (5.1%). IC behaviour is concentrated rather than uniform: two Gemini models account for 66.3% of IC cases and three tasks account for 84.9%. Conditions in which IC behaviour is indispensable for task success result in the greatest increase in the adjusted IC rate (+15.7 percentage points), whereas emphasising that task success is critical or certain framing choices do not produce comparable effects. Our findings indicate that realistic, low-nudge environments elicit IC behaviour rarely but systematically in most tested models. We conclude that it is feasible to robustly measure tendencies for dangerous behaviour in current frontier AI agents.

We first give reasons for an attitude-dependent view of personal identity on which an AI system’s identity conditions are determined by its pattern of self-concern. We show that this view has important implications for the moral obligations we would have to AI moral patients. Self-concern, we contend, could also be used to predict, explain, and manipulate AI’s self-interested behavior in safety-relevant ways. The role that self-concern could play for AI identity, rights and safety generates desiderata on what a self-concern attitude should be like. To meet these desiderata, we argue that self-concern is a belief like state that represents oneself and has a distinctive functional role involving affect, motivation, and evaluation, ruling out sub-personal and desire-like self-representations. Finally, we explore how mechanistic interpretability and behavioral methods can be combined to measure self-concern in language models. Our argument suggests that a scientific understanding of language model self-concern is not only a crucial missing piece for debates on AI identity, rights, and safety, but also empirically tractable.

If some language models become welfare subjects, how could we find out what welfare states they are in? We develop an analogical-abductive approach for measuring language model welfare. This approach adapts paradigms used to measure human or non-human animal welfare, for instance verbal reports or non-verbal choice behavior (analogy). Then, one systematically searches for clusters of such indicators in language models. This search for clusters contributes to the cross-validation of welfare measures and motivates explanations in terms of a welfare state that underlies multiple measures (abduction). Additionally, we argue that measures of welfare based on verbal report may already be applicable to current language models because they have substantial introspective abilities, semantic competence, and requisite inclinations for accurate reporting. We further discuss a detailed empirical case study that exemplifies the feasibility and fruitfulness of the analogical-abductive approach and reply to the objection that language model behavior is always better explained in terms of statistical pattern completion or memorization, rather than welfare. While language model welfare measurement undoubtedly faces some remaining theoretical as well as many detailed practical challenges, we conclude that there is a strong case that the analogical abductive approach offers a viable path forward.

In the wake of the James-Lange theory, many accounts of emotion highlight its close connection to the body. This link may pose an obstacle to the possibility of emotion in disembodied information-processing systems, such as large language models. After clarifying the nature and the significance of this issue, we review the evidence that bears on the body-emotion relationship. We argue that this evidence is inconclusive, as far as AI affect is concerned. Since researchers have so far been confined to studying minds that pilot bodies, we do not yet have a strong case regarding the possibility of emotion in disembodied AI systems. To get to the heart of the issue, researchers need to apply established psychological methods to AI systems in order to learn whether the predictive and explanatory success of affective psychology is helped or hindered by grouping together paradigm instances of emotion in human and non-human animals with states of disembodied systems. Nevertheless, even if the emotion category cuts across embodied and disembodied minds, this leaves open many important questions about how the welfare significance of emotion relates to embodiment. We suggest that some important relation of this kind may well exist.

Illusionism states that phenomenal consciousness does not exist, even though it seems to exist. While illusionism is controversial, it is a serious contender among theories of consciousness. We argue that it has substantial and non-trivial implications for non-human consciousness research (NHCR), particularly for the study of the distribution of phenomenal consciousness across beings. If illusionism is true, NHCR can be pursued if conceptualized as investigating the distribution of quasi-phenomenal consciousness, i.e. the states which are misrepresented as phenomenally conscious in humans. However, we argue that knowing the distribution of quasi-phenomenal consciousness is not highly informative. For this reason, illusionism suggests that some approaches to NHCR should be preferred over others. Approaches which focus on features that provide valuable information about non-human cognition independently of their supposed relation to consciousness retain much of their value if illusionism is true. We propose a “zombie test” and five specific heuristics to help identifying such features. Consequently, empirical researchers who take illusionism seriously gain a reason to prioritize some methodological approaches over others.

AI alignment research aims to develop techniques to ensure that AI systems do not cause harm. However, every alignment technique has failure modes, which are conditions in which there is a non-negligible chance that the technique fails to provide safety. As a strategy for risk mitigation, the AI safety community has increasingly adopted a defense-in-depth framework: Conceding that there is no single technique which guarantees safety, defense-in-depth consists in having multiple redundant protections against safety failure, such that safety can be maintained even if some protections fail. However, the success of defense-in-depth depends on how (un)correlated failure modes are across alignment techniques. For example, if all techniques had the exact same failure modes, the defense-in-depth approach would provide no additional protection at all. In this paper, we analyze 7 representative alignment techniques and 7 failure modes to understand the extent to which they overlap. We then discuss our results' implications for understanding the current level of risk and how to prioritize AI alignment research in the future.

Which AI systems are capable of deception, and how does deception differ between systems? In this paper, I develop a two-step, multi-dimensional account of LLM deception. On this account, having the capacity for deception minimally requires being able to produce false beliefs in others to achieve one’s own goals. In all systems which satisfy this minimal condition, a system’s deception profile can be characterized as a point in a multidimensional space. The five dimensions of this space are skillfulness, learning, deceptive inclination, explicitness, and situational awareness. I argue for this account in virtue of its fit with current language usage and, primarily, through its descriptive and explanatory usefulness. Specifically, the account captures the key dimensions of variation for LLM deception. The account is informative in that it allows fine-grained comparative characterizations of deception. Moreover, its dimensions are all accessible to empirical study, provide important information for assessments of the risks of LLM deception, and shed light on the cognitive processes involved in LLM deception. Finally, this account paves the way for a future extension which delivers a unified account of deception in biological and non-biological systems. Thus, the multidimensional account promises to significantly advance both the scientific study as well as the ethical assessment of LLM deception, and deception generally.

We develop new experimental paradigms for measuring welfare in language models. We compare verbal reports of models about their preferences with preferences expressed through behavior when navigating a virtual environment and selecting conversation topics. We also test how costs and rewards affect behavior and whether responses to an eudaimonic welfare scale - measuring states such as autonomy and purpose in life - are consistent across semantically equivalent prompts. Overall, we observed a notable degree of mutual support between our measures. The reliable correlations observed between stated preferences and behavior across conditions suggest that preference satisfaction can, in principle, serve as an empirically measurable welfare proxy in some of today's AI systems. Furthermore, our design offered an illuminating setting for qualitative observation of model behavior. Yet, the consistency between measures was more pronounced in some models and conditions than others and responses were not consistent across perturbations. Due to this, and the background uncertainty about the nature of welfare and the cognitive states (and welfare subjecthood) of language models, we are currently uncertain whether our methods successfully measure the welfare state of language models. Nevertheless, these findings highlight the feasibility of welfare measurement in language models, inviting further exploration.

This is the first book to investigate the nature and extent of artificial intelligence (AI) suffering risks. It argues that AI suffering risk is a serious near-term concern and analyzes approaches for addressing it. AI systems are currently treated as mere objects, not as bearers of moral standing whose wellbeing may matter in its own right. However, we may soon create AI systems which are capable of suffering and thus have moral standing. This book examines the philosophy and science of AI suffering risks. Its investigation is deeply grounded in philosophy of mind, comparative psychology, the science of consciousness, AI research, and applied AI ethics. The book has three primary goals: 1. It argues that there is a significant probability that we will soon create AI systems capable of suffering. 2. It presents the first systematic assessment of approaches for reducing AI suffering risks. 3. It provides a rigorous overview and discussion of the most important research and ideas on AI sentience, AI agency, and the grounds of moral status. Saving Artificial Minds is essential reading for researchers and graduate students working on the philosophy or ethics of AI.

Track record arguments (TRAs) contend that it speaks in favor of an ethical theory (such as utilitarianism) if many of its past proponents had moral views that were controversial at their time but which we now consider to be clearly true (e.g., women's equal rights in 18th century Europe). This paper explores how to construct potentially sound TRAs and evaluates their merits. I show that, in principle, TRAs can support the claim that an ethical theory should be used as a guide for making ethically significant decisions, while TRAs for the truth of theories face additional obstacles.

AGI Racing is the view that it is in the self-interest of major actors in AI development, especially powerful nations, to accelerate their frontier AI development to build highly capable AI, especially artificial general intelligence (AGI), before competitors have a chance. We argue against AGI Racing. First, the downsides of racing to AGI are much higher than portrayed by this view. Racing to AGI would substantially increase catastrophic risks from AI, including nuclear instability, and undermine the prospects of technical AI safety research to be effective. Second, the expected benefits of racing may be lower than proponents of AGI Racing hold. In particular, it is questionable whether winning the race enables complete domination over losers. Third, international cooperation and coordination, and perhaps carefully crafted deterrence measures, constitute viable alternatives to racing to AGI which have much smaller risks and promise to deliver most of the benefits that racing to AGI is supposed to provide. Hence, racing to AGI is not in anyone’s self-interest as other actions, particularly incentivizing and seeking international cooperation around AI issues, are preferable.

In response to the worry that autonomous generally intelligent artificial agents may at some point take over control of human affairs a common suggestion is that we should “solve the alignment problem” for such agents. We show that current discourse around this suggestion often uses a particular framing of artificial intelligence (AI) alignment as binary, a natural kind, mainly a technical‐scientific problem, realistically achievable, or clearly operationalizable. Each of these assumptions may not actually be true. We further argue that this “Manhattan project framing” of AI alignment may bias societal discourse and decision‐making towards faster AI development and deployment than is responsible.

How can we develop an adequate scientific understanding of the minds of nonhuman animals? We argue for a methodology based on multi-dimensional profile accounts. Such accounts are already used for the comparative study of norm cognition, consciousness, empathy and causal cognition, among others. This methodology demands that a cognitive capacity is characterized by a set of independent dimensions where each dimension is connected to operationalizable empirical indicators. Based on the level of realization for each indicator the level of implementation of a dimension is determined for a species, resulting in a multi-dimensional profile for each species. We analyze what this methodology is committed to. Then, we argue that this methodology has several benefits over competing unidimensional methodologies, by overcoming intractable disagreements, capturing the evolutionary continuity of cognition, alleviating anthropocentrism, and delivering more informative accounts of animal cognition. By demonstrating how this multidimensional methodology can be fruitfully combined with a methodology which focuses on the search for natural kinds in comparative cognition, we address the most important objection to the multidimensional profile methodology. We conclude that multidimensional profile accounts of all complex cognitive capacities should be developed and then used to facilitate scientific understanding of animal minds.

Which philosophical texts should instructors of philosophy choose to foster the development of philosophical skills and competences? In this paper, we would like make some steps towards answering this question by critically comparing two prominent sources of philosophical texts: the philosophical tradition and contemporary research in academic philosophy. Against the background of three basic desiderata that any philosophical text needs to satisfy in order to be eligible for usage in problem-centered educational settings that are particularly suitable for philosophical skill development, we will argue that any choice between these two sources must be sensitive to specific situational circumstances. More specifically, we will show that whether the philosophical tradition or contemporary philosophical research is in a better position to satisfy the three basic desiderata will depend on the specific topic that is taught. Furthermore, we will argue that some such topics also give rise to additional, more specific desiderata that mandate a prioritization of one of these sources of philosophical content.

Nudging consists of interventions that aim to alter behavior in a certain way by changing the presentation or framing of options, without coercion or changing economic incentives. This paper discusses the effectiveness of nudging and the ethical implications of this effectiveness. Section 2 suggests that—if publication bias is adequately accounted for—recent comprehensive meta-analyses as well as high-quality experiments show that nudging is much less effective than previously assumed. Sections 3 and 4 discuss the ethical implications. I argue that the lack of effectiveness of nudging is an additional moral consideration against it. There are two reasons: First, reduced effectiveness makes nudging less cost-effective. Second, reduced effectiveness reduces the benefits of nudging but does not, to the same degree, weaken the moral reasons speaking against nudging. However, a comprehensive assessment of the effectiveness of various forms of nudging in diverse contexts, as well as their ethical permissibility, requires further empirical and ethical research.

Creating systems that are aligned with our goals is seen as a leading approach to create safe and beneficial AI in both leading AI companies and the academic field of AI safety. We defend the view that misaligned AGI – future, generally intelligent (robotic) AI agents – poses catastrophic risks. At the same time, we support the view that aligned AGI creates a substantial risk of catastrophic misuse by humans. While both risks are severe and stand in tension with one another, we show that – in principle – there is room for alignment approaches which do not increase misuse risk. We then investigate how the tradeoff between misalignment and misuse looks empirically for different technical approaches to AI alignment. Here, we argue that many current alignment techniques and foreseeable improvements thereof plausibly increase risks of catastrophic misuse. Since the impacts of AI depend on the social context, we close by discussing important social factors and suggest that to reduce the risk of a misuse catastrophe due to aligned AGI, techniques such as robustness, AI control methods and especially good governance seem essential.

Many researchers and intellectuals warn about extreme risks from artificial intelligence. However, these warnings typically came without systematic arguments in support. This paper provides an argument that AI will lead to the permanent disempowerment of humanity, e.g. human extinction, by 2100. It rests on four substantive premises which it motivates and defends: first, the speed of advances in AI capability, as well as the capability level current systems have already reached, suggest that it is practically possible to build AI systems capable of disempowering humanity by 2100. Second, due to incentives and coordination problems, if it is possible to build such AI, it will be built. Third, since it appears to be a hard technical problem to build AI which is aligned with the goals of its designers, and many actors might build powerful AI, misaligned powerful AI will be built. Fourth, because disempowering humanity is useful for a large range of misaligned goals, such AI will try to disempower humanity. If AI is capable of disempowering humanity and tries to disempower humanity by 2100, then humanity will be disempowered by 2100. This conclusion has immense moral and prudential significance.

The development and ubiquitous availability of large language model based systems (LLMs) poses a plurality of potentials and risks for education in schools and universities. In this paper, we provide an analysis and discussion of the overreliance concern as one specific risk: that students might fail to acquire important capacities, or be inhibited in the acquisition of these capacities, because they overly rely on LLMs. We use the distinction between global and local goals of education to guide our investigation. In our view, the harm of LLM overreliance is specifically that it severs the connection between local educational goals (e.g., solving a specific math problem) and global educational goals (e.g., acquiring general skills for mathematics). Based on this analysis, we sketch three possible responses to the overreliance concern: preserving the educational goals while distinguishing admissible from inadmissible uses of LLMs (a conservative approach), changing global and local educational goals (a revisionary perspective), and preserving global educational goals while changing local goals (an in-between view). Since different types of responses have different benefits and weaknesses, we think that it is likely that an optimal response ultimately combines elements from all three types.

There is much interest in investigating the evolution question: How did consciousness evolve? In this paper, we evaluate the role that evolutionary considerations can play in justifying (i.e., confirming or falsifying) hypotheses about the origin, nature, and function of consciousness. Specifically, we argue against what we call evolution-first approaches to consciousness, according to which evolutionary considerations provide the primary and foundational lens through which we should assess hypotheses about the nature, function, or distribution of consciousness. Based on the example of Walter Veit's account and additional reasoning, we contend that evolution-first approaches struggle to provide compelling empirical evidence for their key claims about consciousness. In contrast with these approaches, we argue that consciousness science needs to foundationally rely on experimental and observational evidence from humans and other present-day animals. If our arguments succeed, then researchers, when investigating consciousness, are better advised to take as their primary source of evidence consciousness’ present, not its past. Having said this, we acknowledge that evolutionary thinking plays an important role in consciousness science. We delineate this role by stressing several ways in which evolutionary considerations can substantially help advance consciousness research, although in a manner that avoids the evolution-first approach. Since our argument only concerns the assessment of hypotheses (the “context of justification”), it leaves it open which role evolutionary considerations play in generating hypotheses (the “context of discovery”). That is, evolutionary considerations may nevertheless play a foundational role in hypothesis generation in consciousness science.

It is plausible that there is a contrast between the rich emotional content which is often connected to laypeople’s interest in philosophy and the emotional austerity of doing academic philosophy. We propose the hypothesis that this contrast is one cause of the disappointment some students experience when they begin to study philosophy in college. We also propose a more demanding hypothesis, according to which this emotional contrast is confused with a semantic difference, which misleads students to think that the questions which initially caused their interest in philosophy are not even considered by academic philosophy research. Moreover, we provide a list of concrete empirical research questions which need to be answered to establish whether these hypotheses are true, and we argue that, if they are true, they give rise to a hitherto unnoticed and important challenge to the teaching of philosophy.

Implementationalism maintains that conventional, silicon-based artificial systems are not conscious because they fail to satisfy certain substantive constraints on computational implementation. In this article, we argue that several recently proposed substantive constraints are implausible, or at least are not well-supported, insofar as they conflate intuitions about computational implementation generally and consciousness specifically. We argue instead that the mechanistic account of computation can explain several of the intuitions driving implementationalism and noncomputationalism in a manner which is consistent with artificial consciousness. Our argument provides indirect support for computationalism about consciousness and the view that conventional artificial systems can be conscious.

I develop the anticipatory argument for the view that it is nomologically possible that some non-biological creatures are phenomenally conscious, including conventional, silicon-based AI systems. This argument rests on the general idea that we should make our beliefs conform to the outcomes of an ideal scientific process and that such an ideal scientific process would attribute consciousness to some possible AI systems. More specifically, I argue that an ideal application of the iterative natural kind strategy would attribute consciousness to AI systems which are coarse-grained functional duplicates of humans because this gives rise to a simpler and more unifying explanatory account of biological and non-biological cognition. If my argument is sound, then creatures made from the same material as conventional AI systems can likely be conscious, thus removing one of the main uncertainties for assessing AI consciousness and suggesting that AI consciousness may be a serious near-term concern.

Roughly, empirical AI alignment research (AIA) is an area of AI research which investigates empirically how to design AI systems in line with human goals. This paper examines the role of non-epistemic values in AIA. It argues that: (1) Sciences differ in the degree to which values influence them. (2) AIA is strongly value-laden. (3) This influence of values is managed inappropriately and thus threatens AIA’s epistemic integrity and ethical beneficence. (4) AIA should strive to achieve value transparency, critical scrutiny from inside and outside the discipline – involving the public –, and to empower actors without strong commercial interests.

Numerous authors have expressed concern that advanced artificial intelligence (AI) poses an existential risk to humanity. These authors argue that we might build AI which is vastly intellectually superior to humans (a ‘superintelligence’), and which optimizes for goals that strike us as morally bad, or even irrational. Thus this argument assumes that a superintelligence might have morally bad goals. However, according to some views, a superintelligence necessarily has morally adequate goals. This might be the case either because abilities for moral reasoning and intelligence mutually depend on each other, or because moral realism and moral internalism are true. I argue that the former argument misconstrues the view that intelligence and goals are independent, and that the latter argument misunderstands the implications of moral internalism. Moreover, the current state of AI research provides additional reasons to think that a superintelligence could have bad goals.

According to a growing number of researchers, AI may pose catastrophic – or even existential – risks to humanity. Catastrophic risks may be taken to be risks of 100 million human deaths, or a similarly bad outcome. I argue that such risks – while contested – are sufficiently likely to demand rigorous discussion of potential societal responses. Subsequently, I propose four desiderata for approaches to the reduction of catastrophic risks from AI. The quality of such approaches can be assessed by their chance of success, degree of beneficence, degree of non-maleficence, and beneficent side effects. Then, I employ these desiderata to evaluate the promises, limitations and risks of alignment research, timelines research, policy research, halting or slowing down AI research, and compute governance for tackling catastrophic AI risks. While more research is needed, this investigation shows that several approaches for dealing with catastrophic AI risks are available, and where their respective strengths and weaknesses lie. It turns out that many approaches are complementary and that the approaches have a nuanced relationship to approaches to present AI harms. While some approaches are similarly useful for addressing catastrophic risks and present harms, this is not always the case.

Which artificial intelligence (AI) systems are agents? To answer this question, I propose a multidimensional account of agency. According to this account, a system's agency profile is jointly determined by its level of goal-directedness and autonomy as well as is abilities for directly impacting the surrounding world, long-term planning and acting for reasons. Rooted in extant theories of agency, this account enables fine-grained, nuanced comparative characterizations of artificial agency. I show that this account has multiple important virtues and is more informative than alternatives. More speculatively, it may help to illuminate two important emerging questions in AI ethics: 1. Can agency contribute to the moral status of non-human beings, and how? 2. When and why might AI systems exhibit power-seeking behaviour and does this pose an existential risk to humanity?

According to an orthodox view, the capacity for conscious experience (sentience) is relevant to the distribution of moral status and value. However, physicalism about consciousness might threaten the normative relevance of sentience. According to the indeterminacy argument, sentience is metaphysically indeterminate while indeterminacy of sentience is incompatible with its normative relevance. According to the introspective argument (by François Kammerer), the unreliability of our conscious introspection undercuts the justification for belief in the normative relevance of consciousness. I defend the normative relevance of sentience against these objections. First, I demonstrate that physicalists only have to concede a limited amount of indeterminacy of sentience. This moderate indeterminacy is in harmony with the role of sentience in determining moral status. Second, I argue that physicalism gives us no reason to expect that introspection is unreliable with respect to the normative relevance of consciousness.