Patrick Butlin

The individuation problem for large language models asks which entities associated with them, if any, should be identified as minds. We approach this problem through mechanistic interpretability, engaging in particular with recent empirical work on persona vectors, persona space, and emergent misalignment. We argue that three views are the strongest candidates: the virtual instance view and two new views we introduce, the (virtual) instance-persona view and the model-persona view. First, we argue for the virtual instance view on the grounds that attention streams sustain quasi-psychological connections across token-time. Then we present the persona literature, organised around three hypotheses about the internal structure underlying personas in LLMs, and show that the two persona-based views are promising alternatives.

Higher-order representations are those that are about other representations. Humans and some other animals form higher-order mental representations concerning representations in our own minds, through the operation of processes of metacognition and introspection. These have been linked with a wide range of mental capacities and attributes, including consciousness. Recent research on large language models (LLMs) has explored their knowledge of their own ‘minds’, sometimes suggesting that these models represent their own inner representational states in activations. This paper surveys this research, arguing that there is some evidence of higher-order representation in LLMs but that substantial empirical and philosophical questions remain unresolved.

Rapid progress in artificial intelligence (AI) capabilities has drawn fresh attention to the prospect of consciousness in AI. There is an urgent need for rigorous methods to assess AI systems for consciousness, but significant uncertainty about relevant issues in consciousness science. We present a method for assessing AI systems for consciousness that involves exploring what follows from existing or future neuroscientific theories of consciousness. Indicators derived from such theories can be used to inform credences about whether particular AI systems are conscious. This method allows us to make meaningful progress because some influential theories of consciousness, notably including computational functionalist theories, have implications for AI that can be investigated empirically.

Modern generative AI systems have shown the capacity to produce remarkably fluent language, prompting debates both about their semantic understanding and, less prominently, about whether they can perform speech acts. This paper addresses the latter question, focusing on assertion. We argue that to be capable of assertion, an entity must meet two requirements: it must produce outputs with descriptive functions, and it must be capable of being sanctioned by agents with which it interacts. The second requirement arises from the nature of assertion as a norm-governed social practice. Pre-trained large language models that have not been subject to fine-tuning fail to meet the first requirement. Language models that have been fine-tuned for “groundedness” or “correctness” may meet the first requirement, but fail the second. We also consider the significance of the point that AI systems can be used to generate proxy assertions on behalf of human agents.

In this report, we argue that there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future. That means that the prospect of AI welfare and moral patienthood — of AI systems with their own interests and moral significance — is no longer an issue only for sci-fi or the distant future. It is an issue for the near future, and AI companies and other actors have a responsibility to start taking it seriously. We also recommend three early steps that AI companies and other actors can take: They can (1) acknowledge that AI welfare is an important and difficult issue (and ensure that language model outputs do the same), (2) start assessing AI systems for evidence of consciousness and robust agency, and (3) prepare policies and procedures for treating AI systems with an appropriate level of moral concern. To be clear, our argument in this report is not that AI systems definitely are — or will be — conscious, robustly agentic, or otherwise morally significant. Instead, our argument is that there is substantial uncertainty about these possibilities, and so we need to improve our understanding of AI welfare and our ability to make wise decisions about this issue. Otherwise there is a significant risk that we will mishandle decisions about AI welfare, mistakenly harming AI systems that matter morally and/or mistakenly caring for AI systems that do not.

Language agents are AI systems that combine large language models with other elements to facilitate interaction with an environment. They include LLM-based chatbots but can have a wide range of additional features to support learning, reasoning and decision-making. Goldstein and Kirk-Giannini. Citationm.s. [AI wellbeing] argue that some language agents have beliefs and desires, but it is not obvious that they are agents at all, since they select outputs by querying language models. This paper investigates agency and desires in language agents.

There is an apparent connection between reinforcement learning and agency. Artificial entities controlled by reinforcement learning algorithms are standardly referred to as agents, and the mainstream view in the psychology and neuroscience of agency is that humans and other animals are reinforcement learners. This article examines this connection, focusing on artificial reinforcement learning systems and assuming that there are various forms of agency. Artificial reinforcement learning systems satisfy plausible conditions for minimal agency, and those which use models of the environment to perform forward search are capable of a form of agency which may reasonably be called action for reasons.

As AI systems become increasingly competent language users, it is an apt moment to consider what it would take for machines to understand human languages. This paper considers whether either language models such as GPT-3 or chatbots might be able to understand language, focusing on the question of whether they could possess the relevant concepts. A significant obstacle is that systems of both kinds interact with the world only through text, and thus seem ill-suited to understanding utterances concerning the concrete objects and properties which human language often describes. Language models cannot understand human languages because they perform only linguistic tasks, and therefore cannot represent such objects and properties. However, chatbots may perform tasks concerning the non-linguistic world, so they are better candidates for understanding. Chatbots can also possess the concepts necessary to understand human languages, despite their lack of perceptual contact with the world, due to the language-mediated concept-sharing described by social externalism about mental content.

Modern generative AI systems have shown the capacity to produce remarkably fluent language, prompting debates both about their semantic understanding and, less prominently, about whether they can perform speech acts. This paper addresses the latter question, focusing on assertion. We argue that to be capable of assertion, an entity must meet two requirements: it must produce outputs with descriptive functions, and it must be capable of being sanctioned by agents with which it interacts. The second requirement arises from the nature of assertion as a norm-governed social practice. Pre-trained large language models that have not been subject to fine-tuning fail to meet the first requirement. Language models that have been fine-tuned for ‘groundedness’ or ‘correctness’ may meet the first requirement, but fail the second. We also consider the significance of the point that AI systems can be used to generate proxy assertions on behalf of human agents.

Machine learning researchers distinguish between reinforcement learning and supervised learning and refer to reinforcement learning systems as “agents”. This paper vindicates the claim that systems trained by reinforcement learning are agents while those trained by supervised learning are not. Systems of both kinds satisfy Dretske’s criteria for agency, because they both learn to produce outputs selectively in response to inputs. However, reinforcement learning is sensitive to the instrumental value of outputs, giving rise to systems which exploit the effects of outputs on subsequent inputs to achieve good performance over episodes of interaction with their environments. Supervised learning systems, in contrast, merely learn to produce better outputs in response to individual inputs.

Affective experience in nonhuman animals is of great interest for both theoretical and practical reasons. This paper highlights research by the psychologists Anthony Dickinson and Bernard Balleine which provides particularly good evidence of conscious affective experience in rats. This evidence is compelling because it implicates a sophisticated system for goal-directed action selection, and demonstrates a contrast between apparently conscious and unconscious evaluative representations with similar content. Meanwhile, the evidence provided by some well-known studies on pain in nonhuman animals is much less convincing. This comparison may offer lessons for the future study of animal consciousness.

Representations may have descriptive content, directive content, or both, but little explicit attention has been given to the problem of distinguishing representations of these three kinds. We do not know, for instance, what determines whether a given representation is a directive instructing its consumer to perform some action or has descriptive content to the effect that the action in question has a certain value. This paper considers what it takes for a representation to have directive content. The first part of the paper presents the Liberal View, which might be taken to be the default position on this issue. The Liberal View has some attractions, but as the second part shows, these are less conclusive than they might at first appear, and there is much to be said for an alternative, the Strict View.

One of the central tasks for naturalistic theories of representation is to say what it takes for something to be a representation, and some leading theories have been criticised for being too liberal. Prominent discussions of this problem have proposed a producer-oriented solution; it is argued that representations must be produced by systems employing perceptual constancy mechanisms. However, representations may be produced by simple transducers if they are consumed in the right way. It is characteristic of representations to be consumed by systems which are capable of independent action. This paper defends this claim; discusses more precise, naturalistic formulations; and shows how it can illuminate the explanatory payoffs which science achieves by appealing to representation.

This paper presents an account of the nature of desire, informed by psychology and neuroscience, which entails that hunger is not a desire. The account is contrasted with Schroeder’s well-known empirically-informed theory of desire. It is argued that one significant virtue of the present account, in comparison with Schroeder’s theory, is that it draws a sharp distinction between desires and basic drives, such as the drive for food. One reason to draw this distinction is that experiments on incentive learning show that desires and basic drives influence action in different ways.