Daniel Herrmann

We consider the questions of whether or not large language models (LLMs) have beliefs, and, if they do, how we might measure them. First, we consider whether or not we should expect LLMs to have something like beliefs in the first place. We consider some recent arguments aiming to show that LLMs cannot have beliefs. We show that these arguments are misguided. We provide a more productive framing of questions surrounding the status of beliefs in LLMs, and highlight the empirical nature of the problem. With this lesson in hand, we evaluate two existing approaches for measuring the beliefs of LLMs, one due to Azaria and Mitchell (The internal state of an llm knows when its lying, 2023) and the other to Burns et al. (Discovering latent knowledge in language models without supervision, 2022). Moving from the armchair to the desk chair, we provide empirical results that show that these methods fail to generalize in very basic ways. We then argue that, even if LLMs have beliefs, these methods are unlikely to be successful for conceptual reasons. Thus, there is still no lie-detector for LLMs. We conclude by suggesting some concrete paths for future work.

Consider two principles of rational decision making. One says that you should never use the value of events that are inconsistent with your evidence to assess what to do. The other says that, if you know what a more informed version of yourself would do, then you should already do likewise with your current information. We show that no decision theory which agrees with evidential decision theory (EDT) and causal decision theory (CDT) whenever they agree can satisfy both principles. In particular, assuming that agents update via Bayesian conditioning, EDT satisfies the former and violates the latter, while CDT violates the former and satisfies the latter. We show that the causalist can satisfy the first (at the cost of then violating the second) if she either uses a learning rule that changes her beliefs about counterfactuals in a particular way or uses a decision rule more in line with a rule proposed by Hitchcock (2016). We then employ these results to illuminate a core tension at the heart of the debate between evidentialists and causalists.

An important feature of many conventions is that the agents use an asymmetry to coordinate their behaviour. We call these ‘correlated conventions’. However, a puzzle arises: since any asymmetry works as well as any other, what are the relevant asymmetries on which a given population founds its correlated conventions? In order to gain traction on this question we need an account of both the invention and evolution of correlated conventions. Invention has remained largely unexplored in the literature. In this article we provide a simple model of the origin and subsequent dynamics of correlated conventions. This model can serve as a base for future investigation.

As large language models (LLMs) continue to demonstrate remarkable abilities across various domains, computer scientists are developing methods to understand their cognitive processes, particularly concerning how (and if) LLMs internally represent their beliefs about the world. However, this field currently lacks a unified theoretical foundation to underpin the study of belief in LLMs. This article begins filling this gap by proposing adequacy conditions for a representation in an LLM to count as belief-like. We argue that, while the project of belief measurement in LLMs shares striking features with belief measurement as carried out in decision theory and formal epistemology, it also differs in ways that should change how we measure belief. Thus, drawing from insights in philosophy and contemporary practices of machine learning, we establish four criteria that balance theoretical considerations with practical constraints. Our proposed criteria include accuracy, coherence, uniformity, and use, which together help lay the groundwork for a comprehensive understanding of belief representation in LLMs. We draw on empirical work showing the limitations of using various criteria in isolation to identify belief representations.

We consider the questions of whether or not large language models (LLMs) have beliefs, and, if they do, how we might measure them. First, we consider whether or not we should expect LLMs to have something like beliefs in the first place. We consider some recent arguments aiming to show that LLMs cannot have beliefs. We show that these arguments are misguided. We provide a more productive framing of questions surrounding the status of beliefs in LLMs, and highlight the empirical nature of the problem. With this lesson in hand, we evaluate two existing approaches for measuring the beliefs of LLMs, one due to Azaria and Mitchell (The internal state of an llm knows when its lying, 2023) and the other to Burns et al. (Discovering latent knowledge in language models without supervision, 2022). Moving from the armchair to the desk chair, we provide empirical results that show that these methods fail to generalize in very basic ways. We then argue that, even if LLMs have beliefs, these methods are unlikely to be successful for conceptual reasons. Thus, there is still no lie-detector for LLMs. We conclude by suggesting some concrete paths for future work.

Grice, Lewis, and Skyrms proposed similar distinctions between kinds of meaning. The meaning of terms in human language, as Lewis and Skyrms had it, is ‘conventional’. Skyrms presented models showing how it is possible for conventional meaning to evolve in a population without reliance on pre-existing meaning. But one might think of conventionality as coming in degrees, based on whether the evolutionary process begins with ‘natural saliences’. We propose a theory of natural salience and several extensions of Skyrms’s models to capture this notion. These models reveal that natural saliences can hinder, as well as help, the evolution of language.

We often need to have beliefs about things on which we are not experts. Luckily, we often have access to expert judgements on such topics. But how should we form our beliefs on the basis of expert opinion when experts conflict in their judgments? This is the core of the novice/2-expert problem in social epistemology. A closely related question is important in the context of policy making: how should a policy maker use expert judgments when making policy in domains in which she is not herself an expert? This question is more complex, given the messy and strategic nature of politics. In this paper we argue that the prediction with expert advice framework from machine learning provides helpful tools for addressing these problems. We outline conditions under which we should expert PWEA to be helpful and those under which we should not expect these methods to perform well.

Signalling games are useful for understanding how language emerges. In the standard models, the dynamics in some sense already know what the signals are, even if they do not yet have meaning. In this article, we relax this assumption and develop a simple model we call an ‘attention game’, in which agents have to learn which feature of their environment is the signal. We demonstrate that simple reinforcement learning agents can still learn to coordinate in contexts where the agents do not already know what the signal is, and the other features in the agents’ environment are uncorrelated with the signal. Furthermore, we show that in cases where other features are correlated with the signal, there is a surprising trade-off between learning what the signal is and success in action. We show that the mutual information between a signal and a feature plays a key role in governing the accuracy and attention of the agent.

Computer scientists have provided a distinct justification of Occam’s Razor. Using the probably approximately correct framework, they provide a theorem that they claim demonstrates that we should favor simpler hypotheses. The argument relies on a philosophical interpretation of the theorem. I argue that the standard interpretation of the result in the literature is misguided and that a better reading does not, in fact, support Occam’s Razor at all. To this end, I state and prove a very similar theorem that, if interpreted the same way, would justify the contradictory Anti-Occam’s Razor—the principle that we should favor more complex hypotheses.