Eliot Du Sordet

Large language models (LLMs) are increasingly attributed with internal representations of the world, a claim often grounded in the teleosemantic account of representation. I examine the core properties this account requires and argue that pre-trained and fine-tuned LLMs face significant challenges in satisfying the selection-history criterion, whereas agentic LLMs may not. I proceed in three stages. First, I argue that in purely pre-trained LLMs, activation vectors and features represent the statistical distribution of the training corpus rather than the world. Second, I argue that fine-tuning does not relocate the representational target to the world, but to the content of human beliefs about the world. Third, I turn to agentic LLMs. Because such systems can act, and because their actions can have real-world consequences, weight updates that respond to those consequences route feedback from the world itself into the training loop, fixing correctness conditions that are not mediated by human belief. This grounds genuinely world-tracking representations, though only for those domains of the world in which the consequences of the agent’s actions have been used to update its parameters.

I contend that, when making claims about LLMs’ world models, we should ask whether a model represents not only the surface properties of an input but also its underlying causes and principles. I then support, drawing on empirical studies in mechanistic interpretability, that features and circuits are strong candidates for realizing both kinds of representation. However, I argue that the pragmatics of human linguistic production can hinder both forms of representation in text-trained LLMs, although chain-of-thought prompting and reinforcement learning from verifiable rewards can substantially mitigate these limitations. Finally, drawing on recent mechanistic interpretability results, I show that just as correct outputs are insufficient evidence for the presence of the relevant internal representations, wrong outputs are correspondingly poor evidence for their absence.