Eliot Du Sordet

It is increasingly claimed that large language models (LLMs) possess internal representations that are grounded in the world, a view often defended within a teleosemantic framework. I examine the conditions that this framework imposes and argue that the selection history of pre-trained and fine-tuned LLMs does not fix the world as their representational target, whereas that of agentic LLMs can. I proceed in three stages. First, I contend that in purely pre-trained LLMs, activation vectors and features represent the statistical distribution of the training corpus. I then show that fine-tuning shifts the representational target not to the world but to the statistics of curated ground truths (Reinforcement learning through verifiable rewards) or to the content of human beliefs (preference tuning). Finally, I turn to agentic LLMs. Unlike the other systems considered here, they act, and their actions can have real-world consequences. These consequences can incorporate feedback from the world itself into the selection of their representations by two separate routes: parameter updates, which can ground features and the activation vectors built from them, and context updates, through external memory or within a single episode, which can ground activation vectors only. Either route establishes correctness conditions that are independent of human beliefs. Agentic LLMs can therefore possess internal representations grounded in the world, but only for the domains, and the aspects of those domains, where the consequences of their own actions have been used to update their parameters or their context.

I contend that, when making claims about LLMs’ world models, we should ask whether a model represents not only the surface properties of an input but also its underlying causes and principles. I then support, drawing on empirical studies in mechanistic interpretability, that features and circuits are strong candidates for realizing both kinds of representation. However, I argue that the pragmatics of human linguistic production can hinder both forms of representation in text-trained LLMs, although chain-of-thought prompting and reinforcement learning from verifiable rewards can substantially mitigate these limitations. Finally, drawing on recent mechanistic interpretability results, I show that just as correct outputs are insufficient evidence for the presence of the relevant internal representations, wrong outputs are correspondingly poor evidence for their absence.