Anders Søgaard

Mechanistic interpretability (MI) aims to explain how neural networks work by uncovering their underlying mechanisms. As the field grows in influence, it is increasingly important to examine not just models themselves, but the assumptions, concepts and explanatory strategies implicit in MI research. We argue that mechanistic interpretability needs philosophy as an ongoing partner in clarifying its concepts, refining its methods, and navigating the epistemic and ethical complexities of interpreting AI systems. There is significant unrealised potential for progress in MI to be gained through deeper engagement with philosophers and philosophical frameworks. Taking three open problems from the MI literature as examples, this paper illustrates the value philosophy can add to MI research, and outlines a path toward deeper interdisciplinary dialogue.

LLMs have been shown to induce representations akin to real-world information structures. According to the standard account, this is because LLMs model the human mind, and minds model the world. Thus, LLMs and minds share properties that ultimately converge upon similar structures. We contribute to this discourse in two ways: a) We argue for an alternative account: LLMs model minds, and minds model worlds, but minds also model LLMs, which in turn extend the capacities of minds. In sum, the world-mind-LLM system constitutes a three-player game, which b) we show does not necessarily converge. We explore the epistemological consequences of non-convergence for world-mind-model systems, introducing the concept of epistemic drift. We offer a new explanatory account of epistemic drift as a social phenomenon, and explore the potential safety concerns of epistemic drift with the introduction of LLMs. Finally, we contribute to existing discourse surrounding LLM agency, arguing that agency is not found in the content of LLM output, but in the way they functionally interact with minds.

In this paper, we argue that current AI (alignment) research operates on a spectrum between two different underlying conceptions of intelligence: Intelligence Realism, which holds that intelligence represents a single, universal capacity measurable across all systems, and Intelligence Pluralism, which views intelligence as diverse, context-dependent capacities that cannot be reduced to a single universal measure. Through an analysis of current debates in AI research, we demonstrate how the conceptions remain largely implicit yet fundamentally shape how empirical evidence gets interpreted across a wide range of areas. More significantly, the underlying views generate fundamentally different research strands across three areas. Methodologically, they produce different approaches to model selection, benchmark design, and experimental validation. Interpretively, they lead to contradictory readings of scaling laws and system limitations. Regarding AI risk, they generate categorically different assessments of risk and alignment approaches: the ones viewing superintelligence as the biggest risk and searching for unified alignment solutions, the others seeing different threats in many different domains and searching for context-specific solutions. We argue that making explicit these underlying assumptions can contribute to a clearer understanding of the disagreements in this research space and, potentially, a more context-sensitive approach to alignment research.

Every explanation faces a trade-off between informativeness and compression (Kinney and Lombrozo, 2022). On the one hand, we want to aim for as much detailed and correct information as possible, informativeness, on the other hand, we want to ensure that a human can process and comprehend the explanation, compression. Current methods in eXplainable AI (XAI) try to satisfy this trade-off statically, outputting one fixed, non-adjustable explanation that sits somewhere on the spectrum between informativeness and compression. However, some current XAI methods fail to meet the expectations of users and developers such that several failures have been reported in the literature which often come with user-specific knowledge gaps and good-enough understanding. In this work, we propose Dynamic XAI to navigate the trade-off interactively. We argue how this simple idea can help overcome the trade-off by eliminating gaps in user-specific understanding and preventing misunderstandings. We conclude by situating our approach within the broader ethical considerations around XAI.

Postmortem avatars (PMAs) — AI systems that simulate a deceased person by being fine-tuned on data they generated or that was generated about them — have attracted growing scholarly attention, yet their potential role in clinical settings remains largely unexplored. This paper examines the ethics of deploying PMAs as therapeutic tools in grief therapy. Drawing on the dual-process model of grief, the theory of continuing bonds, and the philosophical framework of fictionalism, we propose two potential therapeutic applications: incorporating PMAs into established imaginal exercises such as the empty chair exercise, and treating the process of PMA creation as an art-therapeutic exercise in its own right. We consider five ethical objections to these applications and argue that none constitute knock-down arguments against therapeutic use, particularly given the risk-mitigating role of the clinical context. We conclude by identifying outstanding governance challenges and calling for empirical research, without which neither the promise nor the dangers of therapeutic PMAs can be adequately assessed.

Knowledge claims are abundant in the literature on large language models (LLMs); but can we say that GPT-4 truly "knows" the Earth is round? To address this question, we review standard definitions of knowledge in epistemology and we formalize interpretations applicable to LLMs. In doing so, we identify inconsistencies and gaps in how current NLP research conceptualizes knowledge with respect to epistemological frameworks. Additionally, we conduct a survey of 100 professional philosophers and computer scientists to compare their preferences in knowledge definitions and their views on whether LLMs can really be said to know. Finally, we suggest evaluation protocols for testing knowledge in accordance to the most relevant definitions.

Federated learning (FL) is a machine learning (ML) approach that allows multiple devices or institutions to collaboratively train an ML model without sharing their local data with a third-party. It has recently received significant attention as a promising way to overcome longstanding ethical obstacles to training medical ML models with patient health data. This paper examines the promise of FL in healthcare from an ethical perspective. It argues that medical FL generates a new variety of opacity – federation opacity, wherein stakeholders cannot access, analyze, or curate the data on which a model has been trained – which (a) presents distinctive ethical challenges concerning institutional fairness and accountability in medical ML; and (b) makes FL models especially vulnerable to data poisoning attacks. It then identifies several key claims about the expected benefits of FL in healthcare and argues that they may be either exaggerated, misleading, or incomplete – often due to the problem of federation opacity.

Should developers be held responsible for the predictions of their neural networks—and if not, does that introduce a responsibility gap? The claim that neural networks introduce a responsibility gap has seen significant pushback, with philosophers arguing that the gap can be bridged, or did not exist in the first place. We show how the responsibility gap turns on whether we can distinguish between foreseeable and unforeseeable neural network predictions. Empirical facts about neural networks tell us we cannot, which seems to force developers to either assume full responsibility or no responsibility at all, introducing a responsibility gap—unless, of course, the same empirical facts hold true of humans, in which case there is no gap, but the trouble is simply with the classical notion of responsibility. We revisit and revise Mele’s Zygote, as well as the famous Palsgraf case, and argue that in fact, what complicates responsibility assignment for neural networks also complicates responsibility assignment for humans, and humans seem to confront us with the same all-or-nothing dilemma. Thus, we agree there is no technology-induced responsibility gap (there was no gap in the first place), but for slightly different reasons than our predecessors.

Developers of artificial intelligence (AI) often cannot explain the inferences their neural networks make, at least not in ways that satisfy user needs. XAI—explainable artificial intelligence—aims to develop techniques for providing such explanations. XAI researchers have adopted techniques that, to philosophers, seem representationalist–internalist, leading some philosophers to call for more externalist alternatives. But is explaining AI models through causally related external factors feasible? I suggest we compare the idea of externalist XAI to so‐called functionalist XAI. Two common arguments against functionalist XAI are the Epistemic Argument and the Trust Argument, respectively. I discuss whether these arguments can be met by externalists, as well as a new, previously unnoticed challenge from random initialization.

Federated learning (FL) is a machine learning approach that allows multiple devices or institutions to collaboratively train a model without sharing their local data with a third-party. FL is considered a promising way to address patient privacy concerns in medical artificial intelligence. The ethical risks of medical FL systems themselves, however, have thus far been underexamined. This paper aims to address this gap. We argue that medical FL presents a new variety of opacity -- federation opacity -- that, in turn, generates a distinctive *double* black box problem in healthcare AI. We highlight several instances in which the anticipated benefits of medical FL may be exaggerated, and conclude by highlighting key challenges that must be overcome to make FL ethically feasible in medicine.

Developers of artificial intelligence (AI) often cannot explain the inferences their neural networks make, at least not in ways that satisfy user needs. XAI—explainable artificial intelligence—aims to develop techniques for providing such explanations. XAI researchers have adopted techniques that, to philosophers, seem representationalist–internalist, leading some philosophers to call for more externalist alternatives. But is explaining AI models through causally related external factors feasible? I suggest we compare the idea of externalist XAI to so‐called functionalist XAI. Two common arguments against functionalist XAI are the Epistemic Argument and the Trust Argument, respectively. I discuss whether these arguments can be met by externalists, as well as a new, previously unnoticed challenge from random initialization.

Scientists increasingly approach the world through machine learning techniques, but philosophers of science often question their epistemic status. Some philosophers have argued that the use of unsupervised clustering algorithms is more justified than the use of supervised classification, because supervised classification is more biased, and because (parametric) simplicity plays a different and more interesting role in unsupervised clustering. I call these arguments the No-Bias Argument and the Simplicity-Truth Argument. I show how both arguments are fallacious and how, on the contrary, the use of supervised classification is at least as justified as the use of unsupervised clustering.

Deep neural networks are said to be opaque, impeding the development of safe and trustworthy artificial intelligence, but where this opacity stems from is less clear. What are the sufficient properties for neural network opacity? Here, I discuss five common properties of deep neural networks and two different kinds of opacity. Which of these properties are sufficient for what type of opacity? I show how each kind of opacity stems from only one of these five properties, and then discuss to what extent the two kinds of opacity can be mitigated by explainability methods.

I identify a class of arguments against multiple realization (MR): BookofSand arguments. The arguments are in their general form successful under reasonably uncontroversial assumptions, but this, on the other hand, turns the table on identity theory: If arguments from MR can always be refuted by BookofSand arguments, is identity theory falsifiable? In the absence of operational demarcation criteria, it is not. I suggest a parameterized formal demarcation principle for brain state/process types and show how it can be used to identify previously unconsidered contenders for evidence for MR, e.g., binary classification, division, and sorting. For these to be actual instances of MR, the corresponding psychological kinds must be verifiably, relevantly similar. I also briefly discuss possible linguistic, behavioral, and experimental demarcation criteria for psychological kinds.

Most, if not all, philosophers agree that computers cannot learn what words refers to from raw text alone. While many attacked Searle’s Chinese Room thought experiment, no one seemed to question this most basic assumption. For how can computers learn something that is not in the data? Emily Bender and Alexander Koller ( 2020 ) recently presented a related thought experiment—the so-called Octopus thought experiment, which replaces the rule-based interlocutor of Searle’s thought experiment with a neural language model. The Octopus thought experiment was awarded a best paper prize and was widely debated in the AI community. Again, however, even its fiercest opponents accepted the premise that what a word refers to cannot be induced in the absence of direct supervision. I will argue that what a word refers to _is_ probably learnable from raw text alone. Here’s why: higher-order concept co-occurrence statistics are stable across languages and across modalities, because language use (universally) reflects the world we live in (which is relatively stable). Such statistics are sufficient to establish what words refer to. My conjecture is supported by a literature survey, a thought experiment, and an actual experiment.

Landgrebe and Smith :2061–2081, 2021) present an unflattering diagnosis of recent advances in what they call language-centric artificial intelligence—perhaps more widely known as natural language processing: The models that are currently employed do not have sufficient expressivity, will not generalize, and are fundamentally unable to induce linguistic semantics, they say. The diagnosis is mainly derived from an analysis of the widely used Transformer architecture. Here I address a number of misunderstandings in their analysis, and present what I take to be a more adequate analysis of the ability of Transformer models to learn natural language semantics. To avoid confusion, I distinguish between inferential and referential semantics. Landgrebe and Smith ’s analysis of the Transformer architecture’s expressivity and generalization concerns inferential semantics. This part of their diagnosis is shown to rely on misunderstandings of technical properties of Transformers. Landgrebe and Smith also claim that referential semantics is unobtainable for Transformer models. In response, I present a non-technical discussion of techniques for grounding Transformer models, giving them referential semantics, even in the absence of supervision. I also present a simple thought experiment to highlight the mechanisms that would lead to referential semantics, and discuss in what sense models that are grounded in this way, can be said to understand language. Finally, I discuss the approach Landgrebe and Smith advocate for, namely manual specification of formal grammars that associate linguistic expressions with logical form.

Head-driven phrase structure grammar (HPSG) is one of the most prominent theories employed in deep parsing of natural language. Many linguistic theories are arguably best formalized in extensions of modal or dynamic logic (Keller, Feature logics, infinitary descriptions and grammar, 1993; Kracht, Linguistics Philos 18:401–458, 1995; Moss and Tiede, In: Blackburn, van Benthem, and Wolther (eds.) Handbook of modal logic, 2006), and HPSG seems to be no exception. Adequate extensions of dynamic logic have not been studied in detail, however; the most important aspect is the reference to sets of substructures. In this paper, an adequate extension is identified, and some important results are established: Satisfiability is highly undecidable, and model checking is shown to be in EXPTIME and PSPACE-hard. A fragment with polynomial time model checking procedures is identified; it is shown to cover considerable fragments of HPSG.