Keith Stenning

We discuss external and internal graphical and linguistic representational systems. We argue that a cognitive theory of peoples' reasoning performance must account for (a) the logical equivalence of inferences expressed in graphical and linguistic form, and (b) the implementational differences that affect facility of inference. Our theory proposes that graphical representation limit abstraction and thereby aid “processibility”. We discuss the ideas of specificity and abstraction, and their cognitive relevance. Empirical support both comes from tasks which involve the manipulation of external graphics and tasks that do not. For the former, we take Euler's (1772) circles, provide a novel computational reconstruction, show how it captures abstractions, and contrast it with earlier construals and with Johnson‐Laird's (1983) mental models representations. We demonstrate equivalence of the graphical Euler system, and the nongraphical mental models system. For tasks not involving manipulation of external graphics, we discuss text comprehension, and the mental performance of syllogisms. By positing an internal system with the same specificity as Euler's circles, we cover the mental models data, and generate new empirical predictions. Finally, we consider how the architecture of working memory explains why such specific representations are relatively easy to store.

Our long term goal is an understanding of human communication in terms which would provide the basis for rational design. The kernel would be a theory of the cognitive consequences of allocating the same information to different media and modalities, based on the user's information processing characterised in computational terms. Our theory of the cognitive consequences of media/modality allocation starts from an analysis of differences in logical expressiveness of graphical and linguistic representations (Stenning \& Oberlander (1994, 1995)). This semantic approach requires conceptualisations of {\it medium} and {\it modality} that can be related to representation systems. We propose that media are the physical/perceptual aspects of representations; modalities are classes of interpretation function which map media onto meanings. These interpretations of the terms contrast with existing HCI usage. The curtailments of abstraction observed in graphics arise from interactions between medium and modality. Graphical modalities are distinguished from sentential languages by the nature of their interpretation functions. A hierarchy of expressiveness of interpretations of graphics is defined, and compared with interpretations of sentential languages. Using the expressiveness of representations to predict their cognitive properties also requires reference to the availability of constraints of their interpretation to users. Further contrasts between graphics and language emerge in the availability of their constraints. Three example domains of graphical representations are analysed from this perspective---matrix graphics; logic diagrams; and semantic networks. Some empirical evidence of the usability of these notations is reviewed as evidence that the proposed conceptualisation offers powerful generalisations.

Theories of diagrams and diagrammatic reasoning typically seek to account for either the formal semantics of diagrams, or for the advantages which diagrammatic representations hold for the reasoner over other forms of representation. Regrettably, almost no theory exists which accounts for both of these issues together, nor how they affect one another. We do not attempt to provide such an account here. We do, however, seek to lay out larger context than is generally used for examining the processes of using diagrams in reasoning or communication. A context in which detailed studies of sub-problems, such as the formal semantics or cognitive impact of specific diagrammatic systems, may be embedded.Accounts of the embedding of sentential logics in the computational processes of reasoners and communicators are relatively well developed from several decades of research in AI. Analogies between the sentential and the graphical cases are quite revealing about both similarities and differences. To provide a structure for the 'grand context' of diagrammatic representation and reasoning, and to clarify the relations between its component problems, we examine carefully these analogies and the decomposition they provide of subproblems for analysing diagrammatic reasoning.