Kevin J. Lande

The aim of this chapter is to provide a primer on structured mental representations and their place in philosophical and scientific theorizing. We discuss four questions: 1. What does it mean to say that a psychological representation is structured? 2. Why does a representation’s structure matter? 3. What are examples of possible representational structures? 4. How can such representational structures be discovered empirically? We encourage several pluralist perspectives concerning structured mental representation. The first is pluralism about which mental capacities make use of structured representations. Structured representations are not exclusive to evolutionarily and cognitively advanced faculties of language, thought, and reasoning; they underwrite a wide variety of mental capacities. The second perspective is pluralism about the neural underpinnings of representational structures. There are many ways a structured representation might be realized in populations of cells. The third perspective concerns pluralism about types of representational structure. Structured representations are not restricted to sentences and formulas; they can take a wide variety of forms.

How much of what we see is imagined? Perception is a constructive process, supplementing the information available in sensory inputs to build representations of the world, as when one perceives a cat behind a chain-link fence as a whole, intact object, though it produces only a fragmented image on the eye. A recent movement in cognitive science and the philosophy of mind argues that mental imagery supplies much of the material for constructing perceptual representations—filling in the missing parts of the cat, for instance. On this “Constitutive View” of imagery’s role in perception, perceptual representations routinely contain elements of mental imagery. This view rests on an expansive conception of what mental imagery is: roughly, imagery consists in perceptual processing that is not directly connected to sensory inputs. We challenge the Constitutive View. First, we argue that the expansive conception of imagery on which the view relies is problematic, for it fails to capture a unified psychological kind. We then consider whether phenomena such as perceptual completion, in which the occluded parts of the cat are “filled in,” support the Constitutive View. We argue that there is little explanatory utility in construing perceptual completion as a form of imagery. The constructive character of perception, we suggest, is best understood on its own terms, rather than by reference to imagery.

Perception involves the processing of content or information about the world. In what form is this content represented? I argue that perception is widely compositional. The perceptual system represents many stimulus features (including shape, orientation, and motion) in terms of combinations of other features (such as shape parts, slant and tilt, common and residual motion vectors). But compositionality can take a variety of forms. The ways in which perceptual representations compose are markedly different from the ways in which sentences or thoughts are thought to be composed. I suggest that the thesis that perception is compositional is not itself a concrete hypothesis with specific predictions; rather it affords a productive framework for developing and evaluating specific empirical hypotheses about the form and content of perceptual representations. The question is not just whether perception is compositional, but how. Answering this latter question can provide fundamental insights into perception.

It is commonly assumed that images, whether in the world or in the head, do not have a privileged analysis into constituent parts. They are thought to lack the sort of syntactic structure necessary for representing complex contents and entering into sophisticated patterns of inference. I reject this assumption. “Image grammars” are models in computer vision that articulate systematic principles governing the form and content of images. These models are empirically credible and can be construed as literal grammars for images. Images can have rich syntactic structure, though of a markedly different form than sentences in language.

Mental capacities for perceiving, remembering, thinking, and planning involve the processing of structured mental representations. A compositional semantics of such representations would explain how the content of any given representation is determined by the contents of its constituents and their mode of combination. While many have argued that semantic theories of mental representations would have broad value for understanding the mind, there have been few attempts to develop such theories in a systematic and empirically constrained way. This paper contributes to that end by developing a semantics for a ‘fragment’ of our mental representational system: the visual system’s representations of the bounding contours of objects. At least three distinct kinds of composition are involved in such representations: ‘concatenation’, ‘feature composition’, and ‘contour composition’. I sketch the constraints on and semantics of each of these. This account has three principal payoffs. First, it models a working framework for compositionally ascribing structure and content to perceptual representations, while highlighting core kinds of evidence that bear on such ascriptions. Second, it shows how a compositional semantics of perception can be compatible with holistic, or Gestalt, phenomena, which are often taken to show that the whole percept is ‘other than the sum of its parts’. Finally, the account illuminates the format of a key type of perceptual representation, bringing out the ways in which contour representations exhibit domain-specific form of the sort that is typical of structured icons such as diagrams and maps, in contrast to typical discursive representations of logic and language.

When interpreting the meanings of visual features in information visualizations, observers have expectations about how visual features map onto concepts (inferred mappings.) In this study, we examined whether aspects of inferred mappings that have been previously identified for colormap data visualizations generalize to a different type of visualization, Venn diagrams. Venn diagrams offer an interesting test case because empirical evidence about the nature of inferred mappings for colormaps suggests that established conventions for Venn diagrams are counterintuitive. Venn diagrams represent classes using overlapping circles and express logical relationships between those classes by shading out regions to encode the concept of non-existence, or none. We propose that people do not simply expect shading to signify non-existence, but rather they expect regions that appear as holes to signify non-existence (the hole hypothesis.) The appearance of a hole depends on perceptual properties in the diagram in relation to its background. Across three experiments, results supported the hole hypothesis, underscoring the importance of configural processing for interpreting the meanings of visual features in information visualizations.

Perception is a central means by which we come to represent and be aware of particulars in the world. I argue that an adequate account of perception must distinguish between what one perceives and what one's perceptual experience is of or about. Through capacities for visual completion, one can be visually aware of particular parts of a scene that one nevertheless does not see. Seeing corresponds to a basic, but not exhaustive, way in which one can be visually aware of an item. I discuss how the relation between seeing and visual awareness should be explicated within a representational account of the mind. Visual awareness of an item involves a primitive kind of reference: one is visually aware of an item when one's visual perceptual state succeeds in referring to that particular item and functions to represent it accurately. Seeing, by contrast, requires more than successful visual reference. Seeing depends additionally on meta-semantic facts about how visual reference happens to be fixed. The notions of seeing and of visual reference are both indispensable to an account of perception, but they are to be characterized at different levels of representational explanation.

A compositional theory of perceptual representations would explain how the accuracy conditions of a given type of perceptual state depend on the contents of constituent perceptual representations and the way those constituents are structurally related. Such a theory would offer a basic framework for understanding the nature, grounds, and epistemic significance of perception. But an adequate semantics of perceptual representations must accommodate the holistic nature of perception. In particular, perception is replete with context effects, in which the way one perceptually represents one aspect of a scene (including the position, size, orientation, shape, color, motion, or even unity of an object) normally depends on how one represents many other aspects of the scene. The ability of existing accounts of the semantics of perception to analyze context effects is at best unclear. Context effects have even been thought to call into question the very feasibility of a systematic semantics of perception. After outlining a compositional semantics for a rudimentary set of percepts, I draw on empirical models from perceptual psychology to show how such a theory must be modified to analyze context effects. Context effects arise from substantive constraints on how perceptual representations can combine and from the different semantic roles that perceptual representations can have. I suggest that context effects are closely tied to the objectivity of perception. They arise from a perceptual grammar that functions to facilitate the composition of reliably accurate representations in an uncertain but structured world.

An ongoing philosophical discussion concerns how various types of mental states fall within broad representational genera—for example, whether perceptual states are “iconic” or “sentential,” “analog” or “digital,” and so on. Here, I examine the grounds for making much more specific claims about how mental states are structured from constituent parts. For example, the state I am in when I perceive the shape of a mountain ridge may have as constituent parts my representations of the shapes of each peak and saddle of the ridge. More specific structural claims of this sort are a guide to how mental states fall within broader representational kinds. Moreover, these claims have significant implications of their own about semantic, functional, and epistemic features of our mental lives. But what are the conditions on a mental state's having one type of constituent structure rather than another? Drawing on explanatory strategies in vision science, I argue that, other things being equal, the constituent structure of a mental state determines what I call its distributional properties—namely, how mental states of that type can, cannot, or must co‐occur with other mental states in a given system. Distributional properties depend critically on and are informative about the underlying structures of mental states, they abstract in important ways from aspects of how mental states are processed, and they can yield significant insights into the variegation of psychological capacities.

You can perceive things, in many respects, as they really are. For example, you can correctly see a coin as circular from most angles. Nonetheless, your perception of the world is perspectival. The coin looks different when slanted than when head-on, and there is some respect in which the slanted coin looks similar to a head-on ellipse. Many hold that perception is perspectival because you perceive certain properties that correspond to the “looks” of things. I argue that this view is misguided. I consider the two standard versions of this view. What I call the PLURALIST APPROACH fails to give a unified account of the perspectival character of perception, while what I call the PERSPECTIVAL PROPERTIES APPROACH violates central commitments of contemporary psychology. I propose instead that perception is perspectival because of the way perceptual states are structured from their parts.