Zenon Pylyshyn

In _Seeing and Visualizing_, Zenon Pylyshyn argues that seeing is different from thinking and that to see is not, as it may seem intuitively, to create an inner replica of the world. Pylyshyn examines how we see and how we visualize and why the scientific account does not align with the way these processes seem to us "from the inside." In doing so, he addresses issues in vision science, cognitive psychology, philosophy of mind, and cognitive neuroscience. First, Pylyshyn argues that there is a core stage of vision independent from the influence of our prior beliefs and examines how vision can be intelligent and yet essentially knowledge-free. He then proposes that a mechanism within the vision module, called a visual index (or FINST), provides a direct preconceptual connection between parts of visual representations and things in the world, and he presents various experiments that illustrate the operation of this mechanism. He argues that such a deictic reference mechanism is needed to account for many properties of vision, including how mental images attain their apparent spatial character without themselves being laid out in space in our brains. The final section of the book examines the "picture theory" of mental imagery, including recent neuroscience evidence, and asks whether any current evidence speaks to the issue of the format of mental images. This analysis of mental imagery brings together many of the themes raised throughout the book and provides a framework for considering such issues as the distinction between the form and the content of representations, the role of vision in thought, and the relation between behavioral, neuroscientific, and phenomenological evidence regarding mental representations.

In _Things and Places_, Zenon Pylyshyn argues that the process of incrementally constructing perceptual representations, solving the binding problem (determining which properties go together), and, more generally, grounding perceptual representations in experience arise from the nonconceptual capacity to pick out and keep track of a small number of sensory individuals. He proposes a mechanism in early vision that allows us to select a limited number of sensory objects, to reidentify each of them under certain conditions as the same individual seen before, and to keep track of their enduring individuality despite radical changes in their properties -- all without the machinery of concepts, identity, and tenses. This mechanism, which he calls FINSTs (for "Fingers of Instantiation"), is responsible for our capacity to individuate and track several independently moving sensory objects -- an ability that we exercise every waking minute, and one that can be understood as fundamental to the way we see and understand the world and to our sense of space. Pylyshyn examines certain empirical phenomena of early vision in light of the FINST mechanism, including tracking and attentional selection. He argues provocatively that the initial selection of perceptual individuals is our primary nonconceptual contact with the perceptual world (a contact that does not depend on prior encoding of any properties of the thing selected) and then draws upon a wide range of empirical data to support a radical externalist theory of spatial representation that grows out of his indexing theory.

This study investigates a new experimental paradigm called the Modified Traveling Salesman Problem. This task requires subjects to visit once and only once n invisible targets in a 2D display, using a virtual vehicle controlled by the subject. Subjects can only see the directions of the targets from the current location of the vehicle, displayed by a set of oriented segments that can be viewed inside a circular window surrounding the vehicle. Two conditions were compared. In the “allocentric” condition, subjects see the vehicle move across the screen and change orientation under their command. The “egocentric” condition is similar except for how the information is provided: the position and orientation of the vehicle icon remains fixed at the center of the screen and only target directions, as indicated by the oriented segments, change as the subject “moves” the vehicle. The unexpected finding was that this task can be performed, in either condition, for up to 10 targets. We consider two possible strategies that might be used, a location-based strategy and a segment strategy. The location-based strategy relies on spatial memory and attempts to infer the locations of all the targets. The segment strategy is more local and focuses on the directional segments themselves, keeping track of the ones that represent already-visited targets. A number of observations suggest that the segment strategy was used, at least for larger numbers of targets. According to our hypothesis, keeping track of the segments requires one to use indexical reference for associating the segments with their status in the task - given by current status predicates Visited or Not-visited -, perhaps using visual indexes, deictic pointers, or object files.

This paper explores the difference between Connectionist proposals for cognitive a r c h i t e c t u r e a n d t h e s o r t s o f m o d e l s t hat have traditionally been assum e d i n c o g n i t i v e s c i e n c e . W e c l a i m t h a t t h e m a j o r d i s t i n c t i o n i s t h a t , w h i l e b o t h Connectionist and Classical architectures postulate representational mental states, the latter but not the former are committed to a symbol-level of representation, or to a ‘language of thought’: i.e., to representational states that have combinatorial syntactic and semantic structure. Several arguments for combinatorial structure in mental representations are then reviewed. These include arguments based on the ‘systematicity’ of mental representation: i.e., on the fact that cognitive capacities always exhibit certain symmetries, so that the ability to entertain a given thought implies the ability to entertain thoughts with semantically related contents. We claim that such arguments make a powerful case that mind/brain architecture is not Connectionist at the cognitive level. We then consider the possibility that Connectionism may provide an account of the neural (or ‘abstract neurological’) structures in which Classical cognitive architecture is implemented. We survey a n u m b e r o f t h e s t a n d a r d a r g u m e n t s t h a t h a v e b e e n o f f e r e d i n f a v o r o f Connectionism, and conclude that they are coherent only on this interpretation.

Using a novel enumeration task, we examined the encoding of spatial information during subitizing. Observers were shown masked presentations of randomly-placed discs on a screen and were required to mark the perceived locations of these discs on a subsequent blank screen. This provided a measure of recall for object locations and an indirect measure of display numerosity. Observers were tested on three stimulus durations and eight numerosities. Enumeration performance was high for displays containing up to six discs—a higher subitizing range than reported in previous studies. Error in the location data was measured as the distance between corresponding stimulus and response discs. Overall, location errors increased in magnitude with larger numerosities and shorter display durations. When errors were computed as disc distance from display centroid, results suggest a compressed representation by observers. Additionally, enumeration and localization accuracy increased with display regularity.

��In four experiments we address the question whether several visual objects can be selected voluntarily (exogenously) and then tracked in a Multiple Object Tracking paradigm and, if so, whether the selection involves a different process. Experiment 1 showed that items can indeed be selected based on their labels. Experiment 2 showed that to select the complement set to a set that is automatically (exogenously) selected — e.g. to select all objects not flashed — observers require additional time and that given 1080 ms they were able to select and track them as well as those selected automatically. Experiment 3 showed that the additional time needed in the previous experiment cannot be attributed solely to time required to disengage attention from the initially automatic selections. Experiment 4 showed that the added time provides a monotonically greater benefit when there are more targets, suggesting a serial process. These results are discussed in relation to the Visual Index (FINST) theory which assumes that visual indexes are captured by a data-driven process. It is suggested that voluntarily allocated attention can be used to facilitate the automatic attention capture by objects of interest.

This paper describes a programmatic theory of a process in early vision called indexing. The theory hypothesizes that a small number of primitive indexes are available for individuating, tracking and providing direct access to salient visual objects. We discuss some empirical and theoretical arguments in favor of the proposed index as a resource-limited link between an internal visual representation and objects in the visual world. We argue that this link is needed to explain a large range of properties of vision, including the ability to simultaneously track multiple moving objects, to select a subset of visual items to process, as well as such larger issues as how mental images get their apparent metrical properties and why the world appears stable despite constant retinal motion.

Jacques Mehler was notoriously charitable in embracing a diversity of approaches to science and to the use of many different methodologies. One place where his ecumenism brought the two of us into disagreement is when the evidence of brain imaging was cited in support of different psychological doctrines, such as the picture-theory of mental imagery. Jacques remained steadfast in his faith in the ability of neuroscience data (where the main source of evidence has been from clinical neurology and neuro-imaging) to choose among different psychological positions. I personally have seen little reason for this optimism so Jacques and I frequently found ourselves disagreeing on this issue, though I should add that we rarely disagreed on substantive issues on which we both had views. This particular bone of contention, however, kept us busy at parties and during the many commutes between New York and New Jersey, where Jacques was a frequent visitor at the Rutgers Center for Cognitive Science. Now that I am in a position where he is a captive audience it seems an opportune time to raise the question again

For many of the authors in this volume, this is the second attempt to explore what McCarthy and Hayes (1969) first called the “Frame Problem”. Since the first compendium (Pylyshyn, 1987), nicely summarized here by Ronald Loui, there have been several conferences and books on the topic. Their goals range from providing a clarification of the problem by breaking it down into subproblems (and sometimes declaring the hard subproblems to not be the_ real_ Frame Problem), to providing formal “solutions” to certain aspects of the problem. But more often the message has been that the problem is not solvable except in a piecemeal way in special circumstances by some sort of heuristic approximations. It has sometimes also been said that solving the Frame Problem is not only an unachievable goal, but it is also an unnecessary one since_ humans_ do not solve it either; we simply get along as best we can and deal with the problem of planning in ways that, to use Dennett’s phrase, is “good enough for government work”.

After thirty years of the current “imagery debate,” it appears far from resolved, even though there seems to be a growing acceptance that a cortical display cannot be identified directly with the experienced mental image, nor can it account for the experimental findings on imagery, at least not without additional ad hoc assumptions. The commentaries on the target article range from the annoyed to the supportive, with a surprising number of the latter. In this response I attempt to correct some misreadings of the target article and discuss some of the ideas and evidence introduced by the commentators – much of which I found helpful, even though they do not alter my basic thesis. I also further develop the idea that the spatial character of images may come from the way they are connected to our immediate or immediately-recalled environment (by attention or by visual indexes) and towards which we may orient while we are imaging, thus leaving the alleged spatial properties of images outside the head and freeing image-representations from having to be displayed on any surface.

It is generally accepted that there is something special about reasoning by using mental images. The question of how it is special, however, has never been satisfactorily spelled out, despite more than thirty years of research in the post-behaviorist tradition. This article considers some of the general motivation for the assumption that entertaining mental images involves inspecting a picture-like object. It sets out a distinction between phenomena attributable to the nature of mind to what is called the cognitive architecture, and ones that are attributable to tacit knowledge used to simulate what would happen in a visual situation. With this distinction in mind, the paper then considers in detail the widely held assumption that in some important sense images are spatially displayed or are depictive, and that examining images uses the same mechanisms that are deployed in visual perception. I argue that the assumption of the spatial or depictive nature of images is only explanatory if taken literally, as a claim about how images are physically instantiated in the brain, and that the literal view fails for a number of empirical reasons – for example, because of the cognitive penetrability of the phenomena cited in its favor. Similarly, while it is arguably the case that imagery and vision involve some of the same mechanisms, this tells us very little about the nature of mental imagery and does not support claims about the pictorial nature of mental images. Finally, I consider whether recent neuroscience evidence clarifies the debate over the nature of mental images. I claim that when such questions as whether images are depictive or spatial are formulated more clearly, the evidence does not provide support for the picture-theory over a symbol-structure theory of mental imagery. Even if all the empirical claims were true, they do not warrant the conclusion that many people have drawn from them: that mental images are depictive or are displayed in some (possibly cortical) space. Such a conclusion is incompatible with what is known about how images function in thought. We are then left with the provisional counterintuitive conclusion that the available evidence does not support rejection of what I call the “null hypothesis”; namely, that reasoning with mental images involves the same form of representation and the same processes as that of reasoning in general, except that the content or subject matter of thoughts experienced as images includes information about how things would look

Assuming that the vehicle of imaginal thought is a spatial model may not be quite as egregious an error as assuming it is a two-dimensional picture, but it represents no less a reification error. Because the model is not a literal physical layout, one is still owed an explanation of why spatial properties hold in the model – whether because of architectural constraints or by stipulation. The difference is like the difference between explaining behavior from a principle and predicting it by looking it up in a list. In the latter case no purpose is being served by calling it a mental model.