Ronald A. Rensink

It has often been assumed that when we use vision to become aware of an object or event in our surroundings, this must be accompanied by a corresponding visual experience (i.e., seeing). The studies reported here show that this assumption is incorrect. When observers view a sequence of displays alternating between an image of a scene and the same image changed in some way, they often feel (or sense) the change even though they have no visual experience of it. The subjective difference between sensing and seeing is mirrored in several behavioral differences, suggesting that these are two distinct modes of conscious visual perception.

The past few years have seen a resurgence of interest in the scientific study of magic. Despite being only a few years old, this “new wave” has already resulted in a host of interesting studies, often using methods that are both powerful and original. These developments have largely borne out our earlier hopes (Kuhn et al., 2008) that new opportunities were available for scientific studies based on the use of magic. And it would seem that much more can still be done along these lines. But in addition to this, we also suggested that it might be time to consider developing an outright science of magic—a distinct area of study concerned with the experience of wonder that results from encountering an apparently impossible event1. To this end, we proposed a framework as to how this might be achieved (Rensink and Kuhn, 2015). A science can be viewed as a systematic method of investigation involving three sets of issues: (i) the entities considered relevant, (ii) the kinds of questions that can be asked about them, and (iii) the kinds of answers that are legitimate (Kuhn, 1970). In the case of magic, we suggested that this could be done at three different levels, each focusing on a distinct set of issues concerned with the nature of magic itself: (i) the nature of magical experience, (ii) how individual magic tricks create this experience, and (iii) organizing knowledge of the set of known tricks in a more comprehensive way (Rensink and Kuhn, 2015). Our framework also included a base level focused on how the methods of magic could be used as tools to investigate issues in existing fields of study. Lamont (2010) and Lamont et al. (2010) raised a number of concerns about the possibility of such a science, which we have addressed (Rensink and Kuhn, 2015). More recently, Lamont (2015) raised a new objection, arguing that although base-level work (i.e., applications of magic methods) might be useful, there is too little structure in magic tricks for them to be studied in a systematic way at the other levels, ruling out a science of magic. We argue here, however, that although this concern raises some interesting challenges for this science, it does not negate the possibility that it could exist, and could contribute to the study of the mind.

One of the more powerful impressions created by vision is that of a coherent, richly-detailed world where everything is present simultaneously. Indeed, this impression is so compelling that we tend to ascribe these properties not only to the external world, but to our internal representations as well. But results from several recent experiments argue against this latter ascription. For example, changes in images of real-world scenes often go unnoticed when made during a saccade, flicker, blink, or movie cut. This "change blindness" provides strong evidence against the idea that our brains contain a picture-like representation of the scene that is everywhere detailed and coherent

This chapter presents an overview of several recent developments in vision science, and outlines some of their implications for the management of visual attention in graphic displays. These include ways of sending attention to the right item at the right time, techniques to improve attentional efficiency, and possibilities for offloading some of the processing typically done by attention onto nonattentional mechanisms. In addition it is argued that such techniques not only allow more effective use to be made of visual attention, but also open up new possibilities for human-machine interaction.

We show that cast shadows can have a significant influence on the speed of visual search. In particular, we find that search based on the shape of a region is affected when the region is darker than the background and corresponds to a shadow formed by lighting from above. Results support the proposal that an early-level system rapidly identifies regions as shadows and then discounts them, making their shapes more difficult to access. Several constraints used by this system are mapped out, including constraints on the luminance and texture of the shadow region, and on the nature of the item casting the shadow. Among other things, this system is found to distinguish between line elements (items containing only edges) and surface elements (items containing visible surfaces), with only the latter deemed capable of casting a shadow.

We present a rigorous way to evaluate the visual perception of correlation in scatterplots, based on classical psychophysical methods originally developed for simple properties such as brightness. Although scatterplots are graphically complex, the quantity they convey is relatively simple. As such, it may be possible to assess the perception of correlation in a similar way. Scatterplots were each of 5.0  extent, containing 100 points with a bivariate normal distribution. Means were 0.5 of the range of the points, and standard deviations 0.2 of this range. Precision was determined via an adaptive algorithm to find the just noticeable differences (jnds) in correlation, i.e., the difference between two side-by-side scatterplots that could be discriminated 75% of the time. Accuracy was measured by direct estimation, using reference scatterplots with fixed upper and lower values, with a test scatterplot adjusted so that its correlation appeared to be halfway between these. This process was recursively applied to yield several further estimates. Results of the discrimination tests show jnd(r) = k (1/b – r), where r is the Pearson correlation, and parameters 0 < k, b < 1. Integration yields a subjective estimate of correlation g(r) = ln(1 – br) / ln(1– b). The values of b found via discrimination closely match those found via direct estimation. As such, it appears that the perception of correlation in a scatterplot is completely described by two related performance curves, specified by two easily-measured parameters.

Recent developments in vision science have resulted in several major changes in our understanding of human visual perception. For example, attention no longer appears necessary for "visual intelligence"--a large amount of sophisticated processing can be done without it. Scene perception no longer appears to involve static, general-purpose descriptions, but instead may involve dynamic representations whose content depends on the individual and the task. And vision itself no longer appears to be limited to the production of a conscious "picture"--it may also guide processes outside the conscious awareness of the observer.

It is argued here that cognitive science currently neglects an important source of insight into the human mind: the effects created by magicians. Over the centuries, magicians have learned how to perform acts that are perceived as defying the laws of nature, and that induce a strong sense of wonder. This article argues that the time has come to examine the scientific bases behind such phenomena, and to create a science of magic linked to relevant areas of cognitive science. Concrete examples are taken from three areas of magic: the ability to control attention, to distort perception, and to influence choice. It is shown how such knowledge can help develop new tools and indicate new avenues of research into human perception and cognition.

Previous theories of early vision have assumed that visual search is based on simple two-dimensional aspects of an image, such as the orientation of edges and lines. It is shown here that search can also be based on three-dimensional orientation of objects in the corresponding scene, provided that these objects are simple convex blocks. Direct comparison shows that image-based and scene-based orientation are similar in their ability to facilitate search. These findings support the hypothesis that scene-based properties are represented at preattentive levels in early vision.

A modified visual search task demonstrates that humans are very good at resuming a search after it has been momentarily interrupted. This is shown by exceptionally rapid response time to a display that reappears after a brief interruption, even when an entirely different visual display is seen during the interruption and two different visual searches are performed simultaneously. This rapid resumption depends on the stability of the visual scene and is not due to display or response anticipations. These results are consistent with the existence of an iterative hypothesis-testing mechanism that compares information stored in short-term memory (the perceptual hypothesis) with information about the display (the sensory pattern). In this view, rapid resumption occurs because a hypothesis based on a previous glance of the scene can be tested very rapidly in a subsequent glance, given that the initial hypothesis-generation step has already been performed.

Scene Perception is the visual perception of an environment as viewed by an observer at any given time. It includes not only the perception of individual objects, but also such things as their relative locations, and expectations about what other kinds of objects might be encountered. Given that scene perception is so effortless for most observers, it might be thought of as something easy to understand. However, the amount of effort required by a process often bears little relation to its underlying complexity. A closer look shows that scene perception is a highly complex activity, and that any account of it must deal with several difficult issues: What exactly is a scene? What aspects of it do we represent? And what are the processes involved? Finding the answers to these questions has proven to be extraordinarily difficult.

Large changes in a scene often become difficult to notice if made during an eye movement, image flicker, movie cut, or other such disturbance. It is argued here that this _change blindness_ can serve as a useful tool to explore various aspects of vision. This argument centers around the proposal that focused attention is needed for the explicit perception of change. Given this, the study of change perception can provide a useful way to determine the nature of visual attention, and to cast new light on the way that it is?and is not?involved in visual perception. To illustrate the power of this approach, this paper surveys its use in exploring three different aspects of vision. The first concerns the general nature of _seeing_. To explain why change blindness can be easily induced in experiments but apparently not in everyday life, it is proposed that perception involves a _virtual representation_, where object representations do not accumulate, but are formed as needed. An architecture containing both attentional and nonattentional streams is proposed as a way to implement this scheme. The second aspect concerns the ability of observers to detect change even when they have no visual experience of it. This _sensing_ is found to take on at least two forms: detection without visual experience (but still with conscious awareness), and detection without any awareness at all. It is proposed that these are both due to the operation of a nonattentional visual stream. The final aspect considered is the nature of visual attention itself?the mechanisms involved when _scrutinizing_ items. Experiments using controlled stimuli show the existence of various limits on visual search for change. It is shown that these limits provide a powerful means to map out the attentional mechanisms involved.

Experiments are presented showing that visual search for Mueller-Lyer (ML) stimuli is based on complete configurations, rather than component segments. Segments easily detected in isolation were difficult to detect when embedded in a configuration, indicating preemption by low-level groups. This preemption—which caused stimulus components to become inaccessible to rapid search—was an all-or-nothing effect, and so could serve as a powerful test of grouping. It is shown that these effects are unlikely to be due to blurring by simple spatial filters at early visual levels. It is proposed instead that they are due to more sophisticated processes that rapidly bind contour fragments into spatially-extended assemblies. These results support the view that rapid visual search cannot access the primitives formed at the earliest stages of visual processing; rather, it can access only higher-level, more ecologically-relevant structures.

When brief blank fields are placed between alternating displays of an original and a modified scene, a striking failure of perception is induced: the changes become extremely difficult to notice, even when they are large, presented repeatedly, and the observer expects them to occur (Rensink, O'Regan, & Clark, 1997). To determine the mechanisms behind this induced "change blindness", four experiments examine its dependence on initial preview and on the nature of the interruptions used. Results support the proposal that representations at the early stages of visual processing are highly volatile, and that focused attention is needed to stabilize them sufficiently to support the perception of change.

One of the more compelling beliefs about vision is that it is based on representations that are coherent and complete, with everything in the visual field described in great detail. However, changes made during a visual disturbance are found to be difficult to see, arguing against the idea that our brains contain a detailed, picture-like representation of the scene. Instead, it is argued here that a more dynamic, "just-in-time" representation is involved, one with deep similarities to the way that users interact with external displays. It is further argued that these similarities can provide a basis for the design of intelligent display systems that can interact with humans in highly effective and novel ways.

The task of visual search is to determine as rapidly as possible whether a target item is present or absent in a display. Rapidly detected items are thought to contain features that correspond to primitive elements in the human visual system. In previous theories, it has been assumed that visual search is based on simple two-dimensional features in the image. However, visual search also has access to another level of representation, one that describes properties in the corresponding three-dimensional scene. Among these properties are three dimensionality and the direction of lighting, but not viewing direction. These findings imply that the parallel processes of early vision are much more sophisticated than previously assumed.

Human vision briefly retains a trace of a stimulus after it disappears. This trace—iconic memory—is often believed to be a surrogate for the original stimulus, a representational structure that can be used as if the original stimulus were still present. To investigate its nature, a flicker-search paradigm was developed that relied upon a full scan (rather than partial report) of its contents. Results show that for visual search it can indeed act as a surrogate, with little cost for alternating between visible and iconic representations. However, the duration over which it can be used depends on the type of task: some tasks can use iconic memory for at least 240 ms, others for only about 190 ms, while others for no more than about 120 ms. The existence of these different limits suggests that iconic memory may have multiple layers, each corresponding to a particular level of the visual hierarchy. In this view, the inability to use a layer of iconic memory may reflect an inability to maintain feedback connections to the corresponding representation.

Purpose. To determine whether highlights are rapidly identified at early levels of vision. Methods. Visual search experiments were carried out using simple black and white figures corresponding to shiny objects lit from various directions. These included, for example, depictions of cylinders with highlights positioned at various heights (see figure). Targets and distractors differed only in the arrangement of their constituent regions, allowing them to be distinguished by the position of the highlights on the corresponding objects. Results. Three observers were tested. Search is rapid when the target corresponds to a shiny black object with a bottom highlight among objects with top highlights (view figure from rightside up). However, search slows down when the target corresponds to an object with a top highlight among objects with bottom highlights (view figure from upside down). This asymmetry in search speeds is destroyed when the contrast of the figures is reversed. Conclusions. These results indicate that regions in the image can be rapidly identified as highlights at early levels of vision. They also suggest--as for several other rapid recovery processes--that an assumption of lighting from above is used as a simplifying constraint.

Knill, Kersten, & Mamassian (Chapter 6) provide an interesting discussion of how the Bayesian formulation can be used to help investigate human vision. In their view, computational theories can be based on an ideal observer that uses Bayesian inference to make optimal use of available information. Four factors are important here: the image information used, the output structures estimated, the priors assumed (i.e., knowledge about the structure of the world), and the likelihood function used (i.e., knowledge about the projection of the world onto the sensors). Knill & Kersten argue that such a framework not only helps analyze a perceptual task, but can also help investigators to define it. Two examples are provided (the interpretation of surface contour and the perception of moving shadows) to show how this approach can be used in practice. As the authors admit, most (if not all) perceptual processes are ill-suited to a "strong" Bayesian approach based on a single consistent model of the world. Instead, they argue for a "weak" variant that assumes Bayesian inference to be carried out in modules of more limited scope. But how weak is "weak"? Are such approaches suitable for only a few relatively low-level tasks, or can they be applied more generally? Could a weak Bayesian approach, for example, explain how we would recognize the return of Elvis Presley? The formal modelling of human perception To help get a fix on things, it is useful to examine the fate of an earlier attempt to formalize human perception: the application of information theory. It was once hoped that this theory—a close cousin of the Bayesian formulation—would provide a way to uncover information-handling laws that were largely independent of physical implementation. In this approach, the human nervous system was assumed to have..

Large changes that occur in clear view of an observer can become difficult to notice if made during an eye movement, blink, or other such disturbance. This change blindness is consistent with the proposal that focused visual attention is necessary to see change, with a change becoming difficult to notice whenever conditions prevent attention from being automatically drawn to it. It is shown here how the phenomenon of change blindness can provide new results on the nature of visual attention, including estimates of its capacity and the extent to which it can bind visual properties into coherent descriptions. It is also shown how the resultant characterization of attention can in turn provide new insights into the role that it plays in the perception of scenes and events.

In the not-too-distant past, vision was often said to involve three levels of processing: a low level concerned with descriptions of the geometric and photometric properties of the image, a high level concerned with abstract knowledge of the physical and semantic properties of the world, and a middle level concerned with anything not handled by the other two. The negative definition of mid-level vision contained in this description reflected a rather large gap in our understanding of visual processing: How could the here-and-now descriptions of the low levels combine with the enduring knowledge of the high levels to produce our perception of the surrounding world? A number of experimental and theoretical efforts have been made over the past few decades to solve this "mid-level crisis". One of the more recent of these is based on the phenomenon of change blindness—the difficulty in seeing a large change in a scene when the transients accompanying that change no longer convey information about its location (Rensink, O'Regan, & Clark, 1997; Rensink, 2000a). Phenomenologically, this effect is quite striking: the change typically is not seen for several seconds, after which it suddenly snaps into awareness. During the time the change remains "invisible", there is an apparent disconnection of the low-level descriptions (which respond to the change) from subjective visual experience (which does not). As such, this effect would seem to have the potential to help us understand how mid-level mechanisms might knit low- and high-level processes into a coherent representation of our surroundings. It is argued here that this potential can indeed be realized, and that change blindness can teach us much about the nature of mid-level vision.3 A number of studies are first reviewed showing that the perception of a scene does not involve a steady buildup of detailed representation: rather, it is a dynamic process, with focused attention playing one of the main roles, viz., forming coherent object representations whenever needed. It is then argued that change blindness can also shed considerable light on the nature of focused attention itself, such as its speed, capacity, selectivity, and ability to bind together visual properties into coherent structures.

Focused attention is needed to perceive change (Rensink et al., 1997; Psychological Science, 8: 368-373) . But how much attentional processing is given to an item? And does this depend on the nature of the task? To answer these questions, "flicker" displays were created, where an original and a modified image continually alternated, with brief blanks between them. Each image was an array of simple figures, half being horizontal and the other half vertical. In half the trials, one of the items changed orientation; in the remainder, all items remained the same. Observers were asked to detect whether a change was occurring in each trial. Within-observer comparisons for different shapes of items yielded two classes of speed: relatively fast search (50 ms/item) for simple lines, and slower search (90 ms/item) for compound figures made of two or more lines. Evidently, more processing was given to compound figures, even though shape was irrelevant to the task. A similar set of experiments asked observers to detect changes in contrast polarity. Here, no differences were found for different shapes, indicating that geometric information is not processed as much when the task involves only nongeometric properties.

Five aspects of visual change detection are reviewed. The first concerns the concept of change itself, in particular the ways it differs from the related notions of motion and difference. The second involves the various methodological approaches that have been developed to study change detection; it is shown that under a variety of conditions observers are often unable to see large changes directly in their field of view. Next, it is argued that this “change blindness” indicates that focused attention is needed to detect change, and that this can help map out the nature of visual attention. The fourth aspect concerns how these results affect our understanding of visual perception—for example, the implication that a sparse, dynamic representation underlies much of our visual experience. Finally, a brief discussion is presented concerning the limits to our current understanding of change detection