Ronald A. Rensink

Four experiments investigated the extent to which abstract quantitative information can be conveyed by basic visual features. This was done by asking observers to estimate and discriminate Pearson correlation in graphical representations where the first data dimension of each element was encoded by its horizontal position, and the second by the value of one of its visual features; perceiving correlation then requires combining the information in the two encodings via a common abstract representation. Four visual features were examined: luminance, color, orientation, and size. All were able to support the perception of correlation. Indeed, despite the strikingly different appearances of the associated stimuli, all gave rise to performance that was much the same: just noticeable difference was a linear function of distance from complete correlation, and estimated correlation a logarithmic function of this distance. Performance differed only in regards to the level of noise in the feature, with these values compatible with estimates of channel capacity encountered in classic experiments on absolute perceptual magnitudes. These results suggest that quantitative information can be conveyed by visual features that are abstracted at relatively low levels of visual processing, with little representation of the original sensory property. It is proposed that this is achieved via an abstract parameter space in which the values in each perceptual dimension are normalized to have the same means and variances, with perceived correlation based on the shape of the joint probability density function of the resultant elements.

Research over the past half century has produced extensive evidence that observers cannot report or retain all of the details of their visual world from one moment to the next. During the past decade, a new set of studies has illustrated just how pervasive these limits are. For example, early evidence for the failure to detect changes to simple dot patterns (Phillips, 1974) and arrays of letters (Pashler, 1988) generalizes to more naturalistic displays such as photographs and motion pictures (e.g., Levin & Simons, 1997; Rensink, O’Regan, & Clark, 1997; see Rensink, 2002 for a recent review). This failure to report changes (change blindness) can be induced in both simple and naturalistic displays, whenever the change co-occurs with a visual disruption such as an eye movement (Currie, McConkie, Carlson-Radvansky, & Irwin, 1995; Hollingworth, Schrock, & Henderson, 2001), image shift (Blackmore, Brelstaff, Nelson, & Troscianko, 1995), flashed blank screen (Rensink et al., 1997), blink (O’Regan, Deubel, Clark, & Rensink, 2000), transient (O’Regan, Rensink, & Clark, 1999; Rensink, O’Regan, & Clark, 2000), movie cut or pan (Levin & Simons, 1997; Simons, 1996), or an occlusion event (Levin, Simons, Angelone, & Chabris, 2002; Simons & Levin, 1998). Change blindness can also occur in the absence of a disruption, provided the change occurs gradually enough that it does not attract attention (Simons, Franconeri, & Reimer, 2000). Failures of visual awareness have been induced in a number of ways. For example, observers often fail to report a visible but unexpected stimulus when attention is focused on some other object or event in the display (inattentional blindness). As for change blindness, inattentional blindness occurs for both simple (Mack & Rock, 1998; Most et al., 2001) and naturalistic stimuli (Simons & Chabris, 1999). Similarly, repeated instances of an item often go undetected when embedded in a rapid sequence (repetition blindness), a phenomenon that occurs for both words and pictures (Kanwisher, 1987; Kanwisher & Potter, 1990). Observers can also fail to detect a stimulus in a rapid stream of stimuli provided they had to perform an attention-demanding task shortly before the stimulus appeared (Shapiro, Arnell, & Raymond, 1997). The surprisingly large variety of conditions that induce failures of conscious perception reinforces the broad conclusion that we are often unaware of what would otherwise be fully-visible stimuli. Along with an increased appreciation of the limitations of visual awareness has come a renewed interest in using these limitations to explore the nature of the representations and processes underlying our visual experience. Are these limitations due to failures of perception? Of attention? Of memory? What is preserved with and without awareness? The papers in this special issue of the Journal of Vision provide examples of exciting new work on induced failures of visual awareness and the mechanisms that underlie them. For example, several papers examine whether observers are drawn to parts of a display by the stimulus features or whether top-down control or semantic comprehension influence search through a display. Others investigate the nature of and limits on the information that is preserved from one view to the next and how much information needs to be retained for effective perception and action. As guest editors, we hope that this special issue will illustrate how studying the relationship between visual awareness and visual representations can lead to important new insights into the nature of seeing.

A set of visual search experiments tested the proposal that focused attention is needed to detect change. Displays were arrays of rectangles, with the target being the item that continually changed its orientation or contrast polarity. Five aspects of performance were examined: linearity of response, processing time, capacity, selectivity, and memory trace. Detection of change was found to be a self-terminating process requiring a time that increased linearly with the number of items in the display. Capacity for orientation was found to be about 5 items, a value comparable to estimates of attentional capacity. Observers were able to filter out both static and dynamic variations in irrelevant properties. Analysis also indicated a memory for previously-attended locations. These results support the hypothesis that the process needed to detect change is much the same as the attentional process needed to detect complex static patterns. Interestingly, the features of orientation and polarity were found to be handled in somewhat different ways. Taken together, these results not only provide evidence that focused attention is needed to see change, but also show that change detection itself can provide new insights into the nature of attentional processing.

Visual analytics (VA) combines the strengths of human and machine intelligence to enable the discovery of interesting patterns in challenging datasets. Historically, most attention has been given to developing the machine component—for example, machine learning or the human-computer interface. However, it is also essential to develop the abilities of the analysts themselves, especially at the beginning of their careers. For the past several years, we at the University of British Columbia (UBC)—with the support of The Boeing Company—have experimented with various ways of preparing undergraduate students for VA. Although inspired by the need to prepare students to become visual analysts, the result turned out to be fairly general in scope, applicable to other analytical approaches, as well as more general research. In hindsight, this makes considerable sense. Although the visual component of VA is necessary, it is insufficient; many analytical activities at the human end involve nonvisual skills, such as effective decision-making and the ability to quickly focus on the relevant parts of a problem. The result of this experimentation is a third-year undergraduate course titled Cognitive Systems 303 (COGS 303) that focuses on “VA unplugged”—that is, on developing investigative abilities prior to training on the VA systems themselves. It was felt that if students focused on developing basic analytical habits of thought prior to learning VA systems, these habits would be reinforced by subsequent practice on “live” systems.

Advances in neuroscience implicate reentrant signaling as the predominant form of communication between brain areas. This principle was used in a series of masking experiments that defy explanation by feed-forward theories. The masking occurs when a brief display of target plus mask is continued with the mask alone. Two masking processes were found: an early process affected by physical factors such as adapting luminance and a later process affected by attentional factors such as set size. This later process is called masking by object substitution, because it occurs whenever there is a mismatch between the reentrant visual representation and the ongoing lower level activity. Iterative reentrant processing was formalized in a computational model that provides an excellent fit to the data. The model provides a more comprehensive account of all forms of visual masking than do the long-held feed-forward views based on inhibitory contour interactions.

This article discusses several key issues concerning the study of attention and its relation to visual perception, with an emphasis on behavioral and experiential aspects. It begins with an overview of several classical works carried out in the latter half of the 20th century, such as the development of early filter and spotlight models of attention. This is followed by a survey of subsequent research that extended or modified these results in significant ways. It covers current work on various forms of induced blindness and on the capabilities of nonattentional processes. It also includes proposals about how a "just-in-time" allocation of attention can create the impression that we see our surroundings in coherent detail everywhere, as well as how the failure of such allocation can result in various perceptual deficits. The final section examines issues that have received little consideration to date, but may be important for new lines of research in the near future. These include the prospects for a better characterization of attention, the possibility of more systematic explanations, factors that may significantly modulate attentional operation, and the possibility of several kinds of visual attention and visual experience.

East Asians and Westerners differ in various aspects of perception and cognition. For example, visual memory for East Asians is believed to be more influenced by the contextual aspects of a scene than is the case for Westerners (Masuda & Nisbett, 2001). There are also differences in visual search: for Westerners, search for a long line among short is faster than for short among long, whereas this difference does not appear to hold for East Asians (Ueda et al., submitted). However, it is unclear how these group-level differences originate. To investigate the extent to which they depend upon environment, we tested visual search and visual memory in East Asian immigrants who had lived in Canada for different amounts of time. Recent immigrants were found to exhibit no search asymmetry, unlike Westerners who had spent their lives in Canada. However, immigrants who had lived in Canada for more than two years showed performance comparable to that of Westerners. These differences could not be explained by the general analytic/holistic processing distinction believed to differentiate Westerners and East Asians, since all observers showed a strong holistic tendency for visual recognition. Results instead support the suggestion that exposure to a new environment can significantly affect the particular processes used to perceive a given stimulus.

Information management systems improve the retention of information in large collections. As such they act as memory prostheses, implying an ideal basis in human memory models. Since humans process information by association, and situate it in the context of space and time, systems should maximize their effectiveness by mimicking these functions. Since human attentional capacity is limited, systems should scaffold cognitive efforts in a comprehensible manner. We propose the Principles of Mnemonic Associative Knowledge (P-MAK), which describes a framework for semantically identifying, organizing, and retrieving information, and for encoding episodic events by time and stimuli. Inspired by prominent human memory models, we propose associative networks as a preferred representation. Networks are ideal for their parsimony, flexibility, and ease of inspection. Networks also possess topological properties—such as clusters, hubs, and the small world—that aid analysis and navigation in an information space. Our cognitive perspective addresses fundamental problems faced by information management systems, in particular the retrieval of related items and the representation of context. We present evidence from neuroscience and memory research in support of this approach, and discuss the implications of systems design within the constraints of P-MAK’s principles, using text documents as an illustrative semantic domain

As observers, we generally have a strong impression of seeing everything in front of us at any moment. But compelling as it is, this impression is false – there are severe limits to what we can consciously experience in everyday life. Much of the evidence for this claim has come from two phenomena: change blindness (CB) and inattentional blindness (IB). CB refers to the failure of an observer to visually experience changes that are easily seen once noticed. This can happen even if the changes are large, constantly repeat, and the observer has been informed that they will occur. A related phenomenon is IB – the failure to visually experience an object or event when attention is directed elsewhere. For example, observers may fail to notice an unexpected object that enters their visual field, even if this object is large, appears for several seconds, and has important consequences for the selection of action. Both phenomena involve a striking failure to report an object or event that is easily seen once noticed. As such, both are highly counterintuitive, not only in the subjective sense that observers have difficulty believing they could fail so badly at seeing but also in the objective sense that these findings challenge many existing ideas about how we see. But as counterintuitive as these phenomena are, progress has been made in understanding them. Indeed, doing so has allowed us to better understand the limitations of human perception in everyday life and to gain new insights into how our visual systems create the picture of the world that we experience each moment our eyes are open.

It is suggested that the relationship between visual attention and conscious visual experience can be simplified by distinguishing different aspects of both visual attention and visual experience. A set of principles is first proposed for any possible taxonomy of the processes involved in visual attention. A particular taxonomy is then put forward that describes five such processes, each with a distinct function and characteristic mode of operation. Based on these, three separate kinds—or possibly grades—of conscious visual experience can be distinguished, each associated with a particular combination of attentional processes.

We show that early vision can use monocular cues to rapidly complete partially-occluded objects. Visual search for easily detected fragments becomes difficult when the completed shape is similar to others in the display; conversely, search for fragments that are difficult to detect becomes easy when the completed shape is distinctive. Results indicate that completion occurs via the occlusion-triggered removal of occlusion edges and linking of associated regions. We fail to find evidence for a visible filling-in of contours or surfaces, but do find evidence for a "functional" filling-in that prevents the constituent fragments from being rapidly accessed. As such, it is only the completed structures—and not the fragments themselves—that serve as the basis for rapid recognition.

Observers inspected normal, high quality color displays of everyday visual scenes while their eye movements were recorded. A large display change occurred each time an eye blink occurred. Display changes could either involve "Central Interest" or "Marginal Interest" locations, as determined from descriptions obtained from independent judges in a prior pilot experiment. Visual salience, as determined by luminance, color, and position of the Central and Marginal interest changes were equalized. The results obtained were very similar to those obtained in prior experiments showing failure to detect changes occurring simultaneously with saccades, flicker, or “mudsplashes” in the visual scene: Many changes were very hard to detect, and Marginal Interest changes were harder to detect than Central Interest changes. Analysis of eye movements showed, as expected, that the probability of detecting a change depended on the eye’s distance from the change location. However a surprising finding was that both for Central and Marginal Interest changes, evenwhen observers were directly fixating the change locations (within 1 degree),more than 40% of the time they still failed to see the changes. It seems that looking at something does not guarantee you “ see” it.

For scatterplots with gaussian distributions of dots, the perception of Pearson correlation r can be described by two simple laws: a linear one for discrimination, and a logarithmic one for perceived magnitude (Rensink & Baldridge, 2010). The underlying perceptual mechanisms, however, remain poorly understood. To cast light on these, four different distributions of datapoints were examined. The first had 100 points with equal variance in both dimensions. Consistent with earlier results, just noticeable difference (JND) was a linear function of the distance away from r = 1, and the magnitude of perceived correlation a logarithmic function of this quantity. In addition, these laws were linked, with the intercept of the JND line being the inverse of the bias in perceived magnitude. Three other conditions were also examined: a dot cloud with 25 points, a horizontal compression of the cloud, and a cloud with a uniform distribution of dots. Performance was found to be similar in all conditions. The generality and form of these laws suggest that what underlies correlation perception is not a geometric structure such as the shape of the dot cloud, but the shape of the probability distribution of the dots, likely inferred via a form of ensemble coding. It is suggested that this reflects the ability of observers to perceive the information entropy in an image, with this quantity used as a proxy for Pearson correlation.

This paper explores the extent to which a scientific framework for visualization might be possible. It presents several potential parts of a framework, illustrated by application to the visualization of correlation in scatterplots. The first is an extended-vision thesis, which posits that a viewer and visualization system can be usefully considered as a single system that perceives structure in a dataset, much like "basic" vision perceives structure in the world. This characterization is then used to suggest approaches to evaluation that take advantage of techniques used in vision science. Next, an optimal-reduction thesis is presented, which posits that an optimal visualization enables the given task to be reduced to the most suitable operations in the extended system. A systematic comparison of alternative designs is then proposed, guided by what is known about perceptual mechanisms. It is shown that these elements can be extended in various ways—some even overlapping with parts of vision science. As such, a science of some kind appears possible for at least some parts of visualization. It would remain distinct from design practice, but could nevertheless assist with the design of visualizations that better engage human perception and cognition.

Our visual experience of the world is one of diverse objects and events, each with particular colors, shapes, and motions. This experience is so coherent, so immediate, and so effortless that it seems to result from a single system that lets us experience everything in our field of view. But however appealing, this belief is mistaken: there are severe limits on what can be visually experienced. For example, in a display for air-traffic control it is important to track all moving items. For a single item, this can be done without problem. Three or four can also be tracked, although some degree of effort may be needed. As the number is increased further, accurate tracking becomes more and more difficult—and eventually, impossible. Performance is evidently affected by a factor within the observer which enables certain kinds of perception to occur, but is limited in some way. This factor is generally referred to as attention. At various times, attention has been associated with clarity of perception, intensity of perception, consciousness, selection, or the allocation of a limited “resource” enabling various operations (see Hatfield, 1998). During the past several decades, considerable progress has been achieved by focusing on the idea of selection (Broadbent, 1982). In particular, attention can be productively viewed as contingently selective processing. This can be embodied in various ways by various processes—there need not be a single quantity identified with all forms of attention, or a single site where it operates (Allport, 1993; Tsotsos, 2011). Although “paying attention” is often considered to be a unitary operation, it may simply refer to the control of one or more selective processes, ideally in a co-ordinated way. While this view has some cost in terms of conceptual simplicity, it can help make sense of a large set of phenomena. This article surveys several of the major issues in our understanding of attention and how it relates to perception. It focuses on vision, since many—if not all—considerations are similar for all sensory modalities, and the level of understanding achieved in this domain is currently the most advanced.

Over the centuries, magicians have developed extensive knowledge about the manipulation of the human mind—knowledge that has been largely ignored by psychology. It has recently been argued that this knowledge could help improve our understanding of human cognition and consciousness. But how might this be done? And how much could it ultimately contribute to the exploration of the human mind? We propose here a framework outlining how knowledge about magic can be used to help us understand the human mind. Various approaches—both old and new—are surveyed, in terms of four different levels. The first focuses on the methods in magic, using these to suggest new approaches to existing issues in psychology. The second focuses on the effects that magic can produce, such as the sense of wonder induced by seeing an apparently impossible event. Third is the consideration of magic tricks—methods and effects together—as phenomena of scientific interest in their own right. Finally, there is the organization of knowledge about magic into an informative whole, including the possibility of a science centered around the experience of wonder.

Traditionally, vision science and information/data visualization have interacted by using knowledge of human vision to help design effective displays. It is argued here, however, that this interaction can also go in the opposite direction: the investigation of successful visualizations can lead to the discovery of interesting new issues and phenomena in visual perception. Various studies are reviewed showing how this has been done for two areas of visualization, namely, graphical representations and interaction, which lend themselves to work on visual processing and the control of visual operations, respectively. The results of these studies have provided new insights into aspects of vision such as grouping, attentional selection and the sequencing of visual operations. More generally yet, such results support the view that the perception of visualizations can be a useful domain for exploring the nature of visual cognition, inspiring new kinds of questions as well as casting new light on the limits to which information can be conveyed visually.

While some studies suggest cultural differences in visual processing, others do not, possibly because the complexity of their tasks draws upon high-level factors that could obscure such effects. To control for this, we examined cultural differences in visual search for geometric figures, a relatively simple task for which the underlying mechanisms are reasonably well known. We replicated earlier results showing that North Americans had a reliable search asymmetry for line length: Search for long among short lines was faster than vice versa. In contrast, Japanese participants showed no asymmetry. This difference did not appear to be affected by stimulus density. Other kinds of stimuli resulted in other patterns of asymmetry differences, suggesting that these are not due to factors such as analytic/holistic processing but are based instead on the target-detection process. In particular, our results indicate that at least some cultural differences reflect different ways of processing early-level features, possibly in response to environmental factors.

Our recent opinion article [1] examined what change blindness can and cannot tell us about visual representations. Among other things, we argued that change blindness can tell us a lot about how visual representations can be used, but little about their extent. We and others found the ‘sparse representations’ view appealing (and still do), and initially made the overly strong claim that change blindness supports the conclusion of sparse representations [2,3]. We wrote our article because change blindness continues to be taken as evidence for sparse – or even absent – representations, and we used O’Regan and Noë’s influential paper [4] as an example. However, as has been noted for some time [5–8], this conclusion is logically flawed: lack of ability need not be caused by lack of representation.

It has often been assumed that when we use vision to become aware of an object or event in our surroundings, this must be accompanied by a corresponding visual experience (i.e., seeing). The studies reported here show that this assumption is incorrect. When observers view a sequence of displays alternating between an image of a scene and the same image changed in some way, they often feel (or sense) the change even though they have no visual experience of it. The subjective difference between sensing and seeing is mirrored in several behavioral differences, suggesting that these are two distinct modes of conscious visual perception.

When looking at a scene, observers feel that they see its entire structure in great detail and can immediately notice any changes in it. However, when brief blank fields are placed between alternating displays of an original and a modified scene, a striking failure of perception is induced: identification of changes becomes extremely difficult, even when changes are large and made repeatedly. Identification is much faster when a verbal cue is provided, showing that poor visibility is not the cause of this difficulty. Identification is also faster for objects mentioned in brief verbal descriptions of the scene. These results support the idea that observers never form a complete, detailed representation of their surroundings. In addition, results also indicate that attention is required to perceive change, and that in the absence of localized motion signals it is guided on the basis of high-level interest.

Several studies (e.g., Becklen & Cervone, 1983; Mack & Rock, 1998; Neisser & Becklen, 1975) have found that observers attending to a particular object or event often fail to report the presence of unexpected items. This has been interpreted as inattentional blindness (IB), a failure to see unattended items (Mack & Rock, 1998). Meanwhile, other studies (e.g., Pashler, 1988; Phillips, 1974; Rensink et al., 1997; Simons, 1996) have found that observers often fail to report the presence of large changes in a display when these changes occur simultaneously with a transient such as an eye movement or flash of the display. This has been interpreted as change blindness (CB), a failure to see unattended changes (Rensink et al., 1997). In both cases there is a striking failure to report an object or event that would be quite visible under other circumstances. And in both cases there is a widespread (although not universal) belief that the underlying cause has to do with the absence of attention. The question then arises as to how these effects might be related. Is CB the same thing as IB? If not, what is the relation between them? And given that these phenomena deal with failures of subjective perception, what can they teach us about the nature of our visual experience? In particular, what can they teach us about the role played by visual attention?

Adelson & Pentland (Chapter 11) use an engaging metaphor to illustrate their position on scene analysis: interpretations are produced by a workshop that employs a set of specialists, each concerned with a single aspect of the scene. The authors argue that it is too expensive to have a supervisor co-ordinate the specialists and that it is too expensive to let them operate independently. They then show that a careful sequencing of the specialists leads to solutions of minimum cost, at least for their world of Mondrian panels.

Change blindness is the striking failure to see large changes that normally would be noticed easily. Over the past decade this phenomenon has greatly contributed to our understanding of attention, perception, and even consciousness. The surprising extent of change blindness explains its broad appeal, but its counterintuitive nature has also engendered confusions about the kinds of inferences that legitimately follow from it. Here we discuss the legitimate and the erroneous inferences that have been drawn, and offer a set of requirements to help separate them. In doing so, we clarify the genuine contributions of change blindness research to our understanding of visual perception and awareness, and provide a glimpse of some ways in which change blindness might shape future research

It is argued here that cognitive science currently neglects an important source of insight into the human mind: the effects created by magicians. Over the centuries, magicians have learned how to perform acts that are perceived as defying the laws of nature, and that induce a strong sense of wonder. This article argues that the time has come to examine the scientific bases behind such phenomena, and to create a science of magic linked to relevant areas of cognitive science. Concrete examples are taken from three areas of magic: the ability to control attention, to distort perception, and to influence choice. It is shown how such knowledge can help develop new tools and indicate new avenues of research into human perception and cognition.