Ronald A. Rensink

This paper explores the extent to which a scientific framework for visualization might be possible. It presents several potential parts of a framework, illustrated by application to the visualization of correlation in scatterplots. The first is an extended-vision thesis, which posits that a viewer and visualization system can be usefully considered as a single system that perceives structure in a dataset, much like "basic" vision perceives structure in the world. This characterization is then used to suggest approaches to evaluation that take advantage of techniques used in vision science. Next, an optimal-reduction thesis is presented, which posits that an optimal visualization enables the given task to be reduced to the most suitable operations in the extended system. A systematic comparison of alternative designs is then proposed, guided by what is known about perceptual mechanisms. It is shown that these elements can be extended in various ways—some even overlapping with parts of vision science. As such, a science of some kind appears possible for at least some parts of visualization. It would remain distinct from design practice, but could nevertheless assist with the design of visualizations that better engage human perception and cognition.

Scene Perception is the visual perception of an environment as viewed by an observer at any given time. It includes not only the perception of individual objects, but also such things as their relative locations, and expectations about what other kinds of objects might be encountered. Given that scene perception is so effortless for most observers, it might be thought of as something easy to understand. However, the amount of effort required by a process often bears little relation to its underlying complexity. A closer look shows that scene perception is a highly complex activity, and that any account of it must deal with several difficult issues: What exactly is a scene? What aspects of it do we represent? And what are the processes involved? Finding the answers to these questions has proven to be extraordinarily difficult.

Traditionally, vision science and information/data visualization have interacted by using knowledge of human vision to help design effective displays. It is argued here, however, that this interaction can also go in the opposite direction: the investigation of successful visualizations can lead to the discovery of interesting new issues and phenomena in visual perception. Various studies are reviewed showing how this has been done for two areas of visualization, namely, graphical representations and interaction, which lend themselves to work on visual processing and the control of visual operations, respectively. The results of these studies have provided new insights into aspects of vision such as grouping, attentional selection and the sequencing of visual operations. More generally yet, such results support the view that the perception of visualizations can be a useful domain for exploring the nature of visual cognition, inspiring new kinds of questions as well as casting new light on the limits to which information can be conveyed visually.

This chapter presents an overview of several recent developments in vision science, and outlines some of their implications for the management of visual attention in graphic displays. These include ways of sending attention to the right item at the right time, techniques to improve attentional efficiency, and possibilities for offloading some of the processing typically done by attention onto nonattentional mechanisms. In addition it is argued that such techniques not only allow more effective use to be made of visual attention, but also open up new possibilities for human-machine interaction.

The task of visual search is to determine as rapidly as possible whether a target item is present or absent in a display. Rapidly detected items are thought to contain features that correspond to primitive elements in the human visual system. In previous theories, it has been assumed that visual search is based on simple two-dimensional features in the image. However, visual search also has access to another level of representation, one that describes properties in the corresponding three-dimensional scene. Among these properties are three dimensionality and the direction of lighting, but not viewing direction. These findings imply that the parallel processes of early vision are much more sophisticated than previously assumed.

For scatterplots with gaussian distributions of dots, the perception of Pearson correlation r can be described by two simple laws: a linear one for discrimination, and a logarithmic one for perceived magnitude (Rensink & Baldridge, 2010). The underlying perceptual mechanisms, however, remain poorly understood. To cast light on these, four different distributions of datapoints were examined. The first had 100 points with equal variance in both dimensions. Consistent with earlier results, just noticeable difference (JND) was a linear function of the distance away from r = 1, and the magnitude of perceived correlation a logarithmic function of this quantity. In addition, these laws were linked, with the intercept of the JND line being the inverse of the bias in perceived magnitude. Three other conditions were also examined: a dot cloud with 25 points, a horizontal compression of the cloud, and a cloud with a uniform distribution of dots. Performance was found to be similar in all conditions. The generality and form of these laws suggest that what underlies correlation perception is not a geometric structure such as the shape of the dot cloud, but the shape of the probability distribution of the dots, likely inferred via a form of ensemble coding. It is suggested that this reflects the ability of observers to perceive the information entropy in an image, with this quantity used as a proxy for Pearson correlation.

The past few years have seen a resurgence of interest in the scientific study of magic. Despite being only a few years old, this “new wave” has already resulted in a host of interesting studies, often using methods that are both powerful and original. These developments have largely borne out our earlier hopes (Kuhn et al., 2008) that new opportunities were available for scientific studies based on the use of magic. And it would seem that much more can still be done along these lines. But in addition to this, we also suggested that it might be time to consider developing an outright science of magic—a distinct area of study concerned with the experience of wonder that results from encountering an apparently impossible event1. To this end, we proposed a framework as to how this might be achieved (Rensink and Kuhn, 2015). A science can be viewed as a systematic method of investigation involving three sets of issues: (i) the entities considered relevant, (ii) the kinds of questions that can be asked about them, and (iii) the kinds of answers that are legitimate (Kuhn, 1970). In the case of magic, we suggested that this could be done at three different levels, each focusing on a distinct set of issues concerned with the nature of magic itself: (i) the nature of magical experience, (ii) how individual magic tricks create this experience, and (iii) organizing knowledge of the set of known tricks in a more comprehensive way (Rensink and Kuhn, 2015). Our framework also included a base level focused on how the methods of magic could be used as tools to investigate issues in existing fields of study. Lamont (2010) and Lamont et al. (2010) raised a number of concerns about the possibility of such a science, which we have addressed (Rensink and Kuhn, 2015). More recently, Lamont (2015) raised a new objection, arguing that although base-level work (i.e., applications of magic methods) might be useful, there is too little structure in magic tricks for them to be studied in a systematic way at the other levels, ruling out a science of magic. We argue here, however, that although this concern raises some interesting challenges for this science, it does not negate the possibility that it could exist, and could contribute to the study of the mind.

Over the centuries, magicians have developed extensive knowledge about the manipulation of the human mind—knowledge that has been largely ignored by psychology. It has recently been argued that this knowledge could help improve our understanding of human cognition and consciousness. But how might this be done? And how much could it ultimately contribute to the exploration of the human mind? We propose here a framework outlining how knowledge about magic can be used to help us understand the human mind. Various approaches—both old and new—are surveyed, in terms of four different levels. The first focuses on the methods in magic, using these to suggest new approaches to existing issues in psychology. The second focuses on the effects that magic can produce, such as the sense of wonder induced by seeing an apparently impossible event. Third is the consideration of magic tricks—methods and effects together—as phenomena of scientific interest in their own right. Finally, there is the organization of knowledge about magic into an informative whole, including the possibility of a science centered around the experience of wonder.

It has generally been assumed that rapid visual search is based on simple features and that spatial relations between features are irrelevant for this task. Seven experiments involving search for line drawings contradict this assumption; a major determinant of search is the presence of line junctions. Arrow- and Y-junctions were detected rapidly in isolation and when they were embedded in drawings of rectangular polyhedra. Search for T-junctions was considerably slower. Drawings containing T-junctions often gave rise to very slow search even when distinguishing arrow- or Y-junctions were present. This sensitivity to line relations suggests that preattentive processes can extract 3-dimensional orientation from line drawings. A computational model is outlined for how this may be accomplished in early human vision.

East Asians and Westerners differ in various aspects of perception and cognition. For example, visual memory for East Asians is believed to be more influenced by the contextual aspects of a scene than is the case for Westerners (Masuda & Nisbett, 2001). There are also differences in visual search: for Westerners, search for a long line among short is faster than for short among long, whereas this difference does not appear to hold for East Asians (Ueda et al., submitted). However, it is unclear how these group-level differences originate. To investigate the extent to which they depend upon environment, we tested visual search and visual memory in East Asian immigrants who had lived in Canada for different amounts of time. Recent immigrants were found to exhibit no search asymmetry, unlike Westerners who had spent their lives in Canada. However, immigrants who had lived in Canada for more than two years showed performance comparable to that of Westerners. These differences could not be explained by the general analytic/holistic processing distinction believed to differentiate Westerners and East Asians, since all observers showed a strong holistic tendency for visual recognition. Results instead support the suggestion that exposure to a new environment can significantly affect the particular processes used to perceive a given stimulus.

We show that cast shadows can have a significant influence on the speed of visual search. In particular, we find that search based on the shape of a region is affected when the region is darker than the background and corresponds to a shadow formed by lighting from above. Results support the proposal that an early-level system rapidly identifies regions as shadows and then discounts them, making their shapes more difficult to access. Several constraints used by this system are mapped out, including constraints on the luminance and texture of the shadow region, and on the nature of the item casting the shadow. Among other things, this system is found to distinguish between line elements (items containing only edges) and surface elements (items containing visible surfaces), with only the latter deemed capable of casting a shadow.

Recent developments in vision science have resulted in several major changes in our understanding of human visual perception. For example, attention no longer appears necessary for "visual intelligence"--a large amount of sophisticated processing can be done without it. Scene perception no longer appears to involve static, general-purpose descriptions, but instead may involve dynamic representations whose content depends on the individual and the task. And vision itself no longer appears to be limited to the production of a conscious "picture"--it may also guide processes outside the conscious awareness of the observer.

In this paper we examine to what extent the lengths of the links in an animated articulated figure can be changed without the viewer being aware of the change. This is investigated in terms of a framework that emphasizes the role of attention in visual perception. We conducted a set of five experiments to establish bounds for the sen-sitivity to changes in length as a function of several parameters and the amount of attention available. We found that while length changes of 3% can be perceived when the relevant links are given full attention, changes of over 20% can go unnoticed when attention is not focused in this way. These results provide general guidelines for algorithms that produce or process character motion data and also bring to light some of the potential gains that stand to be achieved with attention-based algorithms.

We show that early vision can use monocular cues to rapidly complete partially-occluded objects. Visual search for easily detected fragments becomes difficult when the completed shape is similar to others in the display; conversely, search for fragments that are difficult to detect becomes easy when the completed shape is distinctive. Results indicate that completion occurs via the occlusion-triggered removal of occlusion edges and linking of associated regions. We fail to find evidence for a visible filling-in of contours or surfaces, but do find evidence for a "functional" filling-in that prevents the constituent fragments from being rapidly accessed. As such, it is only the completed structures—and not the fragments themselves—that serve as the basis for rapid recognition.

Previous theories of early vision have assumed that visual search is based on simple two-dimensional aspects of an image, such as the orientation of edges and lines. It is shown here that search can also be based on three-dimensional orientation of objects in the corresponding scene, provided that these objects are simple convex blocks. Direct comparison shows that image-based and scene-based orientation are similar in their ability to facilitate search. These findings support the hypothesis that scene-based properties are represented at preattentive levels in early vision.

Recent advances in our understanding of visual perception have shown it to be a far more complex and counterintuitive process than previously believed. Several important consequences follow from this. First, the design of an effective statistical graphics system is unlikely to succeed based on intuition alone; instead, it must rely on a more sophisticated, systematic approach. The basic elements of such an approach are outlined here, along with several design principles. An overview is then given of recent advances in our understanding of visual perception, including rapid perception, visual attention, and scene perception. It then is argued that the mechanisms involved can be successfully harnessed to allow data to be displayed more effectively than at present. Several directions of development are discussed, including effective use of visual attention, the display of dynamic information, and the effective use of nonattentional and nonconscious perceptual systems

A modified visual search task demonstrates that humans are very good at resuming a search after it has been momentarily interrupted. This is shown by exceptionally rapid response time to a display that reappears after a brief interruption, even when an entirely different visual display is seen during the interruption and two different visual searches are performed simultaneously. This rapid resumption depends on the stability of the visual scene and is not due to display or response anticipations. These results are consistent with the existence of an iterative hypothesis-testing mechanism that compares information stored in short-term memory (the perceptual hypothesis) with information about the display (the sensory pattern). In this view, rapid resumption occurs because a hypothesis based on a previous glance of the scene can be tested very rapidly in a subsequent glance, given that the initial hypothesis-generation step has already been performed.

This past decade has seen a great resurgence of interest in the perception of change. Change has, of course, long been recognized as a phenomenon worthy of study, and vision scientists have given their attention to it at various times in the past (for a review, see Rensink, 2002a). But things seem different this time around. This time, there is an emerging belief that instead of being just another visual ability, the perception of change may be something central to our ‘visual life’, and that the mechanisms that underlie it may provide considerable insight into the operation of much of our visual system. This development may have been sparked by a number of factors: technology that allowed the easy creation of dynamic displays, a feeling in the air that it was time for something new, or it may have simply been a matter of chance. But once underway, this development was fueled by results, results that included both novel behavioral effects and new theoretical insights. Many of these centered around change blindness, the failure of observers to see large changes that are made contingent upon..

Visual analytics (VA) combines the strengths of human and machine intelligence to enable the discovery of interesting patterns in challenging datasets. Historically, most attention has been given to developing the machine component—for example, machine learning or the human-computer interface. However, it is also essential to develop the abilities of the analysts themselves, especially at the beginning of their careers. For the past several years, we at the University of British Columbia (UBC)—with the support of The Boeing Company—have experimented with various ways of preparing undergraduate students for VA. Although inspired by the need to prepare students to become visual analysts, the result turned out to be fairly general in scope, applicable to other analytical approaches, as well as more general research. In hindsight, this makes considerable sense. Although the visual component of VA is necessary, it is insufficient; many analytical activities at the human end involve nonvisual skills, such as effective decision-making and the ability to quickly focus on the relevant parts of a problem. The result of this experimentation is a third-year undergraduate course titled Cognitive Systems 303 (COGS 303) that focuses on “VA unplugged”—that is, on developing investigative abilities prior to training on the VA systems themselves. It was felt that if students focused on developing basic analytical habits of thought prior to learning VA systems, these habits would be reinforced by subsequent practice on “live” systems.

Adelson & Pentland (Chapter 11) use an engaging metaphor to illustrate their position on scene analysis: interpretations are produced by a workshop that employs a set of specialists, each concerned with a single aspect of the scene. The authors argue that it is too expensive to have a supervisor co-ordinate the specialists and that it is too expensive to let them operate independently. They then show that a careful sequencing of the specialists leads to solutions of minimum cost, at least for their world of Mondrian panels.

Knill, Kersten, & Mamassian (Chapter 6) provide an interesting discussion of how the Bayesian formulation can be used to help investigate human vision. In their view, computational theories can be based on an ideal observer that uses Bayesian inference to make optimal use of available information. Four factors are important here: the image information used, the output structures estimated, the priors assumed (i.e., knowledge about the structure of the world), and the likelihood function used (i.e., knowledge about the projection of the world onto the sensors). Knill & Kersten argue that such a framework not only helps analyze a perceptual task, but can also help investigators to define it. Two examples are provided (the interpretation of surface contour and the perception of moving shadows) to show how this approach can be used in practice. As the authors admit, most (if not all) perceptual processes are ill-suited to a "strong" Bayesian approach based on a single consistent model of the world. Instead, they argue for a "weak" variant that assumes Bayesian inference to be carried out in modules of more limited scope. But how weak is "weak"? Are such approaches suitable for only a few relatively low-level tasks, or can they be applied more generally? Could a weak Bayesian approach, for example, explain how we would recognize the return of Elvis Presley? The formal modelling of human perception To help get a fix on things, it is useful to examine the fate of an earlier attempt to formalize human perception: the application of information theory. It was once hoped that this theory—a close cousin of the Bayesian formulation—would provide a way to uncover information-handling laws that were largely independent of physical implementation. In this approach, the human nervous system was assumed to have..

Purpose. To determine whether highlights are rapidly identified at early levels of vision. Methods. Visual search experiments were carried out using simple black and white figures corresponding to shiny objects lit from various directions. These included, for example, depictions of cylinders with highlights positioned at various heights (see figure). Targets and distractors differed only in the arrangement of their constituent regions, allowing them to be distinguished by the position of the highlights on the corresponding objects. Results. Three observers were tested. Search is rapid when the target corresponds to a shiny black object with a bottom highlight among objects with top highlights (view figure from rightside up). However, search slows down when the target corresponds to an object with a top highlight among objects with bottom highlights (view figure from upside down). This asymmetry in search speeds is destroyed when the contrast of the figures is reversed. Conclusions. These results indicate that regions in the image can be rapidly identified as highlights at early levels of vision. They also suggest--as for several other rapid recovery processes--that an assumption of lighting from above is used as a simplifying constraint.