Barbara Tversky

Thinking, collaborating, and communication happen through interactions of the body, the mind, language, and things in the world. Actions of the body actively express intention, thought, and emotion. Points of the fingers and nods of the head can refer to things in the surrounding world. A string of interrelated gestures can represent an environment, a complex system, an arrangement of ideas, a sequence of actions. Those gestures use marks and actions in space to represent thought more directly than words—as do graphics like sketches, maps, graphs, diagrams, and pictures. Seeing or making gestures or graphics can change thought, both in those who view them and in those who make them. Interactions with gestures and graphics and the surrounding world are often internal, invisible, and unique, not in words, and not easily decomposed. Through those interactions, shared and individual meanings emerge and change in real time. These features of human thought present challenges to current multi-modal AI.

Cognitive categories in the geographic realm appear to manifest certain special features as contrasted with categories for objects at surveyable scales. We have argued that these features reflect specific ontological characteristics of geographic objects. This paper presents hypotheses as to the nature of the features mentioned, reviews previous empirical work on geographic categories, and presents the results of pilot experiments that used English-speaking subjects to test our hypotheses. Our experiments show geographic categories to be similar to their non-geographic counterparts in the ways in which they generate instances of different relative frequencies at different levels. Other tests, however, provide preliminary evidence for the existence of important differences in subjects’ categorizations of geographic and non-geographic objects, and suggest further experimental work especially with regard to the role in cognitive categorization of different types of object-boundaries at different scales.

Within psychology, at least two research communities study spatial cognition. One community studies systematic errors in spatial memory and judgement, accounting for them as a consequence of and clue to normal perceptual and cognitive processing. The other community studies navigation in real space, isolating the contributions of various sensory cues and sensori- motor systems to successful navigation. The former group emphasizes error, the latter, selective mechanisms, environmental or evolutionary, that produce fine-tuned correct responses. How can these approaches be reconciled and integrated? First, by showing why errors are impervious to selective pressures. The schematization that leads to errors is a natural consequence of normal perceptual and cognitive processes; it is inherent to the construction of mental spaces and to using them to make judgments in limited capacity working memory. Selection can act on particular instances of errors, yet it is not clear that selection can act on the general mechanisms that produce them. Next, in the wild, there are a variety of correctives. Finally, closer examination of navigation in the wild shows systematic errors, for example, over-shooting in dead reckoning across species. Here, too, environments may provide correctives, specifically, landmarks. General cognitive mechanisms generate general solutions. The errors inevitably produced may be reduced by local specific sensori-motor couplings as well as local environmental cues. Navigation, and other behaviors as well, are a consequence of both

Although people can take spatial perspectives different from their own, it is widely assumed that egocentric perspectives are natural and have primacy. Two studies asked respondents to describe the spatial relations between two objects on a table in photographed scenes; in some versions, a person sitting behind the objects was either looking at or reaching for one of the objects. The mere presence of another person in a position to act on the objects induced a good proportion of respondents to describe the spatial relations from that person’s point of view. When the query about the spatial relations was phrased in terms of action, more respondents took the other’s perspective than their own. The implication of action elicits spontaneous spatial perspective-taking, seemingly in the service of understanding the other’s actions.

When thought overwhelms the mind, the mind uses the body and the world. Several studies reveal ways that people alone or together use gesture and marks on paper to structure and augment their thought for comprehension, inference, and discovery. The studies show that the mapping of thought to gesture or the page is more direct than the arbitrary mapping to language and suggest that these forms of visual/spatial/action representation are used to “translate” language into mental representations. It is argued that actions in space create patterns in the world that reflect abstractions, that the actions are incorporated into gestures and the patterns into diagrams, a network that integrates gesture, action, the designed world, and abstraction dubbedspraction.

Mechanical systems have structural organizations—parts, and their relations—and functional organizations—temporal, dynamic, and causal processes—which can be explained using text or diagrams. Two experiments illustrate the role of arrows in diagrams of mechanical systems. In Experiment 1, people described diagrams with or without arrows, interpreting diagrams without arrows as conveying structural information and diagrams with arrows as conveying functional information. In Experiment 2, people produced sketches of mechanical systems from structural or functional descriptions. People spontaneously used arrows to indicate functional processes in diagrams. Arrows can play a powerful role in augmenting structural diagrams to convey dynamic, causal, or functional information.

Depictive expressions of thought predate written language by thousands of years. They have evolved in communities through a kind of informal user testing that has refined them. Analyzing common visual communications reveals consistencies that illuminate how people think as well as guide design; the process can be brought into the laboratory and accelerated. Like language, visual communications abstract and schematize; unlike language, they use properties of the page (e.g., proximity and place: center, horizontal/up–down, vertical/left–right) and the marks on it (e.g., dots, lines, arrows, boxes, blobs, likenesses, symbols) to convey meanings. The visual expressions of these meanings (e.g., individual, category, order, relation, correspondence, continuum, hierarchy) have analogs in language, gesture, and especially in the patterns that are created when people design the world around them, arranging things into piles and rows and hierarchies and arrays, spatial-abstraction-action interconnections termed spractions. The designed world is a diagram

People are not as fast or as strong as many other creatures that evolved around us. What gives us an evolutionary advantage is working together to achieve common aims. Coordinating joint action begins at a tender age with such cooperative activities as alternating babbling and clapping games. Adult joint activities are far more complex and use multiple means of coordination. Joint action has attracted qualitative analyses by sociolinguists, cognitive scientists, and philosophers as well as empirical analyses and theories by cognitive scientists. Here, we analyze how joint action is spontaneously coordinated from start to finish in a novel complex real‐life joint activity, assembling a piece of furniture, a task that captures the essentials of joint action, collaborators, things in the world, and communicative devices. Pairs of strangers assembled a TV cart from a stack of parts and a photo of the completed cart. Coordination prior to each assembly action was coded as explicit, using speech or gesture, or implicit, actions that both advanced the task and communicated the next step. Initial planning relied on explicit communication about structure, but not action nor division of labor, which were improvised. That served to establish a joint representation of the goal that informed actions and monitored progress. As assembly progressed, coordination was increasingly implicit, through actions alone. Joint action is a dynamic interplay of explicit and implicit signaling with respect to things in the world to coordinate ongoing progress, guided by a shared representation of the goal.

Our earliest tools are our bodies. Our hands raise and turn and toss and carry and push and pull, our legs walk and climb and kick allowing us to move and act in the world and to create the multitude of artifacts that improve our lives. The list of actions made by our hands and feet and other parts of our bodies is long. What is more remarkable is we turn those actions in the world into actions on thought through gestures, language, and graphics, thereby creating cognitive tools that expand the mind. The focus here is gesture; gestures transform actions on perceptible objects to actions on imagined thoughts, carrying meaning with them rapidly, precisely, and directly. We review evidence showing that gestures enhance our own thinking and change the thought of others. We illustrate the power of gestures in studies showing that gestures uniquely change conceptions of time, from sequential to simultaneous, from sequential to cyclical, and from a perspective embedded in a timeline to an external perspective looking on a timeline, and by so doing obviate the ambiguities of an embedded perspective. We draw parallels between representations in gesture and in graphics; both use marks or actions arrayed in space to communicate more immediately than symbolic language.