David Kirsh

The ability to reason about how things were made is a pervasive aspect of how humans make sense of physical objects. Such reasoning is useful for a range of everyday tasks, from assembling a piece of furniture to making a sandwich and knitting a sweater. What enables people to reason in this way even about novel objects, and how do people draw upon prior experience with an object to continually refine their understanding of how to create it? To explore these questions, we developed a virtual task environment to investigate how people come up with step‐by‐step procedures for recreating block towers whose composition was not readily apparent, and analyzed how the procedures they used to build them changed across repeated attempts. Specifically, participants (N = 105) viewed 2D silhouettes of eight unique block towers in a virtual environment simulating rigid‐body physics, and aimed to reconstruct each one in less than 60 s. We found that people built each tower more accurately and quickly across repeated attempts, and that this improvement reflected both group‐level convergence upon a tiny fraction of all possible viable procedures, as well as error‐dependent updating across successive attempts by the same individual. Taken together, our study presents a scalable approach to measuring consistency and variation in how people infer solutions to physical assembly problems.

Time-motion studies revolutionized the design and efficiency of repetitive work last century. Would time-idea studies revolutionize the rules of intellectual/creative work this century? Collaborating with seven professional dancers, we set out to discover if there were any significant temporal patterns to be found in a timeline coded to show when dancers come up with ideas and when they modify or reject them. On each of 3 days, the dancers were given a choreographic problem to help them generate a novel, high quality contemporary dance phrase. They were videoed as they worked on this task for sessions of 15, 30, and 45 min. At the end of each 15 min interval during each session, we had them perform the phrase they were creating. They recorded and then coded the video of themselves dancing during these sessions by using a coding language we developed with them to identify when ideas are introduced, modified, and rejected. We found that most ideas are created early and that though these early ideas are aggressively pruned early on, many still make it into the final product. The two competing accounts of creativity in design research make predictions for the temporal structure of creativity. Our results support neither account, rather showing a more blended version of the two. The iterative design view, arguably the dominant view, is that good ideas are the product of generating many ideas, choosing one fairly early, committing to it, and iteratively improving it. The “fail fast fail often” view is that good ideas are the product of rapidly generating and discarding ideas and holding back from early commitment to any one in particular. The result of holding back commitment, typically, is not that an idea is taken up later and then incrementally improved at the last minute, as much as that later designs are not completely novel, instead incorporating the best parts of the entire sequence of ideas. In our study, we found no evidence that one account or the other was more predictive for the domain of contemporary dance. The behavior of the dancers that we studied revealed elements of both, calling into question how predictive these theories are.

A complementary strategy can be defined as any organizing activity which recruits external elements to reduce cognitive loads. Typical organizing activities include pointing, arranging the position and orientation of nearby objects, writing things down, manipulating counters, rulers or other artifacts that can encode the state of a process or simplify perception. To illustrate the idea of a complementary strategy, a simple experiment was performed in which subjects were asked to determine the dollar value of collections of coins. In the no-hands condition, subjects were not allowed to touch the coin images or to move their hands in any way. In the hands condition, they were allowed to use their hands and fingers however they liked. Significant improvements in time and number of errors were observed when S's used their hands over when they did not. To explain these facts, a brief account of some commonly observed complementary strategies is presented, and an account of their potential benefits to perception, memory and attention.

To explore the question of physical thinking – using the body as an instrument of cognition – we collected extensive video and interview data on the creative process of a noted choreographer and his company as they made a new dance. A striking case of physical thinking is found in the phenomenon of marking. Marking refers to dancing a phrase in a less than complete manner. Dancers mark to save energy. But they also mark to explore the tempo of a phrase, or its movement sequence, or the intention behind it. Because of its representational nature, marking can serve as a vehicle for thought. Importantly, this vehicle is less complex than the version of the same phrase danced ‘full-out’. After providing evidence for distinguishing different types of marking, three ways of understanding marking as a form of thought are considered: marking as a gestural language for encoding aspects of a target movement, marking as a method of priming neural systems involved in the target movement, and marking as a method for improving the precision of mentally projecting aspects of the target.

Why do people create extra representations to help them make sense of situations, diagrams, illustrations, instructions and problems? The obvious explanation— external representations save internal memory and com- putation—is only part of the story. I discuss seven ways external representations enhance cognitive power: they change the cost structure of the inferential landscape; they provide a structure that can serve as a shareable object of thought; they create persistent referents; they facilitate re- representation; they are often a more natural representation of structure than mental representations; they facilitate the computation of more explicit encoding of information; they enable the construction of arbitrarily complex structure; and they lower the cost of controlling thought—they help coordinate thought. Jointly, these functions allow people to think more powerfully with external representations than without. They allow us to think the previously unthinkable.

This paper examines some of the methods animals and humans have of adapting their environment. Because there are limits on how many different tasks a creature can be designed to do well in, creatures with the capacity to redesign their environments have an adaptive advantage over those who can only passively adapt to existing environmental structures. To clarify environmental redesign I rely on the formal notion of a task environment as a directed graph where the nodes are states and the links are actions. One natural form of redesign is to change the topology of this graph structure so as to increase the likelihood of task success or to reduce its expected cost, measured in physical terms. This may be done by eliminating initial states hence eliminating choice points; by changing the action repertoire; by changing the consequence function; and lastly, by adding choice points. Another major method for adapting the environment is to change its cognitive congeniality. Such changes leave the state space formally intact but reduce the number and cost of mental operations needed for task success; they reliably increase the speed, accuracy or robustness of performance. The last section of the paper describes several of these epistemic or complementary actions found in human performance.

In dance, there is a practice called ‘marking’. When dancers mark, they execute a dance phrase in a simplified, schematic or abstracted form. Based on our interviews with professional dancers in the classical, modern, and contemporary traditions, it is fair to assume that most dancers mark in the normal course of rehearsal and practice. When marking, dancers use their body-in-motion to represent some aspect of the full-out phrase they are thinking about. Their stated reason for marking is that it saves energy, avoids strenuous movement such as jumps, and sometimes it facilitates review of specific aspects of a phrase, such as tempo, movement sequence, or intention, all without the mental and physical complexity involved in creating a phrase full-out. It facilitates real-time reflection.

A videographic study of origami is presented in which subjects were observed making four different origami objects under five modes of instruction: photos + captions, illustrations-only, illustrations with small captions, illustrations with large captions, and text-only as control. The objective of the study was to explore the gestures and other actions that subjects produce as they try to follow instructions rather than to determine the most effective style of instruction per se. We found that the task of situating instructions to the context at hand is error prone and that to facilitate it subjects gesture, point, re-orient illustrations, and generally do things that have no function other than to change the epistemic and interactive landscape of activity so they can more easily understand what is to be done. These studies bear on the new questions designers are asking about the placement, timing, and pace of instructions that digital aids now provide and on the fundamental question of how humans embed themselves in an activity by framing their task in a situation specific manner.

Can virtual engagement enable the sort of interactive coupling with objects enjoyed by archaeologists who are physically present at a site? To explore this question I consider three points: 1) Tangible interaction: What role does encounter by muscle and sinew play in experiencing and understanding objects? 2) Thinking with things. What sorts of interactions are involved when we manipulate things to facilitate thought? 3) Projection and imagination. Archaeological inquiry involves processes beyond perception. Material engagement of things stimulates these processes. What must be present in a virtual environment to recreate the feel of material engagement? I conclude that nothing, in principle, prevents future virtual environments from supporting this material engagement of digital versions of artifacts. But, there is much that remains to be understood about how to realize this material engagement, both at a technological and a cognitive level.

We are quickly passing through the historical moment when people work in front of a single computer, dominated by a small CRT and focused on tasks involving only local information. Networked computers are becoming ubiquitous and are playing increasingly significant roles in our lives and in the basic infrastructure of science, business, and social interaction. For human-computer interaction o advance in the new millennium we need to better understand the emerging dynamic of interaction in which the focus task is no longer confined to the desktop but reaches into a complex networked world of information and computer-mediated interactions. We think the theory of distributed cognition has a special role to play in understanding interactions between people and technologies, for its focus has always been on whole environments: what we really do in them and how we coordinate our activity in them. Distributed cognition provides a radical reorientation of how to think about designing and supporting human-computer interaction. As a theory it is specifically tailored to understanding interactions among people and technologies. In this article we propose distributed cognition as a new foundation for human-computer interaction, sketch an integrated research framework, and use selections from our earlier work to suggest how this framework can provide new opportunities in the design of digital work materials.

It is sometimes argued that if PDP networks can be trained to make correct judgements of grammaticality we have an existence proof that there is enough information in the stimulus to permit learning grammar by inductive means alone. This seems inconsistent superficially with Gold's theorem and at a deeper level with the fact that networks are designed on the basis of assumptions about the domain of the function to be learned. To clarify the issue I consider what we should learn from Gold's theorem, then go on to inquire into what it means to say that knowledge is domain specific. I first try sharpening the intuitive notion of domain specific knowledge by reviewing the alleged difference between processing limitations due to shortage of resources vs shortages of knowledge. After rejecting different formulations of this idea, I suggest that a model is language specific if it transparently refer to entities and facts about language as opposed to entities and facts of more general mathematical domains. This is a useful but not necessary condition. I then suggest that a theory is domain specific if it belongs to a model family which is attuned in a law-like way to domain regularities. This leads to a comparison of PDP and parameter setting models of language learning. I conclude with a novel version of the poverty of stimulus argument.

To explore the question of physical thinking – using the body as an instrument of cognition – we collected extensive video and interview data on the creative process of a noted choreographer and his company as they made a new dance. A striking case of physical thinking is found in the phenomenon of marking. Marking refers to dancing a phrase in a less than complete manner. Dancers mark to save energy. But they also mark to explore the tempo of a phrase, or its movement sequence, or the intention behind it. Because of its representational nature, marking can serve as a vehicle for thought. Importantly, this vehicle is less complex than the version of the same phrase danced ‘full-out’. After providing evidence for distinguishing different types of marking, three ways of understanding marking as a form of thought are considered: marking as a gestural language for encoding aspects of a target movement, marking as a method of priming neural systems involved in the target movement, and marking as a method for improving the precision of mentally projecting aspects of the target.

An experiment was performed to test the hypothesis that people sometimes take physical actions to make themselves more effective problem solvers. The task was to generate all possible words that could be formed from seven Scrabble letters. In one condition, participants could use their hands to manipulate the letters, and in another condition, they could not. Results show that more words were generated with physical manipulation than without. However, an interaction was obtained between the physical manipulation conditions and the specific letter sets chosen, indicating that physical manipulation helps more for generating words in some circumstances than in others. Overall, our findings can be explained in terms of an interactive search process in which external, physical activity effectively complements internal, cognitive activity. Within this framework, the interaction can be explained in terms of the relative difficulty of generating words from the letters given in the different sets.

In the course of daily life we solve problems often enough that there is a special term to characterize the activity and the right to expect a scientific theory to explain its dynamics. The classical view in psychology is that to solve a problem a subject must frame it by creating an internal representation of the problem‘s structure, usually called a problem space. This space is an internally generable representation that is mathematically identical to a graph structure with nodes and links. The nodes can be annotated with useful information, and the whole representation can be distributed over internal and external structures such as symbolic notations on paper or diagrams. If the representation is distributed across internal and external structures the subject must be able to keep track of activity in the distributed structure. Problem solving proceeds as the subject works from an initial state in this mentally supported space, actively construction possible solution paths, evaluating them and heuristically choosing the best. Control of this exploratory process is not well understood, as it is not always systematic, but various heuristic search algorithms have been proposed and some experimental support has been provided for them.

This paper introduces the concept of Adaptive Rooms, which are virtual environments able to dynamically adapt to users’ needs, including ‘physical’ and cognitive workflow requirements, number of users, differing cognitive abilities and skills. Adaptive rooms are collections of virtual objects, many of them self-transforming objects, housed in an architecturally active room with information spaces and tools. An ontology of objects used in adap- tive rooms is presented. Virtual entities are classified as passive, reactive, ac- tive, and information entities, and their sub-categories. Only active objects can be self-transforming. Adaptive Rooms are meant to combine the insights of ubiquitous computing -- that computerization should be everywhere, transpar- ently incorporated -- with the insights of augmented reality -- that everyday ob- jects can be digitally enhanced to carry more information about their use. To display the special potential of adaptive rooms, concrete examples are given to show how the demands of cognitive workflow can be reduced.

In the scientific study of mind a distinction is drawn between explicit knowledge–knowledge that can be elicited from a subject by suitable inquiry or prompting, can be brought to consciousness, and externally expressed in words–and implicit knowledge–knowledge that cannot be elicited, cannot be made directly conscious, and cannot be articulated. Michael Polanyi (1967) argued that we usually ‘know more than we can say’. The part we can articulate is explicitly known; the part we cannot is implicit.

Virtual environment landmarks are essential in wayfinding: they anchor routes through a region and provide memorable destinations to return to later. Current virtual environment browsers provide user interface menus that characterize available travel destinations via landmark textual descriptions or thumbnail images. Such characterizations lack the depth cues and context needed to reliably recognize 3D landmarks. This paper introduces a new user interface affordance that captures a 3D representation of a virtual environment landmark into a 3D thumbnail, called a worldlet. Each worldlet is a miniature virtual world fragment that may be interactively viewed in 3D, enabling a traveler to gain first-person experience with a travel destination. In a pilot student conducted to compare textual, image, and worldlet landmark representations within a wayfinding task, worldlet use significantly reduced the overall travel time and distance traversed, virtually eliminating unnecessary backtracking.

Architecture is about to enter its first magical phase: a time when buildings actively co-operate with their inhabitants; when objects know what they are, where they are, what is near them; when social and physical space lose their type coupling; when wall and partitions change with mood and task. As engineers and scientists explore how to digitse the world around us, the classical constraints of design, ruled so long by the physics of space, time, and materials, are starting to crumble. Documents can be laid down in one place, automatically cloned, and a copy picked up in another. Meetings scheduled for 9am to 10am can be joined by latecomers at noon, who then participate in a captured form of the event and are ‘edited into’ the past. People on the West coast of the USA can participate, in a telepresent way, with their colleagues on the East coast, and hold a meeting against a virtual backdrop, such as a production line in their Taiwanese factory. Walls seem to dematerialise, remote objects can be touched virtually, shaped, passed through one another. Technology is moving inexorably so that being in one place at a time no longer need dominate how we work and play. Material boundaries are losing their meaning, and interface and information space are catch words that architects must master.

Using the perspective of situated cognition we studied how people interact with a physical map to help them navigate through an unfamiliar environment. The study used a mixture of cognitive ethnography and traditional experimental methods. We found that the difference between high and low performing navigators showed up in the speed they completed their task and also in the way they use maps. High performers plan routes using a survey method whereas low performers use a route strategy. We suggest that when people are given a task that does not match their cognitive style they try to transform the task to better suit their cognitive abilities and cognitive style.

Individual creativity is standardly treated as an ‘internalist’ process occurring solely in the head. An alternative, more interactionist view is presented here, where working with objects, media and other external things is seen as a fundamental component of creative thought. The value of chance interaction and chance cueing — practices widely used in the creative arts — is explored briefly in an account of the creative method of choreographer Wayne McGregor and then more narrowly in an experimental study that compared performance on a Scrabble-like word discovery problem. Subjects were presented with seven letters and given two minutes to call out three-to-seven-letter English words. There were three conditions: The tiles were fixed in place, subjects were free to move the tiles manually or the tiles could be randomly shuffled. Results showed that random shuffling was best, with manual movement second. Three reasons are provided: Shuffling is faster and cheaper than mentally thinking of candidates; randomizing strings covers the search space better than a deterministic method based on past successes; and randomizing is equivalent to adding diversity to a team, which is known empirically to lead to more creative solutions.

Why do people create extra representations to help them make sense of situations, diagrams, illustrations, instructions and problems? The obvious explanation – external representations save internal memory and computation – is only part of the story. I discuss eight ways external representations enhance cognitive power: they provide a structure that can serve as a shareable object of thought; they create persistent referents; they change the cost structure of the inferential landscape; they facilitate re-representation; they are often a more natural representation of structure than mental representations; they facilitate the computation of more explicit encoding of information; they enable the construction of arbitrarily complex structure; and they lower the cost of controlling thought – they help coordinate thought.

Humans are closely coupled with their environments. They rely on being ‘embedded’ to help coordinate the use of their internal cognitive resources with external tools and resources. Consequently, everyday cognition, even cognition in the absence of others, may be viewed as partially distributed. As cognitive scientists our job is to discover and explain the principles governing this distribution: principles of coordination, externalization, and interaction. As designers our job is to use these principles, especially if they can be converted to metrics, in order to invent and evaluate candidate designs. After discussing a few principles of interaction and embedding I discuss the usefulness of a range of metrics derived from economics, computational complexity, and psychology.

The question of how to conceive and represent the context of work is explored from the theoretical perspective of distributed cognition. It is argued that to understand the office work context we need to go beyond tracking superficial physical attributes such as who or what is where and when and consider the state of digital resources, people’s concepts, task state, social relations, and the local work culture, to name a few. In analyzing an office more deeply, three concepts are especially helpful: entry points, action landscapes, and coordinating mechanisms. An entry point is a structure or cue that represents an invitation to enter an information space or office task. An activity landscape is part mental construct and part physical; it is the space users interactively construct out of the resources they find when trying to accomplish a task. A coordinating mechanism is an artifact, such as a schedule or clock, or an environmental structure such as the layout of papers to be signed, which helps a user manage the complexity of his task. Using these three concepts we can abstract away from many of the surface attributes of work context and define the deep structure of a setting—the invariant structure that many office settings share. A long-term challenge for context-aware computing is to operationalize these analytic concepts.