David Kirsh

The ability to reason about how things were made is a pervasive aspect of how humans make sense of physical objects. Such reasoning is useful for a range of everyday tasks, from assembling a piece of furniture to making a sandwich and knitting a sweater. What enables people to reason in this way even about novel objects, and how do people draw upon prior experience with an object to continually refine their understanding of how to create it? To explore these questions, we developed a virtual task environment to investigate how people come up with step‐by‐step procedures for recreating block towers whose composition was not readily apparent, and analyzed how the procedures they used to build them changed across repeated attempts. Specifically, participants (N = 105) viewed 2D silhouettes of eight unique block towers in a virtual environment simulating rigid‐body physics, and aimed to reconstruct each one in less than 60 s. We found that people built each tower more accurately and quickly across repeated attempts, and that this improvement reflected both group‐level convergence upon a tiny fraction of all possible viable procedures, as well as error‐dependent updating across successive attempts by the same individual. Taken together, our study presents a scalable approach to measuring consistency and variation in how people infer solutions to physical assembly problems.

Time-motion studies revolutionized the design and efficiency of repetitive work last century. Would time-idea studies revolutionize the rules of intellectual/creative work this century? Collaborating with seven professional dancers, we set out to discover if there were any significant temporal patterns to be found in a timeline coded to show when dancers come up with ideas and when they modify or reject them. On each of 3 days, the dancers were given a choreographic problem to help them generate a novel, high quality contemporary dance phrase. They were videoed as they worked on this task for sessions of 15, 30, and 45 min. At the end of each 15 min interval during each session, we had them perform the phrase they were creating. They recorded and then coded the video of themselves dancing during these sessions by using a coding language we developed with them to identify when ideas are introduced, modified, and rejected. We found that most ideas are created early and that though these early ideas are aggressively pruned early on, many still make it into the final product. The two competing accounts of creativity in design research make predictions for the temporal structure of creativity. Our results support neither account, rather showing a more blended version of the two. The iterative design view, arguably the dominant view, is that good ideas are the product of generating many ideas, choosing one fairly early, committing to it, and iteratively improving it. The “fail fast fail often” view is that good ideas are the product of rapidly generating and discarding ideas and holding back from early commitment to any one in particular. The result of holding back commitment, typically, is not that an idea is taken up later and then incrementally improved at the last minute, as much as that later designs are not completely novel, instead incorporating the best parts of the entire sequence of ideas. In our study, we found no evidence that one account or the other was more predictive for the domain of contemporary dance. The behavior of the dancers that we studied revealed elements of both, calling into question how predictive these theories are.

A complementary strategy can be defined as any organizing activity which recruits external elements to reduce cognitive loads. Typical organizing activities include pointing, arranging the position and orientation of nearby objects, writing things down, manipulating counters, rulers or other artifacts that can encode the state of a process or simplify perception. To illustrate the idea of a complementary strategy, a simple experiment was performed in which subjects were asked to determine the dollar value of collections of coins. In the no-hands condition, subjects were not allowed to touch the coin images or to move their hands in any way. In the hands condition, they were allowed to use their hands and fingers however they liked. Significant improvements in time and number of errors were observed when S's used their hands over when they did not. To explain these facts, a brief account of some commonly observed complementary strategies is presented, and an account of their potential benefits to perception, memory and attention.

To explore the question of physical thinking – using the body as an instrument of cognition – we collected extensive video and interview data on the creative process of a noted choreographer and his company as they made a new dance. A striking case of physical thinking is found in the phenomenon of marking. Marking refers to dancing a phrase in a less than complete manner. Dancers mark to save energy. But they also mark to explore the tempo of a phrase, or its movement sequence, or the intention behind it. Because of its representational nature, marking can serve as a vehicle for thought. Importantly, this vehicle is less complex than the version of the same phrase danced ‘full-out’. After providing evidence for distinguishing different types of marking, three ways of understanding marking as a form of thought are considered: marking as a gestural language for encoding aspects of a target movement, marking as a method of priming neural systems involved in the target movement, and marking as a method for improving the precision of mentally projecting aspects of the target.

Why do people create extra representations to help them make sense of situations, diagrams, illustrations, instructions and problems? The obvious explanation— external representations save internal memory and com- putation—is only part of the story. I discuss seven ways external representations enhance cognitive power: they change the cost structure of the inferential landscape; they provide a structure that can serve as a shareable object of thought; they create persistent referents; they facilitate re- representation; they are often a more natural representation of structure than mental representations; they facilitate the computation of more explicit encoding of information; they enable the construction of arbitrarily complex structure; and they lower the cost of controlling thought—they help coordinate thought. Jointly, these functions allow people to think more powerfully with external representations than without. They allow us to think the previously unthinkable.

In dance, there is a practice called ‘marking’. When dancers mark, they execute a dance phrase in a simplified, schematic or abstracted form. Based on our interviews with professional dancers in the classical, modern, and contemporary traditions, it is fair to assume that most dancers mark in the normal course of rehearsal and practice. When marking, dancers use their body-in-motion to represent some aspect of the full-out phrase they are thinking about. Their stated reason for marking is that it saves energy, avoids strenuous movement such as jumps, and sometimes it facilitates review of specific aspects of a phrase, such as tempo, movement sequence, or intention, all without the mental and physical complexity involved in creating a phrase full-out. It facilitates real-time reflection.

A videographic study of origami is presented in which subjects were observed making four different origami objects under five modes of instruction: photos + captions, illustrations-only, illustrations with small captions, illustrations with large captions, and text-only as control. The objective of the study was to explore the gestures and other actions that subjects produce as they try to follow instructions rather than to determine the most effective style of instruction per se. We found that the task of situating instructions to the context at hand is error prone and that to facilitate it subjects gesture, point, re-orient illustrations, and generally do things that have no function other than to change the epistemic and interactive landscape of activity so they can more easily understand what is to be done. These studies bear on the new questions designers are asking about the placement, timing, and pace of instructions that digital aids now provide and on the fundamental question of how humans embed themselves in an activity by framing their task in a situation specific manner.

This paper examines some of the methods animals and humans have of adapting their environment. Because there are limits on how many different tasks a creature can be designed to do well in, creatures with the capacity to redesign their environments have an adaptive advantage over those who can only passively adapt to existing environmental structures. To clarify environmental redesign I rely on the formal notion of a task environment as a directed graph where the nodes are states and the links are actions. One natural form of redesign is to change the topology of this graph structure so as to increase the likelihood of task success or to reduce its expected cost, measured in physical terms. This may be done by eliminating initial states hence eliminating choice points; by changing the action repertoire; by changing the consequence function; and lastly, by adding choice points. Another major method for adapting the environment is to change its cognitive congeniality. Such changes leave the state space formally intact but reduce the number and cost of mental operations needed for task success; they reliably increase the speed, accuracy or robustness of performance. The last section of the paper describes several of these epistemic or complementary actions found in human performance.

We are quickly passing through the historical moment when people work in front of a single computer, dominated by a small CRT and focused on tasks involving only local information. Networked computers are becoming ubiquitous and are playing increasingly significant roles in our lives and in the basic infrastructure of science, business, and social interaction. For human-computer interaction o advance in the new millennium we need to better understand the emerging dynamic of interaction in which the focus task is no longer confined to the desktop but reaches into a complex networked world of information and computer-mediated interactions. We think the theory of distributed cognition has a special role to play in understanding interactions between people and technologies, for its focus has always been on whole environments: what we really do in them and how we coordinate our activity in them. Distributed cognition provides a radical reorientation of how to think about designing and supporting human-computer interaction. As a theory it is specifically tailored to understanding interactions among people and technologies. In this article we propose distributed cognition as a new foundation for human-computer interaction, sketch an integrated research framework, and use selections from our earlier work to suggest how this framework can provide new opportunities in the design of digital work materials.

Can virtual engagement enable the sort of interactive coupling with objects enjoyed by archaeologists who are physically present at a site? To explore this question I consider three points: 1) Tangible interaction: What role does encounter by muscle and sinew play in experiencing and understanding objects? 2) Thinking with things. What sorts of interactions are involved when we manipulate things to facilitate thought? 3) Projection and imagination. Archaeological inquiry involves processes beyond perception. Material engagement of things stimulates these processes. What must be present in a virtual environment to recreate the feel of material engagement? I conclude that nothing, in principle, prevents future virtual environments from supporting this material engagement of digital versions of artifacts. But, there is much that remains to be understood about how to realize this material engagement, both at a technological and a cognitive level.

In the scientific study of mind a distinction is drawn between explicit knowledge— knowledge that can be elicited from a subject by suitable inquiry or prompting, can be brought to consciousness, and externally expressed in words—and implicit knowledge—knowledge that cannot be elicited, cannot be made directly conscious, and can- not be articulated. Michael Polanyi (1967) argued that we usually ‘know more than we can say’. The part we can articulate is explicitly known; the part we cannot is implicit.

Contemporary choreography offers a window onto creative processes that rely on harnessing the power of sensory sys- tems. Dancers use their body as a thing to think with and their sensory systems as engines to simulate ideas non- propositionally. We report here on an initial analysis of data collected in a lengthy ethnographic study of the making of a dance by a major choreographer and show how translating between different sensory modalities can help dancers and choreographer to be more creative.

Metacognition is associated with planning, monitoring, evaluating and repairing performance Designers of elearning systems can improve the quality of their environments by explicitly structuring the visual and interactive display of learning contexts to facilitate metacognition. Typically page layout, navigational appearance, visual and interactivity design are not viewed as major factors in metacognition. This is because metacognition tends to be interpreted as a process in the head, rather than an interactive one. It is argued here, that cognition and metacognition are part of a continuum and that both are highly interactive. The tenets of this view are explained by reviewing some of the core assumptions of the situated and distribute approach to cognition and then further elaborated by exploring the notions of active vision, visual complexity, affordance landscape and cue structure. The way visual cues are structured and the way interaction is designed can make an important difference in the ease and effectiveness of cognition and metacognition. Documents that make effective use of markers such as headings, callouts, italics can improve students' ability to comprehend documents and 'plan' the way they review and process content. Interaction can be designed to improve 'the proximal zone of planning' - the look ahead and apprehension of what is nearby in activity space that facilitates decisions. This final concept is elaborated in a discussion of how e-newspapers combine effective visual and interactive design to enhance user control over their reading experience.

The type of principles which cognitive engineers need to design better work environments are principles which explain interactivity and distributed cognition: how human agents interact with themselves and others, their work spaces, and the resources and constraints that populate those spaces. A first step in developing these principles is to clarify the fundamental concepts of environment, coordination, and behavioural function. Using simple examples, I review changes the distributed perspective forces on these basic notions.

Video data from three large captures of choreographic dance making was analyzed to determine if there is a difference between participant knowledge – the knowledge an agent acquires by being the cause of an action – and observer knowledge – the knowledge an observer acquires through close attention to someone else’s performance. The idea that there might be no difference has been challenged by recent findings about the action observation network and tacitly challenged by certain tenets in enactive perception. We explored why a choreographer ‘riff’s’ when appropriating and evaluating the movements of his dancers. By recruiting his body to help him cognize he is able to understand the possibilities of movement better than observation. He acquires participant knowledge.

The objective of this essay is to provide the beginning of a principled classification of some of the ways space is intelligently used. Studies of planning have typically focused on the temporal ordering of action, leaving as unaddressed questions of where to lay down instruments, ingredients, work-in-progress, and the like. But, in having a body, we are spatially located creatures: we must always be facing some direction, have only certain objects in view, be within reach of certain others. How we manage the spatial arrangement of items around us is not an afterthought: it is an integral part of the way we think, plan, and behave. The proposed classification has three main categories: spatial arrangements that simplify choice; spatial arrangements that simplify perception; and spatial dynamics that simplify internal computation. The data for such a classification is drawn from videos of cooking, assembly and packing, everyday observations in supermarkets, workshops and playrooms, and experimental studies of subjects playing Tetris, the computer game. This study, therefore, focuses on interactive processes in the medium and short term: on how agents set up their workplace for particular tasks, and how they continuously manage that workplace.

Cognitive organisms have three rather different techniques for intelligently regulating their intake of environmental information. In order of the time needed to uncover information they are: 1. control of attention: within an image produced by a given sensor certain elements can be selected for additional processing; 2. control of gaze: the orientation and resolution (center of foveation) of the sensor can be regulated to create a new image; 3. control of activity: certain non-perceptual actions can be performed to increase the probability of unearthing salient information that currently is unavailable, hard to detect, or hard to compute.

Multimedia technology offers instructional designers an unprecedented opportunity to create richly interactive learning environments. With greater design freedom comes complexity. The standard answer to the problems of too much choice, disorientation, and complex navigation is thought to lie in the way we design interactivity in a system. Unfortunately, the theory of interactivity is at an early state of development. After critiquing the decision cycle model of interaction—the received theory in human computer interaction—I present arguments and observational data to show that humans have several ways of interacting with their environments, which resist accommodation in the decision cycle model. These additional ways of interacting include: preparing the environment, maintaining the environment, and reshaping the cognitive congeniality of the environment. Understanding how these actions simplify the computational complexity of our mental processes is the first step in designing the right sort of resources and scaffolding necessary for tractable learner controlled learning environments.

To explore the question of physical thinking – using the body as an instrument of cognition – we collected extensive video and interview data on the creative process of a noted choreographer and his company as they made a new dance. A striking case of physical thinking is found in the phenomenon of marking. Marking refers to dancing a phrase in a less than complete manner. Dancers mark to save energy. But they also mark to explore the tempo of a phrase, or its movement sequence, or the intention behind it. Because of its representational nature, marking can serve as a vehicle for thought. Importantly, this vehicle is less complex than the version of the same phrase danced ‘full-out’. After providing evidence for distinguishing different types of marking, three ways of understanding marking as a form of thought are considered: marking as a gestural language for encoding aspects of a target movement, marking as a method of priming neural systems involved in the target movement, and marking as a method for improving the precision of mentally projecting aspects of the target.

The objective of research in the foundations of Al is to explore such basic questions as: What is a theory in Al? What are the most abstract assumptions underlying the competing visions of intelligence? What are the basic arguments for and against each assumption? In this essay I discuss five foundational issues: (1) Core Al is the study of conceptualization and should begin with knowledge level theories. (2) Cognition can be studied as a disembodied process without solving the symbol grounding problem. (3) Cognition is nicely described in propositional terms. (4) We can study cognition separately from learning. (5) There is a single architecture underlying virtually all cognition. I explain what each of these implies and present arguments from both outside and inside Al why each has been seen as right or wrong.

Contemporary choreography offers a window onto creative processes that rely on harnessing the power of sensory sys- tems. Dancers use their body as a thing to think with and their sensory systems as engines to simulate ideas non- propositionally. We report here on an initial analysis of data collected in a lengthy ethnographic study of the making of a dance by a major choreographer and show how translating between different sensory modalities can help dancers and choreographer to be more creative.

Mechanisms for QoS provisioning in communication networks range from flow-based resource reservation schemes, providing QoS guarantees, through QoS differentiation based on reservation aggregation techniques to adaptation of applications, compensating for incomplete reservations. Scalable, aggregation-based reservations can also be combined with adaptations for a more flexible and robust overall QoS provisioning. Adaptation is particularly important in wireless networks, where reservations schemes are more difficult to realize. It is widely accepted that usability of Cellular or Mobile IP can be largely improved if adaptation is incorporated. Adaptation, in turn, depends on coding techniques involved and on user requirements. Those impose certain preferences and ordering relations on QoS hierarchies or degradation paths to be followed along when adaptation is performed. With the advent of MPEG4 coding techniques, more sophisticated adaptation schemes are more likely to be supported in the future.