David Kirsh

Using the perspective of situated cognition we studied how people interact with a physical map to help them navigate through an unfamiliar environment. The study used a mixture of cognitive ethnography and traditional experimental methods. We found that the difference between high and low performing navigators showed up in the speed they completed their task and also in the way they use maps. High performers plan routes using a survey method whereas low performers use a route strategy. We suggest that when people are given a task that does not match their cognitive style they try to transform the task to better suit their cognitive abilities and cognitive style.

Why do people create extra representations to help them make sense of situations, diagrams, illustrations, instructions and problems? The obvious explanation – external representations save internal memory and computation – is only part of the story. I discuss eight ways external representations enhance cognitive power: they provide a structure that can serve as a shareable object of thought; they create persistent referents; they change the cost structure of the inferential landscape; they facilitate re-representation; they are often a more natural representation of structure than mental representations; they facilitate the computation of more explicit encoding of information; they enable the construction of arbitrarily complex structure; and they lower the cost of controlling thought – they help coordinate thought.

Humans are closely coupled with their environments. They rely on being ‘embedded’ to help coordinate the use of their internal cognitive resources with external tools and resources. Consequently, everyday cognition, even cognition in the absence of others, may be viewed as partially distributed. As cognitive scientists our job is to discover and explain the principles governing this distribution: principles of coordination, externalization, and interaction. As designers our job is to use these principles, especially if they can be converted to metrics, in order to invent and evaluate candidate designs. After discussing a few principles of interaction and embedding I discuss the usefulness of a range of metrics derived from economics, computational complexity, and psychology.

This paper introduces the concept of Adaptive Rooms, which are virtual environments able to dynamically adapt to users’ needs, including ‘physical’ and cognitive workflow requirements, number of users, differing cognitive abilities and skills. Adaptive rooms are collections of virtual objects, many of them self-transforming objects, housed in an architecturally active room with information spaces and tools. An ontology of objects used in adap- tive rooms is presented. Virtual entities are classified as passive, reactive, ac- tive, and information entities, and their sub-categories. Only active objects can be self-transforming. Adaptive Rooms are meant to combine the insights of ubiquitous computing -- that computerization should be everywhere, transpar- ently incorporated -- with the insights of augmented reality -- that everyday ob- jects can be digitally enhanced to carry more information about their use. To display the special potential of adaptive rooms, concrete examples are given to show how the demands of cognitive workflow can be reduced.

The question of how to conceive and represent the context of work is explored from the theoretical perspective of distributed cognition. It is argued that to understand the office work context we need to go beyond tracking superficial physical attributes such as who or what is where and when and consider the state of digital resources, people’s concepts, task state, social relations, and the local work culture, to name a few. In analyzing an office more deeply, three concepts are especially helpful: entry points, action landscapes, and coordinating mechanisms. An entry point is a structure or cue that represents an invitation to enter an information space or office task. An activity landscape is part mental construct and part physical; it is the space users interactively construct out of the resources they find when trying to accomplish a task. A coordinating mechanism is an artifact, such as a schedule or clock, or an environmental structure such as the layout of papers to be signed, which helps a user manage the complexity of his task. Using these three concepts we can abstract away from many of the surface attributes of work context and define the deep structure of a setting—the invariant structure that many office settings share. A long-term challenge for context-aware computing is to operationalize these analytic concepts.

When people make sense of situations, illustrations, instructions and problems they do more than just think with their heads. They gesture, talk, point, annotate, make notes and so on. What extra do they get from interacting with their environment in this way? To study this fundamental problem, I looked at how people project structure onto geometric drawings, visual proofs, and games like tic tac toe. Two experiments were run to learn more about projection. Projection is a special capacity, similar to perception, but less tied to what is in the environment. Projection, unlike pure imagery, requires external structure to anchor it, but it adds ‘mental’ structure to the external scene much like an augmented reality system adds structure to an outside scene. A person projects when they look at a chessboard and can see where a knight may be moved. Because of the cognitive costs of sustaining and extending projection, humans make some of their projections real. They create structure externally. They move the piece, they talk, point, notate, represent. Much of our interactivity during sense making and problem solving involves a cycle of projecting then creating structure.

In the scientific study of mind a distinction is drawn between explicit knowledge— knowledge that can be elicited from a subject by suitable inquiry or prompting, can be brought to consciousness, and externally expressed in words—and implicit knowledge—knowledge that cannot be elicited, cannot be made directly conscious, and can- not be articulated. Michael Polanyi (1967) argued that we usually ‘know more than we can say’. The part we can articulate is explicitly known; the part we cannot is implicit.

Metacognition is associated with planning, monitoring, evaluating and repairing performance Designers of elearning systems can improve the quality of their environments by explicitly structuring the visual and interactive display of learning contexts to facilitate metacognition. Typically page layout, navigational appearance, visual and interactivity design are not viewed as major factors in metacognition. This is because metacognition tends to be interpreted as a process in the head, rather than an interactive one. It is argued here, that cognition and metacognition are part of a continuum and that both are highly interactive. The tenets of this view are explained by reviewing some of the core assumptions of the situated and distribute approach to cognition and then further elaborated by exploring the notions of active vision, visual complexity, affordance landscape and cue structure. The way visual cues are structured and the way interaction is designed can make an important difference in the ease and effectiveness of cognition and metacognition. Documents that make effective use of markers such as headings, callouts, italics can improve students' ability to comprehend documents and 'plan' the way they review and process content. Interaction can be designed to improve 'the proximal zone of planning' - the look ahead and apprehension of what is nearby in activity space that facilitates decisions. This final concept is elaborated in a discussion of how e-newspapers combine effective visual and interactive design to enhance user control over their reading experience.

Contemporary choreography offers a window onto creative processes that rely on harnessing the power of sensory sys- tems. Dancers use their body as a thing to think with and their sensory systems as engines to simulate ideas non- propositionally. We report here on an initial analysis of data collected in a lengthy ethnographic study of the making of a dance by a major choreographer and show how translating between different sensory modalities can help dancers and choreographer to be more creative.

The type of principles which cognitive engineers need to design better work environments are principles which explain interactivity and distributed cognition: how human agents interact with themselves and others, their work spaces, and the resources and constraints that populate those spaces. A first step in developing these principles is to clarify the fundamental concepts of environment, coordination, and behavioural function. Using simple examples, I review changes the distributed perspective forces on these basic notions.

Video data from three large captures of choreographic dance making was analyzed to determine if there is a difference between participant knowledge – the knowledge an agent acquires by being the cause of an action – and observer knowledge – the knowledge an observer acquires through close attention to someone else’s performance. The idea that there might be no difference has been challenged by recent findings about the action observation network and tacitly challenged by certain tenets in enactive perception. We explored why a choreographer ‘riff’s’ when appropriating and evaluating the movements of his dancers. By recruiting his body to help him cognize he is able to understand the possibilities of movement better than observation. He acquires participant knowledge.