David Kirsh

We present data and argument to show that in Tetris - a real-time interactive video game - certain cognitive and perceptual problems are more quickly, easily, and reliably solved by performing actions in the world rather than by performing computational actions in the head alone. We have found that some translations and rotations are best understood as using the world to improve cognition. These actions are not used to implement a plan, or to implement a reaction; they are used to change the world in order to simplify the problem-solving task. Thus, we distinguish pragmatic actions ñ actions performed to bring one physically closer to a goal - from epistemic actions - actions performed to uncover information that is hidden or hard to compute mentally. To illustrate the need for epistemic actions, we first develop a standard information-processing model of Tetris-cognition, and show that it cannot explain performance data from human players of the game - even when we relax the assumption of fully sequential processing. Standard models disregard many actions taken by players because they appear unmotivated or superfluous. However, we describe many such actions that are actually taken by players that are far from superfluous, and that play valuable roles in improving human performance. We argue that traditional accounts are limited because they regard action as having a single function: to change the world. By recognizing a second function of action - an epistemic function - we can explain many of the actions that a traditional model cannot. Although, our argument is supported by numerous examples specifically from Tetris, we outline how the one category of epistemic action can be incorporated into theories of action more generally.

Architecture is about to enter its first magical phase: a time when buildings actively co-operate with their inhabitants; when objects know what they are, where they are, what is near them; when social and physical space lose their type coupling; when wall and partitions change with mood and task. As engineers and scientists explore how to digitse the world around us, the classical constraints of design, ruled so long by the physics of space, time, and materials, are starting to crumble. Documents can be laid down in one place, automatically cloned, and a copy picked up in another. Meetings scheduled for 9am to 10am can be joined by latecomers at noon, who then participate in a captured form of the event and are ‘edited into’ the past. People on the West coast of the USA can participate, in a telepresent way, with their colleagues on the East coast, and hold a meeting against a virtual backdrop, such as a production line in their Taiwanese factory. Walls seem to dematerialise, remote objects can be touched virtually, shaped, passed through one another. Technology is moving inexorably so that being in one place at a time no longer need dominate how we work and play. Material boundaries are losing their meaning, and interface and information space are catch words that architects must master. In this article I will discuss some of the theoretical ideas shaping our new conception of form, function, and interactivity. My view is that of a cognitive scientist interested in how cognition is distributed throughout our environment. Since the ground rules defining the structure of environments are changing, our very idea of how we are embedded in the world is changing. Architecture is at a new frontier.

A complementary strategy can be defined as any organizing activity which recruits external elements to reduce cognitive loads. Typical organizing activities include pointing, arranging the position and orientation of nearby objects, writing things down, manipulating counters, rulers or other artifacts that can encode the state of a process or simplify perception. To illustrate the idea of a complementary strategy, a simple experiment was performed in which subjects were asked to determine the dollar value of collections of coins. In the no-hands condition, subjects were not allowed to touch the coin images or to move their hands in any way. In the hands condition, they were allowed to use their hands and fingers however they liked. Significant improvements in time and number of errors were observed when S’s used their hands over when they did not. To explain these facts, a brief account of some commonly observed complementary strategies is presented, and an account of their potential benefits to perception, memory and attention.

The theory of embodied cognition can provide HCI practitioners and theorists with new ideas about interac-tion and new principles for better designs. I support this claim with four ideas about cognition: (1) interacting with tools changes the way we think and perceive – tools, when manipulated, are soon absorbed into the body schema, and this absorption leads to fundamental changes in the way we perceive and conceive of our environments; (2) we think with our bodies not just with our brains; (3) we know more by doing than by seeing – there are times when physically performing an activity is better than watching someone else perform the activity, even though our motor resonance system fires strongly during other person observa-tion; (4) there are times when we literally think with things. These four ideas have major implications for interaction design, especially the design of tangible, physical, context aware, and telepresence systems.

A startling amount of intelligent activity can be controlled without reasoning or thought. By tuning the perceptual system to task relevant properties a creature can cope with relatively sophisticated environments without concepts. There is a limit, however, to how far a creature without concepts can go. Rod Brooks, like many ecologically oriented scientists, argues that the vast majority of intelligent behaviour is concept-free. To evaluate this position I consider what special benefits accrue to concept-using creatures. Concepts are either necessary for certain types of perception, learning, and control, or they make those processes computationally simpler. Once a creature has concepts its capacities are vastly multiplied.

To explore the question of physical thinking – using the body as an instrument of cognition – we collected extensive video and interview data on the creative process of a noted choreographer and his company as they made a new dance. A striking case of physical thinking is found in the phenomenon of marking. Marking refers to dancing a phrase in a less than complete manner. Dancers mark to save energy. But they also mark to explore the tempo of a phrase, or its movement sequence, or the intention behind it. Because of its representational nature, marking can serve as a vehicle for thought. Importantly, this vehicle is less complex than the version of the same phrase danced ‘full-out’. After providing evidence for distinguishing different types of marking, three ways of understanding marking as a form of thought are considered: marking as a gestural language for encoding aspects of a target movement, marking as a method of priming neural systems involved in the target movement, and marking as a method for improving the precision of mentally projecting aspects of the target.

It is sometimes argued that if PDP networks can be trained to make correct judgements of grammaticality we have an existence proof that there is enough information in the stimulus to permit learning grammar by inductive means alone. This seems inconsistent superficially with Gold's theorem and at a deeper level with the fact that networks are designed on the basis of assumptions about the domain of the function to be learned. To clarify the issue I consider what we should learn from Gold's theorem, then go on to inquire into what it means to say that knowledge is domain specific. I first try sharpening the intuitive notion of domain specific knowledge by reviewing the alleged difference between processing limitations due to shortage of resources vs shortages of knowledge. After rejecting different formulations of this idea, I suggest that a model is language specific if it transparently refer to entities and facts about language as opposed to entities and facts of more general mathematical domains. This is a useful but not necessary condition. I then suggest that a theory is domain specific if it belongs to a model family which is attuned in a law-like way to domain regularities. This leads to a comparison of PDP and parameter setting models of language learning. I conclude with a novel version of the poverty of stimulus argument.

In the course of daily life we solve problems often enough that there is a special term to characterize the activity and the right to expect a scientific theory to explain its dynamics. The classical view in psychology is that to solve a problem a subject must frame it by creating an internal representation of the problem‘s structure, usually called a problem space. This space is an internally generable representation that is mathematically identical to a graph structure with nodes and links. The nodes can be annotated with useful information, and the whole representation can be distributed over internal and external structures such as symbolic notations on paper or diagrams. If the representation is distributed across internal and external structures the subject must be able to keep track of activity in the distributed structure. Problem solving proceeds as the subject works from an initial state in this mentally supported space, actively construction possible solution paths, evaluating them and heuristically choosing the best. Control of this exploratory process is not well understood, as it is not always systematic, but various heuristic search algorithms have been proposed and some experimental support has been provided for them.

An experiment was performed to test the hypothesis that people sometimes take physical actions to make themselves more effective problem solvers. The task was to generate all possible words that could be formed from seven Scrabble letters. In one condition, participants could use their hands to manipulate the letters, and in another condition, they could not. Results show that more words were generated with physical manipulation than without. However, an interaction was obtained between the physical manipulation conditions and the specific letter sets chosen, indicating that physical manipulation helps more for generating words in some circumstances than in others. Overall, our findings can be explained in terms of an interactive search process in which external, physical activity effectively complements internal, cognitive activity. Within this framework, the interaction can be explained in terms of the relative difficulty of generating words from the letters given in the different sets.

In the scientific study of mind a distinction is drawn between explicit knowledge–knowledge that can be elicited from a subject by suitable inquiry or prompting, can be brought to consciousness, and externally expressed in words–and implicit knowledge–knowledge that cannot be elicited, cannot be made directly conscious, and cannot be articulated. Michael Polanyi (1967) argued that we usually ‘know more than we can say’. The part we can articulate is explicitly known; the part we cannot is implicit.

This paper introduces the concept of Adaptive Rooms, which are virtual environments able to dynamically adapt to users’ needs, including ‘physical’ and cognitive workflow requirements, number of users, differing cognitive abilities and skills. Adaptive rooms are collections of virtual objects, many of them self-transforming objects, housed in an architecturally active room with information spaces and tools. An ontology of objects used in adap- tive rooms is presented. Virtual entities are classified as passive, reactive, ac- tive, and information entities, and their sub-categories. Only active objects can be self-transforming. Adaptive Rooms are meant to combine the insights of ubiquitous computing -- that computerization should be everywhere, transpar- ently incorporated -- with the insights of augmented reality -- that everyday ob- jects can be digitally enhanced to carry more information about their use. To display the special potential of adaptive rooms, concrete examples are given to show how the demands of cognitive workflow can be reduced.

Virtual environment landmarks are essential in wayfinding: they anchor routes through a region and provide memorable destinations to return to later. Current virtual environment browsers provide user interface menus that characterize available travel destinations via landmark textual descriptions or thumbnail images. Such characterizations lack the depth cues and context needed to reliably recognize 3D landmarks. This paper introduces a new user interface affordance that captures a 3D representation of a virtual environment landmark into a 3D thumbnail, called a worldlet. Each worldlet is a miniature virtual world fragment that may be interactively viewed in 3D, enabling a traveler to gain first-person experience with a travel destination. In a pilot student conducted to compare textual, image, and worldlet landmark representations within a wayfinding task, worldlet use significantly reduced the overall travel time and distance traversed, virtually eliminating unnecessary backtracking.

Architecture is about to enter its first magical phase: a time when buildings actively co-operate with their inhabitants; when objects know what they are, where they are, what is near them; when social and physical space lose their type coupling; when wall and partitions change with mood and task. As engineers and scientists explore how to digitse the world around us, the classical constraints of design, ruled so long by the physics of space, time, and materials, are starting to crumble. Documents can be laid down in one place, automatically cloned, and a copy picked up in another. Meetings scheduled for 9am to 10am can be joined by latecomers at noon, who then participate in a captured form of the event and are ‘edited into’ the past. People on the West coast of the USA can participate, in a telepresent way, with their colleagues on the East coast, and hold a meeting against a virtual backdrop, such as a production line in their Taiwanese factory. Walls seem to dematerialise, remote objects can be touched virtually, shaped, passed through one another. Technology is moving inexorably so that being in one place at a time no longer need dominate how we work and play. Material boundaries are losing their meaning, and interface and information space are catch words that architects must master.

Individual creativity is standardly treated as an ‘internalist’ process occurring solely in the head. An alternative, more interactionist view is presented here, where working with objects, media and other external things is seen as a fundamental component of creative thought. The value of chance interaction and chance cueing — practices widely used in the creative arts — is explored briefly in an account of the creative method of choreographer Wayne McGregor and then more narrowly in an experimental study that compared performance on a Scrabble-like word discovery problem. Subjects were presented with seven letters and given two minutes to call out three-to-seven-letter English words. There were three conditions: The tiles were fixed in place, subjects were free to move the tiles manually or the tiles could be randomly shuffled. Results showed that random shuffling was best, with manual movement second. Three reasons are provided: Shuffling is faster and cheaper than mentally thinking of candidates; randomizing strings covers the search space better than a deterministic method based on past successes; and randomizing is equivalent to adding diversity to a team, which is known empirically to lead to more creative solutions.

Using the perspective of situated cognition we studied how people interact with a physical map to help them navigate through an unfamiliar environment. The study used a mixture of cognitive ethnography and traditional experimental methods. We found that the difference between high and low performing navigators showed up in the speed they completed their task and also in the way they use maps. High performers plan routes using a survey method whereas low performers use a route strategy. We suggest that when people are given a task that does not match their cognitive style they try to transform the task to better suit their cognitive abilities and cognitive style.