Wim Pouw

Iconicity is a term used in cognitive science and gesture studies to denote an informative relation between the form of an utterance and the meaning of that utterance. With good design, the form of an utterance can directly invite a suitable perceiver with a certain degree of initiation to iconically grasp a meaning in the right direction. Despite the now increasingly touted importance of iconicity in understanding human languages, it proves difficult to define more formally. When the term is defined, researchers tend to base iconicity on resemblances, such that A is iconic of B, if A resembles B in some relevant respect. In the philosophy of depiction fundamental issues have been raised against resemblance-based accounts. Even when barring such metaphysical issues, it has recently been argued that - for all practical research purposes - a 'state-of-the-art' definition of iconicity should also do away with objective resemblances. Instead, iconicity is in the eye of the beholder. This revision of iconicity seems to reflect a philosophical struggle between two extremes: Iconicity as a property of the mind, or iconicity as a property of the environment. In this paper, it is argued that - for all practical purposes - that there is an alternative relational ontology available to us, which is informed broadly by 4E approaches (embodied, embedded, enactive, extended). Taking this road should lead to a 'distributed' view, where iconicity arises in a niche-constructed organism-environment system. This paper aims to further deepen this account by offering a practical philosophical ontology and working definition that keeps iconicity verifiable and context-dependent without reducing it to either objective resemblances or subjective mental states.

In music performance contexts, vocalists tend to gesture with hand and upper body movements as they sing. But how does this gesturing relate to the sung phrases, and how do singers’ gesturing styles differ from each other? In this study, we present a quantitative analysis and visualization pipeline that characterizes the multidimensional co‐structuring of body movements and vocalizations in vocal performers. We apply this to a dataset of performances within the Karnatak music tradition of South India, including audio and motion tracking data of 44 performances by three expert Karnatak vocalists, openly published with this report. Our results show that time‐varying features of head and hand gestures tend to be more similar when the concurrent vocal time‐varying features are also more similar. While for each performer we find clear co‐structuring of sound and movement, they each show their own characteristic salient dimensions (e.g., hand position, head acceleration) through which movement co‐structures with singing. Our time‐series analyses thereby provide a computational approach to characterizing individual vocalists’ unique multimodal vocal‐gesture co‐structuring profiles. We also show that co‐structuring clearly reduces degrees of freedom of the multimodal performance such that motifs that sound alike tend to co‐structure with gestures that move alike. The current method can be applied to any multimodally ensembled signals in both human and nonhuman communication, to determine co‐structuring profiles and explore any reduction in degrees of freedom. In the context of Karnatak singing performance, the current analysis is an important starting point for further experimental study of gesture‐vocal synergies.

Speakers often use gesture to demonstrate how to perform actions—for example, they might show how to open the top of a jar by making a twisting motion above the jar. Yet it is unclear whether listeners learn as much from seeing such gestures as they learn from seeing actions that physically change the position of objects (i.e., actually opening the jar). Here, we examined participants' implicit and explicit understanding about a series of movements that demonstrated how to move a set of objects. The movements were either shown with actions that physically relocated each object or with gestures that represented the relocation without touching the objects. Further, the end location that was indicated for each object covaried with whether the object was grasped with one or two hands. We found that memory for the end location of each object was better after seeing the physical relocation of the objects, that is, after seeing action, than after seeing gesture, regardless of whether speech was absent (Experiment 1) or present (Experiment 2). However, gesture and action built similar implicit understanding of how a particular handgrasp corresponded with a particular end location. Although gestures miss the benefit of showing the end state of objects that have been acted upon, the data show that gestures are as good as action in building knowledge of how to perform an action.

Silent gestures consist of complex multi‐articulatory movements but are now primarily studied through categorical coding of the referential gesture content. The relation of categorical linguistic content with continuous kinematics is therefore poorly understood. Here, we reanalyzed the video data from a gestural evolution experiment (Motamedi, Schouwstra, Smith, Culbertson, & Kirby, 2019), which showed increases in the systematicity of gesture content over time. We applied computer vision techniques to quantify the kinematics of the original data. Our kinematic analyses demonstrated that gestures become more efficient and less complex in their kinematics over generations of learners. We further detect the systematicity of gesture form on the level of thegesture kinematic interrelations, which directly scales with the systematicity obtained on semantic coding of the gestures. Thus, from continuous kinematics alone, we can tap into linguistic aspects that were previously only approachable through categorical coding of meaning. Finally, going beyond issues of systematicity, we show how unique gesture kinematic dialects emerged over generations as isolated chains of participants gradually diverged over iterations from other chains. We, thereby, conclude that gestures can come to embody the linguistic system at the level of interrelationships between communicative tokens, which should calibrate our theories about form and linguistic content.