I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialogue (2025)

Abstract: In face-to-face interaction, we use multiple modalities, including speech and gestures, to communicate information and resolve references to objects. However, how representational co-speech gestures refer to objects remains understudied from a computational perspective. In this work, we address this gap by introducing a multimodal reference resolution task centred on representational gestures, while simultaneously tackling the challenge of learning robust gesture embeddings. We propose a self-supervised pre-training approach to gesture representation learning that grounds body movements in spoken language. Our experiments show that the learned embeddings align with expert annotations and have significant predictive power. Moreover, reference resolution accuracy further improves when (1) using multimodal gesture representations, even when speech is unavailable at inference time, and (2) leveraging dialogue history. Overall, our findings highlight the complementary roles of gesture and speech in reference resolution, offering a step towards more naturalistic models of human-machine interaction.

## introduction referring to objects is common in everyday communication. in face-to-face interaction, when we need to collaborate on new tasks or refer to new objects, we rely on verbal (i.e., speech) and nonverbal (e.g., gestures and gaze) cues to describe salient object features and direct the listener's attention. among the non-verbal cues are representational co-speech gestures, i.e., iconic hand movements semantically and pragmatically related to co-occurring speech @xcite . studies have shown that representational gestures facilitate language comprehension @xcite and help listeners identify referents more quickly than speech alone @xcite there is a circle on the front" figure 1 : example from the cabb dataset @xcite , illustrating how participants resolve references through speech and gestures in face-to-face dialogue. the speaker on the right says "there is a circle on the front" while performing a representational gesture that resembles a circle. the object discussed is shown for illustration but not visible to the listener; the orange arrow points to the referent as annotated by experts. our work draws on these interactions to model multimodal reference resolution. @xcite . along with speech, gestures are used to refer to novel objects and build shared understanding @xcite , as shown in figure 1 . these multimodal abilities are inherent ingredients of our communicative interactions @xcite . hence, developing computational approaches that can interpret such cues is important for naturalistic humanmachine collaboration: in situated dialogue, an artificial agent must recognize multimodal inputs and speaker references to meet human needs @xcite . however, how to computationally represent and interpret co-speech gestures remains an understudied problem, particularly within natural language processing. other gestural forms, such as deictic gestures, i.e., pointing @xcite @xcite or beat gestures, i.e., rhythmic movements without semantic content (e.g., @xcite , have received some attention. in contrast, the challenges posed by iconic, representational gestures and their contribution to reference resolution have hardly been tackled from a data-driven perspective. in this paper, we propose an approach to learning embeddings for representational gestures that exploits not only the body movements that make up a gesture but also the semantically related speech that is typically produced simultaneously with it. we combine contemporary self-supervised learning techniques to train a transformer-based gesture encoder and ground it in information from features extracted by text or speech large language models. then, we test the effectiveness of the pre-trained gesture embeddings in the downstream task of reference resolution in face-to-face dialogue, showing that gestures-as learned by our proposed multimodal approach-have significant predictive power that complements the verbal modality. more concretely, we make the following contributions: ## data for our study, we use the cabb dataset @xcite , which consists of face-to-face conversations in dutch between two dialogue participants who play a reference game. the setup is shown in figure 1 . the participants' task is to identify 16 objects without conventional names that are made up of different geometrical parts (see appendix a). each dyad plays the game for six rounds, exchanging the roles of 'director' (who describes one of the target objects) and 'matcher' (who attempts to identify the director's intended referent among the 16 candidate objects displayed on a screen). the participants are free to communicate as they like, which elicits spontaneous speech and gestures. speakers were video recorded from different angles and we make use of the semifrontal views shown in figure 1 , as well as the audio recordings. we use two different subsets of this data, which we refer to as cabb-s and cabb-l, plus an extension of the latter which we call cabb-xl: cabb-s @xcite consists of 19 dialogues by 38 individuals, corresponding to over 8 hours of recordings. the dataset includes manual speech transcriptions and manual segmentation of gesture strokes, with 4949 gesture segments in total. approximately 97% of these segments are accompanied by concurrent speech. cabb-s also includes manual annotations of gesture strokes with two types of information: foot_0 • referent: the object subpart referred to by the gesture. the candidate objects and their sub-parts are shown in appendix a, figure 9 . cabb-l @xcite ) contains an additional 49 dialogues by 98 different subjects, with about 42.5 hours of recordings. it is therefore much larger than cabb-s. only 42 dialogues are manually transcribed and no manual annotations regarding gestures are present. to identify gestures, we use the segmentation model by @xcite , which has been shown to achieve a mean average precision (map) of 76% on the cabb-s dataset. applying this model to cabb-l results in 30k automatically segmented gestures. 3 to increase the amount of data available for pretraining, we oversample by selecting 1-second time windows overlapping more than 50% with the automatically segmented gestures. this results in approximately 400k data samples, which we refer to as cabb-xl. we use whisper-x @xcite to automatically generate speech transcriptions when manual transcriptions are unavailable. 83% of the gestures are accompanied by speech. we use cabb-l and cabb-xl for pre-training the models introduced in section 4. pre-processing to process body movements, we apply the procedure used by @xcite to cabb-s, cabb-l, and cabb-xl. concretely, we sample 1-second time windows centered around each segmented gesture and use vitpose @xcite to extract skeletal information, i.e., 2d keypoint coordinates for 27 upper body and hand joints. @xcite showed that related speech is often produced before or after gesture strokes (the most meaning-bearing segment of a gesture) by a few hundred milliseconds. to account for this temporal asynchrony between speech and gestures, when processing the verbal modality we extract 2-second windows centered around the sampled gestures, and use both the raw speech and the transcriptions as described in the next section. ## gesture representation learning in this section, we present our approach to learning robust gesture representations in a self-supervised fashion. to do so in the context of multimodal communication, we experiment with three types of input: gestures themselves (i.e., skeletal information corresponding to body movement), raw speech, and text-based semantics (section 4.1). we propose three pre-training model architectures that use these input types to different degrees and with different multimodal integration strategies (section 4.2). we train these models on cabb-l/xl and evaluate them against expert annotations using the unseen gestures in cabb-s (section 4.3). in section 5, we then test the effectiveness of our pre-training approach for the task of reference resolution. ## reference resolution an important question in situated interactions is to what extent representational gestures complement or supplement speech in reference resolution. here we shed light on this question by leveraging our pretrained gesture models for the downstream task of reference resolution. we investigate whether models that have learned gestures by exploiting multimodal information (from body movements and concurrent speech) have more predictive power than models that represent gestures exclusively from body movements. moreover, we test whether gestures contribute complementary information to the verbal modality when identifying referents. ## conclusion in this work, we have studied representational cospeech gestures in collaborative dialogue, using an existing dataset of face-to-face interactions collected by cognitive scientists. we introduced a novel reference resolution task formulated as the problem of identifying the intended referent of a cospeech gesture, while addressing key challenges in gesture representation learning. we proposed a selfsupervised transformer-based approach to learning pre-trained gesture embeddings by jointly exploiting skeletal information and concurrent language encoded with text or speech large language models. our experiments showed that the resulting gesture embeddings effectively contribute to reference resolution. representing gestures by exclusively exploiting skeletal information has significant predictive power, and grounding body movements in concurrent speech during pre-training further improves resolution accuracy, even when speech is not provided at test time. an interesting avenue for future work could be to ground the reference resolution models in the visual properties of the referents, in order to learn mappings between iconic gestures and the objects they represent, which might lead to further improvements. moreover, we showed that reference resolution from representational gestures can benefit from having access to gestures previously used within a dialogue, thus providing empirical support for the presence of gestural entrainment in face-to-face interaction. taken together, our findings emphasize the multimodal character of conversation @xcite and the importance of capturing the complementarity between gestures and speech in naturalistic human-machine interaction. further work is needed to test the extent to which the proposed pre-training approach would transfer to other referential domains and other tasks-a step we leave to future research.