Towards Understanding the Relation between Gestures and Language (2022)

Abstract: In this paper, we explore the relation between gestures and language. Using a multimodal dataset, consisting of Ted talks where the language is aligned with the gestures made by the speakers, we adapt a semi-supervised multimodal model to learn gesture embeddings. We show that gestures are predictive of the native language of the speaker, and that gesture embeddings further improve language prediction result. In addition, gesture embeddings might contain some linguistic information, as we show by probing embeddings for psycholinguistic categories. Finally, we analyze the words that lead to the most expressive gestures and find that function words drive the expressiveness of gestures.

## introduction gestures are often referred to as "non-verbal language" and extensive studies in psychology, sociology, and anthropology have demonstrated the important role they play in communication @xcite @xcite . while language and gesture can occur independently, people often use them together to communicate, suggesting that multimodality plays an important role in understanding gestures. in this work, we consider human gestures together with their corresponding utterances. we jointly learn gesture and word embeddings, and attempt to predict psycholinguistic categories and the language of the speaker from their gesture embeddings. even for humans it is very challenging to predict words from gestures alone (or vice versa), due to the many-to-many relationship between words and gestures. therefore, instead of directly predicting one modality from the other @xcite , we use contrastive pre-training learning to learn a joint embedding space that aligns both modalities @xcite @xcite . this allows our model to learn an association between language and gestures, despite a large amount of uncertainty inherent in the task. the main contributions of this work are as follows: ## data our primary source of data is the youtube gesture dataset @xcite . the dataset consists of over 1,500 ted talk videos of english speakers addressing various topics like science, medicine, society, and others. the camera is usually in front of the speaker, so the gestures are visible. the dataset contains precomputed key points for the head, neck, shoulders, elbows, and wrists, with each pose represented as a 16-dimension vec- radio can help stimulate interest and demand ... ... by playing kenyan music done in english, kiswahili ... voy a hablar de infarto cardíaco ... un hombre entra a una guardia con un infarto ... tor, with x and y coordinates for each key point. the dataset includes subtitles, auto-generated by youtube and aligned by gentle foot_0 .we also use an additional dataset that we compiled ourselves, consisting of 600 videos of spanish speakers. we process videos from the tedx channel using the playlist "tedx talks en español." subtitles for spanish data are auto-generated and aligned by youtube, so we just download the appropriate subtitle file. figure 1 shows an example of gestures aligned with the corresponding language. we split each video into several clips using pyscenedetect, foot_1 which detects changes in the camera angle during the talk and splits a single video into several sub-clips. this is necessary so that the pose movements remain continuous during the short clips, even if the camera angle is changed. ## experiments and results to evaluate the strength of the connection between gestures and language, we perform two types of experiments. first, we perform experiments within one single language only, i.e. how gestures and language interact in either english or spanish. the second type is cross-language, how gestures are different among english and spanish speakers. ## conclusions and lessons learned in this paper, we explored the relation between gestures and language. using a clip-style joint embedding model for gestures and language, applied on a bilingual multimodal dataset consisting of ted talks in english and spanish, we report several findings: first, we found that gestures can be used to infer the corresponding language and conversely that language can be used to infer the corresponding gesture. our proposed model can predict the matching between language and gesture with 65.4% accuracy, compared to the random 50.0% baseline. 7 : accuracy on the language prediction task for raw poses and gesture embeddings unigrams si, ade, y, asking, mas, here, life, po, medicine, ti bigrams ade mas, from the, so that, and it, is to, y con, if your, we just, we can, we 've second, we showed that it is possible to predict several social or psycholinguistic word categories from the gestures with better than random probability. through extensive probing of gesture embeddings for liwc and general inquirer linguistic categories, we were able to identify the categories where gesture embeddings significantly outperform random baselines: the majority of these categories consist of function words, which is a finding that aligns with previous social science findings. we report the results separately for english and spanish. in a similar vein, we showed that gestures can be also predictive of discourse markers. our results indicate that gesture embeddings contain useful information about discourse structure, outperforming both majority and joint-only baselines. finally, we reported that gestures by themselves are predictive of the native language of the speaker, and that gesture embeddings further improve this result. through several analyses, we found that function words are most strongly associated with gestures, which aligns with theories of language evolution that posit that function words are closely connected to the body. there are several limitations of this work. first, this work focuses on hands and head gestures only, ignoring whole body movements and facial expressions. we also assume that two active hands/arms perform a single gesture, while it is also possible to have two separate gestures for two hands. the gestures are also specific for public presentations. for future work, we plan to include hand gestures in our dataset. we also consider compiling an 'in-the-wild' gestures dataset, to extend our findings to more forms of communication, and expand beyond the gestures and language for ted talks.