Spontaneous gestures encoded by hand positions improve language models: An Information-Theoretic motivated study (2023)

Abstract: The multi-modality nature of human communication has been utilized to enhance the performance of language modeling-related tasks. Driven by the development of large-scale end-to-end learning techniques and the availability of multi-modal data, it becomes possible to represent non-verbal communication behaviors through joint-learning, and directly study their interaction with verbal communication. However, there is still gaps in existing studies to better address the underlying mechanism of how non-verbal expression contributes to the overall communication purpose. Therefore, we explore two questions using mixed-modal language models trained against monologue video data: first, whether incorporating gesture representations can improve the language model’s performance (perplexity); second, whether spontaneous gestures demonstrate entropy rate constancy (ERC), which is an empirical pattern found in most verbal language data that supports the rational communication assumption from Information Theory. We have positive and interesting findings for both questions: speakers indeed use spontaneous gestures to convey “meaningful” information that enhances verbal communication, which can be captured with a simple spatial encoding scheme. More importantly, gestures are produced and organized rationally in a similar way as words, which optimizes the communication efficiency.

## introduction human communication is a multi-modal process where both verbal and non-verbal information are expressed simultaneously. this is true in various forms of communication, one-way (speech) or twoway (conversation). it has been revealed in empirical studies that speakers' expression in the visual modality, including gestures, body poses, eye contacts and other types of non-verbal behaviors, play critical roles in face-to-face communication, as they add subtle information that is hard to con-vey in verbal language. it is becoming an emerging sub-area in computational linguistics. however, whether and to what degrees these sparse and random non-verbal signals can be treated as a formal communication channel that transmits "serious" information remains a seldom-validated question, especially with computational methods. we believe a key missing step is to explore whether the nonverbal information can be quantified. the questions that are worth further investigation include (but are not limited to): how rich is the information contained in these non-verbal channels? what are their relationships to verbal information? can we understand the meanings of different gestures, poses, and motions embedded in spontaneous language in a similar way to understanding word meanings? the goal of this study is to propose a simple but straightforward framework to approach the above questions, under the guidance of information theory. some preliminary, yet prospective results are presented. the code and data for this study is published in this repository https://github. com/innerfirexy/life-lessons . ## questions and hypotheses we examine two hypotheses in this study: hypothesis 1: incorporating non-verbal representations as input will improve the performance of language modeling tasks. to test hypothesis 1, we extract non-verbal representations using the output from pose estimation, and then compose discrete tokens to represent the non-verbal information. the non-verbal tokens are inserted into word sequences and form a hybrid type of input data for training language models. the language models are modified to take non-verbal and verbal input sequences simultaneously and compute a fused internal representation. we expect the inclusion of non-verbal information will increase the performance of language models measured by perplexity. ## conclusions our main conclusions are two-fold: first, incorporating gestural features will significantly improve the performance of language modeling tasks, even when gestures are represented with a simplistic method. second, the way gestures are used as a complementary non-verbal communication sidechannel follows the principle of entropy rate constancy (erc) in information theory. it means that the information encoded in hand gestures, albeit subtle, is actually organized in a rational way that enhances the decoding/understanding of information from a receiver's perspective. this is the first work done, to the best of our knowledge, to extend the scope of erc to non-verbal communication. the conclusions are based on empirical results from multi-modal language models trained on monologue speech videos with gesture information represented by discrete tokens. there are two explanations for what causes the observed pattern of increasing entropy: first, more rare gestures (higher entropy) near the later stage of communication; second, the entropy for the same gesture also increases during the communication. while the latter indicates a more sophisticated and interesting theory about gesture usage, both explanations require further investigation. this work is exploratory, but the evidence is promising, as only a small dataset is used, and a simplistic gesture representation method is applied. for future work, we plan to work with a larger and more diverse dataset with a higher variety in genres (public speech, etc.) and examine more advanced representation methods, such as continuous embedding and clustering. another direction to pursue is to interpret the semantic meanings of gestures and other non-verbal features by examining their semantic distance from utterances in vector space. more specifically, non-parametric clustering algorithms can be useful to identify distinct dynamic actions, which provides a different way to extract non-verbal representations.