Gestures Are Used Rationally: Information Theoretic Evidence from Neural Sequential Models (2022)

Abstract: Verbal communication is companied by rich non-verbal signals. The usage of gestures, poses, and facial expressions facilitates the information transmission in verbal channel. However, few computational studies have explored the non-verbal channels with finer theoretical lens. We extract gesture representations from monologue video data and train neural sequential models, in order to study the degree to which non-verbal signals can effectively transmit information. We focus on examining whether the gestures demonstrate the similar pattern of entropy rate constancy (ERC) found in words, as predicted by Information Theory. Positive results are shown to support the assumption, which leads to the conclusion that speakers indeed use simple gestures to convey information that enhances verbal communication, and the production of non-verbal information is rationally organized.

## introduction communication is a multi-modal process, in which information from verbal and non-verbal modalities are mixed into one channel. it has been revealed from a long history of empirical studies that speakers' expression in visual modality, including gestures, body poses, eye contacts and other types of non-verbal behaviors, play critical roles in face-toface communication, as they add subtle information that is hard to convey in verbal language @xcite . however, it remains an untested idea to view these sparse and random non-verbal signals as a formal communication channel that transmits "serious" information, which has seldom been validated by computational studies. a key missing step is to explore whether the non-verbal information can be quantified. the questions that are worth further investigation include: how rich is the information contained in these non-verbal channels? what are their relationships to verbal information? can we understand the meanings of different gestures, poses, and motions embedded in spontaneous language in a similar way to understanding word meanings? the goal of this study is to take a simple yet necessary first step approaching the above questions, by examining a basic information theoretic property of gestures that comes along with verbal language. some preliminary but prospective results are presented. ## question and hypothesis our hypothesis is: non-verbal communication also conforms to the principle of erc. to test it, we approximate the local entropy (h(s|l)) of nonverbal "tokens" using the perplexity scores obtained from neural sequential models, and correlate it with the utterances' relative positions within the monologue data. if we can find that h(s|l) increases with utterance position, is similar to verbal language, then it supports the hypothesis. ## discussion and conclusions our results confirms that the way gestures are used as a complementary non-verbal communication side-channel follows the principle of entropy rate constancy (erc) in information theory. it means that the information encoded in hand gestures, albeit subtle, is actually organized in a rational way that enhances the decoding/understanding of information from a receiver's perspective. the main contribution is that we extend the scope of erc to realm of non-verbal communication. there are two explanations for what causes the observed entropy increasing pattern: first, more rare gestures (higher entropy) near the later stage of communication; second, the entropy for the same gesture also increases during the communication. while the latter indicates a more sophisticated and interesting theory about gesture usage, both explanations requires further investigation. and other implicit messages. for future work, we plan to use a larger dataset with a higher variety in genres (public speech, etc.) and examine more advanced representation method. such as continuous embedding and clustering. it is also interesting to interpret the semantic meanings of gestures and other non-verbal features by examining their semantic distance from words/utterances in vector space. more specifically, non-parametric clustering algorithms can be used to identify distinct the actions or poses of a person, which provides a way to extract more general gesture/action/pose labels for training.