How Much Does Nonverbal Communication Conform to Entropy Rate Constancy?: A Case Study on Listener Gaze in Interaction (2024)

Abstract: According to the Entropy Rate Constancy (ERC) principle, the information density of a text is approximately constant over its length. Whether this principle also applies to nonverbal communication signals is still under investigation. We perform empirical analyses of video-recorded dialogue data and investigate whether listener gaze, as an important nonverbal communication signal, adheres to the ERC principle. Results show (1) that the ERC principle holds for listener gaze; and (2) that the two linguistic factors syntactic complexity and turn transition potential are weakly correlated with local entropy of listener gaze.

## introduction in human communication, information is conveyed via various channels. in addition to conveying meaning through linguistic units, interlocutors may express themselves through multimodal signals such as hand gestures, head movements, or prosody. in face-to-face dialogue, interlocutors monitor signals from different modalities to decide when to take the turn, give feedback, or interject a brief response token @xcite . humans may also change their dialogue strategies based on their listeners' reactions, for example, repeating or explaining what has been said @xcite . by investigating such signals, we can gain insights about (1) how engaged dialogue participants are with the content of a conversation, or (2) how common ground @xcite and mutual understanding evolve over the course of an interaction. one important concept for measuring the growth of common ground is alignment of linguistic information @xcite . paralinguistic information such as prosody, or nonverbal information such as body posture or gaze direction, showing different levels of convergence during interaction, can serve as evidence of linguistic alignment @xcite . recent work introduces methods for investigating alignment of linguistic information through information theoretic measures such as entropy @xcite . it is widely argued that linguistic information is uniformly distributed in language use, e.g., in the uniform information density hypothesis (uid; @xcite , or the entropy rate constancy principle (erc; @xcite . @xcite 's analysis of spoken dialogue data shows that entropy of dialogue partners' speech, interactively evolves and converges as the discourse develops. this phenomenon indicates that mutual understanding has been consistently reached over the shifting of topics. @xcite shows that, while global entropy of dialogue remains constant, topic shift is a linguistic factor which leads to peaks in local entropy. inspired by @xcite and @xcite this paper focuses on listener gaze during interaction. in this paper, we quantify the information conveyed through gaze. based upon and extending our previous work @xcite , we first investigate gaze local entropy to add further evidence that non verbal communication conforms to the entropy rate constancy principle, complementing the finding of @xcite . our motivation for this is based on four points: (1) during interaction, interlocutors spend most of their time looking at each other @xcite ; (2) new gaze behaviour will emerge while the conversation @xcite (3) interlocutors are less likely to shift gaze from one corner of their visual field to the other when listening, as it can be perceived as being distracted (but see @xcite ; (4) if listener gaze targets the speaker, a gaze shift is less likely to happen, unless there is a cognitive processing-based need for gaze aversion @xcite . based on these four points, if we consider gaze as meaningful communicative units (similar to lexical units), listeners' gaze behavior should be predictable to some extent. we also take a closer look at two high level linguistic factors: syntactic complexity @xcite , and transition relevance places (potential turn-taking cues) @xcite in order to investigate if there is a correlation between gaze local entropy and these two linguistic factors. we do this as an initial attempt to explain why nonverbal communication (specifically listener gaze) may conform to the erc principle. in this paper we specifically investigate the following two research questions: the two main contributions of our work, compared to previous studies, are as follow: ## data collection nonverbal communication takes place in parallel to verbal communication @xcite . in order to control the variables in our study, we (roughly) control and constraint verbal information so that we can investigate the variation of nonverbal communication. the interactions we analyze are explanations of a board game (deep sea adventure; @xcite and were collected in an interaction study that resulted in the mundex corpus @xcite . in each interaction an experienced explainer explains the games to an explainee who does not know the it yet (see figure 1 ). the interactions are organized into three phases: in the first phase, the game is explained without the game material being present. in a second phase, the game is put on the table and the explanation continues. in the third phase the two participants start playing the game. the interactions can therefore be considered task-oriented dialogues. in this study we include the videos of 58 interactions from the corpus and extract the first phase (explanation without game). these explanations vary in length from 2:12 min to 17:36 min (mean length 7:04 min, standard deviation 3:15 min). all participants speak german. ## hypothesis our primary hypothesis is as follows: similar to verbal information, which is considered to conform to the entropy rate constancy (erc) principle, nonverbal communication signals such as the listener's gaze -which, similar to gestures, is usually coordinated with speech during interaction -also conform to it. the original hypothesis was posed in our previous work @xcite , but the finding was not conclusive. in this study, we revisit the original hypothesis by refining the experimental setting, e.g., by detecting and removing local outliers and partitioning the video recordings into several groups in a more reasonable way. we further test the erc principle in nonverbal communication by extending the primary hypothesis as follows: the erc principle in nonverbal communication in dialogue is based on the combination of multiple linguistic factors, of which we investigate two: syntactic complexity and turn transition potential of lexical items. ## processing listener gaze and dialogue in interaction for each recorded video from the corpus, we extract the explainee's gaze information using the openface framework @xcite . openface generates two types of gaze features (i) gaze direction values in radians averaged from both eyes (two dimensions) and (ii) gaze vectors in world coordinates (three dimensions for the left and right eye each). we integrate both representations so that the final gaze label does not only reflect gaze dynamics in three dimensions but also incorporates the effects from listener's head orientation during interaction. the computation of a gaze label follows these steps: tran et al., 2020), both horizontally and vertically. the motivation of finding the "dense region" is based on the intuition that interlocutors spend a large proportion of time during the conversation looking at each other. based on the minimal and maximal gaze direction values, a 3 × 3 grid is set up. the gaze direction values that are in this dense region are given the label '5', where the minimum and maximum gaze direction values among all gaze direction values in the dense region are used to decide on its boundary. the gaze direction values that are not in the dense region are assigned eight other number-based labels. the overall process follows the ice algorithm (tran et al., 2020) while the idea of using a 3 × 3 grid is based on xu et al. (2022) and xu and cheng (2023). figure 2 shows the distribution of the 15 most frequent gaze label in our data. the six most frequent labels (41, 37, 38 ,40, 42, 45) represent the explainees looking in the direction of explainers with various distance (closer or more distant). the less frequent labels (68, 50, . . . ) instead start to show explainees averting their gaze from explainers. dialogues were automatically transcribed using 'whisper' @xcite , which generates speech segments with a start and an end time. in order to calculate word timings, we approximated word onset times by calculating the duration of each speech segment, dividing it by its length in words, and approximating word duration (assuming, for this study, that words have uniform length). eye gaze labels are then aligned with words based on the video's frame rate (50 fps). we consider eye movements as sequences of gaze labels (figure 9 shows an example of speech gaze alignment). to be able to compute the entropy for gaze, we train a transformer model @xcite to estimate the underlying probability distribution of the gaze sequences.we have chosen a transformer model since it has a stronger psychometric predictive power compared to lstm-rnns models @xcite . to compute the local entropy of a gaze sequence, we first calculate its negative log probability: ## preliminary result after computing the local entropy, we first calculate statistical significance, removing outliers. we use a simple local outlier detection method by calculating the standard score (z-score) of each data point. if the standard score is greater than a threshold (for the gaze local entropy, we define it as 4 so that only very extreme values are considered outliers), it is considered an outlier and replaced by the median of the local entropy. figure 3 shows the change of local entropy of the combined eye gaze sequences from all 58 interaction videos after removing outliers. the 𝑥-axis represents the dialogue position of the speech segments (with each dialogue position corresponding to about 7 seconds of speech) and the y-axis represents the local entropy of the gaze sequences. although there seems to be a general increasing trend of the local entropy, we also have to consider the different length of the videos, which may bias the result. therefore, we decided to divide the videos into three groups based on length to mitigate the length bias and replot the local entropy (see figure 4 ). we then use a linear model and conduct statistical significance tests between dialogue position and gaze local entropy for each of the three sub groups. the results show us that, coefficient values in all three sub groups are positive, with 𝑝-values smaller than 0.05. the increase of local entropy of gaze (shown in figure 4 ) is thus statistically significant, that is, the preliminary result supports our basic hypothesis. the general increasing trend of local entropy indicates that more new gaze labels appear over the course of the explanations. ## analysis of linguistic factors the goal of this section is to answer the question why listener gaze, as a nonverbal signal in interaction, conforms to the erc principle (see section 6). specifically, we focus on exploring linguistic factors which may influence the information amount conveyed by listeners' gaze during interaction. we select two metricssyntactic complexity and turn transition potential -as the linguistic factors to investigate and perform statistical analyses to check for correlations between gaze local entropy and these two factors. ## conclusion in this study, we extracted listeners' gaze from video-recorded human interaction dialogues (specifically from explainees in explanatory dialogues) and found evidence that listener gaze, as an important nonverbal signal, adheres to the entropy constancy rate principle -a property of human language use @xcite . this finding is supported by an increasing trend of the local entropy of listeners' gaze over the course of the dialogues we analyzed. we believe that this finding can provide future insight into the evaluation of co-speech gaze generation and interpretation in the field of human-agent interaction, e.g., in explanatory settings. we further investigated two linguistic factorssyntactic complexity and turn-taking potential -that may contribute to this increase in gaze entropy, as they have been considered to interact with gaze behavior in previous studies. our statistical analysis provides initial evidence that these two factors are correlated (weakly) with the local entropy value of gaze, which could potentially provide an explanation for why listeners' gaze may adhere to the erc principle.