Does Listener Gaze in Face-to-Face Interaction Follow the Entropy Rate Constancy Principle: An Empirical Study (2023)

Abstract: It is generally assumed that language (written and spoken) follows the entropy rate constancy (ERC) principle, which states that the information density of a text is constant over time. Recently, this has also been found for nonverbal gestures used in monologue, but it is still unclear whether the ERC principle also applies to listeners’ nonverbal signals. We focus on listeners’ gaze behaviour extracted from video-recorded conversations and trained a transformer-based neural sequence model to process the gaze data of the dialogues and compute its information density. We also compute the information density of the corresponding speech using a pre-trained language model. Our results show (1) that listeners’ gaze behaviour in dialogues roughly follows the ERC principle, as well as (2) a congruence between information density of speech and listeners’ gaze behaviour.

## introduction human social interaction is intrinsically multimodal @xcite . face-to-face communication as a multimodal process includes verbal information as well as non-verbal cues, such as gaze, head movements, and speech-accompanying manual gestures from both interlocutors. previous studies have demonstrated that non-verbal behaviours is rich in communicative functions @xcite . in this paper, we look at gaze behaviour of listeners in video-recorded face-to-face-interaction, specifically, explanation dialogues in which a board game is explained by one interlocutor (the 'explainer') to another (the 'explainee'; @xcite . we apply information theoretical measures to the gaze behaviour of the explainees as well as the corresponding utterances of both interlocutors and aim at answering the following questions: (i) how informative is listener gaze from an interactional perspective and does it follow the 'entropy rate constancy' principle (erc; @xcite ? (ii) is there a correlation between verbal information and listener gaze, in terms of local entropy, in dialogue (similar to recent findings on manual gesture in monologue; @xcite )? ## hypothesis in this paper, we investigate the hypothesis that, during face-to-face interaction, listener gaze, being an important non-verbal communication mechanism, also conforms to the erc principle. ## results and discussion figure 3 shows the combined eye gaze sequences from all 58 videos. the x-axes represent the dialogue position of the speech segments (with each dialogue position corresponding to about 7s of speech) and the y-axes represent the local entropy of the gaze sequences. figure 3a , shows a generally rising trend of the local entropy, figure 3b shows a replot of the data without the effect from entropy spikes shown in 3a (which we consider as outlier values), and yields a globally rising trend, but also a decreasing trend for the first 75 dialogue positions. the 58 explanations vary in length (see section 4.1), so plotting them together introduces some bias into the visualisation. we have therefore divided the dataset into four groups (based on the total length of the explanations) and plotted the local entropy of the gaze sequences separately in figure 4 . besides the common rising trend in all of four sub-plots figure 4a-d , they also share a decreasing tendency at the beginning, resulting in (roughly) convex shapes. one explanation for this phenomenon may be that, at the beginning of the interaction, explainees focus on the explanation of the game and direct their gaze (and attention) mainly to the explainer, thus signalling their participation role @xcite . as a result, there is little variation of gaze labels. however, as the explanation progresses, the explainees' cognitive load may increase, a known cause of gaze aversive behaviour @xcite . another explanation for the increasing trend could be that explainers are likely to provide non-verbal feedback to explainers (e.g. head nodding or other head movements; @xcite , signalling understanding etc., which could also lead to changes in gaze behaviour. in any case, the result is a higher diversity of gaze labels and thus an increase in local entropy. we chose a board game explanation so that the linguistic content of the dialogue is limited. this gives us a good opportunity to look at whether different individuals organise their speech in a 'rational' way (following the erc principle; @xcite . according to figure 5 , it turns out that, given the same topic (the explanation of the board game), different explainers organise the explanation in a rational way, as evidenced by the increasing trend of the local entropy. moreover, this increasing trend of local entropy for both the speech segments across dialogues and their corresponding gaze sequences indicates a potential congruence between the information density of speech and the gaze behaviour of explainees/listeners. ## conclusions this study attempts to find out whether (i) gaze as an important non-verbal communication signal follows the entropy rate constancy principle, and whether (ii) the information density of listeners' gaze behaviour correlates with that of the dialogue content. we recorded interaction videos, trained a transformer model, and used a pre-trained language model to approximate the information density of listeners' gaze and dialogue content. we find that listeners' gaze roughly follows the erc principle, which can be taken as further evidence that non-verbal communication generally follows the erc principle (at least to some extent; @xcite -although the result is inconclusive in that the information density of listeners' gaze fluctuates along the interaction. a congruence between dialogues content and listeners' gaze can be roughly confirmed, as the local entropy of listeners' gaze and the local entropy of speech both show an increasing tendency. the fluctuation of the entropy rate value (figure 3a as well as figure 4 ) indicates that the property of non-verbal communication cannot simply be explained by erc principles. as a next step, we plan to look more closely at listeners' eye gaze and further analyse whether sudden changes in the local entropy of gaze behaviour can be aligned with changes in the local entropy of speech, and if so, under what kinds of context. we also plan to perform further linguistic analysis on the entropy spikes, looking at dialogue acts and calculating the syntactic complexity of the corresponding dialogue positions. possible future research includes simplifying or compressing the dialogue context based on changes in the entropy values of the listener's gaze.