Construction Repetition Reduces Information Rate in Dialogue (2022)

Abstract: Speakers repeat constructions frequently in dialogue. Due to their peculiar information-theoretic properties, repetitions can be thought of as a strategy for cost-effective communication. In this study, we focus on the repetition of lexicalised constructions—i.e., recurring multi-word units—in English open-domain spoken dialogues. We hypothesise that speakers use construction repetition to mitigate information rate, leading to an overall decrease in utterance information content over the course of a dialogue. We conduct a quantitative analysis, measuring the information content of constructions and that of their containing utterances, estimating information content with an adaptive neural language model. We observe that construction usage lowers the information content of utterances. This facilitating effect (i) increases throughout dialogues, (ii) is boosted by repetition, (iii) grows as a function of repetition frequency and density, and (iv) is stronger for repetitions of referential constructions.

## introduction the repeated use of particular configurations of structures and lexemes, constructions, is pervasive in conversational language use @xcite . such repetition can be understood as a surface level signal of processes of coordination @xcite or 'interpersonal synergy' between conversational partners @xcite . speakers may use repetitions to successfully maintain common ground with their interlocutors @xcite , because they are primed by their recent linguistic experience @xcite , or to avoid a costly on-the-fly search for alternative phrasings (see, e.g., @xcite . at the same time, repetitions are also advantageous for comprehenders. repeating a sequence of words positively reshapes expectations for those words, allowing comprehenders to process them more rapidly (for a review, see @xcite . as speakers are known to take into consideration both their own production cost and their addressee's processing effort @xcite @xcite , its two-sided processing advantage, as described above, makes construction repetition an efficient, cost-reducing communication strategy. in this paper, we investigate whether and how these information-theoretic properties of repetitions shape patterns of information rate in opendomain spoken dialogue. information theory is the study of the conditions affecting the transmission and processing of information. to the foundations of the field belongs the noisy-channel coding theorem @xcite , which states that for any given degree of noise in a communication channel, it is possible to communicate discrete signals nearly error-free up to a maximum information rate, the channel capacity. if speakers use the communication channel optimally, they might send information at a rate that is always close to the channel capacity. this observation is at the basis of the principle of entropy rate constancy (erc; @xcite , which predicts that the information rate of speaker's utterances, measured as the utterance conditional entropy (i.e., its in-context shannon information content or information density) remains constant throughout discourse. the erc predictions have been empirically confirmed for written language production @xcite @xcite but results on dialogue are mixed @xcite @xcite , with some studies suggesting a decreasing information rate over the course of dialogues @xcite . we hypothesise that this decreasing trend in dialogue may be associated with construction repetition. we conjecture that speakers use construction repetition as a strategy for information rate mitigation, by padding the more information dense parts of their utterances with progressively less information dense constructions-leading to an overall decrease in information rate over the course of a dialogue. we extract occurrences of fully lexicalised constructions (see in sum, our study provides new empirical evidence that dialogue partners use construction repetition as a strategy for information rate mitigation, which can explain why the rate of information transmission in dialogue, in contrast to the constancy predicted by the theory @xcite , is often found to decrease. our findings inform the development of better dialogue models. they indicate, as suggested in related work (e.g., @xcite , that while avoiding degenerate repetitions in utterance generation @xcite is an appropriate strategy, dialogue systems should not suppress human-like patterns of repetition as these make automatic systems be perceived as more natural and more effective in conversational settings. ## data we conduct our study on the spoken british national corpus foot_0 @xcite , a dataset of transcribed open-domain spoken dialogues containing 1,251 contemporary british english conversations, collected in a range of real-life contexts. we focus on the 622 dialogues that feature only two speakers, and randomly split them into a 70% finetuning set (to be used as described in §4) and a 30% analysis set (used in our experiments, as described in §5 and §6). ## experimental setup in this section, we define our information-theoretic measures and present the adaptive language model used to produce information content estimates. foot_5 ## preliminary experiments in this section, we present preliminary experiments on the information content of utterances and constructions, which set the stage for our analysis of the facilitating effect of construction repetition. ## the facilitating effect of construction repetition we now test whether constructions have a positive facilitating effect, i.e., whether they reduce the information content of their containing utterances. we present our main statistical model in §6.1, describe the effects of fe predictors specific to unique construction mentions in §6.2, and analyse differences between types of constructions in §6.3. ## discussion and conclusions construction repetition is a pervasive phenomenon in dialogue; their frequent occurrence gives constructions a processing advantage @xcite . in this paper, we show that the processing advantage of constructions can be naturally modelled as reduced information content and propose that speakers' production of constructions can be seen as a strategy for information rate mitigation. this strategy can explain why utterance information content is often found to decrease over the course of dialogues @xcite , in contrast with the predictions of theories of optimal use of the communication channel @xcite . we observe that, as predicted, construction usage in english open-domain spoken dialogues mitigates the information rate of utterances. furthermore, while constructions are produced at a stable rate throughout dialogues, their facilitating effect-our proposed measure of reduction in utterance information content-increases over time. we find that this increment is led by construction repetition, with facilitating effect being positively affected by repetition frequency, density, and by the contents of a construction. repetitions of referential constructions reduce utterance information content more aggressively, arguably making them a more cost-reducing alternative to the shortening strategy observed in chains of referring expressions @xcite , which instead tends to preserve rate constancy @xcite . 13 relation to cognitive effort we consider repetitions as a way for speakers to make dialogic interaction less cognitively demanding both on the production and on the comprehension side. this is not at odds with the idea that repetitions are driven by interpersonal synergies @xcite and coordination @xcite . we think that the operationalisation of these higher level processes can be described by means of lower level, efficiency-oriented mechanisms, with synergy and coordination both corresponding to reduced collaborative effort. although information content estimates from neural language models have been shown to correlate with human processing effort (cf. §2.2), we cannot claim that our work directly models human cognitive processes as we lack the relevant human data to measure such correlation for the corpus at hand. adaptive language model our decision to use an adaptive neural language model affects information content estimates in two main ways. on the one hand, due to their high frequency, constructions are likely to be assigned higher probabilities by this model, and therefore lower information content. we stress that we do not present constructions' lower information content as a novel result, nor do we make any claims based on this result. as explained in §5.2, this is a precondition for our experiments on the facilitating effect of constructions, which is not determined exclusively by their information content (as empirically shown in §5.4) but rather measures the effect of construction usage on the information content of entire utterances. on the other hand, because our model is adaptive, the probability of constructions is likely to increase as a result of their appearance in the dialogue history. adaptation, however, also contributes to lower utterance information content overall through the exploitation of topical and stylistic cues, as demonstrated by the lower perplexity of the adaptive model on the entire target dialogue as well as on other dialogues from the same dataset (see §4.2 and appendix c.2). in conclusion, while our adaptive language model assigns higher probabilities to frequently repeated tokens-as expected from a psychologically plausible model of utter-ance processing-it is not responsible for the discovered patterns of construction facilitating effect. in future work, the model can be improved, e.g., by conditioning on the linguistic experience of individual speakers. types of dialogue to consolidate our findings, construction repetition patterns should also be studied in dialogues of different genres and on datasets where utterance information content was not found to decrease. we have chosen the spoken bnc for our study as it contains dialogues from a large variety of real-life contexts, which makes it a representative dataset of open-domain dialogue. in task-oriented dialogue, we expect constructions to consist of a more limited, task-specific vocabulary, resulting in longer chains of repetition and potentially more frequent referential construction usage. these peculiarities of task-oriented dialogue may influence the strength of the facilitating effect (as we have seen, facilitating effect is affected by both frequency and referentiality) but we expect our main results to still hold, as they are generally related to the processing advantage of constructions. relevance for dialogue generation models besides contributing new empirical evidence on construction usage in dialogue, our findings inform the development of more naturalistic utterance generation models. they suggest that models should be continually updated for their probabilities to better reflect human expectations; that attention mechanisms targeting contexts of different sizes (local vs. global) may have a significant impact on the naturalness of generated utterances; and that while anomalous repetitions (e.g., generation loops) should be prevented @xcite , it is important to ensure that natural sounding repetitions are not suppressed. we expect dialogue systems that are able to produce humanlike patterns of repetitions to be perceived as more natural overall-with users having the feeling that common ground is successfully maintained (pickering and garrod, 2004)-and to lead to more effective communication @xcite . in our view, such human-like patterns can be reproduced by steering generation models towards the trends of information rate observed in humans.