Is Information Density Uniform in Task-Oriented Dialogues? (2021)

Abstract: The Uniform Information Density principle states that speakers plan their utterances to reduce fluctuations in the density of the information transmitted. In this paper, we test whether, and within which contextual units this principle holds in task-oriented dialogues. We show that there is evidence supporting the principle in written dialogues where participants play a cooperative reference game as well as in spoken dialogues involving instruction giving and following. Our study underlines the importance of identifying the relevant contextual components, showing that information content increases particularly within topically and referentially related contextual units.

## introduction due to production and perception errors, differences between individuals, and other sources of uncertainty, language use for information transmission can be thought to happen through a noisy channel. effective and efficient information exchange under such conditions can be modelled using the tools of information theory @xcite . indeed, information-theoretic models have successfully accounted for surprisal in speech perception @xcite , reading @xcite @xcite , and sentence interpretation @xcite , providing psycholinguistic evidence that the information content of linguistic signals is related to comprehension processing effort. speakers, too, are sensitive to the properties of the communication channel. they are thought to simultaneously minimise their own production effort and the addressee's processing effort @xcite . the most efficient way of dealing with both pressures, according to information theory, is to transmit information at a constant rate @xcite , making linguistic choices that reduce fluctuations in the density of the information transmitted. evidence for the principle of uniform information density (uid; @xcite has been found at many levels of language production: speakers tend to reduce the duration of more predictable sounds @xcite @xcite ; they tend to drop sentential material within more predictable scenarios @xcite @xcite ; in spoken dialogue they are more likely to overlap at turn transitions when information density is low @xcite ; and the rate at which they transmit information in texts is uniform @xcite @xcite . empirically, it is as yet unclear whether information density remains uniform throughout conversations @xcite @xcite . that information density is not always uniform in dialogue may be due to the complex structure of conversational context @xcite , which not only includes previous utterances and world knowledge, but can also comprise preceding interactions between the interlocutors, their perceptual input, and their goals. this paper tests the uid principle on the previously unexplored setting of task-oriented dialogue, with its well-defined structural units and more constrained context than in open domain dialogues. to estimate information density, we use a pre-trained transformer-based language model, which provides more robust measurements than the n-gram models used in prior work. we study whether, and within which structural units the uid principle holds, finding new evidence in support of it in certain structural units of both written cooperative reference games and spoken map navigation dialogues. 1 our study highlights the importance of identifying relevant con-textual structures, showing that topically and referentially related contextual units correspond to more uniform information transmission profiles. ## measuring information content to investigate whether information density is uniform throughout a discourse, each lexical choice can be modelled as a random variable y i and its information density estimated as the shannon information content h(y i ). for the uid principle to hold, the amount of information transmitted with every new word h(y i ) must remain constant. we can rewrite h(y i ) as h(x i |l i , c i ), where c i is the entire relevant context and l i is the local context, both influencing lexical choice x i . typically, h(x i |c i , l i ) is not estimated directly. the term is further decomposed into h(x i |l i ), the information content of x i given the local context, and i(x i ; c i |l i ), the locally conditioned mutual information between x i and the entire relevant context: as the relevant context is built up, i(x i ; c i |l i ) is assumed to increase: next word prediction becomes easier when more contextual cues are available @xcite . so for the uid principle to hold-i.e., for h(x i |c i , l i ) to remain constant in eq. 1-the locally conditioned information content h(x i |l i ) must increase, too, as relevant context accumulates. the local context of a word choice is typically taken to be the utterance or sentence, and these are also considered as the units of information transmission @xcite @xcite @xcite . the information content of an utterance is computed by averaging over the negative logarithms of all locally contextualised word probabilities: to remove the confounding effect of utterance length on information content @xcite , we use @xcite normalised metric of utterance information content: ## data and hypotheses the uid principle is assumed to hold within a structural unit that determines the type and size of the overall relevant context c i as used in eq. 1. @xcite show that, in texts, part of the relevant context is lexical (writers tend to reuse words that have already appeared in the discourse) and topically determined, as given by the paragraph structure of texts. in dialogue, defining a topically relevant contextual unit is not straightforward. @xcite use a topic segmentation algorithm to identify relevant units in unconstrained dialogues and show that information density is influenced by topic shift. here we exploit the inherent (task-related) structure of taskoriented dialogues to test the uid principle within contextual units of different type and size. we analyse two corpora of task-oriented english dialogues: maptask (mt, @xcite foot_1 and photo-book (pb, @xcite foot_2 . dialogue excerpts from both corpora can be found in appendix a. mt contains 128 transcribed spoken dialogues consisting of an instruction giver directing an instruction follower to navigate to a point on a map. the participants cannot see the other's map and their respective maps may contain slightly different landmarks. we consider two types of contextual unit: a) the overall dialogue: a series of landmarks are described in succession to help the instruction follower draw a path towards a goal location; b) a dialogue transaction: a dialogue excerpt related to reaching a certain landmark, manually annotated as part of the corpus. for both types of contextual unit, we also construct versions where we use the mt dialogue act annotation to filter out turns exclusively consisting of backchannels and other grounding acts ('okay', 'mmhmm') common in spoken language. 4 this results in contextual units that focus on information-transmission dialogue acts and are more referentially coherent. pb contains 2,500 dialogues where two participants without specified roles communicate via written chat. each dialogue consists of 5 rounds: in each round, each participant sees a set of photographs which partially overlap with the set of images seen by their dialogue partner. the goal is to find out which images they have in common. the images available to each participant change in each round, but a subset reappears, thus triggering subsequent references to previously described photographs. this task design allows us to investigate the following types of contextual unit: a) the overall dialogue: throughout a game, all the photographs are about a certain domain (e.g., food or dogs); b) a dialogue round: different images are described in succession as participants try to figure out which ones they share in a given round; c) an image reference chain: the (non-adjacent) utterances that refer to a certain image across rounds (we use the automatic annotation of referring utterance chains by @xcite . we hypothesise that, in mt, the uid principle will be more visible at the transaction level, where the context is more topically coherent, than at the dialogue level, where a dozen different landmarks are brought up in succession-in particular when only information-transmission dialogue acts are taken into account. in pb, we expect the strongest effect to be present at the level of reference chains. chains are determined both topically, by the target image, and lexically, by the conceptual pacts established in previous mentions of a target @xcite . in rounds and dialogues, where several different images are described, topic and lexical choices are constrained by the image domain but the vocabulary used in previous turns is more varied. we thus expect the effect to be less pronounced at these two levels. ## modelling to estimate the information content of an utterance we compute the log probabilities in eq. 2 using gpt-2 @xcite , a pre-trained transformer language model, which allows us to obtain more accurate probability estimates than n-gram models. we rely on huggingface's implementation of gpt-2 with default tokenizers and default parameters @xcite . as gpt-2 was pretrained mainly on written text, it is less tuned to the idiosyncrasies of dialogue data. we therefore finetune it separately on a 70% split of each target corpus. 5 as shown in we adopt genzel and charniak's assumption that the mutual information i(x i ; c i |l i ) between an utterance and its context increases with turn position @xcite , see section 2); so for h(s i |c i , l i ) to remain stable, utterance information content h(s i |l i ), too, must increase. consequently, we consider the uid principle to hold when turn position has a significant positive effect on information content. validation to validate our estimates of utterance information content, we replicate @xcite ) and @xcite study on the wall street journal articles of the penn treebank @xcite foot_6 using gpt-2 finetuned on this corpus (see in our validation study, we take both entire articles and paragraphs as structural units and count sentence positions from the beginning of the relevant unit. our linear mixed-effect models show a significant positive effect of sentence position on information content both within articles (β = 1.65e-2, @xmath0.001) and within paragraphs (β = 1.53e-2, @xmath1.01). to reproduce the original experimental setting, we further train an n-gram language model with interpolated kneser-ney smoothing using @xcite data split and select the configuration with the lowest perplexity on the test set, a 3-gram model with a discount value of 0.8. in line with previous work, we find a positive kendall's rank-correlation foot_7 between sentence position and information, as measured with the n-gram model as well as with the transformers. the original results are therefore replicated. foot_8 we test whether the uid principle holds in mt and pb using the procedure presented in section 4. the full results of our statistical analysis can be found in appendix e (tables 6 and 7 ). recall that for the principle to hold, the locally conditioned information content h(x i |l i ) must increase with the position of x i in the relevant context unit c i . the local context l i is defined as a dialogue turn. ## conclusion we investigated to what extent the principle of uniform information density holds in two corpora of english task-oriented dialogues. we have related the properties of task-determined contextual units to patterns of information transmission and have hypothesised that the uid principle holds to a stronger degree in more topically coherent and reference-specific contextual units. our hypotheses are confirmed in photobook, where we find evidence that dialogue participants use rational strategies of information transmission over an entire dialogue. we do not observe uniformity of information in the maptask dialogues and transactions as a whole, similarly to other negative results in interactive settings (e.g., @xcite ). yet the effect is present within maptask transactions when we restrict our analysis to information-transmission dialogue acts: these make for a more topically and referentially coherent contextual unit. indeed, the organisation of context can be complex in dialogues. we have shown that theoretically motivated contextual units such as reference chains in photobook and information-transmission acts in maptask transactions are good candidates to characterise the relevant context over which participants deploy strategies of information compression. we are aware that the assumptions used to test the uid principle, which we have adopted from genzel and charniak's seminal work (2002)-i.e., that context informativeness increases as strongly as sentence entropy as discourse is built up-can be controversial. nevertheless, in this paper we have followed this line of reasoning, used in previous work @xcite @xcite @xcite , and applied it to novel data and contextual units. in @xcite , we go one step further and empirically test these assumptions for the first time, using direct estimates of the contextualised entropy h(s i |c i , l i ) of an utterance and thus of the informativity of its linguistic context i(s i ; c i , l i ). the study presented in this paper provides new empirical evidence on language production in dialogue which we believe can directly inform the development of natural language generation models. our findings suggest that models that take relevant contextual units into account @xcite are better suited for reproducing human patterns of information transmission, and confirm that the use of training objectives that enforce a uniform organisation of information density @xcite is a promising avenue for training language models.