Analysing Human Strategies of Information Transmission as a Function of Discourse Context (2021)

Abstract: Speakers are thought to use rational information transmission strategies for efficient communication (Genzel and Charniak, 2002; Aylett and Turk, 2004; Jaeger and Levy, 2007). Previous work analysing these strategies in sentence production has failed to take into account how the information content of sentences varies as a function of the available discourse context. In this study, we estimate sentence information content within discourse context. We find that speakers transmit information at a stable rate—i.e., rationally—in English newspaper articles but that this rate decreases in spoken open domain and written task-oriented dialogues. We also observe that speakers’ choices are not oriented towards local uniformity of information, which is another hypothesised rational strategy. We suggest that a more faithful model of communication should explicitly include production costs and goal-oriented rewards.

## introduction linguistic communication can be understood as information exchange through a noisy channel. speakers are sensitive to the properties of the channel in two ways @xcite ). on the one hand, they try to reduce the processing effort of the addressee. for example, in the absence of established discourse context, speakers can produce utterances that are easier to process in order to minimise the chance of transmission error. on the other hand, speakers try to reduce their own production effort. for example, given a fixed amount of information that they intend to transmit, speakers can take the risk of producing more concise utterances that are less costly from the production point of view, and expect the addressee to exploit the utterance context for interpretation. effective and efficient information exchange under these two competing pressures can be modelled using the tools of informa-tion theory @xcite . indeed, informationtheoretic models have successfully accounted for surprisal in speech perception @xcite , reading @xcite @xcite , sentence interpretation @xcite , and overlap in turn taking @xcite . the information content of a sentence h(s)i.e., its entropy or the effort it takes to process it out of context-and the informativeness of its discourse context i(s; c) are hypothesised to be related. according to the principle of entropy rate constancy (erc; @xcite , as discourse develops, these two quantities increase at a similar rate; thus, the difference between themi.e., the effort that it takes to process a sentence in context-remains constant over the course of a discourse: h(s|c) ≡ h(s) -i(s; c). a slight relaxation of this prediction is that the information content of a sentence in context remains uniform, rather than constant. this second prediction follows from the principle of uniform information density (uid; @xcite , according to which speakers make rational linguistic choices that avoid peaks in the density of the information transmitted. evidence in favour of these principles has been found in texts @xcite @xcite and, under certain conditions, in conversations @xcite @xcite . however, these studies base their conclusions only on estimates of the decontextualised entropy h(s): under the assumption that a larger context is always more informative, an increase in h(s) suffices as an indication that the erc and uid principles hold. in this work, 1 we dispose of the assumption that context informativeness increases constantly within a discourse, and we test whether the erc and uid principles hold using, for the first time, direct estimates of the contextualised entropy h(s|c) of an utterance and thus of the informativity of its linguistic context i(s; c). using a pre-trained transformer-based language model, which allows us to obtain more accurate probability estimates than the n-gram models used in previous studies and to condition the estimates on discourse context, we replicate @xcite seminal experiments on newspaper articles, and in addition apply the analysis to open domain spoken dialogues and to written task-oriented dialogues to test the principles in interactive settings. the proposed operationalisation allows us to test whether the increase in decontextualised entropy observed in earlier work corresponds to an increase in context informativeness or whether speakers simply change their transmission rate over time. furthermore, this approach allows us to differentiate, for the first time, between the erc and the uid predictions at the level of discourse. by studying language production using information-theoretic tools, this paper directly informs the development of computational models of utterance generation. our findings suggest that architectures and training objectives that enforce a uniform organisation of information density @xcite are better suited for reproducing human strategies of information exchange. ## related work if speakers were to use rational strategies in language production, they would optimise successful communication by transmitting information at a stable rate, close to the capacity of the communication channel @xcite . this rational strategy of information exchange @xcite @xcite ) is employed at many levels of language production. for example, speakers tend to reduce the duration of more predictable sounds @xcite @xcite , they tend to drop sentential material within more predictable scenarios @xcite @xcite , and they are more likely to overlap at turn transitions that are less information dense @xcite . at the level of discourse, it has been shown that the decontextualised information content h(s) of sentences increases in written texts with the amount of relevant discourse @xcite . this has been confirmed across document types and languages @xcite , and it has been hypothesised that an increase in h(s) results in a constant level of information content once the informativeness of discourse context i(s; c) is taken into account. this hypothesis has never been tested directly. in interactive settings, it is as yet empirically unclear whether speakers use this strategy of information transmission. @xcite and @xcite investigate this in spoken dialogue and show that h(s) grows throughout dialogues, too, and that the contribution in information content of interlocutors shows converging patterns @xcite . in contrast, @xcite fail to find uid effects in twitter dialogues and multi-party conversations. again, these studies focus on out-of-context utterance information content and do not measure the contribution of the dialogue context. @xcite , in another study with twitter conversations, are the first to take the informativeness of context into account. they focus on non-linguistic contextual cues (i.e. information about hashtagged events) and show that they have an effect on the overall word-level information transmission profiles of conversations. in this study, we model the effect of linguistic context on the information content of sentences in written monologue as well as in written and spoken dialogue. recent work has extended this view of human communication to computational models of language generation. @xcite show that the successfulness of many common generation algorithms is related to their tendency to discard lexical choices that make the information content of the words in a sentence less uniform. @xcite build on this finding and augment the training objective of language models with various uniform information density regularisers, thus consistently improving the models' perplexity and generating more lexical diverse texts. this suggests that more faithful modelling of information transmission strategies in humans, to which we contribute in this paper, can inform the development of better computational models of language generation. ## data we analyse trends of information content of written and spoken english, in texts and in dialogue. excerpts from our corpus of written texts and from our written dialogue corpus are shown in ## method in this section, we present how we obtain estimates of the information content of sentences (or utterances), both when considered out of context and within their discourse context. we define our main information theoretic measures and describe the computational models that produce the estimates. ## analysis of language model estimates in this section, we report the estimates and patterns of sentence information content directly computed with the finetuned gpt-2 language models for our three corpora. recall that we are directly estimating both h(s) and h(s|c) from data, in contrast to previous work, where h(s|c) is never computed empirically. before using the h(s|c) estimates to test the erc and uid hypotheses, we validate them by comparison with those obtained using random control contexts (see section 4.3). if the language model relies on irrelevant contextual features, we would expect the estimates obtained with the true context to be virtually indistinguishable from those obtained with random contexts. this would mean that our h(s|c) estimates are not reliable. in contrast, if the model does effectively exploit the actual context to estimate a sentence's entropy, we should see a clear difference between the true h(s|c) estimate and the control runs. as can be seen in figure 1 , true and control trends start diverging from sentence position 2. control contexts produce a positive shift in the magnitude of h(s|c) in all corpora: processing a sentence s in a random context is always harder than processing it in its true context. moreover, because the control contexts are incoherent with respect to s, they cause h(s|c) to be higher than h(s): processing a sentence s in an incoherent context is harder than processing it with no context. 9 we also notice that while the magnitude of h(s|c) depends on the veracity of the contexts, its fluctuations are largely determined by h(s). this is particularly true for the control trends of h(s|c), whose slope, too, is determined by h(s). in sum, the h(s|c) trends computed with the true data differ from the trends obtained with control runs according to reasonable expectations: the true h(s|c) estimates are lower, and the control estimates higher, than the h(s) estimates. this provides evidence that our empirical estimates of sentence information content for the three corpora are solid. we can now use these validated estimates to test the erc and uid hypotheses. ## experiments taking the values for h(s) (eq. 4) and h(s|c) (eq. 5) estimated with the language model, we use eq. 6 to compute context informativeness i(s; c) for all sentences in our datasets within a fixed initial context window, as explained in section 4. recall that the erc and uid principles hypothesise that both h(s) and i(s; c) will increase with the position of s within a discourse, and that as a result, h(s|c) will remain stable. this is expressed in eq. 3, repeated here for convenience: in the following experiments we investigate whether this is indeed the case. ## discussion and conclusions in this study, we have examined some central tenets of the classic information-theoretic model of communication. in contrast to previous work, we have used language models to obtain information content estimates (h(s|c)) for sentences within their discourse context, and we have measured context informativeness (i(s; c)) as the reduction in sentence surprisal contributed by discourse with respect to out-of-context estimates (h(s)). this has allowed us to directly model the information transmission profiles of written texts and written and spoken dialogues and, thereby, to test whether they follow the rational communicative strategies predicted by the theory. we have found that in american english newspaper articles, h(s|c) remains stable as predicted by the theory. this is not the case, however, for spoken british english open domain dialogues, nor for written english task-oriented dialogues: here, h(s|c) decreases, albeit moderately, as sentence position grows. we suggest that this is the result of the uneven rates of increase measured for h(s) and i(s; c)-the latter increases faster than the former in all corpora under examination. we find the strongest i(s; c) increase in the photobook dialogues, where topic is determined by a game's image domain (see section 3) and, by task design, participants produce multiple subsequent sentences to describe the same images over game rounds. correct interpretation of subsequent references @xcite requires indeed access to the shared knowledge accumulated by speakers during dialogue. we observe the second strongest i(s; c) increase in the penn treebank articles, where topic is consistent throughout the text but new information keeps being conveyed from the beginning to the end of the discourse. the weakest increase takes place in the spoken bnc: topic is more likely to change during the course of an open domain dialogue and, with topic shifts, the previously established common ground becomes less relevant for the prediction of new linguistic material. the lower rates of increase of h(s), on the other hand, can be due to the limits imposed on lexical choice by grammar and style. in photobook, where participants write freely in a chat interface, the increase is stronger than in the more formal newspaper articles of the penn treebank. however, the stable h(s) trends in the spoken bnc suggest that this is only one side of the coin. the theory predicts that when context is more informative, speakers will increase the density of their sentences to be more efficient, but speakers do not need to be always efficient in open domain conversations, where the pure information transmission goal is perhaps overweighted by more social goals that are not taken into account by the theory. another empirical finding that is not in line with the analysed theoretical framework is that uniformity of information content across consecutive sentences (local predictability) is not a good predictor of the information transmission profiles of the texts and dialogues we analysed. local uniformity may be more relevant for lower-level linguistic signals as they come in a much faster succession: speakers want to avoid sudden changes in information density to reduce comprehension effort; yet, at the discourse level, changes in surprisal are less abrupt as they are spread throughout an entire sentence, thus giving the addressee time to adapt gradually to the higher information content of the larger transmission unit. global centrality seems to be a more faithful criterion of uniformity, in particular for the articles of the penn treebank. in other words, sentences are not so much produced to limit the difference in information content with respect to the previous sentence, but rather to maintain the overall transmission rate stable in the articles. both dialogue corpora show a significantly lower degree of uniformity than the penn treebank, measured both as local predictability and global centrality: in dialogue, an efficient strategy of information exchange needs to be coordinated between two speakers, which can make it more difficult to obtain uniform information profiles. in conclusion, our study suggests that the classical model of communication may be too simplistic for discourse, where the units of information are more complex. a first issue has to do with identifying the relevant contextual components, which are determined, at least, by the internal structure of the discourse @xcite and by topic shifts @xcite . we have indeed shown in related work @xcite that theoretically motivated contextual units exhibit clearer uid trends in task-oriented dialogue. second, the predictions made by this model rely on estimates of comprehension effort of a static addressee whereas true addressees adapt on-the-fly: e.g., @xcite show that endowing a language model with a simple adaptation mechanism improves predictions of human reading times compared to a non-adaptive model. moreover, the classic framework assumes a single addressee across documents while, especially for dialogue, communication is shaped by the identity and the characteristics of multiple addressees @xcite . finally, the framework condenses production and comprehension effort in a single estimate. future work should study strategies of information transmission in discourse using a model of communication, such as the rational speech act model @xcite , that includes production costs more explicitly and that allows accompanying cognitive costs with social costs-e.g., those related to the goal of the linguistic interaction. zaslavsky et al. (2021) recently showed that the rsa model optimises the trade-off between expected utility and communicative effort, and that it is directly related to rate-distortion theory @xcite )-the branch of information theory that formalises the effect of limited transmission resources on communicative success. as the pre-trained model yields relatively high perplexity on the target corpora, we finetune 11 it on 70% of each target corpus and leave out 30% of the dataset to compute the model's evaluation perplexity and to conduct our statistical analysis. the training and held-out portions of the corpora are specified in the main paper. gpt-2 is finetuned for 20 epochs with a learning rate of 1e -04 and batches of size 8. because 20 epochs do not yield a substantial perplexity reduction for the spoken bnc dialogues, we finetuned the model for 20 additional epochs. the perplexity of the pre-trained and finetuned models on the target corpora is reported in the main paper. for our estimates of information content, we include sentence beginning symbols as contextual cues but their information content is not computed. 11 we use huggingface's finetuning script https://github.com/huggingface/ transformers/blob/master/examples/ pytorch/language-modeling/run_clm.py .