Surprise! U niform I nformation D ensity Isn’t the Whole Story: Predicting Surprisal Contours in Long-form Discourse (2024)

Abstract: The Uniform Information Density (UID) hypothesis posits that speakers tend to distribute information evenly across linguistic units to achieve efficient communication. Of course, information rate in texts and discourses is not perfectly uniform. While these fluctuations can be viewed as theoretically uninteresting noise on top of a uniform target, another explanation is that UID is not the only functional pressure regulating information content in a language. Speakers may also seek to maintain interest, adhere to writing conventions, and build compelling arguments. In this paper, we propose one such functional pressure; namely that speakers modulate information rate based on location within a hierarchically-structured model of discourse. We term this the Structured Context Hypothesis and test it by predicting the surprisal contours of naturally occurring discourses extracted from large language models using predictors derived from discourse structure. We find that hierarchical predictors are significant predictors of a discourse’s information contour and that deeply nested hierarchical predictors are more predictive than shallow ones. This work takes an initial step beyond UID to propose testable hypotheses for why the information rate fluctuates in predictable ways.

## introduction linguistic communication takes place in a context, a backdrop of both linguistic and non-linguistic content that can determine how utterances' form @xcite and meaning @xcite are interpreted as well as what words speakers choose to say next @xcite . we investigate the role of context from an informationtheoretic perspective, asking how a linguistic context, i.e., what has been said or written previously, shapes the information content of each linguistic unit, i.e., a novel word or utterance in that context. one influential hypothesis for the relationship between linguistic units and their context is the uniform information density (uid) hypothesis @xcite @xcite @xcite , which posits that, subject to the constraints of the grammar, speakers spread out information as evenly as possible across an utterance. if the uid hypothesis is taken to an extreme, i.e., if it is imposed as a hard constraint, then each linguistic unit would add roughly the same amount of information, when the previous context is taken into account. there is an abundance of empirical support for the uid hypothesis, albeit, in general, for a soft variant of it where there is violable pressure towards uniformity. for instance, @xcite gives evidence across a number of languages that word order is optimized for uid. empirically, however, within a discourse, the information content of individual linguistic units is never observed to be strictly static but rather to fluctuate within a band. we dub this fluctuation in the information content of a discourse its information contour; see fig. 1 for an example. more theoretically, a pressure towards uniformity must naturally be attenuated by other competing functional pressures on linguistic communication. of course, the grammar constrains word choice, which may make uniformity difficult to achieve @xcite . moreover, when an author chooses the next word of a story or a poem, uid might give way to discursive pressures such as a desire for a clean narrative arc or a well-executed rhetorical structure. indeed, some literary devices, such as rhyme and meter, may even ascribe higher aesthetic value to a non-uniform information rate. in this article, we propose an elaboration of the uid hypothesis. in addition to a local pressure for uniformity on information modulated by the grammar, we posit that the information contour of a discourse itself is a meaningful signal that reflects a richer structured notion of context. the idea that there is a relationship between local information content and hierarchical syntax goes back to @xcite and has been expanded more recently @xcite . however, decades of 0 100 200 300 400 500 0 10 20 30 document surprisal computed under the language model predicted document surprisal document surprisal computed under the language model with a 5size rolling window predicted document surprisal with a 5size rolling window to answer these questions, we use neural language models to estimate the local information content of written english and spanish texts. we then consider two different representations for the hierarchical discourse structure of a text. the first is the standard prose-writing convention of dividing the document into (shallow) hierarchically nested paragraphs and sentences. the second is based on rhetorical structure theory (rst; @xcite , which breaks texts into recursively nested spans that are linked by discourse relations. to investigate questions (i) and (ii) above, we apply bayesian regression analysis to determine whether access to the discourse structure helps us better model information contours. we do find evidence that hierarchical discourse structure helps predict information contours across the board and that rst is more predictive than the shallowly nested paragraph and sentence structures. in sum, this work provides preliminary empirical evidence for the structured context hypothesis and paves the way for a theory explaining how and why information contours may be modulated by discourse structure. ## background there are myriad information-theoretic theories of language. this section builds up to and introduces the structured context hypothesis while contextualizing it in light of previous proposals. ## the structured context hypothesis to harmonize uid with the constraints imposed by contour principles, we propose the structured context hypothesis. in most previous work, context is modeled as an essentially sequential object-a succession of paragraphs, topic episodes, dialogue transactions, or dialogue rounds. in contrast, we rely on a different view, considering context as hierarchical representations made up of sentences within paragraphs or deeply nested discourse trees. we hypothesize that the fluctuations observed in surprisal contours of discourse beyond a baseline uniformity can be at least partially accounted for by considering structured representations of the discourse in question. this means that taking into account hierarchical dependencies beyond the sentence level in our theories should increase their ability to predict the information rate of discourse. we express this view through the following hypothesis: hypothesis 1 the structured context hypothesis: the information contour of a discourse is (partially) determined by the hierarchical structure of its constituent discourse units. the objective of our experiments is to empirically test this hypothesis on english and spanish texts. in the remainder of this section, we outline two manners to represent documents' hierarchical dis- r attribution r elaboration-object-attribute-e that is in part because of the effect of having to average the number of shares outstanding, she said. course structure: the conventional prose structure of paragraphs and sentences, and the fine-grained rhetorical structure theory. ## methods the goal of our statistical analyses is to study whether the information contour of a text can be predicted from discourse representations. our models predict measures of information content (dependent variables) based on a number of predictors (independent variables), some of which we designate as baseline predictors while the others designate as independent predictors for convenience. for a summary of all variables, see tab. 1 in app. b. ## empirical findings we overview our empirical results in this section, structuring our presentation in terms of five research questions relating to the structured context hypothesis in q1-4, we move from shallower to deeper discourse structure representations, focusing on rst-based predictors. in q5, we compare rst to conventional prose structure. ## future work we hypothesized that violations of the communicative pressure to communicate at a uniform rate might be predictable, and that part of their predictability is linked to how production choices depend on discourse structure (the structured context hypothesis). while we could not determine violations of the uid hypothesis precisely due to its inherent underspecificity, our predictive modeling framework captured deviations from a constant information rate by design, with the intercept representing the baseline rate and predictors capturing deviations. as fig. 1 shows, the structured context helps predict oscillations around the base rate, though we only account for a small portion of these deviations. there are, however, additional intuitive explanations for violations of uniformity which our predictors do not capture, or only do so partially. ## conclusion we conclude by briefly highlighting the theoretical and empirical contributions of this paper. theoretically, we have enumerated the limitations of the uid hypothesis and have provided an initial hypothesis, the structured context hypothesis, to predict how information fluctuates during a discourse, namely discourse trees based on prose conventions and rst. empirically, we have found support for this hypothesis by evaluating two structured representations of discourse in english and spanish. we view this work as one step in developing theories that can explain the vast variation in discourses, texts, and writing genres observed across human cultures..