Revisiting the U niform I nformation D ensity Hypothesis (2021)

Abstract: The uniform information density (UID) hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal. While its implications on language production have been well explored, the hypothesis potentially makes predictions about language comprehension and linguistic acceptability as well. Further, it is unclear how uniformity in a linguistic signal—or lack thereof—should be measured, and over which linguistic unit, e.g., the sentence or language level, this uniformity should hold. Here we investigate these facets of the UID hypothesis using reading time and acceptability data. While our reading time results are generally consistent with previous work, they are also consistent with a weakly super-linear effect of surprisal, which would be compatible with UID’s predictions. For acceptability judgments, we find clearer evidence that non-uniformity in information density is predictive of lower acceptability. We then explore multiple operationalizations of UID, motivated by different interpretations of the original hypothesis, and analyze the scope over which the pressure towards uniformity is exerted. The explanatory power of a subset of the proposed operationalizations suggests that the strongest trend may be a regression towards a mean surprisal across the language, rather than the phrase, sentence, or document—a finding that supports a typical interpretation of UID, namely that it is the byproduct of language users maximizing the use of a (hypothetical) communication channel.

## introduction the uniform information density (uid) hypothesis @xcite states that language users prefer when information content (measured information-theoretically as 1 analysis pipeline is publicly available and can be found at https://github.com/rycolab/revisiting-uid . figure 1 : correlation coefficient between (negative) sum of surprisals raised to the k th power and linguistic acceptability judgments of a sentence. the higher correlation when @xmath0. in this work, we attempt to shed light on these issues: we first study the relationship between the distribution of information content throughout a sentence and native speakers' (i) sentence-level reading times and (ii) sentence acceptability judgments. while our results for sentence-level reading times do not contradict previous word-level reading time analyses (e.g., @xcite , which have shown a linear effect of surprisal, they suggest that a slight super-linear effect may likewise be a plausible explanationwhich is in line with predictions of the uid hypothesis. for sentence acceptability judgments, we see more concrete signs of a super-linear effect of sentence-level surprisal (see fig. 1 ), consistent with a preference for uid in language. given these findings, we next ask how we can best measure uid. we review previous results supporting uid, in search of an operationalization and find that in most of these studies, adherence to uid is measured via an analysis of individual linguistic units, without direct consideration for the information content carried by surrounding units @xcite @xcite . such a definition fails to account for the distribution across the signal as a whole. consequently, we present and motivate a set of plausible operationalizations-either taken from the literature or newly proposed. given our earlier results, we posit that good operationalizations of uid should provide strong explanatory power for human judgments of linguistic acceptability and potentially reading times. in this search, we additionally explore with respect to which linguistic unit-a phrase, sentence, document, or language as a whole-uniformity should be measured. our results provide initial evidence that the best definition of uid may be a super-linear function of word surprisal. further, we see that a regression towards the mean information content of the entire language, rather than a local information rate, may better capture the pressure for uid in natural language, a theory that falls in line with its information-theoretical interpretation, i.e., that language users maximize the use of a hypothetical noisy channel during communication. ## processing effort in comprehension in psycholinguistics, there are a number of theories that explain how the effort required to process language varies as a function of some perceived linguistic unit. several of these are founded in information theory @xcite , using the notion of language as a communication system in order to build computational models of processing. under such a framework, linguistic units convey information, and the exact amount of information a unit carries can be quantified as its surprisal-also termed shannon information content. formally, let us consider a linguistic signal @xmath1, . . . , u n as a sequence of linguistic units, e.g., words or morphemes; the standard definition of surprisal is then s(u n ) @xmath2, i.e., a unit's negative log-probability conditioned on its prior context. note that under this definition, low probability items are seen as more informative, which reflects the intuition that unpredictable items convey more information than predictable ones. with this background in mind, we now review two prominent examples of information-theoretic models of language processing: surprisal theory and the uniform information density hypothesis. ## discussion most previous works investigating uid have looked for its presence in language production @xcite @xcite , while comprehension has received little attention. collins (2014) and @xcite are perhaps the only other works to find results in support of uid in this setting. our findings are complementary to theirs; we take different analytical approaches but both observe a preference for the uniform distribution of information in a linguistic signal, although a similar analysis should be performed in the spoken domain before stronger conclusions can be drawn. while our reading time results do not refute previous work showing linear effects of surprisal on word-level reading times @xcite @xcite ), 19 we see some suggestions that a super-linear hypothesis is also plausible, especially in the provo corpus. notably, most of these works did not test a parametric space of non-linear functional forms, instead confirming using visual inspection of the results of nonparametric fits. one exception, @xcite , explored the effects of adding a quadratic term for surprisal as a predictor of per-word reading times. yet, if the true k that describes the reading times-surprisal relationship were only slightly greater than 1, as our results suggest, this quadratic test might be too restrictive. our approach, which explores a more fine-grained range of k, is potentially more comprehensive, and indeed we find that values of k slightly greater than 1 often fit the data at least as well as @xmath3, and can certainly not be ruled out. other potential virtues of our analysis are (1) our analysis is performed at the sentence-(rather than word-) level. this is arguably a better method for analyzing a sequence-level phenomenon, i.e., uid, and (2) specifically for eye movement data, we include re-reading times after the first pass. limitations and future directions. a major limitation of this work is that the experimental analysis is limited to english (and dutch, in the appendix); while the pressure for uniformitysince explained by a cognitive process-should hold across languages, further experiments should be performed to verify these findings, especially since the relationship between model quality and psychometric predictive power has recently been 19 notably, @xcite have recently reported a linear effect of word probability on (self-paced) reading times in a controlled experiment where within each experimental item the target word was held constant and predictability was manipulated across a wide range by varying the preceding context. motivated by this result, we repeated our analytic pipeline testing a range of values of k but replacing surprisals with negative raw probabilities. the resulting regression model fits are not as good as those achieved when using surprisals (fig. 11 ; compare y-axis ranges with fig. 3 ). called into question @xcite . as such, while we find convincing preliminary evidence in our analyzed languages, we are not able to fully test the hypothesis that the pressure for uid is at the language-level. further, we have no evidence as to whether there may be pressure towards a cross-linguistic µ c , which would be relevant to cross-linguistic interpretations of uid @xcite . another important limitation of this work is the restriction to psychometric data from the written domain. to fully grasp the effects of the distribution of information in linguistic signals on language comprehension, spoken language data should be similarly analyzed. of course, different factors are likely at play in language comprehension in the spoken domain, including e.g., the cognitive load of the speaker @xcite ; such factors may make it even more difficult to disentangle the contribution of different effects to comprehension. we leave this analysis for future work. ## conclusion in this work, we revisit the uid hypothesis, providing both a quantitative and qualitative assessment of its various interpretations. we find suggestions that the uid formulation proposed in @xcite may better predict processing effort in language comprehension than alternative formulations since proposed. we additionally find that a similar model explains linguistic acceptability judgments well, confirming a preference for uid in written language. we subsequently evaluate different operationalizations of uid, observing that a super-linear function of surprisal best explains psychometric data. further, operationalizations associated with global interpretations of uid appear to provide better explanatory power than those of local interpretations, suggesting that perhaps the most accurate interpretation of uid should be the regression towards the mean information rate of a language. dataset types (m) types (u) sents (m) sents ( u result: the first derivative is always positive, meaning that processing effort is minimized at @xmath4.