Revisiting Entropy Rate Constancy in Text (2023)

Abstract: The uniform information density (UID) hypothesis states that humans tend to distribute information roughly evenly across an utterance or discourse. Early evidence in support of the UID hypothesis came from Genzel and Charniak (2002), which proposed an entropy rate constancy principle based on the probability of English text under n -gram language models. We re-evaluate the claims of Genzel and Charniak (2002) with neural language models, failing to find clear evidence in support of entropy rate constancy. We conduct a range of experiments across datasets, model sizes, and languages and discuss implications for the uniform information density hypothesis and linguistic theories of efficient communication more broadly.

## introduction linguistic functionalists have often claimed that language is optimized for efficient communication (e.g., @xcite . one common argument supporting theories of efficient communication is that humans communicate at or near channel capacity @xcite ; functionalists have argued that because interlocutors can only produce and comprehend a fixed amount of linguistic information per unit of time, and because speakers strategically arrange their utterances to convey as much information as possible, language should typically have uniform surprisal over time @xcite @xcite @xcite . early evidence for this claim comes from @xcite , which used n-gram language models to argue that english documents exhibit entropy rate constancy. in this paper, we identify a limitation in @xcite 's analysis and directly measure its hypothesis using neural language models. we then analyze results across a variety of datasets, languages, and models and discuss their implications for theories of efficient communication. @xcite , averaged across documents per word position. @xcite showed that entropy rate increased under n-gram models and predicted that it would remain constant in models which can condition on long-range context. we replicate the former result but do not find clear evidence supporting the latter. ## models in recent years, neural language models, in particular, transformer-based models @xcite have been shown to greatly outperform ngram models, due to their ability to scale and model long-distance dependencies. in this work, we compare the entropy rate of english text under the transformer-based gpt-2 model @xcite to that of n-gram models. ## experiments for each dataset and model, we compute the pertoken probability of each document in the dataset: for a word w k consisting of subword tokens x start(k) , . . . , x end(k) . in contrast to genzel and charniak (2002), we present results at the wordlevel rather than at the sentence-level to provide additional granularity and insight into trends at the beginnings of documents; additionally, this approach avoids confounding effects of sentence length which were noted in previous work @xcite . we then sum over each position in the dataset to compute the average per-word probability across documents in the dataset, at each word position i: while the standard definition of entropy rate refers a constant, our usage refers a more general trend over the course of documents. further, rather than computing the limit as @xmath0, we estimate the average observed word probabilities for each @xmath1, . . . , n} where n is the length of the document. a strongly negative trend, because word probability under neural models tend to decline sharply at the beginnings of documents, as discussed in the following section. we provide additional discussion of this issue, along with results from the mann-kendall significance test, in appendix e. ## discussion we clarify that the findings presented in this paper do not necessarily invalidate the uniform information density hypothesis. although entropy rate as measured by neural language models may decline over the course of documents, cognitive measures of surprisal might not decline. for example, the recently proposed lossy-context surprisal model of @xcite posits that surprisal is computed with respect to an incomplete representation of context, whereas neural language models may make predictions based on lossless representations of their context windows. this perspective is also consistent with recent findings that the base gpt-2 model (124m) outperforms larger gpt-2 and gpt-3 models as a predictor of human reading time @xcite . in particular, these results point to a discrepancy between surprisal values under a bayes-optimal language model and cognitivelyrelevant measures of surprisal. despite still being worse than humans at a variety of language-related tasks, we consider it likely that large language models outperform humans at the task of raw language modeling, at least as measured by perplexity. as a result, weaker language models may be better correlated with cognitive measures of surprisal. whether or not our results contradict the entropy rate constancy principle is a matter of interpretation. @xcite would predict that neural language models, which are capable of integrating long-distance context, would exhibit roughly constant entropy rate over the course of documents. under certain conditions, however, entropy rate as computed by neural language models seems to decline or even exhibit non-monotonic behavior. while this behavior is mostly isolated to the beginnings of documents, it is impossible for entropy rate to decline forever, because word probabilities cannot be less than zero. at the very least, we can conclude that our analyses do not provide clear support in favor of the entropy rate constancy principle proposed by @xcite . ## related work most relevant to this work are @xcite and @xcite , which explore the role of entropy rate constancy in dialogue datasets such as the spoken british national corpus @xcite , the hcrc map task @xcite , and photobook @xcite . @xcite follows a similar methodological procedure and computes entropy rate using fine-tuned gpt-2 models, claiming to support the entropy rate constancy hypothesis in the penn treebank but not in the dialogue datasets. in contrast, we focus on significantly larger news datasets, which are also more similar to the penn treebank data used in @xcite , and compute results across a wider range of model sizes. using larger datasets enables additional fine-tuning and reduces variance in the results; further, our focus on wordlevel surprisal provides additional granularity at the beginnings of documents, where entropy rate is least constant. finally, we note that perplexity is an extremely sensitive metric (cf. appendix 6.5), and large variation in results may be attributable to small differences in data. in particular, we do not expect the trends we observe in news articles to always transfer to other domains, such as spoken dialogue in @xcite . recent work has also sought to connect cognitive theories of efficient communication with techniques in natural language processing; in particular, operationalizations of the uniform information density hypothesis have been connected to natural language decoding @xcite and used as regularizers for language model training @xcite . we hope that an improved understanding of entropy rate constancy will inform such applications in the future. ## conclusion in this work, we computed entropy rate using trigram models and gpt-2, failing to find clear evidence in support of @xcite 's claim of entropy rate constancy in text. we provide results across various model sizes, with and without fine-tuning, and across several datasets, including the arabic billion words corpus. our work also provides one of the only analyses of entropy rate constancy in a language besides english, although see @xcite for results in russian and spanish. we encourage future work to further investigate the cross-linguistic validity of the uniform information density hypothesis.