A Cognitive Regularizer for Language Modeling (2021)

Abstract: The uniform information density (UID) hypothesis, which posits that speakers behaving optimally tend to distribute information uniformly across a linguistic signal, has gained traction in psycholinguistics as an explanation for certain syntactic, morphological, and prosodic choices. In this work, we explore whether the UID hypothesis can be operationalized as an inductive bias for statistical language modeling. Specifically, we augment the canonical MLE objective for training language models with a regularizer that encodes UID. In experiments on ten languages spanning five language families, we find that using UID regularization consistently improves perplexity in language models, having a larger effect when training data is limited. Moreover, via an analysis of generated sequences, we find that UID-regularized language models have other desirable properties, e.g., they generate text that is more lexically diverse. Our results not only suggest that UID is a reasonable inductive bias for language modeling, but also provide an alternative validation of the UID hypothesis using modern-day NLP tools.

## introduction language has been hypothesized to follow certain information-theoretic constraints. one of the most famous of these constraints is the uniform information density (uid) hypothesis @xcite , which states that, subject to the rules of the grammar, speakers aim to distribute information density across a linguistic signal as uniformly as possible. that is, speakers behaving optimally should structure their utterances such that the differences between the peaks and troughs in information are minimized. in the psycholinguistics literature, the uid hypothesis has been used to explain a variety of linguistic phenomena ranging from how we shorten the phonetic duration of more-predictable linguistic (a) (b) figure 1 : graphical illustration of two examples regarding uid. in (a), many speakers will prefer the version with the relativizer that (dotted blue line). the uid hypothesis posits that this is because, without the relativizer, the first word of the relative clause, we, has high information density; and so including the relativizer distributes the per-word information density more uniformly. in (b), the relativizer that is often omitted because, at the onset of the relative clause, the information density of i is lower and therefore the distribution of information density is already relatively uniform. illustration based on @xcite . in this paper, we propose a new experimental paradigm that uses modern-day nlp models to test the uid hypothesis. whereas prior work has used language modeling as a tool for observing uid, foot_0 we explore the converse-can uid be used as a tool to train better language models? specifically, if the uid hypothesis is true, then we should be able to operationalize uid as a regularizer to help train language models. moreover, observing lower perplexity in language models trained with this regularization would imply that the concept of uid is a good inductive bias for language modeling, thereby providing a new type of evidence for the uid hypothesis at scale. in experiments, we indeed find such evidence: across a variety of languages and dataset sizes, uid regularization consistently improves performance, having a larger effect when training data is limited. moreover, we observe that-in comparison with their unregularized counterparts-uidregularized language models are (1) higher entropy while achieving the same (or better) test set perplexity and ( 2 ) generate text that is longer and more lexically diverse. our work is the first to explore the interaction between uid and training modernday neural language models, and our findings-that a cognitively motivated objective can improve language model performance-open up new avenues for testing other psycholinguistic hypotheses in a similar framework. ## preliminaries: language modeling the task of language modeling aims to estimate a model of the probability of observing any given string in (a subset of) natural language. formally, a language model p is an (unconditional) probability distribution over sequences of words @xmath0, w 2 , . . . , where w consists of tokens from some vocabulary and begins and ends with special tokens bos and eos, respectively. today's language models are typically parameterized by neural networks (e.g., transformers @xcite ), that follow a localnormalization scheme. specifically, the model provides a conditional distribution over the vocabulary at each time step; we can then compute the proba-bility of an entire sequence w as: parameters are estimated by optimizing over some objective l(θ). the standard objective for language modeling is the negative log-likelihood of a dataset w under the model: subsequently, we drop explicit dependence on θ when it is obvious from context. to assess the goodness of fit of a model p, we typically evaluate its perplexity on some held-out dataset w test , where perplexity (ppl) is defined as note that under this definition of perplexity, our evaluation metric is slightly different than the training objective; the former computes an average over each sequence while the later treats all tokens equally, regardless of the length of the sequence in which they are present. ## uniform information density communication via natural language is a complicated and nuanced process that takes place under a host of cognitive and environmental constraints. as a result, speakers have to make (perhaps subconscious) choices to best navigate this communicative dance. a rational speaker would use these choices to optimize the communicative properties of their utterances. one such locus of optimization is outlined by the uniform information density (uid) hypothesis. ## uid-regularized language modeling while prior work has shown evidence that uid can help explain many of the choices we make when generating language, to the best of our knowledge, operationalizations of uid have not been explicitly employed as part of the training objective in modern-day nlp models. this raises the simple question that is central to our paper: can uid serve as an inductive bias for training statistical language models? in an effort to answer this question, we present a scheme for incorporating operationalizations of uid into the language model training objective. formally, we augment the canonical maximum likelihood estimation objective foot_1 in eq. ( 2 ) with uid operationalizations as regularizers r. under this new objective, we minimize ## results in this section, we report results for models trained under the uid-regularized objective. we find that uid regularization consistently improves perplexity for models trained on various languages ( §6.1) and dataset sizes ( §6.2). additionally, we examine properties of text generated by uid-regularized models ( §6.3) and analyze the relationship between our operationalization of uid and perplexity ( §6.4). ## discussion and related work we discussed how operationalizing uid for language modeling leads to better models in a wide variety of settings. these results both provide a new form of evidence for the uid hypothesis and build on prior work exploring uid in modern-day nlp models. evidence for the uid hypothesis. our work extends the body of psycholinguistic research on uniform information density, which has largely corroborated the uid hypothesis by providing evidence that variation in surprisal, as estimated by a lan-guage model, is minimized in natural language. in addition to early studies that used this approach to find evidence for uid in syntactic reduction @xcite , morphosyntactic contractions @xcite , and prosodic structure @xcite , the same line of reasoning has been used by more recent work exploring a variety of other linguistic properties. these studies have found that word duration can be predicted by syntactic surprisal @xcite , construction probability (kuperman and bresnan, 2012), informativity @xcite , and contextual predictability @xcite @xcite . they have also observed that word length is reflected by conceptual complexity @xcite ; word order choice can be predicted by processing cost @xcite ; phonological patterns can be shaped by word predictability @xcite ; and uid computed at the sequence level predicts human preferences for syntactic alternatives of the same sentence. whereas the above prior work has used language modeling as a tool for measuring uid, our paper has explored the exact converse-we have asked whether uid, operationalized as a regularizer, can be used as a tool for training better language models. we argue that if the uid hypothesis holds as a general principle, then we should be able to exploit it as a training criterion that improves language modeling. and accordingly, our results show that-across a variety of languages and dataset sizes-regularization for uid did indeed improve perplexity, which we view as an alternative kind of evidence for the uid hypothesis at scale. notably, figure 3 at first could appear to contradict the uid hypothesis, since models with better uid behavior did not always achieve better perplexity. we do not consider this as evidence against the uid hypothesis, however. rather, we posit that when β is too large, we may be optimizing for uid to the point of tending towards unnatural language-a perfectly uniform dispersion of information across an utterance may come at the cost of strange lexical choices. in this light, such a trade-off should be somewhat expected. uid in modern nlp. in addition to the traditional line of psycholinguistic work, there have also been more-recent studies on uid in the context of modern nlp, although this work is relatively sparse. @xcite leverage infor-mation density encoded as surprisal at the word, part of speech, and syntax levels to help build a state-of-the-art model for mixed-domain translationese detection. @xcite incorporate uid measures across sentences into models designed to detect natural versus manipulated text. perhaps the work that is most related to ours, @xcite , leverages uid to explain why beam search is an effective decoding algorithm and uses operationalizations of uid during beam search to alleviate problems with decoding poorly calibrated machine translation models. whereas meister et al. (2020a) focuses on decoding, our work shows the first evidence that uid can be operationalized to aid training. ## conclusions in closing, we have proposed encoding uniform information density as a regularizer for training language models-a novel manner of incorporating an established psycholinguistic theory into modern statistical language modeling. in experiments on a range of languages and dataset sizes, uid regularization consistently improves perplexity over baselines. our results suggest that uid is a valid inductive bias for improving the canonical maximum likelihood objective in language modeling, providing a new, alternative type of evidence that supports the uid hypothesis at scale. our work opens the door to future research directions such as using similar techniques to validate other psycholinguistic phenomena, applying uid regularization in conditional language generation tasks, and exploring how uid regularized models perform in downstream nlp applications.