On the probability–quality paradox in language generation (2022)

Abstract: When generating natural language from neural probabilistic models, high probability does not always coincide with high quality: It has often been observed that mode-seeking decoding methods, i.e., those that produce high-probability text under the model, lead to unnatural language. On the other hand, the lower-probability text generated by stochastic methods is perceived as more human-like. In this note, we offer an explanation for this phenomenon by analyzing language generation through an information-theoretic lens. Specifically, we posit that human-like language should contain an amount of information (quantified as negative log-probability) that is close to the entropy of the distribution over natural strings. Further, we posit that language with substantially more (or less) information is undesirable. We provide preliminary empirical evidence in favor of this hypothesis; quality ratings of both human and machine-generated text—covering multiple tasks and common decoding strategies—suggest high-quality text has an information content significantly closer to the entropy than we would expect by chance.

## introduction today's probabilistic neural language models are often trained on millions-if not billions-of lines of human text; thus, at least at an intuitive level, we would expect high-probability generations to be human-like. yet the high-quality foot_0 texts these models have become famous for producing (brown et al., 2020; clark et al., 2021) are usually not those assigned the highest probability by the model (fan et al., 2018; holtzman et al., 2020; basu et al., 2021; delucia et al., 2021). rather, the relationship between probability and quality appears to have an inflection point, foot_1 i.e., quality and probability are positively correlated only until a certain threshold, after which the correlation becomes negative. while the existence of such a trend has received informal explanations (see, e.g., ippolito et al. (2019) and zhang et al. (2021) for a qualitative discussion about the trade-off between diversity and quality), it lacks a more fundamental understanding. why does the lower probability text produced by stochastic decoding methods-such as nucleus or top-k sampling-outperform text generated using probability-maximizing approaches? in this note, we take an information-theoretic approach in an attempt to answer this question. in information theory, probability has another interpretation: its negative log quantifies information content. in the context of natural language, the notion of information content is intuitive; humans use strings as a means to convey information. further, less predictable text, i.e., text which would be harder for us to anticipate, conveys more information. if we assume that the goal of human communication is to transmit messages efficiently and reliably (gibson et al., 2019), we may predict that these strings' information content should concentrate inside a specific interval. at one extreme, strings with more-than-expected information may be hard to process, and thus ought to be disfavored when producing language. foot_2 at the other extreme, low-information strings may be seen as boring and uninformative. collectively, these concepts lead us to propose the expected information hypothesis: text perceived as human-like should have an information content within a small interval around the expected information-i.e., the entropy-of natural language strings. such a hypothesis offers an intuitive explanation for the trends observed in natural language generation (nlg), i.e., why desirable text seems to exist not always at the high end of the probability spectrum but around a certain inflection point. @xcite moreover, it also gives us a testable hypothesis: given a language generation model q whose entropy we can empirically estimate, we can evaluate whether high-quality text indeed has an information content that falls within an interval around this quantity. to test our hypothesis, we perform an analysis comparing human and model-generated text, investigating multiple common decoding strategies and nlg tasks. specifically, our analysis focuses exclusively on english text. we indeed observe that the information content of highly ranked text (as judged by humans) often falls within a standard deviation of model entropy; there is statistically significant evidence that this is not due to chance. further, the best-performing decoding methods appear to select strings with an information content within this interval. we take these observations as empirical support for our hypothesis, helping to explain the probability-quality paradox observed in language generation. ## probabilistic language generators in this work, we focus on probabilistic models for language generation tasks. formally, these models are probability distributions q over natural language strings @xmath0, where y is the (countably infinite) set consisting of all possible strings that can be constructed from a set vocabulary v: here, bos and eos stand for special reserved beginning-and end-of-string tokens, respectively, and v * denotes the kleene closure of v. in practice, we limit the set of strings we consider to y n ⊂ y for some maximum sequence length n . note that q may be a conditional model. for instance, we may model q(• | x) where x is an input text, as in the case of machine translation, or an input image, as in the case of image captioning. however, for notational brevity, we omit this explicit dependence in most of our subsequent analyses. in order to estimate q, it is standard practice to maximize the log-probability of a 4 similar ideas have been used to improve language models and language generation before (meister et al., 2020; wei et al., 2021). there are many different decision rules one can employ for generating natural language strings from a model q; such sets of rules are generally referred to as decoding strategies; see wiher et al. (2022) for an in-depth review. given the probabilistic nature of the models we consider, an intuitive strategy for decoding would be to choose the string with the highest probability under q, an approach referred to as maximum-a-posteriori (map) decoding. @xcite yet recent research has shown that solutions to map decoding-or, even more generally, to heuristic mode-seeking methods such as beam search-are often not high-quality, even in state-of-the-art nlg models. for example, in the domain of machine translation, the most probable string under the model is often the empty string (stahlberg and byrne, 2019). similarly, in the domain of openended generation, mode-seeking methods produce dull and generic text (holtzman et al., 2020). where maximization has failed, authors have turned to stochastic methods, taking random samples from q. while the resulting text is often assigned much lower probability than the mode, it can be qualitatively much better. this peculiarity has puzzled the language generation community for the last few years, with only qualitative intuitions being offered as explanation. this paper in turn offers a quantitative explanation. ## language as communication while many aspects of natural language may not perfectly adhere to shannon's mathematical theory of communication, there are several characteristics of human language that can fruitfully be described using an information-theoretic framework. @xcite here we employ this framework for explaining recent phenomena observed in probabilistic nlg. ## experiments our experiments present an analysis of the distribution of information content in text generated by both humans and probabilistic models. specifically, we look at the relationship between information content and quality-as measured by human judgments. we perform experiments on two natural language generation tasks: abstractive summarization and story generation. we present the results for story generation here, while the results for summarization can be found in app. b due to space constraints. a recreation of the probability versus quality plots of zhang et al. (2021) can also be found in app. b. we use the following monte carlo estimator for the entropy, i.e., expected information content, of our model q: ## conclusion in this work, we present the expected information hypothesis, which states that human-like strings typically have negative log-probability close to the expected information content of the probabilistic model from which they were generated. we use this hypothesis to explain why high-quality text seems to exist not necessarily at the high end of the probability spectrum but, rather, close to the entropy of the model. we provide empirical evidence in support of our hypothesis in an analysis of both human and machine-generated text, demonstrating that, overwhelmingly, high-quality text indeed has information content in the proposed region.