Trading Off Diversity and Quality in Natural Language Generation (2021)

Abstract: For open-ended language generation tasks such as storytelling or dialogue, choosing the right decoding algorithm is vital for controlling the tradeoff between generation quality and diversity . However, there presently exists no consensus on which decoding procedure is best or even the criteria by which to compare them. In this paper, we cast decoding as a tradeoff between response quality and diversity, and we perform the first large-scale evaluation of decoding methods along the entire quality-diversity spectrum. Our experiments confirm the existence of the likelihood trap: the counter-intuitive observation that high likelihood sequences are often surprisingly low quality. We also find that when diversity is a priority, all methods perform similarly, but when quality is viewed as more important, nucleus sampling (Holtzman et al., 2019) outperforms all other evaluated decoding algorithms.

## introduction generative language models are applicable for a wide variety of tasks including writing articles, composing shakespearean sonnets, and engaging in conversation @xcite @xcite . this work examines decoding methods, a critical component in language models used in open-ended generative tasks where successful models must generate a diverse spectrum of high quality answers rather than merely a single output @xcite . for many tasks, these two criteria of quality and diversity are not equally important. in machine translation, the most important criteria is to produce an accurate, high-quality translation of the input; generating a variety of alternative translations is also useful, but not if it comes at the cost of correctness. meanwhile, in open domain dialogue figure 1 : the likelihood trap. for a given context, we generate 100 sentences of equal length spanning a variety of model likelihoods and ask human crowdworkers to rate their quality. while model loglikelihoods are generally positively correlated with average human quality judgments, we notice an inflection point after which they become negatively correlated. each point in the graph represents the average crowdworker rating of 5 sentences with similar model likelihoods. the goal is often to sustain an enjoyable conversation with a human conversational partner and as such, a higher premium is placed on diversity. to give a concrete example for the case of dialogue, the phrase "i don't know" is typically a perfectly reasonable remark that appears quite often in the course of normal human conversation. however, a chatbot that only repeats "i don't know" makes for a very poor conversationalist. in such open-ended domains, being able to converse about a wide variety of topics with the occasional odd remark is highly preferred to merely repeating the safest possible remark over and over @xcite . to evaluate both of these criteria, we characterize the performance of decoding algorithms along the entire quality-diversity spectrum instead of simply at individual points. we compare a variety of commonly-used decoding algorithms in the first large-scale study of decoder performance, utilizing over 38,000 ratings on almost 10,000 samples. our results indicate that when diversity is highly valued, all decoders perform similarly, but when quality is viewed as more important, the recently proposed nucleus sampling @xcite outperforms all other evaluated decoding algorithms. additionally, we investigate the commonly held intuition that model likelihood is directly correlated with human quality judgments by explicitly measuring the relationship between the quality of a sentence as judged by human raters and its likelihood under a generative model. our findings confirm the existence of a likelihood trap, the counter-intuitive observation that the highest likelihood sentences are of extremely low quality, despite a generally positive relationship between model likelihoods and human quality judgments. while this finding has been observed across a wide variety of models and tasks from news generation to machine translation @xcite , to our knowledge we are the first to explicitly quantify the relationship between the two across the entire model probability space. ## the likelihood trap sequence likelihood is commonly used as a heuristic for selecting high-quality generations. beam search, the principal approach adopted in machine translation, encapsulates this principle by (approximately) finding the single most likely generation argmax log p model (x). however, prior work has suggested that this assumption of a monotonically positive relationship between sequence likelihood and sequence quality breaks down at the extremes (section 5). we empirically quantify the relationship between sequence likelihoods and human quality judgments by subsampling a large number of context-continuation pairs representing a wide variety of model loglikelihoods. we then request human crowdworkers to rate the quality of each continuation given the context on a five-point "terrible"-to-"high quality" scale. figure 1 plots these ratings as a function of log p model and confirms that on average the highest quality generations are not the most likely. specifically, we find that continuation quality is generally positively related with log p model (x) up until an inflection point after which it becomes negatively related. our findings suggest that while model likelihoods form a good proxy for continuation quality, naively maximizing over sentence likelihood leads to suboptimal continuation quality. we term this phenomenon the likelihood trap. ## evaluation framework we introduce an evaluation framework for measuring the trade off quality and diversity in language generation. we consider autoregressive language models that decompose the likelihood of a sequence x 1:n token-by-token in a left-to-right fashion @xcite . specifically, the (conditional) probability of the sequence is: we evaluate the quality of a single sequence x 1:n by asking humans for a quality judgment hj(x). we can define the quality of a model q(p) = e x∼p [hj(x)] as the expected human "quality" judgment for sentences drawn from it. we measure the diversity of a model via the (conditional) shannon entropy h @xcite , a diversity metric widely used across many fields beyond computer science including biology, economics, chemistry, and physics. conditional shannon entropy is given by h since many metrics for measuring diversity in language generation exist in the literature, we validate our choice of entropy by measuring its correlation with other commonly used metrics of diversity based on n-gram frequency. we find the spearman correlation with distinct-1 and distinct-2 (number of distinct unigrams and bigrams divided by total number of n-grams) to be 0.80 and 0.77 respectively over sentences generated by gpt-2. our choices of using the average human quality judgement to measure quality and entropy to measure diversity guarantee that the optimal pareto frontier trades off monotonically between quality and diversity. optimizing quality with no regard for diversity results in outputting only the single highest quality sentence, whereas optimizing for diversity with no regard for quality results in outputting every utterance with equal probability. typical tasks in language generation (e.g. summarization, machine translation, storytelling) will fall somewhere in between these two extremes. since our models are imperfect, each decoding algorithm will, to the best of its ability, trace out its own estimate of this frontier. as most commonly used decoding strategies offer a knob to control the diversity of the generated text, we compare the performance of decoding algorithms by plotting their performance along various positions on the quality-diversity tradeoff curve. ## experiments we evaluate three commonly used decoding algorithms, sweeping across the quality-diversity curve by considering several hyperparameter settings per decoding algorithm. at the extremes of their hyperparameter ranges, these algorithms all converge to greedy and random sampling, respectively. ## related work encouraging diversity we choose to evaluate three commonly used decoding methods: nucleus sampling @xcite , top-k sampling @xcite , and temperature sampling. all three methods control the relative tradeoff between quality and diversity with a single hyperparameter as described in section 4, though many other decoding methods also exist in the literature. @xcite compares many of these algorithmic advancements on the tasks of open-ended dialog and image captioning, concluding that qualitydiversity tradeoffs make it difficult to say that any one method is ubiquitously best. ## conclusion in this paper, we propose a framework for credibly evaluating decoding algorithms and use it to conduct the first large scale evaluation of decoding algorithms by measuring their performance along the entire quality-diversity frontier. we observe that decoders can be tuned to produce higher-quality text, but that this improved quality comes at the cost of diversity. our findings suggest that existing decoding algorithms are largely interchangeable in high diversity settings, but that nucleus sampling performs best when quality is valued over diversity. we show that when performing a comparison of text generated from multiple decoding algorithms, it is crucial to ensure equivalent diversity to make the comparison fair, a step many evaluations fail to do. finally, we warn against falling for the likelihood trap, as selecting generated text that is too likely results in text that humans judge to be worse.