On Decoding Strategies for Neural Text Generators (2022)

Abstract: When generating text from probabilistic models, the chosen decoding strategy has a profound effect on the resulting text. Yet the properties elicited by various decoding strategies do not always transfer across natural language generation tasks. For example, while mode-seeking methods like beam search perform remarkably well for machine translation, they have been observed to lead to incoherent and repetitive text in story generation. Despite such observations, the effectiveness of decoding strategies is often assessed on only a single task. This work—in contrast—provides a comprehensive analysis of the interaction between language generation tasks and decoding strategies. Specifically, we measure changes in attributes of generated text as a function of both decoding strategy and task using human and automatic evaluation. Our results reveal both previously observed and novel findings. For example, the nature of the diversity–quality trade-off in language generation is very task-specific; the length bias often attributed to beam search is not constant across tasks. https://github.com/gianwiher/decoding-NLG

## introduction modern neural networks constitute an exciting new approach for the generation of natural language text. much of the initial research into neural text generators went into designing different architectures @xcite @xcite . however, recent work has hinted that which decoding strategy (i.e., the method used to generate strings from the model) may be more important than the model architecture itself. for instance, a well replicated recent result is that, under a probabilistic neural text generator trained with the maximum-likelihood objective, the most probable string is often not human-like or high quality @xcite . in light of this finding, a plethora of decoding strategies have been introduced in the literature, each claiming to generate more desirable text than competing approaches. lamentably, empirical studies of decoding strategies are typically evaluated on a single natural language generation task-without investigation into how performance may change across tasksdespite the fact that these tasks differ qualitatively across a large number of axes. these qualitative differences manifest quantitatively as well: for example, we can see in figure 1 that high probability strings are favorable in some tasks, like machine translation (mt), while heavily disfavored in others, like story generation (sg). consequently, we should not a priori expect a strategy that works well for one task to demonstrate the same performance in another. indeed, several cases already show evidence of this: beam search works remarkably well for machine translation but, outside of this context, has been observed to return dull text or degenerate text @xcite . this raises a natural fear that decoding strategies have been optimized for performance on a specific task, and the task-agnostic claims about the effectiveness of one decoding strategy over another are potentially ill-founded. a broader analysis of decoding strategies-both within and across tasks-is needed in order to fully understand the extent of such a problem. our work fills this gap, providing the first comprehensive comparison of decoding strategies across natural language generation tasks. empirically, we compare strategy performance on several axes, taxonomizing methods into groups such as deterministic and stochastic, to understand the importance of various strategy attributes for quantifiable properties of text. in summary, our main findings include the following: we see these results as both a reference point for language generation practitioners, so that they can more confidently choose a decoding strategy that fits their needs, and as an indicator of potential strengths and weaknesses of today's neural probabilistic language generators. we have reason to believe that there is a task-specific optimization happening in the literature whereby many of the proposed and (even celebrated) decoding strategies only outperform their competitors on specific tasks. thus, our paper serves as a cautionary note about proper comparisons. ## probabilistic language generators in this work, we consider models for language generation tasks that define a probability distribution over strings. more formally, these models are probability distributions q over an output space y-(perhaps) conditioned on an input x-where y is the set consisting of all possible strings that can be constructed from the vocabulary v: ## the decoding problem we define the decoding problem as the search for a string y * according to a given model q and a set of decision rules. given the probabilistic nature of most language generators, the natural choice for such a string would be the most probable sequence under the model solving the optimization problem in eq. ( 3 ) is commonly referred to as maximum a posteriori (map) decoding. there are two main reasons why in practice this direct optimization is not used when decoding: first, because of the exponentially large space y and the non-markovian structure of commonly used neural generators, direct optimization is often computationally infeasible. second, recent research has shown that the mode (i.e., the map solution y * ) is often not human-like or high quality text @xcite . for example, in the domain of mt, the most likely string under the model is often the empty string @xcite . for open-ended generation, 2 it has been observed that there is a positive correlation between likelihood and quality up to only a certain inflection point, after which the correlation becomes negative @xcite . thus in practice, y * is almost exclusively approximated using heuristic methods. an overview of such (commonly used) methods is presented below. ## experimental setup the strategies presented in section 3 are compared across a variety of nlg tasks covering open-ended as well as directed generations tasks. we define a task more formally as a triple (x , y out , q) where x denotes the input space, y out ⊆ y the output space foot_2 and q a model that defines a probability distribution over y out for every input @xmath0. a high-level overview of these tasks (and the respective datasets used) can be found in ## discussion when constructing a text generation pipeline, the choice of decoding strategy has a large effect on various aspects of the resulting text. yet when making this choice for a specific language generation task, practitioners are currently limited to either basing their decision on non-comprehensive analyses, using expensive human annotations or even resorting to guesswork. there are potential pitfalls in these practices: as evidenced by various results in this work, certain properties of decoding schemes-especially quality-do not transfer across tasks. this work aims to provide guidance for practitioners in the choice of decoding strategies, revealing their strengths and weaknesses with respect to individual tasks while also giving insights into whether one can expect these properties to transfer to tasks outside of this study. while all of the takeaways from this work cannot be summarized in a few lines, we highlight some key observations below. the relationships and trade-offs between certain properties of text changes notably from one task to another. for example, as depicted in figure 1 , high-probability strings are typically also of high quality for mt while there is an almost inverse relationship between these attributes for sg. as shown in figure 5 , a quality-diversity trade-off exists for directed generation tasks whereas for open-ended generation tasks, the relationship is almost a co-dependence. these task-specific characteristics must be taken into account when both choosing and developing decoding strategies. while decoder performance generally does not transfer faithfully across tasks, we can still identify some rules from our experiments that practitioners can use. for one, we see that on directed generation tasks, mode-seeking methods all perform competitively in terms of quality. further, for stochastic decoders, we observe that restricting the sample space-as done in top-p and top-k decoding-greatly increases quality compared to ancestral sampling, albeit sacrificing some diversity. the ability of a decoder to elicit diversity in text-at least at the set-level-is perhaps the most consistent decoder quality across tasks. there are many other use-case specific insights that can be drawn from the results shown by figures and statistics in this work, which we hope serve as further guidance for practitioners. it is worth noting that the behavior of decoders depends on their respective hyperparameters, for example, k or p in top-k and top-p sampling. this work does not perform a thorough search over hyperparameters, instead utilizing those most widely used in order to optimize for the usefulness of our results to practitioners, who are likely to use similar default settings. while based on the results of other works, these choices should provide representative variants of the text generated according to the respective decoding strategy, this is a limitation of our work worth taking into consideration. ## conclusion this work provides an extensive analysis of the effects of different decoding strategies on generated text across various language generation tasks. we show how different attributes of model-generated text change depending not just on decoding strategy, but also on the task at hand, using both human and automatic evaluations. our results both confirm several prior observations, for example, a trade-off between diversity and quality metrics for specific nlg tasks, while also revealing a number of previously unobserved trends in language generation, both with respect to decoding strategies and the tasks themselves. a main take-away of these results is that decoding strategies are perhaps optimized for specific language generation tasks and that practitioners should take great care in basing their choice of decoding strategy off of results reported for alternate tasks. we release the evaluation framework and generations in the hopes that this type of analysis will be extended, for example, by ablating components of model or training strategies, in order to isolate which artifacts can be attributed to the nature of a specific generation task vs. design choices. we ultimately see this line of research as important for helping practitioners more confidently choose a decoding strategy that fits their needs without the use of valuable resources, for the further development of decoding strategies and for better understanding the shortcomings of probabilistic language generators.