MAP ’s not dead yet: Uncovering true language model modes by conditioning away degeneracy (2024)

Abstract: It has been widely observed that exact or approximate MAP (mode-seeking) decoding from natural language generation (NLG) models consistently leads to degenerate outputs (Holtzman et al., 2019; Stahlberg and Byrne, 2019). Prior work has attributed this behavior to either a fundamental and unavoidable inadequacy of modes in probabilistic models or weaknesses in language modeling. Contrastingly, we argue that degenerate modes can even occur in the absence of any modeling error, due to contamination of the training data. Specifically, we argue that mixing even a tiny amount of low-entropy noise with a population text distribution can cause the data distribution’s mode to become degenerate. We therefore propose to apply MAP decoding to the model’s true conditional distribution where the conditioning variable explicitly avoids specific degenerate behavior. Using exact search, we empirically verify that the length-conditional modes of machine translation models and language models are indeed more fluent and topical than their unconditional modes. For the first time, we also share many examples of exact modal sequences from these models, and from several variants of the LLaMA-7B model. Notably, we observethat various kinds of degenerate modes persist, even at the scale of LLaMA-7B. Although we cannot tractably address these degeneracieswith exact search, we perform a classifier-based approximate search on LLaMA-7B, a model which was not trained for instruction following, and find that we are able to elicit reasonable outputs without any finetuning.

## introduction while it might intuitively be appealing to search for the highest-likelihood response (i.e., the mode) from a language model (lm), approximate map (maximum a posteriori) decoding approaches are known to produce degenerate outputs, such as excessive repetition of phrases, empty outputs, etc. @xcite @xcite . as a result, sampling approaches dominate the natural language generation landscape, reinforcing the belief that mode-seeking is undesirable for decoding from neural language models. most prior work has attributed this degeneracy to either biases/weaknesses in modeling, or has argued that we should not analyze models in terms of their modes due to the small probability it is assigned (e.g., @xcite @xcite . in this work, we argue instead that these degenerate modes will be found even when a model is perfectly trained, and therefore cannot solely be due to a bias found in a particular class of models. this occurs when the training data is contaminated with lowentropy distractors-noisy samples coming from a distribution with much lower entropy than the distribution of desirable outputs. 1 in section 2, we provide a simple theoretical demonstration of the fact that a higher variability of valid outputs necessarily lowers the noise rate required to produce a degenerate mode, regardless of the quality of modeling. this implies that the problem will persist, no matter how well we can fit our nlg models to a population data distribution. we argue that if the set of high-quality sequences is merely hidden beneath a relatively small set of distractors, one could eliminate the bad mode problem by conditioning away these undesirable outputs and finding the "clean" mode underneath. more concretely, consider a population or model distribution which assigns sequence y probability 2 p(y). we claim that even when the modal value of p(y) is degenerate, there often exists a relatively simple attribute function a(y), such that the conditional mode, argmax p(y|a(y) = a), is high-quality. we refer to search for these conditional modes as attribute-conditional search. one might worry that the modes of lms are fundamentally degenerate, and attribute-conditional search would just replace one type of low-quality output with another. for instance, it might avoid empty modes but introduce degenerate repetition in the output. surprisingly, we find that this is not the case for both an mt model and a story generation model. both models have degenerate empty outputs as their unconditional mode for a large fraction of inputs foot_2 , but their length-conditional modes (i.e., the highest scoring output of a given length) are almost universally high quality. our findings from exact length-conditional search are limited in two ways: finding exact modes of a given length is too expensive for realistic output lengths in mt, and furthermore mere length conditioning would not be sufficient to address the multiple failure modes observed in the llama models. to overcome these obstacles, we additionally consider approximate conditional search for these two settings. by using an attribute classifierguided beam search scheme, we show that we can find outputs which, compared to those found are ordinary beam search, are both higher likelihood and judged to be higher quality by automated metrics and human raters. these empirical findings support our claim that map-based methods should not be discarded purely due to the problem of degenerate outputs. in summary, our contributions are: ## causes of output degeneracy prior research has observed that the relationship between the likelihood assigned to a text by nlg models and its quality as assessed by human readers decorrelates in the high-likelihood regime. (see e.g., @xcite . two such well-known results are stahlberg and byrne (2019)'s finding that mt models often prefer the empty sequence to any other output and holtzman et al. (2019)'s demonstration that gpt-2 (radford et al., 2019) produces degenerate outputs under beam search. as we stated in section 1, many explanations have been offered for this phenomenon. we wish to emphasize a point that has been under-discussed in the nlp community: a model can emit high-quality samples with high probability while still having a degenerate mode, even in the absence of model error. we will specifically call this the bad mode problem. in this section, we will explain how lowentropy distractors can lead to this phenomenon and discuss the implications for designing inference algorithms. ## empirical characteristics of exact unconditional modes we study the nature of unconditional modes (highest-scoring outputs) of various kinds of autoregressive models, aiming to characterize the degenerate behaviors observed in modes across context types, model scales, and fine-tuning schemes. specifically, we use the depth-first search (dfs) method introduced by stahlberg and byrne (2019) to find exact modes for the following diverse set of models: (a) the marianmt chinese-english encoder-decoder machine translation model (tiedemann and thottingal, 2020), (b) a 345m-parameter gpt-2 model finetuned on the roc stories dataset (mostafazadeh et al., 2016), (c) a 7b-parameter general-purpose base llama model (touvron et al., 2023a), and (d) finetuned llama models for chat and instruction following, namely alpaca and guanaco (taori et al., 2023; dettmers et al., 2023). while exact decoding in exponentially large search spaces in lms appears intractable, stahlberg and byrne 2019 demonstrated that aggressive pruning makes it possible to find the exact modal output of a model. since the llama models are over 20 times larger than the mt and story generation models, we devise a novel caching strategy to reduce the memory footprint of dfs, which is explained in section c. to our knowledge, this work is the first to find the modes of models of this size. ## exact conditional modes: case study on length in the previous section, we observed that many different kinds of models suffer from degenerate modes and this problem does not seem to be abated by scale or finetuning. in this section, we answer the question of what lies beneath the empty modes by searching for modes of a length-conditional distribution: y * = argmax as such, we restrict our attention to conditioning on output lengths of 12 tokens or less. @xcite ran a similar search, but used a lower bound rather than an exact constraint. however, they did not include any sample outputs of this kind, so the qualitative properties of such sequences are not widely known. in this section, we share a large number of lengthconstrained modes from mt models, and also extend this method to story completion. ## conclusion in this work we argued that low-entropy distractors can lead to degenerate nlg model modes, even in the absence of model error. we investigated the exact modes of several autoregressive models, finding several kinds of degenerate behavior, both replicating and extending prior work. motivated by this, we explored two kinds of conditional search: exact length-conditional search, and approximate lengthconditional and reward-conditional search. these methods showed that these models do support outputs which are simultaneously high-likelihood and high-quality, a fact which is surprising in light of most of the work on this topic. we hope this exposition inspires the community to explore improvements to map-based methods, rather than abandoning them entirely in favor of sampling and reward-maximization methods.