Contrastive Decoding: Open-ended Text Generation as Optimization (2023)

Abstract: Given a language model (LM), maximum probability is a poor decoding objective for open-ended generation, because it produces short and repetitive text. On the other hand, sampling can often produce incoherent text that drifts from the original topics. We propose contrastive decoding (CD), a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint. The contrastive objective returns the difference between the likelihood under a large LM (called the expert, e.g. OPT-13B) and a small LM (called the amateur, e.g. OPT-125M), and the constraint ensures that the outputs are plausible. CD is inspired by the fact that the failures of larger LMs (e.g., repetition, inco- herence) are even more prevalent in smaller LMs, and that this difference signals which texts should be preferred. CD requires zero additional training, and produces higher quality text than decoding from the larger LM alone. It also works across model scales (OPT-13B and GPT2-1.5B) and significantly outperforms four strong decoding algorithms (e.g., nucleus, top-k) in automatic and human evaluations across wikipedia, news and story domains.

## introduction open-ended text generation aims to craft fluent and coherent textual continuations of given prompts, laying foundations for various downstream applications such as writing assistance and story generation @xcite . the canonical approaches often sample from large pre-trained language models @xcite @xcite , but the generated text is prone to incoherence and topic drift as unlucky sampling choices compound over long sequences @xcite . on the other hand, searching for the most likely se-figure 1 : contrastive decoding exploits the contrasts between expert and amateur lm of different sizes by choosing tokens that maximize their log-likelihood difference. cd produces high-quality text that amplifies the good expert behavior and diminishes the undesired amateur behavior. we propose a new search-based approach, contrastive decoding (cd), that can generate fluent and lexically diverse text without compromising coherence. as shown in figure 1 , contrastive decoding takes an off-the-shelf large language model such as opt-13b (that we call the expert) and an off-the-shelf smaller language model such as opt-125m (that we call the amateur). cd searches for text that maximizes the difference between expert log-probabilities and amateur log-probabilities, subject to plausibility constraints which restrict the search space to tokens with sufficiently high probability under the expert lm. contrastive decoding works because many failure modes of language models (short, repetitive, irrelevant or uninteresting strings) are more common under smaller lms than under larger lms. such outputs are further deemphasized by taking the difference between model log-probabilities. conversely, stronger models tend to put more probability mass on desirable outputs, such as those with factual knowledge that has not been learnt by the weaker model, and these strings are emphasized by contrastive decoding. taking figure 1 as an example, the expert model places significant probability mass on previous tokens such as "hawaii" and "honolulu", leading to a highly repetitive continuation from greedy search; and nonsensical tokens such as "washington" may be sampled, leading to an incoherent continuation. a correct continuation "1961" is strongly preferred by contrastive decoding, despite only having a probability of 0.1, and the continuation includes more correct facts. this example suggests that contrastive decoding generates outputs that emphasize the best of the expert lm and remove its amateur tendencies. moreover, we provide a pragmatic interpretation of contrastive decoding in §4. compared to recent training-based methods that improve generation quality such as unlikelihood training @xcite and contrastive learning @xcite , contrastive decoding requires zero additional training. we find that by simply contrasting two frozen language models of different sizes, we are able to decode higher quality text than from the larger lm alone. furthermore, we find that better performance is achieved when the scale difference between expert and amateur is larger ( §7.1). as a result, the optimal amateur model is also cheap to run and incurs very little inference time overhead. we evaluate our contrastive decoding approach for open-ended text generation in three domains: wikipedia, stories, and news, and we evaluate using different teacher-student combinations, including (gpt2-xl v.s. gpt2-small, opt-13b v.s. opt-125m). compared to four decoding baselines (nucleus sampling, top-k, typical decoding and simctg) our contrastive decoding method significantly improves the coherence of generated text, and improves or maintains the same fluency levels, according to both human evaluation and automatic metrics. ## problem statement we consider decoding approaches for open-ended language generation, where the language models receive an input prompt and aim to generate a fluent and coherent continuation. specifically, we consider a relatively short prompt of length n, denoted as x @xmath0, where x i is a token in the vocabulary v. the decoder must generate continuations of length m, denoted as we generate text from a pre-trained autoregressive language model p lm . at decoding time, we iteratively decode one token at a time by conditioning on the preceding context: one canonical decoding approach is to sample from a truncated next token distribution at each time step. for example, nucleus sampling @xcite draws from the top p percentile of the next token distribution; top-k sampling @xcite draws from the top k candidates in the next token distribution. another common approach is to search for the most likely text sequence via greedy decoding or beam search @xcite ; but this leads to repetition and tedious outputs. ## contrastive decoding we propose contrastive decoding as a search-based decoding method that optimizes a novel contrastive objective subject to our plausibility constraint. we first provide intuition and define the constrastive objective ( §3.1). second, we discuss the potential weakness of this objective alone, and introduce the plausibility constraint to correct for the weakness ( §3.2). then we define the full contrastive decoding method as our contrastive objective subject to the plausibility constraint ( §3.3). finally, we elaborate on the design spaces by discussing the choices of amateurs ( §3.4). ## cd as pragmatic communication having formally described contrastive decoding, we now provide a pragmatic interpretation, justifying its validity through pragmatic communication goals . a line of work in pragmatics @xcite characterizes communication as a cooperative process between speakers and listeners. several of these formalisms @xcite describe a tradeoff between speakers and listeners, where a speaker should generally produce language that is high quality (e.g. truthful, fluent, and relevant) while also being informative to a listener. our contrastive objective can be motivated by this tradeoff, with our expert and amateur lms modeling a knowledgable speaker and a lessinformed listener: (1) upweighting tokens by p exp and using our expert-based plausibility constraints generates tokens that have high probability under the expert lm, encouraging generated text to be fluent and relevant (e.g. upweighting '1961' in figure 1 ). (2) downweighting tokens by p ama suppresses language that is predictable by (i.e. less informative to) the amateur lm (e.g. downweighting 'honolulu' and 'washington'), and by proxy encourages the language to be informative to a listener in context. by combining these two criteria, our contrastive decoding method produces high quality text that satisfies the communicative goal of transferring relevant but not predictable information. ## conclusion and future work we propose contrastive decoding, a search-based decoding approach that contrasts lms of different scales. we evaluate our approach on open-ended text generation, and find that it improves over the prevalent methods like nucleus sampling in both fluency and coherence. as future work, the idea of contrasting an expert (larger lm) and an amateur (smaller lm) can be expanded to myriad setups, for instance, contrasting an early checkpoint of an lm and a later checkpoint of the lm. we hope that this paper can encourage more exploration of how to use contrasting language models.