Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LM (2024)

Abstract: Contrastive decoding (CD) (Li et al., 2022) improves the next-token distribution of a large expert language model (LM) using a small amateur LM. Although CD is applied to various LMs and domains to enhance open-ended text generation, it is still unclear why CD often works well, when it could fail, and how we can make it better. To deepen our understanding of CD, we first theoretically prove that CD could be viewed as linearly extrapolating the next-token logits from a huge and hypothetical LM. We also highlight that the linear extrapolation could make CD unable to output the most obvious answers that have already been assigned high probabilities by the amateur LM.To overcome CD’s limitation, we propose a new unsupervised decoding method called Asymptotic Probability Decoding (APD). APD explicitly extrapolates the probability curves from the LMs of different sizes to infer the asymptotic probabilities from an infinitely large LM without inducing more inference costs than CD. In FactualityPrompts, an open-ended text generation benchmark, sampling using APD significantly boosts factuality in comparison to the CD sampling and its variants, and achieves state-of-the-art results for Pythia 6.9B and OPT 6.7B. Furthermore, in five commonsense QA datasets, APD is often significantly better than CD and achieves a similar effect of using a larger LLM. For example, the perplexity of APD on top of Pythia 6.9B is even lower than the perplexity of Pythia 12B in CommonsenseQA and LAMBADA.

## introduction contrastive decoding @xcite (cd) is a simple heuristic that uses the logit of a small lm (amateur lm) to improve the logit of a large figure 1 : given a simple question with clues for which a tiny amateur lm could provide a correct answer, contrastive decoding (cd) could have a "obvious blindness" (i.e., assigning a higher logit to an uncommon answer invertebrate than the most obvious answer bees). in contrast, the proposed asymptotic probability decoding (apd) correctly assigns the highest probability to bees by leveraging the probabilities from multiple lms of different sizes to extrapolate the probabilities from an infinitely large and hypothetical lm. lm (expert lm). 2 the potential of cd has been demonstrated in various open-ended text generation tasks @xcite and reasoning tasks using the expert lms up to @xcite . several variants are also proposed to reduce toxicity @xcite , improve factuality in nlp tasks @xcite @xcite ), text evaluation @xcite , and vision tasks @xcite . however, due to the insufficient theoretical understanding of cd, it is difficult to identify and overcome the failure modes of cd, which hinders its wide applications. scaling law demonstrates that language models (lms) are able to generate more factual next tokens as their sizes increase @xcite . however, their growing energy consumption and costs limit their further applications @xcite @xcite , necessitating techniques that can reduce lm model sizes without compromising their superior performances. in this work, we theoretically demonstrate how contrastive decoding (cd) addresses this lm size-reduction challenge using a simple linear extrapolation. the theory also helps us to identify the limitations of cd and propose apd, a more factual decoding method. first, we discover that cd actually uses the tiny amateur lm to help the large expert lm infer the logit of a huge and hypothetical lm. specifically, the logits from cd could often be viewed as a linear extrapolation of the logit curves from the expert lm and amateur lm. the finding explains several prior empirical observations and also reveals weaknesses of cd. for example, cd tends to neglect the most obvious answer and overemphasize less likely answers in its output distribution instead. we call this tendency "obvious blindness". the rare answers could sometimes degrade the generation's factuality. for example, both amateur and expert lm in figure 1 can identify that bees is the most clear answer suggested by the clues but only the expert lm realizes that there are also some other possible answers such as invertebrates. then, the aggressive linear extrapolation of cd makes invertebrates become the most probable next token. this is not a totally factual answer because many invertebrates cannot fly. motivated by this theoretical explanation, we propose a novel decoding method called asymptotic probability decoding (apd). apd predicts the asymptotic probability from a hypothetical lm with an infinite size. by explicitly modeling the changes of the next-token probabilities as the size of lm increases, apd is able to output the correct probabilities for both easy/common and difficult/uncommon answers. for the example in figure 1 , the probabilities of bees and invertebrates are both increasing as the lm becomes larger. by leveraging the probabilities of mid-size lms, we can reasonably infer that bees should still receive a larger probability from a huge lm than invertebrates. finally, modeling the probability curves for many next tokens on the fly is too time-consuming, so we fine-tune an amateur lm such that the output probability of apd is close to asymptotic probability, which makes apd as efficient as cd. the main goal of our experiments is to check if apd can further improve the factuality compared to cd. we choose our expert lms and amateur lms from the llm families that provide smaller lms, including pythia (6.9b, 70m) @xcite , opt (6.7b, opt 125m) @xcite , and qwen1.5 (4b, 0.5b) @xcite . by comparing different sampling methods in fac-tualityprompts @xcite , we demonstrate that apd consistently and robustly outperforms cd with the best temperature and other stateof-the-art distribution modification methods such as dola @xcite and temperature sampling @xcite . after being combined with dynamically adjusted top-p sampling @xcite , our method can help pythia 6.9b to simultaneously achieve the factuality of top-p sampling @xcite with @xmath0.4 and diversity of top-p with @xmath1.7. we also compare the perplexity of apd and cd using seven datasets. we found that the improvement gap is especially large when cd makes more mistakes on easier tasks. for example, in lambada @xcite and common-senseqa @xcite , apd on top of pythia 6.9b could achieve a similar or better perplexity than pythia 12b, while outperforming cd by a large margin. we plan to release our code to reproduce the results after our work is accepted. our main contributions include • we provide theoretical support for contrastive decoding (cd) and demonstrate that our theory can explain many prior findings from @xcite ; @xcite . ## contrastive decoding as extrapolation first, we review contrastive decoding (cd) and its justification. in cd, the probability of the next token w for context c is determined by log(model size) logit of a next token amateur lm (alm) expert lm (elm) hypothetical lm (hlm) where x is all the words in the vocabulary, and the logit for the token w is here, we provide another intuitive justification of equation (2): elm and alm usually come from the same llm family and they are trained on the same training corpus using similar hyperparameters, so their main difference is the model sizes. l cd c (w) would be smaller when l elm c (w) is small and l alm c (w) is large. in this case, it means that the lms' logits are decreasing for the token w as the lm's model size increases. we can reasonably infer that the larger lm will output an even smaller logit if the trend continues. thus, cd actually infers the logit of a larger lm by subtracting l alm c (w) from l elm c (w). we describe this intuition in a formal way next. setting @xmath2, 2023), so our first assumption often holds in practice. the linear relationship means we can always draw a line to connect all the logits from lms with different sizes as in figure 2 . since t is a global hyperparameter, l cd c (w) is the logit of hlm using the temperature t t -1 . proof. from figure 2 , we can see that the size difference between hlm and elm should be ## asymptotic probability decoding we propose asymptotic probability decoding (apd) to overcome the limitations of cd. in an llm family, there are often lms with sizes between the sizes of the amateur lm (alm) and expert lm (elm). as shown in figure 1 , apd leverages them to improve the extrapolation and models the probability curves instead of the logit curves to avoid outputting a large probability based on a very small logit from alm (e.g., invertebrates). furthermore, for those easy answers (e.g., bees), l elm c (w) and l alm c (w) are both high. apd can extrapolate the high probability curve to output a desired high probability while based on equation (2), the cd might output a low logit. to make apd as efficient as cd foot_2 , apd requires us to fine-tune the alm in cd such that the resulting output probabilities would be close to the asymptotic probabilities (ap) from an hlm with an infinite size. however, we cannot get the ground truth ap to supervise our model directly, so during training on an unsupervised text corpus, we first collect the predicted ap of the top likely next tokens and the empirical probabilities from the lms with different sizes. next, as illustrated in figure 3 , we use an mlp energy network @xcite to output a curve that is close to the observed probabilities and would approach the predicted ap as lm's size goes to infinity. after the training, the apd uses the updated amateur lm (alm') in the same way as the cd to produce the next-token distribution: we provide more details of our procedure of training alm' in the follow subsections. ## experiments in many applications, factuality is arguably the most important aspect @xcite . it is especially difficult to improve factuality in general domains without increasing lm's size, training lms on more high-quality data, or adding more information into the context @xcite . figure 1 shows that cd might assign a lower probability to the most obvious answer. in openended generation tasks, this could cause suboptimal factuality and diversity. furthermore, this could degrade the quality of answers in the questionanswering tasks. thus, we focus on these aspects in our evaluation. in all experiments, apd is trained in a very small training corpus, just 1m lines (1.6%) in wikipedia 2021, to test the generalization capability of our method. pythia, opt, and qwen llm families are the ideal test beds because they provide several smaller lms of different sizes. hence, we select de-duplicated pythia 6.9b and opt-6.7b as the elm (θ s n ) and pythia 70m and opt-125m as the alm (θ s 1 ). the results of qwen-1.5 are presented in the appendix. ## related work besides simulating a huge lm as cd, linear extrapolation could also be used to infer the output of an lm trained on less toxic data @xcite , an lm trained on more context @xcite , an lm trained on less hallucinated data @xcite , an lm trained on more preference data @xcite , and an lm with more hidden layers @xcite . apd shows that an exponential function can improve cd, so similarly, we might be able to use a sophisticated extrapolation function to improve these methods. scaling law shows that the global perplexity of a larger lm could be accurately extrapolated by the perplexities of smaller lms @xcite . furthermore, real sampling is a thresholding method while our method is a distribution modification method, and our experiment shows that they are complementary. ## conclusion can a tiny amateur lm help a large expert lm infer the probabilities of a huge hypothetical lm? yes, we first theoretically show that cd did that and then propose apd to do even better. our theory for cd explains several prior empirical observations and motivates the proposed apd. on the other hand, the apd addresses the limitations of cd and further supports our theoretical explanation. in our experiments, we show that fine-tuning the amateur lm on only 1.6% text of wikipedia is enough to boost the performances of seven qa datasets and the factuality in factualityprompts, judged by both retrieval-based metrics and mturk workers, given the similar generation diversities.