The Benefits of Bad Advice: Autocontrastive Decoding across Model Layers (2023)

Abstract: Applying language models to natural language processing tasks typically relies on the representations in the final model layer, as intermediate hidden layer representations are presumed to be less informative. In this work, we argue that due to the gradual improvement across model layers, additional information can be gleaned from the contrast between higher and lower layers during inference. Specifically, in choosing between the probable next token predictions of a generative model, the predictions of lower layers can be used to highlight which candidates are best avoided. We propose a novel approach that utilizes the contrast between layers to improve text generation outputs, and show that it mitigates degenerative behaviors of the model in open-ended generation, significantly improving the quality of generated texts. Furthermore, our results indicate that contrasting between model layers at inference time can yield substantial benefits to certain aspects of general language model capabilities, more effectively extracting knowledge during inference from a given set of model parameters.

## introduction for a wide range of natural language processing tasks, the standard practice is to rely on deep neural networks with a transformer-based architecture @xcite . such models are composed of multiple transformer layers, where typically the representations of the final layer are used for the downstream task. as shown in prior works, some of the representational knowledge required for performing downstream tasks can already be found within intermediate layers of the model @xcite ; at the same time, relying on the representations of lower model layers does result in decreased performance, specifically for inputs that are more challenging @xcite @xcite @xcite . 24 ) is taken as the expert and contrasted with layer 12, the amateur. as decoding is done token by token, we can only see the direct effect on the first token, where acd leads to selecting an alternative high probability token -"in". recently, @xcite considered a scenario involving two language models; one is a very large pre-trained model, termed the expert, and the other is a much smaller version of the same architecture, termed the amateur. importantly, whereas these models share some failure modes and undesirable behaviors, the expert model clearly outperforms the amateur model in language model tasks. focusing on an open-ended auto-regressive text generation task, they show that it is possible to exploit the contrast between the predictions of the expert and amateur to obtain an improved generated output. they term this method contrastive decoding. specifically, they demonstrate that it is sometimes beneficial to prefer predictions to which only the expert model assigns a high probability, versus predictions to which both the expert and the amateur assign high probabilities. intuitively, since the amateur model has a stronger propensity than the expert for problematic behaviors (e.g., repetitiveness in the case of text generation), we may be able to diminish such behaviors by demoting predictions that are strongly supported by the amateur model. this scenario relies on a delicate balance: on the one hand, when making a prediction in a relatively simpler context, one would expect both the expert and amateur models to be highly confident about the prediction, and justifiably so; in contrast, where both of them assign very low likelihoods to a certain prediction, these prediction probabilities may be uninformative. thus, the aim of considering the amateur's predictions during generation is to better inform a choice between a set of relatively plausible predictions given an input; in other words, the predictions of the amateur can serve as a tiebreaker of sorts, helping to highlight which out of a set of plausible alternative predictions is more "expert-like" and less "amateur-like". inspired by @xcite , in this work we ask whether within a single language model, intermediate hidden layers can similarly be viewed as "amateur" versions of the final "expert" output layer. given indications that model representations gradually improve as an input progresses through its layers @xcite , we aim to examine whether the contrast or gap between the outputs at different model layers can be harnessed to obtain better generation predictions. in other words, we posit that the sub-optimal predictions of intermediate hidden layers carry additional information, which can be utilized during inference to obtain more desirable next-token predictions. our approach, which we term auto-contrastive decoding (acd), redistributes a given model's probability distribution for the next token, by maximizing the difference between the log-probabilities of the final layer and those of an intermediate hidden layer. this setting, where the expert and amateur are situated within the same language model, and their predictions can be carefully contrasted at inference, is a highly practical one and can be easily applied to language models of different sizes. our results show that acd enables getting significantly better predictions out of a given language model, without changing its pre-trained weights. figure 1 illustrates an example of acd applied to gpt2, considering layer 12 as the amateur and layer 24 as the expert. both layers exhibit repetitiveness, but applying acd generates a much improved output altogether. the main contributions of this work are as follows: ## related work there have been a number of studies on analyzing the characteristics of different layers of transformer models. rogers et al. (2020) 2019 also show that similar behavior is observed in other transformer models such as gpt2. @xcite studied the role of feed-forward layers in transformer models. they demonstrate that representations across different layers capture meaningful semantic and syntactic patterns, and describe how model predictions are gradually refined as they progress across the different layers. aiming to reduce the computational load of transformers, multiple works have explored earlyexiting, i.e., performing some calculations without passing through all of the model layers. such works allow for an early (fast) 'exit' from neural network calculations -for simple instances that can be solved with high accuracy by lower layerswhile using a late (slow) 'exit' for more challenging instances @xcite @xcite @xcite . decoding algorithms are commonly classified as search-based and sampling-based. search-based methods @xcite optimize for the language model log-probabilities, while sampling methods @xcite draw the next token from a truncated distribution. the idea of using contrast during decoding has been explored in several studies. @xcite combine a pretrained lm with 'expert' lms and 'anti-expert' lms, where tokens only get high probability if they are considered likely by the experts and unlikely by the anti-experts. @xcite propose constrastive search for decoding, where the generated output is selected from the set of most probable candidates predicted by the model while being discriminative with respect to the context. more recently, @xcite suggested to contrast between the likelihood under a large lm (expert) and a small lm (amateur) during decoding. the present work differs significantly from the aforementioned contrastive approaches, in that we contrast the next-token distributions within a single lm, across expert and amateur layers. ## auto-contrastive decoding we set the goal of applying the contrastive decoding (cd) method, from @xcite , using a single model rather than two different models (in their setting, a large and a small model of the same model architecture). thus, we generally follow the cd approach to calculate the next-token predictions, by contrasting the predictions of the expert with those of the amateur. however, in our setting, both the expert and the amateur are situated in the same model, and are defined by two different layers of that model. we term this new method auto-contrastive decoding (acd). note that this setting is more practical and less computationally demanding, as it does not require passing every input through two different models in parallel. next, we describe how we obtain the expert and the amateur from a single model; and in §3.2, we define the auto-contrastive next-token distribution, given the probability distributions of the expert and the amateur. ## experimental setup to test our approach, we conduct experiments on open-ended text generation, as well as on general language modeling benchmarks, comparing various performance metrics with and without applying auto-contrastive decoding. in order to analyze changes in performance across model layers, we add multiple new linear exit heads; thus, we also report and compare the baseline model behavior at different exit layers. ## discussion in this work we develop an approach that contrasts different model layers, improving the output probabilities of a generative model. applying it to existing pre-trained language models, we demonstrate that intermediate low-performing model layers can in some cases inform the predictions of the highperformance final layer. this setting is of particular interest due to its practicality and flexibility, as it can be applicable to models of different sizes and is utilized during inference via a single forward pass. but more broadly, our findings bring forth an enticing notion, that one would be able to make more out of an existing model simply by considering the predictions of intermediate layers (which are typically ignored). this idea is somewhat counterintuitive, as language models are in a sense optimized -and often in a long pretraining process over massive corpora -for the quality of their final layer representations. at the same time, thematically this notion is in line with works that describe the computations in transformer models as a linear-like progression, where each layer refines the representations of the previous ones, and where even the representations of specific tokens can shift in a consistent direction along with the progression across layers @xcite . loosely speaking, if the changes from one layer to the next can sometimes track a vector of improvement with a discernible direction, then in theory one could try and "extend" this vector; and doing so may help estimate what a larger model, one with additional layers, would have said about a particular instance. we see these as interesting avenues both for theoretical study, and for empirical explorations as to whether surprising findings such as those presented here can be applied to real-world use-cases. here we present an initial, and relatively simple, algorithm for performing the acd contrast between layers. as in @xcite , our formulation still relies on a somewhat arbitrary hyperparameter α; also, contrast is always done with respect to a single particular exit layer, and choosing the most appropriate layer for contrast may not be trivial. here, for simplicity and robustness, we did not attempt to optimize these two important hyperparameters, and used a single configuration throughout our experiments. however, we see much room for future work on improving these details, and finding ways to intelligently choose which layers to contrast and how to combine between them. an interesting avenue for future work concerns the effect of acd when applied not just to a pretrained model, but to one fine-tuned for a particular downstream task. specifically, it may be that specific types of generation tasks may derive more benefit from acd, depending on their reliance on more "high-level" model capabilities, and also on the importance of diversity in generated outputs. the present work focuses specifically on generative models, and on improving the quality of text generation outputs and next-token predictions. however, the basic approach of looking at the outputs of intermediate layers and using them to inform model predictions is a general one, and is thus also worth exploring in other contexts, such as classification tasks. to sum, our findings indicate that our proposed approach, acd, can be of great practical value, in that it significantly boosts the performance of a generative language model with a minimal computational cost. this approach suggests new avenues on how to best extract knowledge from a language model and more efficiently utilize its parameters.