Paper: Contrastive Decoding Improves Reasoning in Large Language Models (ICLR-2024)

Reviewer comment (Evaluation):
Contrastive Decoding (CD) yields highly inconsistent results. For example, it improves on arithmetic reasoning tasks such as GSM8K but not on MATH. It yields small gains on Commonsense Reasoning tasks, but only with large models and on some Contrastive Ranking tasks such as HSwag. Given these mixed results, it is unclear when to use CD effectively.

Author response:
**Consistency of results:** We have highlighted that contrastive decoding yields consistent improvements when coupled with self-consistency on all measured tasks, providing at least one generation paradigm in which it is always shown to help. Further, we include analysis for the generation tasks on which contrastive decoding fails to improve performance -- for instance, the smaller expert models perform worse than chance on AQuA and therefore do not have a clear advantage over the amateur model to contrastively exploit, and the expert’s overall performance on the MATH benchmark is quite low. We also now include more analysis in Section 3.5 about the challenges of CD on ARC-Easy (inconsistent scaling behavior) and MMLU (factual-heavy).

Reviewer comment (Evaluation):
The paper does not provide a detailed error analysis by presenting wins/losses on tasks or offer other insights into why CD performs better/worse on specific tasks.

Author response:
**Error analysis:** We now include example generations with and without CD in the appendix of the paper. Most differences are difficult to attribute to CD for reasonable values of beta, and so we rely on the differences in automatic metrics (e.g. n-gram copying) as well as the manual error analysis in Table 5 to give insight into contrastive decoding’s strengths and weaknesses.

Reviewer comment (Experiments):
Contrastive Decoding (CD) only works with chain-of-thought reasoning. This seems like a significant limitation, and it would be valuable to conduct more ablation studies to understand it better.

Author response:
**Chain-of-Thought Limitation:** We now clarify in both Section 3.3 and 4.2 that contrastive decoding only works when the expert is significantly better than the amateur at a given task. Previous work from Li et al find benefits to open-ended text generation, and our experiments show that non-chain-of-thought performance does not improve as much as chain-of-thought performance with scale, suggesting a lack of a sufficient contrastive gap to exploit. In other words, solving these tasks effectively with an untuned model requires chain-of-thought generations both with and without contrastive decoding.

Reviewer comment (Experiments):
Most of the experiments in the paper (except for the single T5-FLan result) are obtained with LLAMA models. This makes it unclear whether CD would show the same behavior with other pre-trained models.

Author response:
**LLaMA Models:** Contrastive decoding, and contrastive-decoding-like strategies like coherence boosting (Malkin et al 2022) and DoLa (Chuang et al 2023) have been shown to improve performance on open-ended task generation. Most of these works use one to two families of models; we include the FLAN-T5 study to show that our method works out-of-the-box on a family of models without tuning any additional hyperparameters. Further, the original Contrastive Decoding work found improved open-ended generation on OPT and GPT-2, suggesting that the efficacy of scale-based Contrastive Decoding is not unique to the LLaMA family of models.

Reviewer comment (Evaluation):
It would be good to provide error analysis (samples of wins/losses) especially when CD does not work e.g. on MATH.

Author response:
We have now included example generations in the appendix for qualitative analysis. It is important to note that CD does not significantly harm performance on the MATH benchmark (a maximum of -0.4% absolute performance), even though it does not bring the same benefit it does to other arithmetic reasoning tasks.

Reviewer comment (Writing):
In 2.1, it would be good to highlight that \alpha in [0,1] i.e. log \alpha <0.

Author response:
We have refactored 2.1 to give more background on the original method, and in doing so emphasize the range of possible values for alpha.

Reviewer comment (Presentation):
Figure 4: What does 'GSM 8K EM' mean?

Author response:
In this context, GSM8K “EM” is the exact match accuracy on GSM8K results. For clarity, we have changed the label to “Accuracy (GSM8K)”.

Reviewer comment (Evaluation):
P7: In the section titled 'CD reduces copying from the prompt', can you provide statistics from the experiment for the baseline/CD models?

Author response:
We include the n-gram F1 scores for both greedy decoding (blue bars) and contrastive decoding (orange bars), with greedy decoding as the baseline in Figure 6.

Reviewer comment (Theory):
Sec 2.2: It is not immediately obvious why the distribution in Equation 1 collapses to the argmax of pe/pa as beta->\infinity. It would be useful to show the steps in the appendix.

Author response:
As $\beta$ goes to $\infty$, the ratio term dominates the the prior term, assuming there is at least one item in the feasible set which the expert assigns higher probability than the amateur (creating a ratio greater than one which the beta exponent would amplify infinitely above the rest).

Reviewer comment (Novelty):
The experimental results and conclusions of the paper are certainly interesting, but those are mainly based on an idea and concept borrowed from an existing paper (Li and al, 2022). The changes with respect to this previous work are minor in my opinion. The strength of the current submission is the experimental nature of the work, but the approach itself lacks novelty.

Author response:
While the central method of our paper draws on Li et al 2022, we believe that our reformulation of the contrastive objective and application to reasoning tasks constitutes a novel contribution that fills a clear research gap.

Contrastive decoding was originally proposed as an alternative to sampling methods in order to improve open-ended text generation in language models. Given a single generation, most decoding methods that produce more diverse text (e.g. temperature sampling, nucleus sampling) harm performance on reasoning tasks. In showing that contrastive decoding improves reasoning on performance tasks, our findings shift a common understanding of decoding methods: that there is a hard tradeoff between diversity and reasoning ability. 

Beyond our findings, our reformulation constitutes a novel benefit to research on contrastive decoding in the future. The application of too-large amateurs under the original, non-$\beta$ formulation of CD has led other papers to wrongly conclude that contrastive decoding harms reasoning. (DoLa; Chuang et al 2023) The $\beta$-parameter formulation allows better fine-grained control over CD, allowing a continuous reduction to standard greedy decoding in a way that the original formulation does not and making it more straightforward to apply to most problems (with a reasonable search space of [0, 1] rather than [1, $\infty$]).

Besides this reformulation, we combine self-consistency with contrastive decoding for the first time, finding greater benefits than either alone. Negative prompting and contrast against a partially trained amateur are both novel methods that improve results as well.

Thus, we believe that our paper presents several novel methods to study, in addition to a key reformulation of a previous work as well as counterintuitive findings that fill a clear research gap.

Reviewer comment (Writing):
Since the current approach is based on previous work (Contrastive decoding, Li and al, 2022), it would be preferable to first give a recall of this early work, before introducing the revised formulation proposed by the authors. I had to read the original work in order to understand section 2.1, the meaning of the alpha-mask, and the logic/motivation behind the logits-subtraction.

Author response:
Presentation: Thank you for this feedback -- for clarity, we have expanded section 2.1 with more background on the original method prior to our reformulation.

Reviewer comment (Experiments):
Could we use the idea of Contrastive decoding during the training stage itself?

Author response:
Yes; distilling the contrastive objective into a single language model is an ongoing work. It remains out of scope for the purposes of this paper, which focuses on merits of the training-free, inference-time intervention form of contrastive decoding as it pertains to reasoning tasks.