Speculative Contrastive Decoding (2024)

Abstract: Large language models (LLMs) exhibit exceptional performance in language tasks, yet their auto-regressive inference is limited due to high computational requirements and is sub-optimal due to the exposure bias. Inspired by speculative decoding and contrastive decoding, we introduce Speculative Contrastive Decoding (SCD), a straightforward yet powerful decoding approach that leverages predictions from smaller language models (LMs) to achieve both decoding acceleration and quality improvement. Extensive evaluations and analyses on four diverse language tasks demonstrate the effectiveness of SCD, showing that decoding efficiency and quality can compatibly benefit from one smaller LM.

## introduction large language models (llms) have advanced the versatility and proficiency in approaching realworld natural language tasks such as general instruction following @xcite @xcite and reasoning @xcite @xcite . most existing llms @xcite ; @xcite ; @xcite ,inter alia) are built on decoder-only transformers. due to the auto-regressive nature during inference, the runtime of decoding inference can be excessive on general computation infrastructure, and the generation quality can be sub-optimal due to the exposure bias @xcite . improving decoding inference has been the spotlight of the research community in language generation @xcite @xcite . as for decoding acceleration, one prominent method named speculative decoding @xcite has been proposed and leverages relatively smaller language models (lms) to predict several successive token generations of target llms. the llms only require one-time forward computation for checking the validity of predictions from the smaller lms. the decoding method maintains the target llms' token distributions and accelerates more when smaller lms can accurately predict the potential target llms' generations. as for the generation quality, contrastive decoding has been recently proposed @xcite . contrastive decoding assumes that conjugated smaller lms may present higher systematic tendencies to generate erroneous tokens than the larger ones, and the method seeks to eliminate such systematic error by contrasting the token distribution between smaller lms and larger lms. from either inference acceleration or quality improvement, these works have demonstrated a promising direction by integrating smaller lms during auto-regressive generation. inspired by both speculative and contrastive decoding, we propose speculative contrastive decoding (scd), which exploits a single smaller lm for decoding improvement in speed and quality en bloc. comprehensive evaluations of four diverse tasks show that scd can achieve similar acceleration factors of speculative decoding while maintaining the quality improvement from contrastive decoding. by further analyzing the token distributions of the smaller and larger lms in scd, we show the inherent compatibility of decoding acceleration and quality improvement. the contributions of this paper can be summarized as follows: ## related works in terms of inference acceleration, recent research has been devoted to developing various efficient decoding methods @xcite @xcite . speculative decoding @xcite ; @xcite ; @xcite is one of these recent works and utilizes smaller models for acceleration. @xcite ; @xcite propose to organize predictions from small lms into tree structures to accelerate speculative decoding further. in terms of inference quality, rich research has been suggested @xcite @xcite @xcite and contrastive decoding achieves better decoding qualities by similarly integrating smaller lms and devise contrastive token distributions @xcite . it can further be adjusted to other variants such as the token distribution contrasting between model layers @xcite or different inputs @xcite . scd draws inspiration from these works and benefits both decoding speed and quality by incorporating smaller lms into generation. ## preliminaries we follow the terminology in @xcite , and term the target larger lms as the expert lms while the smaller lms as the amateur lms denoted as m e and m a respectively. ## analysis compatibility. results presented in §5 show scd can combine the benefits of cd and sd. we delve deep into the reasons for such compatibility. we calculate the average entropy of token probabilities from m a and m e regarding the accepted and rejected tokens in scd. as shown in fig. 2 , token distribution entropy from both m a and m e of accepted tokens is significantly higher than that of rejected tokens. the phenomenon suggests scd enjoys acceleration from accepting easy tokens of lower entropy while benefiting from contrastive token distribution by rejecting hard tokens of higher entropy. we also present a case study from gsm8k in appx. §c to demonstrate such compatibility. sensitivity. through fig. 3 , we show how performances fluctuate with respect to the hyperparameter α and β. we can see that improved scd is less sensitive to both α and β on gsm8k compared to the original scd. this is possibly due to the better flexibility of manipulating logits than probabilities. results on humaneval are listed in appx. §d presenting similar phenomenons. ## conclusion in this paper, we propose speculative contrastive decoding, a decoding strategy that naturally integrates small amateur lms for inference acceleration and quality improvement of llms. extensive experiments show the effectiveness of scd and our delve-deep analysis also explains the compatibility through the scope of token distribution entropy. our method can be easily deployed to improve the real-world serving of llms.