Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillation (2024)

Abstract: Speculative decoding stands as a pivotal technique to expedite inference in autoregressive (large) language models. This method employs a smaller *draft* model to speculate a block of tokens, which the *target* model then evaluates for acceptance. Despite a wealth of studies aimed at increasing the efficiency of speculative decoding, the influence of generation configurations on the decoding process remains poorly understood, especially concerning decoding temperatures. This paper delves into the effects of decoding temperatures on speculative decoding’s efficacy. Beginning with knowledge distillation (KD), we first highlight the challenge of decoding at higher temperatures, and demonstrate KD in a consistent temperature setting could be a remedy. We also investigate the effects of out-of-domain testing sets with out-of-range temperatures. Building upon these findings, we take an initial step to further the speedup for speculative decoding, particularly in a high-temperature generation setting. Our work offers new insights into how generation configurations drastically affect the performance of speculative decoding, and underscores the need for developing methods that focus on diverse decoding configurations.

## introduction large language models (llms) such as @xcite , claude @xcite , and llama @xcite are revolutionizing the field of natural language processing (nlp) and machine learning (ml). while being powerful tools for various downstream tasks, llms' realtime deployment is still challenging due to the size and the inference cost @xcite . conversely, smaller models have less latency but lower * work partially done during internship at microsoft. generative quality. in a word, efficiency and accuracy form a trade-off. inspired by this, speculative decoding @xcite emerges as a promising token-level solution to reduce the latency of generation for llms. specifically, speculative decoding leverages smaller models as draft models to speculate successive candidate tokens for multiple inference steps with autoregressive generation, which are then verified with the target llm in parallel through a single forward pass. if a token fails to be accepted by the target llm, all the consecutive tokens will be discarded, and the target llm needs resampling for that rejected token. previous studies @xcite generally test speculative decoding in fixed generation configurations, with temperature sampling @xcite being the default setting. compared with other hyperparameters such as top-k @xcite in text generation, temperature has a dominating effect in re-estimating the distribution before top-k sampling @xcite , balancing generation quality and diversity @xcite . however, previous works only test at a coarse-grained level, setting the tempera-ture to binary extremes of either 0 (greedy decoding) or 1.0. on the other hand, accelerating speculative decoding in various generation scenarios is important to better suit user needs in downstream tasks. to this end, this paper investigates from a temperature-centric perspective of speculative decoding for llms. we focus on knowledge distillation (kd) @xcite as the general investigation setting, which has been introduced as an intuitive and general solution to speculative decoding @xcite . particularly, kd aims to align the distributions of draft models better to that of target models. in this way, the acceptance rate of candidate tokens generated by the draft model to the target model could be boosted. our preliminary experiments in figure 1 validate our motivation, highlighting the impacts brought by different temperatures for both decoding and kd stages. notably, the speedup of the decoding processes increases and peaks at a decoding temperature of 0.2 before declining as the temperature approaches 1. the impact of temperature on speedup can reach a relative difference of around 30% ( 2. 23-1.72 1.72 = 29.7%), highlighting its importance. we also notice that kd relieves the degradation of speedup when temperature increases. overall, we explore the impact of temperature on speculative decoding with kd. specifically, we address three pivotal research questions: the experiments are conducted on several commonly used public datasets. our analysis offers a new perspective on understanding speculative decoding by applying fine-grained temperature controls, especially with kd. the key contributions and takeaways can be summarized as follows: ## experimental setup this section outlines the detailed experimental setup, including model architecture, dataset selection, and evaluation metric employed for knowledge distillation (kd) and decoding phases. further details on implementation, including hyperparameter configurations and computation timeframes, are provided in appendix b. ## experiments and analyses our experiments and analyses are organized in the following workflow. we start with an overall investigation of temperature configurations for two kd settings for in-domain testing. leveraging these observations, we further test these insights on outof-domain datasets with out-of-range temperatures. finally, we brought out a simple yet effective solution to further improve the performance of speculative decoding with higher decoding temperatures. ## conclusion in this paper, we have presented a comprehensive investigation into the impact of temperature on speculative decoding, particularly within the context of knowledge distillation (kd), for large language models (llms). through a series of meticulous experiments utilizing the llama series as both target and draft models, we have explored the nuanced interplay between temperature settings during kd and their consequent effect on speculative decoding's efficiency and efficacy. apart from offering empirical findings, we also propose a practical strategy to enhance speculative decoding's performance by leveraging temperature-centric training data assembly. by presenting this work, we aspire to facilitate future works on diverse generation configurations for speculative decoding, and exploring theoretical understanding of the multifaceted relations in between.