Decoding Speculative Decoding (2025)

Abstract: Speculative Decoding is a widely used technique to speed up inference for Large Language Models (LLMs) without sacrificing quality. When performing inference, speculative decoding uses a smaller draft model to generate speculative tokens and then uses the target LLM to verify those draft tokens. The speedup provided by speculative decoding heavily depends on the choice of the draft model. In this work, we perform a detailed study comprising over 350 experiments with LLaMA-65B and OPT-66B using speculative decoding and delineate the factors that affect the performance gain provided by speculative decoding. Our experiments indicate that the performance of speculative decoding depends heavily on the latency of the draft model, and the draft model’s capability in language modeling does not correlate strongly with its performance in speculative decoding. Based on these insights we explore a new design space for draft models and design hardware-efficient draft models for speculative decoding. Our newly designed draft model can provide 111% higher throughput than existing draft models and our approach generalizes further to all LLaMA models (1/2/3.1) and supervised fine-tuned models.

## introduction in recent years, large language models (llms) have emerged as a cornerstone of modern computational linguistics, offering unprecedented capabilities in generating and interpreting human language. as the demand for faster and more efficient language processing grows, understanding and optimizing the inference throughput of these models becomes increasingly crucial. decoder-only llms @xcite use autoregressive decoding to perform inference. autoregressive decoding is known to be hardware inefficient @xcite , leading * correspondence: minghao @xmath0.wisc.@xmath1. several methods @xcite @xcite @xcite have been studied to optimize the serving of llms. one promising approach to improve the throughput for serving llms without accuracy loss is speculative decoding @xcite @xcite . when using speculative decoding to serve an llm (usually 10s to 100s of billion parameters), a draft model (a significantly smaller llm) is used to generate speculative tokens. the target llm model then verifies the output of the draft model and only outputs tokens that match its output. in the case of speculative decoding, the target llm for inference acts as a verifier for the draft model. by leveraging faster inference of smaller draft models, speculative decoding turns autoregressive decoding on the target llm into a more hardware-friendly batched operation (similar to "prefill"), thereby increasing throughput while preserving accuracy. given the promised benefits of speculative decoding, this paper first focuses on understanding the key factors that dictate the throughput improvements that can be obtained. we perform a comprehensive benchmarking study and profile speculative decoding to characterize bottlenecks. we perform over 350 experiments, using llms like llama-65b, opt-66b, and fine-tuned chat models such as vicuna-33b @xcite as target models and llama and opt families as draft models, ranging from ≈ 5× to 528× fewer parameters than the target models. our findings show that the key bottleneck in speculative decoding is the draft model's latency (time to generate candidate tokens from the draft model), highlighting the need to study the design of draft models for speculative decoding. while studying the design of draft models, we observed two interesting phenomena: first we observe that the draft model latency is bottlenecked by model depth, and higher model depth leads to increased latency (section 3.2) even with the same number of parameters. secondly, we observe that draft model accuracy on language modeling tasks does not correlate strongly with its performance in speculative decoding (section 3.3), i.e., a draft model with higher accuracy on language modeling task can have similar tar to a model with lower accuracy. these two phenomena show that existing draft models used for speculative decoding were primarily designed only to achieve maximum accuracy for a given parameter budget and are sub-optimal for maximizing the throughput with speculative decoding. based on these two insights, we propose designing new draft models that trade increased depth for width (thus retaining the same parameter count) and show that our new draft models can boost inference throughput using speculative decoding by over 60%. finally, we show how pruning methods like sheared-llama @xcite can be used to generate smaller draft models with favorable configurations on three different families of models opt, llama, and llama-3.1. ## background and related work first, we provide a high-level overview of llm inference and the use of speculative decoding. ## understanding speculative decoding to study the effects of the choice of the draft model, we first perform a detailed study on serving opt-65b and llama-65b using speculative decoding. metrics: to quantify the performance of different draft models when performing inference on a target model, we measure throughput (tokens generated per second) and tar (figure 2 ). we note that the primary goal of speculative decoding is to improve throughput. figure 4 : this figure shows the performance breakdown of speculative decoding on opt models, look ahead length is set to be optimal for each draft model found empirically. ## draft model design for speculative decoding the above results indicate that to improve the throughput of speculative decoding, it is necessary to improve the latency of draft models, i.e., can we design a model that provides a similar tar at a lower inference cost? in the next section, we study the possibility of such a design. ## conclusion in this work, we conduct a large-scale experimental study to understand how we can optimize the throughput of speculative decoding. using our experiments, we outline the various factors that affect speculative decoding throughput. we observe that draft model accuracy on language modeling does not correlate strongly with its performance in speculative decoding. further, we find that draft model latency is bottlenecked by model depth, and higher model depth increases latency. based on these two insights, we propose new draft models pruned to align with the target model while trading model depth for width. our proposed draft model can increase throughput by up to 111% over existing models. we find that the pruned models can be used for supervised fine-tuned target models without modification and our approach generalizes to state-of-the-art models.