Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion (2025)

Abstract: Speculative decoding has emerged as a widely adopted method to accelerate large language model inference without sacrificing the quality of the model outputs. While this technique has facilitated notable speed improvements by enabling parallel sequence verification, its efficiency remains inherently limited by the reliance on incremental token generation in existing draft models. To overcome this limitation, this paper proposes an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences. This allows parallelization of both the drafting and verification steps, providing significant speedups to the inference process. Our proposed approach, *Speculative Diffusion Decoding (SpecDiff)*, is validated on standard language generation benchmarks and empirically demonstrated to provide up to 7.2x speedups over standard generation processes and up to 1.75x speedups over existing speculative decoding approaches.

## introduction as autoregressive language modeling with transformers @xcite is scaled to larger compute levels, performance improves and new capabilities emerge @xcite . indeed, scaling has been shown to to improve the performance of large language models (llms) for a diverse array of tasks, including code generation, question answering, summarization, and many other use cases @xcite @xcite . for instance, models such as llama 3.2 90b @xcite , @xcite , cohere 52b @xcite , google's gemini-ultra @xcite exemplify the ongoing trend of deploying and releasing increasingly large models, enabling broader access and application across various domains. however, while these desired capability arise, running llms in inference mode for millions of users produces burdensome electricity, time, and monetary demands. many methods exist to mitigate these costs -including sparsity, quantization, and distillation -but they often introduce new tradeoffs, e.g., their application can degrade the performance of the model @xcite . unlike other methods for accelerating llm inference, speculative decoding @xcite can improve llm efficiency by 2-3× with no degradation in the quality of the model outputs. in @xcite , speculative decoding achieves this by sequentially generating multiple tokens with a small, efficient draft model, then running the target, large, llm in parallel on all of the drafted tokens, simultaneously evaluating their consistency with the target llm's output token probabilities. provided that the drafting model's tokens are frequently accepted by the target model and that the drafting model operates substantially faster than the target model, speculative decoding can directly match the sampling output from the target model while significantly reducing runtime @xcite . this functioning is shown in figure 1 @xcite . notably, since both the drafting model's speed and its alignment relative to the target model are critical to the success of speculative decoding, simultaneous improvements in each of these areas are necessary to ensure speculative decoding's relevance to future, more capable target models. for instance, a small gpt-2 @xcite drafting model could produce drafts that are often rejected by gpt-4 @xcite , and simply scaling the drafting model to address its weaker generations risks diminishing the speed advantage necessary to speculative decoding's success. to address this challenge, previous efforts have focused on introducing additional parallelization techniques that incorporate prediction trees and branching to refine the drafting process @xcite @xcite . however, the gain in generation efficiency are at the expense of much increased number of operations and/or memory for each generation. this paper proposes a fundamentally different approach to improve speculative decoding: it proposes to replace the auto-regressive drafter with recently introduced discrete diffusion models @xcite . these models offer several key advantages when used as drafters: firstly, they provide a smooth trade off between the compute cost of generation and the quality of generation (via the number of reverse diffusion steps). second, while they have historically struggled relative to traditional language models, recent diffusion models have been shown to require 32× fewer function evaluations than autoregressive models to produce text with comparable perplexity @xcite , with more recent works reporting even further speedups @xcite . this is a trend which is anticipated to continue. finally, future advances in diffusion model generation quality are highly aligned with their ability to perform strongly as speculative drafters: as drafted tokens are accepted by the target model at a higher rate, a larger number of proposed drafted tokens becomes optimal from an efficiency/speed point of view, and (unlike sequential drafters) diffusion models can easily accommodate generation of many more tokens since they are able to generate entire sequences in a single step. contributions. more specifically, this paper makes the following contributions: 1. it introduces a novel integration of generative diffusion language models with speculative decoding, schematically illustrated in figure 1 (right). ## related work while autoregressive language models provide state-of-the-art performance on language generation tasks, the incremental decoding used by these architectures results in significant overhead at inference time @xcite . this is largely a result of the inability to parallelize the sequential process of generating tokens in the output sequence as each token generation is dependent upon the preceding tokens in the sequence; consequentially, scaling the compute associated with the inference cannot directly reduce this overhead when using standard decoding schemes. in recent literature studying how to accelerate large language model generation, two primary approaches have been explored: (1) advanced decoding implementations that better parallelize token generation and speculative decoding. speculative decoding accelerates autoregressive generation by leveraging a smaller autoregressive models of the same architecture (the drafter model) to predict candidate sequences, which the original model (the target model) then verifies @xcite . notably, the earliest literature on speculative diffusion adapted a non-autoregressive model to act as the drafter model @xcite , using a masked language model with a bidirectional decoder @xcite . however, the integration of non-autoregressive draft models has not received much attention due to the difficulty introduced by the necessary additional training in existing approaches and the modest speedups that were previously reported using these methods (less than 2x speedup over vanilla decoding schemes). thus, recent advancements in speculative decoding have focused on overcoming memory-related constraints, with improvements achieved through various approaches: drafting directly with the target model @xcite , enhancing draft algorithms @xcite , and introducing additional parallelization techniques that incorporate branching to refine the drafting process @xcite @xcite . non-autoregressive language models. models which stray from the autoregressive paradigm have been shown to speedup generation by generating blocks or even entire sequences simultaneously. @xcite propose a method of adapting traditional autoregressive models to sample blocks of tokens, improving inference time over similarly scaled models. in a similar vein, diffusion language models have been recognized for their efficiency in generating extended token sequences concurrently, offering even greater speed enhancements. these models recast language generation as a diffusion process either across the embedding space @xcite or, more recently, through the probability distributions of generated tokens @xcite . recent models report up to a 32x speedup over similarly sized gpt-2 models @xcite , and current state-of-the-art further improves runtime speed @xcite . however, despite they have been shown to dramatically accelerate the inference time for language generation, diffusion models typically perform less effectively than state-of-the-art autoregressive models in terms of standard language metrics, often exhibiting significantly higher perplexity scores @xcite . in the following section, we will demonstrate, for the first time, how the speed of these models can be leveraged without being subject to this critical limitation. ## preliminaries and settings for open-ended language generation, we focus on the task of token generation, where given a sequence of tokens x 1 , x 2 , . . . , x i , denoted here with shorthand notation x 1:i , the goal is to generate the next n tokens x i+1 , . . . , x i+n from the conditional distributions p(x i+1 |x 1:i ), . . . , p(x i+n |x 1:i+n-1 ) or more succinctly p i+1 , . . . , p i+n . speculative decoding. speculative decoding leverages two llms, m p and m q , to parallelize token generation: this process follows a draft-then-verify approach @xcite , where m q efficiently computes a candidate sequence of tokens, which m p then verifies in parallel. during each speculative decoding iteration, m q generates a subset of the total n tokens that are required for the generation task. the size of this subset is denoted as γ. as shown in figure 1 (left), the tokens x i+1:i+γ sampled from m q are then used by m p to generate the corresponding probability distributions p i+1 , . . . , p i+γ . the distributions q i+1 , . . . , q i+γ from m q are stored for evaluating acceptance in subsequent steps. critically, the target model's inference over p i+1 , . . . , p i+γ can now be run in parallel as the model has access to tokens x i+1:i+γ , alleviating the sequential dependency for generation with m p . to ensure high-quality outputs despite potential discrepancies between m p and m q , tokens are subjected to an acceptance criterion. for each token x j with @xmath0, denoted α, and theoretically demonstrate that α = 1 -e(d lk (p, q)) where d lk represents a divergence measure between distributions @xcite . this has led to the prevalent use of drafters taken from the same series as the target models, a paradigm that we challenge in this paper. ## speculative diffusion models speculative decoding has provided state-of-the-art results for improving language generation inference time but requires meticulous tuning of the associated hyperparameters to achieve optimal results. particularly γ, the sequence length generated by the drafter model, needs to be appropriately calibrated not only to maximize potential speedup but to even outperform standard autoregressive decoding. this is an important consideration when using current autoregressive draft models, provided that the inference time to generate m q (x), the draft logits, is directly scaled by the size of γ. increasing this value too high reduces the number of operations that are conducted in parallel, potentially leading to speculative decoding increasing inference time, while reducing this value too low results in speculative decoding "missing out" on token generations that could have been handled by the draft model. @xcite has conducted theoretical analysis on how to best optimize the value of γ, however, it has been contingent upon accurately estimating the percentage of tokens in a the sequence that will be accepted by the target model. by their own acknowledgment, it would be necessary to predict this value for each draft and numerically solve for the optimal value of γ to fully realize the potential speedup of speculative decoding. thus, a significant portion of the residual suboptimality in current implementations can be attributed directly to the sensitivity of this hyperparameter. diffusion language models are juxtaposed to conventional language models in that they do not sample token sequences in a sequential manner, rather generating entire sequences in parallel. this has resulted in significant speedup over similarly sized autoregressive models when generating extended sequences @xcite . this can particularly be observed in longer sequence generations as scaling the draft length γ results in minimal overhead due to the ability to directly parallelize token generation. ## comparative analysis of speculative decoders according to the work-depth parallel computation model @xcite , an algorithm can be represented by a dag, in which each node is an operation and each edge is a dependency. the longest shortest path in this computational dag (i.e., the longest dependency chain) is called the depth. it follows that the average parallelism of a computation is the total number of nodes divided by the depth, and that higher depth means fewer opportunities for parallelism. we represent the operations by the flops of the model. the table reasons about the approaches through three parameters for m p and m q : c (memory consumption), f (arithmetic/floating-point operations, or flops), and d (depth). due to the quadratic memory and computation requirements of transformers, and for simplicity, c {p,q} and f {p,q} are represented as functions of the number of tokens after the initial prompt, i.e., c p (k) is the memory cost for generating p + k tokens, where p is the number of prompt tokens. the depth d {p,q} is generally independent of the number of tokens computed and is thus represented by a scalar. in medusa, the prediction tree is set by a number of fixed hyperparameters @xcite , which we aggregate by using the number of nodes and the depth of the tree t n and t d . medusa-2 exhibits the same parallel properties as medusa-1, but requires a joint fine-tuning of m p along with m q . hydra is an adaptation of medusa, in which the heads are applied in sequence. it thus only affects the depth of the computations. eagle uses a drafter model that contains the target embedding layer, one autoregressive layer, and the target lm head. the fixed tree size is 5 levels deep and contains 25 nodes @xcite . eagle-2 (li et al., 2024a) dispenses with the fixed tree defined in eagle, and the induced dynamic tree used in generation is represented with t . notice that in our method, the depth of the algorithm is dependent on t , the number of diffusion steps, rather than γ. empirically, t is smaller than γ, which enables more parallelism in the computation w.r.t. the number of generated tokens. combined with diffusion models enabling longer prediction horizons, this allows specdiff to produce more speculative predictions faster. practically, we observe that this enables specdiff to produce near state-of-the-art inference acceleration while requiring significantly less computational overhead and reduced memory requirements. ## evaluation 6.1 experimental setup settings. to empirically evaluate the improvements provided by using specdiff, the paper provides an empirical analysis on three standard natural language processing tasks: (1) text summarization using the cnn/dm dataset @xcite , (2) text generation on the openwebtext (owt) dataset @xcite , and (3) text generation using mt bench @xcite . additionally, we assess the performance of our method against the current state-of-the-art using specbench, a unified evaluation platform for speculative decoding techniques @xcite . in each setting, the model is queried for 1024 tokens using a greedy decoding scheme (@xmath1. for the experiments, we evaluate the pretrained target models m p gpt-2 xl (1.5b), gpt-neo (2.7b), and vicuna (33b). we use masked diffusion language model (110m) as our drafter model m q , which is a comparable size to the baseline drafter gpt-2 (86m) employed for our standard speculative decoding baseline @xcite . all evaluation is conducted on two nvidia a100 series gpus (80gb) using cuda 12.2. additionally, flashattention @xcite is used to optimize the performance in all experiments. ## conclusion motivated by the costly inference time of current large language models, this paper has proposed the novel integration of discrete diffusion models with autoregressive language models. the proposed method, speculative diffusion decoding, alters existing speculative decoding schemes to integrate a non-autoregressive diffusion model as the draft model. as shown by the empirical evaluation on standard language generation benchmarks, the proposed method leverages the dramatic runtime advantages of discrete diffusion models while also maintaining the dramatically higher generation quality of autoregressive target models. the reported results demonstrate the utility of this approach in effectively accelerating runtime, outperforming vanilla decoding by over 7x and speculative decoding methods by over 1.75x.