Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoff (2025)

Abstract: Speculative Decoding (SD) enforces strict distributional equivalence to the target model when accepting candidate tokens. While it maintains the target model’s generation quality, this strict equivalence limits the speedup achievable by SD and prevents users from trading deviations from the target distribution in exchange for further inference speed gains. To address these limitations, we introduce Fuzzy Speculative Decoding (FSD) - a decoding algorithm that generalizes SD by accepting candidate tokens based on the divergences between the target and draft model distributions. By allowing for controlled divergence from the target model, FSD enables users to flexibly trade generation quality for inference speed. Across several benchmarks, our method is able to achieve significant runtime improvements of over 5 tokens per second faster than SD at only an approximate 2% absolute reduction in benchmark accuracy. In many cases, FSD is even able to match SD benchmark accuracy at over 2 tokens per second faster, demonstrating that distributional equivalence is not necessary to maintain target model performance. Furthermore, FSD can be seamlessly integrated into existing SD extensions; we demonstrate this by applying FSD to EAGLE-2, greatly enhancing this existing extension’s efficiency while allowing it to leverage FSD’s tunable quality-speed trade-off.

## introduction speculative decoding (sd), introduced by @xcite and @xcite , is a large language model (llm) inference acceleration algorithm that leverages a smaller, faster draft model to generate sequences of candidate tokens which are then verified and accepted in parallel by a larger target model. the speculative sampling rule that sd employs to determine which candidates to accept enforces a strict equivalence of the final sampling distribution and the original target model distribution. thus, by cutting out the expensive sequential generation from the large target model, sd can lead to inference time reductions of around 2-3x while maintaining the same generation quality as the target model. despite this impressive speedup, sd suffers from two major flaws. firstly, in order to maintain strict distributional equivalence to the target model, the sd candidate acceptance rule is overly strict, and in many cases may reject tokens that if accepted would have no impact on final generation quality @xcite , thereby unnecessarily limiting potential speed-ups. secondly, the enforced distributional equivalence means that users cannot tune the sd acceptance rule to be more or less lenient in its candidate acceptance, preventing users from trading deviations from the target model distribution in exchange for further inference speed gains. however, the flexibility for users to tune their llm generation along an inference speedgeneration quality tradeoff would be highly beneficial in real-world applications, as the relative importance of inference speed compared to generation quality may vary across different contexts within an application. to address these limitations of sd, we introduce fuzzy speculative decoding -a generalized sd algorithm that determines token acceptance based on the divergence between the target and draft model distributions, allowing users to tune the generation quality -inference time tradeoff of their model. with fsd, users have the flexibility to tune a threshold parameter t that determines how lenient candidate acceptance should be, and thus can control how much they are willing to deviate from the target model's distribution in exchange for further runtime reductions. as it doesn't enforce strict distributional equivalence, fsd can achieve significant runtime improvements over sd by accepting a higher percentage of candidate tokens. sd accepts candidate token with a probability dependent on the candidate's relative likelihood under m d and m t . fsd determines candidate acceptance deterministically based on whether the divergence between the m d and m t distributions at the candidate's position exceeds a given threshold t , allowing users to determine how many candidate tokens to accept by setting the threshold t accordingly. we conduct extensive experiments across four diverse benchmarks-spanning factoid qa, math, and coding-using three different model pairs. our key findings are: we also perform a broad range of ablation studies, demonstrating that fsd's performance shares many similarities with sd, including dependence on draft and target model alignment for a given text and the ability to use both sample-based and greedy decoding strategies. we also show that fsd can be applied on top of existing sd extensions like eagle-2, bringing the aforementioned tunability to these otherwise inflexible methods. ## previous works several works have sought to improve speculative decoding, primarily by increasing the acceptance rate of draft-generated tokens, including but not limited to (1) verifying more tokens with treestructured proposals: some methods improve efficiency by allowing the draft model to propose tokens in a tree structure, enabling the target model to verify multiple candidates in parallel using tree attention mechanisms. this expands the search space and increases the likelihood of accepting a valid token @xcite @xcite @xcite . ( 2 ) aligning the draft model with the target model: methods include fine-tuning the draft model to mimic the target model's outputs @xcite , granting the draft model access to additional representation information from the target model @xcite @xcite , or even using a partial version of the target model as the draft model itself-such as using partial layers @xcite @xcite or augmenting the target model with lightweight extensions to improve alignment @xcite @xcite . the most similar method to ours is concurrent work judge decoding (jd) @xcite , an sd variant where a compact module is trained on token embeddings to 'judge' and accept candidate tokens based on correctness rather than strict alignment with the target model. this allows jd to accept more tokens than sd with minimal performance loss. however, jd has two major limitations. first, it generalizes poorly to unseen data, as token acceptance relies on a trained judgment module. its performance drops significantly on out-of-distribution text @xcite . 1 second, jd requires per-model training, preventing out-of-the-box use for new model pairs. in contrast, fsd is training-free, generalizes across datasets, and can be applied to any model pair out-of-thebox, effectively addressing jd's weaknesses. ## speculative decoding we start by reviewing how sd works in order to properly introduce fsd as an extension of this method. consider a larger target model m t and a smaller draft model m d . the biggest bottleneck when generating from m t individually is that tokens are sequentially dependent, and therefore each token will require a full m t forward pass to be generate conditional on the previously generated tokens. sd mitigates this bottleneck by first generating a sequence of candidate tokens sequentially from the faster m d , and then uses a single m t forward pass only to verify which of these tokens to accept. provided that m d is a good enough approximation of m t such that a significant portion of these candidates are accepted, the runtime saved by avoiding sequential generation from m t outweighs the additional runtime of running m d , resulting in an overall speedup. in order to maintain m t 's full generation quality, sd accepts candidate tokens based on an acceptance rule that guarantees the final sequence of sampled tokens will still be distributed the same as they would under m t . specifically, at each sd step m d first generates a sequence of l candidate tokens, @xmath0, x 1 ...x l ], which are then passed through m t to calculate the likelihood of each candidate token x i under m t . using this likelihood, each candidate x i is accepted with the probability: once sd reaches the first rejection of the candidate sequence, it resamples a token at the rejected candidate position from the adjusted distribution: by accepting tokens that are more likely under m d than under m t with a probability of and resampling rejected tokens from an adjusted distribution, sd corrects for the bias introduced by m d , ensuring that the final distribution remains the same as that of m t . ## fuzzy speculative decoding the defining difference of fsd is that it employs a different token acceptance rule that can be tuned to be more or less lenient in its acceptance decisions based on a threshold parameter t , which can be arbitrarily set by the user. this effectively allows users to determine how much they are willing to diverge from the target distribution m t in exchange for a higher percentage of candidates accepted, resulting in speed-ups beyond sd. while sd determines acceptance based on the likelihood of candidate x i under p m t and p m d , fsd calculates the distribution-level divergence between these two distributions at each candidate position. then, based on the tunable divergence threshold t , fsd will accept a candidate token if the models' divergence at the corresponding position is less than t . this makes the fsd acceptance rule: in the case of candidate token rejection, fsd will sample from p m t [i], that is the original target model distribution at the rejected position, with whatever sampling method the user sets for the generation. the full fsd algorithm is depicted if figure 1 as a side-by-side comparison with sd. ## conclusion we have introduced fsd -a modified sd algorithm that can accept divergence allows users to tune how much divergence from m t they are willing to accept in exchange for runtime improvement beyond sd. this flexibility to achieve significantly higher runtimes, in addition an ability to match sd generation quality at faster inference in certain scenarios, makes fsd novel alternative to sd that we expect can be valuable in many llm applications. we have shown that fsd is able to achieve very strong empirical results on-par with sd, and is able to achieve considerably higher generation speeds the cost of only minor deteriorations in generation quality.