Consultant Decoding: Yet Another Synergistic Mechanism (2025)

Abstract: The synergistic mechanism based on Speculative Decoding (SD) has garnered considerable attention as a simple yet effective approach for accelerating the inference of large language models (LLMs). Nonetheless, the high rejection rates require repeated LLMs calls to validate draft tokens, undermining the overall efficiency gain of SD.In this work, we revisit existing verification mechanisms and propose a novel synergetic mechanism Consultant Decoding (CD). CD achieves up to a 2.5-fold increase in inference speed compared to the target model, while maintaining comparable generation quality (~100% of the target model’s performance). Interestingly, this is achieved by combining models whose parameter sizes differ by two orders of magnitude.In addition, CD reduces the call frequency of the large target model to below 10%, particularly in more demanding tasks.CD’s performance was even found to surpass that of the large target model, which theoretically represents the upper bound for speculative decoding.

## introduction large language models (llms) have attracted widespread attention and adoption across diverse domains, owing to their exceptional performance and robust generalization abilities @xcite @xcite @xcite @xcite . as their capabilities continue to evolve, so does their scale, with some reaching unprecedented sizes. notable examples include llama3.1-405b @xcite and deepseek-v3/r1 models @xcite growing size of large models is making inference efficiency an increasingly pressing concern. recently, speculative decoding (sd) @xcite , based on the draftand-verify paradigm, has been introduced to accelerate the decoding process of large language models (llms). it exploits the key property that parallel processing can improve gpu utilization and reduce the average latency during autoregressive decoding. sd maintains distribution consistency between the sampling results and target models. this advantage has spurred the development of numerous improved methods @xcite @xcite @xcite @xcite . however, these methods rely on a rigid verification process akin to speculative sampling, leading to an excessive rejection of draft tokens and fre-quent target model calls. these challenges raise a crucial question: can we devise a better verification approach that boosts the acceptance rate and lessens reliance on target models, while maintaining a high level of performance? in this paper, we examine existing speculative decoding verification approaches @xcite and observe that they predominantly evaluate the draft tokens by likelihood ratio or its variants between the target and draft models, as illustrated in figure 1 @xcite . this metric is derived from sd's importance sampling to ensure that the final sampling distribution is well aligned with that of the llm. however, the likelihood ratio does not reflect well the quality of the generated token. for example, as shown in figure 1(a) , when the target model's prediction probability marginally exceeds that of the draft model -even if both probabilities are substantially low -the candidate tokens will still be accepted, potentially compromising output quality. in addition, the target model's best token may still be rejected, particularly when the target language model has a relatively flat probability distribution. the enhanced sd approaches, such as mentored decoding (md) @xcite , which continue to exhibit suboptimal problems, as it relies on likelihood ratio. furthermore, it is unclear on how to choose the parameters, such as threshold and tolerance. to address these problems, we introduce a novel approach that directly evaluates the correctness of candidate tokens using likelihood of the target model. essentially, we leverage the negative log-likelihood (nll) of candidate tokens predicted by target models to directly evaluate their correctness, a method we refer to as consultant decoding (cd). as illustrated in figure 1 (b), a draft token is deemed correct if its nll approaches or falls below the convergence loss observed during the training of target models, signaling that the distribution of draft tokens closely mirrors the training data distribution of targe models. consultant decoding is named for its approach of consulting the target model's judgment to validate draft tokens, similar to seeking expert advice while retaining the ability to make independent decisions. this approach strikes a balance between independence (the draft model's autonomy) and collaboration (the target model's validation), leading to faster inference without sacrificing accuracy much even exceeding target models. we conduct extensive empirical experiments us-ing both publicly available datasets (covering areas such as math, coding, conversation, and instruction following) and real-world industrial scenarios, and observe significant benefits. additionally, we compare the consistency of cd and top-p sampling and found theoretical evidence that cd can maintain the performance of target models. our analysis of speedup ratios and llm-calls ratios also revealed that cd shows better scalability. ## related work speculative decoding @xcite @xcite has been widely adopted and extended across various works, yielding notable speedups in numerous model families and datasets. several strategies for optimizing the draft model to accelerate the generation of draft tokens have also been explored in the literature. these approaches include training a smaller draft model specifically optimized for faster token generation @xcite @xcite and utilizing non-causal algorithms to generate draft tokens @xcite @xcite , which enable parallelization of the token generation process, reasonably boosting throughput. additionally, several research efforts have focused on improving the acceptance rate of draft tokens by relaxing the verification criteria @xcite @xcite . instead of strictly requiring that the distribution from speculative sampling aligns perfectly with the distribution of llms, these approaches allow for a certain degree of deviation within an acceptable range. these methods achieve a speedup by moderately sacrificing predictive accuracy. research efforts have also been undertaken to examine the structure of the drafts with the aim of boosting the acceptance rate of draft tokens: 1) multi-candidate draft: these methods @xcite @xcite @xcite involve using tree-based attention mechanisms to generate multiple candidate draft token sequences during each iteration for chances of accepting more tokens. 2) block-wise verify: researchers have also explored moving beyond the token-level verification of sd, opting instead for block-wise conditional probabilities @xcite . 3) dynamic draft length: in an effort to minimize the rejection of draft tokens for im-proving the acceptance rate @xcite . existing works, such as bild @xcite , have employed similar verification methods to ours by evaluating token accuracy through the distribution distance between the hard label of the draft model and the soft label of the target model. our work focuses on the novel non-distribution verification mechanism, which differs remarkably from the above diverse research threads. ## conclusion this paper develops consultant decoding (cd), a novel decoding method based on nll, that improves speculative decoding for model inference speedup. unlike traditional methods that trade efficiency for performance, cd aligns with the convergence loss of target model training, achieving higher speedup without compromising or even enhancing the target model performance by theoretical analysis and case study. comprehensive experiments show the acceleration advantage, generalization, resilience and scalability of cd. furthermore, we provide a comparison of cd with top-p sampling, providing a theoretical basis for how cd maintains the inference accuracy of the target model with a threshold ε of 2.0. this advantage provides convenience for the use of cd in practical scenarios. the generalization of cd to other architectures (e.g., eagle-2) further highlights its robustness and scalability.