EMS - SD : Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models (2025)

Abstract: Speculative decoding emerges as a pivotal technique for enhancing the inference speed of Large Language Models (LLMs). Despite recent research aiming to improve prediction efficiency, multi-sample speculative decoding has been overlooked due to varying numbers of accepted tokens within a batch in the verification phase. Vanilla method adds padding tokens in order to ensure that the number of new tokens remains consistent across samples. However, this increases the computational and memory access overhead, thereby reducing the speedup ratio. We propose a novel method that can resolve the issue of inconsistent tokens accepted by different samples without necessitating an increase in memory or computing overhead. Furthermore, our proposed method can handle the situation where the prediction tokens of different samples are inconsistent without the need to add padding tokens. Sufficient experiments demonstrate the efficacy of our method. Our code will be released later.

## introduction large language models (llms) @xcite @xcite have demonstrated considerable capabilities, particularly in the realm of natural language processing. autoregressive large language models generate a token in a single pass, whereas speculative decoding allows large models to generate multiple tokens in a single pass, thereby greatly improving inference speed. it is crucial to highlight that the inference time of llms on a single token and multiple tokens is approximate. consequently, reducing the number of inference steps can significantly reduce the inference time. a plethora of efficient speculative decoding methods have been proposed recently. however, none of these methods provide a comprehensive figure 1 : speedup ratio of opt-6.7b on the cnn/daily mail dataset for greedy settings when batch @xmath0, utilizing llma @xcite as the basic speculative decoding method. our method demonstrates superior performance to the vanilla method under varying batch sizes. the larger the batch size, the more pronounced the advantage of our method. can we perform multi-sample speculative decoding without increasing computational and memory access overhead? padding tokens. furthermore, in anticipation of the potential discrepancy in the number of predicted tokens across different samples, we proposed the unpad input tokens method as a solution in the prediction phase. this method concatenates all input tokens prior to inference and expands these tokens during the calculation of attention. the main contributions are as follows: ## conclusions in this paper, we present the first study of multisample speculative decoding. we introduce an effective method, called ems-sd. ems-sd is an effective solution to the inconsistency problem of different samples in the prediction and verification stages, without the need of padding tokens. the proposed method is flexibly integrated with almost any basic speculative decoding method. extensive comparisons show that ems-sd exhibits superior performance compared to the vanilla method in multi-sample speculative decoding.