Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2023)

Abstract: We propose Speculative Decoding (SpecDec), for the first time ever, to formally study exploiting the idea of speculative execution to accelerate autoregressive (AR) decoding. Speculative Decoding has two innovations: Spec-Drafter – an independent model specially optimized for efficient and accurate drafting – and Spec-Verification – a reliable method for verifying the drafted tokens efficiently in the decoding paradigm. Experimental results on various seq2seq tasks including machine translation and abstractive summarization show our approach can achieve around 5x speedup for the popular Transformer architectures with comparable generation quality to beam search decoding, refreshing the impression that the draft-then-verify paradigm introduces only 1.4x~2x speedup. In addition to the remarkable speedup, we also demonstrate 3 additional advantages of SpecDec, revealing its practical value for accelerating generative models in real-world applications. Our models and codes are available at https://github.com/hemingkx/SpecDec.

## introduction as the de facto method for text generation, au-toregressive (ar) decoding is widely blamed for its poor inference efficiency due to its low level of parallelism, which fails to utilize the full potential of modern parallel computing devices like gpus. this inefficiency not only leads to high deployment figure 1 : compared with autoregressive decoding (left) that generates token by token, the draft-then-verify paradigm (right) first drafts multiple tokens efficiently and then verifies these tokens in parallel. drafted tokens after the bifurcation position (e.g., y 5 ) will be discarded to guarantee the generation quality. in this work, we study the draft-then-verify paradigm for accelerating seq2seq generation of an existing ar model foot_0 . as shown in figure 1 , the draft-then-verify paradigm first generates a number of drafted tokens efficiently and then verifies these tokens using the existing ar model in parallel to ensure the decoding result matches ar decoding. however, previous attempts in the "draftthen-verify" paradigm such as blockwise decoding @xcite and aggressive decoding @xcite tend to lack in-depth investigation of this paradigm. their modest speedup (i.e., 1.4×∼2.0×) or limitation to certain seq2seq tasks like grammatical error correction (gec) has caused this paradigm to be underestimated, resulting in it not receiving much attention and remaining dormant for years. to fully exploit the draft-then-verify paradigm, we propose speculative decoding (specdec), drawing inspiration from speculative execution 3 in com-puter architecture, with two key innovations that improve drafting and verification processes respectively. for drafting, we derive two principles for designing the drafting model 4 : the capability principle and the latency principle. following these two principles, we propose spec-drafter -a specialized independent model optimized in the draftthen-verify paradigm, which can accurately and efficiently fulfill the drafting task. for verification, we propose an advanced method -spec-verification that relaxes the vanilla verification strategy. spec-verification allows the decoding results of specdec to be slightly different from ar greedy decoding, offering an opportunity to accept more drafted tokens without sacrificing generation quality and leading to higher decoding efficiency. we conduct extensive experiments on various seq2seq generation tasks like machine translation and abstractive summarization. results show our approach can achieve around 5× speedup for the popular transformer architectures with comparable generation quality to beam search decoding, largely outperforming previous draft-thenverify work (1.4×∼2.0× speedup). moreover, we demonstrate that specdec has several additional advantages that enhance its practicality for accelerating generative models in real-world applications. our contributions can be summarized as follows: ## background: draft-then-verify decoding the "draft-then-verify" paradigm first drafts multiple tokens efficiently, as a speculation of ar decoding results; then, it verifies these tokens in parallel to ensure they match the ar decoding result, as illustrated in figure 1 . it is an implicit implementation of speculative execution in transformer inference. draft there are different approaches to drafting tokens, including model-based @xcite and input-(context-) based foot_2 methods @xcite . take blockwise decoding -the most representative work attempting the draft-then-verify paradigm -as an example (illustrated in figure 2 (a)): it introduces additional k -1 feedforward network (ffn) heads on top of an existing ar model, enabling the model to predict the next k drafted tokens in parallel during inference. ## speculative decoding to fully exploit speculative execution for transformer inference, we propose speculative decoding (specdec) with two innovations -spec-drafter and spec-verification that substantially improve drafting (section 3.1) and verification (section 3.2) respectively. transformer block 𝑦 ! 𝑦 " 𝑦 # 𝑦 $ 𝑦 % [𝑀] [𝑀] [𝑀] ## conclusion we present speculative decoding (specdec), the first work to explicitly embrace the idea of speculative execution for seq2seq generation acceleration with a formal study and extensive discussion of both drafting and verification phases. contrary to the common belief that an increase in model complexity tends to hamper inference speed, specdec's introduction of an appropriately invested auxiliary drafter model substantially speeds up transformer inference, owing to higher computational parallelism introduced by speculative execution to better utilize computing resources. the remarkable acceleration performance, combined with the advantages demonstrated in our experiments, clearly illustrates that specdec is a practical acceleration method for model deployment in real-world applications. we hope that our preliminary study could draw more attention to this promising decoding paradigm that may potentially evolve into a de facto standard for efficient transformer decoding in the near future.