Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (2024)

Abstract: We present a novel inference scheme, self-speculative decoding, for accelerating Large Language Models (LLMs) without the need for an auxiliary model. This approach is characterized by a two-stage process: drafting and verification. The drafting stage generates draft tokens at a slightly lower quality but more quickly, which is achieved by selectively skipping certain intermediate layers during drafting. Subsequently, the verification stage employs the original LLM to validate those draft output tokens in one forward pass. This process ensures the final output remains identical to that produced by the unaltered LLM. Moreover, the proposed method requires no additional neural network training and no extra memory footprint, making it a plug-and-play and cost-effective solution for inference acceleration. Benchmarks with LLaMA-2 and its variants demonstrated a speedup up to 1.99 \times .

## introduction transformer-based large language models (llms), such as gpt-3/4, palm, and llama, have been widely adopted in various real-world applications @xcite @xcite @xcite . however, their inference costs have raised significant concerns, especially for latency-sensitive scenarios @xcite . the main efficiency bottleneck is the autoregressive decoding process. this process decodes each output token sequentially, leading to a high number of transformer calls; furthermore, each transformer call is typically memory bandwidth-bound, resulting in low computation utility and thus longer wall-clock time @xcite . for instance, decoding 128 tokens autoregressively using llama-2-13b on an a100 gpu can take up to 100× longer than a sequence-level forward pass on the same number of tokens, highlighting the substantial inefficiency inherent in the current decoding process. established model compression techniques such as quantization @xcite , pruning @xcite , and distillation @xcite have been employed to alleviate these costs. while these solutions have proven extremely effective, they usually require changing the model architecture, changing the training procedure, retraining or fine-tuning the models, and do not maintain identical outputs. in parallel to model compression, speculative execution is being explored to accelerate the autoregressive decoding process @xcite . these methods train an auxiliary draft model that can quickly generate some draft output tokens. subsequently, the original llm, referred to as the verify model, then checks the acceptability of these draft tokens with one single forward pass. this verification step ensures that the outputs are derived from the original llm's probability distribution. however, an essential issue of existing speculative execution methods is the need to identify or train a suitable draft model that can generate outputs consistent with the verify model. it becomes more tricky when the llm is already a finetuned model, e.g. llama-2-chat @xcite , @xcite . how to find or train a draft model that can effectively mimic the outputs of such a tailored model is a formidable task, with no straightforward or guaranteed solutions. furthermore, the introduction of an additional draft model escalates the gpu memory overhead, increasing deployment challenges particularly on devices with restricted memory capacity. in this paper, we present self-speculative decoding, a novel approach to accelerate the inference of llms. this method builds on the principles of mlp mlp mlp ## method in this section, we first go through the standard autoregressive decoding. subsequently, we provide a detailed depiction of our proposed method, including selectively skipping layers during drafting, and adaptively determining the number of draft tokens. ## conclusion in this paper, we introduced self-speculative decoding, a novel and efficient inference scheme that accelerates transformer-based llms. our method does not depend on additional neural network training and incurs no extra device memory, making it a highly practical and cost-effective solution for inference acceleration. moreover, we used bayesian optimization to search for layers to skip in drafting and proposed an adaptive draft-exiting mechanism to improve the end-to-end inference speed. benchmark tests with llama-2 and its fine-tuned models demonstrated a speedup of up to 1.99×. for future work, we aim to explore more sophisticated model compression strategies to further accelerate the drafting stage for low-resource scenarios.