SL i M : Speculative Decoding with Hypothesis Reduction (2024)

Abstract: Speculative decoding has emerged as a prominent alternative to autoregressive decoding for expediting inference in large language models (LLMs). However, prevailing assumptions often focus solely on latency reduction, neglecting the computational expenses. In this paper, we present S peculate L ess, val i date M ore (SLiM), a speculative decoding enhancement to reduce the speculation set while validating more effective tokens. SLiM is designed to mitigate LLMs’ computation costs associated with the token verification by introducing hypothesis reduction based on a fast posterior estimation. It consistently surpasses counterparts lacking cost reduction across a spectrum from CPU to GPU. Our evaluation with diverse conversational datasets shows that SLiM can achieve a substantial 70% reduction in FLOPs while generating more effective predictions on top of prior arts.

## introduction recent advancements in large language models (llms), such as llama @xcite , @xcite , and vicuna @xcite , have showcased their tremendous potential as proficient artificial intelligence (ai) assistants @xcite across diverse domains. despite their widespread adoption, these models are severely limited by inference speed due to their serial decoding mechanisms. to address this concern, several methods have been proposed to expedite token generation. in particular, speculative decoding @xcite @xcite @xcite has emerged as a prominent strategy by leveraging the speculate and verify mechanism, resulting in substantial inference time reduction. it is a dual-stage procedure wherein (1) lightweight models generate hypothesized token sequences speculatively, and at the party where at our house and at the house and at your party and at our party and 1 ) generating multiple hypotheses with a lightweight drafting mechanism and (2) verifying the hypotheses with powerful hardware in parallel. our work introduces a hypothesis reduction stage to drastically reduce the computation, making our approach friendly for resource-constrained devices. two criteria need to be met for speculative decoding to gain speed: (1) the lightweight model must draft the predictions much faster than the original llm, and (2) the device must have sufficient computation throughput for the parallel verification of multiple hypotheses. while the first criterion (light drafting mechanism) has garnered attention and inspired several works, the second criterion (speedy verification) has been generally overlooked in the community. this becomes a substantial concern since recent methods tend to generate a relatively larger number of hypotheses @xcite . in many practical scenarios, the inference has very limited computational budgets, and therefore a high-complexity verification step can offset the speed-up benefits gained from speculation. to mitigate the above caveat, we introduce the three-stage speculative decoding process which inserts a novel hypothesis reduction stage between the two existing stages. figure 1 illustrates the process, in which our hypothesis reduction performs a fast posterior estimation to eliminate unlikely candidates, yielding high computation savings in the verification stage. in other words, we propose to have a lightweight verification strategy before performing the expensive one. our lightweight verification computes the corrected posterior estimation based on a simple bigram correlation function, providing a more confident assessment to prune the hypotheses. figure 2 provides a preview of how our strategy significantly reduces the number of floating-point operations (flops) per speculation step needed to achieve the same speed. in summary, we make the following contributions: ## related works given the widespread use of large language models (llms) across diverse applications, the acceleration of their inference speed has garnered significant attention @xcite . extensive efforts have been dedicated to replacing the autoregressive decoding method with parallel decoding (or non-autoregressive decoding) @xcite @xcite @xcite @xcite . while these works focus on machine translation, @xcite ; @xcite ; @xcite ; @xcite consider sentence generation task. however, these approaches often necessitate intricate model training, which limits their practical applicability in the context of large models. recently, speculative decoding (chen et al., 2023; @xcite has stood out as a prominent approach to accelerate llms' inferences. this method adopts a "speculate and verify" strategy, utilizing a draft model-a smaller model with faster inference capabilities-to propose tokens for verification by the original model. this approach enables the generation of multiple tokens simultaneously, enhancing overall inference speed. while earlier works predominantly focus on single-sequence speculation @xcite @xcite @xcite , recent advancements leverage parallel computing for batchsequence speculations to further accelerate inferences @xcite @xcite . notably, @xcite introduces a novel lossless parallel decoding method based on jacobi iteration. however, these approaches often come at the cost of increased computation due to the enlargement of the hypothesis set. in the context of lossy acceleration methods, where the generated output deviates from the original model, bild @xcite ) employs a large model to enhance and refine the generation of a smaller model. on the other hand, sot @xcite takes a distinct approach by initially creating a skeleton of points and subsequently completing each point in parallel. slim differs from them by preserving the original outputs. in contrast to existing research, slim incorporates the batch-speculative strategy with sequence posterior estimation to reduce the hypothesis set without compromising the acceptance rate. the estimation has been guided with a correlation function learned from online corpora. while @xcite also leverages online corpora for speculation, it necessitates an exact match of context between the corpus and model inputs, limiting its applicability to specific types of problems like retrieval-augmented generation. in contrast, slim is application agnostic. when selecting the draft model, slim follows the strategy introduced by @xcite and @xcite to mitigate additional complexity. this method entails integrating additional prediction heads atop the base model, allowing it to predict extra tokens. notably, while these approaches share a common foundation, neither incorporates any hypothesis reduction technique. slim stands out as the first method specifically designed to reduce the complexity of both speculative and verification stages. ## conclusions in this work, we introduced slim, a speculative decoding enhancement framework designed to alleviate the computational burden associated with token verification. our method leverages sequence posterior estimation as a lightweight verifier by incorporating bigram information. starting with an exponentially large speculative set, we judiciously eliminate most speculations with low posteriors. subsequently, only the sequences in the reduced hypothesis set are verified using the llm. this approach allows slim to speculate fewer sequences while validating more tokens. empirically, our results demonstrate that slim's acceleration surpasses alternative methods lacking this augmentation across diverse devices. on an rtx 4090, slim achieves a no