Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training (2024)

Abstract: Existing speculative decoding methods typically require additional model structure and training processes to assist the model for draft token generation. This makes the migration of acceleration methods to the new model more costly and more demanding on device memory. To address this problem, we propose the Make Some Noise (MSN) training framework as a replacement for the supervised fine-tuning stage of the large language model. The training method simply introduces some noise at the input for the model to learn the denoising task. It significantly enhances the parallel decoding capability of the model without affecting the original task capability. In addition, we propose a tree-based retrieval-augmented Jacobi (TR-Jacobi) decoding strategy to further improve the inference speed of MSN models. Experiments in both the general and code domains have shown that MSN can improve inference speed by 2.3-2.7x times without compromising model performance. The MSN model also achieves comparable acceleration ratios to the SOTA model with additional model structure on Spec-Bench.

## introduction large language models (llms) represented by @xcite and llama @xcite have made great breakthroughs to artificial intelligence @xcite . however, llms suffer from high inference latency due to the autoregressive (ar) decoding paradigm, which constrains the model to generate only one token per decoding step. it significantly limits the applications of llms when needs lengthy response. to address the bottleneck introduced by ar, speculative decoding @xcite is proposed to get more than one token in one decoding step. it first guesses multi-step † equal contribution. * corresponding author. ## conclusion in this paper, we propose an effective training framework make some noise (msn) to be used as a replacement for the sft stage. it enhances the denoising ability of the model without affecting the sft training performance. combined with our proposed tr-jacobi decoding strategy, the msn model is able to achieve 2.3-2.7x speedup in the general and code domains without additional structure and training.