Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference (2025)

Abstract: The auto-regressive decoding of Large Language Models (LLMs) results in significant overheads in their hardware performance. While recent research has explored various speculative decoding techniques for multi-token generation, these methods introduce high memory costs from the additional weights and KV cache of separate draft models, limiting efficiency in edge and long-context scenarios. To overcome these limitations in edge-scale LLMs, we propose a novel parallel prompt decoding that requires only runtime memory overhead by employing a unified single model for both speculation and verification. Inspired by the human natural language generation process, PPD approximates outputs generated at future timesteps in parallel by using multiple prompt tokens. Furthermore, we present a hardware-aware two-stage tree pruning algorithm that adaptively optimizes this decoding scheme to fully leverage the computational capacities on different GPUs. Through extensive experiments across LLMs ranging from MobileLlama to Vicuna-13B on a wide range of benchmarks, our approach demonstrates up to 2.49 times speedup. Moreover, our parallel prompt decoding can serve as an orthogonal optimization for synergistic integration with existing speculative decoding, showing up to 1.22 times further speed improvement. To support future development, we have included our code implementation with this submission.

## introduction the recent advances in large language models (llms) are increasingly shaping and influencing a wide range of ai applications. however, autoregressive generation, the de facto approach employed in llm inference, suffers from inadequate hardware performance due to its inherent sequential nature @xcite . speculative decoding @xcite @xcite , an emerging acceleration technique, employs a guess-and-verify framework for llm inference, where a smaller draft model first predicts multiple tokens sequentially and then the original llm verifies them in parallel. despite its potential, speculative decoding is constrained by high training costs and memory overhead, as shown in figure 1 . particularly, this paper reveals that the extra runtime memory required for draft model storage and the kv cache-which scales linearly with sequence length-limits its practicality in edge devices, mobile environments, and long-context scenarios. therefore, given the growing need for user privacy and personalization, this paper focuses on accelerating edge-scale llms via a novel more memory-and cost-efficient acceleration solution. recent efforts have explored the possibility of generating multiple tokens in parallel without relying on a separate transformer draft model @xcite . approaches such as inserting additional decoding heads @xcite and retrieving frequently used tokens @xcite are employed to enhance performance. however, these methods either make strong conditional independence assumptions for tokens generated within a single step @xcite or rely on unparameterized trained tokens that do not scale effectively @xcite . therefore, they often suffer from low acceptance rates during inference. to alleviate the complexity and overhead associated with the use of draft models while maintaining a high acceptance rate, we propose parallel prompt decoding (ppd), a novel architecture-agnostic and memory-efficient framework that adopts prompt tuning for non-autoregressive llm inference. inspired by the human natural language generation process where continuous words like common expressions and phrases are produced simultaneously, ppd introduces the use of ensemble prompt tokens, the meticulously trained embeddings, for multi-token prediction. specifically, these trained tokens are appended to the original input sequence in parallel, enabling the concurrent generation of multiple output tokens in a single forward pass. the key intuition of ppd lies in the observation that if trained properly, prompt tokens appended to the input can approximate tokens generated at future timesteps, thereby partially recovering the missing conditional dependency information for multi-token generation. by strategically positioning trained prompt tokens, ppd achieves up to a 28% higher acceptance rate when predicting longrange tokens. moreover, to facilitate the optimized implementation of ppd across different hardware platforms, we propose a hardware-aware two-stage tree pruning technique that adaptively refines the prompt structure during runtime based on the computational resources available on the specific hardware. as shown in figure 1a , ppd achieves a comparable speedup to the state-of-the-art speculative decoding approaches with negligible memory overhead and reduced training cost. to demonstrate the effectiveness of our approach, we evaluate ppd on mobilellama @xcite , @xcite . running on a single gpu using the a100-40gb and rtx 4090, our method achieves a speedup ratio for inference from 2.12× to 2.49× across a diverse range of popular datasets including mt-bench, humaneval, and gsm8k. our experiments demonstrate that ppd not only achieves comparable throughput to the state-of-theart speculative decoding method, but it also manages this with significantly fewer trainable parameters-specifically, 0.0002% of trainable parameters-and incurs only a minimal memory overhead (0.0004%), showcasing that ppd is remarkably cost-and memory-efficient. the training of prompt tokens can be completed in 16 hours using one a100 gpu, 8 hours using four geforce rtx 3090 gpus, compared to the 1-2 days on four a100 gpus required for eagle @xcite . furthermore, since ppd does not require the modification of the original llm or the addition of extra networks, it is highly adaptable and orthogonal to other decoding techniques. for instance, it can be effectively combined with a draft model to further reduce inference latency. our contributions are summarized as follows: ## background and related work to enhance the inference speed of llm, various approaches adopt an iterative guess-and-verify strategy to enable multi-token generation. in the guessing phase, potential future tokens are proposed at a faster speed than in traditional autoregressive implementations. subsequently, a parallelized verification process assesses which guessed tokens should be accepted. depending on how tokens are generated during the guess stage, these approaches can generally be categorized as i) speculative decoding and ii) parallel decoding. ## parallel prompt decoding (ppd) the primary advantage of ppd lies in training embeddings for prompt tokens rather than developing a separate model. our method integrates three substeps into a single decoding step, following the guess-and-verify strategy: (1) candidate generation, where multiple candidate continuations foot_0 are predicted by strategically inserting the prompt tokens into the input sequence. we adopt tree attention @xcite to merge the processing of multiple candidates into a single forward pass; ## conclusion we introduced ppd, a memory-efficient and costeffective parallel decoding method that incorporates a hardware-aware online tree optimization. utilizing specially trained prompt tokens to predict long-range tokens accurately, ppd achieves a speedup of up to 2.49× in inference while employing negligible additional trainable parameters and memory overhead. we showcased that ppd offers a novel perspective on the capabilities of parallel decoding. importantly, it could be synergized with other speculative or parallel decoding techniques to expedite inference even further.