P ipe S pec: Breaking Stage Dependencies in Hierarchical LLM Decoding (2025)

Abstract: Speculative decoding accelerates large language model inference by using smaller draft models to generate candidate tokens for parallel verification. However, current approaches are limited by sequential stage dependencies that prevent full hardware utilization. We present PipeSpec, a framework that generalizes speculative decoding to use multiple models arranged in a hierarchical pipeline, enabling asynchronous execution with lightweight coordination for prediction verification and rollback. Our analytical model characterizes token generation rates across pipeline stages and proves guaranteed throughput improvements over traditional decoding for any non-zero acceptance rate. We further derive closed-form expressions for steady-state verification probabilities that explain the empirical benefits of pipeline depth. We validate PipeSpec across text summarization, mathematical reasoning, and code generation tasks using LLaMA 2 and 3 models, demonstrating that pipeline efficiency increases with model depth, providing a scalable approach to accelerating LLM inference on multi-device systems. Our code is available at https://github.com/BradMcDanel/PipeSpec.

## introduction large language models (llms) have transformed natural language processing through their remarkable ability to understand and generate human-like text. however, the fundamental requirement of autoregressive token generation, where each token must be generated sequentially based on all previous tokens, creates significant performance bottlenecks. this limitation is particularly pronounced in modern llms with 100b or more parameters @xcite , making real-time applications challenging. recent advances in speculative decoding have shown promise by leveraging smaller, faster models to draft candidate tokens for verification by larger models. however, current as illustrated in figure 1 , traditional autoregressive decoding using a single large model is limited to 1 token per unit time due to strict sequential dependencies. standard speculative decoding improves throughput to 1.5 tokens per unit time by employing a small draft model (10 tokens/unit) to generate candidates for batch verification by the large model. however, this approach still suffers from alternating idle periods where either the draft or verify model must wait for the other to complete. our key insight is that these limitations can be overcome through pipelining of multiple models. pipespec introduces a novel k-model architecture where each consecutive pair of models operates in an asynchronous producer-consumer relationship. in the three-model configuration shown in figure 1 (bottom), an initial small model (m 0 ) rapidly generates draft tokens (10 tokens/unit), which are progressively refined by a medium-sized model (m 1 , 5 tokens/unit) before final verification by the large model (m k , 2.25 tokens/unit). this hierarchical structure provides two key advantages: (1) each stage operates asynchronously, enabling continuous parallel execution without idle periods, and (2) the intermediate models provide higher-quality draft tokens compared to single-draft approaches while still benefiting from their own draft-verify speedups. pipespec operates through optimistic execution, where each model generates tokens assuming downstream acceptance. when a model rejects a prediction (marked as ✗), it triggers a rollback cascade -all subsequent predictions in earlier pipeline stages must be discarded and regenerated. this enables pipespec to maintain higher throughput than traditional speculative decoding. the main contributions of this work are: recent research has targeted hardware-level optimizations for both phases. for prefill, flashattention @xcite optimizes attention computation through tiling and recomputation strategies, particularly important for long sequences where naive implementations would exceed gpu memory bandwidth. other approaches focus on gpu utilization @xcite @xcite and efficient keyvalue cache management @xcite @xcite . while these approaches optimize individual model execution, they are complementary to our proposed pipespec framework, which focuses on algorithmic speedups through pipelined speculative execution. ## hierarchical pipelined speculative decoding in this section, we first describe the operation of hierarchical pipelined speculative decoding (pipespec) as a k-stage pipeline (section 3.1). we then present the core algorithm of pipespec (section 3.2). finally, we develop a theoretical framework to analyze pipespec's performance characteristics and compare it with existing approaches (section 3.3). ## evaluation our evaluation examines four aspects: end-to-end performance across summarization and code generation tasks (4.2), token acceptance patterns and timing characteristics (4.3), the impact of lookahead window sizes on throughput (4.4), and gpu resource utilization (4.5). ## conclusion we introduced pipespec, a novel framework that fundamentally rethinks speculative decoding by breaking traditional sequential dependencies in llm inference through asynchronous, hierarchical pipelined execution. our analytical model not only proves guaranteed throughput improvements over autoregressive decoding for any non-zero acceptance rate but also provides insights into the steady-state behavior of such pipelined systems. empirically, pipespec consistently delivers substantial speedups, often achieving two to three-fold improvements over autoregressive baselines and notable gains over standard speculative decoding across various models and tasks. this performance is achieved by effectively utilizing multi-gpu setups and minimizing idle hardware time. a key finding is that pipeline efficiency often increases with depth; for instance, three-model configurations generally outperformed two-model setups by enabling progressive token refinement and better draft quality for the final, largest model. this suggests a promising architectural paradigm for future inference systems, particularly as models continue to grow in scale and complexity. by facilitating continuous parallel execution and adaptable draft quality through its hierarchical structure, pipespec offers a robust and scalable approach to accelerating llm inference.