Mitigating Sequential Dependencies: A Survey of Algorithms and Systems for Generation-Refinement Frameworks in Autoregressive Models (2025)

Abstract: Sequential dependencies present a fundamental bottleneck in deploying large-scale autoregressive models, particularly for real-time applications. While traditional optimization approaches like pruning and quantization often compromise model quality, recent advances in generation-refinement frameworks demonstrate that this trade-off can be significantly mitigated.This survey presents a comprehensive taxonomy of generation-refinement frameworks, analyzing methods across autoregressive sequence tasks. We categorize methods based on their generation strategies (from simple n-gram prediction to sophisticated draft models) and refinement mechanisms (including single-pass verification and iterative approaches). Through systematic analysis of both algorithmic innovations and system-level implementations, we examine deployment strategies across computing environments and explore applications spanning text, images, and speech generation. This systematic examination of both theoretical frameworks and practical implementations provides a foundation for future research in efficient autoregressive decoding. In the appendix A, we additionally provide experimental comparisons of various baseline methods.

## introduction large models (lms) have demonstrated remarkable capabilities across diverse domains, from text generation @xcite @xcite and translation @xcite @xcite to image synthesis @xcite @xcite and video generation @xcite @xcite . however, these models face a critical challenge: their inherently sequential nature creates significant latency bottlenecks, particularly for real-time applications. while traditional optimization approaches 1 equal contributions. 2 corresponding author. like quantization and pruning often compromise model quality for speed, recent research has focused on maintaining output quality while breaking sequential dependencies through novel algorithmic and system-level innovations. generation-refinement frameworks have emerged as a promising family of solutions that directly address these sequential bottlenecks. these approaches encompass a range of methods, from speculative decoding with draft models to iterative refinement techniques inspired by numerical optimization. the common thread among these approaches is their division of the generation process into two phases: an initial generation step that produces draft tokens in parallel, followed by a refinement step that ensures output quality. the implementation of these frameworks presents unique system-level challenges across different deployment scenarios. edge devices require careful optimization of memory usage and computation patterns @xcite , while distributed systems must manage complex communication patterns and load balancing. these system-level considerations have driven innovations in areas like kernel design, hardware acceleration, and batch processing optimization, significantly influencing both algorithmic choices and practical performance. this survey synthesizes research across these approaches, examining both algorithmic innovations and their system implementations. we present a systematic taxonomy of generation-refinement methods, analyze deployment strategies across computing environments, and explore applications spanning text, images @xcite , and speech @xcite . our contributions include comprehensive analysis of system-level implementations and optimizations, detailed examination of applications across modalities, and identification of key research challenges in efficient neural sequence generation. ## the sequential bottleneck in large model inference traditional approaches to accelerating lm inference have focused on reducing computational costs through model compression, knowledge distillation, and architectural optimizations. however, these methods primarily address individual computation costs rather than the fundamental sequential dependency that requires each token to wait for all previous tokens. speculative decoding (sd) @xcite ) has emerged as a promising solution that directly targets this sequential bottleneck. as illustrated in figure 1 , this approach introduces a two-phase process where a smaller, faster draft model first predicts multiple tokens in parallel, followed by verification using the target model. the draft model enables parallel token generation, breaking away from traditional token-by-token generation, while the target model's verification step maintains output quality through accept/reject decisions. this strategy has proven particularly valuable for real-time applications like interactive dialogue systems, where response latency directly impacts user experience. the verification mechanism provides a crucial balance between generation speed and output quality, accepting correct predictions to maintain throughput while falling back to sequential generation when necessary to preserve accuracy. while sd represents one successful approach to breaking sequential dependencies in autoregressive (ar) models, it belongs to a broader family of generation-refinement methods. the following sections present a systematic taxonomy of these approaches, examining how different techniques balance the trade-offs between generation paral- ## a taxonomy for generation and refinement frameworks to systematically analyze approaches for breaking sequential dependencies in large models, we propose a unified taxonomy that categorizes methods based on their generation and refinement strategies. as shown in figure 2 , our taxonomy decomposes these frameworks into two fundamental phases: sequence generation and sequence refinement. this decomposition not only encompasses traditional sd approaches but also captures a broader range of emerging methods that trade off between generation parallelism and output quality. the sequence generation phase focuses on different strategies for producing draft tokens more efficiently than conventional auto-regressive decoding using a single larger model. these strategies range from simple approaches like random token sampling (used in conjunction with iterative decoding) to more sophisticated methods like retrieval-based generation and draft model prediction. each generation method offers trade-offs in terms of computational cost and prediction quality. the sequence refinement phase then determines how these candidates are processed -either accepting them directly (with possible poorer quality), verifying a subset of tokens in a single pass, or refining the draft tokens through multiple iterations until convergence. ## conclusion this survey analyzes generation-refinement frameworks for mitigating sequential dependencies in autoregressive models, highlighting how these approaches are fundamentally changing efficient neural sequence generation across text, speech, and vi-sual domains. through examining both algorithmic innovations and system-level implementations, we have demonstrated their broad applicability while providing crucial deployment insights for practitioners. moving forward, significant challenges persist in constructing solid theoretical foundations to grasp the balance between parallelism and quality, as well as in developing comprehensive approaches that span different modalities-efforts that could narrow the divide between the capabilities of large models and their actual implementation. additionally, it remains crucial to examine the scalability of the speculative decoding system as the quantity of draft and target models increases.