S eq1 F 1 B : Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training (2025)

Abstract: Training large language models (LLMs) heavily relies on distributed training strategies, among which pipeline parallelism (PP) plays a crucial role. As training sequences extend to 32k or even 128k tokens, current PP methods face severe bottlenecks, including substantial pipeline bubbles and high memory footprint, greatly hindering training throughput and model scalability. This paper introduces a sequence-level one-forward-one-backward (1F1B) PP method, named Seq1F1B, tailored for training LLMs on long sequences with high training throughput and memory efficiency. Unlike typical PP methods, which adopt batch-level pipeline schedule, Seq1F1B schedules the pipeline of training LLMs at the sequence level. It uses a computational strategy to partition sequences appropriately, significantly reducing pipeline bubbles and memory footprint. Compared to competitive PP baselines such as Megatron 1F1B PP, Seq1F1B achieves 1.14X training throughput with half memory footprint.Notably, Seq1F1B trains an LLM with 30B parameters on sequences up to 64k tokens using 64X NVIDIA A100 GPUs without using recomputation strategies, a feat unachievable with existing methods.We have released our code on GitHub to facilitate further research and development in LLM training on long sequences: https://github.com/thunlp/Seq1F1B.

## introduction efficient distributed strategies @xcite @xcite play a crucial role in training large language models (llms), and these llms have revolutionized various nlp tasks in recent years @xcite @xcite . among these strategies, pipeline parallelism (pp) @xcite stands out due to its low communication bandwidth requirement and great computing resource scalability, and it can be easily integrated with other strategies such as data parallelism (dp) @xcite and tensor parallelism (tp) @xcite . pp involves partitioning a model into multiple stages, with each computing device processing a stage consisting of consecutive layers. this paradigm inherently leads to "bubbles"-the idle time caused by the execution topology between the sharded layers. several ingenious pipeline schedule strategies have been proposed to address this bubble problem. gpipe @xcite reduces bubbles by splitting each batch of training sequences into micro-batches, coming at the cost of increased memory usage, as each pipeline stage must store the intermediate states of all micro-batches generated during forward passes until backward passes are completed. to address the high memory demand of gpipe, one-forwardone-backward (1f1b) methods are proposed @xcite @xcite . 1f1b methods make backward passes have higher execution priority than forward passes and schedule backward passes in advance without affecting final results. owing to this, the memory demand for storing intermediate states can be reduced without adding extra bubbles. generally, optimizing pp relies on handling the trade-off between bubble ratio and memory footprint. recent studies @xcite have highlighted the advantages of longsequence training for llms in various aspects. however, training llms on long sequences remains challenging due to the quadratic time and memory complexities with respect to the input sequence in transformer attention modules @xcite . some works, such as @xcite , propose parallelizing attention computation across workers in a distributed cluster to enable efficient training of llms on long sequences. in this approach, each worker must communicate activations during attention computation and synchronize the model's weights. consequently, these approaches exhibit poor performance when communication bandwidth is limited. compared to these methods, pp incurs significantly lower communication overhead and is thus a more suitable choice when communication bandwidth is constrained. however, long sequence data presents new challenges, such as increased memory demands and higher bubble ratios, making it difficult to achieve effective training with pp methods. for gpipe and existing 1f1b methods, whose minimal schedulable unit is micro-batch, inevitably face the memory overflow caused by just a single microbatch, as training sequences extend to extremely long lengths. long sequences make balancing bubble ratio and memory footprint more challenging for pp methods. in this paper, we introduce a sequence-level 1f1b (seq1f1b) pp method. this method capitalizes on the causal self-attention mechanism of llms to schedule pipeline stages at the sequence level. in contrast to existing 1f1b methods @xcite , seq1f1b offers significant efficiency and memory benefits. specifically, splitting sequences into sub-sequences allows for a significant reduction in memory footprint since only the intermediate states of sub-sequences rather than micro-batches need to be retained. scheduling the pipeline at the sequence level yields more stages and thus reduces the bubble ratio. while, the causal nature of llms also causes a dependency between the forward and backward passes of different sub-sequences, i.e., the forward passes of later sub-sequences rely on earlier ones, and vice versa for the backward passes of early sub-sequences, bringing the challenge for the pipeline schedule. to this end, we introduce a partially ordered queue in seq1f1b to replace the first-in-first-out (fifo) queue used in existing 1f1b methods and reorganize the pipeline schedule, so that we can preserve the exact execution dependencies between forward and backward passes while providing synchronous parallelism. to further improve seq1f1b, we propose a strategy for balancing the workload across sub-sequences rather than simply splitting sequences evenly along the sequence dimension. sufficient experiments demonstrate that seq1f1b significantly outperforms recent popular 1f1b methods @xcite in terms of memory efficiency and training throughput for training llms, with the sequence length ranging from 16k to 128k and the model size ranging from 2.7b to 32b. as the sequence length increases, the efficiency of seq1f1b becomes more pronounced. seq1f1b supports efficiently training an llm with 30b parameters on sequences up to 64k tokens using 64×nvidia a100 gpus without using recomputation strategies, which is unachievable with existing pp methods. ## related work training llms requires using a mixture of parallelism strategies, the most important of which are dp, tp, and pp @xcite . for pp, pipeline schedules can be broadly categorized into two main types: synchronous schedules and asynchronous schedules. asynchronous schedules such as asynchronous pipedream @xcite and pipemare @xcite can achieve bubblefree results but suffer from the performance degradation of final trained models because they use outdated parameters to compute gradient updates. as for synchronous schedules, gpipe @xcite and 1f1b @xcite are the most commonly used pipeline schedules following synchronous settings. they achieve much fewer bubbles as the number of micro-batch increases and guarantee mathematical equivalent to the original training process. given this, our work focuses on improving synchronous pipeline schedules as they ensure consistent semantics across different parallelism strategies. the original gpipe @xcite simply divides a batch into several micro-batches, and its scheduling process has only two phases: the forward phase and the backward phase. the backward passes are executed only after the forward passes of all micro-batches within a batch are completed. during the forward phase, the intermediate states of each micro-batch are enqueued into a fifo queue q. during the backward phase, these intermediate states are dequeued for their corresponding backward passes. since the backward phase happens after all intermediate states are queued, gpipe exhibits an o(m ) memory consumption, where m represents the number of micro-batches. ter-apipe @xcite relies on the observation of different from gpipe, 1f1b @xcite alternates between forward and backward passes (adopting a 1f1b pattern) to keep the number of intermediate states in the fifo queue q constant. regardless of the number of micro-batches, 1f1b mitigates excessive memory usage. based on 1f1b, 1f1b with interleaved stages (1f1b-i) @xcite enlarges the number of pipeline stages and assigns each device multiple stages. by interleaving stages among devices, 1f1b-i reduces the bubble ratio at the cost of adding more communication operators and slightly increasing memory consumption. zerobubble-pipeline (zb1p) @xcite divides the backward pass into weight gradient computation and input gradient computation separately. this approach achieves higher pipeline efficiency by delaying weight gradient computation and optimizing the schedule using dynamic programming. zb1p nearly achieves zero-bubble pipeline efficiency but brings more memory footprint caused by delaying memory release. 1f1b methods are the most popular for training llms, yet still suffer from difficulties in balancing bubble ratio and memory footprint, which is the issue we want to solve. ## methodology in this section, we first give a preliminary overview to introduce the characteristics of the 1f1b schedule and language modeling. then, we prove why it is feasible to schedule the pipeline of training llms at the sequence level for micro-batches in 1f1b. finally, we explain how seq1f1b works in detail and how it meets the exact semantics of original language modeling. ## conclusion in this paper, we present seq1f1b, an efficient 1f1b pipeline parallel schedule orienting to training transformer-based llms on long sequences by decomposing the batch-level schedulable units used by typical 1f1b methods into more finegrained sequence-level units. to achieve a better workload balance of the sequence-level pipeline, we design a computation-wise sequence partition strategy to partition the sequences that evenly distribute computational load across devices. mean-while, seq1f1b can integrate with other pipeline parallel methods such as 1f1b with interleaved stage or zero-bubble-pipeline. our evaluations demonstrate that seq1f1b outperforms the 1f1b and 1f1b-i schedules regarding memory efficiency and training throughput under variable sequence lengths and model sizes. moreover, seq1f1b can support the efficient training of a 30b gpt model on sequences up to 64k in length using 64× a100 gpus without recomputation strategies, which is unachievable with existing pipeline parallel methods. in the future, we will thoroughly combine our method with other distributed methods to achieve better llm training acceleration.