Original: Decentralized Training of Foundation Models in Heterogeneous Environments
Revised: Decentralized Training of Foundation Models in Heterogeneous Environments

Original: Abstract
Revised: Abstract

Original: Training foundation models, such as GPT-3 and PaLM, can be extremely expensive, often involving tens of thousands of GPUs running continuously for months.
Revised: Training foundation models, such as GPT-3 and PaLM, can be extremely expensive, often involving tens of thousands of GPUs running continuously for months.

Original: These models are typically trained in specialized clusters featuring fast, homogeneous interconnects and using carefully designed software systems that support both data parallelism and model/pipeline parallelism.
Revised: These models are typically trained in specialized clusters featuring fast, homogeneous interconnects and using carefully designed software systems that support both data parallelism and model/pipeline parallelism.

Original: Such dedicated clusters can be costly and difficult to obtain.
Revised: Such dedicated clusters can be costly and difficult to obtain.

Original: Can we instead leverage the much greater amount of decentralized, heterogeneous, and lower-bandwidth interconnected compute?
Revised: Can we instead leverage the much greater amount of decentralized, heterogeneous, and lower-bandwidth interconnected compute?

Original: Previous works examining the heterogeneous, decentralized setting focus on relatively small models that can be trained in a purely data parallel manner.
Revised: Previous works examining the heterogeneous, decentralized setting focus on relatively small models that can be trained in a purely data parallel manner.

Original: State-of-the-art schemes for model parallel foundation model training, such as Megatron and Deepspeed, only consider the homogeneous data center setting.
Revised: State-of-the-art schemes for model parallel foundation model training, such as Megatron and Deepspeed, only consider the homogeneous data center setting.

Original: In this paper, we present the first study of training large foundation models with model parallelism in a decentralized regime over a heterogeneous network.
Revised: In this paper, we present the first study of training large foundation models with model parallelism in a decentralized regime over a heterogeneous network.

Original: Our key technical contribution is a scheduling algorithm that allocates different computational “tasklets” in the training of foundation models to a group of decentralized GPU devices connected by a slow heterogeneous network.
Revised: Our key technical contribution is a scheduling algorithm that allocates different computational “tasklets” in the training of foundation models to a group of decentralized GPU devices connected by a slow heterogeneous network.

Original: We provide a formal cost model and further propose an efficient evolutionary algorithm to find the optimal allocation strategy.
Revised: We provide a formal cost model and further propose an efficient evolutionary algorithm to find the optimal allocation strategy.

Original: We conduct extensive experiments that represent different scenarios for learning over geo-distributed devices simulated using real-world network measurements.
Revised: We conduct extensive experiments that represent different scenarios for learning over geo-distributed devices simulated using real-world network measurements.

Original: In the most extreme case, across 8 different cities spanning 3 continents, our approach is 4.8 × faster than prior state-of-the-art training systems.
Revised: In the most extreme case, across 8 different cities spanning 3 continents, our approach is 4.8 × faster than prior state-of-the-art training systems.

Original:  Introduction
Revised:  Introduction

Original: Recent years have witnessed the rapid development of deep learning models, particularly foundation models (FMs) [1] such as GPT-3 [2] and PaLM [3].
Revised: Recent years have witnessed the rapid development of deep learning models, particularly foundation models (FMs) [1] such as GPT-3 [2] and PaLM [3].

Original: Along with these rapid advancements, however, comes computational challenges in training these models: the training of these FMs can be very expensive — a single GPT3-175B training run takes 3.6K Petaflops-days [2]— this amounts to $4M on today’s
Revised: Along with these rapid advancements, however, comes computational challenges in training these models: the training of these FMs can be very expensive — a single GPT3-175B training run takes 3.6K Petaflops-days [2]— this amounts to $4M on today’s

Original: AWS on demand instances, even assuming 50% device utilization (V100 GPUs peak at 125 TeraFLOPS)!
Revised: AWS on demand instances, even assuming 50% device utilization (V100 GPUs peak at 125 TeraFLOPS)!

Original: Even the smaller scale language models, e.g., GPT3-1.3B (1.3 billion parameters), on which this paper evaluates, require 64 Tesla V100 GPUs to run for one week, costing $32K on
Revised: Even the smaller scale language models, e.g., GPT3-1.3B (1.3 billion parameters), on which this paper evaluates, require 64 Tesla V100 GPUs to run for one week, costing $32K on

Original: AWS.
Revised: AWS.

Original: As a result, speeding up training and decreasing the cost of FMs have been active research areas.
Revised: As a result, speeding up training and decreasing the cost of FMs have been active research areas.

Original: Due to their vast number of model parameters, state-of-the-art systems (e.g., Megatron[4],
Revised: Due to their vast number of model parameters, state-of-the-art systems (e.g., Megatron[4],

Original: Deepspeed[5], Fairscale[6]) leverage multiple forms of parallelism [4, 7, 8, 9, 10, 11].
Revised: Deepspeed[5], Fairscale[6]) leverage multiple forms of parallelism [4, 7, 8, 9, 10, 11].

Original: However, their design is only tailored to fast , homogeneous data center networks.
Revised: However, their design is only tailored to fast , homogeneous data center networks.

Original: [Figure]
Revised: [Figure]

Original: On the other hand, decentralization is a natural and promising direction.
Revised: On the other hand, decentralization is a natural and promising direction.

Original: Jon Peddie Research reports that the PC and AIB GPU market shipped 101 million units in Q4 2021 alone [12].
Revised: Jon Peddie Research reports that the PC and AIB GPU market shipped 101 million units in Q4 2021 alone [12].

Original: Furthermore, many of these GPUs are underutilized.
Revised: Furthermore, many of these GPUs are underutilized.

Original: Leveraging this fact, volunteer computing projects such as Folding@Home [13] have sourced upwards of 40K Nvidia and AMD GPUs continuously [14].
Revised: Leveraging this fact, volunteer computing projects such as Folding@Home [13] have sourced upwards of 40K Nvidia and AMD GPUs continuously [14].

Original: Moreover, the incremental electricity and HVAC costs of running a V100 GPU for a volunteer are 50–100 × lower than the spot prices for an equivalent device on AWS [15].
Revised: Moreover, the incremental electricity and HVAC costs of running a V100 GPU for a volunteer are 50–100 × lower than the spot prices for an equivalent device on AWS [15].

Original: If we could make use of these devices in a decentralized open-volunteering paradigm for foundation model training, this would be a revolutionary alternative to the expensive solutions offered by data centers.
Revised: If we could make use of these devices in a decentralized open-volunteering paradigm for foundation model training, this would be a revolutionary alternative to the expensive solutions offered by data centers.

Original: This vision inspired many recent efforts in decentralized learning, including both those that are theoretical and algorithmic [16, 17, 18], as well as recent prototypes such as Learning@Home
Revised: This vision inspired many recent efforts in decentralized learning, including both those that are theoretical and algorithmic [16, 17, 18], as well as recent prototypes such as Learning@Home

Original: [19] and DeDLOC [20].
Revised: [19] and DeDLOC [20].

Original: However, efforts to-date in decentralized training either focus solely on data parallelism [16, 17, 18, 20], which alone is insufficient for FMs whose parameters exceed the capacity of a single device, or orient around alternative architectures, e.g., mixture of experts [19].
Revised: However, efforts to-date in decentralized training either focus solely on data parallelism [16, 17, 18, 20], which alone is insufficient for FMs whose parameters exceed the capacity of a single device, or orient around alternative architectures, e.g., mixture of experts [19].

Original: These alternative architectures provide promising directions for decentralized learning, however, they are currently only trained and evaluated on smaller datasets and at a smaller computational scale (e.g.,
Revised: These alternative architectures provide promising directions for decentralized learning, however, they are currently only trained and evaluated on smaller datasets and at a smaller computational scale (e.g.,

Original: MNIST and WikiText-2 in [19]) than their state-of-the-art counterparts, e.g., GLaM [21].
Revised: MNIST and WikiText-2 in [19]) than their state-of-the-art counterparts, e.g., GLaM [21].

Original: In this paper, we focus on a standard GPT-style architecture, without considering any changes that might alter the model architecture or the convergence behaviour during training.
Revised: In this paper, we focus on a standard GPT-style architecture, without considering any changes that might alter the model architecture or the convergence behaviour during training.

Original: To fulfill the potential of decentralization for the training of FMs, we need to be able to (1) take advantage of computational devices connected via heterogeneous networks with limited bandwidth and significant latency, and (2) support forms of parallelism beyond pure data parallelism.
Revised: To fulfill the potential of decentralization for the training of FMs, we need to be able to (1) take advantage of computational devices connected via heterogeneous networks with limited bandwidth and significant latency, and (2) support forms of parallelism beyond pure data parallelism.

Original: In this paper, we tackle one fundamental aspect of this goal — how can we assign different computational “tasklets”, corresponding to a micro-batch and a subset of layers, to a collection of geo-distributed devices connected via heterogeneous, slow networks?
Revised: In this paper, we tackle one fundamental aspect of this goal — how can we assign different computational “tasklets”, corresponding to a micro-batch and a subset of layers, to a collection of geo-distributed devices connected via heterogeneous, slow networks?

Original: This is not an easy task — even for fast and homogeneous data center networks, such assignments are still an open ongoing research challenge [22, 23, 24, 25, 26].
Revised: This is not an easy task — even for fast and homogeneous data center networks, such assignments are still an open ongoing research challenge [22, 23, 24, 25, 26].

Original: For the heterogeneous setting, it becomes even more challenging as the size of the search space increases dramatically.
Revised: For the heterogeneous setting, it becomes even more challenging as the size of the search space increases dramatically.

Original: In the homogeneous setting, the homogeneity of the edges in the communication graph reduces the search space into many equivalent classes representing allocation strategies with the same communication costs, enabling efficient polynomial runtime algorithms [23, 24, 22, 25, 26]; however, in the heterogeneous setting, one has to consider potentially exponentially many more distinct allocation strategies — as we will see later, because of the heterogeneity of the communication matrix, even the sub-problem of finding the best pipeline parallelism strategy equates to a hard open loop travelling salesman problem [27].
Revised: In the homogeneous setting, the homogeneity of the edges in the communication graph reduces the search space into many equivalent classes representing allocation strategies with the same communication costs, enabling efficient polynomial runtime algorithms [23, 24, 22, 25, 26]; however, in the heterogeneous setting, one has to consider potentially exponentially many more distinct allocation strategies — as we will see later, because of the heterogeneity of the communication matrix, even the sub-problem of finding the best pipeline parallelism strategy equates to a hard open loop travelling salesman problem [27].

Original: In this paper, we focus on this challenging scheduling problem of decentralized training of FMs over slow, heterogeneous networks, and make the following contributions:
Revised: In this paper, we focus on this challenging scheduling problem of decentralized training of FMs over slow, heterogeneous networks, and make the following contributions:

Original: • We study the problem of allocating distributed training jobs over a group of decentralized GPU devices connected via a slow heterogeneous network.
Revised: • We study the problem of allocating distributed training jobs over a group of decentralized GPU devices connected via a slow heterogeneous network.

Original: More specifically: – To capture the complex communication cost for training FMs, we propose a natural, but novel, formulation involving decomposing the cost model into two levels: the first level is a balanced
Revised: More specifically: – To capture the complex communication cost for training FMs, we propose a natural, but novel, formulation involving decomposing the cost model into two levels: the first level is a balanced

Original: graph partitioning problem corresponding to the communication cost of data
Revised: graph partitioning problem corresponding to the communication cost of data

Original: parallelism, whereas the second level is a joint graph matching and traveling salesman problem corresponding to the communication cost of pipeline parallelism.
Revised: parallelism, whereas the second level is a joint graph matching and traveling salesman problem corresponding to the communication cost of pipeline parallelism.

Original: – We propose a novel scheduling algorithm to search for the optimal allocation strategy given our cost model.
Revised: – We propose a novel scheduling algorithm to search for the optimal allocation strategy given our cost model.

Original: Developing a direct solution to this optimization problem is hard; thus, we propose an efficient evolutionary algorithm based on a collection of novel heuristics, going beyond the traditional heuristics used in standard graph partitioning methods [28].
Revised: Developing a direct solution to this optimization problem is hard; thus, we propose an efficient evolutionary algorithm based on a collection of novel heuristics, going beyond the traditional heuristics used in standard graph partitioning methods [28].

Original: • We carefully designed and implemented a collection of system optimizations to hide communication within the computation to further reduce the impact of slow connections.•
Revised: • We carefully designed and implemented a collection of system optimizations to hide communication within the computation to further reduce the impact of slow connections.•

Original: We conduct extensive experiments that represent different scenarios of collaborative decentralized learning, simulated by using network measurements from different geographical regions of AWS.
Revised: We conduct extensive experiments that represent different scenarios of collaborative decentralized learning, simulated by using network measurements from different geographical regions of AWS.

Original: In the worldwide setting with 64 GPUs across 8 regions (Oregon, Virginia, Ohio, Tokyo, Seoul, London, Frankfurt, Ireland), we show that our system is 3.8-4.8 × faster, in end-to-end runtime, than the state-of-the-art systems, for training GPT3-1.3B, without any difference in what is computed or convergence dynamics .
Revised: In the worldwide setting with 64 GPUs across 8 regions (Oregon, Virginia, Ohio, Tokyo, Seoul, London, Frankfurt, Ireland), we show that our system is 3.8-4.8 × faster, in end-to-end runtime, than the state-of-the-art systems, for training GPT3-1.3B, without any difference in what is computed or convergence dynamics .

Original: In addition, we also provide careful ablation studies to show the individual effectiveness of the scheduler and system optimizations.
Revised: In addition, we also provide careful ablation studies to show the individual effectiveness of the scheduler and system optimizations.

Original: • We shed light on the potential of decentralized learning — our prototype in the global heterogeneous setting is only 1.7-3.5 × slower than Megatron/Deepspeed in data centers even though its network can be 100 × slower .
Revised: • We shed light on the potential of decentralized learning — our prototype in the global heterogeneous setting is only 1.7-3.5 × slower than Megatron/Deepspeed in data centers even though its network can be 100 × slower .

Original: We hope this paper can inspire future explorations of decentralized learning for FMs, over geo-distributed servers, desktops, laptops, or even mobile devices.
Revised: We hope this paper can inspire future explorations of decentralized learning for FMs, over geo-distributed servers, desktops, laptops, or even mobile devices.

Original: Limitations and Moving Forward.
Revised: Limitations and Moving Forward.

Original: In this paper, we tackle one foundational aspect of decentralized learning but leave as future work many problems that are important for a practical system.
Revised: In this paper, we tackle one foundational aspect of decentralized learning but leave as future work many problems that are important for a practical system.

Original: We assume that communication between devices is relatively stable for a reasonable amount of time and that all devices are always online without failure or eviction.
Revised: We assume that communication between devices is relatively stable for a reasonable amount of time and that all devices are always online without failure or eviction.

Original: Note that we also do not train a full system to full convergence, instead running partial training to confirm intermediate result equivalence across regimes.
Revised: Note that we also do not train a full system to full convergence, instead running partial training to confirm intermediate result equivalence across regimes.

Original: Scheduling over a dynamic, heterogeneous environment and providing fault tolerance, potentially with checkpointing, while training to convergence are directions for future exploration.
Revised: Scheduling over a dynamic, heterogeneous environment and providing fault tolerance, potentially with checkpointing, while training to convergence are directions for future exploration.

Original:  Decentralized
Revised:  Decentralized

Original: Training of Foundation Models: Problem Formulation
Revised: Training of Foundation Models: Problem Formulation

Original: T r a n s f o r m e r
Revised: T r a n s f o r m e r

Original: B l o ck
Revised: B l o ck

Original: T r a n s f o r m e r
Revised: T r a n s f o r m e r

Original: B l o ck
Revised: B l o ck

Original: T r a n s f o r m e r
Revised: T r a n s f o r m e r

Original: B l o ck
Revised: B l o ck

Original: T r a n s f o r m e r
Revised: T r a n s f o r m e r

Original: B l o ck
Revised: B l o ck

Original: T r a n s f o r m e r
Revised: T r a n s f o r m e r

Original: B l o ck
Revised: B l o ck

Original: T r a n s f o r m e r
Revised: T r a n s f o r m e r

Original: B l o ck
Revised: B l o ck

Original: Forward Activation
Revised: Forward Activation

Original: Backward Gradient
Revised: Backward Gradient

Original: Gradient
Revised: Gradient

Original: Micro-batch Micro-batch
Revised: Micro-batch Micro-batch

Original: D ev i ce D ev i ce
Revised: D ev i ce D ev i ce

Original: D ev i ce D ev i ce
Revised: D ev i ce D ev i ce

Original: We first introduce concepts, technical terms, and the procedure of decentralized training.
Revised: We first introduce concepts, technical terms, and the procedure of decentralized training.

Original: Then we formally define the scheduling problem this paper tackles.
Revised: Then we formally define the scheduling problem this paper tackles.

Original: Decentralized setting.
Revised: Decentralized setting.

Original: We assume a group of devices (GPUs) participating in collaborative training of a foundation model.
Revised: We assume a group of devices (GPUs) participating in collaborative training of a foundation model.

Original: Each pair of devices has a connection with potentially different delay and bandwidth.
Revised: Each pair of devices has a connection with potentially different delay and bandwidth.

Original: These devices can be geodistributed, as illustrated in Figure 1, with vastly different pairwise communication bandwidth and latency.
Revised: These devices can be geodistributed, as illustrated in Figure 1, with vastly different pairwise communication bandwidth and latency.

Original: In decentralized training, all layers of a model are split into multiple stages , where each device handles a consecutive sequence of layers, e.g., several transformer blocks [29].
Revised: In decentralized training, all layers of a model are split into multiple stages , where each device handles a consecutive sequence of layers, e.g., several transformer blocks [29].

Original: In addition, since the input for foundation model pre-training is huge, e.g., a few millions of tokens, it is also split into multiple micro-batches that can be handled in parallel.
Revised: In addition, since the input for foundation model pre-training is huge, e.g., a few millions of tokens, it is also split into multiple micro-batches that can be handled in parallel.

Original: Problem definition.
Revised: Problem definition.

Original: We define tasklets as a collection of computational tasks in foundation model training — Tasklet t i,j is the forward and backward computation for a stage j with a micro-batch i of training data in a training iteration.
Revised: We define tasklets as a collection of computational tasks in foundation model training — Tasklet t i,j is the forward and backward computation for a stage j with a micro-batch i of training data in a training iteration.

Original: We aim to design an effective scheduler to assign each tasklet to a particular device so that the training throughput is maximized in decentralized training.
Revised: We aim to design an effective scheduler to assign each tasklet to a particular device so that the training throughput is maximized in decentralized training.

Original: Parallelism.
Revised: Parallelism.

Original: The above setting involves two forms of parallelism, pipeline and data .
Revised: The above setting involves two forms of parallelism, pipeline and data .

Original: In pipeline parallelism, the compute in multiple stages is parallelized — each device handles activation or gradient computation for different micro-batches in parallel and the results can be communicated or passed to subsequent stages.
Revised: In pipeline parallelism, the compute in multiple stages is parallelized — each device handles activation or gradient computation for different micro-batches in parallel and the results can be communicated or passed to subsequent stages.

Original: Data parallelism means that devices compute the gradient for different micro-batches independently, but need to synchronize these gradients through communication.
Revised: Data parallelism means that devices compute the gradient for different micro-batches independently, but need to synchronize these gradients through communication.

Original: In a decentralized environment, the training procedure is communication-bounded .
Revised: In a decentralized environment, the training procedure is communication-bounded .

Original: The scheduling problem is to accelerate the communication procedure by allocating tasklets that require high communication volumes between them to devices with faster connections.
Revised: The scheduling problem is to accelerate the communication procedure by allocating tasklets that require high communication volumes between them to devices with faster connections.

Original: [Figure]
Revised: [Figure]

Original: Formalization of the scheduling problem.
Revised: Formalization of the scheduling problem.

Original: Formally, our scheduling problem is as follows.
Revised: Formally, our scheduling problem is as follows.

Original: • Let D = { d 1 . . .
Revised: • Let D = { d 1 . . .

Original: d N } be a set of N devices; A ∈ R N × N + and B ∈ R N × N + be the communication matrix between these devices describing the delay and bandwidth respectively, where the delay and bandwidth between device d and d ′ is α d,d ′ and β d,d ′ .
Revised: d N } be a set of N devices; A ∈ R N × N + and B ∈ R N × N + be the communication matrix between these devices describing the delay and bandwidth respectively, where the delay and bandwidth between device d and d ′ is α d,d ′ and β d,d ′ .

Original: • Given the communication matrix A and B , we construct a communication graph G
Revised: • Given the communication matrix A and B , we construct a communication graph G

Original: (Figure 2(a))
Revised: (Figure 2(a))

Original: — each device corresponds to a node in G and each edge between d and d ′ is labeled with the average latency and bandwidth between d and d ′ :
Revised: — each device corresponds to a node in G and each edge between d and d ′ is labeled with the average latency and bandwidth between d and d ′ :

Original: (( α d,d ′ + α d ′ ,d ) / 2 , ( β d,d ′ + β d ′ ,d ) / 2) .
Revised: (( α d,d ′ + α d ′ ,d ) / 2 , ( β d,d ′ + β d ′ ,d ) / 2) .

Original: Even though A and B are asymmetric (i
Revised: Even though A and B are asymmetric (i

Original: [Figure]
Revised: [Figure]

Original: e., upload and download speeds might be different), the communication graph G is symmetric because in our workloads all communications between two devices happen to involve the same amount of upload and download.
Revised: e., upload and download speeds might be different), the communication graph G is symmetric because in our workloads all communications between two devices happen to involve the same amount of upload and download.

Original: • The number of stages that a micro-batch needs to go through is D PP (noted as pipeline parallel degree); the number of batch partition that needs to run model gradient synchronization is D DP (noted as data parallel degree); we have D DP × D PP = N , i.e., the total number of devices.
Revised: • The number of stages that a micro-batch needs to go through is D PP (noted as pipeline parallel degree); the number of batch partition that needs to run model gradient synchronization is D DP (noted as data parallel degree); we have D DP × D PP = N , i.e., the total number of devices.

Original: • c PP (resp.
Revised: • c PP (resp.

Original: c DP ) represent the number of bytes of activations for a micro-batch (resp.
Revised: c DP ) represent the number of bytes of activations for a micro-batch (resp.

Original: parameters/gradients for a stage) communicated in pipeline parallelism (resp.
Revised: parameters/gradients for a stage) communicated in pipeline parallelism (resp.

Original: data parallelism).
Revised: data parallelism).

Original: • We denote a training tasklet as t i,j , where i ∈ { 1 , ..., D DP } and j ∈ { 1 , ..., D PP } , each of which corresponds to one specific micro-batch i and pipeline stage j . •
Revised: • We denote a training tasklet as t i,j , where i ∈ { 1 , ..., D DP } and j ∈ { 1 , ..., D PP } , each of which corresponds to one specific micro-batch i and pipeline stage j . •

Original: An assignment strategy σ ∈ D D DP × D PP assigns, for each tasklet t i,j , a device σ i,j ∈ D , which means that device σ i,j runs the training tasklet t i,j .
Revised: An assignment strategy σ ∈ D D DP × D PP assigns, for each tasklet t i,j , a device σ i,j ∈ D , which means that device σ i,j runs the training tasklet t i,j .

Original: A valid assignment needs to be unique, i.e., ∀
Revised: A valid assignment needs to be unique, i.e., ∀

Original: ( i, j ) ̸
Revised: ( i, j ) ̸

Original: = ( i ′ , j ′ ) :
Revised: = ( i ′ , j ′ ) :

Original: σ i,j ̸ = σ i ′ ,j ′ .
Revised: σ i,j ̸ = σ i ′ ,j ′ .

Original: We use Σ to denote the set of all valid assignments.
Revised: We use Σ to denote the set of all valid assignments.

Original: • An optimal assignment strategy is an assignment σ that minimizes the communication cost
Revised: • An optimal assignment strategy is an assignment σ that minimizes the communication cost

Original: [Equation]
Revised: [Equation]

Original: Challenges and Goals.
Revised: Challenges and Goals.

Original: Our goal is to find the optimal assignment strategy, which involves two challenges: (1) How to effectively model the communication cost C OMM -C OST ( σ ) for a given assignment σ under a heterogeneous network environment?
Revised: Our goal is to find the optimal assignment strategy, which involves two challenges: (1) How to effectively model the communication cost C OMM -C OST ( σ ) for a given assignment σ under a heterogeneous network environment?

Original: and (2) How to effectively search for the optimal assignment strategy that minimizes such a cost?
Revised: and (2) How to effectively search for the optimal assignment strategy that minimizes such a cost?

Original: We tackle these two questions in Section 3.
Revised: We tackle these two questions in Section 3.

Original:  Scheduling in Heterogeneous Environments
Revised:  Scheduling in Heterogeneous Environments

Original: Scheduling in the heterogeneous setting is a challenging task, as the size of the search space increases dramatically compared to that of the homogeneous case.
Revised: Scheduling in the heterogeneous setting is a challenging task, as the size of the search space increases dramatically compared to that of the homogeneous case.

Original: In the homogeneous data-center case, the network delay can be usually ignored (e.g., A = 0 ) and the bandwidth B are assumed to be formed by just a few constants — e.g., the communication bandwidths between different machines on the same rack are assumed to be same [23, 24, 22, 22, 26].
Revised: In the homogeneous data-center case, the network delay can be usually ignored (e.g., A = 0 ) and the bandwidth B are assumed to be formed by just a few constants — e.g., the communication bandwidths between different machines on the same rack are assumed to be same [23, 24, 22, 22, 26].

Original: This significantly constrains the search space — one can ignore the influence of communication given uniform connections [23, 24, 22], or organize the device with a hierarchical structure [22, 26], making the scheduling problem solvable in polynomial time in terms of the number of machines.
Revised: This significantly constrains the search space — one can ignore the influence of communication given uniform connections [23, 24, 22], or organize the device with a hierarchical structure [22, 26], making the scheduling problem solvable in polynomial time in terms of the number of machines.

Original: In contrast, in the fully heterogeneous scenario the communication matrix A and B consists of distinct values, which can make the search space grows exponentially.
Revised: In contrast, in the fully heterogeneous scenario the communication matrix A and B consists of distinct values, which can make the search space grows exponentially.

Original: In this section, we describe our scheduler that searches for an optimal strategy in the complex search space.
Revised: In this section, we describe our scheduler that searches for an optimal strategy in the complex search space.

Original: 3.1 Overview of the scheduler
Revised: 3.1 Overview of the scheduler

Original: We carefully design a bi-level scheduling algorithm based on extended balanced graph partition problem (see Figure 2), and solve this problem by an evolutionary algorithm with a carefully designed local search strategy.
Revised: We carefully design a bi-level scheduling algorithm based on extended balanced graph partition problem (see Figure 2), and solve this problem by an evolutionary algorithm with a carefully designed local search strategy.

Original: Given an assignment strategy σ = { σ i,j } for all tasklets { t i,j } , we first model its communication cost.
Revised: Given an assignment strategy σ = { σ i,j } for all tasklets { t i,j } , we first model its communication cost.

Original: During the training of FMs, the communication costs come from two different sources: (1) Data parallel : All devices that are assigned with the tasklets dealing with the same stage j (handling the same layers) of different micro-batches need to communicate within themselves to exchange gradients of these layers.
Revised: During the training of FMs, the communication costs come from two different sources: (1) Data parallel : All devices that are assigned with the tasklets dealing with the same stage j (handling the same layers) of different micro-batches need to communicate within themselves to exchange gradients of these layers.

Original: For layer j , we call these devices its data parallel group : C j = { σ i,j | ∀ i ∈
Revised: For layer j , we call these devices its data parallel group : C j = { σ i,j | ∀ i ∈

Original: [ D DP ] } .
Revised: [ D DP ] } .

Original: We can implement the communication using different primitives, e.g., AllReduce [30], ScatterGatter [31], or other decentralized average protocols [16].
Revised: We can implement the communication using different primitives, e.g., AllReduce [30], ScatterGatter [31], or other decentralized average protocols [16].

Original: (2) Pipeline parallel : All devices that are assigned with the tasklets dealing with the same micro-batch i of different stages need to form a pipeline, communicating within themselves to exchange activations and backward gradients.
Revised: (2) Pipeline parallel : All devices that are assigned with the tasklets dealing with the same micro-batch i of different stages need to form a pipeline, communicating within themselves to exchange activations and backward gradients.

Original: For micro-batch i , these devices are P i = { σ i,j | ∀
Revised: For micro-batch i , these devices are P i = { σ i,j | ∀

Original: j ∈
Revised: j ∈

Original: [ D PP ] } .
Revised: [ D PP ] } .

Original: Because these devices need to form a linear pipeline, any permutation over P i corresponds to one strategy of how these machines can conduct pipeline parallelism within them.
Revised: Because these devices need to form a linear pipeline, any permutation over P i corresponds to one strategy of how these machines can conduct pipeline parallelism within them.

Original: Scheduling Problem.
Revised: Scheduling Problem.

Original: The goal of our scheduler is to minimize both costs.
Revised: The goal of our scheduler is to minimize both costs.

Original: One design decision that we made is to decompose this complex optimization problem into two levels.
Revised: One design decision that we made is to decompose this complex optimization problem into two levels.

Original: At the first level, we consider the best way of forming C j ’s, incurring data parallel communication costs within them.
Revised: At the first level, we consider the best way of forming C j ’s, incurring data parallel communication costs within them.

Original: At the second level, we consider the cost of pipeline parallelism given an layout from the first level:
Revised: At the second level, we consider the cost of pipeline parallelism given an layout from the first level:

Original: [Equation]
Revised: [Equation]

Original: where computing P IPELINE P-C OST ( C
Revised: where computing P IPELINE P-C OST ( C

Original: [Equation]
Revised: [Equation]

Original: C D PP ) involves finding the optimal pipeline structure.
Revised: C D PP ) involves finding the optimal pipeline structure.

Original: In Section 3.2 and Section 3.3, we provide details on C OMM -C OST ( C 1 ... C D PP ) .
Revised: In Section 3.2 and Section 3.3, we provide details on C OMM -C OST ( C 1 ... C D PP ) .

Original: Notice that this modified objective makes our problem different from the textbook graph partition problem; thus, we need a carefully designed evolutionary algorithm for finding such a solution introduced in Section 3.4.
Revised: Notice that this modified objective makes our problem different from the textbook graph partition problem; thus, we need a carefully designed evolutionary algorithm for finding such a solution introduced in Section 3.4.

Original: 3.2 Modelling data parallel communication cost
Revised: 3.2 Modelling data parallel communication cost

Original: Given the communication graph G forming data parallel groups C 1 ... C D PP corresponds to a partition of G — In Figure 2(b), different colors correspond to devices in the same C j .
Revised: Given the communication graph G forming data parallel groups C 1 ... C D PP corresponds to a partition of G — In Figure 2(b), different colors correspond to devices in the same C j .

Original: The data parallel cost within C j only relies on all communication channels (edges in the communication graph) connecting devices in C j .
Revised: The data parallel cost within C j only relies on all communication channels (edges in the communication graph) connecting devices in C j .

Original: If we assume a colocated sharded parameter server [31] implementation for communicating within each C j , and recall that c DP represents the total amount of data (in bytes) that needs to be exchanged during gradient aggregation — each device in C j needs to manage c DP /D DP bytes of parameter shard.
Revised: If we assume a colocated sharded parameter server [31] implementation for communicating within each C j , and recall that c DP represents the total amount of data (in bytes) that needs to be exchanged during gradient aggregation — each device in C j needs to manage c DP /D DP bytes of parameter shard.

Original: Once the gradient is ready, each device needs to send each of its local shards to the corresponding device; next, each device can aggregate the gradients it receives from all other devices in C j ; and finally, each device will send the aggregated gradient shard to all other devices.
Revised: Once the gradient is ready, each device needs to send each of its local shards to the corresponding device; next, each device can aggregate the gradients it receives from all other devices in C j ; and finally, each device will send the aggregated gradient shard to all other devices.

Original: Therefore, we can model the data parallel cost for C j as follows:
Revised: Therefore, we can model the data parallel cost for C j as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, the total cost is bounded by the slowest device ( max d ∈ C j ), which needs to exchange data with all other machines ( (cid:80) d ′ ∈ C j −{ d } ).
Revised: Here, the total cost is bounded by the slowest device ( max d ∈ C j ), which needs to exchange data with all other machines ( (cid:80) d ′ ∈ C j −{ d } ).

Original: Because the communication of these different data parallel groups
Revised: Because the communication of these different data parallel groups

Original: C 1 ...
Revised: C 1 ...

Original: C D PP can be conducted in parallel and we are only bounded by the slowest data parallel group.
Revised: C D PP can be conducted in parallel and we are only bounded by the slowest data parallel group.

Original: This allows us to model the total communication cost for data parallelism as:
Revised: This allows us to model the total communication cost for data parallelism as:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: 3.3 Modeling pipeline parallel communication cost
Revised: 3.3 Modeling pipeline parallel communication cost

Original: Given C 1 ... C D PP , to model the communication cost of pipeline parallelism, we need to consider two factors: (1) each permutation π of { C 1 ... C D PP } corresponds to a specific pipeline strategy — devices in C π j and devices in C π j +1 communicates to exchange activations (during forward pass) and gradients on activations (during backward pass); and (2) devices in C π j and devices in C π j +1 need to be “matched” — only devices that are dealing with the same micro-batch needs to communicate.
Revised: Given C 1 ... C D PP , to model the communication cost of pipeline parallelism, we need to consider two factors: (1) each permutation π of { C 1 ... C D PP } corresponds to a specific pipeline strategy — devices in C π j and devices in C π j +1 communicates to exchange activations (during forward pass) and gradients on activations (during backward pass); and (2) devices in C π j and devices in C π j +1 need to be “matched” — only devices that are dealing with the same micro-batch needs to communicate.

Original: This makes modeling the cost of pipeline parallel communication more complex.
Revised: This makes modeling the cost of pipeline parallel communication more complex.

Original: To model the cost of pipeline parallel communication, we first consider the best possible way that devices in C j and C j ′ can be matched.
Revised: To model the cost of pipeline parallel communication, we first consider the best possible way that devices in C j and C j ′ can be matched.

Original: We do this by creating a coarsened communication graph (Figure 2(c)).
Revised: We do this by creating a coarsened communication graph (Figure 2(c)).

Original: A coarsened communication graph (cid:98) G C 1 ... C D PP is a fully connected graph, and each partition C j in the original communication graph G corresponds to a node in (cid:98) G C 1 ... C D PP .
Revised: A coarsened communication graph (cid:98) G C 1 ... C D PP is a fully connected graph, and each partition C j in the original communication graph G corresponds to a node in (cid:98) G C 1 ... C D PP .

Original: In the coarsened graph (cid:98) G , the weight on an edge between C j and C j ′ corresponds to the following — if C j and C j ′ need to communicate in a pipeline, what is the communicate cost of the optimal matching strategy between devices in C j and devices in C j ′ ?
Revised: In the coarsened graph (cid:98) G , the weight on an edge between C j and C j ′ corresponds to the following — if C j and C j ′ need to communicate in a pipeline, what is the communicate cost of the optimal matching strategy between devices in C j and devices in C j ′ ?

Original: Recall that c PP represents the amount of data between two devices for pipeline parallel communication, we can model this cost by
Revised: Recall that c PP represents the amount of data between two devices for pipeline parallel communication, we can model this cost by

Original: [Equation]
Revised: [Equation]

Original: where M is a perfect matching between C j and C j ′ — ( d, d ′ )
Revised: where M is a perfect matching between C j and C j ′ — ( d, d ′ )

Original: ∈ M means that device d ∈ C
Revised: ∈ M means that device d ∈ C

Original: j will communicate with device d ′ ∈ C j ′ (i.e., they deal with the same micro-batch).
Revised: j will communicate with device d ′ ∈ C j ′ (i.e., they deal with the same micro-batch).

Original: Computing this value is similar to the classical minimal sum weight perfect matching problem (MinSumWPM) in bipartite graphs [32], with the only difference being that we compute the max instead of the sum .
Revised: Computing this value is similar to the classical minimal sum weight perfect matching problem (MinSumWPM) in bipartite graphs [32], with the only difference being that we compute the max instead of the sum .

Original: As we will show in the supplementary material, similar to MinSumWPM, Eq 3 can also be solved in PTIME.
Revised: As we will show in the supplementary material, similar to MinSumWPM, Eq 3 can also be solved in PTIME.

Original: The coarsened communication graph captures the pipeline parallel communication cost between two groups of devices, assuming they become neighbors in the pipeline.
Revised: The coarsened communication graph captures the pipeline parallel communication cost between two groups of devices, assuming they become neighbors in the pipeline.

Original: Given this, we need to find an optimal permutation of C 1 ... C D PP , corresponds to the structure of the pipeline.
Revised: Given this, we need to find an optimal permutation of C 1 ... C D PP , corresponds to the structure of the pipeline.

Original: This becomes the open-loop traveling salesman problem [27] over this condensed graph (Figure 2(e)).
Revised: This becomes the open-loop traveling salesman problem [27] over this condensed graph (Figure 2(e)).

Original: Formally, we have the following definition of the pipeline parallel cost:
Revised: Formally, we have the following definition of the pipeline parallel cost:

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) G C 

Revised: where (cid:98) G C

Original: C D PP is the coarsened graph defined above.
Revised: C D PP is the coarsened graph defined above.

Original: 3.4 Searching via hybrid genetic algorithm
Revised: 3.4 Searching via hybrid genetic algorithm

Original: The scheduling problem solves the optimization problem in Eq 1, which corresponds to a balanced graph partition problem with a complex objective corresponding to the communication cost.
Revised: The scheduling problem solves the optimization problem in Eq 1, which corresponds to a balanced graph partition problem with a complex objective corresponding to the communication cost.

Original: Balanced graph partition problem is a challenging NP-hard problem [33].
Revised: Balanced graph partition problem is a challenging NP-hard problem [33].

Original: Over the years, researchers have been tackling this problem via different ways [34, 35, 36].
Revised: Over the years, researchers have been tackling this problem via different ways [34, 35, 36].

Original: We follow the line of research that uses hybrid genetic algorithm [37, 28] since it provides us the flexibility in dealing with complex objective.
Revised: We follow the line of research that uses hybrid genetic algorithm [37, 28] since it provides us the flexibility in dealing with complex objective.

Original: Hybrid Genetic Algorithm.
Revised: Hybrid Genetic Algorithm.

Original: A hybrid genetic algorithm for balanced graph partition usually follows a structure as as follows.
Revised: A hybrid genetic algorithm for balanced graph partition usually follows a structure as as follows.

Original: The input is a set of candidate balanced graph partitions which serves as the initial population.
Revised: The input is a set of candidate balanced graph partitions which serves as the initial population.

Original: The algorithm generates the next generation as follows.
Revised: The algorithm generates the next generation as follows.

Original: It first generates a new “offspring” o given two randomly selected “parents” p 1 and p 2 .
Revised: It first generates a new “offspring” o given two randomly selected “parents” p 1 and p 2 .

Original: One popular way is to randomly swap some nodes between these two parents (we follow [28]).
Revised: One popular way is to randomly swap some nodes between these two parents (we follow [28]).

Original: Given this offspring o , we then conduct local search starting at o to find a new balanced partitioning strategy
Revised: Given this offspring o , we then conduct local search starting at o to find a new balanced partitioning strategy

Original: o ∗ that leads to better cost.
Revised: o ∗ that leads to better cost.

Original: We then add o ∗ to the population and remove the worst partition candidate in the population if o ∗ has a better cost.
Revised: We then add o ∗ to the population and remove the worst partition candidate in the population if o ∗ has a better cost.

Original: As suggested by [37], the combination of heuristic-based local search algorithms and genetic algorithm can accelerate convergence by striking the balance between local and global optimum.
Revised: As suggested by [37], the combination of heuristic-based local search algorithms and genetic algorithm can accelerate convergence by striking the balance between local and global optimum.

Original: Existing Local Search Strategy.
Revised: Existing Local Search Strategy.

Original: The key in designing this algorithm is to come up with a good local search strategy.
Revised: The key in designing this algorithm is to come up with a good local search strategy.

Original: For traditional graph partitioning task, one popular choice is to use the Kernighan-Lin
Revised: For traditional graph partitioning task, one popular choice is to use the Kernighan-Lin

Original: Algorithm [38].
Revised: Algorithm [38].

Original: Which, at each iteration, tries to find a pair of nodes: d in partition C j and d ′ in partition C j ′ , to swap.
Revised: Which, at each iteration, tries to find a pair of nodes: d in partition C j and d ′ in partition C j ′ , to swap.

Original: To find such a pair to swap, it uses the following “gain” function:
Revised: To find such a pair to swap, it uses the following “gain” function:

Original: [Equation]
Revised: [Equation]

Original: where w i,j corresponds to the weight between node i and j in the graph.
Revised: where w i,j corresponds to the weight between node i and j in the graph.

Original: However, directly applying this local search strategy, as we will also show in the experiment (Section 4) does not work well.
Revised: However, directly applying this local search strategy, as we will also show in the experiment (Section 4) does not work well.

Original: Greedily following GAIN KL does not decrease the communication cost of foundation model training.
Revised: Greedily following GAIN KL does not decrease the communication cost of foundation model training.

Original: Therefore, we have to design a new local search strategy tailored to our cost model.
Revised: Therefore, we have to design a new local search strategy tailored to our cost model.

Original: Improving Local Search Strategy.
Revised: Improving Local Search Strategy.

Original: Our local search strategy is inspired by two observations:
Revised: Our local search strategy is inspired by two observations:

Original: Removing the device d 1 with a fast connection (
Revised: Removing the device d 1 with a fast connection (

Original: say
Revised: say

Original: with d 2 ) within
Revised: with d 2 ) within

Original: partition
Revised: partition

Original: C j will not tend to change the data parallel cost within C j , since it is only bounded by the slowest connections.
Revised: C j will not tend to change the data parallel cost within C j , since it is only bounded by the slowest connections.

Original: Once d 1 is moved to C j ′ , highly likely the pipeline parallel matching between C j and C j ′ will consist of the link d 1 ↔ d 2 , since it is a fast connection.
Revised: Once d 1 is moved to C j ′ , highly likely the pipeline parallel matching between C j and C j ′ will consist of the link d 1 ↔ d 2 , since it is a fast connection.

Original: Therefore, in our local search strategy we only consider the fastest connection within C j : d 1 ↔ dand the fastest connection within C j ′ :
Revised: Therefore, in our local search strategy we only consider the fastest connection within C j : d 1 ↔ dand the fastest connection within C j ′ :

Original: d ′ 1 ↔ d ′ 2 and generate four swap candidates: d 1 ↔ d ′ 1 , d 1 ↔ d ′ 2 , d 2 ↔ d ′ 1 , d 2 ↔ d ′ 2 .
Revised: d ′ 1 ↔ d ′ 2 and generate four swap candidates: d 1 ↔ d ′ 1 , d 1 ↔ d ′ 2 , d 2 ↔ d ′ 1 , d 2 ↔ d ′ 2 .

Original: We use the following gain function (take d 1 ↔ d ′ 1 as an example):
Revised: We use the following gain function (take d 1 ↔ d ′ 1 as an example):

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: w d
Revised: w d

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: measures the expected pipeline parallel cost of connecting d 1 with other devices in C j ′ before the swap , and w d 1 ,d 2 is the cost of connecting d 1 with other devices in C j ′ after the swap , assuming this fast link d 1 ↔ d 2 will now be used for pipeline parallelism.
Revised: measures the expected pipeline parallel cost of connecting d 1 with other devices in C j ′ before the swap , and w d 1 ,d 2 is the cost of connecting d 1 with other devices in C j ′ after the swap , assuming this fast link d 1 ↔ d 2 will now be used for pipeline parallelism.

Original: Just like how Kernighan-Lin Algorithm [38] can be extended to a circular version [28] to swap multiple nodes beyond a pair, we can also extend our method into a circular one, following procedure as circular KL with our new gain function.
Revised: Just like how Kernighan-Lin Algorithm [38] can be extended to a circular version [28] to swap multiple nodes beyond a pair, we can also extend our method into a circular one, following procedure as circular KL with our new gain function.

Original: 3.5 Other System Optimizations
Revised: 3.5 Other System Optimizations

Original: We also have some system optimizations to further improve the performance.
Revised: We also have some system optimizations to further improve the performance.

Original: The most important optimization involves pipelining of communications and computations.
Revised: The most important optimization involves pipelining of communications and computations.

Original: We divide each stage in the pipeline into three slots: a receiving slot, a computation slot, and a sending slot.
Revised: We divide each stage in the pipeline into three slots: a receiving slot, a computation slot, and a sending slot.

Original: The receiving slot of stage j needs to build connections to receive activations from the stage j − 1 in forward propagation and to receive gradients of activations from stage j +1 .
Revised: The receiving slot of stage j needs to build connections to receive activations from the stage j − 1 in forward propagation and to receive gradients of activations from stage j +1 .

Original: The computation slot handles the computation in forward and backward propagation.
Revised: The computation slot handles the computation in forward and backward propagation.

Original: Symmetric to the receiving slot, the sending slot of stage j needs to build connections to send activations to stage j + 1 in the forward propagation and send gradients of activations to stage j − 1 in the backward propagations.
Revised: Symmetric to the receiving slot, the sending slot of stage j needs to build connections to send activations to stage j + 1 in the forward propagation and send gradients of activations to stage j − 1 in the backward propagations.

Original: These three slots are assigned to three CUDA streams so that they will be further pipelined efficiently; as a result, communication will overlap with computation.
Revised: These three slots are assigned to three CUDA streams so that they will be further pipelined efficiently; as a result, communication will overlap with computation.

Original: In the decentralized scenario (communication bound), computation can be fully hidden inside the communication time.
Revised: In the decentralized scenario (communication bound), computation can be fully hidden inside the communication time.

Original:  Evaluation
Revised:  Evaluation

Original: We demonstrate that our system can speed up foundation model training in decentralized setting.
Revised: We demonstrate that our system can speed up foundation model training in decentralized setting.

Original: Specifically, (1) We show that our system is 4.8 × faster, in end-to-end runtime, than the state-of-theart systems (Megatron and Deepspeed) training GPT3-1.3B in world-wide geo-distributed setting.
Revised: Specifically, (1) We show that our system is 4.8 × faster, in end-to-end runtime, than the state-of-theart systems (Megatron and Deepspeed) training GPT3-1.3B in world-wide geo-distributed setting.

Original: Surprisingly, it is only 1 .
Revised: Surprisingly, it is only 1 .

Original: 7 − 2 .
Revised: 7 − 2 .

Original: 3 × slower than these systems in data centers.
Revised: 3 × slower than these systems in data centers.

Original: This indicates that we can bridge the gap between decentralized and data center training (up to 100 × slower networks) through scheduling and system optimization; (2) We demonstrate the necessity of our scheduler through an ablation study.
Revised: This indicates that we can bridge the gap between decentralized and data center training (up to 100 × slower networks) through scheduling and system optimization; (2) We demonstrate the necessity of our scheduler through an ablation study.

Original: We show that with the scheduler, our system is 2.7 × faster in world-wide geo-distributed setting.
Revised: We show that with the scheduler, our system is 2.7 × faster in world-wide geo-distributed setting.

Original: 4.1 Experimental Setup.
Revised: 4.1 Experimental Setup.

Original: To simulate the decentralized setting, we use 8 different AWS regions (Oregon, Virginia, Ohio, Tokyo,
Revised: To simulate the decentralized setting, we use 8 different AWS regions (Oregon, Virginia, Ohio, Tokyo,

Original: Seoul, London, Frankfurt, and Ireland) and measure the latency and bandwidth between these regions (we consider the bandwidth that we can realistically obtain using NCCL and UDP hole punching between these regions).
Revised: Seoul, London, Frankfurt, and Ireland) and measure the latency and bandwidth between these regions (we consider the bandwidth that we can realistically obtain using NCCL and UDP hole punching between these regions).

Original: Given these measurements, we use 64 Tesla V100 GPUs and control their pairwise communication latency and bandwidth for five different cases:
Revised: Given these measurements, we use 64 Tesla V100 GPUs and control their pairwise communication latency and bandwidth for five different cases:

Original: C a s e
Revised: C a s e

Original: D a t a
Revised: D a t a

Original: C e n t e r on D e m a nd
Revised: C e n t e r on D e m a nd

Original: C a s e
Revised: C a s e

Original: D a t a
Revised: D a t a

Original: C e n t e r
Revised: C e n t e r

Original: S po t I n s t a n ce s
Revised: S po t I n s t a n ce s

Original: C a s e
Revised: C a s e

Original: [Figure]
Revised: [Figure]

Original: M u l t i p l e D a t a
Revised: M u l t i p l e D a t a

Original: C e n t e rs
Revised: C e n t e rs

Original: C a s e
Revised: C a s e

Original: W o r l dw i d e
Revised: W o r l dw i d e

Original: D i s t
Revised: D i s t

Original: r
Revised: r

Original: i bu t e d
Revised: i bu t e d

Original: C a s e
Revised: C a s e

Original: R eg i on a l D
Revised: R eg i on a l D

Original: i s t
Revised: i s t

Original: r
Revised: r

Original: i bu t e d
Revised: i bu t e d

Original: [Figure]
Revised: [Figure]

Original: 6x
Revised: 6x

Original: [Figure]
Revised: [Figure]

Original: Case 1.
Revised: Case 1.

Original: Data center on demand.
Revised: Data center on demand.

Original: This is a standard setting that a user can obtain to train foundation models.
Revised: This is a standard setting that a user can obtain to train foundation models.

Original: we use
Revised: we use

Original: [Figure]
Revised: [Figure]

Original: s ti m a t e d
Revised: s ti m a t e d

Original: [Figure]
Revised: [Figure]

Original: s t
Revised: s t

Original: [Figure]
Revised: [Figure]

Original: Case 2.
Revised: Case 2.

Original: Data center spot instances.
Revised: Data center spot instances.

Original: Spot GPUs are cheaper in a data center, but can be located on different types of machine.
Revised: Spot GPUs are cheaper in a data center, but can be located on different types of machine.

Original: In this case, we rent 4 AWS p3.8xlarge nodes (each with 4 V100) and 32 p3.2xlarge nodes (each with 1 V100); the intra- p3.8xlarge node connection has a bandwidth of 100 Gbps, and the inter-node connection has a bandwidth of 10 Gbps.
Revised: In this case, we rent 4 AWS p3.8xlarge nodes (each with 4 V100) and 32 p3.2xlarge nodes (each with 1 V100); the intra- p3.8xlarge node connection has a bandwidth of 100 Gbps, and the inter-node connection has a bandwidth of 10 Gbps.

Original: We do not manually control latency and bandwidth in this case.
Revised: We do not manually control latency and bandwidth in this case.

Original: Case 3.
Revised: Case 3.

Original: Multiple Data Centers.
Revised: Multiple Data Centers.

Original: We consider two organizations, one in Ohio and another in Virginia, each organization contributes 32 V100 GPUs; within each organization, the bandwidth is 10 Gbps, and connections cross different campuses have a delay of 10 ms and bandwidth of 1.12 Gbps.
Revised: We consider two organizations, one in Ohio and another in Virginia, each organization contributes 32 V100 GPUs; within each organization, the bandwidth is 10 Gbps, and connections cross different campuses have a delay of 10 ms and bandwidth of 1.12 Gbps.

Original: Case 4.
Revised: Case 4.

Original: Regional geo-distributed.
Revised: Regional geo-distributed.

Original: We consider individual GPUs cross four different regions in US (California, Ohio, Oregon, and Virginia) ; within each region, the delay is 5 ms and bandwidth is
Revised: We consider individual GPUs cross four different regions in US (California, Ohio, Oregon, and Virginia) ; within each region, the delay is 5 ms and bandwidth is

Original: Gbps; cross different regions, the delay is 10 ∼ 70 ms and the bandwidth is 1 .
Revised: Gbps; cross different regions, the delay is 10 ∼ 70 ms and the bandwidth is 1 .

Original: ∼ 1 . 3 Gbps.
Revised: ∼ 1 . 3 Gbps.

Original: Case 5.
Revised: Case 5.

Original: World-wide geo-distributed.
Revised: World-wide geo-distributed.

Original: We consider individual GPUs cross eight different regions world-wide (Oregon, Virginia, Ohio, Tokyo, Seoul, London, Frankfurt, and Ireland); within each region, the delay is 5 ms and bandwidth is 2 Gbps; cross different regions, the delay is 10 ∼ 250 ms and the bandwidth is 0 .
Revised: We consider individual GPUs cross eight different regions world-wide (Oregon, Virginia, Ohio, Tokyo, Seoul, London, Frankfurt, and Ireland); within each region, the delay is 5 ms and bandwidth is 2 Gbps; cross different regions, the delay is 10 ∼ 250 ms and the bandwidth is 0 .

Original: 3 ∼ 1 . 3 Gbps.
Revised: 3 ∼ 1 . 3 Gbps.

Original: Metrics and Model Architecture.
Revised: Metrics and Model Architecture.

Original: Since we do not introduce any optimizations that might change the computation or convergence, we can compare all methods by its throughput, we can compare all systems by the total number of floating point operations per second (PFLOPS), which is inverse proportional to the runtime of each iteration (which we show in Appendix).
Revised: Since we do not introduce any optimizations that might change the computation or convergence, we can compare all methods by its throughput, we can compare all systems by the total number of floating point operations per second (PFLOPS), which is inverse proportional to the runtime of each iteration (which we show in Appendix).

Original: We use the standard
Revised: We use the standard

Original: GPT3-1.3B architecture [2], while also benchmarked different number of layers { 24 , 32 , 40 } , and batch sizes { 1024 , 2048 , 4096 } .
Revised: GPT3-1.3B architecture [2], while also benchmarked different number of layers { 24 , 32 , 40 } , and batch sizes { 1024 , 2048 , 4096 } .

Original: Tuning of Megatron and Deepspeed.
Revised: Tuning of Megatron and Deepspeed.

Original: We did a careful grid search of different parallelism settings and report the optimal results in each case—in Case 1, the optimal setting includes tensor model parallelism in Megatron and ZeRO-S3 in Deepspeed; in all other cases, the optimal settings are based on pipeline and data parallelism.
Revised: We did a careful grid search of different parallelism settings and report the optimal results in each case—in Case 1, the optimal setting includes tensor model parallelism in Megatron and ZeRO-S3 in Deepspeed; in all other cases, the optimal settings are based on pipeline and data parallelism.

Original: We discuss more details in Appendix.
Revised: We discuss more details in Appendix.

Original: 4.2 Results
Revised: 4.2 Results

Original: End-to-end Comparison.
Revised: End-to-end Comparison.

Original: Figure 3(c) shows the end-to-end comparison in terms of averaged
Revised: Figure 3(c) shows the end-to-end comparison in terms of averaged

Original: PFLOPS achieved across different settings and different batch sizes and number of layers.
Revised: PFLOPS achieved across different settings and different batch sizes and number of layers.

Original: In the world-wide geo-distributed cases, we achieve an 4 . 8 × speedup of Megatron, ( 3 . 6 × speedup of
Revised: In the world-wide geo-distributed cases, we achieve an 4 . 8 × speedup of Megatron, ( 3 . 6 × speedup of

Original: Deepspeed).
Revised: Deepspeed).

Original: While in all other cases, our system can be 1 . 2 − 2 .
Revised: While in all other cases, our system can be 1 . 2 − 2 .

Original: 5 × faster.
Revised: 5 × faster.

Original: If we compare our system in Case 5 (world-wide geo-distributed) and Megatron/Deepspeed in Case 1 (data center on demand), it is exciting to see that the performance slowdown caused by decentralization is only 1 . 7 − 3 .
Revised: If we compare our system in Case 5 (world-wide geo-distributed) and Megatron/Deepspeed in Case 1 (data center on demand), it is exciting to see that the performance slowdown caused by decentralization is only 1 . 7 − 3 .

Original: 5 × !
Revised: 5 × !

Original: This illustrates the great potential of decentralized training for foundation models.
Revised: This illustrates the great potential of decentralized training for foundation models.

Original: Additionally,
Revised: Additionally,

Original: Figure 3(c) illustrates another interesting behavior pattern.
Revised: Figure 3(c) illustrates another interesting behavior pattern.

Original: As increasing the batch size does not increases the communication cost of data parallelism and increasing # layers per device does not increases the communication cost of pipeline parallelism, with a larger batch size and a deeper model, the gap between centralized Megatron/Deepspeed and our decentralized system is even smaller.
Revised: As increasing the batch size does not increases the communication cost of data parallelism and increasing # layers per device does not increases the communication cost of pipeline parallelism, with a larger batch size and a deeper model, the gap between centralized Megatron/Deepspeed and our decentralized system is even smaller.

Original: Effectiveness of Scheduler.
Revised: Effectiveness of Scheduler.

Original: To evaluate the effectiveness of the scheduler, we disable it and use a random assignment in all cases and the results are also illustrated in Figure 3(c).
Revised: To evaluate the effectiveness of the scheduler, we disable it and use a random assignment in all cases and the results are also illustrated in Figure 3(c).

Original: We see that with our scheduler provides up to 2 .
Revised: We see that with our scheduler provides up to 2 .

Original: 7 × speeds up.
Revised: 7 × speeds up.

Original: To evaluate our local search strategy, we also compare our scheduler with a scheduler that uses the standard Kernighan-Lin algorithm for local search, illustrated in Figure 4.
Revised: To evaluate our local search strategy, we also compare our scheduler with a scheduler that uses the standard Kernighan-Lin algorithm for local search, illustrated in Figure 4.

Original: We see that, while both outperform random, our carefully designed local search strategy significantly outperforms Kernighan-Lin.
Revised: We see that, while both outperform random, our carefully designed local search strategy significantly outperforms Kernighan-Lin.

Original:  Related Work
Revised:  Related Work

Original: Foundation models.
Revised: Foundation models.

Original: Foundation models[1] refer to models that are trained on large-scale data and can be adapted (e.g., fine-tuned) to a wide range of downstream tasks.
Revised: Foundation models[1] refer to models that are trained on large-scale data and can be adapted (e.g., fine-tuned) to a wide range of downstream tasks.

Original: Current examples include
Revised: Current examples include

Original: BERT [39], GPT-3 [2], and CLIP[40].
Revised: BERT [39], GPT-3 [2], and CLIP[40].

Original: Foundation models are usually trained in a data center, where the connection between GPUs is fast and homogeneous.
Revised: Foundation models are usually trained in a data center, where the connection between GPUs is fast and homogeneous.

Original: ML infrastructures such as Megatron[4] and ZeRO[10, 11] have been proposed to distribute the training of these foundation models in a data center.
Revised: ML infrastructures such as Megatron[4] and ZeRO[10, 11] have been proposed to distribute the training of these foundation models in a data center.

Original: Megatron uses AllReduce to synchronize activations in tensor model parallelism; ZeRO adopts ScatterGather to dispatch sharded parameters for layer-wise data parallelism.
Revised: Megatron uses AllReduce to synchronize activations in tensor model parallelism; ZeRO adopts ScatterGather to dispatch sharded parameters for layer-wise data parallelism.

Original: However, such collective communication paradigms would cause serious performance problems with slow and heterogeneous connections (see Appendix for detailed discussions).
Revised: However, such collective communication paradigms would cause serious performance problems with slow and heterogeneous connections (see Appendix for detailed discussions).

Original: Decentralized optimization .
Revised: Decentralized optimization .

Original: Decentralized training is first proposed within the scope of data parallelism, where each worker only synchronizes gradients with its neighbors (instead of all workers) to remove the latency bottleneck [17, 41, 16, 42, 43, 44].
Revised: Decentralized training is first proposed within the scope of data parallelism, where each worker only synchronizes gradients with its neighbors (instead of all workers) to remove the latency bottleneck [17, 41, 16, 42, 43, 44].

Original: Recently, [20] has also modified the implementation of data parallelism to support training in an open collaborative environment.
Revised: Recently, [20] has also modified the implementation of data parallelism to support training in an open collaborative environment.

Original: Varuna [45] is released by Microsoft to support the training of GPT models in spot instances from a cloud service provider, which has the potential to be extended to the open collective scenario, but there is limited consideration with respect to the challenges of heterogeneous connections.
Revised: Varuna [45] is released by Microsoft to support the training of GPT models in spot instances from a cloud service provider, which has the potential to be extended to the open collective scenario, but there is limited consideration with respect to the challenges of heterogeneous connections.

Original: Volunteer computing .
Revised: Volunteer computing .

Original: Distributing computationally intensive tasks over an open collaborative environment has been advocated for a few decades since the development of BOINC[46]; for example, the folding@home project [13] has been running simulations about protein dynamics on volunteers’ personal computers for more than 20 years.
Revised: Distributing computationally intensive tasks over an open collaborative environment has been advocated for a few decades since the development of BOINC[46]; for example, the folding@home project [13] has been running simulations about protein dynamics on volunteers’ personal computers for more than 20 years.

Original: Recently, the learning@home project[19] starts to consider training of mixture-of-expert transformers in such a volunteer computing setting.
Revised: Recently, the learning@home project[19] starts to consider training of mixture-of-expert transformers in such a volunteer computing setting.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we probe the opportunity to train foundation models via a decentralized training regime with devices connected over a heterogeneous network.
Revised: In this paper, we probe the opportunity to train foundation models via a decentralized training regime with devices connected over a heterogeneous network.

Original: We propose an effective scheduling algorithm to assign tasklets from the foundation model pre-train computation.
Revised: We propose an effective scheduling algorithm to assign tasklets from the foundation model pre-train computation.

Original: Empirical studies suggest that, in the worldwide geo-distributed scenario, our proposed scheduling algorithm enables a 4 . 8 × speedup compared to prior state-of-the-art training systems.
Revised: Empirical studies suggest that, in the worldwide geo-distributed scenario, our proposed scheduling algorithm enables a 4 . 8 × speedup compared to prior state-of-the-art training systems.

Original: We believe that the decentralization and democratization of the training of FMs can shift the balance of power positively, but also necessitate new governance structures to help ensure the responsible development and deployment of FMs.
Revised: We believe that the decentralization and democratization of the training of FMs can shift the balance of power positively, but also necessitate new governance structures to help ensure the responsible development and deployment of FMs.

Original: Acknowledgments
Revised: Acknowledgments

Original: CZ and the DS3Lab gratefully acknowledge the support from the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number MB22.00036 (for European Research Council (ERC) Starting
Revised: CZ and the DS3Lab gratefully acknowledge the support from the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number MB22.00036 (for European Research Council (ERC) Starting

Original: Grant TRIDENT 101042665), the Swiss National Science Foundation (Project Number 200021_184628, and 197485), Innosuisse/SNF BRIDGE
Revised: Grant TRIDENT 101042665), the Swiss National Science Foundation (Project Number 200021_184628, and 197485), Innosuisse/SNF BRIDGE

Original: Discovery (Project Number 40B2-0_187132), European Union Horizon 2020 Research and Innovation Programme (DAPHNE, 957407), Botnar Research Centre for Child Health,
Revised: Discovery (Project Number 40B2-0_187132), European Union Horizon 2020 Research and Innovation Programme (DAPHNE, 957407), Botnar Research Centre for Child Health,

Original: Swiss Data Science Center, Alibaba, Cisco, eBay, Google Focused Research Awards, Kuaishou Inc., Oracle
Revised: Swiss Data Science Center, Alibaba, Cisco, eBay, Google Focused Research Awards, Kuaishou Inc., Oracle

Original: Labs, Zurich Insurance, and the Department of Computer Science at ETH Zurich.
Revised: Labs, Zurich Insurance, and the Department of Computer Science at ETH Zurich.

Original: CR gratefully acknowledges the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos.
Revised: CR gratefully acknowledges the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos.

Original: CCF

Revised: CCF

Original: (Beyond Sparsity),
Revised: (Beyond Sparsity),

Original: CCF1563078 (Volume to Velocity), and 1937301 (RTML);
Revised: CCF1563078 (Volume to Velocity), and 1937301 (RTML);

Original: ARL under No.
Revised: ARL under No.

Original: W911NF-21-2-0251 (Interactive
Revised: W911NF-21-2-0251 (Interactive

Original: Human-AI Teaming)
Revised: Human-AI Teaming)

Original: ; ONR under No. N

Revised: ; ONR under No. N

Original: (Unifying Weak Supervision); ONR N00014-20-1-2480:
Revised: (Unifying Weak Supervision); ONR N00014-20-1-2480:

Original: Understanding and Applying Non-Euclidean Geometry in Machine Learning; N000142012275 (NEPTUNE);
Revised: Understanding and Applying Non-Euclidean Geometry in Machine Learning; N000142012275 (NEPTUNE);

Original: NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture,
Revised: NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture,

Original: Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for
Revised: Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for

Original: Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project:
Revised: Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project:

Original: Facebook, Google, and VMWare.
Revised: Facebook, Google, and VMWare.

Original: The U.S distributeChecklist
Revised: The U.S distribute

Original: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:

Original: • Did you include the license to the code and datasets?

Original: [N/A] • Did you include the license to the code and datasets?

Original: [N/A] • Did you include the license to the code and datasets?

Original: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] In lines 122-124 of the Problem Formulation, we highlight the core limitations and simplifying assumptions.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] In lines 364-366 of the Conclusion, we discuss the potential negative impact of decentralization and democratization, noting that while there are positives, this can also lead to acceleration and lack of control.

Original: We look forward to actively engaging with the community on questions of governance.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] We fully follow the standard settings for GPT3-1.3B training.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We repeated all the experiments for at least three times and reported the average in the paper.

Original: Details are reported in Appendix.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] Our systems was built on PyTorch and CuPy and was evaluated using publicly available datasets.

Original: All of them use open-source licenses and can be used for non-commercial educational purposes.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: A Social Impact of Decentralized Training

Original: In this paper, we find that decentralized training shows great potential for foundation models — such a technique would lead to significant positive social impacts.

Original: For example, decentralized training can utilize more inexpensive computational resources, which can significantly reduce the budget for the training of foundation models.

Original: This would increase the accessibility of foundation models for small research and commercial institutions.

Original: In fact, if the expense would be significantly reduced, large organizations would also receive some benefit from adopting such technology.

Original: On the other hand, we also notice that decentralization and democratization can also lead to a lack of control of cheaper computing resources and accelerate the risks of foundation models [1].

Original: We look forward to actively engaging the community on governance questions.

Original: B Limitation and Future Work

Original: There are still some limitations of the current approach that could be explored further in future work.

Original: First, we assume a homogeneous compute GPU resource in the scheduling algorithm, and in practice, different types of GPU could join the training computations.

Original: A native extension of the current solution could be to further split the tasklets into smaller pieces and to assign different numbers of pieces in different types of GPUs considering their memory budget and compute power as constraints.

Original: However, there are many opportunities for further improvement.

Original: Second, there is still lots of room for the improvement of our scheduling algorithm, and strengthen our argument about the end-to-end speedup.

Original: In this paper, we hope to first provide some positive insights to the community that the potential of decentralized training can make it deployable for giant foundation models.

Original: On the other hand, there are many recent advances in system optimization that could lead to further improvement.

Original: For example, exploring recent advances in reinforcement learning [47] to solve our scheduling algorithm would be an interesting future direction.

Original: We believe that any improvement there can only improve the decentralized performance, which is consistent with the central message we try to share in this paper.

Original: Last but not least, there are still some important open questions on the system side to handle the dynamics in decentralized environments.

Original: For example, some mechanism should be necessary to handle the dynamic join and leave of GPU nodes.

Original: On the other hand, failure happens more frequently in the decentralized environment, as fault tolerance should be considered for deployment, we believe that the current strategy such as checkpointing the model periodically could be adopted for this problem, but there would be some suitable solutions for the decentralized training runtime.

Original: C Anatomy of the Current ML Systems for Foundation Model Training

Original: Training foundation models [1] is a challenging task due to the enormous scale of such models — even the most powerful GPU cannot hold a complete copy of parameters for such models [48].

Original: Thus, one cannot train such a model without distribution or using vanilla data parallelism.

Original: Two popular approaches have been proposed to distribute the training of such foundation models in a data center:

Original: • Megatron [4] distributes training by combining its proposed tensor model parallelism with pipeline parallelism [49, 22] and data parallelism [50].

Original: The tensor model parallelism partitions individual layers across a group of workers and must run one AllReduce for the output activations of each layer in forward propagation and one AllReduce for the corresponding gradients in backward propagation for each layer.

Original: • ZeRO [10] can be viewed as an effective optimization for data parallelism.

Original: The most effective mode is called ZeRO stage-3 from the Deepspeed implementation [5]

Original: and the equivalent implementation is known as Fully Sharded Data Parallelism (FSDP) from Fairscale [6]).

Original: In this mode, the parameter is sharded among all workers — in forward propagation, each worker conducts one AllGather to collect the parameters demanded for the current layer and discard the parameter after the forward computation; in backward propagation, each worker uses one AllGather to collect the parameter again and run one ReduceScatter to synchronize the gradients of this layer after the backward computation.

Original: Both Megatron and ZeRO take a heavy usage of collective communication paradigms [51], which leads to two fundamental problems when it is deployed with heterogeneous and slow connections:

Original: • Demanding high bandwidth connections.

Original: Both Megatron and ZeRO require high bandwidth connections for collective communications, since the compute cores are idled during communication slots.

Original: As long as communication takes an increasing share (due to lower bandwidth) of the execution time, the hardware efficiency drops dramatically.

Original: In fact, tensor model parallelism is recommended only within a single DGX server equipped with high-bandwidth NVLinks [48].

Original: • Sensitive to straggler.

Original: The design and implementation of state-of-the-art collective communication libraries, e.g., NCCL [52], assume highly homogeneous connections within a data center, thus there is not sufficient robustness to handle the straggler among workers due to the heterogeneity of the open collective runtime.

Original: Furthermore, the layer-wise usage of collective communications in both Megatron and ZeRO has intensified this problem.

Original: To bridge the performance gap between the data center and the decentralized open environment we need to rethink the communication paradigms in different parallel strategies.

Original: • Pipeline parallelism is communication efficient.

Original: Pipeline parallelism [49, 22, 53] partitions the model into multiple stages and a batch into multiple mini-batches, where once a worker finished the forward computation of a micro-batch, this worker will send the activations to the worker running the next stages; on the other hand, a worker needs to send the gradients of the activation back to the last stage in the backward propagation.

Original: Notice that pipeline parallelism utilizes point-to-point communications instead of collective paradigms.

Original: As long as one can put an increasing amount of computation inside a stage, the ratio of communication cost will also drop, leading to more efficient utilization of compute cores.

Original: On the other hand, pipeline parallelism has its own limitation — one can only partition a model to a limited number of stages, which cannot scale out to lots of GPUs.

Original: We need to combine pipeline parallelism with data parallelism to scale out the training.

Original: • Scheduling is essential.

Original: The point-to-point communication pattern in pipeline parallelism provides good opportunities to assign the training procedure on the decentralized environment that utilizes fast links and

Original: avoids slow links by a carefully designed scheduler, as presented in Section 3.

Original: D Additional Details of Experimental Evaluation

Original: We enumerate some additional details about our experiments.

Original: D.1 Multiple Execution of the Benchmark

Original: We repeated all the benchmarks of 5 different scenarios listed in Section 4 three times.

Original: For our system with scheduler, since the scheduled layout is the same, we simply issued three independent executions in each scenarios; For our system without scheduler, we used three different random seeds (2022, 2023, 2024) to generate three layouts, and issued one execution for each layout in each scenario.

Original: The number in Figure 3 is based on an average of three different runs for each scenario — to avoid visual confusion, we did not plot the error bar within this line plot.

Original: We also repeated the scheduling algorithms three times with random seeds (0, 1, 2) to generate scheduled layouts and reported the average estimated cost (seconds) in Figure 4.

Original: In Figure 6, we plot runtime of each iteration as a bar chart with error bars.

Original: Notice that the variance of all executions in each setting is within 5% .

Original: D.2 Tuning of Megatron and Deepspeed

Original: Megatron.

Original: We carefully tuned Megatron to perform a fair comparison with our system.

Original: As we mentioned in Section C. Megatron has three free degrees of parallel strategies: tensor model parallelism , pipeline parallelism , and data parallelism , we note the degrees of these parallel strategies as D TP , D PP , and D DP respectively.

Original: We conduct a complete grid search of the combinations of these hyper-parameters in the space of:

Original: [Equation]

Original: D

Original: [Equation]

Original: D DP ∈ { 1 , 2 , 4 , 8 } and D TP × D PP × D DP = 64 } .

Original: [Figure]

Original: And we reported the optimal setting as the results for Megatron.

Original: Interestingly, only in Case 1 (data center on demand), the optimal setting includes tensor model parallelism (i.e., D TP ̸ = 1 ), where

Original: D TP = 2 , D PP = 4 , D DP = 8 ; in all other scenarios, the optimal setting is D TP = 1 , D PP = 8 , D DP = 8 .

Original: This illustrates that tensor model parallelism is not suitable in slow and heterogeneous settings, consistent with our analysis in Section C. Since Megatron does not include a similar scheduler as its own components, we use the same random layouts as what we test for our system without scheduler.

Original: Deepspeed.

Original: To run Deepspeed, we start with ZeRO-S3, which is usually viewed as the most significant technical contribution of the Deepspeed system.

Original: Under the same settings, the execution time of ZeRO-S3 is much longer comparing to both Megatron and our system (See Figure 5).

Original: This is consistent with our analysis in Appendix B. We then try to combine the pipeline parallel implementation in Deepspeed with its different implementations of data parallelism (e.g., ZeRO-S 1, 2 and 3)—it turns out that even the latest version of Deepspeed (0.6.7) only allows ZeRO-S1 to combine with pipeline parallelism.

Original: We find this combination outperforms ZeRO-S3 in most of the settings in Case 1 and all settings in Case 2,3,4 and 5.

Original: Notice that results of Deepspeed we report in

Original: Figure 3 is based on the optimal result of these two settings.

Original: D.3 Network Benchmark.

Original: To obtain the network delay and bandwidth between different regions across the world, we rent

Original: AWS instances in 9 different data centers (California, Oregon, Virginia, Ohio, Tokyo, Seoul, London,

Original: Frankfurt, and Ireland).

Original: Instead of using AWS VPC, we setup our own VPN (using StrongSwan [54]) established on the public IP of these instances—any GPU machine connected to Internet can be linked in the same way.

Original: The strongSwan VPN would expose a private IP associated with a visible network interface, we can bind the NCCL communication on this network interface.

Original: The delay and bandwidth we obtained for cross-region NCCL connections are summarized in Table 1 for Caseregional geo-distributed scenario and and Table 2 for Case 5 world-wide geo-distributed scenario.

Original: D.4 Other Presentation of Experimental Results

Original: In Figure 6, we plot runtime of each iteration for each scenario; this is a supplement to Figure

Original: . D.5 Deployment on FluidStack

Original: We believe that having more realistic measurements and an end-to-end run can provide more pervasive statements for decentralized training.

Original: To this end, we conducted an additional experiment.

Original: [Table]

Original: [Table]

Original: We rent 32 A40 GPUs (each with 48GB GPU memory, and 149.7 peak FP16 TFLOPS) from

Original: FluidStack [55], which consists of a group of geo-distributed GPU clusters, in (1) US Mid and (2)

Original: US East.

Original: We get the communication delay and bandwidths between GPUs as below:

Original: • Intra-US Mid: delay 0 .

Original: 5 ± 0 .

Original: 1 ms; bandwidth 10 .

Original: 40 ± 1 . 11 Gbps; • Intra-US East: delay 0 .

Original: 5 ± 0 .

Original: 1 ms; bandwidth 11 .

Original: 98 ± 1 .

Original: 92 Gbps; • US Mid to US East: delay 21 .

Original: 8 ± 0 .

Original: 3 ms; bandwidth 3 .

Original: [-87 ± 1 . 
-]
Revised: [-87 ± 1 . 
-]

Original: Gbps; • US East to US Mid: delay 21 .

Original: 8 ± 0 .

Original: 3 ms; bandwidth 3 .

Original: 73 ± 1 .

Original: 38 Gbps.

Original: We conducted an end-to-end run of the same training task of GPT3-1.3B without artificially controlling the bandwidth and latency.

Original: We also explore the training tasks of larger scale GPT3 models, including

Original: GPT3-6.7B, and GPT3-13B with a batch size of 1024.

Original: The performance in terms of the total number of floating point operations per second (PFLOPS) and runtime of each iteration are illustrated in

Original: Figure 7.

Original: This is a promising result of decentralized training — for GPT3-1.3B model with

Original: C a s e

Original: D a t a C e n t e r on D e m a nd

Original: C a s e

Original: D a t a C e n t e r S po t I n s t a n ce s

Original: C a s e

Original: M u l t i p l e

Original: D a t a C e n t e rs

Original: C a s e

Original: W o r l dw i d e

Original: D i s t

Original: r

Original: i bu t e d

Original: C a s e

Original: R eg i on a l D

Original: i s t

Original: r

Original: i bu t e d

Original: [Figure]

Original: layers and 4K batch size, we archive 27 .

Original: 4% of the peak FLOPS of the cluster, for GPT3-6.7B and

Original: GPT3-13B, we obtain 26 .

Original: 4% and 29 .

Original: 7% respectively.

Original: [Figure]