Original: Decentralized Training of Foundation Models in Heterogeneous Environments
Revised: Decentralized Training of Foundation Models in Heterogeneous Environments

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Training foundation models, such as GPT-3 and PaLM, can be extremely expensive,often involving tens of thousands of GPUs running continuously for months.
Revised: Training foundation models, such as GPT-3 and PaLM, can be extremely expensive,often involving tens of thousands of GPUs running continuously for months.

Original: Thesemodels are typically trained in specialized clusters featuring fast, homogeneousinterconnects and using carefully designed software systems that support bothdata parallelism and model/pipeline parallelism.
Revised: Thesemodels are typically trained in specialized clusters featuring fast, homogeneousinterconnects and using carefully designed software systems that support bothdata parallelism and model/pipeline parallelism.

Original: Such dedicated clusters can becostly and difficult to obtain.
Revised: Such dedicated clusters can becostly and difficult to obtain.

Original: Can we instead leverage the much greater amount ofdecentralized, heterogeneous, and lower-bandwidth interconnected compute?
Revised: Can we instead leverage the much greater amount ofdecentralized, heterogeneous, and lower-bandwidth interconnected compute?

Original: Previous works examining the heterogeneous, decentralized setting focus on relativelysmall models that can be trained in a purely data parallel manner.
Revised: Previous works examining the heterogeneous, decentralized setting focus on relativelysmall models that can be trained in a purely data parallel manner.

Original: State-of-the-artschemes for model parallel foundation model training, such as Megatron, onlyconsider the homogeneous data center setting.
Revised: State-of-the-artschemes for model parallel foundation model training, such as Megatron, onlyconsider the homogeneous data center setting.

Original: In this paper, we present the firststudy of training large foundation models with model parallelism in a decentralizedregime over a heterogeneous network.
Revised: In this paper, we present the firststudy of training large foundation models with model parallelism in a decentralizedregime over a heterogeneous network.

Original: Our key technical contribution is a scheduling algorithm that allocates different computational “tasklets” in the training offoundation models to a group of decentralized GPU devices connected by a slowheterogeneous network.
Revised: Our key technical contribution is a scheduling algorithm that allocates different computational “tasklets” in the training offoundation models to a group of decentralized GPU devices connected by a slowheterogeneous network.

Original: We provide a formal cost model and further proposean efficient evolutionary algorithm to find the optimal allocation strategy.
Revised: We provide a formal cost model and further proposean efficient evolutionary algorithm to find the optimal allocation strategy.

Original: Weconduct extensive experiments that represent different scenarios for learning overgeo-distributed devices simulated using real-world network measurements.
Revised: Weconduct extensive experiments that represent different scenarios for learning overgeo-distributed devices simulated using real-world network measurements.

Original: In themost extreme case, across 8 different cities spanning 3 continents, our approach is4.8 × faster than prior state-of-the-art training systems (Megatron).
Revised: In themost extreme case, across 8 different cities spanning 3 continents, our approach is4.8 × faster than prior state-of-the-art training systems (Megatron).

Original:  Introduction
Revised:  Introduction

Original: Recent years have witnessed the rapid development of deep learning models, particularly foundationmodels (FMs) [1] such as GPT-3 [2] and PaLM [3].
Revised: Recent years have witnessed the rapid development of deep learning models, particularly foundationmodels (FMs) [1] such as GPT-3 [2] and PaLM [3].

Original: Along with these rapid advancements, however,comes computational challenges in training these models: the training of these FMs can be veryexpensive — a single GPT3-175B training run takes 3.6K Petaflops-days [2]— this amounts to $4Mon today’s AWS on demand instances, even assuming 50% device utilization (V100 GPUs peak at125 TeraFLOPS)!
Revised: Along with these rapid advancements, however,comes computational challenges in training these models: the training of these FMs can be veryexpensive — a single GPT3-175B training run takes 3.6K Petaflops-days [2]— this amounts to $4Mon today’s AWS on demand instances, even assuming 50% device utilization (V100 GPUs peak at125 TeraFLOPS)!

Original: Even the smaller scale language models, e.g., GPT3-1.3B (1.3 billion parameters),on which this paper evaluates, require 64 Tesla V100 GPUs to run for one week, costing $32K on
Revised: Even the smaller scale language models, e.g., GPT3-XL (1.3 billion parameters),on which this paper evaluates, require 64 Tesla V100 GPUs to run for one week, costing $32K on

Original: AWS.
Revised: AWS.

Original: As a result, speeding up training and decreasing the cost of FMs have been active researchareas.
Revised: As a result, speeding up training and decreasing the cost of FMs have been active researchareas.

Original: Due to their vast number of model parameters, state-of-the-art systems (e.g., Megatron[4],
Revised: Due to their vast number of model parameters, state-of-the-art systems (e.g., Megatron[4],

Original: Deepspeed[5], Fairscale[6]) leverage multiple forms of parallelism [4, 7, 8, 9, 10, 11].
Revised: Deepspeed[5], Fairscale[6]) leverage multiple forms of parallelism [4, 7, 8, 9, 10, 11].

Original: However, theirdesign is only tailored to fast , homogeneous data center networks.
Revised: However, theirdesign is only tailored to fast , homogeneous data center networks.

Original: On the other hand, decentralization is a natural and promising direction.
Revised: On the other hand, decentralization is a natural and promising direction.

Original: Jon Peddie Research reports that the PC and AIB GPU market shipped 101 million units in Q4 2021 alone [12].
Revised: Jon Peddie Research reportsthat the PC and AIB GPU market shipped 101 million units in Q4 2021 alone [12].

Original: Furthermore,
Revised: Furthermore,

Original: [Figure]
Revised: [Figure]

Original: many of these GPUs are underutilized.
Revised: many of these GPUs are underutilized.

Original: Leveraging this fact, volunteer computing projects suchas Folding@Home
Revised: Leveraging this fact, volunteer computing projects suchas Folding@Home

Original: [13] have sourced upwards of 40K Nvidia and AMD GPUs continuously [14].
Revised: [13] have sourced upwards of 40K Nvidia and AMD GPUs continuously [14].

Original: Moreover, The incremental electricity and HVAC costs of running a V100 GPU for a volunteer are50–100 × lower than the spot prices for an equivalent device on AWS [15].
Revised: Moreover, The incremental electricity and HVAC costs of running a V100 GPU for a volunteer are50–100 × lower than the spot prices for an equivalent device on AWS [15].

Original: If we could make useof these devices in a decentralized open-volunteering paradigm for foundation model training, thiswould be a revolutionary alternative to the expensive solutions offered by data centers.
Revised: If we could make useof these devices in a decentralized open-volunteering paradigm for foundation model training, thiswould be a revolutionary alternative to the expensive solutions offered by data centers.

Original: This vision inspired many recent efforts in decentralized learning, including both those that aretheoretical and algorithmic [16, 17, 18], as well as recent prototypes such as Learning@Home
Revised: This vision inspired many recent efforts in decentralized learning, including both those that aretheoretical and algorithmic [16, 17, 18], as well as recent prototypes such as Learning@Home

Original: [19]and DeDLOC
Revised: [19]and DeDLOC

Original: However, efforts to-date in decentralized training either focus solely on dataparallelism [16, 17, 18, 20], which alone is insufficient for FMs whose parameters exceed thecapacity of a single device, or orient around alternative architectures, e.g., mixture of experts [19].
Revised: However, efforts to-date in decentralized training either focus solely on dataparallelism [16, 17, 18, 20], which alone is insufficient for FMs whose parameters exceed thecapacity of a single device, or orient around alternative architectures, e.g., mixture of experts [19].

Original: These alternative architectures provide promising directions for decentralized learning, however, theyare currently only trained and evaluated on smaller datasets and at a smaller computational scale (e.g.,
Revised: These alternative architectures provide promising directions for decentralized learning, however, theyare currently only trained and evaluated on smaller datasets and at a smaller computational scale (e.g.,

Original: MNIST and WikiText-2 in [19]) than their state-of-the-art counterparts, e.g., GLaM [21].
Revised: MNIST and WikiText-2 in [19]) than their state-of-the-art counterparts, e.g., GLaM [21].

Original: In thispaper, we focus on a standard GPT-style architecture, without considering any changes that mightalter the model architecture or the convergence behaviour during training.
Revised: In thispaper, we focus on a standard GPT-style architecture, without considering any changes that mightalter the model architecture or the convergence behaviour during training.

Original: To fulfill the potential of decentralization for the training of FMs, we need to be able to (1) takeadvantage of computational devices connected via heterogeneous networks with limited bandwidthand significant latency, and (2) support forms of parallelism beyond pure data parallelism.
Revised: To fulfill the potential of decentralization for the training of FMs, we need to be able to (1) takeadvantage of computational devices connected via heterogeneous networks with limited bandwidthand significant latency, and (2) support forms of parallelism beyond pure data parallelism.

Original: In thispaper, we tackle one fundamental aspect of this goal — how can we assign different computational“tasklets”, corresponding to a micro-batch and a subset of layers, to a collection of geo-distributeddevices connected via heterogeneous, slow networks?
Revised: In thispaper, we tackle one fundamental aspect of this goal — how can we assign different computational“tasklets”, corresponding to a macro-batch and a subset of layers, to a collection of geo-distributeddevices connected via heterogeneous, slow networks?

Original: This is not an easy task — even for fast andhomogeneous data center networks, such assignments are still an open ongoing research challenge [22,23, 24, 25, 26].
Revised: This is not an easy task — even for fast andhomogeneous data center networks, such assignments are still an open ongoing research challenge [22,23, 24, 25, 26].

Original: For the heterogeneous setting, it becomes even more challenging as the size of thesearch space increases dramatically.
Revised: For the heterogeneous setting, it becomes even more challenging as the size of thesearch space increases dramatically.

Original: In the homogeneous setting, the homogeneity of the edges in thecommunication graph reduces the search space into many equivalent classes representing allocationstrategies with the same communication costs, enabling efficient polynomial runtime algorithms [23,24, 22, 25, 26]; however, in the heterogeneous setting, one has to consider potentially exponentiallymany more distinct allocation strategies — as we will see later, because of the heterogeneity of thecommunication matrix, even the sub-problem of finding the best pipeline parallelism strategy equatesto a hard open loop travelling salesman problem [27].
Revised: In the homogeneous setting, the homogeneity of the edges in thecommunication graph reduces the search space into many equivalent classes representing allocationstrategies with the same communication costs, enabling efficient polynomial runtime algorithms [23,24, 22, 25, 26]; however, in the heterogeneous setting, one has to consider potentially exponentiallymany more distinct allocation strategies — as we will see later, because of the heterogeneity of thecommunication matrix, even the sub-problem of finding the best pipeline parallelism strategy equatesto a hard open loop travelling salesman problem [27].

Original: In this paper, we focus on this challenging scheduling problem of decentralized training of FMs overslow, heterogeneous networks, and make the following contributions:
Revised: In this paper, we focus on this challenging scheduling problem of decentralized training of FMs overslow, heterogeneous networks, and make the following contributions:

Original: • We study the problem of allocating distributed training jobs over a group of decentralized GPU
Revised: • We study the problem of allocating distributed training jobs over a group of decentralized GPU

Original: devices
Revised: devices

Original: connected
Revised: connected

Original: via a
Revised: via a

Original: slow
Revised: slow

Original: heterogeneous network
Revised: heterogeneous network.

Original: More specifically
Revised: More specifically

Original: :– To capture the complex communication cost for training FMs, we propose a natural, butnovel, formulation involving decomposing the cost model into two levels: the first level isa balanced graph partitioning problem corresponding to the communication cost of
Revised: :– To capture the complex communication cost for training FMs, we propose a natural, butnovel, formulation involving decomposing the cost model into two levels: the first level isa balanced graph partitioning problem corresponding to the communication cost of

Original: data
Revised: data

Original: parallelism, whereas the second level is a joint graph matching and traveling salesman [Table] problem corresponding to the communication cost of pipeline parallelism.
Revised: parallelism, whereas the second level is a joint graph matching and traveling salesmanproblem corresponding to the communication cost of pipeline parallelism.

Original: [Table]
Revised: [Table]

Original: – We propose a novel scheduling algorithm to search for the optimal allocation strategy givenour cost model.
Revised: – We propose a novel scheduling algorithm to search for the optimal allocation strategy givenour cost model.

Original: Developing a direct solution to this optimization problem is hard; thus, wepropose an efficient evolutionary algorithm based on a collection of novel heuristics, goingbeyond the traditional heuristics used in standard graph partitioning methods[28]. [Table]
Revised: Developing a direct solution to this optimization problem is hard; thus, wepropose an efficient evolutionary algorithm based on a collection of novel heuristics, goingbeyond the traditional heuristics used in standard graph partitioning methods

Original: • We carefully designed and implemented a collection of system optimizations to hide communi-
Revised: [28].• We carefully designed and implemented a collection of system optimizations to hide communi-

Original: [Table]
Revised: [Table]

Original: cation within the computation to further reduce the impact of slow connections. [Table] • We conduct extensive experiments that represent different scenarios of collaborative decentral-
Revised: cation within the computation to further reduce the impact of slow connections.• We conduct extensive experiments that represent different scenarios of collaborative decentralized learning, simulated by using network measurements from different geographical regions ofAWS.

Original: [Table] ized learning, simulated by using network measurements from different geographical regions ofAWS. In the worldwide setting with 64 GPUs across 8 regions (Oregon, Virginia, Ohio, Tokyo,Seoul, London, Frankfurt, Ireland), we show that our system is 3.8-4.8 × faster, in end-to-endruntime, than the state-of-the-art system, Megatron, for training GPT3-1.3B, without any difference in what is computed or convergence dynamics .
Revised: In the worldwide setting with 64 GPUs across 8 regions (Oregon, Virginia, Ohio, Tokyo,Seoul, London, Frankfurt, Ireland), we show that our system is 3.8-4.8 × faster, in end-to-endruntime, than the state-of-the-art system, Megatron, for training GPT3-XL, without any difference in what is computed or convergence dynamics .

Original: In addition, we also provide careful ablationstudies to show the individual effectiveness of the scheduler and system optimizations. [Table]
Revised: In addition, we also provide careful ablationstudies to show the individual effectiveness of the scheduler and system optimizations.•

Original: • We shed light on the potential of decentralized learning — our prototype in the global heteroge-
Revised: We shed light on the potential of decentralized learning — our prototype in the global heteroge-

Original: [Table]
Revised: [Table]

Original: neous setting is only 1.7-3.5 × slower than Megatron in data centers even though its network canbe 100 × slower .
Revised: neous setting is only 1.7-3.5 × slower than Megatron in data centers even though its network canbe 100 × slower .

Original: We hope this paper can inspire future explorations of decentralized learningfor FMs, over geo-distributed servers, desktops, laptops, or even mobile devices.
Revised: We hope this paper can inspire future explorations of decentralized learningfor FMs, over geo-distributed servers, desktops, laptops, or even mobile devices.

Original: Limitations and Moving Forward.
Revised: Limitations and Moving Forward.

Original: In this paper we tackle one foundational aspect of decentralized learning but leave as future work many problems that are important for a practical system.
Revised: In this paper we tackle one foundational aspect of decentralized learning but leave as future work many problems that are important for a practical system.

Original: We assume
Revised: We assume

Original: [Table]
Revised: [Table]

Original: that communication between devices is relatively stable for a reasonable amount of time and that alldevices are always online without failure or eviction.
Revised: that communication between devices is relatively stable for a reasonable amount of time and that all [Table] devices are always online without failure or eviction.

Original: Note that we also do not train a full system to
Revised: Note that we also do not train a full system to

Original: [Table]
Revised: [Table]

Original: full convergence, instead running partial training to confirm intermediate result equivalence across
Revised: full convergence, instead running partial training to confirm intermediate result equivalence acrossregimes.

Original: [Table] regimes. Scheduling over a dynamic, heterogeneous environment and providing fault tolerance,potentially with checkpointing, while training to convergence are directions for future exploration.
Revised: Scheduling over a dynamic, heterogeneous environment and providing fault tolerance,potentially with checkpointing, while training to convergence are directions for future exploration.

Original:  Decentralized Training of Foundation Models: Problem Formulation
Revised:  Decentralized Training of Foundation Models: Problem Formulation

Original: T r a n s f o r m e r
Revised: T r a n s f o r m e r

Original: B l o ck
Revised: B l o ck

Original: T r a n s f o r m e r
Revised: T r a n s f o r m e r

Original: B l o ck
Revised: B l o ck

Original: T r a n s f o r m e r
Revised: T r a n s f o r m e r

Original: B l o ck
Revised: B l o ck

Original: T r a n s f o r m e r
Revised: T r a n s f o r m e r

Original: B l o ck
Revised: B l o ck

Original: T r a n s f o r m e r
Revised: T r a n s f o r m e r

Original: B l o ck
Revised: B l o ck

Original: T r a n s f o r m e r
Revised: T r a n s f o r m e r

Original: B l o ck
Revised: B l o ck

Original: [Table]
Revised: [Table]

Original: Backward 
Revised: Backward Gradient

Original: [Table]

Original: Gradient
Revised: Gradient

Original: [Table]
Revised: [Table]

Original: Micro

Original: [Table]

Original: D ev i ce D ev i ce
Revised: D ev i ce D ev i ce

Original: D ev i ce D ev i ce
Revised: D ev i ce D ev i ce

Original: We first introduce concepts, technical terms, and the procedure ofdecentralized training.
Revised: We first introduce concepts, technical terms, and the procedure ofdecentralized training.

Original: Then we formally define the schedulingproblem this paper tackles.
Revised: Then we formally define the schedulingproblem this paper tackles.

Original: Decentralized setting.
Revised: Decentralized setting.

Original: We assume a group of devices (GPUs)participating in collaborative training of a foundation model.
Revised: We assume a group of devices (GPUs)participating in collaborative training of a foundation model.

Original: Eachpair of devices has a connection with potentially different delay andbandwidth.
Revised: Eachpair of devices has a connection with potentially different delay andbandwidth.

Original: These devices can be geo-distributed, as illustrated in
Revised: These devices can be geo-distributed, as illustrated in

Original: Figure 1, with vastly different pairwise communication bandwidthand latency.
Revised: Figure 1, with vastly different pairwise communication bandwidthand latency.

Original: In decentralized training, all layers of a model aresplit into multiple stages , where each device handles a consecutive sequence of layers, e.g., severaltransformer blocks [29].
Revised: In decentralized training, all layers of a model aresplit into multiple stages , where each device handles a consecutive sequence of layers, e.g., severaltransformer blocks [29].

Original: In addition, since the input for foundation model pre-training is huge, e.g., afew millions of tokens, it is also split into multiple micro-batches that can be handled in parallel.
Revised: In addition, since the input for foundation model pre-training is huge, e.g., afew millions of tokens, it is also split into multiple macro-batches that can be handled in parallel.

Original: Problem definition.
Revised: Problem definition.

Original: We define tasklets as a collection of computational tasks in foundation modeltraining — Tasklet t i,j is the forward and backward computation for a stage j with a micro-batch i oftraining data in a training iteration.
Revised: We define tasklets as a collection of computational tasks in foundation modeltraining — Tasklet t i,j is the forward and backward computation for a stage j with a macro-batch iof training data in a training iteration.

Original: We aim to design an effective scheduler to assign each tasklet to aparticular device so that the training throughput is maximized in decentralized training.
Revised: We aim to design an effective scheduler to assign each taskletto a particular device so that the training throughput is maximized in decentralized training.

Original: Parallelism.
Revised: Parallelism.

Original: The above setting involves two forms of parallelism, pipeline and data .
Revised: The above setting involves two forms of parallelism, pipeline and data .

Original: In pipelineparallelism, the compute in multiple stages is parallelized — each device handles activation orgradient computation for different micro-batches in parallel and the results can be communicated orpassed to subsequent stages.
Revised: In pipelineparallelism, the compute in multiple stages is parallelized — each device handles activation orgradient computation for different macro-batches in parallel and the results can be communicated orpassed to subsequent stages.

Original: Data parallelism means that devices compute the gradient for differentmicro-batches independently, but need to synchronize these gradients through communication.
Revised: Data parallelism means that devices compute the gradient for differentmacro-batches independently, but need to synchronize these gradients through communication.

Original: Ina decentralized environment, the training procedure is communication-bounded .
Revised: Ina decentralized environment, the training procedure is communication-bounded .

Original: The schedulingproblem is to accelerate the communication procedure by allocating tasklets that require highcommunication volumes between them to devices with faster connections.
Revised: The schedulingproblem is to accelerate the communication procedure by allocating tasklets that require highcommunication volumes between them to devices with faster connections.

Original: [Figure]
Revised: [Figure]

Original: Formalization of the scheduling problem.
Revised: Formalization of the scheduling problem.

Original: Formally, our scheduling problem is as follows.
Revised: Formally, our scheduling problem is as follows.

Original: [Figure]
Revised: [Figure]

Original: • Let D = { d 1 . . .
Revised: • Let D = { d 1 . . .

Original: d N } be a set of N devices; A ∈ R N × N + and B ∈ R N × N + be the communication
Revised: d N } be a set of N devices; A ∈ R N × N + and B ∈ R N × N + be the communication

Original: [Figure]
Revised: [Figure]

Original: matrix between these devices describing the delay and bandwidth respectively, where the delay
Revised: matrix between these devices describing the delay and bandwidth respectively, where the delay

Original: [Figure]
Revised: [Figure]

Original: and bandwidth between device d and d ′ is α d,d ′ and β d,d ′ .
Revised: and bandwidth between device d and d ′ is α d,d ′ and β d,d ′ .

Original: [Figure]
Revised: [Figure]

Original: • Given the communication matrix A and B , we construct a communication graph G (Figure 2(a))
Revised: • Given the communication matrix A and B , we construct a communication graph G (Figure 2(a))

Original: [Figure]
Revised: [Figure]

Original: — each device corresponds to a node in G and each edge between d and d ′ is labeled with the
Revised: — each device corresponds to a node in G and each edge between d and d ′ is labeled with the

Original: average latency and bandwidth between d and d ′ :
Revised: average latency and bandwidth between d and d ′ :

Original: (( α d,d ′ + α d ′ ,d ) / 2 , ( β d,d ′ + β d ′ ,d ) / 2) .
Revised: (( α d,d ′ + α d ′ ,d ) / 2 , ( β d,d ′ + β d ′ ,d ) / 2) .

Original: Even
Revised: Even

Original: though A and B are asymmetric (i.e., upload and download speed might be different), the
Revised: though A and B are asymmetric (i.e., upload and download speed might be different), the

Original: communication graph G is symmetric because in our workloads all communications between
Revised: communication graph G is symmetric because in our workloads all communications between

Original: two devices happen to involve the same amount of upload and download.
Revised: two devices happen to involve the same amount of upload and download.

Original: • The number of stages that a micro-batch needs to go through is D PP (noted as pipeline paralleldegree); the number of batch partition that needs to run model gradient synchronization is D DP
Revised: • The number of stages that a macro-batch needs to go through is D PP (noted as pipeline paralleldegree); the number of batch partition that needs to run model gradient synchronization is D DP

Original: (noted as data parallel degree); we have D DP × D PP = N , i.e., the total number of devices.
Revised: (noted as data parallel degree); we have D DP × D PP = N , i.e., the total number of devices.

Original: • c PP (resp.
Revised: • c PP (resp.

Original: c DP ) represent the number of bytes of activations for a micro-batch (resp.
Revised: c DP ) represent the number of bytes of activations for a macro-batch (resp.

Original: parame-
Revised: parame-

Original: ters/gradients for a stage) communicated in pipeline parallelism (resp.
Revised: ters/gradients for a stage) communicated in pipeline parallelism (resp.

Original: data parallelism).
Revised: data parallelism).

Original: • We denote a training tasklet as t i,j , where i ∈ { 1 , ..., D DP } and j ∈ { 1 , ..., D PP } , each of which
Revised: • We denote a training tasklet as t i,j , where i ∈ { 1 , ..., D DP } and j ∈ { 1 , ..., D PP } , each of which

Original: corresponds to one specific micro-batch i and pipeline stage j .
Revised: corresponds to one specific macro-batch i and pipeline stage j .

Original: • An assignment strategy σ ∈ D D DP × D PP assigns, for each tasklet t i,j , a device σ i,j ∈ D , whichmeans that device σ i,j runs the training tasklet t i,j .
Revised: • An assignment strategy σ ∈ D D DP × D PP assigns, for each tasklet t i,j , a device σ i,j ∈ D , whichmeans that device σ i,j runs the training tasklet t i,j .

Original: An valid assignment needs to be unique,
Revised: An valid assignment needs to be unique,

Original: i
Revised: i

Original: e.,∀ ( i, j ) ̸
Revised: e.,∀ ( i, j ) ̸

Original: = ( i ′ , j ′ ) :
Revised: = ( i ′ , j ′ ) :

Original: σ i,j ̸ = σ i ′ ,j ′ .
Revised: σ i,j ̸ = σ i ′ ,j ′ .

Original: We use Σ to denote the set of all valid assignments.
Revised: We use Σ to denote the set of all valid assignments.

Original: • An optimal assignment strategy is an assignment σ that minimizes the communication cost
Revised: • An optimal assignment strategy is an assignment σ that minimizes the communication cost

Original: [Equation]
Revised: [Equation]

Original: Challenges and Goals.
Revised: Challenges and Goals.

Original: Our goal is to find the optimal assignment strategy, which involves twochallenges: (1) How to effectively model the communication cost C OMM -C OST ( σ ) for a givenassignment σ under a heterogeneous network environment?
Revised: Our goal is to find the optimal assignment strategy, which involves twochallenges: (1) How to effectively model the communication cost C OMM -C OST ( σ ) for a givenassignment σ under a heterogeneous network environment?

Original: and (2) How to effectively search for theoptimal assignment strategy that minimizes such a cost?
Revised: and (2) How to effectively search for theoptimal assignment strategy that minimizes such a cost?

Original: We tackle these two questions in Section 3.
Revised: We tackle these two questions in Section 3.

Original:  Scheduling in Heterogeneous Environments
Revised:  Scheduling in Heterogeneous Environments

Original: Scheduling in the heterogeneous setting is a challenging task, as the size of the search space increasesdramatically compared to that of the homogeneous case.
Revised: Scheduling in the heterogeneous setting is a challenging task, as the size of the search space increasesdramatically compared to that of the homogeneous case.

Original: In the homogeneous data-center case, thenetwork delay can be usually ignored (e.g., A = 0 ) and the bandwidth B are assumed to be formedby just a few constants — e.g., the communication bandwidths between different machines on thesame rack are assumed to be same [23, 24, 22, 22, 26].
Revised: In the homogeneous data-center case, thenetwork delay can be usually ignored (e.g., A = 0 ) and the bandwidth B are assumed to be formedby just a few constants — e.g., the communication bandwidths between different machines on thesame rack are assumed to be same [23, 24, 22, 22, 26].

Original: This significantly constrains the searchspace — one can ignore the influence of communication given uniform connections [23, 24, 22], ororganize the device with a hierarchical structure [22, 26], making the scheduling problem solvable inpolynomial time in terms of the number of machines.
Revised: This significantly constrains the searchspace — one can ignore the influence of communication given uniform connections [23, 24, 22], ororganize the device with a hierarchical structure [22, 26], making the scheduling problem solvable inpolynomial time in terms of the number of machines.

Original: In contrast, in the fully heterogeneous scenario the communication matrix A and B consists ofdistinct values, which can make the search space grows exponentially.
Revised: In contrast, in the fully heterogeneous scenario the communication matrix A and B consists ofdistinct values, which can make the search space grows exponentially.

Original: In this section, we describeour scheduler that searches for an optimal strategy in the complex search space.
Revised: In this section, we describeour scheduler that searches for an optimal strategy in the complex search space.

Original: 3.1 Overview of the scheduler
Revised: 3.1 Overview of the scheduler

Original: We carefully design a bi-level scheduling algorithm based on extended balanced graph partitionproblem (see Figure 2), and solve this problem by an evolutionary algorithm with a carefully designedlocal search strategy.
Revised: We carefully design a bi-level scheduling algorithm based on extended balanced graph partitionproblem (see Figure 2), and solve this problem by an evolutionary algorithm with a carefully designedlocal search strategy.

Original: Given an assignment strategy σ = { σ i,j } for all tasklets { t i,j } , we first model its communication cost.
Revised: Given an assignment strategy σ = { σ i,j } for all tasklets { t i,j } , we first model its communication cost.

Original: During the training of FMs, the communication costs come fromtwo different sources: (1) Data parallel : All devices that are assigned with the tasklets dealingwith the same stage j (handling the same layers) of different micro-batches need to communicatewithin themselves to exchange gradients of these layers.
Revised: During the training of FMs, the communication costs come fromtwo different sources: (1) Data parallel : All devices that are assigned with the tasklets dealingwith the same stage j (handling the same layers) of different macro-batches need to communicatewithin themselves to exchange gradients of these layers.

Original: For layer j , we call these devices itsdata parallel group : C j = { σ i,j | ∀ i ∈
Revised: For layer j , we call these devices itsdata parallel group : C j = { σ i,j | ∀ i ∈

Original: [ D DP ] } .
Revised: [ D DP ] } .

Original: We can implement the communication usingdifferent primitives, e.g., AllReduce [30], ScatterGatter [31], or other decentralized averageprotocols [16].
Revised: We can implement the communication usingdifferent primitives, e.g., AllReduce [30], ScatterGatter [31], or other decentralized averageprotocols [16].

Original: (2) Pipeline parallel : All devices that are assigned with the tasklets dealing with thesame micro-batch i of different stages need to form a pipeline, communicating within themselves toexchange activations and backward gradients.
Revised: (2) Pipeline parallel : All devices that are assigned with the tasklets dealing with thesame macro-batch i of different stages need to form a pipeline, communicating within themselves toexchange activations and backward gradients.

Original: For micro-batch i , these devices are P i = { σ i,j | ∀
Revised: For macro-batch i , these devices are P i = { σ i,j | ∀

Original: j ∈
Revised: j ∈

Original: [ D PP ] } .
Revised: [ D PP ] } .

Original: Because these devices need to form a linear pipeline, any permutation over P i correspondsone strategy of how these machines can conduct pipeline parallelism within them.
Revised: Because these devices need to form a linear pipeline, any permutation over P i correspondsone strategy of how these machines can conduct pipeline parallelism within them.

Original: Scheduling Problem.
Revised: Scheduling Problem.

Original: The goal of our scheduler is to minimize both costs.
Revised: The goal of our scheduler is to minimize both costs.

Original: One design decision that 177 we made is to decompose this complex optimization problem into two levels.
Revised: One design decision that 177 we made is to decompose this complex optimization problem into two levels.

Original: At the first level, weconsider the best way of forming C j ’s, incurring data parallel communication costs within them.
Revised: At the first level, weconsider the best way of forming C j ’s, incurring data parallel communication costs within them.

Original: At 179 the second level, we consider the cost of pipeline parallelism given an layout from the first level:
Revised: At 179 the second level, we consider the cost of pipeline parallelism given an layout from the first level:

Original: [Equation]
Revised: [Equation]

Original: (1) where computing P IPELINE P-C OST ( C 1 ... C D PP ) involves finding the optimal pipeline structure.
Revised: (1) where computing P IPELINE P-C OST ( C 1 ... C D PP ) involves finding the optimal pipeline structure.

Original: In Section 3.2 and Section 3.3, we provide details on C OMM -C OST ( C 1 ... C D PP ) .
Revised: In Section 3.2 and Section 3.3, we provide details on C OMM -C OST ( C 1 ... C D PP ) .

Original: Notice that thismodified objective makes our problem different from the textbook graph partition problem; thus, weneed a carefully designed evolutionary algorithm for finding such a solution introduced in Section 3.4.
Revised: Notice that thismodified objective makes our problem different from the textbook graph partition problem; thus, weneed a carefully designed evolutionary algorithm for finding such a solution introduced in Section 3.4.

Original: 3.2 Modelling data parallel communication cost
Revised: 3.2 Modelling data parallel communication cost

Original: Given the communication graph G forming data parallel groups C 1 ... C D PP corresponds to a partition 186 of G — In Figure 2(b), different colors correspond to devices in the same C j .
Revised: Given the communication graph G forming data parallel groups C 1 ... C D PP corresponds to a partition 186 of G — In Figure 2(b), different colors correspond to devices in the same C j .

Original: The data parallelcost within C j only relies on all communication channels (edges in the communication graph) 188 connecting devices in C j .
Revised: The data parallelcost within C j only relies on all communication channels (edges in the communication graph) 188 connecting devices in C j .

Original: If we assume a colocated sharded parameter server [31] implementation 189 for communicating within each C j , and recall that c DP represents the total amount of data (in bytes)that needs to be exchanged during gradient aggregation — each device in C j needs to managec DP /D DP bytes of parameter shard.
Revised: If we assume a colocated sharded parameter server [31] implementation 189 for communicating within each C j , and recall that c DP represents the total amount of data (in bytes)that needs to be exchanged during gradient aggregation — each device in C j needs to managec DP /D DP bytes of parameter shard.

Original: Once the gradient is ready, each device needs to send each of its 192 local shards to the corresponding device; next, each device can aggregate the gradients it receivesfrom all other devices in C j ; and finally, each device will send the aggregated gradient shard to all 194 other devices.
Revised: Once the gradient is ready, each device needs to send each of its 192 local shards to the corresponding device; next, each device can aggregate the gradients it receivesfrom all other devices in C j ; and finally, each device will send the aggregated gradient shard to all 194 other devices.

Original: Therefore, we can model the data parallel cost for C j as follows:
Revised: Therefore, we can model the data parallel cost for C j as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, the total cost is bounded by the slowest device ( max d ∈ C j ), which needs to exchange data withall other machines ( (cid:80) d ′ ∈ C j −{ d } ).
Revised: Here, the total cost is bounded by the slowest device ( max d ∈ C j ), which needs to exchange data withall other machines ( (cid:80) d ′ ∈ C j −{ d } ).

Original: Because the communication of these different data parallel groups
Revised: Because the communication of these different data parallel groups

Original: C 1 ...
Revised: C 1 ...

Original: C D PP can be conducted in parallel and we are only bounded by the slowest data parallel group. 

Revised: C D PP can be conducted in parallel and we are only bounded by the slowest data parallel group.

Original: This allows us to model the total communication cost for data parallelism as:
Revised: This allows us to model the total communication cost for data parallelism as:

Original: [Equation]
Revised: [Equation]

Original: 3.3 Modeling pipeline parallel communication cost
Revised: 3.3 Modeling pipeline parallel communication cost

Original: Given C 1 ... C D PP , to model the communication cost of pipeline parallelism, we need to considertwo factors: (1) each permutation π of { C 1 ... C D PP } corresponds to a specific pipeline strategy —devices in C π j and devices in C π j +1 communicates to exchange activations (during forward pass) andgradients on activations (during backward pass); and (2) devices in C π j and devices in C π j +1 needto be “matched” — only devices that are dealing with the same micro-batch needs to communicate.
Revised: Given C 1 ... C D PP , to model the communication cost of pipeline parallelism, we need to considertwo factors: (1) each permutation π of { C 1 ... C D PP } corresponds to a specific pipeline strategy —devices in C π j and devices in C π j +1 communicates to exchange activations (during forward pass) andgradients on activations (during backward pass); and (2) devices in C π j and devices in C π j +1 needto be “matched” — only devices that are dealing with the same macro-batch needs to communicate.

Original: This makes modeling the cost of pipeline parallel communication more complex.
Revised: This makes modeling the cost of pipeline parallel communication more complex.

Original: To model the cost of pipeline parallel communication, we first consider the best possible way thatdevices in C j and C j ′ can be matched.
Revised: To model the cost of pipeline parallel communication, we first consider the best possible way thatdevices in C j and C j ′ can be matched.

Original: We do this by creating a coarsened communication graph(Figure 2(c)).
Revised: We do this by creating a coarsened communication graph(Figure 2(c)).

Original: A coarsened communication graph (cid:98) G C 1 ... C D PP is a fully connected graph, and eachpartition C j in the original communication graph G corresponds to a node in (cid:98) G C 1 ... C D PP .
Revised: A coarsened communication graph (cid:98) G C 1 ... C D PP is a fully connected graph, and eachpartition C j in the original communication graph G corresponds to a node in (cid:98) G C 1 ... C D PP .

Original: In the coarsened graph (cid:98) G , the weight on an edge between C j and C j ′ corresponds to the following 211 — if C j and C j ′ need to communicate in a pipeline, what is the communicate cost of the optimal 212 matching strategy between devices in C j and devices in C j ′ ?
Revised: In the coarsened graph (cid:98) G , the weight on an edge between C j and C j ′ corresponds to the following 211 — if C j and C j ′ need to communicate in a pipeline, what is the communicate cost of the optimal 212 matching strategy between devices in C j and devices in C j ′ ?

Original: Recall that c PP represents the amountof data between two devices for pipeline parallel communication, we can model this cost by
Revised: Recall that c PP represents the amountof data between two devices for pipeline parallel communication, we can model this cost by

Original: [Equation]
Revised: [Equation]

Original: where M is a perfect matching between C j and C j ′ — ( d, d ′ )
Revised: where M is a perfect matching between C j and C j ′ — ( d, d ′ )

Original: ∈ M means that device d ∈ C
Revised: ∈ M means that device d ∈ C

Original: j willcommunicate with device d ′ ∈ C j ′ (i.e., they deal with the same micro-batch).
Revised: j willcommunicate with device d ′ ∈ C j ′ (i.e., they deal with the same macro-batch).

Original: Computing this valueis similar to the classical minimal sum weight perfect matching problem (MinSumWPM) in bipartitegraphs [32], with the only difference being that we compute the max instead of the sum .
Revised: Computing this valueis similar to the classical minimal sum weight perfect matching problem (MinSumWPM) in bipartitegraphs [32], with the only difference being that we compute the max instead of the sum .

Original: As we willshow in the supplementary material, similar to MinSumWPM, Eq 3 can also be solved in PTIME.
Revised: As we willshow in the supplementary material, similar to MinSumWPM, Eq 3 can also be solved in PTIME.

Original: The coarsened communication graph captures the pipeline parallel communication cost between two 220 groups of devices, assuming they become neighbors in the pipeline.
Revised: The coarsened communication graph captures the pipeline parallel communication cost between two 220 groups of devices, assuming they become neighbors in the pipeline.

Original: Given this, we need to find anoptimal permutation of C 1 ... C D PP , corresponds to the structure of the pipeline.
Revised: Given this, we need to find anoptimal permutation of C 1 ... C D PP , corresponds to the structure of the pipeline.

Original: This becomes the 222 open-loop traveling salesman problem [27] over this condensed graph (Figure 2(e)).
Revised: This becomes the 222 open-loop traveling salesman problem [27] over this condensed graph (Figure 2(e)).

Original: Formally, wehave the following definition of the pipeline parallel cost:
Revised: Formally, wehave the following definition of the pipeline parallel cost:

Original: [Equation]
Revised: [Equation]

Original: 4) where (cid:98) G C 

Revised: 4) where (cid:98) G C

Original: C D PP is the coarsened graph defined above.
Revised: C D PP is the coarsened graph defined above.

Original: 3.4 Searching via hybrid genetic algorithm
Revised: 3.4 Searching via hybrid generic algorithm

Original: The scheduling problem solves the optimization problem in Eq 1, which corresponds to a balancedgraph partition problem with a complex objective corresponding to the communication cost.
Revised: The scheduling problem solves the optimization problem in Eq 1, which corresponds to a balancedgraph partition problem with a complex objective corresponding to the communication cost.

Original: Balancedgraph partition problem is a challenging NP-hard problem [33].
Revised: Balancedgraph partition problem is a challenging NP-hard problem [33].

Original: Over the years, researchers have beentackling this problem via different ways [34, 35, 36].
Revised: Over the years, researchers have beentackling this problem via different ways [34, 35, 36].

Original: We follow the line of research that uses hybridgenetic algorithm [37, 28] since it provides us the flexibility in dealing with complex objective.
Revised: We follow the line of research that uses hybridgenetic algorithm [37, 28] since it provides us the flexibility in dealing with complex objective.

Original: Hybrid Genetic Algorithm.
Revised: Hybrid Genetic Algorithm.

Original: A hybrid genetic algorithm for balanced graph partition usually followsa structure as as follows.
Revised: A hybrid genetic algorithm for balanced graph partition usually followsa structure as as follows.

Original: The input is a set of candidate balanced graph partitions which serves as theinitial population.
Revised: The input is a set of candidate balanced graph partitions which serves as theinitial population.

Original: The algorithm generates the next generation as follows.
Revised: The algorithm generates the next generation as follows.

Original: It first generates a new“offspring” o given two randomly selected “parents” p 1 and p 2 .
Revised: It first generates a new“offspring” o given two randomly selected “parents” p 1 and p 2 .

Original: One popular way is to randomly swapsome nodes between these two parents (we follow [28]).
Revised: One popular way is to randomly swapsome nodes between these two parents (we follow [28]).

Original: Given this offspring o , we then conduct localsearch starting at o to find a new balanced partitioning strategy o ∗ that leads to better cost.
Revised: Given this offspring o , we then conduct localsearch starting at o to find a new balanced partitioning strategy o ∗ that leads to better cost.

Original: We thenadd o ∗ to the population and remove the worst partition candidate in the population if o ∗ has a bettercost.
Revised: We thenadd o ∗ to the population and remove the worst partition candidate in the population if o ∗ has a bettercost.

Original: As suggested by [37], the combination of heuristic-based local search algorithms and geneticalgorithm can accelerate convergence by striking the balance between local and global optimum.
Revised: As suggested by [37], the combination of heuristic-based local search algorithms and genericalgorithm can accelerate convergence by striking the balance between local and global optimum.

Original: Existing Local Search Strategy.
Revised: Existing Local Search Strategy.

Original: The key in designing this algorithm is to come up with a good localsearch strategy.
Revised: The key in designing this algorithm is to come up with a good localsearch strategy.

Original: For traditional graph partitioning task, one popular choice is to use the Kernighan-Lin
Revised: For traditional graph partitioning task, one popular choice is to use the Kernighan-Lin

Original: Algorithm [38].
Revised: Algorithm [38].

Original: Which, at each iteration, tries to find a pair of nodes: d in partition C j and d ′ inpartition C j ′ , to swap.
Revised: Which, at each iteration, tries to find a pair of nodes: d in partition C j and d ′ inpartition C j ′ , to swap.

Original: To find such a pair to swap, it uses the following “gain” function:
Revised: To find such a pair to swap, it uses the following “gain” function:

Original: [Equation]
Revised: [Equation]

Original: where w i,j corresponds to the weight between node i and j in the graph.
Revised: where w i,j corresponds to the weight between node i and j in the graph.

Original: However, directly applyingthis local search strategy, as we will also show in the experiment (Section 4) does not work well.
Revised: However, directly applyingthis local search strategy, as we will also show in the experiment (Section 4) does not work well.

Original: Greedily following GAIN KL does not decrease the communication cost of foundation model training.
Revised: Greedily following GAIN KL does not decrease the communication cost of foundation model training.

Original: Therefore, we have to design a new local search strategy tailored to our cost model.
Revised: Therefore, we have to design a new local search strategy tailored to our cost model.

Original: Improving Local Search Strategy.
Revised: Improving Local Search Strategy.

Original: Our local search strategy is inspired by two observations:
Revised: Our local search strategy is inspired by two observations:

Original: Removing the device d 1 with a fast connection (say with d 2 ) within partition C j will not tend tochange the data parallel cost within C j , since it is only bounded by the slowest connections.2.
Revised: Removing the device d 1 with a fast connection (say with d 2 ) within partition C j will not tend tochange the data parallel cost within C j , since it is only bounded by the slowest connections.2.

Original: Once d 1 is moved to C j ′ , highly likely the pipeline parallel matching between C j and C j ′ willconsist of the link d 1 ↔ d 2 , since it is a fast connection.
Revised: Once d 1 is moved to C j ′ , highly likely the pipeline parallel matching between C j and C j ′ willconsist of the link d 1 ↔ d 2 , since it is a fast connection.

Original: Therefore, in our local search strategy we only consider the fastest connection within C j : d 1 ↔ d 2and the fastest connection within C j ′ :
Revised: Therefore, in our local search strategy we only consider the fastest connection within C j : d 1 ↔ d 2and the fastest connection within C j ′ :

Original: d ′ 1 ↔ d ′ 2 and generate four swap candidates: d 1 ↔ d ′ 1 ,d 1 ↔ d ′ 2 , d 2 ↔ d ′ 1 , d 2 ↔ d ′ 2 .
Revised: d ′ 1 ↔ d ′ 2 and generate four swap candidates: d 1 ↔ d ′ 1 ,d 1 ↔ d ′ 2 , d 2 ↔ d ′ 1 , d 2 ↔ d ′ 2 .

Original: We use the following gain function (take d 1 ↔ d ′ 1 as an example):
Revised: We use the following gain function (take d 1 ↔ d ′ 1 as an example):

Original: G AIN (( d, C j ) ↔ ( d ′ , C j ′ ))
Revised: G AIN (( d, C j ) ↔ ( d ′ , C j ′ ))

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: where 1 | C j ′ | (cid:80) d ′′ ∈ C j ′ w d 1 ,d ′′ measures the expected pipeline parallel cost of connecting d 1 withother devices in C j ′ before the swap , and w d 1 ,d 2 is the cost of connecting d 1 with other devices in
Revised: where 1 | C j ′ | (cid:80) d ′′ ∈ C j ′ w d 1 ,d ′′ measures the expected pipeline parallel cost of connecting d 1 withother devices in C j ′ before the swap , and w d 1 ,d 2 is the cost of connecting d 1 with other devices in

Original: C j ′ after the swap , assuming this fast link d 1 ↔ d 2 will now be used for pipeline parallelism.
Revised: C j ′ after the swap , assuming this fast link d 1 ↔ d 2 will now be used for pipeline parallelism.

Original: Just like how Kernighan-Lin Algorithm [38] can be extended to a circular version [28] to swapmultiple nodes beyond a pair, we can also extend our method into a circular one, following procedureas circular KL with our new gain function.
Revised: Just like how Kernighan-Lin Algorithm [38] can be extended to a circular version [28] to swapmultiple nodes beyond a pair, we can also extend our method into a circular one, following procedureas circular KL with our new gain function.

Original: 3.5 Other System Optimizations
Revised: 3.5 Other System Optimizations

Original: We also have some system optimizations to further improve the performance.
Revised: We also have some system optimizations to further improve the performance.

Original: The most importantoptimization involves pipelining of communications and computations.
Revised: The most importantoptimization involves pipelining of communications and computations.

Original: We divide each stage in thepipeline into three slots: a receiving slot, a computation slot, and a sending slot.
Revised: We divide each stage in thepipeline into three slots: a receiving slot, a computation slot, and a sending slot.

Original: The receiving slot ofstage j needs to build connections to receive activations from the stage j − 1 in forward propagationand to receive gradients of activations from stage j +1 .
Revised: The receiving slot ofstage j needs to build connections to receive activations from the stage j − 1 in forward propagationand to receive gradients of activations from stage j +1 .

Original: The computation slot handles the computationin forward and backward propagation.
Revised: The computation slot handles the computationin forward and backward propagation.

Original: Symmetric to the receiving slot, the sending slot of stage jneeds to build connections to send activations to stage j + 1 in the forward propagation and sendgradients of activations to stage j − 1 in the backward propagations.
Revised: Symmetric to the receiving slot, the sending slot of stage jneeds to build connections to send activations to stage j + 1 in the forward propagation and sendgradients of activations to stage j − 1 in the backward propagations.

Original: These three slots are assigned tothree CUDA streams so that they will be further pipelined efficiently; as a result, communication willoverlap with computation.
Revised: These three slots are assigned tothree CUDA streams so that they will be further pipelined efficiently; as a result, communication willoverlap with computation.

Original: In the decentralized scenario (communication bound), computation can befully hidden inside the communication time.
Revised: In the decentralized scenario (communication bound), computation can befully hidden inside the communication time.

Original:  Evaluation
Revised:  Evaluation

Original: We demonstrate that our system can speed up foundation model training in decentralized setting.
Revised: We demonstrate that our system can speed up foundation model training in decentralized setting.

Original: Specifically, (1) We show that our system is 4.8 × faster, in end-to-end runtime, than the state-of-theart systems (Megatron and Deepspeed) training GPT3-1.3B in world-wide geo-distributed setting.
Revised: Specifically, (1) We show that our system is 4.8 × faster, in end-to-end runtime, than the state-of-theart system Megatron training GPT3-XL in world-wide geo-distributed setting.

Original: Surprisingly, it is only 1 .
Revised: Surprisingly, it is only1 .

Original: 7 − 2 .
Revised: 7 − 2 .

Original: 3 × slower than these systems in data centers.
Revised: 3 × slower than Megatron in data centers.

Original: This indicates that wecan bridge the gap between decentralized and data center training (up to 100 × slower networks)through scheduling and system optimization; (2) We demonstrate the necessity of our schedulerthrough an ablation study.
Revised: This indicates that we can bridge the gap betweendecentralized and data center training (up to 100 × slower networks) through scheduling and systemoptimization; (2) We demonstrate the necessity of our scheduler through an ablation study.

Original: We show that with the scheduler, our system is 2.7 × faster in world-widegeo-distributed setting.
Revised: We showthat with the scheduler, our system is 2.7 × faster in world-wide geo-distributed setting.

Original: Experimental Setup To simulate the decentralized setting, we use 8 different AWS regions (Oregon,
Revised: Experimental Setup To simulate the decentralized setting, we use 8 different AWS regions (Oregon,

Original: Virginia, Ohio, Tokyo, Seoul, London, Frankfurt, and Ireland) and measure the latency and bandwidthbetween these regions (we consider the bandwidth that we can realistically obtain using NCCL and
Revised: Virginia, Ohio, Tokyo, Seoul, London, Frankfurt, and Ireland) and measure the latency and bandwidthbetween these regions (we consider the bandwidth that we can realistically obtain using NCCL and

Original: UDP hole punching between these regions).
Revised: UDP hole punching between these regions).

Original: Given these measurements, we use 64 Tesla V100 GPUsand control their pairwise communication latency and bandwidth for five different cases:
Revised: Given these measurements, we use 64 Tesla V100 GPUsand control their pairwise communication latency and bandwidth for five different cases:

Original: Case 1.
Revised: Case 1.

Original: Data center on demand.
Revised: Data center on demand.

Original: This is a standard setting that a user can obtain 
Revised: This is a standard setting that a user can obtain to train foundationmodels.

Original: to train foundation

Original: models.

Original: we use 8 AWS p3.16xlarge nodes (each with 8 V100 GPUs); the intra-node connection is NVLink of 300 GB/s bi-directional bandwidth (150 GB/s unidirectional) and the inter-node connection has a bandwidth of 25 Gbps.
Revised: we use 8 AWS p3.16xlarge nodes (each with 8 V100 GPUs); the intra-node connectionhas a bandwidth of 100 Gbps, and the inter-node connection has a bandwidth of 25 Gbps.

Original: We do not manually control latency and bandwidth here.
Revised: We do notmanually control latency and bandwidth in this case.

Original: Case 2.
Revised: Case 2.

Original: Data center spot instances.
Revised: Data center spot instances.

Original: Spot GPUs are cheaper in a data center, but can be located on different types of machine.
Revised: Spot GPUs are cheaper in a data center, but can be located ondifferent types of machine.

Original: In this case, we rent 4 AWS p3.8xlarge nodes (each with 4 V100)
Revised: In this case, we rent 4 AWS p3.8xlarge nodes (each with 4 V100)

Original: and32 p3.2xlarge nodes (each with 1 V100); the intra- p3.8xlarge node connection has a bandwidthof 100 Gbps, and the inter-node connection has a bandwidth of 10 Gbps.
Revised: and32 p3.2xlarge nodes (each with 1 V100) ; the intra- p3.8xlarge node connection has a bandwidthof 100 Gbps, and the inter-node connection has a bandwidth of 10 Gbps.

Original: We do not manually controllatency and bandwidth in this case.
Revised: We do not manually control latency and bandwidth in this case.

Original: Case 3.
Revised: Case 3.

Original: Multiple Data Centers.
Revised: Multiple Data Centers.

Original: We consider two organizations, one in Ohio and another in Virginia,each organization contributes 32 V100 GPUs; within each organization, the bandwidth is 10 Gbps,and connections cross different campuses have a delay of 10 ms and bandwidth of 1.12 Gbps.
Revised: We consider two organizations, one in Ohio and another in Virginia,each organization contributes 32 V100 GPUs; within each organization, the bandwidth is 10 Gbps,and connections cross different campuses have a delay of 10 ms and bandwidth of 1.12 Gbps.

Original: Case 4.
Revised: Case 4.

Original: Regional geo-distributed.
Revised: Regional geo-distributed.

Original: We consider individual GPUs cross four different regions in US(California, Ohio, Oregon, and Virginia) ; within each region, the delay is 5 ms and bandwidth is [-
 Gbps;-] cross different regions, the delay is 10 ∼ 70 ms and the bandwidth is 1 .
Revised: We consider individual GPUs cross four different regions in US(California, Ohio, Oregon, and Virginia) ; within each region, the delay is 5 ms and bandwidth is [-
 Gbps;-]2Gbps; cross different regions, the delay is 10 ∼ 70 ms and the bandwidth is 1 .

Original: ∼ 1 . 3 Gbps.
Revised: ∼ 1 . 3 Gbps.

Original: C a s e
Revised: C a s e

Original: D a t a C e n t e r onD
Revised: D a t a C e n t e r on

Original: [Figure] nd
Revised: D e m a nd

Original: C a s e
Revised: C a s e

Original: D a t a C e n t e r S po t I n s t a n ce s
Revised: D a t a C e n t e r S po t I n s t a n ce s

Original: C a s e
Revised: C a s e

Original: M u l t i p l e
Revised: M u l t i p l e

Original: D a t a C e n t e rs
Revised: D a t a C e n t e rs

Original: C a s e
Revised: C a s e

Original: W o r l d - w i d e D i s t
Revised: W o r l d - w i d e D i s t

Original: r
Revised: r

Original: i bu t e d
Revised: i bu t e d

Original: C a s e
Revised: C a s e

Revised: R

Revised: [Figure]

Revised: i

Original: [Figure]
Revised: [Figure]

Original: i on a l D i s t
Revised: a l D i s t

Original: r
Revised: r

Original: i bu t e d 
Revised: i [Figure] t e d [Figure]

Revised: L

Original: [Figure]
Revised: [Figure]

Original: Case 5.
Revised: Case 5.

Original: World-wide geo-distributed.
Revised: World-wide geo-distributed.

Original: We consider individual GPUs cross eight different regionsworld-wide (Oregon, Virginia, Ohio, Tokyo, Seoul, London, Frankfurt, and Ireland); within eachregion, the delay is 5 ms and bandwidth is 2 Gbps; cross different regions, the delay is 10 ∼ 250 msand the bandwidth is 0 .
Revised: We consider individual GPUs cross eight different regionsworld-wide (Oregon, Virginia, Ohio, Tokyo, Seoul, London, Frankfurt, and Ireland); within eachregion, the delay is 5 ms and bandwidth is 2 Gbps; cross different regions, the delay is 10 ∼ 250 msand the bandwidth is 0 .

Original: 3 ∼ 1 . 3 Gbps.
Revised: 3 ∼ 1 . 3 Gbps.

Original: Metrics and Model Architecture.
Revised: Metrics and Model Architecture.

Original: Since we do not introduce any optimizations that might changethe computation or convergence, we can compare all methods by its throughput, we can compareall systems by the total number of floating point operations per second (PFLOPS), which is inverseproportional to the runtime of each iteration (which we show in Appendix).
Revised: Since we do not introduce any optimizations that might changethe computation or convergence, we can compare all methods by its throughput, we can compareall systems by the total number of floating point operations per second (PFLOPS), which is inverseproportional to the runtime of each iteration (which we show in Appendix).

Original: We use the standard
Revised: We use the standard

Original: GPT3-1.3B architecture [2], while also benchmarked different number of layers { 24 , 32 , 40 } , andbatch sizes { 1024 , 2048 , 4096 } .
Revised: GPT3-XL architecture [2], while also benchmarked different number of layers { 24 , 32 , 40 } , andbatch sizes { 1024 , 2048 , 4096 } .

Original: Tuning of Megatron and Deepspeed.
Revised: Tuning of Megatron.

Original: We did a careful grid searchof different parallelism settings  and report the optimal results in each case—in Case 1, the optimalsetting includes tensor model parallelism in Megatron and ZeRO-S3 in Deepspeed; in all other cases,the optimal settings are based on pipeline and data parallelism.
Revised: When comparing with Megatron, we did acareful grid search of different parallelism settings in Megatron and report the optimal results in eachcase—in Case 1, the optimal setting includes tensor model parallelism; all other cares the optimalsettings are based on pipeline and data parallelism.

Original: We discuss more details in Appendix.
Revised: We discuss more details in Appendix.

Original: 4.1 Results
Revised: 4.1 Results

Original: [Figure]

Original: End-to-end Comparison.
Revised: End-to-end Comparison.

Original: Figure 3(c) shows the end-to-end comparison  in terms of averaged PFLOPS achieved across different settings and different batch sizes and number of layers.
Revised: Figure 3(c) shows the end-to-end comparison between Megatron and oursystem in terms of averaged PFLOPS achieved across different settings and different batch sizes andnumber of layers.

Original: In theworld-wide geo-distributed cases, we achieve an 4 . 8 × speedup of Megatron, ( 3 . 6 × speedup of Deepspeed).
Revised: From the world-wide geo-distributed cases, we observe that our system achieves an4 . 8 × speedup, as compared with Megatron.

Original: While in all other cases, our system can be 1 . 2 − 2 .
Revised: While in all other cases, our system can be 1 . 2 − 2 .

Original: 5 × faster.
Revised: 5 ×faster.

Original: If we compare our systemin Case 5 (world-wide geo-distributed) and Megatron/Deepspeed in Case 1 (data center on demand),it is exciting to see that the performance slowdown caused by decentralization is only 1 .
Revised: If we compare our system in Case 5 (world-wide geo-distributed) and Megatron in Case 1 (datacenter on demand), it is exciting to see that the performance slowdown caused by decentralization isonly 1 .

Original: 7 − 3 .
Revised: 7 − 3 .

Original: 5 × !
Revised: 5 × !

Original: This illustrates the great potential of decentralized training for foundation models.
Revised: This illustrates the great potential of decentralized training for foundation models.

Original: Additionally, Figure 3(c) illustrates another interesting behavior pattern.
Revised: Additionally, Figure 3(c) illustrates another interesting behavior pattern.

Original: As increasing the batch size does notincreases the communication cost of data parallelism and increasing # layers per device does notincreases the communication cost of pipeline parallelism, with a larger batch size and a deeper model,the gap between centralized Megatron/Deepspeed and our decentralized system is even smaller.
Revised: As increasing the batch sizedoes not increases the communication cost of data parallelism and increasing # layers per device doesnot increases the communication cost of pipeline parallelism, with a larger batch size and a deepermodel, the gap between centralized Megatron and our decentralized system is even smaller.

Original: [Figure]
Revised: [Figure]

Original: Effectiveness of Scheduler.
Revised: Effectiveness of Scheduler.

Original: To evaluate the effectiveness of the scheduler, we disable it and use a random assignment in all cases and the results are alsoillustrated in Figure 3(c).
Revised: To evaluate the effectiveness of the scheduler, we disable it and use a randomassignment in all cases and the results are also illustratedin Figure 3(c).

Original: We see that with our scheduler provides up to 2 .
Revised: We see that with our scheduler provides upto 2 .

Original: 7 × speeds up.
Revised: 7 × speeds up.

Original: To evaluate ourlocal search strategy, we also compare our schedulerwith a scheduler that uses the standard KernighanLin algorithm for local search, illustrated in Figure4.
Revised: To evaluate our local search strategy,we also compare our scheduler with a scheduler that usesthe standard Kernighan-Lin algorithm for local search,illustrated in Figure 4 (a - e).

Original: We see that, while both outperform random, ourcarefully designed local search strategy significantlyoutperforms Kernighan-Lin.
Revised: We see that, while bothoutperform random, our carefully designed local searchstrategy significantly outperforms Kernighan-Lin.

Original:  Related Work
Revised:  Related Work

Original: Foundation models.
Revised: Foundation models.

Original: Foundation models[1] refer to models that are trained on large-scale data andcan be adapted (e.g., fine-tuned) to a wide range of downstream tasks.
Revised: Foundation models[1] refer to models that are trained on large-scale data andcan be adapted (e.g., fine-tuned) to a wide range of downstream tasks.

Original: Current examples include
Revised: Current examples include

Original: BERT [39], GPT-3 [2], and CLIP[40].
Revised: BERT [39], GPT-3 [2], and CLIP[40].

Original: Foundation models are usually trained in a data center, wherethe connection between GPUs is fast and homogeneous.
Revised: Foundation models are usually trained in a data center, wherethe connection between GPUs is fast and homogeneous.

Original: ML infrastructures such as Megatron[4]and ZeRO[10, 11] have been proposed to distribute the training of these foundation models in a datacenter.
Revised: ML infrastructures such as Megatron[4]and ZeRO[10, 11] have been proposed to distribute the training of these foundation models in a datacenter.

Original: Megatron uses AllReduce to synchronize activations in tensor model parallelism; ZeROadopts ScatterGather to dispatch sharded parameters for layer-wise data parallelism.
Revised: Megatron uses AllReduce to synchronize activations in tensor model parallelism; ZeROadopts ScatterGather to dispatch sharded parameters for layer-wise data parallelism.

Original: However,such collective communication paradigms would cause serious performance problems with slow andheterogeneous connections (see Appendix for detailed discussions).
Revised: However,such collective communication paradigms would cause serious performance problems with slow andheterogeneous connections (see Appendix for detailed discussions).

Original: Decentralized optimization .
Revised: Decentralized optimization .

Original: Decentralized training is first proposed within the scope of data parallelism, where each worker only synchronizes gradients with its neighbors (instead of all workers) toremove the latency bottleneck [17, 41, 16, 42, 43, 44].
Revised: Decentralized training is first proposed within the scope of data parallelism, where each worker only synchronizes gradients with its neighbors (instead of all workers) toremove the latency bottleneck [17, 41, 16, 42, 43, 44].

Original: Recently, [20] has also modified the implementation of data parallelism to support training in an open collaborative environment.
Revised: Recently, [45] has also modified the implementation of data parallelism to support training in an open collaborative environment.

Original: Varuna [45]is released by Microsoft to support the training of GPT models in spot instances from a cloud serviceprovider, which has the potential to be extended to the open collective scenario, but there is limitedconsideration with respect to the challenges of heterogeneous connections.
Revised: Varuna [46]is released by Microsoft to support the training of GPT models in spot instances from a cloud serviceprovider, which has the potential to be extended to the open collective scenario, but there is limitedconsideration with respect to the challenges of heterogeneous connections.

Original: Volunteer computing .
Revised: Volunteer computing .

Original: Distributing computationally intensive tasks over an open collaborativeenvironment has been advocated for a few decades since the development of BOINC[46]; forexample, the folding@home project [13] has been running simulations about protein dynamics onvolunteers’ personal computers for more than 20 years.
Revised: Distributing computationally intensive tasks over an open collaborativeenvironment has been advocated for a few decades since the development of BOINC[47]; forexample, the folding@home project [13] has been running simulations about protein dynamics onvolunteers’ personal computers for more than 20 years.

Original: Recently, the learning@home project[19]starts to consider training of mixture-of-expert transformers in such a volunteer computing setting.
Revised: Recently, the learning@home project[19]starts to consider training of mixture-of-expert transformers in such a volunteer computing setting.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we probe the opportunity to train foundation models via a decentralized training regimewith devices connected over a heterogeneous network.
Revised: In this paper, we probe the opportunity to train foundation models via a decentralized training regimewith devices connected over a heterogeneous network.

Original: We propose an effective scheduling algorithm to assign tasklets from the foundation model pre-train computation.
Revised: We propose an effective scheduling algorithmto assign tasklets from the foundation model pre-train computation.

Original: Empirical studies suggest that, in the worldwide geo-distributed scenario, our proposed scheduling algorithm enables a 4 . 8 × speed- up compared to prior state-of-the-art training systems.
Revised: Empirical studies suggest that, inthe worldwide geo-distributed scenario, our proposed scheduling algorithm enables a 4 . 8 × speedup compared to prior state-of-the-art training systems.

Original: We believe that the decentralization anddemocratization of the training of FMs can shift the balance of power positively, but also necessitate new governance structures to help ensure the responsible development and deployment of FMs
Revised: We believe that the decentralization anddemocratization of the training of FMs can shift the balance of power positively, but also necessitatenew governance structures to help ensure the responsible development and deployment of FMs.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A] • Did you include the license to the code and datasets?
Revised: [N/A]• Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] In lines 122-124 of the Problem Formulation, we highlight the core limitations and simplifying assumptions.
Revised: [Yes] In lines 122-124 of the ProblemFormulation, we highlight the core limitations and simplifying assumptions.(c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] In lines 364-366 of the Conclusion, we discuss the potential negative impact of decentralization and democratization, noting that while there are positives, this can also lead to acceler- ation and lack of control.
Revised: [Yes] In lines364-366 of the Conclusion, we discuss the potential negative impact of decentralizationand democratization, noting that while there are positives, this can also lead to acceleration and lack of control.

Original: We look forward to actively engaging with the community on questions of governance.
Revised: We look forward to actively engaging with the communityon questions of governance.(d)

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A](b)

Original: (b) Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] We fully follow the standard settings for GPT3-1.3B training.
Revised: [Yes] We fully follow the standard settings for GPT3-XL training.

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] We repeated all the experiments for at least three times
Revised: [Yes] We repeated all the experiments for at least three times

Original: and reported the average in the paper.
Revised: and reported the average in the paper.

Original: Details are reported in Appendix.
Revised: Details are reported in Appendix.

Original: (d) Did you include the total amount of compute and the type of resources used (e g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] Our systems was built on PyTorch andCuPy and was evaluated using publicly available datasets.
Revised: [Yes] Our systems was built on PyTorch and CuPy and was evaluated using publicly available abstracts.

Original: All of them use open-source
Revised: All of them use open-source

Original: licenses and can be used for non-commercial educational purposes.
Revised: licenses and can be used for non-commercial educational purposes.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects..

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A] [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]

Original: Board (IRB) approvals, if applicable?

Original: [N/A] [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?

Original: [N/A] [Table]
Revised: [Table]

Original: A Social Impact of Decentralized Training

Original: In this paper, we find that decentralized training shows great potential for foundation models — sucha technique would lead to significant positive social impacts.

Original: For example, decentralized trainingcan utilize more inexpensive computational resources, which can significantly reduce the budget forthe training of foundation models.

Original: This would increase accessibility of foundation models for smallresearch and commercial institutions.

Original: In fact, if the expense would be significantly reduced, largeorganizations would also get some benefit from adopting such technology.

Original: On the other hand, wealso notice that decentralization and democratization can also lead to a lack of control of cheapercompute resources and accelerate the risks of foundation models [1].

Original: We look forward to activelyengaging with the community on governance questions.

Original: B Limitation and Future Work

Original: There are still some limitations of the current approach that could be explored further in future work.

Original: First, we assume a homogeneous compute GPU resource in the scheduling algorithm, and in practice,different types of GPU could join the training computations.

Original: A native extension of the currentsolution could be to further split the tasklets into smaller pieces and to assign different numbers ofpieces in different types of GPUs considering their memory budget and compute power as constraints.

Original: However, there are many opportunities for further improvement.

Original: Secondly, there are still lots of room for the improvement of our scheduling algorithm, and strengthenour argument about the end-to-end speedup.

Original: In this paper, we hope to first provide some positiveinsights to the community that the potential of decentralized training can make it deployable forgiant foundation models.

Original: On the other hand, there are lots of recent advances in system optimizationthat could lead to further improvement.

Original: For example, exploring recent advances in reinforcementlearning [47] to solve our scheduling algorithm would be an interesting future direction.

Original: We believethat any improvement there can only improve the decentralized performance, which is consistentwith the central message we try to share in this paper.

Original: Last but not least, there are still some important open questions on the system side to handle thedynamics in decentralized environments.

Original: For example, some mechanism should be necessary tohandle the dynamic join and leave of GPU nodes.

Original: On the other hand, failure happens more frequentlyin the decentralized environment, as fault tolerance should be considered for deployment, we believethat the current strategy such as checkpointing the model periodically could be adopted for thisproblem, but there would be some suitable solutions for the decentralized training runtime.

Original: C Anatomy of the Current ML Systems for Foundation Model Training

Original: Training foundation models [1] is a challenging task due to the enormous scale of such models —even the most powerful GPU cannot hold a complete copy of parameters for such models [48].

Original: Thus,one cannot train such a model without distribution or using vanilla data parallelism.

Original: Two popular approaches have been proposed to distribute the training of such foundation models in adata center:

Original: • Megatron [4] distributes training by combining its proposed tensor model parallelism with pipelineparallelism [49, 22] and data parallelism [50].

Original: The tensor model parallelism partitions individuallayers across a group of workers and must run one AllReduce for the output activations of eachlayer in forward propagation and one AllReduce for the corresponding gradients in backwardpropagation for each layer.• ZeRO [10] can be viewed as an effective optimization for data parallelism.

Original: The most effective modeis called ZeRO stage-3 from the Deepspeed implementation [5], and the equivalent implementation

Original: is known as Fully Sharded Data Parallelism (FSDP) from Fairscale [6]).

Original: In this mode, the parameteris sharded among all workers — in forward propagation, each worker conducts one AllGather tocollect the parameters demanded for the current layer and discard the parameter after the forwardcomputation; in backward propagation, each worker uses one AllGather to collect the parameteragain and run one ReduceScatter to synchronize the gradients of this layer after the backwardcomputation.

Original: Both Megatron and ZeRO take a heavy usage of collective communication paradigms [51], whichleads to two fundamental problems when it is deployed with heterogeneous and slow connections:

Original: • Demanding high bandwidth connections.

Original: Both Megatron and ZeRO require high bandwidth connec-

Original: tions for collective communications, since the compute cores are idled during communication slots.

Original: As long as communication takes an increasing share (due to lower bandwidth) of the execution

Original: time

Original: the hardware efficiency drops dramatically.

Original: In fact, tensor model parallelism is recommended

Original: only within a single DGX server equipped with high-bandwidth NVLinks [48].

Original: • Sensitive to straggler.

Original: The design and implementation of state-of-the-art collective communicationlibraries, e

Original: g., NCCL [52], assume highly homogeneous connections within a data center, thusthere is not sufficient robustness to handle the straggler among workers due to the heterogeneity of

Original: the open collective runtime.

Original: Furthermore, the layer-wise usage of collective communications in

Original: both Megatron and ZeRO has intensified this problem.

Original: To bridge the performance gap between the data center and the decentralized open environment,

Original: we

Original: need to rethink the communication paradigms in different parallel strategies.

Original: • Pipeline parallelism is communication efficient.

Original: Pipeline parallelism [49, 22, 53] partitions themodel into multiple stages and a batch into multiple mini-batches, where once a worker finished theforward computation of a micro-batch, this worker will send the activations to the worker running

Original: the next stages; on the other hand, a worker needs to send the gradients of the activation back to

Original: the last stage in the backward propagation.

Original: Notice that pipeline parallelism utilizes point-to-pointcommunications instead of collective paradigms.

Original: As long as one can put an increasing amount of

Original: computation inside a stage, the ratio of communication cost will also drop, leading to more efficient

Original: utilization of compute cores.

Original: 1 On the other hand, pipeline parallelism has its own limitation — onecan only partition a model to a limited number of stages, which cannot scale out to lots of GPUs.

Original: We need to combine pipeline parallelism with data parallelism to scale out the training.

Original: • Scheduling is essential.

Original: The point-to-point communication pattern in pipeline parallelism providesgood opportunities to assign the training procedure on the decentralized environment that utilizesfast links and avoids slow links by a carefully designed scheduler, as presented in Section 3.

Original: D Additional Details of Experimental Evaluation

Original: We enumerate some additional details about our experiments.

Original: D.1 Multiple Execution of the Benchmark

Original: We repeated all the benchmarks of 5 different scenarios listed in Section 4 three times.

Original: For our systemwith scheduler, since the scheduled layout is the same, we simply issued three independent executionsin each scenarios; For our system without scheduler, we used three different random seeds (2022,2023, 2024) to generate three layouts, and issued one execution for each layout in each scenario.

Original: The number in Figure 3 is based on an average of three different runs for each scenario — to avoidvisual confusion, we did not plot the error bar within this line plot.

Original: We also repeated the schedulingalgorithms three times with random seeds (0, 1, 2) to generate scheduled layouts and reported the g

Original: [Figure]

Original: average estimated cost (seconds) in Figure 4.

Original: In Figure 6, we plot runtime of each iteration as a barchart with error bars.

Original: Notice that the variance of all executions in each setting is within 5% .

Original: D.2 Tuning of Megatron and Deepspeed

Original: Megatron.

Original: We carefully tuned Megatron to perform a fair comparison with our system.

Original: As wementioned in Section C. Megatron has three free degrees of parallel strategies: tensor model parallelism , pipeline parallelism , and data parallelism , we note the degrees of these parallel strategiesas D TP , D PP , and D DP respectively.

Original: We conduct a complete grid search of the combinations of thesehyper-parameters in the space of:

Original: [Equation]

Original: and D

Original: [Equation]

Original: =

Original: [Equation]

Original: And we reported the optimal setting as the results for Megatron.

Original: Interestingly, only in Case 1 (datacenter on demand), the optimal setting includes tensor model parallelism (i.e., D TP ̸ = 1 ), where

Original: D TP = 2 , D PP = 4 , D DP = 8 ; in all other scenarios, the optimal setting is D TP = 1 , D PP = 8 , D DP = 8 .

Original: This illustrates that tensor model parallelism is not suitable in slow and heterogeneous settings,consistent with our analysis in Section C. Since Megatron does not include a similar scheduler as itsown components, we use the same random layouts as what we test for our system without scheduler.

Original: Deepspeed.

Original: To run Deepspeed, we start with ZeRO-S3, which is usually viewed as the mostsignificant technical contribution of the Deepspeed system.

Original: Under the same settings, the executiontime of ZeRO-S3 is much longer comparing to both Megatron and our system (See Figure 5).

Original: This is consistent with our analysis in Appendix B. We then try to combine the pipeline parallelimplementation in Deepspeed with its different implementations of data parallelism (e.g., ZeRO-S1, 2 and 3)—it turns out that even the latest version of Deepspeed (0.6.7) only allows ZeRO-S1 tocombine with pipeline parallelism.

Original: We find this combination outperforms ZeRO-S3 in most of thesettings in Case 1 and all settings in Case 2,3,4 and 5.

Original: Notice that results of Deepspeed we report in

Original: Figure 3 is based on the optimal result of these two settings.

Original: D.3 Network Benchmark.

Original: To obtain the network delay and bandwidth between different regions across the world, we rent

Original: AWS instances in 9 different data centers (California, Oregon, Virginia, Ohio, Tokyo, Seoul, London,

Original: [Table][Table]
Revised: [Table]

Original: Frankfurt, and Ireland).

Original: Instead of using AWS VPC, we setup our own VPN (using StrongSwan [54])established on the public IP of these instances—any GPU machine connected to Internet can belinked in the same way.

Original: The strongSwan VPN would expose a private IP associated with a visiblenetwork interface, we can bind the NCCL communication on this network interface.

Original: The delay andbandwidth we obtained for cross-region NCCL connections are summarized in Table 1 for Case 4regional geo-distributed scenario and and Table 2 for Case 5 world-wide geo-distributed scenario.

Original: D.4 Other Presentation of Experimental Results

Original: In Figure 6, we plot runtime of each iteration for each scenario; this is a supplement to Figure 3.

Original: D.5 Deployment on FluidStack

Original: C a s e

Original: D a t a C e n t e r on D e m a nd

Original: C a s e

Original: D a t a C e n t e r S po t I n s t a n ce s

Original: C a s e

Original: M u l t i p l e

Original: D a t a C e n t e rs

Original: C a s e

Original: W o r l dw i d e

Original: D i s t

Original: r

Original: i bu t e d

Original: C a s e

Original: R eg i on a l D

Original: i s t r i bu t e d [Figure]
Revised: [Table]

Original: We believe that having more realistic measurements and an end-to-end run can provide more pervasivestatements for decentralized training.

Original: To this end, we conducted an additional experiment.

Original: We rent 32 A40 GPUs (each with 48GB GPU memory, and 149.7 peak FP16 TFLOPS) from

Original: FluidStack [55], which consists of a group of geo-distributed GPU clusters, in (1) US Mid and (2)

Original: US East.

Original: We get the communication delay and bandwidths between GPUs as below:

Original: [Figure]

Original: • Intra-US Mid: delay 0 .

Original: 5 ± 0 .

Original: 1 ms; bandwidth 10 .

Original: 40 ± 1 .

Original: 11 Gbps;• Intra-US East: delay 0 .

Original: 5 ± 0 .

Original: 1 ms; bandwidth 11 .

Original: 98 ± 1 .

Original: 92 Gbps;• US Mid to US East: delay 21 .

Original: 8 ± 0 .

Original: 3 ms; bandwidth 3 .

Original: 87 ± 1 .

Original: 07 Gbps;• US East to US Mid: delay 21 .

Original: 8 ± 0 .

Original: 3 ms; bandwidth 3 .

Original: 73 ± 1 .

Original: 38 Gbps.

Original: We conducted an end-to-end run of the same training task of GPT3-1.3B without artificially controllingthe bandwidth and latency.

Original: We also explore the training tasks of larger scale GPT3 models, including

Original: GPT3-6.7B, and GPT3-13B with a batch size of 1024.

Original: The performance in terms of the total numberof floating point operations per second (PFLOPS) and runtime of each iteration are illustrated in

Original: Figure 7.

Original: This is a promising result of decentralized training—for GPT3-1.3B model with 40 layersand 4K batch size, we archive 27 .

Original: 4% of the peak FLOPS of the cluster, for GPT3-6.7B and GPT3-13B,we obtain 26 .

Original: 4% and 29 .

Original: 7% respectively.