Original: SWARM P ARALLELISM : T RAINING L ARGE M ODELS
Revised: SWARM P ARALLELISM : T RAINING L ARGE M ODELS

Original: C AN B E S URPRISINGLY C OMMUNICATION -E FFICIENT
Revised: C AN B E S URPRISINGLY C OMMUNICATION -E FFICIENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many deep learning applications benefit from using large models with billions of parameters.
Revised: Many deep learning applications benefit from using large models with billions of parameters.

Original: These models can only be trained with specialized distributed training algorithms that require low-latency and high-bandwidth interconnect.
Revised: These models can only be trained with specialized distributed training algorithms that require low-latency and high-bandwidth interconnect.

Original: As a result, large models are typically trained in dedicated GPU clusters that can be extremely costly to deploy and operate.
Revised: As a result, large models are typically trained in dedicated GPU clusters that can be extremely costly to deploy and operate.

Original: In contrast, there are more affordable distributed training setups, such as using cheap “preemptible” instances or pooling together existing resources from multiple regions.
Revised: In contrast, there are more affordable distributed training setups, such as using cheap “preemptible” instances or pooling together existing resources from multiple regions.

Original: However, both these setups come with unique challenges that make it impractical to train large models using conventional model parallelism.
Revised: However, both these setups come with unique challenges that make it impractical to train large models using conventional model parallelism.

Original: In this work, we carefully analyze these challenges and find configurations where training larger models becomes less communication-intensive .
Revised: In this work, we carefully analyze these challenges and find configurations where training larger models becomes less communication-intensive .

Original: Based on these observations, we propose SWARM Parallelism 1 — a model-parallel training algorithm designed for swarms of poorly connected, heterogeneous unreliable devices.
Revised: Based on these observations, we propose SWARM Parallelism 1 — a model-parallel training algorithm designed for swarms of poorly connected, heterogeneous unreliable devices.

Original: SWARM creates temporary randomized pipelines between available nodes that are rebalanced in case of failure.
Revised: SWARM creates temporary randomized pipelines between available nodes that are rebalanced in case of failure.

Original: To further reduce the network usage of our approach, we develop several compression-aware architecture modifications and evaluate their tradeoffs.
Revised: To further reduce the network usage of our approach, we develop several compression-aware architecture modifications and evaluate their tradeoffs.

Original: Finally, we combine our insights to train a large
Revised: Finally, we combine our insights to train a large

Original: Transformer language model with 1.1B shared parameters ( ≈ 13 B before sharing) on a swarm of preemptible T4 GPUs with less than 400Mb/s network throughput.
Revised: Transformer language model with 1.1B shared parameters ( ≈ 13 B before sharing) on a swarm of preemptible T4 GPUs with less than 400Mb/s network throughput.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: For the past several years, the deep learning community has been growing ever more reliant on large pretrained models.
Revised: For the past several years, the deep learning community has been growing ever more reliant on large pretrained models.

Original: Perhaps the easiest example of this trend is natural language processing, where the parameter count of models grew from hundreds of millions (Vaswani et al., 2017;
Revised: Perhaps the easiest example of this trend is natural language processing, where the parameter count of models grew from hundreds of millions (Vaswani et al., 2017;

Original: Radford et al., 2018;
Revised: Radford et al., 2018;

Original: Devlin et al., 2019) to billions (Narayanan et al., 2021; Rosset; Raffel et al., 2020; Wang &
Revised: Devlin et al., 2019) to billions (Narayanan et al., 2021; Rosset; Raffel et al., 2020; Wang &

Original: Komatsuzaki, 2021; Sun et al., 2021) to hundreds of billions (Brown et al., 2020;
Revised: Komatsuzaki, 2021; Sun et al., 2021) to hundreds of billions (Brown et al., 2020;

Original: Lepikhin et al., 2020; Fedus et al., 2021) with consistent gains in quality (Kaplan et al., 2020).
Revised: Lepikhin et al., 2020; Fedus et al., 2021) with consistent gains in quality (Kaplan et al., 2020).

Original: Likewise, many models in computer vision are reaching the billion-parameter scale (Henighan et al., 2020;
Revised: Likewise, many models in computer vision are reaching the billion-parameter scale (Henighan et al., 2020;

Original: Ramesh et al., 2021; Zhai et al., 2021; Riquelme et al., 2021;
Revised: Ramesh et al., 2021; Zhai et al., 2021; Riquelme et al., 2021;

Original: Dai et al., 2021; Dhariwal & Nichol, 2021).
Revised: Dai et al., 2021; Dhariwal & Nichol, 2021).

Original: At this scale, the model no longer fits into a single accelerator and requires specialized training algorithms that partition model parameters across devices (Krizhevsky et al., 2012; Dean et al., 2012).
Revised: At this scale, the model no longer fits into a single accelerator and requires specialized training algorithms that partition model parameters across devices (Krizhevsky et al., 2012; Dean et al., 2012).

Original: While these model-parallel algorithms use different partitioning strategies, they all share the need to perform intensive device-to-device communication (Narayanan et al., 2019; 2021).
Revised: While these model-parallel algorithms use different partitioning strategies, they all share the need to perform intensive device-to-device communication (Narayanan et al., 2019; 2021).

Original: Furthermore, if a single device fails, it will cause the entire training to break down.
Revised: Furthermore, if a single device fails, it will cause the entire training to break down.

Original: As a result, model-parallel algorithms are typically deployed in dedicated HPC 2 clusters or supercomputers (Shoeybi et al., 2019;
Revised: As a result, model-parallel algorithms are typically deployed in dedicated HPC 2 clusters or supercomputers (Shoeybi et al., 2019;

Original: Rajbhandari et al., 2020; Narayanan et al., 2021).
Revised: Rajbhandari et al., 2020; Narayanan et al., 2021).

Original: This kind of infrastructure is notoriously expensive to build and operate, available only to few wellfunded universities and large corporations (Larrea et al., 2019; Strohmaier et al., 2021; Langston, 2020).
Revised: This kind of infrastructure is notoriously expensive to build and operate, available only to few wellfunded universities and large corporations (Larrea et al., 2019; Strohmaier et al., 2021; Langston, 2020).

Original: Most researchers, especially in developing nations, simply cannot afford to run the necessary experiments to properly evaluate their ideas.
Revised: Most researchers, especially in developing nations, simply cannot afford to run the necessary experiments to properly evaluate their ideas.

Original: This ultimately limits the scientific progress for many important research areas, such as solving NLP problems in “non-mainstream” languages.
Revised: This ultimately limits the scientific progress for many important research areas, such as solving NLP problems in “non-mainstream” languages.

Original: Several recent works propose more cost-efficient distributed training strategies that use fleets of temporary “preemptible” instances that can be dynamically allocated in regions with low demand for hardware and electricity, making them 2–10 times cheaper than their dedicated counterparts (Harlap et al., 2017).
Revised: Several recent works propose more cost-efficient distributed training strategies that use fleets of temporary “preemptible” instances that can be dynamically allocated in regions with low demand for hardware and electricity, making them 2–10 times cheaper than their dedicated counterparts (Harlap et al., 2017).

Original: Another solution is to train in “collaborations” by pooling together preexisting resources or using the help of volunteers (Diskin et al., 2021; Atre et al., 2021; Ryabinin & Gusev, 2020).
Revised: Another solution is to train in “collaborations” by pooling together preexisting resources or using the help of volunteers (Diskin et al., 2021; Atre et al., 2021; Ryabinin & Gusev, 2020).

Original: However, training in either of those setups requires specialized algorithms that can adapt to the changing number of workers, utilize heterogeneous devices and recover from hardware and network failures.
Revised: However, training in either of those setups requires specialized algorithms that can adapt to the changing number of workers, utilize heterogeneous devices and recover from hardware and network failures.

Original: While there are several practical algorithms for unreliable hardware (Kijsipongse et al., 2018; Lin et al., 2020; Ryabinin et al., 2021), they can only train relatively small models that fit into the memory of the smallest device .
Revised: While there are several practical algorithms for unreliable hardware (Kijsipongse et al., 2018; Lin et al., 2020; Ryabinin et al., 2021), they can only train relatively small models that fit into the memory of the smallest device .

Original: This limits the practical impact of cost-efficient strategies, as most computationally demanding workloads typically train models with billions of parameters.
Revised: This limits the practical impact of cost-efficient strategies, as most computationally demanding workloads typically train models with billions of parameters.

Original: In this work, we aim to find a practical way of training large models using unreliable heterogeneous devices and slow interconnect.
Revised: In this work, we aim to find a practical way of training large models using unreliable heterogeneous devices and slow interconnect.

Original: Our contributions can be summarized as such:
Revised: Our contributions can be summarized as such:

Original: • We carefully analyze the existing model-parallel training techniques and formulate the “Square-Cube Law” of distributed training: a counterintuitive observation that, for some methods, training larger models can actually decrease the network overhead .
Revised: • We carefully analyze the existing model-parallel training techniques and formulate the “Square-Cube Law” of distributed training: a counterintuitive observation that, for some methods, training larger models can actually decrease the network overhead .

Original: • We develop SWARM parallelism : a decentralized model-parallel algorithm 3 that replaces traditional pipelines with randomized temporary routing between swarms of peers, allowing it to operate with heterogeneous devices and recover from node failures.
Revised: • We develop SWARM parallelism : a decentralized model-parallel algorithm 3 that replaces traditional pipelines with randomized temporary routing between swarms of peers, allowing it to operate with heterogeneous devices and recover from node failures.

Original: To the best of our knowledge, this is the first general-purpose algorithm capable of billion-scale training on heterogeneous unreliable devices with slow interconnect.
Revised: To the best of our knowledge, this is the first general-purpose algorithm capable of billion-scale training on heterogeneous unreliable devices with slow interconnect.

Original: • We study compression-aware architecture variants that utilize bottlenecks and activation sparsity to significantly reduce the network bandwidth requirements.
Revised: • We study compression-aware architecture variants that utilize bottlenecks and activation sparsity to significantly reduce the network bandwidth requirements.

Original: We evaluate the performance and quality tradeoffs of two different methods and find that our best compression technique, that uses the maxout activation function for compression, reduces bandwidth requirements by 4x while degrading performance by only 4–8%.
Revised: We evaluate the performance and quality tradeoffs of two different methods and find that our best compression technique, that uses the maxout activation function for compression, reduces bandwidth requirements by 4x while degrading performance by only 4–8%.

Original: • Combining SWARM parallelism and compression-aware techniques, we demonstrate that it is possible to train a billion-scale Transformer language model with high training throughput on preemptible low-power T4 GPUs and < 400 Mb/s network.
Revised: • Combining SWARM parallelism and compression-aware techniques, we demonstrate that it is possible to train a billion-scale Transformer language model with high training throughput on preemptible low-power T4 GPUs and < 400 Mb/s network.

Original:  B ACKGROUND & R ELATED W ORK 2. M ODEL - PARALLEL TRAINING
Revised:  B ACKGROUND & R ELATED W ORK 2. M ODEL - PARALLEL TRAINING

Original: Deep learning community came up with multiple algorithms for training large models.
Revised: Deep learning community came up with multiple algorithms for training large models.

Original: Most of them work by somehow dividing the model between multiple workers, which is known as model parallelism.
Revised: Most of them work by somehow dividing the model between multiple workers, which is known as model parallelism.

Original: The exact way in which these algorithms divide the model significantly affects their training performance and the maximum model size they can support.
Revised: The exact way in which these algorithms divide the model significantly affects their training performance and the maximum model size they can support.

Original: Traditional model parallelism.
Revised: Traditional model parallelism.

Original: Historically, the first general strategy for training large models was to assign each device to compute a subset of each model layer (e.g. a subset of neurons), then communicate activations between each other (Krizhevsky et al., 2012); see surveys by Ben-Nun &
Revised: Historically, the first general strategy for training large models was to assign each device to compute a subset of each model layer (e.g. a subset of neurons), then communicate activations between each other (Krizhevsky et al., 2012); see surveys by Ben-Nun &

Original: Hoefler (2019) and Tang et al.
Revised: Hoefler (2019) and Tang et al.

Original: for a more comprehensive overview.
Revised: for a more comprehensive overview.

Original: Since each device stores a fraction of layer parameters, this technique can train extremely wide layers that would not fit into a single GPU.
Revised: Since each device stores a fraction of layer parameters, this technique can train extremely wide layers that would not fit into a single GPU.

Original: However, applying traditional model parallelism to deep models comes with a significant performance penalty, as it requires all-to-all communication after each layer.
Revised: However, applying traditional model parallelism to deep models comes with a significant performance penalty, as it requires all-to-all communication after each layer.

Original: As a result, while intra-layer parallelism schemes are still widely used (Shazeer et al., 2018; Rajbhandari et al., 2020), they are usually applied within one physical server in combination with other strategies (Krizhevsky, 2014;
Revised: As a result, while intra-layer parallelism schemes are still widely used (Shazeer et al., 2018; Rajbhandari et al., 2020), they are usually applied within one physical server in combination with other strategies (Krizhevsky, 2014;

Original: Chilimbi et al., 2014;
Revised: Chilimbi et al., 2014;

Original: Jia et al., 2019; Narayanan et al., 2021).
Revised: Jia et al., 2019; Narayanan et al., 2021).

Original: Pipeline parallelism circumvents the need for expensive all-to-all communication by assigning each device with one or several full layers (Huang et al., 2019).
Revised: Pipeline parallelism circumvents the need for expensive all-to-all communication by assigning each device with one or several full layers (Huang et al., 2019).

Original: During the forward pass, each stage applies its subset of layers to the inputs supplied by the previous stage, then sends the outputs of the final layer to the next stage.
Revised: During the forward pass, each stage applies its subset of layers to the inputs supplied by the previous stage, then sends the outputs of the final layer to the next stage.

Original: For backpropagation, this process is reversed, with each pipeline stage passing the gradients to the same device that previously supplied it with input activations.
Revised: For backpropagation, this process is reversed, with each pipeline stage passing the gradients to the same device that previously supplied it with input activations.

Original: /swarm
Revised: /swarm

Original: To better utilize the available devices, the pipeline must process multiple micro-batches per step, allowing each stage to run in parallel on a different batch of inputs.
Revised: To better utilize the available devices, the pipeline must process multiple micro-batches per step, allowing each stage to run in parallel on a different batch of inputs.

Original: In practice, the number of microbatches is limited by the available device memory, which results in the reduced device utilization when processing first and last micro-batches, which is known as the “bubble” overhead (Huang et al., 2019).
Revised: In practice, the number of microbatches is limited by the available device memory, which results in the reduced device utilization when processing first and last micro-batches, which is known as the “bubble” overhead (Huang et al., 2019).

Original: To combat this issue, subsequent studies propose using activation checkpointing, interleaved scheduling, and even asynchronous training (Narayanan et al., 2019; 2021; Huang et al., 2019;
Revised: To combat this issue, subsequent studies propose using activation checkpointing, interleaved scheduling, and even asynchronous training (Narayanan et al., 2019; 2021; Huang et al., 2019;

Original: Shoeybi et al., 2019; Yang et al., 2019).
Revised: Shoeybi et al., 2019; Yang et al., 2019).

Original: Another limitation of pipeline parallelism is that it needs to evenly split the compute load across the pipeline stages (Narayanan et al., 2019), which makes it difficult to apply when model layers have uneven computational cost.
Revised: Another limitation of pipeline parallelism is that it needs to evenly split the compute load across the pipeline stages (Narayanan et al., 2019), which makes it difficult to apply when model layers have uneven computational cost.

Original: Finally, there are two types of algorithms that can be used to train large models: data parallelism with dynamic parameter loading (Rajbhandari et al., 2020;
Revised: Finally, there are two types of algorithms that can be used to train large models: data parallelism with dynamic parameter loading (Rajbhandari et al., 2020;

Original: Pudipeddi et al., 2020;
Revised: Pudipeddi et al., 2020;

Original: Ren et al., 2021) and layer-dependent algorithms such as distributed Mixture-of-Experts (Jacobs et al., 1991; Shazeer et al., 2017;
Revised: Ren et al., 2021) and layer-dependent algorithms such as distributed Mixture-of-Experts (Jacobs et al., 1991; Shazeer et al., 2017;

Original: Lepikhin et al., 2020; Fedus et al., 2021).
Revised: Lepikhin et al., 2020; Fedus et al., 2021).

Original: We review these algorithms in Appendix A.
Revised: We review these algorithms in Appendix A and compare our approach with pipeline parallelism and offloading in Appendices E and F.

Original: 2. D ISTRIBUTED T RAINING O UTSIDE HPC
Revised: D ISTRIBUTED T RAINING O UTSIDE HPC

Original: The distributed training techniques described in Section 2.1 are designed for clusters of identical devices with rapid and reliable communication, making them a natural fit for the High-Performance
Revised: The distributed training techniques described in Section 2.1 are designed for clusters of identical devices with rapid and reliable communication, making them a natural fit for the High-Performance

Original: Computing (HPC) environment.
Revised: Computing (HPC) environment.

Original: As we discussed earlier, this infrastructure is not always available or affordable.
Revised: As we discussed earlier, this infrastructure is not always available or affordable.

Original: A more cost-efficient alternative is to use “preemptible” instances (Li et al., 2019;
Revised: A more cost-efficient alternative is to use “preemptible” instances (Li et al., 2019;

Original: Zhang et al., 2020; Harlap et al., 2017) or volunteer computing (Diskin et al., 2021; Kijsipongse et al., 2018; Atre et al., 2021; Ryabinin & Gusev, 2020).
Revised: Zhang et al., 2020; Harlap et al., 2017) or volunteer computing (Diskin et al., 2021; Kijsipongse et al., 2018; Atre et al., 2021; Ryabinin & Gusev, 2020).

Original: However, these environments are more difficult for distributed training, as each machine can leave training abruptly due to hardware failure or spikes in demand.
Revised: However, these environments are more difficult for distributed training, as each machine can leave training abruptly due to hardware failure or spikes in demand.

Original: Furthermore, since there is a limited number of available instances per region, training at scale often requires operating in multiple geographic regions or using multiple instance types.
Revised: Furthermore, since there is a limited number of available instances per region, training at scale often requires operating in multiple geographic regions or using multiple instance types.

Original: To summarize, there are five main limitations for distributed training in non-HPC environments: (1) nodes joining and leaving over time, (2) node and network failures, (3) uneven performance from heterogeneous devices, and, finally, (4) low network bandwidth and (5) high latency between regions.
Revised: To summarize, there are five main limitations for distributed training in non-HPC environments: (1) nodes joining and leaving over time, (2) node and network failures, (3) uneven performance from heterogeneous devices, and, finally, (4) low network bandwidth and (5) high latency between regions.

Original: In order to train with a dynamic number of workers, deep learning practitioners came up with elastic training algorithms (TorchElastic; ElasticHorovod).
Revised: In order to train with a dynamic number of workers, deep learning practitioners came up with elastic training algorithms (TorchElastic; ElasticHorovod).

Original: If one peer leaves or fails midway through training, these algorithms dynamically re-balance the load between the remaining devices and continue the training (Harlap et al., 2017; Ryabinin et al., 2021)
Revised: If one peer leaves or fails midway through training, these algorithms dynamically re-balance the load between the remaining devices and continue the training (Harlap et al., 2017; Ryabinin et al., 2021)

Original: If new devices join midway through training, they download the latest training parameters from their peers and train alongside them.
Revised: If new devices join midway through training, they download the latest training parameters from their peers and train alongside them.

Original: Another line of research tackles distributed training on heterogeneous devices with uneven performance.
Revised: Another line of research tackles distributed training on heterogeneous devices with uneven performance.

Original: One way to solve this problem is to use asynchronous training, where devices compute gradients at their own pace and aggregate them using either a parameter server (Recht et al., 2011; Kijsipongse et al., 2018) or a decentralized communication network (Lian et al., 2017).
Revised: One way to solve this problem is to use asynchronous training, where devices compute gradients at their own pace and aggregate them using either a parameter server (Recht et al., 2011; Kijsipongse et al., 2018) or a decentralized communication network (Lian et al., 2017).

Original: This idea allows asynchronous algorithms to fully utilize each individual device, but may reduce the convergence rate due to using “stale” gradients (Recht et al., 2011; Aji & Heafield, 2019).
Revised: This idea allows asynchronous algorithms to fully utilize each individual device, but may reduce the convergence rate due to using “stale” gradients (Recht et al., 2011; Aji & Heafield, 2019).

Original: Finally, several studies (Li et al., 2020; Ryabinin et al., 2021; Ren et al., 2021; Diskin et al., 2021) propose hybrid techniques that remove some synchronization bottlenecks while maintaining per-iteration convergence.
Revised: Finally, several studies (Li et al., 2020; Ryabinin et al., 2021; Ren et al., 2021; Diskin et al., 2021) propose hybrid techniques that remove some synchronization bottlenecks while maintaining per-iteration convergence.

Original: However, all of these use cases rely on data-parallel training and thus share a fundamental limitation: each computing node must be able to run the entire model.
Revised: However, all of these use cases rely on data-parallel training and thus share a fundamental limitation: each computing node must be able to run the entire model.

Original: In contrast, the most compute-intensive tasks require training models with billions of parameters, which is way outside both GPU and host memory of most low-end GPUs.
Revised: In contrast, the most compute-intensive tasks require training models with billions of parameters, which is way outside both GPU and host memory of most low-end GPUs.

Original: Unlike data-parallel training, the model-parallel algorithms from
Revised: Unlike data-parallel training, the model-parallel algorithms from

Original: Section 2.1 are not redundant, which makes them more vulnerable to hardware and network failures.
Revised: Section 2.1 are not redundant, which makes them more vulnerable to hardware and network failures.

Original: In fact, even a single failed node can cause permanent loss of trained parameters and leave others unable to run forward and backward passes.
Revised: In fact, even a single failed node can cause permanent loss of trained parameters and leave others unable to run forward and backward passes.

Original: To the best of our knowledge, there is only one study about training large models with unreliable devices (Ryabinin & Gusev, 2020), but it supports only
Revised: To the best of our knowledge, there is only one study about training large models with unreliable devices (Ryabinin & Gusev, 2020), but it supports only

Original: Mixture-of-Experts and requires at least 1Gb/s interconnect even in toy experiments.
Revised: Mixture-of-Experts and requires at least 1Gb/s interconnect even in toy experiments.

Original: 2. C OMMUNICATION EFFICIENCY & C OMPRESSION STRATEGIES
Revised: 2. C OMMUNICATION EFFICIENCY & C OMPRESSION STRATEGIES

Original: In this section, we discuss techniques that address training with limited network bandwidth and/or high latency, such as gradient compression or overlapping computation with communication phases.
Revised: In this section, we discuss techniques that address training with limited network bandwidth and/or high latency, such as gradient compression or overlapping computation with communication phases.

Original: Efficient parameter gradient communication.
Revised: Efficient parameter gradient communication.

Original: Data-parallel training requires the synchronization of gradients after each backpropagation pass, which can be costly if the model has many parameters or network bandwidth is limited.
Revised: Data-parallel training requires the synchronization of gradients after each backpropagation pass, which can be costly if the model has many parameters or network bandwidth is limited.

Original: Deep Gradient Compression (Lin et al., 2018) approaches this problem by sparsifying gradients before synchronizing and corrects the momentum to work with sparse updates.
Revised: Deep Gradient Compression (Lin et al., 2018) approaches this problem by sparsifying gradients before synchronizing and corrects the momentum to work with sparse updates.

Original: PowerSGD (Vogels et al., 2019) compresses gradients via factorization and uses error feedback to counterbalance approximation errors over time.
Revised: PowerSGD (Vogels et al., 2019) compresses gradients via factorization and uses error feedback to counterbalance approximation errors over time.

Original: Techniques like 1-bit SGD (Seide et al., 2014), 1-bit Adam (Tang et al., 2021), and 1-bit LAMB (Li et al., 2021) do not synchronize gradients directly, but instead synchronize error-corrected 1-bit local optimizer statistics.
Revised: Techniques like 1-bit SGD (Seide et al., 2014), 1-bit Adam (Tang et al., 2021), and 1-bit LAMB (Li et al., 2021) do not synchronize gradients directly, but instead synchronize error-corrected 1-bit local optimizer statistics.

Original: Dettmers (2015) uses nonlinear 8-bit quantization to compress gradients before communication.
Revised: Dettmers (2015) uses nonlinear 8-bit quantization to compress gradients before communication.

Original: We use this technique in conjunction with compression-aware architectures.
Revised: We use this technique in conjunction with compression-aware architectures.

Original: Besides direct compression of gradients, another effective technique is to use layer sharing (Lan et al., 2020) which reduces the amount of parameter gradients that need to be synchronized by the factor of how many times a layer is reused.
Revised: Besides direct compression of gradients, another effective technique is to use layer sharing (Lan et al., 2020) which reduces the amount of parameter gradients that need to be synchronized by the factor of how many times a layer is reused.

Original: Overlapping communication and computation.
Revised: Overlapping communication and computation.

Original: Model, pipeline, and data parallelism all have specific synchronization points at each layer, stage, and parameter update, respectively, and require communication-intensive transfers of gradients or activities.
Revised: Model, pipeline, and data parallelism all have specific synchronization points at each layer, stage, and parameter update, respectively, and require communication-intensive transfers of gradients or activities.

Original: Usually, model training cannot progress until these transfers are completed.
Revised: Usually, model training cannot progress until these transfers are completed.

Original: One effective way to reduce this cost is to overlap communication with computation and thereby hiding the transfer hiding the synchronization latency.
Revised: One effective way to reduce this cost is to overlap communication with computation and thereby hiding the transfer hiding the synchronization latency.

Original: This can be achieved by combining multiple parallelization techniques (Krizhevsky, 2014; Rajbhandari et al., 2020), by synchronizing parameter gradients layer-by-layer in lockstep with backpropagation (Paszke et al., 2019), or by using pure pipeline parallelism (Huang et al., 2019; Narayanan et al., 2019).
Revised: This can be achieved by combining multiple parallelization techniques (Krizhevsky, 2014; Rajbhandari et al., 2020), by synchronizing parameter gradients layer-by-layer in lockstep with backpropagation (Paszke et al., 2019), or by using pure pipeline parallelism (Huang et al., 2019; Narayanan et al., 2019).

Original: However, pure pipeline parallelism requires many pipeline stages to hide the communication effectively.
Revised: However, pure pipeline parallelism requires many pipeline stages to hide the communication effectively.

Original: To overcome this problem, we develop compression-aware architectures that work well even with relatively few pipeline stages.
Revised: To overcome this problem, we develop compression-aware architectures that work well even with relatively few pipeline stages.

Original: Another approach is to improve the computation/communication ratio instead of overlapping communication.
Revised: Another approach is to improve the computation/communication ratio instead of overlapping communication.

Original: This ratio can be improved by using gradient accumulation (Ott et al., 2018) or large batch sizes (You et al., 2020).
Revised: This ratio can be improved by using gradient accumulation (Ott et al., 2018) or large batch sizes (You et al., 2020).

Original: Model Compression methods.
Revised: Model Compression methods.

Original: A seminal study of compression is work by Han et al.
Revised: A seminal study of compression is work by Han et al.

Original: (2016) which uses a pipeline of pruning, quantization, and Huffman coding to compress neural networks for inference.
Revised: (2016) which uses a pipeline of pruning, quantization, and Huffman coding to compress neural networks for inference.

Original: Courbariaux et al.
Revised: Courbariaux et al.

Original: (2014) explores trade-offs between the numeric precision during training and the final test error.
Revised: (2014) explores trade-offs between the numeric precision during training and the final test error.

Original: Work on 1-bit quantization seeks to trade predictive performance for speedups (Rastegari et al., 2016; Courbariaux et al., 2015).
Revised: Work on 1-bit quantization seeks to trade predictive performance for speedups (Rastegari et al., 2016; Courbariaux et al., 2015).

Original: While extreme compression methods have been used for smaller models and convolutional networks successfully (Qin et al., 2020), quantization of large models or Transformers has been more challenging (Ramesh et al., 2021).
Revised: While extreme compression methods have been used for smaller models and convolutional networks successfully (Qin et al., 2020), quantization of large models or Transformers has been more challenging (Ramesh et al., 2021).

Original:  C OMMUNICATION - EFFICIENT MODEL PARALLELISM 3. T HE SQUARE - CUBE LAW OF DEEP LEARNING
Revised:  C OMMUNICATION - EFFICIENT MODEL PARALLELISM 3. T HE SQUARE - CUBE LAW OF DEEP LEARNING

Original: To better understand the general scaling properties of model parallelism, we must abstract away from the application-specific parameters, such as model architecture, minibatch size, and system design.
Revised: To better understand the general scaling properties of model parallelism, we must abstract away from the application-specific parameters, such as model architecture, minibatch size, and system design.

Original: To that end, we first consider a simplified model of pipeline parallelism.
Revised: To that end, we first consider a simplified model of pipeline parallelism.

Original: Our “pipeline” consists of k stages,
Revised: Our “pipeline” consists of k stages,

Original: each represented by an n × n matrix.
Revised: each represented by an n × n matrix.

Original: Intuitively, the first stage represents input data and all subsequent stages are applying “layers” to that data.
Revised: Intuitively, the first stage represents input data and all subsequent stages are applying “layers” to that data.

Original: This model does not account for specifics of any single application, we can use it do capture the intuition that will later apply to real-world cases.
Revised: This model does not account for specifics of any single application, we can use it do capture the intuition that will later apply to real-world cases.

Original: During “training”, stages iteratively perform matrix multiplication, then send the output to the subsequent pipeline stage over a throughput-limited network.
Revised: During “training”, stages iteratively perform matrix multiplication, then send the output to the subsequent pipeline stage over a throughput-limited network.

Original: These two operations have different scaling properties.
Revised: These two operations have different scaling properties.

Original: The compute time for naive matrix multiplication scales as O ( n 3 ) .
Revised: The compute time for naive matrix multiplication scales as O ( n 3 ) .

Original: While this can be reduced further in theory (Coppersmith & Winograd, 1990; Alman & Williams, 2021), it is only used for very large matrices (Zhang & Gao, 2015; Fatahalian et al., 2004; Huang et al., 2020), the standard for neural networks is to use O ( n 3 ) algorithms.
Revised: While this can be reduced further in theory (Coppersmith & Winograd, 1990; Alman & Williams, 2021), it is only used for very large matrices (Zhang & Gao, 2015; Fatahalian et al., 2004; Huang et al., 2020), the standard for neural networks is to use O ( n 3 ) algorithms.

Original: In turn, the communication phase requires at most O ( n 2 ) time to transfer a batch of activations (or gradients).
Revised: In turn, the communication phase requires at most O ( n 2 ) time to transfer a batch of activations (or gradients).

Original: Therefore, as we increase model size, the computation time grows faster than communication time, regardless of which matrix multiplication algorithm we use.
Revised: Therefore, as we increase model size, the computation time grows faster than communication time, regardless of which matrix multiplication algorithm we use.

Original: We refer to this idea as the square-cube law after the eponymous principle in physics (Galileo, 1638; Allen, 2013).
Revised: We refer to this idea as the square-cube law after the eponymous principle in physics (Galileo, 1638; Allen, 2013).

Original: “input”n×n ⇒ × =
Revised: “input”n×n ⇒ × =

Original: ⇒ × “input”n×n “weight”n×n “output”n×n
Revised: ⇒ × “input”n×n “weight”n×n “output”n×n

Original: O(n 3 ) computation
Revised: O(n 3 ) computation

Original: O(n 2 ) communication
Revised: O(n 2 ) communication

Original: Stage k
Revised: Stage k

Original: Stage k+
Revised: Stage k+

Original: [Figure]
Revised: [Figure]

Original: This principle applies to many real-world neural networks architectures, albeit with some confounding variables.
Revised: This principle applies to many real-world neural networks architectures, albeit with some confounding variables.

Original: In convolutional neural networks (Fukushima, 1980), the computation time scales as
Revised: In convolutional neural networks (Fukushima, 1980), the computation time scales as

Original: O ( BHWC 2 ) and the communication is O ( BHWC ) , where B, H, W and C stand for batch size, height, width and the number of channels.
Revised: O ( BHWC 2 ) and the communication is O ( BHWC ) , where B, H, W and C stand for batch size, height, width and the number of channels.

Original: Recurrent neural networks (Rumelhart et al., 1986; Hochreiter & Schmidhuber, 1995) need O ( BLH 2 ) compute in terms of batch size, sequence length and hidden size, respectively, and O ( BLH ) or O ( BH ) communication, depending on the architecture.
Revised: Recurrent neural networks (Rumelhart et al., 1986; Hochreiter & Schmidhuber, 1995) need O ( BLH 2 ) compute in terms of batch size, sequence length and hidden size, respectively, and O ( BLH ) or O ( BH ) communication, depending on the architecture.

Original: With the same notation, transformers (Vaswani et al., 2017) require O ( BL 2 H ) compute for attention layers, O ( BLH 2 ) compute for feedforward layers, but only O ( BLH ) communication in both cases.
Revised: With the same notation, transformers (Vaswani et al., 2017) require O ( BL 2 H ) compute for attention layers, O ( BLH 2 ) compute for feedforward layers, but only O ( BLH ) communication in both cases.

Original: Based on these observations, we conclude that pipeline parallelism naturally grows more communication-efficient with model size.
Revised: Based on these observations, we conclude that pipeline parallelism naturally grows more communication-efficient with model size.

Original: More precisely, increasing the hidden dimension will reduce the communication load per device per unit of time, making it possible to train the model efficiently with lower network bandwidth and higher latency 4 .
Revised: More precisely, increasing the hidden dimension will reduce the communication load per device per unit of time, making it possible to train the model efficiently with lower network bandwidth and higher latency 4 .

Original: While the exact practical ramifications depend on the use case, Section 4.1 demonstrates some of the larger models trained with pipeline parallelism can already train at peak efficiency with only hundreds of Mb/s bandwidth.
Revised: While the exact practical ramifications depend on the use case, Section 4.1 demonstrates some of the larger models trained with pipeline parallelism can already train at peak efficiency with only hundreds of Mb/s bandwidth.

Original: Curiously, the square-cube principle also applies to intra-layer parallelism (Krizhevsky et al., 2012;
Revised: Curiously, the square-cube principle also applies to intra-layer parallelism (Krizhevsky et al., 2012;

Original: Jacobs et al., 1991), but using this technique at 500Mb/s only becomes feasible for impractically large layer sizes in excess of 65536 units.
Revised: Jacobs et al., 1991), but using this technique at 500Mb/s only becomes feasible for impractically large layer sizes in excess of 65536 units.

Original: Data-parallel training with sharding and/or offloading does not scale as well as its communication time scales the size of model parameters instead of activations.
Revised: Data-parallel training with sharding and/or offloading does not scale as well as its communication time scales the size of model parameters instead of activations.

Original: However, it may be possible to achieve similar scaling with another type of data-parallel training using low-rank gradient approximations (Vogels et al., 2019).
Revised: However, it may be possible to achieve similar scaling with another type of data-parallel training using low-rank gradient approximations (Vogels et al., 2019).

Original: 3. SWARM PARALLELISM
Revised: 3. SWARM PARALLELISM

Original: Traditional pipeline-parallelism can be communication-efficient, but this alone is not enough.
Revised: Traditional pipeline-parallelism can be communication-efficient, but this alone is not enough.

Original: Since each swarm participant has different compute and network capabilities, a pipeline formed out of such devices would be bottlenecked by the single weakest link, i.e. the participant with the smallest training throughput.
Revised: Since each swarm participant has different compute and network capabilities, a pipeline formed out of such devices would be bottlenecked by the single weakest link, i.e. the participant with the smallest training throughput.

Original: As a result, the more powerful nodes along the pipeline would be underutilized due to either lack of inputs or slow subsequent stages.
Revised: As a result, the more powerful nodes along the pipeline would be underutilized due to either lack of inputs or slow subsequent stages.

Original: On top of that, if any single peer fails or leaves training prematurely, it will stall the entire training procedure.
Revised: On top of that, if any single peer fails or leaves training prematurely, it will stall the entire training procedure.

Original: In order to overcome these two challenges, we replace the rigid pipeline structure with randomized temporary “pipelines” formed on the fly during each iteration.
Revised: In order to overcome these two challenges, we replace the rigid pipeline structure with randomized temporary “pipelines” formed on the fly during each iteration.

Original: Each participant can send their outputs to any peer that serves the next pipeline stage.
Revised: Each participant can send their outputs to any peer that serves the next pipeline stage.

Original: Thus, if one peer has significantly more compute than others, it can process inputs from multiple predecessors and distribute its outputs across several weaker peers to maximize utilization.
Revised: Thus, if one peer has significantly more compute than others, it can process inputs from multiple predecessors and distribute its outputs across several weaker peers to maximize utilization.

Original: Furthermore, if a participant leaves during training, its predecessors can reroute their requests to its neighbors, allowing the training to proceed as long as there is at least one active participant at every pipeline stage (we elaborate on this Appendix C).
Revised: Furthermore, if a participant leaves during training, its predecessors can reroute their requests to its neighbors, allowing the training to proceed as long as there is at least one active participant at every pipeline stage (we elaborate on this Appendix C).

Original: The resulting infrastructure consists of multiple consecutive swarms, as depicted in Figure 2.
Revised: The resulting infrastructure consists of multiple consecutive swarms, as depicted in Figure 2.

Original: Peers within one swarm serve the same pipeline stage (i.e. the same subset of model layers).
Revised: Peers within one swarm serve the same pipeline stage (i.e. the same subset of model layers).

Original: During forward pass, peers receive micro-batches of inputs from random predecessors and send activations a random peers in the next stage.
Revised: During forward pass, peers receive micro-batches of inputs from random predecessors and send activations a random peers in the next stage.

Original: For backward pass, peers receive gradients w.r.t.
Revised: For backward pass, peers receive gradients w.r.t.

Original: outputs and compute gradients w.r.t.
Revised: outputs and compute gradients w.r.t.

Original: layer inputs and accumulates gradients w.r.t.
Revised: layer inputs and accumulates gradients w.r.t.

Original: parameters.
Revised: parameters.

Original: Each peer has two queues for incoming and outgoing requests to maintain high GPU utilization even with a noticeable communication latency via buffering.
Revised: Each peer has two queues for incoming and outgoing requests to maintain high GPU utilization even with a noticeable communication latency via buffering.

Original: Similarly to other pipeline implementations (Huang et al., 2019; Narayanan et al., 2021), SWARM uses activation checkpointing (Griewank & Walther, 2000;
Revised: Similarly to other pipeline implementations (Huang et al., 2019; Narayanan et al., 2021), SWARM uses activation checkpointing (Griewank & Walther, 2000;

Original: Chen et al., 2016) to reduce the memory footprint.
Revised: Chen et al., 2016) to reduce the memory footprint.

Original: [Figure]
Revised: [Figure]

Original: ; Diskin
Revised: ; Diskin

Original: Stochastic wiring.
Revised: Stochastic wiring.

Original: To better utilize the capacity of each individual device, we dynamically “wire” each input through each stage and pick devices in proportion to their training throughput.
Revised: To better utilize the capacity of each individual device, we dynamically “wire” each input through each stage and pick devices in proportion to their training throughput.

Original: SWARM achieves this by creating several “trainer” processes that route through different pipeline stages using interleaved weighted round-robin algorithm (Katevenis et al., 1991; Tabatabaee et al., 2020) with randomized initial order; these processes do not use GPU and have no trainable parameters.
Revised: SWARM achieves this by creating several “trainer” processes that route through different pipeline stages using interleaved weighted round-robin algorithm (Katevenis et al., 1991; Tabatabaee et al., 2020) with randomized initial order; these processes do not use GPU and have no trainable parameters.

Original: Appendix D contains a detailed description of this component.
Revised: Appendix D contains a detailed description of this component.

Original: Adaptive swarm rebalancing.
Revised: Adaptive swarm rebalancing.

Original: As we described earlier in Section 2.2, our workers can join and leave training at any time.
Revised: As we described earlier in Section 2.2, our workers can join and leave training at any time.

Original: If any single pipeline stage loses too many peers, the remaining ones will have to deal with an increased processing load, which will inevitably form a bottleneck.
Revised: If any single pipeline stage loses too many peers, the remaining ones will have to deal with an increased processing load, which will inevitably form a bottleneck.

Original: SWARM parallelism addresses this problem by allowing peers to dynamically switch between pipeline stages to rebalance the training throughput.
Revised: SWARM parallelism addresses this problem by allowing peers to dynamically switch between pipeline stages to rebalance the training throughput.

Original: Every T =300 seconds, peers measure the utilization rate of each pipeline stage as the queue size.
Revised: Every T =300 seconds, peers measure the utilization rate of each pipeline stage as the queue size.

Original: Peers from the most under-utilized pipeline stage will then switch to the most over-utilized one (see Figure 2), download the latest training state from their new neighbors and continue training.
Revised: Peers from the most under-utilized pipeline stage will then switch to the most over-utilized one (see Figure 2), download the latest training state from their new neighbors and continue training.

Original: Similarly, if a new peer joins midway through training, it automatically finds the most over-utilized pipeline stage by following the same protocol.
Revised: Similarly, if a new peer joins midway through training, it automatically finds the most over-utilized pipeline stage by following the same protocol.

Original: As a side-effect, this technique allows SWARM to operate with uneven pipeline stages by allocating peers in proportion to the compute requirements of each pipeline stage.
Revised: As a side-effect, this technique allows SWARM to operate with uneven pipeline stages by allocating peers in proportion to the compute requirements of each pipeline stage.

Original: 3. C OMPRESSION - AWARE ARCHITECTURES
Revised: 3. C OMPRESSION - AWARE ARCHITECTURES

Original: Since pipeline parallelism has several distinct points of communication, the network overhead can be reduced considerably by reducing the size of features at these communication points.
Revised: Since pipeline parallelism has several distinct points of communication, the network overhead can be reduced considerably by reducing the size of features at these communication points.

Original: To exploit this, we develop compression-aware architectures that apply extreme compression at these points.
Revised: To exploit this, we develop compression-aware architectures that apply extreme compression at these points.

Original: We study two distinct ways of compression: compression through a linear bottleneck layer and compression through a bottleneck induced by the maxout activation function (Goodfellow et al., 2013).
Revised: We study two distinct ways of compression: compression through a linear bottleneck layer and compression through a bottleneck induced by the maxout activation function (Goodfellow et al., 2013).

Original: We combine our compression-aware architectures with 8-bit quantization and Huffman coding to reduce the communication overhead by another 62%.
Revised: We combine our compression-aware architectures with 8-bit quantization and Huffman coding to reduce the communication overhead by another 62%.

Original: Fully Connected Layers – The Baseline: Fully connected layers in transformers consist of a multi-layer perceptron with a single hidden layer and a non-linear activation function.
Revised: Fully Connected Layers – The Baseline: Fully connected layers in transformers consist of a multi-layer perceptron with a single hidden layer and a non-linear activation function.

Original: Without biases and with a residual connection (He et al., 2015) from the inputs to the outputs this can be described as:
Revised: Without biases and with a residual connection (He et al., 2015) from the inputs to the outputs this can be described as:

Original: [Equation]
Revised: [Equation]

Original: where x ∈ R b × s × m , w 1 ∈ R m × h , w 2 ∈ R h × m , and σ ( · ) , is a non-linear activation function, such as ReLU (Krizhevsky et al., 2012), and b , s , m , and h are the batch, sequence, model, and hidden dimension of the transformer neural network.
Revised: where x ∈ R b × s × m , w 1 ∈ R m × h , w 2 ∈ R h × m , and σ ( · ) , is a non-linear activation function, such as ReLU (Krizhevsky et al., 2012), and b , s , m , and h are the batch, sequence, model, and hidden dimension of the transformer neural network.

Original: To compress the output of the MLP layer, we want to apply a compression layer between two stages.
Revised: To compress the output of the MLP layer, we want to apply a compression layer between two stages.

Original: For example, if we have 24 layers and 4 stages, we have 3 compression layers at layer 6, 12, and 18 that bridge these stages.
Revised: For example, if we have 24 layers and 4 stages, we have 3 compression layers at layer 6, 12, and 18 that bridge these stages.

Original: Bottleneck layers: We experiment with simple bottleneck layers that work by compressing the output features of the MLP by linear projection:
Revised: Bottleneck layers: We experiment with simple bottleneck layers that work by compressing the output features of the MLP by linear projection:

Original: [Equation]
Revised: [Equation]

Original: where w c ∈ R m × c , w d ∈ R c × m are compression and decompression parameters with compression dimension c < m .
Revised: where w c ∈ R m × c , w d ∈ R c × m are compression and decompression parameters with compression dimension c < m .

Original: We find it critical to use a layer norm (Ba et al., 2016) to ensure training without divergence.
Revised: We find it critical to use a layer norm (Ba et al., 2016) to ensure training without divergence.

Original: The parameter tensor w c resides in one stage and its outputs are transferred to the next stage which holds the parameter w d , which requires m/c times less communication compared to the original model.
Revised: The parameter tensor w c resides in one stage and its outputs are transferred to the next stage which holds the parameter w d , which requires m/c times less communication compared to the original model.

Original: Note that adding a bottleneck only adds two linear layers for the forward pass and decreases the size of MLP activations; thus, its computational overhead is negligible.
Revised: Note that adding a bottleneck only adds two linear layers for the forward pass and decreases the size of MLP activations; thus, its computational overhead is negligible.

Original: Maxout Compression: Compared to bottleneck compression, maxout compression works by using the maxout activation function (Goodfellow et al., 2013) for compression rather than a linear projection.
Revised: Maxout Compression: Compared to bottleneck compression, maxout compression works by using the maxout activation function (Goodfellow et al., 2013) for compression rather than a linear projection.

Original: The maxout function of factor k takes inputs with a hidden dimension of d and reduces this dimension by the factor of k by computing the maximum value for each non-overlapping window of k features.
Revised: The maxout function of factor k takes inputs with a hidden dimension of d and reduces this dimension by the factor of k by computing the maximum value for each non-overlapping window of k features.

Original: We use maxout compression as follows:
Revised: We use maxout compression as follows:

Original: [Equation]
Revised: [Equation]

Original: where the output is reduced by a factor of k through the maxout function in the previous stage, and these outputs are sent to the next stage which holds the decompression matrix w d ∈ R m/k × m .
Revised: where the output is reduced by a factor of k through the maxout function in the previous stage, and these outputs are sent to the next stage which holds the decompression matrix w d ∈ R m/k × m .

Original:  E XPERIMENTS 4. C OMMUNICATION EFFICIENCY AT SCALE
Revised:  E XPERIMENTS 4. C OMMUNICATION EFFICIENCY AT SCALE

Original: Before we can meaningfully evaluate SWARM parallelism, we must verify our theoretical observations on communication efficiency.
Revised: Before we can meaningfully evaluate SWARM parallelism, we must verify our theoretical observations on communication efficiency.

Original: Here we run several controlled experiments that measure the
Revised: Here we run several controlled experiments that measure the

Original: GPU utilization and network usage for different model sizes, using vanilla transformer architecture (Vaswani et al., 2017) with batch size 1 and sequence length 512.
Revised: GPU utilization and network usage for different model sizes, using vanilla transformer architecture (Vaswani et al., 2017) with batch size 1 and sequence length 512.

Original: To decouple the performance impact from other factors, we run these experiments on homogeneous V100 GPU nodes that serve one pipeline stage over the network with varying latency and bandwidth.
Revised: To decouple the performance impact from other factors, we run these experiments on homogeneous V100 GPU nodes that serve one pipeline stage over the network with varying latency and bandwidth.

Original: The full experimental configuration is deferred to Appendix B.
Revised: The full experimental configuration is deferred to Appendix B.

Original: First, we measure how model size affects the computation to communication ratio at 500 Mb/s network bandwidth in both directions.
Revised: First, we measure how model size affects the computation to communication ratio at 500 Mb/s network bandwidth in both directions.

Original: We consider 4 different model configurations: the base configuration from the BERT paper (Devlin et al., 2019), “xxlarge" (“large” with d model =4096 ), which is used in several recent works (Lan et al., 2020; Sun et al., 2021;
Revised: We consider 4 different model configurations: the base configuration from the BERT paper (Devlin et al., 2019), “xxlarge" (“large” with d model =4096 ), which is used in several recent works (Lan et al., 2020; Sun et al., 2021;

Original: He et al., 2020), the full
Revised: He et al., 2020), the full

Original: GPT-3 model with d model =

Revised: GPT-3 model with d model =

Original: (Brown et al., 2020) and our modified Transformer architecture from Section 4.3 with d model =4096 , 3 layers per pipeline stage and 8-bit quantized activations.
Revised: (Brown et al., 2020) and our modified Transformer architecture from Section 4.3 with d model =4096 , 3 layers per pipeline stage and 8-bit quantized activations.

Original: In the first three configurations, the model consists of 12 Transformer layers placed on 12 servers with a single GPU; in the last one, there are 4 servers, each hosting 3 layers.
Revised: In the first three configurations, the model consists of 12 Transformer layers placed on 12 servers with a single GPU; in the last one, there are 4 servers, each hosting 3 layers.

Original: In addition, we evaluate how the same models perform under different network latency in Table 1.
Revised: In addition, we evaluate how the same models perform under different network latency in Table 1.

Original: As depicted in Figure 1 (right), larger models can indeed achieve a better GPU utilization rate, since their communication load grows slower than computation.
Revised: As depicted in Figure 1 (right), larger models can indeed achieve a better GPU utilization rate, since their communication load grows slower than computation.

Original: More importantly, even at a modest 500Mb/s network, the resulting GPU idle time can be pushed into the 10–20% range: either naturally for GPT-3-sized models, or with smaller models through activation compression.
Revised: More importantly, even at a modest 500Mb/s network, the resulting GPU idle time can be pushed into the 10–20% range: either naturally for GPT-3-sized models, or with smaller models through activation compression.

Original: Furthermore, these models maintain most of their training efficiency at 100ms latency (Figure 3), which is roughly equivalent to training on different continents (Verizon, 2021).
Revised: Furthermore, these models maintain most of their training efficiency at 100ms latency (Figure 3), which is roughly equivalent to training on different continents (Verizon, 2021).

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Experimental Setup:
Revised: Experimental Setup:

Original: As a baseline we train a transformer language model (Vaswani et al., 2017) on the OpenWebText corpus (Gokaslan & Cohen, 2019).
Revised: As a baseline we train a transformer language model (Vaswani et al., 2017) on the OpenWebText corpus (Gokaslan & Cohen, 2019).

Original: We use the following transformer model: sequence size 512, 16 layers with model dimension 1024, and hidden dimension 4096 for a total of 253M parameters.
Revised: We use the following transformer model: sequence size 512, 16 layers with model dimension 1024, and hidden dimension 4096 for a total of 253M parameters.

Original: We use BPE encoding (Radford et al., 2019) with vocabulary size ofsymbols.
Revised: We use BPE encoding (Radford et al., 2019) with vocabulary size ofsymbols.

Original: We do not use dropout or other regularization since our models underfit.
Revised: We do not use dropout or other regularization since our models underfit.

Original: We run our models in fairseq (Ott et al., 2019).
Revised: We run our models in fairseq (Ott et al., 2019).

Original: We test bottleneck and maxout compression for a compression factor of 50% and 75% compression compared to the original size over two and four stages.
Revised: We test bottleneck and maxout compression for a compression factor of 50% and 75% compression compared to the original size over two and four stages.

Original: We look at how these compression aware architectures degrade performance compared to the compression that they achieve.
Revised: We look at how these compression aware architectures degrade performance compared to the compression that they achieve.

Original: Results: The results of our compression aware architectures are shown in Table 2.
Revised: Results: The results of our compression aware architectures are shown in Table 2.

Original: We can see that while the bottleneck architecture is competitive with maxout for a compression factor of 2x with two stages, maxout has better perplexities if more stages or a higher compression ratio is used.
Revised: We can see that while the bottleneck architecture is competitive with maxout for a compression factor of 2x with two stages, maxout has better perplexities if more stages or a higher compression ratio is used.

Original: The out-of-distribution perplexities vary consistently with the in-distribution perplexity, which suggest compression-aware architectures do not degrade out-of-distribution performance more than in-distribution performance.
Revised: The out-of-distribution perplexities vary consistently with the in-distribution perplexity, which suggest compression-aware architectures do not degrade out-of-distribution performance more than in-distribution performance.

Original: As such, maxout compression is an effective technique to reduce the bandwidth requirements of pipeline parallel training further.
Revised: As such, maxout compression is an effective technique to reduce the bandwidth requirements of pipeline parallel training further.

Original: Together with the square-cube law for distributed training, and SWARM parallelism, compressionaware architectures allow for better scaling of large neural networks trained over preemptible low network bandwidth peers.
Revised: Together with the square-cube law for distributed training, and SWARM parallelism, compressionaware architectures allow for better scaling of large neural networks trained over preemptible low network bandwidth peers.

Original: Thus, compression-aware architectures improve accessibility and affordability to train large neural networks outside of HPC environments.
Revised: Thus, compression-aware architectures improve accessibility and affordability to train large neural networks outside of HPC environments.

Original: 4. R EAL - WORLD CONVERGENCE AND SCALING
Revised: 4. R EAL - WORLD CONVERGENCE AND SCALING

Original: Finally, to verify the efficiency of SWARM parallelism in a practical scenario, we conduct a series of large-scale distributed experiments using preemptible cloud T4 and A100 GPUs over the nonspecialized network.
Revised: Finally, to verify the efficiency of SWARM parallelism in a practical scenario, we conduct a series of large-scale distributed experiments using preemptible cloud T4 and A100 GPUs over the nonspecialized network.

Original: In all experiments below, we train a Transformer language model similar to prior work (Brown et al., 2020; Wang & Komatsuzaki, 2021; Black et al., 2021) with 1.08 billion parameters in total: because of layer sharing, it is equivalent to a 12 times larger network in terms of compute requirements.
Revised: In all experiments below, we train a Transformer language model similar to prior work (Brown et al., 2020; Wang & Komatsuzaki, 2021; Black et al., 2021) with 1.08 billion parameters in total: because of layer sharing, it is equivalent to a 12 times larger network in terms of compute requirements.

Original: The detailed setup of all experiments is given in Appendix B.
Revised: The detailed setup of all experiments is given in Appendix B.

Original: Specifically, our model consists of 4 stages, each containing a single Transformer decoder block with d model = 4096 and 12 layers per pipeline stage, which decreases the size of data that needs to be sent during the aggregation step.
Revised: Specifically, our model consists of 4 stages, each containing a single Transformer decoder block with d model = 4096 and 12 layers per pipeline stage, which decreases the size of data that needs to be sent during the aggregation step.

Original: We also use 8-bit compression (Dettmers, 2015) for activations and gradients to reduce the communication intensity of training.
Revised: We also use 8-bit compression (Dettmers, 2015) for activations and gradients to reduce the communication intensity of training.

Original: One additional advantage of using the activation compression (compared to other evaluated approaches) is that it does not require to modify the architecture and evaluate its performance; therefore, we hope to demonstrate that can be a simple yet practical solution for bandwidth-constrained environments.
Revised: One additional advantage of using the activation compression (compared to other evaluated approaches) is that it does not require to modify the architecture and evaluate its performance; therefore, we hope to demonstrate that can be a simple yet practical solution for bandwidth-constrained environments.

Original: First, to verify that model parallelism with asynchronous updates does not cause any significant convergence issues, we train the model on the Pile (Gao et al., 2020) dataset with 400 T4 cloud instances, each hosting a single accelerator.
Revised: First, to verify that model parallelism with asynchronous updates does not cause any significant convergence issues, we train the model on the Pile (Gao et al., 2020) dataset with 400 T4 cloud instances, each hosting a single accelerator.

Original: As a baseline, we use regular data-parallel training with offloading on 128 A100 GPUs.
Revised: As a baseline, we use regular data-parallel training with offloading on 128 A100 GPUs.

Original: We report the per-iteration convergence of two setups to amortize the computational performance difference between the setups.
Revised: We report the per-iteration convergence of two setups to amortize the computational performance difference between the setups.

Original: Figure 4 shows the results of this experiment: it can be seen that the training dynamics of two approaches are indeed similar, which demonstrates the viability of SWARM parallelism for heterogeneous and poorly-connected devices.
Revised: Figure 4 shows the results of this experiment: it can be seen that the training dynamics of two approaches are indeed similar, which demonstrates the viability of SWARM parallelism for heterogeneous and poorly-connected devices.

Revised: We also use the T4 node preemption data of this run to demonstrate the necessity of adaptive rebalancing in a pipeline of unreliable devices; refer to Appendix G for the description.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: In the next experiment, we aim to measure the pipeline throughput in different hardware conditions and to compare it with theoretical best-case performance.
Revised: In the next experiment, we aim to measure the pipeline throughput in different hardware conditions and to compare it with theoretical best-case performance.

Original: For that, we use several setups: first, we use the same 400 preemptible T4 nodes; also, we use 7 instances with 8 A100 GPU each; finally, we combine these fleets to create a heterogeneous setup.
Revised: For that, we use several setups: first, we use the same 400 preemptible T4 nodes; also, we use 7 instances with 8 A100 GPU each; finally, we combine these fleets to create a heterogeneous setup.

Original: We measure the number of samples processed by the pipeline both in our infrastructure and the ideal use case which ignores all network-related operations.
Revised: We measure the number of samples processed by the pipeline both in our infrastructure and the ideal use case which ignores all network-related operations.

Original: As demonstrated in the left two columns of Table 3, asynchronous training of computeintensive models with compressed activations allows us to achieve high performance without a dedicated networking solution.
Revised: As demonstrated in the left two columns of Table 3, asynchronous training of computeintensive models with compressed activations allows us to achieve high performance without a dedicated networking solution.

Original: Furthermore, the load balancing algorithm of SWARM allows us to dynamically and efficiently utilize different hardware without being bottlenecked by slower devices.
Revised: Furthermore, the load balancing algorithm of SWARM allows us to dynamically and efficiently utilize different hardware without being bottlenecked by slower devices.

Original: Finally, we use the same load testing scenario to estimate the bandwidth required to fully utilize each device type in the above infrastructure.
Revised: Finally, we use the same load testing scenario to estimate the bandwidth required to fully utilize each device type in the above infrastructure.

Original: For this, we measure the average incoming and outgoing bandwidth on the nodes that serve the intermediate stage of the pipeline.
Revised: For this, we measure the average incoming and outgoing bandwidth on the nodes that serve the intermediate stage of the pipeline.

Original: We summarize our findings in the right two columns of Table 3: it turns out that with layer sharing and 8-bit compression, medium-performance GPUs (such as T4) can be saturated even with moderate network speeds.
Revised: We summarize our findings in the right two columns of Table 3: it turns out that with layer sharing and 8-bit compression, medium-performance GPUs (such as T4) can be saturated even with moderate network speeds.

Original: Although training over the Internet with more efficient hardware might indeed underutilize the accelerator, this issue can be offset by advanced compression strategies such as compression-aware architectures or more layer sharing iterations.
Revised: Although training over the Internet with more efficient hardware might indeed underutilize the accelerator, this issue can be offset by advanced compression strategies such as compression-aware architectures or more layer sharing iterations.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we have have analyzed and evaluated the feasibility of high-throughput training of neural models on unreliable, preemptible peers with low network bandwidth.
Revised: In this work, we have have analyzed and evaluated the feasibility of high-throughput training of neural models on unreliable, preemptible peers with low network bandwidth.

Original: Through analysis guided by the Square-Cube Law of distributed training, we found that this is feasible by training very large models with pipeline parallelism.
Revised: Through analysis guided by the Square-Cube Law of distributed training, we found that this is feasible by training very large models with pipeline parallelism.

Original: Specifically, we propose SWARM parallelism to overcome the challenges of pipeline parallelism for preemptible devices with heterogeneous network bandwidths and computational throughputs.
Revised: Specifically, we propose SWARM parallelism to overcome the challenges of pipeline parallelism for preemptible devices with heterogeneous network bandwidths and computational throughputs.

Original: We have shown that SWARM parallelism is highly effective at re-balancing peers and maximizing the aggregate throughput of the entire model-parallel pipeline.
Revised: We have shown that SWARM parallelism is highly effective at re-balancing peers and maximizing the aggregate throughput of the entire model-parallel pipeline.

Original: We showed that the combination of SWARM parallelism and compression-aware architectures allows to train very large models on cheap preemptible cloud instances or on volunteered resources.
Revised: We showed that the combination of SWARM parallelism and compression-aware architectures allows to train very large models on cheap preemptible cloud instances or on volunteered resources.

Original: As such, our work makes the training of large models accessible and affordable to small research teams that do not have access to dedicated infrastructure such as supercomputers.
Revised: As such, our work makes the training of large models accessible and affordable to small research teams that do not have access to dedicated infrastructure such as supercomputers.

Original: S UPPLEMENTARY M ATERIAL A A DDITIONAL R ELATED W ORK
Revised: S UPPLEMENTARY M ATERIAL A A DDITIONAL R ELATED W ORK

Original: In this section we review two additional types of distributed training algorithms that can support training large models.
Revised: In this section we review two additional types of distributed training algorithms that can support training large models.

Original: Dynamic parameter loading.
Revised: Dynamic parameter loading.

Original: Several recent studies propose alternative execution algorithms that allow training large models with data parallelism.
Revised: Several recent studies propose alternative execution algorithms that allow training large models with data parallelism.

Original: Since neural networks typically use a small fraction of weights at any given moment, the remaining “inactive” weights can be sharded (Rajbhandari et al., 2020) or offloaded to external memory (Pudipeddi et al., 2020;
Revised: Since neural networks typically use a small fraction of weights at any given moment, the remaining “inactive” weights can be sharded (Rajbhandari et al., 2020) or offloaded to external memory (Pudipeddi et al., 2020;

Original: Ren et al., 2021; Rajbhandari et al., 2021).
Revised: Ren et al., 2021; Rajbhandari et al., 2021).

Original: In sharded data-parallelism (Rajbhandari et al., 2020) inactive tensors are distributed across all devices such that each device stores 1 /n th of all parameters.
Revised: In sharded data-parallelism (Rajbhandari et al., 2020) inactive tensors are distributed across all devices such that each device stores 1 /n th of all parameters.

Original: For active layers, the shards are gathered such that each device holds the entire tensor just-in-time for computation.
Revised: For active layers, the shards are gathered such that each device holds the entire tensor just-in-time for computation.

Original: After the computation, the parameters’ memory is freed such that only the sharded memory remains (1/Nth per device).
Revised: After the computation, the parameters’ memory is freed such that only the sharded memory remains (1/Nth per device).

Original: This makes it very memory efficient to store model and optimizer states for inactive layers if many devices are available.
Revised: This makes it very memory efficient to store model and optimizer states for inactive layers if many devices are available.

Original: Similarly to tensor parallelism, these algorithms can support arbitrary models without the need for layer partitioning and can, in principle, run a large model on a single GPU which is useful for fine-tuning and inference.
Revised: Similarly to tensor parallelism, these algorithms can support arbitrary models without the need for layer partitioning and can, in principle, run a large model on a single GPU which is useful for fine-tuning and inference.

Original: Architecture-specific methods.
Revised: Architecture-specific methods.

Original: Finally, some distributed training algorithms take advantage of specific layers, such as locally-connected layers (Dean et al., 2012;
Revised: Finally, some distributed training algorithms take advantage of specific layers, such as locally-connected layers (Dean et al., 2012;

Original: Coates et al., 2013), Mixture-ofExperts (Jacobs et al., 1991; Shazeer et al., 2017;
Revised: Coates et al., 2013), Mixture-ofExperts (Jacobs et al., 1991; Shazeer et al., 2017;

Original: Lepikhin et al., 2020), Switch layers (Fedus et al., 2021) or Product Key Memory (Lample et al., 2019).
Revised: Lepikhin et al., 2020), Switch layers (Fedus et al., 2021) or Product Key Memory (Lample et al., 2019).

Original: These layers contain many near-independent parts that can be assigned to different devices.
Revised: These layers contain many near-independent parts that can be assigned to different devices.

Original: They can easily scale to an extremely large number of parameters with a relatively small increase in compute (Shazeer et al., 2017).
Revised: They can easily scale to an extremely large number of parameters with a relatively small increase in compute (Shazeer et al., 2017).

Original: However, they are also less parameter-efficient (Fedus et al., 2021) and may not apply to all architectures.
Revised: However, they are also less parameter-efficient (Fedus et al., 2021) and may not apply to all architectures.

Original: B D ETAILED EXPERIMENTAL SETUPS B. A DDITIONAL DETAILS FOR S ECTION 4. 
Revised: B D ETAILED EXPERIMENTAL SETUPS B. A DDITIONAL DETAILS FOR S ECTION 4.

Original: We benchmark three popular versions of the Transformer layer:
Revised: We benchmark three popular versions of the Transformer layer:

Original: • Transformer-base: d model =
Revised: • Transformer-base: d model =

Original: 768 , d FFN = 3072 , 12 heads; • Transformer-xxlarge: d model = 4096 , d FFN = 16384 , 32 heads; • “the GPT3” (Brown et al., 2020): d model = 12288 , d FFN = 49152 , 96 heads.
Revised: 768 , d FFN = 3072 , 12 heads; • Transformer-xxlarge: d model = 4096 , d FFN = 16384 , 32 heads; • “the GPT-3” (Brown et al., 2020): d model = 12288 , d FFN = 49152 , 96 heads.

Original: For simplicity, we set up each experiment with 12 Transformer layers using 12 servers with a single
Revised: For simplicity, we set up each experiment with 12 Transformer layers using 12 servers with a single

Original: V100-PCIE GPU each.
Revised: V100-PCIE GPU each.

Original: The servers communicate at 500Mbps under 3–6ms latency.
Revised: The servers communicate at 500Mbps under 3–6ms latency.

Original: Finally, we evaluate GPT-3 with colocated layers — the same architecture as in GPT-3, but this time we placelayers per server (4 servers total).
Revised: Finally, we evaluate GPT-3 with colocated layers — the same architecture as in GPT-3, but this time we placelayers per server (4 servers total).

Original: Due to modest communication bandwidth, smaller models spend most of the time waiting for the network.
Revised: Due to modest communication bandwidth, smaller models spend most of the time waiting for the network.

Original: However, that same bandwidth allows for > 80% GPU utilization when dealing with
Revised: However, that same bandwidth allows for > 80% GPU utilization when dealing with

Original: GPT-3-sized layers and over 90% with GPT-3 layers per device.
Revised: GPT-3-sized layers and over 90% with GPT-3 layers per device.

Original: The training time reported above is the time required to run forward and backward pass for all layers with a batch of 4×128 tokens, not including the Adam updates.
Revised: The training time reported above is the time required to run forward and backward pass for all layers with a batch of 4×128 tokens, not including the Adam updates.

Original: All results are averaged over 1000 consecutive batches, the standard deviations are below 0.1%.
Revised: All results are averaged over 1000 consecutive batches, the standard deviations are below 0.1%.

Original: All four GPUs are in the same data center, but different servers.
Revised: All four GPUs are in the same data center, but different servers.

Original: Each layer is a TransformerEncoderLayer (torch==1.7.0), wrapped with torch.utils.checkpoint.
Revised: Each layer is a TransformerEncoderLayer (torch==1.7.0), wrapped with torch.utils.checkpoint.

Original: We use hivemind==0.8.

Revised: We use hivemind==0.8.

Original: (Ryabinin & Gusev, 2020) with a single synchronous trainer based on huggingface BERT training code.
Revised: (Ryabinin & Gusev, 2020) with a single synchronous trainer based on huggingface BERT training code.

Original: However, these results are not specific to hivemind and are likely reproducible in FairScale (Baines et al., 2021) or PyTorch RPC.
Revised: However, these results are not specific to hivemind and are likely reproducible in FairScale (Baines et al., 2021) or PyTorch RPC.

Original: The only important detail is that the training code should run as much communication as possible in the background, while the GPUs are busy processing batches.
Revised: The only important detail is that the training code should run as much communication as possible in the background, while the GPUs are busy processing batches.

Original: It is important to reuse the same connection for multiple RPC calls, so the TCP buffer does not have to warm up again during each call.
Revised: It is important to reuse the same connection for multiple RPC calls, so the TCP buffer does not have to warm up again during each call.

Original: Also, our implementation performs quantization asynchronously with communication and other computations.
Revised: Also, our implementation performs quantization asynchronously with communication and other computations.

Original: B. A DDITIONAL DETAILS FOR S ECTION 4. 
Revised: B. A DDITIONAL DETAILS FOR S ECTION 4.

Original: For the language model training experiment, we use the standard Transformer architecture with two modifications.
Revised: For the language model training experiment, we use the standard Transformer architecture with two modifications.

Original: First, we use Rotary Positional Embeddings (Su et al., 2021) to increase the stability of our training runs.
Revised: First, we use Rotary Positional Embeddings (Su et al., 2021) to increase the stability of our training runs.

Original: Second, we divide the model into four stages: two stages in the middle consist of regular stacked Transformer layers, the first stage also contains the embedding layer and the last stage includes the language modeling head.
Revised: Second, we divide the model into four stages: two stages in the middle consist of regular stacked Transformer layers, the first stage also contains the embedding layer and the last stage includes the language modeling head.

Original: To reduce the communication intensity of All-Reduce during the data-parallel steps, we leverage weight sharing (similarly to ALBERT from Lan et al., 2020) in each stage.
Revised: To reduce the communication intensity of All-Reduce during the data-parallel steps, we leverage weight sharing (similarly to ALBERT from Lan et al., 2020) in each stage.

Original: C A NSWERS TO COMMON QUESTIONS
Revised: C A NSWERS TO COMMON QUESTIONS

Original: In this section, we provide answers to several assorted questions about our study and address some of the limitations of SWARM parallelism.
Revised: In this section, we provide answers to several assorted questions about our study and address some of the limitations of SWARM parallelism.

Original: Why not just use data parallelism with offloading?
Revised: Why not just use data parallelism with offloading?

Original: Regular data parallelism requires all-reduce steps where peers exchange gradients.
Revised: Regular data parallelism requires all-reduce steps where peers exchange gradients.

Original: This can be prohibitively expensive for large models.
Revised: This can be prohibitively expensive for large models.

Original: For example, a 1B parameter model with 16-bit gradients requires 2 GB of data to be synchronized between all n devices.
Revised: For example, a 1B parameter model with 16-bit gradients requires 2 GB of data to be synchronized between all n devices.

Original: We need at least n messages to perform this synchronization.
Revised: We need at least n messages to perform this synchronization.

Original: If we have, for example, 100 devices with bidirectional communication, each client would need to send 2 GB of data to finish the synchronization.
Revised: If we have, for example, 100 devices with bidirectional communication, each client would need to send 2 GB of data to finish the synchronization.

Original: Thus, with slow interconnects such synchronizations are not practical.
Revised: Thus, with slow interconnects such synchronizations are not practical.

Original: Why not just use fully sharded data parallelism with elasticity?
Revised: Why not just use fully sharded data parallelism with elasticity?

Original: Sharded data parallelism requires all-to-all communication of parameter buffers at each layer.
Revised: Sharded data parallelism requires all-to-all communication of parameter buffers at each layer.

Original: Each of these all-to-all communications can be in parallel and has a size of parameters/ n and in total n messages are required.
Revised: Each of these all-to-all communications can be in parallel and has a size of parameters/ n and in total n messages are required.

Original: Thus for 1B parameters in 16-bit precision, a total of 2GB of data need to be synchronized for both the forward and backward pass.
Revised: Thus for 1B parameters in 16-bit precision, a total of 2GB of data need to be synchronized for both the forward and backward pass.

Original: If using low-bandwidth devices with 100 Mbit/s speed, this would entail an overhead of 5.5 minutes per forward/backward pass which is difficult to hide under computation.
Revised: If using low-bandwidth devices with 100 Mbit/s speed, this would entail an overhead of 5.5 minutes per forward/backward pass which is difficult to hide under computation.

Original: This is exacerbated further because all-to-all communication latency is determined by the lowest peer.
Revised: This is exacerbated further because all-to-all communication latency is determined by the lowest peer.

Original: Thus sharded data parallelism can be particularly ineffective for environments where peers have heterogeneous network bandwidths.
Revised: Thus sharded data parallelism can be particularly ineffective for environments where peers have heterogeneous network bandwidths.

Original: Zero-offload allows one to train 13B parameters on a single V100, so why do I need SWARM?
Revised: Zero-offload allows one to train 13B parameters on a single V100, so why do I need SWARM?

Original: Training with zero-offload can slow down training due to the slow bandwidth between external memory and the accelerator.
Revised: Training with zero-offload can slow down training due to the slow bandwidth between external memory and the accelerator.

Original: Training with SWARM can accelerate training while also allowing the training of large models.
Revised: Training with SWARM can accelerate training while also allowing the training of large models.

Original: Also, see Appendix E for a detailed comparison.
Revised: Also, see Appendix E for a detailed comparison.

Original: Should I use SWARM in a supercomputer?
Revised: Should I use SWARM in a supercomputer?

Original: By default, SWARM is worse than traditional parallelism due to extra complexity.
Revised: By default, SWARM is worse than traditional parallelism due to extra complexity (see experiments in Section F.

Original: However, supercomputers often have heterogeneous devices, in which case SWARM is useful.
Revised: However, supercomputers often have heterogeneous devices, in which case SWARM is useful.

Original: When should I avoid using SWARM?
Revised: When should I avoid using SWARM?

Original: SWARM is very efficient at training large models with more than 1B parameters.
Revised: SWARM is very efficient at training large models with more than 1B parameters.

Original: For smaller models a sharded data-parallel approach can be effective.
Revised: For smaller models a sharded data-parallel approach can be effective.

Original: For
Revised: For

Original: HPC environments with homogeneous networking, standard sharded data parallel or pipeline parallel training will be more effective than SWARM because the environment is stable and predictable such that re-balancing is not required.
Revised: HPC environments with homogeneous networking, standard sharded data parallel or pipeline parallel training will be more effective than SWARM because the environment is stable and predictable such that re-balancing is not required.

Original: For HPC environments which are so extensive that failure of a nodes is likely, the effectiveness of swarm depends on how many nodes are expected to fail.
Revised: For HPC environments which are so extensive that failure of a nodes is likely, the effectiveness of swarm depends on how many nodes are expected to fail.

Original: Elastic sharded data parallelism is better than SWARM if the number of expected failures is relatively low.
Revised: Elastic sharded data parallelism is better than SWARM if the number of expected failures is relatively low.

Original: How much failure can SWARM handle?
Revised: How much failure can SWARM handle?

Original: As long as there is at least one operational peer at every pipeline stage and at least one trainer SWARM will work without any issues.
Revised: As long as there is at least one operational peer at every pipeline stage and at least one trainer SWARM will work without any issues.

Original: The same is true for network partitioning which requires at least one trainer and peer in each pipeline stage.
Revised: The same is true for network partitioning which requires at least one trainer and peer in each pipeline stage.

Original: Can I use SWARM without layer sharing or quantization?
Revised: Can I use SWARM without layer sharing or quantization?

Original: Yes, SWARM can still be effective in this scenarios.
Revised: Yes, SWARM can still be effective in this scenarios.

Original: Our bandwidth experiments in the main paper give some idea what the network overhead is.
Revised: Our bandwidth experiments in the main paper give some idea what the network overhead is.

Original: By using no quantization, that means using regular 16-bit activations, the network overhead increases roughly by a factor of two.
Revised: By using no quantization, that means using regular 16-bit activations, the network overhead increases roughly by a factor of two.

Original: Without layer sharing the overhead within each pipeline stage to synchronize the gradient is increased by the number of layers not being shared.
Revised: Without layer sharing the overhead within each pipeline stage to synchronize the gradient is increased by the number of layers not being shared.

Original: As such, a rough estimate of the efficiency of SWARM in these scenarios can be estimated by taking our model-size-network-bandwidth-requirements data and multiplying it by the relevant factor.
Revised: As such, a rough estimate of the efficiency of SWARM in these scenarios can be estimated by taking our model-size-network-bandwidth-requirements data and multiplying it by the relevant factor.

Original: Do the compression-aware architecture modifications apply only to transformers?
Revised: Do the compression-aware architecture modifications apply only to transformers?

Original: Bottleneck and maxout compression are general compression techniques that can be applied to any layer in any architecture.
Revised: Bottleneck and maxout compression are general compression techniques that can be applied to any layer in any architecture.

Original: However, their effectiveness may vary depending on where in the model they are applied and what kind of model these are applied to (CNN vs RNN vs Transformer, etc.)
Revised: However, their effectiveness may vary depending on where in the model they are applied and what kind of model these are applied to (CNN vs RNN vs Transformer, etc.)

Original: Some configurations in Section 4.1 measure less than 20% GPU idle time, while many HPC systems only achieve ≈ 80% GPU utilization.
Revised: Some configurations in Section 4.1 measure less than 20% GPU idle time, while many HPC systems only achieve ≈ 80% GPU utilization.

Original: Does this mean that SWARM is 30% faster?
Revised: Does this mean that SWARM is 30% faster?

Original: No, because these are different measurement types.
Revised: No, because these are different measurement types.

Original: Narayanan et al.
Revised: Narayanan et al.

Original: (2021) measures GPU utilization as a fraction of theoretical peak flop/s of their GPUs.
Revised: (2021) measures GPU utilization as a fraction of theoretical peak flop/s of their GPUs.

Original: In contrast, we only measure what fraction of time is GPU running the model, regardless of efficiency.
Revised: In contrast, we only measure what fraction of time is GPU running the model, regardless of efficiency.

Original: Since no deep learning workload can achieve 100% peak flop/s, 20% GPU idle time for SWARM means that it can achieve ≈ 0 .
Revised: Since no deep learning workload can achieve 100% peak flop/s, 20% GPU idle time for SWARM means that it can achieve ≈ 0 .

Original: 8 × the training throughput compared to training with a infinitely fast network.
Revised: 8 × the training throughput compared to training with a infinitely fast network.

Original: As a rule of thumb, one can say that
Revised: As a rule of thumb, one can say that

Original: SWARM will run at 20% slower than Narayanan et al.
Revised: SWARM will run at 20% slower than Narayanan et al.

Original: (2021) using the infrastructure that is several times cheaper.
Revised: (2021) using the infrastructure that is several times cheaper.

Original: D S TOCHASTIC WIRING DETAILS
Revised: D S TOCHASTIC WIRING DETAILS

Original: Our approach uses stochastic wiring , a specialized routing algorithm designed around heterogeneous unreliable devices and high network latency.
Revised: Our approach uses stochastic wiring , a specialized routing algorithm designed around heterogeneous unreliable devices and high network latency.

Original: The core idea of stochastic wiring is to route each training microbatch through randomly chosen pipeline stages such that the workload of each peer is proportional to its performance.
Revised: The core idea of stochastic wiring is to route each training microbatch through randomly chosen pipeline stages such that the workload of each peer is proportional to its performance.

Original: From a system design perspective, each device runs a separate trainer process that forms microbatches and routes them through pipeline stages (forward and backward pass).
Revised: From a system design perspective, each device runs a separate trainer process that forms microbatches and routes them through pipeline stages (forward and backward pass).

Original: As we describe earlier in
Revised: As we describe earlier in

Original: Section 3.2, trainers run Interleaved
Revised: Section 3.2, trainers run Interleaved

Original: Weighted Round Robin (Katevenis et al., 1991; Tabatabaee et al., 2020) scheduling to dynamically assign microbatches to peers based on each peer’s training throughput (“samples per second”) in a balanced way.
Revised: Weighted Round Robin (Katevenis et al., 1991; Tabatabaee et al., 2020) scheduling to dynamically assign microbatches to peers based on each peer’s training throughput (“samples per second”) in a balanced way.

Original: To achieve this, trainers maintain a moving average estimate of each peer’s training throughput based on how long does the peer take to process the batch.
Revised: To achieve this, trainers maintain a moving average estimate of each peer’s training throughput based on how long does the peer take to process the batch.

Original: This moving average estimate is updated each time when the corresponding peer responds to a request.
Revised: This moving average estimate is updated each time when the corresponding peer responds to a request.

Original: If a peer does not respond or otherwise fails to process the minibatch, trainer will temporarily ban this peer and reset its throughput estimate.
Revised: If a peer does not respond or otherwise fails to process the minibatch, trainer will temporarily ban this peer and reset its throughput estimate.

Original: Curiously, different trainers can have different throughput estimates based on the network topology.
Revised: Curiously, different trainers can have different throughput estimates based on the network topology.

Original: For instance, if peers are located in two cloud regions, a given peer’s trainer will have higher throughput estimate for peers in the same data center.
Revised: For instance, if peers are located in two cloud regions, a given peer’s trainer will have higher throughput estimate for peers in the same data center.

Original: In other words, trainers automatically adjust to the network topology by routing more traffic to peers that are “nearby”.
Revised: In other words, trainers automatically adjust to the network topology by routing more traffic to peers that are “nearby”.

Original: Another important observation is that stochastic wiring allows SWARM to mitigate network latency .
Revised: Another important observation is that stochastic wiring allows SWARM to mitigate network latency .

Original: Unlike existing pipeline algorithms (Huang et al., 2019), SWARM workers do not get blocked if their neighbors take too long to process a minibatch.
Revised: Unlike existing pipeline algorithms (Huang et al., 2019), SWARM workers do not get blocked if their neighbors take too long to process a minibatch.

Original: Instead, each SWARM device maintains a queue of microbatches assigned by trainers.
Revised: Instead, each SWARM device maintains a queue of microbatches assigned by trainers.

Original: In case of a latency spike, workers keep processing previously queued microbatches, maintaining high device utilization.
Revised: In case of a latency spike, workers keep processing previously queued microbatches, maintaining high device utilization.

Original: E O N  RELATION BETWEEN SWARM AND Z ERO -O FFLOAD
Revised: E O N THE RELATION BETWEEN SWARM AND Z ERO -O FFLOAD

Original: As we outlined earlier, one key property of SWARM is that it produces exactly the same stepwise updates as conventional distributed training algorithms and will therefore achieve a solution in the same number of steps.Each iteration of SWARM-parallel training performs exactly the same computation as training with ZeRO-Offload (Ren et al., 2021).
Revised: In this section, we argue that depending on the use of DPU, SWARM-parallel training is equivalent to either fully synchronous training or ZeRO-Offload (Ren et al., 2021). That is, SWARM produces exactly the same stepwise updates as conventional distributed training algorithms and will therefore achieve a solution in the same number of steps.

Original: This observation is similar to how many advanced distributed training techniques (Huang et al., 2019; Rajbhandari et al., 2020) are computationally equivalent to regular synchronous training on a single device.
Revised: This observation is similar to how many advanced distributed training techniques (Huang et al., 2019; Rajbhandari et al., 2020) are computationally equivalent to regular synchronous training on a single device.

Original: For instance, despite using advanced distributed computation strategies, GPipe (Huang et al., 2019) computes exactly the same mathematical expression to obtain gradients and applies those gradients in the same order as any other synchronous training algorithm.
Revised: For instance, despite using advanced distributed computation strategies, GPipe (Huang et al., 2019) computes exactly the same mathematical expression to obtain gradients and applies those gradients in the same order as any other synchronous training algorithm.

Original: On the other hand,
Revised: On the other hand,

Original: PipeDream (Narayanan et al., 2019) changes the order in which the updates are applied, introducing the so-called stale gradients (Recht et al., 2011).
Revised: PipeDream (Narayanan et al., 2019) changes the order in which the updates are applied, introducing the so-called stale gradients (Recht et al., 2011).

Original: This allows PipeDream to improve device utilization, but it has been shown to reduce the final model quality in some setups (Wang et al., 2020).
Revised: This allows PipeDream to improve device utilization, but it has been shown to reduce the final model quality in some setups (Wang et al., 2020).

Original: Despite using randomized routing and asynchronous communication between pipeline stages,
Revised: Despite using randomized routing and asynchronous communication between pipeline stages,

Original: SWARM still performs optimizer steps synchronously after peers collectively accumulate a global batch size (hyperparameter).
Revised: SWARM still performs optimizer steps synchronously after peers collectively accumulate a global batch size (hyperparameter).

Original: While different peers may accumulate a different number of samples, they will all use the same gradient after averaging.
Revised: While different peers may accumulate a different number of samples, they will all use the same gradient after averaging.

Original: Finally, any peer that fails or does not meet this condition for any reason is considered a straggler and must reload its state from neighbors before it can resume training.
Revised: Finally, any peer that fails or does not meet this condition for any reason is considered a straggler and must reload its state from neighbors before it can resume training.

Original: This ensures that all surviving peers use non-stale aggregated gradient over the specified batch size when they perform the optimizer step.
Revised: This ensures that all surviving peers use non-stale aggregated gradient over the specified batch size when they perform the optimizer step.

Original: The only deviation from fully synchronous training is that
Revised: The only deviation from fully synchronous training is that

Original: SWARM uses the same approach for CPU offloading as ZeRO-Offload, and by extension, delayed parameter updates (DPU).
Revised: SWARM uses the same approach for CPU offloading as ZeRO-Offload, and by extension, delayed parameter updates (DPU).

Original: While DPU were shown not to affect convergence (Ren et al., 2021), one can disable this functionality and make SWARM fully equivalent to normal training.
Revised: While DPU were shown not to affect convergence (Ren et al., 2021), one can disable this functionality and make SWARM fully equivalent to normal training.

Original: Naturally, these guarantees come at the cost of reduced compute utilization as a small portion of devices will need to wait after every step.
Revised: Naturally, these guarantees come at the cost of reduced compute utilization as a small portion of devices will need to wait after every step.

Original: However, as we show in Section 4.3, SWARM can still train with competitive training throughput due to the fact that large models are trained with increased batch sizes (Brown et al., 2020). 
Revised: However, as we show in Section 4.3, SWARM can still train with competitive training throughput due to the fact that large models are trained with increased batch sizes (Brown et al., 2020). F D ETAILED PERFORMANCE COMPARISON

Revised: Here, we investigate how SWARM parallelism compares to existing systems for training large models:

Revised: GPipe (Huang et al., 2019) and ZeRO-Offload (Ren et al., 2021).

Revised: The purpose of this section is to compare the training throughput in “ideal” conditions (with homogeneous reliable devices and balanced layers), as deviating from these conditions makes it infeasible to train with baseline systems.

Revised: We evaluate training performance for sequences of 4 Transformer layers of identical size distributed over 16 workers.

Revised: The pipeline does not contain embeddings or language modeling heads, as it would result in imbalance between the stages.

Revised: Similarly to Section 4.1, we use two layer configurations: “xxlarge” ( d model =4096 , d FFN =16384 , 32 heads) and “GPT-3” ( d model =12288 , d FFN =49152 ,heads).

Revised: The microbatch size is 4 for “xxlarge” and 1 for “GPT-3”, and the sequence length is 512.

Revised: To provide a more detailed view of the training performance, we measure two separate performance statistics: the training throughput and the All-Reduce time.

Revised: The training throughput measures the rate at which the system can process training sequences, i.e. run forward and backward passes.

Revised: In turn, the All-Reduce time is the time each system spends to aggregate those accumulated gradients across devices.

Revised: The total time per step can be computed as batch_size / throughput + all_reduce_time .

Revised: Intuitively, training with small batch sizes is more sensitive to the All-Reduce time since the algorithm needs to run All-Reduce more frequently and vice versa.

Revised: Hardware setup: Each worker uses a V100-PCIe GPU with 16 CPU threads (E5 v5-2660v4) and 128 GB RAM.

Revised: The only exception is for ZeRO-Offload with “GPT-3” layers, where we had to double the RAM size because the system required 190GB at peak.

Revised: Similarly to Section 4.1, each worker can communicate at a 500Mb/s bandwidth for both upload and download for a total of 1Gb/s.

Revised: In terms of latency, we consider two setups: with no latency , where workers communicate normally within the same rack, and with latency , where we inject additional 100 ± 50 ms latency in the kernel 5 .

Revised: GPipe configuration: We use a popular PyTorch-based implementation of GPipe 6 .

Revised: The model is partitioned into 4 stages repeated over 4 model-parallel groups.

Revised: To fit into the GPU memory for the “GPT-3” configuration, we offload the optimizer into RAM using ZeRO-Offload.

Revised: Before averaging, we use PyTorch’s built-in All-Reduce to aggregate gradients.

Revised: ZeRO-Offload configuration: Each worker runs the entire model individually, then aggregates gradients with peers.

Revised: For “xxlarge”, we use the official implementation from Ren et al.

Revised: However, for “GPT-3”, we found that offloading the optimizer still does not allow us to fit 4 layers into the GPU memory even with batch size 1.

Revised: For this reason, we also offload parameters into RAM using “offload_param” with device set to “cpu” 7 .

Revised: [Table]

Revised: [Table]

Revised: We report our measurements in Tables 4 and 5.

Revised: Table 4 demonstrates that for larger model size,

Revised: SWARM and GPipe have approximately the same performance without latency.

Revised: When training with latency, SWARM significantly outperforms GPipe, which is likely caused by the asynchronous queueing (see Appendix D).

Revised: In turn, when training smaller models (Table 5), ZeRO-Offload outperforms both SWARM and GPipe.

Revised: This result coincides with our earlier observations in Figure 1, where the same model spent the most of the time waiting for communication between pipeline stages.

Revised: We also observe that ZeRO-Offload takes longer to aggregate gradients, likely because each peer must aggregate the entire model, whereas in SWARM and GPipe peers aggregate a single pipeline stage.

Revised: The variation between All-Reduce time in GPipe and SWARM is due to implementation differences.

Revised: G A DAPTIVE REBALANCING EVALUATION

Revised: In this experiment, we evaluate the efficiency of adaptive peer rebalancing between stages proposed in

Revised: Section 3.2.

Revised: We use actual statistics of the number of active T4 nodes from the 32-hour segment of the experiment described in Section 4.3 for a representative sample of training dynamics with unstable participation.

Revised: We use these data to simulate training dynamics as follow: we use a sequence of events, each consisting of a timestamp and the change in the number of peers (which can be positive or negative).

Revised: When a worker is removed from the pipeline, we randomly choose the stage it was removed from: that is, removing N peers corresponds to N samples from the uniform distribution over four pipeline stages.

Revised: To compare our method with the baseline without rebalancing, we runsimulations over different random seeds and average the resulting trajectories.

Revised: The results of this evaluation are available in Figure 5; for reference, we also provide the performance of a theoretically optimal rebalancing strategy that maintains the highest possible throughput at every moment.

Revised: It can be seen that even with rebalancing period T = 300 , adding dynamic rebalancing helps significantly improve the overall throughput of the pipeline during the experiment.

Revised: When the total number of peers is approximately stable, the rebalanced pipeline also reaches the optimal one in terms of throughput, which shows the efficiency of our strategy even when moving only one node at a time.

Revised: In addition, we observed that for some brief periods, the performance of the unbalanced pipeline exceeded the throughput of the balanced one due to random choice of disconnecting peers (dropping more from “overrepresented” stages affects the imbalanced pipeline less).

Revised: However, this held true only for ≈ 4 .

Revised: 5% of the entire experiment and was quickly mitigated by adaptive rebalancing.

Revised: As expected, decreasing T from 300 to 60 seconds improves both the overall throughput and the speed of convergence to optimal pipeline performance; however, the effect is not as drastic compared to the increase in DHT data transfer.

Revised: This is also demonstrated by Table 6, which shows the relative throughput of the three configurations compared to the optimal one.

Revised: Furthermore, the table displays that although initially there is little difference between rebalancing choices, it becomes more pronounced later on as the imbalanced version drifts further from the optimal state.

Revised: [Figure]

Revised: Rebalancing

Revised: [Table]