Original: 1- BIT LAMB: C OMMUNICATION E FFICIENT L ARGES CALE
Revised: 1- BIT LAMB: C OMMUNICATION E FFICIENT L ARGES CALE

Original: L ARGE -B ATCH
Revised: L ARGE B ATCH

Original: T RAINING WITH LAMB’ S
Revised: T RAINING WITH LAMB’ S

Original: C ONVERGENCE S PEED
Revised: C ONVERGENCE S PEED

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: To train large models (like BERT and GPT-3) on hundreds of GPUs, communication has become a major bottleneck, especially on commodity systems with limitedbandwidth TCP network.
Revised: To train large models (like BERT and GPT-3) on hundreds of GPUs, communication has become a major bottleneck, especially on commodity systems with limitedbandwidth TCP network.

Original: On one side large batch-size optimization such as LAMB algorithm was proposed to reduce the frequency of communication.
Revised: On one side large batch-size optimization such as LAMB algorithm was proposed to reduce the frequency of communication.

Original: On the other side, communication compression algorithms such as 1-bit Adam help to reduce the volume of each communication.
Revised: On the other side, communication compression algorithms such as 1-bit Adam help to reduce the volume of each communication.

Original: However, we ﬁnd that simply using one of the techniques is not sufﬁcient to solve the communication challenge, especially under low network bandwidth.
Revised: However, we ﬁnd that simply using one of the techniques is not sufﬁcient to solve the communication challenge, especially under low network bandwidth.

Original: Motivated by this we aim to combine the power of large-batch optimization and communication compression, but we ﬁnd that existing compression strategies cannot be directly applied to LAMB due to its unique adaptive layerwise learning rates.
Revised: Motivated by this we aim to combine the power of large-batch optimization and communication compression, but we ﬁnd that existing compression strategies cannot be directly applied to LAMB due to its unique adaptive layerwise learning rates.

Original: To this end, we design a new communicationefﬁcient algorithm, 1-bit LAMB, which introduces a novel way to support adaptive layerwise learning rates under compression.
Revised: To this end, we design a new communicationefﬁcient algorithm, 1-bit LAMB, which introduces a novel way to support adaptive layerwise learning rates under compression.

Original: In addition, we introduce a new system implementation for compressed communication using the NCCL backend of PyTorch distributed, which improves both usability and performance.
Revised: In addition, we introduce a new system implementation for compressed communication using the NCCL backend of PyTorch distributed, which improves both usability and performance.

Original: For BERTLarge pre-training task with batch sizes from 8K to 64K, our evaluations on up to 256 GPUs demonstrate that 1-bit LAMB with NCCL-based backend is able to achieve up to 4.6x communication volume reduction, up to 2.8x end-to-end timewise speedup, and the same sample-wise convergence speed (and same ﬁne-tuning task accuracy) compared to uncompressed LAMB.
Revised: For BERTLarge pre-training task with batch sizes from 8K to 64K, our evaluations on up to 256 GPUs demonstrate that 1-bit LAMB with NCCL-based backend is able to achieve up to 4.6x communication volume reduction, up to 2.8x end-to-end timewise speedup, and the same sample-wise convergence speed (and same ﬁne-tuning task accuracy) compared to uncompressed LAMB.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Training large-scale deep learning models in a distributed fashion is computation-heavy and expensive (Brown et al., 2020).
Revised: Training large-scale deep learning models in a distributed fashion is computation-heavy and expensive (Brown et al., 2020).

Original: In addition to computation, communication overhead becomes a serious system challenge for such large models.
Revised: In addition to computation, communication overhead becomes a serious system challenge for such large models.

Original: A recent study of BERT pre-training with Adam demonstrates that the allreduce communication can take up to 94% and 75% of total training time per step on clusters with Ethernet and InﬁniBand inter-node connections, respectively (Tang et al., 2021).
Revised: A recent study of BERT pre-training with Adam demonstrates that the allreduce communication can take up to 94% and 75% of total training time per step on clusters with Ethernet and InﬁniBand inter-node connections, respectively (Tang et al., 2021).

Original: To achieve communication efﬁcient distributed training, there are two promising directions: large batch optimization and communication compression.
Revised: To achieve communication efﬁcient distributed training, there are two promising directions: large batch optimization and communication compression.

Original: LAMB optimizer, which can be viewed as
Revised: LAMB optimizer, which can be viewed as

Original: Adam with adaptive layerwise learning rates, is an example of large batch optimization (You et al., 2020).
Revised: Adam with adaptive layerwise learning rates, is an example of large batch optimization (You et al., 2020).

Original: LAMB can scale the batch size of BERT pre-training to 64K without losing accuracy, thereby greatly reducing the total training time as larger batch sizes leads to less frequent communication.
Revised: LAMB can scale the batch size of BERT pre-training to 64K without losing accuracy, thereby greatly reducing the total training time as larger batch sizes leads to less frequent communication.

Original: On the other hand, recent works on communication compression such as 1-bit Adam demonstrate that it is possible to combine 1-bit compression with Adam’s convergence speed, thereby reduce BERT pre-training communication volume by 5x (Tang et al., 2021).
Revised: On the other hand, recent works on communication compression such as 1-bit Adam demonstrate that it is possible to combine 1-bit compression with Adam’s convergence speed, thereby reduce BERT pre-training communication volume by 5x (Tang et al., 2021).

Original: Both LAMB and 1-bit Adam demonstrate great beneﬁt for distributed training.
Revised: Both LAMB and 1-bit Adam demonstrate great beneﬁt for distributed training.

Original: Unfortunately, our studies show that simply using one of them is not sufﬁcient to fully address the communication issue, especially under limited network bandwidth and large number of GPUs/machines (Section 3).
Revised: Unfortunately, our studies show that simply using one of them is not sufﬁcient to fully address the communication issue, especially under limited network bandwidth and large number of GPUs/machines (Section 3).

Original: We ﬁnd that communication is still a non-trivial overhead when running large-scale distributed training with LAMB, even with the larger batch sizes.
Revised: We ﬁnd that communication is still a non-trivial overhead when running large-scale distributed training with LAMB, even with the larger batch sizes.

Original: Previous study shows that Adam provides slower convergence speed compared to LAMB at batch sizes 16K or larger for BERT pre-training (You et al., 2020).
Revised: Previous study shows that Adam provides slower convergence speed compared to LAMB at batch sizes 16K or larger for BERT pre-training (You et al., 2020).

Original: Using the same methodology, our BERT experiments show that 1-bit Adam, similar to Adam, also has slower convergence speed compared to LAMB at batch size
Revised: Using the same methodology, our BERT experiments show that 1-bit Adam, similar to Adam, also has slower convergence speed compared to LAMB at batch size

Original: 16K.
Revised: 16K.

Original: Even with the communication compression, this batch size limitation would hurt the communication efﬁciency when the number of GPUs/machines is large.
Revised: Even with the communication compression, this batch size limitation would hurt the communication efﬁciency when the number of GPUs/machines is large.

Original: LAMB and 1-bit Adam are two unique optimizers.
Revised: LAMB and 1-bit Adam are two unique optimizers.

Original: However, the techniques behind them are complementary: large batch optimization reduces the frequency of communication, and compression reduces the volume of communication.
Revised: However, the techniques behind them are complementary: large batch optimization reduces the frequency of communication, and compression reduces the volume of communication.

Original: Motivated by this we aim to combine LAMB’s large batch optimization algorithm with compression strategies behind 1-bit Adam.
Revised: Motivated by this we aim to combine LAMB’s large batch optimization algorithm with compression strategies behind 1-bit Adam.

Original: However, we ﬁnd that they are not directly compatible due to LAMB’s unique layerwise learning rate update strategy, which requires information that are missing when communication and optimizer states are compressed (Section 3).
Revised: However, we ﬁnd that they are not directly compatible due to LAMB’s unique layerwise learning rate update strategy, which requires information that are missing when communication and optimizer states are compressed (Section 3).

Original: The studies and challenges above motivate us to design a new algorithm called 1-bit LAMB (Section 4).
Revised: The studies and challenges above motivate us to design a new algorithm called 1-bit LAMB (Section 4).

Original: Learning from the insights behind 1-bit Adam, 1-bit LAMB is a 2-stage algorithm which uses
Revised: Learning from the insights behind 1-bit Adam, 1-bit LAMB is a 2-stage algorithm which uses

Original: LAMB (warmup stage) to “pre-condition” a communication compressed momentum SGD algoirthm (compression stage).
Revised: LAMB (warmup stage) to “pre-condition” a communication compressed momentum SGD algoirthm (compression stage).

Original: At compression stage where original LAMB algorithm cannot be used to update the layerwise learning rates, 1-bit LAMB employs a novel way to adaptively scale layerwise learning rates based on information from both warmup and compression stages.
Revised: At compression stage where original LAMB algorithm cannot be used to update the layerwise learning rates, 1-bit LAMB employs a novel way to adaptively scale layerwise learning rates based on information from both warmup and compression stages.

Original: As a result, 1-bit
Revised: As a result, 1-bit

Original: LAMB is able to achieve large batch optimization (LAMB)’s convergence speed under compressed communication, which is impossible using existing approaches.
Revised: LAMB is able to achieve large batch optimization (LAMB)’s convergence speed under compressed communication, which is impossible using existing approaches.

Original: In addition to the 1-bit LAMB algorithm, we propose a new NCCL-based compressed communication backend which provides better usability and performance than previous work (Section 5).
Revised: In addition to the 1-bit LAMB algorithm, we propose a new NCCL-based compressed communication backend which provides better usability and performance than previous work (Section 5).

Original: This backend can be applied to 1-bit LAMB, 1-bit Adam, and other communication compression algorithms.
Revised: This backend can be applied to 1-bit LAMB, 1-bit Adam, and other communication compression algorithms.

Original: We evaluate 1-bit LAMB using BERT pre-training and GLUE/SQuAD ﬁne-tuning tasks (Section 6).
Revised: We evaluate 1-bit LAMB using BERT pre-training and GLUE/SQuAD ﬁne-tuning tasks (Section 6).

Original: Results show that under different batch sizes from 8K to 64K and with up to 256 GPUs, 1-bit LAMB with NCCL-based backend is able to achieve up to 4.6x communication volume reduction and up to 2.8x end-to-end time-wise speedup for BERT pre-training compared to uncompressed LAMB, together with the same sample-wise convergence speed and same GLUE/SQuAD ﬁne-tuning task accuracy.
Revised: Results show that under different batch sizes from 8K to 64K and with up to 256 GPUs, 1-bit LAMB with NCCL-based backend is able to achieve up to 4.6x communication volume reduction and up to 2.8x end-to-end time-wise speedup for BERT pre-training compared to uncompressed LAMB, together with the same sample-wise convergence speed and same GLUE/SQuAD ﬁne-tuning task accuracy.

Original: The 1-bit LAMB optimizer as well as the NCCL-based communication backend has been open sourced in a deep learning optimization library (name hidden to maintain anonymity).
Revised: The 1-bit LAMB optimizer as well as the NCCL-based communication backend has been open sourced in a deep learning optimization library (name hidden to maintain anonymity).

Original:  R ELATED W ORK AND B ACKGROUND
Revised:  R ELATED W ORK AND B ACKGROUND

Original: To achieve communication efﬁcient distributed training, techniques include decentralization (Lian et al., 2017; Koloskova* et al., 2020;
Revised: To achieve communication efﬁcient distributed training, techniques include decentralization (Lian et al., 2017; Koloskova* et al., 2020;

Original: Li et al., 2018), asynchronous communication (Zheng et al., 2016; Chaturapruek et al., 2015), and gradient compression/quantization which we focus on in this paper.
Revised: Li et al., 2018), asynchronous communication (Zheng et al., 2016; Chaturapruek et al., 2015), and gradient compression/quantization which we focus on in this paper.

Original: Before communication, we could compress the original gradient g into C ω
Revised: Before communication, we could compress the original gradient g into C ω

Original: [ g ] , where
Revised: [ g ] , where

Original: C ω [ · ] is the compress operator 1 .
Revised: C ω [ · ] is the compress operator 1 .

Original: As a result the communication volume could be greatly reduced.
Revised: As a result the communication volume could be greatly reduced.

Original: Compression can be achieved by quantization, sparsiﬁcation, sketching, etc.
Revised: Compression can be achieved by quantization, sparsiﬁcation, sketching, etc.

Original: (Ye & Abbe, 2018;
Revised: (Ye & Abbe, 2018;

Original: Alistarh et al., 2017; Agarwal et al., 2018; Yu et al., 2019; Spring et al., 2019; Ivkin et al., 2019;
Revised: Alistarh et al., 2017; Agarwal et al., 2018; Yu et al., 2019; Spring et al., 2019; Ivkin et al., 2019;

Original: Shi et al., 2021).
Revised: Shi et al., 2021).

Original: Several works focus on unbiased compression methods (original and compressed tensors have the same expectation), such as centralized compressed parallel SGD (Alistarh et al., 2017) and many others (Wangni et al., 2018;
Revised: Several works focus on unbiased compression methods (original and compressed tensors have the same expectation), such as centralized compressed parallel SGD (Alistarh et al., 2017) and many others (Wangni et al., 2018;

Original: Shen et al., 2018; Zhang et al., 2017; Wen et al., 2017; Jiang & Agrawal, 2018).
Revised: Shen et al., 2018; Zhang et al., 2017; Wen et al., 2017; Jiang & Agrawal, 2018).

Original: On the other hand, recent works about biased compression methods demonstrate better compression rate and the same convergence rate by using an error compensation technique (Seide et al., 2014;
Revised: On the other hand, recent works about biased compression methods demonstrate better compression rate and the same convergence rate by using an error compensation technique (Seide et al., 2014;

Original: Bernstein et al., 2019; Stich et al., 2018; Zheng et al., 2019; Phuong &
Revised: Bernstein et al., 2019; Stich et al., 2018; Zheng et al., 2019; Phuong &

Original: Phong, 2020; Yu et al., 2019; Shi et al., 2019; Ivkin et al., 2019; Sun et al., 2019; Basu et al., 2019;
Revised: Phong, 2020; Yu et al., 2019; Shi et al., 2019; Ivkin et al., 2019; Sun et al., 2019; Basu et al., 2019;

Original: Vogels et al., 2019; Tang et al., 2021).
Revised: Vogels et al., 2019; Tang et al., 2021).

Original: The error-compensated compression is proposed in the 1-bit SGD work (Seide et al., 2014): instead of compressing the gradient at each iteration directly, they compress the sum of the gradient and the last step’s compression error.
Revised: The error-compensated compression is proposed in the 1-bit SGD work (Seide et al., 2014): instead of compressing the gradient at each iteration directly, they compress the sum of the gradient and the last step’s compression error.

Original: By using error compensation the training can achieve promising convergence speed even with 1-bit compression (representing the gradient by ± 1 signs and a scale).
Revised: By using error compensation the training can achieve promising convergence speed even with 1-bit compression (representing the gradient by ± 1 signs and a scale).

Original: Recent works provide theoretical guarantee of this method (Bernstein et al., 2019), and also demonstrate that it admits the same asymptotic convergence rate as the uncompressed one (Stich et al., 2018).
Revised: Recent works provide theoretical guarantee of this method (Bernstein et al., 2019), and also demonstrate that it admits the same asymptotic convergence rate as the uncompressed one (Stich et al., 2018).

Original: In addition, error compensation method enables almost any compression methods (Stich et al., 2018), either biased or unbiased, to converge as fast as the uncompressed case.
Revised: In addition, error compensation method enables almost any compression methods (Stich et al., 2018), either biased or unbiased, to converge as fast as the uncompressed case.

Original: Adam (Kingma & Ba, 2015) can be viewed as SGD with momentum and adaptive learning rate scaling on each coordinate of the gradient.
Revised: Adam (Kingma & Ba, 2015) can be viewed as SGD with momentum and adaptive learning rate scaling on each coordinate of the gradient.

Original: It has demonstrated promising convergence speed and hyperparameter robustness on many deep learning tasks.
Revised: It has demonstrated promising convergence speed and hyperparameter robustness on many deep learning tasks.

Original: Recently, Tang et al.
Revised: Recently, Tang et al.

Original: (2021) proposed 1-bit Adam which combines the efﬁciency of error-compensated 1-bit compression with Adam’s convergence speed.
Revised: (2021) proposed 1-bit Adam which combines the efﬁciency of error-compensated 1-bit compression with Adam’s convergence speed.

Original: They show that error-compensated compression does not work for Adam directly, because Adam is non-linearly dependent on the gradient (the variance term).
Revised: They show that error-compensated compression does not work for Adam directly, because Adam is non-linearly dependent on the gradient (the variance term).

Original: On the other hand, they ﬁnd that Adam’s variance becomes stable at an early stage of training.
Revised: On the other hand, they ﬁnd that Adam’s variance becomes stable at an early stage of training.

Original: To this end, they design a new 2-stage algorithm, 1-bit Adam: At warmup stage, vanilla Adam is used.
Revised: To this end, they design a new 2-stage algorithm, 1-bit Adam: At warmup stage, vanilla Adam is used.

Original: At compression stage, they stop updating the variance and use it as a ﬁxed precondition, and communicate based on the momentum applied with error-compensated 1-bit compression.
Revised: At compression stage, they stop updating the variance and use it as a ﬁxed precondition, and communicate based on the momentum applied with error-compensated 1-bit compression.

Original: Their experiments on up to 256 GPUs show that 1-bit Adam achieve the same convergence behaviour and ﬁnal accuracy as Adam, together with up to 5x less communication volume and 3.3x faster end-to-end throughput.
Revised: Their experiments on up to 256 GPUs show that 1-bit Adam achieve the same convergence behaviour and ﬁnal accuracy as Adam, together with up to 5x less communication volume and 3.3x faster end-to-end throughput.

Original: To further improve training efﬁciency at large scale, being able to support large minibatches while keeping the convergence speed is a critical factor.
Revised: To further improve training efﬁciency at large scale, being able to support large minibatches while keeping the convergence speed is a critical factor.

Original: Recently You et al.
Revised: Recently You et al.

Original: ﬁnd that it is difﬁcult to keep Adam’s convergence speed at batch sizes 16K or larger for BERT pre-training.
Revised: ﬁnd that it is difﬁcult to keep Adam’s convergence speed at batch sizes 16K or larger for BERT pre-training.

Original: To this end they proposed LAMB which can be viewed as Adam with adaptive layerwise learning rates.
Revised: To this end they proposed LAMB which can be viewed as Adam with adaptive layerwise learning rates.

Original: By using
Revised: By using

Original: LAMB, they are able to scale the batch size of BERT pre-training to 64 K without losing accuracy, thereby, reducing the BERT training time from 3 days to around 76 minutes.
Revised: LAMB, they are able to scale the batch size of BERT pre-training to 64 K without losing accuracy, thereby, reducing the BERT training time from 3 days to around 76 minutes.

Original: The major idea of
Revised: The major idea of

Original: LAMB is that it utilizes a layerwise scaling coefﬁcient to regulate the update of each layer, and the updating rule can be summarized as 2 :
Revised: LAMB is that it utilizes a layerwise scaling coefﬁcient to regulate the update of each layer, and the updating rule can be summarized as 2 :

Original: [Equation]
Revised: [Equation]

Original: Here g ( l ) t = ∇ F ( x t ; ξ t ) , m ( l ) t , v ( l ) t , x ( l ) t denote the stochastic gradient, momentum, second moment (i.e., the variance), and the model parameters at the model’s l -th layer at step t ; β 1 and β 2 are the decaying factor; γ is the learning rate; η is an additive constant to avoid division by 0; clip ( x, a, b ) :
Revised: Here g ( l ) t = ∇ F ( x t ; ξ t ) , m ( l ) t , v ( l ) t , x ( l ) t denote the stochastic gradient, momentum, second moment (i.e., the variance), and the model parameters at the model’s l -th layer at step t ; β 1 and β 2 are the decaying factor; γ is the learning rate; η is an additive constant to avoid division by 0; clip ( x, a, b ) :

Original: = min { max { x, a } , b } is the clipping operation 3 ; c ( l ) t is a layer-wise scaling factor that regulates the update of x ( l ) t into certain range.
Revised: = min { max { x, a } , b } is the clipping operation 3 ; c ( l ) t is a layer-wise scaling factor that regulates the update of x ( l ) t into certain range.

Original: One thing to note is that within each layer, each tensor (e.g., weight and bias) will have its own scaling coefﬁcient c ( l ) t .
Revised: One thing to note is that within each layer, each tensor (e.g., weight and bias) will have its own scaling coefﬁcient c ( l ) t .

Original: The underlying intuition of LAMB’s scaling coefﬁcient is that when the update is relatively large compared to the parameter, we should apply a lower learning rate to that layer (and vice versa).
Revised: The underlying intuition of LAMB’s scaling coefﬁcient is that when the update is relatively large compared to the parameter, we should apply a lower learning rate to that layer (and vice versa).

Original:  M OTIVATION AND I NSIGHTS 3. 1- BIT A DAM IS NOT SUFFICIENT FOR LARGE - BATCH DISTRIBUTED TRAINING
Revised:  M OTIVATION AND I NSIGHTS 3. 1- BIT A DAM IS NOT SUFFICIENT FOR LARGE - BATCH DISTRIBUTED TRAINING

Original: 1-bit Adam demonstrates the same convergence speed as Adam for BERT pre-training task with batch size 4K (Tang et al., 2021).
Revised: 1-bit Adam demonstrates the same convergence speed as Adam for BERT pre-training task with batch size 4K (Tang et al., 2021).

Original: On the other hand, the LAMB work shows that it is difﬁcult to keep Adam’s convergence speed at batch sizes 16K or larger for BERT pre-training (You et al., 2020).
Revised: On the other hand, the LAMB work shows that it is difﬁcult to keep Adam’s convergence speed at batch sizes 16K or larger for BERT pre-training (You et al., 2020).

Original: To ﬁnd out whether 1-bit Adam is sufﬁcient for large-batch distributed training, we perform a similar experiment using BERT pre-training task at batch size 16K.
Revised: To ﬁnd out whether 1-bit Adam is sufﬁcient for large-batch distributed training, we perform a similar experiment using BERT pre-training task at batch size 16K.

Original: Using You et al.
Revised: Using You et al.

Original: (2020)’s training parameters and tuning procedure (details in Appendix A.1) for LAMB and Adam, we perform BERT pre-training with LAMB and 1-bit Adam, respectively.
Revised: (2020)’s training parameters and tuning procedure (details in Appendix A.1) for LAMB and Adam, we perform BERT pre-training with LAMB and 1-bit Adam, respectively.

Original: Then we use the two pre-trained BERT model to perform SQuAD 1.1 ﬁne-tuning (details in Section 6).
Revised: Then we use the two pre-trained BERT model to perform SQuAD 1.1 ﬁne-tuning (details in Section 6).

Original: Results in Table 1 show that similar to
Revised: Results in Table 1 show that similar to

Original: Adam, 1-bit Adam has slower convergence speed compared to LAMB at larger batch size.
Revised: Adam, 1-bit Adam has slower convergence speed compared to LAMB at larger batch size.

Original: [Table] [Figure] 
Revised: [Figure] [Table]

Original: [Table] [Figure] [Table] [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:  C OMMUNICATION IS STILL AN OVERHEAD FOR LARGE - BATCH DISTRIBUTED TRAINING 
Revised: 3. C OMMUNICATION IS STILL AN OVERHEAD FOR LARGE - BATCH DISTRIBUTED TRAINING Tang

Original: Tang et al.
Revised: et al.

Original: (2021) demonstrate that when pre-training BERT with Adam at batch sizes 64 to 4K, the communication may take up to 94% and 75% of total training time per step on clusters with Ethernet and InﬁniBand networks.
Revised: (2021) demonstrate that when pre-training BERT with Adam at batch sizes 64 to 4K, the communication may take up to 94% and 75% of total training time per step on clusters with Ethernet and InﬁniBand networks.

Original: To investigate whether the communication overhead still exists, we conduct a similar proﬁling experiments but with LAMB optimizer and batch sizes 8K to 64K. We evaluate two clusters: one with 4 NVIDIA V100 GPUs per node and 40 Gigabit Ethernet inter-node network (4.
Revised: To investigate whether the communication overhead still exists, we conduct a similar proﬁling experiments but with LAMB optimizer and batch sizes 8K to 64K. We evaluate two clusters: one with 4 NVIDIA V100 GPUs per node and 40 Gigabit Ethernet inter-node network (4.

Original: Gbps effective bandwidth); the other one with 8 V100 GPUs per node and 100 Gigabit InﬁniBand
Revised: Gbps effective bandwidth); the other one with 8 V100 GPUs per node and 100 Gigabit InﬁniBand

Original: EDR inter-node network (close to theoretical peak effective bandwidth).
Revised: EDR inter-node network (close to theoretical peak effective bandwidth).

Original: Results show that even with larger batch sizes the communication still contributes up to 91% and 52% of the training time on two clusters (details in Appendix A.2), indicating the opportunities to improve large-batch distributed training efﬁciency by communication compression.
Revised: Results show that even with larger batch sizes the communication still contributes up to 91% and 52% of the training time on two clusters (details in Appendix A.2), indicating the opportunities to improve large-batch distributed training efﬁciency by communication compression.

Original: 3. I NVESTIGATING BERT PRE - TRAINING WITH BASELINE LAMB
Revised: 3. I NVESTIGATING BERT PRE - TRAINING WITH BASELINE LAMB

Original: Tang et al.
Revised: Tang et al.

Original: ﬁnd that Adam’s variance term becomes stable at an early stage of training (after around 15% of total training for BERT), which is why 1-bit Adam can “freeze” the variance after the warmup stage and use it as a ﬁxed precondition during compression.
Revised: ﬁnd that Adam’s variance term becomes stable at an early stage of training (after around 15% of total training for BERT), which is why 1-bit Adam can “freeze” the variance after the warmup stage and use it as a ﬁxed precondition during compression.

Original: LAMB also has the variance term as a non-linearly gradient dependency, and LAMB’s scaling coefﬁcient ( c ( l ) t in (1)) depends on the variance.
Revised: LAMB also has the variance term as a non-linearly gradient dependency, and LAMB’s scaling coefﬁcient ( c ( l ) t in (1)) depends on the variance.

Original: Thus we investigate how LAMB’s scaling coefﬁcient and variance change during training using BERT pre-training task.
Revised: Thus we investigate how LAMB’s scaling coefﬁcient and variance change during training using BERT pre-training task.

Original: Figure 1 presents LAMB’s scaling coefﬁcients and variance norms for different layers in the ﬁrst
Revised: Figure 1 presents LAMB’s scaling coefﬁcients and variance norms for different layers in the ﬁrst

Original: BertLayer (other layers in the model have similar patterns, details in Appendix A.3).
Revised: BertLayer (other layers in the model have similar patterns, details in Appendix A.3).

Original: Results demonstrate that the scaling coefﬁcients keep increasing until reaching plateaus, because the update tends to become smaller compared to the parameter during training.
Revised: Results demonstrate that the scaling coefﬁcients keep increasing until reaching plateaus, because the update tends to become smaller compared to the parameter during training.

Original: In addition, many scaling coefﬁcients become stable at an early stage.
Revised: In addition, many scaling coefﬁcients become stable at an early stage.

Original: Results also demonstrate that LAMB provides adaptive learning rate in two folds: 1) different layers may reach scaling coefﬁcient plateaus at different time; 2) different layers may reach different scaling coefﬁcient plateaus.
Revised: Results also demonstrate that LAMB provides adaptive learning rate in two folds: 1) different layers may reach scaling coefﬁcient plateaus at different time; 2) different layers may reach different scaling coefﬁcient plateaus.

Original: We believe that this is one of the reasons why LAMB can provide better convergence (or reach the same convergence with less hyperparameter tuning) at larger batch sizes compared with Adam.
Revised: We believe that this is one of the reasons why LAMB can provide better convergence (or reach the same convergence with less hyperparameter tuning) at larger batch sizes compared with Adam.

Original: On the other hand, LAMB’s varaince terms are less stable compared with Adam: many layers have their variance norms constantly decreasing during the whole training, up to two orders of magnitude difference.
Revised: On the other hand, LAMB’s varaince terms are less stable compared with Adam: many layers have their variance norms constantly decreasing during the whole training, up to two orders of magnitude difference.

Original: As we see in next section this makes 1-bit Adam’s strategy affect convergence when directly applied to LAMB. 
Revised: As we see in next section this makes 1-bit Adam’s strategy affect convergence when directly applied to LAMB. [Table]

Original: [Table]
Revised: [Table]

Original: E XISTING COMPRESSION STRATEGY AFFECTS LAMB’ S CONVERGENCE
Revised: E XISTING COMPRESSION STRATEGY AFFECTS LAMB’ S CONVERGENCE

Original: To combine 1-bit compression and large-batch training’s communication efﬁciency, ﬁrst we attempt to directly apply 1-bit Adam’s strategy to LAMB.
Revised: To combine 1-bit compression and large-batch training’s communication efﬁciency, ﬁrst we attempt to directly apply 1-bit Adam’s strategy to LAMB.

Original: We design and evaluate an experimental two-stage algorithm “LAMB + basic 1-bit”: At the warmup stage we use vanilla LAMB.
Revised: We design and evaluate an experimental two-stage algorithm “LAMB + basic 1-bit”: At the warmup stage we use vanilla LAMB.

Original: At the compression stage we stop updating the variance and LAMB’s scaling coefﬁcient and use them as precondition, and communicate based on 1-bit compressed momentum (details about this algorithm design in
Revised: At the compression stage we stop updating the variance and LAMB’s scaling coefﬁcient and use them as precondition, and communicate based on 1-bit compressed momentum (details about this algorithm design in

Original: Appendix A.4).
Revised: Appendix A.4).

Original: For simplicity we only apply this experimental algorithm to BERT pre-training seqlen 128 phase, and still only use vanilla LAMB in seqlen 512 phase.
Revised: For simplicity we only apply this experimental algorithm to BERT pre-training seqlen 128 phase, and still only use vanilla LAMB in seqlen 512 phase.

Original: Table 2 presents the BERT pre-training ﬁnal validation loss when using vanilla LAMB, the experimental algorithm described in this section, and the proposed 1-bit LAMB algorithm (Section 4).
Revised: Table 2 presents the BERT pre-training ﬁnal validation loss when using vanilla LAMB, the experimental algorithm described in this section, and the proposed 1-bit LAMB algorithm (Section 4).

Original: We also use the pre-trained models to ﬁne-tune SQuAD 1.1 and present the F1 scores.
Revised: We also use the pre-trained models to ﬁne-tune SQuAD 1.1 and present the F1 scores.

Original: Results show that simply freezing both variance and LAMB’s scaling coefﬁcient would affect the convergence speed and lead to lower SQuAD scores (and lower GLUE scores in Section 6).
Revised: Results show that simply freezing both variance and LAMB’s scaling coefﬁcient would affect the convergence speed and lead to lower SQuAD scores (and lower GLUE scores in Section 6).

Original: This is because LAMB’s variance term is less stable as demonstrated in our study.
Revised: This is because LAMB’s variance term is less stable as demonstrated in our study.

Original: On the other hand, the proposed 1-bit
Revised: On the other hand, the proposed 1-bit

Original: LAMB is able to provide the same convergence speed as vanilla LAMB, and next we will describe its algorithm design.
Revised: LAMB is able to provide the same convergence speed as vanilla LAMB, and next we will describe its algorithm design.

Original:  1- BIT LAMB A LGORITHM
Revised:  1- BIT LAMB A LGORITHM

Original: The proposed 1-bit LAMB optimizer introduces a novel way to update the adaptive layerwise learning rate during the compression stage.
Revised: The proposed 1-bit LAMB optimizer introduces a novel way to update the adaptive layerwise learning rate during the compression stage.

Original: There are two major differences between 1-bit LAMB and the original LAMB: 1) During compression stage, 1-bit LAMB updates the layerwise learning rate based on a novel “reconstructed gradient” based on the compressed momentum.
Revised: There are two major differences between 1-bit LAMB and the original LAMB: 1) During compression stage, 1-bit LAMB updates the layerwise learning rate based on a novel “reconstructed gradient” based on the compressed momentum.

Original: This makes 1-bit
Revised: This makes 1-bit

Original: LAMB compatible with error compensation and be able to keep track of the training dynamic under compression.
Revised: LAMB compatible with error compensation and be able to keep track of the training dynamic under compression.

Original: 2) 1-bit LAMB also introduces extra stabilized soft thresholds when updating layerwise learning rate at compression stage, which makes training more stable under compression.
Revised: 2) 1-bit LAMB also introduces extra stabilized soft thresholds when updating layerwise learning rate at compression stage, which makes training more stable under compression.

Original: Problem setting In this paper, we focus on the following optimization task:
Revised: Problem setting In this paper, we focus on the following optimization task:

Original: [Equation]
Revised: [Equation]

Original: where d is the dimension of the input model x , D i is the data distribution of individual data sample ξ ( i ) on the i -th worker, F ( x ; ξ ) is the loss function.
Revised: where d is the dimension of the input model x , D i is the data distribution of individual data sample ξ ( i ) on the i -th worker, F ( x ; ξ ) is the loss function.

Original: Notations and deﬁnitions Throughout this paper, we use the following notations:
Revised: Notations and deﬁnitions Throughout this paper, we use the following notations:

Original: • ∇ f ( · ) denotes the gradient of a function f .
Revised: • ∇ f ( · ) denotes the gradient of a function f .

Original: • f i ( x ) :=
Revised: • f i ( x ) :=

Original: E ξ ∼D i F ( x ; ξ ) .
Revised: E ξ ∼D i F ( x ; ξ ) .

Original: • (cid:107) · (cid:107) p denotes the l p -norm for vectors and matrices.
Revised: • (cid:107) · (cid:107) p denotes the l p -norm for vectors and matrices.

Original: Notice p = ∞ means the inﬁnity norm.
Revised: Notice p = ∞ means the inﬁnity norm.

Original: • C ω ( · ) denotes the randomized compressing operator, where ω denotes the random variable.
Revised: • C ω ( · ) denotes the randomized compressing operator, where ω denotes the random variable.

Original: One example is the randomized quantization operator, for example, C ω (0 . 7)
Revised: One example is the randomized quantization operator, for example, C ω (0 . 7)

Original: = 1 with probability 0 .
Revised: = 1 with probability 0 .

Original: 7 and C ω (0 . 7)
Revised: 7 and C ω (0 . 7)

Original: = 

Revised: =

Original: with probability 0 . 3 . • √· denotes the square root of the argument.
Revised: with probability 0 . 3 . • √· denotes the square root of the argument.

Original: In this paper if the argument is a vector, then it returns a vector taking the element-wise square root.
Revised: In this paper if the argument is a vector, then it returns a vector taking the element-wise square root.

Original: • ( x ) 2 denotes the element-wise square operation if x is a vector.
Revised: • ( x ) 2 denotes the element-wise square operation if x is a vector.

Original: • ab or a / b denotes the element-wise division operation if both a and b are vectors and their dimension matches. 
Revised: • ab or a / b denotes the element-wise division operation if both a and b are vectors and their dimension matches. [Table] [Table] [Table]

Revised: by δ ( i ) t

Original:  Take the average over all ˆ m ( i ) t it receives and compress it into m t =
Revised: = m ( i ) t + δ ( i ) t − 1 − ˆ m ( i ) t . Send the ˆ m ( i ) t to the server. 9: (On server) 10: Take the average over all ˆ m ( i ) t it receives and compress it into m t =

Original: C ω (cid:104) 1 n (cid:80) nj
Revised: C ω (cid:104) 1 n (cid:80) nj

Original: =1 ˆ m ( i ) t
Revised: =1 ˆ m ( i ) t

Original: + δ t − 1 (cid:105) , and update the compression error accordingly by δ t = 1 n (cid:80) nj
Revised: + δ t − 1 (cid:105) , and update the compression error accordingly by δ t = 1 n (cid:80) nj

Original: =1 ˆ m ( i ) t
Revised: =1 ˆ m ( i ) t

Original: + δ t − 1 − m t .
Revised: + δ t − 1 − m t .

Original: Send m t to all the workers.
Revised: Send m t to all the workers.

Original: 12: (On j -th node) 13:
Revised: 12: (On j -th node) 13:

Original: Set m t = m t .
Revised: Set m t = m t .

Original: 14: for the l -th layer do 15:
Revised: 14: for the l -th layer do 15:

Original: Reconstruct global gradient g ( l ) t = ( m ( l ) t − β 1 m ( l ) t − 1 ) / (1 − β 1 ) .
Revised: Reconstruct global gradient g ( l ) t = ( m ( l ) t [Equation] 1 m ( l ) t [Equation]

Original: 16: v ( l ) t = β 2 v ( l ) t − 1 + (1 − β 2 ) (cid:16) g
Revised: [Equation] [Equation]

Original: ( l ) t (cid:17) 2 .
Revised: [Equation] 17: r

Original: 17: r ( l ) t = (cid:13)(cid:13)(cid:13) v ( l ) Tw / v ( l ) t (cid:13)(cid:13)(cid:13) ∞ .
Revised: [Equation] 18: r ( l ) t = clip (cid:16) [Equation] [Equation]

Original: 18: r ( l ) t = clip (cid:16) r ( l ) t , (1 − r threshold ) × r ( l ) t − 1 , (1 + r threshold )
Revised: [Equation] [Equation] threshold )

Original: × r ( l ) t − 1 (cid:17) .
Revised: [Equation] ( l [Equation] [Equation]

Revised: r threshold

Revised: [Equation]

Revised: ×

Revised: [Equation]

Revised: ( l

Revised: [Equation]

Original:  19: r ( l ) t = clip (cid:16) r ( l ) t , r min , r max (cid:17) .
Revised: t [Equation] (cid:17) . 19: r ( l ) t = clip (cid:16) r ( l ) t , r min , r max (cid:17) .

Original: 20: c ( l ) t = r ( l ) t c ( l ) avg .
Revised: 20: c [Equation] = [Equation]

Original:  Update model of the l -th layer x ( l ) t = x ( l ) t − 1 − γc ( l ) t m ( l) t (cid:114) v ( l ) Tw .
Revised: avg . Update model of the l -th layer x ( l ) t = x ( l ) t − 1 − γc ( l ) [Equation] m ( l

Revised: [Equation]

Revised: (cid:114) v ( l ) Tw .

Original: 22: end for 23: end for 24: Output : x .
Revised: 22: end for 23: end for 24: Output : x .

Original: We summarize 1-bit LAMB in Algorithm 1.
Revised: We summarize 1-bit LAMB in Algorithm 1.

Original: During the compression stage, we freeze the variance in order to apply the error compensation mechanism correctly (Tang et al., 2021).
Revised: During the compression stage, we freeze the variance in order to apply the error compensation mechanism correctly (Tang et al., 2021).

Original: However, this brings two challenges for LAMB’s case: 1) We cannot update LAMB’s scaling coefﬁcients ( c ( l ) t in (1)) during compression stage based on LAMB algorithm, because it requires uncompressed momentum and up-to-date variance; 2) In vanilla LAMB the scaling coefﬁcients become stable during training.
Revised: However, this brings two challenges for LAMB’s case: 1) We cannot update LAMB’s scaling coefﬁcients ( c ( l ) t in (1)) during compression stage based on LAMB algorithm, because it requires uncompressed momentum and up-to-date variance; 2) In vanilla LAMB the scaling coefﬁcients become stable during training.

Original: However, because LAMB’s variance term is not stable for some layers but we freeze them during compression, we need to adjust scaling coefﬁcients during compression to compensate this.
Revised: However, because LAMB’s variance term is not stable for some layers but we freeze them during compression, we need to adjust scaling coefﬁcients during compression to compensate this.

Original: To this end, 1-bit LAMB uses a novel way to adaptively update LAMB scaling coefﬁcients during the compression stage to compensate the difference between frozen and actual variance.
Revised: To this end, 1-bit LAMB uses a novel way to adaptively update LAMB scaling coefﬁcients during the compression stage to compensate the difference between frozen and actual variance.

Original: During the warmup stage, we use vallia LAMB and keep track of the moving average of each layer’s scaling coefﬁcient (used during the compression stage because the scaling coefﬁcient is not stable at beginning).
Revised: During the warmup stage, we use vallia LAMB and keep track of the moving average of each layer’s scaling coefﬁcient (used during the compression stage because the scaling coefﬁcient is not stable at beginning).

Original: At the end of warmup, we stop updating the moving average, and store the frozen variance to be used during compression stage.
Revised: At the end of warmup, we stop updating the moving average, and store the frozen variance to be used during compression stage.

Original: On the other hand, we still keep updating another “fresh” variance by reconstructing the global gradient based on this and last step’s compressed momentum.
Revised: On the other hand, we still keep updating another “fresh” variance by reconstructing the global gradient based on this and last step’s compressed momentum.

Original: To update LAMB scaling coefﬁcient during compression stage, we compute a 1-bit LAMB scaling ratio which is the max element among the ( frozen variance / fresh variance ).
Revised: To update LAMB scaling coefﬁcient during compression stage, we compute a 1-bit LAMB scaling ratio which is the max element among the ( frozen variance / fresh variance ).

Original: We use the max element as the scaling ratio because it is effective and cheap to compute based on our experiments.
Revised: We use the max element as the scaling ratio because it is effective and cheap to compute based on our experiments.

Original: To avoid extreme ratios and dramatic change between ratios, we use two kinds of clipping conﬁgurable by the user.
Revised: To avoid extreme ratios and dramatic change between ratios, we use two kinds of clipping conﬁgurable by the user.

Original: (The clipping from vanilla LAMB algorithm are not used during the compression stage.)
Revised: (The clipping from vanilla LAMB algorithm are not used during the compression stage.)

Original: Then we compute this step’s LAMB scaling coefﬁcient using the 1-bit LAMB scaling ratio and the moving average at the end of warmup, and use it to update the model.
Revised: Then we compute this step’s LAMB scaling coefﬁcient using the 1-bit LAMB scaling ratio and the moving average at the end of warmup, and use it to update the model.

Original: Our study shows that those layers with less stable variance tend to have more dynamic 1-bit LAMB scaling ratios, which demonstrate the desired adaptiveness (details in Appendix A.5).
Revised: Our study shows that those layers with less stable variance tend to have more dynamic 1-bit LAMB scaling ratios, which demonstrate the desired adaptiveness (details in Appendix A.5).

Original: Appendix A.7 provides a theoretical analysis for 1-bit LAMB.
Revised: Appendix A.9 provides a theoretical analysis for 1-bit LAMB.

Original:  P ROPOSED S YSTEM D ESIGN
Revised:  P ROPOSED S YSTEM D ESIGN

Original: To realize the compressed communication at system level, Tang et al.
Revised: To realize the compressed communication at system level, Tang et al.

Original: (2021) designed a custom collective primitive, called “compressed allreduce”, using Message Passing Interface (MPI).
Revised: (2021) designed a custom collective primitive, called “compressed allreduce”, using Message Passing Interface (MPI).

Original: In addition to implementing 1-bit LAMB using this MPI-based backend, we introduce a new system implementation for compressed communication using the NCCL backend of PyTorch distributed, following the 3-phase design in the MPI-based backend: 1)
Revised: In addition to implementing 1-bit LAMB using this MPI-based backend, we introduce a new system implementation for compressed communication using the NCCL backend of PyTorch distributed, following the 3-phase design in the MPI-based backend: 1)

Original: The gather step, where each worker sends its i -th chunk to worker i , is implemented using NCCL’s Alltoall and AllGather.
Revised: The gather step, where each worker sends its i -th chunk to worker i , is implemented using NCCL’s Alltoall and AllGather.

Original: The average step, where each worker averages all chunks it receives.
Revised: The average step, where each worker averages all chunks it receives.

Revised: Here each worker acts as the server deﬁned in

Revised: Algorithm 1. 3)

Original: The scatter step, where each worker receives the average of all i -th chunks from worker i , is implemented using NCCL’s AllGather.
Revised: The scatter step, where each worker receives the average of all i -th chunks from worker i , is implemented using NCCL’s AllGather.

Original: Compared to the MPI-based, this new NCCL-based implementation signiﬁcantly improves the usability since NCCL is integrated with PyTorch distributed.
Revised: Compared to the MPI-based, this new NCCLbased implementation signiﬁcantly improves the usability since NCCL is integrated with PyTorch distributed.

Original: In addition, evaluations show that the performance of the NCCL-based implementation is better than the MPI-based for Ethernet-based systems and on-par for InﬁniBand-based systems.
Revised: In addition, evaluations show that the performance of the NCCL-based implementation is better than the MPI-based for Ethernet-based systems and on-par for InﬁniBand-based systems.

Original: Thus we will mainly present the results with NCCL-based backend, but we also include a comparison between MPI and NCCL-based implementations in Appendix A.6.
Revised: Thus we will mainly present the results with NCCL-based backend, but we also include a comparison between MPI and NCCL-based implementations in Appendix A.6.

Original:  E VALUATION
Revised:  E VALUATION

Original: Dataset and models We evaluate the convergence and performance of 1-bit LAMB and uncompressed LAMB for BERT-Large ( L = 24 , H = 1024 , A = 16 , 340 M params) pre-training task.
Revised: Dataset and models We evaluate the convergence and performance of 1-bit LAMB and uncompressed LAMB for BERT-Large ( L = 24 , H = 1024 , A = 16 , 340 M params) pre-training task.

Original: We use the same dataset as Devlin et al.
Revised: We use the same dataset as Devlin et al.

Original: (2019), which is a concatenation of Wikipedia and BooksCorpus with 2.5B and 800M words respectively.
Revised: (2019), which is a concatenation of Wikipedia and BooksCorpus with 2.5B and 800M words respectively.

Original: Compared to the original BERT model, one notable change is that we applied PreLN instead of PostLN for better training stability (Zhang & He, 2020; Xiong et al., 2020).
Revised: Compared to the original BERT model, one notable change is that we applied PreLN instead of PostLN for better training stability (Zhang & He, 2020; Xiong et al., 2020).

Original: We use the GLUE ﬁne-tuning benchmark (Wang et al., 2018) and SQuAD 1.1 ﬁne-tuning task 4 to evaluate the convergence of the BERT models trained by LAMB and 1-bit LAMB.
Revised: We use the GLUE ﬁne-tuning benchmark (Wang et al., 2018) and SQuAD 1.1 ﬁne-tuning task 4 to evaluate the convergence of the BERT models trained by LAMB and 1-bit LAMB.

Original: Hardware We use the two clusters described in Section 3.2.
Revised: Hardware We use the two clusters described in Section 3.2.

Original: We use 8 to 256 GPUs for BERT pre-training tasks to measure 1-bit LAMB’s performance gain.
Revised: We use 8 to 256 GPUs for BERT pre-training tasks to measure 1-bit LAMB’s performance gain.

Original: For ﬁne-tuning tasks we use 4 GPUs.
Revised: For ﬁne-tuning tasks we use 4 GPUs.

Original: One thing to note is that because 1-bit LAMB introduces additional memory overhead (one persistent copy of the fresh varaince and one temporary copy of last step’s momentum) and because the V
Revised: One thing to note is that because 1-bit LAMB introduces additional memory overhead (one persistent copy of the fresh varaince and one temporary copy of last step’s momentum) and because the V

Original: GPUs on the Ethernet cluster have 16GB memory instead of 32GB, we were not able to ﬁt large batch sizes for BERT pre-training when the number of GPUs is small.
Revised: GPUs on the Ethernet cluster have 16GB memory instead of 32GB, we were not able to ﬁt large batch sizes for BERT pre-training when the number of GPUs is small.

Original: For seqlen 128 and 512, we need to use at least 8 and 16 GPUs on the Ethernet cluster, respectively.
Revised: For seqlen 128 and 512, we need to use at least 8 and 16 GPUs on the Ethernet cluster, respectively.

Original: Training parameters For BERT pre-training, we set the parameters in (1) as β 1 = 0 .
Revised: Training parameters For BERT pre-training, we set the parameters in (1) as β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , c min = 0 .
Revised: 999 , c min = 0 .

Original: 01 and c max = 0 .
Revised: 01 and c max = 0 .

Original: 3 for LAMB and 1-bit LAMB.
Revised: 3 for LAMB and 1-bit LAMB.

Original: For 1-bit LAMB, we set β 3 = 0 .
Revised: For 1-bit LAMB, we set β 3 = 0 .

Original: 9 , r min = 0 .
Revised: 9 , r min = 0 .

Original: 5 , r max = 4 . 0 , and r threshold = 0 .
Revised: 5 , r max = 4 . 0 , and r threshold = 0 .

Original: 1 in Algorithm 1.
Revised: 1 in Algorithm 1.

Original: For convergence analysis, we set total batch size as 64K for seqlen 128 and 32K for seqlen 512.
Revised: For convergence analysis, we set total batch size as 64K for seqlen 128 and 32K for seqlen 512.

Original: For performance analysis, we test different batch sizes from 8K to 64K.
Revised: For performance analysis, we test different batch sizes from 8K to 64K.

Original: For BERT pre-training seqlen 128, the learning rate starts from 1 × 10 − 3 , exponentially increases to 12 × 10 − 3 as a warmup in the ﬁrst 450 steps, then decays into 0.9 of the original after everysteps.
Revised: For BERT pre-training seqlen 128, the learning rate starts from 1 × 10 − 3 , exponentially increases to 12 × 10 − 3 as a warmup in the ﬁrst 450 steps, then decays into 0.9 of the original after everysteps.

Original: The total number of steps is 5993.
Revised: The total number of steps is 5993.

Original: For 1-bit LAMB we use the ﬁrst 1000 steps (16.7%) as the warmup stage.
Revised: For 1-bit LAMB we use the ﬁrst 1000 steps (16.7%) as the warmup stage.

Original: For BERT pre-training seqlen 512, the learning rate starts from 0 , exponentially increases to 2 × 10 − 3 as a warmup in the ﬁrst 150 steps, then decays into 0.9 of the original after
Revised: For BERT pre-training seqlen 512, the learning rate starts from 0 , exponentially increases to 2 × 10 − 3 as a warmup in the ﬁrst 150 steps, then decays into 0.9 of the original after

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: every we use the as the warmup

Original: [Table]
Revised: [Table]

Original: For 1-bit LAMB’s hyperparameter tuning, we ﬁnd that the only parameter that requires nontrivial tuning is the number of warmup steps, which depends on when do the LAMB’s scaling coefﬁcients become stable as discussed in Section 3.3 thus could potentially be automated.
Revised: For 1-bit LAMB’s hyperparameter tuning, we ﬁnd that the only parameter that requires nontrivial tuning is the number of warmup steps, which depends on when do the LAMB’s scaling coefﬁcients become stable as discussed in Section 3.3 thus could potentially be automated.

Original: For the other parameters ( β 3 , r min , r max , and r threshold in Algorithm 1) we ﬁnd tuning them does not lead to very different convergence since they are just used to avoid extreme value/dramatic change of LAMB’s scaling coefﬁcients.
Revised: For the other parameters ( β 3 , r min , r max , and r threshold in Algorithm 1) we ﬁnd tuning them does not lead to very different convergence since they are just used to avoid extreme value/dramatic change of LAMB’s scaling coefﬁcients.

Original: Similarly the clipping parameters c min and c max in (1) for LAMB and 1-bit
Revised: Similarly the clipping parameters c min and c max in (1) for LAMB and 1-bit

Original: LAMB do not require non-trivial tuning.
Revised: LAMB do not require non-trivial tuning.

Original: For GLUE benchmarks we use Adam optimizer and perform single-task training on the dev set.
Revised: For GLUE benchmarks we use Adam optimizer and perform single-task training on the dev set.

Original: Following the setup in the BERT paper (Devlin et al., 2019), we use a batch size of 32 and ﬁnetune for 3 epochs for all GLUE tasks.
Revised: Following the setup in the BERT paper (Devlin et al., 2019), we use a batch size of 32 and ﬁnetune for 3 epochs for all GLUE tasks.

Original: For each task, we select the best learning rate among { 2 × 10 − 5 , 3 × 10 − 5 , 4 × 10 − 5 , 5 × 10 − 5 } .
Revised: For each task, we select the best learning rate among { 2 × 10 − 5 , 3 × 10 − 5 , 4 × 10 − 5 , 5 × 10 − 5 } .

Original: For SQuAD ﬁne-tuning we use Adam optimizer and the same parameters as published by HuggingFace (batch size = 24 , learning rate = 3 × 10 − 5 , dropout = 0 .
Revised: For SQuAD ﬁne-tuning we use Adam optimizer and the same parameters as published by HuggingFace (batch size = 24 , learning rate = 3 × 10 − 5 , dropout = 0 .

Original: 1 , 2 epochs).
Revised: 1 , 2 epochs).

Original: Convergence analysis Figure 2 presents the BERT pre-training sample-wise convergence results.
Revised: Convergence analysis Figure 2 presents the BERT pre-training sample-wise convergence results.

Original: For both seqlen 128 and 512, 1-bit LAMB provides the same convergence speed as LAMB, while it takes much less time to process each batch due to communication compression.
Revised: For both seqlen 128 and 512, 1-bit LAMB provides the same convergence speed as LAMB, while it takes much less time to process each batch due to communication compression.

Original: We already presented the BERT pre-training validation loss and SQuAD ﬁne-tuning results in Section 3.4 Table 2.
Revised: We already presented the BERT pre-training validation loss and SQuAD ﬁne-tuning results in Section 3.4 Table 2.

Original: And
Revised: And

Original: Table 3 presents the GLUE results.
Revised: Table 3 presents the GLUE results.

Original: For both SQuAD and GLUE, results show that 1-bit LAMB provides the same ﬁne-tuning task accuracy as LAMB, while simply freezing both the varaince and
Revised: For both SQuAD and GLUE, results show that 1-bit LAMB provides the same ﬁne-tuning task accuracy as LAMB, while simply freezing both the varaince and

Original: LAMB’s scaling coefﬁcients would hurt the accuracy.
Revised: LAMB’s scaling coefﬁcients would hurt the accuracy.

Original: Performance analysis Computed as 1/(warmup_ratio + (1 - warmup_ratio)/16) for FP16 training, 1-bit LAMB offers 4 .
Revised: Performance analysis Computed as 1/(warmup_ratio + (1 - warmup_ratio)/16) for FP16 training, 1-bit LAMB offers 4 .

Original: 6 × and 4 .
Revised: 6 × and 4 .

Original: 1 × end-to-end communication volume reduction for BERT-Large seqlen 128 and 512, respectively.
Revised: 1 × end-to-end communication volume reduction for BERT-Large seqlen 128 and 512, respectively.

Original: To measure the actual end-to-end performance gain, ﬁrst we perform a throughput analysis where we run the warmup (i.e., baseline LAMB’s performance) and compression stage of 1-bit LAMB for 200 steps each, and measure the average throughput of the two stages.
Revised: To measure the actual end-to-end performance gain, ﬁrst we perform a throughput analysis where we run the warmup (i.e., baseline LAMB’s performance) and compression stage of 1-bit LAMB for 200 steps each, and measure the average throughput of the two stages.

Original: Figure 3 presents the results with NCCL-based compressed communication backend under different batch sizes and number of GPUs.
Revised: Figure 3 presents the results with NCCL-based compressed communication backend under different batch sizes and number of GPUs.

Original: For seqlen 128, 1-bit LAMB provides up to 4 .
Revised: For seqlen 128, 1-bit LAMB provides up to 4 .

Original: 5 × speedup during the compression stage, which is equivalent to 2 .
Revised: 5 × speedup during the compression stage, which is equivalent to 2 .

Original: 8 × end-to-end speedup (computed as 1/(warmup_ratio + (1 - warmup_ratio)/compression_stage_speedup)).
Revised: 8 × end-to-end speedup (computed as 1/(warmup_ratio + (1 - warmup_ratio)/compression_stage_speedup)).

Original: For seqlen 512, 1-bit LAMB provides up to 3 . 

Revised: For seqlen 512, 1-bit LAMB provides up to 3 .

Original: × speedup during the compression stage, which is equivalent to 2 .
Revised: × speedup during the compression stage, which is equivalent to 2 .

Original: 2 × end-to-end speedup.
Revised: 2 × end-to-end speedup.

Original: This demonstrates 1-bit LAMB’s better scalability compared to LAMB.
Revised: This demonstrates 1-bit LAMB’s better scalability compared to LAMB.

Original: It is also worth mentioning that 1-bit LAMB on Ethernet (4.1 Gbps effective bandwidth, 4 GPUs per node) is able
Revised: It is also worth mentioning that 1-bit LAMB on Ethernet (4.1 Gbps effective bandwidth, 4 GPUs per node) is able

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: to achieve comparable throughput as LAMB on InﬁniBand (near 100 Gbps effective bandwidth,
Revised: to achieve comparable throughput as LAMB on InﬁniBand (near 100 Gbps effective bandwidth,

Original: GPUs per node), which demonstrates 1-bit LAMB’s efﬁciency considering the hardware differences.
Revised: GPUs per node), which demonstrates 1-bit LAMB’s efﬁciency considering the hardware differences.

Original: In addition to throughput analysis, we also measure the total runtime of pre-training at batch size 64 / 32 K for both 1-bit LAMB and LAMB.
Revised: In addition to throughput analysis, we also measure the total runtime of pre-training at batch size 64 / 32 K for both 1-bit LAMB and LAMB.

Original: As shown in Table 4, overall 1-bit LAMB is able to provide 2 .
Revised: As shown in Table 4, overall 1-bit LAMB is able to provide 2 .

Original: 2 × and 1 .
Revised: 2 × and 1 .

Original: 5 × speedup for seqlen 128 and 512.
Revised: 5 × speedup for seqlen 128 and 512.

Original: These numbers are consistent with the endto-end speedup calculated in the throughput analysis ( 2 . 

Revised: These numbers are consistent with the endto-end speedup calculated in the throughput analysis ( 2 .

Original: × and 1 . 5 ×
Revised: × and 1 . 5 ×

Original: based on results in Figure 3(d) and 3(g)).
Revised: based on results in Figure 3(d) and 3(g)).

Original: For seqlen 128, the end-to-end speedup based on runtime is slightly higher than the speedup based on throughput.
Revised: For seqlen 128, the end-to-end speedup based on runtime is slightly higher than the speedup based on throughput.

Original: We ﬁnd that it is because uncompressed LAMB’s larger communication volume makes it more sensitive to the occasional ﬂuctuation of the actual network bandwidth.
Revised: We ﬁnd that it is because uncompressed LAMB’s larger communication volume makes it more sensitive to the occasional ﬂuctuation of the actual network bandwidth.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: To reduce both the frequency and volume of communications for large-scale training, we propose an error-compensated LAMB preconditioned momentum SGD algorithm, 1-bit LAMB, which combines the power of large batch optimization and communication compression by introducing an novel way to support adaptive layerwise learning rates during communication compression.
Revised: To reduce both the frequency and volume of communications for large-scale training, we propose an error-compensated LAMB preconditioned momentum SGD algorithm, 1-bit LAMB, which combines the power of large batch optimization and communication compression by introducing an novel way to support adaptive layerwise learning rates during communication compression.

Original: We also introduce an easier-to-use and more efﬁcient compressed communication backend system based on NCCL.
Revised: We also introduce an easier-to-use and more efﬁcient compressed communication backend system based on NCCL.

Original: Evaluations show that 1-bit LAMB with NCCL-based backend is able to achieve up to 4 .
Revised: Evaluations show that 1-bit LAMB with NCCL-based backend is able to achieve up to 4 .

Original: 6 × communication volume reduction and up to 2 .
Revised: 6 × communication volume reduction and up to 2 .

Original: 8 × end-to-end time-wise speedup for BERT pre-training compared to uncompressed LAMB, together with the same sample-wise convergence speed and ﬁne-tuning task accuracy.R
Revised: 8 × end-to-end time-wise speedup for BERT pre-training compared to uncompressed LAMB, together with the same sample-wise convergence speed and ﬁne-tuning task accuracy.

Original: A A PPENDIX A. T RAINING PARAMETERS FOR 1- BIT A DAM EXPERIMENTS IN S ECTION 3. 
Revised: A A PPENDIX A. T RAINING PARAMETERS FOR 1- BIT A DAM EXPERIMENTS IN S ECTION 3.

Original: For experiments in Section 3.1, for both LAMB and 1-bit Adam we use batch size = 16K, 28125/ steps for seqlen 128/512, weight decay = 0.01, linear LR warmup and decay.
Revised: For experiments in Section 3.1, for both LAMB and 1-bit Adam we use batch size = 16K, 28125/ steps for seqlen 128/512, weight decay = 0.01, linear LR warmup and decay.

Original: For LAMB, we use learning rate = 3 .
Revised: For LAMB, we use learning rate = 3 .

Original: 54 × 10 − 3 , 10% LR warmup, clipping conﬁgs ( c min and c max in (1)) as 0.
Revised: 54 × 10 − 3 , 10% LR warmup, clipping conﬁgs ( c min and c max in (1)) as 0.

Original: and 1.
Revised: and 1.

Original: For 1-bit Adam, we use learning rates ∈ { 1 × 10 − 4 , 2 × 10 − 4 , 3 × 10 − 4 } , LR warmup ∈ { 5% , 10% , 20% } .
Revised: For 1-bit Adam, we use learning rates ∈ { 1 × 10 − 4 , 2 × 10 − 4 , 3 × 10 − 4 } , LR warmup ∈ { 5% , 10% , 20% } .

Original: All of these training parameters (except LAMB clipping conﬁgs) are from the
Revised: All of these training parameters (except LAMB clipping conﬁgs) are from the

Original: LAMB paper.
Revised: LAMB paper.

Original: For 1-bit Adam, following the original work’s strategy we set the number of warmup steps as 4000 (out of total 28125 steps) for seqlen 128 and 475 (out of 3125) for seqlen 512.
Revised: For 1-bit Adam, following the original work’s strategy we set the number of warmup steps as 4000 (out of total 28125 steps) for seqlen 128 and 475 (out of 3125) for seqlen 512.

Original: A. D ETAILED RESULTS FOR PROFILING EXPERIMENTS IN S ECTION 3. 
Revised: A. D ETAILED RESULTS FOR PROFILING EXPERIMENTS IN S ECTION 3.

Original: Table 5 presents the detailed proﬁling results.
Revised: Table 5 presents the detailed proﬁling results.

Original: Results show that even with larger batch sizes the allreduce communication still contributes to a great portion of the training time per step, up to 91% and 52% on two different kinds of clusters.
Revised: Results show that even with larger batch sizes the allreduce communication still contributes to a great portion of the training time per step, up to 91% and 52% on two different kinds of clusters.

Original: And this overhead is larger when the number of nodes is larger, when the batch size is smaller, when the network bandwidth is lower.
Revised: And this overhead is larger when the number of nodes is larger, when the batch size is smaller, when the network bandwidth is lower

Original:  A. D ETAILED RESULTS FOR LAMB EXPERIMENTS IN S ECTION 3. 
Revised: . A. D ETAILED RESULTS FOR LAMB EXPERIMENTS IN S ECTION 3.

Original: Figure 4 presents LAMB’s scaling coefﬁcients for different layers during BERT pre-training sequence length 128 (sequence length 512 has similar patterns).
Revised: Figure 4 presents LAMB’s scaling coefﬁcients for different layers during BERT pre-training sequence length 128 (sequence length 512 has similar patterns).

Original: Only the cls.seq_relationship.bias has a very unstable scaling coefﬁcient.
Revised: Only the cls.seq_relationship.bias has a very unstable scaling coefﬁcient.

Original: This is because this bias only has two elements, representing the two states of whether the two sequences are next to each other.
Revised: This is because this bias only has two elements, representing the two states of whether the two sequences are next to each other.

Original: Figure 5 presents LAMB’s variance norms for different layers during BERT pre-training sequence length 128 (sequence length 512 has similar patterns).
Revised: Figure 5 presents LAMB’s variance norms for different layers during BERT pre-training sequence length 128 (sequence length 512 has similar patterns).

Original: We believe that there are two reasons why
Revised: We believe that there are two reasons why

Original: LAMB’s varaince terms are less stable: 1) LAMB has larger batch size, smaller number of steps, and layerwise adaptive learning rates compared to Adam. 2) Because LAMB requires the calculation of the scaling coefﬁcient for each layer, we cannot fuse all the variance together as in Adam.
Revised: LAMB’s varaince terms are less stable: 1) LAMB has larger batch size, smaller number of steps, and layerwise adaptive learning rates compared to Adam. 2) Because LAMB requires the calculation of the scaling coefﬁcient for each layer, we cannot fuse all the variance together as in Adam.

Original: And each separate variance could have less stable norm compared to a single fused variance.
Revised: And each separate variance could have less stable norm compared to a single fused variance.

Revised: A. D ESIGN OF THE EXPERIMENTAL ALGORITHM IN S ECTION 3.

Revised: Because LAMB’s scaling coefﬁcient is essentially part of the learning rate (and it’s just a scalar), updating this coefﬁcient would not affect communication compression’s error compensation mechanism

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Since all 24 BertLayer have similar patterns, we just present the ﬁrst one. The y-axis is in log scale.{+small enough between two steps. 
+}
Revised: Since all 24 BertLayer have similar patterns, we just present the ﬁrst one as long as the change is {+small enough between two steps. 
+}

Original: Because LAMB’s scaling coefﬁcient is essentially part of the learning rate (and it’s just a scalar), updating this  coefﬁcient would not affect communication compression’s error compensation mechanism as long as the change is small enough between two steps.
Revised: However, we ﬁnd that it’s challenging to update this coefﬁcient during compression stage because in LAMB’s algorithm, updating this scaling coefﬁcient requires both momentum and variance term. However, the error compensation mechanism requires freezing the variance term at the beginning of the compression stage due to its nonlinear dependency to the gradient.

Original: However, we ﬁnd that it’s challenging toupdate this coefﬁcient during compression stage because in LAMB’s algorithm, updating this scaling coefﬁcient requires both momentum and variance term. However, the error compensation mechanism
Revised: In addition, we ﬁnd that due to the error compensation mechanism, the norm of some layers’ momentum term could become larger/smaller compared to the uncompressed case. As a result, we ﬁnd that updating LAMB’s scaling coefﬁcients during compression stage based

Original: [Figure]
Revised: [Figure]

Original: layers during BERT pretraining sequence length 128.
Revised: layers during BERT pretraining sequence length 128.

Original: Since all 24 BertLayer have similar patterns, we just present the ﬁrst one.
Revised: Since all 24 BertLayer have similar patterns, we just present the ﬁrst one.

Original: The number of warmup steps is 1 K out of 5993 total steps.
Revised: The number of warmup steps is 1 K out of 5993 total steps.

Original: We set the clipping conﬁgurations of the scaling ratio ( r min , r max , r threshold in Algorithm 1) at 0.5, 4.0, 0.1.
Revised: We set the clipping conﬁgurations of the scaling ratio ( r min , r max , r threshold in Algorithm 1) at 0.5, 4.0, 0.1.

Original: requires freezing the variance term at the beginning of the compression stage due to its nonlinear dependency to the gradient. In addition, we ﬁnd that due to the error compensation mechanism, the norm of some layers’ momentum term could become larger/smaller compared to the uncompressed case. As a result, we ﬁnd that updating LAMB’s scaling coefﬁcients during compression stage based on original LAMB algorithm would produce suboptimal results and slower convergence speed.
Revised: on original LAMB algorithm would produce suboptimal results and slower convergence speed.

Original: Thus for Section 3.4’s experimental algorithm, we stop updating LAMB’s scaling coefﬁcients during compression stage, and just use a calculated scaling coefﬁcient moving average at the end of the warmup stage.
Revised: Thus for Section 3.4’s experimental algorithm, we stop updating LAMB’s scaling coefﬁcients during compression stage, and just use a calculated scaling coefﬁcient moving average at the end of the warmup stage.

Original: For compression stage we also tried communicating based on 1-bit compressed gradient, but it leads to much slower convergence speed.
Revised: For compression stage we also tried communicating based on 1-bit compressed gradient, but it leads to much slower convergence speed.

Original: We believe it’s because gradients are less stable, which could lead to higher compression error and slower convergence.
Revised: We believe it’s because gradients are less stable, which could lead to higher compression error and slower convergence.

Original: A. 1- BIT LAMB ALGORITHM DESIGN AND IMPLEMENTATION DETAILS
Revised: A. 1- BIT LAMB ALGORITHM DESIGN AND IMPLEMENTATION DETAILS

Original: Figure 6 presents 1-bit LAMB scaling ratios ( r ( l ) t in Algorithm 1) for different layers during BERT pre-training sequence length 128 (sequence length 512 has similar patterns).
Revised: Figure 6 presents 1-bit LAMB scaling ratios ( r ( l ) t in Algorithm 1) for different layers during BERT pre-training sequence length 128 (sequence length 512 has similar patterns).

Original: When comparing with
Revised: When comparing with

Original: Figure 5, we ﬁnd that for those layers with less stable varaince (e.g., 3 kinds of embeddings, weights in BertLayer), the corresponding 1-bit LAMB scaling ratios are also larger.
Revised: Figure 5, we ﬁnd that for those layers with less stable varaince (e.g., 3 kinds of embeddings, weights in BertLayer), the corresponding 1-bit LAMB scaling ratios are also larger.

Original: As a result 1-bit LAMB is able to adaptively update LAMB scaling coefﬁcients during compression according to the difference between the frozen and fresh variance.
Revised: As a result 1-bit LAMB is able to adaptively update LAMB scaling coefﬁcients during compression according to the difference between the frozen and fresh variance.

Original: A.5. R EDUCING NUMBER OF COMMUNICATIONS BY MOMENTUM FUSION
Revised: A.5. R EDUCING NUMBER OF COMMUNICATIONS BY MOMENTUM FUSION

Original: Different from Adam, LAMB has the scaling coefﬁcients that need to be updated separately for each layer.
Revised: Different from Adam, LAMB has the scaling coefﬁcients that need to be updated separately for each layer.

Original: For the communication of the compressed momentum during 1-bit LAMB’s compression stage, if we also communicate separately for each layer, the number of communications (which is 302 for BERT-Large model) will greatly affect the overall performance.
Revised: For the communication of the compressed momentum during 1-bit LAMB’s compression stage, if we also communicate separately for each layer, the number of communications (which is 302 for BERT-Large model) will greatly affect the overall performance.

Original: Thus we fuse all layers’ momentum into a contiguous 1D buffer and just do one communication over this fused momentum.
Revised: Thus we fuse all layers’ momentum into a contiguous 1D buffer and just do one communication over this fused momentum.

Original: This fused momentum is not a duplication, but a different “view” of all layers’ momentum.
Revised: This fused momentum is not a duplication, but a different “view” of all layers’ momentum.

Original: We implement this momentum fusion by torch._utils._flatten_dense_tensors and torch._utils._unflatten_dense_tensors .
Revised: We implement this momentum fusion by torch._utils._flatten_dense_tensors and torch._utils._unflatten_dense_tensors .

Original: [Figure]

Original: A.5. R EDUCING COMPRESSION ERROR BY MOMENTUM SCALING
Revised: A.5. R EDUCING COMPRESSION ERROR BY MOMENTUM SCALING

Original: We ﬁnd that after momentum fusion, the compression error increase a lot for some layers which greatly increase the chance of divergence.
Revised: We ﬁnd that after momentum fusion, the compression error increase a lot for some layers which greatly increase the chance of divergence.

Original: This is because: 1) When communicating base on the fused momentum, all layers’ momentum will be compressed to the same scale due to the nature of  1-bit compression (representing the tensor by ± 1 signs and a single scale).
Revised: This is because: 1) When communicating base on the fused momentum, all layers’ momentum will be compressed to the same scale due to the nature of [Figure] 1-bit compression (representing the tensor by ± 1 signs and a single scale).

Original: Thus the layers with very small/large momentum scale will have larger compression error; 2) Due to LAMB’s layerwise adaptive learning rate, each layer is learning at different speed, which could further increase the variance of momentum scale among layers.
Revised: Thus the layers with very small/large momentum scale will have larger compression error; 2) Due to LAMB’s layerwise adaptive learning rate, each layer is learning at different speed, which could further increase the variance of momentum scale among layers.

Original: We solve this issue by computing an average momentum scale at the end of warmup stage and using it to compute a momentum scale coefﬁcient for each layer.
Revised: We solve this issue by computing an average momentum scale at the end of warmup stage and using it to compute a momentum scale coefﬁcient for each layer.

Original: During the compression stage, we multiply each layer’s local momentum by its scale coefﬁcient, then do the compressed communication, then divide each layer’s global momentum by the same scale coefﬁcient.
Revised: During the compression stage, we multiply each layer’s local momentum by its scale coefﬁcient, then do the compressed communication, then divide each layer’s global momentum by the same scale coefﬁcient.

Original: By performing this momentum scaling, all layers’ momentum will have similar scales when passed to the compressed communication.
Revised: By performing this momentum scaling, all layers’ momentum will have similar scales when passed to the compressed communication.

Original: Thus we are able to greatly reduce the compression error and chance of divergence.
Revised: Thus we are able to greatly reduce the compression error and chance of divergence.

Original: Since the momentum scale coefﬁcients are ﬁxed during the compression stage, it won’t affect 1-bit compression’s error compensation mechanism
Revised: Since the momentum scale coefﬁcients are ﬁxed during the compression stage, it won’t affect 1-bit compression’s error compensation mechanism.

Original: . A. C OMPARING MPI AND NCCL- BASED COMMUNICATION BACKEND
Revised: A. C OMPARING MPI AND NCCL- BASED COMMUNICATION BACKEND

Original: All of the results in main paper are evaluated with the NCCL-based compressed communication backend implementation.
Revised: All of the results in main paper are evaluated with the NCCL-based compressed communication backend implementation.

Original: Figure 7 presents the performance comparison between MPI and NCCLbased implementations.
Revised: Figure 7 presents the performance comparison between MPI and NCCLbased implementations.

Original: Compared to the MPI-based implementation introduced in the 1-bit Adam work, our NCCL-based implementation is able to provide better performance on the Ethernet cluster (where OpenMPI library is used for MPI backend) and on par performance on the InﬁniBand cluster (where MVAPICH2-GDR is used for MPI backend).
Revised: Compared to the MPI-based implementation introduced in the 1-bit Adam work, our NCCL-based implementation is able to provide better performance on the Ethernet cluster (where OpenMPI library is used for MPI backend) and on par performance on the InﬁniBand cluster (where MVAPICH2-GDR is used for MPI backend).

Original: A.T HEORETICAL A NALYSIS
Revised: A. A DDITIONAL CONVERGENCE ANALYSIS ABOUT NUMBER OF WORKERS AND BATCH SIZE

Revised: This section provides additional convergence analysis on BERT pre-training and SQuAD ﬁne-tuning. First we investigate whether the number of workers/GPUs affect the convergence of 1-bit LAMB. Table 6 presents the SQuAD average/max F1 scores using the BERT models pre-trained with LAMB and 1-bit LAMB optimizers, with different number of GPUs, and with the same training parameters described in Section 6.

Revised: Results show that both LAMB and 1-bit LAMB achieve similar SQuAD F scores under different number of workers, and 1-bit LAMB always provides better results.

Revised: Next, we investigate whether the batch size affects the convergence of 1-bit LAMB.

Revised: In main paper we compared LAMB and 1-bit LAMB when pre-training BERT with batch size 64K/32K at seqlen 128/512.

Revised: Here we compare them at batch size 16K and 4K. At batch size 16K, for LAMB we use the same training parameters described in Appendix A.1, and for 1-bit LAMB we set the number of warmup steps as 4000 (14%) for seqlen 128 and 475 (15%) for seqlen 512.

Revised: At batch size 4K, for

Revised: LAMB we follow the original work and use 112500/12500 steps for seqlen 128/512, weight decay = 0.01, linear LR warmup and decay, learning rate = 1 .

Revised: 77 × 10 − 3 , 2.5% LR warmup, clipping conﬁgs ( c min and c max in (1)) as 0.1 and 1.

Revised: For 1-bit LAMB we set the number of warmup steps as(14%) for seqlen 128 and 1900 (15%) for seqlen 512.

Revised: For 1-bit LAMB’s other hyperparameters in

Revised: Algorithm 1, we keep them the same under all batch sizes: β 3 = 0 .

Revised: 9 , r min = 0 .

Revised: 5 , r max = 4 . 0 , and r threshold = 0 .

Revised: As shown in Table 7, results show that 1-bit LAMB provides better SQuAD F scores than LAMB under all batch sizes.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: A. A DDITIONAL CONVERGENCE ANALYSIS : R ES N ET -50 ON CIFAR

Revised: This section provides additional analysis of training ResNet-50 on CIFAR100.

Revised: Because ResNetis much smaller than BERT-Large and because we use 8 GPUs on a single node for the training, 1-bit LAMB and LAMB have very similar throughput.

Revised: Thus this experiment focus on whether 1-bit

Revised: LAMB can achieve the same convergence as LAMB under the same training hyperparameters.

Revised: For both LAMB and 1-bit LAMB, we use 250 epochs (about 12K steps), batch size 1K, learning rates ∈ { 0 . 005 , 0 . 01 , 0 .

Revised: 02 } , linear LR warmup in the ﬁrst 1K steps, no LR decay, weight decay = 0.01.

Revised: For

Revised: LAMB we set the clipping conﬁgs ( c min and c max in (1)) as 0.01 and 0.3.

Revised: For 1-bit LAMB we set the number of warmup steps as 1K (8%), and set other hyperparameters in Algorithm 1 as: β 3 = 0 .

Revised: 9 , r min = 0 .

Revised: 5 , r max = 2 . 0 , and r threshold = 0 .

Revised: Figure 8 presents the sample-wise training loss and testing accuracy, and Table 8 presents the ﬁnal testing accuracy.

Revised: Results show that under all three learning rates, 1-bit LAMB provides on par or better convergence and ﬁnal accuracy than LAMB.

Revised: A. T HEORETICAL A NALYSIS

Original: Notice that for 1-bit LAMB, we only use original LAMB at warm-up, and then we essentially run error-compensated momentum SGD with coordinate-dependent learning rate γ √ v T w .
Revised: Notice that for 1-bit LAMB, we only use original LAMB at warm-up, and then we essentially run error-compensated momentum SGD with coordinate-dependent learning rate γ √ v Tw .

Original: Therefore here we consider the LAMB-based warm-up phase as a way to ﬁnd a good precondition variance term v Tw to be used in the compression phase.
Revised: Therefore here we consider the LAMB-based warm-up phase as a way to ﬁnd a good precondition variance term v Tw to be used in the compression phase.

Original: Below we are going to introduce the convergence rate for the compression phase after warm-up.
Revised: Below we are going to introduce the convergence rate for the compression phase after warm-up.

Original: We ﬁrst introduce some necessary assumptions, then we present the theoretical guarantee of the convergence rate for 1-bit LAMB.
Revised: We ﬁrst introduce some necessary assumptions, then we present the theoretical guarantee of the convergence rate for 1-bit LAMB.

Original: Assumption 1.
Revised: Assumption 1.

Original: We make the following assumptions:
Revised: We make the following assumptions:

Original: Lipschitzian gradient: f ( · ) is assumed to be with L -Lipschitzian gradients, which means (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) ≤ L (cid:107) x − y (cid:107) , ∀ x , ∀ y , 2.
Revised: Lipschitzian gradient: f ( · ) is assumed to be with L -Lipschitzian gradients, which means (cid:107)∇ [Equation]

Original: Bounded variance:
Revised: Bounded variance:

Original: The variance of the stochastic gradient is bounded E ζ ( i )
Revised: The variance of the stochastic gradient is bounded [Equation]

Original: ∼D i (cid:107)∇ F ( x ; ζ ( i ) ) − ∇ f ( x ) (cid:107) 2 ≤ σ 2 , ∀ x , ∀ i. 
Revised: ∀ i. 3.

Original: Bounded magnitude of error for C ω [ · ] : The magnitude of worker’s local errors δ ( i ) t and the server’s global error δ t , are assumed to be bounded by a constant (cid:15)
Revised: Bounded magnitude of error for C ω [ · ] : The magnitude of worker’s local errors δ ( i ) t and the server’s global error δ t , are assumed to be bounded by a constant (cid:15)

Original: [Equation]
Revised: [Equation]

Revised: The most important reason we use the third assumption above is that for 1-bit compression, the signal-to-noise assumption does not hold if there is only one non-zero element in the tensor.

Revised: In (Tang

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: et al., 2019)

Revised: [Figure]

Revised: authors proved that our assumption is equivalent to the signal-to-noise assumption when the gradient magnitude is upper bounded.

Original: Next we present the main theorem for 1-bit LAMB.
Revised: Next we present the main theorem for 1-bit LAMB.

Original: Theorem 1.
Revised: Theorem 1.

Original: Under Assumption 1, for 1-bit LAMB, we have the following convergence rate 
Revised: Under Assumption 1, for 1-bit LAMB, we have the following convergence rate (cid:18)

Original: [Equation]
Revised: [Equation]

Original: where V = diag (cid:16) 1 / v (1) T w , 1 / v (2) T w , · · · , 1 / v ( d ) T w (cid:17) is a diagonal matrix spanned by v Tw .
Revised: where V = diag (cid:16) 1 / v (1) T w , 1 / v (2) T w , · · · , 1 / v ( d ) T w (cid:17) is a diagonal matrix spanned by v Tw .

Revised: Here 1 / v is a moving sum over all history ( ∇ f ( x )) 2 , not (cid:107)∇ f ( x ) (cid:107) .

Revised: Therefore it will not approach to 0 for the following two reasons: i)

Revised: We stop updating v after the warmup stage, which means ∇ f ( x ) at this time will not approach 0 since it has not converged yet; ii) Since β 2 = 0 .

Revised: 999 , this means v is heavily dependent to the history gradients, therefore even though ∇ f ( x ) approaches to 0 , v itself will still taking a time to decay to 0 .

Revised: Therefore it is reasonable for us to assume an upper bound for

Original:  Given the generic result in Theorem 1, we obtain the convergence rate for 1-bit LAMB with appropriately chosen learning rate γ .
Revised: V max Given the generic result in Theorem 1, we obtain the convergence rate for 1-bit LAMB with appropriately chosen learning rate γ .

Original: Corollary 1.
Revised: Corollary 1.

Original: Under Assumption 1, for 1-bit LAMB, choosing γ = γ max = γ min = 1 − β 16 LV max + σ √ Tn + T 13 (cid:15) 23 , we have the following convergence rate
Revised: Under Assumption 1, for 1-bit LAMB, choosing γ = γ max = γ min = 1 − β 16 LV max + σ √ Tn + T 13 (cid:15) 23 , we have the following convergence rate

Original: [Equation]
Revised: [Equation]

Original: where we treat f ( x 1 ) − f ∗ , β and L as constants.
Revised: where we treat f ( x 1 ) − f ∗ , β and L as constants.

Original: This result suggests that: 1-bit LAMB essentially admits the same convergence rate as distributed
Revised: This result suggests that: 1-bit LAMB essentially admits the same convergence rate as distributed

Original: SGD in the sense that both of them admit the asymptotical convergence rate O (1 / √ nT ) , which means we can still achieve linear speedup w.r.t.
Revised: SGD in the sense that both of them admit the asymptotical convergence rate O (1 / √ nT ) , which means we can still achieve linear speedup w.r.t.

Original: the number of workers n .
Revised: the number of workers n .

Original: For theoretical analysis, we follow the framework of previous work (Tang et al., 2021).
Revised: For theoretical analysis, we follow the framework of previous work (Tang et al., 2021).

Original: Below we ﬁrst summarize the updating rule of 1-bit LAMB, followed by the proof of Theorem 1, then the proof of Corollary 1.
Revised: Below we ﬁrst summarize the updating rule of 1-bit LAMB, followed by the proof of Theorem 1, then the proof of Corollary 1.

Original: A.7. P ROOF TO THE UPDATING FORM
Revised: A.9. P ROOF TO THE UPDATING FORM

Original: Since our algorithm is equivalent to running a parameter-server prototype communication on each chunk of the gradient, so below we will assume a parameter-server model (which means the tensor is not required to be divided into n chunks) for simplicity.
Revised: Since our algorithm is equivalent to running a parameter-server prototype communication on each chunk of the gradient, so below we will assume a parameter-server model (which means the tensor is not required to be divided into n chunks) for simplicity.

Original: Although we use a layer-wise updating strategy in 1-bit LAMB, for the simplicity of theoretical analysis, we consider the case where we use the same updating rule for all the layer, but our results can be easily applied to this layer-wise case.
Revised: Although we use a layer-wise updating strategy in 1-bit LAMB, for the simplicity of theoretical analysis, we consider the case where we use the same updating rule for all the layer, but our results can be easily applied to this layer-wise case.

Original: According to the algorithm description in Algorithm 1, at iteration t + 1 , the updating step of the momentum term m t +1 can be divided into two steps:
Revised: According to the algorithm description in Algorithm 1, at iteration t + 1 , the updating step of the momentum term m t +1 can be divided into two steps:

Original: Local Update and Compress: each worker locally update m t and use the error-compensate strategy for compressing.
Revised: Local Update and Compress: each worker locally update m t and use the error-compensate strategy for compressing

Original: [Equation]
Revised: [Equation]

Original: All workers send its m ( i ) t
Revised: All workers send its m ( i ) t

Original: + 12 to the server.
Revised: + 12 to the server.

Original: The server takes the average over them and compress it again using error-compensation.
Revised: The server takes the average over them and compress it again using error-compensation.

Original: [Equation]
Revised: [Equation]

Original: The server broadcast m t + 1 to all workers, and all workers update the local model according to
Revised: The server broadcast m t +1 to all workers, and all workers update the local model according to

Original: [Equation]
Revised: [Equation]

Original: Notice that here we do not include the updating rule for { c t } because for the following part we will only view it as a scaling factor.
Revised: Notice that here we do not include the updating rule for { c t } because for the following part we will only view it as a scaling factor.

Original: So actually the updating rule above can be summarized as
Revised: So actually the updating rule above can be summarized as

Original: [Equation]
Revised: [Equation]

Original: (from the deﬁnition ofδ t +1 )
Revised: from the deﬁnition of

Original: [Equation]
Revised: [Equation]

Original: (from the deﬁnition of
Revised: from the deﬁnition of

Original: [Equation]
Revised: [Equation]

Original: i = δ ( i ) t + δ t (cid:33) − (cid:32)n n (cid:88) i = δ ( i) t +1 +
Revised: ( i

Original: [Equation]
Revised: [Equation]

Original: t + (cid:33) .

Original: Denoteg
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: the update rule of m t can be summarized as
Revised: the update rule of m t can be summarized as

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where V = diag (1 / √ v 1 , 1 / √ v 2 , · · · , 1 / √ v d ) is the a diagonal matrix that spanned with v T w .
Revised: where V = diag (1 / √ v 1 , 1 / √ v 2 , · · · , 1 / √ v d ) is the a diagonal matrix that spanned with v T w .

Original: In order to simlify the notation, we deﬁne γ t := γc t − 1 as a time-varying learning rate, which reduces the updating rule above into
Revised: In order to simlify the notation, we deﬁne γ t := γc t − 1 as a time-varying learning rate, which reduces the updating rule above into

Original: [Equation]
Revised: [Equation]

Revised: A.9. P ROOF TO T HEOREM

Original: Notice that in for 1-bit LAMB, the learning rate for each coordinate is different.
Revised: Notice that in for 1-bit LAMB, the learning rate for each coordinate is different.

Original: In order to simplify our analysis, we instead consider another function that is deﬁned as
Revised: In order to simplify our analysis, we instead consider another function that is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: also
Revised: also

Original: [Equation]
Revised: [Equation]

Original: where V is a diagonal matrix.
Revised: where V is a diagonal matrix.

Original: In this case we have
Revised: In this case we have

Original: [Equation]
Revised: [Equation]

Original: Therefore the updating rule of 1-bit LAMB in the view of h ( · ) is
Revised: Therefore the updating rule of 1-bit LAMB in the view of h ( · ) is

Original: [Equation]
Revised: [Equation]

Original: It can be easily veriﬁed that
Revised: It can be easily veriﬁed that

Original: [Equation]

Original: [Equation]

Original: s

Original: [Equation]
Revised: [Equation]

Original: β

Original: [Equation]

Original: n n (cid:88)

Original: [Equation]

Original: s

Original: [Equation]

Original: [Equation]

Original: ) which means
Revised: which means

Original: [Equation]

Original: n

Original: [Equation]

Original: [Equation]

Original: ) =(1 − β ) t (cid:88) s =

Original: β t − s g s

Original: [Equation]
Revised: [Equation]

Original: where g s ( z ) is the corresponding averaged stochastic gradient computed in the view of loss function h ( · ) .
Revised: where g s ( z ) is the corresponding averaged stochastic gradient computed in the view of loss function h ( · ) .

Original: Then, if we deﬁne m t ( z ) =
Revised: Then, if we deﬁne m t ( z ) =

Original: V 12 m t , the updating rule of m t ( z ) admits
Revised: V 12 m t , the updating rule of m t ( z ) admits

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: From (4) and (5) we shall see that using different learning rate for each coordinate is equivalent to optimizing a new loss function deﬁned on scaling the original coordinate and using a uniform learning for all coordinates.
Revised: From (4) and (5) we shall see that using different learning rate for each coordinate is equivalent to optimizing a new loss function deﬁned on scaling the original coordinate and using a uniform learning for all coordinates.

Original: Therefore below we ﬁrst study the behavior of the error-compensated momentum SGD using a coordinate-independent time-varying learning rate.
Revised: Therefore below we ﬁrst study the behavior of the error-compensated momentum SGD using a coordinate-independent time-varying learning rate.

Original: Below are some critical lemmas for the proof of Theorem 1.
Revised: Below are some critical lemmas for the proof of Theorem 1.

Original: Lemma 1.
Revised: Lemma 1.

Original: Given two non-negative sequences { a t } ∞ t =1 and { b t } ∞ t =1 that satisfying
Revised: Given two non-negative sequences { a t } ∞ t =1 and { b t } ∞ t =1 that satisfying

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: with ρ ∈ [0 , 1) , we have
Revised: with ρ ∈ [0 , 1) , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From the deﬁnition, we have
Revised: From the deﬁnition, we have

Original: S k = k (cid:88) t = t (cid:88) s
Revised: S k = k (cid:88) t = t (cid:88) s

Original: =
Revised: =

Original: ρ t − s b s = k (cid:88) s =
Revised: ρ t − s b s = k (cid:88) s =

Original: k (cid:88) t =
Revised: k (cid:88) t =

Original: sρ t − s b s = k (cid:88) s =
Revised: s

Original: k − s (cid:88) t
Revised: [Equation] (cid:88) s

Revised: [Equation]

Original: = ρ t b s ≤ k (cid:88) s
Revised: (cid:88) [Equation] (cid:88)

Original: = b s 1 − ρ, (7)
Revised: [Equation] − [Equation]

Original: D k = k (cid:88) t
Revised: D k = k (cid:88) t

Original: = t (cid:88) s
Revised: = t (cid:88) s

Original: =
Revised: =

Original: ρ t − s b s t (cid:88) r
Revised: ρ t − s b s t (cid:88) r

Original: = ρ t − r b r = k (cid:88) t
Revised: = ρ t − r b r = k (cid:88) t

Original: = t (cid:88) s
Revised: = t (cid:88) s

Original: = t (cid:88)
Revised: = t (cid:88)

Original: r
Revised: r

Original: = ρ 2 t − s − r b s b r ≤ k (cid:88) t
Revised: = ρ 2 t − s − r b s b r ≤ k (cid:88) t

Original: = t (cid:88) s = t (cid:88)
Revised: = t (cid:88) s = t (cid:88)

Original: r
Revised: r

Original: = ρ 2 t − s − r b 2 s + b 2 r
Revised: = ρ 2 t − s − r b 2 s + b 2 r

Original: =
Revised: =

Original: k (cid:88) t
Revised: k (cid:88) t

Original: = t (cid:88) s = t (cid:88)
Revised: = t (cid:88) s = t (cid:88)

Original: r
Revised: r

Original: = ρ 2 t − s − r b 2 s ≤1 − ρ k (cid:88) t
Revised: = ρ 2 t − s − r b 2 s ≤1 − ρ k (cid:88) t

Original: = t (cid:88) s
Revised: = t (cid:88) s

Original: =
Revised: =

Original: ρ t − s b 2 s ≤ (1 − ρ )k (cid:88) s =
Revised: ρ t − s b 2 s ≤ (1 − ρ )k (cid:88) s =

Original: b 2 s , (due to (7)) which completes the proof.
Revised: b 2 s , (due to (7)) which completes the proof.

Original: Lemma 2.
Revised: Lemma 2.

Original: Under Assumption 1, for any sequence that follows the updating rule of
Revised: Under Assumption 1, for any sequence that follows the updating rule of

Original: [Equation]
Revised: [Equation]

Original: if
Revised: if

Revised: [Equation]

Revised: E

Revised: [Equation]

Revised: δ

Original: [Equation]
Revised: [Equation]

Revised: ∀

Revised: [Equation]

Revised: (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) ≤ L (cid:107) x − y (cid:107) ,

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: y

Revised: [Equation]

Original: then we can guarantee that
Revised: then we can guarantee that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Instead of investigating x t directly
Revised: Instead of investigating x t directly

Original: [Equation]
Revised: [Equation]

Original: we introduce the following sequence
Revised: we introduce the following sequence

Original: [Equation]
Revised: [Equation]

Original: The updating rule of y t admits
Revised: The updating rule of y t admits

Original: [Equation]
Revised: [Equation]

Original: Since f ( · ) is with L-Lipschitzian
Revised: Since f ( · ) is with L-Lipschitzian

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: E

Revised: [Equation]

Revised: y

Original: [Equation]
Revised: [Equation]

Original: Summing up the equation above from t = 0 to t
Revised: Summing up the equation above from t = 0 to t

Original: =
Revised: =

Original: T we get
Revised: T we get

Original: E f

Original: ( y T +1 )

Original: − E f ( y 0 ) ≤

Original: [Equation]
Revised: [Equation]

Original: Therefore if we γ t < 12 L and it’s within certain range γ t ∈ [ γ min
Revised: Therefore if we γ t < 12 L and it’s within certain range γ t ∈ [ γ min

Original: [Equation]
Revised: [Equation]

Original: γ max ]
Revised: γ max ]

Original: [Equation]
Revised: [Equation]

Original: the equation above leads to (1 − γ max L
Revised: the equation above leads to (1 − [Equation] max L

Original: [Equation]
Revised: [Equation]

Original: Notice that we have
Revised: Notice that we have

Original: [Equation]
Revised: [Equation]

Original: which by using Lemma
Revised: which by using Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: T

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Combing (8) and (9) together we get
Revised: Combing (8) and (9) together we get

Original: [Equation]
Revised: [Equation]

Original: Proof to Theorem 1 Since using a per-coordinate learning rate for loss function f ( · ) is equivalent to use a constant learning for all coordinates but for loss function h ( · ) , the only two thing that change are
Revised: Proof to Theorem 1 Since using a per-coordinate learning rate for loss function f ( · ) is equivalent to use a constant learning for all coordinates but for loss function h ( · ) , the only two thing that change are

Original: • Different L-Lipschitzian coefﬁcient : the L-Lipschitzian coefﬁcient for h ( · ) is
Revised: • Different L -Lipschitzian coefﬁcient : the L-Lipschitzian coefﬁcient for h ( · ) is

Original: [Equation]
Revised: [Equation]

Original: Therefore the effective L-Lipschitzian coefﬁcient of h ( x ) is LV max • Different deﬁnition of δ t : from (4) we shall see that actually the compression error in the view of h ( · ) is V 12 δ t , so in this case we have
Revised: Therefore the effective L-Lipschitzian coefﬁcient of h ( x ) is LV max • Different deﬁnition of δ t : from (4) we shall see that actually the compression error in the view of h ( · ) is V 12 δ t , so in this case we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From Lemma
Revised: From Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: A.7. P ROOF TO C OROLLARY
Revised: A.9. P ROOF TO C OROLLARY

Original: Proof.
Revised: Proof.

Original: By choosing γ max = γ min = [Equation] 16 LV max [Equation]
Revised: By choosing γ max = γ min = 1 − β 16 LV max + σ √ Tn + T 13 (cid:15) 23 , we can guarantee that

Original: σ

Original: [Equation]
Revised: [Equation]

Original: Tn

Original: [Equation]

Original: [Equation]

Original: 23 , we can guarantee that

Original: [Equation]

Original: So (3) leads to
Revised: So (3) leads to

Original: T(cid:88) [Equation] [Equation]
Revised: T

Original: [Equation]
Revised: [Equation]

Original: Treating f ( y 1 ) − f ∗ , β and L as constants [Equation] from the inequality above we get[Equation]
Revised: Treating f ( y 1 ) − f ∗ , β and L as constants, from the inequality above we get

Original: f

Original: [Equation]
Revised: [Equation]

Original: It completes the proof.
Revised: It completes the proof.