Original: O N THE V ARIANCE OF THE A DAPTIVE L EARNING R ATE AND B EYOND
Revised: O N THE V ARIANCE OF THE A DAPTIVE L EARNING R ATE AND B EYOND

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam.
Revised: The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam.

Original: Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate – its variance is problematically large in the early stage, and presume warmup works as a variance reduction technique.
Revised: Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate – its variance is problematically large in the early stage and presume warmup works as a variance reduction technique.

Original: We provide both empirical and theoretical evidence to verify our hypothesis.
Revised: We provide both empirical and theoretical evidence to verify our hypothesis.

Original: We further propose Rectiﬁed Adam (RAdam), a novel variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate.
Revised: We further propose Rectiﬁed Adam (RAdam), a novel variant of Adam by introducing a term to rectify the variance of the adaptive learning rate.

Original: Experimental results on image classiﬁcation, language modeling, and neural machine translation verify our intuition and demonstrate the efﬁcacy and robustness of RAdam
Revised: Experimental results on image classiﬁcation, language modeling, and neural machine translation verify our intuition and demonstrate the efﬁcacy and robustness of RAdam

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: Fast and stable optimization algorithms are what generations of researchers have been pursuing (Gauss, 1823; Cauchy, 1847).
Revised: Fast and stable optimization algorithms are what generations of researchers have been pursuing (Gauss, 1823; Cauchy, 1847).

Original: Remarkably, stochastic gradient-based optimization, such as stochastic gradient descent (SGD), has witnessed tremendous success in many ﬁelds of science and engineering despite its simplicity.
Revised: Remarkably, stochastic gradient-based optimization, such as stochastic gradient descent (SGD), has witnessed tremendous success in many ﬁelds of science and engineering despite its simplicity.

Original: Recently, many efforts have been made to accelerate optimization by applying adaptive learning rate .
Revised: Recently, many efforts have been made to accelerate optimization by applying adaptive learning rate .

Original: In particular, Adagrad (Duchi et al., 2011) and its variants, e.g. , RMSprop (Tieleman & Hinton, 2012), Adam (Kingma & 
Revised: In particular, Adagrad (Duchi et al., 2010) and its variants, e.g. , RMSprop (Hinton et al., 2012), Adam (Kingma & Ba, 2014),

Original: Ba, 2014), Adadelta (Zeiler, 2012) and Nadam (Dozat, 2016), stand out due to their fast convergence, and have been considered as the optimizer of choice in many applications.
Revised: Adadelta (Zeiler, 2012) and Nadam (Dozat, 2016), stand out due to their fast convergence, and have been considered as the optimizer of choice in many applications.

Original: However, it has been observed that these optimization methods may converge to bad/suspicious local optima, and have to resort to a warmup heuristic – using a small learning rate in the ﬁrst few epochs of training– to mitigate such problem (Vaswani et al., 2017; Popel & Bojar, 2018).
Revised: However, it has been observed that these optimization methods may converge to bad/suspicious local optima, and have to resort to a warmup heuristic – using a small learning rate in the ﬁrst few epochs of training to mitigate such problem (Vaswani et al., 2017; Popel & Bojar, 2018).

Original: For example, when training typical Transformers based neural machine translation models on the De-En
Revised: For example, when training typical Transformers based neural machine translation models on the De-En

Original: IWSLT’14 dataset, removing the warmup stage increases the training loss from 3 to around 10, as shown in Figure 1.
Revised: IWSLT’14 dataset, removing the warmup stage increases the training loss from 3 to around 10, as shown in Figure 1.

Original: Similar phenomena are observed in other scenarios like BERT (a bidirectional transformer language model) pre-training (Devlin et al., 2018).
Revised: Similar phenomena are observed in other scenarios like BERT (a bidirectional transformer language model) pre-training (Devlin et al., 2019).

Original: Duo to the lack of the theoretical underpinnings, there is neither guarantee that warmup would bring consistent improvements for various machine learning settings nor guidance on how we should conduct warmup.
Revised: Duo to the lack of the theoretical underpinnings, there is neither guarantee that warmup would bring consistent improvements for various machine learning settings nor guidance on how we should conduct warmup.

Original: Thus, researchers typically use different settings in different applications and have to take a trial-and-error approach, which can be tedious and time-consuming.
Revised: Thus, researchers typically use different settings in different applications and have to take a trial-and-error approach, which can be tedious and time-consuming.

Original: In this paper, we conduct both empirical and theoretical analysis of the convergence issue to identify its origin.
Revised: In this paper, we conduct both empirical and theoretical analysis of the convergence issue to identify its origin.

Original: We show that its root cause is: the adaptive learning rate has undesirably large variance in the early stage of model training, due to the limited amount of training samples being used.
Revised: We show that its root cause is: the adaptive learning rate has undesirably large variance in the early stage of model training, due to the limited amount of training samples being used.

Original: Thus, to reduce such variance, it is better to use smaller learning rates in the ﬁrst few epochs of training, which justiﬁes the warmup heuristic.
Revised: Thus, to reduce such variance, it is better to use smaller learning rates in the ﬁrst few epochs of training, which justiﬁes the warmup heuristic.

Original: Inspired by our analysis results, we propose a new variant of Adam, called Rectiﬁed Adam (RAdam), which explicitly rectiﬁes the variance of the adaptive learning rate based on derivations.We conduct I t e r a t i o n
Revised: Inspired by our analysis results, we propose a new variant of Adam, called Rectiﬁed Adam (RAdam), which explicitly rectiﬁes the variance of the adaptive learning rate based on derivations.

Original: Adam with warmup Adam without warmup 6.76×10 ' 9.38×10 '

Original: I t e r a t i o n

Original: I t e r a t i o n 4.08×10 ' 4.08×10 '

Original: [-I t e r a t i o n <- ./0 - .1' - .1/ - .2 <- ./0 - .1' - .1/ - .
-]
Revised: [-I t e r a t i o n <- ./0 - .1' - .1/ - .2 <- ./0 - .1' - .1/ - .
-]

Original: [-<- ./ - .1' - .1/ - .
-]
Revised: [-<- ./ - .1' - .1/ - .
-]

Original: <- ./0 - .1' - .1/ - .2 - .

Original: 50 540K 70k 40K 70k

Original: The distribution is distorted within 10 updates.

Original:  extensive experiments on language modeling, image classiﬁcation, and neural machine translation.
Revised: We conduct extensive experiments on language modeling, image classiﬁcation, and neural machine translation.

Original: RAdam brings consistent improvement over the vanilla Adam, which veriﬁes the variance issue generally exists on various tasks across different network architectures.
Revised: RAdam brings consistent improvement over the vanilla Adam, which veriﬁes the variance issue generally exists on various tasks across different network architectures.

Original: In summary, our main contributions are two-fold:
Revised: In summary, our main contributions are two-fold:

Original: • We identify the variance issue of the adaptive learning rate and present a theoretical justiﬁcation for the warmup heuristic.
Revised: • We identify the variance issue of the adaptive learning rate and present a theoretical justiﬁcation for the warmup heuristic.

Original: We show that the convergence issue is due to the undesirably large variance of the adaptive learning rate in the early stage of model training.
Revised: We show that the convergence issue is due to the undesirably large variance of the adaptive learning rate in the early stage of model training.

Original: • We propose a new variant of Adam ( i
Revised: • We propose a new variant of Adam ( i

Original: e. , RAdam), which not only explicitly rectiﬁes the variance and is theoretically sound, but also compares favorably with the heuristic warmup.
Revised: e. , RAdam), which not only explicitly rectiﬁes the variance and is theoretically sound, but also compares favorably with the heuristic warmup.

Original:  P RELIMINARIES AND M OTIVATIONS
Revised:  P RELIMINARIES AND M OTIVATIONS

Original: Generic adaptive methods.
Revised: Generic adaptive methods.

Original: Algorithm 1 is a generic framework (all operations are element-wise).
Revised: Algorithm 1 is a generic framework (all operations are element-wise).

Original: It describes various popular stochastic gradient descent algorithms (Reddi et al., 2019).
Revised: It describes various popular stochastic gradient descent algorithms (Reddi et al., 2018).

Original: Speciﬁcally, different optimization algorithms can be speciﬁed by different choices of φ ( . ) and ψ ( . ) , where φ ( . ) speciﬁes how the momentum at time step t is calculated, and ψ ( . )
Revised: Speciﬁcally, different optimization algorithms can be speciﬁed by different choices of φ ( . ) and ψ ( . ) , where φ ( . ) speciﬁes how the momentum at time step t is calculated, and ψ ( . )

Original: how the adaptive learning rate at t is calculated.
Revised: how the adaptive learning rate at t is calculated.

Original: For example, in the Adam algorithm, we have:
Revised: For example, in the Adam algorithm, we have:

Original: [Equation]
Revised: [Equation]

Original: and ψ 
Revised: 1 ) (cid:80) ti =1 β t − i 1 g t 1 − β tand ψ ( g 1 , · · · , g t )

Original: [Equation]
Revised: = (cid:115) 1 − β t(1 − β 2 ) (cid:80) ti = 1 β t − i 2 g 2 i .

Original: For numerical stability, the function ψ ( . ) in Equation 1 is usually calculated as (cid:98) ψ ( g 1 , · · · , g t ) = √ 1 − β t
Revised: For numerical stability, the function ψ ( . ) in Equation 1 is usually calculated as (cid:98) ψ ( g 1 , · · · , g t ) = √ 1 − β t(cid:15) + √ (1 − β 2 ) (cid:80) ti

Original: [Equation] g 2 i , where (cid:15) is a relatively small / negligible value ( e.g. , 1 × 10 − 8 ).
Revised: =1 β t − i 2 g 2 i , where (cid:15) is a relatively small / negligible value ( e.g. , 1 × 10 − 8 ).

Original: Algorithm
Revised: [Table]

Original: Input
Revised: [Table]

Original: Output parameters 1 while t = 1 to T do g t ← ∆ θ f t ( θ t − 1 ) (Calculate gradients w.r.t.
Revised: [Table] [Table] [Table] [Table]

Original: stochastic objective at timestep t) m t ← φ t ( g 1 , · · · , g t ) (Calculate momentum) l t ← ψ t ( g 1 , · · · , g t ) (Calculate adaptive learning rate) θ t ← θ t − 1 − α t m t l t (Update parameters) 6 return θ T
Revised: [Table] [Table]

Original: Learning rate warmup.
Revised: Learning rate warmup.

Original: Instead of setting the learning rate α t as a constant or in a decreasing order, a learning rate warmup strategy sets α t as smaller values in the ﬁrst few steps, thus not satisfying ∀ t α t +1 ≤ α t .
Revised: Instead of setting the learning rate α t as a constant or in a decreasing order, a learning rate warmup strategy sets α t as smaller values in the ﬁrst few steps, thus not satisfying ∀ t α t +1 ≤ α t .

Original: For example, linear warmup sets α t =
Revised: For example, linear warmup sets α t =

Original: t α 0 when t < T w .
Revised: t α 0 when t < T w .

Original: Warmup has been demonstrated to be beneﬁcial in many deep learning applications.
Revised: Warmup has been demonstrated to be beneﬁcial in many deep learning applications.

Original: For example, in the NMT experiments in Figure 1, the training loss convergences around 10 when warmup is not applied (Adam-vanilla), and it surprisingly decreases to below 3 after applying warmup (Adam-warmup).
Revised: For example, in the NMT experiments in Figure 1, the training loss convergences around 10 when warmup is not applied (Adam-vanilla), and it surprisingly decreases to below 3 after applying warmup (Adam-warmup).

Original: To further analyze this phenomenon, we visualize the histogram of the absolute value of gradients on a log scale in Figure 2.
Revised: To further analyze this phenomenon, we visualize the histogram of the absolute value of gradients on a log scale in Figure 2.

Original: We observe that, without applying warmup, the gradient distribution 
Revised: We observe that, without applying warmup, the gradient distribution is distorted to have a mass center in relatively small values within 10 updates.

Original: [-Adam-2k 5.72×
 RAdam 6.82×
 Adam-eps 5.42×1010 − 𝑒 − 𝑒 − 𝑒 − <𝑒 − 𝑒 − 𝑒 − 𝑒 − <𝑒 −20 𝑒 − 𝑒 − 𝑒 −-]
Revised: [-Adam-2k 5.72×
 RAdam 6.82×
 Adam-eps 5.42×1010 − 𝑒 − 𝑒 − 𝑒 − <𝑒 − 𝑒 − 𝑒 − 𝑒 − <𝑒 −20 𝑒 − 𝑒 − 𝑒 −-]Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the ﬁrst few [Figure]

Original: I t e r a t io n
Revised: I t e r a t i

Revised: [Figure]

Original: I t e r a t i on
Revised: I t e r a t i o

Revised: [Figure]

Original: I t e r a t i o n <𝑒 −  40K 70k 40K 70k 40K 70k is distorted to have a mass center in relatively small values within 10 updates.
Revised: I t e r a t i o n <- ./0 - . [Figure] [Figure] - .2 <- ./0 - .

Original: Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the ﬁrst few updates.
Revised: [Figure] [Figure] - .2 <- ./ [Figure]

Original:  Warmup essentially reduces the impact of these problematic updates to avoid the convergence problem.
Revised: [Figure] [Figure] [Figure] Warmup essentially reduces the impact of these problematic updates to avoid the convergence problem.

Original: In the following sections, we focus our analysis on learning rate warmup for the
Revised: In the following sections, we focus our analysis on learning rate warmup for the

Original: Adam algorithm, while it can be applied to other algorithms that use similar adaptive learning rate ( ψ ( . ) ) designs, e.g. , RMSprop (Tieleman & Hinton, 2012) and Nadam (Dozat, 2016).
Revised: Adam algorithm, while it can be applied to other algorithms that use similar adaptive learning rate ( ψ ( . ) ) designs, e.g. , RMSprop (Hinton et al., 2012) and Nadam (Dozat, 2016).

Original:  V ARIANCE OF THE A DAPTIVE L EARNING R ATE
Revised:  V ARIANCE OF THE A DAPTIVE L EARNING R ATE

Original: In this section, we ﬁrst introduce empirical evidence, then analyze the variance of the adaptive learning rate to support our hypothesis – Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima .
Revised: In this section, we ﬁrst introduce empirical evidence, then analyze the variance of the adaptive learning rate to support our hypothesis – Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima .

Original: To convey our intuition, we begin with a special case.
Revised: To convey our intuition, we begin with a special case.

Original: When t = 1 , we have ψ ( g 1 ) =
Revised: When t = 1 , we have ψ ( g 1 ) =

Original: (cid:112) 1 /g 21 .
Revised: (cid:112) 1 /g 21 .

Original: We view { g 1 , · · · , g t } as i.i.d.
Revised: We view { g 1 , · · · , g t } as i.i.d.

Original: Gaussian random variables following N (0 , σ 2 ) 1 .
Revised: Gaussian random variables following N (0 , σ 2 ) 2 .

Original: Therefore, 1 /gis subject to the scaled inverse chi-squared distribution, Scale-inv-
Revised: Therefore, 1 /gis subject to the scaled inverse chi-squared distribution, Scale-inv-

Original: X 2 (1 , 1 /σ 2 ) , and Var[ (cid:112) 1 /g 21 ] is divergent.
Revised: X 2 (1 , 1 /σ 2 ) , and Var[ (cid:112) 1 /g 21 ] is divergent.

Original: It means that the adaptive ratio can be undesirably large in the ﬁrst stage of learning.
Revised: It means that the adaptive ratio can be undesirably large in the ﬁrst stage of learning.

Original: Meanwhile, setting a small learning rate at the early stage can reduce the variance ( Var[ αx ] =
Revised: Meanwhile, setting a small learning rate at the early stage can reduce the variance ( Var[ αx ] =

Original: α 2 Var[ x ] ), thus alleviating this problem.
Revised: α 2 Var[ x ] ), thus alleviating this problem.

Original: Therefore, we suggest it is the unbounded variance of the adaptive learning rate in the early stage that causes the problematic updates.
Revised: Therefore, we suggest it is the unbounded variance of the adaptive learning rate in the early stage that causes the problematic updates.

Original: 3. W ARMUP AS V ARIANCE R EDUCTION
Revised: 3. W ARMUP AS V ARIANCE R EDUCTION

Original: In this section, we design a set of controlled experiments to verify our hypothesis.
Revised: In this section, we design a set of controlled experiments to verify our hypothesis.

Original: Particularly, we design two variants of Adam that reducing the variance of the adaptive learning rate: Adam-2k and
Revised: Particularly, we design two variants of Adam that reducing the variance of the adaptive learning rate: Adam-2k and

Original: Adam-eps .
Revised: Adam-eps .

Original: We compare them to vanilla Adam with and without warmup on the IWSLT’14 German to English translation dataset (Cettolo et al., 2014).
Revised: We compare them to vanilla Adam with and without warmup on the IWSLT’14 German to English translation dataset (Cettolo et al., 2014).

Original: In order to reduce the variance of the adaptive learning rate ( ψ ( . ) ), Adam-2k only updates ψ ( . )
Revised: In order to reduce the variance of the adaptive learning rate ( ψ ( . ) ), Adam-2k only updates ψ ( . )

Original: in the ﬁrst two thousand iterations, while the momentum ( φ ( . ) ) and parameters ( θ ) are ﬁxed 2 ; other than this, it follows the original Adam algorithm.
Revised: in the ﬁrst two thousand iterations, while the momentum ( φ ( . ) ) and parameters ( θ ) are ﬁxed 3 ; other than this, it follows the original Adam algorithm.

Original: To make comparison with other methods, its iterations are indexed from -1999 instead of 1.
Revised: To make comparison with other methods, its iterations are indexed from -1999 instead of 1.

Original: In Figure 1, we observe that, after getting these additional two thousand samples for estimating the adaptive learning rate, Adam-2k avoids the convergence problem of the vanilla-Adam.
Revised: In Figure 1, we observe that, after getting these additional two thousand samples for estimating the adaptive learning rate, Adam-2k avoids the convergence problem of the vanilla-Adam.

Original: Also, comparing Figure 2 and Figure 3, getting large enough samples prevents the gradient distribution from being distorted.
Revised: Also, comparing Figure 2 and Figure 3, getting large enough samples prevents the gradient distribution from being distorted.

Original: These observations verify our hypothesis that the lack of sufﬁcient data samples in the early stage is the root cause of the convergence issue.
Revised: These observations verify our hypothesis that the lack of sufﬁcient data samples in the early stage is the root cause of the convergence issue.

Original: Another straightforward way to reduce the variance is to increase the value of (cid:15) in (cid:98) ψ ( g 1 , · · · , g t ) = √ 1 − β t(cid:15) + √ (1 − β 2 ) (cid:80) ti
Revised: Another straightforward way to reduce the variance is to increase the value of (cid:15) in (cid:98) ψ ( g 1 , · · · , g t ) = √ 1 − β t(cid:15) + √ (1 − β 2 ) (cid:80) ti

Original: =1 β t − i 2 g 2 i .
Revised: =1 β t − i 2 g 2 i .

Original: Actually, if we assume (cid:98) ψ ( . ) is subject to the uniform distribution, its variance equals to 1 12 (cid:15) 2 .
Revised: Actually, if we assume (cid:98) ψ ( . ) is subject to the uniform distribution, its variance equals to 1 12 (cid:15) 2 .

Original: Therefore, we design Adam-eps, which uses a non-negligibly large (cid:15) =
Revised: Therefore, we design Adam-eps, which uses a non-negligibly large (cid:15) =

Original: 10 − 4 , while (cid:15) =
Revised: 10 − 4 , while (cid:15) =

Original: 10 − 8 for vanilla Adam.
Revised: 10 − 8 for vanilla Adam.

Original: Its performance is summarized in Figure 1.
Revised: Its performance is summarized in Figure 1.

Original: We observe that it does not suffer from the serious convergence problem of vanilla-Adam.
Revised: We observe that it does not suffer from the serious convergence problem of vanilla-Adam.

Original: This further demonstrates that the convergence problem can be alleviated by reducing the variance of the adaptive learning rate, and also explains why tuning (cid:15) is important in practice (Liu et al., 2019).
Revised: This further demonstrates that the convergence problem can be alleviated by reducing the variance of the adaptive learning rate, and also explains why tuning (cid:15) is important in practice (Liu et al., 2019).

Original: Besides, similar to
Revised: Besides, similar to

Original: Adam-2k, it prevents the gradient distribution from being distorted (as shown in Figure 3).
Revised: Adam-2k, it prevents the gradient distribution from being distorted (as shown in Figure 3).

Original: However, as in Figure 1, it produces a much worse performance comparing to Adam-2k and Adam-warmup.
Revised: However, as in Figure 1, it produces a much worse performance comparing to Adam-2k and Adam-warmup.

Original: We conjecture that this is because large (cid:15) induces a large bias into the adaptive learning rate and slows down the optimization process.
Revised: We conjecture that this is because large (cid:15) induces a large bias into the adaptive learning rate and slows down the optimization process.

Original: Thus, we need a more principled and rigorous way to control the variance of the adaptive learning rate.
Revised: Thus, we need a more principled and rigorous way to control the variance of the adaptive learning rate.

Original: In the next subsection, we will present a theoretical analysis of the variance of the adaptive learning rate.
Revised: In the next subsection, we will present a theoretical analysis of the variance of the adaptive learning rate.

Original: 3. A NALYSIS OF A DAPTIVE L EARNING R ATE V ARIANCE
Revised: 3. A NALYSIS OF A DAPTIVE L EARNING R ATE V ARIANCE

Original: As mentioned before, Adam uses the exponential moving average to calculate the adaptive learning rate.
Revised: As mentioned before, Adam uses the exponential moving average to calculate the adaptive learning rate.

Original: For gradients { g 1 , · · · , g t } , their exponential moving average has a larger variance than their simple average.
Revised: For gradients { g 1 , · · · , g t } , their exponential moving average has a larger variance than their simple average.

Original: Also, in the early stage ( t is small), the difference of the exponential weights of { g 1 , · · · , g t } is relatively small (up to 1 − β t − 1).
Revised: Also, in the early stage ( t is small), the difference of the exponential weights of { g 1 , · · · , g t } is relatively small (up to 1 − β t − 1).

Original: Therefore, for ease of analysis, we approximate the distribution of the exponential moving average as the distribution of the simple average (Nau, 2014), i.e. , p ( ψ ( . ))
Revised: Therefore, for ease of analysis, we approximate the distribution of the exponential moving average as the distribution of the simple average (Nau, 2014), i.e. , p ( ψ ( . ))

Original: = p ( (cid:114)
Revised: = p ( (cid:114)

Original: [Equation]
Revised: [Equation]

Original: Since g i ∼ N (0 , σ 2 ) , we have t
Revised: ti

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i ) .

Original: [Equation] 2 i ∼ Scale-inv- X 2 ( t, 1 σ 2 ) .
Revised: Since g i ∼ N (0 , σ 2 ) , we have t (cid:80) ti =1 g 2 i ∼ Scale-inv- X 2 ( t, 1 σ 2 ) .

Original: Therefore, we assume
Revised: Therefore, we assume

Original: [Equation]
Revised: [Equation]

Original:  also subjects to a scaled inverse chi-square distribution with ρ degrees of freedom (further analysis on this approximation is conducted in Section 5.3).
Revised: i also subjects to a scaled inverse chi-square distribution with ρ degrees of freedom (further analysis on this approximation is conducted in Section 5.3).

Original: Based on this assumption, we can calculate Var[ ψ 2 ( . )] and the PDF of ψ 2 ( . ) .
Revised: Based on this assumption, we can calculate Var[ ψ 2 ( . )] and the PDF of ψ 2 ( . ) .

Original: Now, we proceed to the analysis of its square root variance, i.e. , Var[ ψ ( . )] , and show how the variance changes with ρ (which corresponds to number of used training samples).
Revised: Now, we proceed to the analysis of its square root variance, i.e. , Var[ ψ ( . )] , and show how the variance changes with ρ (which corresponds to number of used training samples).

Original: Theorem 1.
Revised: Theorem 1.

Original: If ψ 2 ( . ) ∼ Scale-inv- X 2 ( ρ, 1 σ 2 ) , Var[ ψ ( . )] monotonically decreases as ρ increases.
Revised: If ψ 2 ( . ) ∼ Scale-inv- X 2 ( ρ, 1 σ 2 ) , Var[ ψ ( . )] monotonically decreases as ρ increases.

Original: Proof.
Revised: Proof.

Original: For ∀ ρ > 4 , we have:
Revised: For ∀ ρ > 4 , we have:

Original: [Equation]
Revised: [Equation]

Original: where B ( . ) is the beta function.
Revised: where B ( . ) is the beta function.

Original: By analyzing the derivative of Var[ ψ ( . )] , we know it monotonically decreases as ρ increases.
Revised: By analyzing the derivative of Var[ ψ ( . )] , we know it monotonically decreases as ρ increases.

Original: The detailed derivation is elaborated in the Appendix A.
Revised: The detailed derivation is elaborated in the Appendix A.

Original: Theorem 1 gives a qualitative analysis of the variance of the adaptive learning rate.
Revised: Theorem 1 gives a qualitative analysis of the variance of the adaptive learning rate.

Original: It shows that, due to the lack of used training samples in the early stage, Var[ ψ ( . )] is larger than the late stage (Figure 8).
Revised: It shows that, due to the lack of used training samples in the early stage, Var[ ψ ( . )] is larger than the late stage (Figure 8).

Original: To rigorously constraint the variance, we perform a quantiﬁed analysis on Var[ ψ ( . )] by estimating the degree of freedoms ρ .
Revised: To rigorously constraint the variance, we perform a quantiﬁed analysis on Var[ ψ ( . )] by estimating the degree of freedoms ρ .

Original:  R ECTIFIED A DAPTIVE L EARNING R ATE
Revised:  R ECTIFIED A DAPTIVE L EARNING R ATE

Original: In the previous section, Equation 2 gives the analytic form of Var[ ψ ( . )] , where ρ is the degree of freedoms.
Revised: In the previous section, Equation 2 gives the analytic form of Var[ ψ ( . )] , where ρ is the degree of freedoms.

Original: Here, we ﬁrst give an estimation of ρ based on t to conduct a quantiﬁed analysis for
Revised: Here, we ﬁrst give an estimation of ρ based on t to conduct a quantiﬁed analysis for

Original: Var[ ψ ( g 1 , · · · , g t )] , then we describe the design of the learning rate rectiﬁcation, and compare it to the heuristic warmup strategies.
Revised: Var[ ψ ( g 1 , · · · , g t )] , then we describe the design of the learning rate rectiﬁcation, and compare it to the heuristic warmup strategies.

Original: 4. E STIMATION OF ρ
Revised: 4. E STIMATION OF ρ

Original: The exponential moving average (EMA) can be interpreted as an approximation to the simple moving average (SMA) in real application (Nau, 2014), i.e. ,
Revised: The exponential moving average (EMA) can be interpreted as an approximation to the simple moving average (SMA) in real application (Nau, 2014), i.e. ,

Original: [Equation]
Revised: [Equation]

Original: where f ( t, β 2 ) is the length of the SMA which allows the SMA to have the same “center of mass” with the EMA. In other words, f ( t, β 2 ) satisﬁes: [Equation] By solving Equation 4, we have:
Revised: 2 ρ

Original: [Equation]

Original: In the previous section, we assume:

Original: [Equation] ρ [Equation] Here, since g i ∼ N (0 , σ 2 ) , we have
Revised: 3 while t = {

Original: [Equation]

Original: ∼ Scale-inv- X 2 ( f ( t, β 2 ) , 1 σ 2 ) .

Original: Thus, Equation 3 views Scale-inv-

Original: X 2 ( f ( t, β 2 ) , 1 σ 2 ) 2 ρ 3 while t dog t ← ∆ θ f t ( θ t 
Revised: T dog t ← ∆ θ f t ( θ t − 1 ) (

Original: v t ← 1 /
Revised: v t ← 1 /

Original: 2 v t − 1 + 1 − β g
Revised: 2 v t − 1 + (1 − β g

Original: m t ← β  m t
Revised: m t ← β 1 m t

Original: + 1  − βm m
Revised: 1 + (1 − β

Revised: ) (cid:99) m m

Original: ρ
Revised: ρ

Original:  if the variance is tractable, i.e., ρ t > 4 then l t ← 
Revised: [Table] tβ if the variance is tractable, i.e., ρ t > 4 then l t ← (cid:112) (1 − β t 2 ) /v

Original: /v
Revised: t (Compute adaptive learning rate) r t ← (cid:113) ( ρ [Table] 4)( ρ

Original: (Compute
Revised: [Table]

Original: r t ← (cid:113) ( ρt
Revised: − 2) ρ [Table] ( ρ

Original:  θ
Revised: [Table] [Table] t (Compute ) θ

Original: ← m l
Revised: ← θ t

Revised: [Table]

Revised: 1 − r

Revised: (cid:99)

Revised: m l

Original: else θ
Revised: else θ

Revised: (cid:99)

Revised: 15 return θ where f ( t, β 2 ) is the length of the SMA which allows the SMA to have the same “center of mass” with the EMA.

Revised: In other words, f ( t, β 2 ) satisﬁes:

Revised: [Equation]

Revised: By solving Equation 4, we have: f ( t, β 2 ) =

Original: 15 return as an approximation to Scale-inv-
Revised: [Equation] In the previous section, we assume: [Equation] ∼ Scale-inv-

Original: X 2 ( ρ, 1 σ 2 ) .
Revised: X 2 ( ρ, 1 σ 2 ) .

Revised: Here, since g i ∼ N (0 , σ 2 ) , we have

Revised: [Equation]

Revised: ∼ Scale-inv- X 2 ( f ( t, β 2 ) , 1 σ 2 ) .

Revised: Thus, Equation 3 views Scale-inv-

Revised: X 2 ( f ( t, β 2 ) , 1 σ 2 ) as an approximation to Scale-inv-

Revised: X 2 ( ρ, 1 σ 2 ) .

Original: Therefore, we treat f ( t, β 2 ) as an estimation of ρ .
Revised: Therefore, we treat f ( t, β 2 ) as an estimation of ρ .

Original: For ease of notation, we mark f
Revised: For ease of notation, we mark f

Original: ( t, β 2 ) as ρ t .
Revised: ( t, β 2 ) as ρ t .

Original: Also, we refer 2 1 − β 2 − 1 as ρ ∞ (maximum length of the approximated SMA), due to the inequality f ( t, β 2 ) ≤ lim t →∞ f
Revised: Also, we refer 2 1 − β 2 − 1 as ρ ∞ (maximum length of the approximated SMA), due to the inequality f ( t, β 2 ) ≤ lim t →∞ f

Original: ( t, β 2 ) =
Revised: ( t, β 2 ) =

Original: 2 1 − β 2 − 1 .
Revised: 2 1 − β 2 − 1 .

Original:  V ARIANCE E STIMATION AND R ECTIFICATION
Revised: 4. V ARIANCE E STIMATION AND R ECTIFICATION

Original: Based on previous estimations, we have Var[ ψ ( . )]
Revised: Based on previous estimations, we have Var[ ψ ( . )]

Original: =
Revised: =

Original: τ 2 ( ρ t ρ t − 2 − ρ t 2 2 ρt − 5 π
Revised: τ 2 ( ρ t ρ t − 2 − ρ t 2 2 ρt − 5 π

Original: B ( ρ t − 1 2 , ρ t − 1 2 ) 2 ) .
Revised: B ( ρ t − 1 2 , ρ t − 1 2 ) 2 ) .

Original: The value of this function in the early stage is signiﬁcantly larger than the late stage (as analyzed later, it decays roughly at the speed of O ( 1 ρ t ) ).
Revised: The value of this function in the early stage is signiﬁcantly larger than the late stage (as analyzed later, it decays roughly at the speed of O ( 1 ρ t ) ).

Original: For example, the variance at ρ t = 5 is over 100 times larger than the variance at ρ t = 500 .
Revised: For example, the variance at ρ t = 5 is over 100 times larger than the variance at ρ t = 500 .

Original: Additionally, based on Theorem 1, we know min ρ t Var[ ψ ( . )]
Revised: Additionally, based on Theorem 1, we know min ρ t Var[ ψ ( . )]

Original: =
Revised: =

Original: Var[ ψ ( . )]
Revised: Var[ ψ ( . )]

Original: | ρ t = ρ ∞ and mark this minimal value as C var.
Revised: | ρ t = ρ ∞ and mark this minimal value as C var.

Original: In order to ensure that the adaptive learning rate ( ψ ( . ) ) has consistent variance, we rectify the variance at the t -th timestamp as below,
Revised: In order to ensure that the adaptive learning rate ( ψ ( . ) ) has consistent variance, we rectify the variance at the t -th timestamp as below,

Original: [Equation]
Revised: [Equation]

Original: where r t = (cid:112) C var / Var[ ψ ( g 1 , · · · , g t )] .
Revised: where r [Equation] = [Equation]

Revised: [Equation]

Original: Although we have the analytic form of Var[ ψ ( . )]
Revised: Although we have the analytic form of Var[ ψ ( . )]

Original: ( i.e. , Equation 2), it is not numerically stable.
Revised: ( i.e. , Equation 2), it is not numerically stable.

Original: Therefore, we use the ﬁrst-order approximation to calculate the rectiﬁcation term.
Revised: Therefore, we use the ﬁrst-order approximation to calculate the rectiﬁcation term.

Original: Speciﬁcally, by approximating (cid:112) ψ 2 ( . ) to the ﬁrst order (Wolter, 2007),(cid:112) ψ 2 ( . ) ≈ (cid:112) E [ ψ 2 ( . )] + 2 (cid:112) E [ ψ 2 ( . )]( ψ 2 ( . )
Revised: Speciﬁcally, by approximating (cid:112) ψ 2 ( . ) to the ﬁrst order (Wolter, 2007),

Original: − E [ ψ 2 ( . )]) and

Original: Var[ ψ ( . )]

Original: ≈ Var[ ψ 2 ( . )]

Original: 4 E [ ψ

Original: [Equation]
Revised: [Equation]

Original: Since ψ 2 ( . ) ∼ Scale-inv- X 2 ( ρ t , 1 σ 2 ) , we have:
Revised: Since ψ 2 ( . ) ∼ Scale-inv- X 2 ( ρ t , 1 σ 2 ) , we have:

Original: [Equation]
Revised: [Equation]

Original: In Section 5.3, we conduct simulation experiments to examine Equation 5 and ﬁnd that it is a reliable approximation.
Revised: In Section 5.3, we conduct simulation experiments to examine Equation 5 and ﬁnd that it is a reliable approximation.

Original: Based on Equation 5, we know that Var[ (cid:112) ψ ( . )] decreases approximately at the  speed of O ( 1 ρ t ) .
Revised: Based on Equation 5, we know that Var[ (cid:112) ψ ( . )] decreases approximately at the [Figure] [Figure] speed of O ( 1 ρ t ) .

Original: With this approximation, we can calculate the rectiﬁcation term as:
Revised: With this approximation, we can calculate the rectiﬁcation term as:

Original: [Equation]
Revised: [Equation]

Revised: Applying our rectiﬁcation term to Adam

Original: Applying our rectiﬁcation term to Adam, we come up with a new variant of Adam, Rectiﬁed Adam (RAdam), as summarized in Algorithm 2.
Revised: [Figure] we come up with a new variant of Adam [Figure] Rectiﬁed Adam (RAdam), as summarized in Algorithm 2.

Original: Speciﬁcally, when the length of the approximated SMA is less or equal than 4, the variance of the adaptive learning rate is intractable and the adaptive learning rate is inactivated.
Revised: Speciﬁcally, when the length of the approximated SMA is less or equal than 4, the variance of the adaptive learning rate is intractable and the adaptive learning rate is inactivated.

Original: Otherwise, we calculate the variance rectiﬁcation term and update parameters with the adaptive learning rate.
Revised: Otherwise, we calculate the variance rectiﬁcation term and update parameters with the adaptive learning rate.

Original: It is worth mentioning that, if β 2 ≤ 0 .
Revised: It is worth mentioning that, if β 2 ≤ 0 .

Original: 6 , we have ρ ∞ ≤ 4 and
Revised: 6 , we have ρ ∞ ≤ 4 and

Original: RAdam is degenerated to SGD with momentum.
Revised: RAdam is degenerated to SGD with momentum.

Original: [Figure]

Original: [Figure]

Original: 4. I N C OMPARISON WITH W ARMUP AND O THER S TABILIZATION T ECHNIQUES
Revised: 4. I N C OMPARISON WITH W ARMUP AND O THER S TABILIZATION T ECHNIQUES

Original: Different from the analysis in this paper, warmup is originally proposed to handle training with very large batches for SGD (Goyal et al., 2017;
Revised: Different from the analysis in this paper, warmup is originally proposed to handle training with very large batches for SGD (Goyal et al., 2017;

Original: Gotmare et al., 2019; Bernstein et al., 2018; Xiao et al., 2019).
Revised: Gotmare et al., 2019; Bernstein et al., 2018; Xiao et al., 2017).

Original: Wefurther notice that r t has a similar form to the heuristic linear warmup, which can be viewed as setting the rectiﬁcation term as min ( t,T w ) T w .
Revised: We notice that r t has a similar form to the heuristic linear warmup, which can be viewed as setting the rectiﬁcation term as min ( t,T w ) T w .

Original: It veriﬁes our intuition that warmup works as a variance reduction technique.
Revised: It veriﬁes our intuition that warmup works as a variance reduction technique.

Original: Comparing these two strategies, RAdam deactivates the adaptive learning rate when its variance is divergent, thus avoiding undesired instability in the ﬁrst few updates.
Revised: RAdam deactivates the adaptive learning rate when its variance is divergent, thus avoiding undesired instability in the ﬁrst few updates.

Original: Besides, our method does not require an additional hyperparameter ( i.e. , T w )to control the variance reduction and can automatically adapt to different moving average rules.
Revised: Besides, our method does not require an additional hyperparameter ( i.e. , T w ) and can automatically adapt to different moving average rules.

Original: Here, we identify and address an underlying issue of adaptive optimization methods independent of (neural) model architectures.
Revised: Here, we identify and address an underlying issue of adaptive optimization methods independent of (neural) model architectures.

Original: Thus, the proposed rectiﬁcation term is orthogonal to other training stabilization techniques such as gradient clipping (Bengio et al., 2013), smoothing the adaptive learning rate ( i.e. , increasing (cid:15) or applying geometric mean ﬁlter (Chen & Gu, 2018)), initialization (Balduzzi et al., 2017; Zhang et al., 2019) and normalization (Ba et al., 2016; Ioffe & Szegedy, 2015).
Revised: Thus, the proposed rectiﬁcation term is orthogonal to other training stabilization techniques such as gradient clipping (Bengio et al., 2013), smoothing the adaptive learning rate ( i.e. , increasing (cid:15) , applying geometric mean ﬁlter (Chen & Gu, 2018), or adding range constraints (Luo et al., 2019)), initialization (Balduzzi et al., 2017; Zhang et al., 2019) and normalization (Ba et al., 2016; Ioffe & Szegedy, 2015).

Original: Indeed, these techniques can be combined with the proposed variance rectiﬁcation method.
Revised: Indeed, these techniques can be combined with the proposed variance rectiﬁcation method.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate RAdam on several benchmarks: One Billion Word for language modeling; Cifar and ImageNet for image classiﬁcation; IWSLT’14 De-En/EN-DE and WMT’16 EN-De for neural machine translation.
Revised: We evaluate RAdam on several benchmarks: One Billion Word for language modeling; Cifar and ImageNet for image classiﬁcation; IWSLT’14 De-En/EN-DE and WMT’16 EN-De for neural machine translation.

Original: Following Loshchilov & Hutter (2017), we decouple weight decays in the vanilla Adam, Adam with warmup and RAdam in our experiments.
Revised: Following Loshchilov & Hutter (2018), we decouple weight decays in the vanilla Adam, Adam with warmup and RAdam in our experiments.

Original: Details are in Appendix B.
Revised: Details are in Appendix B.

Original: 5. C OMPARING TO V ANILLA A DAM
Revised: 5. C OMPARING TO V ANILLA A DAM

Original: As analyzed before, the adaptive learning rate has undesirably large variance in the early stage of training and leads to suspicious/bad local optima on NMT.
Revised: As analyzed before, the adaptive learning rate has undesirably large variance in the early stage of training and leads to suspicious/bad local optima on NMT.

Revised: One question we are interested in

Original: One question we are interested in is: whether such an issue widely exits in other similar tasks and applications.
Revised: [Figure] [Figure] [Figure] is: whether such an issue widely exits in other similar tasks and applications.

Original: Thus, we conduct a set of experiments with two classical tasks of NLP and CV, i.e. , language modeling and image classiﬁcation.
Revised: Thus, we conduct a set of experiments with two classical tasks of NLP and CV, i.e. , language modeling and image classiﬁcation.

Original: RAdam not only results in consistent improvements over the vanilla Adam, but also demonstrates its robustness to the change of learning rates.
Revised: RAdam not only results in consistent improvements over the vanilla Adam, but also demonstrates its robustness to the change of learning rates.

Original: It veriﬁes that the variance issue exists in various machine learning applications, and has a big impact on the model behavior.
Revised: It veriﬁes that the variance issue exists in various machine learning applications, and has a big impact on the model behavior.

Original: Performance Comparison.
Revised: Performance Comparison.

Original: The performances on language modeling ( i.e. , One Billion
Revised: The performances on language modeling ( i.e. , One Billion

Original: Word (Chelba et al., 2013)) and image classiﬁcation ( i.e. , CIFAR10 (Krizhevsky et al., 2009) and
Revised: Word (Chelba et al., 2013)) and image classiﬁcation ( i.e. , CIFAR10 (Krizhevsky et al., 2009) and

Original: ImageNet (Deng et al., 2009)) are presented in Figure 4, 5.
Revised: ImageNet (Deng et al., 2009)) are presented in Figure 4, 5.

Original: The results show that RAdam outperforms Adam in all three datasets.
Revised: The results show that RAdam outperforms Adam in all three datasets.

Original: As shown in Figure 4, although the rectiﬁcation term makes[Figure] [Figure]
Revised: As shown in Figure 4, although the rectiﬁcation term makes

Original: RAdam slower than the vanilla Adam in the ﬁrst few epochs, it allows RAdam to converge faster after that.
Revised: RAdam slower than the vanilla Adam in the ﬁrst few epochs, it allows RAdam to converge faster after that.

Original: In other words, by reducing the variance of the adaptive learning rate in the early stage, it gets both faster convergence and better performance, which veriﬁes the impact of the variance issue.
Revised: In other words, by reducing the variance of the adaptive learning rate in the early stage, it gets both faster convergence and better performance, which veriﬁes the impact of the variance issue.

Original: We also observe that RAdam obtains consistent improvements over Adam on image classiﬁcation.
Revised: We also observe that RAdam obtains consistent improvements over Adam on image classiﬁcation.

Original: It is worth noting that, on both ImageNet and CIFAR10, although RAdam fails to outperform SGD in terms of test accuracy, it results in a better training performance ( e [Figure] g. , the training accuracy of
Revised: It is worth noting that, on both ImageNet and CIFAR10, although RAdam fails to outperform SGD in terms of test accuracy, it results in a better training performance ( e.g. , the training accuracy of

Original: SGD, Adam, and RAdam on ImageNet are 69 .
Revised: SGD, Adam, and RAdam on ImageNet are 69 .

Original: 12 and 70 . 30 respectively).
Revised: 12 and 70 . 30 respectively).

Original: Robustness to Learning Rate Change.
Revised: Robustness to Learning Rate Change.

Original: Besides performance improvements, RAdam also improves the robustness of model training.
Revised: Besides performance improvements, RAdam also improves the robustness of model training.

Original: We use different initial learning rates, conduct experiments with
Revised: We use different initial learning rates, conduct experiments with

Original: ResNet-20 on the CIFAR10 datasets, and summarize their performance in Figure 6.
Revised: ResNet-20 on the CIFAR10 datasets, and summarize their performance in Figure 6.

Original: For learning rates within a broad range ( i.e. , { 0 . 1 , 0 . 03 , 0 . 01 , 0 . 003 } ), RAdam achieves consistent model performances (their test accuracy curves highly overlap with each other), while Adam and SGD are shown to be more sensitive to the learning rate.
Revised: For learning rates within a broad range ( i.e. , { 0 . 1 , 0 . 03 , 0 . 01 , 0 . 003 } ), RAdam achieves consistent model performances (their test accuracy curves highly overlap with each other), while Adam and SGD are shown to be more sensitive to the learning rate.

Original: The observation can be interpreted that by rectifying the variance of the adaptive learning rate, RAdam improves the robustness of model training and can adapt to different learning rates of a broader range.
Revised: The observation can be interpreted that by rectifying the variance of the adaptive learning rate, RAdam improves the robustness of model training and can adapt to different learning rates of a broader range.

Original: 5. C OMPARING TO H EURISTIC W ARMUP
Revised: 5. C OMPARING TO H EURISTIC W ARMUP

Original: To examine the effectiveness of RAdam, we ﬁrst conduct comparisons on neural machine translation, on which the state-of-the-art employs Adam with the linear warmup.
Revised: To examine the effectiveness of RAdam, we ﬁrst conduct comparisons on neural machine translation, on which the state-of-the-art employs Adam with the linear warmup.

Original: Speciﬁcally, we conduct experiments on three datasets, i.e., IWSLT’14 De-En, IWSLT’14 En-De, and WMT’16 En-De. 
Revised: Speciﬁcally, we conduct experiments on three datasets, i.e., IWSLT’14 De-En, IWSLT’14 En-De, and WMT’16 En-De. Due

Original: Due to the limited size of the IWSLT’14 dataset, we conduct experiments using 5 different random seeds and report their mean and standard derivation.
Revised: [Table] to the limited size of the IWSLT’14 dataset, we conduct experiments using 5 different random seeds and report their mean and standard derivation.

Original: As discussed before, the vanilla Adam algorithm leads to suspicious/bad local optima (i.e., converges to a training perplexity around 500), and needs a learning rate warmup stage to stabilize the training.
Revised: As discussed before, the vanilla Adam algorithm leads to suspicious/bad local optima (i.e., converges to a training perplexity around 500), and needs a learning rate warmup stage to stabilize the training.

Original: We summarize the performance obtained with the heuristic warmup and our proposed rectiﬁcation term in Table 2 and visualize the training curve of IWSLT De-En in Figure 1.
Revised: We summarize the performance obtained with the heuristic warmup and our proposed rectiﬁcation term in Table 2 and visualize the training curve of IWSLT De-En in Figure 1.

Original: With a consistent adaptive learning rate variance, our proposed method achieves similar performance to that of previous state-of-the-art warmup heuristics.
Revised: With a consistent adaptive learning rate variance, our proposed method achieves similar performance to that of previous state-of-the-art warmup heuristics.

Original: It veriﬁes our intuition that the problematic updates of Adam are indeed caused by the undesirably large variance in the early stage.[Figure] [Table] [Figure]
Revised: It veriﬁes our intuition that the problematic updates of Adam are indeed caused by the undesirably large variance in the early stage.

Original: [Figure] Moreover [Figure] we applied Adam with warmup on the CIFAR10 dataset.
Revised: Moreover, we applied Adam with warmup on the CIFAR10 dataset.

Original: Its best accuracy on the test set is 91 . 29 , which is similar to RAdam ( 91 . 38 ).
Revised: Its best accuracy on the test set is 91 . 29 , which is similar to RAdam ( 91 . 38 ).

Original: However, we found that RAdam requires less hyperparameter tuning.
Revised: However, we found that RAdam requires less hyperparameter tuning.

Original: Speciﬁcally, we visualize their learning curves in Figure 7.
Revised: Speciﬁcally, we visualize their learning curves in Figure 7.

Original: For some warmup steps, Adam with warmup is relatively more sensitive to the choice of the learning rate.
Revised: For some warmup steps, Adam with warmup is relatively more sensitive to the choice of the learning rate.

Original: RAdam, at the same time, is not only more robust [Figure] but also can automatically control the warmup behavior ( i.e. , without requiring the length of warmup).
Revised: RAdam, at the same time, is not only more robust, but also can automatically control the warmup behavior ( i.e. , without requiring the length of warmup).

Original: For example, when setting the learning rate as 0 . 1 , Adam with 100 steps of warmup fails to get satisfying performance and only results in an accuracy of 90 . 13 ; RAdam successfully gets an accuracy of 91 .
Revised: For example, when setting the learning rate as 0 . 1 , Adam with 100 steps of warmup fails to get satisfying performance and only results in an accuracy of 90 . 13 ; RAdam successfully gets an accuracy of 91 .

Original: 06 , with the original setting of the moving average calculation ( i.e. , β 1 = 0 .
Revised: 06 , with the original setting of the moving average calculation ( i.e. , β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: We conjecture the reason is due to the fact that RAdam, which is based on a rigorous variance analysis, explicitly avoids the extreme situation where the variance is divergent, and rectiﬁes the variance to be consistent in other situations.
Revised: We conjecture the reason is due to the fact that RAdam, which is based on a rigorous variance analysis, explicitly avoids the extreme situation where the variance is divergent, and rectiﬁes the variance to be consistent in other situations.

Original: 5. S IMULATED V ERIFICATION
Revised: 5. S IMULATED V ERIFICATION

Original: In Sections 3 and 4, we approximate Var[ (cid:113) t/ (cid:80) ti =1 g 2 i ] to the ﬁrst order, and assume ψ 2 ( . )
Revised: In Sections 3 and 4, we approximate Var[ (cid:113) t/ (cid:80) ti =1 g 2 i ] to the ﬁrst order, and assume ψ 2 ( . )

Original: = 1 − β [Equation] [Equation]
Revised: = 1 − β t 2 (1 − β 2 ) (cid:80) ti =

Original: − β 2 ) (cid:80) ti =1 β t − i 2 g 2 i subjects to a scaled inverse chi-square distribution (this assumption covers the approximation from EMA to SMA).
Revised: 1 β t − i 2 g 2 i subjects to a scaled inverse chi-square distribution (this assumption covers the approximation from EMA to SMA).

Original: Here, we examine these two approximations using simulations.
Revised: Here, we examine these two approximations using simulations.

Original: First Order Approximation of Var[ (cid:113) t/ (cid:80) ti =
Revised: First Order Approximation of Var[ (cid:113) t/ (cid:80) ti =

Original: 1 g 2 i ] .
Revised: 1 g 2 i ] .

Original: To compare Equations 5 and 2, we assume τ = 1 and plot their values and difference for ν = { 5 , · · · , 500 } in Figure 8.
Revised: To compare Equations 5 and 2, we assume τ = 1 and plot their values and difference for ν = { 5 , · · · , 500 } in Figure 8.

Original: The curve of the analytic form and the ﬁrst-order approximation highly overlap, and their difference is much smaller than their value.
Revised: The curve of the analytic form and the ﬁrst-order approximation highly overlap, and their difference is much smaller than their value.

Original: This result veriﬁes that our ﬁrst-order approximation is very accurate.
Revised: This result veriﬁes that our ﬁrst-order approximation is very accurate.

Original: Scaled Inverse Chi-Square Distribution Assumption.
Revised: Scaled Inverse Chi-Square Distribution Assumption.

Original: In this paper, we assume g i accords to a
Revised: In this paper, we assume g i accords to a

Original: Normal distribution with a zero mean.
Revised: Normal distribution with a zero mean.

Original: We also assume ψ 2 ( . )
Revised: We also assume ψ 2 ( . )

Original: accords to the scaled inverse chi-square distribution to derive the variance of Var[ ψ ( . )] , based on the similarity between the exponential moving average and simple moving average.
Revised: accords to the scaled inverse chi-square distribution to derive the variance of Var[ ψ ( . )] , based on the similarity between the exponential moving average and simple moving average.

Original: Here, we empirically verify this assumption.
Revised: Here, we empirically verify this assumption.

Original: Speciﬁcally, since g i in the optimization problem may not be zero-mean, we assume its expectation is µ and sample g i from N ( µ, 1) .
Revised: Speciﬁcally, since g i in the optimization problem may not be zero-mean, we assume its expectation is µ and sample g i from N ( µ, 1) .

Original: Then, based on these samples, we calculate the variance of the original adaptive learning rate and the proposed rectiﬁed adaptive learning rate, i.e. , Var[ 1 (cid:98) v t ] and
Revised: Then, based on these samples, we calculate the variance of the original adaptive learning rate and the proposed rectiﬁed adaptive learning rate, i.e. , Var[ 1 (cid:98) v t ] and

Original: Var[ r t (cid:98) v t ] respectively.
Revised: Var[ r t (cid:98) v t ] respectively.

Original: We set β 2 to 0 .
Revised: We set β 2 to 0 .

Original: 999 , the number of sampled trajectories to 5000 , the number of iterations to 6000 , and summarize the simulation results in Figure 9.
Revised: 999 , the number of sampled trajectories to 5000 , the number of iterations to 6000 , and summarize the simulation results in Figure 9.

Original: Across all six settings with different µ , the adaptive learning rate has a larger variance in the ﬁrst stage and the rectiﬁed adaptive learning rate has relative consistent variance.
Revised: Across all six settings with different µ , the adaptive learning rate has a larger variance in the ﬁrst stage and the rectiﬁed adaptive learning rate has relative consistent variance.

Original: This veriﬁes the reliability of our assumption.
Revised: This veriﬁes the reliability of our assumption.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we explore the underlying principle of the effectiveness of the warmup heuristic used for adaptive optimization algorithms.
Revised: In this paper, we explore the underlying principle of the effectiveness of the warmup heuristic used for adaptive optimization algorithms.

Original: Speciﬁcally, we identify that, due to the limited amount of samples in the early stage of model training, the adaptive learning rate has an undesirably large variance and can cause the model to converge to suspicious/bad local optima.
Revised: Speciﬁcally, we identify that, due to the limited amount of samples in the early stage of model training, the adaptive learning rate has an undesirably large variance and can cause the model to converge to suspicious/bad local optima.

Original: We provide both empirical and theoretical evidence to support our hypothesis and further propose a new variant of
Revised: We provide both empirical and theoretical evidence to support our hypothesis, and further propose a new variant [Figure] [Figure] µ

Revised: [Figure]

Original: Adam, whose adaptive learning rate is rectiﬁed so as to have a consistent variance.
Revised: µ [Figure] [Figure] µ

Revised: [Figure]

Revised: [Figure]

Revised: of Adam

Revised: , whose adaptive learning rate is rectiﬁed so as to have a consistent variance.

Original: Empirical results demonstrate the effectiveness of our proposed method.
Revised: Empirical results demonstrate the effectiveness of our proposed method.

Original: In future work, we plan to apply the proposed method to other applications such as Named Entity Recognition (Reimers & Gurevych, 2017; Lin et al., 2019).
Revised: In future work, we plan to replace the rectiﬁcation strategy by sharing the second moment estimation across similar parameters. A CKNOWLEDGE

Original: Another interesting direction to pursue is to adapt the choice of β based on the variance estimation of different parameters, i e. , use a larger β for parameters with a larger variance.
Revised: We thank Aeyuan Allen-Zhu for valuable discussions and comments, Microsoft Research Technology Engineering team for setting up GPU machines. Research was sponsored in part by DARPA No. W911NF-17-C-0099 and FA8750-19-2-1004, National Science Foundation IIS 16-18481, IIS 17-04532, and IIS-17-41317, and DTRA HDTRA11810026.

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: For ease of notation, we refer ψ 2 ( . ) as x and 1 σ 2 as τ 2 .
Revised: For ease of notation, we refer ψ 2 ( . ) as x and 1 σ 2 as τ 2 .

Original: Thus, x ∼ Scale-inv-
Revised: Thus, x ∼ Scale-inv-

Original: X 2 ( ρ, τ 2 ) and:
Revised: X 2 ( ρ, τ 2 ) and:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where Γ( . ) is the gamma function.
Revised: where Γ( . ) is the gamma function.

Original: Therefore, we have:
Revised: Therefore, we have:

Original: [Equation]
Revised: [Equation]

Original: Based on Equation 6 and 7, for ∀ ρ > 4 , we have:
Revised: Based on Equation 6 and 7, for ∀ ρ > 4 , we have:

Original: [Equation]
Revised: [Equation]

Original: where B ( . ) is the beta function.
Revised: where B ( . ) is the beta function.

Original: To prove the monotonic property of Var[ ψ ( . )] , we need to show:
Revised: To prove the monotonic property of Var[ ψ ( . )] , we need to show:

Original: Lemma 1.
Revised: Lemma 1.

Original: for t ≥ 4 , ∂∂t ( t t − 2 − t 2 2 t − 5 π
Revised: for t ≥ 4 , ∂∂t ( t t − 2 − t 2 2 t − 5 π

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The target inequality can be re-wrote as
Revised: The target inequality can be re-wrote as

Original: [Equation]
Revised: [Equation]

Original: This inequality is equivalent to: 64 π
Revised: This inequality is equivalent to: 64 π

Original: [Equation]
Revised: [Equation]

Original: where ( i ) is derived from Legendre duplication formula.
Revised: where ( i ) is derived from Legendre duplication formula.

Original: Simplify the above inequality
Revised: Simplify the above inequality

Original: [Equation]
Revised: [Equation]

Original: we get:
Revised: we get:

Original: [Equation]
Revised: [Equation]

Original: We only need to show 64 π
Revised: We only need to show 64 π

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is from ln( x ) − 1 / (2 x ) > Ψ( x ) >
Revised: where the ﬁrst inequality is from ln( x ) − 1 / (2 x ) > Ψ( x ) >

Original: ln( x + 0 . 5) − 1 /x .
Revised: ln( x + 0 . 5) − 1 /x .

Original: Therefore, we only need to show
Revised: Therefore [Equation] we only need to show

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to
Revised: which is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where ( i ) is from Legendre duplication formula.
Revised: where ( i ) is from Legendre duplication formula.

Original: So we only need to show
Revised: So we only need to show

Original: [Equation]
Revised: [Equation]

Original: Using Gautschi’s inequality ( Γ( x
Revised: Using Gautschi’s inequality ( Γ( x

Original: +1) Γ( x + s ) <
Revised: +1) Γ( x + s ) <

Original: ( x + 1) 1 − s ), we have
Revised: ( x + 1) 1 − s ), we have

Original: [Equation]
Revised: [Equation]

Original: B I MPLEMENTATION D ETAILS B. L ANGUAGE M ODELING
Revised: B I MPLEMENTATION D ETAILS B. L ANGUAGE M ODELING

Original: Our implementation is based on the previous work (Liu et al., 2018).
Revised: Our implementation is based on the previous work (Liu et al., 2018).

Original: Speciﬁcally, we use two-layer
Revised: Speciﬁcally, we use two-layer

Original: LSTMs with 2048 hidden states with adaptive softmax to conduct experiments on the one billion words dataset.
Revised: LSTMs with 2048 hidden states with adaptive softmax to conduct experiments on the one billion words dataset.

Original: Word embedding (random initialized) of 300 dimensions is used as the input and the adaptive softmax is incorporated with a default setting (cut-offs are set to [4000 , 40000 , 200000] ).
Revised: Word embedding (random initialized) of 300 dimensions is used as the input and the adaptive softmax is incorporated with a default setting (cut-offs are set to [4000 , 40000 , 200000] ).

Original: Additionally, as pre-processing, we replace all tokens occurring equal or less than 3 times with as
Revised: Additionally, as pre-processing, we replace all tokens occurring equal or less than 3 times with as

Original: UNK, which shrinks the dictionary from 7.9M to 6.4M. Dropout is applied to each layer with a ratio of 0 . 1 , gradients are clipped at 5.0.
Revised: UNK, which shrinks the dictionary from 7.9M to 6.4M. Dropout is applied to each layer with a ratio of 0 . 1 , gradients are clipped at 5.0.

Original: We use the default hyper-parameters to update moving averages, i.e. β 1 = 0 .
Revised: We use the default hyper-parameters to update moving averages, i.e. β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: The learning rate is set to start from 0.001, and decayed at the start of 10th epochs.
Revised: The learning rate is set to start from 0.001, and decayed at the start of 10th epochs.

Original: LSTMs are unrolled for 20 steps without resetting the LSTM states and the batch size is set to 128.
Revised: LSTMs are unrolled for 20 steps without resetting the LSTM states and the batch size is set to 128.

Original: All models are trained on one NVIDIA Tesla V100 GPU.
Revised: All models are trained on one NVIDIA Tesla V100 GPU.

Original: B. I MAGEINE C LASSIFICATION
Revised: B. I MAGEINE C LASSIFICATION

Original: We use the default ResNet architectures (He et al., 2016) in a public pytorch re-implementation 3 .
Revised: We use the default ResNet architectures (He et al., 2016) in a public pytorch re-implementation 4 .

Original: Speciﬁcally, we use 20 -layer ResNet ( 9 Basic Blocks) for CIFAR-10 and 18-layer ResNet ( 8 Basic
Revised: Speciﬁcally, we use 20 -layer ResNet ( 9 Basic Blocks) for CIFAR-10 and 18-layer ResNet ( 8 Basic

Original: Blocks) for ImageNet.
Revised: Blocks) for ImageNet.

Original: Batch size is 128 for CIFAR-10 and 256 for ImageNet.
Revised: Batch size is 128 for CIFAR-10 and 256 for ImageNet.

Original: The model is trained for 186 epoches and the learning rate decays at the 81 -th and the 122 -th epoches by 0 .
Revised: The model is trained for 186 epoches and the learning rate decays at the 81 -th and the 122 -th epoches by 0 .

Original: 1 on CIFAR10, while the model is trained for 90 epoches and the learning rate decays at the 31 -th and the 61 -th epoch by 0 .
Revised: 1 on CIFAR10, while the model is trained for 90 epoches and the learning rate decays at the 31 -th and the 61 -th epoch by 0 .

Original: 1 on ImageNet.
Revised: 1 on ImageNet.

Original: For Adam and RAdam, we set β 1 = 0 .
Revised: For Adam and RAdam, we set β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: For SGD, we set the momentum factor as 0 . 9 .
Revised: For SGD, we set the momentum factor as 0 . 9 .

Original: The weight decay rate is 10 − 4 .
Revised: The weight decay rate is 10 − 4 .

Original: Random cropping and random horizontal ﬂipping are applied to training data.
Revised: Random cropping and random horizontal ﬂipping are applied to training data.

Original: B. N EURAL M ACHINE T RANSLATION
Revised: B. N EURAL M ACHINE T RANSLATION

Original: Our experiments are based on the default Transformers (Vaswani et al., 2017) implementation from the fairseq package (Ott et al., 2019).
Revised: Our experiments are based on the default Transformers (Vaswani et al., 2017) implementation from the fairseq package (Ott et al., 2019).

Original: Speciﬁcally, we use word embedding with 512 dimensions and 6-layer encoder / decoder with 4 head and 1024 hidden dimensions on the IWSLT14’ dataset; use word embedding with 512 dimension and 6-layer encoder / decoder with 8 heads and 2048 hidden dimensions.
Revised: Speciﬁcally, we use word embedding with 512 dimensions and 6-layer encoder / decoder with 4 head and 1024 hidden dimensions on the IWSLT14’ dataset; use word embedding with 512 dimension and 6-layer encoder / decoder with 8 heads and 2048 hidden dimensions.

Original: Label smoothed cross entropy is used as the objective function with an uncertainty = 0 .
Revised: Label smoothed cross entropy is used as the objective function with an uncertainty = 0 .

Original: 1 (Szegedy et al., 2016).
Revised: 1 (Szegedy et al., 2016).

Original: We use linear learning rate decay starting from 3 e − 4 , and the checkpoints of the last 20 epoches are averaged before evaluation.
Revised: We use linear learning rate decay starting from 3 e − 4 , and the checkpoints of the last 20 epoches are averaged before evaluation.

Original: As to the wamrup strategy, we use a linear warmup for Adam in the ﬁrst 4000 updates, and set β 2 to satisfy ν = 4000 ( β 2 = 0 . 9995 ).
Revised: As to the wamrup strategy, we use a linear warmup for Adam in the ﬁrst 4000 updates, and set β 2 to satisfy ν = 4000 ( β 2 = 0 . 9995 ).

Original: In the
Revised: In the

Original: IWSLT’14 dataset, we conduct training on one NVIDIA Tesla V100 GPU, set maximum batch size as 4000 , apply dropout with a ratio 0 .
Revised: IWSLT’14 dataset, we conduct training on one NVIDIA Tesla V100 GPU, set maximum batch size as 4000 , apply dropout with a ratio 0 .

Original: 3 , using weight decay of 0 . 

Revised: 3 , using weight decay of 0 .

Original: and clip the gradient norm at 25 .
Revised: and clip the gradient norm at 25 .

Original: In the WMT’16 dataset, we conduct training on four NVIDIA Quadro R8000 GPUs and set maximum batch size as 8196 .
Revised: In the WMT’16 dataset, we conduct training on four NVIDIA Quadro R8000 GPUs and set maximum batch size as 8196 .

Revised: C D OWNGRADING TO SGDM

Revised: As a byproduct determined by math derivations, we degenerated RAdam to SGD with momentum in the ﬁrst several updates.

Revised: Although this stage only contains several gradient updates, these up- dates could be quite damaging (e.g., in our Figure 2, the gradient distribution is distorted withingradient updates).

Revised: Intuitively, updates with divergent adaptive learning rate variance could be more damaging than the ones with converged variance, as divergent variance implies more instability.

Revised: As a case study, we performed experiments on the CIFAR10 dataset.

Revised: Five-run average results are summarized in Table 3.

Revised: The optimizer fails to get an equally reliably model when changing the ﬁrst 4 updates to Adam, yet the inﬂuence of switching is less deleterious when we change 5-8 updates instead.

Revised: This result veriﬁes our intuition and is in agreement with our theory the ﬁrst few updates could be more damaging than later updates.

Revised: By saying that, we still want to emphasize that this part (downgrading to SGDM) is only a minor part of our algorithm design whereas our main focus is on the mechanism of warmup and the derivation of the rectiﬁcation term.

Revised: [Table]