Original: P ADAM : C LOSING THE G ENERALIZATION G AP OF A DAPTIVE G RADIENT M ETHODS IN T RAINING D EEP N EURAL N ETWORKS
Revised: P ADAM : C LOSING THE G ENERALIZATION G AP OF A DAPTIVE G RADIENT M ETHODS IN T RAINING D EEP N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks.
Revised: Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks.

Original: This leaves how to close the generalization gap of adaptive gradient methods an open problem.
Revised: This leaves how to close the generalization gap of adaptive gradient methods an open problem.

Original: In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes “over adapted”.
Revised: In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes “over adapted”.

Original: We design a new algorithm, called Partially adaptive momentum estimation method (Padam), which uniﬁes the Adam/Amsgrad with SGD by introducing a partial adaptive parameter p , to achieve the best from both worlds.
Revised: We design a new algorithm, called Partially adaptive momentum estimation method (Padam), which uniﬁes the Adam/Amsgrad with SGD by introducing a partial adaptive parameter p , to achieve the best from both worlds.

Original: Experiments on standard benchmarks show that Padam can maintain fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks.
Revised: Experiments on standard benchmarks show that Padam can maintain fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks.

Original: These results would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks.
Revised: These results would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Stochastic gradient descent (SGD) is now one of the most dominant approaches for training deep neural networks (Goodfellow et al., 2016).
Revised: Stochastic gradient descent (SGD) is now one of the most dominant approaches for training deep neural networks (Goodfellow et al., 2016).

Original: In each iteration, SGD only performs one parameter update on a mini-batch of training examples.
Revised: In each iteration, SGD only performs one parameter update on a mini-batch of training examples.

Original: SGD is simple and has been proved to be efﬁcient, especially for tasks on large datasets.
Revised: SGD is simple and has been proved to be efﬁcient, especially for tasks on large datasets.

Original: In recent years, adaptive variants of SGD have emerged and shown their successes for their convenient automatic learning rate adjustment mechanism.
Revised: In recent years, adaptive variants of SGD have emerged and shown their successes for their convenient automatic learning rate adjustment mechanism.

Original: Adagrad (Duchi et al., 2011) is probably the ﬁrst along this line of research, and signiﬁcantly outperforms vanilla SGD in the sparse gradient scenario.
Revised: Adagrad (Duchi et al., 2011) is probably the ﬁrst along this line of research, and signiﬁcantly outperforms vanilla SGD in the sparse gradient scenario.

Original: Despite the ﬁrst success, Adagrad was later found to demonstrate degraded performance especially in cases where the loss function is nonconvex or the gradient is dense.
Revised: Despite the ﬁrst success, Adagrad was later found to demonstrate degraded performance especially in cases where the loss function is nonconvex or the gradient is dense.

Original: Many variants of Adagrad, such as RMSprop (Hinton et al., 2012), Adam (Kingma & Ba, 2015), Adadelta (Zeiler, 2012), Nadam (Dozat, 2016), were then proposed to address these challenges by adopting exponential moving average rather than the arithmetic average used in
Revised: Many variants of Adagrad, such as RMSprop (Hinton et al., 2012), Adam (Kingma & Ba, 2015), Adadelta (Zeiler, 2012), Nadam (Dozat, 2016), were then proposed to address these challenges by adopting exponential moving average rather than the arithmetic average used in

Original: Adagrad.
Revised: Adagrad.

Original: This change largely mitigates the rapid decay of learning rate in Adagrad and hence makes this family of algorithms, especially Adam, particularly popular on various tasks.
Revised: This change largely mitigates the rapid decay of learning rate in Adagrad and hence makes this family of algorithms, especially Adam, particularly popular on various tasks.

Original: Recently, it has also been observed (Reddi et al., 2018) that Adam does not converge in some settings where rarely encountered large gradient information quickly dies out due to the “short momery” problem of exponential moving average.
Revised: Recently, it has also been observed (Reddi et al., 2018) that Adam does not converge in some settings where rarely encountered large gradient information quickly dies out due to the “short momery” problem of exponential moving average.

Original: To address this issue, Amsgrad (Reddi et al., 2018) has been proposed to keep an extra “long term memory” variable to preserve the past gradient information and to correct the potential convergence issue in Adam.
Revised: To address this issue, Amsgrad (Reddi et al., 2018) has been proposed to keep an extra “long term memory” variable to preserve the past gradient information and to correct the potential convergence issue in Adam.

Original: On the other hand, people recently found that for largely over-parameterized neural networks, e.g., more complex modern convolutional neural network (CNN) architectures such as VGGNet (He et al., 2016), ResNet (He et al., 2016), Wide ResNet (Zagoruyko & Komodakis, 2016), DenseNet (Huang et al., 2017), training with Adam or its variants typically generalizes worse than SGD with momentum, even when the training performance is better.
Revised: On the other hand, people recently found that for largely over-parameterized neural networks, e.g., more complex modern convolutional neural network (CNN) architectures such as VGGNet (He et al., 2016), ResNet (He et al., 2016), Wide ResNet (Zagoruyko & Komodakis, 2016), DenseNet (Huang et al., 2017), training with Adam or its variants typically generalizes worse than SGD with momentum, even when the training performance is better.

Original: In particular, people found that carefullytuned SGD, combined with proper momentum, weight decay and appropriate learning rate decay strategies, can signiﬁcantly outperform adaptive gradient algorithms eventually (Wilson et al., 2017).
Revised: In particular, people found that carefullytuned SGD, combined with proper momentum, weight decay and appropriate learning rate decay strategies, can signiﬁcantly outperform adaptive gradient algorithms eventually (Wilson et al., 2017).

Original: As a result, even though adaptive gradient methods are relatively easy to tune and converge faster at the early stage, recent advances in designing neural network structures are all reporting their performances by training their models with SGD-momentum (He et al., 2016; Zagoruyko & Komodakis,
Revised: As a result, even though adaptive gradient methods are relatively easy to tune and converge faster at the early stage, recent advances in designing neural network structures are all reporting their performances by training their models with SGD-momentum (He et al., 2016; Zagoruyko & Komodakis,

Original: ; Huang et al., 2017; Simonyan & Zisserman, 2014; Ren et al., 2015; Xie et al., 2017; Howard et al., 2017).
Revised: ; Huang et al., 2017; Simonyan & Zisserman, 2014; Ren et al., 2015; Xie et al., 2017; Howard et al., 2017).

Original: Moreover, it is difﬁcult to apply the same learning rate decay strategies that work well in SGD with momentum to adaptive gradient methods, since adaptive gradient methods usually require a much smaller base learning rate that will soon die out after several rounds of decay.
Revised: Moreover, it is difﬁcult to apply the same learning rate decay strategies that work well in SGD with momentum to adaptive gradient methods, since adaptive gradient methods usually require a much smaller base learning rate that will soon die out after several rounds of decay.

Original: We refer to it as the “small learning rate dilemma” (see more details in Section 3).
Revised: We refer to it as the “small learning rate dilemma” (see more details in Section 3).

Original: With all these observations, a natural question is:
Revised: With all these observations, a natural question is:

Original: Can we take the best from both Adam and SGD-momentum, i.e., design an algorithm that not only enjoys the fast convergence rate as Adam, but also generalizes as well as SGD-momentum?
Revised: Can we take the best from both Adam and SGD-momentum, i.e., design an algorithm that not only enjoys the fast convergence rate as Adam, but also generalizes as well as SGD-momentum?

Original: In this paper, we answer this question afﬁrmatively.
Revised: In this paper, we answer this question afﬁrmatively.

Original: We close the generalization gap of adaptive gradient methods by presenting a new algorithm, called p artially ada ptive m omentum estimation (Padam) method, which uniﬁes Adam/Amsgrad with SGD-momentum to achieve the best of both worlds.
Revised: We close the generalization gap of adaptive gradient methods by presenting a new algorithm, called p artially ada ptive m omentum estimation (Padam) method, which uniﬁes Adam/Amsgrad with SGD-momentum to achieve the best of both worlds.

Original: In particular, we make the following contributions:
Revised: In particular, we make the following contributions:

Original: • We propose a new algorithm, Padam, which uniﬁes Adam/Amsgrad and SGD with momentum by a partially adaptive parameter.
Revised: • We propose a new algorithm, Padam, which uniﬁes Adam/Amsgrad and SGD with momentum by a partially adaptive parameter.

Original: We show that Adam/Amsgrad can be seen as a special fully adaptive instance of Padam.
Revised: We show that Adam/Amsgrad can be seen as a special fully adaptive instance of Padam.

Original: The intuition behind our algorithm is that, by controlling the degree of adaptiveness, the learning rate in Padam does not need to be as small as other adaptive gradient methods to prevent the gradient update from exploding.
Revised: The intuition behind our algorithm is that, by controlling the degree of adaptiveness, the learning rate in Padam does not need to be as small as other adaptive gradient methods to prevent the gradient update from exploding.

Original: This resolves the “small learning rate dilemma” for adaptive gradient methods and allows for faster convergence, hence closing the gap of generalization.
Revised: This resolves the “small learning rate dilemma” for adaptive gradient methods and allows for faster convergence, hence closing the gap of generalization.

Original: • We show that the Padam’s performance is also theoretically guaranteed.
Revised: • We show that the Padam’s performance is also theoretically guaranteed.

Original: We provide a convergence analysis of Padam in the convex setting, based on the analysis of Kingma & Ba (2015); Reddi et al.
Revised: We provide a convergence analysis of Padam in the convex setting, based on the analysis of Kingma & Ba (2015); Reddi et al.

Original: (2018), and prove a data-dependent regret bound.
Revised: (2018), and prove a data-dependent regret bound.

Original: • We also provide thorough experiments about our proposed Padam method on training modern deep neural architectures.
Revised: • We also provide thorough experiments about our proposed Padam method on training modern deep neural architectures.

Original: We empirically show that Padam achieves the fastest convergence speed while
Revised: We empirically show that Padam achieves the fastest convergence speed while

Original: generalizing as well as SGD with momentum.
Revised: generalizing as well as SGD with momentum.

Original: These results suggest that practitioners should pick up adaptive gradient methods once again for faster training of deep neural networks.
Revised: These results suggest that practitioners should pick up adaptive gradient methods once again for faster training of deep neural networks.

Original: Notation: Scalars are denoted by lower case letters, vectors by lower case bold face letters, and matrices by upper case bold face letters.
Revised: Notation: Scalars are denoted by lower case letters, vectors by lower case bold face letters, and matrices by upper case bold face letters.

Original: For a vector θ ∈ R d , we denote the (cid:96) 2 norm of θ by (cid:107) θ (cid:107) 2 = (cid:113)(cid:80) di
Revised: For a vector θ ∈ R d , we denote the (cid:96) 2 norm of θ by (cid:107) θ (cid:107) 2 = (cid:113)(cid:80) di

Original: =1 θ 2 i , the (cid:96) ∞ norm of θ by (cid:107) θ (cid:107) ∞ =
Revised: =1 θ 2 i , the (cid:96) ∞ norm of θ by (cid:107) θ (cid:107) ∞ =

Original: max di =1 | θ i | .
Revised: max di =1 | θ i | .

Original: For a sequence of vectors { θ j } tj =1 , we denote by θ j,i the i -th element in θ j .
Revised: For a sequence of vectors { θ j } tj =1 , we denote by θ j,i the i -th element in θ j .

Original: We also denote θ 1: t,i =
Revised: We also denote θ 1: t,i =

Original: [ θ 1 ,i , θ 2 ,i , . . .
Revised: [ θ 1 ,i , θ 2 ,i , . . .

Original: , θ t,i ] (cid:62) .
Revised: , θ t,i ] (cid:62) .

Original: With slightly abuse of notation, for any two vectors a and b , we denote a 2 as the element-wise square, a p as the element-wise power operation, a / b as the element-wise division and max( a , b ) as the element-wise maximum.
Revised: With slightly abuse of notation, for any two vectors a and b , we denote a 2 as the element-wise square, a p as the element-wise power operation, a / b as the element-wise division and max( a , b ) as the element-wise maximum.

Original: We denote by diag( a ) a diagonal matrix with diagonal entries a 1 , . .
Revised: We denote by diag( a ) a diagonal matrix with diagonal entries a 1 , . .

Original: . , a d .
Revised: . , a d .

Original: Let S d ++ be the set of all positive deﬁnite d × d matrices.
Revised: Let S d ++ be the set of all positive deﬁnite d × d matrices.

Original: We denote by Π X , A ( b ) the projection of b onto a convex set X , i.e., argmin a ∈X (cid:107)
Revised: We denote by Π X , A ( b ) the projection of b onto a convex set X , i.e., argmin a ∈X (cid:107)

Original: A 1 / 2 ( a − b ) (cid:107) 2 for b ∈ R d , A ∈ S d ++ .
Revised: A 1 / 2 ( a − b ) (cid:107) 2 for b ∈ R d , A ∈ S d ++ .

Original: Given two sequences { a n } and { b n } , we write a n = O ( b n ) if there exists a constant 0 < C < + ∞ such that a n ≤ C b n .
Revised: Given two sequences { a n } and { b n } , we write a n = O ( b n ) if there exists a constant 0 < C < + ∞ such that a n ≤ C b n .

Original: We use notation (cid:101) O ( · ) to hide logarithmic factors.
Revised: We use notation (cid:101) O ( · ) to hide logarithmic factors.

Original: We write a n = o ( b n ) if a n /b n → 0 as n → ∞ .
Revised: We write a n = o ( b n ) if a n /b n → 0 as n → ∞ .

Original:  R EVIEW OF A DAPTIVE G RADIENT M ETHODS
Revised:  R EVIEW OF A DAPTIVE G RADIENT M ETHODS

Original: Various adaptive gradient methods have been proposed in order to achieve better performance on various stochastic optimization tasks.
Revised: Various adaptive gradient methods have been proposed in order to achieve better performance on various stochastic optimization tasks.

Original: Adagrad (Duchi et al., 2011) is among the ﬁrst methods with adaptive learning rate for each individual dimension, which motivates the research on adaptive gradient methods in the machine learning community.
Revised: Adagrad (Duchi et al., 2011) is among the ﬁrst methods with adaptive learning rate for each individual dimension, which motivates the research on adaptive gradient methods in the machine learning community.

Original: In detail, Adagrad 1 adopts the followingupdate form:
Revised: In detail, Adagrad 1 adopts the following

Original: [Equation]
Revised: [Equation]

Original: Here g t stands for  stochastic  t ( θ t ) , and α t = α/ √ t is the step size (a.k.a., learning rate).
Revised: Here g t stands for the stochastic gradient ∇ f t ( θ t ) , and α t = α/ √ t is the step size (a.k.a., learning rate).

Original: Adagrad is proved to enjoy a huge gain in terms of convergence especially in sparse gradient situations.
Revised: Adagrad is proved to enjoy a huge gain in terms of convergence especially in sparse gradient situations.

Original: Empirical studies also show a performance gain even for non-sparse gradient settings.
Revised: Empirical studies also show a performance gain even for non-sparse gradient settings.

Original: RMSprop (Hinton et al., 2012) follows the idea of adaptive learning rate and it changes the arithmetic averages used for v t in Adagrad to exponential moving averages.
Revised: RMSprop (Hinton et al., 2012) follows the idea of adaptive learning rate and it changes the arithmetic averages used for v t in Adagrad to exponential moving averages.

Original: Even though RMSprop is an empirical method with no theoretical guarantee, the outstanding empirical performance of RMSprop raised people’s interests in exponential moving average variants of Adagrad.
Revised: Even though RMSprop is an empirical method with no theoretical guarantee, the outstanding empirical performance of RMSprop raised people’s interests in exponential moving average variants of Adagrad.

Original: Adam (Kingma & Ba, 2015) 2 is the most popular exponential moving average variant of Adagrad.
Revised: Adam (Kingma & Ba, 2015) 2 is the most popular exponential moving average variant of Adagrad.

Original: It combines the idea of
Revised: It combines the idea of

Original: RMSprop and momentum acceleration, and takes the following update form:
Revised: RMSprop and momentum acceleration, and takes the following update form:

Original: [Equation]
Revised: [Equation]

Revised: , where

Revised: [Equation]

Original: Adam also requires α t = α/ √ t for the sake of convergence analysis.
Revised: Adam also requires α t = α/ √ t for the sake of convergence analysis.

Original: In practice, any decaying step size or even constant step size works well for Adam.
Revised: In practice, any decaying step size or even constant step size works well for Adam.

Original: Note that if we choose β 1 = 0 , Adam basically reduces to RMSprop.
Revised: Note that if we choose β 1 = 0 , Adam basically reduces to RMSprop.

Original: Reddi et al.
Revised: Reddi et al.

Original: identiﬁed a non-convergence issue in Adam, and  proposed a modiﬁed algorithm namely Amsgrad.
Revised: identiﬁed a non-convergence issue in Adam. Speciﬁcally, Adam does not collect long-term memory of past gradients and therefore the effective learning rate could be increasing in some cases. They proposed a modiﬁed algorithm namely Amsgrad.

Original: More speciﬁcally, Amsgrad adopts an additional step to ensure the decay of the effective learning rate α t / √ (cid:98) v t , and its key update formula is as follows:
Revised: More speciﬁcally, Amsgrad adopts an additional step to ensure the decay of the effective learning rate α t / √ (cid:98) v t , and its key update formula is as follows:

Original: [Equation]
Revised: [Equation]

Original: The deﬁnitions of m t and v t are the same as Adam.
Revised: The deﬁnitions of m t and v t are the same as Adam.

Original: Note that the modiﬁed (cid:98) v t ensures the convergence of Amsgrad.
Revised: Note that the modiﬁed (cid:98) v t ensures the convergence of Amsgrad.

Original: Reddi et al.
Revised: Reddi et al.

Original: (2018) also corrected some mistakes in the original proof of Adam and proved an O (1 / √ T ) convergence rate of Amsgrad for convex optimization.
Revised: (2018) also corrected some mistakes in the original proof of Adam and proved an O (1 / √ T ) convergence rate of Amsgrad for convex optimization.

Original: Note that all the theoretical guarantees on adaptive gradient methods (Adagrad, Adam, Amsgrad) are only proved for convex functions.
Revised: Note that all the theoretical guarantees on adaptive gradient methods (Adagrad, Adam, Amsgrad) are only proved for convex functions.

Original:  T HE P ROPOSED A LGORITHM
Revised:  T HE P ROPOSED A LGORITHM

Original: In this section, we propose a new algorithm that not only inherits the O (1 / √ T ) convergence rate from Adam/Amsgrad, but also has comparable or even better generalization performance than SGD with momentum.
Revised: In this section, we propose a new algorithm that not only inherits the O (1 / √ T ) convergence rate from Adam/Amsgrad, but also has comparable or even better generalization performance than SGD with momentum.

Original: Speciﬁcally, we introduce the a partial adaptive parameter p to control the level of adaptivity of the optimization procedure.
Revised: Speciﬁcally, we introduce the a partial adaptive parameter p to control the level of adaptivity of the optimization procedure.

Original: The proposed algorithm is displayed in in Algorithm 1.
Revised: The proposed algorithm is displayed in in Algorithm 1.

Original:  input: initial point θ 1 ∈ X ; step sizes { α t } ; momentum parameters { β 1 t } , β 2 ; partially adaptive parameter p ∈ (0 , 1 / 2] set m [-
-] = 0 , v 0 = 0 , (cid:98) v 0 =for t = 1 , . .
Revised: Algorithm input: initial point θ 1 ∈ X ; step sizes { α t } ; momentum parameters { β 1 t } , β 2 ; partially adaptive parameter p ∈ (0 , 1 / 2] set m [-
-]0 = 0 , v 0 = 0 , (cid:98) v 0 =for t = 1 , . .

Original: , T do g t = ∇ f t ( θ t ) m t =
Revised: , T do g t = ∇ f t ( θ t ) m t =

Original: [Equation]
Revised: [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Revised: g

Original: [Equation]
Revised: [Equation]

Original: t v t = 
Revised: v t = β

Original: [Equation]
Revised: [Equation]

Original: + [Equation] g
Revised: v t

Original: [Equation]
Revised: [Equation]

Original:  t
Revised: + (1 − β 2 ) g 2 t (cid:98) v t = max( (cid:98) v t

Original: [Equation]
Revised: [Equation]

Original:  ) θ 
Revised: 1 , v t ) θ t +1 = Π X , diag(

Original: [Equation]
Revised: [Equation]

Original: =
Revised: pt

Original: [Equation]
Revised: [Equation]

Original: end for
Revised: end for

Original: In Algorithm 1, g t denotes the stochastic gradient and (cid:98) v t can be seen as a moving average over the second order momentum of the stochastic gradients.
Revised: In Algorithm 1, g t denotes the stochastic gradient and (cid:98) v t can be seen as a moving average over the second order momentum of the stochastic gradients.

Original: As we can see from Algorithm 1, the key difference between Padam and Amsgrad (Reddi et al., 2018) is that: while m t is still the ﬁrst order momentum as in Adam/Amsgrad, it is now “partially adapted” by the second order momentum, i.e., θ t +

Revised: As we can see from Algorithm 1, the key difference between Padam and Amsgrad (Reddi et al., 2018) is that: while m t is still the ﬁrst order momentum as in Adam/Amsgrad, it is now “partially adapted” by the second order momentum, i.e., θ t +

Original: =
Revised: =

Original: θ t − α t m t (cid:98) v p t , where (cid:98) [Equation]
Revised: θ t − α t m t (cid:98) v pt , where (cid:98) v t = max( (cid:98) v t − 1 , v t ) (Padam)

Original: We call p ∈ (0 , 1 / 2] the partially adaptive parameter.
Revised: We call p ∈ (0 , 1 / 2] the partially adaptive parameter.

Original: Note that 1 / 2 is the largest possible value for p and a larger p will result in non-convergence in the proof (see the details of the proof in the appendix).
Revised: Note that 1 / 2 is the largest possible value for p and a larger p will result in non-convergence in the proof (see the details of the proof in the appendix).

Original: When p → 0 , Algorithm 1 reduces to SGD with momentum 3 and when p = 1 / 2 , Algorithm 1 is exactly Amsgrad.
Revised: When p → 0 , Algorithm 1 reduces to SGD with momentum 3 and when p = 1 / 2 , Algorithm 1 is exactly Amsgrad.

Original: Therefore, Padam indeed uniﬁes Amsgrad and SGD with momentum.
Revised: Therefore, Padam indeed uniﬁes Amsgrad and SGD with momentum.

Original: Now the question is, what value for p should we choose?
Revised: Now the question is, what value for p should we choose?

Original: Or in another way, is p = 1 / 2 the best choice?
Revised: Or in another way, is p = 1 / 2 the best choice?

Original: The answer is negative.
Revised: The answer is negative.

Original: The intuition behind this is simple: it is very likely that
Revised: The intuition behind this is simple: it is very likely that

Original: Adam/Amsgrad is “over-adaptive”.
Revised: Adam/Amsgrad is “over-adaptive”.

Original: One notable fact that people found when using adaptive gradient methods to train deep neural networks is that the learning rate needs to be much smaller than that of
Revised: One notable fact that people found when using adaptive gradient methods to train deep neural networks is that the learning rate needs to be much smaller than that of

Original: SGD-momentum (Keskar & Socher, 2017; Wilson et al., 2017).
Revised: SGD-momentum (Keskar & Socher, 2017; Wilson et al., 2017).

Original: For many tasks, the base learning rate for SGD-momentum is usually set to be 0 .
Revised: For many tasks, the base learning rate for SGD-momentum is usually set to be 0 .

Original: 1 while that of Adam is usually set to be 0 .
Revised: 1 while that of Adam is usually set to be 0 .

Original: In fact, the key reason that prohibits Adam from adopting a more aggressive learning rate is the large adaptive term 1 / √ (cid:98) v t .
Revised: In fact, the key reason that prohibits Adam from adopting a more aggressive learning rate is the large adaptive term 1 / √ (cid:98) v t .

Original: The existence of such a large adaptive term makes the effective learning rate ( α t / √ (cid:98) v t ) easily explode with a larger α t .
Revised: The existence of such a large adaptive term makes the effective learning rate ( α t / √ (cid:98) v t ) easily explode with a larger α t .

Original: Moreover, the learning rate decaying strategy used in modern deep neural network training makes things worse.
Revised: Moreover, the learning rate decaying strategy used in modern deep neural network training makes things worse.

Original: More speciﬁcally, after several rounds of decaying, the learning rates of the adaptive gradient methods are too small to make any signiﬁcant progress in the training process.
Revised: More speciﬁcally, after several rounds of decaying, the learning rates of the adaptive gradient methods are too small to make any signiﬁcant progress in the training process.

Original: We call it “small learning rate dilemma”.
Revised: We call it “small learning rate dilemma”.

Original: This explains the relatively weak performances of adaptive gradient methods at the later stage of the training process, where the non-adaptive gradient methods like SGD start to outperform them.
Revised: This explains the relatively weak performances of adaptive gradient methods at the later stage of the training process, where the non-adaptive gradient methods like SGD start to outperform them.

Original: The above discussion suggests that we should consider using Padam with a proper adaptive parameter p , which will enable us to adopt a larger learning rate to avoid the “small learning rate dilemma”.
Revised: The above discussion suggests that we should consider using Padam with a proper adaptive parameter p , which will enable us to adopt a larger learning rate to avoid the “small learning rate dilemma”.

Original: And we will show in our experiments (Section 5) that Padam can adopt an equally large base learning rate as SGD. 
Revised: And we will show in our experiments (Section 5) that Padam can adopt an equally large base learning rate as SGD. [Figure]

Revised: Figure 1 shows the comparison of test error performances under the different partial adaptive parameter p for ResNet on both CIFAR-10 and CIFAR-100 datasets.

Revised: We can observe that a larger p will lead to fast convergence at early stages and worse generalization performance later, while a smaller p behaves more like SGD with momentum: slow in early stages but ﬁnally catch up.

Revised: While a proper choice of p (e.g., 1 / 8 in this case), could obtain the best of both worlds.

Original: Note that besides Algorithm 1, our partially adaptive idea can also be applied to other adaptive gradient methods such as Adagrad, Adadelta, RMSprop, AdaMax (Kingma & Ba, 2015).
Revised: Note that besides Algorithm 1, our partially adaptive idea can also be applied to other adaptive gradient methods such as Adagrad, Adadelta, RMSprop, AdaMax (Kingma & Ba, 2015).

Original: For the sake of conciseness, we do not list the partially adaptive versions for other adaptive gradient methods here.
Revised: For the sake of conciseness, we do not list the partially adaptive versions for other adaptive gradient methods here.

Original: We also would like to comment that Padam is totally different from the p -norm generalized version of Adam in Kingma & Ba (2015), which induces AdaMax method when p → ∞ .
Revised: We also would like to comment that Padam is totally different from the p -norm generalized version of Adam in Kingma & Ba (2015), which induces AdaMax method when p → ∞ .

Original: In their case, p -norm is used to generalize 2 -norm of their current and past gradients while keeping the scale of adaptation unchanged.
Revised: In their case, p -norm is used to generalize 2 -norm of their current and past gradients while keeping the scale of adaptation unchanged.

Original: In sharp contrast, we intentionally change (reduce) the scale of the adaptive term in Padam to get better generalization performance.
Revised: In sharp contrast, we intentionally change (reduce) the scale of the adaptive term in Padam to get better generalization performance.

Original: Finally, note that in Algorithm 1 we remove the bias correction step used in the original Adam paper following Reddi et al.
Revised: Finally, note that in Algorithm 1 we remove the bias correction step used in the original Adam paper following Reddi et al.

Original: Nevertheless, our arguments and theory are applicable to the bias correction version as well.
Revised: Nevertheless, our arguments and theory are applicable to the bias correction version as well.

Original:  C ONVERGENCE A NALYSIS OF THE P ROPOSED A LGORITHM
Revised:  C ONVERGENCE A NALYSIS OF THE P ROPOSED A LGORITHM

Original: In this section, we establish the theory of convergence for our proposed algorithm in the online optimization setting (Zinkevich, 2003), where we try to minimize the cumulative objective value of a sequence of loss functions: f 1 , f 2 , . . .
Revised: In this section, we establish the theory of convergence for our proposed algorithm in the online optimization setting (Zinkevich, 2003), where we try to minimize the cumulative objective value of a sequence of loss functions: f 1 , f 2 , . . .

Original: , f T .
Revised: , f T .

Original: In particular, at each time step t , the optimization algorithm picks a point θ t ∈ X , where
Revised: In particular, at each time step t , the optimization algorithm picks a point θ t ∈ X , where

Original: X ∈ R
Revised: X ∈ R

Original: d is the feasible set.
Revised: d is the feasible set.

Original: A loss function f t is then revealed, and the algorithm incurs loss f t ( θ t ) .
Revised: A loss function f t is then revealed, and the algorithm incurs loss f t ( θ t ) .

Original: Let θ ∗ be optimal solution to the cumulative objective function as follows
Revised: Let θ ∗ be optimal solution to the cumulative objective function as follows

Original: [Equation]
Revised: [Equation]

Original: where X is a feasible set for all steps.
Revised: where X is a feasible set for all steps.

Original: We evaluate our algorithm using the regret, which characterizes the sum of all previous loss function values f t ( θ t ) relative to the performance of the best ﬁxed parameter θ ∗ from a feasible set.
Revised: We evaluate our algorithm using the regret, which characterizes the sum of all previous loss function values f t ( θ t ) relative to the performance of the best ﬁxed parameter θ ∗ from a feasible set.

Original: Speciﬁcally, the regret is deﬁned as
Revised: Speciﬁcally, the regret is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: and our goal is to predict the unknown parameter θ t and minimize the overall regret R T .
Revised: and our goal is to predict the unknown parameter θ t and minimize the overall regret R T .

Original: Our theory is established for convex loss functions, where the following assumption holds.
Revised: Our theory is established for convex loss functions, where the following assumption holds.

Original: with momentum is an extra constant factor (1 − β 1 ) , which can be moved into the learning rate such that the update rules for these two algorithms are identical.

Original: Assumption 4.1 (Convex function) .
Revised: Assumption 4.1 (Convex function) .

Original: All f t ( θ ) are convex functions on X for 1 ≤ t ≤ T , i.e., for all x , y ∈ X ,
Revised: All f t ( θ ) are convex functions on X for 1 ≤ t ≤ T , i.e., for all x , y ∈ X ,

Original: [Equation]
Revised: [Equation]

Original: Assumption 4.1 is a standard assumption in online learning and the same assumption has been used in the analysis of Adagrad (Duchi et al., 2011), Adam (Kingma & Ba, 2015) and Amsgrad (Reddi et al., 2018).
Revised: Assumption 4.1 is a standard assumption in online learning and the same assumption has been used in the analysis of Adagrad (Duchi et al., 2011), Adam (Kingma & Ba, 2015) and Amsgrad (Reddi et al., 2018).

Original: Next we provide the main convergence rate result for our proposed algorithm.
Revised: Next we provide the main convergence rate result for our proposed algorithm.

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Under Assumption 4.1, if the convex feasible set X has bounded diameters, i.e., (cid:107) θ − θ ∗ (cid:107) ∞ ≤ D ∞
Revised: Under Assumption 4.1, if the convex feasible set X has bounded diameters, i.e., (cid:107) θ − θ ∗ (cid:107) ∞ ≤ D ∞

Original: for all θ ∈ X , and f t has bounded gradients, i.e., (cid:107)∇ f t ( θ ) (cid:107) ∞ ≤ G ∞ for all θ ∈ X , 1 ≤ t ≤ T , let α t =
Revised: for all θ ∈ X , and f t has bounded gradients, i.e., (cid:107)∇ f t ( θ ) (cid:107) ∞ ≤ G ∞ for all θ ∈ X , 1 ≤ t ≤ T , let α t =

Original: α/ √ t , β 1 t = β 1 λ t − 1 , λ ∈ (0 , 1) , 0 ≤ β 1 , β 2 < 1 , p ∈ (0 , 0 . 5] , the regret of Algorithm 1 satisﬁes:
Revised: α/ √ t , β 1 t = β 1 λ t − 1 , λ ∈ (0 , 1) , 0 ≤ β 1 , β 2 < 1 , p ∈ (0 , 0 . 5] , the regret of Algorithm 1 satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: where γ = β 1 / √ β 2 < 1 .
Revised: where γ = β 1 / √ β 2 < 1 .

Original: Remark 4.3.
Revised: Remark 4.3.

Original: Theorem 4.2 suggests that, similar to Adam (Kingma & Ba, 2015) and Amsgrad (Reddi et al., 2018), the regret of Padam can be considerably better than online gradient descent (which is known to have a regret bound of O ( √ dT ) )
Revised: Theorem 4.2 suggests that, similar to Adam (Kingma & Ba, 2015) and Amsgrad (Reddi et al., 2018), the regret of Padam can be considerably better than online gradient descent (which is known to have a regret bound of O ( √ dT ) )

Original: when (cid:80) di
Revised: when (cid:80) di

Original: =1 (cid:107) g 1: T,i (cid:107) 

Revised: =1 (cid:107) g 1: T,i (cid:107)

Original: (cid:28) √ dT (Duchi et al., 2011) and (cid:80) di =1 (cid:98) v pT,i (cid:28) √ d .
Revised: (cid:28) √ dT (Duchi et al., 2011) and (cid:80) di =1 (cid:98) v pT,i (cid:28) √ d .

Original: Also, from the proof of Theorem 4.2 in Appendix A, we can see that the regret bound can remain in the same order even with a more modest momentum decay β 1 t = β 1 /t .
Revised: Also, from the proof of Theorem 4.2 in Appendix A, we can see that the regret bound can remain in the same order even with a more modest momentum decay β 1 t = β 1 /t .

Original: The following corollary demonstrates that our proposed algorithm enjoys a regret bound of (cid:101) O ( √ T ) , which is comparable to the best known bound for general convex online learning problems.
Revised: The following corollary demonstrates that our proposed algorithm enjoys a regret bound of (cid:101) O ( √ T ) , which is comparable to the best known bound for general convex online learning problems.

Original: Corollary 4.4.
Revised: Corollary 4.4.

Original: Under the same conditions of Theorem 4.2, for all T ≥ 1 , the regret of Algorithmsatisﬁes R T = (cid:101) O [Equation] (cid:1) .
Revised: Under the same conditions of Theorem 4.2, for all T ≥ 1 , the regret of Algorithmsatisﬁes R T = (cid:101) O (cid:0) √ T (cid:1) .

Original: Corollary 4.4 suggests that Padam attains R T
Revised: Corollary 4.4 suggests that Padam attains R T

Original: = o ( T ) for all situations (no matter whether the data features are sparse or not).
Revised: = o ( T ) for all situations (no matter whether the data features are sparse or not).

Original: This suggests that Algorithm 1 indeed converges to the optimal solution when the loss functions are convex, as shown by the fact that lim T →∞ R T /T
Revised: This suggests that Algorithm 1 indeed converges to the optimal solution when the loss functions are convex, as shown by the fact that lim T →∞ R T /T

Original: → 0 .
Revised: → 0 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we empirically evaluate our proposed algorithm for training various modern deep learning models and test them on several standard benchmarks 4 .
Revised: In this section, we empirically evaluate our proposed algorithm for training various modern deep learning models and test them on several standard benchmarks 4 .

Original: We show that for nonconvex loss functions in deep learning, our proposed algorithm still enjoys a fast convergence rate, while its generalization performance is as good as SGD with momentum and much better than existing adaptive gradient methods such as Adam and Amsgrad.
Revised: We show that for nonconvex loss functions in deep learning, our proposed algorithm still enjoys a fast convergence rate, while its generalization performance is as good as SGD with momentum and much better than existing adaptive gradient methods such as Adam and Amsgrad.

Original: 5. E NVIRONMENTAL S ETUP
Revised: 5. E NVIRONMENTAL S ETUP

Original: All experiments are conducted on Amazon AWS p3.8xlarge servers which come with Intel Xeon E
Revised: All experiments are conducted on Amazon AWS p3.8xlarge servers which come with Intel Xeon E

Original: CPU and 4 NVIDIA Tesla V100 GPUs (16G RAM per GPU).
Revised: CPU and 4 NVIDIA Tesla V100 GPUs (16G RAM per GPU).

Original: All experiments are implemented in
Revised: All experiments are implemented in

Original: Pytorch platform version 0 .
Revised: Pytorch platform version 0 .

Original: 4 . 1 within Python 3 . 6 . 4 .
Revised: 4 . 1 within Python 3 . 6 . 4 .

Original: 5. B ASELINE A LGORITHMS
Revised: 5. B ASELINE A LGORITHMS

Original: We compare our proposed algorithm against several state-of-the-art algorithms, including: (1) SGD with momentum (2) Adam (Kingma & Ba, 2015)
Revised: We compare our proposed algorithm against several state-of-the-art algorithms, including: (1) SGD with momentum (2) Adam (Kingma & Ba, 2015)

Original: (3) Amsgrad (Reddi et al., 2018). [Table]
Revised: (3) Amsgrad (Reddi et al., 2018) (4) AdamW (Loshchilov & Hutter, 2017).

Original: Note that we do not perform the projection step explicitly in all experiments as in Reddi et al.
Revised: Note that we do not perform the projection step explicitly in all experiments as in Reddi et al.

Original: Also both Amsgrad and Padam will perform the bias correction step as in Adam for a fair comparison.
Revised: Also both Amsgrad and Padam will perform the bias correction step as in Adam for a fair comparison.

Original: 5. D ATASETS / P ARAMETER S ETTINGS / CNN A RCHITECTURES
Revised: 5. D ATASETS / P ARAMETER S ETTINGS / CNN A RCHITECTURES

Original: We use several popular datasets for image classiﬁcations: CIFAR-10 (Krizhevsky & Hinton, 2009),
Revised: We use several popular datasets for image classiﬁcations: CIFAR-10 (Krizhevsky & Hinton, 2009),

Original: CIFAR-100 (Krizhevsky & Hinton, 2009), ImageNet dataset (ILSVRC2012)
Revised: CIFAR-100 (Krizhevsky & Hinton, 2009), ImageNet dataset (ILSVRC2012)

Original: (Deng et al., 2009).
Revised: (Deng et al., 2009).

Original: We adopt three popular CNN architectures to evaluate the performance of our proposed algorithms:
Revised: We adopt three popular CNN architectures to evaluate the performance of our proposed algorithms:

Original: VGGNet (Simonyan & Zisserman, 2014), Residual Neural Network (ResNet) (He et al., 2016),
Revised: VGGNet (Simonyan & Zisserman, 2014), Residual Neural Network (ResNet) (He et al., 2016),

Original: Wide Residual Network(Zagoruyko & Komodakis, 2016).
Revised: Wide Residual Network(Zagoruyko & Komodakis, 2016).

Original: We test all tasks for 100 epochs.
Revised: We test CIFAR-10 and CIFAR-100 tasks for 200 epochs.

Original: For all experiments, we use a ﬁxed multi-stage learning rate decaying scheme: we decay the learning rate by 0 . 
Revised: For all experiments, we use a ﬁxed multi-stage learning rate decaying scheme: we decay the learning rate by 0 . 1 at the 50 th, 100 th and 150 th epochs.

Original: 1 at the 30 th, 60 th and 80 th epochs.
Revised: We test ImageNet tasks forepochs with the same multi-stage learning rate decaying scheme at the 30 th, 60 th and 80 th epochs.

Original: We perform grid search on validation set to choose the best base learning rate for all algorithms, the partially adaptive parameter p and also the second order momentum parameter β 2 for all the adaptive gradient methods. 
Revised: We perform grid search on validation set to choose the best base learning rate for all algorithms and also the second order momentum parameter β 2 for all the adaptive gradient methods. In terms of the choice of p , we recommend doing binary search from 1 / 4 to 1 / 8 , to 1 / 16 , etc.

Revised: Yet in most cases we tested, 1 / 8 is a stable and reliable choice of p .

Original: Details about the datasets, CNN architectures and all the speciﬁc model parameters used in the following experiments can be found in the supplementary materials.[Figure]
Revised: Details about the datasets, CNN architectures and all the speciﬁc model parameters used in the following experiments can be found in the supplementary materials.

Original: 5. E XPERIMENTAL R ESULTS
Revised: 5. E XPERIMENTAL R ESULTS

Original: We compare our proposed algorithm with other baselines on training the aforementioned three modern CNN architectures for image classiﬁcation on the CIFAR-10, CIFAR-100 and also ImageNet datasets.
Revised: We compare our proposed algorithm with other baselines on training the aforementioned three modern CNN architectures for image classiﬁcation on the CIFAR-10, CIFAR-100 and also ImageNet [Figure]

Original:  Due to the paper length limit, we leave all our experimental results regarding CIFARdataset in the supplementary materials.
Revised: datasets. Due to the paper length limit, we leave all our experimental results regarding CIFAR100 dataset in the supplementary materials.

Original: Figures 1 plots the train loss and test error (top- 1 error) against training epochs on the CIFAR-10 dataset.
Revised: Figure 2 plots the train loss and test error (top- 1 error) against training epochs on the CIFAR-10 dataset.

Original: As we can see from the ﬁgure, at the early stage

Original: [Figure] of the training process, (partially) adaptive gradient methods including Padam, make rapid progress lowing both the train loss and the test error [Figure] while SGD with momentum converges relatively slowly.
Revised: As we can see from the ﬁgure, at the early stage of the training process, (partially) adaptive gradient methods including Padam, make rapid progress lowing both the train loss and the test error, while SGD with momentum converges relatively slowly.

Original: After the ﬁrst learning rate decaying at the 30 -th epoch, different algorithms start to behave differently.
Revised: After the ﬁrst learning rate decaying at the 50 -th epoch, different algorithms start to behave differently.

Original: SGD with momentum makes a huge drop while fully adaptive gradient methods (Adam and
Revised: SGD with momentum makes a huge drop while fully adaptive gradient methods (Adam and

Original: Amsgrad) start to generalize badly.
Revised: Amsgrad, AdamW) start to generalize badly.

Original: Padam, on the other hand, maintains relatively good generalization performance and also holds the lead over other algorithms.
Revised: Padam, on the other hand, maintains relatively good generalization performance and also holds the lead over other algorithms.

Original: After the second decaying at the 60 -th epoch, Adam and Amsgrad basically lose all the power of traversing through the parameter space due to the “small learning dilemma”, while the performance of SGD with momentum ﬁnally catches up with Padam.
Revised: After the second decaying at the 100 -th epoch, Adam and Amsgrad basically lose all the power of traversing through the parameter space due to the “small learning dilemma”, while the performance of SGD with momentum ﬁnally catches up with Padam.

Revised: AdamW improves the performance comparing with original Adam but there are still generalization gaps left behind, at least in our test settings.

Original: Overall we can see that Padam indeed achieves the best from both worlds(i.e., Adam and SGD with momentum): it maintains faster convergence rate while also generalizing as well as SGD with momentum in the end.
Revised: Overall we can see that Padam indeed achieves the best from both worlds (i.e., Adam and SGD with momentum): it maintains faster convergence rate while also generalizing as well as SGD with momentum in the end.

Original: Table 1 shows the test accuracy of VGGNet on CIFAR-10 dataset, from which we can also observe the fast convergence of Padam.
Revised: Table 1 shows the test accuracy of VGGNet on CIFAR-10 dataset, from which we can also observe the fast convergence of Padam.

Original: Speciﬁcally, Padam achieves the best test accuracy at 50 th, 75 th and 100 th epochs for CIFAR-10, except for the 25 th, where fully adaptive methods like Adam and Amsgrad  still enjoy their fast early stage convergence performances.
Revised: Speciﬁcally, Padam achieves the best test accuracy at 100 th, 150 th and 200 th epochs for CIFAR-10, except for the 50 th, where fully adaptive methods like Adam, Amsgrad and AdamW still enjoy their fast early stage convergence performances.

Original: This suggests that practitioners should once again, use fast to converge partially adaptive gradient methods for training deep neural networks, without worrying about the generalization performances.
Revised: This suggests that practitioners should once again, use fast to converge partially adaptive gradient methods for training deep neural networks, without worrying about the generalization performances.

Original: [Figure]
Revised: [Table]

Original: Figure 2 plots the Top- 1 and Top- 5 error against training epochs on the ImageNet dataset for both
Revised: Figure 3 plots the Top- 1 and Top- 5 error against training epochs on the ImageNet dataset for both

Original: VGGNet and ResNet 5 .
Revised: VGGNet and ResNet 5 .

Original: We can see that on ImageNet, all methods behave similarly as in our CIFAR10 experiments.
Revised: We can see that on ImageNet, all methods behave similarly as in our CIFAR10 experiments.

Original: Padam method again obtains the best from both worlds by achieving the fastest convergence while generalizing as well as SGD with momentum.
Revised: Padam method again obtains the best from both worlds by achieving the fastest convergence while generalizing as well as SGD with momentum.

Original: Even though Amsgrad has greatly improve the performance comparing with  Adam, it still suffers from the generalization gap on ImageNet.
Revised: Even though Amsgrad and AdamW have greatly improve the performance comparing with standard Adam, it still suffers from the generalization gap on ImageNet.

Original: We also test the inﬂuence of different partial adaptive parameter p on the performance of our proposed method.

Original: Figure 3(a) shows the comparison of test error performances under the different partial adaptive parameter p for ResNet on CIFAR-10 dataset.

Original: We can observe that a larger p will lead to fast convergence at early stages and worse generalization performance later, while a smaller p behaves more like SGD with momentum: slow in early stages but ﬁnally catch up.

Original: While a proper choice of p (e.g., 1 / 8 ), could obtain the best of both worlds.

Original: As a side product of our experiments, we also examine the distance that θ t has traversed in the parameter space, i.e., (cid:107) θ t − θ 0 (cid:107) 2 , as shown in many other studies (Hoffer et al., 2017; Xing et al., 2018) discussing the generalization performance of various optimization algorithms for deep learning.

Original: Figure 3(b) shows the plot of the Euclidean distance of weight vector θ t from initialization θ 0 , against training epochs for ResNet on CIFARdataset.

Original: First, we would like to emphasize that these are the plots only regrading the Euclidean distance, which does not contain any directional information.

Original: In other words, the cross in the plot does not mean that the weight vectors in different optimization algorithms actually meet somewhere during the training process. As we can see from the plots, SGD with momentum tends to overshoot a lot at the early stage of the entire training process. This could explain why the convergence of SGD with momentum is slower at early stages. And for Adam/Amsgrad, despite the quick start and less overshooting, the “small learning rate dilemma” (see Section 3) largely limits the weight vector’s ability of exploring the parameter space in the middle and later stages, which could explain the bad generalization performance of Adam/Amsgrad.
Revised: [Figure]

Original: On the other hand, Padam behaves inbetween SGD with momentum and Adam/Amsgrad, with less severe overshooting compared with SGD with momentum, while maintaining a better capability of traversing through the parameter space compared with Adam/Amsgrad.

Original: This partly justiﬁes the outstanding generalization performance of Padam.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We brieﬂy review the related work in this section.
Revised: We brieﬂy review the related work in this section.

Original: There are only several studies closely related to improving the generalization performance of Adam.
Revised: There are only several studies closely related to improving the generalization performance of Adam.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2017) proposed a normalized direction-preserving Adam (ND-Adam), which changes the adaptive terms from individual dimensions to the whole gradient vector.
Revised: (2017) proposed a normalized direction-preserving Adam (ND-Adam), which changes the adaptive terms from individual dimensions to the whole gradient vector.

Original: This makes it more like a SGD-momentum with learning rate adjusted in each iteration, rather than an adaptive gradient algorithm.
Revised: This makes it more like a SGD-momentum with learning rate adjusted in each iteration, rather than an adaptive gradient algorithm.

Original: The empirical result also shows that its performance resembles SGD with momentum.
Revised: The empirical result also shows that its performance resembles SGD with momentum.

Original: Keskar & Socher (2017) proposed to improve the generalization performance by switching from Adam to SGD.
Revised: Keskar & Socher (2017) proposed to improve the generalization performance by switching from Adam to SGD.

Original: Yet the empirical result shows that it actually sacriﬁces some of the convergence speed for better generalization rather than achieving the best from both worlds.
Revised: Yet the empirical result shows that it actually sacriﬁces some of the convergence speed for better generalization rather than achieving the best from both worlds.

Original: Also deciding the switching learning rate and the best switching point requires extra efforts on parameter tuning since they can be drastically different for different tasks according to the paper.
Revised: Also deciding the switching learning rate and the best switching point requires extra efforts on parameter tuning since they can be drastically different for different tasks according to the paper.

Original: Loshchilov & Hutter (2017) proposed to ﬁx the weight decay regularization in Adam by decoupling the weight decay from the gradient update and this improves the generalization performance of Adam.
Revised: Loshchilov & Hutter (2017) proposed to ﬁx the weight decay regularization in Adam by decoupling the weight decay from the gradient update and this improves the generalization performance of Adam.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: In this paper, we proposed Padam, which uniﬁes Adam/Amsgrad with SGD-momentum.
Revised: In this paper, we proposed Padam, which uniﬁes Adam/Amsgrad with SGD-momentum.

Original: With an appropriate choice of the partially adaptive parameter, we show that Padam can achieve the best from both worlds, i.e., maintaining fast convergence rate while closing the generalization gap.
Revised: With an appropriate choice of the partially adaptive parameter, we show that Padam can achieve the best from both worlds, i.e., maintaining fast convergence rate while closing the generalization gap.

Original: We also provide a theoretical analysis towards the convergence rate of Padam and show a similar datadependent regret bound as in Duchi et al.
Revised: We also provide a theoretical analysis towards the convergence rate of Padam and show a similar datadependent regret bound as in Duchi et al.

Original: Reddi et al.
Revised: Reddi et al.

Original: It would be interesting to see how well Padam performs in other types of nerual networks, such as recurrent neural networks (RNNs) (Hochreiter & Schmidhuber, 1997) and generative adversarial networks (GANs) (Goodfellow et al., 2014).
Revised: It would be interesting to see how well Padam performs in other types of nerual networks, such as recurrent neural networks (RNNs) (Hochreiter & Schmidhuber, 1997) and generative adversarial networks (GANs) (Goodfellow et al., 2014).

Original: We leave it as a future work.
Revised: We leave it as a future work.

Original: A P ROOF OF THE M AIN R ESULTS A. P ROOF OF T HEOREM 4. 
Revised: A P ROOF OF THE M AIN R ESULTS A. P ROOF OF T HEOREM 4.

Original: The proof of Theorem 4.2 needs the following lemmas.
Revised: The proof of Theorem 4.2 needs the following lemmas.

Original: Lemma A.1.
Revised: Lemma A.1.

Original: Under the same conditions as in Theorem 4.2, we have
Revised: Under the same conditions as in Theorem 4.2, we have

Original: [Equation]
Revised: [Equation]

Original: Lemma A.1 basically describes the bound of the key quantity ( (cid:80)
Revised: Lemma A.1 basically describes the bound of the key quantity ( (cid:80)

Original: Tt =1 (cid:80) di
Revised: Tt =1 (cid:80) di

Original: =1 α t · m 2 t,i / (cid:98) v pt,i ) in the convergence analysis of Algorithm 1.
Revised: =1 α t · m 2 t,i / (cid:98) v pt,i ) in the convergence analysis of Algorithm 1.

Original: Its proof is inspired by Lemma 2 in Reddi et al.
Revised: Its proof is inspired by Lemma 2 in Reddi et al.

Original: Note that the original proof in Kingma & Ba (2015) for the corresponding quantity contains some mistakes, which result in a bound without the logarithmic term in (A.1).
Revised: Note that the original proof in Kingma & Ba (2015) for the corresponding quantity contains some mistakes, which result in a bound without the logarithmic term in (A.1).

Original: Here we follow the corrected version in Reddi et al.
Revised: Here we follow the corrected version in Reddi et al.

Original: Lemma A.2.
Revised: Lemma A.2.

Original: For any V ∈ S d ++ and convex feasible set X ⊂ R d , suppose a 1 = Π X , V p ( b 1 ) , a 2 = Π X , V p ( b 2 ) for p ∈ (0 , 1 / 2] , we have
Revised: For any V ∈ S d ++ and convex feasible set X ⊂ R d , suppose a 1 = Π X , V p ( b 1 ) , a 2 = Π X , V p ( b 2 ) for p ∈ (0 , 1 / 2] , we have

Original: [Equation]
Revised: [Equation]

Original: where p ∈ (0 , 1 / 2] is an absolute constant.
Revised: where p ∈ (0 , 1 / 2] is an absolute constant.

Original: Lemma A.2 is an adaptation of Lemma 5 in McMahan & Streeter (2010) (or Lemma 4 in Reddi et al.
Revised: Lemma A.2 is an adaptation of Lemma 5 in McMahan & Streeter (2010) (or Lemma 4 in Reddi et al.

Original: Now we are ready to prove Theorem 4.2.
Revised: Now we are ready to prove Theorem 4.2.

Original: Proof
Revised: Proof

Original: of Theorem 4.2.
Revised: of Theorem 4.2.

Original: By Assumption 4.1, all f t ’s are convex, we have
Revised: By Assumption 4.1, all f t ’s are convex, we have

Original: [Equation]
Revised: [Equation]

Original: Consider the update rule for Algorithm 1, let (cid:98) V t =
Revised: Consider the update rule for Algorithm 1, let (cid:98) V t =

Original: diag( (cid:98) v t ) , then we have θ t
Revised: diag( (cid:98) v t ) , then we have θ t

Original: +

Revised: +

Original: =
Revised: =

Original: Π X , (cid:98) V pt (cid:0) θ t − α t (cid:98) V − p t · m t (cid:1) .
Revised: Π X , (cid:98) V pt (cid:0) θ t − α t (cid:98) V − p t · m t (cid:1) .

Original: Since Π X , (cid:98) V pt ( θ ∗ ) =
Revised: Since Π X , (cid:98) V pt ( θ ∗ ) =

Original: θ ∗ for all θ ∗ ∈ X , by Lemma A.2 we have
Revised: θ ∗ for all θ ∗ ∈ X , by Lemma A.2 we have

Original: [Equation]
Revised: [Equation]

Original: Now expand the square term on the right hand side of (A.4), we have
Revised: Now expand the square term on the right hand side of (A.4), we have

Original: [Equation]
Revised: [Equation]

Original: V
Revised: V

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: α
Revised: α

Original: [Equation]
Revised: [Equation]

Original: V −
Revised: V −

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: where the second equality follows from the deﬁnition for m t .
Revised: where the second equality follows from the deﬁnition for m t .

Original: Rearrange the items in (A.5), we have (cid:104) g t , θ t − θ ∗ (cid:105) =
Revised: Rearrange the items in (A.5), we have (cid:104) g t , θ t − θ ∗ (cid:105) =

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: where the inequality holds due to Young’s inequality.
Revised: where the inequality holds due to Young’s inequality.

Original: Summing over (A.6) for all 1 ≤ t ≤ T and submitting it back into (A.3), we obtain
Revised: Summing over (A.6) for all 1 ≤ t ≤ T and submitting it back into (A.3), we obtain

Original: T (cid:88)
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds due to the fact that β 1 t = β 1 λ t − 1 is monotonically decreasing with t .
Revised: where the last inequality holds due to the fact that β 1 t = β 1 λ t − 1 is monotonically decreasing with t .

Original: For term
Revised: For term

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the inequality follows from the bounded diameter condition and the deﬁnition of (cid:98) v t ensures that (cid:98) v pt,i /α t − (cid:98) v pt − 1 ,i /α t − 1 > 0 , and the second equality holds due to simple telescope sum.
Revised: where the inequality follows from the bounded diameter condition and the deﬁnition of (cid:98) v t ensures that (cid:98) v pt,i /α t − (cid:98) v pt − 1 ,i /α t − 1 > 0 , and the second equality holds due to simple telescope sum.

Original: For term I 2 , note that simple calculation yields that (cid:98) v pt,i ≤ G 2 p ∞ , we have
Revised: For term I 2 , note that simple calculation yields that (cid:98) v pt,i ≤ G 2 p ∞ , we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from ﬁrst relaxing √ t to t and then using the geometric series summation rule.
Revised: where the last inequality follows from ﬁrst relaxing √ t to t and then using the geometric series summation rule.

Original: For term I 3 , by Lemma A.1 we have
Revised: For term I 3 , by Lemma A.1 we have

Original: [Equation]
Revised: [Equation]

Original: (A
Revised: (A

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Submitting (A.8), (A.9) and (A.10) back into (A.7) yields the desired result.
Revised: Submitting (A.8), (A.9) and (A.10) back into (A.7) yields the desired result.

Original: A. P ROOF OF C OROLLARY 4. 
Revised: A. P ROOF OF C OROLLARY 4.

Original: Proof of Corollary 4.4.
Revised: Proof of Corollary 4.4.

Original: By Theorem 4.2, we have
Revised: By Theorem 4.2, we have

Original: [Equation]
Revised: [Equation]

Original: Note that we have
Revised: Note that we have

Original: [Equation]
Revised: [Equation]

Original: and by deﬁnition we also have
Revised: and by deﬁnition we also have

Original: [Equation]
Revised: [Equation]

Original: Combine the above results we can easily have
Revised: Combine the above results we can easily have

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: B P ROOF OF T ECHNICAL L EMMAS IN A PPENDIX A B. P ROOF OF L EMMA A. 
Revised: B P ROOF OF T ECHNICAL L EMMAS IN A PPENDIX A B. P ROOF OF L EMMA A.

Original: Proof.
Revised: Proof.

Original: Consider
Revised: Consider

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: B.1) where the ﬁrst inequality holds due to the deﬁnition of (cid:98) v T,i and the second inequality follows from the fact that β 1 t = β 1 λ t − 1 and λ ∈ (0 , 1) .
Revised: B.1) where the ﬁrst inequality holds due to the deﬁnition of (cid:98) v T,i and the second inequality follows from the fact that β 1 t = β 1 λ t − 1 and λ ∈ (0 , 1) .

Original: (B.1) can be further bounded as:
Revised: (B.1) can be further bounded as:

Original: [Equation]
Revised: [Equation]

Original: − jg (1+2 p ) j,i (cid:1) (cid:0) (cid:80)
Revised: − jg (1+2 p ) j,i (cid:1) (cid:0) (cid:80)

Original: Tj =1 β T − jg 2 j,i (cid:1) p ≤
Revised: Tj =1 β T − jg 2 j,i (cid:1) p ≤

Original: T − 1 (cid:88) t = d (cid:88) i
Revised: T − 1 (cid:88) t = d (cid:88) i

Original: = α t · m 2 t,i (cid:98) v pt,i + αG (1 − 2 p ) ∞ √
Revised: = α t · m 2 t,i (cid:98) v pt,i + αG (1 − 2 p ) ∞ √

Original: T (1 − β 1 )(1 − β 2 ) p
Revised: T (1 − β 1 )(1 − β 2 ) p

Original: d (cid:88) i = (cid:0) (cid:80)
Revised: d (cid:88) i = (cid:0) (cid:80)

Original: Tj =1 β T − jg (1+2 p )
Revised: Tj =1 β T − jg (1+2 p )

Original: j,i (cid:1) (cid:0) (cid:80)
Revised: j,i (cid:1) (cid:0) (cid:80)

Original: Tj =1 β T − jg 2 j,i (cid:1) p ≤
Revised: Tj =1 β T − jg 2 j,i (cid:1) p ≤

Original: T − 1 (cid:88) t = d (cid:88) i
Revised: T − 1 (cid:88) t = d (cid:88) i

Original: = α t · m 2 t,i (cid:98) v pt,i + αG (1 − 2 p ) ∞ √
Revised: = α t · m 2 t,i (cid:98) v pt,i + αG (1 − 2 p ) ∞ √

Original: T (1 − β 1 )(1 − β 2 ) p
Revised: T (1 − β 1 )(1 − β 2 ) p

Original: T (cid:88)
Revised: T (cid:88)

Original: j = d (cid:88)
Revised: j = d (cid:88)

Original: i = (cid:0) β T − jg (1+2 p )
Revised: i = (cid:0) β T − jg (1+2 p )

Original: j,i (cid:1) (cid:0) β T − jg 2 j,i (cid:1) p ≤
Revised: j,i (cid:1) (cid:0) β T − jg 2 j,i (cid:1) p ≤

Original: T − 1 (cid:88) t = d (cid:88) i
Revised: T − 1 (cid:88) t = d (cid:88) i

Original: = α t · m 2 t,i (cid:98) v pt,i + αG (1 − 2 p ) ∞ √
Revised: = α t · m 2 t,i (cid:98) v pt,i + αG (1 − 2 p ) ∞ √

Original: T (1 − β 1 )(1 − β 2 ) p
Revised: T (1 − β 1 )(1 − β 2 ) p

Original: T (cid:88)
Revised: T (cid:88)

Original: j = d (cid:88) i
Revised: j = d (cid:88) i

Original: = γ T − j | g j,i | , where the ﬁrst inequality holds due to Cauchy-Schwarz inequality and the fact that 0 ≤ β 1 j < 1 , the second inequality follows from the bounded gradient condition and (cid:80)
Revised: = γ T − j | g j,i | , where the ﬁrst inequality holds due to Cauchy-Schwarz inequality and the fact that 0 ≤ β 1 j < 1 , the second inequality follows from the bounded gradient condition and (cid:80)

Original: Tj =1 β T − j≤ 1 / (1 − β 1 ) , and the last inequality is due to the fact that β 1 /β p 2 ≤ β 1 / √ β 2 = γ .
Revised: Tj =1 β T − j≤ 1 / (1 − β 1 ) , and the last inequality is due to the fact that β 1 /β p 2 ≤ β 1 / √ β 2 = γ .

Original: Now repeat the above process for 1 ≤ t ≤ T , we have
Revised: Now repeat the above process for 1 ≤ t ≤ T , we have

Original: [Equation]
Revised: [Equation]

Original: where the equality holds due to the change of the order of summation, the last inequality follows from the fact that (cid:80)
Revised: where the equality holds due to the change of the order of summation, the last inequality follows from the fact that (cid:80)

Original: Tj = t γ j − t ≤ 1 / (1 − γ ) .
Revised: Tj = t γ j − t ≤ 1 / (1 − γ ) .

Original: By Cauchy-Schwarz inequality , (B.2) can be further written as
Revised: By Cauchy-Schwarz inequality , (B.2) can be further written as

Original: [Equation]
Revised: [Equation]

Original: 1 )(1 − γ )(1 − β 2 ) p d (cid:88)
Revised: 1 )(1 − γ )(1 − β 2 ) p d (cid:88)

Original: i =
Revised: i =

Original: (cid:107) g 1: T,i (cid:107)(cid:118)(cid:117)(cid:117)(cid:116) T (cid:88) t =  t ≤ αG (1 − 2 p ) ∞ √ 1 + log T (1 − β 1 )(1 − γ )(1 − β 2 ) p
Revised: (cid:107) g 1: T,i (cid:107)(cid:118)(cid:117)(cid:117)(cid:116) T (cid:88) t =  t ≤ αG (1 − 2 p ) ∞ √ 1 + log T (1 − β 1 )(1 − γ )(1 − β 2 ) p

Original: d (cid:88) i =
Revised: d (cid:88) i =

Original: (cid:107) g 1: T,i (cid:107) 2 .
Revised: (cid:107) g 1: T,i (cid:107) 2 .

Original: This completes the proof.
Revised: This completes the proof.

Original: B. P ROOF OF L EMMA A. 
Revised: B. P ROOF OF L EMMA A.

Original: Proof.
Revised: Proof.

Original: The proof is inspired from Lemma 4 in Reddi et al.
Revised: The proof is inspired from Lemma 4 in Reddi et al.

Original: (2018) and Lemma 5 in McMahan &
Revised: (2018) and Lemma 5 in McMahan &

Original: Streeter (2010).
Revised: Streeter (2010).

Original: Since a 1 = argmin x ∈X (cid:107) V p ( x − b 1 ) (cid:107) 2 and a 2 = argmin x ∈X (cid:107) V p ( x − b 2 ) (cid:107) 2 , by projection property we have
Revised: Since a 1 = argmin x ∈X (cid:107) V p ( x − b 1 ) (cid:107) 2 and a 2 = argmin x ∈X (cid:107) V p ( x − b 2 ) (cid:107) 2 , by projection property we have

Original: [Equation]
Revised: [Equation]

Original: Combine the above inequalities we have
Revised: Combine the above inequalities we have

Original: [Equation]
Revised: [Equation]

Original: Rearrange (B.3) yields that
Revised: Rearrange (B.3) yields that

Original: [Equation]
Revised: [Equation]

Original: Also since V is positive deﬁnite, the fact that ( b 2 − b 1 − ( a 2 − a 1 )) (cid:62) V 2 p ( b 2 − b 1 − ( a
Revised: Also since V is positive deﬁnite, the fact that ( b 2 − b 1 − ( a 2 − a 1 )) (cid:62) V 2 p ( b 2 − b 1 − ( a

Original: [Equation]
Revised: [Equation]

Original: a 1 )) ≥immediately implies that
Revised: a 1 )) ≥immediately implies that

Original: [Equation]
Revised: [Equation]

Original: Combining (B.4) and (B.5) we have the desired result.
Revised: Combining (B.4) and (B.5) we have the desired result.

Original: C E XPERIMENT D ETAILS C. D ATASETS
Revised: C E XPERIMENT D ETAILS C. D ATASETS

Original: We use several popular datasets for image classiﬁcations.
Revised: We use several popular datasets for image classiﬁcations.

Original: • CIFAR-10 (Krizhevsky & Hinton, 2009): it consists of a training set of 50 , 000 32 ×color images from 10 classes, and also 10 , 000 test images.
Revised: • CIFAR-10 (Krizhevsky & Hinton, 2009): it consists of a training set of 50 , 000 32 ×color images from 10 classes, and also 10 , 000 test images.

Original: • CIFAR-100 (Krizhevsky & Hinton, 2009): it is similar to CIFAR-10 but contains 100 image classes with 600 images for each.
Revised: • CIFAR-100 (Krizhevsky & Hinton, 2009): it is similar to CIFAR-10 but contains 100 image classes with 600 images for each.

Original: • ImageNet dataset
Revised: • ImageNet dataset

Original: (ILSVRC2012)
Revised: (ILSVRC2012)

Original: (Deng et al., 2009): ILSVRC2012 contains 1 .
Revised: (Deng et al., 2009): ILSVRC2012 contains 1 .

Original: 28 million training images, and 50 k validation images over 1000 classes.
Revised: 28 million training images, and 50 k validation images over 1000 classes.

Original: C. P ARAMETER S ETTINGS
Revised: C. P ARAMETER S ETTINGS

Original: We perform grid searches to choose the best base learning rate for all algorithms over { 0 . 0001 , 0 . 001 , 0 . 01 , 0 . 1 , 1 } , the partial adaptive parameter p from { 1 / 4 , 1 / 8 , 1 / 16 } and the second order momentum parameter β 2 from { 0 . 9 , 0 . 99 , 0 . 999 } .
Revised: We perform grid searches to choose the best base learning rate for all algorithms over { 0 . 0001 , 0 . 001 , 0 . 01 , 0 . 1 , 1 } , the partial adaptive parameter p from { 1 / 4 , 1 / 8 , 1 / 16 } and the second order momentum parameter β 2 from { 0 . 9 , 0 . 99 , 0 . 999 } .

Original: For SGD with momentum, the base learning rate is set to be 0 .
Revised: For SGD with momentum, the base learning rate is set to be 0 .

Original: 1 with a momentum parameter of 0 .
Revised: 1 with a momentum parameter of 0 .

Original: For Adam and Amsgrad, we set the base learning rate as 0 .
Revised: For Adam and Amsgrad, we set the base learning rate as 0 .

Original: For Padam, the base learning rate is set to be 0 .
Revised: For Padam, the base learning rate is set to be 0 .

Original: 1 and the partially adaptive parameter p is set to be 1 / 8 due to its best empirical performance.
Revised: 1 and the partially adaptive parameter p is set to be 1 / 8 due to its best empirical performance.

Original: We also tune the momentum parameters for all the adaptive gradient methods.
Revised: We also tune the momentum parameters for all the adaptive gradient methods.

Original: After tuning, for Adam, Amsgrad, the momentum parameters are set to be β 1 = 0 .
Revised: After tuning, for Adam, Amsgrad, the momentum parameters are set to be β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: And for Padam, we set β 1 = 0 .
Revised: And for Padam, we set β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: All the algorithms tested also come with  a weight decay factor of 5 × 10 − 4  . 
Revised: Padam and SGD with momentum use a weight decay factor of 5 × 10 − 4 and for Adam/ Amsgrad we use a weight decay factor of 1 × 10 − 4 . For AdamW, the normalized weight decay weight decay factor is set to 2 . 5 × 10 − 2 for CIFAR and 5 × 10 − 2 for ImageNet.

Original: All experiments use cross-entropy as the loss function.
Revised: All experiments use cross-entropy as the loss function.

Original: The minibatch sizes for CIFAR-10 and CIFAR-100 are set to be 128 and for ImageNet dataset we set it to be 256 .
Revised: The minibatch sizes for CIFAR-10 and CIFAR-100 are set to be 128 and for ImageNet dataset we set it to be 256 .

Original: C. CNN A RCHITECTURES
Revised: C. CNN A RCHITECTURES

Original: VGGNet (Simonyan & Zisserman, 2014):
Revised: VGGNet (Simonyan & Zisserman, 2014):

Original: We use a modiﬁed VGG-16 architecture for this experiment.
Revised: We use a modiﬁed VGG-16 architecture for this experiment.

Original: The VGG-16 network uses only 3 × 3 convolutional layers stacked on top of each other for increasing depth and adopts max pooling to reduce volume size.
Revised: The VGG-16 network uses only 3 × 3 convolutional layers stacked on top of each other for increasing depth and adopts max pooling to reduce volume size.

Original: Finally, two fully-connected layers 6 are followed by a softmax classiﬁer.
Revised: Finally, two fully-connected layers 6 are followed by a softmax classiﬁer.

Original: ResNet (He et al., 2016): Residual Neural Network (ResNet) introduces a novel architecture with “skip connections” and features a heavy use of batch normalization.
Revised: ResNet (He et al., 2016): Residual Neural Network (ResNet) introduces a novel architecture with “skip connections” and features a heavy use of batch normalization.

Original: Such skip connections are also known as gated units or gated recurrent units and have a strong similarity to recent successful elements applied in RNNs.
Revised: Such skip connections are also known as gated units or gated recurrent units and have a strong similarity to recent successful elements applied in RNNs.

Original: We use ResNet-18 for this experiment, which contains 2 blocks for each type of basic convolutional building blocks in He et al.
Revised: We use ResNet-18 for this experiment, which contains 2 blocks for each type of basic convolutional building blocks in He et al.

Original: Wide ResNet (Zagoruyko & Komodakis, 2016):
Revised: Wide ResNet (Zagoruyko & Komodakis, 2016):

Original: Wide Residual Network further exploits the “skip connections” used in ResNet and in the meanwhile increases the width of residual networks.
Revised: Wide Residual Network further exploits the “skip connections” used in ResNet and in the meanwhile increases the width of residual networks.

Original: In detail, we use the 16 layer Wide ResNet with 4 multipliers (WRN-16-4) in our experiments.
Revised: In detail, we use the 16 layer Wide ResNet with 4 multipliers (WRN-16-4) in our experiments.

Original: D A DDITIONAL E XPERIMENTS
Revised: D A DDITIONAL E XPERIMENTS

Original: Figure 4 plots the train loss and test error (top- 1 error) against training epochs on the CIFARdataset and the CIFAR-100 dataset respectively.
Revised: Figure 4 plots the train loss and test error (top- 1 error) against training epochs on the CIFARdataset.

Original: We can see that Padam achieves the best from both worlds by maintaining faster convergence rate while also generalizing as well as SGD with momentum in the end.
Revised: We can see that Padam achieves the best from both worlds by maintaining faster convergence rate while also generalizing as well as SGD with momentum in the end.

Original: [Table]
Revised: [Table]

Original: Table 2 shows the test accuracy of VGGNet on CIFAR-100 dataset.
Revised: Table 2 shows the test accuracy of VGGNet on CIFAR-100 dataset.

Original: We can see that  Padam achieves the best test accuracy at 25 th, 50 th, 75 th and 100 th epochs for CIFAR-100 dataset.
Revised: We can see that for CIFARdataset, Padam achieves the best test accuracy at 100 th, and also keep really close to SGD with momentum at 150 th and 200 th epochs (differences less than 0 .

Original: Figure 5(a) shows the comparison of test error performances under the different partial adaptive parameter p for ResNet on CIFAR-100 dataset. We can observe that a larger p will lead to fast convergence at early stages and worse generalization performance later, while a smaller p leads to slow convergence but ﬁnally good generalization performances. While a proper choice of p could obtain the best of both worlds.
Revised: As a side product of our experiments, we also examine the distance that θ t has traversed in the parameter space, i.e., (cid:107) θ t − θ 0 (cid:107) 2 , as shown in many other studies (Hoffer et al., 2017; Xing et al.,

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: 2018) discussing the generalization performance of various optimization algorithms for deep learning.

Original: Figure 5(b) shows the plot of the Euclidean distance of weight vector θ t from initialization θ 0 , against training epochs for ResNet on CIFAR-100 dataset.
Revised: Figure 5 shows the plot of the Euclidean distance of weight vector θ t from initialization θ 0 , against training epochs for ResNet on CIFAR-10 dataset 7 .

Revised: First, we would like to emphasize that these are the plots only regrading the Euclidean distance, which does not contain any directional information.

Revised: In other words, the cross in the plot does not mean that the weight vectors in different optimization algorithms actually meet somewhere during the training process.

Original: As we can see from the plots, SGD with momentum tends to overshoot a lot at the early stage of the entire training process.
Revised: As we can see from the plots, SGD with momentum tends to overshoot a lot at the early stage of the entire training process.

Original: And for
Revised: This could explain why the convergence of SGD with momentum is slower at early stages.

Original:  Adam/Amsgrad, despite the quick start and less overshooting, the “small learning rate dilemma”  largely limits the weight vector’s ability of exploring the parameter space in the middle and later stages.
Revised: And for Adam/Amsgrad, despite the quick start and less overshooting, the “small learning rate dilemma” (see Section 3) largely limits the weight vector’s ability of exploring the parameter space in the middle and later stages, which could explain the bad generalization performance of Adam/Amsgrad.

Original: On the other hand, Padam behaves inbetween SGD with momentum and Adam/Amsgrad, with less severe overshooting compared with SGD with momentum, while maintaining a better capability of traversing through the parameter space compared with Adam/Amsgrad.
Revised: On the other hand, Padam behaves inbetween SGD with momentum and Adam/Amsgrad, with less severe overshooting compared with SGD with momentum, while maintaining a better capability of traversing through the parameter space compared with Adam/Amsgrad.

Revised: This partly justiﬁes the outstanding generalization performance of Padam.