D ECAYING MOMENTUM HELPS NEURAL NETWORK TRAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Momentum is a simple and popular technique in deep learning for gradient-based optimizers.
We propose a decaying momentum (D EMON ) rule, motivated by decaying the total contribution of a gradient to all future updates.
Applying D EMON to Adam leads to signiﬁcantly improved training, notably competitive to momentum SGD with learning rate decay, even in settings in which adaptive methods are typically non-competitive.
Similarly, applying D EMON to momentum SGD rivals momentum SGD with learning rate decay and in some cases leads to improved performance.
D EMON is trivial to implement and incurs limited extra computational overhead, compared to the vanilla counterparts.
I NTRODUCTION
Deep Neural Networks (DNNs) have drastically advanced the state-of-the-art performance in many computer science applications, including computer vision (Krizhevsky et al., 2012), (He et al., 2016;
Ren et al., 2015), natural language processing (Mikolov et al., 2013; Bahdanau et al., 2014; Gehring et al., 2017) and speech recognition (Sak et al., 2014; Sercu et al., 2016).
Yet, in the face of such signiﬁcant developments, the age-old stochastic gradient descent (SGD), and the accelerated variant
SGD with momentum (SGDM), algorithm remains one of the most, if not the most, popular method for training DNNs (Sutskever et al., 2013; Goodfellow et al., 2016; Wilson et al., 2017).
Adaptive methods (Duchi et al., 2011; Zeiler, 2012; Hinton et al., 2012; Kingma & Ba, 2014; Ma &
Yarats, 2018) sought to simplify the training process, while providing similar performance.
However, while they are often used by practitioners, there are cases where their use leads to a performance gap (Wilson et al., 2017; Shah et al., 2018).
At the same time, much of the state-of-the-art performance on highly contested benchmarks—such as the image classiﬁcation dataset ImageNet—have been produced with SGDM (Krizhevsky et al., 2012;
He et al., 2016;
Xie et al., 2017; Zagoruyko &
Komodakis, 2016; Huang et al., 2017; Ren et al., 2015; Howard et al., 2017).
Nevertheless, a key factor in any algorithmic success still lies in hyperparameter tuning.
For example, in the literature above, they obtain such performance with a well-tuned SGD with momentum and a learning rate decay schedule, or with a proper hyperparameter tuning in adaptive methods.
Slight changes in learning rate, learning rate decay, momentum, and weight decay (amongst others) can drastically alter performance.
Hyperparameter tuning is arguably one of the most time consuming parts of training DNNs, and researchers often resort to a costly grid search.
Thus, ﬁnding new and simple hyper-parameter tuning routines that boost the performance of state of the art algorithms is of ultimate importance and one of the most pressing problems in machine learning.
The focus of this work is on the momentum parameter and how we can boost the performance of training methods with a simple technique.
Momentum helps speed up learning in directions of low curvature, without becoming unstable in directions of high curvature.
Minimizing the objective function L ( · ) , the simplest and most common momentum method, SGDM, is given by the following recursion for variable vector θ t ∈ R p :
[Equation]
The coefﬁcient β —traditionally, selected constant in [0 , 1] —controls how quickly the momentum decays, g t represents a stochastic gradient, usually E [ g t ] =
∇L ( θ t ) , and η > 0 is the step size.
But how do we select β ?
The most prominent choice among practitioners is β = 0 .
This is supported by recent works that prescribe it (Chen et al., 2016; Kingma & Ba, 2014; Hinton et al.,
2012; Reddi et al., 2019), and by the fact that most common softwares, such as PyTorch (Paszke et al., 2017), declare β = 0 .
9 as the default value in their optimizer implementations.
However, there is no indication that this choice is universally well-behaved.
There are papers that attempt to tune the momentum parameter.
Under an asynchronous distributed setting, (Mitliagkas et al., 2016) observe that running SGD asynchronously is similar to adding a momentum-like term to SGD; they also provide experimental evidence that naively setting β
=
.would result in a momentum “overdose”, leading to suboptimal performance.
As another example,
YellowFin (Zhang & Mitliagkas, 2017) is a learning rate and momentum adaptive method for both the synchronous and asynchronous setting, motivated by a quadratic model analysis and some robustness insights.
The main message of that work is that, like η , momentum acceleration needs to be carefully selected based on properties of the objective, the data, and the underlying computational resources.
Finally, moving from classical DNN settings towards generative adversarial networks (GANs), the proposed momentum values tend to decrease from β = 0 .
9 (Mirza & Osindero, 2014;
Radford et al., 2015; Arjovsky et al., 2017), taking even negative values (Gidel et al., 2018).
In this paper, we introduce a novel momentum decay rule which signiﬁcantly surpasses the performance of both Adam and SGDM (as they are used currently), in addition to other state-of-the-art adaptive learning rate and adaptive momentum methods, across a variety of datasets and networks.
In particular, our ﬁndings can be summarized as follows:
i )
We propose a new momentum decay rule, motivated by decaying the total contribution of a gradient to all future updates, with limited overhead and additional computation.
ii ) Using the momentum decay rule with Adam, we observe large performance gains—relative to vanilla Adam—where the network continues to learn for far longer after Adam begins to plateau, and suggest that the momentum decay
rule
should be used as
default
for this method.
iii )
We observe comparative performance for SGDM between momentum decay and learning rate decay; an interesting result given the unparalleled effectiveness of learning rate decay schedule
Experiments are provided on various datasets, including MNIST, CIFAR-10, CIFAR-100, STL-10,
Penn Treebank (PTB), and networks, including Convolutional Neural Networks (CNN) with Residual architecture (ResNet) (He et al., 2016),
Wide Residual architecture (Wide ResNet) (Zagoruyko & Komodakis, 2016), Non-Residual architecture (VGG-16) (Simonyan & Zisserman, 2014), Recurrent Neural Networks (RNN) with Long Short-Term Memory architecture (LSTM) (Hochreiter &
Schmidhuber, 1997), Variational AutoEncoders (VAE) (Kingma & Welling, 2015), and the recent
Noise Conditional Score Network (NCSN) (Song & Ermon, 2019).
P RELIMINARIES
Plain stochastic gradient descent motions.
Let θ t ∈ R p be the parameters of the network at time step t , where η ∈ R is the learning rate/step size, and g t is the stochastic gradient w.r.t.
θ t for empirical loss L ( · ) , such that E [ g t ] =
∇L ( θ t ) .
Then, plain stochastic gradient descent (SGD) uses the recursion: θ t +1 =
θ t − η · g t , ∀ t .
Here, the step size η could also be time dependent, η t , but practice shows that decreasing the value of η at regular or predeﬁned intervals works favorably compared to decreasing the value of η at every iteration.
SGDM is parameterized by β ∈ R , the momentum coefﬁcient, and follows the recursion:
[Equation]
where v t ∈ R p accumulates momentum.
Observe that for β = 0 , the above recursion is equivalent to SGD.
Common values for β are closer to one, with β = 0 .
9 the most used value (Ruder, 2016).
Adaptive gradient descent motions.
These algorithms utilize current and past gradient information to design preconditioning matrices that better approximate the local curvature of L ( · ) .
Beginning with AdaGrad (Duchi et al., 2011), the SGD recursion, per coordinate i of θ , becomes:
[Equation]
where G t ∈ R p × p is usually a diagonal preconditioning matrix as a summation of squares of past gradients, and ε > 0 a small constant.
RMSprop (Hinton et al., 2012) substitutes the ever accumulating matrix G t with a root mean squared operation.
Denoting the average of squared gradients as E g ◦ g t , per iteration we compute: E g ◦ g t +1 = β 2 ·E g ◦ g t +(1 − β 2 ) · ( g t ◦ g t ) , where β 2 was ﬁrst proposed as 0 . 9 .
Here, ◦ denotes the per-coordinate multiplication.
Then, RMSprop updates as—where a momentum term can also be optionally added:
[Equation]
∀
[Equation]
Finally, Adam (Kingma & Ba, 2014), in addition, keeps an exponentially decaying average of past gradients: E gt +
= β 1 · E gt + (1 − β 1 ) · g t , leading to the recursion:
[Equation]
t
[Equation]
where usually β 1 = 0 .
9 and β 2 = 0 .
Observe that Adam is equivalent to RMSprop when β 1 = 0 , and when no bias correction is applied results in the same recursion.
D EMON : D ECAYING MOMENTUM ALGORITHM
Algorithm
EMON
1: Parameters : # of iterations T , step size η , momentum initial value β init .
2: v t = θ t = 0 , otherwise randomly initialized.
3: for t = 0 , . . .
, T do 4: β t = β init · (cid:16) 1 − tT (cid:17) (1 − β init )
+ β init ( 1 − tT ) 5: θ t +1 =
θ t − ηg t + β t v t 6: v t +1 = β t v t − ηg t 7: end for
Algorithm initialized t do β t = β init · (cid:16) 1 − tT (cid:17) (1 − β init )+ β init ( 1 − tT ) 5:
E g ◦ g t +1 = β 2 · E g ◦ g t + (1 − β 2 ) · ( g t ◦ g t ) 6: m t,i =
g t, 1 + β t m t − 1 ,i 7: θ t +1 ,i = θ t,i − η √ E g ◦ g t
+
,i + ε · m t,i 8: end for
Motivation and interpretation.
D EMON is motivated by linear learning rate decay models which reduce the impact of a gradient to current and future updates.
By decaying the momentum parameter, we decay the total contribution of a gradient to all future updates.
Our goal here is to present a concrete, effective, and easy-to-use momentum decay procedure which we show in the experimental section.
By Occam’s Razor, a linear decay achieves such a goal since it is simple and requires no tuning in most cases.
The key component is the momentum decay schedule:
[Equation]
Above, the fraction (1 − t/T ) refers to the proportion of iterations remaining.
The interpretation of this rule comes from the following argument: Assume ﬁxed momentum parameter β t ≡ β ; e.g., β = 0 .
9 , as literature dictates.
For our discussion, we will use the SGDM recursion.
We know that v 0 = 0 , and v t = βv
t − 1 − g t .
Then, the main recursion can be unrolled into:
[Equation]
Interpreting the above recursion, a particular gradient term g t contributes a total of η (cid:80) i β i of its “energy” to all future gradient updates.
Moreover, for an asymptotically large number of iterations, we know that β contributes on up to t − 1 terms.
Then, (cid:80) ∞ i =1 β i = β (cid:80) ∞
i =
β i =
β/ (1 − β ) .
Thus, in our quest for a decaying schedule and for a simple linear momentum decay, it is natural to consider a scheme where the cumulative momentum is decayed to 0 .
Let β init be the initial β ; then at current step t with total T steps, we design the decay routine such that: β/
(1 − β ) =
(1 − t/T ) β init / (1 − β init ) .
This leads to equation 1.
Although β changes in subsequent iterations, this is typically a very close approximation since β i β
i +1 . . .
β t for a particular g i diminishes much faster than β changes.
Connection to previous algorithms.
D EMON introduces an implicit discount factor.
The main recursions of the algorithm are the same with standard algorithms in machine learning.
E.g., for β t = β = 0 .
9 we obtain SGD with momentum, and for β =
we obtain plain SGD in Algorithm 1; in Algorithm 2, for β 1 = 0 .
9 with a slightly adjustment of learning rate we obtain Adam, while for β 1 =
we obtain a non-accumulative AdaGrad algorithm.
We choose to apply D EMON to a slightly adjusted Adam—instead of vanilla Adam—to isolate the effect of the momentum parameter, since the momentum parameter adjusts the magnitude of the current gradient as well in vanilla Adam.
Efﬁciency.
D EMON requires only limited extra overhead and computation in comparison to the vanilla counterparts, for the computation of β t .
Practical suggestions.
For settings in which β init is typically large, such as image classiﬁcation, we advocate for decaying momentum from β init at t = 0 , to 0 at t =
T as a general rule.
We also observe and report improved performance by delaying momentum decay till later epochs.
In many cases, performance can be further improved by decaying to a small negative value, such as 0.3.
R ELATED WORK
There are numerous techniques for automatic hyperparameter tuning.
The most widely used are learning rate adaptive methods, starting with AdaGrad (Duchi et al., 2011), AdaDelta (Zeiler, 2012),
RMSprop (Hinton et al., 2012), and Adam (Kingma & Ba, 2014).
Adam (Kingma & Ba, 2014), the most popular, introduced a momentum term, which is combined with the current gradient before multiplying with an adaptive learning rate.
Interest in closing the generalization difference between adaptive methods and SGDM led to AMSGrad (Reddi et al., 2019), replacing the exponential moving average of squared gradients with the maximum, QHAdam (Ma & Yarats, 2018), a variant of
QHM which is capable of recovering a variety of optimization algorithms, AdamW (Loshchilov &
Hutter, 2017), by ﬁxing the weight decay of Adam, and Padam (Chen & Gu, 2018), by lowering the exponent of the second moment.
Asynchronous methods are commonly used in deep learning, and (Mitliagkas et al., 2016) show that running SGD asynchronously is similar to adding a momentum-like term to SGD without assumptions of convexity of the objective function.
They demonstrate this natural connection empirically on CNNs.
This implies that the momentum parameter needs to be tuned according to the level of asynchrony.
YellowFin (Zhang & Mitliagkas, 2017) is a learning rate and momentum adaptive method for both the synchronous and asynchronous setting motivated by a quadratic model analysis and robustness insights.
In the non-convex setting, STORM (Cutkosky & Orabona, 2019) uses a variant of momentum for variance reduction.
There is substantial research, both empirical and theoretical, into the convergence of momentum methods (Wibisono & Wilson, 2015; Wibisono et al., 2016; Wilson et al., 2016; Kidambi et al., 2018).
In addition, (Sutskever et al., 2013) explored momentum schedules, with even increasing momentum schedules during training, inspired by Nesterov’s routines for convex optimization.
(Smith et al., 2017) explores increasing batch size instead of decreasing the learning rate, as well as scaling batch size to increasing learning rate or increasing momentum.
(Smith, 2018) introduces cycles of simultaneously increasing learning rate and decreasing momentum followed by simultaneously decreasing learning rate and increasing momentum.
There is some work into reducing oscillations during training, by adapting the momentum (Odonoghue & Candes, 2015).
There is also work into adapting momentum in well-conditioned convex problems as opposed to setting to zero (Srinivasan et al., 2018).
Another approach in this area is to keep several momentum vectors according to different β and combining them (Lucas et al., 2018).
We are aware of the theoretical work of (Yuan, 2016) which prove under certain conditions that SGDM is equivalent to SGD with a rescaled learning rate, however our experiments in the deep learning setting show slightly different behavior and understanding why is an exciting direction of research.
Smaller values of β have gradually been employed for Generative Adversarial Networks (GAN), and recent developments in game dynamics (Gidel et al., 2018) show a negative momentum is helpful.
E XPERIMENTS
We separate experiments into those with adaptive learning rate and those with adaptive momentum.
All settings, with exact hyper-parameters, are brieﬂy summarized in Table 1 and comprehensively detailed in Appendix A.
We report improved performance by delaying the application of D EMON where applicable, and report performance across different number of total epochs to demonstrate effectiveness regardless of the training budget.
Note that the predeﬁned number of epochs we run all experiments affects the proposed decaying momentum routine, by deﬁnition of β t .
[Table] [Table]
5. A DAPTIVE METHODS
At ﬁrst
[Table]
we apply D EMON Adam (Algorithm 2) to a variety of models and tasks.
We select vanilla
Adam as the baseline algorithm and include more recent state-of-the-art adaptive learning rate methods Quasi-Hyperbolic Adam (QHAdam) (Ma & Yarats, 2018) and AMSGrad (Reddi et al., 2019) in our comparison.
See Appendix A.2.1 for details.
We tune all learning rates in roughly multiples of 3 and try to keep all other parameters close to those recommended in the original literature.
For
D EMON Adam, we leave β init = 0 .
9 , β 2 = 0 .
999 and decay from β init to 0 in all experiments.
Residual Neural Network ( RN18-CIFAR10-DEMONAdam ).
We train a ResNet18 (He et al., 2016) model on the CIFAR-10 dataset.
With D EMON Adam, we achieve the generalization error reported in the literature (He et al., 2016) for this model, attained using SGDM and a curated learning rate decay schedule, whilst all other methods are non-competitive.
Refer to Table 2 for exact results.
In Figure 2 ( Top row, two left-most plots ), D EMON Adam is able to learn in terms of both loss and accuracy after other methods have plateaued.
Running 5 seeds, D EMON Adam outperforms all other methods by a large 2%-5% generalization error margin with a small and large number of epochs.
Non-Residual Neural Network ( VGG16-CIFAR100-DEMONAdam ).
For the CIFAR-100 dataset, we train an adjusted VGG-16 model (Simonyan & Zisserman, 2014).
Similarly to the previous setting, we observe similar learning behavior of D EMON Adam, where it continues to improve after other methods appear to begin to plateau.
We note that this behavior results in a 1-3% decrease in generalization error than typically reported results with the same model and task (Sankaranarayanan et al., 2018), which are attained using SGDM and a curated learning rate decay schedule.
Running 5 seeds, D EMON Adam achieves an improvement of 3%-6% generalization error margin over all other methods, both for a small and large number of epochs.
Refer to Figure 2 ( Top row, right-most plot ) and Table 3 for more details.
Wide Residual Neural Network ( WRN-STL10-DEMONAdam ).
The STL-10 dataset presents a different challenge with a signiﬁcantly smaller number of images than the CIFAR datasets, but in higher resolution.
We train a Wide Residual 16-8 model (Zagoruyko & Komodakis, 2016) for this task.
In this setting, we note again the behavior of D EMON Adam signiﬁcantly outperforming other methods in the latter stages of training.
Running 5 seeds, D EMON Adam outperforms all other methods by a 0.5%-2% generalization error margin with a small and large number of epochs.
Refer to Figure 2 ( Bottom row, left-most plot ) and
Table 3 for more details.
[Table] [Table]
LSTM (PTB-LSTM-DEMONAdam) .
Language modeling can have gradient distributions which are sharp; for example, in the case of rare words.
We use an LSTM (Hochreiter & Schmidhuber, 1997) model to this task.
We observe overﬁtting for all adaptive methods.
Similar to above, running 5 seeds, D EMON Adam outperforms all other methods by a 6-14 generalization perplexity margin .
Refer to Figure 2 ( Bottom row, middle plot ) and Table 4 for details.
Variational AutoEncoder (VAE-MNIST-DEMONAdam) .
Generative models are a branch of unsupervised learning that try to learn the data distribution.
VAEs (Kingma & Welling, 2015) pair a generator network with a second Neural Network, a recognition model that performs approximate inference, and can be trained with backpropagation.
We train VAEs on the MNIST dataset.
Running 5 seeds, D EMON Adam outperforms all other methods, particularly for smaller number of epochs.
Refer to Figure 2 ( Bottom row, right-most plot ) and Table 4 for more details.
Noise Conditional Score Network (NCSN-CIFAR10-DEMONAdam) .
NCSN (Song & Ermon, 2019) is a recent generative network achieving state-of-the-art inception score on CIFAR10.
NCSN estimates the gradients of the data distribution with score matching.
Samples are then produced via
Langevin dynamics using those gradients.
We train a NCSN on the CIFAR10 dataset and, using the ofﬁcial implementation, were unable to reproduce the reported score in the literature.
NSCN trained with Adam achieves a superior inception score in Table 4, however the produced images in Figureexhibit a noticeably unnatural green compared to those produced by D EMON Adam.
5. A DAPTIVE MOMENTUM METHODS
We apply D EMON SGDM (Algorithm 1) to a variety of models and tasks.
Since SGDM with learning rate decay is most often used to achieve the state-of-the-art results with the architectures and tasks in question, we include SGDM with learning rate decay as the target to beat.
SGDM with learning rate decay is implemented with a decay on validation error plateau, where we hand-tune the number of epochs to deﬁne plateau.
Recent adaptive momentum methods included in this section are Aggregated Momentum (AggMo)
(Lucas et al., 2018), and Quasi-Hyperbolic Momentum
[Figure]
(QHM) (Ma & Yarats, 2018).
We exclude accelerated SGD (Jain et al., 2017) due to difﬁculties in tuning.
See Appendix A.2.2 for details.
Similar to the last section, we tune all learning rates in roughly multiples of 3 and try to keep all other parameters close to those recommended in the original literature.
For D EMON SGDM, we leave β init = 0 .
9 for most experiments and generally decay from β init to 0 .
Residual Neural Network ( RN18-CIFAR10-DEMONSGDM ).
We train a ResNet18 model on the
CIFAR-10 dataset.
With D EMON SGDM, we achieve better generalization error than SGDM with learning rate decay, the optimizer for producing state-of-the-art results with ResNet architecture.
The better performance of decaying momentum relative to learning rate decay is surprising.
Running 5 seeds, D EMON SGDM outperforms all other adaptive momentum methods by a large 3%-8% validation error margin with a small and large number of epochs and is competitive or better than SGDM with learning rate decay.
In Figure 3 ( Top row, two left-most plots ), D EMON SGDM is observed to continue learning after other adaptive momentum methods appear to begin to plateau.
Non-Residual Neural Network ( VGG16-CIFAR100-DEMONSGDM ).
For the CIFAR-100 dataset, we train an adjusted VGG-16 model.
In Figure 3 ( Top row, right-most plot ), we observe D EMON
SGDM to learn slowly initially in loss and error, but similar to the previous setting it continues to learn after other methods begin to plateau, resulting in superior ﬁnal generalization error.
Running 5 seeds, D EMON SGDM achieves an improvement of 1%-8% generalization error margin over all other methods.
Refer to Table 6 for more details.
Wide Residual Neural Network ( WRN-STL10-DEMONSGDM ).
We train a Wide Residual 16model for the STL-10 dataset.
In Figure 3 ( Bottom row, left-most plot ), training in both loss and error slows down quickly for other adaptive momentum methods with a large gap with SGDM learning rate decay.
D EMON SGDM continues to improve and eventually catches up to SGDM learning rate decay.
Running 5 seeds, D EMON SGDM outperforms all other methods by a 1.5%-2% generalization error margin with a small and large number of epochs.
Refer to Table 6 for more details.
[Table]
[Table] [Table]
-LSTM-DEMONSGDM ).
We train an RNN with LSTM architecture for the PTB language modeling task.
Running 5 seeds, D EMON SGDM slightly outperforms other adaptive momentum methods in generalization perplexity, and is competitive with SGDM with learning rate decay.
Refer to Figure 3 ( Bottom row, middle plot ) and Table 7 for more details.
Variational AutoEncoder (VAE-MNIST-DEMONSGDM) .
We train the generative model VAE on the MNIST dataset.
Running 5 seeds, D EMON SGDM outperforms all other methods by a 2%- 6% generalization error for a small and large number of epochs.
Refer to Figure 3 ( Bottom row, right-most plot ) and Table 7 for more details.
C ONCLUSION
We show the effectiveness of the proposed momentum decay rule, D EMON , across a number of datasets and architectures.
The adaptive optimizer Adam combined with D EMON is empirically substantially superior to the popular Adam, in addition to other state-of-the-art adaptive learning rate algorithms, suggesting a drop-in replacement.
Surprisingly, it is also demonstrated that D EMON
SGDM is comparable to SGDM with learning rate decay.
In cases where budget is limited, D EMON
SGDM may be preferable.
D EMON is computationally cheap, easy to understand and use, and we hope it is useful in practice and as a subject of future research.
A E XPERIMENTS
We evaluated the momentum decay rule with Adam and SGDM on Residual CNNs, Non Residual
CNNS, RNNs and generative models.
For CNNs, we used the image classiﬁcation datasets CIFAR10, CIFAR100 and STL10 datasets.
For RNNs, we used the language modeling dataset PTB.
For generative modeling, we used the MNIST and CIFAR10 datasets.
For each network dataset pair other than NSCN, we evaluated Adam, QHAdam, AMSGrad, D EMON Adam, AggMo, QHM, D EMON SGDM, and SGDM with learning rate decay.
For adaptive learning rate methods and adaptive momentum methods, we generally perform a grid search over the learning rate.
For SGDM, we generally perform a grid search over learning rate and initial momentum.
For SGDM learning rate decay, the learning rate is decayed by a factor of 0.1 after there is no improvement in validation loss for the best of { 1 , 2 , 3 , 5 , 10 , 20 , 30 , 40 } epochs.
A. S ETUP
We describe the six test problems in this paper.
• CIFAR10 - ResNet18 CIFAR10 contains 60,000 32x32x3 images with a 50,000 training set, 10,000 test set split.
There are 10 classes.
ResNet18 (He et al., 2016) is an 18 layers deep CNN with skip connections for image classiﬁcation.
Trained with a batch size of 128. • CIFAR100 - VGG16 CIFAR100 is a ﬁne-grained version of CIFAR-10 and contains 60, 32x32x3 images with a 50,000 training set, 10,000 test set split.
There are 100 classes.
VGG (Simonyan & Zisserman, 2014) is a 16 layers deep CNN with extensive use of 3x3 convolutional ﬁlters.
Trained with a batch size of• STL10 - Wide ResNet 16-8 STL10 contains 1300 96x96x3 images with a 500 training set,test set split.
There are 10 classes.
Wide ResNet 16-8 (Zagoruyko & Komodakis, 2016) is alayers deep ResNet which is 8 times wider.
Trained with a batch size of 64. • PTB - LSTM PTB is an English text corpus containing 929,000 training words, 73,000 validation words, and 82,000 test words.
There are 10,000 words in the vocabulary.
The model is stacked LSTMs (Hochreiter & Schmidhuber, 1997) with 2 layers, 650 units per layer, and dropout of 0.5.
Trained with a batch size of 20. • MNIST - VAE MNIST contains 60,000 32x32x1 grayscale images with a 50,000 training set, 10,000 test set split.
There are 10 classes of 10 digits.
VAE (Kingma & Welling, 2015) with three dense encoding layers and three dense decoding layers with a latent space of size 2.
Trained with a batch size of 100.
• CIFAR10 - NCSN CIFAR10 contains 60,000 32x32x3 images with a 50,000 training set, 10, test set split.
There are 10 classes.
NCSN (Song & Ermon, 2019) is a recent state-of-the-art generative model which achieves the best reported inception score.
We compute inception scores based on a total of 50000 samples.
We follow the exact implementation in and defer details to the original paper.
A. M ETHODS A.2. A DAPTIVE LEARNING RATE
Adam (Kingma & Ba, 2014), as previously introduced in section 2, keeps an exponentially decaying average of squares of past gradients to adapt the learning rate.
It also introduces an exponentially decaying average of gradients.
The Adam algorithm is parameterized by learning rate η > 0 , discount factors β 1 < 1 and β 2 < 1 , a small constant (cid:15) , and uses the update rule:
[Equation]
i
[Equation]
ε
[Equation]
i , ∀ t.
AMSGrad (Reddi et al., 2019) resolves an issue in the proof of Adam related to the exponential moving average E g ◦ g t , where Adam does not converge for a simple optimization problem.
Instead of an exponential moving average, AMSGrad keeps a running maximum of E g ◦ g .
The AMSGrad algorithm is parameterized by learning rate η > 0 , discount factors β 1 < 1 and β 2 < 1 , a small constant (cid:15) , and uses the update rule:
[Equation]
where E gt +1 and E g ◦ g t +1 are deﬁned identically to Adam.
QHAdam (Quasi-Hyperbolic Adam) (Ma & Yarats, 2018) extends QHM (Quasi-Hyperbolic Momentum), introduced further below, to replace both momentum estimators in Adam with quasihyperbolic terms.
This quasi-hyperbolic formulation is capable of recovering Adam and NAdam (Dozat, 2016), amongst others.
The QHAdam algorithm is parameterized by learning rate η > 0 , discount factors β 1 < 1 and β 2 < 1 , ν 1 , ν 2 ∈ R , a small constant (cid:15) , and uses the update rule:
[Equation]
where E gt +1 and E g ◦ g t +1 are deﬁned identically to Adam.
A.2. A DAPTIVE MOMENTUM
AggMo (Aggregated Momentum)
(Lucas et al., 2018) takes a linear combination of multiple momentum buffers.
It maintains K momentum buffers, each with a different discount factor, and averages them for the update.
The AggMo algorithm is parameterized by learning rate η > 0 , discount factors β ∈ R K , and uses the update rule:
[Equation]
QHM (Quasi-Hyperbolic Momentum) (Ma & Yarats, 2018) is a weighted average of the momentum and plain SGD.
QHM is capable of recovering Nesterov Momentum (Nesterov, 1983), Synthesized
Nesterov Variants (Lessard et al., 2016), accSGD (Jain et al., 2017) and others.
The QHM algorithm is parameterized by learning rate η > 0 , discount factor β < 1 , immediate discount factor ν ∈ R , and uses the update rule:
[Equation]
A. O PTIMIZER HYPERPARAMETERS
[Table]
[Table] [Table] [Table] [Table]
B C ONVERGENCE A NALYSIS
We analyze the global convergence of D EMON CM in the convex setting, following (Ghadimi et al., 2014).
For an objective function f which is convex, continuously differentiable, its gradient ∇ f ( · ) is Lipschitz continuous with constant L , our goal is to show that f (¯ θ T ) converges to the optimum f ∗ with decreasing momentum, where ¯ θ T is the average of θ t for t = 1 , ..., T .
Our following theorem holds for a constant learning rate and β t decaying with t .
Theorem 1.
Assume that f is convex, continuously differentiable, its gradient ∇ f ( · ) is Lipschitz continuous with constant L , with a decreasing momentum, but constant step size, as in:
[Equation]
We consider the CM iteration in non-stochastic settings, where:
[Equation]
Then, the sequence { θ t } Tt =1 generated by the CM iteration, with decreasing momentum, satisﬁes:
[Equation]
where ¯ θ T is the Cesaro average of the iterates:
[Equation] Proof. Let β t = 1 t · t +1 t +2 and [Equation]
We consider the CM iteration in non-stochastic settings, where:
[Equation]
Using the deﬁnition of p t above, one can easily prove that:
[Equation]
Using this expression
[Equation]
we will analyze the term (cid:107) θ
[Equation] [Equation] Since f is convex [Equation]
continuously differentiable
[Equation]
its gradient is Lipschitz continuous with constant L
[Equation]
then
[Equation]
Substituting the above inequalities leads to
[Equation]
where λ ∈ (
[Equation]
1] is a parameter weighting (2) and (3)
[Equation]
Grouping together terms yields
[Equation]
The last term is non-positive when α ∈ [0 , t +1 t +2 ( 2 − λL )] so it can be dropped.
Summing over t = 1 , ..., T yields
[Equation]
implying that:
[Equation]
Since:
[Equation]
we further have:
[Equation]
Due to the convexity of f ,
[Equation]
observe that
[Equation]
Since f ( θ 1 ) − f ∗ ≤ L 2 (cid:107) θ 1 − θ (cid:63) (cid:107) 2 by Lipschitz continuous gradients, setting λ = 1 and observing ( t + 1) / ( t + 2) ≥ 2 / 3 gives the result.
C A DDITIONAL PLOTS