A GGREGATED M OMENTUM : S TABILITY T HROUGH P ASSIVE D AMPING
Anonymous authors
Paper under double-blind review
A BSTRACT
Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions.
Its performance depends crucially on a damping coefﬁcient β .
Large β values can potentially deliver much larger speedups, but are prone to oscillations and instability; hence one typically resorts to small values such as 0.5 or 0.9.
We propose Aggregated Momentum (AggMo) , a variant of momentum which combines multiple velocity vectors with different β parameters.
AggMo is trivial to implement, but signiﬁcantly dampens oscillations, enabling it to remain stable even for aggressive β values such as 0.999.
We reinterpret Nesterov’s accelerated gradient descent as a special case of AggMo and analyze rates of convergence for quadratic objectives.
Empirically, we ﬁnd that AggMo is a suitable drop-in replacement for other momentum methods, and frequently delivers faster convergence with little to no tuning.
Introduction
In spite of a wide range of modern optimization research, gradient descent with momentum and its variants remain the tool of choice in machine learning.
Momentum methods can help the optimizer pick up speed along low curvature directions without becoming unstable in high-curvature directions.
The simplest of these methods, classical momentum (Polyak, 1964), has an associated damping coefﬁcient, 0 ≤ β < 1 , which controls how quickly the momentum vector decays.
The choice of β imposes a tradoff between speed and stability: in directions where the gradient is small but consistent, the terminal velocity is proportional to 1 / (1 − β ) , suggesting that β slightly less than 1 could deliver much improved optimization performance.
However, large β values are prone to oscillations and instability (O’Donoghue & Candes, 2015; Goh, 2017), requiring a smaller learning rate and hence slower convergence.
Finding a way to dampen the oscillations while preserving the high terminal velocity of large beta values could dramatically speed up optimization.
Sutskever et al.
(2013) found that Nesterov accelerated gradient descent (Nesterov, 1983), which they reinterpreted as a momentum method, was more stable than classical momentum for large β values and gave substantial speedups for training neural networks.
However, the reasons for the improved performance remain somewhat mysterious.
O’Donoghue & Candes (2015) proposed to detect oscillations and eliminate them by resetting the velocity vector to zero.
But in practice it is difﬁcult to determine an appropriate restart condition.
In this work, we introduce Aggregated Momentum (AggMo) , a variant of classical momentum which maintains several velocity vectors with different β parameters.
AggMo averages the velocity vectors when updating the parameters.
We ﬁnd that this combines the advantages of both small and large β values: the large values allow signiﬁcant buildup of velocity along low curvature directions, while the small values dampen the oscillations, hence stabilizing the algorithm.
AggMo is trivial to implement and incurs almost no computational overhead.
We draw inspiration from the physics literature when we refer to our method as a form of passive damping .
Resonance occurs when a system is driven at speciﬁc frequencies but may be prevented through careful design (Goldstein, 2011).
Passive damping can address this in structures by making use of different materials with unique resonant frequencies.
This prevents any single frequency from producing catastrophic resonance.
By combining several momentum velocities together we achieve a similar effect — no single frequency is driving the system and so oscillation is prevented.
In this paper we analyze rates of convergence on quadratic functions.
We also provide theoretical convergence analysis showing that AggMo achieves converging average regret in online convex pro- gramming (Zinkevich, 2003).
To evaluate AggMo empirically we compare against other commonly used optimizers on a range of deep learning architectures: deep autoencoders, convolutional networks, and long-term short-term memory (LSTM).
In all of these cases, we ﬁnd that AggMo works as a drop-in replacement for classical momentum, in the sense that it works at least as well for a given β parameter.
But due to its stability at higher β values, it often delivers substantially faster convergence than both classical and Nesterov momentum when its maximum β value is tuned.
Background: momentum-based optimization
Classical momentum We consider a function f : R d → R to be minimized with respect to some variable θ .
Classical momentum (CM) minimizes this function by taking some initial point θ 0 and running the following iterative scheme,
[Equation]
where γ t denotes a learning rate schedule, β is the damping coefﬁcient and we set v 0 = 0 .
Momentum can speed up convergence but it is often difﬁcult to choose the right damping coefﬁcient, β .
Even with momentum, progress in a low curvature direction may be very slow.
If the damping coefﬁcient is increased to overcome this then high curvature directions may cause instability and oscillations.
Nesterov momentum Nesterov’s Accelerated Gradient (Nesterov, 1983; 2013) is a modiﬁed version of the gradient descent algorithm with improved convergence and stability.
It can be written as a momentum-based method (Sutskever et al., 2013),
[Equation]
Nesterov momentum seeks to solve stability issues by correcting the error made after moving in the direction of the velocity, v .
In fact, it can be shown that for a quadratic function Nesterov momentum adapts to the curvature by effectively rescaling the damping coefﬁcients by the eigenvalues of the quadratic (Sutskever et al., 2013).
Quadratic convergence We begin by studying convergence on quadratic functions, which have been an important test case for analyzing convergence behavior (Sutskever et al., 2013; O’Donoghue & Candes, 2015; Goh, 2017), and which can be considered a proxy for optimization behavior near a local minimum (O’Donoghue & Candes, 2015).
We analyze the behavior of these optimizers along the eigenvectors of a quadratic function in Figure 1.
In the legend, λ denotes the corresponding eigenvalue.
In (a) we use a low damping coefﬁcient ( β = 0 . 9 ) while (b) shows a high damping coefﬁcient ( β = 0 . 999 ).
When using a low damping coefﬁcient it takes many iterations to ﬁnd the optimal solution.
On the other hand, increasing the damping coefﬁcient from 0.9 to 0.999 causes oscillations which prevent convergence.
When using
CM in practice we seek the critical damping coefﬁcient which allows us to rapidly approach the optimum without becoming unstable (Goh, 2017).
On the other hand, Nesterov momentum with β = 0 .
999 is able to converge more quickly within high curvature regions than CM but retains oscillations for the quadratics exhibiting lower curvature.
Passive damping through Aggregated Momentum
Aggregated Momentum
We propose Aggregated Momentum (AggMo), a variant of gradient descent which aims to improve stability while providing the convergence beneﬁts of larger damping coefﬁcients.
We modify the gradient descent algorithm by including several velocity vectors each with their own damping coefﬁcient.
At each optimization step these velocities are updated and then averaged to produce the ﬁnal velocity used to update the parameters.
This updated iterative procedure can be written as follows,
[Figure] [Equation]
where v ( i )
= 0 for each i .
We refer to the vector β =
[ β (1) , . .
, β ( K ) ] as the damping vector .
By taking advantage of several damping coefﬁcients, AggMo is able to optimize well over illconditioned curvature.
Figure 1 (d) shows the optimization along the eigenvectors of a quadratic function using AggMo.
AggMo dampens oscillations quickly for all eigenvalues and converges faster than CM and Nesterov in this case.
In Figure 2 we display the AggMo velocities during optimization.
At point (1) the velocities are aligned towards the minima, with the β = 0 .
999 velocity contributing substantially more to each update.
By point (2) the system has begun to oscillate.
While the β = 0 .
999 velocity is still pointed away from the minima, the β = 0 .
9 velocity has changed direction and is damping the system.
Combining the velocities allows AggMo to achieve fast convergence while reducing the impact of oscillations caused by large β values.
3. Using AggMo
Choosing the damping vector Recall that in a direction with small but steady gradient, the terminal velocity is proportional to 1 / (1 − β ) .
We found that a good choice of damping vectors was therefore to space the terminal velocities exponentially.
To do so, we specify an exponential scale-factor, a , and a count K .
The damping vector is then constructed as β ( i ) = 1 − a
i − 1 , for i = 1 . . .
K .
We ﬁx a = 0 .
1 throughout and vary only K .
A good default choice is K = 3 which corresponds to β =
We found this setting to be both stable and effective in all of our experiments.
Computational/Memory overhead There is very little additional computational overhead when using AggMo compared to CM, as it only requires a handful of extra addition and multipliciation operations on top of the single gradient evaluation.
There is some memory overhead due to storing the
K velocity vectors, which are each the same size as the parameter vector.
However, for most modern deep learning applications, the memory cost at training time is dominated by the activations rather than the parameters (Gomez et al., 2017; Chen et al., 2016; Werbos, 1990; Hochreiter & Schmidhuber, 1997), so the overhead will generally be small.
Recovering Nesterov momentum
In this section we show that we can recover Nesterov Momentum (Equation 2) using a simple generalization of Aggregated Momentum (Equation 3).
We now introduce separate learning rates for each velocity, γ ( i ) , so that the iterate update step from Equation 3 is replaced with,
[Equation]
with each velocity updated as in Equation 3.
To recover Nesterov momentum we consider the special case of β = [0 , β ] and γ (1) t
[Equation]
=
[Equation]
βγ .
The AggMo update rule can now be written as,
[Equation]
[Equation]
Similarly, we may write the Nesterov momentum update with constant learning rate γ t = γ as,
[Equation]
Now we consider Equation 6 when using the reparameterization given by φ t = θ t
+ γβ [Equation] t
[Equation]
It follows that the update to φ from Nesterov is identical to the AggMo update to θ , and we have φ
= θ 0 .
We can think of the φ reparameterization as taking a half-step forward in the Nesterov optimization allowing us to directly compare the iterates at each time step.
We note also that if γ (1) t = γ (2) t = 2 γ then the equivalence holds approximately when β is sufﬁciently close to 1 .
We demonstrate this equivalence empirically in Appendix B.
This formulation allows us to reinterpret Nesterov momentum as a weighted average of a gradient update and a momentum update.
Moreover, by showing that AggMo recovers Nesterov momentum we gain access to the same theoretical convergence results that Nesterov momentum achieves.
Convergence analysis 5. Analyzing quadratic convergence
We can learn a great deal about optimizers by carefully reasoning about their convergence on quadratic functions.
O’Donoghue & Candes (2015) point out that in practice we do not know the condition number of the function to be optimized and so we aim to design algorithms which work well over a large possible range.
Sharing this motivation, we consider the convergence behaviour of momentum optimizers on quadratic functions with ﬁxed hyperparameters over a range of condition numbers.
To compute the convergence rate, || θ t − θ ∗ || 2 , we model each optimizer as a linear dynamical systems as in Lessard et al.
The convergence rate is then determined by the eigenvalues of this system.
We leave details of this computation to appendix B.
Figure 3 displays the convergence rate of each optimizer for quadratics with condition numbers ( κ ) from 10 1 to 10 7 .
The blue dashed line displays the optimal convergence rate achievable by CM with knowledge of the condition number — an unrealistic scenario in practice.
The two curves corresponding to CM (red and purple) each meet the optimal convergence rate when the condition number is such that β is critical.
On the left of this critical point, where the convergence rates for
CM are ﬂat, the system is ”under-damped” meaning there are complex eigenvalues corresponding to oscillations.
We observe that the convergence rate of AggMo interpolates smoothly between the convergence rates of CM with β = 0 .
9 and β = 0 .
99 as the condition number varies.
AggMo’s ability to quickly kill oscillations leads to an approximately three-times faster convergence rate than Nesterov momentum in the under-damped regime without sacriﬁcing performance on larger condition numbers.
We evaluate the convergence rate of AggMo in the setting of online convex programming, as proposed in Zinkevich (2003). This is an increasingly common setting to analyze optimization algorithms tailored to machine learning (Duchi et al., 2011; Kingma & Ba, 2014; Reddi et al., 2018). Notably, this is equivalent to analyzing the convergence rate in the setting of stochastic convex optimization.
We consider a sequence of unknown convex cost functions, f 1 ( θ ) , . . . , f T ( θ ) . At each time t , our goal is to predict the parameter θ t which minimizes the regret, [Equation]
where θ ∗ is the ﬁxed point parameter minimizing (cid:80)
Tt =1 f t ( θ ∗ ) .
We are able to show that AggMo has regret bounded by O ( √ T ) - a result asymptotically comparable to the best known bound (Duchi et al., 2011).
We adopt the following deﬁnitions from Duchi et al.
(2011) to simplify the notation.
We write g t = ∇ f t ( θ t ) with g t,i as the i th element of this vector.
Additionally, we write g 1: t,i ∈
R t as the vector containing the i th element of the gradient over the ﬁrst t iterations; g 1: t,i =
[ g 1 ,i , . . . , g t,i ] .
Then the following theorem holds,
Theorem 1.
Assume that f t has bounded gradients, ||∇ f t ( θ )
|| 2 < G, ||∇ f t ( θ )
|| ∞ <
G ∞ , ∀ θ ∈ R d .
Moreover, assume that each θ t generated by AggMo satisﬁes || θ n − θ m || 2 ≤ D, || θ n − θ m || ∞ ≤ D ∞ for all m, n ∈ { 1 , . .
, T } .
Let γ t = γ √ t and β ( i ) t
= β ( i ) λ t , λ ∈ (0 , 1) .
Then AggMo achieves the following regret bound, for all T ≥ 1 .
[Equation]
+ γ (cid:112)
[Equation]
log( T ) 2 K d (cid:88)
j = || g 1: T,j ||
K (cid:88) i = 1 + β ( i ) (1 − β ( i ) ) 2 +
D2 Kγ (1 − λ ) K (cid:88) i = β ( i ) . It immediately follows that the average regret of AggMo converges, i.e. that R ( T ) /T
→ 0 , by observing that
|| g 1: T,j || 24 ≤ G 2 ∞ √ T, ∀ j .
The full proof is given in Appendix C alongside some open questions on the convergence of AggMo.
While the statement of Theorem 1 requires strict assumptions we note that this result is certainly non-trivial.
Reddi et al.
(2018) showed that the average regret of Adam (Kingma & Ba, 2014) is not guaranteed to converge under the same assumptions.
Related work
The convergence of momentum methods has been studied extensively, both theoretically and empirically (Wibisono & Wilson, 2015; Wibisono et al., 2016; Wilson et al., 2016;
Kidambi et al., 2018).
By analyzing the failure modes of existing methods these works motivate successful momentum schemes.
Sutskever et al.
(2013) explored the effect of momentum on the optimization of neural networks and introduced the momentum view of Nesterov’s accelerated gradient.
They focused on producing good momentum schedules during optimization to adapt to ill-conditioned curvature.
Despite strong evidence that this approach works well, practitioners today still typically opt for a ﬁxed momentum schedule and vary the learning rate instead.
Adaptive gradient methods have been introduced to deal with the ill-conditioned curvature that we often observe in deep learning (Duchi et al., 2011; Kingma & Ba, 2014; Zeiler, 2012; Tieleman &
Hinton, 2012).
These methods typically approximate the local curvature of the objective to adapt to the geometry of the data.
Wilson et al.
(2017) highlight the inability of adaptive methods to generalize as well as classical momentum on deep learning tasks.
Natural gradient descent (Amari, 1998) preconditions by the Fisher information matrix, which can be shown to approximate the Hessian under certain assumptions (Martens, 2014).
Several methods have been proposed to reduce the computational and memory cost of this approach (Martens & Grosse, 2015; Martens, 2010) but these are difﬁcult to implement and introduce additional hyperparameters and computational overhead compared to SGD.
Another line of adaptive methods seeks to detect when oscillations occur during optimization.
O’Donoghue & Candes (2015) proposed using an adaptive restarting scheme to remove oscillations whenever they are detected.
In its simplest form, this is achieved by setting the momentum velocity to zero whenever the loss increases.
Further work has suggested using an adaptive momentum schedule instead of zeroing (Srinivasan et al., 2018).
Although this technique works well for well-conditioned convex problems it is difﬁcult to ﬁnd an appropriate restart condition for stochastic optimization where we do not have an accurate computation of the loss.
On the other hand, AggMo’s passive damping approach addresses the oscillation problem without the need to detect its occurrence.
Evaluation
We evaluated the AggMo optimizer on the following deep learning architectures; deep autoencoders, convolutional networks, and LSTMs.
To do so we used four datasets: MNIST (LeCun et al., 1998), CIFAR-10, CIFAR-100 (Krizhevsky & Hinton, 2009) and Penn Treebank (Marcus et al., 1993).
In each experiment we compared AggMo to classical momentum, Nesterov momentum, and
Adam.
These optimizers are by far the most commonly used and even today remain very difﬁcult to outperform in a wide range of tasks.
For each method, we performed a grid search over the learning rate and the damping coefﬁcient.
For AggMo, we keep the scale a = 0 .
1 ﬁxed and vary
K as discussed in Section 3.1.
Full details of the experimental set up for each task can be found in
Appendix D with additional results given in Appendix E.
For each of the following experiments we choose to report the validation and test performance of the network in addition to the ﬁnal training loss when it is meaningful to do so.
We include these generalization results because recent work has shown that the choice of optimizer may have a signiﬁcant effect on the generalization error of the network in practice (Wilson et al., 2017). [Table]
7. Autoencoders
We trained fully-connected autoencoders on the MNIST dataset using a set-up similar to that of
Sutskever et al.
While their work focused on ﬁnding an optimal momentum schedule we instead kept the momentum ﬁxed and applied a simple learning rate decay schedule.
For CM and
Nesterov we evaluated damping coefﬁcients in the range: { 0 . 0 , 0 . 9 , 0 . 99 , 0 . 999 } .
For Adam, it is standard to use β 1 = 0 .
9 and β 2 = 0 .
Since β 1 is analogous to the momentum damping parameter, we considered β 1 ∈ { 0 . 9 , 0 . 99 , 0 . 999 } and kept β 2 = 0 . 999 .
For AggMo, we explored
K in { 2,3,4 } .
Each model was trained for 1000 epochs.
We report the training, validation, and test errors in Table 1.
Results are displayed for the hyperparameters that achieved the best training loss and also for those that achieved the best validation loss.
While Adam is able to perform well on the training objective it is unable to match the performance of
AggMo or Nesterov on the validation/test sets.
AggMo achieves the best performance in all cases.
In these experiments the optimal damping coefﬁcient for both CM and Nesterov was β = 0 .
99 while the optimal damping vector for AggMo was β = [0 . 0 , 0 .
999] , given by K = 4 .
In Figurewe compare the convergence of each of the optimizers under the optimal hyperparameters for the training loss.
Increasing damping coefﬁcients During our experiments we observed that AggMo remains stable during optimization for learning rates an order of magnitude (or more) larger than is possible for CM and Nesterov with β equal to the max damping coefﬁcient used in AggMo.
We further investigated the effect of increasing the maximum damping coefﬁcient of AggMo in
Figure 5.
The learning rate is ﬁxed at 0.1 and we vary K from 2 to 5.
We compared to Nesterov with damping coefﬁcients in the same range (max of 0 . 9999 ) and a ﬁxed learning rate of 0.05 (to be consistent with our analysis in Section 4).
We do not include the curves for which training is unstable:
Nesterov with β ∈ { 0 . 999 , 0 . 9999 } and AggMo with K = 5 .
AggMo is able to take advantage of the larger damping coefﬁcient of 0.999 and achieves the fastest overall convergence.
7. Classiﬁcation
For the following experiments we evaluated AggMo using two network architectures: a neural network with 5 convolutional layers (CNN-5) and the ResNet-32 architecture (He et al., 2016).
We use data augmentation and regularization only for the latter.
Each model was trained for 400 epochs.
[Table]
For each optimizer we report the accuracy on a randomly held out validation set and the test set.
All of the models achieve near-perfect accuracy on the training set and so we do not report this.
The results are displayed in Table 2.
On the small convolutional network without regularization, AggMo signiﬁcantly out performed the other methods. For both of the ResNet-32 experiments we observed the best validation accuracy with CM.
This is perhaps expected as the model architecture and hyperparameters were likely to have been tuned using CM. Despite this, we observed that AggMo performed consistently well and had the fastest overall convergence.
We found that our proposed default hyperparameters for AggMo ( a = 0 . 1 , K = 3 ) led to much faster convergence than CM and Nesterov with β = 0 .
9 , a common default choice.
Figure 6 shows the training loss and validation accuracy during training for each optimizer used to train the ResNet32 model.
The hyperparameters used for each plot are those which obtained the best validation accuracy.
AggMo converged most quickly on the training objective without sacriﬁcing ﬁnal validation performance.
Surprisingly, we found that using AggMo we were also able to train the ResNet-32 architecture on CIFAR-100 without using batch normalization. With a limited search over learning rates we achieved 69.32% test error compared to a best value of 67.26% using CM.
We also found that, with batch normalization removed, optimization with AggMo remained stable at larger learning rates than with CM.
We note that the additional network hyperparameters (e.g. weight decay) are defaults which were likely picked as they work well with classical momentum.
This may disadvantage the other optimizers, including our own.
Despite this, we found that we are able to outperform CM with the AggMo and
Nesterov optimizers without additional tuning of any of these hyperparameters.
7. Language modeling
We trained LSTM Language Models on the Penn Treebank dataset.
We followed the experimental setup of Merity et al.
and made use of the code provided by the authors.
We used the optimal hyperparameter settings described by the authors and vary only the learning rate, momentum and whether gradient clipping is used.
The network hyperparameters were tuned using SGD and may not be optimal for the other optimizers we evaluate (including our own).
We followed only the base model training used in Merity et al.
and do not include the ﬁne-tuning and continuous cache optimization steps.
Each model was trained for 750 epochs.
As noted in Merity et al.
(2017), it is typically observed that SGD without momentum performs better than momentum-based methods in language modeling tasks.
However, in our experiments we [Table] observed all momentum-based optimizers but CM outperform SGD without momentum.
Surprisingly, we found that Adam is well-suited to this task and achieves the best training, validation [Table] and test performance.
We believe that the heavy regularization used when training the network makes Adam a good choice.
AggMo is very close in terms of ﬁnal performance to Adam.
Table 3 contains the results for the hyperparameter settings which achieved the best validation error for each optimizer.
The ﬁrst row (denoted *) uses the scheme suggested in Merity et al.
(2017): once the validation loss plateaus we switch to the ASGD (Polyak & Juditsky, 1992) optimizer.
The other rows instead decay the learning rate when the validation loss plateaus.
Figure 7 compares the convergence of the training and validation perplexity of each optimizer.
While the momentum methods converge after 300 epochs, the momentum-free methods converged much more slowly.
Surprisingly, we found that SGD worked best without any learning rate decay.
Adam converged most quickly and achieved a validation perplexity which is comparable to that of AggMo.
While gradient clipping is critical for SGD without momentum, which utilizes a large learning rate, we found that all of the momentum methods perform better without gradient clipping.
In short, while existing work encourages practitioners to avoid classical momentum we found that using other momentum methods may signiﬁcantly improve convergence rates and ﬁnal performance.
AggMo worked especially well on this task over a large range of damping coefﬁcients and learning rates.
Conclusion
Aggregated Momentum is a simple extension to classical momentum which is easy to implement and has negligible computational overhead on modern deep learning tasks.
We showed empirically that
AggMo is able to remain stable even with large damping coefﬁcients and enjoys faster convergence rates as a consequence of this.
Nesterov momentum can be viewed as a special case of AggMo.
(Incidentally, we found that despite its lack of adoption by deep learning practitioners, Nesterov momentum also showed substantial advantages compared to classical momentum.)
On the tasks we explored, AggMo could be used as a drop-in replacement for existing optimizers with little-to-no additional hyperparameter tuning.
But due to its stability at higher β values, it often delivered substantially faster convergence than both classical and Nesterov momentum.
Appendices A Nesterov Equivalence
In this section we demonstrate this equivalence on two toy problems.
In each of the ﬁgures included here we take β = 0 .
We ﬁrst consider a 2D quadratic function, f ( x ) =
x T A x , where A has eigenvalues 1.0 and 0.001.The learning rates for each optimizer are set as described in Section 4.
Each optimizer is initialized at the same position.
Figure 8 shows both optimizers following the same optimization trajectories.
In this setting, the two paths are also visually indistinguishable with γ (1) t
[Equation]
We now optimize the Rosenbrock function, given by,
[Equation]
This function has a global minimum at ( x, y )
= 1 .
Once again the optimizers are initialized at the same point but for this example we take γ (1) t
[Equation]
γ for AggMo.
Figure 9 shows the optimization trajectories of both algorithms.
In this case we see that the updates are initially indistinguishable but begin to differ as the algorithms approach the origin.
B Quadratic Convergence Analysis
In this section we present details of the convergence rate computations in Figure 3.
We also present some additional supporting results.
We ﬁrst note that for quadratic functions of the form f ( x ) =
1 2 x T
Ax + b T x we can write the AggMo optimization procedure as a linear dynamical systems in K + 1 variables:
[Equation]
x t − x ∗  
The spectral norm of the matrix B determines the rate at which the linear dynamical system converges and thus bounds || x t − x ∗
||
(Lessard et al., 2016).
We can write down the exact form of B as follows,
[Equation]

We note in particular that in the special case of K = 1 (CM) we recover the characteristic equation of O’Donoghue & Candes (2015):
[Equation]
Which in turn yields the critical damping coefﬁcient and optimal rate, with
[Equation]
When β < β ∗ the system is over-damped and exhibits slow monotone convergence (Figure 1 (a)).
When β > β ∗ the system is under-damped and the characteristic equation yields imaginary solutions that correspond to oscillations (Figure 1 (b)) with convergence rate equal to 1 − | β | .
At the critical damping coefﬁcient the convergence is optimal at 1 . 0 −
[Equation]
We can combine this analysis with Theorem 2 from Sutskever et al.
(2013) to recover similar convergence bounds for Nesterov momentum.
Producing Figure 3 To produce the curves in Figure 3 we compute the eigenvalues directly from the matrix B for matrices A with varying condition numbers.
While we can ﬁnd the optimal learning rate for CM and Nesterov momentum in closed form we have been unable to do so for AggMo.
Therefore, we instead perform a ﬁne-grained grid search to approximate the optimal learning rate for each condition number.
[Figure]
Studying Velocity We now present a brief study illustrating how using multiple velocities can break oscillations during optimization.
Figure 10 shows the optimization of a 1-D quadratic function with CM, Nesterov, and AggMo.
The shaded region around each curve represents the direction and relative magnitude of the velocities term during optimization.
CM (a) has a single velocity and oscillates at a near-constant amplitude.
For Nesterov momentum (b) we display the velocity and the ”error-correcting” term.
AggMo (c) has shaded regions for each velocity.
For AggMo, the velocity with β = 0 .
9 oscillates at a higher frequency and thus damps the whole system.
C Convergence Proof
Here we present the proof of Theorem 5 1 .
We introduce some simplifying notation used in Duchi et al.
We write g t = ∇ f ( θ t ) , with g t,i denoting the i th element of the vector g t .
We further write g 1: t,i ∈ R t for the i th dimension of gradients up to iteration t .
We begin with the following lemma,
Lemma 1.
We write v it,j to indicate the j th element of the i th velocity at time t .
Assume g t is bounded, then the following holds for all j ,
[Equation]
Proof We begin by expanding the
[Equation]
The ﬁrst inequality is obtained via Cauchy-Schwarz and by noting that β ( i ) t ≤ β for all t .
The second inequality follows directly from the fact that (cid:80)
Th =1 ( β ( i ) )
T − h < 1 / (1 − β ( i ) ) .
We can apply this upper bound to each term of the sum over t ,
[Equation] i ) ) [Equation]
i [Equation] = [Equation]
[Equation]
i ) [Equation] =
[Equation] [Equation]
[Equation]
h ≤ [Equation]
[Equation]
i ) [Equation] =
[Equation] i ) [Equation]
√ t [Equation] = [Equation]
[Equation]
= [Equation]
g [Equation]
(cid:118)(cid:117)(cid:117)(cid:116)
[-T (cid:88) t = g 2 t,j 1 √ t ≤ K (cid:88) i =  (1 − β ( i ) ) 2 || g 1: T,j ||(cid:118)(cid:117)(cid:117)(cid:116) T (cid:88) t =  t ≤ || g 1: T,j || 24 (cid:112) 
 + log( T )-][Equation]
(cid:88) [Equation] =  t [Equation]
Under equality we swap the order of sums and collect terms under g t .
The third inequality follows from (cid:80) tj =1 ( β ( i ) )
j − t < 1 / (1 − β ) .
The fourth inequality is an application of Cauchy-Schwarz.
The ﬁnal inequality is from the harmonic sum bound: (cid:80) Tt =1 1 /t ≤ 1 + log( T ) .
This completes the proof.
Proof of Theorem 1 From the update equations we may write,
[Equation]
We now shift focus to only the j th dimension.
We subtract θ ∗ j from both sides and square,
[Equation]
We can rearrange this expression and bound as follows,
[Equation]
The ﬁrst inequality is an application of Young’s inequality.
For the second inequality we use the sum-of-squares inequality.
We now make use of convexity, and take the sum over dimensions and time,
[Table]
We now make use of the bounding assumptions, || θ m − θ n || 2 ≤ D and || θ m − θ n || ∞ ≤ D ∞ ,
R ( T ) ≤ D 2 ∞ √ T γ + γ (cid:112)
[Equation]
i
[Equation]
[Equation]
D
[Equation]
γ
K
K (cid:88) i =
T (cid:88) t = β (
[Equation]
) λ t
[Equation]
t The ﬁrst two terms are collapsed using a telescoping sum.
Using (cid:80) t λ t − 1 √ t ≤ 1 / (1 − λ ) 2 , we achieve the following bound,
[Equation]
C. Open Questions on Convergence
While studying the convergence properties of AggMo we made several interesting observations which presented theoretical challenges.
We present some of these observations here to shed light on key differences between AggMo and existing momentum methods.
We hope that these will provoke further study.
Further reduction of B
In Appendix B we derived the matrix B in order to get bounds on the convergence.
We can further reduce B to block diagonal form, where the j th block takes the form,
B j =    β (1) · · · − λ j β (2) ...
− λ j [Equation] [Equation] j
[Equation]
K [Equation] K [Equation]
K
[Equation]
  
From this relatively simple form we may be able to derive a closed-form solution for the eigenvalues which would allow us to reason theoretically about the quadratic convergence properties of AggMo.
An easier goal would be ﬁnding suitable conditions under which the eigenvalues are complex and the system is under-damped.
Finite Difference Equation
In this section we demonstrate that the dynamics of AggMo can be written as a ( K + 1) -th order ﬁnite difference equation.
While most momentum methods can be viewed as the discretization of second order ODEs (Wilson et al., 2016)
it seems that AggMo does not fall into this class of algorithms.
As a consequence, it becomes difﬁcult to apply existing convergence proof techniques to AggMo.
For simplicity, we assume a ﬁxed learning rate γ for all time steps.
We will ﬁrst tackle the special case K = 2 .
From the AggMo update rule, we have
[Equation]
 (
[Equation]
Denoting the matrices as symbols correspondingly, it becomes v =
Bv − g , therefore
[Equation]
Denote δ t = θ t − θ (cid:63) , then θ t − θ t − 1 =
δ t − δ t − 1 .
Note that θ t +
=
θ t + γ t
+1 2 ( v (1) t +1 + v (2) t +1 ) , plugging
Eq 10 into it, we have
[Equation]
[Equation]
Which reduces to the following ﬁnite difference equation,
[Equation]
For K ≥ 2 , we only need to change Eq 9 accordingly, follow the remaining derivations, and recover a ( K +1)
-th order difference equation.
We could also derive the same result using sequence elimination, made simpler with some sensible variable substitutions.
This result is of considerable importance.
Existing momentum methods can generally be rewritten as a second order difference equation (Section 2 in O’Donoghue & Candes (2015)) which then induce a second order ODE (Su et al., 2014; Wibisono & Wilson, 2015).
The momentum optimization procedure can then be thought of as a discretization of a Hamiltonian ﬂow.
On the other hand, AggMo does not obviously lend itself to the analytical tools developed in this setting - it is not obvious whether the form in AggMo is indeed a discretization of a Hamiltonian ﬂow.
D Experiments
All of our experiments are conducted using the pytorch library Paszke et al.
In each experiment we make use of early stopping to determine the run with the best validation performance.
D. Autoencoders
For the autoencoders we train fully connected networks with encoders using the following architecture:
The decoder reverses this architecture.
We use relu activations throughout the network.
We train for a total of 1000 epochs using a multiplicative learning rate decay of 0. at
400, and 800 epochs.
We train using batch sizes of
For these experiments the training set consists of 90% of the training data with the remaining 10% being used for validation.
For each optimizer we searched over the following range of learning rates: { 0.1, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001, 0.00005, 0.00001 } .
D. Classiﬁcation
For each of the classiﬁcation tasks we train for a total of 400 epochs using batchsizes of 128.
We make use of a multiplicative learning rate decay of 0.1 at 150 and 250 epochs.
For each of these experiments we use 80% of the training data for training and use the remaining 20% as validation.
In these experiments we searched over the following learning rates for all optimizers: { 0.1, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001 } .
We searched over the same damping coefﬁcients as in the autoencoder experiments.
Each model was trained for a total of 500 epochs.
When training without batch normalization we explored a smaller range of learning rates for both
CM and AggMo: { 0.1, 0.05, 0.01, 0.005 } .
CNN-
The CNN-5 model uses relu activations throughout and 2x2 max pooling with stride 2.
The ﬁrst convolutional layer uses an 11x11 kernel with a stride of 4.
This is followed by a max pooling layer.
There is then a 5x5 convolutional kernel followed by max pooling.
The network then uses three 3x3 convolutional layers and a ﬁnal max pooling layer before feeding into a fully connected output layer.
We do not use any regularization when training this model.
ResNet-32 We use the ResNet-32 architecture on both CIFAR-10 and CIFAR-100.
We make use of a weight decay of 0.0005 and use batch normalization (Ioffe & Szegedy, 2015).
We introduce data augmentation by using random crops with a padding of 4 and use random horizontal ﬂips with probability 0.5.
D. LSTM Language Modelling
We train LSTMs with 3-layers containing 1150 hidden units per layer, and a 400 embedding size.
Within the network we use dropout on the layers with probability 0.4.
The hidden layers use dropout with probability 0.3 and the input embedding layers use dropout with probability 0.
while the embedding layer itself uses dropout with probability 0.1.
We also apply the weight drop method proposed in Merity et al.
(2017) with probability 0.5.
L2 regularization is applied on the
RNN activations with a scaling of 2.0, we also use temporal activation regularization (slowness regularization) with scaling 1.0.
Finally, all weights receive a weight decay of 1.2e-6.
We train the model using variable sequence lengths and batch sizes of 80.
We measure the validation loss during training and decrease the learning rate if the validation loss has not decreased forepochs.
We found that a learning rate decay of 0.5 worked best for all optimizers except for SGD which achieved best performance with a ﬁxed learning rate.
For SGD, CM, AggMo and Nesterov we searched over learning rates in the range { 50, 30, 10, 5, 2.5, 1, 0.1, 0.01 } .
We found that Adam required much smaller learning rates in this setting and so searched over values in the range { 0.1, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001 } .
We searched over the damping coefﬁcients as in the previous experiments.
Each model was trained for 750 epochs, as in Merity et al.
E Additional Results
In this section we display some of the experimental results which we are unable to ﬁt in the main paper.
E. Toy Problem
To better understand how AggMo is able to help in non-convex settings we explore its effectiveness on a simple non-convex toy problem.
The function we aim to optimize is deﬁned as follows,
[Equation]
[Table] where a and b are constants which may be varied.
We choose this function because it features ﬂat regions and a series of non-convex funnels with varied curvature.
The optimizer must traverse the ﬂat regions quickly whilst remaining stable within the funnels.
This function has an optimal value at ( x, y ) =
Figure 11 compares the performance of classical momentum and aggregated momentum when optimizing Equation 13 with a = 8 , b = 10 .
We see that GD with β = 0 and β = 0 .
9 are unable to leave the ﬂat region around x < − 1 .
For GD with β = 0 .
999 the optimizer enters the funnels but frequently becomes unstable with oscillations and ﬁnally overshoots the optimum.
Compared to
GD, AggMo is able to quickly traverse both the ﬂat region and the funnels while remaining stable.
AggMo also successfully slows down quickly once reaching the optimum.
E. Comparison at default damping settings
In this section we present results using the default damping coefﬁcient settings for the autoencoder and LSTM experiments.
The default settings for CM, Nesterov, and AggMo are compared in Table 4.
The default settings of AggMo outperform both CM and Nesterov signiﬁcantly.
Moreover, while the AggMo default settings perform similarly to the best results in Table 1 there is a large gap for the CM and Nesterov defaults.
This suggests that for this task AggMo is less sensitive to hyperparameter tuning than the other methods.
For the LSTM experiments we found that all methods worked best with their default damping coefﬁcients except for Nesterov momentum which used β = 0 . 99 .
For Nesterov momentum with β = 0 .
9 the validation perplexity was 63.67 and the test perplexity was 61.45.
AggMo with default settings achieved better training, validation and test perplexity than both the CM and Nesterov defaults.
F Beta-Averaged Momentum
In this section we present a continuous analog of AggMo which provides additional insight into its effectiveness.
The AggMo update rule features the average of several velocities with some chosen damping coefﬁcients, β .
A natural extension to this formulation instead considers a mapping from beta values to velocities with the space of velocities being integrated over instead of summed.
Explicitly, we write this update rule as,
[Equation]
Where π ( b ) is a probability density deﬁned on [0 , 1] .
We can link this back to aggregated momentum in the following way.
If we sampled b ( i ) under the density π for i = 1 : M then the procedure described by Equation 3 is approximating Equation 14 via Monte Carlo Integration.
Although this seems like a reasonable idea, it is not obvious whether we can compute this integral in closed form.
We can understand this update rule by expanding v t recursively,
[Equation]
[Equation]
Thus we can write the update rule for x t as,
[Equation]
Thus to compute the update rule we must compute the raw moments of b .
Fortunately, for the special case where π is the density function of a Beta distribution then we have closed form solutions for the raw moments of b ∼ Beta ( α, β ) (note that β here is not referring to a damping coefﬁcient) then these raw moments have a closed form:
[Equation]
This provides a closed form solution to compute θ t given θ t − 1 and the history of all previous gradients.
We refer to this update scheme as Beta-Averaged Momentum .
Unfortunately, each update requires the history of all previous gradients to be computed.
We may ﬁnd some reasonable approximation to the update rule.
For example, we could keep only the T most recently computed gradients.
Figure 12 shows the optimization of 1D quadratics using Beta-Averaged Momentum.
The trajectories are similar to those achieved using the original AggMo formulation.