O N THE CONVERGENCE OF A DAM AND B EYOND
A BSTRACT
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMS PROP , A DAM , A DADELTA ,
N ADAM are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients.
In many applications, e g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings).
We show that one cause for such failures is the exponential moving average used in the algorithms.
We provide an explicit example of a simple convex optimization setting where A DAM does not converge to the optimal solution and describe the precise problems with the previous analysis of A DAM algorithm.
Our analysis suggests that the convergence issues can be ﬁxed by endowing such algorithms with “long-term memory” of past gradients, and propose new variants of the A DAM algorithm which not only ﬁx the convergence issues but often also lead to improved empirical performance.
I NTRODUCTION
Stochastic gradient descent (S GD ) is the dominant method to train deep networks today.
This method iteratively updates the parameters of a model by moving them in the direction of the negative gradient of the loss evaluated on a minibatch.
In particular, variants of S GD that scale coordinates of the gradient by square roots of some form of averaging of the squared coordinates in the past gradients have been particularly successful, because they automatically adjust the learning rate on a per-feature basis.
The ﬁrst popular algorithm in this line of research is A DAGRAD (Duchi et al., 2011; McMahan & Streeter, 2010), which can achieve signiﬁcantly better performance compared to vanilla S GD when the gradients are sparse, or in general small.
Although A DAGRAD works well for sparse settings, its performance has been observed to deteriorate in settings where the loss functions are nonconvex and gradients are dense due to rapid decay of the learning rate in these settings since it uses all the past gradients in the update.
This problem is especially exacerbated in high dimensional problems arising in deep learning.
To tackle this issue, several variants of A DAGRAD , such as RMS PROP (Tieleman & Hinton, 2012), A DAM (Kingma & Ba, 2015), A DADELTA (Zeiler, 2012), N ADAM (Dozat, 2016), etc, have been proposed which mitigate the rapid decay of the learning rate using the exponential moving averages of squared past gradients, essentially limiting the reliance of the update to only the past few gradients.
While these algorithms have been successfully employed in several practical applications, they have also been observed to not converge in some other settings.
It has been typically observed that in these settings some minibatches provide large gradients but only quite rarely, and while these large gradients are quite informative, their inﬂuence dies out rather quickly due to the exponential averaging, thus leading to poor convergence.
In this paper, we analyze this situation in detail.
We rigorously prove that the intuition conveyed in the above paragraph is indeed correct; that limiting the reliance of the update on essentially only the past few gradients can indeed cause signiﬁcant convergence issues.
In particular, we make the following key contributions:
• We elucidate how the exponential moving average in the RMS PROP and A DAM algorithms can cause non-convergence by providing an example of simple convex optimization prob- lem where RMS PROP and A DAM provably do not converge to an optimal solution.
Our analysis easily extends to other algorithms using exponential moving averages such as A DADELTA and N ADAM as well, but we omit this for the sake of clarity.
In fact, the analysis is ﬂexible enough to extend to other algorithms that employ averaging squared gradients over essentially a ﬁxed size window (for exponential moving averages, the inﬂuences of gradients beyond a ﬁxed window size becomes negligibly small) in the immediate past.
We omit the general analysis in this paper for the sake of clarity.
• The above result indicates that in order to have guaranteed convergence the optimization algorithm must have “long-term memory” of past gradients.
Speciﬁcally, we point out a problem with the proof of convergence of the A DAM algorithm given by Kingma & Ba (2015).
To resolve this issue, we propose new variants of A DAM which rely on long-term memory of past gradients, but can be implemented in the same time and space requirements as the original A DAM algorithm.
We provide a convergence analysis for the new variants in the convex setting, based on the analysis of Kingma & Ba (2015), and show a datadependent regret bound similar to the one in A DAGRAD .
• We provide a preliminary empirical study of one of the variants we proposed and show that it either performs similarly, or better, on some commonly used problems in machine learning.
P RELIMINARIES
Notation.
We use S + d to denote the set of all positive deﬁnite d × d matrices.
With slight abuse of notation, for a vector a ∈ R d and a positive deﬁnite matrix M ∈ R d × R d , we use a/M to denote
M − 1 a , (cid:107) M i (cid:107) 2 to denote (cid:96) 2 -norm of i th row of M and √ M to represent M 1 / 2 .
Furthermore, for any vectors a, b ∈ R d , we use √ a for element-wise square root, a 2 for element-wise square, a/b to denote element-wise division and max( a, b ) to denote element-wise maximum.
For any vector θ i ∈ R d , θ i,j denotes its j th coordinate where j ∈ [ d ] .
The projection operation Π F ,A ( y ) for A ∈ S d
+ is deﬁned as arg min x ∈F (cid:107)
A 1 / 2 ( x − y )
(cid:107) for y ∈ R d .
Finally, we say F has bounded diameter D ∞ if (cid:107) x − y (cid:107) ∞ ≤ D ∞ for all x, y ∈ F .
Optimization setup.
A ﬂexible framework to analyze iterative optimization methods is the online optimization problem in the full information feedback setting.
In this online setup, at each time step t , the optimization algorithm picks a point (i.e. the parameters of the model to be learned) x t ∈ F , where F ∈ R d is the feasible set of points.
A loss function f t (to be interpreted as the loss of the model with the chosen parameters in the next minibatch) is then revealed, and the algorithm incurs loss f t ( x t ) .
The algorithm’s regret at the end of T rounds of this process is given by R T =
(cid:80)
Ti =1 f t ( x t ) − min x ∈F (cid:80)
Ti =1 f t ( x ) .
Throughout this paper, we assume that the feasible set F has bounded diameter and (cid:107)∇ f t ( x ) (cid:107) ∞ is bounded for all t ∈ [ T ] and x ∈ F .
Our aim to is to devise an algorithm that ensures R T
= o ( T ) , which implies that on average, the model’s performance converges to the optimal one.
The simplest algorithm for this setting is the standard online gradient descent algorithm (Zinkevich, 2003), which moves the point x t in the opposite direction of the gradient g t = ∇ f t ( x t ) while maintaining the feasibility by projecting onto the set F via the update rule x t
+1 =
Π F ( x t − α t g t ) , where Π F ( y ) denotes the projection of y ∈ R d onto the set F i.e., Π F ( y ) =
min x ∈F (cid:107) x − y (cid:107) , and α t is typically set to α/ √ t for some constant α .
The aforementioned online learning problem is closely related to the stochastic optimization problem: min x ∈F E z
[ f ( x, z )] , popularly referred to as empirical risk minimization (ERM), where z is a training example drawn training sample over which a model with parameters x is to be learned, and f ( x, z ) is the loss of the model with parameters x on the sample z .
In particular, an online optimization algorithm with vanishing average regret yields a stochastic optimization algorithm for the ERM problem (Cesa-Bianchi et al., 2004).
Thus, we use online gradient descent and stochastic gradient descent (S GD ) synonymously.
Generic adaptive methods setup.
We now provide a framework of adaptive methods that gives us insights into the differences between different adaptive methods and is useful for understanding the ﬂaws in a few popular adaptive methods.
Algorithm 1 provides a generic adaptive framework that encapsulates many popular adaptive methods.
Note the algorithm is still abstract because the
Algorithm
Input: x 1 ∈ F , step size { α t > 0 } Tt =1 , sequence of functions { φ t , ψ t } Tt =1 for t = 1 to T do g t = ∇ f t ( x t ) m t =
φ t ( g 1 , . . . , g t ) and V t =
ψ t ( g 1 , . . . , g t ) ˆ x t +1 =
x t − α t m t / √ V t x t +1 =
Π F , √ V t (ˆ x t +1 ) end for “averaging” functions φ t and ψ t have not been speciﬁed. Here φ t : F t → R d and ψ t : F t → S d + .
For ease of exposition, we refer to α t as step size and α t V − 1 / 2 t as learning rate of the algorithm and furthermore, restrict ourselves to diagonal variants of adaptive methods encapsulated by Algorithmwhere V t = diag ( v t ) .
We ﬁrst observe that standard stochastic gradient algorithm falls in this framework by using:
[Equation]
and α t = α/ √ t for all t ∈ [ T ] .
While the decreasing step size is required for convergence, such an aggressive decay of learning rate typically translates into poor empirical performance.
The key idea of adaptive methods is to choose averaging functions appropriately so as to entail good convergence.
For instance, the ﬁrst adaptive method A DAGRAD (Duchi et al., 2011), which propelled the research on adaptive methods, uses the following averaging functions:
[Equation]
and step size α t = α/ √ t for all t ∈
[ T ] .
In contrast to a learning rate of α/ √ t in S GD , such a setting effectively implies a modest learning rate decay of α/ (cid:113)(cid:80)
i g 2 i,j for j ∈ [ d ] .
When the gradients are sparse, this can potentially lead to huge gains in terms of convergence (see Duchi et al.
These gains have also been observed in practice for even few non-sparse settings.
Adaptive methods based on Exponential Moving Averages.
Exponential moving average variants of A DAGRAD are popular in the deep learning community.
RMS PROP , A DAM , N ADAM , and
A DADELTA are some prominent algorithms that fall in this category.
The key difference is to use an exponential moving average as function ψ t instead of the simple average function used in
A DAGRAD .
A DAM 1 , a particularly popular variant, uses the following averaging functions:
[Equation]
for some β 1 , β 2 ∈ [0 , 1) .
This update can alternatively be stated by the following simple recursion:
[Equation]
[Equation]
and m 0 ,i =
and v 0 ,i =
for all i ∈ [ d ] .
and t ∈ [ T ] .
A value of β 1 = 0 .
9 and β 2 = 0 .is typically recommended in practice.
We note the additional projection operation in Algorithmin comparison to A DAM .
When F = R d , the projection operation is an identity operation and this corresponds to the algorithm in (Kingma & Ba, 2015).
For theoretical analysis, one requires α t = 1 / √ t for t ∈ [ T ] , although, a more aggressive choice of constant step size seems to work well in practice.
RMS PROP , which appeared in an earlier unpublished work (Tieleman & Hinton, 2012) is essentially a variant of A DAM with β 1 = 0 .
In practice, especially in deep learning applications, the momentum term arising due to non-zero β 1 appears to signiﬁcantly boost the performance.
We will mainly focus on A DAM algorithm due to this generality but our arguments also apply to
RMS PROP and other algorithms such as A DADELTA , N ADAM .
T HE N ON -C ONVERGENCE OF A DAM
With the problem setup in the previous section, we discuss fundamental ﬂaw in the current exponential moving average methods like A DAM .
We show that A DAM can fail to converge to an optimal solution even in simple one-dimensional convex settings.
These examples of non-convergence contradict the claim of convergence in (Kingma & Ba, 2015), and the main issue lies in the following quantity of interest:
[Equation]
This quantity essentially measures the change in the inverse of learning rate of the adaptive method with respect to time.
One key observation is that for S GD and A DAGRAD , Γ t (cid:23) 0 for all t ∈ [ T ] .
This simply follows from update rules of S GD and A DAGRAD in the previous section.
In particular, update rules for these algorithms lead to “non-increasing” learning rates.
However, this is not necessarily the case for exponential moving average variants like A DAM and RMS PROP i.e., Γ t can potentially be indeﬁnite for t ∈
[ T ] .
We show that this violation of positive deﬁniteness can lead to undesirable convergence behavior for A DAM and RMS PROP .
Consider the following simple sequence of linear functions for F = [ − 1 , 1] :
[Equation]
where C > 2 .
For this function sequence, it is easy to see that the point x = − 1 provides the minimum regret.
Suppose β 1 = 0 and β 2 = 1 / (1 + C 2 ) .
We show that A DAM converges to a highly suboptimal solution of x = +1 for this setting.
Intuitively, the reasoning is as follows.
The algorithm obtains the large gradient C once every 3 steps, and while the other 2 steps it observes the gradient − 1 , which moves the algorithm in the wrong direction.
The large gradient C is unable to counteract this effect since it is scaled down by a factor of almost C for the given value of β 2 , and hence the algorithm converges to 1 rather than − 1 .
We formalize this intuition in the result below.
Theorem 1.
There is an online convex optimization problem where A DAM has non-zero average regret i.e., R T /T (cid:57) 0 as T → ∞ .
We relegate all proofs to the appendix.
A few remarks are in order.
One might wonder if adding a small constant in the denominator of the update helps in circumventing this problem i.e., the update for A DAM in Algorithm 1 of ˆ x t +1 is modiﬁed as follows:
[Equation]
The algorithm in (Kingma & Ba, 2015) uses such an update in practice, although their analysis does not.
In practice, selection of the (cid:15) parameter appears to be critical for the performance of the algorithm.
However, we show that for any constant (cid:15) > 0 , there exists an online optimization setting where, again, A DAM has non-zero average regret asymptotically (see Theorem 6 in Section F of the appendix).
The above examples of non-convergence are catastrophic insofar that A DAM and RMS PROP converge to a point that is worst amongst all points in the set [ − 1 , 1] .
Note that above example also holds for constant step size α t
= α .
Also note that classic S GD and A DAGRAD do not suffer from this problem and for these algorithms, average regret asymptotically goes to 0.
This problem is especially aggravated in high dimensional settings and when the variance of the gradients with respect to time is large.
This example also provides intuition for why large β 2 is advisable while using A DAM algorithm, and indeed in practice using large β 2 helps.
However the following result shows that for any constant β 1 and β 2 with β 1 < √ β 2 , we can design an example where A DAM has non-zero average rate asymptotically.
Theorem 2.
For any constant β 1 , β 2 ∈ [0 , 1) such that β 1 < √ β 2 , there is an online convex optimization problem where A DAM has non-zero average regret i.e., R T /T (cid:57) 0 as T → ∞ .
The above results show that with constant β 1 and β 2 , momentum or regularization via (cid:15) will not help in convergence of the algorithm to the optimal solution.
Note that the condition β 1 < √ β 2 is benign and is typically satisﬁed in the parameter settings used in practice.
Furthermore, such condition is assumed in convergence proof of Kingma & Ba (2015).
We can strengthen this result by providing a similar example of non-convergence even in the easier stochastic optimization setting: Input: x 1 ∈ , step size { t } Tt { β
} Tt
2 Set m to do m = β 1 t m t − 1 + t v t = β 2 v t − 1 + − g t ˆ v
− 1 , and x t +1 =
Π F
V t − t for Theorem 3. For any constant β 1 , β 2 ∈ [0 , 1) such that β 1 < √ β 2 , there is a stochastic convex optimization problem for which A DAM does not converge to the optimal solution.
These results have important consequences insofar that one has to use “problem-dependent” (cid:15), βand β 2 in order to avoid bad convergence behavior.
In high-dimensional problems, this typically amounts to using, unlike the update in Equation (3), a different (cid:15), β 1 and β 2 for each dimension.
However, this defeats the purpose of adaptive methods since it requires tuning a large set of parameters.
We would also like to emphasize that while the example of non-convergence is carefully constructed to demonstrate the problems in A DAM , it is not unrealistic to imagine scenarios where such an issue can at the very least slow down convergence.
We end this section with the following important remark.
While the results stated above use constant β 1 and β 2 , the analysis of A DAM in (Kingma & Ba, 2015)
actually relies on decreasing β 1 over time.
It is quite easy to extend our examples to the case where β 1 is decreased over time, since the critical parameter is β 2 rather than β 1 , and as long as β 2 is bounded away from 1 , our analysis goes through.
Thus for the sake of clarity, in this paper we only prove non-convergence of A DAM in the setting where β 1 is held constant.
A N EW E XPONENTIAL M OVING A VERAGE V ARIANT : AMSG RAD
In this section, we develop a new principled exponential moving average variant and provide its convergence analysis.
Our aim is to devise a new strategy with guaranteed convergence while preserving the practical beneﬁts of A DAM and RMS PROP .
To understand the design of our algorithms, let us revisit the quantity Γ t in (2).
For A DAM and RMS PROP , this quantity can potentially be negative.
The proof in the original paper of A DAM erroneously assumes that Γ t is positive semi-deﬁnite and is hence, incorrect (refer to Appendix D for more details).
For the ﬁrst part, we modify these algorithms to satisfy this additional constraint.
Later on, we also explore an alternative approach where Γ t can be made positive semi-deﬁnite by using values of β 1 and β 2 that change with t .
AMSG RAD uses a smaller learning rate in comparison to A DAM and yet incorporates the intuition of slowly decaying the effect of past gradients on the learning rate as long as Γ t is positive semideﬁnite.
Algorithm 2 presents the pseudocode for the algorithm.
The key difference of AMSG RAD with A DAM is that it maintains the maximum of all v t until the present time step and uses this maximum value for normalizing the running average of the gradient instead of v t in A DAM .
By doing this, AMSG RAD results in a non-increasing step size and avoids the pitfalls of A DAM and
RMS PROP i.e., Γ t (cid:23) 0 for all t ∈
[ T ] even with constant β 2 .
Also, in Algorithm 2, one typically uses a constant β 1 t in practice (although, the proof requires a decreasing schedule for proving convergence of the algorithm).
To gain more intuition for the updates of AMSG RAD , it is instructive to compare its update with
A DAM and A DAGRAD .
Suppose at particular time step t and coordinate
i ∈
[ d ] , we have v t − 1 ,i > g 2 t,i > 0 , then A DAM aggressively increases the learning rate, however, as we have seen in the previous section, this can be detrimental to the overall performance of the algorithm.
On the other hand, A DAGRAD slightly decreases the learning rate, which often leads to poor performance in practice since such an accumulation of gradients over a large time period can signiﬁcantly decrease the learning rate.
In contrast, AMSG RAD neither increases nor decreases the learning rate and furthermore, decreases v t which can potentially lead to non-decreasing learning rate even if gradient is large in the future iterations.
For rest of the paper, we use g 1: t =
[ g 1 . . .
g t ] to denote the matrix obtained by concatenating the gradient sequence.
We prove the following key result for AMSG RAD .
Theorem 4.
Let { x t } and { v t } be the sequences obtained from Algorithm 2, α t = α/ √ t , β 1 = β 11 , β 1 t ≤ β 1 for all t ∈
[ T ] and γ = β 1 / √ β 2 < 1 .
Assume that F has bounded diameter D ∞ and (cid:107)∇ f t ( x ) (cid:107) ∞ ≤ G ∞ for all t ∈
[ T ] and x ∈ F .
For x t generated using the AMSG RAD (Algorithm 2), we have the following bound on the regret
R
[Equation]
The following result falls as an immediate corollary of the above result.
Corollary 1.
Suppose β 1 t = β 1 λ t − 1 in Theorem 4, then we have
[Equation]
The above bound can be considerably better than O ( √ dT ) regret of S GD when (cid:80) di
=1 ˆ v 1 / 2 T,i (cid:28) √ d and (cid:80) di =1 (cid:107) g 1: T,i (cid:107)
(cid:28) √ dT (Duchi et al., 2011).
Furthermore, in Theorem 4, one can use a much more modest momentum decay of β 1 t = β 1 /t and still ensure a regret of O ( √ T ) .
We would also like to point out that one could consider taking a simple average of all the previous values of v t instead of their maximum.
The resulting algorithm is very similar to A DAGRAD except for normalization with smoothed gradients rather than actual gradients and can be shown to have similar convergence as A DAGRAD .
E XPERIMENTS
In this section, we present empirical results on both synthetic and real-world datasets.
For our experiments, we study the problem of multiclass classiﬁcation using logistic regression and neural networks, representing convex and nonconvex settings, respectively.
Synthetic Experiments : To demonstrate the convergence issue of A DAM , we ﬁrst consider the following simple convex setting inspired from our examples of non-convergence:
[Equation]
with the constraint set F = [ − 1 , 1] .
We ﬁrst observe that, similar to the examples of nonconvergence we have considered, the optimal solution is x = − 1 ; thus, for convergence, we expect the algorithms to converge to x = − 1 .
For this sequence of functions, we investigate the regret and the value of the iterate x t for A DAM and AMSG RAD .
To enable fair comparison, we set β 1 = 0 .and
β 2 = 0 .
99 for A DAM and AMSG RAD algorithm, which are typically the parameters settings used for A DAM in practice.
Figure 1 shows the average regret ( R t /t ) and value of the iterate ( x t ) for this problem.
We ﬁrst note that the average regret of A DAM does not converge to 0 with increasing t .
Furthermore, its iterates x t converge to x = 1 , which unfortunately has the largest regret amongst all points in the domain.
On the other hand, the average regret of AMSG RAD converges to 0 and its iterate converges to the optimal solution.
Figure 1 also shows the stochastic optimization setting: [Equation] probability
[Equation]
Similar to the aforementioned online setting, the optimal solution for this problem is x = − 1 .
Again, we see that the iterate x t of A DAM converges to the highly suboptimal solution x = 1 .
Logistic Regression : To investigate the performance of the algorithm on convex problems, we compare AMSG RAD with A DAM on logistic regression problem.
We use MNIST dataset for this experiment, the classiﬁcation is based on 784 dimensional image vector to one of the 10 class labels.
The step size parameter α t is set to α/ √ t for both A DAM and AMSG RAD in for our experiments, consistent with the theory.
We use a minibatch version of these algorithms with minibatch size set to 128 .
We set β 1 = 0 .
9 and β 2 is chosen from the set { 0 . 99 , 0 . 999 } , but they are ﬁxed throughout the experiment.
The parameters α and β 2 are chosen by grid search.
We report the train and test loss with respect to iterations in Figure 2.
We can see that AMSG RAD performs better than A DAM with respect to both train and test loss.
We also observed that AMSG RAD is relatively more robust to parameter changes in comparison to A DAM .
Neural Networks :
For our ﬁrst experiment, we trained a simple 1-hidden fully connected layer neural network for the multiclass classiﬁcation problem on MNIST.
Similar to the previous experiment, we use β 1 = 0 .
9 and β 2 is chosen from { 0 . 99 , 0 . 999 } .
We use a fully connected 100 rectiﬁed linear units (ReLU) as the hidden layer for this experiment.
Furthermore, we use constant α t = α throughout all our experiments on neural networks.
Such a parameter setting choice of A DAM is consistent with the ones typically used in the deep learning community for training neural networks.
A grid search is used to determine parameters that provides the best performance for the algorithm.
Finally, we consider the multiclass classiﬁcation problem on the standard CIFAR-10 dataset, which consists of 60,000 labeled examples of 32 × 32 images.
We use C IFARNET , a convolutional neural network (CNN) with several layers of convolution, pooling and non-linear units, for training a multiclass classifer for this problem.
In particular, this architecture has 2 convolutional layers with 64 channels and kernel size of 6 × 6 followed by 2 fully connected layers of size 384 and 192.
The network uses 2 × 2 max pooling and layer response normalization between the convolutional layers (Krizhevsky et al., 2012).
A dropout layer with keep probability of 0.5 is applied in between the fully connected layers (Srivastava et al., 2014).
The minibatch size is also set to 128 similar to previous experiments.
The results for this problem are reported in Figure 2.
The parameters for
A DAM and AMSG RAD are selected in a way similar to the previous experiments.
We can see that
AMSG RAD performs considerably better than A DAM on train loss and accuracy.
Furthermore, this performance gain also translates into good performance on test loss.
5. E XTENSION : A DAM N C ALGORITHM
An alternative approach is to use an increasing schedule of β 2 in A DAM .
This approach, unlike
Algorithm 2 does not require changing the structure of A DAM but rather uses a non-constant β 1 and β 2 .
The pseudocode for the algorithm, A DAM N C , is provided in the appendix (Algorithm 3).
We show that by appropriate selection of β 1 t and β 2 t , we can achieve good convergence rates.
Theorem 5.
Let { x t } and { v t } be the sequences obtained from Algorithm 3, α t = α/ √ t , β 1 = βand β 1 t ≤ β 1 for all t ∈
[ T ] .
Assume that F has bounded diameter D ∞ and (cid:107)∇ f t ( x ) (cid:107) ∞ ≤ G ∞ for all t ∈
[ T ] and x ∈ F .
Furthermore, let { β 2 t } be such that the following conditions are satisﬁed:
1 α T (cid:113)(cid:80) tj =1 Π t − j k
=1 β 2( t − k +1) (1 − β 2 j ) g 2 j,i ≥ 1 ζ (cid:113)(cid:80) tj =1 g 2 j,i for some ζ > 0 and all t ∈ [ T ] , j ∈
[ d ] . 2.
v 1 / 2 t,i α t ≥ v 1 / 2 t − 1 ,
i α t − 1 for all t ∈ { 2 , · · · , T } and i ∈ [ d ] .
Then for x t generated using the A DAM N C (Algorithm 3), we have the following bound on the regret
[Equation]
The above result assumes selection of { ( α t , β 2 t ) } such that Γ t (cid:23) 0 for all t ∈ { 2 , · · · , T } .
However, one can generalize the result to deal with the case where this constraint is violated as long as the violation is not too large or frequent.
Following is an immediate consequence of the above result.
Corollary 2.
Suppose β 1 t = β 1 λ t − 1 and β 2 t = 1 − 1 /t in Theorem 5, then we have
[Equation]
The above corollary follows from a trivial fact that v t,i = (cid:80) tj =1 g 2 j,i /t for all i ∈
[ d ] when β 2 t = 1 − 1 /t .
This corollary is interesting insofar that such a parameter setting effectively yields a momentum based variant of A DAGRAD .
Similar to A DAGRAD , the regret is data-dependent and can be considerably better than O ( √ dT ) regret of S GD when (cid:80) di
=1 (cid:107) g 1: T,i (cid:107)
(cid:28) √ dT (Duchi et al., 2011).
It is easy to generalize this result for setting similar settings of β 2 t .
Similar to Corollary 1, one can use a more modest decay of β 1 t = β 1 /t and still ensure a data-dependent regret of O ( √ T ) .
D ISCUSSION
In this paper, we study exponential moving variants of A DAGRAD and identify an important ﬂaw in these algorithms which can lead to undesirable convergence behavior.
We demonstrate these problems through carefully constructed examples where RMS PROP and A DAM converge to highly suboptimal solutions.
In general, any algorithm that relies on an essentially ﬁxed sized window of past gradients to scale the gradient updates will suffer from this problem.
We proposed ﬁxes to this problem by slightly modifying the algorithms, essentially endowing the algorithms with a long-term memory of past gradients.
These ﬁxes retain the good practical performance of the original algorithms, and in some cases actually show improvements.
The primary goal of this paper is to highlight the problems with popular exponential moving average variants of A DAGRAD from a theoretical perspective.
RMS PROP and A DAM have been immensely successful in development of several state-of-the-art solutions for a wide range of problems.
Thus, it is important to understand their behavior in a rigorous manner and be aware of potential pitfalls while using them in practice.
We believe this paper is a ﬁrst step in this direction and suggests good design principles for faster and better stochastic optimization.
A PPENDIX A P ROOF OF T HEOREM
Proof.
We consider the setting where f t are linear functions and F = [ − 1 , 1] .
In particular, we deﬁne the following function sequence:
[Equation]
where C ≥ 2 .
For this function sequence, it is easy to see that the point x = − 1 provides the minimum regret.
Without loss of generality, assume that the initial point is x 1 = 1 .
This can be assumed without any loss of generality because for any choice of initial point, we can always translate the coordinate system such that the initial point is x 1 = 1 in the new coordinate system and then choose the sequence of functions as above in the new coordinate system.
Also, since the problem is one-dimensional, we drop indices representing coordinates from all quantities in
Algorithm 1.
Consider the execution of A DAM algorithm for this sequence of functions with
[Equation]
and
[Equation]
where α < √ 1 − β 2 . Note that since gradients of these functions are bounded, F has bounded
L ∞ diameter and β 21 / √ β 2 < 1 .
Hence, the conditions on the parameters required for A DAM are satisﬁed (refer to (Kingma & Ba, 2015) for more details).
Our main claim is that for iterates { x t } ∞ t =1 arising from the updates of A DAM , we have x t > 0 for all t ∈ N and furthermore, x 3 t +1 = 1 for all t ∈ N ∪{ 0 } .
For proving this, we resort to the principle of mathematical induction.
Since x 1 = 1 , both the aforementioned conditions hold for the base case.
Suppose for some t ∈ N ∪ { 0 } , we have x i > 0 for all i ∈ [3 t + 1] and x 3 t +1 = 1 .
Our aim is to prove that x 3 t +2 and x 3 t +3 are positive and x 3 t +4 = 1 .
We ﬁrst observe that the gradients have the following form:
[Equation]
, for i mod 3 =− 1 , otherwise
From (3 t + 1) th update of A DAM in Equation (1), we obtain ˆ x 3 t
[Equation]
=
[Equation]
The equality follows from the induction hypothesis.
We observe the following:
[Equation]
The second inequality follows from the step size choice that α < √ 1 − β 2 .
Therefore, we have 0 <
ˆ x 3 t +2 < 1 and hence x 3 t +2 =
ˆ x 3 t +2 > 0 .
Furthermore, after the (3 t + 2) th and (3 t + 3) th updates of A DAM in Equation (1), we have the following:
[Equation]
Since x 3 t +2 > 0 , it is easy to see that x 3 t +3 > 0 .
To complete the proof, we need to show that x 3 t +4 = 1 .
In order to prove this claim, we show that ˆ x 3 t +4 ≥ 1 , which readily translates to x 3 t +4 = 1 because x 3 t +4 =
Π F (ˆ x 3 t +4 ) and F = [ − 1 , 1] here Π F is the simple Euclidean projection (note that in one-dimension, Π F , √ V t = Π F ).
We observe the following:
[Equation]
The above equality is due to the fact that ˆ x 3 t +3 > 0 and property of projection operation onto the set F = [ − 1 , 1] .
We consider the following two cases:
Suppose ˆ x 3 t +3 ≥ 1 , then it is easy to see from the above equality that ˆ x 3 t +4 > 1 . 2.
Suppose ˆ x 3 t +3 < 1 , then we have the following:
[Equation]
The third equality is due to the fact that x 3 t +2 =
ˆ x 3 t +2 .
Thus, to prove ˆ x 3 t +4 > 1 , it is enough to the prove:
[Equation]
We have the following bound on term T 1 from Equation (4):
[Equation]
Furthermore, we lower bound T 2 in the following manner:
[Equation]
The ﬁrst inequality is due to the fact that v t ≤ C 2 for all t ∈ N .
The last inequality follows from inequality in Equation (5).
The last equality is due to following fact:
[Equation]
for the choice of β 2 = 1 / (1 + C 2 ) .
Therefore, we have T 2 ≥ T 1 and hence, ˆ x 3 t +4 ≥ 1 .
Therefore, from both the cases, we see that x 3 t +4 = 1 .
Therefore, by the principle of mathematical induction it holds for all t ∈ N ∪ { 0 } .
Thus, we have f
[Equation]
Therefore, for every 3 steps, A DAM suffers a regret of at least 2 C − 4 .
More speciﬁcally, R T ≥ (2 C − 4) T/ 3 .
Since C ≥ 2 , this regret can be very large and furthermore, R T /T (cid:57) 0 as T → ∞ , which completes the proof.
B P ROOF OF T HEOREM
Proof.
The proof generalizes the optimization setting used in Theorem 1.
Throughout the proof, we assume β 1 < √ β 2 , which is also a condition (Kingma & Ba, 2015) assume in their paper.
In this proof, we consider the setting where f t are linear functions and F = [ − 1 , 1] .
In particular, we deﬁne the following function sequence:
[Equation]
where C ∈ N , C mod 2 = 0 satisﬁes the following:
[Equation]
where γ = β 1 / √ β 2 < 1 .
It is not hard to see that these conditions hold for large constant C that depends on β 1 and β 2 .
Since the problem is one-dimensional, we drop indices representing coordinates from all quantities in Algorithm 1.
For this function sequence, it is easy to see that the point x = − 1 provides the minimum regret since C ≥ 2 .
Furthermore, the gradients have the following form:
[Equation]
Our ﬁrst observation is that m kC ≤ 0 for all k ∈ N ∪ { 0 } .
For k = 0 , this holds trivially due to our initialization.
For the general case, observe the following:
[Equation]
If m kC ≤ 0 , it can be easily shown that m kC + C ≤ 0 for our selection of C in Equation (7) by using the principle of mathematical induction.
With this observation we continue to the main part of the proof.
Let T (cid:48) be such that t + C ≤ τ 2 t for all t ≥ T (cid:48) where τ ≤ 3 / 2 .
All our analysis focuses on iterations t ≥ T (cid:48) .
Note that any regret before T (cid:48) is just a constant because T (cid:48) is independent of T and thus, the average regret is negligible as T → ∞ .
Consider an iterate at time step t of the form kC after T (cid:48) .
Our claim is that
[Equation]
for some c t > 0 .
To see this, consider the updates of A DAM for the particular sequence of functions we considered are:
[Equation]
i ∈ { 2 , · · · , C }
[Equation]
For i ∈ { 2 , · · · , C } , we use the following notation:
[Equation]
Note that if δ t + j ≥ 0 for some j ∈ { 1 , · · · , C − 1 } then δ t + l ≥ 0 for all l ∈ { j, · · · , C − 1 } .
This follows from the fact that the gradient is negative for all time steps i ∈ { 2 , · · · , C } .
Using Lemmafor { x t +1 , · · · , x t + C } and { δ t , · · · , δ t + C − 1 } , we have the following:
[Equation]
Let i (cid:48) =
C/ 2 .
In order to prove our claim in Equation (10)
[Equation]
we need to prove the following:
[Equation]
To this end
[Equation]
we observe the following:
[Equation]
The ﬁrst equality follows from the deﬁnition of m t
+ i +1 .
The ﬁrst inequality follows from the fact that m t ≤ 0 when t mod C =
(see Equation (9) and arguments based on it).
The second inequality follows from the deﬁnition of τ that t + C ≤ τ 2 t for all t ≥ T (cid:48) .
The third inequality is due to the fact that v t + i − 1 ≥ (1 − β 2 ) β i −
C 2 .
The last inequality follows from our choice of C .
The fourth inequality is due to the following upper bound that applies for all i (cid:48) ≤ i ≤ C :
[Equation]
[Equation]
The ﬁrst inequality follows from online problem setting for the counter-example i.e., gradient is C once every C iterations and − 1 for the rest.
The last inequality follows from the fact that β i (cid:48) −
C 2 ≤
1 and β C 2 ≤ β 2 .
Furthermore, from the above inequality, we have
[Equation] [Equation]
Note that from our choice of C , it is easy to see that λ ≥ 0 .
Also, observe that λ is independent of t .
Thus, x t + C ≥ min { 1 , x t + λ/ √ t } .
From this fact, we also see the following:
If x t = 1 , then x t + C = 1 for all t ≥ T (cid:48) such that t mod C = 0 .
There exists constant T (cid:48) 1 ≥ T (cid:48) such that x T (cid:48) 1 = 1 where T (cid:48)
1 mod C = 0 .
The ﬁrst point simply follows from the relation x t + C ≥ min { 1 , x t + λ/ √ t } .
The second point is due to divergent nature of the sum (cid:80) ∞
t = t (cid:48) 1 / √ t .
Therefore, we have
[Equation]
where kC ≥ T (cid:48) 1 .
Thus, when t ≥ T (cid:48) 1 , for every C steps, A DAM suffers a regret of at least 2 .
More speciﬁcally, R T ≥ 2( T − T (cid:48) 1 ) /C .
Thus, R T /T (cid:57) 0 as T → ∞ , which completes the proof.
C P ROOF OF T HEOREM
Proof.
Let δ be an arbitrary small positive constant, and C be a large enough constant chosen as a function of β 1 , β 2 , δ that will be determined in the proof.
Consider the following one dimensional stochastic optimization setting over the domain [ − 1 , 1] .
At each time step t , the function f t ( x ) is chosen i.i.d.
as follows:
[Equation]
The expected function is F ( x ) = δx ; thus the optimum point over [ − 1 , 1] is x (cid:63) = − 1 .
At each time step t the gradient g t equals C with probability p and − 1 with probability 1 − p .
Thus, the step taken by A DAM is
[Equation]
We now show that for a large enough constant C , E [∆ t ] ≥ 0 , which implies that the A DAM ’s steps keep drifting away from the optimal solution x (cid:63) = − 1 .
Lemma 1.
For a large enough constant C (as a function of β 1 , β 2 , δ ), we have E [∆ t ] ≥ 0 .
Proof.
Let E t [ · ] denote expectation conditioned on all randomness up to and including time t − 1 .
Taking conditional expectation of the step, we have
[Equation]
We will bound the expectation of the terms T 1 , T 2 and T 3 above separately.
First, for T 1 , we have
[Equation]
Next, we bound E
[ T 2 ] .
Deﬁne k
=
(cid:100) log( C +1)
log(1 /β 1 ) (cid:101) .
This choice of k ensures that β k 1 C ≤ 1 − β k 1 .
Now, note that
[Equation]
Let E denote the event that for every i = t − 1 , t − 2 , . .
, max { t − k, 1 } , g i = − 1 .
Note that
Pr[ E ] ≥ 1 − kp .
Assuming E happens, we can bound m t − 1 as follows:
[Equation]
and so T 2 ≥ 0 .
With probability at most kp , the event E doesn’t happen.
In this case, we bound T 2 as follows.
We ﬁrst bound m t − 1 in terms of v t − 1 using the Cauchy-Schwarz inequality as follows:
[Equation]
·√
[Equation]
Thus, v t − 1 ≥ m 2 t − 1 /A 2 .
Thus, we have
[Equation]
Hence, we have
[Equation]
Finally, we lower bound E
[ T 3 ] using Jensen’s inequality applied to the convex function 1 √ x :
[Equation]
The last inequality follows by using the facts v t − 1 = (1 − β 2 ) (cid:80) t − 1 i =1 β t − 1 − ig 2 i , and the random variables g 21 , g 22 , . .
. , g 2 t − 1 are i.i.d., and so
[Equation]
Combining the bounds in (12), (13), and (14) in the expression for A DAM ’s step, (11), and plugging in the values of the parameters k and p we get the following lower bound on E [∆ t ] : −
[Equation]
It is evident that for C large enough (as a function of δ, β 1 , β 2 ), the above expression can be made non-negative.
For the sake of simplicity, let us assume, as is routinely done in practice, that we are using a version of A DAM that doesn’t perform any projection steps 2 .
Then the lemma implies that E [ x t +1 ] ≥ E [ x t ] .
Via a simple induction, we conclude that E [ x t ] ≥ x 1 for all t .
Thus, if we assume that the starting point x 1 ≥ 0 , then E [ x t ] ≥ 0 .
Since F is a monotonically increasing function, we have E
[ F ( x t )] ≥
F (0) = 0 , whereas F ( − 1) = − δ .
Thus the expected suboptimality gap is always δ > 0 , which implies that A DAM doesn’t converge to the optimal solution.
D P ROOF OF T HEOREM
The proof of Theorem 4 presented below is along the lines of the Theorem 4.1 in (Kingma &
Ba, 2015) which provides a claim of convergence for A DAM .
As our examples showing nonconvergence of A DAM indicate, the proof in (Kingma & Ba, 2015) has problems.
The main issue in their proof is the incorrect assumption that Γ t deﬁned in their equation (3) is positive semideﬁnite, and we also identiﬁed problems in lemmas 10.3 and 10.4 in their paper.
The following proof ﬁxes these issues and provides a proof of convergence for AMSG RAD .
Proof.
We begin with the following observation:
[Equation]
Furthermore, Π F , √ ˆ V t ( x ∗ ) =
x ∗ for all x ∗ ∈ F .
In this proof
[Equation]
we will use x ∗ i to denote the i th coordinate of x ∗ .
Using Lemma 4 with u 1 = x t +1 and u 2 = x ∗
[Equation]
we have the following:
[Equation]
Rearranging the above inequality
[Equation]
we have
[Equation]
The second inequality follows from simple application of Cauchy-Schwarz and Young’s inequality.
We now use the standard approach of bounding the regret at each step using convexity of the function f t in the following manner:
[Equation]
The ﬁrst inequality is due to convexity of function f t .
The second inequality follows from the bound in Equation (15).
For further bounding this inequality, we need the following intermediate result.
Lemma 2.
For the parameter settings and conditions assumed in Theorem
[Equation]
we have
[Equation]
Proof.
We start with the following:
[Equation]
The ﬁrst inequality follows from the deﬁnition of ˆ v T
[Equation]
i
[Equation]
which is maximum of all v T,i until the current time step.
The second inequality follows from the update rule of Algorithm 2.
We further bound the above inequality in the following manner:
[Equation]
The ﬁrst inequality follows from Cauchy-Schwarz inequality.
The second inequality is due to the fact that β 1 k ≤ β 1 for all k ∈ [ T ] .
The third inequality follows from the inequality (cid:80)
Tj =1 β T − j≤ 1 / (1 − β 1 ) .
By using similar upper bounds for all time steps, the quantity in Equation (17) can further be bounded as follows:
[Equation]
d (cid:88) i
= (cid:107) g 1: T,i (cid:107)
The third inequality follows from the fact that (cid:80)
Tj = t γ j − t ≤ 1 / (1 − γ ) .
The fourth inequality is due to simple application of Cauchy-Schwarz inequality.
The ﬁnal inequality is due to the following bound on harmonic sum: (cid:80) T t =1 1 /t ≤ (1 + log T ) .
This completes the proof of the lemma.
We now return to the proof of Theorem 4.
Using the above lemma in Equation (16) , we have:
T (cid:88)
[Equation]
The ﬁrst inequality and second inequality use the fact that β 1 t ≤ β 1 .
In order to further simplify the bound in Equation (18), we need to use telescopic sum.
We observe that, by deﬁnition of ˆ v t,i , we have
[Equation]
Using the L ∞ bound on the feasible region and making use of the above property in Equation (18), we have:
T (cid:88) t
= f t ( x t ) − f t ( x ∗ ) ≤ 2 α 1 (1 − β 1 ) d (cid:88) i
= ˆ v 1 / 2 1 ,i D 2 ∞ + 2(1 − β 1 )
T (cid:88) t = d (cid:88) i =
D 2 ∞ (cid:34) ˆ v 1 / 2 t,i α t − ˆ v 1 / 2 t − 1 ,i α t −(cid:35)
+ 2(1 − β 1 )
T (cid:88) t = d (cid:88) i =
D 2 ∞ β 1 t ˆ v 1 / 2 t,i α t + α √
+ log T (1 − β 1 ) 2 (1 − γ ) (cid:112) (1 − β 2 ) d (cid:88) i
= (cid:107) g 1: T,i (cid:107)=
D 2 ∞ 2 α T (1 − β 1 ) d (cid:88)
i
= ˆ v 1 / 2 T,i +
D 2 ∞ 2(1 − β 1 )
T (cid:88) t = d (cid:88) i =
β 1 t ˆ v 1 / 2 t,i α t + α √
+ log T (1 − β 1 ) 2 (1 − γ ) (cid:112) (1 − β 2 ) d (cid:88) i
= (cid:107) g 1: T,i (cid:107) 2 .
The equality follows from simple telescopic sum, which yields the desired result.
One important point to note here is that the regret of AMSG RAD can be bounded by O ( G ∞ √ T ) .
This can be easily seen from the proof of the aforementioned lemma where in the analysis the term (cid:80) Tt =1 | g t,i | / √ t can also be bounded by O ( G ∞ √ T ) .
Thus, the regret of AMSG RAD is upper bounded by minimum of
O ( G ∞ √ T ) and the bound in the Theorem 4 and therefore, the worst case dependence of regret on
T in our case is O ( √ T ) .
E P ROOF OF T HEOREM
Proof.
Using similar argument to proof of Theorem 4 until Equation (15), we have the following (cid:104) g t , x t − x ∗ (cid:105) ≤
[Equation]
The second inequality follows from simple application of Cauchy-Schwarz and Young’s inequality.
We now use the standard approach of bounding the regret at each step using convexity of the function f t in the following manner:
[Equation]
The inequalities follow due to convexity of function f t and Equation (19).
For further bounding this inequality, we need the following intermediate result
[Equation]
Lemma 3.
For the parameter settings and conditions assumed in Theorem
[Equation]
we have
[Equation]
Proof.
We start with the following:
[Equation]
The ﬁrst inequality follows from the update rule of Algorithm 2.
We further bound the above inequality in the following manner:
[Equation]
The ﬁrst inequality follows from Cauchy-Schwarz inequality.
The second inequality is due to the fact that β 1 k ≤ β 1 for all k ∈ [ T ] .
The third inequality follows from the inequality (cid:80)
Tj =1 β T − j≤ 1 / (1 − β 1 ) .
Using similar argument for all time steps, the quantity in Equation (21) can be bounded as follows:
[Equation]
The ﬁnal inequality is due to Lemma 5.
This completes the proof of the lemma.
Using the above lemma in Equation (20) , we have:
[Equation]
The ﬁrst inequality and second inequality use the fact that β 1 t ≤ β 1 .
Furthermore, from the theorem statement, we know that that { ( α t .β 2 t ) } are selected such that the following holds:
[Equation]
Using the L ∞ bound on the feasible region and making use of the above property in Equation (22), we have:
[Equation]
The equality follows from simple telescopic sum, which yields the desired result.
F P ROOF OF T HEOREM
Theorem 6.
For any (cid:15) > 0 , A DAM with the modiﬁed update in Equation (3) and with parameter setting such that all the conditions in (Kingma & Ba, 2015) are satisﬁed can have non-zero average regret i.e., R T /T (cid:57) 0 as T → ∞ for convex { f i } ∞ i =1 with bounded gradients on a feasible set F having bounded D ∞ diameter.
Proof.
Let us ﬁrst consider the case where (cid:15)
=
1 (in fact, the same setting works for any (cid:15) ≤ 1 ).
The general (cid:15) case can be proved by simply rescaling the sequence of functions by a factor of √ (cid:15) .
We show that the same optimization setting in Theorem 1 where f t are linear functions and F = [ − 1 , 1] , hence, we only discuss the details that differ from the proof of Theorem 1.
In particular, we deﬁne the following function sequence:
[Equation]
where C ≥ 2 .
Similar to the proof of Theorem 1, we assume that the initial point is x 1 = 1 and the parameters are:
[Equation]
where α < √ 1 − β 2 .
The proof essentially follows along the lines of that of Theorem 1 and is through principle of mathematical induction.
Our aim is to prove that x 3 t +2 and x 3 t +3 are positive and x 3 t +4 = 1 .
The base case holds trivially.
Suppose for some t ∈ N ∪ { 0 } , we have x i > 0 for all i ∈ [3 t +1] and x 3 t +1 = 1 .
For (3 t +1) th update, the only change from the update of in Equation (1) is the additional (cid:15) in the denominator i.e., we have
[Equation]
The last inequality follows by simply dropping v 3 t term and using the relation that α < √ 1 − β 2 .
Therefore, we have 0 <
ˆ x 3 t +2 < 1 and hence x 3 t +2 =
ˆ x 3 t +2 > 0 .
Furthermore, after the (3 t + 2) th and (3 t + 3) th updates of A DAM in Equation (1), we have the following:
[Equation]
Since x 3 t +2 > 0 , it is easy to see that x 3 t +3 > 0 .
To complete the proof, we need to show that x 3 t +4 = 1 .
The only change here from the proof of Theorem 1 is that we need to show the following:
[Equation]
The ﬁrst inequality is due to the fact that v t ≤ C 2 for all t ∈ N .
The last equality is due to following fact:
[Equation]
for the choice of β 2 = 2 / [(1 + C 2 ) C 2 ] and (cid:15) = 1 .
Therefore
[Equation]
we see that x 3 t +4 = 1 .
Therefore, by the principle of mathematical induction it holds for all t ∈ N ∪ { 0 } .
Thus
[Equation]
we have f
[Equation]
t +
[Equation]
+
[Equation] [Equation]
− 1) − f 3 t +2 ( − 1) − f 3 t +3 ( − 1) ≥ 2 C − 4 .
Therefore, for every 3 steps, A DAM suffers a regret of at least 2 C − 4 .
More speciﬁcally, R T ≥ (2 C − 4) T/ 3 .
Since C ≥ 2 , this regret can be very large and furthermore, R T /T (cid:57) 0 as T → ∞ , which completes the proof of the case where (cid:15) = 1 .
For the general (cid:15) case, we consider the following sequence of functions:
[Equation]
The functions are essentially rescaled in a manner so that the resultant updates of A DAM correspond to the one in the optimization setting described above.
Using essentially the same argument as above, it is easy to show that the regret R T ≥ (2 C − 4) √ (cid:15)T/ 3 and thus, the average regret is non-zero asymptotically, which completes the proof.
G A UXILIARY L EMMA
Lemma 4 ((McMahan & Streeter, 2010)) .
For any Q ∈ S d + and convex feasible set F ⊂ R d , suppose u 1 = min x ∈F (cid:107) Q 1 / 2 ( x − z 1 ) (cid:107) and u 2 = min x ∈F (cid:107) Q 1 / 2 ( x − z 2 ) (cid:107) then we have (cid:107) Q 1 / 2 ( u 1 − u 2 ) (cid:107) ≤ (cid:107) Q 1 / 2 ( z 1 − z 2 ) (cid:107) .
Proof.
We provide the proof here for completeness.
Since u 1 = min x ∈F (cid:107) Q 1 / 2 ( x − z 1 ) (cid:107) and u 2 = min x ∈F (cid:107) Q 1 / 2 ( x − z 2 ) (cid:107) and from the property of projection operator we have the following:
[Equation]
− z 1 ) (cid:105)
[Equation]
0 and (cid:104) z
[Equation]
u 2 , Q
[Equation]
z
[Equation]
z
[Equation]
(cid:105)
[Equation]
Combining the above inequalities, we have
[Equation]
Also, observe the following:
[Equation]
The above inequality can be obtained from the fact that (cid:104) ( u 2 − u 1 ) − ( z 2 − z 1 ) , Q (( u 2 − u 1 ) − ( z 2 − z 1 )) (cid:105) ≥ 0 as Q ∈ S d + and rearranging the terms.
Combining the above inequality with
Equation (24), we have the required result.
Lemma 5 ((Auer & Gentile, 2000)) .
For any non-negative real numbers y 1 , · · · , y t , the following holds:
[Equation]
Lemma 6.
Suppose F =
[ a, b ] for a, b ∈ R and
[Equation]
for all the t ∈ [ T ] , y 1 ∈ F and furthermore, there exists
i ∈
[ T ] such that δ j ≤ 0 for all j ≤ i and δ j > 0 for all j > i .
Then we have,
[Equation]
Proof.
It is ﬁrst easy to see that y i +1 ≥
y
+ (cid:80) ij =1 δ j since δ j ≤ 0 for all j ≤ i .
Furthermore, also observe that y T +1 ≥ min { b, y i +1 + (cid:80) T
j = i +1 δ j } since δ j ≥ 0 for all j > i .
Combining the above two inequalities gives us the desired result.