META-STORM: G ENERALIZED F ULLY -A DAPTIVE V ARIANCE R EDUCED SGD FOR U NBOUNDED F UNCTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the application of variance reduction (VR) techniques to general nonconvex stochastic optimization problems.
In this setting, the recent work STORM (Cutkosky & Orabona, 2019) overcomes the drawback of having to compute gradients of “mega-batches” that earlier VR methods rely on.
There, STORM utilizes recursive momentum to achieve the VR effect and is then later made fully adaptive in STORM+ (Levy et al., 2021), where full-adaptivity removes the requirement for obtaining certain problem-speciﬁc parameters such as the smoothness of the objective and bounds on the variance and norm of the stochastic gradients in order to set the step size.
However, STORM+ crucially relies on the assumption that the function values are bounded, excluding a large class of useful functions.
In this work, we propose META-STORM, a generalized framework of STORM+ that removes this bounded function values assumption while still attaining the optimal convergence rate for non-convex optimization.
META-STORM not only maintains full-adaptivity, removing the need to obtain problem speciﬁc parameters, but also improves the convergence rate’s dependency on the problem parameters.
Furthermore, META-STORM can utilize a large range of parameter settings that subsumes previous methods allowing for more ﬂexibility in a wider range of settings.
Finally, we demonstrate the effectiveness of META-STORM through experiments across common deep learning tasks.
Our algorithm improves upon the previous work STORM+ and is competitive with widely used algorithms after the addition of per-coordinate update and exponential moving average heuristics.
I NTRODUCTION
In this paper, we consider the stochastic optimization problem in the form
[Equation]
(1) where F : R d → R is possibly non-convex.
We assume only access to a ﬁrst-order stochastic oracle via sample functions f
( x, ξ ) , where ξ comes from a distribution D representing the randomness in the sampling process.
Optimization problems of this form are ubiquitous in machine learning and deep learning.
Empirical risk minimization (ERM) is one instance, where F ( x ) is the loss function that can be evaluated by a sample or a minibatch represented by ξ .
An important advance in solving Problem (1) is the recent development of variance reduction (VR) techniques that improve the convergence rate to critical points of vanilla SGD from O (1 /T 1 / 4 ) to O (1 /T 1 / 3 )
(Fang et al., 2018; Li et al., 2021) for the class of mean-squared smooth functions (Arjevani et al., 2019).
In contrast to earlier VR algorithms which often require the computation of the gradients over large batches, recent methods such as Cutkosky & Orabona (2019); Levy et al.
; Huang et al.
(2021) avoid this drawback by using a weighted average of past gradients, often known as momentum .
When the weights are selected appropriately, momentum reduces the error in the gradient estimates which improves the convergence rate.
A different line of work on adaptive methods (Duchi et al., 2011; Kingma & Ba, 2014), some of which incorporate momentum techniques, have shown tremendous success in practice.
These adap-
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
tive methods remove the burden of obtaining certain problem-speciﬁc parameters, such as smoothness, in order to set the right step size to guarantee convergence.
STORM+ (Levy et al., 2021) is the ﬁrst algorithm to bridge the gap between fully-adaptive algorithms and VR methods, achieving the variance-reduced convergence rate of O (1 /T 1 / 3 ) while not requiring knowledge of any problemspeciﬁc parameter.
This is also the ﬁrst work to demonstrate the interplay between adaptive momentum and step sizes to adapt to the problem’s structure, while still achieving the VR rate.
However,
STORM+ relies on a strong assumption that the function values are bounded, which generally does not hold in practice.
Moreover, the convergence rate of STORM+ has high polynomial dependencies on the problem parameters, compared to what can be achieved by appropriately conﬁguring the step sizes and momentum parameters given knowledge of the problem parameters (see Section 3.1).
Our contributions: In this work, we propose META-STORM-SG and META-STORM, two ﬂexible algorithmic frameworks that attain the optimal variance-reduced convergence rate for general nonconvex objectives.
Both of them generalize STORM+ by allowing a wider range of parameter selection and removing the restrictive bounded function value assumption while maintaining its desirable fully-adaptive property – eliminating the need to obtain any problem-speciﬁc parameter.
These have been enabled via our novel analysis framework that also establishes a convergence rate with much better dependency on the problem parameters.
We present a comparison of META-STORM and its sibling META-STORM-SG against recent VR methods in Table 1.
In the appendix, we propose another algorithm, META-STORM-NA, with even less restrictive assumptions; however, with a tradeoff of losing the adaptivity to the variance parameter.
We complement our theoretical results with experiments across three common tasks: image classiﬁcation, masked language modeling, and sentiment analysis.
Our algorithms improve upon the previous work, STORM+.
Furthermore, the addition of heuristics such as exponential moving average and per-coordinate updates improves our algorithms’ generalization performance.
These versions of our algorithms are shown to be competitive with widely used algorithms such as Adam and AdamW.
1. R ELATED WORK
Variance reduction methods for stochastic non-convex optimization: Variance reduction is introduced for non-convex optimization by Allen-Zhu & Hazan (2016); Reddi et al.
in the context of ﬁnite sum optimization, achieving faster convergence over the full gradient descent method.
These methods are ﬁrst improved by Lei et al.
(2017) and later by Fang et al.
; Li et al.
(2021) both of which achieve an O (1 /T 1 / 3 ) convergence rate, matching the lower bounds in Arjevani et al.
However, these earlier methods periodically need to compute the full gradient (in the ﬁnitesum case) or a giant batch at a check point, which can be quite costly.
Shortly after, Cutkosky &
Orabona (2019) and Tran-Dinh et al.
(2019) introduce a different approach that utilizes stochastic gradients from previous time steps instead of computing the full gradient at a checkpoints.
These methods are framed as momentum-based methods as they are similar to using a weighted average of the gradient estimates to achieve the variance reduction.
Recently, SUPER-ADAM (Huang et al., 2021) integrates STORM in a larger framework of adaptive algorithms, but loses adaptivity to the variance parameter σ .
At the same time, STORM+ (Levy et al., 2021) proposes a fully adaptive version of STORM, which our work builds upon.
Adaptive methods for stochastic non-convex optimization: Classical methods, like SGD (Ghadimi & Lan, 2013), typically require the knowledge of problem parameters, such as the smoothness and the variance of the stochastic gradients, to set the step sizes.
In contrast, adaptive methods (Duchi et al., 2011; Tieleman et al., 2012; Kingma & Ba, 2014) forgo this requirement: their step sizes only rely on the stochastic gradients obtained by the algorithms.
Although these adaptive methods are originally designed for convex optimization, they enjoy great successes and popularity in highly non-convex practical applications such as training deep neural networks, often making them the method of choice in practice.
As a result, theoretical understanding of adaptive methods for non-convex problems has received signiﬁcant attention in recent years.
The works by Ward et al.
(2019); Kavis et al.
(2021) propose a convergence analysis of AdaGrad under various assumptions.
Among VR methods, STORM+ is the only fully adaptive algorithm that does not require knowledge of any problem parameter.
Our work builds on and generalizes STORM+, removing the bounded function value assumption while obtaining much better dependencies on the problem parameters.
1. P ROBLEM DEFINITION AND ASSUMPTIONS
We study stochastic non-convex optimization problems for which the objective function F : R d → R that has form F ( x ) :=
E ξ ∼D [ f ( x, ξ )] and f ( · , ξ ) is a sampling function depending on a random variable ξ drawn from a distribution D .
We will omit the writing of D in E ξ ∼D [ f ( x, ξ )] for simplicity in the remaining paper.
(cid:107) · (cid:107) represents (cid:107) · (cid:107) 2 for brevity.
[ T ] is deﬁned as { 1 , 2 , · · · , T } .
The analysis of our algorithms relies on the following assumptions 1–5:
Lower bounded function value : F ∗ := inf x ∈ R d F ( x ) >
−∞ . 2.
Unbiased estimator with bounded variance : We assume to have access to ∇ f ( x, ξ ) satisfying E ξ
[ ∇ f ( x, ξ )] = ∇ F ( x ) , E ξ (cid:2) (cid:107)∇ f ( x, ξ ) − ∇ F ( x ) (cid:107) 2 (cid:3) ≤ σ 2 for some σ ≥ 0 .
Averaged β -smoothness : E ξ (cid:2) (cid:107)∇ f ( x, ξ ) − ∇ f ( y, ξ ) (cid:107) 2 (cid:3) ≤ β 2 (cid:107) x − y (cid:107) 2 , ∀ x, y ∈ R d . 4.
Bounded stochastic gradients: (cid:107)∇ f ( x, ξ ) (cid:107) ≤ (cid:98) G, ∀ x ∈ R d , ξ ∈ support ( D ) for some (cid:98) G ≥
Bounded stochastic gradient differences :
(cid:107)∇ f ( x, ξ ) − ∇ f ( x, ξ (cid:48) ) (cid:107) ≤ 2 (cid:98) σ, ∀ x ∈ R d , , ξ (cid:48) ∈
support ( D ) for some (cid:98) σ ≥ 0 .
Assumptions 1, 2 and 3 are standard in the VR setting (Arjevani et al., 2019).
Assumption 5 is weaker than the assumptions made in the prior works based on the STORM framework (Cutkosky & Orabona, 2019; Levy et al., 2021).
These works assume that the stochastic gradients are bounded, i.e., Assumption 4.
We note that assumption 4 implies that assumption 5 holds by replacing (cid:98) σ by (cid:98) G , thus we only have to consider (cid:98) σ =
O ( (cid:98) G ) .
To better understand assumption 5, we ﬁx ξ ∈ support ( D ) and consider another ξ (cid:48) ∼ D , then due to the convexity of (cid:107)·(cid:107) , (cid:107)∇ f ( x, ξ ) −∇ F ( x ) (cid:107)
=
(cid:107)∇ f ( x, ξ ) − E ξ (cid:48) [ ∇ f ( x, ξ (cid:48) )]
(cid:107) ≤ E ξ (cid:48)
[ (cid:107)∇ f ( x, ξ ) − ∇ f ( x, ξ ) (cid:48) (cid:107) ] ≤ 2 (cid:98) σ .
This means assumptionimplies a stronger version of assumption 2.
For this reason, we can consider σ =
O ( (cid:98) σ ) .
Input: Initial point x 1 ∈ R d
Parameters: a 0 , b 0 , η, p ∈ [ 14 , 12 ] , p + 2 q =
Sample ξ 1 ∼ D , d 1 = ∇ f ( x 1 , ξ 1 ) for t = 1 , · · · , T do: a t +1 = (cid:16) 1 + (cid:80) ti =1 (cid:107)∇ f
( x i ,ξ i ) (cid:107) 2 a(cid:17) −b t = ( b 1 /p+ (cid:80) ti =1 (cid:107)
d i (cid:107) 2 ) p /a
qt + x t +1 =
x t − ηb t d t
Sample ξ t +1 ∼ D d t +1 = ∇ f ( x t +1 , ξ t +1 )
+ (1 − a t +1 )
( d t − ∇ f ( x t , ξ t +1 )) end for
Output x out = x t where t ∼ Uniform ([ T ]) .
Algorithm
Input: Initial point x 1 ∈ R d
Parameters: a 0 , b 0 , η, p ∈ [ 3 −√ 7 2 , 12 ] , p + 2 q =
Sample ξ 1 ∼ D , d 1 = ∇ f ( x 1 , ξ 1 ) , a 1 =for t = 1 , · · · , T do: b t = ( b 1 /p+ (cid:80) ti =1 (cid:107)
d i (cid:107) 2 ) p /a
qt x t +1 =
x t − ηb t d t
Sample ξ t +1 ∼ D a t +1 = (cid:16) 1 + (cid:80) ti =1 (cid:107)∇ f ( x i ,ξ i ) −∇ f
( x i ,ξ i +1 ) (cid:107) 2 a(cid:17) −d t +1 = ∇ f
( x t +1 , ξ t +1 )
+ (1 − a t +1 )
( d t − ∇ f ( x t , ξ t +1 )) end for Output x out = x t where t ∼ Uniform ([ T ]) .
Additional assumptions made in the prior works (Cutkosky & Orabona, 2019; Levy et al., 2021;
Huang et al., 2021) include the following: 3’
Almost surely β -smooth : (cid:107)∇
[Equation]
ξ ∈ support ( D ) . Bounded function values: There exists B ≥ 0 such that | F ( x ) − F ( y ) | ≤ B for all x, y ∈ R d .
We remark that 3’ is strictly stronger than 3 and it is NOT a standard assumption in Arjevani et al.
Moreover, assumption 6, which plays a critical role in the analysis of Levy et al.
(2021), is relatively strong and cannot be always satisﬁed in non-convex optimization.
Our work removes these two restrictive assumptions and also improves the dependency on the problem parameters.
O UR ALGORITHMS
In this section, we introduce our two main algorithms, META-STORM-SG and META-STORM, shown in Algorithm 1 and Algorithm 2 respectively.
Our algorithms follow the generic framework of momentum-based variance-reduced SGD put forward by STORM (Cutkosky & Orabona, 2019).
The STORM template incorporates momentum and variance reduction as follows:
[Equation]
The ﬁrst variant, META-STORM-SG, similar to prior works, uses the gradient norms when setting a t and similarly, requires the strong assumption on the boundedness of the stochastic gradients.
The major difference lies in the structure of the momentum parameters and the step sizes and their relationship, which is further developed in the second algorithm META-STORM so that assumption 4 can be relaxed to assumption 5.
We now highlight our key algorithmic contributions and how they depart from prior works.
A ﬁrst point of departure is our use of stochastic gradient differences when setting the momentum parameter a t in META-STORM: prior works set a t based on the stochastic gradients, while
META-STORM sets a t based on the difference of two gradient estimators taken at two different time step ξ t − 1 and ξ t at the same point x t − 1 .
The gradient difference can be viewed as a proxy for the variance σ 2 , which allows us to require the mild assumption 5 in the analysis.
With this choice, our algorithm obtains the best dependency on the problem parameters.
On the other hand, the coefﬁcient 1 − a t +1 in the update for d t +1 now depends on ξ t +1 , and addressing this correlation requires a more careful analysis.
The second point of departure is the setting of the step sizes b t and their relationship to the momentum parameters a t in both META-STORM-SG and META-STORM.
We propose a general update rule b t = ( b 1 /p+ (cid:80) ti =1 (cid:107)
d i (cid:107) 2 ) p /a qt that allows for a broad range of choices for p and q that subsume prior works.
In practice, different problem domains may beneﬁt from different choices of p and q .
Our framework allows us to capture prior works such as the STORM+ update b t = ( (cid:80) ti =1 (cid:107) d i (cid:107) 2 /a
i +1 ) 1 / 3 using a different but related choice of momentum parameters and a simpler update that uses only the current momentum value a t instead of all the previous momentum values a i +1 with i ≤ t .
We further motivate and provide intuition for our algorithmic choices in Section 3.
We note that our algorithm uses only the stochastic gradient information received, and it does not require any knowledge of the problem parameters.
We provide an overview and intuition for our algorithm in Section 3, and give the complete analysis in the appendix.
Our analysis departs signiﬁcantly from prior works such as STORM+, and it allows us to forgo the bounded function value assumption and improve the convergence rate’s dependency on the problem parameters.
It remains an interesting open question to determine the best convergence rate that can be achieved when the function values are bounded.
We can further alleviate assumption 5 in another new algorithm, META-STORM-NA (Algorithm 4), provided in Section H in the appendix.
To the best of our knowledge, META-STORM-NA is the only adaptive algorithm that enjoys the convergence rate (cid:101) O
(1 /T 1 / 3 ) under only the weakest assumptions 1-3.
It also allows a wide range of choices for p ∈ (cid:0) 0 , 12 (cid:3) .
However, the tradeoff is that the algorithm does not adapt to the variance parameter σ .
For the detailed analysis, we refer readers to Section H.
Finally, we show the convergence rate obtained by Algorithms 1 and 2 in the following theorems.
The convergence rates for general p are given in the appendix.
Theorem 2.1.
Under the assumptions 1-4 in Section 1.2, with the choice p = 12 and setting a 0 = b 0 = η = 1 to simplify the ﬁnal bound, META-STORM-SG ensures that
E
[Equation] [Equation]
W
[Equation]
{+where W 1 = O (cid:0) F ( x 1 ) − F ∗ + σ 
+}
+ (cid:98) G 2 + β
[Equation]
1+
[Equation]
G
[Equation]
log (cid:0) β +
[Equation]
G 2 β
[Equation]
, W 2 = O (cid:0) ( F ( x 1 ) − F ∗ ) 2 / 3 +
[Equation]
W
[Equation]
O
[Equation]
We note that when σ 2 > 0 and T is large enough, the effect of W 1 can be eliminated.
Combining
Theorem 2.1 and Markov’s inequality, we immediately have the following corollary.
Corollary 2.2.
Under the same setting in Theorem 2.1, additionally we assume σ 2 >
and T is large enough, then for any 0 < δ < 1 , with probability 1 − δ
(cid:107)∇ F ( x out ) (cid:107) ≤ O (cid:16) κ 1 + κ 2 log (cid:0)
[Equation]
+ σ 2 T (cid:1) δ 3 /(cid:16)
[Equation]
[Equation]
(cid:17)(cid:17) where κ 1 = O (cid:0) F ( x 1 ) − F ∗ + σ
+ (cid:98) G
+ κ 2 log κ 2 (cid:1) and κ 2 =
O (cid:0)(cid:0)
+ (cid:98)
[Equation]
β
[Equation]
Theorem 2.3.
Under the assumptions 1–3 and 5 in Section 1.2, with the choice p = 12 and setting a 0 = b 0 = η = 1 to simplify the ﬁnal bound, META-STORM ensures that
[Equation]
where Q 1 = O (cid:0)(cid:0) F ( x 1 ) − F ∗ (cid:1) 6 / 7 + (cid:0)(cid:98) σσ (cid:1) 6 / 7 + σ 12 / 7 + (cid:98) σ 18 / 7 + (cid:0) 1 + (cid:98) σ 18 / 7 (cid:1) β 6 / 7 log 6 / 7 (cid:0) β + (cid:98) σ 3 β (cid:1) and Q 2 =
O (cid:0)(cid:0)
+ (cid:98) σ 18 / 7 (cid:1) β 6 / 7 (cid:1) .
Combining Theorem 2.3 and Markov’s inequality, we also have the following corollary.
Corollary 2.4.
Under the same setting in Theorem 2.3, then, for any 0 < δ < 1 , with probability 1 − δ
[Equation]
where κ 1 = O (cid:0) F ( x 1 ) − F ∗ + (cid:98)
[Equation]
+
[Equation]
σ
[Equation]
+ κ [Equation] [Equation] and κ {+
+}
= O (cid:0)(cid:0)
+ (cid:98) σ 3 (cid:1) β (cid:1) .
We emphasize that the aim of our analysis is to provide a convergence in expectation or with constant probability.
In particular, we state Corollaries 2.2 and 2.4 only to give a more intuitive way to see the dependency on the problem parameters.
To boost the success probability and achieve a log 1 δ dependency on the probability margin, a common approach is to perform log
δ independent repetitions of the algorithms.
We brieﬂy discuss the difference between the convergence rate of the two algorithms.
We note that these two rates cannot be compared directly since assumption 4 is stronger than assumption 5.
Additionally, as pointed out in Section 1.2, we have (cid:98) σ =
O ( (cid:98) G ) and thus the term O ( (cid:98) σ 3 ) in Corollary 2.4 is O
( (cid:98) G 3 ) , whereas Corollary 2.2 has a O ( (cid:98) G 2 ) term.
To give an intuition why an extra higher order term W 1 appears in Theorem 2.1 when σ = 0 compared with Theorem 2.3, we note that when σ = 0 , d t in both algorithms degenerates to ∇ F ( x t ) .
However, the coefﬁcient a t +1 becomes 1 in
META-STORM but does not in META-STORM-SG.
This discrepancy leads to b t being larger in
META-STORM-SG than in META-STORM, and moreover the META-STORM b t becomes exactly the same as the stepsize used in AdaGrad.
Due to the larger b t when σ = 0 , it is reasonable to expect a slower convergence rate for META-STORM-SG.
The appearance of the term W 1 reﬂects that.
O VERVIEW OF MAIN IDEAS AND ANALYSIS
In this section, we an overview of our novel analysis framework.
We ﬁrst give a basic non-adaptive algorithm and its analysis to motivate the algorithmic choices made by our adaptive algorithms.
We then discuss how to turn the non-adaptive algorithm into an adaptive one.
Section D in the appendix gives a proof sketch for Theorem 2.3 for the special case p = 1 2 that illustrates the main ideas used in the analyses of all of our algorithms.
We give the complete analyses in the appendix.
3. N ON - ADAPTIVE ALGORITHM
As a warm-up towards our fully adaptive algorithms and their analysis, we start with a basic nonadaptive algorithm and analysis that will guide our algorithmic choices and provide intuition for our analysis.
The algorithm instantiates the STORM template using ﬁxed choices a t = a and b t /η =
b for the momentum and step size.
In the following, we outline an analysis for the algorithm and derive appropriate choices for the values a and b .
Algorithm :
As noted above, the algorithm performs the following updates:
[Equation]
To make it simpler, we assume d 1 = ∇ F ( x 1 ) .
Alternatively, one can use a standard mini-batch setting to set d 1 = 1 m (cid:80) mi =1 ∇ f ( x 1 ; ξ i ) with a proper m leading to small variance as in previous non-adaptive analysis (Fang et al., 2018;
Zhou et al., 2018; Tran-Dinh et al., 2019).
Key idea: We start by introducing some convenient notation.
Let (cid:15) t =
d t −∇ F ( x t ) be the stochastic error (in particular, (cid:15) 1 = 0 ) and
[Equation]
First, to bound E [ (cid:107)∇ F ( x out ) (cid:107) ]
where x out is an iterate chosen uniformly at random, it sufﬁces to upper bound E [ H T ] .
Then, we can translate this term to a convergence guarantee for E
[ (cid:107)∇ F ( x out ) (cid:107) ] .
An important intuition from STORM/STORM+ is the incorporation of VR in (2), leading to a decrease over time of the error term (cid:15) t .
Thus, we can view d t as a proxy for ∇ F ( x t ) .
It is then natural to decompose H T in terms of D T and E T .
By the deﬁnition of (cid:15) t , we can write H T ≤ 2 D T +2 E T .
Therefore, to upper bound E [ H T ] , it sufﬁces to upper bound E [ D T ] and E [ E T ] , which will be the essential steps in the analysis framework.
A key insight is that E [ D T ] and E [ E T ] can be upper bounded in terms of each other, as we now show.
Bounding D T : Starting from the function value analysis, using smoothness, the update rule x t +
= x
t − 1 b d t , the deﬁnition of (cid:15) t =
d t − ∇ F ( x t ) , and Cauchy-Schwarz, we obtain
[Equation]
Suppose that we choose b so that b ≥ 2 β , which ensures that β 2 b 2 ≤ 14 b .
By rearranging the previous inequality, summing up over all iterations, and taking expectation, we obtain
[Equation]
Bounding E T :
By the standard calculation for the stochastic error (cid:15) t used in STORM, we have
[Equation]
Summing up over all iterations, rearranging, and using that a ∈ [0 , 1] and (cid:15) 1 = 0 , we obtain
E
[Equation]
By combining inequalities (4) and (5), we obtain
[Equation]
Ideal non-adaptive choices for a, b : Here, we set a and b to optimize the overall bound, and obtain choices that depend on the problem parameters.
In the next section, we build upon these choices to obtain adaptive algorithms that use only the stochastic gradient information received by the algorithm.
We observe that (6) and (7) bound E [ D T ] and E [ E T ] in terms of themselves, and the coefﬁcient on the right-hand side is 4 β 2 ab 2 .
Suppose that we set a so that this coefﬁcient is 12 , i.e., we set a = 8 β 2 b 2 , so that 4 β 2 ab 2 = 12 (note that this requires setting b ≥ 2 √ 2 β, so that a ≤ 1 ).
By plugging this choice into (6) and (7), we obtain
[Equation]
The best choice for b is the one that balances the two terms above: b = Θ (cid:0) β 2 σ 2 T F ( x 1 )
− F ∗ (cid:1) 1 / 3 .
Since we also need b ≥ Ω( β ) , we can set b to the sum of the two.
Hence, we obtain
[Equation] [Equation]
3. A DAPTIVE ALGORITHM
In this section, we build on the non-adaptive algorithm and its analysis from the previous section.
We ﬁrst motivate the algorithmic choices made by our algorithm via a thought experiment where we pretend that H T , D T , E T are deterministic quantities.
Towards adaptive algorithms: To develop an adaptive algorithm, we would like to pick a, b without an explicit dependence on the problem parameters by using quantities that the algorithm can track.
We break this down by ﬁrst considering choices that do not depend on β , but on σ , and then removing the dependency on σ .
As a thought experiment, let us pretend that H T , D T , E T are deterministic quantities.
A natural choice for a that mirrors the non-adaptive choice (8) is a = (1 + σ 2 T ) −
[Equation]
Since we are pretending that D T is a deterministic quantity, we can set b by inspecting (5):
[Equation]
If we set b = D 1 / 2 T /a 1 / 4 , we ensure that D T cancels and we obtain the desired upper bound on E T .
More precisely, by plugging in a = (1 + σ 2 T ) − 2 / 3 and b =
D 1 / 2 T /a 1 / 4 into (5), we obtain
[Equation]
We now consider two cases for D T .
If D T ≤ 16 β 2 (1 + σ 2 T ) 1 / 3 , the above inequality together with H T ≤ 2 D T + 2 E T imply that H T ≤ O ((1 + β 2 )(1 + σ 2 T ) 1 / 3 ) .
Otherwise, we have D T ≥ 16 β 2 (1 + σ 2 T ) 1 / 3 and thus ab 2 ≥ 16 β 2 .
Plugging into (6), we obtain
[Equation]
which solves to D T ≤ O ((1 + σ 2 T ) 1 / 3 ( F ( x 1 ) − F ∗ ) 2 )
We can again bound H T using H T ≤ 2 D T + 2 E T .
In both cases, we have the bound
[Equation]
We now turn to removing the dependency on σ 2 T in a .
The algorithm can also track (cid:101) H T :
= (cid:80)
Tt =
(cid:107)∇ f ( x t
[Equation]
ξ t ) − ∇ f ( x t [Equation] ξ t +1 ) (cid:107) 2 , which can be viewed as a proxy for σ 2 T .
Replacing σ 2 T by this proxy and making a and b be time dependent give the update rules employed by our algorithm in the special case p = 12 .
Our update rule for general p follows from a similar thought experiment.
Analysis : Using a similar approach as in the non-adaptive analysis, we can turn the above argument into a rigorous analysis.
In the appendix, we give the complete analysis as well as a proof sketch in
Section D that gives an overview of our main analysis techniques.
E XPERIMENTS
We examine the empirical performance of our methods against the previous work STORM+ (Levy et al., 2021) and popular algorithms (Adam, AdamW, AdaGrad, and SGD) on three tasks: (1) Image classiﬁcation with the CIFAR10 dataset (Krizhevsky et al., 2009) using ResNet18 (Ren et al., 2016) models; (2) Masked language modeling via the BERT pretraining loss (Devlin et al., 2018) with the IMDB dataset (Maas et al., 2011) using distill-BERT models (Sanh et al., 2019), where we employ the standard cross entropy loss for MLM ﬁne tuning (with whole word masking and ﬁxed test masks) with maximum length 128; and (3) Sentiment analysis with the SST2 dataset (Socher et al., 2013) via ﬁnetuning BERT models (Devlin et al., 2018).
We use the standard train/validation split and run all algorithms for 4 epochs.
We use the default implementation of AdaGrad, Adam, AdamW, and SGD from Pytorch.
For
STORM+, we follow the authors’ original implementation.
We give the complete implementation details and tables of hyperparameters for all algorithms in Section B.1 of the Appendix.
Heuristics.
For our algorithms, we further examine whether heuristics like exponential moving average (EMA) of the gradient sums (or often called online moment estimation) and per-coordinate update would be beneﬁcial.
This version with heuristics is further denoted (H) in our results below.
This is discussed in full details in Section B.1.1 of the Appendix.
Results.
We perform our experiments on the standard train/test splits of each dataset.
We tune for the best learning rate across a ﬁxed grid for all algorithms and perform each run 5 times.
For readability, we omit error bars in the plot.
Full plots with error bars and tabular results with standard deviation as well as further discussions are presented in Section B.2 of the Appendix.1.
CIFAR10 (Figure 1).
Overall, META-STORM-SG achieves the lowest training loss with METASTORM and STORM+ coming in close.
META-STORM with heuristics attains the
best test accuracy, with Adam coming in close.
IMDB (Figure 2).
AdamW attains the best training loss.
However, META-STORM with heuristics achieve the best test loss (with AdamW coming in close).
META-STORM-SG and the heuristic algorithms outperform STORM+ in both minimizing training loss and test loss.
SST2 (Figure 3).
META-STORM with heuristics attain the best training loss and accuracy, above Adam and AdamW.
It also achieves the best validation accuracy out of all the algorithms.
Furthermore, non-heuristic META-STORM and META-STORM-SG outperform STORM+.
We
The mind variance- require twice the amount queries, algorithms exhibit does not come remark that STORM+ appears to be rather unstable for this task as some of the random runs do not converge to good stationary points
[Figure]
[Figure] [Figure]
C ONCLUSION
In this paper, we propose META-STORM-SG and META-STORM, two fully-adaptive momentumbased variance-reduced SGD frameworks that generalize upon STORM+ and remove STORM+’s restrictive bounded function values assumption.
META-STORM and its sibling META-STORM-SG attain the optimal convergence rate with better dependency on the problem parameters than previous methods and allow for a wider range of conﬁgurations.
Experiments demonstrate our algorithms’ effectiveness across common deep learning tasks against the previous work STORM+, and when heuristics are further added, achieve competitive performance against state-of-the-art algorithms.
Reproducibility Statement.
We include the full proofs of all theorems in the Appendix.
For our experiments, full implementation details including hyperparameter selection and algorithm development are included in Section B of the Appendix.
We also make our source code available.
A A PPENDIX OUTLINE
The appendix is organized as follows.
• Section B presents the full implementation details for our algorithms and hyperparameters used.
This section also includes additional ablation studies and experiments.
• Section C introduces the notations used in the analysis of our algorithms.
• Section D presents the proof sketch of Theorem 2.3.
• Section E establishes some basic results that are used in our full analysis.
• Section F gives the analysis of META-STORM for general p .
• Section G gives the analysis of META-STORM-SG for general p .
• Section H introduces META-STORM-NA and gives the analysis for general p .
• Section I gives several basic inequalities that are used in our analysis.
B E XPERIMENTAL DETAILS AND ADDITIONAL EXPERIMENTS
In this section, we present the complete implementation details along with the full experimental setup.
All of our experiments were conducted on two NVIDIA RTX3090.
B. I MPLEMENTATION DETAILS AND HYPERPARAMETER TUNING
In this section, we present the full implementation details of the heuristics version, parameter selection, and hyperparameter tuning for all 3 datasets.
B.1. H EURISTICS VERSIONS OF META-STORM AND META-STORM-SG
b
[Equation]
=   [Equation] + D
[Equation]
(cid:17) p /a qt for META-STORM (H) (cid:16) b 1 /p+ D t (cid:17) p /a qt + for META-STORM-SG (H) a t +1 = (cid:0) 1 + G t /a 20 (cid:1) − 2 /where D t =
αD t − 1 + (1 − α )
d 2 t
G t = (cid:40) αG t − 1 + (1 − α ) ( ∇ f ( x t , ξ t )
− ∇ f ( x t , ξ t +1 ))for META-STORM (H)
αG t − 1 + (1 − α ) ( ∇ f ( x t , ξ t ))for META-STORM-SG (H)
For our algorithms, we employ the common heuristic of using an exponential moving average (EMA) scheme in the momentum and the step size.
We also perform a per-coordinate update instead of simply using the norm.
With this, our update rules for x t +1 = x
t − ηd t /b t becomes coordinatewise division with the update rules as in Algorithm 3, where all the operations between vectors here are coordinate-wise multiplication, exponentiation, and division.
In our experiments, we set α = 0 .
99 , a 0 = 1 , b 0 = 10 − 8 as selected by the criterion detailed next.
B.1. A LGORITHM DEVELOPMENT AND DEFAULT PARAMETERS SELECTION
We develop our algorithm on MNIST and tune for p, a 0 , and b 0 .
For a 0 , we tune on MNIST across a range of values from 1 to 10 8 and found that larger values of a 0 are helpful.
For b 0 , we simply need a small number for numerical stability so we pick 10 − 8 .
For the heuristic versions of our algorithms, a 0 = 1 gives the best results.
This might be due to the effects of per-coordinate operations removing the need to scale down the gradient-accumulated step-size.
Effects of varying p .
In Figures 4 and 5, we show the training loss and test accuracy of different values of p of our algorithms on MNIST (with a 0 = 10 8 and b 0 = 10 − 8 ).
For each conﬁguration, we tune the base learning rate η across (cid:8) 10 − 3 , 10 − 2 , 10 − 1 , 1 , 10 (cid:9) .
The results suggest that the lower values of p tend to perform better.
While p = 1 / 3 has comparable performance to the lowest setting of p , this choice is somewhat analogous to STORM+.
Hence
[Figure]
we select the lowest possible value p for our algorithms in the subsequent experiments (with p = 0 .
20 for META-STORM and p = 0 .for
META-STORM-SG)
[Figure] [Figure]
p values.
For the heuristics versions of our algorithms, we perform the same experiments and show the results in Figures 6 and 7.
Since p = 0 .
50 attains the lowest training loss for both heuristics versions of our algorithms, we select such value for all our experiments.
Default parameters.
[Table]
[Figure] [Figure]
The discussion above leads to the choice of a 0 = 10 8 and b 0 = 10 − 8 by default for our algorithms with p = 0 .
20 for META-STORM and p = 0 .
25 for META-STORM-SG on the benchmarks present in this section.
For the heuristic versions of META-STORM, we use p = 0 .
50 , a 0 =
[Figure]
and b 0 = 10 − 8 for our algorithm with heuristics.
This version with heuristics is further denoted (H) in our results below.
For STORM+, we use the original authors’ implementation of setting a 0 to the number of parameters of the model (which is roughly 10 8 for ResNet18 for example).
For other baseline algorithms, we use the default parameters from Pytorch implementation.
Hyperparameter tuning.
For all algorithms, we tune only the learning rate while using the default values for the other parameters for all algorithms.
For STORM+, the default a 0 is equal to the number of parameters of the model and b 0 = 1 .
For learning rate tuning, we perform a grid search across values (cid:8) 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 (cid:9) for CIFAR10 and IMDB and across values (cid:8) 10 − 5 , 2 × 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 (cid:9) for SST (due to 2 × 10 − 5 being the default learning rate for AdamW on SST2 and also more practical due to
SST2 being a smaller dataset).
For Adam on IMDB, the learning rate in our grid search is not small enough to converge, requiring additional tuning for decreasing training loss.
Table 3 includes the selected learning rate we used for each algorithm across the datasets.
After obtaining the best learning rate, we additionally run each algorithm across 5 different seeds to obtain error bars.
[Table]
B. F ULL RESULTS FOR EXPERIMENTS IN S ECTION 4 AND ADDITIONAL EXPERIMENTS
In this section, we show complete plots and tabular results along with more detailed discussions for our experiments.
The reader should note that STORM-based methods require twice the amount of oracle access over the baselines.
The plots show average across 5 seeds along with min/max bars.
The tables show the average across 5 seeds across a range of selected epochs and one standard deviation is included at the last epoch.
In the plots and tables below: (H) denotes the version of the algorithm with the heuristics (EMA and per-coordinate update) employed.
B.2. CIFAR10: RESULTS AND DISCUSSIONS
Figure 8 shows all 4 plots of the main experiments in Section 4 in Figure 8.
[Figure]
Tables
Tables 4 and 5 show the training loss and accuracy for CIFAR10.
Tables 6 and 7 show the test loss and accuracy for CIFAR10.
[Table] [Table]
[Table] [Table]
Discussion.
META-STORM-SG achieves the lowest training loss and best training accuracy (with
META-STORM and STORM+ coming in close).
META-STORM-SG maintains the best training loss and accuracy for longest before the ﬁnal epoch.
For test loss and test accuracy, META-STORM (H) attains the best test accuracy (with Adam coming in close) while Adam attains the best test loss.
While META-STORM-SG and META-STORM achieve low training loss, their generalization performance seems worse than their heuristic counterparts.
To further study this generalization gap among different algorithms, Table 8 shows the generalization gap of different algorithms.
META-STORM with heuristics and Adam achieve the smallest gap among all the algorithms.
For our algorithms, the version with heuristics exhibit a smaller generalization gap than the version without the heuristics while STORM+ lies in between.
Interestingly,
Adagrad and SGD exhibit larger generalization gaps.
[Table]
B.2. IMDB: RESULTS AND DISCUSSIONS
Figure 2 from Section 4 shows the train and test loss of the algorithms used.
We include Figurehere that includes the error bars across 5 random seeds.
[Figure]
Tables.
Tables 9 and 10 show the train and test loss for our experiments.
[Table] [Table]
Discussion.
Here, AdamW achieves the best training loss with the heuristic algorithms coming in close.
For the test loss, these algorithms also have similar performances.
All META-STORM algorithms (with and without heuristics) perform better than STORM+ in minimizing training loss.
For test loss, META-STORM-SG performs better than STORM+ but META-STORM does not.
Both the heuristic versions of META-STORM and META-STORM-SG outperform STORM+.
B.2. SST2: FULL RESULTS AND DISCUSSIONS
Figure 10 shows all 4 plots of the main experiments for SST2.
Tables.
Tables 11 and 12 present the training loss and accuracy for the experiments for SST2.
Tables 13 and 14 show the validation loss and accuracy for the experiments for SST2.
[Figure] [Figure]
[Table]
[Table]
[Table]
[Table]
Discussions.
Similarly to CIFAR
[Table]
we examine the generalization gap of different algorithms in
Table 15.
Here, we see that MS-SG attains the lowest generalization gap between training accuracy and test accuracy while Adam suffers from the largest generalization gap among the algorithms compared in our experiments.
[Table]
C A SSUMPTIONS AND NOTATIONS C. A SSUMPTIONS
We recall the assumptions in Section 1.2 we rely on:
Lower bounded function value : F ∗ : = inf x ∈ R d F ( x ) >
−∞ .
Unbiased estimator with bounded variance : We assume to have access to ∇ f ( x, ξ ) satisfying E ξ
[ ∇ f ( x, ξ )] = ∇ F ( x ) , E ξ (cid:2) (cid:107)∇ f ( x, ξ ) − ∇ F ( x ) (cid:107) 2 (cid:3) ≤ σ 2 for some σ ≥ 0 .
Averaged β -smoothness : E ξ (cid:2) (cid:107)∇ f ( x, ξ ) − ∇ f ( y, ξ ) (cid:107) 2 (cid:3) ≤ β 2 (cid:107) x − y (cid:107) 2 , ∀ x, y ∈ R d . 4.
Bounded stochastic gradients: (cid:107)∇ f ( x, ξ ) (cid:107) ≤ (cid:98) G, ∀ x ∈ R d , ξ ∈ support ( D ) for some (cid:98) G ≥ 0 .
Bounded stochastic gradient differences : (cid:107)∇ f ( x, ξ ) − ∇ f ( x, ξ (cid:48) )
(cid:107) ≤ 2 (cid:98) σ, ∀ x ∈ R d , ξ, ξ (cid:48) ∈ support ( D ) for some (cid:98) σ ≥ 0 .
We remind the reader that σ = O ( (cid:98) σ ) and (cid:98) σ = O ( (cid:98) G ) .
C. N OTATIONS
In the analysis below, we employ the following notations β max := max { β, 1 } ;
D
[Equation]
(cid:107)
[Equation]
(cid:107) [Equation]
E
[Equation]
:=
[Equation]
(cid:88) i = a si
[Equation]
(cid:107) (cid:15) [Equation] (cid:107) 2 ;
H t :=
t (cid:88) i =
(cid:107)∇ F ( x i ) (cid:107) 2 ; (cid:98) H t :=
t (cid:88) i =
(cid:107)∇
[Equation]
We will also write E t :=
E t, 0 = (cid:80) ti =1 (cid:107) (cid:15)
i (cid:107) 2 .
We denote F t = σ ( ξ i , 1 ≤ i ≤ t ) as the sigma algebra generated by the ﬁrst t samples.
Besides, we deﬁne 0 0 :
= 1 .
In Section I, we will list and prove all inequalities used in the subsequent proofs.
D P ROOF SKETCH FOR T HEOREM 2.
In this section, to give an overview of the proof techniques, we present the proof sketch for Theorem 2.3 for the special case p = 12 .
For simplicity, we assume β ≥ 1 to simplify the notation.
The analysis of the fully adaptive algorithms follows a similar approach to the non-adaptive analysis given in Section 3.
As before, towards our ﬁnal goal of bounding (cid:107)∇ F ( x out ) (cid:107) , we will translate to
H T and upper bound it via D T and E T .
Bounding E T : As in existing VR algorithms, we need to calculate how the stochastic error (cid:15) t changes with each iteration.
By a standard calculation, we obtain
[Equation]
11) where
[Equation]
We note that, in META-STORM, a t +1 ∈ F t +1 , which implies E [ M t +1 | F t ] (cid:54) = 0 .
This extra term
M t +1 makes our analysis more challenging compared with previous works.
Now, we highlight some challenges and point out how to solve them:
C HALLENGE 1.
How to obtain a term as close to E T as possible with a proper upper bound?
In the L.H.S. of (11), we can see an extra coefﬁcient a t +1 appear in front of (cid:107) (cid:15) t (cid:107) 2 .
A straightforward option is to divide both sides by a t
+1 then sum up to get E T .
However, if we do so, the following problem arises.
Let us focus on the term (cid:107) Z t
[Equation]
1 (cid:107) 2 /a t
+1 .
The averaged β -smoothness assumption gives
[Equation]
However, we cannot apply this result to (cid:107) Z t +1 (cid:107) 2 /a t +1 since a t +1 ∈ F t +1 as noted above.
If we temporarily think a − 1 t +1 ≤ ca − 1 t for some constant c (we can expect this because the change from a t to a t +1 is not too large due to the bounded differences assumption), we will get E [ a − 1 t +1 |(cid:107)
Z t +1 (cid:107) 2 |
F t ] ≤ E [ ca − 1 t (cid:107) Z t +1 (cid:107) 2 | F t ] ≤ η 2 β 2 (cid:107) d t (cid:107) 2 a t b 2 t .
If we plug in the update rule of b t = ( b 20 + D T ) 1 / 2 /a 1 / 4 t , then we obtain E [ (cid:107) Z t +1 (cid:107) 2 | /a
t +1 | F t ] ≤ η 2 β 2 a − 1 / 2 t (cid:107) d t (cid:107) 2 b 20 + D t .
It can be shown that (cid:80) Tt =1 (cid:107) d t (cid:107) 2 b 20 + D T can be upper bounded by log D T b 20 , but now we still have the extra a − 1 / 2 t coefﬁcent.
To remove it, it is reasonable to divide both sides of (11) by a 1 / 2 t +1 rather than a t +1 .
C HALLENGE 2.
How to get rid of the term involving M t
+1 ?
As discussed in Challenge 1, we want to divide both sides by a 1 / 2 t +1 .
Now we focus on the term a − 1 / 2 t +1 M t +1 .
Again, due to a t +1 ∈ F t +1 , E [ a − 1 / 2 t +1 M t +1 | F t ] (cid:54) = 0 .
An important observation here is that, if we replace a t +1 by a t in M t +1 , we will have a martingale difference sequence.
Formally, we deﬁne
[Equation]
Then E [ N t +1 | F t ] and E [ a − 1 / 2 t
N t +1 | F t ] are both 0 .
This observation tells us that, in order to bound E [ (cid:80) Tt =1 a − 1 / 2 t +1 M t +1 ] , it sufﬁces to bound E [ (cid:80)
Tt =1 a − 1 / 2 t +1 M t +1 − a − 1 / 2 t
N t +1 ] .
Using the Cauchy-Schwartz inequality, we show that the term (cid:80) Tt =1 a − 1 / 2 t +1 M t +1 − a − 1 / 2 t
N t + can be bounded by terms related to (cid:80) T t =1 ( a − 1 / 2 t + − a − 1 / 2 t ) (cid:107) (cid:15) t (cid:107) 2 , (cid:80) T t =1 a − 1 / 2 t +1 (cid:107) Z t
+1 (cid:107) 2 and (cid:80) Tt
=1 a 3 / 2 t (cid:107)∇ f ( x t +1 , ξ t +1 )
− ∇ F ( x t +1 ) (cid:107) 2 .
We then bound these latter terms in turn, and eliminate the term involving M t +1 .
After overcoming the two challenges above, we can ﬁnally show the following inequality, where
K 1 , K 2 , K 4 are constants that depend only on σ, (cid:98) σ, β, a 0 , b 0 , η and are independent of T .
[Equation]
Bounding D T :
By following the standard non-adaptive analysis via smoothness, we obtain
[Equation]
Here we proceed similarly to the non-adaptive analysis from Section 3.1, but start to diverge from the analysis approach used in STORM+.
The STORM+ analysis proceeds by splitting −(cid:104)∇ F
[Equation]
+
[Equation]
2 , multiplying both sides of (13) with b t /η , and summing up over all iterations.
This gives the following upper bound on H T :
[Equation]
This analysis requires F ( x ) to be bounded so that the sum (cid:80) Tt =1 2 η ( F ( x t ) − F ( x t +1 ))
b t can telescope.
To remove this assumption, we go back to (13), split −(cid:104)∇ F ( x t ) , d t (cid:105)
= −(cid:107)
d t (cid:107) 2 + (cid:104) (cid:15) t , d t (cid:105) , and upper bound the inner product via the Cauchy-Schwartz inequality and the inequality ab ≤ γ 2 a 2 + 12 γ b 2 which holds for any γ > 0 :
[Equation]
where λ > 0 is a constant (setting λ based on (cid:98) σ yields the best dependence on (cid:98) σ ).
We note that this choice will need a bound on E [ (cid:80)
Tt =1 a 1 / 2 t
+1 (cid:107) (cid:15) t (cid:107) 2 ] , and 1 / 2 turns out to be the smallest choice of c which makes E [ (cid:80)
Tt =1 a ct +1 (cid:107) (cid:15) t (cid:107) 2 ] have a constant order.
The intuition for setting γ = λa 1 / 2 t +1 b t ηβ is that this coefﬁcient ensures a constant split if a t and b t correspond to the non-adaptive choices we derived in Section 3.1, which were set so that a 1 / 2 b = Θ ( β ) .
We obtain
E
[Equation]
The term ( (cid:63) ) can be bounded using standard techniques used in the analyses of adaptive algorithms.
The term ( (cid:63)(cid:63) ) has already been bounded in the previous analysis.
Now we only need to simplify the term on the L.H.S. to D T .
But due to the randomness of b t , this is not achievable.
However, the same as for the ﬁrst inequality in (12), we can bridge this gap by aiming for a slightly weaker inequality that bounds D 1 / 2 T instead of D T .
More precisely, we connect the left-hand side of (14) to
D 1 / 2 T as follows:
[Equation]
By plugging in (15) into (14) and setting λ appropriately, we can ﬁnally obtain the following upper bound:
[Equation]
where K 5 , K 6 , K 7 , K 8 , K 9 depend only on σ, (cid:98) σ, β, a 0 , b 0 , η and are independent of T .
Combining the bounds: The ﬁnal part of the analysis is to combine (12) and (16).
In contrast to the simpler non-adaptive analysis, these inequalities bound a 1 / 2 T +1 E T and a 1 / 4 T +1 D 1 / 2 T instead of
D T and E T .
In order to obtain an upper bound on H T via the inequality H T ≤ 2 D T + 2 E T , we need to connect a 1 / 2 T +1 E T and a 1 / 4 T +1 D 1 / 2 T and D T and E T .
The bounded variance assumption on the stochastic gradients gives us a bound on E [ a − 3 / 2 T +1 ]
=
E [1 + (cid:101) H T /a 20 ]
= O (1 + σ 2 T ) (note that this − 3 / 2 is the smallest c to make sure we can upper bound E [ a ct +1 ] ).
Combining this result and
Holder’s inequality gives us the bound
[Equation]
where 3 / 7 is chosen to ensure that we ﬁnally can use the bound on E [ a − 3 / 2 T +1 ] .
Thus we obtain an upper bound on E [ H 3 / 7 T ] .
Finally, applying the concavity of x 3 / 7 to E [ H 3 / 7 T ] gives Theorem 2.3.
E B ASIC ANALYSIS
As discussed in Section 3, we aim to use E T and D T to bound H T .
Here, we apply this framework to give some basic results which will be used frequently for the full analysis of every algorithm.
We ﬁrst state the following decomposition in our analysis framework.
The reason we use (cid:98) p ≤ 1 here is that we can not always bound H T directly because of the randomness of a t and b t in our algorithms.
Lemma E.1.
Given (cid:98) p ≤ 1 , we have
[Equation]
Proof.
By the deﬁnition of H T , E T and D T , we have H T ≤ 2 E T + 2 D T .
Hence
[Equation]
where ( a ) and ( b ) are both by due to (cid:98) p ≤ 1 .
E. V ARIANCE REDUCTION ANALYSIS FOR E T
The same as in all existing momentum-based VR methods, we need to analyze how the error term (cid:15) t changes in the algorithm.
Based on our notations
[Equation]
we give the following two standard lemmas.
Lemma E.2.
∀ t ≥ 1 , we have
[Equation]
where
[Equation]
Proof.
Starting from the deﬁnition of (cid:15) t +1 , we have
[Equation]
where ( a ) is by ( x + y ) 2 ≤ 2 x 2 + 2 y 2 , ( b ) is by 0 ≤ 1 − a t +
[Equation]
≤ 1 .
Adding a t +
[Equation]
t
[Equation]
2 −
[Equation]
+1 (cid:107)to both sides, we get the desired result.
Lemma E.3.
∀ t ≥ 1 , we have
[Equation]
Proof.
From the deﬁnition of Z t +1 , we have
E (cid:2) (cid:107)
[Equation]
where ( a ) is by E (cid:2) (cid:107) X
− E
[ X ] (cid:107) 2 (cid:3) ≤ E (cid:2) (cid:107) X (cid:107) 2 (cid:3) , ( b ) is by the averaged β -smooth assumption, ( c ) is by the fact x t
+1 − x t = − ηb t d t .
E. O N THE WAY TO BOUND D T
We choose to bound the terms D T instead of starting from H T as done in AdaGradNorm or
STORM+.
The latter also requires the bounded function value assumption in the analysis.
Lemma E.4.
For any of META-STORM-SG, META-STORM or META-STORM-NA, we have, for any λ >
[Equation]
ηβ max .
Proof.
Using smoothness, the update rule x t +
= x
t − ηb t d t and the deﬁnition of (cid:15) t = d t −∇ F ( x t ) , we obtain
[Equation]
First we use Cauchy-Schwarz to separate the stochastic gradient and the stochastic error terms
[Equation]
Taking
[Equation]
for some λ > 0 .
We have
[-+
-]
[Equation]
⇒ E (cid:34)
[Equation]
ηβ
[Equation]
The ﬁnal step is to relate the L.H.S. to D T .
Recall for META-STORM-SG and META-STORM-NA, we have
[Equation]
Hence
T (cid:88) t =
[Equation]
The same result holds for META-STORM by a similar proof.
By using this bound, the proof is ﬁnished.
To ﬁnish section, we prove a technical result, Lemma E.5, which will be very useful in the proof of every algorithm.
The motivation to prove it is because we want to bound the term inside the expectation part in Lemma E.4.
Lemma E.5.
Given A, B ≥ 0 .
We have for META
STORM
SG and META STORM-NA
[Equation]
for
META
-STORM
[Equation]
Proof.
In META-STORM-SG and META-STORM-NA, we have
[Equation]
where p + 2 q = 1 .
Deﬁne the set
[Equation]
and let s = max S .
We know
T (cid:88) t =
[Equation]
d
[Equation]
(cid:107)
[Equation]
b
[Equation]
(cid:32)
[Equation]
a
[Equation]
b t (cid:33)
a
[Equation]
b
[Equation]
pt − a q/
[Equation]
where ( a ) is by a t ≥ a t +1 .
Note that
[Equation]
where ( b ) holds by a t +1 ≤ 1 , ( c ) is due to qp − 12 = 2 q − p 2 p = 1 − 2 p 2 p = 12 p − 1 by p + 2 q
=
and ( d ) is by applying Lemma I.8.
Thus we know
T (cid:88) t = (cid:32)
A + B a 1 /
[Equation]
( f ) ≤ ( A + B ) 1 p −1 − p s (cid:88) t = log b t b t −= ( A + B ) 1 p −1 − p
log b s b( g ) ≤ ( A + B ) 1 p −1 − p log
A + a − 1 / 2 T +1 B bwhere ( e ) is by taking x = ( b t /b t − 1 ) 1 /p in 1 − 1 x ≤ log x , ( f ) is because b t is increasing.
The reason ( g ) is true is that b s ≤ A + a − 1 / 2 s +1 B ≤
A + a − 1 / 2 T +1 B where the ﬁrst inequality is due to s ∈ S and the second one holds by that a − 1 / 2 t is increasing.
Now we ﬁnish the proof for META-STORM-SG and
META-STORM-NA.
The proof for META-STORM is essentially the same hence omitted here.
F A NALYSIS OF META-STORM FOR GENERAL p
In this section, we give a general analysis for our Algorithm META-STORM.
We will see that p =is a special corner case.
First we recall the choices of a t and b t
[Equation]
where p, q satisfy p + 2 q = 1 , p ∈ (cid:104) 3 −√ 7 2 , 12 (cid:105) .
a 0 > 0 and b 0 > 0 are absolute constants.
Naturally, we have a 1 = 1 .
We will ﬁnally prove the following theorem.
Theorem F.1.
Under the assumptions 1-3 and 5, by deﬁning (cid:98) p = 3(1 − p ) 4 − p ∈ (cid:2) 37 , √ 7 − 2 (cid:3) , we have
[Equation]
 
[Equation]
where K i , i ∈ [9] are some constants only depending on a 0 , b 0 , η, σ, (cid:98) σ, β, p, q, F ( x 1 ) − F ∗ .
To simplify our ﬁnal bound, we only indicate the dependency on β and F ( x 1 ) − F ∗ .
[Equation]
Remark F.2 .
For all i ∈ [9] , the constant K i will be deﬁned in the proof that follows.
By using the concavity of x (cid:98) p , we state the following convergence theorem without proof.
Theorem F.3.
Under the assumptions 1-3 and 5, by deﬁning (cid:98) p = 3(1 − p ) 4 − p ∈ (cid:2) 37 , √ 7 − 2 (cid:3) , we have
[Equation]
Here, we give a more explicit convergence dependency for p = 1 2 used in Theorem 2.
Theorem F.4.
Under the assumptions 1-3 and 5, when p = 12 , by setting λ = min (cid:8) 1 , ( a 0 / (cid:98) σ ) 7 / 3 (cid:9) (which is used in K 5 to K 9 ) we get the best dependency on (cid:98) σ .
For simplicity, under the setting a 0 = b 0 =
η = 1 , we have
[Equation]
where Q 1 = O (cid:16) ( F ( x 1 ) − F ∗ ) 6 / 7 + σ 12 / 7 + ( (cid:98) σσ ) 6 / 7 + (cid:98) σ 18 / 7 + (cid:0) 1 + (cid:98) σ 18 / 7 (cid:1)
β 6 / 7 log 6 / 7 (cid:0) β
+ (cid:98) σ 3 β (cid:1)(cid:17) and Q 2 =
O (cid:0)(cid:0)
+ (cid:98) σ 18 / 7 (cid:1) β 6 / 7 (cid:1) .
To start with, we ﬁrst state the following useful bound for a t :
Lemma F.5.
∀ α ∈ (0 , 3 / 2] and ∀ t ≥ 1 , there is [Equation] a
[Equation]
a
[Equation]
(cid:98)
[Equation]
a αt .
Especially, taking α ∈ { 1 / 2 , 1 , 3 / 2 } , we have (cid:18) a t a t + (cid:19) 1 /≤ 1 + 4 1 / 3 (cid:98)
[Equation]
Proof.
Note that
[Equation]
where the last inequality is because 2 α/ 3 ≤
[Equation]
Lemma F.5 allows us to obtain some other properties of a t .
Lemma F.6.
For t ≥
[Equation]
Proof.
Let a t +
= x, a t = y and note that x ≤ y ≤ 1 .
For the ﬁrst inequality,
[Equation]
[Equation]
For the second inequality, we have
[Equation]
F.
A NALYSIS OF E T
Following a similar approach
[Equation]
we ﬁrst deﬁne a random time τ satisfying
[Equation]
, where
[Equation]
One thing we need to emphasize here is that, in our current choice
[Equation]
a t ∈ F t , which implies {
[Equation] [Equation]
1 } ∈ F t .
This means τ + 1 is a stopping time instead of τ itself.
We now prove a useful proposition for τ :
Lemma F.7.
We have
[Equation]
where
[Equation]
Proof.
First, by the deﬁnition of τ , we know a t ≥ K − 1 ≥ K 0 , ∀ t ≤ τ .
For time τ , we have a −
[Equation] [Equation]
[Equation]
⇒ a − 1 τ +1 ≤ ( a − 3 / 2 τ
[Equation]
σ 2 /a 20 ) 2 / 3 ≤
[Equation]
K − 3 / 2 −
[Equation]
a
[Equation] [Equation] [Equation]
−
[Equation] [Equation]
which implies a τ +1 ≥ K 0 .
For the second proposition, let h ( y ) =
y 2 / 3 .
Due to the concavity of h , we know h ( y 1 ) − h
( y 2 ) ≤ h (cid:48)
( y 2 )( y 1 − y 2 )
=
2( y 1 − y 2 ) 3 y 1 / 3.
Now we have
[Equation]
where the last step is by a t ≤ a τ +1 < K − 1 ≤ a 4 0 / (144 (cid:98) σ 4 ) .
F.1. B OUND ON E (cid:2)
E τ, 3 / 2 − 2 (cid:96) (cid:3) FOR (cid:96) ∈ (cid:2) 14 , 12 (cid:3)
Unlike STORM+ in which they bound E [ E τ ] , we choose to bound E (cid:2) E τ, 3 / 2 − 2 (cid:96) (cid:3) .
We ﬁrst prove the following bound on E (cid:2)
E τ, 3 / 2 − 2 (cid:96) (cid:3) :
Lemma F.8.
For any (cid:96) ∈ (cid:2) 14 , 12 (cid:3) , we have
[Equation]
Proof.
We start from Lemma E.
[Equation]
Summing up from 1 to τ and taking expectations on both sides, we will have
E [ E τ, 1 ] ≤
[Equation]
First we bound E [ (cid:80) τt =1 M t +1 ] .
From the deﬁnition of M t +1 , we have
E [ M t +1 ]
[Equation]
[Equation]
Now for t ≥ 1 , we deﬁne
N
[Equation] [Equation]
t , Z
[Equation]
[Equation]
+
[Equation]
∇ f
[Equation]
x
[Equation]
) − ∇ F
[Equation]
[Equation]
∈ F
[Equation]
+ with N 1 := 0 .
A key observation is that
[Equation]
This is because N t := (cid:80) ti =1 N t is a martingale and τ + 1 is a bounded stopping time.
Then by optional sampling theorem, we have
E
[Equation]
By subtracting E [ (cid:80) τt =
1 M t +1 ] by E [ (cid:80) τt =
1 N t +1 ] , we obtain
E (cid:34) τ (cid:88) t =
[Equation] [Equation]
Using Cauchy-Schwarz inequality for each term, we have
[Equation]
a
[Equation]
1 (cid:107) 2 , 2 ((1 − a t +1 ) a t +1 − (1 − a t ) a t ) (cid:104) (cid:15) t , ∇ f ( x t +1 , ξ t +1 )
− ∇ F ( x t +1 ) (cid:105) ≤ 2 | (1 − a t +1 ) a t +1 − (1 − a t ) a t | (cid:107) (cid:15) t (cid:107)(cid:107)∇ f ( x t +1 , ξ t +1 ) − ∇ F ( x t +1 ) (cid:107) ≤ a t + 4 (cid:107) (cid:15) t (cid:107) 2 + 4 ((1 − a t +1 )
a t +1 − (1 − a t ) a t )a t + (cid:107)∇ f ( x t +1 , ξ t +1 )
− ∇ F ( x t +1 ) (cid:107) 2 .
Plugging the above bounds into (18), we obtain
[Equation]
Plugging the bounds for ( i ) and ( ii ) from Lemma F.6 into (19)
[Equation]
the following bound on E [
[Equation]
τt =1 M t +1 ] comes up
[Equation]
Then from (17), we have
[Equation]
which will give us
[Equation]

[Equation] [Equation]
For term ( iii ) , Lemma E
[Equation]
3 tells us
[Equation]
For term ( iv ) , we know
[Equation] [Equation]
Note that
[Equation]
then we have
[Equation] [Equation]
where, for the ﬁrst inequality
[Equation]
we use Lemma I
[Equation]
4 and Lemma I
[Equation]
Plugging (21) and (23) into (20), we obtain
[Equation]
Note that by Lemma F.7, we have for t ≤ τ , a t +1 ≥ K 0 .
By using this property and noticing 2 (cid:96) − 1 / 2 ≥ 0 , we can obtain
[Equation]
E
[Equation]
which will give the desired bound immediately.
F.1. B OUND ON E [ E T, 1 − 2 (cid:96) ] FOR (cid:96) ∈ (cid:2) 14 , 12 (cid:3)
With the previous result on E (cid:2) E τ, 3 / 2 − 2 (cid:96) (cid:3) , we can bound E [ E T, 1 − (cid:96)
Lemma F.9.
For any (cid:96) ∈ (cid:2) 14 , 12 (cid:3) , we have
E [ E T, 1 − 2 (cid:96) ] ≤ K 1 ( (cid:96) )
+ K 2 ( (cid:96)
)   
E
[Equation]
(cid:96) > E
[Equation]
log (cid:16) 1 + (cid:101) H
[Equation]
/a 20 (cid:17)(cid:105)
[Equation]
where
[Equation]
Proof.
We use a similar strategy as in the previous proof in which we bound E (cid:2) E τ,
[Equation]
2 − 2 (cid:96) (cid:3) .
Starting from Lemma E.
[Equation]
Dividing both sides by a 2 (cid:96)t +1 , taking the expectations on both sides and summing up from 1 to T to get
[Equation]
As before, we bound E (cid:104)
M t +1 a 2 (cid:96)t +1 (cid:105) ﬁrst.
From the deﬁnition of M t +1 , we have
E
[Equation]
A similar key observation is that, if we replace a t +1 by a t , we can ﬁnd
[Equation]
a
[Equation]
x
[Equation]
(cid:105) (cid:21) = 0 .
By subtracting E (cid:104) M t +1 a 2 (cid:96)t +1 (cid:105) by 0 , we know
E
[Equation]
M t + a
[Equation]
+
[Equation]
a
[Equation]
Schwarz term
[Equation]
[Table]
Plugging these two bounds into (25)
[Table]
we obtain
[Table]
[Equation]
To bound ( i ) and ( ii ) , let a (cid:96)t +
= x, a (cid:96)t =
y and note
that 0 ≤ x ≤ y ≤ 1 .
By Lemma I.6, we have for ( i )
[Equation] [Equation]
For ( ii ) , by Lemma I.7,
[Equation]
[Equation]
Plugging (27) and (28) into (26), we will have
[Equation]
Now combining this with (24), we obtain
E [ E T, 1 − 2 (cid:96) ] ≤ σ 2 + E  
T (cid:88) t =
[Equation]
2 (cid:96) t +1 − a − 2 (cid:96) t (cid:1) (cid:107) (cid:15) t (cid:107)(cid:124) (cid:123)(cid:122) (cid:125) ( iii ) +
T (cid:88) t =
[Equation]
1 +
[Equation]
(cid:96)
[Equation]
(cid:96) [Equation] a 2 (cid:96)t +
[Equation]
For ( iii ) , we split the sum according to τ then use Lemma F.7 and Lemma F.5,
T (cid:88) t =
[Equation]
t
[Equation]
[Table]
[Equation]
[Table]
(cid:1) [Table] (cid:1)
[Table]
Note that 3 / 2 − 2 (cid:96) ∈ (cid:2) 12 , 1 (cid:3) , we have a
[Equation]
[Table]
[Equation]
+
[Equation]
a − 2 (cid:96) t
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
, ( Lemma F.5 )
and we can use Lemma F.7 to bound for t ≥ τ +a
[Equation]
t =
[Equation] [Equation]
−
[Equation]
[Equation]
[Table]
Thus
[Table]
For ( iv ) , note that
[Equation]
where the last step is by Lemma E.3.
Hence we obtain
E
[Equation]
a 2 (cid:96)t (cid:33) η 2 β 2 (cid:107) d t (cid:107)a 2 (cid:96)t b 2 t (cid:35) .
For ( v ) , by the same argument when bounding (22), we know
E [( v )] ≤ E (cid:34) T (cid:88) t = (cid:32) a 2 − 2 (cid:96) t (cid:96)+ 2 a 2 − 2 (cid:96) t + (cid:33) (cid:107)∇ f ( x t +1 , ξ t +1 )
− ∇ f ( x t +1 , ξ t +2 ) |(cid:107)(cid:35) .
Now we use Lemma I.2 and Lemma I.3 to get
T (cid:88) t = (cid:32) a 2 − 2 (cid:96) t (cid:96)+ 2 a 2 − 2 (cid:96) t + (cid:33) (cid:107)∇ f ( x t +1 , ξ t +1 ) − ∇ f ( x t +1 , ξ t +2 ) (cid:107)
[Table]
 
[Table]
[Equation]
Plugging the bounds on ( iii ) , ( iv ) and ( v ) into (29), we get
[Equation]
which gives us
[Equation]
Now we plug in the bound on E (cid:2)
E τ, 3 / 2 − 2 (cid:96) (cid:3) in Lemma F.8 to get the ﬁnal result
E [ E T, 1 − 2 (cid:96) ] ≤
[Equation] [Equation]
The following bound on E (cid:2) E T, 1 / 2 (cid:3) will be useful when we bound D T .
Corollary F.10.
We have
[Equation]
t = [Equation] (cid:16)
[Equation]
(cid:17)
[Equation]
(cid:107) d
[Equation]
(cid:107)
[Equation]
[Equation]
(cid:35) .
Proof.
Take (cid:96) = 14 in Lemma F.9.
F.1.
B OUND ON E (cid:104) a 1 − 2 q T +1 E T (cid:105)
With the previous result on E [ E T, 1 − 2 (cid:96) ] , we can bound E (cid:104) a 1 − 2 q T +1 E T (cid:105) immediately.
Lemma F.11.
Given p + 2 q = 1 , p ∈ (cid:104) 3 −√ 7 2 , 12 (cid:105) , we have
[Equation]
where
[Equation]
Proof.
When q > 14 ⇔ p < 12 , by Lemma F.9, taking (cid:96) =
q , we know
[Equation] [Equation]
where ( a ) is by
[Equation]
( b ) is by Lemma I.1, ( c ) is by 1 − 2 p = 4 q − 1 .
When q = 1 4 , by a similar argument, we have
[Equation]
Now we can deﬁne
[Equation]
The ﬁnal step is by noticing for 1 − 2 q = p > 0 ,
[Equation]
F. A NALYSIS OF D T
We will prove the following bound
Lemma F.12.
Given p + 2 q = 1 , p ∈ (cid:104) 3 −√ 7 2 , 12 (cid:105) , we have
[Equation]
where
[Equation]
λ > 0 can be any number
[Equation]
Proof.
We start from Lemma E.
[Equation] [Equation]
where λ > 0 is used to reduce the order of (cid:98) σ in the ﬁnal bound.
In the proof of the general case
[Equation]
we don’t choose λ explicitly anymore.
Plugging in the bound on E (cid:2) E T, 1 / 2 (cid:3) in Corollary F.10, we have
[Equation]

[Equation]
where, in the last step, we use Lemma F.5.
Next, we apply Lemma E.5 to ( i ) to get ( i ) ≤
[Equation]
By plugging the above bound into (30), we get the desired result.
F.
C OMBINE THE BOUNDS AND THE FINAL P ROOF
From Lemma F.11, we have
[Equation]
From Lemma F.12, we have
[Equation]
Now let
[Equation]
Apply Lemma E.1, we can obtain
[Equation]
Now we can give the ﬁnal proof of Theorem F
[Equation]
Proof.
First, we have
[Equation]
where the second equation is by the independency of ξ i and ξ i +1 .
Now we consider following two cases:
[Equation]
In this case
[Equation]
we will ﬁnally prove
[Equation]

[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Note that by Holder inequality
[Equation]
where ( a ) is by 1 − 2 q = p , ( b ) is due to 2 p (cid:98)
[Equation]
First, if q (cid:54) = 14 , we have
[Equation]
where ( c ) is by 4 q − 1= 1 − 2 p≤ 1 and p ≥ 3 −√ 7⇒
− 2 p ≤ 3(1 − p )
4 − p = (cid:98) p , ( d ) is by E (cid:104) (cid:101) H T (cid:105) ≤ 2 σ 2 T and E (cid:104) D (cid:98) p T (cid:105) ≤ E (cid:104) E (cid:98) p T (cid:105) .
Then we know
[Equation]
[Table] [Table]
Combining two results, we know when q (cid:54) =
[Equation]
Following a similar approach
[Equation]
we can prove for q =
[Equation]
there is
[Equation]
Now we use Lemma I.9 to get
[Equation]
Finally
[Equation]
we have
[Equation]

[Equation]
q =.
[Equation] [Table]
[Equation]
pT (cid:105) .
In this case, we will ﬁnally prove
[Equation]
(cid:104)
[Equation]
[Table]
[Equation]
where the last step is by
[Equation]
We know
[Equation]
where ( e ) is by the concavity of log function, ( f ) holds due to E (cid:2)
X 1 / 3 (cid:3) ≤ E 1 /
[ X ] for X ≥ 0 , ( g ) is by E (cid:104) (cid:101) H T (cid:105) ≤ 2 σ 2 T .
Then we have
[Equation]
Finally
[Equation]
combining Case 1 and Case 2 and using (31)
[Equation]
we get the desired result and ﬁnish the proof
[Equation]
(cid:19) (cid:98)
p.
G A NALYSIS OF META-STORM-SG FOR GENERAL p
In this section, we give a general analysis for our Algorithm META-STORM-SG.
Readers will see p = 12 is a very special corner case.
First we recall the choices of a t and b t :
[Equation]
where p, q satisfy p + 2 q = 1 , p ∈ (cid:2) 14 , 12 (cid:3) .
a 0 > 0 and b 0 > 0 are absolute constants.
Naturally, we have a 1 = 1 .
We will ﬁnally prove the following theorem.
Theorem G.1.
Under the assumptions 1-4, by deﬁning (cid:98) p = 2(1 − p )∈ (cid:2) 13 , 12 (cid:3) , we have
[Equation]
 
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
where C i , i ∈ [10] are some constants only depending on a 0 , b 0 , σ, (cid:98) G, β, p, q, F ( x 1 ) − F ∗ .
To simplify our ﬁnal bound, we only indicate the dependency on β and F ( x 1 ) − F ∗ when σ (cid:54) =
and T is big enough to eliminate C 9 and C
[Equation]
Remark G.2 .
For all i ∈ [10] , the constant C i will be deﬁned in the proof that follows.
Again, by the concavity of x (cid:98) p , we have the following convergence theorem, of which the proof is omitted.
Theorem G.3.
Under the assumptions 1-4 by deﬁning (cid:98) p = 2(1 − p )∈ (cid:2) 13 , 12 (cid:3) , when σ (cid:54) =
and T is big enough, we have
[Equation]
Here, we give a more explicit convergence dependency for p = 12 used in Theorem 2.1.
Theorem G.4.
Under the assumptions 1-4, when p = 12 , by setting λ = min (cid:110) 1 , ( a 0 / (cid:98) G ) 2 (cid:111) (which is used in C 4 to C 8 and C 10 ) we get the best dependency on (cid:98) G .
For simplicity, under the setting a 0 = b 0 =
η = 1 , we have
[Equation]
2 / 9 (cid:19) where
[Equation]
and
[Equation]
To start with
[Equation]
we ﬁrst state the following useful bound for a t :
Lemma G.5.
∀ t ≥ 1 , there is
[Equation]
Proof.
[Equation]
G.
[Equation]
A NALYSIS OF E T
Following a similar approach
[Equation]
we deﬁne a random time τ satisfying
[Equation]
where
[Equation]
Note that { τ = t } = { a t ≥ C 0 , a t +1 < C 0 } ∈ F t , this means τ is a stopping time.
We now prove a useful proposition of τ :
Lemma G.6.
∀ t ≥ τ + 1 , we have
[Equation]
Proof.
Let h ( y ) =
y 2 / 3 .
Due to the concavity, we know h ( y 1 ) − h
( y 2 ) ≤ h (cid:48)
( y
[Equation]
( y 1 − y 2 ) = 2( y 1 − y
[Equation]
3 y
[Equation] [Equation]
Now we have
[Equation]
where the last step is by a t ≤ a τ +1 < C 0 ≤ ( a 0 / (cid:98) G ) 4 .
G.1. B OUND ON E (cid:2) E τ, 3 / 2 − 2 (cid:96) (cid:3) FOR (cid:96) ∈ (cid:2) 14 , 12 (cid:3)
Similar to the analysis of META-STORM, we choose to bound E (cid:2) E τ, 3 / 2 − 2 (cid:96) (cid:3) .
We ﬁrst prove the following bound on E (cid:2)
E τ, 3 / 2 − 2 (cid:96) (cid:3) :
Lemma G.7.
For any (cid:96) ∈ (cid:2) 14 , 12 (cid:3) , we have
[Equation]
Proof.
We start from Lemma E.2,
[Equation]
[Table]
≤ (cid:107)
[Table]
−
[Table]
[Equation]
(cid:107)
[Equation]
[Table]
+
[Table]
(cid:107)
[Equation]
Summing up from 1 to τ − 1 and taking the expectations on both sides, we obtain
E
[Equation]
=
[Equation]
(cid:107) (cid:15)
[Equation]
(cid:107) (cid:15)
[Equation]
+1 (cid:107)
[Equation]
(cid:107)
[Equation]
1 (cid:107)
[Equation]
1 (cid:107)∇
[Equation]
x
[Equation]
[Equation]
[Equation]
x
[Equation]
) (cid:107)
[Equation]
(cid:21) =
[Equation]
(cid:20) (cid:107) (cid:15) 1 (cid:107) [Equation] − (cid:107) (cid:15) [Equation]
(cid:107) 2 +
[Equation]
−
[Equation]
(cid:88)
[Equation]
=
[Equation]
(cid:107) Z
[Equation]
1 (cid:107) [-Z t +1 (cid:107)+ 2 a 2 t +
-][Equation] [Equation] [Equation]
1 (cid:107)∇ [Equation] ( x t [Equation]
1 , ξ [Equation] [Equation] ( x t +1 ) (cid:107)[-
-]
[Equation] M t + (cid:21)
[Equation] (cid:20) (cid:107) (cid:15) 1 (cid:107) [-2 − (cid:107) (cid:15) τ (cid:107) 2 + τ − 1 (cid:88) t = 2 (cid:107) Z t +1 (cid:107)+ 2 a 2 t +
-][Equation]
(cid:107) (cid:15)
[Equation] (cid:107) [-
-]2 +
[Equation] (cid:88) t = 2 (cid:107) Z [Equation] +
[Equation] (cid:107)+ 2 a 2 t [-+
-]+1 (cid:107)∇ f ( x t +1 , ξ [Equation]
+1 ) [Equation] ( x [Equation]
+ [Equation] ) (cid:107) [-
-]2 + [Equation]
−
[Equation] (cid:88) [Equation] =
M [Equation] + (cid:21) ⇒ E (cid:2) E
[Equation]
− 1 , 1 + (cid:107) (cid:15) τ (cid:107) 2 (cid:3) ≤ σ 2 + E (cid:20)
T (cid:88) t = 2 (cid:107) Z t +1 (cid:107)+ 2 a
[Equation]
Because C 0 ≤ 1 , a τ +1 ≤ 1 , 2 (cid:96) − 1 / 2 ≥ 0 and 3 / 2 − 2 (cid:96) ≥ 0 , so we have
C 2 (cid:96) − 1 /
[Equation]
Besides, for t ≤ τ − 1 , by the deﬁnition of τ , we have C 0 ≤ a t +1 , then we know
[Equation]
These two results give us
[Equation]
which implies
[Equation]
M
[Equation]
Let M t := (cid:80) ti =1 M i ∈ F t with M 1 = 0 .
For s ≤ t , we know E [ M t
|F s ] = 0 , hence M t is a martingale.
Note that τ is a bounded stopping time, hence by optional sampling theorem
E
[Equation]
Now we have
[Equation]
By Lemma E.
[Equation]
Besides, under our current choice
[Equation]
a t +1 ∈ F t ,
[Equation]
Using these two bounds
[Equation]
we have
[Equation]
where the last inequality holds by Lemma I.4.
Dividing both sides by C 2 (cid:96) − 1 / 2, we get the desired bound immediately
[Equation]
G.1. B OUND ON E [ E T, 1 − 2 (cid:96) ] FOR (cid:96) ∈ (cid:2) 14 , 12 (cid:3)
With the previous result on E (cid:2) E τ, 3 / 2 − 2 (cid:96) (cid:3) , we can bound E [ E T, 1 − 2 (cid:96) ] .
Lemma G.8.
For any (cid:96) ∈ (cid:2) 14 , 12 (cid:3) , we have
[Equation]
where
[Equation]
Proof.
Starting from Lemma E
[Equation]
2 as well
[Equation]
Dividing both sides by a 2 (cid:96)t +1 and taking expectations
[Equation]
we have
[Equation] [Equation]
Note that under our current choice
[Equation]
a t +1 ∈ F t , hence we have
[Equation]
where the second bound holds by Lemma E.3.
Plugging these three bounds into (32), we know
E
[Equation]
Now sum up from 1 to T to get
E
[Equation]
] ≤ E
[Equation]

[Equation]
(cid:125)
[Equation]
ii [Equation]
  .
For ( i ) , we split the time by τ
T (cid:88)
[Equation]
=
[Equation]
where the second inequality is by Lemma G.5 and Lemma G.6.
Next, for ( ii ) , we use Lemma I.2 to get
T
[Equation] [Equation]
Plugging these two bounds into (33)
[Equation]
we have
[Equation]
=.
Thus
E
[Equation]

[Equation]
Plugging the bound on E (cid:2) E τ, 3 / 2 − 2 (cid:96) (cid:3) in Lemma G.
[Equation]
we ﬁnally get
E [
[Equation] [Equation]
where
[Equation]
G.1.
B OUND ON E (cid:2) E T, 1 / 2 (cid:3)
The following bound on E (cid:2) E T, 1 / 2 (cid:3) will be useful when we bound D T .
Corollary G.9.
We have
[Equation]
Proof.
Take (cid:96) = 1 4 in Lemma G.8.
[Equation]
ON
[Equation]
Lemma G.10.
Given p + 2 q = 1 , p ∈ (cid:2) 14 , 12 (cid:3) , we have
[Equation]
where
[Equation]
Proof.
When p (cid:54) = 1 2 ⇔ q > 1 4 , by Lemma G.8, taking (cid:96) =
q , we know
[Equation]
, where ( a ) is by
[Equation]
( b ) is by Lemma I.1, ( c ) is by 1 − 2 p = 4 q − 1 .
When p = 12 ⇔ q = 14 , by a similar argument, we have
[Equation]
Now we can deﬁne
[Equation]
The ﬁnal step is by noticing for 1 − 2 q = p > E
[Equation] [Equation]
G. A NALYSIS OF D T
We will prove the following bound
Lemma G.11.
Given p + 2 q = 1 , p ∈ (cid:2) 14 , 12 (cid:3) , we have
[Equation]
where
[Equation]
λ > 0 can be any number .
Proof.
The same as before
[Equation]
we start from Lemma E.
[Equation]
where λ > 0 is used to reduce the order of (cid:98) G in the ﬁnal bound.
In the proof of the general case , we don’t choose λ explicitly anymore.
Plugging in the bound on E (cid:2) E T, 1 / 2 (cid:3) in Corollary G.9, we know
[Equation]
Applying Lemma E.5 to ( i ) , we get
[Equation]
By using this bound to (34), the proof is completed
[Equation]
From Lemma G.10, we have
[Equation]
From Lemma G.11, we have
[Equation]
Now let
[Equation]
Apply Lemma E.1, we have
[Equation]
Now we can give the ﬁnal proof of Theorem G
[Equation] [Equation]
Proof.
First, we have
[Equation]
Now we consider following two cases:
Case 1: E
[Equation]
E
[Equation]
pT
[Equation]
≥ E
[Equation]
D
[Equation]
pT
[Equation]
In this case, we will ﬁnally prove
[Equation]
where C 9 is a constant.
Note that by Holder inequality
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
where ( a ) is by 1 − 2 q = p , ( b ) is due to
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
First, if q (cid:54) = 14 , we have
[Equation]
where ( c ) is by 4 q − 1= 1 − 2 p≤ 2 − 2 p= (cid:98) p and p ≥ 14 ⇒ 1 − 2 p ≤ 2 − 2 p= (cid:98) p , ( d ) is by (36) and
E (cid:104) D (cid:98) pT (cid:105) ≤ E (cid:104) E (cid:98) pT (cid:105) .
Then we know
[Equation]
 
[Equation]
 
[Equation]
If 4 E (cid:104) E (cid:98) pT (cid:105) ≤ (cid:0) 2 σ 2 T (cid:1) (cid:98) p , we will get
E (cid:104) E (cid:98) pT (cid:105) ≤ 
[Equation]
a
[Equation]
[Table] , we [Table] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
we [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
Now if 4 E (cid:104) E (cid:98) p T
[Equation]
≥ (cid:0) 2 σ 2 T (cid:1) (cid:98) p , then we have
[Equation]
We claim there is a constant C 9 such that E (cid:104) E (cid:98) pT (cid:105)
≤ C 9 because the highest order of E (cid:104) E (cid:98) pT (cid:105) is only 1 − 2 p + 2 p 3 = 1 − 4 p 3 < 1 .
Here we give the order of C 9 directly without proof
[Equation]
Hence, when q (cid:54) = 14 , we ﬁnally have
[Equation]
Following a similar approach
[Equation]
we can prove for q =
[Equation]
there is
[Equation]
where
[Equation]
Finally
[Equation]
we have
[Equation]
 
[Equation]
Case 2: E (cid:104) E
[Equation]
pT
[Equation]
≤ E
[Equation]
D (cid:98) pT (cid:105) .
In this case
[Equation]
we will ﬁnally prove
[Equation]
where C 10 is a constant.
Note that by Holder inequality
[Equation]
where ( e ) is by
[Equation]

[Table]
[Equation]
where ( f ) is by ( x + y ) p ≤ x p + y p , ( x + y ) q ≤ (2 x ) q
+ (2 y ) q for 0 ≤ x, y, 0 ≤ p ≤ 1 , q ≥ 0 , ( g ) holds by the concavity of log function, ( h ) is due to (36) and E
(cid:104) E (cid:98) pT (cid:105)
≤ E (cid:104) D (cid:98) pT (cid:105) .
Then we know
[Equation]

[Equation]
a
[Equation]
If 4 E (cid:104) D (cid:98) pT (cid:105) ≤ (cid:0) 2 σ 2 T (cid:1) (cid:98) p , we will get
[Equation] [Equation]
 
[Equation]
×
[Equation] [Equation]
If 4 E (cid:104) D (cid:98) pT (cid:105) ≥ (cid:0) 2 σ 2 T (cid:1) (cid:98) p , we have
[Equation]
38) which implies there is a constant C 10 such that E (cid:104) D (cid:98) pT (cid:105) ≤ C 10 .
Here we give the order of Cdirectly without proof
[Equation]
Combining these two results, we know
[Equation]
Finally
[Equation]
combining Case 1 and Case 2 and using
[Equation]
we get the desired result and the ﬁnish the proof
[Equation]
 
[Equation]
H A LGORITHM META-STORM-NA AND ITS ANALYSIS FOR GENERAL p
Algorithm META-STORM-NA is shown in Algorithm 4.
To highlight the differences with
META-STORM-SG and META-STORM, we set a t only based on the time round t , not using the stochastic gradients.
This is the reason that the convergence of this algorithm does not depend on bounded stochastic gradients or bounded stochastic gradients differences assumptions.
Moreover, the requirement of p ∈ (cid:0) 0 , 12 (cid:3) is also more relaxed compared with our previous algorithms.
Input: Initial point x 1 ∈ R d
Parameters: a 0 > (cid:113) 23 , b 0 , η, p ∈ (cid:0) 0 , 12 (cid:3) , p + 2 q =
Sample ξ 1 ∼ D , d 1 = ∇ f ( x 1 , ξ 1 ) for t = 1 , · · · , T do: a t +1 = (cid:0) 1 + t/a 20 (cid:1) −b t = ( b 1 /p+ (cid:80) ti =1 (cid:107)
d i (cid:107) 2 ) p /a
qt + x t +1 =
x t − ηb t d t
Sample ξ t +1 ∼ D d t +1 = ∇ f ( x t +1 , ξ t +1 )
+ (1 − a t +1 )
( d t − ∇ f ( x t , ξ t +1 )) end for
Output x out = x t where t ∼ Uniform ([ T ]) .
Now we give the main convergence result, Theorem H.1, of META-STORM-NA.
As we discussed before, it can achieve the rate (cid:101)
O
(1 /T 3 )
under the weakest assumptions 1-3, however, with losing the adaptivity to the variance parameter σ as a tradeoff.
Theorem H.1.
Under the assumptions 1-3, by deﬁning (cid:98) p = 1 − p ∈ (cid:2) 12 , 1 (cid:1) , we have (omitting the dependency on η, a 0 and b 0 )
[Equation]
By combining the above theorem with the concavity of x (cid:98) p , we give the following convergence guarantee omitting the proof:
Theorem H.2.
There is
[Equation]
Note that 2 (cid:98) p ≥ 1 , hence the criterion, E (cid:2) (cid:107)∇ F ( x out ) (cid:107) 2 (cid:98) p (cid:3) , used in Theorem H.2 is strictly stronger than E
[ (cid:107)∇ F ( x out ) (cid:107) ] .
In the following sections, we will give a proof of Theorem H.1.
H. B OUND ON E (cid:2) E T, 1 / 2 (cid:3)
Lemma H.3.
Given p + 2 q = 1 , p ∈ (cid:0) 0 , 12 (cid:3) , we have
[Equation]
Proof.
We start from Lemma E.2,
[Equation]
Dividing both sides by a 1 / 2 t +1 , summing up from 1 to T and taking the expectations on both sides, we obtain
[Equation] [Equation]
Because a t +1 is not random
[Equation]
we know
[Equation]
where the ﬁrst inequality is by Lemma E.3.
Besides, by the concavity of x 2 / 3 and a 0 > (cid:113) 23 , we know
[Equation]
Then we have
[Equation]
Note that
[Equation]
So we know
[Equation]
H.
B OUND ON E [ E T ]
Lemma H.4.
Given p + 2 q = 1 , p ∈ (cid:0) 0 , 12 (cid:3) , we have
E [ E T ]
[Equation]
a
[Equation]
(cid:0)
[Equation]
T
[Equation]
a
[Equation]
(cid:1)
[Equation] [Equation]
[Equation] [Equation]
Proof.
We start from Lemma E.2,
[Equation]
Dividing both sides by a t +1 , summing up from 1 to T and taking the expectations on both sides, we obtain
[Equation]
Because a t +1 is not random
[Equation]
we know
[Equation]
where the ﬁrst inequality is by Lemma E
[Equation]
Then we know
[Equation]
Note that there is
T
[Equation]
where ( a ) is by 1 − 2 q = p , ( b ) is by Lemma I.
[Equation]
Besides
T (cid:88)
[Equation]
So we know
[Equation] [Equation]
Lemma H.5.
Given p + 2 q = 1 , p ∈ (cid:0) 0 , 12 (cid:3) , we have E
[Equation]
p T
[Equation]
Proof.
The same as before, we start from Lemma E.
[Equation]
D 1 − p
[Equation]
Now we simply take λ = 1 and use Lemma H.3 to get
[Equation]
Applying Lemma E.5 to ( i ) , we get
[Equation]
i
[Equation]
Note that a qT +1 = a 1 − p 2 T +
=
(cid:0)
+ T/a 20 (cid:1) − 1 − p 3 is deterministic, by multiplying both sides of (39) by (cid:0)
+ T/a 20 (cid:1) 1 − p 3 , we get the desired result.
H. C OMBINE THE BOUNDS AND THE FINAL PROOF
From Lemma H.4, we have
[Equation]
From Lemma H.5, we have
[Equation]
b
[Equation]
Now let
[Equation]
(cid:19) .
Apply Lemma E.1, we know
[Equation]
Now we can give the ﬁnal proof of Theorem H.1.
Proof.
Now we consider following two cases:
[Equation]
p
[Equation]
Note that by Holder inequality
[Equation]
So we know
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Now if
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Then if
[Equation] [Equation]
, we know
[Equation]
If
[Equation]
≥ (cid:18) 6 a 20 σ 2 ( 1+ T/a 20 ) 1 /1 − 2 / (3 a 20 ) (cid:19) (cid:98) p , we know
E (cid:104) E (cid:98) pT (cid:105) ≤(cid:32)
η 2 β 2 (1 + T/a 20 ) 2 p(1 − 2 / (3 a 20 ))(1 − 2 p ) (cid:33) (cid:98) p
E 1 − 2 p (cid:104) E (cid:98) pT (cid:105) ⇒ E (cid:104) E (cid:98) pT (cid:105) ≤ (cid:32)
2 1 (cid:98) p 2 η 2 β(1 − 2 / (3 a 2 0 ))
(1 − 2 p ) (cid:33) (cid:98) p 2 p (cid:0)
+ T/a 20 (cid:1) (cid:98) p 3 .
Hence under E (cid:104) E (cid:98) pT (cid:105) ≥ E (cid:104) D (cid:98) pT (cid:105) , we get
E (cid:104) E (cid:98) pT (cid:105) ≤   (cid:32) 2 1 (cid:98) p 2 η 2 β(1 − 2 / (3 a 20 ))
(1 − 2 p ) (cid:33) (cid:98) p 2 p + (cid:32)
2 1 (cid:98) p 6 a 20 σ1 − 2 / (3 a 20 ) (cid:33) (cid:98) p 
 (cid:0) 1 + T/a 20 (cid:1) (cid:98) p 3 .
Then by using (40), we know
E (cid:104) H (cid:98) pT (cid:105) ≤ 4 max (cid:110) E (cid:104) E (cid:98) p T (cid:105) , E (cid:104) D (cid:98) p T (cid:105)(cid:111)
[Equation]
×
[Equation] [Equation]
Case 2: p = 12 .
By a similar proof, we still have
[Equation]
I B ASIC INEQUALITIES
In this section
[Equation]
we prove some technical lemmas used in our proof
[Equation]
Lemma I.1.
For c 0 > 0 , c i ≥ 1 ≥ 0 , p ∈ (0 , 1] , we have
T (cid:88)
[Equation]
Proof.
We ﬁrst prove the case p (cid:54)
= 1 .
From Lemma 3 in Levy et al.
(2021), for b 1 > 0 , b i ≥ 2 ≥ 0 , p ∈ (0 , 1) , we have
T (cid:88) t =
[Equation]
t ( (cid:80) ti =
[Equation]
i
[Equation]
≤
[Equation]
p (cid:32) T (cid:88) i =
b i (cid:33) 1 − p .
Now we deﬁne
T 0 = min { t ∈ [ T ] : c t > 0 } .
By the deﬁnition of T 0 , we know for any 1 ≤ t ≤ T 0 − 1 , c t = 0 .
Then we have
T
[Equation]
[Equation]
≤
[Equation]
=
[Equation]
p
[Equation]
i =
[Equation]
c
[Equation]
− p .
For p = 1 , we know
[Equation]
≤
[Equation]
where the inequality holds by 1 − 1 x ≤ log x .
Lemma I.2.
For c 0 > 0 , c i ≥ 1 ∈ (0 , c ] , p ∈ (0 , 1] , we have
[Equation]
Proof.
Deﬁne
[Equation]
, then we know
T (cid:88) t =
c t + ( c 0 + (cid:80) ti =
1 c i ) p ≤
T − 1 (cid:88) t =
c t + ( [Equation] (cid:80) ti =
[Equation] i ) p + [Equation] =
[Equation]
c
[Equation]
T 0 − 1 (cid:88) [Equation] c [Equation]
+
[Equation] ( c [Equation] + (cid:80) ti =1 c
[Equation]
) p + T −
[Equation]
(cid:88) [Equation]
= Tc
[Equation] + [Equation] [Equation]
{+[Equation] (cid:80) T 
 i =1+} c [Equation]
+ (cid:80) ti =
T
+1 c i ) p ≤ [Equation]
[Equation] [Equation] T − [Equation]
= T [Equation] + [Equation]
p ≤ [Equation] c c
[Equation] T − [Equation] (cid:88) t =
Tc t
+ ([-c 0 + c + (cid:80) ti = T 
 +1 c i ) p ≤ 3 c c p 0 +-]
[Equation]
[Equation] (cid:80) t +1 i
=
T
+1 c i ) p (
[Equation]
where ( a ) is by Lemma
I
[Equation]
[Equation]
Lemma I.3.
For c 0 > 0 , c i ≥ 1 ∈ (0 , c ] , p ∈ (0 , 1] , we have
[Equation]
Proof.
Deﬁne
[Equation]
Then we know
[Equation]
[-T 
-]
c i ) p ≤
T 0 − 1 (cid:88) t =
c t + c p 0 +
T (cid:88) t =
Tc t
+ ( c 0 + c + (cid:80) t − 1 i =
T
c i ) p ≤ 3 c c p 0 +
T (cid:88) t =
Tc t
+ ( c 0 + (cid:80) ti =
T 0 c i ) p ( a ) ≤ 6 c c p 0 +   11 − p (cid:16)(cid:80)
Ti = T
c i (cid:17) 1 − p p (cid:54) =log (cid:18) 1 + (cid:80)
Ti = T
c i c(cid:19) p =
[Equation]
where ( a ) is by Lemma I.2.
Lemma I.4.
(Lemma 6 in Levy et al.
(2021)), for c i ≥ 1 ∈ (0 , c ] , we have
T (cid:88)
[Equation]
=
[Equation]
Lemma I.5.
For c i ≥ 1 ∈ (0 , c ] , we have, we have
[Equation]
Proof.
Deﬁne
T 0 = min (cid:40) t ∈ [ T ] : t − 1 (cid:88) i
=
c
i ≥ c (cid:41) .
Then we know
T (cid:88)
[Equation]
c t + (
[Equation]
=
[Equation]
c
[Equation]
/≤ 3 c [Equation] T (cid:88) t = Tc t
[Equation] [Equation] ti [Equation]
T 0 c i )
[Equation]
/≤ 12 + 5 c, where the last inequality is by Lemma I.4.
Lemma I.6.
Given 0 ≤ x ≤ y ≤ 1 , 0 < (cid:96) ≤ 1 , we have
[Equation] [Equation]
Proof.
Note that
[Equation]
, now let h ( x ) = 1 − x 1 /l x , we can ﬁnd h (cid:48)
( x ) = − (1 − (cid:96) )
x 1 /(cid:96)
+ (cid:96) (cid:96)x≤ 0 .
Hence 1 − x 1 /(cid:96)
x − 1 − y 1 /(cid:96)
y
= h ( x ) − h ( y ) ≥ 0 .
Besides, let g ( x )
= h ( x ) − 1 (cid:96)x , we can ﬁnd that
[Equation]
This means
[Equation]
which implies
[Equation]
Thus we ﬁnally have
[Equation]
Lemma I.7.
Given 0 ≤ x ≤ y ≤ 1 , 0 < (cid:96) ≤ 12 , we have
[Equation]
Proof.
If (cid:96) = 12 , then we know
[Equation]
If (cid:96) (cid:54) = 12 , let h ( x ) denote (1 − x 1 /(cid:96) )
x 1 /(cid:96) − 2 , then we know h (cid:48)
( x ) = x 1 /(cid:96)
− 3 2( (cid:96) − 1) x 1 /(cid:96) − 2 (cid:96) +1 (cid:96) .
By
Taylor’s expansion, there exists x ≤ z ≤ y , such that
[Equation]
This will give us
[Equation]
Lemma I.8.
Given m, n ≥ 0 , For 0 ≤ x ≤ m , we have
[Equation]
Proof.
Note that
[Equation]
where ( a ) is by the concavity of log function.
Then we know ( m − x ) x n ≤ (cid:16) mn +1 (cid:17) n +1 n n .
Lemma I.9.
Given X, A, B ≥ 0 , C > 0 , D ≥ 0 , 0 ≤ u ≤ 1 ,if we have
[Equation]
C
[Equation]
D
[Equation]
then there is
X
[Equation]
Especially, when D ≥ 1 , we know
[Equation]
Proof.
Let Y =
( X/D ) 1 /u , then we know
[Equation] [Equation]
where ( a ) is by ( x + y ) p ≤ (2 x ) p + (2 y ) p , for x, y ≥ 0 , p ≥ 1 .
( b ) is by log x ≤ x − 1 ≤ x .
Then we know
[Equation]