U NIVERSAL S TAGEWISE L EARNING FOR N ONC ONVEX P ROBLEMS WITH C ONVERGENCE ON A VERAGED S OLUTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Although stochastic gradient descent (S GD ) method and its variants (e.g., stochastic momentum methods, A DA G RAD ) are algorithms of choice for solving nonconvex problems (especially deep learning), big gaps still remain between the theory and the practice with many questions unresolved.
For example, there is still a lack of theories of convergence for S GD and its variants that use stagewise step size and return an averaged solution in practice.
In addition, theoretical insights of why adaptive step size of A DA G RAD could improve non-adaptive step size of S GD is still missing for non-convex optimization.
This paper aims to address these questions and ﬁll the gap between theory and practice.
We propose a universal stagewise optimization framework for a broad family of non-smooth nonconvex problems with the following key features: (i) at each stage any suitable stochastic convex optimization algorithms (e.g., S GD or A DA G RAD ) that return an averaged solution can be employed for minimizing a regularized convex problem; (ii) the step size is decreased in a stagewise manner; (iii) an averaged solution is returned as the ﬁnal solution.
Our theoretical results of stagewise A DA G RAD exhibit its adaptive convergence, therefore shed insights on its faster convergence than stagewise S GD for problems with slowly growing cumulative stochastic gradients.
To the best of our knowledge, this is the ﬁrst work for addressing the unresolved issues of existing theories mentioned earlier.
Besides theoretical contributions, our empirical studies show that our stagewise S GD and A DA G RAD improve the generalization performance of existing variants/implementations of
S GD and A DA G RAD .
I NTRODUCTION
Non-convex optimization has recently received increasing attention due to its popularity in emerging machine learning tasks, particularly for learning deep neural networks.
One of the keys to the success of deep learning for big data problems is the employment of simple stochastic algorithms such as S GD or A DA G RAD (Krizhevsky et al., 2012; Dean et al., 2012).
Analysis of these stochastic algorithms for non-convex optimization is an important and interesting research topic, which already attracts much attention from the community of theoreticians (Ghadimi & Lan, 2013; 2016; Yang et al., 2016; Davis & Drusvyatskiy, 2018b; Ward et al., 2018; Li & Orabona, 2018).
However, one issue that has been largely ignored in existing theoretical analysis is that the employed algorithms in practice usually differ from their plain versions that are well understood in theory.
Below, we will discuss several important heuristics used in practice for training deep neural networks and the gap between the practice and the theory to motivate this work.
First, a trick for setting the step size in training deep neural networks is to change it in a stagewise manner from a large value to a small value, i.e., a constant step size is used in a stage for a number of iterations and is decreased for the next stage (Krizhevsky et al., 2012; Ren et al., 2018).
Although this trick has been adopted by most open-sourced libraries, e.g., Caffe (Jia et al., 2014), TensorFlow (Abadi et al., 2015), Pytorch (Paszke et al., 2017), it still lacks theoretical analysis to date for non-convex optimization.
A related question that stands out is how and when to decrease the step size.
On standard benchmark datasets for academic use such as CIFAR-10, CIFAR-100 (Krizhevsky et al.), people could follow the setting reported in previous studies to get a good result, which however might not work well for new datasets.
Hence, a better solution to setting the stagewise step size with insights from theory would be much preferred.
However, in the existing literature of theory for non-convex optimization (Ghadimi & Lan, 2013; Davis & Drusvyatskiy, 2018b), only strategies based on an iteratively decreasing step size or a small constant step size have been well analyzed.
For example, the existing theory usually suggests an iteratively decreasing step size proportional to 1 / √ t at the t -th iteration or a small constant step size, e.g., proportional to (cid:15) 2 with (cid:15) (cid:28) 1 for ﬁnding an (cid:15) -stationary solution whose gradient’s magnitude (in expectation) is small than (cid:15) .
Second, the averaging heuristic is usually used in practice, i.e., an averaged solution is returned for prediction (Bottou, 2010), which could yield improved stability and generalization (Hardt et al., 2016).
However, existing theory for many stochastic non-convex optimization algorithms only provides guarantee on a uniformly sampled solution or a non-uniformly sampled solution with decreasing probabilities for latest solutions (Ghadimi & Lan, 2013; Yang et al., 2016; Davis & Drusvyatskiy, 2018b).
In particular, if an iteratively decreasing step size proportional to 1 / √ t at the t -th iteration is employed, the convergence guarantee was provided for a random solution that is non-uniformly selected from all iterates with a sampling probability proportional to 1 / √ t for the t -th iterate.
This means that the latest solution always has the smallest probability to be selected as the ﬁnal solution, which contradicts to the common wisdom.
If a small constant step size is used, then usually a uniformly sampled solution is provided with convergence guarantee.
However, both options are rarely used in practice and cannot justify the heuristic that returns the last solution.
A third common approach in practice is to use adaptive coordinate-wise step size of A DAG RAD
(Dean et al., 2012).
Although adaptive step size has been well analyzed for convex problems (e.g., why and when it could yield faster convergence than S GD ) (Duchi et al., 2011; Chen et al., 2018b), it still remains an mystery for non-convex optimization with missing insights from theory.
Several recent studies have attempted to analyze A DA G RAD for non-convex problems (Ward et al., 2018; Li & Orabona, 2018;
Chen et al., 2018a; Zou & Shen, 2018).
Nonetheless, none of them are able to exhibit the adaptive convergence of A DA G RAD to data as in the convex case and its potential advantage over S GD for non-convex problems.
To overcome the shortcomings of existing theories for stochastic non-convex optimization, this paper analyzes new algorithms that employ some or all of these commonly used heuristics in a systematic framework, aiming to ﬁll the gap between theory and practice.
The main results and contributions are summarized below:
• We propose a universal stagewise optimization framework for solving a family of non-convex problems, i.e., weakly convex problems, which is broader than smooth non-convex problems and includes some non-smooth non-convex problems.
At each stage, any suitable stochastic convex optimization algorithms (e.g., S GD , A DA G RAD ) with a constant step size parameter can be employed for optimizing a regularized convex problem with a number of iterations, which usually return an averaged solution.
The step size parameter is decreased in a stagewise manner following a polynomial decaying scheme.
• We analyze several variants of the proposed framework by employing different basic algorithms, including S GD , A DA G RAD , stochastic heavy-ball (S HB ) method, and stochastic Nesterov’s accelerated gradient (S NAG ) method.
We prove the convergence of their stagewise versions for an averaged solution that is randomly selected from all stagewise averaged solutions.
• To justify a heuristic approach that returns the last averaged solution in stagewise learning, we present and analyze a non-uniform sampling strategy over stagewise averaged solutions with sampling probabilities increasing as the stage number.
• Regarding the convergence results, for stagewise S GD , S HB , S NAG , we establish the same order of iteration complexity for ﬁnding a nearly stationary point as the existing theories of their nonstagewise variants.
For stagewise A DA G RAD , we establish an adaptive convergence for ﬁnding a nearly stationary point, which is provably better than (stagewise) S GD , S HB , and S NAG when the cumulative growth of stochastic gradient is slow.
• Besides theoretical contributions, we also empirically verify the effectiveness of the proposed stagewise algorithms.
In particular, our empirical studies show that (i) the stagewise A DA G RAD dramatically improves the generalization performance of existing variants of A DA G RAD , (ii) stagewise S GD , S HB , S NAG also outperform their plain variants with an iteratively decreasing step size; (iii) the proposed stagewise algorithms achieve similar if not better generalization performance than their heuristic variants implemented in existing libraries on standard benchmark datasets.
R ELATED W ORK
S GD for unconstrained smooth non-convex problems was ﬁrst analyzed by Ghadimi & Lan (2013), who established an O (1 /(cid:15) 4 ) iteration complexity for ﬁnding an (cid:15) -stationary point x in expectation satisfying E
[ (cid:107)∇ f ( x ) (cid:107) ] ≤ (cid:15) , where f ( · ) denotes the objective function.
As mentioned earlier, the returned solution is either a uniformly sampled solution or a non-uniformly sampled one with sampling probabilities proportional to the decreasing step size.
Similar results were established for the stochastic momentum variants of S GD (i.e., S HB , S NAG ) by Yang et al.
; Ghadimi & Lan (2016).
Recently, SGD was also analyzed for (constrained) weakly convex problems, whose objective function is non-convex and not necessarily smooth, by Davis & Drusvyatskiy (2018b).
However, none of these studies provide results for algorithms that return an averaged solution, and these analyzed algorithms differ signiﬁcantly from that used in practice for achiving the state-of-the-art results (Krizhevsky et al., 2012;
Ren et al., 2018; Loshchilov & Hutter, 2017).
Although adaptive variants of S GD , e.g., A DA G RAD (Duchi et al., 2011), A DAM (Kingma & Ba, 2015; Reddi et al., 2018), were widely used for training deep neural networks, there are few studies on theoretical analysis of these algorithms for non-convex problems.
Several recent studies attempted to analyze A DA G RAD for non-convex problems (Ward et al., 2018; Li & Orabona, 2018;
Chen et al., 2018a; Zou & Shen, 2018).
Although these studies have established an iteration complexity of O (1 /(cid:15) 4 ) for different variants of A DA G RAD for ﬁnding an (cid:15) -stationary solution of a stochastic non-convex optimization problem, none of them can exhibit the potential adaptive advantage of A DA G RAD over S GD as in the convex case.
Besides that, these studies also suffer from the following shortcomings: (i) they all assume smoothness of the objective function, while we consider non-smooth and non-convex problems; (ii) their convergence is provided on a solution with minimum magnitude of gradient that is expensive to compute, though their results also imply a convergence on a random solution selected from all iterates with decreasing sampling probabilities.
In contrast, these shortcomings do not exist in this paper.
To the best of our knowledge, our result is the ﬁrst one that explicitly shows that coordinate-wise adaptive step size could yield faster convergence than using non-adaptive step size for non-smooth non-convex problems , which is similar to that in the convex case and was observed in practice for deep learning (Dean et al., 2012).
The proposed stagewise algorithm is similar to several existing algorithms in design (Xu et al., 2017; Davis & Grimmer, 2017), which are originated from the proximal point algorithm (Rockafellar, 1976).
I.e., at each stage a regularized convex subproblem is formed and then a stochastic algorithm is employed for optimizing the regularized subproblem inexactly with a number of iterations.
Xu et al.
(2017) used this idea for solving problems that satisfy a local error bound condition, aiming to achieve faster convergence than vanilla S GD .
Davis & Grimmer (2017) are probably the ﬁrst who analyzed this idea for solving non-smooth weakly convex problems.
In these two papers S GD with decreasing step sizes for a strongly convex problem is employed at each stage for solving the regularized subproblem.
Our stagewise algorithm is developed following the similar idea.
The key differences from (Xu et al., 2017; Davis & Grimmer, 2017) are that (i) we focus on non-convex problems instead of convex problems considered in (Xu et al., 2017); (ii) we analyze a non-uniform sampling strategy with sampling probabilities increasing as the stage number to justify a heuristic approach that uses the last averaged solution for prediction, unlike the uniform sampling used in (Davis & Grimmer, 2017); (iii) we present a uniﬁed algorithmic framework and convergence analysis, which enable one to employ any suitable stochastic convex optimization algorithms at each stage.
Importantly, it brings us several interesting variants including stagewise stochastic momentum methods and stagewise A DA G RAD .
Finally, we note that a similar idea that adds a strongly convex term into the non-convex objective has been considered in several recent works (Allen-Zhu, 2017;
Lan & Yang, 2018; Carmon et al., 2016).
However, the key difference between these works and the present work is that they need to assume the objective function is smooth or has a smooth component.
In addition, Allen-Zhu (2017); Lan & Yang (2018) consider ﬁnite-sum problems, and Carmon et al.
(2016) consider deterministic problems.
In contrast, we consider more general stochastic problems without assuming the objective function is smooth.
P RELIMINARIES
The problem of interest in this paper is:
[Equation]
where Ω ⊆ R d is a closed convex set, ξ ∈ U is a random variable, φ ( x ) and φ ( x ; ξ ) are non-convex functions, with the basic assumptions on the problem given in Assumption 1.
To state the convergence property of an algorithm for solving the above problem.
We need to introduce some deﬁnitions.
These deﬁnitions can be also found in related literature, e.g., Davis &
Grimmer (2017); Davis & Drusvyatskiy (2018b).
In the sequel, we let (cid:107) · (cid:107) denote an Euclidean norm, [ S ] = { 1 , . . .
, S } denote a set, and δ Ω ( · ) denote the indicator function of the set Ω .
Deﬁnition 1.
(Fr´echet subgradient)
For a non-smooth and non-convex function f ( · ) ,
[Equation]
denotes the Fr´echet subgradient of f .
Deﬁnition 2.
(First-order stationarity)
For problem (1), a point x ∈ Ω is a ﬁrst-order stationary point if 0 ∈ ∂ F ( φ + δ Ω )( x ) , where δ Ω denotes the indicator function of Ω .
Moreover, a point x is said to be (cid:15) -stationary if dist (0 , ∂ F ( φ + δ Ω )( x )) ≤ (cid:15) , where dist denotes the Euclidean distance from a point to a set.
Deﬁnition 3.
(Moreau Envelope and Proximal Mapping)
For any function f and λ > 0 , the following function is called a Moreau envelope of f
[Equation]
Further, the optimal solution to the above problem denoted by
[Equation]
is called a proximal mapping of f .
Deﬁnition 4.
(Weakly convex)
A function f is µ -weakly convex ( µ > 0 ), if f ( x ) + µ 2 (cid:107) x (cid:107) 2 is convex.
It is known that if f ( x ) is µ -weakly convex and γ < µ − 1 , then its Moreau envelope f γ ( x ) is
C 1 -smooth with the gradient given by (see e.g., Davis & Drusvyatskiy (2018b))
[Equation]
The tool of Moreau envelope is introduced to measure the convergence for optimizing non-smooth and non-convex functions.
A small norm of ∇ f γ ( x ) has an interpretation that x is close to a point that is nearly stationary.
In particular for any x ∈ R d , let (cid:98) x
= prox γf
( x ) , then we have (Davis &
Drusvyatskiy, 2018b)
[Equation]
where the ﬁrst inequality follows from the deﬁnition of prox λf ( x ) such that f ( (cid:98) x ) + 12 λ (cid:107) (cid:98) x − x (cid:107) 2 ≤ f ( x ) (Please see Appendix H for a detailed proof of (3)).
This means that a point x satisfying (cid:107)∇ f γ ( x ) (cid:107) ≤ (cid:15) is close to a point in distance of O ( (cid:15) ) that is (cid:15) -stationary.
It is notable that for a non-smooth function f ( · ) , there could exist a sequence of solutions { x k } such that ∇ f γ ( x k ) converges while dist (0 , ∂f ( x k )) may not converge (Drusvyatskiy & Paquette, 2018).
A simple example is to consider min x ∈ R | x | .
As long as x (cid:54) = 0 , dist (0 , ∂f ( x ))
= 1 (cid:54) =
no matter how close is x to the stationary point 0 .
To handle such a challenging issue for non-smooth non-convex problems, we will follow existing works (Davis & Drusvyatskiy, 2018a; Drusvyatskiy & Paquette, 2018; Davis & Grimmer, 2017) to prove the near stationarity in terms of ∇ f γ ( x ) .
In the case when f is smooth, (cid:107)∇ f γ ( x ) (cid:107) is closely related to the magnitude of the gradient.
In particular, let us deﬁne the projected gradient below, which is used as a criterion for non-convex smooth optimization in the presence of constraints (Ghadimi & Lan, 2016; Allen-Zhu, 2017):
[Equation]
Note that the projected gradient becomes the normal gradient when the constraint x ∈ Ω is absent.
It was shown that when f ( · ) is smooth with L -Lipschitz continuous gradient (Drusvyatskiy & Lewis, 2016):
[Equation]
1: Initialize: a sequence of decreasing step size parameters { η s } , x 0 ∈ Ω , γ < µ −2: for s = 1 , . . .
, S do 3:
Let f s ( · ) =
φ ( · ) + 12 γ (cid:107) · − x s − 1 (cid:107)4: x s = SA ( f s , x s − 1 , η s , T s ) (cid:5) x s is usually an averaged solution 5: end for
Thus, the near stationarity in terms of ∇ f γ ( x ) implies the near stationarity in terms of G γ ( x ) for a smooth function f ( · ) for a properly chosen γ > 0 (e.g., γ = 1 / (2 L ) ).
In this work, we deﬁne φ γ ( x ) as the Moreau envelope of φ ( x ) + δ Ω ( x ) as in (2) with f ( x ) replaced by φ ( x ) + δ Ω ( x ) and study the convergence in terms of of ∇ φ γ ( x ) .
Now, we are ready to state the basic assumptions of the considered problem (1).
Assumption 1.
(i)
There is a measurable mapping g : Ω × U
→ R such that E ξ [ g ( x ; ξ )] ∈
∂ F φ ( x ) for any x ∈ Ω ; (ii) for any x ∈ Ω , E[ (cid:107) g ( x ; ξ ) (cid:107) 2 ] ≤ G 2 ; (iii) there exists ∆ φ > 0 such that φ ( x )
− min z ∈ Ω
φ ( z ) ≤ ∆ φ for any x ∈ Ω ; (iv) the objective function φ is µ -weakly convex;
Remark: Assumption 1(i), (ii) assume a stochastic subgradient is available for the objective function and its Euclidean norm square is bounded in expectation, which are standard assumptions for nonsmooth optimization.
Assumption 1(iii) assumes that the objective value with respect to the optimal value is bounded.
Assumption 1(iv) assumes weak convexity of the objective function, which is weaker than assuming smoothness.
Below, we present some examples of objective functions in machine learning that are weakly convex.
Ex. 1: Smooth Non-Convex Functions.
If φ ( · ) is a L -smooth function (i.e., its gradient is LLipschitz continuous), then it is L -weakly convex.
This will include the objective function for neural networks with a smooth activation function (e.g., the sigmoid function) and a smooth loss function (e.g., softmax loss).
Ex. 2: Convex and Smooth Composition.
Consider φ ( x ; ξ )
=
h ( c ( x ; ξ )) where h ( · ) :
R m → R is closed convex and M -Lipschitz continuous, and c ( x ; ξ ) : R d → R m is nonlinear smooth mapping with L -Lipschitz continuous gradient.
This class of functions has been considered in (Drusvyatskiy & Paquette, 2018) and it was proved that φ ( x ; ξ ) is ML -weakly convex.
An interesting example is phase retrieval, where φ ( x ; a, b ) =
| ( x (cid:62) a ) 2 − b | .
Another example related to deep learning is that if c ( x ; ξ ) denotes the mapping function of a neural network parameterized by x with a smooth activation function and h denotes a non-smooth Lipschitz continuous convex loss function (e.g., hinge loss), then the resulting loss h ( c ( x ; ξ )) is a weakly convex function of x .
More examples of this class can be found in (Davis & Drusvyatskiy, 2018a).
S TAGEWISE O PTIMIZATION : A LGORITHMS AND C ONVERGENCE
In this section, we will present the proposed stagewise algorithms and their convergence results.
We will ﬁrst present a Meta algorithmic framework highlighting the key features of the proposed algorithms and then present several variants of the Meta algorithm.
The Meta algorithmic framework is described in Algorithm 1.
There are several key features that differentiate Algorithm 1 from existing stochastic algorithms that come with theoretical guarantee.
First, the algorithm is run with multiple stages.
At each stage, a basic stochastic algorithm (SA) is called to optimize a regularized problem f s ( x ) inexactly that consists of the original objective function and a quadratic term, which is guaranteed to be convex due to the weak convexity of φ and γ < µ − 1 .
The convexity of f s allows one to employ any suitable existing stochastic algorithms (cf.
Theorem 1) that have convergence guarantee for convex problems.
The returned solution from the ( s − 1) -th stage is used as a reference point for constructing f s and as an initial solution for warm-start.
It is notable that SA usually returns an averaged solution x s at each stage.
Second, a decreasing sequence of step size parameters η s is used.
At each stage, the SA uses a constant step size parameter η s and runs the updates for a number of T s iterations.
We do not initialize T s as it might be adaptive to the data as in stagewise A DA G RAD .
The setup of η s and T s will depend on the speciﬁc choice of SA, which will be exhibited later for different variants.
To illustrate that Algorithm 1 is a universal framework such that any suitable SA algorithm can be employed, we present the following result by assuming that SA has an appropriate convergence for a convex problem.
Theorem 1.
Let f ( · ) be a convex function, x ∗ =
arg min x ∈ Ω f
( x ) and Θ denote some problem dependent constants.
Suppose for x + = SA ( f, x 0 , η, T ) , we have
[Equation]
[Equation]
Under Assumption 1(i), (iii) and (iv), by running Algorithm 1 with γ = 1 / (2 µ ) , and with η s , T s satisfying ε 1 ( η s , T s , Θ) ≤ 1 / (48 γ ) , ε 2 ( η s , T s , Θ) ≤ 1 / 2 , ε 3 ( η s , T s , Θ) ≤ c 3 /s for some c 3 > 0 , we have
[Equation]
where τ is randomly selected from { 0 , . .
. , S } with probabilities p τ ∝ ( τ + 1) α , α ≥ 1 .
Remark: It is notable that the convergence guarantee is provided on a stagewise average solution x τ .
To justify a heuristic approach that returns the ﬁnal average solution for prediction, we analyze a new sampling strategy that samples a solution among all stagewise average solutions with sampling probabilities increasing as the stage number increases.
This sampling strategy is better than uniform sampling strategy or a strategy with decreasing sampling probabilities in the existing literature.
The convergence upper bound in (6) of SA covers the results of a broad family of stochastic convex optimization algorithms.
When ε 2 ( η s , T s , Θ) =
(as in S GD ), the upper bound can be improved by a constant factor.
Moreover, we do not optimize the value of γ .
Indeed, any γ < 1 /µ will work, which only has an effect on constant factor in the convergence upper bound.
Next, we present several variants of the Meta algorithm by employing S GD , A DA G RAD , and stochastic momentum methods as the basic SA algorithm, to which we refer as stagewise S GD , stagewise A DA G RAD , and stagewise stochastic momentum methods, respectively.
It is worth mentioning that one can follow similar analysis to analyze other stagewise algorithms by using their basic convergence for stochastic convex optimization, including RMSProp (Mukkamala & Hein, 2017), AMSGrad (Reddi et al., 2018).
4. S TAGEWISE S GD
In this subsection, we analyze the convergence of stagewise S GD , in which S GD shown in Algorithm 3 in the Appendix is employed in the Meta framework.
Besides Assumption 1, we impose the following bounded domain assumption in this subsection.
Assumption 2.
There exists D > 0 such that (cid:107) x − y (cid:107) ≤ D for any x , y ∈ Ω .
It is worth mentioning that bounded domain assumption is imposed for simplicity, which is usually assumed in convex optimization.
For machine learning problems, one usually imposes some bounded norm constraint to achieve a regularization.
Nevertheless, the bounded domain assumption is not essential for the proposed algorithm.
We present a more involved analysis in subsection 4.3 for unbounded domain Ω = R d .
The following is a standard basic convergence result of S GD (Zinkevich, 2003), which clearly satisﬁes the bound in (6).
Lemma 1.
For Algorithm 3, assume that f ( · ) is convex and E (cid:107) g t (cid:107) 2 ≤ G 2 , ∀ t , then for any x ∈ Ω
[Equation]
The following theorem exhibits the convergence of stagewise S GD .
Theorem 2.
Suppose Assumption 1 and 2 hold.
By setting γ = 1 / (2 µ ) , η s = η 0 /s, T s = 12 γs/ηwhere η 0 > 0 is a free parameter, then stagewise S GD (Algorithm 1 employing S GD ) guarantees that
[Equation]
where ˆ G 2 = 2 G 2 + 2 γ − 2 D 2 , and τ is similarly deﬁned as in Theorem 1.
Remark:
To ﬁnd a solution with E (cid:2) (cid:107)∇ φ γ ( x τ ) (cid:107) 2 (cid:3) ≤ (cid:15) 2 , we can set S
= O (1 /(cid:15) 2 ) and the total iteration complexity (cid:80) Ss =1 T s is in the order of O (1 /(cid:15) 4 ) .
The above theorem is essentially a corollary of Theorem 1 by applying Lemma 1 to f s ( · ) at each stage.
We present a complete proof in the appendix.
1: Initialize: x 1 =
0 , g 1:0 = [] , H d 2: while T does not satisfy the condition in Theorem 3 do 3:
Compute a stochastic subgradient g for
Update g 1: t =
[ g 1: t − 1 , g ( x t )] , s t,i = (cid:107) g 1: t,i (cid:107)5:
Set H t
= H 0 + diag
( s t ) and ψ t ( x ) = 12 ( x − x 1 ) (cid:62) H t ( x − x 1 ) 6:
Let x t +1 = arg min x ∈ Ω η x (cid:62) (cid:16) 1 t (cid:80) tτ
=1 g τ (cid:17) + 1 t ψ t ( x ) 7: end while 8: Output : x T =
Tt =1 x t /T
S TAGEWISE A DA G RAD
One of the main contributions of the present work is to develop a variant of A DA G RAD with adaptive convergence to data for stochastic non-convex optimization.
In this subsection, we analyze stagewise
A DA G RAD and establish its adaptive complexity.
In particular, we consider the Meta algorithm that employs A DA G RAD in Algorithm 2 to optimize each f s .
The key difference of stagewise A DA G RAD from stagewise S GD is that the number of iterations T s at each stage is adaptive to the history of learning.
It is this adaptiveness that makes the proposed stagewise A DA G RAD achieve adaptive convergence.
It is worth noting that such adaptive scheme has been also considered in (Chen et al., 2018b) for solving stochastic strongly convex problems.
In contrast, we consider stochastic weakly convex problems.
Similar to previous analysis of A DA G RAD (Duchi et al., 2011; Chen et al., 2018b), we assume (cid:107) g
( x ; ξ ) (cid:107) ∞ ≤ G, ∀ x ∈ Ω in this subsection.
Note that this is stronger than Assumption(ii).
We formally state this assumption required in this subsection below.
Assumption 3.
(cid:107) g
( x ; ξ ) (cid:107) ∞ ≤ G for any x ∈ Ω .
The convergence analysis of stagewise A DA G RAD is build on the following lemma, which is attributed to Chen et al.
(2018b)
with a proof sketch provided in the Appendix.
Lemma 2.
Let f ( x ) be a convex function, H
=
GI with G ≥ max t (cid:107) g t (cid:107) ∞ , and iteration number
T satisfy T ≥ M max { G +max i (cid:107) g 1: T,i (cid:107) 2 c , c (cid:80) di =1 (cid:107) g 1: T,i (cid:107)} for some c > 0 .
Algorithm 2 returns an averaged solution (cid:98) x T such that
[Equation]
where x ∗ = arg min x ∈ Ω f ( x ) , g 1: t
=
( g ( x 1 ) , . . .
, g ( x t )) and g 1: t,i denotes the i -th row of g 1: t .
The convergence property of stagewise A DA G RAD is described by following theorem.
Theorem 3.
Suppose Assumption 1(i), (iii), (iv), Assumption 2 and Assumption 3 hold.
By setting γ = 1 / (2 µ ) , η s = η 0 / √ s , T s ≥ M s max { ( ˆ G + max
i
(cid:107) g s 1: T s ,i (cid:107) ) /
(2 c ) , c (cid:80) di =1 (cid:107) g s 1: T s ,i (cid:107)} where η 0 , c > 0 are free parameters, and M s η s ≥ 24 γc , then stagewise A DA G RAD guarantees that
[Equation]
where ˆ G = G + γ − 1 D , g s 1: t,i denotes the cumulative stochastic gradient of the i -th coordinate at the s -th stage, and τ is similarly deﬁned as in Theorem 1.
Remark:
Note that the free parameter c is introduced to balance the two terms in the lower bound of T s due to that ( ˆ G + max
i (cid:107) g s 1: T s ,i (cid:107) )
∝ ˆ G √ T s and (cid:80) di =1 (cid:107) g s 1: T s ,i (cid:107)
∝ d ˆ G √ T s have different orders when d is very large.
One way to balance these two terms is to set c in the order of (cid:112) 1 /d , resulting O ( d/ ( S + 1)) for the second term in the above convergence bound.
Another way is to choose c in the s -th stage such that the two terms in the max of the lower bound of T s match each other.
One can derive a similar order of O ( d/ ( S + 1)) for the second term in the above convergence bound.
It is obvious that the total number of iterations (cid:80) Ss =1 T s is adaptive to the data.
Next, let us present more discussion on the iteration complexity.
Note that M s =
O ( √ s ) by setting c as a constant.
By the boundness of stochastic gradient (cid:107) g s 1: T s ,i (cid:107) ≤ O ( √ T s ) , therefore T s in the order of O ( s ) will satisfy the condition in Theorem 3.
Thus, in the worst case the iteration complexity for ﬁnding E[ (cid:107)∇ φ γ ( x τ ) (cid:107) 2 ] ≤ (cid:15) 2 is in the order of (cid:80) Ss =1 O ( s ) ≤
O (1 /(cid:15) 4 ) .
To show the potential advantage of adaptive step size as in the convex case, let us consider a good case
[Figure] [Figure]
when the cumulative growth of stochastic gradient is slow, e.g., assuming (cid:107) g s 1: T s ,i (cid:107) ≤ O ( T sα ) with α < 1 / 2 .
Then T s =
O ( s 1 / (2(1 − α )) ) will work, and then the total number of iterations (cid:80) S s =1 T s ≤ S 1+1 / (2(1 − α ))
≤ O (1 /(cid:15)
+1 / (1 − α ) ) , which is better than O (1 /(cid:15) 4 ) .
4. S TAGEWISE STOCHASTIC MOMENTUM (SM) METHODS
Finally, we present stagewise stochastic momentum (SM) methods and their convergence results.
In the literature, there are two popular variants of stochastic momentum methods, namely, stochastic heavy-ball method (S HB ) and stochastic Nesterov’s accelerated gradient method (S NAG ).
Both methods have been used for training deep neural networks (Krizhevsky et al., 2012; Sutskever et al., 2013), and have been analyzed by (Yang et al., 2016) for non-convex optimization.
To contrast with the results in (Yang et al., 2016), we will consider the same uniﬁed stochastic momentum methods that subsume S HB , S NAG and S GD as special cases when Ω = R d .
The updates are presented in
Algorithm 4 in the Appendix.
There are two additional parameters: β ∈ (0 , 1) is the momentum parameter and ρ is a parameter that can vary between [0 , 1 / (1 − β )] .
By changing the value of ρ , we can obtain the three variants, S HB ( ρ = 0 ), S NAG ( ρ = 1 ) and S GD ( ρ = 1 / (1 − β ) ).
Due to the limit of space, we only present the convergence of stagewise SM methods below.
Theorem 4.
Suppose Assumption 1 holds.
By setting γ = 1 / (2 µ ) , η s = (1 − β ) γ/ (96 s ( ρβ + 1)) ,
T s ≥ 2304( ρβ + 1) s , with τ similarly deﬁned as in Theorem 1 stagewise SM methods guarantee
[Equation]
Remark:
The bound in the above theorem is in the same order as that in Theorem 2.
The total iteration complexity for ﬁnding a solution x τ with E (cid:2) (cid:107)∇ φ γ ( x τ ) (cid:107) 2 (cid:3) ≤ (cid:15) 2 is O (1 /(cid:15) 4 ) similar to that achieved in (Yang et al., 2016).
E XPERIMENTS
In this section, we present some empirical results to verify the effectiveness of the proposed stagewise algorithms.
We use two benchmark datasets, namely CIFAR-10 and CIFAR-100 (Krizhevsky et al.)
for our experiments.
We implement the proposed stagewise algorithms in TensorFlow.
We compare different algorithms for learning ResNet-20 (He et al., 2016) with batch normalization (Ioffe & Szegedy, 2015) adopted after each convolution and before ReLu activation.
Baselines.
We compare the proposed stagewise algorithms with their variants implemented in TensorFlow.
It is notable that A DA G RAD has a step size (aka learning rate) parameter 1 , which is a constant in theory (Li & Orabona, 2018; Chen et al., 2018a; Zou & Shen, 2018).
However, in the deep learning community a heuristic ﬁxed frequency decay scheme for the step size parameter is commonly adopted (Ren et al., 2018; Wilson et al., 2017).
We thus compare two implementations of A DA G RAD - one with a constant learning rate parameter and another one with a ﬁxed frequency decay scheme, which are referred to as A DA G RAD (theory) and A DA G RAD (heuristic).
For each baseline algorithms of S GD , S HB , S NAG , we also implement two versions - a theory version with iteratively decreasing size η 0 / √ t suggested by previous theories and a heuristic approach with ﬁxed frequency decay scheme used in practice, using (theory) and (heuristic) to indicate them.
The ﬁxed frequency decay scheme used in the heuristic variants is the same as that in (Ren et al., 2018), i.e., the step size parameter is decreased by 10 at 40k, 60k iterations.
We also compare stagwise A DA G RAD with AMSGrad (Reddi et al., 2018)
- a corrected version of Adam.
Parameters.
The stagewise step size η s = η 0 / √ s is used in stagwise A DA G RAD , the number of iterations T s in stagwise A DA G RAD is set according to Theorem 3 with some simpliﬁcations for dealing with unknown ˆ G , in particular we set T s to the smallest value larger than
T 0 (cid:113) s max i (cid:107) g s 1: T s ,i (cid:107) (cid:80) i (cid:107) g s 1: T s ,i (cid:107) .
For stagewise S GD , S HB , S NAG , the stagewise step size and iteration number is set to η s = η 0 /s and T s = T 0 s , respectively.
The involved parameters in the compared algorithms are tuned for the best performance, including the initial step size parameter ηof all algorithms, the value of T 0 and γ for our stagewise algorithms.
Results.
We consider two settings - with/without an (cid:96) 2 norm regularization on weights.
For comparison, we evaluate the training error and testing error of obtained solutions in the process of training.
For our stagewise algorithms, the evaluation is done based on the current averaged solution, and for other baselines the evaluation is done based on the current solution.
Due to the limit of space, we only show the results of testing error on CIFAR-10 and CIFAR-100 for the setting without regularization in Figure 1.
All results are included in the Appendix.
From all results, we have several observations.
(i)
The proposed stagewise algorithms perform much better in terms of testing error than the existing theoretical versions reported in the literature (marked with theory in the legend).
This indicates the proposed stagewise step size scheme is better than iteratively decreasing step size scheme.
(ii)
The proposed stagewise algorithms achieve similar and sometimes even better testing error than the heuristic approaches with a ﬁxed frequency decay scheme used in practice.
However, the heuristic approaches usually give smaller training error.
This seems to indicate that the proposed algorithms are less vulnerable to the overﬁtting.
In another word, the proposed algorithms have smaller generalization error, i.e., the difference between the testing error and the training error.
C ONCLUSION & F UTURE W ORK
In this paper, we have proposed a universal stagewise learning framework for solving stochastic nonconvex optimization problems, which employs well-known tricks in practice that have not been well analyzed theoretically.
We provided theoretical convergence results for the proposed algorithms for non-smooth non-convex optimization problems.
We also established an adaptive convergence of a stochastic algorithm using data adaptive coordinate-wise step size of A DA G RAD , and exhibited its faster convergence than non-adaptive stepsize when the growth of cumulative stochastic gradients is slow similar to that in the convex case.
For future work, one may consider developing more variants of the proposed meta algorithm, e.g., stagewise AMSGrad, stagewise RMSProp, etc.
We will also consider the empirical studies on the large-scale ImageNet data set.
A M ORE E XPERIMENTAL R ESULTS
In this section, we present more experimental results.
Comparison of training and testing error in the two settings (w/o regularization) on the two data sets are plotted in Figure 2, 3, 4, 5.
We also report the ﬁnal testing error (after running 80 k iterations) of different algorithms in the two settings on the two datasets in Table 1.
For parameter tuning, the initial step sizes of all algorithms are tuned in { 0 .
The value of γ of stagewise algorithms is tuned in { 1 , 10 , 100 , 500 , 1000 , 1500 , 2000 } .
The initial value T 0 for stagewise S GD , S HB , S NAG is tuned in { 10 , 100 , 1 k , 5 k , 6 k , 7 k , 10 k } , and that for stagewise A DA G RAD is tuned in { 1 , 10 , 15 , 20 , 25 } .
Note that the training error plotted in the Figure is evaluated on a randomly sampled subset of data around with size about 1 / 5 of the full size.
B P ROOF OF T HEOREM
Proof.
Below, we use E s to denote expectation over randomness in the s -th stage given all history before s -th stage.
Deﬁne
[Equation]
Then ∇ φ γ ( x s − 1 )
= γ − 1 ( x s − 1 − z s ) .
By applying the convergence bound of SA to f s ( x ) , we have
E
[Equation]
It then follows that
[Equation]
On the other hand, we have that
[Equation]
[Table]
[Figure] [Figure]
where the inequality follows from the Young’s inequality with 0 < α s < 1 .
Combining the above inequality with (9) we have that
E
[Equation]
≤
[Equation]
≤
[Equation]
(cid:21) + E s (cid:20)
α − 1 s − γµ (
[Equation]
[Figure] [Figure]
[Figure]
where the second inequality uses the strong convexity of f s ( x ) , whose strong convexity parameter is γ − 1 − µ .
Next
[Figure]
we bound f s ( x s − 1 ) − f s ( z s ) given that x
s − 1 is ﬁxed.
According to the deﬁnition of f s ( · ) , we have
[Equation]
Taking expectation over randomness in the s -th stage on both sides
[Equation]
we have
[Equation]
Thus,
[Equation]
+ ε 3 ( η s , T s , Θ) . Assuming that ε 2 ( η s , T s , Θ) ≤ 1 / 2 , we have
[Equation]
Plugging this upper bound into (10), we have
[Equation]
η
[Equation]
, T
[Equation] Θ [Equation] By setting α s = 1 / 2 , γ = 1 / (2 µ ) and assuming ε 1 ( η s , T s , Θ) ≤ 1 / (48 γ ) , we have
[Equation]
Deﬁne w s =
s α .
Multiplying both sides by w s , we have that
[Equation]
By summing over s = 1 , . . .
, S + 1 , we have
[Equation]
1 (cid:88)
[Equation]
w
[Equation] η [Equation] , Θ
[Equation]
(cid:21)
Taking the expectation w.r.t.
τ ∈ { 0 , . .
, S } , we have that
[Equation]
For the ﬁrst term on the R.H.S, we have that
S +
[Equation]
s =
[Equation]
−
[Equation]
) −
[Equation]
x
[Equation]
S +1 (cid:88) s
= ( w s − w s − 1 ) φ ( x s − 1 ) =
w 0 φ ( x 0 ) − w S
+1 φ ( x S +1 )
+
S +1 (cid:88) s =
[Equation]
w s − w s − 1 ) φ ( x s − 1 ) =
S +1 (cid:88) s
= ( w s − w s − 1 )( φ ( x s − 1 ) − φ ( x S +1 ))
≤ ∆ φ
S +1 (cid:88) s
= ( w s
[Equation]
Then,
[Equation]
The standard calculus tells that
[Equation]
Combining these facts and the assumption ε 3 ( η s , T s , Θ) ≤ c/s [Equation] we have that
[Equation]
In order to have E[ (cid:107)∇ φ γ ( x τ ) (cid:107) 2 ] ≤ (cid:15) 2 , we can set S
= O (1 /(cid:15) 2 ) .
The total number of iterations is
[Equation]
=
[Equation]
C P ROOF OF T HEOREM
Proof.
The proof is almost a duplicate to that of Theorem 1.
Deﬁne w s =
s α .
We apply Lemmato each call of S GD in stagewise S GD ,
[Equation]
where ˆ G 2 is the upper bound of E[ (cid:107) g ( x ; ξ ) + γ − 1 ( x − x s − 1 ) (cid:107) 2 ] , which exists and can be set to 2 G [-
-]2 + 2 γ − 2 D 2 due to the Assumption 1-(ii) and the bounded assumption of the domain.
Then following the same analysis as that in the proof of Theorem
[Equation]
we have
[Equation]
Combining the above inequalities
[Equation]
we have that
[Equation]
Multiplying both sides by w s , we have that
[Equation]
By setting α s = 1 / 2 and γ = 1 / (2 µ ) [Equation] T s η s ≥ 12 γ , we have
[Equation]
By summing over s = 1 , . . .
, S + 1 , we have
[Equation]
Taking the expectation w.r.t.
τ ∈ { 0 , . .
, S } , we have that
[Equation]
By similar analysis, we have that
[Equation]
In order to have E[ (cid:107)∇ φ γ ( x τ ) (cid:107) 2 ] ≤ (cid:15) 2 , we can set S
= O (1 /(cid:15) 2 ) .
The total number of iterations is
S (cid:88) s =
[Equation]
D P ROOF OF T HEOREM
We need the following lemma for the convergence bound of stochastic momentum methods for a strongly convex problem, whose proof is postponed to Section F.
Lemma 3.
For Algorithm 4, assume f ( x ) =
φ ( x ) + 12 γ (cid:107) x − x 0 (cid:107) 2 is a λ -strongly convex function, g t =
g
( x t ; ξ )
+ 1 γ ( x t − x 0 ) where g ( x ; ξ ) ∈
∂ F φ ( x t ) such that E[ (cid:107) g
( x ; ξ ) (cid:107) 2 ] ≤ G 2 , and η ≤ (1 − β ) γ 2 λ/ (8 ρβ + 4) , then we have that
[Equation]
where (cid:98) x T = (cid:80) Tt =
0 x t / (1 + T ) and x ∗ ∈ arg min x ∈ R d f ( x ) .
Remark: It is notable that in the above result, we do not use the bounded domain assumption since we consider Ω
= R d for the uniﬁed momentum methods in this subsection.
The key to get rid of bounded domain assumption is by exploring the strong convexity of f ( x )
=
φ ( x ) + 12 γ (cid:107) x − x 0 (cid:107) 2 .
Proof. of Theorem
According to the deﬁnition of z s in (8) and Lemma 3, we have that
[Equation]
≤ φ ( x s − 1 ) + E s , where the last inequality uses the value of η s = (1 − β ) γ/ (96 s ( ρβ +1)) ≤ (1 − β ) γ/ (96( ρβ +1)) , which also satisﬁes the condition in Lemma 3 by noting that λ = γ − 1 − µ = 1 / (2 γ ) .
Similar to the proof of Theorem 1, we have
[Equation]
[Equation]
Plugging the expression of E s and rearranging above inequality, we have that
[Equation]
The deﬁnition of f s gives that
[Equation]
On the other hand, the µ -weakly convexity of φ gives that
[Equation]
where g ( x s − 1 ) ∈
∂
F φ ( x s − 1 ) .
Combing these two inequalities we have that
[Equation]
where the second inequality follows from Jensen’s inequality for (cid:107) · (cid:107) and Young’s inequality.
Combining above inequalities and multiplying both side by w s , we have that
[Equation]
By setting α s = 1 /
[Equation]
η s ( T s + 1) ≥ 24(1 − β ) γ
[Equation]
we have that
[Equation]
Summing over s = 1 , . .
, S + 1 and rearranging, we have
S +1 (cid:88)
[Equation]
Following similar analysis as in the proof of Theorem
[Equation]
we can ﬁnish the proof.
E P ROOF OF T HEOREM
Proof.
Applying Lemma 2 with T s ≥ M s max { ˆ G +max i
[Equation]
g s 1: T s ,i (cid:107)} M s > 0 , and the fact that φ ( x s − 1 ) ≥ φ ( z s ) + 12 γ (cid:107) x s − 1 − z s (cid:107) 2 in s th stage, we have that
[Equation]
According to (14), we have that
[Equation]
Rearranging above inequality then multiplying both side by w s , we have that
[Equation]
By using M s η s ≥ 24 γc and summing over s = [Equation]
, S + 1 , we have that
[Equation]
1 (cid:88)
[Equation]
By the deﬁnition of τ in the theorem, taking expectation of (cid:107)∇ φ γ ( x τ ) (cid:107) 2 w.r.t.
τ ∈ { 0 , . .
. , S } we have that
[Equation]
where I ( α < 1) is 1 if α < 1 and 0 otherwise.
F P ROOF OF L EMMA
Proof.
Following the analysis in Yang et al.
(2016), we directly have the following inequality,
E[ (cid:107)
[Equation]
We also note that
[Equation]
where the ﬁrst two inequalities are due to the strong convexity of f ( · ) and the last three inequalities are due to the boundness assumption.
Thus
[Equation]
k + p k − x (cid:107) 2 ] − 2 η 1 − β E
[( f ( x k ) − f ( x ))] −
[Equation]
[Equation] β [Equation] [( f ( x k
[Equation]
f (
[Equation]
k
[Equation]
+
[Equation]
By summarizing the above inequality over k = 0 , . . .
, T , we have
[Equation]
When η ≤ (1 − β ) γ [Equation] λ/ (8 ρβ + 4)
[Equation] we have
[Equation]
G P ROOF OF L EMMA
The proof is almost a duplicate of the proof of Proposition 1 in Chen et al.
(2018b).
For completeness, we present a proof here.
Proof.
Let ψ 0 ( x ) =
and (cid:107) x (cid:107) H = √ x
(cid:62) H x .
First, we can see that ψ t +1 ( x ) ≥ ψ t ( x ) for any t ≥ 0 .
Deﬁne ζ t = (cid:80) tτ
=1 g t and ∆ τ = ( ∂F ( x t ) − g t ) (cid:62) ( x t − x ) .
Let ψ ∗ t be deﬁned by
[Equation]
Taking the summation of objective gap in all iterations, we have
[Equation]
∆
[Equation]
Note that ψ
[Equation]
where the last inequality uses the fact that ψ t ( x ) is 1-strongly convex w.r.t (cid:107) · (cid:107) ψ t = (cid:107) · (cid:107) H t and consequentially ψ ∗ t ( x ) is η -smooth wr.t.
(cid:107) · (cid:107) ψ ∗ t = (cid:107) · (cid:107) H − 1 t .
Thus, we have
T (cid:88)
[Equation]
By repeating this process, we have
[Equation]
Then
[Equation]
Following the analysis in Duchi et al.
(2011), we have
[Equation]
=
[Equation]
Thus
T (cid:88) t =
[Equation]
≤ G + max i [Equation] g [Equation]
i = [Equation] (cid:107)
[Equation]
t = ∆ t
Now by the value of T ≥ M max { ( G +max
i
(cid:107) g 1: T,i ) (cid:107) 2 c , c (cid:80) di
=1 (cid:107) g 1: T,i (cid:107)} , we have ( G + max
i (cid:107) g 1: T,i (cid:107) 2 )
[Equation]
Mc Dividing by T on both sides and setting x = x ∗ , following the inequality (3) and the convexity of f ( x ) we have
[Equation]
Let {F t } be the ﬁltration associated with Algorithm 1 in the paper.
Noticing that T is a random variable with respect to {F t } , we cannot get rid of the last term directly.
Deﬁne the Sequence { X t } t ∈ N + as
[Equation]
where E[ g i ] ∈ ∂f ( x i ) .
Since E [ g t +1 − E[ g t +1 ]] = 0 and x t +1 = arg
min x ∈ Ω η x (cid:62) (cid:16) 1 t (cid:80) tτ
=1 g τ (cid:17) + 1 t ψ t ( x ) , which is measurable with respect to g 1 , . . . , g t and x 1 , . . .
, x t , it is easy to see { ∆ t } t ∈ N is a martingale difference sequence with respect to {F t } , e.g. E[∆ t |F t − 1 ] = 0 .
On the other hand, since (cid:107) g t (cid:107) 2 is upper bounded by G , following the statement of T in the theorem, T ≤ N =
M 2 max { ( G +1) 2, d 2 G 2 } < ∞ always holds.
Then following Lemma 1 in (Chen et al., 2018b) we have that E[ X T ] = 0 .
Now taking the expectation we have that
[Equation]
Then we ﬁnish the proof.
H P ROOF OF I NEQUALITY (3)
Let us recall the deﬁnition of (cid:98)
x
= prox γf
( x ) .
[Equation]
First, note that when f ( z ) is µ -weakly convex and γ < µ − 1 , the above problem is strongly convex and (cid:98) x
= prox γf
( x ) is well-deﬁned and unique.
By the optimality condition of (cid:98) x , it is clear that f ( (cid:98) x ) + 12 γ (cid:107) (cid:98) x − x (cid:107) 2 ≤ f ( x ) , which proves the ﬁrst inequality f
( (cid:98) x ) ≤ f ( x ) .
Note that when x = x ∗ ∈ arg min z f ( z ) , we can prove that (cid:98) x = x = x ∗ .
This is because that by deﬁnition of (cid:98) x , it is unique and satisﬁes
0 ∈ ∂f ( (cid:98) x )+ 1 γ ( (cid:98) x − x ) .
When x = x ∗ , we have 0 ∈ ∂f ( x ) and x ∗ clearly satisifes the ﬁrst-order condition
0 ∈ ∂f ( x ∗ ) + 1 γ ( x ∗ − x ) .
By the uniqueness of (cid:98) x , it follows that (cid:98) x
= x ∗ .
The ﬁrst-order condition 0 ∈ ∂f ( (cid:98) x ) + 1 γ ( (cid:98) x − x ) also gives that dist (0 , ∂f ( (cid:98) x )) ≤ 1 γ (cid:107) (cid:98) x − x (cid:107) .
Also, we have ∇ f γ ( x ) = 1 γ ( (cid:98) x − x ) (Rockafellar, 1970), which implies the second and the third inequality in (3).