W HY GRADIENT CLIPPING ACCELERATES TRAINING : A THEORETICAL JUSTIFICATION FOR ADAPTIVITY
A BSTRACT
We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks.
The key ingredient is a new smoothness condition derived from practical neural network training examples.
We observe that gradient smoothness, a concept central to the analysis of ﬁrst-order optimization algorithms that is often assumed to be a constant, demonstrates signiﬁcant variability along the training trajectory of deep neural networks.
Further, this smoothness positively correlates with the gradient norm, and contrary to standard assumptions in the literature, it can grow with the norm of the gradient.
These empirical observations limit the applicability of existing theoretical analyses of algorithms that rely on a ﬁxed bound on smoothness.
These observations motivate us to introduce a novel relaxation of gradient smoothness that is weaker than the commonly used
Lipschitz smoothness assumption.
Under the new condition, we prove that two popular methods, namely, gradient clipping and normalized gradient , converge arbitrarily faster than gradient descent with ﬁxed stepsize.
We further explain why such adaptively scaled gradient methods can accelerate empirical convergence and verify our results empirically in popular neural network training settings.
I NTRODUCTION
We study optimization algorithms for neural network training and aim to resolve the mystery of why adaptive methods converge fast.
Speciﬁcally, we study gradient-based methods for minimizing a differentiable nonconvex function f : R d → R , where f ( x ) can potentially be stochastic, i e., f ( x ) =
E ξ [ F ( x, ξ )] .
Such choices of f cover a wide range of problems in machine learning and their study motivates a vast body of current optimization literature.
A widely used (and canonical) approach for minimizing f is the (stochastic) gradient descent (GD) algorithm.
Despite its simple form, GD often achieves superior empirical (Wilson et al., 2017) performances and theoretical (Carmon et al., 2017) guarantees.
However, in many tasks such as reinforcement learning and natural language processing (NLP), adaptive gradient methods (e.g.,
Adagrad (Duchi et al., 2011), ADAM (Kingma and Ba, 2014), and RMSProp (Tieleman and Hinton, 2012)) outperform SGD.
Despite their superior empirical performance, our understanding of the fast convergence of adaptive methods is limited.
Previous analysis has shown that adaptive methods are more robust to variation in hyper-parameters (Ward et al., 2018) and adapt to sparse gradients (Duchi et al., 2011) (a more detailed literature review is in Appendix A).
However, in practice, the gradient updates are dense, and even after extensively tuning the SGD hyperparameters, it still converges much slower than adaptive methods in NLP tasks.
We analyze the convergence of clipped gradient descent and provide an explanation for its fast convergence.
Even though gradient clipping is a standard practice in tasks such as language models (e.g.
Merity et al., 2018; Gehring et al., 2017; Peters et al., 2018), it lacks a ﬁrm theoretical grounding.
Goodfellow et al.
; Pascanu et al.
(2013; 2012) discuss the gradient explosion problem in recurrent models and consider clipping as an intuitive work around.
We formalize this intuition and prove that clipped GD can converge arbitrarily faster than ﬁxed-step gradient descent .
This result is shown to hold under a novel smoothness condition that is strictly weaker than the standard Lipschitzgradient assumption pervasive in the literature.
Hence our analysis captures many functions that are not globally Lipschitz smooth.
Importantly, the proposed smoothness condition is derived on the basis of extensive NLP training experiments, which are precisely the same type of experiments for which adaptive gradient methods empirically perform superior to gradient methods.
By identifying a a new smoothness condition through experiments and then using it to analyze the convergence of adaptively-scaled methods, we reduce the following gap between theory and practice.
On one hand, powerful techniques such as Nesterov’s momentum and variance reduction theoretically accelerate convex and nonconvex optimization.
But, at least for now, they seem to have limited applicability in deep learning (Defazio and Bottou, 2018).
On the other hand, some widely used techniques (e.g., heavy-ball momentum, adaptivity) lack theoretical acceleration guarantees.
We suspect that a major reason here is the misalignment of the theoretical assumptions with practice.
Our work demonstrates that the concept of acceleration critically relies on the problem assumptions and that the standard global Lipschitz-gradient condition may not hold in the case of some applications and thus must be relaxed to admit a wider class of objective functions.
1. C ONTRIBUTIONS
In light of the above background, the main contributions of this paper are the following:
(cid:4) Inspired and supported by neural network training experiments, we introduce a new smoothness condition that allows the local smoothness constant
to
increase
with
the gradient norm
This condition is strictly weaker than the pervasive Lipschitz-
gradient assumption (cid:4) We provide a convergence rate for clipped GD under our smoothness assumption (Theorem 3).
(cid:4) We prove an upper-bound (Theorem 6) and a lower-bound (Theorem 4) on the convergence rate of GD under our relaxed smoothness assumption.
The lower-bound demonstrates that GD with ﬁxed step size can be arbitrarily slower than clipped GD.
(cid:4) We provide upper bounds for stochastic clipped GD (Theorem 7) and SGD (Theorem 8).
Again, stochastic clipped GD can be arbitrarily faster than SGD with a ﬁxed step size.
We support our proposed theory with realistic neural network experiments.
First, in the state of art
LSTM language modeling (LM) setting, we observe the function smoothness has a strong correlation with gradient norm (see Figure 2).
This aligns with the known fact that gradient clipping accelerates
LM more effectively compared to computer vision (CV) tasks.
Second, our experiments in CV and LM demonstrate that clipping accelerates training error convergence and allows the training trajectory to cross non-smooth regions of the loss landscape.
Furthermore, gradient clipping can also achieve good generalization performance even in image classiﬁcation (e.g., 95 . 2% test accuracy in 200 epochs for ResNet20 on Cifar10).
Please see Section 5 for more details.
A N EW R ELAXED S MOOTHNESS C ONDITION
In this section, we motivate and develop a relaxed smoothness condition that is weaker (and thus, more general) than the usual global Lipschitz smoothness assumption.
We start with the traditional deﬁnition of smoothness.
2. F UNCTION SMOOTHNESS (L IPSCHITZ GRADIENTS )
Recall that f denotes the objective function that we want to minimize.
We say that f is L -smooth if
[Equation]
For twice differentiable functions, condition (1) is equivalent to (cid:107)∇ 2 f ( x ) (cid:107) ≤ L, ∀ x ∈ R d .
This smoothness condition enables many important theoretical results.
For example, Carmon et al.
(2017) show that GD with h = 1 /L is up to a constant optimal for optimizing smooth nonconvex functions.
But the usual L -smoothness assumption (1) also has its limitations.
Assuming existence of a global constant L that upper bounds the variation of the gradient is very restrictive.
For example, simple polynomials such as f ( x ) =
x 3 break the assumption.
One workaround is to assume that L exists in a compact region, and either prove that the iterates do not escape the region or run projectionbased algorithms.
However, such assumptions can make L very large and slow down the theoretical convergence rate.
In Section 4, we will show that a slow rate is unavoidable for gradient descent with ﬁxed step size, whereas clipped gradient descent can greatly improve the dependency on L .
The above limitations force ﬁxed-step gradient descent (which is tailored for Lipschitz smooth functions) to converge slowly in many tasks.
In Figure 1, we plot the estimated function smoothness at different iterations during training neural networks.
We ﬁnd that function smoothness varies greatly at different iterations.
From Figure 1, we further ﬁnd that local smoothness positively correlates with the full gradient norm, especially in the language modeling experiment.
A natural question is:
Can we ﬁnd a ﬁne-grained smoothness condition under which we can design theoretically and empirically fast algorithms at the same time?
To answer this question, we introduce the relaxed smoothness condition in the next section, which is developed on the basis of extensive experiments— Figure 1 provides an illustrative example.
2. A NEW RELAXED SMOOTHNESS CONDITION
We observe strong positive correlation between function smoothness and gradient norm in language modeling experiments (Figure 1(a)).
This observation leads us to propose the following smoothness condition that allows local smoothness to grow with function gradients.
Deﬁnition 1.
A second order differentiable function f is ( L 0 , L 1 ) -smooth
if
[Equation]
Deﬁnition 1 strictly relaxes the usual (and widely used)
L -smoothness.
There are two ways to interpret the relaxation: First, when we focus on a compact region, we can balance the constants Land L 1 such that L 0 (cid:28) L while L 1 (cid:28) L .
Second, there exist functions that are ( L 0 , L 1 ) -smooth globally, but not L -smooth.
Hence the constant L for L -smoothness gets larger as the compact set increases but L 0 and L 1 stay ﬁxed.
An example is given in Lemma 2.
Remark 1.
It is worth noting that we do not need the Hessian operator norm and gradient norm to necessarily satisfy the linear relation (2).
As long as these norms are positively correlated, gradient clipping can be shown to achieve faster rate than ﬁxed step size gradient descent.
We use the linear relationship (2) for simplicity of exposition.
Lemma 2.
Let f be the univariate polynomial f ( x ) = (cid:80) di
[-i =
-]=1 a i x i .
When d ≥ 3 , then f is ( L 0 , L 1 ) - smooth for some L 0 and L 1 but not L -smooth.
Proof.
The ﬁrst claim follows from lim
[Equation]
= ∞ .
The second claim follows by the unboundedness of f (cid:48)(cid:48)
( x ) .
2. S MOOTHNESS IN NEURAL NETWORKS
We saw that our smoothness condition relaxes the traditional smoothness assumption and is motivated empirically (Figure 1).
Below we develop some intuition for this phenomenon.
We conjecture that the proposed positive correlation results from the common components in expressions of the gradient and the Hessian.
We illustrate the reasoning behind this conjecture by considering an (cid:96) - layer linear network with quadratic loss—a similar computation also holds for nonlinear networks.
The L 2 regression loss of a deep linear network is L ( Y, f ( X )) :
= (cid:107) Y − W (cid:96) · · · W 1 X (cid:107) 2 , where
Y denotes labels, X denotes the input data matrix, and W i denotes the weights in the i th layer.
By (Lemma 4.3 Kawaguchi, 2016), we know that
[Equation]
where vec( · ) ﬂattens a matrix in R m × n into a vector in R mn ; ⊗ denotes the Kronecker product.
For constants i, j such that (cid:96) ≥ j > i > 0 , the second order derivative
[Equation]
When j = i , the second term equals 0 .
Based on the above expressions, we notice that the gradient norm and Hessian norm may be positively correlated due to the following two observations.
First, the gradient and the Hessian share many components such as the matrix product of weights across layers.
Second, if one naively upper bounds the norm using Cauchy-Schwarz, then both upperbounds would be monotonically increasing with respect to (cid:107) W
i (cid:107) and (cid:107) f
( X ) − Y (cid:107) .
P ROBLEMS SETUP AND ALGORITHMS
In this section, we state the optimization problems and introduce gradient based algorithms for them that work under the new smoothness condition (2).
Convergence analysis follows in Section 4.
Recall that we wish to solve the nonconvex optimization problem min x ∈ R d f ( x ) .
Since in general this problem is intractable, following common practice we also seek an (cid:15) -stationary point, i.e., a point x such that (cid:107)∇ f ( x ) (cid:107) ≤ (cid:15) .
Furthermore, we make the following assumptions to regularize the function class studied and subsequently provide nonasymptotic convergence rate analysis.
Assumption 1.
The function f is lower bounded by f ∗ > −∞ .
Assumption 2.
The function f is twice differentiable.
Assumption 3 ( ( L 0 , L 1 ) -smoothness).
The function f is ( L 0 , L 1 ) -smooth, i.e., there exist positive constants L 0 and L 1 such that (cid:107)∇ [Equation] ≤ [Equation]
+
[Equation] 1 (cid:107)∇ f [Equation] x )
[Equation]
—see condition (2).
The ﬁrst assumption is standard.
Twice differentiability in Assumption 2 can relaxed to ﬁrst-order differentiability by modifying the deﬁnition of ( L 0 , L 1 ) -smoothness as
[Equation]
The above inequality implies ∇ f ( x ) is locally Lipschitz, and hence almost everywhere differentiable.
Therefore, all our results can go through by handling the integrations more carefully.
But to avoid complications and simplify exposition, we assume that the function is twice differentiable.
To further relax the global assumptions, by showing that GD and clipped GD are monotonically decreasing in function value, we require the above assumptions to hold just in a neighborhood determined by the sublevel set S 1 for a given initialization x 0 , where [Equation] { x | ∃ y such that f ( y ) ≤ f ( x 0 ) , and (cid:107) x − y (cid:107) ≤
[Equation]
3. G RADIENT DESCENT ALGORITHMS
In this section, we review a few well-known variants of gradient based algorithms that we analyze.
We start with the ordinary gradient descent with a ﬁxed step size η ,
[Equation]
This algorithm (pedantically, its stochastic version) is widely used in neural network training.
Many modiﬁcations of it have been proposed to stabilize or accelerate training.
One such technique of particular importance is clipped gradient descent , which performs the following updates:
[Equation]
Another algorithm that is less common in practice but has attracted theoretical interest is normalized gradient descent .
The updates for normalized GD method can be written as
[Equation]
The stochastic version of the above algorithms replace the gradient with a stochastic estimator.
We note that Clipped GD and NGD are almost equivalent.
Indeed, for any given η n and β , if we set γη c = η n and η c = η n /β , then we have
[Equation]
Therefore, clipped GD is equivalent to NGD up to a constant factor in the step size choice.
Consequently, the nonconvex convergence rates in Section 4 and Section 4.2 for clipped GD also apply to
NGD.
We omit repeating the theorem statements and the analysis for conciseness.
T HEORETICAL ANALYSIS
In this section, we analyze the oracle complexities of GD and clipped GD under our relaxed smoothness condition.
All the proofs are in the appendix.
We highlight the key theoretical challenges that needed to overcome in Appendix B (e.g., due to absence of Lipschitz-smoothness, already the ﬁrststep of analysis, the so-called “descent lemma” fails).
Since we are analyzing the global iteration complexity, let us recall the formal deﬁnition being used.
We follow the notation from Carmon et al.
For a deterministic sequence { x k } k ∈ N , deﬁne the complexity of { x k } k ∈ N for a function f as
[Equation]
For a random process { x k } k ∈ N , we deﬁne the complexity of { x k } k ∈ N for function f as
[Equation]
In particular, if the condition is never satisﬁed, then the complexity is ∞ .
Given an algorithm A θ , where θ denotes hyperparameters such as step size and momentum coefﬁcient, we denote A θ
[ f, x 0 ] as the sequence of (potentially stochastic) iterates generated by A when operating on f with initialization x 0 .
Finally, we deﬁne the iteration complexity of an algorithm class parameterized by p hyperparameters,
A = { A θ } θ ∈ R p on a function class F as
[Equation]
The deﬁnition in the stochastic setting simply replaces the expression (7) with the expression (8).
In the rest of the paper, “iteration complexity” refers to the quantity deﬁned above.
4. C ONVERGENCE IN THE DETERMINISTIC SETTING
In this section, we present the convergence rates for GD and clipped GD under deterministic setting.
We start by analyzing the clipped GD algorithm with update deﬁned in equation (5).
Theorem 3.
Let F denote the class of functions that satisfy Assumptions 1, 2, and 3 in set S deﬁned in (3) .
Recall f ∗ is a global lower bound for function value.
With η c =
[Equation]
[Equation]
, γ = min { 1 η c , 1 10 L 1 η c } , we can prove that the iteration complexity of clipped GD (Algorithm 5) is upper bounded by
[Equation]
The proof of Theorem 3 is included in Appendix C.
Now, we discuss the convergence of vanilla GD.
The standard GD is known to converge to ﬁrst order (cid:15) -stationary points in O (( L ( f ( x 0 ) − f ∗ ))
(cid:15) − 2 ) iterations for ( L, 0) − smooth nonconvex functions.
By Theorem 1 of Carmon et al.
(2017), this rate is up to a constant optimal.
However, we will show below that gradient descent is suboptimal under our relaxed ( L 0 , L 1 ) - smoothness condition.
In particular, to prove the convergence rate for gradient descent with ﬁxed step size, we need to permit it beneﬁt from an additional assumption on gradient norms.
Assumption 4.
Given an initialization x 0 , we assume that
M
[Equation]
= sup { [Equation] f ( x )
[Equation]
| x such that f ( x ) ≤ f ( x 0 ) } < ∞ .
This assumption is in fact necessary , as our next theorem reveals.
Theorem 4.
Let F be the class of objectives satisfying Assumptions 1, 2, 3, and 4 with ﬁxed constants L 0 ≥ 1 , L 1 ≥ 1 , M > 1 .
The iteration complexity for the ﬁxed-step gradient descent algorithms parameterized by step size h is at least
[Equation]
The proof can be found in Appendix D.
Remark 5.
Theorem 1 of Carmon et al.
(2017) and
Theorem 4 together show that gradient descent with a ﬁxed step size cannot converge to an (cid:15) -stationary point faster than Ω (cid:0) ( L 1 M/ log( M ) + L 0 )
( f ( x 0 ) − f ∗ ) (cid:15) − 2 (cid:1) .
Recall that clipped GD algorithm converges as
O (cid:0) L 0 ( f ( x 0 ) − f ∗ ) (cid:15) − 2 + L 21 ( f ( x 0 ) − f ∗ )
L − 1 0 (cid:1) .
Therefore, clipped GD can be arbitrarily faster than GD when L 1 M is large, or in other words, when the problem has a poor initialization .
Below, we provide an iteration upper bound for the ﬁxed-step gradient descent update (4).
Theorem 6.
Suppose assumptions 1, 2, 3 and 4 hold in set S deﬁned in (3) .
If we pick parameters such that h = 1 (2( ML 1 + L 0 )) , then we can prove that the iteration complexity of GD with a ﬁxed step size deﬁned in Algorithm 4 is upper bounded by
[Equation]
Please refer to Appendix E for the proof.
Theorem 6 shows that gradient descent with a ﬁxed step size converges in O (( ML 1 + L 0 )
( f ( x 0 ) − f ∗ ) /(cid:15) 2 ) iterations.
This suggests that the lower bound in Remark 5 is tight up to a log factor in M .
4. C ONVERGENCE IN THE STOCHASTIC SETTING
In the stochastic setting, we assume GD and clipped GD have access to an unbiased stochastic gradient ∇ ˆ f ( x ) instead of the exact gradient ∇ f ( x ) .
For simplicity, we denote g k = ∇
ˆ
f
( x k ) below.
To prove convergence, we need the following assumption.
Assumption 5.
There exists τ > 0 , such that (cid:107)∇ ˆ f ( x ) − ∇ f ( x ) (cid:107) ≤ τ almost surely.
Bounded noise can be relaxed to sub-gaussian noise if the noise is symmetric.
Furthermore, up to our knowledge, this is the ﬁrst stochastic nonconvex analysis of adaptive methods that does not require the gradient norm (cid:107)∇ f ( x ) (cid:107) to be bounded globally.
The main result of this section is the following convergence guarantee for stochastic clipped GD (based on the stochastic version of the update (5)).
Theorem 7.
Let Assumptions 1–3 and 5 hold globally with L> 0 .
Let h = min (cid:8)
[Equation]
, η (cid:9) where η = min (cid:8) 1 {+
+} L 0 ,
[Equation]
[Equation]
T (cid:9) .
Then we can show that iteration complexity for stochastic clipped GD after of update (5) is upper bounded by
[Equation]
where ∆ =
( f ( x 0 ) − f ∗ + (5 L 0 + 2 L 1 τ ) τ 2 + 9 τL 20 /L 1 ) .
In comparison, we have the following upper bound for ordinary SGD.
Theorem 8.
Let Assumptions 1–3, and 5 hold globally with L 1 > 0 .
Let h = min (cid:8) 1 √ T ,
[Equation]
Then the iteration complexity for the stochastic version of GD (4) is upper bounded by
[Equation]
We cannot provide a lower bound for this algorithm.
In fact, lower bound is not known for SGD even in the global smoothness setting.
However, the deterministic lower bound in Theorem 4 is still valid, though probably loose.
Therefore, the convergence of SGD still requires additional assumption and can again be arbitrarily slower compared to clipped SGD when M is large.
[Figure] [Figure]
E XPERIMENTS
In this section
[Figure]
we summarize our empirical ﬁndings on the positive correlation between gradient norm and local smoothness.
We then show that clipping accelerates convergence during neural network training.
Our experiments are based on two tasks: language modeling and image classiﬁcation.
We run language modeling on the Penn Treebank (PTB) (Mikolov et al., 2010) dataset with AWD-LSTM models (Merity et al., 2018) 2 .
We train ResNet20 (He et al., 2016) on the Cifar dataset (Krizhevsky and Hinton, 2009).
Details about the smoothness estimation and experimental setups are in Appendix H.
An additional synthetic experiment is discussed in Appendix I.
First, our experiments test whether the local smoothness constant increases with the gradient norm, as suggested by the relaxed smoothness conditions deﬁned in (2) (Section 2).
To do so, we evaluate both quantities at points generated by the optimization procedure.
We then scatter the local smoothness constants against the gradient norms in Figure 2 and Figure 3.
Note that the plots are on a log-scale.
A linear scale plot is shown in Appendix Figure 5.
We notice that the correlation exists in the default training procedure for language modeling (see
Figure 2a) but not in the default training for image classiﬁcation (see Figure 3a).
This difference aligns with the fact that gradient clipping is widely used in language modeling but is less popular in
ResNet training, offering empirical support to our theoretical ﬁndings .
We further investigate the cause of correlation.
The plots in Figures 2 and 3 show that correlation appears when the models are trained with clipped GD and large learning rates.
We propose the following explanation.
Clipping enables the training trajectory to stably traverse non-smooth regions.
Hence, we can observe that gradient norms and smoothness are positively correlated in Figures 2a and 3c.
Without clipping, the optimizer has to adopt a small learning rate and stays in a region where local smoothness does not vary much, otherwise the sequence diverges, and a different learning rate is used.
Therefore, in other plots of Figures 2 and 3, the correlation is much weaker.
As positive correlations are present in both language modeling and image classiﬁcation experiments with large step sizes, our next set of experiments checks whether clipping helps accelerate convergence as predicted by our theory.
From Figure 4, we ﬁnd that clipping indeed accelerates convergence.
Because gradient clipping is a standard practice in language modeling, the LSTM models trained with clipping achieve the best validation performance and the fastest training loss conver-
/ why-clipping-accelerates [Figure]
gence as expected. For image classiﬁcation, surprisingly, clipped GD also achieves the fastest convergence and matches the test performance of SGD+momentum.
These plots show that clipping can accelerate convergence and achieve good test performance at the same time.
D ISCUSSION
Much progress has been made to close the gap between upper and lower oracle complexities for ﬁrst order smooth optimization.
The works dedicated to this goal provide important insights and tools for us to understand the optimization procedures.
However, there is another gap that separates theoretically accelerated algorithms from empirically fast algorithms.
Our work aims to close this gap.
Speciﬁcally, we propose a relaxed smoothness assumption that is supported by empirical evidence.
We analyze a simple but widely used optimization technique known as gradient clipping and provide theoretical guarantees that clipping can accelerate gradient descent.
This phenomenon aligns remarkably well with empirical observations.
There is still much to be explored in this direction.
First, though our smoothness condition relaxes the usual Lipschitz assumption, it is unclear if there is an even better condition that also matches the experimental observations while also enabling a clean theoretical analysis.
Second, we only study convergence of clipped gradient descent.
Studying the convergence properties of other techniques such as momentum, coordinate-wise learning rates (more generally, preconditioning), and variance reduction is also interesting.
Finally, the most important question is: “can we design fast algorithms based on relaxed conditions that achieve faster convergence in neural network training?”
Our experiments also have noteworthy implications.
First, though advocating clipped gradient descent in ResNet training is not a main point of this work, it is interesting to note that gradient descent and clipped gradient descent with large step sizes can achieve a similar test performance as momentum-SGD.
Second, we learned that the performance of the baseline algorithm can actually beat some recently proposed algorithms.
Therefore, when we design or learn about new algorithms, we need to pay extra attention to check whether the baseline algorithms are properly tuned
A CKNOWLEDGEMENT
SS acknowledges support from an NSF-CAREER Award (Number 1846088) and an Amazon Research Award.
AJ acknowledges support from an MIT-IBM-Exploratory project on adaptive, robust, and collaborative optimization.
A M ORE RELATED WORK ON ACCELERATING GRADIENT METHODS
Variance reduction.
Many efforts have been made to accelerate gradient-based methods.
One elegant approach is variance reduction (e.g. Schmidt et al., 2017; Johnson and Zhang, 2013; Defazio et al., 2014; Bach and Moulines, 2013; Koneˇcn`y and Richt´arik, 2013; Xiao and Zhang, 2014;
Gong and Ye, 2014; Fang et al., 2018;
Zhou et al., 2018b).
This technique aims to solve stochastic and ﬁnite sum problems by averaging the noise in the stochastic oracle via utilizing the smoothness of the objectives.
Momentum methods.
Another line of work focuses on achieving acceleration with momentum.
Polyak (1964) showed that momentum can accelerate optimization for quadratic problems; later,
Nesterov (1983) designed a variation that provably accelerate any smooth convex problems.
Based on Nesterov’s work, much theoretical progress was made to accelerate different variations of the original smooth convex problems (e.g. Ghadimi and Lan, 2016; 2012; Beck and Teboulle, 2009;
Shalev-Shwartz and Zhang, 2014; Jin et al., 2018;
Carmon et al., 2018; Allen-Zhu, 2017; Lin et al., 2015; Nesterov, 2012).
Adaptive step sizes.
The idea of varying step size in each iteration has long been studied.
Armijo (1966) proposed the famous backtracking line search algorithm to choose step size dynamically.
Polyak (1987) proposed a strategy to choose step size based on function suboptimality and gradient norm.
More recently, Duchi et al.
(2011) designed the Adagrad algorithm that can utilize the sparsity in stochastic gradients.
Since 2018, there has been a surge in studying the theoretical properties of adaptive gradient methods.
One starting point is (Reddi et al., 2019), which pointed out that ADAM is not convergent and proposed the AMSGrad algorithm to ﬁx the problem.
Ward et al.
; Li and Orabona (2018) prove that Adagrad converges to stationary point for nonconvex stochastic problems.
Zhou et al.
(2018a) generalized the result to a class of algorithms named Padam.
Zou et al.
; Staib et al.
; Chen et al.
; Zhou et al.
(2018c);
Agarwal et al.
; Zhou et al.
(2018b)
; Zou and Shen (2018) also studied different interesting aspects of convergence of adaptive methods.
In addition, Levy (2016) showed that normalized gradient descent may have better convergence rate in presence of injected noise.
However, the rate comparison is under dimension dependent setting.
Hazan et al.
(2015) studied the convergence of normalized gradient descent for quasi-convex functions.
B C HALLENGES IN THE PROOFS
In this section, we highlight a few key challenges in our proofs.
First, the analysis convergence under the relaxed smoothness condition is more difﬁcult than the traditional setup.
In particular, classical analyses based on Lipschitz-smooth gradients frequently exploit the descent condition:
[Equation]
However, under our relaxed smoothness condition, the last term will increase exponentially in (cid:107) y − x (cid:107) 2 .
To solve this challenge, we bound the distance moved by clipping and apply Gr¨onwall’s inequality.
Second, our algorithm speciﬁc lower bound proved in Theorem 4 is novel and tight up to a log factor.
To our knowledge, the worst case examples used have not been studied before.
Last, proving the convergence of adaptive methods in the nonconvex stochastic setting suffers from a fundamental challenge: the stochastic gradient is dependent on the update step size.
This problem is usually circumvented by either assuming gradients have bounded norms or by using a lagging-byone step-size to decouple the correlation.
The situation is even worse under the relaxed smoothness assumption.
In our case, we overcome this challenge by a novel analysis that divides the proof into the large gradient scenario and the small gradient scenario.
C P ROOF OF T HEOREM
We start by proving a lemma that is repeatedly used in later proofs.
The lemma bounds the gradient in a neighborhood of the current point by Gr¨onwall’s inequality (integral form).
Lemma 9.
Given x such that f ( x ) ≤ f ( x 0 ) , for any x + such that (cid:107) x
+ − x (cid:107) ≤ min { 1 /L 1 , 1 } , we have (cid:107)∇ f
[Equation]
≤
[Equation]
L
[Equation]
f ( x
[Equation]
Remark 10.
Note that the constant “1” comes from the deﬁnition of S in (3).
If Assumption 3 holds globally, then we do not need to constrain (cid:107) x
+ − x (cid:107) ≤ 1 .
This version will be used in Theorem 7.
Proof.
Let γ ( t ) be a curve deﬁned below,
[Equation]
Then we have
[Equation]
By Cauchy-Schwarz’s inequality, we get
[Equation]
The second inequality follows by Assumption 3.
Then we can apply the integral form of Gr ¨ onwall’s inequality and get
[Equation]
The Lemma follows by setting t = 1 .
C. P ROOF OF THE THEOREM
We parameterize the path between x k and its updated iterate x k +1 as follows:
[Equation]
Since x k +
=
x
k − h k ∇ f ( x k ) , using Taylor’s theorem, the triangle inequality, and Cauchy-Schwarz, we obtain
[Equation]
Since
[Equation]
we know by Lemma
[Equation]
Then by Assumption 3, we obtain the “descent inequality”:
[Equation]
Therefore, as long as h k ≤ 1 / (5 L 0 + 4 L 1 (cid:107)∇ f ( x k ) (cid:107) ) (which follows by our choice of η, γ ), we can quantify the descent to be
[Equation]
When (cid:107)∇ f ( x k ) (cid:107) ≥ L 0 /L 1 , we have h
[Equation]
max
[Equation]
When (cid:15) ≤ (cid:107)∇ f ( x k ) (cid:107) ≤
L 0 /L 1 , we have
[Equation]
Therefore,
[Equation]
Assume that (cid:15) ≤ (cid:107)∇ f ( x k ) (cid:107) for k ≤ T iterations.
By doing a telescopic sum, we get
T
[Equation] [Equation]
Rearranging we get
[Equation]
D P ROOF OF T HEOREM
We will prove a lower bound for the iteration complexity of GD with ﬁxed step size.
The high level idea is that if GD converges for all functions satisfying the assumptions, then the step size needs to be small.
However, this small step size will lead to very slow convergence for another function.
Recall that the ﬁxed step size GD algorithm is parameterized by the scaler: step size h .
First, we show that when h > 2log( M )+
[Equation]
= ∞
We start with a function that grows exponentially.
Let L 1 > 1 , M > 1 be ﬁxed constants.
Pick the initial point x 0 = (log( M ) + 1) /L 1 .
Let the objective be deﬁned as follows,
[Equation]
, for x < − 1 L 1 ,
L 1 x 2+ 12 L 1 , for x ∈ [ − 1 L 1 , 1 L 1 ] , e L 1 x L 1 e , for x > 1 L 1 .
We notice that the function satisﬁes the assumptions with constants
[Equation]
When h > 2 x 0 /M , we would have | x 1 | > | x 0 | .
By symmetry of the function and the super-linear growth of the gradient norm, we know that the iterates will diverge.
Hence, in order for gradient descent with a ﬁxed step size h to converge, h must be small enough.
Formally,
[Equation]
Second, we show that when h ≤ 2log
[Equation]
Now, let’s look at a different objective that grows slowly.
[Equation]
for x ∈ [
[Equation] [Equation]
[Equation]
[Equation]
(cid:15)
[Equation]
for x > 1 .
This function is also second order differentiable and satisﬁes the assumptions with constants in (11).
If we set x 0 = 1 + ∆ /(cid:15) for some constant ∆ > 0 , we know that f ( x 0 ) − f ∗ = 2∆ + 5 (cid:15)/ 4 .
With the step size choice h ≤ (2 log M + 2) /
( ML 1 ) , we know that in each step, x k +1 ≥ x k − (4 (cid:15) (log M + 1)) /
( L 1 M ) .
Therefore, for k ≤ ∆ L 1 M/ (4 (cid:15) 2 (log M + 1)) , (cid:107)∇ f ( x k ) (cid:107) = 2 (cid:15).
After combining these two points, we proved the theorem by deﬁnition (9).
E P ROOF OF T HEOREM
We start by parametrizing the function value along the update,
[Equation]
Note that with this parametrization, we have γ (0) =
x k , γ (1) =
x k +1 .
Now we would like to argue that if f ( x k ) ≤ f ( x 0 ) , then (cid:107)∇ f ( x ( t )) (cid:107) ≤ M, ∀ t ≤ 1 .
Assume by contradiction that this is not true.
Then there exists (cid:15) > 0 , t ∈ [0 , 1] such that (cid:107)∇ f ( x ( t ))
(cid:107) ≥ M + (cid:15) .
Since (cid:15) can be made arbitrarily small below a threshold, we assume (cid:15) <
M .
Denote
[Equation]
The value t ∗ exists by continuity of (cid:107)∇ f ( x ( t ))
(cid:107) as a function of t .
Then we know by Assumptionthat f
( x ( t ∗ ))
> f ( x k ) .
However, by Taylor expansion, we know that
[Equation]
The last inequality follows by h = 1 / (2( ML 1 + L 0 )) .
Hence we get a contradiction and conclude that for all t ≤ 1 , (cid:107)∇ f ( x ( t ))
(cid:107) ≤ M .
Therefore, following the above inequality and Assumption 3, we get
[Equation]
The conclusion follows by the same argument as in Theorem 3 via a telescopic sum over k .
F P ROOF OF T HEOREM
Recall that we set the following parameters
[Equation]
Similar to proof of Theorem
[Equation]
we have
[Equation]
(cid:107)
E [ h 2 k (cid:107) g k (cid:107) 2 ] (13) ≤ f ( x k )
[Equation]
E [ h k (cid:104) g k , ∇ f ( x k ) (cid:105) ]
+ 5 L 0 + 4 L 1 (cid:107)∇ f ( x k ) (cid:107)
E [ h 2 k ( (cid:107)∇ f ( x k ) (cid:107)+ (cid:107) g k − ∇ f ( x k ) (cid:107) 2 + 2 (cid:104)∇ f ( x k ) , g k − ∇ f ( x k ) (cid:105) )] ≤ f ( x k ) + E [ − h k + 5 L 0 + 4 L 1 (cid:107)∇ f ( x k ) (cid:107)h 2 k ] (cid:107)∇ f ( x k ) (cid:107)+ E [ h k ( − 1 + (5 L 0 + 4 L 1 (cid:107)∇ f ( x k ) (cid:107) )
h k ) (cid:104)∇ f ( x k ) , g k − ∇ f ( x k ) (cid:105) ]
+ 5 L 0 + 4 L 1 (cid:107)∇ f ( x k ) (cid:107)
E [ h 2 k ( (cid:107) g k − ∇ f ( x k ) (cid:107) 2 )]
First we show (5 L 0 + 4 L 1 (cid:107)∇ f ( x k ) (cid:107) )
h k ≤ 12 .
This follows by 5 L 0 h k ≤ 14 , h k 4 L 1 (cid:107)∇ f ( x k ) (cid:107) ≤ h k 4 L 1 ( (cid:107) g k (cid:107) + τ ) ≤ 14 .
Substitute in (15) and we get
E [
[Equation]
Then we bound T 1 , T 2 , T 3 in Lemma 11,12,13 and get
[Equation]
Rearrange and do a telescopic sum, we get
E
[Equation]
k
[Equation]
Furthermore, we know
[Equation]
Hence along with η ≤ T − 1 / 2 , we get
E [ (cid:88)
[Equation]
Let U = { k | η (cid:107)∇ f k (cid:107) 2 ≤ (cid:107)∇ f k (cid:107) 16 L 1 } , we know that
E [ (cid:88)
[Equation]
5 L
[Equation]
[Equation]
[Equation]
[Equation]
1 ) , and
E [ (cid:88) k ∈U
[Equation]
(cid:107)∇ f
[Equation]
(cid:107) [Equation] [Equation] [Equation]
[Equation]
L
[Equation]
Therefore,
E
[Table]
[Equation]
[Table]
[Equation]
The last inequality follow by the fact that either |U| ≥ T/ 2 or |U c | ≥ T/ 2 .
This implies that
E [min k ≤ T (cid:107)∇ f ( x k ) (cid:107) ] ≤ 2 (cid:15) when
[Equation]
where ∆ =
( f ( x 0 ) − f ∗ + (5 L 0 + 2 L 1 τ ) τ 2 + 9 τL 20 /L 1 ) .
By Markov inequality,
[Equation]
The theorem follows by the deﬁnition in (9).
F. T ECHNICAL LEMMAS
Lemma 11.
[Equation]
Proof.
By unbiasedness of g k and the fact that η is a constant, we have
[Equation]
The second last inequality follows by h k ≤ η and Cauchy-Schwartz inequality.
The last inequality follows by
[Equation]
The equality above holds because h k =
[Equation]
+
[Equation]
The lemma follows by 32 ηL 1 τ ≤ 1 / 4 .
Lemma 12.
[Equation]
Proof.
When (cid:107)∇ f ( x k ) (cid:107) ≥ L 0 /L 1 ,
[Equation]
The last inequality follows by (12).
When (cid:107)∇
[Equation]
[Equation]
≤
[Equation] [Equation]
Lemma 13.
[Equation]
Proof.
When (cid:107)∇ f ( x k ) (cid:107) ≥ L 0 /L
+ τ , we get
[Equation]
The ﬁrst inequality follows by h k ≤ η and (cid:107) g k − ∇ f ( x k ) (cid:107) ≤ τ ≤ (cid:107)∇ f ( x k ) (cid:107) .The
last inequality follows by (12).
When (cid:107)∇ f ( x k ) (cid:107) ≤ L 0 /L
+ τ , we get
[Equation] [Equation]
5 L
[Equation]
+ 2 L 1 τ
[Equation]
G P ROOF OF T HEOREM
Similar to proof of Theorem 3, we have
[Equation]
Sum across k ∈ { 0 , ..., T − 1 } and take expectations, then we can get
[Equation]
Rearrange and we get
[Equation]
By Jensen’s inequality,
[Equation]
By Markov inequality,
[Equation]
The theorem follows by the deﬁnition in (9) and Jensen’s inequality.
H E XPERIMENT DETAILS
In this section, we ﬁrst brieﬂy overview the tasks and models used in our experiment.
Then we explain how we estimate smoothness of the function.
Lastly, we describe some details for generating the plots in Figure 2 and Figure 3.
H. L ANGUAGE MODELLING
Clipped gradient descent was introduced in (vanilla) recurrent neural network (RNN) language model (LM) (Mikolov et al., 2010) training to alleviate the exploding gradient problem, and has been used in more sophisticated RNN models (Hochreiter and Schmidhuber, 1997) or seq2seq models for language modelling or other NLP applications (Sutskever et al., 2014; Cho et al., 2014).
In this work we experiment with LSTM LM (Sundermeyer et al., 2012), which has been an important building block for many popular NLP models (Young et al., 2017).
The task of language modelling is to model the probability of the next word w t +1 based on word history (or context).
Given a document of length T (words) as training data, the training objective is to minimize negative log-likelihood of the data − 1 T Σ
Tt =1 log P ( w t | w 1 ...
w t − 1 ) .
We run LM experiments on the Penn Treebank (PTB) (Mikolov et al., 2010) dataset, which has been a popular benchmark for language modelling.
It has a vocabulary of size 10k, and 887k/70k/78k words for training/validation/testing.
To train the LSTM LM, we follow the training recipe from 3 (Merity et al., 2018).
The model is a 3-layer LSTM LM with hidden size of 1150 and embedding size of 400.
Dropout (Srivastava et al., 2014) of rate 0.4 and DropConnect (Wan et al., 2013) of rate 0.5 is applied.
For optimization, clipped SGD with clip value of 0.25 and a learning rate of 30 is used, and the model is trained for 500 epochs.
After training, the model reaches a text-set perplexity of 56.5, which is very close to the current state-of-art result (Dai et al., 2019) on the PTB dataset.
H. I MAGE CLASSIFICATION
As a comparison
[Figure]
we run the same set of experiments on image classiﬁcation tasks.
We train the
ResNet20 (He et al., 2016) model on Cifar10 (Krizhevsky and Hinton, 2009) classiﬁcation dataset.
The dataset contains 50k training images and 10k testing images in 10 classes.
Unless explicitly state
[Figure]
we use the standard hyper-parameters based on the Github repository 4 .
Our baseline algorithm runs SGD momentum with learning rate 0.1, momentum 0.9 for 200 epochs.
We choose weight decay to be 5 e − 4 .
The learning rate is reduced by 10 at epoch 100 and 150.
Up to our knowledge, this baseline achieves the best known test accuracy (95.0%) for Resnet20 on Cifar10.
The baseline already beats some recently proposed algorithms which claim to improve upon SGD momentum.
[Figure]
[Figure] [Figure]
H.
E STIMATING SMOOTHNESS
Our smoothness estimator follows a similar implementation as in (Santurkar et al., 2018).
More precisely, given a sequence of iterates generated by training procedure { x k } k , we estimate the smoothness ˆ L ( x k ) as follows.
For some small value δ ∈ (0 , 1) , d = x k +1 − x k ,
[Equation]
This suggests that we only care about the variation of gradient along x k +1 − x k .
The motivation is based on the function upper bound (10), which shows that the deviation of the objective from its linear approximation is determined by the variation of gradient between x k +1 and x k .
H. A DDITIONAL PLOTS
The plots in Figure 2a show log-scale scattered data for iterates in the ﬁrst epoch.
To supplement this result, we show in Figure 5a
the linear scale plot of the same data as in Figure 2a.
In Figure 5b, we run the same experiment as in Figure 2a for 200 epochs instead of 1 epoch and plot the gradient norm and estimated smoothness along the trajectory.
In Figure 2, we plot the correlation between gradient norm and smoothness in LSTM LM training.
We take snapshots of the model every 5 iterations in the ﬁrst epoch, and use 10% of training data to estimate gradient norm and smoothness.
As shown in Figure 6, using 10% of the data provides a very accurate estimate of the smoothness computed from the entire data.
I A S YNTHETIC E XPERIMENT
In this section, we demonstrate the different behaviors of gradient descent versus clipped gradient descent by optimizing a simple polynomial f ( x ) = x 4 .
We initialize the point at x 0 = 30 and run both algorithms.
Within the sublevel set [ − 30 , 30] , the function satisﬁes
[Equation]
Therefore, we can either pick L 1 = 0 , L 0 = 1 .
08 4 for gradient descent or L 1 = 10 .L
= 0 .
1 for clipped GD.
Since the theoretical analysis is not tight with respect to constants, we scan the step sizes to pick the best parameter for both algorithms.
For gradient descent, we scan step size by halving the current steps.
For clipped gradient descent, we ﬁx threshold to be 0.01 and pick the step size in the same way.
The convergence results are shown in Figure 7.
We can conclude that clipped gradient descent converges much faster than vanilla gradient descent, as the theory suggested
[Figure]
J A QUANTITATIVE COMPARISON OF THEOREMS AND EXPERIMENTS
To quantify how much the result align with the theorem, we assume that the leading term in the iteration complexity is the (cid:15) dependent term.
For GD, the term scales as O ( ML 1 + L 0 √ T ) , while for
Clipped GD, the term scales as O ( L 0 √ T ) .
First, we start with the synthetic experiment presented in I. From theory, we infer that the improvement of f (cid:48) ( x GD ) f (cid:48) ( x CGD ) ≈
L 0 ML 1 + L 0 ≈ 1 e 5 .
In experiment, the best performing GD reaches f (cid:48) ( x T )
= 0 .
36 , while for clipped GD, the value is 1 .
3 e − 8 , and the ratio is f (cid:48)
( x GD ) f (cid:48) ( x CGD ) ≈ 1 e 7 .
This suggests that in this very adversarial (against vanilla GD) synthetic experiment, the theorem is correct but conservative.
Next, we test how well the theory can align with practice in neural network training.
To do so, we rerun the PTB experiment in 5 with a smaller architecture (2-Layer LSTM with 200 embedding dimension and 512 inner dimension).
We choose hyperparameters based on Figure 4a.
For clipped
GD, we choose clipping threshold to be 0.25 and learning rate to be 10.
For GD, we use a learning rate 2.
One interesting observation is that, though GD makes steady progress in minimizing function value, its gradient norm is not decreasing.
To quantify the difference between theory and practice, we follow the procedure as in the synthetic experiment.
First, we estimate ML 1 + L 0 = 25 for GD from Figure 8(c).
Second, we estimate
L 1 = 10 , L 0 = 5 for clipped GD’s trajectory from subplot (d).
Then the theory predicts that the ratio between gradients should be roughly 25 / 5 = 5 .
Empirically, we found the ratio to be ≈by taking the average (as in Theorem 3 and Theorem 6).
This doesn’t exactly align but is of the same scale.
From our view, the main reason for the difference could be the presence of noise in this experiment.
As theorems suggested, noise impacts convergence rates but is absent in our rough estimates
[Figure]