A New Accelerated Gradient Method Inspired by ContinuousTime Perspective
Anonymous authors
Paper under double-blind review
Abstract
Nesterov’s accelerated method are widely used in problems with machine learning background including deep learning.
To give more insight about the acceleration phenomenon, an ordinary differential equation was obtained from Nesterov’s accelerated method by taking step sizes approaching zero, and the relationship between
Nesterov’s method and the differential equation is still of research interest.
In this work, we give the precise order of the iterations of Nesterov’s accelerated method converging to the solution of derived differential equation as step sizes go to zero.
We then present a new accelerated method with higher order.
The new method is more stable than ordinary method for large step size and converges faster.
We further apply the new method to matrix completion problem and show its better performance through numerical experiments
Introduction
Optimization is a core component of statistic and machine learning problems.
Recently, gradientbased algorithms are widely used in such optimization problems due to its simplicity and efficiency for large-scale situations.
For solving convex optimization problem
[Equation]
where F ( x ) is convex and sufficiently smooth, a classical first-order method is gradient descent.
We assume that f ( x ) =
∇ F ( x ) satisfies L -Lipschitz condition, that is, there exists constant L such that
[Equation]
Under these conditions, gradient descent achieves a convergence rate of O ( n (cid:0) 1 ) , i.e., ∥ F ( x n ) (cid:0)
F ( x (cid:3) )
∥ decreases to zero at a rate of O ( n (cid:0) 1 ) , where x n denotes the n th iteration and x (cid:3) denotes the minimum point of F ( x ) in R d .
Nesterov’s accelerated method (Nesterov, 1983) is a more efficient first-order algorithm than gradient descent, of which we will use the following form: starting with x 0 = x 1 ,
[Equation]
for n (cid:21) 1 .
It is shown that under abovementioned conditions, Nesterov’s accelerated method converges at a rate of O ( n (cid:0) 2 ) .
Accelerated gradient method has been successful in training deep and recurrent neural networks (Sutskever et al., 2013) and is widely used in problems with machine learning background to avoid sophisticated second-order methods (Cotter et al., 2011; Hu et al., 2009; Ji & Ye, 2009).
To provide more theorical understanding, an important research topic of Nesterov’s accelerated method is to find an explanation of the acceleration.
On this topic, Nesterov’s method was studied via a continuous-time perspective (Su et al., 2014).
They considered a curve x ( t ) , introduced the ansatz x n (cid:25) x ( n p s ) and substituted it to (1.1).
Letting s ! 0 , they obtained the following differential equation.
[Equation]
The differential equation was used as a tool for analyzing and generalizing Nesterov’s scheme.
Furthermore, this idea has been studied from different directions.
A class of accelerated methods have been generated in continuous-time (Wibisono et al., 2016).
ODE (1.2) can also be discretized directly using Runge-Kutta method to achieve acceleration (Zhang et al., 2018).
Although many results have been achieved, the process of obtaining the differential equation (1.2) has not been rigorous, and the method is still time-consuming for large-scale problems.
In this work, we give the precise order of the iterations of Nesterov’s accelerated method converging to solution of the differential equation (1.2) with initial conditions
[Equation]
as step size s goes to zero.
Inspired from this perspective, we present a new accelerated method to make this convergence faster.
As we expected, iterations of the new method are closer to the solution x ( t ) of differential equation (1.2) than original Nesterov’s method.
Moreover, we find the new method is more stable than original Nesterov’s method when step size is large.
Based on abovementioned observations, we try to take advantage of the new method in more practical problems.
We apply the new method to matrix completion problem.
We combine the new method with proximal operator (Parikh & Boyd, 2014) into a new algorithm, which we call modified FISTA.
We find that the new method performs better than FISTA (Beck & Teboulle, 2009) and acclerated proximal gradient method (Parikh & Boyd, 2014) because it can work with larger step sizes.
This paper is organized as follows.
In section 2 , we prove that iterations of Nesterov’s accelerated method converge to solution of the differential equation (1.2).
In section 3 , we present a new method to make the convergence faster and show its better stablity through two simple examples.
In section 4 , we apply the new method to matrix completion problem.
A strict analysis of the relation between Nesterov’s method and its continuous-time limit
We refer to x ( t ) as the solution of differential equation (1.2) with initial conditions (1.3).
Existance and uniqueness of such solutions have been proved (Su et al., 2014).
In this section, We give the order of the iterations of Nesterov’s accelerated method converging to x ( t ) as step sizes go to zero.
For convenience, we substitute the first equation in Nesterov’s method (1.1) to the second one to get
[Equation]
We write s = h 2 and rewrite the above recurrence relation as
[Equation]
Inspired by the ansatz x n (cid:25) x
( n p s ) (Su et al., 2014), we consider the convergence between x n and x ( nh ) .
More precisely, we show that for fixed time t , x n converges to x ( t ) as h goes to zero, where n = th .
2.1 Truncation error
Firstly, we consider the following ’truncation error’:
[Equation]
(2.2) is obtained from (2.1) by replacing x n +1 ; x n ; x n (cid:0) 1 with x ( t + h ) ; x ( t ) ; x ( t (cid:0) h ) and substituting the relation n = th .
Our first result is the order of truncation error L [ x ( t ); h ] .
Theorem 1.
Assume f satisfies L -Lipschitz condition, and solution x ( t ) of the derived differential equation (1.2) has a continuous third derivative.
For fixed time t , the truncation error (2.2) satisfies
[Equation]
Theorem 1 shows the size of error caused by a single iteration when the starting point is just on x ( t ) .
Then we have to add up these errors to prove the convergence proporty we need.
2.2 Convergence theorem
We now come to the convergence theorem.
In this theorem, we give the precise order of the iterations of Nesterov’s method converging to solution of the derived differential equation.
Theorem 2.
Under conditions in Theorem 1, for fixed time t , x t=h converges to x ( t ) as h goes to zero at a rate of O (
h ln 1 h ) if x 0 = x (0) and x 1 = x ( h ) .
Theorem 2 coincides with derivation of ODE (1.2) (Su et al., 2014).
New accelerated method 3.1 Derivation of the new method and analysis of truncation error
Inspired from the continuous-time perspective and our proof of the convergence from iterations of
Nesterov’s method to its continuous-time limit, we present a new method to make this convergence faster.
Precisely, the new method has a higher truncation order.
We need one more step in our scheme than in Nesterov’s method to achieve higher truncation order in the following analysis, so we consider a recurrence relation with form
[Equation]
where f (cid:11) i g , f (cid:12) i g and f (cid:13)
i g are to be determined.
Now we expand x
( t (cid:0) h ) to first order.
Calculation shows that
[Equation]
Substitute this expansion to truncation error
[Equation]
and choose parameters appropriately to eliminate low-order terms [Equation] we get the following recurrence relation
[Equation]
Here we rewrite this scheme as Algorithm
[Equation]
[Equation] For truncation order of this new method, we have the following theorem.
The abovementioned procedure is presented in Appendix A.4 detailedly, as proof of Theorem 3.
Theorem 3.
If f has continuous second order derivative, the first and second derivative are bounded, and x ( t ) has continuous fourth derivative, then for fixed t , truncation error of (3.2) satisfies
[Equation]
The convergence of the new method and x ( t ) can be proved similar to Theorem 2.
3.2 Advantage of the new method
Since the new method has a truncation error of higher order than original Neaterov’s method, the iterations of the new method converge to the differential equation (1.2) when those of original
Nesterov’s method diverge.
In another word, the new method is more stable for large step size.
We present two numerical results in Figure 1 to confirm it.
Quadratic. F ( x ) =
x T Ax is a strongly convex function, in which x 2 R 2 and A is a 2 (cid:2) 2 matrix.
Linear regression.
F ( x ) =
n ∑ i =
( y i (cid:0) w T i x ) 2 , where n is the number of samples and ( w i ; y i ) is the i th sample.
[Figure]
In Figure 1(a), step size s = 0 : 03705 ; in Figure 1(b), step size s = 0 : 00565 .
In these examples, iterations of the new method converge to the minimum point, while those of original Nesterov’s method diverge, which confirms that the new method is more stable for large step size. 3.3 Absolute stability of Nesterov’s method and the new method In this subsection, we explain the better stability of the new method with absolute stability theory. Firstly, recall the scheme of our new method
[Equation]
We use linear approximation f [Equation] and the characteristic equation of this finite scheme is approximately (cid:21) 3 (cid:0) [Equation]
For large n
[Equation]
we can ignore the high order terms and the characteristic equation becomes
[Equation]
According to the absolute stability theory, the numerical stability of Nesterov’s scheme with respect to accumulated roundoff error is equivalent to this: all the roots of the characteristic equation lie in the unit circle (Leader, 2004).
Noticing that the left hand of the equation can be factorized to (
[Equation]
the largest modulu of the roots is 1 when 0 (cid:20) s (cid:1) ∇ 2 F (cid:20) 4 , and the absolutely stable region of the new method is s (cid:1) ∇ 2 F 2 [0 ; 4] .
When s (cid:1)∇ 2 F lies in the absoletely stable region, the related theory guarantees that the error caused by every iteration will not be magnified as the iteration number increases.
To make the analysis more precise, we should consider the difference of the scheme between iterations caused by different n .
We define the transfer matrix
[Equation]
CA and Q n = P n P n (cid:0) 1 (cid:1) (cid:1) (cid:1) P 1 .
Error analysis shows that if the largest modulu of eigenvalues of Q n goes to zero, then error caused by iterations will be eliminated as the iteration number increases.
Figure 2 presents the largest module of eigenvalues of Q n for different values of s (cid:1) ∇ 2 F .
From the experiment we can see that the above condition is satisfied.
[Figure]
We then apply the same method to Nesterov’s method discussed in (Su et al., 2014) and conclude that the absolutely stable region of Nesterov’s method is [0 ; 43 ] .
According to the above analysis, the absolutely stable region of the new method is four times as large as Nesterov’s method, so the new method is more stable, and we can choose larger step sizes to achieve faster convergence.
Application to matrix completion problem: modified FISTA
Our theory and numerical results show that the new method is more stable than original Nestrov’s method.
So we can choose larger step size for new method and convergence to the optimal solution can be faster, compared with original Nesterov’s method.
In this section we apply the new method to matrix completion problem.
We present a new algorithm which can be viewed as a modification of the well-konwn fast iterative shrinkage-thresholding algorithm (FISTA) (Beck & Teboulle, 2009).
The performance of modified FISTA can also confirm the advantage of the new method.
For matrix completion problem there exists a ’true’ low rank matrix M .
We are given some entries of M and asked to fill missing entries.
There have been various algorithms to solve such problem (Candès & Recht, 2009; Keshavan et al., 2010).
Besides, it is proposed that matrix completion can be transformed to the following unconstrained optimization problem (Mazumder et al., 2010)
[Equation]
Notice that F ( X ) is composed of a smooth term and a non-smooth term, so gradient-based algorithms cannot be used directly.
Proximal gradient algorithms (Parikh & Boyd, 2014) are widely used in such composite optimization problems, and fast iterative shrinkage-thresholding algorithm (FISTA) is a successful algorithm.
Moreover, FISTA has been extended to matrix completion case (Ji & Ye, 2009).
For convenience, we set G ( X ) =
∥
X obs (cid:0) M obs ∥ 2 , H ( X ) = (cid:21) ∥
X ∥ (cid:3) , and g ( X ) = ∇ G ( X ) :
The idea of FISTA builds on Nesterov’s method.
We also apply acclerated proximal gradient method (Parikh & Boyd, 2014) for our numerical experiment, which is composed of Nesterov’s method and proximal gradient descent.
These two algorithms are presented in Appendix A.5.
We find the performances of them are similar in our experiments.
Our contribution is the third method (Algorithm 2), the new method (3.2) combined with proximal operator, which we call modified FISTA.
Input: step size s
Initial value: X 2 = X
1 = X 0 2 M 100 .
( k (cid:0) 1) th iteration ( k (cid:21) 2) .
Compute
[Equation]
Notice that the minimizing problems in interations of above three algorithms can be solved directly by singular value decomposition (Cai & Candès, 2010).
We take experiments on a simulated data set.
We use fixed step sizes in the above three algorithms, and the performances are presented in Figure 3.
We find empirically that for all methods, convergence is faster when step size is larger, so we choose the largest step sizes for all methods to compare their fastest convergence speed.
Through experiments, we find the largest step size that makes modified FISTA convergent is 4 : 1 (accurate to one decimal place), while those for the first two algorithms are both 1 : 3 .
We also compare performances of the three methods with step sizes reduced from the largest in equal proportion.
We find that when step sizes are chosen to be the largest or reduced from the largest in equal proportion ( 80% , 50% , 10% ), convergence of modified FISTA is faster than the other two methods. [Figure]
We also combine the three methods with backtracking (Beck & Teboulle, 2009) to choose step sizes automatically.
We present modified FISTA with backtracking as Algorithm [Figure] and the other two algorithms are similar.
Performances of the three algorithms with backtracking on abovementioned data set are presented in Figure 4.
Convergence of modified FISTA is faster than the other two methods.
Moreover, we find that the final step size of modified FISTA is larger.
Discussion
In this paper we prove that iterations of Nesterov’s accelerated method converge to solution of the derived differential equation as step sizes go to zero.
We present a new accelerated method to make this convergence faster.
We use numerical results to show that the new method is more stable, especially for large step sizes, and explan it using the order of truncation error.
We then apply the new method to matrix completion problem and present a new algorithm which we call modified FISTA.
Numerical experiments show that modified FISTA performs better than existing algorithms based on Nesterov’s acceleration because it can work with larger step sizes.
We will also combine our new method with stochastic gradient-based algorithms and apply the new method to deep networks in the future.
Algorithm
Input: some (cid:12) <
Initial value.
X
= X
1 =
X 0 2 M 100 , step size s 2 .
( k (cid:0) 1) th iteration ( k (cid:21) 2) .
[Equation]
Find the smallest positive integer i k +1 such that with s = (cid:12)
i
k +1 s k
[Equation]
where f X = arg min
[Equation]
Set s k
+1 =
(cid:12)
i
k +
s k and compute
[Equation] [Figure]
) (cid:0) F ( x (cid:3) )
j . Step sizes are chosen by backtracking. Our work shows that for an accelerated gradient method, the rate at which it converges to the derived differential equation is possibly related to its property as an optimization algorithm. We think this work suggests that more consideration should be given to the corresponding differential equations when studying optimization algorithms.
A Appendix A.1 Proof of Theorem
Theorem 1.
Assume f satisfies L -Lipschitz condition, and solution x ( t ) of the derived differential equation (1.2) has a continuous third derivative.
For fixed time t , the truncation error (2.2) satisfies
L [
[Equation]
A
[Equation] [Equation]
Proof.
Notice that
[Equation]
Substitute this equation to the last term of L [ x ( t ); h ] to get
[Equation]
Since f satisfies L -Lipschitz condition, we know
[Equation]
To get the second equality, we substitute the differential equation (1.2).
Then we expend the first and third terms of L [ x ( t ); h ] to third order to get
[Equation]
Substitute these three equations to (2.2), we have
[Equation]
Remark 1.
(A.1) can also be written as
[Equation]
where M 1 depends on sup s (cid:20) t j x (1) ( s ) j and sup s (cid:20)
t
j x (3) ( s ) j .
Remark 2.
Theorem 1 deals with the problem for fixed time t .
To finish the proof of the convergence, we have to consider the situation that t n = nh , where n (cid:21) 1 is fixed.
We set a fixed time t 0 and assume that t n = nh < t 0 .
Since x ( t ) has a continuous third derivative, x ( t ) and its first to third derivative are bounded in [0 ; t 0 ] .
We replace time t in the above proof by t n and expend the terms of (2.2).
Now the term
[Equation]
obtained from the expansion of x ( t n (cid:0) 1 ) cannot be viewed as O ( h 3 ) , but there exists M 2 > 0 such that
[Equation]
As a consequence, we have
[Equation]
where M 1 and M 2 rely on t 0 .
A.2 Two lemmas for Theorem
For the proof of Theorem 2, we need the following two lemmas.
Lemma 1.
(Holte, 2009)
For constant (cid:11); (cid:12) > 0 and positive sequence f (cid:17) n g n (cid:21) 0 satisfying
[Equation]
; the following inequality holds
[Equation]
The above lemma is a classic result and refered to as discrete Gronwall inequality.
Lemma 2.
We define matrices C n and D n;l as
[Equation]
where n (cid:21) 0 and 0 < l (cid:20) n + 1 .
In addition, we set D n; 0 = I 2 .
Then there exist positive constants
M; M 3 such that for all n , the following two inequalities hold, where the matrix norm is 2-norm.
[Equation]
Proof.
Since
C
[Equation]
we notice that when n (cid:21) 2 ,
D n;n (cid:0) 1 = (    ) ;
D n;n = (    ) ;
D n;n +1 =
(    ) ; having nothing to do with the value of n .
So it is obvious that there exists M 3 to make (??) holds and M 4 > 0 such that for all n < 2 or n (cid:21) 2 ; l > n (cid:0) 2 or l =
[Equation]
Then we consider the condition when n (cid:21) 2 ; 0 < l (cid:20) n (cid:0) 2 .
Notice that
[Equation]
For convenience, we write
[Equation]
Assume we have alreagy got
[Equation]
satisfying
[Equation]
then since
[Equation]
and 0 (cid:20) n
[Equation]
l
[Equation]
n [Equation]
D n; [Equation] +1 has the same form
[Equation]
satisfying
[Equation]
Then for fixed n , induce from l = 1 , we get
[Equation]
satisfying
[Equation]
for all n (cid:21) 2 ; 0 < l (cid:20) n (cid:0) 2 .
Then we can estimate ∥
D n;l ∥ .
Notice that
[Equation]
The eigenvalues of this matrix are
[Equation]
Combining this representation with (A.4), we get the estimation
[Equation]
So there exists M 5 > 0 , such that for all n (cid:21)
[Equation]
holds.
Combining (A.3) with (A.5), we finish the proof.
A.3 Proof of Theorem
Theorem 2.
Under conditions in Theorem 1, for fixed time t , x t=h converges to x ( t ) as h goes to zero at a rate of O ( h ln 1 h ) if x 0 = x (0) and x 1 = x ( h ) .
Proof.
In this proof, we first calculate the error caused by a single iteration, which can be divided into an accumulation term and a truncation term.
Then we use the estimation given by Theorem 1 and apply discrete Gronwall inequality to prove the convergence.
Recall the recurrence relation
[Equation]
and the definition of truncation error
[Equation]
where t n = nh .
Subtract the above two equations, and introduce overall error
[Equation]
we have
[Equation]
which can also be written as
[Equation]
where
[Equation]
We will also use the notation
[Equation]
Then we rewrite (A.6) into a form that is convenient for recurrence.
We set
[Equation]
Then (A.6) can be written as
[Equation]
By recursive method
[Equation]
we have
[Equation]
With the notations introduced in Lemma 2, this equation can be written as
[Equation]
Now we need to estimate ∥ B n ∥ .
Since f satisfies L -Lipschitz condition, from (A.7) we have
[Equation]
and
[Equation]
Take norm on both sides of (A.8) and substitute (A.9) and conclusion of Lemma 2, we have the following estimation
[Equation]
Now we deal with truncation errors.
Recall (A.2) in remark of Theorem
[Equation]
Take sum to obtain
[Equation]
Notice the classic inequality n
[Equation]
where M e refers to a positive constant.
Substitute it to (A.11), we have
[Equation]
Substitute this inequality to (A.10), we get a control of ∥ E n ∥ ∥
[Equation]
Using discrete Gronwall inequality, we have
[Equation]
Then for fixed t , we choose n = th to get ∥ E
[Equation]
Notice that
[Equation]
so if E 0 = 0 , then the vector form of overall error E t=h satisfies
[Equation]
.4 Proof of Theorem
Theorem 3.
If f has continuous second order derivative, the first and second derivative are bounded, and x ( t ) has continuous fourth derivative, then for fixed t , truncation error of (3.2) satisfies
[Equation]
Proof.
Recall the proof of Throrem 1.
Now we expand x
( t (cid:0) h ) to first order x ( t (cid:0) h ) =
x ( t ) + hx (1) ( t ) + O ( h 2 ) :
Then we have
[Equation]
We now expand f :
[Equation]
To do this
[Equation]
we need f has continuous second derivative and the second derivative is bounded.
Take derivetive on both sides of differential equation
[Equation]
we have
[Equation]
So
[Equation]
Expand x ( t + h ) ; x ( t (cid:0) h ) ; x ( t (cid:0) 2 h ) to the third order, we have (
[Equation]
Substitute these three equations and (A.12) to truncation error of recurrence relation ( [Equation]
[Equation]
h )) ) ;
then simple calculation shows that terms with order less than four will be eliminated if we choose coefficients according to the following equations
[Equation]
where k; m 1 ; m 2 can be chosen randomly.
Notice that coefficients of recurrence relation (3.2) satisfy above equations.
A.5 Algorithms
Algorithm
Input: step size s
Initial value: Y 1 =
X 0 2 M 100 , t 1 = 1 .
k th iteration ( k (cid:21) 1) .
Compute
[Equation]
Input: step size s
Initial value: X 1 = X 0 2 M 100 .
k th iteration ( k (cid:21) 1) .
Compute
Y k
= X k (cid:0) 1 +
[Equation]
A.6 Details about Numerical Experiments in Section
Here we produce some details for our numerical experiments in Section 4.
Our experiments are taken on a simulated data set.
Firstly, we generated the ’true’ low rank matrix M .
To do this, we generate a random matrix M 0 .
Entries of M 0 are independent and uniformly distributed on (0 ; 20) .
{+Then we compute the singular value decomposition of M 0 , that is, M 
+}
= U (cid:6) V T .
{+After that, we set M = U (cid:6) 
+}
V T , where (cid:6) 0 is a diagonal matrix with only three nonzero diagonal elements.
It is not difficult to prove that M has rank 3 .
Secondly, we generate the observation set.
For every row of M , we choose randomly ten entrys to be observed.
As a consequence, 10% entries are observed in total.
After data generation step, we apply the abovementioned algorithms (accelerated proximal gradient method, FISTA and our modified FISTA) with fixed step sizes and backtracking to this data set.
{+The parameter of the loss function (4.1) is (cid:21) = 
+}
For initial point, we simply choose the zero matrix (every entry equals to zero).
For backtracking, we set the initial step size as 10 and the decay factor (cid:12) = 0 : 1 .