B ETTER G ENERALIZATION BY E FFICIENT T RUST R EGION M ETHOD
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we develop a trust region method for training deep neural networks.
At each iteration, trust region method computes the search direction by solving a non-convex subproblem.
Solving this subproblem is non-trivial—existing methods have only sub-linear convergence rate.
In the ﬁrst part, we show that a simple modiﬁcation of gradient descent algorithm can converge to a global minimizer of the subproblem with an asymptotic linear convergence rate.
Moreover, our method only requires Hessian-vector products, which can be computed efﬁciently by back-propagation in neural networks.
In the second part, we apply our algorithm to train large-scale convolutional neural networks, such as VGG and MobileNets.
Although trust region method is about 3 times slower than SGD in terms of running time, we observe it ﬁnds a model that has lower generalization (test) error than SGD, and this difference is even more signiﬁcant in large batch training.
We conduct several interesting experiments to support our conjecture that the trust region method can avoid sharp local minima.
I NTRODUCTION
Despite having many optimization algorithms in the literature (Nesterov, 2013; Wright & Nocedal, 1999; Boyd & Vandenberghe, 2004), when talking about neural network optimization, by far the most popular methods are SGD and its variants such as AdaDelta (Zeiler, 2012), AdaGrad (Duchi et al., 2011), Adam (Kingma & Ba, 2014) and RMSProp (Hinton et al., 2006).
Compared to these stochastic ﬁrst order methods, second order methods are rarely used in training deep neural networks due to large data ﬁles and high dimensional parameter space.
Moreover, neural network has nonconvex loss function and its Hessian is often ill-conditioned, making it difﬁcult to apply second order methods.
Prior to our work, there aren’t many works on second order methods for neural network optimization (Martens, 2010; Kiros, 2013; Botev et al., 2017), and most of them are tested on multilayer perceptron so it is unclear whether second order methods can be useful for more advanced networks such as convolutional neural networks.
In this paper, we develop an efﬁcient trust region method for training deep neural networks, where the objective function is approximated by a quadratic term with a norm constraint, known as the trust-region subproblem.
The main difﬁculty in large-scale applications is how to solve this nonconvex subproblem efﬁciently (Yuan, 2000).
Approximate solvers such as (Steihaug, 1983; Powell, 1970) cannot converge to the exact minimizer, while the best method for computing the global minimizer has sub-linear convergence rate (Hazan & Koren, 2016).
We propose a new solver that is guaranteed to converge to a global minimum of the nonconvex trust region sub-problem with an asymptotic linear convergence rate.
What’s more, our method only requires Hessian-vector product, which only costs two forward-backward operations in neural networks.
Empirically we ﬁnd this technique making trust region method comparable to SGD in terms of running time.
Apart from convergence rate, generalization ability is another important issue to practitioners.
Starting from (Zhang et al., 2016) which proposed the generalization paradox of deep models, intensive works have been done to ﬁnd ways to explain and close generalization gap in large batch SGD (Keskar et al., 2016; Kawaguchi et al., 2017; Wilson et al., 2017).
Among them, (Keskar et al., 2016) empirically ﬁnds the generalization gap between large and small batch and later (Wilson et al., 2017) notices that adaptive methods like Adam, AdaGrad and RMSprop are even worse than SGD concerning large batch generalization.
To close the generalization gap, (Chaudhari et al., 2016) changes
[Equation]
−
−
• If q t ≥ 0 .
9 , then x t +1 = x t + z t , r t +1 = 2 r t .
If q t ≥ 0 .
1 , then x t +1 = x t + z t , r t +1 =
r t .
• If q t < 0 .
1 , then x t +1 = x t , r t +1 = 0 .
5 r t .
• If q t ≥ 0 . 9 and (cid:107) g (cid:107) ≥ r t , then x t +1 = x t + z t , r t +1 = min { 2 r t , r max } . • Otherwise, x t +1 = x t , r t +1 = 0 . 5 r t
the loss function to eliminate sharp local minima, (De et al., 2016) selects batch size adaptively and (Hoffer et al., 2017) combines adaptive step size with new batch normalization layer.
In contrast to these works, we ﬁnd through experiments that using trust region method alone can effectively escape from sharp minima and achieve lower test error than SGD, especially on largebatch training.
For CIFAR-10 with VGG network, our trust region based method achieves 86.8% accuracy and our hybrid method have 88.1%, both of them are better than SGD’s solution with 86.7% accuracy when batch size is 128.
As batch size grows to 2048, our trust region method still has 85.5% accuracy and our hybrid method is 87.0%, in contrast, SGD only has 80.3% accuracy.
Arguably, our method is a more natural solution to large batch training because it can match the small batch accuracy simply by switching to trust region method without changing network structure, loss function and batch size.
B ACKGROUND AND R ELATED W ORK 2. (S TOCHASTIC ) T RUST REGION METHOD
Trust region (TR) method (Conn et al., 2000) (Algorithm 1) is a classical second order method for solving nonlinear optimization problems.
At each iteration, TR forms a second order approximation around the current solution and updates the solution by minimizing this quadratic function within the “trust region”.
The trust region, usually a hyper ball, is chosen to make the second order approximation close enough to the original one and thus by minimizing the subproblem, the objective function will very likely to descent.
More formally, suppose F ( x ) is the objective function, at iteration x t the subproblem can be written as: [Equation]
[Equation]
Where H = ∇ 2 F ( x t ) , g = ∇ F ( x t ) , and r > 0 is the radius of trust region.
If objective function F has ﬁnite sum structure, F ( x ) = 1 n (cid:80) ni =1 f i ( x ) or more generally F ( x ) =
E ξ f ( x, ξ ) where ξ is random variable with bounded variance, then we can turn to stochastic trust region method discussed in (Chen et al., 2015).
Both deterministic and stochastic trust region method have very similar algorithm as one can see in Algorithm 1.
As to convergence rate, (Blanchet et al., 2016) shows that the stochastic version is guaranteed to ﬁnd a stationary point (cid:107)∇ f ( x ) (cid:107) ≤ (cid:15) in O ( (cid:15) − 2 ) iterations, while the deterministic version is able to ﬁnd (cid:15) -suboptimal stationary point within O ( (cid:15) − 3 / 2 ) iterations (Curtis et al., 2017).
For large scale problem when we cannot calculate the gradient and Hessian on the whole dataset, stochastic method is the only choice.
To apply trust region method to large-scale non-convex optimization (e.g., training deep networks), there are two difﬁculties: (a) The Hessian matrix H may not be positive deﬁnite, so (2) is a nonconvex problem with constraint and it is hard to compute its global minimizer.
(b) H is very large (millions by millions) and cannot be formed explicitly.
For (a), several algorithms have been proposed in the literature.
Traditionally, dogleg method (Powell, 1970) for H (cid:31) 0 and Steihaugs method (Steihaug, 1983) for indeﬁnite H can be applied to get an approximated solution.
However if a more precise solution is desired, we need to ﬁnd other ways, e.g. (Hazan & Koren, 2016) is able to ﬁnd an (cid:15) -suboptimal
solution in ˜ O
( M/ √ (cid:15) ) time, where M is the matrix size (number of nonzero elements).
In this paper, we design an efﬁcient solver based purely on gradient information that can ﬁnd an (cid:15) -suboptimal solution in asymptotically linear time
˜ O
( M log(1 /(cid:15) )) .
For (b), due to high dimensionality of H , matrix decomposition such as SVD will be in-feasible.
Therefore, an efﬁcient subproblem solver should only involve computations of Hessian-vector product, which can be computed by back-propagation in deep networks (Pearlmutter, 1994;
Baydin et al., 2015).
Furthermore, since machine learning problems can usually be formulated as ﬁnite sum structure F ( x ) =
1 N (cid:80) Ni =
1 f i ( x ) , instead of computing the exact Hessian we can compute the subsampled Hessian on the selected batch, so the subsampled Hessian-vector product computation is only slower than SGD computation by a constant factor.
Our proposed subproblem solver only involves
Hessian-vector product, so it can be applied to large-scale problems and replace SGD seamlessly.
This subsampled version is justiﬁed by the stochastic trust region method described above.
2. O THER S ECOND O RDER M ETHODS FOR T RAINING N EURAL N ETWORKS
Another related technique for non-convex optimization is adaptive regularization with cubic (ARC)
(Griewank, 1981).
Similar to TR method, ARC uses cubic term of positive coefﬁcient
[Equation]
to replace constraint in (2):
[Equation] [Equation]
ARC method can be seen as the “twin” method of TR, since they share many similar properties (e.g. converge rates).
Intuitively the radius parameter r in (2) plays the same role as σ in ARC: a larger r and smaller σ both mean larger “trust-region”.
However, since the ARC subproblem is unconstrained, the treatment is relatively easy: for example, generalized Lanczos-type iterations (Gould et al., 1999) or gradient descent (Carmon & Duchi, 2016) can be used.
Our proposed subproblem solver for TR is also based on gradient descent, but due to constraint in the TR subproblem our algorithm and proof are different from (Carmon & Duchi, 2016).
Recently, several works have applied second order methods to neural network optimization, including (Martens, 2010; Botev et al., 2017; Xu et al., 2017b;a).
However, all of them only consider small-scale fully connected networks with few layers.
Among them, (Xu et al., 2017b;a) adopt full gradient and subsampled Hessian and solve the trust region subproblem in the subspace spanned by Cauchy point and minimum eigenvector v 1 ( H ) .
We are different from this recent paper in two aspects.
First, we develop a new trust region subproblem solver with better theoretical guarantee.
Second, the full gradient computation in their algorithm is too expensive for large-scale deep network training.
In contrast, we implement our algorithm with subsampled gradient and Hessian, thus we are able to run large-scale experiments and observe better generalization compared with SGD.
2. G ENERALIZATION A BILITY FOR F IRST O RDER M ETHODS
SGD and other stochastic ﬁrst order methods are popular for training large scale neural networks since gradient for each training sample can be computed in one forward-backward cycle.
Recently there are many works on the effect of batch size on training time as well as generalization ability.
Indeed, a larger batch makes more use of parallel processors, so the program can scale perfectly to more GPUs.
Moreover, more samples make the gradient less noisy and thus a larger learning rate and higher converge rate are possible, see (Goyal et al., 2017;
You et al., 2017a;b) and references therein.
At the same time, large batch in SGD is harmful to generalization ability, experiments in (Keskar et al., 2016) show that batch size can affect the geometry around the solution that SGD ﬁnds.
Specifically, SGD with a larger batch size tends to ﬁnd sharper local minima, so even a tiny change of data from training set to test set will lead to high loss, i.e. bad generalization ability.
As to the converge rate to local minima, although (Lee et al., 2016) shows gradient descent always converges to local minima, (Du et al., 2017) proves the converge process can take exponential time.
On the other hand by adding extra noise, (Ge et al., 2015) claims SGD is able to escape strict saddle in poly ( d/(cid:15) ) iterations, and further (Jin et al., 2017) design a full gradient with noise method that improves the result to poly- log( d ) /(cid:15) 2 steps.
Notations : Throughout this paper, we use S n − 1 to denote the unit sphere in R n .
The model parameters x ∈ R d and objective function F ( x ) = 1 N (cid:80) Ni =
1 f i ( x ) is in C 2 ; its gradient and Hessian at step x t are denoted as g t = ∇ F ( x t ) and H t = ∇ 2 F ( x t ) , and if the index t is unimportant we will simplify them as g and H respectively.
For the Hessian matrix H , its eigenvalue decomposition is H = (cid:80) di
=1 λ i v i v (cid:124)
i such that v (cid:124)
i v j = δ ij and λ 1 ≤ λ 2 ≤ · · · ≤ λ d , accordingly we decompose any vector a ∈ R d as a = (cid:80) di
=1 a ( i ) v i .
Denote operator norm β = (cid:107) H (cid:107) op = max {| λ 1 | , | λ d |} .
(cid:107) · (cid:107) represents 2-norm if not stated explicitly, I d ∈ R d × d is the identity matrix.
Finally, following (Carmon & Duchi, 2016)
we use s to represent the global minimizer of trust region subproblem (2).
E FFICIENT T RUST R EGION SUBPROBLEM SOLVER
Algorithm
Input: Gradient g , Hessian matrix H
Output: Approximated solution z 1: Initialize z 0 and η so that Assumption 1,2 are met.
2: boundary=false 3: for t = 0 , 1 , . . .
, N 1 do 4: if (cid:107) z t (cid:107) < 1 then 5: z t +1 =
z t − η ( Hz t + g ) 6: else 7: boundary = true 8: break 9: end if 10: end for 11: if boundary = false then return z N 1 + 12: end if 13: for t (cid:48)
=
t, t + 1 , . .
. , N 2 do 14:
Choose α t (cid:48) by Armijo line search 15:
[Equation]
=
[Equation]
16: end for 17: return z N 2 +
In this section, we show an efﬁcient gradient-descent based algorithm with proper initialization can ﬁnd the global minimum of the trust region subproblem (2).
If the global minimum lies inside of the sphere then gradient descent itself is guaranteed to ﬁnd it; Otherwise we ﬁrst conduct gradient descent until the iterate hits the spherical constraint, then a manifold gradient descent on the sphere can converge to the solution.
We prove this simple procedure can return the global minimum of the non-convex trust region subproblem and has asymptotically linear convergence rate.
The details are shown in Algorithm 2.
3. P ROPERTIES OF GLOBAL MINIMUM
The necessary and sufﬁcient condition of global minimum comes from KKT condition, see (Wright & Nocedal, 1999) for details.
Lemma 1.
(Global minimum) s is the global minimum of (2) if and only if (cid:107) s (cid:107) ≤ 1 and there is a scalar λ ≥ 0 such that: [Equation]
Proposition 1.
Based on (4) we can describe the solution(s) as follows:
• λ 1 > 0 : only one global minimum.
If (cid:107) H − 1 g (cid:107) <
1 then (cid:107) s (cid:107) < 1 , otherwise (cid:107) s (cid:107) = 1 .
• λ 1 =
: if g (1) (cid:54) = 0 , only one solution with (cid:107) s (cid:107) = 1 ; Otherwise if g (1) = 0 and (cid:80)
ni
=2 ( g ( i ) λ i ) 2 ≥ 1 , one solution with (cid:107) s (cid:107) = 1 ; If g (1) = 0 and (cid:80) ni =2 ( g ( i ) λ i ) 2 < 1 , there are multiple solutions.
• λ 1 < 0 , g (1) (cid:54) = 0 : only one solution.
• λ 1 < 0 , g (1) =
: if (cid:80) ni =2 ( g ( i ) λ
i − λ 1 ) 2 ≥ 1 then only one global minimum, otherwise there are multiple ones.
Proof details are postponed to appendix.
Since when λ 1 ≥ 0 every stationary points are global minimum, and further if (cid:107) s (cid:107) < 1 then we only need to use gradient descent to solve it (that the case for Line 11 in Algorithm 2), otherwise (cid:107) s (cid:107) =
1 and our manifold based algorithm also applies, since in this case subproblem is strongly convex, we don’t need to worry about converging to suboptimal point.
Now we only need to consider about λ 1 < 0 case as follows.
Notice that in this case we always have (cid:107) s (cid:107) = 1 .
In the following lemma we try to distinguish global minimum from other stationary points, restricted to g (1) (cid:54) = 0 case (we call it “easy-case”, as oppose to “hard-case” in (Wright &
Nocedal, 1999)).
As to the “hard-case” g (1) = 0 , in theory we can apply a small perturbation to b as (Carmon & Duchi, 2016) does: b (cid:48)
=
b + ε where ε is a small Gaussian noise.
In practice due to rounding error this case is hardly seen, for best efﬁciency we choose to ignore it.
The following lemma gives a sufﬁcient condition of global minimum:
Lemma 2.
When g (1) (cid:54) = 0 and λ 1 < 0 , among all stationary points if s (1) g (1) ≤ 0 then s is the global minimum.
3. C ONVERGENCE OF ITERATION
Now it left to see how the solution found by Algorithm 2 will meet Lemma 2.
To this end, we need to enforce following assumptions:
Assumption 1.
(Bounded step size)
Step size η < 1 /β .
Assumption 2.
(Initialize) z
= − α g (cid:107) g (cid:107) , 0 < α < 1 .
We remark that Assumption 2 is a good guess of global minimum if ignoring the curvature information, and using this alone as subproblem solution will reduce trust region method to gradient descent.
Under these assumptions, Algorithm 2 is guaranteed to ﬁnd (one of) the global minimum, which is formally stated in the following theorem:
Theorem 3.
Under proximal gradient descent update: z t +1 =
Prox (cid:107)·(cid:107) (cid:0) z t − η ∇ f ( z t ) (cid:1) , and Assumption 1 if z ( i ) t g ( i ) ≤ 0 then z ( i ) t +1 g
( i ) ≤ 0 .
Combining with Assumption 2 and Lemma 2, if λ 1 < 0 , g (1) (cid:54) =
then z t converges to a global minimum s .
To see the converging process more clearly, we divide the iterations into two phases: in the ﬁrst phase z t stays strictly inside the sphere S n − 1 : (cid:107) z t (cid:107) < 1 , and during this phase we will show that (cid:107) z t (cid:107) is monotone increasing until it hits the sphere and that is when the second phase begins.
In the second phase the iterates adhere to the sphere and converge to the global minimum with asymptotically linear rate.
First of all we show the monotone increasing property based on following lemma:
Lemma 4.
For z t deﬁned above, we have z (cid:124)
t H ∇ f
( z t ) ≥ βz (cid:124) t ∇ f
( z t ) (recall we deﬁne β as the operator norm of H ).
Now imagine there is another iterate ˜
z t that does “plain” GD (i.e. without projection): ˜ z t +1 = ˜
z t − η ∇ f
(˜ z t ) , using the same step size as z t and same initialization ˜
z
= z 0 .
Such a iteration rule guarantees that as long as (cid:107) z t (cid:107) <
1 for all z ≤ τ then ˜
z t = z t .
By showing (cid:107) ˜
z t (cid:107) is monotone increasing, we actually prove monotone increasing property in the ﬁrst phase.
Theorem 5.
Suppose z t is in the region such that proximal gradient update equals to plain GD:
˜ z t +1 = ˜
z t − η ∇ f
(˜ z t ) , then under this update rule, (cid:107) ˜
z t (cid:107) is monotone increasing.
Another key observation from Theorem 5 is the unboundedness of (cid:107) ˜
z t (cid:107) , i.e. (cid:107) ˜
z t (cid:107) → ∞ as t → ∞ .
Indeed we have the following lemma:
Lemma 6.
(Finite phase I)
Assuming λ 1 < 0 , suppose t ∗ is the index that (cid:107)
˜
z t ∗ (cid:107) < 1 and (cid:107) ˜
z t ∗ +1 (cid:107) ≥ 1 , then t ∗ is bounded by:
[Equation]
So after t ∗ iterations the algorithm will reach phase II.
Then we turn to manifold optimization method, which is discussed extensively in (Absil et al., 2009) and for the manifold theory we refer (Do Carmo & Flaherty Francis, 1992).
Here we list some concepts and its explicit form in our problem: denote M as the smooth manifold, and z ∈ M can be any point on manifold M .
The tangent space of z , denoted as T z , is the set of all tangent vectors to M at z .
When M = S n −then T z = { ξ ∈ R n : z (cid:124) ξ = 0 } ; Retraction is a mapping R z ( ξ ) : ξ ∈ T z (cid:55)→ M , in S n − 1 manifold, one of the commonly used retraction is R z ( ξ ) =
z + ξ (cid:107) z + ξ (cid:107) ; Projection onto T z , denoted as P z ξ is a mapping from R n to T z , for S n − 1 the projection is simply P z ( ξ )
=
(I d − zz (cid:124) ) ξ .
Based on above concepts, we can write the gradient descent on M as: z t +
=
R z t ( − α t grad f
( z t )) , and α t is tactically selected by line search such that f ( z t ) − f ( z t +1 ) ≥ σα t (cid:107) grad f ( x t ) (cid:107) , where σ ∈ (0 , 1) .
Because we already know at least one of the global minimum lies on S n − 1 , and according to Lemma 6 after at most t ∗ iterations we start to use manifold based optimization, i.e. the subproblem becomes:
[Equation]
Therefore, our method shrinks the search space from (cid:107) z (cid:107) ≤ 1 to (cid:107) z (cid:107) = 1 , by doing so we can apply well studied manifold optimization theory (Absil et al., 2009; Udriste, 1994) to our problem.
Indeed we have following theorems:
Theorem 7.
Let { z t } be an inﬁnite sequence of iterates generated by line search gradient descent, then every accumulation point of { z t } is a stationary point of the cost function f .
This above only guarantees convergence to stationary points, however, according to Theorem 3 if the step size α t is not too large, it actually converges to global minimum.
Then it remains to show a linear convergence rate, as guaranteed by the following theorem:
Theorem 8.
Let { z t } be an inﬁnite sequence of iterates generated by line search gradient descent, suppose it converges to s .
Let λ H, min and λ H, max be the smallest and largest eigenvalues of the
Hessian at s .
Assume that s is a local minimizer then λ H, min > 0 and given r in the interval ( r ∗ , 1) with r ∗ = 1 − min (cid:0) 2 σ ¯ αλ [Equation] , min
[Equation]
4 σ
[Equation]
[Equation]
(cid:1) , there exists an integer K such that:
[Equation]
for all t ≥ K .
Remarks : λ H, min and λ H, max are the minimum and maximum eigenvalue of Riemannian Hessian.
Speciﬁcally the Riemannian Hessian can be calculated by(Proposition 5.5.4, Absil et al.
[Equation]
By direct calculation(in appendix) we shall see: (cid:104)
Hess f ( s )[ ξ ] , ξ (cid:105) =
− s (cid:124)
Hs + ξ (cid:124)
Hξ − g (cid:124) s , where ξ ∈ Null ( s ) , (cid:107) ξ (cid:107)
= 1 .
By optimal condition (4), we have:
[Equation]
) so (cid:104)
Hess f ( s )[ ξ ] , ξ (cid:105) =
λ + ξ (cid:124)
Hξ ≥ λ
+ λ 1 ! > 0 .
Where ! > is guaranteed by gradient condition in (4):
[Equation]
in “easy-case”, g (1) (cid:54) = 0 , so λ 1 + λ (cid:54) = 0 and Hessian condition in (4) can be improved to λ 1 + λ > 0 .
Based on above discussion, we know λ H, min ≥ λ + λ 1 > 0 and λ H, max ≤ λ + λ n .
E XPERIMENTS
In this section, we examine the performance of our algorithm.
First of all, we use a random generated problem to check the dynamics of our proposed trust region subproblem solver, then we compare the performance of our trust region method with SGD on deep convolutional neural network.
After that, we focus on generalization ability of the solution that trust region method returns.
We include following algorithms in our experiments:
• SGD : We tune the step size to get the fastest, stable convergence. • TR : Trust region method with our proposed subproblem solver. For each subproblem, we choose N 1 = N 2 = 1 (deﬁned in Algorithm 2). The trust region radius is set to 1 and the step size of subproblem solver is set to 0 .
1 . • Hybrid :
Our proposed hybrid method, it runs trust
region method until the accuracy no longer increases, and then uses the last epoch to initialize SGD method.
When the training accuracy stabilizes, it returns the ﬁnal model.
Both TR and SGD here follow the same hyper-parameters above.
4. S OLVING THE T RUST R EGION S UBPROBLEM
We sample an indeﬁnite random matrix by H = BB (cid:124)
− λ I n , where B ∈ R n × ( n − 1) and B ij iid ∼
N (0 , 1) , obviously λ min ( H ) =
λ 1 = − λ .
Afterwards we sample a vector g by g i iid ∼ N (0 , 1) .
By changing the value of λ in { 10 , 30 , 50 , 70 , 90 , 110 } , we plot the function value decrement with respect to number of iterations in Figure 1(left).
As we can see, the iterates ﬁrst stay inside of the sphere (phase I) for a few iterations and then stay on the boundary (phase II).
To inspect how λ changes the duration of phase I, we then plot the number of iterations it takes to reach phase II, under different λ values shown in Figure 1(middle).
Recall in (5), number of iterations is bounded as a function of λ , which can be further simpliﬁed to:
[Equation]
where we set ˜ z (1) = 0 to simplify the formula.
By ﬁtting the data point with function T ( λ ) =
log(1+ c 1 λ ) log(1+ c 2 λ ) , we ﬁnd our bounds given by Lemma 6 is quite accurate.
4. O PTIMIZING DEEP CNN S
Next, we test the performance of our proposed trust region method (actually stochastic is used, denoted as TR) on training deep CNNs, including VGG (Simonyan & Zisserman, 2014) and MobileNets (Howard et al., 2017).
VGG/MobileNets are adopted to classify CIFAR10/STL10 data.
During all experiments, we only compare trust region method with plain SGD.
We did not include other SGD variants because (1) We want to focus on whether second order information is useful without considering the effect of momentum/acceleration.
(2) It is known that SGD generalize better than adaptive methods in large-batch (Wilson et al., 2017).
For reference, we also list the best accuracy obtained by Adam with small batch size in Table 1, which shows Adam’s accuracy is only slightly better than SGD but still worse than our hybrid method. [Figure] i v
[Figure] un c t i [Figure] We have some tricks to make it possible to apply trust region method to a deep convolutional network: First of all, rather than taking ReLU as the activation function, we use a similar yet everywhere continuously differentiable function, namely SReLU (see Figure 1(right)):
[Equation]
Although we can also deﬁne Hessian on ReLU function, it is not well supported on major platforms (Theano/PyTorch).
Likewise, we ﬁnd max-pooling is also not supported by platforms to calculate higher order derivative, one way to walk around is to change all the max-pooling layers to avgpooling—it hurts accuracy a little bit (we observe 1 ∼ 2% accuracy loss compared with (Keskar et al., 2016)), albeit this is not our primary concern.
Secondly, the oracle in TR method is Hessianvector product, or Hv .
Theoretically, the cost of Hv is comparable to gradient oracle because Hv = ∂∂w ( v (cid:124) ∂f∂w ) .
However, due to inefﬁcient implementation, in many deep learning frameworks the cost of Hv is ∼ 7x slower than gradient oracle, making running time comparison unfavorable, so we use the following numerical differentiation to replace chain-rule:
[Equation]
Experiments show the relative error is controllable ( (cid:107) (cid:99) Hv − Hv (cid:107) / (cid:107) Hv (cid:107) ≤ 1% ).
Nevertheless, the Hessian-vector operation is still a bottleneck to performance, thus if we care about running time then 1 ∼ 2 inner iteration(s) for each subproblem is more suitable.
Lastly, although in theory we need full gradient and full Hessian to guarantee convergence (in Xu et al.
(2017b)
they only need full gradient), calculating them in each iteration is not practical, so we calculate both Hessian and gradient on subsampled data to replace the whole dataset.
In the ﬁrst experiment, we compare the training/testing accuracy w.r.t.
running time and epoch on
VGG16+CIFAR10/STL10 dataset.
For SGD we choose step size η = 0 .
1 , which is the best step size in { 10 − n | n = 1 , 2 , . . . } .
For trust region subproblem we choose step size η (cid:48)
= 0 .
1 for all the experiments.
Both of them use batch size B = 512 (for CIFAR10) and B = 1024 (for STL10).
Our machine has 4 Titan Xp GPUs and Xeon E5-2620 CPU.
The experimental results are shown in
Figure 2.
Without surprising, the trust region method is ∼ 3x slower than SGD, partly because we use subsampled gradient/Hessian (currently there is no theoretical result to guarantee convergence in this situation) and Hv operation is too expensive.
Despite the slower training time, we observe
TR converges to a solution with better test error.
4. B ETTER G ENERALIZATION A BILITY IN L ARGE B ATCH T RAINING
Training neural network with larger batch size has become an important issue for faster training.
To fully exploit the computational power of multi-GPU systems, we need to increase the batch size, [Figure] at least proportional to number of cores.
However, large batch size can affect the generalization ability (Keskar et al., 2016) [Figure] so it would be interesting to see if this dilemma can be solved merely by choosing a different optimization method
[Figure]
To do so, we design an experiment that compares testing accuracy under different batch sizes in
Figure 3.
We observe that our method (TR) converges to solutions with much better test error when batch size is larger than 128 .
We postulate this is because SGD has not enough noise to escape from a sharp local minimum, especially when the large batch is used. For smaller batch size, TR method is not as good as SGD method, this is because the approximation of gradient and Hessian is not good enough by only a few samples.
However, our Hybrid method combines the beneﬁts of TR and SGD methods to make it both fast and accurate, for both small and large batch size. 4. L OSS L ANDSCAPE OF D IFFERENT S OLUTIONS To explore whether TR/Hybrid methods ﬁnd the wide local minimum, we follow the experiment techniques in (Keskar et al., 2016) to draw the loss curve that connects two models. Speciﬁcally, if we use w (cid:63) SGD , w (cid:63) TR and w (cid:63) Hybrid to denote models of SGD, trust region and hybrid method respectively, then the interpolated model in between is
[Equation]
The loss and accuracy can be evaluated on both training set and test set by parameters w (cid:63)
[Figure] and the result is displayed in Figure 4. We also calculate the (cid:96) 2 -distance between model A and B, shown in titles of sub-ﬁgures.
[Figure]
Figure 4 clearly shows the width and relative location of local minima: models of TR and hybrid methods lie in the same wide local minimum, while SGD ﬁnds a much sharper local minimum.
Moreover, we notice Dist(TR, SGD)
≈ Dist(Hybrid, SGD)
(cid:29) Dist(Hybrid, TR), implying that hybrid method is a reﬁnement of TR method (both of them are close to the exact wide local minimum, but hybrid method is even closer).
In contrast, SGD converges to a very different solution, where the loss curve is sharp and the model does not generalize as good as the other two.
4. P ROPERTIES OF L ARGE B ATCH SGD VS TR
By far, we can only conclude that trust region method generalize better than SGD, especially when batch size is large.
In what follows, we design a novel experiment that help us identifying the differences between those two methods.
Rather than running a homogeneous algorithm, we interleave
SGD and TR as two tracks, for simplicity we denote SGD( n ) as running n epochs SGD, and TR( n ) as running n epochs TR.
In the ﬁrst track, we run SGD(50) in the beginning, then we use the last iteration in SGD(50) to initialize TR(100), lastly based on the latest iterate, run SGD(50) again.
In the second track we change the order: Begin with TR(100), then SGD(50), and end with TR(50).
The purpose of switching between TR and SGD methods is to detect the character of respective solutions they ﬁnd.
Before running experiment we do such “thought experiment”: Imagine if large batch SGD converges to a sharp local minimum, and TR method can successfully escape it, then initialized by the solution that SGD ﬁnds, TR method will “climb over” the loss hill and down to a wide minimum, during this process the loss function (as well as accuracy) ﬁrst increases and then decreases.
On the contrary if TR method already ﬁnds a wide local minimum, then initialized by that solution SGD shall not escape to other minima, so we don’t expect a sharp change in either loss or accuracy.
This thought experiment is illustrated in Figure 5.
Our experimental results are presented in Table 1, where we run VGG16+CIFAR10 and choose batch size B = 2048 .
The overall running process is shown in the picture, and we further extract some important data into the table in order to see the differences more clearly.
We explain the results from following angles:
As expectation, there is a sharp raise in both train loss and test error at the 50th epoch in SGD(50)-TR(100)-SGD(50) track, meaning trust region method does escape from the solution that SGD(50) converges to.
Meanwhile both loss function and test error change smoothly at the 100th epoch in TR(100)-SGD(50)-TR(50) track, which is understandable because SGD can’t escape the basin of wide minimum obtained by TR.
[Figure]
[Figure] [Table]
method
[Table]
method accuracy method accuracy Track SGD(50) TR(100) 86.2%
SGD(50)
Track
TR(100) 85.6%
SGD(50)
TR(50) 86.5%
Adam
ADAM(100) 86.
[Table]
Warm-started by TR(100), SGD can reach the best testing accuracy, moreover, the accuracy merely change by enlarging batch size.
This sets the foundation of our hybrid method in
Figure 3, where the testing accuracy only drops 0 .
35% from B = 64 to B = 1024 , in comparison SGD testing accuracy drops 6 . 97% .
Due to subsampled Hessian and gradient, TR method can only ﬂuctuate around the minimizer, this causes a 1 . 6% drop in testing accuracy compared to the best result.
Another clue is the 1 . 0% drop in testing accuracy and raise of training loss at 150th epoch in TR(100)- SGD(50)-TR(50) track, when SGD(50) → TR(50) happens.
We want to emphasize that even though we ﬁxed the step size in SGD while in practice we should damp the step size after every few epochs, this won’t change the results above, since a smaller step size makes it even harder to escape from sharp minima.
In appendix we have supplementary experiments by using other networks combined with other datasets to show that these ﬁndings are not speciﬁc to one network.
D ISCUSSION
In this paper we ﬁrst show that a simple gradient based method can effectively ﬁnd the global minimum of trust region subproblem, even if it is nonconvex.
By examining the convergence rate as well as generalization ability, we ﬁnd our algorithm is comparable to SGD with respect to running time, but can converge to a solution with better generalization error.
We suggest to combine trust region with SGD to enjoy both fast and accurate properties.
As an important future direction, it would be interesting to see why trust region based algorithm can escape sharp local minima and try to establish convergence results on subsampled trust region method
A P ROOFS A. P ROOF OF P ROPOSITION
Proof.
For clarity, we repeat (4) in Lemma 1 here:
[Equation]
i)
If λ 1 > 0 then we always have s = −
( H + λ I d ) − 1 g , since (cid:107) s (cid:107) ≤ 1 , if (cid:107) H − 1 g (cid:107) >
1 then it must be that λ > 0 and (cid:107) s (cid:107) =
(cid:107) ( H + λ I d ) − 1 g (cid:107) = 1 . ii)
If λ 1 = 0 and g (1) (cid:54) = 0 , considering about λs (1) + g (1) =
we can infer λ (cid:54) = 0 and further (cid:107) s (cid:107) = 1 .
So in this case the only solution is s = −
( H + λ I d ) − 1 g where λ is the solution of (cid:107) ( H + λ I d ) − 1 g (cid:107) = 1 . iii)
If λ 1 = 0 and g (1) = 0 , in this case, either λ = 0 or s (1) = 0 .
If λ = 0 then Hs + g = 0 , this is equivalent to s ( i ) =
− g ( i )
λ
i for i ≥ 2 (suppose λ i > 0 since i ≥ 2 ).
On the other hand (cid:107) s (cid:107) ≤ 1 , this requires (cid:80) n i =2 ( g ( i ) λ i ) 2 ≤ 1 .
Otherwise λ (cid:54) =
and (cid:107) s (cid:107) = 1 , and λ is the solution of (cid:80) ni =
( g ( i ) λ i ) 2 = 1 , such λ > 0 must exists if (cid:80) ni
=2 ( g ( i ) λ i ) 2 > 1 .
iv)
If λ
<
0 and g (1) (cid:54) = 0 , then λ ≥ − λ 1 and (cid:107) s (cid:107) = 1 .
Because g (1) (cid:54) = 0 , which implies λ 1 > − λ 1 .
Immediately we know s = −
( H + λ I d ) − 1 g
and λ is the solution of (cid:80) ni =1 ( g ( i ) λ
i + λ ) 2 = 1 , v)
If λ 1 < 0 and g (1) = 0 .
In this condition, λ > 0 and (cid:107) s (cid:107)
=
By gradient condition, we see ( λ 1 + λ ) s (1) =
so either λ = − λ 1 or s (1) =
(or both).
This is determined by (cid:107) s (cid:107) =
s (1)2 + (cid:80) ni
=
( g ( i ) λ
+ λ i ) 2 = 1 , if (cid:80) ni
=2 ( g ( i ) λ
i − λ 1 ) 2 ≤ 1 then it is appropriate to set λ = − λ
and s (1) =
± (cid:113) 1 − (cid:80) ni
=
s ( i )2 .
Otherwise if (cid:80) ni
=2 ( g ( i ) λ
i − λ 1 ) 2 >
then it must be that s (1) =
and λ > − λ 1 such that (cid:80) ni
=2 ( g ( i ) λ
+ λ i ) 2 = 1 holds, one can see such equation has only one solution.
A. P ROOF OF L EMMA
Proof.
Suppose s is a stationary point, by formulating the Lagrange multiplier and using KKT condition, there exists a λ ≥ 0 such that ( H + λ I d ) s + g = 0 , and further by complementary slackness if (cid:107) s (cid:107) < 1 then ∇ f ( s ) =
Hs + g =
this leads to λ = 0 , otherwise (cid:107) s (cid:107) =
so in both cases the ﬁrst and second conditions in (4) are reached.
According to Lemma 1 if s is not global minimum then the third condition should be violated, implying λ 1 + λ < 0 , by gradient condition ( λ 1 + λ ) s (1) + g (1)
= 0 ,
because g (1) (cid:54) =
indicating s (1) (cid:54) = 0 , multiply both sides by s (1) we get s (1) g (1) > 0 , that is the case for stationary points excluding global minimum.
On the contrary for global minimum, we must have s (1) g (1) ≤ 0 .
A. P ROOF OF T HEOREM
Proof.
Notice the projection onto sphere will not change the sign of z ( i ) t +1 , so:
[Equation]
η t < 1 /λ n ensures 1 − ηλ i > 0 for all i ∈ [ n ] .
From Assumption 2 we know z ( i )
g ( i ) = − α g
( i )2 (cid:107) g (cid:107) ≤ 0 , so z ( i )
t g ( i ) ≤ 0 for all t .
A. P ROOF OF L EMMA
Proof.
Deﬁne w ( i ) t =
z ( i ) t / ( − ηg ( i ) ) , then by iteration rule:
[Equation]
solving this geometric series, we get:
[Equation]
[Equation]
suppose at t -th iteration, w ( i ) t ≥ w ( i ) t +1 which is equivalent to:
[Equation]
because from Assumption 2 we know w ( i ) 0 ≥ 0 , if w ( i ) 0 − 1 ηλ i ≤ 0 , i.e. 0 < ηλ i ≤ 1 /w ( i ) {+
+} then by (15) we know 1 − ηλ i ≥ 1 ⇔ ηλ i ≤ 0 leading to a contradiction, so w ( i )
− 1 ηλ i > 0 and 1 − ηλ i ≤ 1 .
On the other hand, λ j ≥ λ i for j ≥ i , so 1 − ηλ j ≤ 1 , together with w ( j ) 0 − 1 ηλ j = z ( i )
− 1 ηλ j ≥ w ( i ) 0 − 1 ηλ i >
we conclude:
[Equation]
For any t , suppose i ∗ ∈ [ n ] is the smallest coordinate index such that w ( i ∗ ) t ≥ w ( i ∗ ) t +1 , which implies w it < w it +1 for any i < i ∗ and w it ≥ w it
+1 for any i ≥ i ∗ (such a i ∗ may not exist, but it doesn’t matter).
By analyzing the sign of z t we know:
[Equation]
ﬁnally we have:
[Equation]
A. P ROOF OF T HEOREM
Proof.
First of all, notice (cid:107) ˜
z t +1 (cid:107) 2 = (cid:107) ˜
z t (cid:107) 2 − 2 η ˜ z
(cid:124) t ∇ f
(˜ z t ) + η 2 (cid:107)∇ f (˜ z t ) (cid:107) 2 , so it remains to show ˜
z (cid:124) t ∇ f
(˜ z t ) ≤ 0 .
We prove this by induction rule, suppose ˜ z (cid:124) t − 1 ∇ f
(˜ z t − 1 ) ≤ 0 and from ˜ z t = ˜
z t − 1 − η ∇ f
(˜ z t − 1 ) we know:
[Equation]
(cid:124)
[Equation]
(cid:125)
[Equation]
From Lemma 4 we know (1) ≥ β ˜ z t − 1 ∇ f
(˜ z t − 1 ) and recall β is the operator norm of A , we have (2) ≤ β (cid:107)∇ f (˜
z t − 1 ) (cid:107) 2 , so:
[Equation]
naturally, by choosing η < 1 /β we have ˜ z (cid:124) t ∇ f
(˜ z t ) ≤ 0 for all t , based on this observation (cid:107) ˜
z t +1 (cid:107) is monotone increasing.
A. P ROOF OF L EMMA
Proof.
This directly follows from:
[Equation]
together with (14) immediately comes to (5)
[Equation]
A. P ROOF OF T HEOREM 7,
See Theorem 4.3.1 and Theorem 4.5.6 in (Absil et al., 2009)
[Equation]
A. C ALCULATING λ H, min AND λ H, max
This is mainly brute force calculation.
By deﬁnition (7), we know for ξ ∈ T x M ,
[Equation]
we then expand f
( R x ( tξ )) to,
[Equation]
By differentiating t twice and set t =
(this can be done by software)
[Equation]
we ﬁnally get
[Equation]
[Equation]
Thus the eigenvalue of Riemannian Hessian comes from deﬁnition:
[Equation]
[Table]
B S UPPLEMENTARY E XPERIMENTS B. M OBILE N ETS +CIFAR
We apply MobileNets (Howard et al., 2017) to classify CIFAR10 data, MobileNets is a light-weight network that designed for mobile and embedded vision applications.
Here we still use η = 0 .
1 as the step size for SGD, for TR we use step size η (cid:48)
= 0 .
01 , the batch size is B = 2048 .
Other settings are the same with VGG16 experiment in the main text.
We ﬁnd the overall experiment outcome are quit similar to VGG16 network (see Figure 6)
[Table]
B. VGG16+STL
Now we change our dataset to STL10 but still using VGG16 model.
The parameter setting are the same with VGG16+CIFAR10.
We choose batch size B = 1024 , note that STL10 has onlytraining samples, so even the batch size is as small as 1k can have considerable generalization loss.
As can be seen in Figure 7 and Table 3.
[Figure]
[Figure]
Stage III method accuracy method accuracy method accuracy
Track
SGD(150) 62.4%
TR(300) 64.6%
SGD(50) 66.0%
Track
TR(300) 63.6%
SGD(50)
TR(150) 65.8%
[Figure]