A DAPTIVE REGULARIZED CUBICS USING QUASIN EWTON APPROXIMATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Stochastic gradient descent and other ﬁrst-order variants, such as Adam and AdaGrad, are commonly used in the ﬁeld of deep learning due to their computational efﬁciency and low-storage memory requirements.
However, these methods do not exploit curvature information.
Consequently, iterates can converge to saddle points and poor local minima.
To avoid this, directions of negative curvature can be utilized, which requires computing the second-derivative matrix.
In Deep
Neural Networks (DNNs), the number of variables ( n ) can be of the order of tens of millions, making the Hessian impractical to store ( O ( n 2 ) ) and to invert ( O ( n 3 ) ).
Alternatively, quasi-Newton methods compute Hessian approximations that do not have the same computational requirements.
Quasi-Newton methods re-use previously computed iterates and gradients to compute a low-rank structured update.
The most widely used quasi-Newton update is the L-BFGS, which guarantees a positive semi-deﬁnite Hessian approximation, making it suitable in a line search setting.
However, the loss function in DNNs are non-convex, where the Hessian is potentially non-positive deﬁnite.
In this paper, we propose using a Limited-Memory Symmetric Rank-1 quasi-Newton approach which allows for indeﬁnite Hessian approximations, enabling directions of negative curvature to be exploited.
Furthermore, we use a modiﬁed Adaptive Regularized Cubics approach, which generates a sequence of cubic subproblems that have closed-form solutions.
We investigate the performance of our proposed method on autoencoders and feed-forward neural network models and compare our approach to state-of-the-art ﬁrst-order adaptive stochastic methods as well as L-BFGS.
I NTRODUCTION
Most deep learning problems involve minimization of the empirical risk of estimation
[Equation]
where Θ ∈ R n is the set of weights and f is some scalar-valued loss function.
To solve (1), various optimization approaches have been implemented, which we describe below.
Throughout this paper, we write f (Θ) and f ( x ; Θ) interchangeably.
Gradient and adaptive gradient methods are widely used for training deep neural networks (DNN) for their computational efﬁciency.
The most common approach is Stochastic Gradient Descent (SGD) which, despite its simplicity, performs well over a wide range of applications.
However, in a sparse training data setting, SGD performs poorly due to limited training speed (Luo et al.
To address this problem, adaptive methods such as AdaGrad (Duchi et al.
, AdaDelta (Zeiler (2012)), RMSProp (Hinton et al.
and Adam (Kingma & Ba (2014)) have been proposed.
These methods take the root mean square of the past gradients to inﬂuence the current step.
Amongst all of these adaptive methods, Adam is arguably the most widely used in a deep learning setting due to it rapid training speed.
Newton’s method has the potential to exploit curvature information from the second-order derivative (Hessian) matrix (see e.g., Gould et al.
Generally, the iterates are deﬁned by Θ k +1 =
Θ k − α k ∇ 2 f (Θ k ) − 1 ∇ f (Θ k ) , where α k > 0 is a steplength deﬁned by a linesearch criterion (Nocedal & Wright (2006)).
In a DNN setting, we know that the number of parameters ( n ) of the network can be of the order of millions.
Thus storing the Hessian which takes O ( n 2 ) memory, becomes impractical.
In addition, the inversion of the Hessian matrix, which takes O ( n 3 ) operations, is also impractical.
Even though Newton’s method achieves convergence in fewer steps, the method becomes computationally intractable to use on large-scale DNNs.
Quasi-Newton methods are alternatives to Newton methods.
They compute Hessian approximations, B k
+1 , that satisfy the secant condition given by y k
= B
k
+1 s k , where s k = Θ k +1 − Θ k and y k = ∇ f (Θ k +1 )
− ∇ f (Θ k ) .
The most commonly used quasi-Newton method, including in the realm of deep learning, is the limited-memory BFGS update, or L-BFGS (see e.g., Liu & Nocedal (1989)), where the Hessian approximation is given by
[Equation]
The generic L-BFGS quasi-Newton update scheme is described in Algorithm 1, and numerous variants of L-BFGS exist (see Goldfarb et al.
; Moritz et al.
; Gower et al.
One advantage of using an L-BFGS update is that the Hessian approximation can be guaranteed to be deﬁnite, which is highly suitable in line-search settings because the update s k is guaranteed to be a descent direction, meaning there is some step length along this direction that results in a decrease in the objective function (see Nocedal & Wright (2006), Algorithm 6.1).
However, because the L-BFGS update is positive deﬁnite, it does not readily detect directions of negative curvature for avoiding saddle points.
In contrast, the Symmetric-Rank One (SR1) quasi-Newton update is not guarateed to be positive deﬁnite and can result in ascent directions for line-search methods.
However, in trust-region settings where indeﬁnite Hessian approximations are an advantage because they capture directions of negative curvature, the limited-memory SR1 (L-SR1) has been shown to outperform L-BFGS in DNNs for classiﬁcation (see Erway et al.
We discuss this in more detail in Section 2 but in the context of Adaptive Regularization using Cubics.
Algorithm 1 L-BFGS Quasi-Newton Method with Line Search
Require:
Initial weights Θ 0 , batch size d , learning rate α , dataset D , loss function f (Θ) .
for k = 0 , 1 , 2 , . . .
do
Sample mini-batch of size d : D k ⊆ D Perform the forward backward pass over the current mini-batch
Compute the limited memory approximation B k using (2)
Compute step s k = α B − 1 k ∇ Θ f (Θ k ) , where α is the line-search step length end for
L-SR1 A DAPTIVE R EGULARIZATION USING C UBICS M ETHOD
We begin by discussing the L-SR1 update and the adaptive regularizion using cubics methods for large-scale optimization.
Unlike the BFGS update (2), which is a rank-two update, the SR1 update is a rank-one update, which is given by
[Equation]
(3) (see Khalfan et al.
As previously mentioned, B k +1 in (3) is not guaranteed to be deﬁnite.
However, it can be shown that the SR1 matrices can converge to the true Hessian (see Conn et al.
(1991) for details).
We note that the pair ( s k , y k ) is accepted only when | s (cid:62) k
( y k − B k s k ) | > ε (cid:107) y k − B k s k (cid:107) 22 , for some constant ε > 0 (see Nocedal & Wright (2006), Sec. 6.2, for details).
The SR1 update can be deﬁned recursively as
[Equation]
In limited-memory SR1 (L-SR1) settings, only the last m (cid:28) n pairs of ( s j , y j ) are stored and used.
If S k +1 =
[ s 0 s 1 · · · s k ] and Y k
+
=
[ y 0 y 1 · · · y k ] , then B k +1 admits a compact representation of the form
[Equation]
where Ψ
[Equation]
[-Y k +1  B 0 S k +
-]
[-+
-]
[-( D k +1 + L k +
 + L > k +1  S > k +1 B 
 S k +1 )  1 , (6) with-]where L k +1 is the strictly lower triangular part, U k +1 is the strictly upper triangular part, and D k + is the diagonal part of S (cid:62)
k +1 Y k
+
=
L k +
+ D
k
+
+ U k +
(see Byrd et al.
for further details).
Because of the compact representation of B k +1 , its partial eigendecomposition can be computed (see Burdakov et al.
In particular, if we compute the QR decomposition of Ψ k +1 , then we can write B k +
= B
= U (cid:107) ˆ Λ k +1 U (cid:62)(cid:107) , where U (cid:107) ∈ R n × ( k +1) has orthonormal columns and ˆ Λ ∈ R ( k +1) ×
( k +1) is a diagonal matrix.
If B 0 = δ k
I (see e.g., Lemma 2.4 in Erway et al.
where 0 < δ k < δ max is some scalar and I is the identity matrix, then we obtain the eigendecomposition B k +1 =
U
[-+
 ⇤-]k +1 Λ k +1 U (cid:62) k +1 , where U k +1 =
[ U (cid:107) U ⊥ ] , with U ⊥ ∈ R n × ( n − ( k +1)) and U (cid:62) k [-+
-]+1 U
k
+
= I .
Here, ( Λ k +1 )
i =
δ k
+ ˆ λ
i for i ≤ k + 1 , where ˆ λ i is the i th diagonal in ˆ Λ k +1 , and ( Λ ) i = δ k for i > k + 1 .
Since the SR1 Hessian approximation can be indeﬁnite, some safeguard must be implemented to ensure that the resulting search direction s k is a descent direction.
One such safeguard is to use a “regularization” term.
The Adaptive Regularization using Cubics (ARCs) method (Griewank (1981); Cartis et al.
(2011)) can be viewed as an alternative to line-search and trust-region methods.
At each iteration, an approximate global minimizer of a local (cubic) model,
[Equation]
is determined, where g k = ∇ f (Θ k ) , µ k > 0 is a regularization parameter, and Φ k is a function (norm) that regularizes s .
Typically, the Euclidean norm is used.
In this work, we propose an alternative “shape-changing” norm that allows us to solve each subproblem (7) exactly.
This shape-changing norm was proposed in Burdakov et al.
and it is based on the partial eigendecomposition of B k .
Speciﬁcally, if B k = U k Λ k U (cid:62) k is the eigendecomposition of B k , then we can deﬁne the norm (cid:107) s (cid:107) U
k def = (cid:107) U (cid:62) k s (cid:107) 3 .
Applying a change of basis with ¯ s = U (cid:62) k s and ¯ g k
= U (cid:62) k g k , we can redeﬁne the cubic subproblem as
[Equation]
With this change of basis, we can ﬁnd a closed-form solution of (8) easily.
The proposed Adaptive
Regularization using Cubics with L-SR1 (ARCSLSR1) algorithm is given in Algorithm 2.
The main contributions of this paper are as follows: 1. L-SR1 quasi-Newton methods: The most commonly used quasi-Newton approach is the L-BFGS method. In this work, we use the LSR1 update to better model potentially indeﬁnite Hessians of the non-convex loss function.
Adaptive Regularization using Cubics (ARCs): Given that the quasi-Newton approximation is allowed to be indeﬁnite, we use an Adaptive Regularized using Cubics approach to safeguard each search direction. Shape-changing regularizer: We use a shape-changing norm to deﬁne the cubic regularization term, which allows us to compute the closed form solution to the cubic subproblem (7). Computational complexity: Let m be the number of previous iterates and gradients stored in memory. The proposed LSR1 ARC approach is comparable to L-BFGS in terms of storage and compute complexity (see Table 1).
1: Given: Θ 0 , γ 2 ≥ γ 1 , 1 > η 2 ≥ η 1 > 0 , and σ 0 >2: for k = 0 , 1 , 2 . . . do 3:
Obtain S k
= [ s 0 · · · s k ] , Y k
=
[ y 0 · · · y k ] 4:
Solve the generalized eigenproblem S (cid:62) k Y k u
= ˆΛ S (cid:62) k S k u and let δ k = min { ˆ λ i } 5:
Compute Ψ k
= Y k − δ k S k 6:
Perform QR decomposition of Ψ = QR 7:
Compute the eigendecomposition RMR (cid:62) =
PΛP (cid:62) 8: Assign U (cid:107) = QP and U (cid:62)(cid:107)
=
P (cid:62) Q (cid:62) 9:
Deﬁne C (cid:107) = diag ( c 1 , . . .
, c m ) , where c i =
2 λ i + √ λ 2 i +4 µ | ¯ g i |
and ¯ g (cid:107) = U (cid:62)(cid:107) g 10:
Compute α ∗ = {+
 δ+} k + √ δ 2 k +4 µ (cid:107) g ⊥ (cid:107) where g ⊥ =
g − U (cid:107) ¯ g (cid:107) 11:
Compute step s ∗ = − α ∗ g
+ U (cid:107) ( α ∗ I m − C (cid:107) )
U (cid:62)(cid:107) 12: Compute m ( s ∗ ) and ρ k = ( f (Θ k ) − f
(Θ k +1 )) /m
( s ∗ ) 13:
Set Θ k +1 = (cid:26) Θ k + s k , if ρ k ≥ η 1 , Θ k , otherwise and µ k +1 =    0 . 5 µ k if ρ k > η 2 , 0 . 5 µ k (1 + γ 1 ) if η 1 ≤ ρ k ≤ η 2 , 0 .
5 µ k ( γ 1 + γ 2 ) otherwise 14: end for [Table]
[Equation]
2. I MPLEMENTATION
Because full gradient computation is very expensive to perform, we impement a stochastic version of the proposed ARCs-LSR1 method.
In particular, we use the batch gradient approximation
[Equation]
In deﬁning the SR1 matrix, we use the quasi-Newton pairs ( s k , ˜ y k ) , where ˜ y
k = ˜ g k +1 − ˜ g k (see e.g., Erway et al.
C ONVERGENCE ANALYSIS
In this section, we prove convergence properties of the proposed method (ARCs-LSR1 in Algorithm 2). The following theoretical guarantees follow the ideas from Cartis et al. and Benson & Shanno (2018).
First, we make the following mild assumptions: A1. The loss function f (Θ) is continuously differentiable, i
e., f ∈ C 1 ( R n ) . A2. The loss function f (Θ) is bounded below. Next, we prove that the matrix B k in (4) is bounded.
{+Lemma 
 The SR1 matrix B k +1 in (4) satsiﬁes (cid:107) B k +1 (cid:107) F ≤ κ B for all k ≥for some κ B > 0. Proof: Using the limited-memory SR1 update with memory parameter m in (4), we have+}
[Equation] Using a property of the Frobenius norm, namely, for real matrices A , (cid:107) A (cid:107) 2 F = trace ( AA (cid:62) ) , we have that (cid:107) ( y j − B j s j )
( y j − B j s j ) (cid:62) (cid:107) F = (cid:107) y j − B j s j (cid:107) 22 . Since the pair ( s j , y j ) is accepted only when | s (cid:62) j ( y j − B j s j ) | > ε (cid:107) y j − B j s j (cid:107) 22 , for some constant ε > 0 , and B 0 = δ k
I for some 0 < δ k < δ max , we have [Equation] Given the bound on (cid:107) B k +1 (cid:107) F , we obtain the following result, which is similar to Theorem 2.5 in
Cartis et al. Theorem 1 Under Assumptions A1 and A2 , if Lemma 1 holds, then [Equation] Finally, we consider the following assumption, which can be satisﬁed when the gradient, g (Θ) , is
Lipschitz continuous on Θ . A3. If { Θ t i } and { Θ l i } are subsequences of { Θ k } , then (cid:107) g t i − g l i (cid:107) → 0 whenever (cid:107) Θ t i − Θ l i (cid:107) →as i → ∞ .
If we further make Assumption A3 , we have the following stronger result (which is based on Corollary 2.6 in Cartis et al. Corollary 1 Under Assumptions A1 , A2 , and A3 , if Lemma 1 holds, then
[Equation]
E XPERIMENTS
To empirically compare the efﬁciency of the method against popular optimization methods like
SGD, ADAGRAD, ADAM, RMSProp and L-BFGS, we focus on two broad deep learning problems: image classiﬁcation and image reconstruction.
We choose these tasks due to their broad importance and availability of reproducible model architectures.
We run each experiments on an average oftimes with a random initialization in each experiment.
The number of parameters, convolutional layers and fully connected layers are mentioned in Table 3.
Dataset: We measure the classiﬁcation performance of each optimization method on 4 image datasets: MNIST (LeCun et al.
, FashionMNIST (Xiao et al.
(2017)), IRIS (Dua & Graff (2017)) and CIFAR10 (Krizhevsky et al.).
We have provided a comprehensive view of the experiments in Table
Dataset
[Table]
Hyperparameter tuning: We empirically ﬁne-tune the hyperparameters and select the best for each update scheme.
We have made a comprehensive list of all the learning rates for the gradient and adaptive gradient based algorithms in Table 4 in the Appendix.
The additional parameters are deﬁned as follows:
• ADAM:
We apply an (cid:15) perturbation of 1 . {+
 ×+} 10 − 6 .
{+β 
+} and β 1 are chosen to be 0.9 and 0.999, respectively.
• ADAGRAD:
The initial accumulator value is set to 0.
The perturbation (cid:15) is set to 1 . {+
 ×+} 10 − 10 .
• SGD: We use a momentum of 0 .
• RMSPROP:
We set α = 0 .
The perturbation (cid:15) is set 1 . {+
 ×+} 10 − 8 .
• L-BFGS: The table 4 in Appendix A presents the initial learning rate for the stochastic step in L-BFGS.
We set the default learning rate to 1 . 0 .
We choose a history size m of 10 and max iterations to 10 .
The tolerance on function value/parameter change is set to 1 . {+
 ×+} 10 − 9 and the ﬁrst-order optimality condition for termination is deﬁned as 1 . {+
 ×+} 10 − 9 .
• ARC-LSR1: We choose the same parameters as L-BFGS.
Network architecture: For each problem, we deﬁne the model architecture in Table 3 in the appendix
We deﬁne the process of the forward and backward pass of a DNN in Algorithm 3 in the
[-Given input ( x , y ), weights (and biases) ⇥ l and activations  l where l 2 [1 , L ] is the layer and L is the number of layers a 
 = x ; for l = 1 , . . . , L do h l = ⇥-]appendix.
[Table]
Testbed and software: All experiments were conducted using open-source software PyTorch (Paszke et al.
, SciPy (Virtanen et al.
and
NumPy (Harris et al.
We use an Intel Core i7-8700 CPU with a clock rate of 3.20 GHz and an NVIDIA RTX 2080 Ti graphics card.
R ESULTS
We have divided the sections into two categories: classiﬁcation and image reconstruction.
We present both the training results and the testing results for all methods.
5. C LASSIFICATION RESULTS
For each classiﬁcation problem, we deﬁne the network architecture, the corresponding hyperparameters (other than the learning rate) for each optimization scheme.
IRIS:
Since this dataset is relatively small, we assume a small network for our deep-learning model.
The model is described in 3.
We set the history size for the proposed approach and L-BFGS to 10 and the number of iterations to 10.
Figure 1 shows the comparative performance of all the methods.
Note that our proposed method (ARCLSR1) achieves the highest classiﬁcation accuracy in the fewest number of epochs.
MNIST: We trained the network for 20 epochs with a batch size of 256 images each.
We keep the same history size and number of iterations as the IRIS dataset for L-BFGS and the proposed ARCLSR1 approach.
For training, it can be seen in Figure 2 that nearly all methods achieve optimal training accuracy.
However, closely inspecting the testing curve, we notice that the proposed approach achieves higher accuracy than all the existing methods.
FMNIST: We train the network for 20 epochs with a batch size of 256 images.
We keep the history size the same as the IRIS and MNIST experiments for the proposed approach and L-BFGS.
For this method, the proposed ARCLSR1 approach is comparable to L-BFGS but outperforms the adaptive methods (see Figure 3).
CIFAR10:
We use the same parameters presented in Table 4 in the previous section for the adaptive methods.
For ARCLSR1 and L-BFGS, we have a history size of 100 with a maximum number of iterations of 100 and a batch size of 1024.
Figure 4(a) represents the training loss (cross-entropy
[Figure]
[Figure] [Figure]
loss).
Figure 4(b) represents the testing accuracy, i e., number of sample correctly predicted in the testing set.
To demonstrate the efﬁcacy of the proposed method on larger networks, additional experimentation on the ResNet50 architecture can be found in the appendix (Figure 8).
[Figure]
5. I MAGE R ECONSTRUCTION R ESULTS
The image reconstruction problem involves feeding a feedforward convolutional autoencoder model (with randomly initialized weights) a batch of the dataset.
It follows the same deep learning convention as mentioned in Algorithm 3 in Appendix A.
The loss function is deﬁned between the reconstructed image and the original image.
MNIST:
An image x ∈ R n is fed to the network, compressed into a latent space z ∈ R l , where l (cid:28) n , and reconstructed back to its original image size ¯ x ∈ R n .
We compute the mean-squared loss error between the reconstruction and the true image.
The weights are initialized randomly.
Each experiment has been conducted 5 times and we considered a batch size of 256 images each withepochs.
The results for the image reconstruction can be seen in Figure 5.
[Figure]
One can notice that the initial descent provided by the proposed approach provides a signiﬁcant decrease in the objective function.
To understand better, we provide the details of the results during the initial epoch (Figure 9(a)) and the ﬁnal epoch (Figure 9(b)).
We notice that the ARCLSR method has minimized efﬁciently in the ﬁrst half of the ﬁrst epoch.
This is empirical evidence that the method converges to the minimizer in fewer steps in comparison to the adaptive methods.
In
Figure 5 (b), we notice that all the adaptive methods eventually converge to the same point.
For training response results on the F-MNIST dataset, see Section B in the appendix.
[Figure]
5. T IME COMPLEXITY ANALYSIS
We understand that the proposed approach performs competitively against all existing methods.
We now analyze the time-constraints of each method.
We choose to clock the computationally demanding algorithm here - CIFAR10 classiﬁcation.
We chose a maximum iterations of 100 with a history size of 100 for L-BFGS and the ARCs LSR1, with a batch size of 1024 images.
Figureplots the time required by each of the methods to reach non-overtrained minima with a batch size of 1024 images.
As can be seen, the proposed approach is able to reach the desired minima in much less time than the rest of the algorithms.
L-BFGS ﬁnds it hard to converge due to a very noisy loss function and a small batch size, thus causing the algorithm to break.
Ozyildirim & Kiran (2020) argue that a large batch size is required for quasi-Newton methods to perform well.
However, the ARCLSR1 method performs well with a small batch size as well.
C ONCLUSION
In this paper, we proposed a novel quasi-Newton approach in a modiﬁed adaptive regularized cubic setting.
We were able to empirically and theoretically show how an L-SR1 quasi-Newton approximation in an ARCs setting was able to perform either better or comparably to most of the state of the art optimization schemes.
Even though the approach has yielded exceptional results, we need to test the method’s efﬁcacy when the network size and dataset size is large and when availability of data is sparse.