D OUBLY A DAPTIVE S CALED A LGORITHM FOR M ACHINE L EARNING U SING 2 nd O RDER I NFORMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a novel adaptive optimization algorithm for large-scale machine learning problems.
Equipped with a low-cost estimate of local curvature and Lipschitz smoothness, our method dynamically adapts the search direction and step-size.
The search direction contains gradient information preconditioned by a well-scaled diagonal preconditioning matrix that captures the local curvature information.
Our methodology does not require the tedious task of learning rate tuning, as the learning rate is updated automatically without adding an extra hyperparameter.
We provide convergence guarantees on a comprehensive collection of optimization problems, including convex, strongly convex, and nonconvex problems, in both deterministic and stochastic regimes.
We also conduct an extensive empirical evaluation on standard machine learning problems, justifying our algorithm’s versatility and demonstrating its strong performance compared to other start-of-theart ﬁrst-order and second-order methods.
I NTRODUCTION
This paper presents an algorithm for solving empirical risk minimization problems of the form:
[Equation] [Equation]
where w is the model parameter/weight vector, { ( x i , y i ) } ni =1 are the training samples, and f i : R d → R is the loss function.
Usually, the number of training samples, n , and dimension, d , are large, and the loss function F is potentially nonconvex, making this minimization problem difﬁcult to solve.
In the past decades, signiﬁcant effort has been devoted to developing optimization algorithms for machine learning.
Due to easy implementation and low per-iteration cost, (stochastic) ﬁrst-order methods (Robbins & Monro, 1951; Duchi et al., 2011; Schmidt et al., 2017; Johnson & Zhang, 2013; Nguyen et al., 2017; 2019; Kingma & Ba, 2014; Jahani et al., 2021a; Recht et al., 2011) have become prevalent approaches for many machine learning applications.
However, these methods have several drawbacks: ( i ) they are highly sensitive to the choices of hyperparameters, especially learning rate; ( ii ) they suffer from ill-conditioning that often arises in large-scale machine learning; and ( iii ) they offer limited opportunities in distributed computing environments since these methods usually spend more time on “communication” instead of the true “computation.”
The main reasons for the aforementioned issues come from the fact that ﬁrst-order methods only use the gradient information for their updates.
On the other hand, going beyond ﬁrst-order methods, Newton-type and quasi-Newton methods (Nocedal & Wright, 2006; Dennis & Mor´e, 1977; Fletcher, 1987) are considered to be a strong family of optimizers due to their judicious use of the curvature information in order to scale the gradient.
By exploiting the curvature information of the objective function, these methods mitigate many of the issues inherent in ﬁrst-order methods.
In the deterministic regime, it is known that these methods are relatively insensitive to the choices of the hyperparameters, and they handle ill-conditioned problems with a fast convergence rate.
Clearly, this does not come for free, and these methods can have memory requirements up to O
( d 2 ) with computational complexity up to O ( d 3 ) (e.g., with a naive use of the Newton method).
There are, of course, efﬁcient ways to solve the Newton system with signiﬁcantly lower costs (e.g., see Nocedal & Wright (2006)).
Moreover, quasi-Newton methods require lower memory and computational complexities than Newton-type methods.
Recently, there has been shifted attention towards stochastic second-order (Roosta-Khorasani & Mahoney, 2018;
Byrd et al., 2011; Martens, 2010; Jahani et al., 2020a; Xu et al., 2017; Roosta et al., 2018; Yao et al., 2018) and quasi-Newton methods (Curtis, 2016; Berahas et al., 2016; Mokhtari & Ribeiro, 2015;
Jahani et al., 2021b; Berahas et al., 2019; Jahani et al., 2020b) in order to approximately capture the local curvature information.
These methods have shown good results for several machine learning tasks (Xu et al., 2020;
Berahas et al., 2020; Yao et al., 2019).
In some cases, however, due to the noise in the Hessian approximation, their performance is still on par with the ﬁrst-order variants.
One avenue for reducing the computational and memory requirements for capturing curvature information is to consider just the diagonal of the Hessian.
Since the Hessian diagonal can be represented as a vector, it is affordable to store its moving average, which is useful for reducing the impact of noise in the stochastic regime.
To exemplify this, AdaHessian Algorithm (Yao et al., 2020) uses Hutchinson’s method (Bekas et al., 2007) to approximate the Hessian diagonal, and it uses a second moment of the Hessian diagonal approximation for preconditioning the gradient.
AdaHessian achieves impressive results on a wide range of state-of-the-art tasks.
However, its preconditioning matrix approximates the Hessian diagonal only very approximately, suggesting that improvements are possible if one can better approximate the Hessian diagonal.
[Figure]
in Euclidean norm) between the true diagonal of matrix A and the diagonal approximation by AdaHessian, Hutchinson’s method, and OASIS ( random vectors sampled from Rademacher distribution, see Section
Moreover
In this paper, we propose the d O ubly A daptive
S caled algor I thm for machine learning using
S econd-order information ( OASIS ).
OASIS approximates the Hessian diagonal in an efﬁcient way, providing an estimate whose scale much more closely approximates the scale of the true
Hessian diagonal (see Figure 1).
Due to this improved scaling, the search direction in OASIS contains gradient information, in which the components are well-scaled by the novel preconditioning matrix.
Therefore, every gradient component in each dimension is adaptively scaled based on the approximated curvature for that dimension.
For this reason, there is no need to tune the learning rate, as it would be updated automatically based on a local approximation of the Lipschitz smoothness parameter (see Figure 2).
The well-scaled preconditioning matrix coupled with the adaptive learning rate results in a fully adaptive step for updating the parameters.
Here, we provide a brief summary of our main contributions:
[Figure]
• Novel Optimization Algorithm.
We propose OASIS as a fully adaptive method that preconditions the gradient information by a well-scaled Hessian diagonal approximation.
The gradient component in each dimension is adaptively scaled by the corresponding curvature approximation.
• Adaptive Learning Rate.
Our methodology does not require us to tune the learning rate, as it is updated automatically via an adaptive rule.
The rule approximates the Lipschitz smoothness parameter, and it updates the learning rate accordingly.
• Comprehensive Theoretical Analysis.
We derive convergence guarantees for OASIS with respect to different settings of learning rates, namely the case with adaptive learning rate for convex and strongly convex cases.
We also provide the convergence guarantees with respect to ﬁxed learning rate and line search for both strongly convex and nonconvex settings.
• Competitive Numerical Results.
We investigate the empirical performance of OASIS on a variety of standard machine learning tasks, including logistic regression, nonlinear least squares problems, and image classiﬁcation.
Our proposed method consistently shows competitive or superior performance in comparison to many ﬁrst- and second-order state-of-the-art methods.
Notation.
By considering the positive deﬁnite matrix D , we deﬁne the weighted Euclidean norm of vector x ∈ R d with (cid:107) x (cid:107) 2 D = x T Dx .
Its corresponding dual norm is shown as (cid:107) · (cid:107) ∗ D .
The operator (cid:12) is used as a component-wise product between two vectors.
Given a vector v , we represent the corresponding diagonal matrix of v with diag ( v ).
R ELATED W ORK
In this paper, we analyze algorithms with the generic iterate updates:
[Equation]
where ˆ D k is the preconditioning matrix, m k is either g k (the true gradient or the gradient approximation) or the ﬁrst moment of the gradient with momentum parameter β 1 or the bias corrected ﬁrst moment of the gradient, and η k is the learning rate.
The simple interpretation is that, in order to update the iterates, the vector m k would be rotated and scaled by the inverse of preconditioning matrix ˆ D k , and the transformed information would be considered as the search direction.
Due to limited space, here we consider only some of the related studies with a diagonal preconditioner.
For more general preconditioning, see Nocedal & Wright (2006).
Clearly, one of the beneﬁts of a well-deﬁned diagonal preconditioner is the easy calculation of its inverse.
There are many optimization algorithms that follow the update in (2).
A well-known method is stochastic gradient descent (SGD).The idea behind SGD is simple yet effective: the preconditioning matrix is set to be ˆ D k
= I d , for all k ≥ 0 .
There are variants of SGD with and without momentum.
The advantage of using momentum is to smooth the gradient (approximation) over the past iterations, and it can be useful in the noisy settings.
In order to converge to the stationary point(s), the learning rate in SGD needs to decay.
Therefore, there are many important hyperparameters that need to be tuned, e.g., learning rate, learning-rate decay, batch size, and momentum.
Among all of them, tuning the learning rate is particularly important and cumbersome since the learning rate in SGD is considered to be the same for all dimensions.
To address this issue, one idea is to use an adaptive diagonal preconditioning matrix, where its elements are based on the local information of the iterates.
One of the initial methods with a non-identity preconditioning matrix is Adagrad (Duchi et al., 2011).
In Adagrad, the momentum parameter is set to be zero ( m k = g k ), and the preconditioning matrix is deﬁned as: [Equation]
[Equation]
As is clear from the preconditioning matrix ˆ D k in (3), every gradient component is scaled with the accumulated information of all the past squared gradients.
It is advantageous in the sense that every component is scaled adaptively.
However, a signiﬁcant drawback of ˆ D k in (3) has to do with the progressive increase of its elements, which leads to rapid decrease of the learning rate.
To prevent Adagrad’s aggressive, monotonically decreasing learning rate, several approaches, including Adadelta (Zeiler, 2012) and RMSProp (Tieleman & Hinton, 2012), have been developed.
Speciﬁcally, in RMSProp, the momentum parameter β 1 is zero (or m k = g k ) and the preconditioning matrix is as follows:
[Equation]
where β 2 is the momentum parameter used in the preconditioning matrix.
As we can see from the difference between the preconditioning matrices in (3) and (4), in RMSProp an exponentially decaying average of squared gradients is used, which prevents rapid increase of preconditioning components in (3).
Another approach for computing the adaptive scaling for each parameter is Adam (Kingma & Ba, 2014).
Besides storing an exponentially decaying average of past squared gradients like Adadelta and
RMSprop, Adam also keeps ﬁrst moment estimate of gradient, similar to SGD with momentum.
In
Adam, the bias-corrected ﬁrst and second moment estimates, i.e., m k
and ˆ D k in (2), are as follows:
[Equation]
There have been many other ﬁrst-order methods with adaptive scaling (Loshchilov & Hutter, 2017;
Chaudhari et al., 2019; Loshchilov & Hutter, 2016; Shazeer & Stern, 2018).
The methods described so far have only used the information of the gradient for preconditioning m k in (2).
The main difference of second-order methods is to employ higher order information for scaling and rotating the m k in (2).
To be precise, besides the gradient information, the (approximated) curvature information of the objective function is also used.
As a textbook example, in Newton’s method ˆ D k = ∇ 2 F ( w k ) and m k = g k with η k = 1 .
Diagonal Approximation.
Recently, using methods from randomized numerical linear algebra, the
AdaHessian method was developed (Yao et al., 2020).
AdaHessian approximates the diagonal of the
Hessian, and it uses the second moment of the diagonal Hessian approximation as the preconditioner ˆ
D k in (2).
In AdaHessian, Hutchinson’s method 1 is used to approximate the Hessian diagonal as follows:
[Equation]
where z k is a random vector with Rademacher distribution.
Needless to say, 2 the oracle ∇ 2 F ( w k )
z k , or Hessian-vector product, can be efﬁciently calculated; in particular, for AdaHessian, it is computed with two back-propagation rounds without constructing the Hessian explicitly.
In a nutshell, the ﬁrst momentum for AdaHessian is the same as (5), and its second order momentum is:
[Equation]
The intuition behind AdaHessian is to have a larger step size for the dimensions with shallow loss surfaces and smaller step size for the dimensions with sharp loss surfaces.
The results provided by AdaHessian show its strength by using curvature information, in comparison to other adaptive ﬁrst-order methods, for a range of state-of-the-art problems in computer vision, natural language processing, and recommendation systems (Yao et al., 2020).
However, even for AdaHessian, the preconditioning matrix ˆ D k in (7) does not approximate the scale of the actual diagonal of the Hessian particularly well (see Figure 1).
One might hope that a better-scaled preconditioner would enable better use of curvature information.
This is one of the main focuses of this study.
Adaptive Learning Rate.
In all of the methods discussed previously, the learning rate η k in (2) is still a hyperparameter which needs to be manually tuned, and it is a critical and sensitive hyperparameter.
It is also necessary to tune the learning rate in methods that use approximation of curvature information (such as quasi-Newton methods like BFGS/LBFGS, and methods using diagonal Hessian approximation like AdaHessian).
The studies (Loizou et al., 2020; Vaswani et al., 2019; Chandra et al., 2019; Baydin et al., 2017; Malitsky & Mishchenko, 2020) have tackled the issue regarding tuning learning rate, and have developed methodologies with adaptive learning rate, η k , for ﬁrst-order methods.
Speciﬁcally, the work (Malitsky & Mishchenko, 2020) ﬁnds the learning rate by approximating the Lipschitz smoothness parameter in an affordable way without adding a tunable hyperparameter which is used for GD-type methods (with identity norm).
Extending the latter approach to the weighted-Euclidean norm is not straightforward.
In the next section, we describe how we can extend the work (Malitsky & Mishchenko, 2020) for the case with weighted Euclidean norm.
This is another main focus of this study.
In fact, while we focus on AdaHessian, any method with a positive-deﬁnite preconditioning matrix and bounded eigenvalues can beneﬁt from our approach.
OASIS
In this section, we present our proposed methodology.
First, we focus on the deterministic regime, and then we describe the stochastic variant of our method.
For a general symmetric matrix A, E [ z (cid:12) Az ] equals the diagonal of A (Bekas et al., 2007)
3. D ETERMINISTIC OASIS
Similar to the methods described in the previous section, our OASIS Algorithm generates iterates
Input: w 0 , η 0 , D 0 , θ 0 = + ∞ 1: w 1 = w 0 − η 0 ˆ D 0 − 1 ∇ F ( w 0 ) 2: for k = 1 , 2 , . . .
do 3:
Form D k via (8) and ˆ D k via (9) 4:
Update η k based on (10) 5:
Set w k +
=
w k − η k ˆ D k − 1 ∇ F ( w k ) 6:
Set θ k = η k η k −7: end for according to (2).
Motivated by AdaHessian, and by the fact that the loss surface curvature is different across different dimensions, we use the curvature information for preconditioning the gradient.
We now describe how the preconditioning matrix ˆ D k can be adaptively updated at each iteration as well as how to update the learning rate η k automatically for performing the step.
To capture the curvature information, we also use Hutchinson’s method and update the diagonal approximation as follows:
[Equation]
Before we proceed, we make a few more comments about the Hessian diagonal D k in (8).
As is clear from (8), a decaying exponential average of Hessian diagonal is used, which can be very useful in the noisy settings for smoothing out the Hessian noise over iterations.
Moreover, it approximates the scale of the Hessian diagonal with a satisfactory precision, unlike AdaHessian Algorithm (see
Figure 1).
More importantly, the modiﬁcation is simple yet very effective.
Similar simple and efﬁcient modiﬁcation happens in the evolution of adaptive ﬁrst-order methods (see Section 2).
Further, motivated by (Paternain et al., 2019; Jahani et al., 2021b), in order to ﬁnd a well-deﬁned preconditioning matrix ˆ D k , we truncate the elements of D k by a positive truncation value α .
To be more precise:
[Equation]
The goal of the truncation described above is to have a well-deﬁned preconditioning matrix that results in a descent search direction; note the parameter α is equivalent to (cid:15) in Adam and AdaHessian).
Next, we discuss the adaptive strategy for updating the learning rate η k in (2).
By extending the adaptive rule in (Malitsky & Mishchenko, 2020) and by deﬁning θ k :=
η k η k − 1 , ∀ k ≥ 1 , our learning rate needs to satisfy the inequalities: i) η 2 k ≤ (1 + θ k [Equation]
[Equation]
η 2 k − 1 , and ii) η
[Equation]
≤
[Equation]
(These inequalities come from the theoretical results.)
Thus, the learning rate can be adaptively calculated as follows:
[Equation]
As is clear from (10), it is only required to store the previous iterate with its corresponding gradient and the previous learning rate (a scalar) in order to update the learning rate.
Moreover, the learning rate in (10) is controlled by the gradient and curvature information.
As we will see later in the theoretical results, η k ≥ α 2 L where L is the Lipschitz smoothness parameter of the loss function.
It is noteworthy to highlight that due to usage of weighted-Euclidean norms the required analysis for the cases with adaptive learning rate is non-trivial (see Section 4).
Also, by setting β 2 = 1 , α = 1 , and
D
= I d , our OASIS algorithm covers the algorithm in (Malitsky & Mishchenko, 2020).
3. S TOCHASTIC OASIS
In every iteration of OASIS , as presented in the previous section, it is required to evaluate the gradient and Hessian-vector product on the whole training dataset.
However, these computations are prohibitive in the large-scale setting, i.e., when n and d are large.
To address this issue, we present a stochastic variant of OASIS (see Appendix B for details) that only considers a small mini-batch of training data in each iteration.
The Stochastic OASIS chooses sets I k , J k ⊂ [ n ] randomly and independently, and the new iterate is computed as: w k +1 =
w k − η k ˆ D k − 1 ∇ F I k ( w k ) , where ∇ F I k ( w k ) = 1 |I k | (cid:80) i ∈I k ∇ F i ( w k ) and ˆ D k is the truncated variant of D k =
β 2 D k − 1 + (1 − β 2 ) diag ( z k (cid:12) ∇ 2 F J k ( w k ) z k ) with ∇ 2 F J k ( w k )
=
|J k
| (cid:80) j ∈J k ∇ 2 F j ( w k ) .
3. W ARMSTARTING AND C OMPLEXITY OF OASIS
Both deterministic and stochastic variants of our methodology share the need to obtain an initial estimate of D 0 .
The importance of this is illustrated by the rule (10), which regulates the choice of η k , which is highly dependent on D k .
In order to have a better approximation of D 0 , we propose to sample some predeﬁned number of Hutchinson’s estimates before the training process.
The main overhead of our methodology is the Hessian-vector product used in Hutchinson’s method for approximating the Hessian diagonal.
With the current advanced hardware and packages, this computation is not a bottleneck anymore.
To be more speciﬁc, the Hessian-vector product can be efﬁciently calculated by two rounds of back-propagation.
We also present how to calculate the
Hessian-vector product efﬁciently for various well-known machine learning tasks in Appendix B.
T HEORETICAL A NALYSIS
In this section, we present our theoretical results for OASIS .
We show convergence guarantees for different settings of learning rates, i.e., ( i ) adaptive learning rate, ( ii ) ﬁxed learning rate, and ( iii ) with line search.
Before the main theorems are presented, we state the following assumptions and lemmas that are used throughout this section.
For brevity, we present the theoretical results using line search in Appendix A.
The proofs and other auxiliary lemmas are in Appendix A.
Assumption 4.1.
(Convex).
The function F is convex.
Assumption 4.2.
( L − smooth).
The gradients of F are L − Lipschitz continuous for all w ∈ R d , i.e., ∃ L > 0 such that ∀ w, w (cid:48)
∈ R d , F ( w ) ≤ F ( w (cid:48) )
+ (cid:104)∇ F ( w (cid:48)
[Equation]
w − w (cid:48)
[Equation]
−
[Equation]
(cid:107) 2 .
Assumption 4.3.
The function F is twice continuously differentiable.
Assumption 4.4.
( µ − strongly convex).
The function F is µ − strongly convex, i.e., there exists a constant µ > 0 such that ∀ w, w (cid:48)
∈ R d , F ( w ) ≥ F ( w (cid:48) )
+ (cid:104)∇ F ( w (cid:48) ) , w − w (cid:48) (cid:105) + µ 2 (cid:107) w − w (cid:48) (cid:107) 2 .
Here is a lemma regarding the bounds for Hutchinson’s approximation and the diagonal differences.
Lemma 4.5.
(Bound on change of D k ).
Suppose that Assumption 4.2 holds, then i) | ( v k ) i | ≤ Γ ≤ √ dL , where v k
= z k (cid:12) ∇ 2 F ( w k )
z k ; ii) ∃ δ ≤ 2(1 − β 2 )Γ such that ∀ k : (cid:107) D
k
+
− D k (cid:107) ∞
≤ δ.
4. A DAPTIVE L EARNING R ATE
Here, we present theoretical convergence results for the case with adaptive learning rate using (10).
Theorem 4.6.
Suppose that Assumptions 4.1, 4.2 and 4.3 hold.
Let { w k } be the iterates generated by Algorithm OASIS , then we have: F ( ˆ w k ) − F ∗ ≤ LCk + 2 L (1 − β 2 ) Γ Q k k , where [Equation] ( w
[Equation]
and Q k = (cid:80) ki = [Equation] 2 η i
[Equation]
Remark 4.7.
The following remarks are made regarding Theorem 4.6:
If β 2 = 1 , α = 1 , and D 0 = I d , the results in Theorem 4.6 completely match with (Malitsky & Mishchenko, 2020).
By considering an extra assumption as in (Reddi et al., 2019; Duchi et al., 2011) regarding bounded iterates, i.e., (cid:107) w k − w ∗ (cid:107) 2 ≤ B ∀ k ≥ 0 , one can easily show the convergence of OASIS to the neighborhood of optimal solution(s)
The following lemma provides the bounds for the adaptive learning rate for smooth and stronglyconvex loss functions.
The next theorem provides the linear convergence rate for the latter setting.
Lemma 4.8.
Suppose that Assumptions 4.2, 4.3, and 4.4 hold, then η k ∈ [ α 2 L , Γ2 µ ] .
Theorem 4.9.
Suppose that Assumptions 4.2, 4.3, and 4.4 hold, and let w ∗ be the unique solution for (1).
Let { w k } be the iterates generated by Algorithm 1.
Then, for all k ≥ 0 and β 2 ≥ max { 1 −
[Equation]
we have
[Equation]
Ψ[-k +
 = (cid:107) [Equation] + θ-]
[Equation]
where Ψ
[Equation]
[Equation]
4. F IXED L EARNING R ATE
Here, we provide theoretical results for ﬁxed learning rate for deterministic and stochastic regimes.
Remark 4.10.
For any k ≥ 0 , we have αI (cid:22)
ˆ D k (cid:22)
Γ I
where 0 < α ≤ Γ .
4.2. D ETERMINISTIC R EGIME
Strongly Convex.
The following theorem provides the linear convergence for the smooth and strongly-convex loss functions with ﬁxed learning rate.
Theorem 4.11.
Suppose that Assumptions 4.2, 4.3, and 4.4 hold, and let F ∗ =
F ( w ∗ ) , where w ∗ is the unique minimizer.
Let { w k } be the iterates generated by Algorithm 1, where 0 < η k = η ≤ α 2 L Γ , and w 0 is a starting point.
Then, for all k ≥
we have: F ( w k ) − F ∗ ≤ (1 − ηµ Γ ) k
[ F ( w 0 ) − F ∗ ] .
Nonconvex.
The following theorem provides the convergence to the stationary points for the nonconvex setting with ﬁxed learning rate.
Assumption 4.12.
The function F ( . ) is bounded below by a scalar ˆ F .
Theorem 4.13.
Suppose that Assumptions 4.2, 4.3, and 4.12 hold.
Let { w k } be the iterates generated by Algorithm , where 0 < η k = η ≤ α 2 L Γ , and w 0 is a starting point.
Then, for all T > 1 we have:
[Equation]
4.2. S TOCHASTIC R EGIME
Here, we use E I k
[ . ] to denote conditional expectation given w k , and E [ . ] to denote the full expectation over the full history.
The following standard assumptions as in (Bollapragada et al., 2019; Berahas et al., 2016) are considered for this section.
Assumption 4.14.
There exist a constant γ such that E I [ (cid:107)∇ F I ( w ) − ∇ F ( w ) (cid:107) 2 ] ≤ γ 2 .
Assumption 4.15.
There exist a constant σ 2 < ∞ such that E I [ (cid:107)∇ F I ( w [Equation]
Assumption 4.16.
∇ F I ( w ) is an unbiased estimator of the gradient, i.e., E I [ ∇ F I ( w )]
= ∇ F ( w ) .
Strongly Convex.
The following theorem presents the convergence to the neighborhood of the optimal solution for the smooth and strongly-convex case in the stochastic setting.
Theorem 4.17.
Suppose that Assumptions 4.2, 4.3, 4.4, 4.15 and 4.16 hold.
Let { w k } be the iterates generated by Algorithm 1 with η k = η ∈ (0 , α 2 µ Γ L 2 ) , then, for all k ≥ 0 ,
[Equation]
where c
= 2 ηµ Γ − 2 η 2 L 2 α∈ (0 , 1) .
Moreover, if η k = η ∈ (0 , α 2 µ 2Γ L 2 ) then E
[Equation]
Nonconvex.
The next theorem provides the convergence to the stationary point for the nonconvex loss functions in the stochastic regime.
Theorem 4.18.
Suppose that Assumptions 4.2, 4.3, 4.12, 4.14 and 4.16 hold.
Let { w k } be the iterates generated by Algorithm 1, where 0 < η k = η ≤ η 2 L Γ , and w 0 is the starting point.
Then, for all k ≥ 0 ,
[Equation]
The previous two theorems provide the convergence to the neighborhood of the stationary points.
One can easily use either a variance reduced gradient approximation or a decaying learning rate strategy to show the convergence to the stationary points (in expectation).
OASIS’s analyses are similar to those of limited-memory quasi-Newton approaches which depends on λ max and λ min (largest and smallest eigenvalues) of the preconditioning matrix (while in (S)GD λ max = λ min = 1 ).
These methods in theory are not better than GD-type methods.
In practice, however, they have shown their strength.
E MPIRICAL R ESULTS
In this section, we present empirical results for several machine learning problems to show that
our
OASIS methodology outperforms state-of-the-art ﬁrst- and second-order methods in both deterministic and stochastic regimes.
We considered: (1) deterministic (cid:96) 2 -regularized logistic regression (strongly convex); (2) deterministic nonlinear least squares (nonconvex), and we report results on 2 standard machine learning datasets rcv1 and ijcnn1 3 ; and (3) image classiﬁcation tasks on [Figure] [Figure]
MNIST , CIFAR10 , and CIFAR100 datasets on standard network structures.
In the interest of space, we report only a subset of the results in this section.
The rest can be found in Appendix C.
To be clear, we compared the empirical performance of OASIS with algorithms with diagonal preconditioners.
In the deterministic regime, we compared the performance of OASIS with AdGD (Malitsky & Mishchenko, 2020) and AdaHessian (Yao et al., 2020).
Further, for the stochastic regime, we provide experiments comparing SGD (Robbins & Monro, 1951), Adam (Kingma & Ba, 2014),
AdamW (Loshchilov & Hutter, 2017), and AdaHessian.
For the logistic regression problems, the regularization parameter was chosen from the set λ ∈ { 110 n , 1 n , 10 n } .
It is worth highlighting that we ran each method for each of the following experiments from 10 different random initial points.
Moreover, we separately tuned the hyperparameters for each algorithm, if needed.
See Appendix C for details.
The proposed OASIS is robust with respect to different choices of hyperparameters, and it has a narrow spectrum of changes (see Appendix C).
Logistic Regression.
We considered (cid:96) 2 -regularized logistic regression problems, F ( w ) = 1 n (cid:80) ni
=1 log(1 + e −
y
i
x Ti w ) + λ 2 (cid:107) w (cid:107) 2 .
Figure 3 shows the performance of the methods in terms of optimality gap and test accuracy versus number of effective passes (number of gradient and
Hessian-vector evaluations).
As is clear, the performance of OASIS (with adaptive learning rate and without any hyperparameter tuning) is on par or better than that of the other methods.
Non-linear Least Square.
We considered non-linear least squares problems (described in Xu et al.
F ( w ) = 1 n (cid:80) ni
=
( y i − 1 / (1 + e − x Ti w )) 2 .
Figure 4 shows that our OASIS Algorithm always outperforms the other methods in terms of training loss function and test accuracy.
Moreover, the behaviour of OASIS is robust with respect to the different initial points.
Image Classiﬁcation.
We illustrate the performance of OASIS on standard bench-marking neural network training tasks: MNIST , CIFAR10 , and CIFAR100 .
The results for MNIST and the details of the problems are given in Appendix C.
We present the results regarding CIFAR10 and CIFAR100 .
CIFAR10.
We use standard ResNet-20 and ResNet-32 (He et al., 2015) architectures for comparing the performance of OASIS with SGD, Adam, AdamW and AdaHessian 4 .
Speciﬁcally, we report 3 variants of OASIS : ( i ) adaptive learning rate, ( ii ) ﬁxed learning rate (without ﬁrst moment) and ( iii ) ﬁxed learning rate with gradient momentum tagged with “Adaptive LR,” “Fixed
LR,” and “Momentum,” respectively.
For settings with ﬁxed learning rate, no warmstarting is used in order to obtain an initial D 0 approximation.
For the case with adaptive learning case, we used the warmstarting strategy to approximate the initial Hessian diagonal.
More details regarding the exact parameter values and hyperparameter search can be found in the Appendix C.
The results on
CIFAR10 are shown in the Figure 5 (the left and middle columns) and Table 1.
As is clear, the simplest variant of OASIS with ﬁxed learning rate achieves signiﬁcantly better results, as compared to Adam, and a performance comparable to SGD.
For the variant with an added momentum, we get similar accuracy as AdaHessian, while getting better or the same loss values, highlighting the advantage of using different preconditioning schema.
Another important observation is that OASISAdaptive LR, without too much tuning efforts, has better performance than Adam with sensitive [Figure] hyperparameters.
All in all, the performance of OASIS variants is on par or better than the other state-of-the-art methods especially SGD.
As we see from Figure 5 (left and middle columns),
the lack of momentum produces a slow, noisy training curve in the initial stages of training, while OASIS with momentum works better than the other two variants in the early stages.
All three variants of OASIS get satisfactory results in the end of training.
More results and discussion regarding CIFAR10 dataset are in Appendix C.
[Table] [Table]
We use the hyperparameter settings obtained by training on CIFAR10 on
ResNet-20/32 to train CIFAR100 on ResNet-18 network structure.
5 We similarly compare the performance of our method and its variants with SGD, Adam, AdamW and AdaHessian.
The results are shown in Figure 5 (right column) and Table 2.
In this setting, without any hyperparameter tuning, fully adaptive version of our algorithm immediately produces results surpassing other state-of-the-art methods especially SGD.
F INAL R EMARKS
This paper presents a fully adaptive optimization algorithm for empirical risk minimization.
The search direction uses the gradient information, well-scaled with a novel Hessian diagonal approximation, which itself can be calculated and stored efﬁciently.
In addition, we do not need to tune the learning rate, which instead is automatically updated based on a low-cost approximation of the
Lipschitz smoothness parameter.
We provide comprehensive theoretical results covering standard optimization settings, including convex, strongly convex and nonconvex; and our empirical results highlight the efﬁciency of OASIS in large-scale machine learning problems.
Future research avenues include: (1) deriving the theoretical results for stochastic regime with adaptive learning rate; (2) employing variance reduction schemes in order to reduce further the noise in the gradient and Hessian diagonal estimates; and (3) providing a more extensive empirical investigation on other demanding machine learning problems such as those from natural language processing and recommendation system (such as those from the original AdaHessian paper (Yao et al., 2020)).
E THICS S TATEMENT
This work presents a new algorithm for training machine learning models.
We do not foresee any ethical concerns.
All datasets used in this work are from the public domain and are commonly used benchmarks in ML papers.
R EPRODUCIBILITY S TATEMENT
We uploaded all the codes used to make all the experiments presented in this paper.
We have used random seeds to ensure that one can start optimizing the ML models from the same initial starting point as was used in the experiments.
We have used only datasets that are in the public domain, and one can download them from the following website https://www.csie.ntu.edu.tw/ ˜cjlin/libsvmtools/datasets/ .
After acceptance, we will include a link to the GitHub repository where we will host the source codes.
A T HEORETICAL R ESULTS AND P ROOFS A. A SSUMPTIONS
Assumption 4.1.
(Convex).
The function F is convex, i.e., ∀ w, w
[Equation]
Assumption 4.2.
( L − smooth).
The gradients of F are L − Lipschitz continuous for all w ∈ R d , i.e., there exists a constant L > 0 such that ∀ w, w (cid:48)
∈ R d ,
[Equation]
Assumption 4.3.
The function F is twice continuously differentiable.
Assumption 4.4.
( µ − strongly convex).
The function F is µ − strongly convex, i.e., there exists a constant µ > 0 such that ∀ w, w (cid:48)
∈ R d
[Equation]
Assumption 4.12.
The function F ( . ) is bounded below by a scalar ˆ F .
Assumption 4.14.
There exist a constant γ such that E I [ (cid:107)∇ F I ( w ) − ∇ F ( w ) (cid:107) 2 ] ≤ γ 2 .
Assumption 4.16.
∇ F I ( w ) is an unbiased estimator of the gradient, i.e., E I [ ∇ F I ( w )]
= ∇ F ( w ) , where the samples I are drawn independently.
A.
P ROOF OF L EMMA 4.
Lemma 4.5.
(Bound on change of D k ).
Suppose that Assumption 4.2 holds, i.e., ∀ w : ∇ 2 f ( w ) (cid:22) LI , then
| ( v k ) i | ≤ Γ ≤ √ dL , where v k =
z k (cid:12) ∇ 2 F ( w k )
z k . 2.
there ∃ δ ≤ 2(1 − β 2 )Γ such that
[Equation]
Proof.
By Assumption 4.2, we have that (cid:107)∇ 2 F ( w ) (cid:107) 2 ≤ L and hence
[Equation]
which ﬁnishes the proof of case 1.
Now, from (8) we can derive
[Equation]
and hence
[Equation]
A. L EMMA R EGARDING S MOOTHNESS WITH W EIGHTED N ORM
In the following, we present a lemma for smoothness with weighted norm.
Theorem 2.1.5 in (Nesterov et al., 2018) provides the same analysis for any norm, and the following lemma can be seen as a special case of Theorem 2.1.5 (Nesterov et al., 2018) with respect to the weighted Euclidean norm.
In the following, we provide the proof for completeness.
Lemma A.1.
(Smoothness with norm D )
Suppose that Assumptions 4.1 and 4.2 hold, and D (cid:31) 0 , then we have:
[Equation]
17) where ˜ L =
L λ min ( D ) .
Proof.
By the equality ˜ L =
L λ min ( D )
[Equation]
we conclude that ˜ LD (cid:23) LI which results in:
[Equation]
Extending proof of Theorem 2.1.
[Equation]
in (Nesterov
[Equation]
we deﬁne φ ( y ) =
F ( y ) − (cid:104)∇ F ( x ) , y (cid:105) .
Then, clearly, x ∈ arg min φ ( y ) and (18) is still valid for φ ( y ) .
Therefore
[Equation]
Thus
[Equation]
Adding (19) with itself with x swapped with y we obtain
[Equation]
which implies that
[Equation]
A. P ROOF OF T HEOREM 4.
Lemma A.2.
Let f : R d
→ R be a convex function
[Equation]
and x ∗ is one of the optimal solutions for (1).
Then, for the sequence of { w k } generated by Algorithm 1 we have:
[Equation]
Proof.
We extend the proof in (Malitsky & Mishchenko, 2020).
We have
[Equation]
where the third equality comes from the OASIS ’s step and the inequality follows from convexity of
F ( w ) .
Now, let’s focus on (cid:107) w k +1 − w k (cid:107) 2ˆ D k .
We have
[Equation]
w
[Equation]
+
[Equation]
Now,
[Equation]
where the ﬁrst inequality comes from Cauchy-Schwarz and the third one follows Young’s inequality
[Equation]
Further,
[Equation]
where the ﬁrst two qualities are due the OASIS ’s update step.
The second term in the above equality can be upperbounded as follows:
[Equation]
where multiplier on the left is obtained via the bound on (cid:107) · (cid:107) ∞ norm of the diagonal matrix ˆ D k − 1 ˆ
[Equation]
α , which also represents a bound on the operator norm of the same matrix difference.
Next we can use the Young’s inequality and ∇ F ( w ∗ ) = 0 to get
[Equation]
Therefore
[Equation]
we have
[Equation]
Also,
[Equation]
Finally
[Equation]
we have
[Equation]
By simplifying the above inequality
[Equation]
we have:
[Equation]
Theorem
[Equation]
Suppose that Assumptions 4.1, 4.2 and 4.3 hold.
Let { w k } be the iterates generated by Algorithm 1, then we have:
[Equation]
where
[Equation]
Proof.
By telescoping inequality (21) in Lemma A.2 we have:
[Equation]
Moreover
[Equation]
by the rule for adaptive learning rule we know η i (1 + θ i ) − η i +
θ i +1 ≥ 0 , ∀ i .
Therefore
[Equation]
we have
[Equation]
By setting ˆ w = η k (1 + θ
[Equation]
) w
[Equation]
+
[Equation]
−
[Equation] [Equation]
θ
[Equation]
S
[Equation]
, where S k :
=
η k (1 + θ k ) + (cid:80) k − 1 i =1 ( η i (1 + θ i ) − η i +
θ i +1 ) , and by using Jensens’s inequality, we have:
[Equation]
By the fact that η k ≥ 1 2 L thus 1 S k ≤ 2 L k , we have
[Equation]
A. P ROOF OF L EMMA 4.
Lemma 4.8.
Suppose that Assumptions 4.2, 4.3 and 4.4 hold, then η k ∈ (cid:104) α 2 L, Γ 2 µ (cid:105) .
Proof.
By (17) and the point that ˜ L =
L λ min ( ˆ D k ) , we conclude that:
[Equation]
Now, in order to ﬁnd the upperbound for
[Equation]
, we use the following inequality which comes from Assumption 4.4:
[Equation]
By setting ˜ µ = µ λ max ( ˆ D k ) , we conclude that ˜ µ ˆ D k (cid:22) µI which results in:
[Equation] [Equation]
The above inequality results in:
[Equation]
Therefore
[Equation]
we obtain that
[Equation]
and therefore, by the update rule for η k
[Equation]
we conclude that η k ∈ (cid:104) α 2 L, Γ 2 µ (cid:105) .
A. P ROOF OF T HEOREM 4.
Lemma A.3. .
Suppose Assumptions 4.2, 4.3 and 4.4 hold and let w ∗ be the unique solution of (1).
Then for ( w k ) generated by Algorithm 1 we have:
[Equation]
Proof.
By the update rule in Algorithm 1 we have:
[Equation] [Equation]
where the inequality follows from convexity of F ( w ) .
By strong convexity of F ( . ) we have:
[Equation]
In the lights of the strongly convex inequality we can change (29) as follows
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the ﬁrst inequality is due to Cauchy–Schwarz inequality
[Equation]
and the second inequality comes from the choice of learning rate such that η k ≤
[Equation]
By plugging (33) into (32) we obtain:
[Equation]
Now
[Equation]
we can summarize that
[Equation] [Equation]
Next, let us bound (cid:104)∇ F ( w k − 1 ) , w k − w k +1 (cid:105) .
By the OASIS ’s update rule, w k +1 =
w k − η k ˆ D k − 1 ∇ F ( w k ) , we have,
[Equation]
The second term in the above equality can be bounded from above as follows:
[Equation]
where multiplier on the left is obtained via the bound on (cid:107) · (cid:107) ∞ norm of the diagonal matrix
[Equation]
which also represents a bound on the operator norm of the same matrix difference.
Next we can use the Young’s inequality and ∇ F ( w ∗ ) = 0 to get
[Equation]
By the inequalities (72), (36), (37) and (38)
[Equation]
and the deﬁnition θ k =
η k η k − 1 we have:
[Equation]
Theorem 4.9.
Suppose that Assumptions 4.2, 4.4, and 4.3 hold and let w ∗ be the unique solution for (1).
Let { w k } be the iterates generated by Algorithm 1.
Then, for all k ≥
we have: If β 2 ≥ max { 1 − α
[Equation]
where
[Equation]
Proof.
By Lemma A.3, we have:
[Equation]
Lemma 4.8 gives us η k ∈ (cid:104) α 2 L, Γ 2 µ (cid:105) , so we can choose large enough β 2 ∈ (0 , 1) such that
[Equation]
In other words
[Equation]
we have the following bound for β 2 :
[Equation]
However, we can push it one step further
[Equation]
by requiring a stricter inequality to hold
[Equation]
to get a recursion which can allows us to derive a linear convergence rate as follows
[Equation]
which requires a correspondingly stronger bound on β 2 :
[Equation]
Combining this with the condition for η k and deﬁnition for θ k
[Equation]
we obtain
[Equation]
and
[Equation]
where κ = Lµ .
Using it to supplement the main statement of the lemma
[Equation]
we get
[Equation]
Mirroring the result in (Malitsky & Mishchenko, 2020) we get a contraction in all terms
[Equation]
since for function value differences we also can show
[Equation]
A. P ROOF OF T HEOREM 4.
Theorem 4.11.
Suppose that Assumptions 4.2, 4.4, and 4.3 hold, and let F ∗ =
F ( w ∗ ) where w ∗ is the unique minimizer.
Let { w k } be the iterates generated by Algorithm 1, where 0 < η k = η ≤ α 2 L Γ , and w 0 is a starting point.
Then, for all k ≥
we have:
[Equation]
Proof.
By smoothness of F ( . ) we have:
[Equation]
(44) where the ﬁrst inequality comes from Assumption 4.2, and the second inequality is due to Remark 4.10, and ﬁnally, the last inequality is by the choice of η .
Since F ( . ) is strongly convex, we have 2 µ
[Equation]
F
[Equation]
w k )
[Equation]
F ∗ ) ≤ (cid:107)∇ F ( w k ) (cid:107) 2 , and therefore,
[Equation]
Also, we have:
[Equation]
A. P ROOF OF T HEOREM 4.
Theorem 4.13.
Suppose that Assumptions 4.3, 4.12 and 4.2 hold.
Let { w k } be the iterates generated by Algorithm 1, where 0 < η k = η ≤ α 2 L Γ , and w 0 is a starting point.
Then, for all T > 1 we have:
[Equation]
Proof.
By starting from (44), we have:
[Equation]
By summing both sides of the above inequality from k = 0 to T − 1 we have:
[Equation]
By simplifying the left-hand-side of the above inequality, we have
[Equation]
where the inequality is due to ˆ F ≤ F ( w T ) (Assumption 4.12).
Using the above, we have
[Equation]
A. P ROOF OF T HEOREM 4.
Lemma A.4 (see Lemma 1 (Nguyen et al., 2018) or Lemma 2.4 (Gower et al., 2019)) .
Assume that ∀ i the function F i ( w ) is convex and L-smooth.
Then, ∀ w ∈ R d the following hold:
[Equation]
Theorem A.5.
Suppose that Assumptions 4.2, 4.3, 4.4, 4.15 and 4.16 hold.
Let parameters α > 0 , η > 0 , β 2 ∈ (0 , 1] are chosen such that α < 2Γ Lµ , η ≤ α 2 L , β 2 > 1 −
[Equation]
) > 0 .
Let { w k } be the iterates generated by Algorithm 1, then, for all k ≥ 0 ,
[Equation]
49) where c = ηµα − (Γ − ηµ )2(1 − β 2 )
Γ Γ α ∈ (0 , 1) .
Remark A.6.
If we choose β 2 := 1 −
[Equation]
4Γ(Γ − ηµ ) > 0 then
[Equation]
Proof.
In order to bound (cid:107) r k +1 (cid:107) 2ˆ
D k +1 by (cid:107) r k +1 (cid:107) 2ˆ
D k we will use that 0 ≺
[Equation]
Then
[Equation]
Now, taking an expectation with respect to I k conditioned on the past
[Equation]
we obtain
E [ (cid:107) r k +1 (cid:107)
ˆ
D
k
+1 ]
[Equation]
where we used the fact that 2 L ηα − 1 ≤ 0 → η ≤ α 2 L .
In order to achieve a convergence we need to have
[Equation]
We have
[Equation]
Now
[Equation]
we need to choose β 2 such that
[Equation]
With such a choice we can conclude that
[Equation]
Theorem
[Equation]
Suppose that Assumptions 4.2, 4.3, 4.4, 4.15 and 4.16 hold.
Let { w k } be the iterates generated by Algorithm 1 with η k = η ∈ (0 , α 2 µ Γ L 2 ) , then, for all k ≥ 0 ,
[Equation] [Equation]
where c
= 2 ηµ Γ − 2 η 2 L 2 α∈
[Equation]
Moreover, if η k = η ∈ (0 , α 2 µ 2Γ L 2 ) then
[Equation]
Proof.
First, we will upper-bound the F ( w k +1 ) using smoothness of F ( w )
[Equation]
where the ﬁrst inequality is because of Assumptions 4.2 and 4.
[Equation]
and the second inequality is due to
Remark 4.10.
By taking the expectation over the sample I k , we have
[Equation]
Subtracting F ∗ from both sides
[Equation]
we obtain
[Equation]
By taking the total expectation over all batches I 0 , I 1 , I 2 ,... and all history starting with w 0 , we have
[Equation]
and the (53) is obtained.
Now, if η ≤ α 2 µ 2Γ L 2 then
[Equation]
and (54) follows.
A. P ROOF OF T HEOREM 4.
Theorem 4.18.
Suppose that Assumptions 4.3, 4.12, 4.2, 4.14 and 4.16 hold.
Let { w k } be the iterates generated by Algorithm 1, where 0 < η k = η ≤ η 2 L Γ , and w 0 is the starting point.
Then, for all k ≥ 0 ,
[Equation]
Proof.
We have that
[Equation]
where the ﬁrst inequality is because of Assumptions 4.2 and 4.
[Equation]
and the second inequality is due to
Remark 4.10.
By taking the expectation over the sample I k
[Equation]
we have
[Equation]
58) where the second inequality is due to Remark 4.10 and Assumption 4.14, and the third inequality is due to the choice of the step length.
By inequality (58) and taking the total expectation over all batches I 0 , I 1 , I 2 ,... and all history starting with w 0 , we have
[Equation]
By summing both sides of the above inequality from k = 0 to T − 1 ,
[Equation] [Equation]
By simplifying the left-hand-side of the above inequality, we have
[Equation]
where the inequality is due to ˆ F ≤ F ( w T ) (Assumption 4.12).
Using the above, we have
[Equation]
A. P ROOFS WITH L INE S EARCH
Given the current iterate w k , the steplength is chosen to satisfy the following sufﬁcient decrease condition
[Equation]
where p k = − ˆ D k − 1 ∇ F ( w k ) and c 1 ∈ (0 , 1) .
The mechanism works as follows.
Given an initial steplength (say η k = 1 ), the function is evaluated at the trial point w k + α k p k and condition (59) is checked.
If the trial point satisﬁes (59), then the step is accepted.
If the trial point does not satisfy (59), the steplength is reduced (e.g., η k = τη k for τ ∈ (0 , 1) ).
This process is repeated until a steplength that satisﬁes (59) is found.
[Table] [Table]
[Table]
) > F ( w k ) + c 1 η k ∇ F ( w k )
T p k do 4:
Set
η j +
= τη
j 5:
Set j = j +6: end while
Output: η k = η
j
By following from the study (Berahas et al., 2019), we have the following theorems.
A.11. D ETERMINITIC R EGIME - S TRONGLY C ONVEX
Theorem A.7.
Suppose that Assumptions 4.3 and 4.2 and 4.4 hold.
Let { w k } be the iterates generated by Algorithm 1, where η k is the maximum value in { τ − j : j = 0 , 1 , . . .
} satisfying (59) with 0 < c 1 < 1 , and w 0 is the starting point.
Then for all k ≥ 0 ,
[Equation]
Proof.
Starting with (43) we have
[Equation]
From the Armijo backtracking condition (59), we have
[Equation]
[Equation]
Looking at (60) and (61), it is clear that the Armijo condition is satisﬁed when
[Equation]
Thus, any η k that satisﬁes (62) is guaranteed to satisfy the Armijo condition (59).
Since we ﬁnd η k using a constant backtracking factor of τ < 1 , we have that
[Equation]
Therefore, from (60) and by (62) and (63) we have
[Equation]
By strong convexity, we have 2 µ ( F ( w ) − F (cid:63) ) ≤ (cid:107)∇ F ( w ) (cid:107) 2 , and thus
[Equation]
Subtracting F (cid:63) from both sides, and applying (65) recursively yields the desired result.
A.11. D ETERMINISTIC R EGIME - N ONCONVEX
Theorem A.8.
Suppose that Assumptions 4.3, 4.12 and 4.2 hold.
Let { w k } be the iterates generated by Algorithm 1, where η k is the maximum value in { τ − j : j = 0 , 1 , . . .
} satisfying (59) with 0 < c 1 < 1 , and where w 0 is the starting point.
Then,
[Equation]
and, moreover, for any T > 1 ,
[Equation]
Proof.
We start with
[Equation] [Equation]
Summing both sides of the above inequality from k = 0 to T − 1 ,
[Equation]
The left-hand-side of the above inequality is a telescopic sum and thus,
[Equation]
where the inequality is due to ˆ F ≤ F ( w T ) (Assumption 4.12)
[Equation]
Using the above
[Equation]
we have
[Equation]
Taking limits we obtain,
[Equation]
which implies (66).
Dividing (67) by T we conclude
[Equation]
A. P ROOFS WITH DECAYING STEP - SIZE
In this section, we provide the analysis with decaying stepsize for both strongly convex and nonconvex cases in the stochastic regime.
The proofs for the following theorems follows from the proofs in
Theorems 4.17 and 4.18, and Theorems 4.7 and 4.9 in Bottou et al.
A.12. S TOCHASTIC R EGIME – S TRONGLY C ONVEX
Theorem A.9.
Suppose that Assumptions 4.2, 4.3, 4.4, 4.15 and 4.16 hold., and let F (cid:63)
=
F ( w (cid:63) ) , where w (cid:63) is the minimizer of F . Let { w k } be the iterates generated by Algorithm 1, where η k is a sequence of stepsize such that, for all k ≥ 0 ,
[Equation]
for some φ > Γ µ and ζ > 0 such that η 0 ≤ α
[Equation]
µ 2Γ L(68)
Then, for all k ≥ 0 , the expected optimality gap satisﬁes
[Equation]
where
[Equation]
Proof.
By taking total expectation from (56), and replacing η with η k we have:
[Equation]
By the assumption η 0 ≤ α 2 µ 2Γ L 2 , and the way that η k is designed, i.e., in a decaying way, we have η k < η 0 ≤ α 2 µ 2Γ L 2 , therefore
[Equation]
Thus, we have:
[Equation]
We do the rest of the analysis of (69) with induction.
First, for k = 0 , (69) holds by the deﬁnition of ν .
Next, assuming (69) holds for k ≥ 0 , by (72) we have:
[Equation]
[Table]
[Equation] [Equation]
A.12. S TOCHASTIC R EGIME – N ONCONVEX
Theorem A.10.
Suppose that Assumptions 4.3, 4.12, 4.2, 4.14 and 4.16 hold.
Let { w k } be the iterates generated by Algorithm 1, where η k is a steplength sequence satisfying
[Equation]
and w 0 is the starting point.
Then, with A T := (cid:80)
[Equation]
E [
[Equation]
Proof.
The steplength sequence { η k } goes to zero due to the second condition in (74).
meaning that w.l.o.g., we may assume that η k ≤ 2 α 2 2Γ L for all k ≥ 0 .
By starting from and taking total expectation from (57), we have
[Equation]
Summing both sides of this inequality for k ∈ { 0 , . .
, T − 1 } gives:
[Equation]
Multiplying by 2Γ , and rearranging we get
[Equation]
The second condition in (74) guarantees that the right-hand side of the above inequality converges to a ﬁnite limit when T increases, therefore, it implies (75).
By the ﬁrst condition in (74), we conclude that A T → ∞ as T → ∞ , which results in (76).
A. C ONVERGENCE B OUNDS I NTERPRETATION
In this section, we provide the interpretation behind the convergence bounds in our theoretical results.
• In Theorem 4.6 there is the term (1 − β 2 )Γ Q k k , which essentially highlights that the parameter β 2 controls the size of the neighborhood of optimal solution(s).
In other words, if β 2 is very close to 1 (which is the case for our algorithm), the convergence neighborhood would be very small.
Meaning that, with the assumption regarding bounded iterates, our algorithm converges to a very small neighborhood of optimal solution as β 2 is very close to 1 (while in SGD the step-size controls the size of this neighborhood and accordingly, the step-size needs to go zero in order to converge to the small neighborhood of the optimal solution(s)).
Note this convergence guarantee is for smooth and convex case with adaptive learning rate .
• Theorem 4.9 provides a linear convergence rate for the smooth and strongly convex case with adaptive learning rate .
The analysis behind this theorem is based on the Lyapunov energy that decreases linearly.
• Remark 4.10 guarantees that the OASIS preconditioning matrix is positive deﬁnite and its eigenvalues are uniformly bounded above and below.
• Theorems 4.11 and 4.13 provides linear and sublinear rate to the stationary point(s) for deterministinc strongly convex and nonconvex cases, respectively.
The learning rate is ﬁxed in both mentioned cases, and the convergence rates are similar to those of limited-memory quasi-Newton approaches (e.g., L-BFGS).
• Theorems 4.17 and 4.18 provides linear and sublinear rate to the neighborhood of stationary point(s) (in expectation) for deterministinc strongly convex and nonconvex cases , respectively.
Please note that we provided another convergence guarantee for the stochastic strongly convex case (please see Theorem A.5).
Again, as the deterministic cases, the provided convergence guarantees completely match with those of the classical quasi-Newton methods (such as L-BFGS); the learning rate is ﬁxed in the latter cases.
Please note that these methods (such as OASIS and classical limited-memory quasi-Newton methods) in theory are not better than GD-type methods.
In practice, however, they have shown their strength.
• Theorems A.9 and A.10 provides convergence guarantees for the stochastic setting with decaying learning rate.
The mentioned theorems provides the convergence guarantees to the stationary point(s) (in expectation) for the stochastic strongly convex and nonconvex cases.
• Theorems A.7 and A.8 provides linear and sublinear convergence rates to the stationary point(s) for the deterministic strongly convex and nonconvex cases with utilizing linesearch.
B A DDITIONAL A LGORITHM D ETAILS B. R ELATED W ORK
As was mentioned in Section 2, we follow the generic iterate update:
[Equation]
Table 3 summarizes the methodologies discussed in Section 2.
[Table]
SGD (Robbins & Monro, 1951)
[Table]
[Equation]
[Table]
[Equation]
[Table]
Adagrad (Duchi et al., 2011)
g
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
and z
[Table]
∼ Rademacher
[Equation]
B. S TOCHASTIC OASIS
Here we describe a stochastic variant of OASIS in more detail.
As mentioned in Section 3, to estimate gradient and Hessian diagonal, the choices of sets I k , J k ⊂ [ n ] , are independent and correspond to only a fraction of data.
This change results in Algorithm 3.
Input: w 0 , η 0 , I k , D 0 , θ 0 = + ∞ , β 2 , α 1: w 1 = w 0 − η 0 ˆ D 0 − 1 ∇ F ( w 0 )
I k 2: for k = 1 , 2 , . . .
do 3:
Calculate D k =
β 2 D k − 1 + (1 − β 2 ) diag ( z k (cid:12) ∇ 2 F J k ( w k ) z k ) 4:
Calculate ˆ D k by setting ( ˆ D k ) i,i = max {| D k | i,i , α } , ∀ i ∈
[ d ] 5:
Update η k = min { (cid:112)
+ θ k − 1 η k − 1 , (cid:107) w k − w k − 1 (cid:107) ˆ
Dk 2 (cid:107)∇ F I k ( w k ) −∇ F I k
( w k − 1 ) (cid:107) ∗ ˆ Dk } 6:
Set w k +
=
w k − η k ˆ D k − 1 ∇ F I k ( w k ) 7:
Set θ k = η k η k −8: end for
Additionally, in order to compare the performance of our preconditioner schema independent of the adaptive learning-rate rule, we also consider variants of OASIS with ﬁxed η .
We explore two modiﬁcations, denoted as “Fixed LR” and “Momentum” in Section 5.
“Fixed LR” is obtained from Algorithm 3 by simply having a ﬁxed scalar η for all iterations, which results in Algorithm 4.
“Momentum” is obtained from “Fixed LR” by considering a simple form of ﬁrst-order momentum with a parameter β 1 , and this results in Algorithm 5.
In Section C.5, we show that OASIS is robust with respect to the different choices of learning rate, obtaining a narrow spectrum of changes.
Input: w 0 , η , I k , D 0 , β 2 , α 1: w 1 = w 0 − η ˆ D 0 − 1 ∇ F I k ( w 0 ) 2: for k = 1 , 2 , . . .
do 3:
Calculate D k =
β 2 D k − 1 + (1 − β 2 ) diag ( z k (cid:12) ∇ 2 F J k ( w k ) z k ) 4:
Calculate ˆ D k by setting ( ˆ D k ) i,i = max {| D k | i,i , α } , ∀ i ∈
[ d ] 5:
Set w k +
=
w k − η ˆ D k − 1 ∇ F I k ( w k ) 6: end for
Input: w 0 , η , I k , D 0 , β 1 , β 2 α 1: Set m 0 = ∇ F I k ( w 0 ) 2: w 1 = w 0 − η ˆ D 0 − 1 m3: for k = 1 , 2 , . . .
do 4:
Calculate D k =
β 2 D k − 1 + (1 − β 2 ) diag ( z k (cid:12) ∇ 2 F J k ( w k )
z k ) 5:
Calculate ˆ D k by setting ( ˆ D k ) i,i = max {| D k | i,i , α } , ∀ i ∈
[ d ] 6:
Calculate m k = β 1 m k − 1 + (1 − β 1 ) ∇ F I k ( w k ) 7:
Set w k +
=
w k − η ˆ D k − 1 m k 8: end for
In our experiments, we used a biased version of the algorithm with I k
= J k .
One of the beneﬁts of this choice is to compute the Hessian-vector product efﬁciently.
By reusing the computed gradient, the overhead of computing gradients with respect to different samples is signiﬁcantly reduced (see
Section B.3).
The ﬁnal remark is related to a strategy to obtain D 0 , which is required at the start of OASIS
Algorithm.
One option is to do warmstarting, i.e., spend some time before training in order to sample some number of Hutchinson’s estimates.
The second option is to use bias corrected rule for D k
[Equation]
which allows to obtain D 0 by deﬁning D − 1 to be a zero diagonal.
B. E FFICIENT H ESSIAN - VECTOR C OMPUTATION
In the OASIS Algorithm, similar to AdaHessian (Yao et al., 2020) methodology, the calculation of the Hessian-vector product in Hutchinson’s method is the main overhead.
In this section, we present how this product can be calculated efﬁciently.
First, lets focus on two popular machine learning problems: ( i ) logistic regression; and ( ii ) non-linear least squares problems.
Then, we show the efﬁcient calculation of this product in deep learning problems.
Logistic Regression.
One can note that we can show the (cid:96) 2 -regularized logistic regression as:
[Equation]
where 1 n is the vector of ones with size 1 × n , ∗ is the standard multiplication operator between two matrices, and X is the feature matrix and Y is the label matrix.
Further, the Hessian-vector product for logistic regression problems can be calculated as follows (for any vector v ∈ R d ):
[Equation]
The above calculation shows the order of computations in order to compute the Hessian-vector product without constructing the Hessian explicitly.
Non-linear Least Squares.
The non-linear least squares problems can be written as following:
[Equation]
The Hessian-vector product for the above problem can be efﬁciently calculated as:
[Equation]
Deep Learning.
In general, the Hessian-vector product can be efﬁciently calculated by:
[Equation]
As is clear from (83), two rounds of back-propagation are needed.
In fact, the round regarding the gradient evaluation is already calculated in the corresponding iteration; and thus, one extra round of back-propagation is needed in order to evaluate the Hessian-vector product.
This means that the extra cost for the above calculation is almost equivalent to one gradient evaluation.
[Table]
C D ETAILS OF E XPERIMENTS C. T ABLE OF A LGORITHMS
Table 4 summarizes the algorithms implemented in Section
[Table]
To display the optimality gap for logistic regression problems
[Table]
we used Trust Region (TR) Newton Conjugate Gradient (Newton-CG) method (Nocedal & Wright, 2006) to ﬁnd a w such that (cid:107)∇ F ( w ) (cid:107) 2 < 10 − 19 .
Hereafter, we denote F ( w ) − F ( w ∗ ) the optimality gap, where we refer w ∗ to the solution found by TR Newton-CG.
C. P ROBLEM D ETAILS
Some metrics for the image classiﬁcation problems are given in Table 5.
The number of parameters for ResNet architectures is particularly important, showcasing that OASIS is able to operate in very high-dimensional problems, contrary to the widespread belief that methods using second-order information are fundamentally limited by the dimensionality of the parameter space.
C. B INARY C LASSIFICATION
In Section 5 of the main paper
[Table]
we studied the empirical performance of OASIS on binary classiﬁcation problems in a deterministic setting, and compared it with AdGD and AdaHessian.
Here, we present the extended details on the experiment.
C.3. P ROBLEM AND D ATA
As a common practice for the empirical research of the optimization algorithms, LIBSVM datasetsare chosen for the exercise.
Speciﬁcally, we chose 5 popular binary class datasets, ijcnn1, rcv1, news20, covtype and real-sim .
Table 6 summarizes the basic statistics of the datasets.
[Table]
Let ( x i , y i ) be a training sample indexed by i ∈ [ n ] := { 1 , 2 , ..., n } , where x i ∈ R
d is a feature vector and y i ∈ {− 1 , +1 } is a label.
The loss functions are deﬁned in the forms
[Equation]
where (84) is a regularized logistic regression of a particular choice of λ >
(and we used λ ∈ { 0 .
1 /n, 1 /n, 10 /n } in the experiment), and hence a strongly convex function; and (85) is a non-linear least square loss, which is apparently non-convex.
The problem we aimed to solve is then deﬁned in the form
[Equation] [Equation]
and we denote w ∗ the global optimizer of (86) for logistic regression.
C.3. C ONFIGURATION OF A LGORITHM
To
better evaluate the performance of the algorithms, we conﬁgured OASIS , AdGD and AdaHessian with different choices of parameters.
• OASIS was conﬁgured with different values of β 2 and α , where β 2 can be any values in { 0 . 95 , 0 . 99 , 0 . 995 , 0 . 999 } and α can be any value in the set { 10 − 3 , 10 − 5 , 10 − 7 } (see Section C.5 where OASIS has a narrow spectrum of changes with respect to different values of α and β 2 ).
In addition, we adopted
a warmstarting approach to evaluate the diagonal of Hessian at the
starting point w 0 .
• AdGD was conﬁgured with 12 different values of the initial learning rate, i.e., η 0 ∈ { 10 − 11 , 10 − 10 , ..., 0 .
• AdaHessian was conﬁgured with different values of the ﬁxed learning rate: for logistic regression, we used 12 values between 0 . 1 and 5 ; for non-linear least square, we used 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 .and 2 . 0 .
To take into account the randomness of the performance, we used 10 distinct random seeds to initialize w 0 for each algorithm, dataset and problem.
C.3. E XTENDED E XPERIMENTAL R ESULTS
This section presents the extended results on our numerical experiments.
For the whole experiments in this paper, we ran each method 10 times starting from different initial points.
For logistic regression , we show the evolution of the optimality gap in Figure 6 and the ending gap in Figure 7; the evolution of the testing accuracy and the maximum achieved ones are shown in Figure 8 and Figure 9 respectively.
For non-linear least square , the evolution of the objective and its ending values are shown in
Figure 10; the evolution of the testing accuracy along with the maximum achived ones are shown in
Figure 11.
[Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
CIFAR
[Figure]
Weight Decay
[Figure]
C. I MAGE C LASSIFICATION
In the following sections
[Figure]
we provide the results on standard bench-marking neural network training tasks: CIFAR10 , CIFAR100 , and MNIST .
C.4. CIFAR
In our experiments, we compared the performance of the algorithm described in Table 4 in 2 settings - with and without weight decay.
The procedure for choosing parameter values differs slightly for these two cases, so they are presented separately.
Implementations of ResNet20/32 architectures are taken from AdaHessian repository.
Analogously to (Malitsky & Mishchenko, 2020), we also modify terms in the update formula for η k η k =
[Equation] [Equation]
+ θ k −
[Equation]
η
[Equation]
Speciﬁcally, we incorporate parameter γ into (cid:112) 1 + γθ k − 1 and use more optimistic bound of 1 /L
k instead of 1 / 2 L k , which results in a slightly modiﬁed rule
[Equation]
In order to ﬁnd the best value for γ , we include it into hyperparameter tuning procedure, ranging the values in the set { 1 , 0 . 1 , 0 .
01 } (see Section C.5 where OASIS has a narrow spectrum of changes with respect to different values of γ ).
Moreover, we used a learning-rate decaying strategy as considered in (Yao et al., 2020) to have the same and consistent settings.
In the aforementioned setting, η k would be decreased by a multiplier on some speciﬁc epochs common for every algorithms (the epochs 80 and 120).
[Figure]
Setting
[Table]
90.10 ± 0.
91.30 ± 0.
91.03 ± 0.
AdamW 91.99 ± 0.
90.25 ± 0.
92.58 ± 0.
91.10 ± 0.
AdaHessian 92.03 ± 0.
91.22 ± 0.
92.71 ± 0.
92.19 ± 0.
OASIS -Adaptive LR 91.20 ± 0.
91.19 ± 0.
92.61 ± 0.
91.97 ± 0.
OASIS -Fixed LR 91.96 ± 0.
89.94 ± 0.
93.01 ± 0.
90.88 ± 0.
OASIS -Momentum 92.01 ± 0.
90.23 ± 0.
92.77 ± 0.
91.11 ± 0.
[Table]
With weight decay : It is important to mention that due to variance in stochastic setting, the learning rate needs to decay (regardless of the learning rate rule is adaptive or ﬁxed) to get convergence.
One can apply adaptive learning rate rule without decaying by using variance-reduced methods (future work).
Experimental setup is generally taken to be similar to the one used in (Yao et al., 2020).
Parameter values for SGD, Adam, AdamW and AdaHessian are taken from the same source, meaning that learning rate is set to be 0 .
15 and, where they are used, β 1 and β 2 are set to be 0 .
9 and 0 .
For OASIS , due to a signiﬁcantly different form of the preconditioner aggregation, we conducted a small scale search for the best value of β 2 , choosing from the set { 0 . 999 , 0 . 99 , 0 .
95 } , which produced values of 0 .
98 for “Fixed LR,” “Momentum” and adaptive variants correspondingly for ResNet20 and 0 .
99 / 0 .for
“Fixed LR,” “Momentum” and adaptive variants correspondingly for ResNet32, while β 1 for momentum variant is still set to 0 .
Additionally, in the experiments with ﬁxed learning rate, it is tuned for each architecture, producing values 0 .
1 for ResNet20 and 0 .
1 for ResNet32 for “Fixed LR” and “Momentum” variants of OASIS correspondingly.
For the adaptive variant, γ is set to 0 .
for ResNet20/32.
α is set to 0 .
1 for all OASIS variants.
We train all methods for 160 epochs and for all methods.
With ﬁxed learning rate we employ identical scheduling, reducing step size by a factor of 10 at epochs 80 and 120.
For the adaptive variant scheduler analogue is implemented, multiplying step size by ρ at epochs 80 and 120, where ρ is set to be 0 .
5 for ResNet20/32.
Weight decay value for all optimizers with ﬁxed learning rate is 0 . 0005 and decoupling for OASIS is done similarly to AdamW and AdaHessian.
For adaptive OASIS weight decay is set to 0 .
001 without decoupling.
Batch size for all optimizers is 256 .
Without weight decay : In this setting we tune learning rate for SGD, Adam, AdamW and AdaHessian, obtaining the values 0 .
25 and 0 .
25 for ResNet20/32 accordingly.
Where relevant, β 1 , β 2 are taken to be 0 . 9 , 0 . 999 .
For OASIS we still try to tune β 2 , which produces values of 0 .
999 for ResNet20 and adaptive case of ResNet32 and 0 .
99 for “Fixed LR,” “Momentum” in the case of ResNet32; for the momentum variant β 1 = 0 .
Learning rates are 0 .
for ResNet20 and 0 .
1 for ResNet32 for “Fixed LR” and “Momentum” variants of
OASIS correspondingly.
For adaptive variant γ is set to 0 .
01 for ResNet20/32.
α is set to 0 .for
adaptive OASIS variant, while hyperoptimization showed, that for “Fixed LR” and “Momentum” a value of 0 .
01 can be used.
We train all methods for 160 epochs and for all methods with ﬁxed learning rate we employ identical scheduling, reducing step size by a factor of 10 at epochs 80 and 120.
For the adaptive variant scheduler analogue is implemented, multiplying step size by ρ at epochs 80 and 120, where ρ is set to be 0 .
1 for ResNet20/32.
Batch size for all optimizers is 256 .
Finally, all of the results presented are based on 10 runs with different seeds, where parameters are chosen amongst the best runs produced at the preceding tuning phase.
All available results can be seen in Table 7 and Figures 12 and 13.
We ran our experiments on an
NVIDIA V100 GPU.
C.4. CIFAR
Analogously to CIFAR10, we do experiments in settings with and without weight decay.
In both cases we took best learning rates from the corresponding CIFAR10 experiment with ResNet20 architecture without any additional tuning, which results in 0 .
1 for all.
β 2 is set to 0 .
99 in the case with weight decay and to 0 .
999 in the case without it.
ResNet18 architecture implementation is taken from a github repository.
8 We train all methods for 200 epochs.
For all optimizers learning rate is decreased (or effectively decreased in case of fully adaptive OASIS ) by a factor of 5 at epochs 60, 120 and 160.
All of the results presented are based off of 10 runs with different seeds.
All available results can be seen in Table 8 and Figure 14.
We ran our experiments on an NVIDIA V100 GPU.
C.4. MNIST
Similar to the previous results, we ran the methods with 10 different random seeds.
We considered
Net DNN which has 2 convolutional layers and 2 fully connected layers with ReLu non-linearity, mentioned in the Table 5.
In order to tune the hyperparameters for other algorithms, we considered the set of learning rates { 10 0 , 10 − 1 , 10 − 2 , 10 − 3 } .
For OASIS , we used the set of { 10 − 1 , 10 − 2 } and the
[Figure]
[Table]
OASIS -
[Table]
OASIS -
[Table]
OASIS -
[Table]
[Figure]
Net DNN on MNIST With Weight Decay
[Figure]
[Table]
set for truncation parameter α ∈ { 10 − 1 , 10 − 2 } .
As is clear from the results shown in Figure 15 and
Table 9, OASIS with momentum has the best performance in terms of training (lowest loss function), and and it is comparable with the best test accuracy for the cases with and without weight decay.
It is worth mentioning that OASIS with adaptive learning rate got satisfactory results with lower number of parameters, which require tuning, which is vividly important, especially in comparison to ﬁrst-order methods that are sensitive to the choice of learning rate.
We ran our experiments on a
Tesla K80 GPU.
C. S ENSITIVITY A NALYSIS OF OASIS
It is worth noting that we designed and analyzed OASIS for the deterministic setting with adaptive learning rate.
It is safe to state that no sensitive tuning is required in the deterministic setting (the learning rate is updated adaptively, and the performance of OASIS is completely robust even if the rest of the hyperparameters are not perfectly hand-tuned); see Figures 16 and 17.
[Figure] [Figure]
[Figure]
cc
[Figure]
c y
[Figure] [Figure]
We also provide sensitivity analysis for the image classiﬁcation tasks.
These problems are in stochastic setting, and we need to tune some of the hyperparameters in OASIS .
As is clear from the following ﬁgures, OASIS is robust with respect to different settings of hyperparameters, and the spectrum of changes is narrow enough; implying that even if the hyperparameters aren’t properly tuned, we still get acceptable results that are comparable to other state-of-the-art ﬁrst- and second-order approaches with less tuning efforts.
Figure 18 shows that OASIS is robust for different values of learning rate (left ﬁgure), while SGD is completely sensitive with respect to learning rate choices.
One possible reason for this is because OASIS uses well-scaled preconditioning, which scales each gradient component with regard to the local curvature at that dimension, whereas SGD treats all components equally.
Furthermore, in order to use an adaptive learning rate in the stochastic setting, an extra hyperparameter, γ , is used in OASIS (similar to (Malitsky & Mishchenko, 2020)).
Figure 19 shows that OASIS is also robust with respect to different values of γ (unlike the study in (Malitsky & Mishchenko, 2020)).
The aforementioned ﬁgures are for CIFAR10 dataset on the ResNet20 architecture; the same behaviour is observed for the other network architectures.
Finally, we show here that the performance of OASIS is also robust with respect to different values of β 2 .
Figure 20 shows the robustness of OASIS across a range of β 2 values.
This ﬁgure is for
CIFAR100 dataset on the ResNet18 architecture.
[Figure] [Figure]