A First Approach to Universal Second-Order Acceleration for Convex Minimization
Anonymous Author(s)
AffiliationAddress email
Abstract
This work proposes a universal and adaptive second-order method for minimizingsecond-order smooth, convex functions.
Our algorithm achieves O ( σ/ √ T ) convergence when the oracle feedback is stochastic with variance σ 2 , and improvesits convergence to O (1 /T 3 ) with deterministic oracles, where T is the number ofiterations.
Our method also interpolates these rates without knowing the nature ofthe oracle apriori, which is enabled by a parameter-free adaptive step-size that isoblivious to the knowledge of smoothness modulus, variance bounds and the diameter of the constrained set.
To our knowledge, this is the first universal algorithmwith such global guarantees within the second-order optimization literature
Introduction
Over the last few decades, first-order (convex) minimization methods have gained popularity formodern machine learning and optimization problems due to their efficient per-iteration cost andglobal convergence properties.
The literature on first-order methods is rather dense and extensivewith a concrete, thorough understanding of the optimal global convergence behavior.
Focusing on themore relevant settings of smooth, convex minimization, the lower bounds have been well-established;
O ( σ/ √ T ) when the gradient feedback is noisy with variance σ 2 , and O (1 /T 2 ) under deterministicfirst-order oracles [38, 43].
Under slight variations of the aforementioned problem setting, thereexists an extensive amount of work that enjoys the latter, “accelerated” rate [41, 42, 39, 50, 52, 29, 2,31, 51, 13, 12, 26, 22].
On the contrary to its first-order analogue, the literature on global convergence of second-order,smooth methods is notably sparse with many open questions standing even in the simplest problemformulations.
Following the pioneering works of Bennett [6], Kantorovich [24], Newton’s method andits variations [30, 33] are considered as the staple of second-order methods in optimization.
Althoughits powerful local convergence behavior has been repeatedly demonstrated [11, 28], studies on itsglobal behavior are relatively limited.
Prior attempts at tackling global convergence mostly makeadditional structural assumptions on the objective function [46, 34, 28] or assume extra regularityconditions on the Hessian [25] beyond the simplest smooth and convex setting.
Over the last decade,we have witnessed important progress towards a more complete theory of globally-convergent secondorder methods (more on this shortly), and yet there remains many important questions unanswered,which we will delve into in this paper.
To motivate the perspective in our technical endeavour, we take a small detour to introduce the idea ofuniversality , which we particularly characterize as adaptation to the level of noise in oracle feedback .
Enabled by the recent advances in online optimization, universal first-order algorithms essentially attain the O ( σ/ √ T + 1 /T 2 ) convergence, interpolating between stochastic and deterministic rates.
There exist a plethora of algorithms that enjoy this rate under different sets of assumptions [29, 26,16, 22] and we have a quite solid understanding of this optimal behavior.
However, we observe thatsuch universal results do not exist in second-order literature, hence, it is only natural to ask,
Can we design a simple second-order method that will achieveaccelerated universal rates beyond O ( σ/ √ T + 1 /T 2 ) ?
More recently, global sub-linear convergence rates for second-order methods have been characterizedby [44] for second-order smooth and convex setting.
Essentially, the so-called Cubic Regularized
Newton’s Method combines the quadratic Taylor approximation in the typical Newton update witha cubic regularization term.
At the expense of solving a cubic problem, this method achieves
O (1 /T 2 ) convergence rate.
Shortly after, Nesterov [40] proposes an accelerated version of thecubic regularization idea with O (1 /T 3 ) value convergence, pioneering a new direction of researchin the study of globally-convergent second-order methods [35].
This idea has been studied furtherfor different settings in convex optimization [19, 20] with the same accelerated O (1 /T 3 ) rate andextended to non-convex realm [8, 9], obtaining the analogous rates of O (1 /T 2 / 3 ) and O (1 /T 1 / 3 )for finding first-order and second-order stationary points, respectively, leading the way for furtherinvestigations [5, 15, 10].
Notice that accelerated cubic regularization is sub -optimal such that recent studies prove a respectivelower-bound for second-order smooth, convex problems as O (1 /T 7 / 2 ) [1, 4].
A different line ofresearch, which we will call “bisection-type” methods, shrinks the gap between the upper andlower bounds for achieving an almost -optimal (more on this shortly) convergence [45].
Pioneeredby Monteiro and Svaiter [36], these class of algorithms propose a conceptual method where thestep-size of the algorithm implicitly depends on the next iterate.
To resolve, the authors propose abisection procedure that simultaneously finds a step-size/next iterate pair that satisfies the conditionsof the iterative update, which enables the convergence rate of O (1 /T 7 / 2 ) , modulo the complexityof bisection procedure.
This idea was very recently generalized for higher-order tensor methods[17].
Not so surprisingly, the same construction finds application in variational inequality (VI) andmin-max optimization literature [7, 21].
Although there are promising results with an increasing interest into second-order –and also higherorder– methods, we identify three main shortcomings in the literature, which we will systematicallyaddress in the sequel.
First, bisection-type methods achieve the optimal convergence rate however,the search procedure is computationally very prohibitive [45, 32] and the resulting algorithms arecomplicated with many interconnected components.
Indeed, finding a computationally-efficient,optimal second-order method is still an open problem according to [45].
On the other hand, cubicregularization-based ideas propose a simple construction that achieves acceleration beyond O (1 /T 2 )however, similar to bisection methods, they either require the knowledge of smoothness constant orneed to execute a standard line-search procedure to estimate it locally.
A common drawback for bothapproaches is that the algorithmic constructions are designed for handling only deterministic oracleinformation and it is an open question whether such frameworks could accommodate stochastic firstand second-order information.
Our contributions: To address the aforementioned issues, we developed the first universal andadaptive second-order algorithm, E XTRA N EWTON , for convex minimization.
We summarize ourcontributions as follows:
We prove the global convergence rate of E XTRA N EWTON that interpolates simultaneously betweenO ( LD 2 /T 3 ) for deterministic and O ( σ/ √ T ) for stochastic oracles, achieving the first universalconvergence result in the literature.2.
Our method is completely oblivious to any problem-dependent parameters including smoothnessmodulus, variance bounds on stochastic oracles, diameter of the
AGD [41]
UniXGrad [26]
Reg.
Newton [35]
AcceleratedCubicReg.
ANPE[36]
ExtraNewton [ours]
Convergence Rate 1 Tσ √ T + 1 T1 T1 T
[Table]
[Table] + [Table] Bisection ✗ ✗ ✗ ✗ ✓ ✗
Adapts to L ✗ ✓ ✗
Partial ✗ ✓
Adapts to Oracle ✗ ✓ ✗ ✗ ✗ ✓ 3.
We design the first adaptive step-size, in the sense of [14, 48], that successfully incorporatessecond-order information “on-the-fly”.
While doing so, we bypass any bisection or linesearchprocedure, and propose a simple, intuitive algorithmic framework.
From a technical point of view, what will allow us to achieve these results is the combination ofthree principal ingredients: ( i) proposing appropriate adjustments to Extra-Gradient [27] that wasoriginally designed for solving variational inequalities and min/max problems; ( ii) an “optimistic”weighted iterate averaging scheme accompanied by an appropriate gradient rescaling strategy in thespirit of [51, 13, 26] which allows us to obtain an accelerated rate of convergence by means of ageneralized online-to-batch conversion (Theorem 3.3), and ( iii) the glue that holds these elementstogether is an adaptive learning rate inspired by [48, 26, 3] which automatically rescales aggregatedgradients and second order information.
In what follows, we shall explicate these arguments.
Problem setup
Throughout the sequel, we will be focusing on solving (constrained) convex minimization problemsof the general form:minimize f ( x ) , subject to x ∈ X .
(Opt)
Formally, in the above X is a convex and compact subset of a d - dimensional normed space V ∼ = R d ,and f : V → R ∪ { + ∞} is a proper, lower semi-continuous, convex function with dom f = { x ∈ 98 R d : f ( x ) <
+ ∞} ⊂ X .
To that end, we make the following blanket assumptions for (Opt).
First inorder to avoid trivialities we assume that the said problem admits at least a solution, i.e.
Assumption 2.1.
The solution set X ∗ = arg min x ∈X f ( x ) of (Opt) is non-empty.
Furthermore, we assume that there exists a Lipschitz continuous selection x (cid:55)→ ∇ 2 f ( x ) ∈ R d × d , i.e.,∥∇ 2 f ( x ) − ∇
2 f ( x ′ )
[Equation]
L
[Equation]
x
[Equation]
X (H-smooth) and in addition it satisfies the second order approximation:
[Equation]
Taylor)
To that end, combining (H-smooth) and (Taylor) we readily get the following inequality:
[Equation]
The above equivalences are well-established and hence we omit their proofs (we defer for a panoramicview to [53])
Oracle feedback structure From an algorithmic point of view, we aim to solve (Opt) by usingmethods that require access to a (stochastic) first and second order-oracle.
Before we move forwardwith the methodology,we shall introduce the definitions and notations for this oracle model which we will use in algorithm definitions and technical discussions.
For notational simplicity we denotestochastic gradients by g ( · ) and stochastic Hessian by H ( · ) such that
[Equation]
Due to space constraints, we will also define an operator that accommodates second-order informationand its respective stochastic counterpart.
[Equation]
where F is essentially the gradient (with respect to x ) of the second-order Taylor polynomial.
Bydefinition, the operator F satisfies the second-order smoothness property in Eq.
Method
In this section, we shall establish our universal second-order framework.
Our presentation evolvesaround three key components: choosing the appropriate algorithmic template with the key motivationsbehind it, solving implementability issues that commonly arise in higher-order methods and finallydesigning a universal algorithm that can handle deterministic and noisy oracle feedback simultaneously without having prior knowledge.
Our point of departure is the popular Extra-Gradient (EG)template; originally introduced by Korpelevich [27] and further developed in Nemirovski [37],
[Equation]
where Π X ( x ) =
arg min
z ∈X ∥ x − z ∥ 2 is the standard Euclidean projection onto the set X .
In termsof output, the candidate solution returned by (EG) after T iterations is the so-called “ergodic average”
[Equation]
Then, taking b t = γ t and assuming the method’s step-size γ t is chosen appropriately, ¯ X T enjoys thefollowing universal guarantee [23, 47]:
[Equation]
where σ signifies the effect of the noisy feedback.
However, as it becomes apparent, the vanilla(EG) template is not capable of matching the iconic 1 /T 2 for the smooth deterministic case.
It iswell-established in the literature of smooth, convex minimization that iterate averaging (or momentumin the sense of Nesterov [41]) is essential for matching the O (1 /T 2 ) lower bounds.
In fact, plainuniform averaging is not sufficient; one needs to introduce new iterates with increasing weights.
Precisely, this is equivalent to computing an average by taking b t
= O ( t ) .
However, we cannot fullycharacterize the acceleration machinery without what we like to call “gradient weighting”.
On top of(weighted) iterate averaging, gradients must be multiplied by the same order of weights to achieveacceleration, which is a recurring theme in the literature of accelerated and universal optimization[50, 52, 29, 2, 31, 51, 12, 26, 22].
Going back to discussion on (EG), Wang and Abernethy [51] and Kavis et al.
[26] provide usefulinsights into acceleration within the context of (EG).
Wang and Abernethy [51] identifies a 2-playergame with a particular structure called F ENCHEL G AME framework, which essentially reduces tominimizing a smooth, convex function when the players cooperate.
By introducing an “optimistic”weighted iterate averaging along with a complementary gradient weighting strategy, the frameworkrecovers different acceleration schemes of Nesterov [41, 42, 39].
On a related front, Diakonikolasand Orecchia [13] proposes the first acceleration of (EG) by appropriately integrating the optimisticaveraging idea [51] into the (EG) template as follows:
[Equation]
where b t =
O ( t ) is the “iterate averaging” parameter.
Later on, Kavis et al.
[26] designs an adaptive,universal variant of accelerated Mirror-Prox following the same optimistic averaging idea as in
Eq.
All in all, it is a recurring theme among accelerated algorithms to adopt weighted iterateaveraging ( b t =
O ( t ) ) with proportionate gradient weighting, and not so surprisingly, prior workestablishes clear connections between the degree of weighting and convergence rate.
Cutkosky [12]designs a black-box reduction that accelerated a class of online algorithms and proves that the rate ofconvergence of the reduction is O (1 / (cid:80) Tt =1 b t ) for b t ∈ [1 , t ] .
In retrospect, we aim at answering thefollowing question; What algorithmic construction would enable acceleration beyond O (1 /T 2 ) ?
3.1 Implicit algorithm
We give a first affirmative answer to the above question by presenting our implicit acceleratedalgorithm which is constructed upon (EG), and establish its convergence properties.
Note thatthe implicitness of the scheme serves as a gentle introduction to the actual explicit second orderacceleration, which shall follow.
Formally, our scheme is given via the following recursion:
[Equation]
(Implicit Second-Order) with ˜ X t and ¯ X t +1 / 2 being defined as in (6) and the adaptive step-size policy γ t defined as:
[Equation]
The implicit nature of (Implicit Second-Order) originates from X t +1 / 2 update (which we shall referto as (corrected) extrapolation step at times) since ¯ X t +1 / 2 depends upon X t +1 / 2 itself.
However, thisscheme exhibits several key differences from the vanilla (EG), which constitute the fundamental partsof our second-order acceleration machinery.
In particular, we have:
(i) integration of second-order updates for sharper extrapolation steps - first step of acceleration.
(ii) interplay between averaging ( b t ) and gradient weighting ( a t ) which allows more aggressiveaveraging - second step of acceleration.
(iii) adaptive step-size in the sense of Rakhlin and Sridharan [48] - key to adaptivity and universality.
Second-order updates: First, we will consider the particular interpretation of (EG) as an approximation to the Proximal Point method [49] which serves as motivation for the accommodation ofsecond-order information in our scheme.
[Equation] In particular, (EG) tries to approximate X t +1 by generating the extrapolated point X t +1 / 2 , and makeuse of the gradient at X t +1 / 2 to take a step from X t to X t +1 .
Therefore, if the algorithm is ableto compute a sharper estimate in the extrapolation step, it should be able live up to the fame of(PP) and display faster convergence.
To this end, we augment the extrapolation step by introducingsecond-order term.
Essentially, our algorithm makes use of second-order Taylor approximation , asopposed to first-order expansion, only for the extrapolation step, trading-off sharper approximationwith second-order information.
Iterate averaging and gradient weighting: Now, we turn our attention to the second componentin our acceleration machinery; averaging and weighting.
Recall that the acceleration framework of
Cutkosky [12] guarantees a value convergence rate of O (1 /t p +1 ) when weighting factor satisfiesb t
= O ( t p ) with p ∈ [0 , 1] .
We take this result one step beyond in two fronts; our algorithm exploitshigher-order smoothness in order to extend this bound for p ∈ [0 , 2] , implying the accelerated rateof O (1 /T 3 ) .
Second, we observe that previous work restricts the choice of gradient weights andaveraging weights by taking a t ≈ b t .
We decouple those weights by allowing the sequences a t andb t to be different , which in turn equips us with more aggressive iterate averaging when necessary.
Adaptive step-size: As the final component, we study the adaptive step-size (7) from the parameteradaptation perspective (i.e., adaptation to the Lipschitz modulus) and expand on its universal properties in the next section.
The vast literature on adaptive methods predominantly rely on constructionsof AdaGrad-like decreasing step-size policies by accumulating the observed gradient norms in itsdenominator.
The intuition behind this choice is that whenever the method approaches a solution, thevanishing gradients bring about stabilization, ensuring progress around the solution’s neighborhood.
However, this idea fails for (compactly) constrained problems; when the solution lies on the boundary.
So inspired by [48], we overcome this issue by accumulating ∥∇ f ( ¯ X t +1 / 2 ) − F ( ¯ X t +1 / 2 ; ˜ X t )
∥ 2which converges to 0 as ¯ X t +1 /
2 − ˜ X t → 0 ; which in turn implies convergence of the algorithm.
Toour knowledge, this is the first adaptive step-size that accommodates second order information.
Having established the core components of our design, we are in position to present the first accelerated convergence rate guarantee for (Implicit Second-Order).
Formally, this is given by thefollowing.
Theorem 3.1.
Let { X t +1 / 2 } Tt =1 be generated by (Implicit Second-Order) run with the adaptivestep-size policy (7) where a t = t 2 , b t = t 3 .
Assume that f satisfies (H-smooth) and denoteσ = max { σ g , σ H } .
Then, the following guarantee holds:
[Equation]
When γ = D , we obtain the converge rate O (cid:16) LD 2 T 3 (cid:17) .
Remark 3.1 .
We emphasize that the above rate does not require any prior knowledge of problemparamaters such as L , D , time-horizon T and any bounds on gradient/Hessian norms.
In order tohave better dependence on D one could set γ = D , and our rate of O (1 /T 3 ) holds irrespective of γ .
3.2 Explicit algorithm
Despite the fact that (Implicit Second-Order) improves upon the accelerated rate of O (1 /T 2 ) , onemay easily observe that it exhibits the following drawbacks:
(Implicit Second-Order) is a conceptual algorithm and therefore, not implementable in practice.2.
A fortiori, it cannot provide rate interpolation guarantees as it does not have the machinery tosimultaneously cope with deterministic and stochastic feedback.
As discussed earlier, a common strategy for overcoming this implicit construction is using abisection/line-search procedure [21, 36, 7].
Depending on the context, this procedure serves twodistinct purposes.
Primarily, it tackles the implicit nature of the update rule by simultaneously findinga pair of ( γ t , X t +1 / 2 ) and secondly, it enables adaptation to the second-order smoothness modulus.
However, one may identify major setbacks with these approaches; first, it is not clear how to handlestochastic oracles for executing the search procedure, so it is not capable of satisfying any universalguarantees.
Moreover, it yields a rather complicated procedure as a byproduct that has many movingparts.
To that end, we propose an alternative approach which not only yields a simple scheme, butalso provides a universal algorithm that is able to handle noisy feedback on-the-fly.
Without furtherado, we display our explicit algorithm, E XTRA N EWTON , with appropriate modifications.
Input: X 1 ∈ X , gradient weight { a t } Tt =1 , averaging weight { b t } Tt =1 , γ >1: for t = 1 to T do 2: X t
[Equation]
/ 2 = arg min x
[Equation]
⟨ a
[Equation]
x
[Equation]
B [Equation] X t + = arg min x ∈X
[Table]
Having established our main algorithmic template, Algorithm 1, we would like to provide a moredetailed description of its core components.
Universal step-size We modify our step-size (see Eq.
(8)) in order to operate in the stochasticregime while making it noise-adaptive for achieving rate interpolation.
Using the same optimisticweighted averaging scheme in Eq.
(6), we define the universal counterpart of the adaptive step-size,
[Equation]
Note that γ t is independent of any variable/randomness generated at iteration t ; it accumulatesa
[Equation]
g ( ¯ X s +1 / 2 ) − ˜F ( ¯ X s +1 / 2 ; ˜ X s ) ∥ 2 up to t − 1 .
Therefore, the step-size is decoupled from theexplicit update, a priori .
Then, what remains is a new algorithmic design that will retain theaccelerated convergence properties demonstrated by (Implicit Second-Order) while having an explicitconstruction that is capable of automatically adjusting to noise level in the oracle feedback.
Before expanding upon the technical details of our proof strategy, let us take our time to explain theconsequences of our explicit design in comparison to (Implicit Second-Order).
From implicit to explicit To obtain the explicit algorithm, ( i) we write the projection sub-problemin the arg min form; ( ii) replace X t +1 / 2 with the free variable x ; ( iii) introduce stochastic evaluationsinto the quadratic, second-order term; then, ( iv) simplify:
[Equation]
Given the bisection-type conceptual methods [36, 21, 7], it is surprising how smoothly we couldtransition from implicit to explicit once we decouple the step-size from the current iteration apriori .
Moreover, the resulting update rule for the extrapolation step retains the quadratic structure as the
X t +1 update rule.
Having analyzed the components of the explicit scheme, we will first present the universal convergencerates then provide a concise explanation of the proof strategy with particular emphasis on the principalcomponents of the analysis.
Theorem 3.2.
Let { X t +1 / 2 } Tt =1 be generated by Algorithm 1, run with the adaptive step-size policy(8) and a t = t 2 , b t = t 3 .
Assume that f satisfies (H-smooth) , and that Assumptions (2) hold withσ = max { σ g , σ H } .
Then, the following universal guarantee holds:
[Equation]
T
[Equation]
When γ = D , we obtain the target rate O (cid:16) σ √ T + LD 2 T 3 (cid:17) .
Remark 3.2 .
Similar to Theorem 3.1, E XTRA N EWTON achieves the preceding convergence rateindependent of the knowledge of problem parameters.
Compatible with the (EG)-based algorithmic design, our proof has the following main steps
We perform an offline regret analysis of Alg. 1 and show adaptive regret bounds - see Prop. 3.1.
We prove an anytime online-to-batch conversion framework, which generalizes that of Cutkosky
[12], through decoupling iterate averaging from gradient weighting - see Theorem 3.3.
Combining the adaptive regret bound with the conversion theorem immediately implies universal,
accelerated value convergence of O ( σ √ T + LD 2 T 3 ) - see Theorem 3.2.
Let us begin with clarifying what offline regret means for Algorithm 1.
We define the (linear) regretconsidering the convention in both online learning [48, 12] and first-order acceleration literature[51, 26, 22].
We measure the performance of our decisions for the extrapolation sequence such thatafter playing X t +1 / 2 , our algorithm observes and suffers the linear (weighted) loss with respect toa t ∇ f
( ¯ X t +1 / 2 ) .
Hence, we define the regret as
[Equation]
where we run the algorithm for T rounds.
Next up, we provide our generalized conversion result.
Theorem 3.3.
Let R EG T ( x ∗ ) denote the anytime regret for the decision sequence { X t +1 / 2 } Tt =1 as in(Reg) , and define two sequences of non-decreasing weights a t and b t such that a t , b t ≥ 1 .
As long asa
t /b t is ensured to be non-increasing,
[Equation]
Remark 3.3 .
This conversion result holds independent of the order of smoothness of the objectiveas long as f is convex.
Moreover, it allows averaging parameter b t to be asymptotically larger thangradient weights a t , enabling a more aggressive averaging strategy when necessary.
To complement the lower bound to the regret R EG T ( x ∗ ) , we will provide a template upper boundthat will help us explain how we exploit second-order smoothness for a more aggressive weighting,hence the rate O (1 /T 3 ) .
Proposition 3.1.
Let { X t +1 / 2 } Tt =1 be generated by Algorithm 1, run with a non-increasing step-sizesequence γ t and non-decreasing sequences of weights a t , b t ≥ 1 such that a t /b t is also nonincreasing.
Then, the following guarantee holds:
[Equation]
It is important to note that the template inequality in Proposition 3.1 is agnostic to the design of ouradaptive step-size as in Eq.
(8) as well as the selection of the weights as we described in Theorem 3.2;it holds for Algorithm 1 in the most general parameter selection.
It essentially applies to any nonincreasing sequence of step-sizes and non-decreasing gradient weight sequence a t ≥ 1 .
To obtain it,we neither used convexity nor the smoothness of the objective f .
In fact, the structure of the objectivefunction, i.e., its convexity, will not be needed for upper-bounding the regret expression, and we useit only for the conversion in Theorem 3.3.
Now, let us explain how we make use of second-order smoothness for enjoying faster rates, andgive a brief discussion of how the regret bound will look in its final form.
First, we decompose thestochastic term ∥ g ( ¯ X t +1 / 2 ) − ˜F ( ¯ X t +1 / 2 ; ˜ X t ) ∥ 2 into deterministic feedback and noise.
Then, weargue that the noisy component grows as O ( σT 5 / 2 ) where we define σ = max { σ g , σ H } for brevity.
On the other hand, achieving the accelerated O (1 /T 3 ) component of the universal rate amountsto showing that the regret has a constant, O (1) , component.
In the worst-case sense, however, thedeterministic component itself grows as O ( T 5 / 2 ) , too.
Fortunately, we identify that the negativeterm is “large enough” in magnitude to control the growth of the deterministic term, which permits aconstant component of order O ( LD 2 ) for the regret bound.
Although the regret bound of O ( LD 2 + σT 5 / 2 ) seems counter-intuitive from an online-learningperspective, it will make perfect sense when we discuss how second-order smoothness leads to “faster”conversion through more aggressive averaging.
As a matter of fact, we will conclude our discussionwith how second-order smoothness helps us accelerate.
It turns out that using (H-smooth), iterateaveraging as in Eq.(6) and compactness of X , we can bound the negative term as,
[Equation]
Observe that to seamlessly combine the positive and negative terms, our analysis enforces thata t
= O ( t 2 ) and b t
= Ω( t 2 ) .
Then, the conversion implies a convergence rate of R EG T ( x ∗ ) /T
3 ,hence the recipe for acceleration.
Therefore, the constant component of the regret amounts to
O (1 /T 3 ) convergence rate, while the stochastic component of the regret implies O ( σ/ √ T ) rate,giving us the first universal acceleration beyond first-order smoothness.
Experiments
[Figure] In this section, we will present practical performance of E XTRA N EWTON against a set of algorithms,e.g., G D , S GD , A DA G RAD [14], A CCELE G RAD [31], U NIX G RAD [26], N EWTON ’ S , for least squaresand logistic regression problems over a LIBSVM dataset, a1a .
Our main objective is to convey twomotivational ideas.
First, when the objective has a favorable structure as in least squares, second-ordermethod has cheap oracle costs (Hessian will be constant across the whole space) and display superiorconvergence behavior.
Second, we want to demonstrate the improved rates of our algorithm againstaccelerated and non-accelerated first-order methods through the ℓ 2 -regularized logistic regressionproblem.
In the plots, the x-axis denotes # of oracle calls which counts any gradient or Hessiancomputation as one oracle call.
[Figure]
When the problem is suitable, second-order methods show promising performance with truly superiorrun time.
In Figure 1a, we display the result for least squares setting.
Second-order methods areknown to be suitable for quadratic problems, and our method exploits its hybrid construction toconverge significantly faster than first-order methods, matching the behavior of N EWTON ’ S .
For the logistic regression problem, we regularize it with g ( x ) = 1 / 2 ∥ x ∥ 2 , but use a very smallregularization constant to render the problem ill-conditioned, making things slightly more difficultfor the algorithms [34, 35].
Although we implement N EWTON ’ S with line-search, we actuallyobserved a sporadic convergence behavior; when the initial point is close to the solution it convergessimilarly to E XTRA N EWTON , however when we initialize further away it doesn’t converge.
Thisnon-convergent behavior has been known for N EWTON ’ S , even with line-search present [18].
On thecontrary, E XTRA N EWTON consistently converges; even if we perturb the initial step-size and make itadversarially large, it manages to recover due to its adaptive step-size.
Conclusion
In this work, we present the first universal, second-order algorithm, E XTRA N EWTON , which enjoysthe value convergence rate of O ( σ/ √ T + 1 /T 3 ) .
To that end, an important open question is whetherwe could design a method that achieves an improved rate interpolation guarantee O ( σ/ √ T +1 /T 7 / 2 )without depending on any line-search/bisection mechanism.
We defer this to a future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2 andTheorem 3.1, 3.3, 3.2(b)
Did you include complete proofs of all theoretical results?
[Yes] See the Appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[Yes] We provide mean and standard deviation on the curves over multipleruns(d)
Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[N/A](b)
Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A](d)
Did you discuss whether
and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[-Appendix Table of Contents A Preface 
 B Notation and Definitions 
-]information or offensive content?
[-C Further Experimental Evaluation 
-]
[-Proposition 3.1) 
-]
[-G Implicit Accelerated Second-order Algorithm (Theorem 3.1) 
-]
[-The First Universal Second-order Accelerated Method (Theorem 3.2) 
-]
[N/A]
If you used crowdsourcing
or conducted research with human subjects...
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]