Approximate Secular Equations for the Cubic Regularization Subproblem
Anonymous Author(s)
AffiliationAddress
Abstract
The cubic regularization method (CR) is a popular algorithm for unconstrainednon-convex optimization.
At each iteration, CR solves a cubically regularizedquadratic problem, called the cubic regularization subproblem (CRS).
One wayto solve the CRS relies on solving the secular equation, whose computationalbottleneck lies in the computation of all eigenvalues of the Hessian matrix.
In thispaper, we propose and analyze a novel CRS solver based on an approximate secularequation, which requires only some of the Hessian eigenvalues and is thereforemuch more efficient.
Two approximate secular equations (ASEs) are developed.
For both ASEs, we first study the existence and uniqueness of their roots and thenestablish an upper bound on the gap between the root and that of the standardsecular equation.
Such an upper bound can in turn be used to bound the distancefrom the approximate CRS solution based ASEs to the true CRS solution, thusoffering a theoretical guarantee for our CRS solver.
A desirable feature of our
CRS solver is that it requires only matrix-vector multiplication but not matrixinversion, which makes it particularly suitable for high-dimensional applicationsof unconstrained non-convex optimization, such as low-rank recovery and deeplearning.
Numerical experiments with synthetic and real data-sets are conducted toinvestigate the practical performance of the proposed CRS solver.
Experimentalresults show that the proposed solver outperforms two state-of-the-art methods.
Introduction
The cubic regularization method (CR) is a variant of Newton’s method proposed by Griewank [8], andlater independently by Nesterov and Polyak [10] and Weiser et al.
It gained significant attentionover the last decade due to its attractive theoretical properties, such as convergence to second-ordercritical points[10] and quadratic convergence rate under mild assumptions [15].
Each iteration of CRsolves a problem of the following form, called the cubic regularization subproblem (CRS):
[Equation]
where ρ > 0 is the regularization parameter, b ∈ R n , and A ∈ R n × n is a symmetric matrix, notnecessarily positive semidefinite.
Many variants and generalizations of CR are developed, includingthe Adaptive Regularization Using Cubics (ARC) which allows for a dynamic choice of ρ and inexact
CRS solutions [3, 4], accelerated CR using momentum [13] and stochastic CR for solving stochasticoptimization [12].
Despite the theoretical success, the practicality of CR and its variants reliescritically on the efficiency of the CRS solver.
The goal of this paper is to develop a novel, efficient
CRS solver along with theoretical guarantees.
A popular approach for solving the CRS is via solving the so-called secular equation.
We now review this approach.
Towards that, we denote by λ 1 ≤ · · · ≤ λ n the eigenvalues of A and by v 1 , · · · , v n the corresponding eigenvectors.
In other words, we have the eigendecomposition
A = (cid:80) ni
=
λ i v i v Ti = VΛV T , where Λ = diag ( λ 1 , . . .
λ n ) and V =
[ v 1 , · · · , v n ] .
Note thateigenvalues λ
i are not necessarily positive due to the indefiniteness of the matrix A .
Also, we denotethe Euclidean norm by ∥ · ∥ .
Proposition 1 ([10, 3]) .
A vector x ∗ solves the CRS (1)
if and only if it satisfies the system(cid:26)
[Equation]
Moreover, if A + ρ ∥ x ∗ ∥ I ≻ 0 , then x ∗ is the unique solution (and hence a critical point).
Proposition 2 ([1]) .
Let x ∗ be a global solution of CRS (1) and the eigendecomposition for A = 41 (cid:80) ni
=
λ i v i v Ti = VΛV T , where Λ = diag ( λ 1 , . . .
λ n ) and V =
[ v 1 , · · · , v n ] .
If b T v 1 ̸ = 0 , then
A + ρ ∥ x ∗ ∥ I ≻ 0 and the solution x ∗ is the unique critical point (and hence the unique solution).
Conversely, if b T v 1 = 0 , then the CRS (1) has multiple optimal solutions.
From Proposition 2, if b T v 1 ̸ = 0 , then there is only one critical point, which is also the optimalsolution, and hence the gradient norm ∥∇ f A , b ,ρ ( x ) ∥ serves as an optimality measure.
Throughoutthe paper, we assume b T v 1 ̸ = 0 , under which the CRS is said to be in the easy case.
This is withoutmuch loss of generality as this holds generically true in practice.
Moreover, we could easily avoid thehard case ( b T v 1 = 0 ) by slightly perturbing the vector b , see [10, 2].
To introduce the secular equation, note that in the easy case, conditions (2) and (3) can be written as
[Equation]
where σ =: ρ ∥ x ∗ ∥ , [ y ∗ 1 , · · · , y ∗ n ] T :=
y ∗ =
V T x ∗ and [ c 1 , · · · , c n ] T :=
c = − V T b .
Therefore,y ∗ i
[Equation]
=
[Equation] [Equation]
Since the Euclidean norm is invariant to orthogonal transformation, we have
[Equation]
Consequently, instead of solving the complicated nonlinear system (2)-(3), we could solve the CRS (1)by first finding the (unique) root σ > max {− λ 1 , 0 } of the equation
[Equation]
called the secular equation, and then solves the linear system ( A + σ I ) x
= − b .
The first step can bedone efficiently by using existing root-finding algorithms (e.g., the bisection method and Newton’smethod etc.).
The disadvantage of the above CRS solver, based on the secular equation (4), is that it requires the fullspectrum of A , which costs O ( n 3 ) .
This approach is viable only for low- to moderate-dimensionalproblems.
However, when n is large, computing all eigenvalues of A is prohibitive.
Worse still, afterthe root σ is solved, we still need to apply iterative methods (e.g., Lanczos method) to solve thelarge-scale linear system (2).
We are thus motivated approximate the secular equation by using onlysome of the eigenvalues of A , as opposed to all.
As our main contribution, we developed two different approximate secular equations (ASEs), both ofwhich require computing m < n eigenvalues of A .
The cost for forming the approximate secularequations is only O ( mn 2 ) , and hence the resulting CRS solver is much more efficient and scalable.
On the theoretical side, for each of the proposed approximate secular equations, we first studied theexistence and uniqueness of its root, and then derived an upper bound on the gap between the root andthat of the standard secular equation (4).
This upper bound can in turn be used to bound the distancefrom the approximate CRS solution based ASEs to the true CRS solution, thus offering a theoreticalguarantee for the resulting CRS solver.
A desirable feature of our CRS solver is that it requiresonly matrix-vector multiplication but not matrix inversion, which makes it particularly suitable for high-dimensional applications of unconstrained non-convex optimization, such as low-rank recoveryand deep learning.
On the empirical side, we conducted experiments with both synthetic and realproblem instances to investigate the practical performance of the proposed CRS solver and theassociated CR.
Experimental results showed that the proposed solver outperforms two state-of-the-artmethods.
The selection of m for the proposed ASEM is an interesting and crucial topic. We willdiscuss related issues in section 4 and some numerical explorations are also conducted in section 5.
The First-Order Truncated Secular Equation
We define the first-order truncated secular equation by
[Equation]
where µ ≥ λ m is an input parameter that approximates the unobserved eigenvalues λ m +1 , · · · , λ n ,c i = − b T v i and (cid:80) ni
= m +1 c 2 i =
∥
b ∥ 2 − (cid:80) mi =1 c 2 i .
Note that only m eigenvalues λ 1 , · · · , λ m andtheir corresponding eigenvectors v 1 , · · · , v m are needed to form (5), which is computationallyfriendlier compared with (4).
The name first-order truncated secular equation comes from the factthat w 1 ( σ, µ ) is the first-order Taylor approximation to the function w ( σ ) .
Below we will first studythe existence and uniqueness for the root of (5).
Then, we derive an error bound for the root.
2.1 Existence and Uniqueness for the Root
In the easy case that b T v 1 ̸ = 0 (equivalently, c 1 ̸ = 0 ), the solution x ∗ to the CRS (1) is unique, whichimplies the existence and uniqueness for the root σ ∗ of (4).
To show that our proposed CRS solveris also well-defined, we prove the existence and uniqueness of the root of the first-order truncatedsecular equation (5).
Lemma 1.
For any µ ≥ λ m , the function w 1 ( · ; µ ) as defined in (5) admits a unique root.
Proof.
Existence.
We first consider the case when λ 1 ≤ 0 .
Then, for any fixed µ ≥ λ m ,lim σ →
[Equation]
+
[Equation]
[Equation] [Equation] [Equation] ∞ and
[Equation]
By the intermediate value theorem, w 1 ( · ; µ ) has a root in ( − λ 1 , + ∞ ) .
For λ 1 > 0 , we havew 1 (0; µ ) > 0 and [Equation] ) =
[Equation]
Therefore, w 1 ( σ ; µ ) has a root in (0 , + ∞ ) .
Uniqueness.
Note that w 1 ( σ ; µ ) is monotonically decreasing for σ ∈ ( − λ 1 , + ∞ ) and σ ∈ (0 , + ∞ )when λ 1 ≤ 0 and λ 1 > 0 , respectively.
Therefore, the uniqueness of the root for w 1 ( σ ; µ ) isguaranteed.
2.2 Error Analysis
In order to study the quality of the CRS solution based on our proposed solver using approximatesecular equations, we need the quality of the root to the first-order truncated secular equation, denotedby σ ∗ 1 .
Towards that end, we provide an upper bound on the gap | σ ∗ 1 − σ ∗ | between σ ∗ 1 and the root σ ∗of the exact secular equation (4).
Theorem 1.
Let σ ∗ 1 and σ ∗ be the unique roots of w 1 ( σ ; µ ) and w ( σ ) , respectively.
Then
[Equation]
[Equation]
where C m > 0 is a constant, upper bounded by
[Equation]
[Equation]
with B 1 =− λ 1 + √ λ
21 +4 ρ ·∥ b ∥being an upper bound for | σ ∗ 1 | .
We clearly see that the right-hand side of inequality (1) is decreasing in m .
This confirms that usingmore eigen information (i.e., larger m ) helps to reduce the error | σ ∗ 1 − σ ∗ | .
The proof of Theorem 1is technical and quite long and hence relegated to Appendix A.
The approximation quality of our
CRS solver is guaranteed by combining Theorem 1 with the following proposition.
Proposition 3.
Let x ∗ and ˜ x be solutions to the equations ( A + σ ∗
I ) x ∗ = − b and ( A + σ ∗ 1 I )
˜
x
=− b , respectively.
Then, ∥ ˜ x − x ∗ ∥ =
O ( | σ ∗ 1 − σ ∗ | ) .
Proof.
By definition, we have
[Equation]
and
[Equation]
then
[Equation]
This completes the proof.
Before ending this section, some remarks are in order.
First, the parameter µ acts as an approximationto n − m unknown eigenvalues λ m +1 , · · · , λ n .
An intuitive choice of µ that works well in practiceand is computationally cheap is the average of unknown eigenvalues, i.e.,
[Equation]
Second, the error bound C m · max m +1 ≤ i ≤ n | λ i − µ | in Theorem 1 depends on the distribution ofeigenvalues of A .
If the unobserved eigenvalues λ m +1 , · · · , λ n cluster around a small interval, thenwith a suitable choice of µ ∈ [ λ m +1 , λ n ] , max m +1 ≤ i ≤ n |
λ i − µ | is small.
Conversely, if the unknowneigenvalues spread over a large interval, then it is hard to make the error max m +1 ≤ i ≤ n | λ i − µ | small.
Third, it is instructive to study the error bound (6) under some random matrix model for A .
Supposethat A =
(cid:101)
A / √ 2 n , where (cid:101)
A is a symmetric random matrix with i.i.d.
entries on and above thediagonal.
By the Wigner semicircle law [6], as n → ∞ , the eigenvalues of A distribute according toa density of a semi-circle shape.
In particular, we can deduce that with probability of 1 − o (1) ,
[Equation]
The detailed proof of (8) and further discussions under random A can be found in Appendix C.
The Second-Order Truncated Secular Equation
Similarly to the equation (5), but with the second-order Taylor approximation, we define the secondorder truncated secular equation by
[Equation]
where µ ≥ λ m is an input parameter that approximates the unobserved eigenvalues λ m +1 , · · · , λ n .
3.1 Existence and Uniqueness for the Root
The lemma blow shows the existence and uniqueness of the root of w 2 ( · ; µ ) .
Lemma 2.
With
[Equation]
the function w 2 ( · ; µ ) as defined in (9) admits a unique root.
Proof.
When
[Equation]
the third summation in the definition (9) vanishes, and hence w 2 ( σ, µ ) becomes the same as w 1 ( σ, µ ) ,except with a specific choice of µ .
The desired conclusion then follows from Lemma 1.
Unlike its first-order counterpart, we do not develop the existence and uniqueness of the root of thesecond-order truncated secular equation for arbitrary µ .
The reason is that when λ 1 > 0 , w 2 (0; µ )can potentially be positive or negative.
3.2 Error Analysis
Similar to that for the first-order truncated secular equation, we can also derive an error bound for theroot of the second-order truncated secular equation.
Theorem 2.
Let σ ∗ 2 and σ ∗ be the unique root of w 2 ( σ ; µ ) and w ( σ ) , respectively, and
[Equation]
Then,
[Equation]
(11) where C m > 0 is a constant bounded by
[Equation]
with B 1 =−
[Equation]
+
[Equation] [Equation]
∥being an upper bound for | σ ∗ 2 | .
The proof of Theorem 2 can be found in Appendix B. We can similarly estimate the approximationquality by combining Theorem 2 and Proposition 3.
Again, the right-hand side of the error bound (11)is decreasing in m .
We should also point out that the CRS solver based on the second-order secularequation outperforms the first-order counterpart only if max m +1 ≤ i ≤ n | λ i − µ | / | λ m − λ 1 | is smallenough.
The computation of µ here requires c m +1 , · · · , c n , which seem to be inaccessible.
Weprovide a tractable form for µ in (13) and will discuss it in the next part.
Implementation Details
We now discuss the implementation details for solving the proposed first-order secular equationfor CRS.
First, we obtain the partial eigen information { λ 1 , · · · , λ m } and { v 1 , · · · , v m } by Krylovsubspace methods.
Note that only Hessian-vector products are required.
This is computationallyfriendlier than other methods that rely on matrix inversions and particularly suitable for modern, highdimensional applications.
Then, we solve the first-order secular equation (5) with µ defined in (7) or(10), using anyroot-finding algorithm, such as Newton’s method.
Finally, we solve the linear system( A + σ ∗ I ) x
= − b by iterative algorithms, e.g., the Lanczos method and the conjugate gradientmethod.
The resulting CRS solver, namely the approximate secular equation method (ASEM), issummarized as follows:
Step 1: obtaining the partial eigen information { λ 1 , · · · , λ m } and { v 1 , · · · , v m } of A .
Step 2: solving the secular equation (5) with µ defined in (7) or (10); we get σ ∗ .Step 3: iteratively solving the linear system ( A + σ ∗ I ) x
+ b = 0 .
Output: the solution x .
Details for step 1.
The Krylov subspace is one of the most popular iterative methods in solving eigenproblems with O ( mn 2 ) computational cost [11].
The Lanczos decomposition for a real symmetricmatrix B satisfies
[Equation]
where U k ∈ R n × k is an orthonormal matrix (i.e., U T
k U
k
= I k ), T k ∈ R k × k is a symmetrictridiagonal matrix and e k ∈ R
k is the k -th standard basis vector in R k .
Lanczos observed thateven for comparatively small k , T k has significant approximation to eigenvalues and eigenvectorsof B .
Specifically, for a suitable eigenpair ( γ, w ) of T k with T k w = γ · w , the pair ( γ, U k w )is an approximation to an eigenpair of B , i.e., Bz ≈ γ · z with z
= U k w .
Here, the Krylovsubspace is constructed by u 1 , the first column of U k
=
[ u 1 , · · · , u k ] , i.e., K k ( B , u 1 ) .
Note that
T k approximates B for eigenvalues with largest modulus (or absolute values) and the correspondingeigenvectors.
Empirically, for calculating m eigenvalues of B with largest absolute values and thecorresponding eigenvectors, we usually construct the Krylov subspace K k ( B , u 1 ) with dimensionk = max { 2 m, 20 } .
Moreover, the base vector u 1 is also essential for the Krylov subspace method.
Moreover, restarting is adopted to iteratively update the base vector u 1 .
To the best of our knowledge,the mentioned iterative method for partial eigen informnation is supported in many softwares, e.g.,
Matlab (eigs function) and Python (Scipy package) etc.
For more details, please refer to ARPACK[9].
Returning back to the proposed algorithm with A , instead of calculating the largest (in termsof the absolute value) m eigenvalues of A , we aim to get m (algebraically) smallest eigenvalues{ λ 1 , · · · , λ m } and the corresponding eigenvectors { v 1 , · · · , v m } .
We first roughly calculate a shiftvalue β ≥ ∥
A ∥ by several steps of power iteration (Hessian-vector products).
Then, let B = β · I − A ,whose eigenvalues { β − λ n , · · · , β − λ 1 } are non-negative and the corresponding eigenvectors are{ v n , · · · , v 1 } .
Applying the mentioned Krylov subspace algorithm, we first obtain an estimate ofshifted eigenvalues { λ 1 , · · · , λ m } and the corresponding eigenvectors { v 1 , · · · , v m } for A , since{ β − λ m , · · · , β − λ 1 } are largest eigenvalues of B .
To further lower the computational cost, wemay adopt k = m dimensional Krylov subspace K k ( B , u 1 ) for m eigenvalues without restarting inimplementation.
Details for step 2.
Instead of directly solving (5), Cartis et al.
[3] recommended to find the root forthe equivalent equation:
[Equation]
which is convex on ( − λ 1 , + ∞ ) .
Moreover, Newton method is proved to achieve (locally) quadraticconvergence with perfect initialization.
However, we numerically find that it depends much on theinitialization and may converge to a point outside the feasible domain ( − λ 1 , + ∞ ) if it has imperfectinitialization.
Here, we recommend to use the bisection method to find the root of (5) or (12) due toits linear convergence, stability (not beyond the feasible domain) and simple implementation.
For theweighted average µ defined in (10), we can rewrite it as a more tractable but equivalent form that
[Equation]
where V m =
[ v 1 , · · · , v m ] and Λ m =
diag
( λ 1 , · · · , λ m ) .
Details for step 3.
There are a bunch of and well-behaved iterative methods for (real symmetric) linearsystems, e.g., Krylov subspace (Lanczos) methods and conjugate gradient methods etc.
We do notdiscuss any details here since solving linear systems has been investigated for decades and achievedgreat success.
We adopt the Lanczos method for solving the linear system ( A + σ ∗ I ) x
+ b = 0 ,where multiple steps of Hessian-vector products are required.
In summary, the main computational cost comes from step 1 and step 3 for Hessian-vector products( O ( mn 2 )) , since w 1 ( σ ; µ ) is a one dimensional problem for σ in step 2 and is of O ( n ) .
Therefore,the total computational cost for the proposed algorithm is of O ( mn 2 ) , much lower than the methodwith full eigendecomposition ( O ( n 3 ) ).
The selection of m .
We seek to set proper m in ASEM for solving CRS because overly large mwould cause computational waste while too small m cannot achieve our expected accuracy.
Toachieve ε -accuracy (i.e., | σ ∗ 1 − σ ∗ | ≤ ε ), with random Gaussian matrix A , we may set m ≤ n such that
[Equation]
according to (8).
However, the error bound (8) may be overestimated for m .
Moreover, for generalproblems without priors for eigen distribution of A , it is hard to estimate m based on expectedε -accuracy, because the error bounds (6) and (11) are implicit for m .
Therefore, adaptive methods(or heuristic methods) for selecting m are essential and urgent in practice.
An intuitive way isto check the suboptimality (gradient norm) in each step, and enlarge m by m 0 (i.e., we set m =max { m + m 0 , m max } , where m max is the maximal number of eigenvalues we adopt in ASEM), ifthe output does not satisfy the given condition for suboptimality.
Moreover, numerical experimentson CUTEst problems (see Experiment 6 in section 5) shows that m = 1 is enough for most of thecases.
We left the study for the selection of m as a future work.
To the best of our knowledge, it isalso unsolved for the Krylov subspace method [3, 2] in solving CRS.
Experimental Results
Without the loss of generality (see Appendix D), we adopt diagonal matrix A in the synthetic cubicregularized quadratic problem, for simplicity and fair comparison.
Furthermore, we also test theproposed ASEM on CUTEst library [7].
All experiments were run on a Macbook Pro M1 laptop.
Formore experimental details, please refer to Appendix E.
Experiment 1.
The distribution for eigenvalues of the matrix A .
In the error analysis (Theorem 1 and
Theorem 2), the error is controlled by the approximation of µ to unobserved eigenvalues { λ i } ni = m +1 ,i.e., max m +1 ≤ i ≤ n | λ i − µ | and max m +1 ≤ i ≤ n ( λ i − µ ) 2 .
It further implies that the distribution ofeigenvalues { λ i } ni = m
+1 is essential for the proposed method.
Intuitively, if eigenvalues { λ i } ni = m +1 236 cluster around a small interval, then the rough estimate (7) for µ is enough to approximate theunknown eigenvalues well.
Conversely, if eigenvalues { λ i } ni
= m +1 are spread over a large interval,then we cannot expect a single µ to estimate all eigenvalues { λ i } ni = m +1 .
Here, we have four speciallydesigned cases for distributions of eigenvalues of the matrix A to illustrate our theoretical observationsfor the proposed method.
Case 1 (evenly spaced): all eigenvalues { λ i } ni =1 are evenly spaced in[ − 1 , 1] ; Case 2 (separated): half of eigenvalues are far away from the remaining, i.e., eigenvalues{ λ i } n/ 2 i =1 are evenly spaced in [ − 1 , − 4 / 5] and the remaining eigenvalues { λ i } ni = n/ 2+1 are evenlyspaced in [4 / 5 , 1] ; Case 3 (right centered): the minimal 2% of eigenvalues and the remaining 98%of eigenvalues gather together respectively, i.e., eigenvalues { λ i } n/ 50 i =1 and { λ i } ni = n/ 50+1 are spacedevenly in [ − 1 , 4 / 5] and [4 / 5 , 1] respectively; Case 4 (left centered): the maximal 2% of eigenvaluesand the remaining 98% of eigenvalues gather together respectively, i.e., eigenvalues { λ i } 49 / 50 n i = and{ λ i } n i =49 / 50 n +1 are evenly spaced in [ − 1 , 4 / 5] and [4 / 5 , 1] respectively.
The vector b is proportionalto [1 , · · · , 1] T with ∥ b ∥ = 0 .
The remaining parameters are n = 5 × 10 3 and ρ = 0 .
We adoptthe first-order method (i.e., µ is defined in (7)) here.
Figure 1 validates our theories that the proposedalgorithm converges fast if unknown eigenvalues { λ i } ni = m +1 are close.
Moreover, without the needto compute all eigenvalues, partial eigen information is enough to achieve satisfactory solutions inpractice, except the hard case (e.g., case 4).
[Figure] [Figure]
∥∇ f A , b ,ρ ( x ) ∥ ) with different µ in Experiment
[Figure]
Experiment 2.
The effect of the parameter µ (first-order and second-order methods).
For thefirst-order method, we select (7) for µ , while (10) for the second-order method based on our previousanalysis.
Here, we test the effect of µ for the proposed method in solving the cubic regularizedquadratic problem, with other parameters fixed.
Case 1 (first-order method): µ is adopted as themean value of unknown eigenvalues, defined in (7); Case 2 (second-order method): µ is selectedas the weighted average of unknown eigenvalues with respected to c 2 i (please see (10)); Case 3(first-order method): µ = λ m , the maximal eigenvalue we observe; Case 4: µ = 10 6 , much largerthan eigenvalues of A , as an approximation to + ∞ .
Different from the previous two experiments, thevector b is proportional to [ λ 1 , · · · , λ n ] T with length ∥ b ∥ = 0 .
Eigenvalues of the matrix A areevenly spaced in [ − 1 , 1] .
The remaining parameters are d = 5 × 10 3 and ρ = 0 .
Figure 2 shows thesuperiority of second-order method over the first-order method that it is more stable and convergesfaster when m is large, consistent with the theoretical analysis Theorem 1 and Theorem 2.
Moreover,the results further imply the significance of the selection for µ .
There are several observations from
Figure 2.
Firstly, we cannot discard the residual term with the unknown eigenvalues { λ i } ni
= m +1 ,where they still contain much information, as is shown in case 1 and case 4.
Secondly, randomselection of µ is not acceptable and may cause non-convergence (e.g., see case 3 and case 4).
Furthermore, proper selection of µ contributes to the well-behaved algorithm (e.g., see case 1 andcase 2).
Experiment 3.
Approximation capabilities of the Krylov subspace method for ASEM.
As is introduced in section 4, we adopt m dimensional Krylov subspace K m ( B , u 1 ) to approximatelycalculate m algebraically smallest eigenvalues { λ 1 , · · · , λ m } and the corresponding eigenvectors{ v 1 , · · · , v m } .
Here, we investigate the performance of ASEM with estimated eigenvalues andeigenvectors.
The vector b is proportional to the vector [1 , · · · , 1] T with length
∥ b ∥ = 0 . 1 .
Eigenvalues of the matrix
A are evenly spaced in [ − 1 , 1] .
The remaining parameters are n = 5 × 10 3 andρ = 0 .
We adopt the first-order method (i.e., µ is defined in (7)) here.
Trajectories for suboptimalitywith exact and approximated eigenvalues and eigenvectors are shown in Figure 3.
The Krylovsubspace K m ( B , u 1 ) with relatively low dimension m for ASEM matches well with ASEM withexact eigenvalues.
It provides confidence that we can use the m dimensional Krylov subspace for meigenvalues and eigenvectors in ASEM.
Experiment 4.
Comparison of ASEM with the Krylov subspace method [3, 2] and the gradientdescent method [1] on synthetic problems.
For large scale problems, the Krylov subspace methodand the gradient descent method are two state-of-the-art methods in solving the target problem(1).
In this experiment, we compare the proposed ASEM (with partial eigen information) with the
Krylov subspace method and the gradient descent method.
The dominant computation steps for thosethree methods are Hessian-vector products ( O ( mn 2 ) ).
The vector b is proportional to the vector[1 , · · · , 1] T with length ∥ b ∥ = 0 .
Eigenvalues of the matrix
A are evenly spaced in [ − 1 , 1] .
Similarto the setting in [2], we define the condition number for (1) as κ = λ n + ρ ·∥ x ∗ ∥ λ 1 + ρ ·∥ x ∗ ∥ =
λ n + σ ∗ λ 1 + σ ∗ .
Then, wehave σ ∗ =
λ n − κ · λ 1 κ −and ρ
=
σ ∗ ∥
( A + σ ∗ I ) − 1 b ∥ .
Case 1: easy case that κ = 10 3 ; Case 2: harder case thatκ = 10 6 .
The remaining parameter is n = 5 × 10 3 .
We adopt the first-order method (i.e., µ is definedin (7)) here.
As are shown in Figure 4, ASEM outperforms both the gradient descent method andthe Krylov subspace method when m is relatively large.
It is reasonable that ASEM underperforms
[Figure]
when m is small since the m dimensional Krylov subspace cannot well approximate eigenvalues andeigenvectors of A .
The results further display the performance of the proposed ASEM method.
Experiment 5.
Comparison of ASEM with the Cauchy point method [5], the gradient descentmethod, and the Krylov subspace method on CUTEst problems [7].
CUTEst library collects variousunconstrained and constrained optimization problems that arise in real applications.
In this part, wecompare the numerical performances of the ARC algorithm [3] on four unconstrained optimizationproblems from the CUTEst library, where subproblems are solved by the Cauchy point method (ARCCP), the gradient descent method (ARC-GD), the Krylov subspace method (ARC-Krylov( k ), wherek is the number of Lanczos basis vectors) and the ASEM method (ARC-ASEM( m ), where m is thenumber of eigenvalues for ASEM).
The architecture of the ARC algorithm is provided in Appendix
E.2.
Four unconstrained optimization problems (e.g., TOINTGSS, BRYBAND, DIXMAANG and
TQUARTIC) in the CUTEst library are adopted for testing, where their dimensions are 1000, 2000,3000 and 5000 respectively.
We use the first order ASEM ( µ is defined in (7)) here, since we foundthat the performances of the first order method and the second order methods in ASEM does notdiffer so much for these problems.
Numerical results are reported in Table 1, where x out , ∥∇ f ( x out ) ∥ ,λ 1 ( ∇ 2 f ( x out )) , iter and time represent the output of the ARC algorithm, the suboptimality (gradientnorm) of the objective function, the minimal eigenvalue of the Hessian matrix, numbers of iterationfor the ARC and CPU time, respectively.
Here are several observations.
Firstly, the proposed
ASEM algorithm outperforms others in most of cases and is comparible with the Krylov subspacemethod sometimes, where the ASEM achieves lower suboptimality (gradient norms) or CPU time.
Furthermore, only one eigenvalue is enough for the well-behaved ASEM (i.e., ARC-ASEM( m ) withm = 1 ), which is a surprising result not observed from theoretical analysis.
The main reason is that
Hessian matrices in these problems are not left centered (i.e., case 4 in experiment 1), consistent withour analysis.
For more experimental details, please refer to Appendix E.
Conclusion
We develop the first-order and the second-order truncated secular equations as surrogates to thesecular equation with full eigendecomposition in solving the cubic regularized non-convex quadraticproblem (1).
The proposed ASEM algorithm is feasible since it reduces the computational cost from
O ( n 3 ) to O ( mn 2 ) .
The theoretical analysis that more eigen information contributes to lower errorshows the properties and reasonableness of the algorithm.
We also introduce the implementationdetails for ASEM algorithm that multiple steps of Hessian-vector products are requisite for estimatingeigenvalues and eigenvectors rather than dealing with a large matrix.
Numerical experiments furtherinvestigate properties and performances of the proposed ASEM algorithm and validate the derivedtheoretical results.
Moreover, the ASEM algorithm converges faster than the state-of-the-art methodson synthetic and CUTEst problems.
[Table]
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] As is mentioned in the lastparagraph of section 4, the proposed ASEM algorithm imvolves in the selection of m ,which balances the accuracy and computational cost of ASEM.
We left the (adaptiveand heuristic) methods for the selection of m as a future work.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] Theproposed ASEM is for general optimization problems, and it belongs to the foundationalresearch.
We do not expect it to involve negative societal impacts.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Yes
If you are including theoretical results...(
a
) Did you state the
full
set of assumptions of all theoretical results
[Yes]
(b) Did
you
include complete proofs
of all
theoretical results?
[Yes] We provide sketch of
proof following theorems and detailed proofs are available in Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Our results are
reproducible and we will share our code on github later.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Details of experiments are specified in the paper, see section 5.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No] Our experiments do not involve random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] All experiments are conducted on
MacBook Pro M1.
Moreover, we also report the CPUs time.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] Most of works are reproduced by
ourselves.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes](d) Did you discuss whether
and how consent was obtained from people whose data you’re
using/curating?
[Yes] We mention that we test the proposed algorithm on synthetic
problems and open datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] We mention that we test the proposed algorithmon synthetic problems and open datasets.
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[No] Our paper does not involve researches with human subjects.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[No] Our paper does not involve researches withhuman subjects.
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[No] Our paper does not involve researches withhuman subjects.
[Table]
A Proof of Theorem 1
The secular equation (4) can be decomposed into two parts:
[Equation]
) where e 1 ( σ ; µ ) = w ( σ ) − w 1 ( σ ; µ ) represents the error between two secular equations.
By the Taylor’s theorem, we have
[Equation]
where ξ ∈ ( σ ∗ , σ ∗ 1 ) .
By the definition that w 1 ( σ ∗ 1 ; µ ) = 0 and w ( σ ) is monotonically decreasing inthe domain (i.e., ddσ w ( ξ ) ̸ = 0 ), we have
[Equation]
For the error term, we can further rewrite it as
[Equation]
for some ς i ∈ ( µ, λ i ) , i = m + 1 , · · · , n .
Due to µ ≥ λ m
and λ i ≥ λ m , we have ς i > λ m and thusς i + σ > λ m − λ 1 > 0 .
Therefore, the error term can be bounded by
[Equation]
The derivative of the secular equation is
[Equation]
, for σ > max {− λ 1 , 0 } .
Then, we have
[Equation]
d
[Equation]
and
[Equation]
n
[Equation]
Moreover, σ ∗ 1 is a root of w 1 ( σ ; µ ) = 0 , which implies that
[Equation]
and thus
[Equation]
Similarly, σ ∗ is a root of w ( σ ) = 0 , which further implies that
[Equation]
and thus 0 ≤ σ ∗ ≤ B 1 .
Due to ξ ∈ ( σ ∗ , σ ∗ 1 ) (or ξ ∈ ( σ ∗ 1 , σ ∗ ) ), we have
[Equation]
Equipped with the above results, we can bound the derivative term by
[Equation]
Note that B 1 > − λ 1 and thus ( λ n + B 1 ) 3 > 0 , which implies that (18) makes sense.
Combiningwith (15), (16) and (18), we have
[Equation]
where the constant C m is bounded by
[Equation]
B Proof of Theorem 2
Similar steps as in the proof for Theorem 1.
The secular equation (4) can be decomposed into twoparts:
[Equation]
19) where e 2 ( σ ; µ ) =
w ( σ ) − w 2 ( σ ; µ ) represents the error between two secular equations.
By Taylor’s
Theorem, we have
[Equation]
where ξ ∈ ( σ ∗ , σ ∗ 2 ) .
By the definition that w 2 ( σ ∗ 2 ; µ ) = 0 and w ( σ ) is monotonically decreasing inthe domain (i.e., ddσ w ( ε ) ̸ = 0 ), we have
[Equation]
For the error term
[Equation]
− µ
[Equation]
for some ς i ∈ ( µ, λ i ) , i = m + 1 , · · · , n .
Due to µ > λ m , we have ς i > λ m and thus ς
i + σ >λ m − λ 1 > 0 .
Therefore, the error term can be bounded by
[Equation] [Equation]
Due to the special choice of µ = µ 2 defined in (10), same upper bound for the derivative term isachieved as that in Theorem 1, i.e.,
[Equation]
[Equation]
Combining with (20), (21) and (22), we have
[Equation]
C Random Gaussian Matrix
In this part, we further elaborate the error analysis in Theorem 1 for random Gaussian matrix, wherewe can access the eigen distribution.
We first introduce the definition for a class of Gaussian randommatrix.
Definition 1.
A n × n matrix M with entries m ij is a Wigner matrix if m ij = m ji and m ij arerandomly i.i.d.
up to symmetry.
Definition 2.
Let M be a real Wigner matrix with diagonal entries m ii ∼ N (0 , 1) and m ij ∼
N (0 , 1 / 2) , i ̸
= j .
Then, M is said to belong to the Gaussian orthogonal ensemble (GOE).
Remark.
Consider a random n × n matrix W with all entries coming from N (0 , 1) , then M =
W + W Tis a Gaussian orthogonal ensemble (GOE) matrix.
Theorem 3.
Suppose that the n × n matrix A =
(cid:101)
A √ 2 n with (cid:101)
A belonging to GOE, then with probabilityof 1 − o (1) , we have
[Equation]
where µ ∈ [ λ m +1 , λ n ] and n is large enough.
Proof.
According to section 1.4 in [6], eigenvalues { λ 1 , · · · , λ n } of A follows the Wigner’s semicircle law that p.d.f.
of λ ∈ { λ 1 , · · · , λ n } satisfies
[Equation]
where I [ − 1 , 1] ( λ ) is the indicator function that values 1 if λ ∈ [ − 1 , 1] and 0 if λ / ∈ [ − 1 , 1] .
Therefore,
[Equation]
Furthermore,
[Equation]
where the last equation comes from π 2 − arcsin( λ ∗ ) ≈ (cid:112) 2(1 − λ ∗ )
=
O (cid:0) √ 1 − λ ∗ (cid:1) when λ ∗ → 1 − ,and we may select λ ∗ with 1 − λ ∗ =
O (cid:16)(cid:0) 1 − m +1 n (cid:1) 2 / 3 (cid:17) ≈ (cid:16) 3 π 4 √ 2 (cid:17) 2 / 3 ·
[Equation]
1 − m +1 n (cid:1) 2 / 3 such that
P ( λ ∗ ≤ λ ≤ λ m +1 ) > 0 .
Therefore, with probability of 1 − o (1) , we have λ ∗ ≤ λ m +1 ≤ µ ≤λ n ≤ 1 and thusmax
[Equation]
Remark.
The above theorem also holds for other kinds of symmetric random Gaussian matrix, e.g.,the Gaussian unitary ensemble (GUE) matrix and the Gaussian symplectic ensemble (GSE) matrixetc., because eigen distribution for both GUE and GSE matrix follow the Wigner’s semicircle law [6].
D Equivalence of Cubic Regularized Quadratic Problems
For the cubic regularized non-convex quadratic problem
[Equation]
24) the Hessian matrix A is diagonalizable, i.e., there exist a diagonal matrix Λ ∈ R n × n =diag ( λ 1 , · · · , λ n ) and a unitary matrix V ∈ R n × n , such that
[Equation]
= VΛV
[Equation]
Then (24) is equivalent to the following problem (25) with diagonal Hessian matrix Λ :
[Equation]
25) where c = V T b .
Therefore, we adopt diagonal matrix A in numerical experiments since these twoproblems are equivalent.
E Experimental Details
E.1 The Cauchy Point
The Cauchy radius [5] for the cubic regularized quadratic problem
[Equation]
is the solution of
[Equation]
and the Cauthy point is defined as
[Equation]
Note that f A , b ,ρ ( x c ) ≤ f A , b ,ρ ( 0 ) = 0 .
E.2 The ARC Algorithm
We present the ARC algorithm [3] (Algorithm 1) here for completeness and clarification.
E.3 Hyperparameters
In this subsection, we provide hyperparameters of the ARC algorithm in solving four optimizationproblems (e.g., TOINTGSS, BRYBAND, DIXMAANG and TQUARTIC) from the CUTEst library.
In all experiments, we set γ 1 = γ 2 = 2 , η 1 = 0 .
1 , η 2 = 0 .
9 and ρ 0 = 10 3 .
The initial guess x 0 isavailable for each problem (you can access it by prob.x command in MATLAB).
For TOINTGSS,
BRYBAND, DIXMAANG and TQUARTIC, we select the top 1000 , 2000 , 3000 and 5000 variablesas unknown variables for optimization, respectively.
E.4 Additional Experiments
Experiment 6.
The effect of the length ∥ b ∥ .
The derived upper bound in (6) and (11) are proportionalto ∥
b ∥ 3 / 2 , which implies that larger ∥ b ∥ may cause worse approximation for σ ∗ and thus theunattainable optimality.
Here, we tested the convergence of the proposed method for different length∥ b ∥ with other parameters fixed.
In case 1 to 6, we set ∥ b ∥ ∈ { 1 , 0 . 5 , 0 . 2 , 0 . 1 , 0 . 05 , 0 . 01 } .
Thevector b is proportional to the vector [1 , · · · , 1] T .
Eigenvalues of the matrix A are evenly spaced in[ − 1 , 1] .
The remaining parameters are n = 5 × 10 3 and ρ = 0 .
We adopt the first-order method(i.e.
, µ is defined in (7)) here.
As is shown in Figure 5, the proposed method is stable for both largeand small ∥ b ∥ while gradient descent [1] is ill-conditioned if ∥ b ∥ is tiny.
Consistent with the erroranalysis, the proposed method converge faster for smaller length ∥ b ∥ .
Target: a local minimizer of the objective function f ( x ) , where x ∈ R n .
Input: initial guess x 0 , γ 2 ≥ γ 1 > 1 , 1 > η 2 ≥ η 1 > 0 , ρ 0 > 0 and the integer T > 0 .
Output: the approximate solution x out ∈ R n .
1: for t = 0 , · · · , T do 2: load b t = ∇ f ( x t ) , A t = ∇ 2 f ( x t ) 3: compute the Cauchy point s c of f A t , b t ,ρ t ( s ) :
=
b T t s + 12 s T A t s + ρ t 3 ∥ s ∥4: solve f A t , b t ,ρ t ( s ) by the gradient descent, the Krylov subspace or the ASEM method, the approximate solution is denoted as s t 5: if f A t , b t ,ρ t ( s c ) ≤ f A t , b t ,ρ t ( s t ) then 6: s t =
s c ; 7: end if 8: compute κ t = f ( x t ) − f ( x t + s t ) − f A t , b t ,ρ t ( s t ) ; 9: set x t +1 = (cid:26) x t + s t if κ t ≥ ηx t otherwise 10: set ρ t +1 ∈ (cid:40) (0 , ρ t ] if κ t > η(very successful iteration)
[ ρ t , γ 1 ρ t ] if η 1 ≤ κ t ≤ η(successful iteration) [ γ 1 ρ t , γ 2 ρ t ] otherwise (unsuccessful iteration) 11: end for   m 0.
g r a d i e n t n o r m 1ecase 1: ||
b ||
=case 2: ||
b || = 0.
case 3: ||
b || = 0.
case 4: ||
b ||
[Figure]