A Variant of Anderson Mixing with Minimal Memory Size
Anonymous Author(s)
AfﬁliationAddress
Abstract
Anderson mixing (AM) is an acceleration method for ﬁxed-point problems byexploring the information from historical iterations.
Despite its numerical successin various applications, the memory requirement in AM remains a bottleneckwhen solving large-scale optimization problems in a resource-limited machine.
Toaddress this problem, we propose a novel variant of AM method, called Min-AM,by storing only one vector pair, that is the minimal memory size requirement in AM.
Our method forms a symmetric approximation to the inverse Hessian matrix and isproved to be equivalent to the full-memory Type-I AM for solving strongly convexquadratic optimization.
Moreover, for general nonlinear optimization problems,we establish the convergence properties of Min-AM under reasonable assumptionsand show that the mixing parameters can be adaptively chosen by estimatingthe eigenvalues of the Hessian.
Finally, we extend Min-AM to solve stochasticprogramming problems.
Experimental results on logistic regression and networktraining problems validate the effectiveness of the proposed Min-AM.
Introduction
Anderson mixing (AM) [2] is a powerful method for accelerating ﬁxed-point iterations, and has beenwidely used in scientiﬁc computing [1, 5] and machine learning [30, 57].
In each iteration, AMextrapolates a new iterate that satisﬁes certain optimality property by using the historical iterations,and exhibits empirical acceleration over classical ﬁxed-point iterations [59].
Thus, AM can reducethe computation cost when evaluating the ﬁxed-point map is a time-consuming process.
Also, theinspiring connections of AM with GMRES [59, 47] and multisecant quasi-Newton methods [22]provide certain theoretical properties for the acceleration beneﬁt from AM.
In recent years, there havebeen many works applying AM to solve various optimization problems [55, 61, 40].
Their promisingresults in different tasks suggest that these AM-based methods are very competitive with ﬁrst-ordermethods while being more economical than second-order methods.
However, as commented by therecent review [3], the existing convergence analysis of AM is still unsatisfactory and it is worth notingthat AM does not converge in general [40].
Consequently, exploring theoretical properties of AMand its variants and their applications for machine learning problems deserves deep research.
One major concern of AM is the heavier memory overhead compared with the ﬁxed-point iteration.
In AM( m ), it has to store vector pairs about historical information from the m previous iterations,where m is the memory size.
Speciﬁcally, AM( m ) is called as the limited-memory AM if m < ∞ ,and full-memory AM if m = ∞ .
The AM( 0 ) is the simple ﬁxed-point iteration.
In practice, choosingproper memory size m is heuristic and it is observed in [46] that a small m can deteriorate the efﬁcacy.
However, for practical applications, a large m can lead to memory issue for solving large-scale andhigh-dimensional problems when the memory resource is limited.
In this paper, we address the memory issue of AM for solving optimization problems by proposing
Min-AM, a novel variant of AM( 1 ) that has minimal memory size in AM.
Min-AM is built uponthe Type-I AM [22, 59], a variant of AM.
Let AM-I( m ) denote the Type-I AM with memory size m .
Using a projection-mixing framework of AM-I, the Min-AM retains a recursively modiﬁed vectorpair and adds one extra projection step in each step of AM-I( 1 ).
Compared to AM(1), Min-AM formssymmetric approximations to the Hessian and is equivalent to AM-I( ∞ ) when the objective functionis strongly convex quadratic.
By incorporating the restarting strategy, the restarted Min-AM has fasterconvergence rate than AM( 1 ) for general nonlinear optimization and the spectrum of Hessian can becheaply estimated that leads to adaptive choices of the mixing parameters.
Min-AM is also extendedto stochastic optimization.
In summary, we highlight the main contributions of this work as follows.
• We propose Min-AM, a variant of AM(1) that achieves the minimal memory size in AM,
for solving optimization problems.
Compared to AM( 1 ), Min-AM forms symmetric approximations to the Hessian and incorporates more historical information to obtain the update.
Thus, Min-AM can signiﬁcantly reduce the memory requirement and computational cost of
AM while maintaining the fast convergence property.
• We show that the local convergence rate of the restarted Min-AM has optimal dependenceon the condition number of the objective function.
We provide adaptive choices of themixing parameters by estimating the spectrum of the Hessian economically in each iteration.
Also, we give the convergence and complexity analysis of a stochastic extension of Min-AM
for solving nonconvex stochastic optimization.
• We verify the properties of Min-AM, and apply the restarted Min-AM to logistic regression
and the stochastic Min-AM to train neural networks.
The experimental results are consistent
with our theoretical analysis and show that the Min-AM is competitive with the limited-
memory quasi-Newton methods with large memory size in deterministic optimization
andachieves promising results in training neural networks.
Related work.
Fixed-point problems can be recast as solving systems of (non)linear equations.
AM is a practical alternate for Newton’s method when the Jacobian is unavailable or difﬁcult tocompute [34].
In the linear case, it is established in [59, 47] that the full-memory AM is equivalent to
GMRES [50], a classical Krylov subspace method for linear systems.
In the nonlinear case, AM isrecognized as a multisecant quasi-Newton method and a variant, called Type-I AM, has been proposedin [22].
From the perspective of convergence analysis, it is proved that the local convergence rateof the limited-memory AM is no worse than that of ﬁxed-point iteration [58, 16] and the potentialimprovement depends on the quality of extrapolation [21, 45].
However, a counter example [40]shows the divergence behaviour of AM( 1 ) and the recent review [3] indicates the need of furtherresearch on the theoretical properties of AM.
The inspiring performance and the easy implementation of AM motivate many new variants of AMfor various applications in machine learning.
In [54], a regularized variant of AM is proposed toaccelerate gradient descent for unconstrained optimization, which is adapted in [23] for nonsmoothconvex optimization and in [57] for reinforcement learning.
In [61], a stochastic version of AM isdeveloped and has convergence guarantee for stochastic optimization.
The Type-I AM is modiﬁedin [66] to solve nonsmooth ﬁxed-point problems.
A more recent work, the short-term recurrence
AM (ST-AM) [62], stores two vector pairs to reduce the memory cost of AM.
Our Min-AM methodachieves the minimal memory size, i.e. m = 1 , further reducing the memory size of ST-AM.
The memory overhead is a common issue of quasi-Newton methods.
It is desirable to keep memorysize as small as possible without sacriﬁcing the efﬁcacy of the algorithm.
Some works [36, 9, 8]study the choice of memory size in L-BFGS [38], but the performance with minimal memory sizeis unsatisfactory.
In stochastic programming, the sublinear convergence rate is optimal when onlystochastic gradients can be accessed [42].
Thus the heavier memory and computational cost makesquasi-Newton methods [12, 27, 60, 10] less appealing than the more economical ﬁrst-order methods[48, 35].
In this work, our proposed Min-AM method exhibits the fast convergence as quasi-Newtonmethods while the memory cost is close to ﬁrst-order methods.
Notations.
Let ∆ denote the forward difference operator, e.g. ∆ x k = x k +1 − x k .
range( X ) denotesthe subspace spanned by the columns of a matrix X .
For every matrix A ∈ R d × d , the Krylovsubspace K m ( A, v ) ≡ span { v, Av, . . .
A m − 1 v } , and (cid:107) x (cid:107)
A =
( x T Ax ) 1 / 2 denotes the A
-normwhen
A is symmetric positive deﬁnite (SPD).
“ † ” is the Penrose-Moore inverse.
The Anderson mixing scheme
In this work, we consider the minimization problem
[Equation]
where f : R d → R is continously differentiable.
The ﬁrst-order optimality condition of (1) isequivalent to x = g ( x ) , where g ( x ) = x − ∇ f ( x ) .
We describe the Anderson mixing scheme withthe projection-mixing framework [61].
Deﬁne r k = g
x k )
− x k = −∇ f
( x k ) to be the residual.
AM( m ) maintains m ( m ≤ min { d, k } ) vector pairs stored in X k , R k ∈
[Equation]
[Equation]
Each update can be decoupled into two steps, namely the projection step and the mixing step :
[Equation]
where ¯ r k :=
r k − R k Γ k , β k > 0 is the mixing parameter, and Γ k = arg min Γ ∈
R m (cid:107) r k − R k Γ (cid:107) 2 .
It 98 is easy to know that the Galerkin’s projection condition ¯ r k ⊥ range( R k ) holds.
Assuming R T k R k is 99 nonsingular, the resulting one-step update of AM( m ) is x k +
= x k + H k r k where
[Equation]
The AM-I [22, 59] falls into the same projection-mixing framework, and the only difference lies inthe choice of Γ k .
The Γ k of AM-I is determined by another Galerkin’s condition:
[Equation] [Equation]
Substituting the solution of (5) into (3), the approximated inverse Hessian matrix H k of AM-I is
[Equation]
by assuming X T k R k is nonsingular.
AM-I has a close relation to Newton’s method and an alternativeinterpretation of AM-I is given in Appendix A.1.
Remark 1.
It is established in [22] that the H k of AM solves min H k (cid:107)
H
k − β k
I (cid:107) F s.t.
H k R k =− X k and the H k of AM-I satisﬁes H k
= B − 1 k , where B k solves min B k (cid:107) B
k − β − 1 k I (cid:107) F s.t.
B
k X k =− R k .
Thus, both AM and AM-I are multisecant methods.
Besides, from (4) and (6) , the approximatedinverse Hessian matrices H k are generally not symmetric.
The Min-AM methods
In each iteration, AM( m ) has to store two matrices X k ,
R k ∈ R d × m , which dramatically increasesthe memory burden in large-scale problems.
To reduce the memory requirement, we consider theminimal memory case, i.e. m = 1 .
The proposed Min-AM is a variant of AM( 1 ) and the convergenceof Min-AM is established in this section.
3.1 The basic Min-AM
In this section, we consider the simple case of (1): the objective function is f ( x ) =
12 x T Ax − b T
xwhere
A is SPD.
Let s k :=
∆ x k − 1 , y k :
=
∆ r k − 1 , and choose m = 1 in (4).
The approximatedinverse Hessian of AM(1) is H k =
β k
I − ( s k
+ β k y k )
( y T k y k ) − 1 y T k , which is non-symmetric.
However, the Hessian matrix ∇ 2 f is naturally symmetric.
To achieve a better approximation of ∇ 2 f ,we ﬁrst recursively modify the vector pair s k , y k to incorporate more information from the previousiterations.
Let p 1 = ∆ x 0 , q 1 = ∆ r 0 ∈ R d .
For k ≥ 2 , we construct p k and q k by
[Equation]
where ζ k =
( p T k − 1 q k − 1 ) − 1 ∆ r T k − 1 p k − 1 , assuming p T k − 1 q k − 1 (cid:54) = 0 .
Then, inspired by the Two-Grid 122 Cycle method [52, Algorithm 13.2] in the multigrid techniques [11], we add an extra projection stepto (3), and the resulting scheme is
[Equation]
where r (1) k :=
r k − q k Γ (1) k and β k > 0 .
Deﬁne r (2) k
= r (1)
k − β k Ar (1) k and r (3) k
= r (2) k − q k Γ (2) k .
Itcan be veriﬁed that r (1) k , r (2) k , and r (3) k are equal to the residuals at x (1) k , x (2) k , and x k +1 , respectively.
The coefﬁcients Γ (1) k and Γ (2) k are determined by applying the Galerkin’s condition of AM-I:
[Equation]
Assume p T k q k (cid:54) = 0 .
The scheme formulated by (8a)-(8c) gives the x k +1 by
[Equation]
leading to the basic Min-AM.
The derivation of (10) is in Appendix B.1.
It is worth mentioningthat the approximated inverse Hessian H k in (10) is symmetric.
Due to the space limit, we give thedetails of the basic Min-AM in Algorithm 2 in Appendix B. Deﬁne P k
= ( p 1 , p 2 , . .
. , p k ) , Q k =( q 1 , q 2 , . .
. , q k ) , we summarize the properties of the basic Min-AM in the next theorem.
Theorem 1.
Let f ( x ) =
12 x T Ax − b T x where A is SPD and { x k } be the sequence generated by thebasic Min-AM for solving (1) .
x ∗ is the exact solution.
Then, the following properties hold:
(i) (cid:107) p k (cid:107) 2 > 0 , range( P k ) =
K k ( A, r 0 ) , range( Q k ) =
A K k ( A, r 0 );
(ii) Q k = − AP k , p i ⊥
q j (1 ≤ i (cid:54) =
j ≤ k );
(iii) r (1) k ⊥ range( P k ) and x (1) k
= x 0 + z k , where z k = arg min
z ∈K k
( A,r 0 ) (cid:107) x 0 + z − x ∗ (cid:107) A .
Moreover, if
(cid:107) r (1) k
(cid:107)
= 0 , then x k +
= x
∗
The proof is deferred to Appendix B.2.
The property (i) ensures p T k q k (cid:54)
=
during the iterations, soζ k and the update (10) are well deﬁned.
The property (iii) indicates x (1) k
= x CG k , where x CG k is thek -th iterate of the conjugate gradient (CG) method [31].
Also, due to Proposition 1 in Appendix B.2,the intermediate iterate ¯ x k of AM-I( ∞ ) satisﬁes ¯ x k = x CG k , so the basic Min-AM is essentiallyequivalent to the full-memory AM-I when solving unconstrained strongly convex quadratic problems.
In this sense, Min-AM can incorporate more historical information than AM(1).
Remark 2.
Although Min-AM and CG are equivalent for solving SPD linear systems, their numericalperformance has a large difference for general nonlinear programming as shown in our experiments.
Remark 3.
The H k in (10) is reminiscent of the memoryless BFGS method [4], but the key differencelies in the construction of p k and q k in (7) .
In memoryless BFGS, there are no correction terms( ζ k ≡ 0 ), and it is generally not equivalent to BFGS [44] and CG.
See Appendix A.3 for more details.
3.2 The restarted Min-AM
For general nonlinear optimization, since global convergence may be unavailable for Min-AM asa counter example exists for AM [40], we introduce the restarting strategy to the basic Min-AMand establish the local convergence analysis.
Our purpose is to prove the faster convergence of therestarted Min-AM than that of gradient descent (GD), and sharpen the existing results of AM [58, 21].
Restarting Strategy.
Let m k be the number of iterations from the last restart.
Min-AM restarts, i.e.,setting m k = 0 and p k = q k = 0 , when any of the following conditions is violated:
[Equation]
where m is a constant positive integer, and 0 < τ < 1 , η > 0 are constants.
In the interval betweentwo successive restarts, the iterations are the basic Min-AM iterations.
The check (11a) is to limit the number of high-order terms of errors accumulated during iterations.
The check (11b) is for the numerical stability of Min-AM since p T k q k appears in the denominatorsin (10).
The check (11c) is to control divergence.
The algorithm is shown in Algorithm 1.
Let
B ρ ( x ) :=
{ y ∈ R d |(cid:107) y − x (cid:107) 2 ≤ ρ } denote the ball with centre x and radius ρ .
We impose the 162 following assumptions on f that are standard in the convergence analysis of quasi-Newton methods[44] and the regularized nonlinear acceleration method [55].
Assumption 1.
f : R d → R is twice Lipschitz continuously differentiable in a neighbourhood of alocal minima x ∗ , and the Hessian matrix A :
= ∇ 2 f ( x ∗ ) is SPD.
Input :
[Equation] [Equation]
[Equation] [Equation]
1: p
[Equation] [Equation]
[Equation]
2: for k = 0 , 1 , . . .
, until convergence do 3: r
[Equation]
=
[Equation] [Equation]
) 4: if m k > m or (cid:107) r k (cid:107)
[Equation]
> η (cid:107)
[Equation]
(cid:107) 2 then 5: m k =6: end if 7: if m k > 0 then 8:
[Equation]
[Equation] [Equation]
11: if | p T k q k | < τ | p T k − m k +
q k − m k +
| then 12: m k =13: end if 14: end if 15: if m k =
then 16: p
[Equation]
=
[Equation]
=
[Equation]
d 17: end if 18:
[Equation]
19: m
[Equation]
m
[Equation]
+ 1 20: end for 21: return x k
From Assumption 1, there exist positive constants ˆ ρ, µ, L , and ˆ κ such that for all x ∈ B ˆ ρ ( x ∗ ) ,µ ≤ y T ∇ 2 f ( x ) y y T y ≤ L, for all y ∈ R d and y (cid:54)
= 0 , (12a) (cid:107)∇ f ( x ) − ∇ 2 f ( x ∗ )( x − x
[Equation]
Here, µ is the strong convexity constant, and L, ˆ κ are Lipschitz constants of ∇ f and ∇ 2 f respectively.
With the restarting strategy, we apply a multi-step analysis and has the following convergence theory:
Theorem 2.
Suppose that the Assumption 1 holds.
Let { x k } be the sequence generated by the 170 restarted Min-AM for solving (1) and θ k = max {| 1 − β k L | , | 1 − β k µ |} ≤ θ for a constant θ ∈ (0 , 1) .
Then there exists a positive constant ρ ≤ ˆ ρ such that for any x 0 ∈ B ρ ( x ∗ ) , we have
[Equation]
Moreover, if m k = d , then (cid:107) x k +1 − x ∗ (cid:107)
= ˆ κ O ( (cid:107) x k − m k − x ∗ (cid:107) 22 ) .
The proof is in Appendix C.1, where a more general theorem (Theorem 6) is given.
The result from
Theorem 2 suggests that the restarted Min-AM method has a local linear-quadratic convergence rate,and the rate has optimal dependence on the condition number.
Compared to the existing results thatshow the local linear convergence rate of AM is no worse than that of gradient descent (GD) method[58, 16], the bound (13) clearly improves the convergence rate of GD.
Remark 4.
Classical nonlinear CG methods [17] are also memory-efﬁcient with recursively modiﬁedsearching directions, but the step size relies on line-search or Hessian-vector products which can betime-consuming for solving large-scale optimization problems.
Remark 5.
The momentum based methods, e.g. Nesterov’s accelerated gradient (NAG) method [43],also achieve the optimal convergence rate if the parameters µ and L are known.
In restarted Min-AM,we give an economical method that approximates the best step size 2 / ( L + µ ) in the next subsection.
3.3 Eigenvalue estimates and the choice of mixing parameter
The mixing parameter β k is critical for AM and AM-I, and an improper choice of β k
can leadto divergence [22].
Fortunately, the spectrum of the Hessian can be economically estimated as a by-product of Min-AM.
To better explain the idea, we consider min f ( x ) =
12 x T
Ax − b T x where
A is SPD.
We apply a projection method [53] to estimate the eigenvalues:
[Equation]
where v ∈ R d is the approximate eigenvector sought in the Krylov subspace K k
( A, r 0 ) , and λ ∈ Ris an eigenvalue estimate, which is known as a generalized Ritz value [41].
Our goal is to compute λ .
Since range( P k ) =
K k ( A, r 0 ) as shown in Theorem 1, we can select v = P k y, y ∈ R k .
Then theprojection condition (14) leads to
[Equation] [Equation]
From the construction of P k and Q k , we can derive the three-term recurrence relations (see details in
Appendix D.1): there exists a tridiagonal matrix ¯ T k ∈ R ( k +1)
× k , such that
[Equation]
Then, we have P T k AAP k
= P T k
AP k
+1 ¯ T k .
By Theorem 1, P T k
Ap k +1 =
− Q T k p k +1 =
Thus,
[Equation]
where T k ∈ R k × k is obtained from ¯ T k by deleting its last row.
By (15) and (17), we obtain T k
y
= λy,which indicates that the generalized eigenvalue problem (15) is reduced to solving the eigenvalues ofa tridiagonal matrix T k .
Moreover, the three elements of the k -th column of ¯ T k are easy to obtainfrom the coefﬁcients Γ (1) k , Γ (2) k , β k − 1 , and β k on the ﬂy (see details in Appendix D.1):
[Equation]
where Γ (2)0 := 0 .
For solving general nonlinear optimization with the restarted Min-AM, suppose thatm k ≥ 1 .
We can also construct a tridiagonal matrix T k ∈ R m k × m k starting from the ( k − m k ) -thiteration with the nonzero elements deﬁned by (18) (see Deﬁnition 2 in Appendix D.2).
We use theeigenvalues of T k as the eigenvalue estimates of the Hessian.
Theorem 3.
Suppose that Assumption 1 holds, and { x k } is the sequence generated by the restarted 205 Min-AM for solving (1) .
Assume that β j ∈ [ β, β (cid:48) ] ( ∀ j ≥ 0 ) for some positive constants β and β (cid:48) , 206 and there are positive constants η 0 , τ 0 such that (cid:107)∇ f ( x j ) (cid:107) 2 ≤ η 0 (cid:107)∇ f ( x 0 ) (cid:107) 2 (0 ≤ j ≤ k + 1) ,| 1 − Γ (1) j | ≥ τ 0 (1 ≤ j ≤ k ) ; λ is an eigenvalue of T k constructed by restarted Min-AM.
Let Θ ( m k )denote the set of Ritz values computed by an m k -step
A -norm based Lanczos algorithm (Algorithm 3in Appendix D.2) for ∇ f 2 ( x ∗ ) with starting vector ∇ 2 f ( x ∗ )( x ∗ − x k − m k ) .
Then there is a positiveconstant ρ ≤ ˆ ρ such that for any x 0 ∈ B ρ ( x ∗ ) , we have
[Equation]
[Equation]
The proof is in Appendix D.2.
Since the Ritz value ˆ λ approximates a true eigenvalue of ∇ 2 f ( x ∗ )
Theorem 3 suggests that the restarted Min-AM can give reliable eigenvalue estimates if (cid:107) x 0 − x ∗ (cid:107) 2 issufﬁciently small.
At the k -th iteration, where m k ≥ 2 , let ˜ µ and ˜ L be the eigenvalues of T k − 1 withthe smallest absolute value and the largest absolute value, respectively.
We use | ˜ µ | and | ˜ L | to estimateµ and L .
Then we can take the mixing parameter as β k = 2 / ( | ˜ µ | + | ˜ L | ) to estimate the optimal value2 / ( µ + L ) .
The total computational cost is O ( m 3 k − 1 ) ﬂops as it requires solving the eigenvaluesof T k − 1 [24].
Since m k (cid:28) d in practice, this strategy can be a useful and economical option.
Remark 6.
The spectrum of Hessian is important for determining the step sizes.
Once we obtain theestimates of µ and L by Min-AM, we can apply them to many ﬁrst-order methods.
3.4 The stochastic Min-AM
For many applications in deep learning, the large data size prohibits the evaluation of the full gradient,and the stochastic optimization stands out as a solution.
Speciﬁcally, we consider developing astochastic version of Min-AM to solve the nonconvex optimization problem: [Equation]
[Equation]
where f i : R d → R may be nonconvex.
Since T can be extremely large, we sample a mini-batch
S k ⊆
[ T ] := { 1 , 2 , . .
, T } and use f S k ( x k ) :
= 1 n k (cid:80) i ∈ S k f i ( x k ) to estimate the full gradient∇ f ( x k ) , where n k :=
| S k | is the batch size.
Then, r k := −∇ f S k ( x k ) is an estimate of −∇ f ( x k ) .
Due to the noise in the gradient estimates and the nonconvexity of f , a direct application of thedeterministic Min-AM to (20) may be problematic.
For example, the searching direction may benot a descent direction since H k can be indeﬁnite.
Hence, we should introduce some fundamentalmodiﬁcations to the basic Min-AM.
Regularization.
Inspired by [23], we deﬁne the function
[Equation]
Let Φ k :=
Φ( p k , q k , δ (2) k ) for some δ (2) k > 0 .
We substitute Φ k for the p T k q k appeared in (10) anddeﬁne ρ k =
Φ † k , i.e. ρ k = 0 if Φ k = 0 and ρ k =
Φ − 1 k if Φ k (cid:54) = 0 .
The resulting regularized Min-AMupdate is x k +
= x k + H Ak r k , where
[Equation]
Damping.
We propose a specialized damping technique for Min-AM, which interpolates the updatesof stochastic gradient descent [49] (SGD) and Min-AM, and is described as
[Equation]
where x Gk is the SGD update, x Ak is the Min-AM update, and α k is the damping parameter.
It canbe seen that SGD and Min-AM serve as the predicted step, and x k +1 is a weighted average of them.
Moreover, the inverse Hessian approximation is
[Equation]
provided that α k ∈ [0 , 1) and H Ak (cid:23) 0 .
By incorporating damping and regularization, we obtain the stochastic Min-AM (sMin-AM), and thealgorithm is summarized in Algorithm 4 in Appendix E.
To analyze the convergence properties, weﬁrst impose the following assumptions on the objective function f .
Assumption 2.
f : R d → R is continuously differentiable, and f ( x ) is lower bounded by a realnumber f low for ∀ x ∈ R d . ∇ f is globally L -Lipschitz continuous.
Assumption 3.
At each iteration k , the gradient estimate ∇ f ξ k ( x k ) satisﬁes E
[Equation]
k [ ∇ f
[Equation]
k ( x k )]
=∇ f ( x k ) , E ξ k [ (cid:107)∇ f ξ k ( x k ) − ∇ f ( x k ) (cid:107) 22 ] ≤ σ 2 , where σ > 0 , and ξ k ∈ [ T ] , k = 0 , 1 , . .
areindependent samples, and for a given k , the random variable ξ k is independent of { x j } kj =0 .
The diminishing condition about β k is
[Equation]
With chosen β k , we choose the α k and δ (2) k of sMin-AM to meet the following conditions:
Assumption 4.
Given constants µ ∈ (0 , 1) , C 1 > 0 , and C 2 > 0 , we have α k ∈ [0 , min { C 1 β 1 / 2 k , 1 −µ } ] , ρ k ≤ 0 , and − ( ρ k /β k ) (cid:107)
p k
(cid:107) 22 − 2 ρ k (cid:107)
p k
(cid:107) 2 (cid:107) q k (cid:107) 2 + ρ 2 k (cid:107)
p k
(cid:107) 22 (cid:107) q k (cid:107) 22 ≤ C 2 .
We establish the convergence results of sMin-AM for nonconvex stochastic optimization and theproofs can be found in Appendix E.2 and Appendix E.3.
Theorem 4.
Suppose that Assumptions 2-4 hold and { x k } is the sequence generated by sMin-AMwith batch size n k ≡ n ≤ T .
If β k ∈ (0 , µ 2 L
[Equation]
1+ C 2 ) 2 ] and satisﬁes (25), then
[Equation] [Equation]
26) and for all k , there exists M f > 0 such that E [ f ( x k )] ≤ M f .
If E ξ k
[ (cid:107)∇ f ξ k ( x k ) (cid:107) 22 ] ≤ M g , ∀ k where M g > 0 is a constant, we have
[Equation]
[Figure]
Theorem 5.
Suppose that Assumptions 2-4 hold.
{ x k } N − 1 k =0 are the ﬁrst N iterations generatedby sMin-AM with batch size n k ≡ n ≤ T , and β k = min {
[Equation]
Dσ √ N } , where ˜ D is aproblem-independent constant.
R is a random variable following P R ( k ) :
= Prob { R = k } = 1 /N ,k = 0 , . . . , N − 1 .
Then
[Equation]
where D f := f
( x 0 ) − f low and the expectation is taken with respect to R and { S j } N − 1 j =0 .
To ensure
E [ (cid:107)∇ f ( x R ) (cid:107) 22 ] ≤ (cid:15) , the number of iterations is O (1 /(cid:15) 2 ) .
Remark 7.
The Theorem 4 establishes the global convergence of sMin-AM for solving the problem(20) , and Theorem 5 indicates that sMin-AM attains the O (1 /(cid:15) 2 ) iteration complexity which isasymptotically optimal for optimization methods based on a stochastic ﬁrst-order oracle [42].
Experiments
We ﬁrst tested the basic Min-AM in solving strongly convex quadratic optimization (cf.
Theorem 1)and the restarted Min-AM on logistic regression (cf.
Theorem 2).
Then, we applied the sMin-AMto train deep neural networks on CIFAR [37] and ImageNet [19].
Details about the experimentalsettings and hyperparameters can be found in Appendix F.
Strongly convex quadratic optimization.
As shown in Figure 1(a), the ﬁrst intermediate step x (1) kof Min-AM exactly matches the k -th iterate of CG, which veriﬁes Theorem 1.
The eigenvalueestimates computed by Min-AM also well approximate the true eigenvalues of the Hessian matrix.
Regularized logistic regression.
We conducted the regularized logistic regression on the datasets“madelon” and “a9a” from LIBSVM [14].
The compared methods were GD, nonlinear CG (NCG)with line-search, Nesterov’s accelerated gradient (NAG), L-BFGS( m ), AM( m ), and the ST-AM.
Weapplied the standard k -step Lanczos algorithm
[24] to compute 100 Ritz values of ∇ 2 f ( x ∗ ) , in whichthe smallest and largest ones were used for the setting of µ, L in NAG.
The results in Figure 1 showthat Min-AM is competitive with AM(20).
It converges much faster than AM(1), L-BFGS(1)
(i.e. memoryless BFGS), NCG, and NAG, and outperforms ST-AM.
Also, in both datasets, the eigenvalueestimates from Min-AM are accurate enough to approximate the largest and the smallest Ritz values,even though the objective function is non-quadratic.
[Table] [Table]
[Table] [Table]
[Figure]
80 Epochs
T e s t
A cc u r ac
y %
SGDMsMin-AM 8075 75.8176.
(c) Test accuracy
Experiments on CIFAR.
We applied sMin-AM to train VGG
[56], ResNet18/20/44/56 [29],
WideResNet16-4 [65] (abbr.
WRN16-4) on CIFAR-10, and ResNet18, ResNeXt50 [63], DenseNet121[33] on CIFAR-100.
The baseline optimizers were SGDM [48], Adam [35], AdaHessian
[64],stochastic AM (SAM) [61], and ST-AM [62].
As shown in Table 1(a), the sMin-AM improves
SAM(1)
and has comparable accuracy to SAM(10).
By setting the accuracy of SGDM as baseline,
Table 1(b) shows the memory, per-epoch time, and the total time for an optimizer to achieve acomparable accuracy to SGDM (more details can be found in Table 2 in Appendix F.3.2).
sMin-AMsigniﬁcantly reduces the memory cost of SAM(10) and the total training time is less than SGDM.
Experiments on ImageNet.
We trained ResNet50 on ImageNet with SGDM and sMin-AM.
Figure 2shows that the training process of sMin-AM is faster than SGDM.
sMin-AM can achieve both highertrain accuracy and higher test accuracy in fewer epochs.
Conclusion
In this paper, we propose Min-AM, a variant of AM(1) with minimal memory size.
Min-AM onlystores one recursively modiﬁed vector pair and is essentially equivalent to the full-memory Type-I
AM in strongly convex quadratic optimization.
We establish the convergence properties of Min-AMunder deterministic and stochastic settings.
For deterministic optimization, we also propose aneconomical method to estimate the spectrum of Hessian, which can be used to adaptively choose themixing parameters.
The experimental results validate the effectiveness of our methods.
A Comparisons with conventional methods
A.1 Relationship between AM-I, Min-AM, and Newton’s method . . . . . . . . . . . .
A.2 Comparison with the conjugate gradient method . . . . . . . . . . . . . . . . . . .
A.3 Comparison with the BFGS method . . . . . . . . . . . . . . . . . . . . . . . . .
A.4 Comparison with the momentum-based method . . . . . . . . . . . . . . . . . . .
A.5 Comparison with related variants of Anderson mixing . . . . . . . . . . . . . . . .
B Details of the basic Min-AM
B
.1 Derivation of the basic Min-AM
B.
Proof
of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Convergence analysis . . .
C Details of the restarted Min-AM
C.1 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D Details of the eigenvalue estimation procedure
D.1 Analysis of the quadratic case . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E Details of the stochastic Min-AM
E.1 Some useful results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.2 Proof of Theorem 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3 Proof of Theorem 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F Experimental details
F.
Strongly convex quadratic problem . .
F.
Regularized logistic regression problem . .
F.
Deep neural network training problem . . . . . . . . . . . . . . . . . . . . . . . .
F.3.
Hyperparameter setting of the stochastic Min-AM . . . . . . . . . . . . .
F.3.
Experiments on CIFAR-10 and CIFAR-100 . . . . . . . . . . . . . . . . .
F.3.
Experiment of training ResNet50 on ImageNet . . . . . . . . . . . . . . . .
F.3.
An additional experiment: adversarial training . . . . . . . . . . . . . . .
G
{+Limitations 
+}
A Comparisons with conventional methods
We compare the proposed Min-AM method with other related methods, including Newton’s method,the (nonlinear) conjugate gradient method, BFGS method, momentum-based method, and some othervariants of Anderson mixing.
A.1 Relationship between AM-I, Min-AM, and Newton’s method
We ﬁrst give a new interpretation of the AM-I method here, which reveals the relationship between
AM-I and Newton’s method.
Recall that the historical information is stored in X k ,
R k ∈ R d
× m .
We have the decomposition of thesolution space by R d = range( X k ) (cid:76) range( X k ) ⊥ .
Deﬁne V k ∈ R d × ( d − m ) whose columns forman orthonormal basis of range( X k ) ⊥ .
Then for any x ∈ R d , we have x = x k − X k γ − V k η , whereγ ∈ R m , η ∈ R d − m .
As a result,
[Equation]
Suppose that f is twice differentiable and X T k ∇ 2 f ( x k )
X k is nonsingular.
We apply Newton’smethod in the low dimensional subspace range( X k ) followed by a gradient descent with stepsize β kin the complementary subspace range( X k ) ⊥ .
Then
[Equation]
where γ k =
( X T k ∇ 2 f ( x k )
X k ) − 1 X T k ∇ f ( x k ) , and η k = β k V T k ∇ f
(¯ x k ) .
When m (cid:28) d , the scheme(30) is more economical than applying the Newton’s method in the whole solution space R d .
Now, consider the simple case that f is a quadratic function: f ( x ) = 12 x T Ax − b T x .
The residualr k = −∇ f ( x k ) =
b − Ax k .
Then r k − r k − 1 = − A ( x k − x k − 1 ) , which implies R k = − AX k .
It 545 follows that γ k =
( X T k R k ) − 1 X T k r k .
Also, ¯ r k :=
r k − R k γ k
= −∇ f
(¯ x k ) .
So ¯ r k ⊥ range( X k ) due 546 to the choice of γ k , i.e., a kind of the Galerkin’s projection condition [52].
Let U k be the matrixwhose columns form an orthonormal basis of range( X k ) .
Then V k V T k
= I − U
k U T k
and U
T
k ¯ r k
=
For the gradient descent step, it has η k =
− β k V T
k ¯ r k , which implies
[Equation]
namely a mixing step.
Therefore, the AM-I method coincides with the scheme (30) for minimizinga quadratic function.
Since Min-AM is essentially equivalent to the full-memory AM-I whenminimizing a strongly convex quadratic function, we know Min-AM is also closely related to thescheme (30).
In the general case, if the objective function f can be well approximated by a quadraticfunction in a local region around the optima, it is expected that AM-I and Min-AM can behavesimilarly to the scheme (30) when the iterates enter this local region.
A.2 Comparison with the conjugate gradient method
The conjugate gradient (CG) method is a classical method for solving SPD linear systems and solvingunconstrained optimization.
Consider the problem
[Equation]
where f : R d → R is continuously differentiable and bounded from below.
The nonlinear CG (NCG)generates sequence { x k } from an initial guess x 0 ∈ R d , by the update scheme
[Equation]
where the step size α k is usually obtained by a line search, and the searching direction p k isconstructed by
[Equation]
Here, r k :=
−∇ f ( x k ) and the β k is the momentum term.
There are several choices of β k , for example, the Fletcher-Reeves variant β k = (cid:107) r k +1 (cid:107) 22 (cid:107) r k (cid:107) 22 and the Polak-Ribière variant β k
=
r
T
k
+
( r k +1 − r k ) (cid:107) r k
(cid:107).(See [44].)
However, the step size α k is not easy to obtain.
If f is a quadratic function, let A be the
Hessian of f .
The computation of α k involves a matrix-vector product, e.g.,
[Equation]
which ensures r k +1 ⊥ p k .
If ﬁnite difference is used to compute Ap k , it requires two gradientevaluations.
If f is a general nonlinear function, a line search is necessary to ensure the convergenceof NCG:
α k is chosen as an approximate solution to the problem
[Equation] [Equation]
where p k is supposed to be a descent direction.
The d -step quadratic convergence proved in [17],which is similar to our result of the restarted Min-AM, was only established under the assumptionof an exact line search, namely, (33) is exactly solved.
More practical choices of α k require someconditions to be satisﬁed.
For example, the strong Wolfe conditions are
[Equation]
where 0 < c 1 < c 2 < 1 .
Since the line search can incur many times of the backtracking procedure,the total number of gradient evaluations for NCG can be large.
In contrast, our proposed Min-AM method does not need explicit Hessian-vector products or linesearch to determine the step size, while the convergence results (cf.
Theorem 2) match those of
NCG with exact line search.
Therefore, Min-AM is more economical than NCG, and can have betterconvergence than the NCG with inexact line search.
Also, it should be pointed out that even instrongly convex quadratic optimization, Min-AM generally does not generate the same sequence ofiterates as CG: They are equivalent in the sense that x (1) k = x CG k , where x (1) k is the intermediate stepin (8a).
A.3 Comparison with the BFGS method
The BFGS constructs the approximate inverse Hessian by solving
[Equation]
34) where s k :=
x k − x k − 1 , y k := ∇ f ( x k ) − ∇ f ( x k − 1 ) .
The norm (cid:107) · (cid:107) F ( W ) is the weighted Frobeniusnorm (i.e., (cid:107) X (cid:107) 2 F ( W ) :
= (cid:107) W 1 / 2 XW 1 / 2 (cid:107) F for a matrix X ∈ R d × d ) and the weight matrix
Wsatisﬁes Ws k
= y k .
The solution is
[Equation]
Note that the H k ∈ R d
×
d is recursively constructed and can be dense during the later iterations.
Toreduce memory overhead, the limited-memory BFGS [38] is often used.
In memoryless BFGS, theprevious approximate Hessian is always reset to I d , i.e.,
[Equation]
[Equation]
Memoryless BFGS has minimal memory size in L-BFGS, but has no equivalence to BFGS.
It ispointed in [44] that for the update x k +1 =
x
k − α k H
k ∇ f ( x k ) of memoryless BFGS, if α k is chosen 592 by an exact line search, then memoryless BFGS is equivalent to CG for quadratic function.
However,an exact line search can incur prohibitive cost in practice.
The Min-AM method has a similar form to that of memoryless BFGS, but uses a recursivelyconstructed vector pair.
Min-AM is essentially equivalent to the full-memory AM-I and CG forminimizing strongly convex quadratic functions.
In fact, let P k
=
( p 1 , . . .
, p k ) , Q k = ( q 1 , . . .
, q k ) ,and deﬁne
[Equation]
assuming P T k Q k is nonsingular.
We call the iterations deﬁned by x k +
= x k + H Ak
r k as Scheme A.
Clearly, H Ak Q k = − P k holds.
Moreover, it can be proved by direct computation (using the propertiesin Theorem 1) that for strongly convex quadratic optimization, if ﬁxed β k is used, i.e., β k ≡ β , whereβ is a constant, we have
[Equation]
starting from H A 0 :=
βI .
Hence, the H Ak in (37) solves
[Equation]
[Equation]
where the weight matrix satisﬁes Wp k
= − q k .
The (39) is similar to (34), but based on modiﬁedvector pairs.
(Note that we use r k − r k − 1 = − ( ∇ f ( x k ) − ∇ f ( x k − 1 )) to construct q k , so there 605 is a difference of sign.)
It can also be veriﬁed that for strongly convex quadratic optimization,
H k r k = H
Ak r k , where H k is deﬁned in (10).
So the iterations of Min-AM are identical to those of
Scheme A in this case.
In this sense, though only using one vector pair, Min-AM implicitly constructs
H Ak which can well approximate the inverse Hessian.
A.4 Comparison with the momentum-based method
The momentum-based method is similar to the scheme deﬁned in (31) and (32), but the momentumβ k is different from that in CG.
We take the Nesterov’s accelerated gradient (NAG) method as anexample.
The update scheme of NAG is deﬁned as
[Equation]
where L and µ are the Lipschitz constant and the strong convexity constant of ∇ f , respectively.
Unlike Min-AM, NAG generally does not form a symmetric approximation to the Hessian.
In
Figure 3, we show the convergence behaviours of NAG with different settings of µ and L .
It is foundthat the setting of L and µ can have a large effect on the convergence.
On the contrary, Min-AM formssymmetric Hessian approximations and can give useful information of L and µ via an economicaleigenvalue estimation procedure.
[Figure]
A.5 Comparison with
related variants of Anderson mixing
Compared with the original AM, AM-I, the regularized nonlinear acceleration scheme [54], andthe stochastic AM [61], the proposed Min-AM has the minimal memory size, the same as that of
AM(1).
Nonetheless, Min-AM incorporates more historical information than AM(1) and AM-I(1),and is equivalent to the full-memory AM-I in solving strongly convex quadratic optimization.
Theconvergence analysis in Section 3.4 also shows that the stochastic Min-AM has similar convergenceto that of the stochastic AM in theory.
The short-term recurrence AM (ST-AM) [62] stores two vector pairs and is equivalent to the fullmemory AM in solving strongly convex quadratic optimization, but the approximated Hessian from
ST-AM is generally not symmetric.
It is also worth pointing out that for AM and ST-AM, theirexisting theoretical results do not show the signiﬁcantly better convergence rate than the simplegradient descent method for unconstrained optimization.
Our proposed Min-AM further reduces the memory size of ST-AM and the faster convergence thangradient descent for general nonlinear optimization is rigorously justiﬁed in theory.
B Details of the basic Min-AM
The procedure of the basic Min-AM is described in Algorithm 2.
Next, we give the derivation of thebasic Min-AM, and prove the related theoretical properties.
Algorithm 2 Min-AM for strongly convex quadratic optimization
Input : x 0 ∈ R d , β k > 0 , 0 < max _ iter ≤ d 1: p 0 , q 0 = 0 ∈ R d 2: for k = 0 , 1 , . . .
, max _ iter do 3: r k = −∇ f ( x k ) 4:
if k > 0 then 5:
[Equation]
r
[Equation]
(Compute ∆ x k − 1 , ∆ r k − 1 ) 6:
[Equation]
7: p
[Equation] [Equation]
( p
[Equation]
) 8: end if 9:
[Equation]
k 10:
[Equation]
(Projection step: r (1) k
[Equation] [Equation]
(Mixing step) 12:
[Equation] [Equation]
(Projection step: r k +1 ⊥ p
[Equation]
) 14: if (cid:107) r (1) k (cid:107)
=
then 15: break 16: end if 17: end for 18: return x k
B.1 Derivation of the basic Min-AM
We give the derivation of the update scheme (10) for solving the quadratic optimizationmin x ∈ R d f ( x ) = 12 x T
Ax − b T x here.
Recall that the steps in one iteration of the basic Min-AM are:
[Equation]
41) where r (1) k
= r k − q k Γ (1) k and β k > 0 .
Deﬁne r (2) k
= r (1)
k − β k Ar (1) k and r (3) k
= r (2) k − q k Γ (2) k .
TheΓ (1) k and Γ (2) k are determined by imposing the projection conditions:
[Equation]
Here, we deﬁne p 1 = ∆ x 0 , q 1 = ∆ r 0 , and for k ≥ 2 , the construction of p k , q k at the beginning ofthe k -th iteration is
[Equation]
where ζ k =
( p T k − 1 q k − 1 ) − 1 ∆ r T k − 1 p k − 1 , assuming p T k − 1 q k − 1 (cid:54) = 0 .
We ﬁrst prove that
[Equation]
42) by induction.
By the deﬁnition of f and r k = −∇ f ( x k ) , it is clear that r k − r k − 1 = − A ( x k − x k − 1 ) .
So (42) holds for k = 1 .
For k ≥ 2 , suppose that (42) holds for k − 1 .
Then q k = ∆ r k − 1 − q k − 1 ζ k =
[Equation]
+
[Equation]
= − Ap k , which completes the induction.
Suppose that p T k q k (cid:54) = 0 .
we have Γ (1) k
= ( p T k q k ) − 1 p T k r k .
For Γ (2) k , the exact solution is
[Equation]
[Equation]
due to p T k r (1) k = 0 , the symmetry of A , and (42).
As a result,
[Equation]
The corresponding inverse Hessian approximation is
[Equation]
The reason to use the projection condition of AM-I. Note that the Galerkin’s projection conditionof AM-I is used to determine Γ (1) k and Γ (2) k .
On the other side, if the Galerkin’s condition of theoriginal AM is used to determine Γ (1) k and Γ (2) k , the scheme (41) leads to
[Equation]
which is not a practical method since Aq k needs to be explicitly computed.
B.2 Proof of Theorem 1
Recall that the strongly convex quadratic optimization is formulated as
[Equation]
[Equation]
where A ∈ R d × d is SPD, b ∈ R d .
Solving (47) is equivalent to solving the SPD linear system
[Equation]
We ﬁrst state the relationship of the AM-I with the full orthogonalization method [51] (FOM) in thefollowing propositions.
Let x FOM k and r FOM k
[Equation]
k denote the k -th FOM iterate and residual,respectively.
Deﬁne e j :=
, 1) T ∈ R j for j ≥ 1 .
The main results of the full-memory AM-Iare stated in Proposition 1 and Proposition 2.
(Proposition 1 is a known result in [59].)
Proposition 1 (General linear system) .
For solving a general linear system Ax = b with the fullmemory AM-I, suppose that β k > 0 and the ﬁxed-point map is g ( x ) =
( I − A ) x + b .
If the initialpoint of AM-I is x 0 = x FOM, and X T j R j is nonsingular for j = 1 , . . .
, k , then the intermediateiterate ¯ x k satisﬁes ¯ x k = x FOM k .
Proof.
The deﬁnition of the ﬁxed-point map suggests that the residual r k = g ( x k ) − x k =
b − Ax k .
From (5), for j = 1 , . . .
, k , the nonsingularity of X T j R j ensures that each Γ j is uniquely determined.
Thus the updates of AM-I are well deﬁned.
Since X T k R k is nonsingular, we have rank( X k )
=
rank( R k )
=
k .
We ﬁrst show
[Equation]
by induction.
We abbreviate K k ( A, r FOM) as K k in this proof.
First, ∆ x 0 = β 0 r 0 = β 0 r
FOMsince x 1 = x 0 + β 0 r 0 .
If k = 1 , then the proof is complete.
Then,suppose that k > 1 and, as an inductive hypothesis, that range( X k − 1 ) =
K k − 1 .
From (3), we havex k +
= x k + β k r k −
X k + β k R k )Γ k .
Noting that R k
= − AX k , it follows that
[Equation]
Since r 0 ∈ K k − 1 , and by the inductive hypothesis range( X k − 1 )
=
K k − 1 which also implies 675 range( AX k − 1 ) ⊆
K k , we know ∆ x k − 1 ∈ K k .
Thus, range( X k ) ⊆
K k .
Since rank( X k ) =
k , 676 namely dim(range( X k ))
=
dim( K k ) , we have range( X k ) =
K k , thus completing the induction.
As a result, we also have
[Equation]
Recalling that to determine Γ k , we solve the projection condition:
[Equation]
[Equation]
The nonsingluarity of X T k R k ensures that Γ k is uniquely determined by
[Equation]
Also, since r k = b − Ax k = b − A ( x 0 + X k e k ) =
r 0 − AX k e k , we have r k − R k Γ =
r k + AX k Γ =r 0 − AX k e k + AX k Γ = r 0 − AX k ˜Γ , where ˜Γ =
e k − Γ , for ∀ Γ ∈ R k .
So Γ k solves (52) if andonly if ˜Γ k = e k − Γ k solves
[Equation]
According to (49), the condition (54) is equivalent to
[Equation] [Equation]
where range( X k ) =
K k .
Since the initializations are identical, the condition (55) for AM-I is the
Petrov-Galerkin condition for FOM.
Due to the nonsingularity of X T k R k , the solution of (54) is alsounique.
Therefore, we have
[Equation]
In Proposition 1, the assumption that X T k R k is nonsingular is critical to ensure no stagnation occurs atthe k -th iteration for solving a general linear system.
In fact, for SPD linear systems (48) or stronglyconvex quadratic optimization (47), when AM-I breaks down, i.e. X T k R k is singular, AM-I obtainsthe exact solution, as shown in the next proposition.
Proposition 2 (SPD) .
For applying the AM-I to minimize a strongly convex quadratic problem (47),or equivalently, solve an SPD linear system (48), suppose that β k > 0 and the ﬁxed-point map isg ( x ) =
( I − A ) x + b .
If the condition X T k R k is nonsingular holds for 1 ≤ k < s while failing tohold for k = s , where s ≥ 1 , then the residual of AM-I satisﬁes r s =
¯ r s − 1 = 0 .
Proof.
The deﬁnition of g suggests that the residual r k = g
( x k )
− x k
= b − Ax k .
The relation
R k
= − AX k holds during the iterations and the nonsingularity of A implies rank( X k ) =
rank( R k ) .
Since A is SPD and X T k R k = − X T k AX k , it follows that X T k R k being nonsingular ⇔ rank( X k ) =
rank( R k )
=
k .
Hence rank( X k ) =
k holds for 1 ≤ k < s while failing to hold for k = s .
For s = 1 , since the ﬁrst step of AM is x 1 = x 0 + β 0 r 0 , the assumption rank( X 1 ) = 0 implies thatrank( r 0 )
=
rank( X 1 ) = 0 , then r 1 = ¯ r 0 := 0 .
For s > 1 , ∆ x s − 1 = x s − x s − 1 = − X s − 1 Γ s − 1 + β s − 1 ¯ r s − 1 .
The rank deﬁciency of X s implies 703 ∆ x s − 1 ∈ range( X s − 1 ) , which further implies ¯ r s − 1 ∈ range( X s − 1 ) .
So there exists ζ ∈ R s − 1 , such 704 that ¯ r s −
=
X s − 1 ζ .
Due to ¯ r s − 1 ⊥ X s − 1 , we have
[Equation]
Because rank( X s − 1 ) = s − 1 , we have ζ = 0 , which implies ¯ r s − 1 = 0 .
Hence x s = ¯ x s − 1 andr s = ¯ r s − 1 = 0 .
It is also known that FOM method can be simpliﬁed to the conjugate gradient (CG) method [52]
inthis case.
Now we give the proof of Theorem 1.
Proof of Theorem 1 .
The A -norm minimization problem in the property (iii) is equivalent to the
Galerkin condition [52], i.e.
[Equation]
Also, r (3) k
= r k +
=
r (2) k − q k Γ (2) k .
Besides relations (i)-(iii), we add an auxiliary relation here:(iv)
[Equation]
=
[Equation] [Equation]
Q
[Equation]
k ∈ K k +
( A, r 0 ) , where ¯Γ k ∈ R k .
We prove the relations (i)-(iv) by induction.
For k = 1 , since r 0 = r (1) (cid:54) = 0 , we have rank(∆ x 0 )
=
rank( X 1 ) = 1 and rank(∆ r 0 )
=
rank( R 1 )
= 1 .
So p k (cid:54) = 0 and p T k q k = − p T k
Ap k < 0 .
The relation (i) holds.
Since p 1 = ∆ x 0 , q 1 =∆ r 0 , and ∆ r 0 = − A ∆ x 0 , the equality Q 1 = − AP 1 also holds.
Since r 1 = r 0 − β 0 Ar 0 and 718 range( Q 1 ) = span { Ar 0 } , it is clear that r 1 = r 0 − Q 1 ¯Γ 1 ∈ K 2 ( A, r 0 ) , namely relation (iv).
Dueto the projection step (Line 10 in Algorithm 2), r (1) ⊥ range( P 1 ) and r (1) is unique, which isguaranteed by p T1 q 1 (cid:54)
= 0 .
Also, r (1)1 = r 1 − q 1 Γ (1)1 = r 0 − β
Ar 0 − q 1 Γ (1)1 =
r
− Q 1 η 1 , where thelast equality is due to span { Ar 0 } =
range( Q 1 ) .
For r FOM= r 0 − Az 1 , where z 1 ∈ K 1 ( A, r 0 ) , itholds r FOM⊥ K 1 ( A, r 0 )
=
range( P 1 ) .
As a result, both r (1)1 and r FOMare the oblique projectionsof r 0 onto the subspace range( P 1 ) ⊥ along range( Q 1 ) .
Since p T1 q 1 (cid:54) = 0 , the projection exists and isunique, which implies r (1) =
r FOM.
So x (1)1 = x FOM= x 0 + z 1 because their residuals are equaland A is nonsingular.
Hence the relation (iii) holds.
Suppose that k > 1 , and as an inductive hypothesis, the relations (i)-(iv) hold for j = 1 , . .
, k − 1 .
Consider the k -th iteration.
From Line 7 in Algorithm 2, q k ∈ span { ∆ r k − 1 , q k − 1 } , and p k ∈ 728 span { ∆ x k − 1 , p k − 1 } .
Due to p T k − 1 q k − 1 = − p T k − 1 Ap k − 1 (cid:54) = 0 , we have q T k p k − 1 = 0 .
We ﬁrst prove 729 that p k (cid:54) = 0 by contradiction.
If p k = 0 , then from Line 7 in Algorithm 2, ∆ x k − 1 ∈ span { p k − 1 } .
From Line 10, Line 11, and 731 Line 13, we have
[Equation]
[Equation]
So r (1) k − 1 ∈ span { p k − 1 } since ∆ x k − 1 ∈ span { p k − 1 } .
Hence there exists ζ ∈ R , such that r (1) k − 1 =
p k − 1 ζ .
From the Line 10, we know r (1) k − 1 ⊥ span { p k − 1 } , so we have
[Equation]
Since p k − 1 (cid:54) = 0 , it follows that ζ = 0 which implies r (1) k − 1 = 0 .
It is impossible otherwise Algorithm 2has terminated in the ( k − 1) -th iteration.
So p k (cid:54) = 0 .
Moreover p T k q k = − p T k
Ap k
(cid:54)
=
since A is 736 SPD.
Since r (1) k − 1 =
r k − 1 − q k − 1 Γ (1) k − 1 , and r k − 1 ∈ K k
( A, r 0 ) , q k − 1 ∈ range( Q k − 1 ) =
A K k − 1 ( A, r 0 )due to the inductive hypothesis, we have r (1) k − 1 ∈ K k
( A, r 0 ) , which together with (57) and p k − 1 ∈range( P k − 1 ) =
K k − 1 ( A, r 0 ) infers ∆ x k − 1 ∈
K k ( A, r 0 ) .
As a result, p k ∈ span { ∆ x k − 1 , p k − 1 } ⊆
K k ( A, r 0 ) .
So range( P k ) =
range( P k − 1 , p k ) ⊆
K k ( A, r 0 ) .
Moreover, p k / ∈ range( P k − 1 ) .
We 741 prove it by contradiction.
If p k ∈ range( P k − 1 ) , then from Line 7 in Algorithm 2, ∆ x k − 1 ∈ span { p k − 1 , p k } ⊆
range( P k − 1 ) ,which together with (57) leads to r (1)
k − 1 ∈ range( P k − 1 ) .
Hence there exists ξ ∈ R k − 1 such that r (1) k − 1 =
P k − 1 ξ .
From the inductive hypothesis, we know r (1) k − 1 ⊥ range( P k − 1 ) .
So we have
[Equation]
Since − P T k − 1 AP k − 1 = P T k − 1 Q k − 1 is nonsingular due to the inductive hypothesis p i ⊥ q j (1 ≤ i (cid:54) =
j ≤ k − 1) and p T i q i = − p T i Ap i (cid:54) =
0(1 ≤ i ≤ k − 1) as p i (cid:54) = 0 , it follows that rank( P k − 1 ) =
k − 1 .
So P T k − 1 P k − 1 is nonsingular which implies that ξ = 0 .
Then r (1) k − 1 =
P k − 1 ξ = 0 .
But it is impossibleotherwise Algorithm 2 has terminated in the ( k − 1) -th iteration.
Hence p k / ∈ range( P k − 1 ) .
As aresult, range( P k )
=
K k ( A, r 0 ) .
Because ∆ r k − 1 = − A ∆ x k − 1 and q k − 1 = − Ap k − 1 , Line 7 in Algorithm 2 infers q k = − Ap k .
So 751 Q k = − AP k .
Hence range( Q k ) =
A K k ( A, r 0 ) .
To prove p i ⊥ q j for 1 ≤ i (cid:54)
=
j ≤ k , it sufﬁces to show q k ⊥ P
k − 1 .
From the construction of q k in
Line 7 in Algorithm 2 and p T k − 1 q k − 1 (cid:54) = 0 , we know q T k p k − 1 = 0 .
To further prove q k ⊥ range( P k − 2 )( k ≥ 3) , note that
[Equation]
where the second equality is a direct substitution with (57).
Therefore,
[Equation]
where the second equality is due to q k − 1 ⊥ range( P k − 2 ) and A is SPD, and the third equality is dueto r (1)
k − 1 ⊥ range( P k − 1 ) =
K k − 1 ( A, r 0 ) and range( AP k − 2 )
=
A K k − 2 ( A, r 0 ) ⊆
K k − 1 ( A, r 0 ) .
As 758 a result, we obtain
[Equation]
which is due to (58) and q k − 1 ⊥ range( P k − 2 ) .
Therefore, q k ⊥ range( P k − 1 ) .
Since A is SPD, we 760 also have p T
[Equation]
Hence relation (ii) holds in 761 the k -th iteration.
Next, we prove the relation (iv).
We have
[Equation]
k
[Equation]
where the last equality is due to r k −
= r
+ Q k − 1 ¯Γ k − 1 by the inductive hypothesis.
Sincerange( Q k − 1 ) =
A K k − 1 ( A, r 0 ) ⊆
A K k
( A, r 0 ) , q k − 1 ∈ range( Q k − 1 ) ,
Ar 0 ∈
A K k ( A, r 0 ) , 765 range( AQ k − 1 ) ⊆
A 2 K k − 1 ( A, r 0 ) ⊆
A K k
( A, r 0 ) , and Aq k − 1 ∈ range( AQ k − 1 ) , it is clear thatr k
= r 0 + Q k ¯Γ k ∈ K k +
( A, r 0 ) for some ¯Γ k ∈ R k .
The relation (iv) is proved.
Finally, we prove the relation (iii).
For proving r (1) k ⊥ range( P k ) , note that r (1) k ⊥ span { p k } 768 already holds due to the projection step (Line 9 and Line 10 in Algorithm 2) and p T k q k (cid:54) = 0 .
Itsufﬁces to prove r (1) k ⊥ range( P k − 1 ) .
We ﬁrst prove r k ⊥ range( P k − 1 ) .
The projection step 770 (Line 12 and Line 13 in Algorithm 2) at the ( k − 1) -iteration ensures r k ⊥ p
k − 1 .
It remains toprove r k ⊥ range( P k − 2 ) .
Because r k = r (2) k − 1 − q k − 1 Γ (2) k − 1 = r (1) k − 1 − β k − 1 Ar (1) k − 1 − q k − 1 Γ (2) k − 1 ,and r (1) k − 1 ⊥ range( P k − 2 ) , q k − 1 ⊥ range( P k − 2 ) , ( Ar (1) k − 1 ) T P k − 2 =
( r (1) k − 1 ) T ( AP k − 2 ) =
0 asrange( AP k − 2 ) ⊆
K k − 1 ( A, r 0 ) =
range( P k − 1 ) , it follows that r k ⊥ range( P k − 2 ) .
Therefore, r k ⊥range( P k − 1 ) .
Also noting that q k ⊥ range( P k − 1 ) , we have r (1) k
= r k − q k Γ (1) k ⊥ range( P k − 1 ) .
Sor (1) k ⊥ range( P k ) .
To prove x (1) k
= x FOM k := x 0 + z k , where z k ∈ K k
( A, r 0 ) should satisfy r 0 − Az k ⊥ K k
( A, r 0 ) ,ﬁrst we have r k = r
+ Q k ¯Γ k , where ¯Γ k ∈ R k .
Hence r (1) k
= r k − q k Γ (1) k
= r 0 + Q k ¯Γ k − q k Γ (1) k
= r 0 − Q k η k , where η k ∈ R k .
Since P T k Q k is nonsingular, there exists a unique η k ∈ R k such thatr (1) k ⊥ range( P k ) , i.e., r (1) k is the oblique projection of r 0 onto the subspace range( P k ) ⊥ alongrange( Q k ) .
On the other side, for FOM, r FOM k = r 0 − Az k ⊥ K k ( A, r 0 )
=
range( P k ) , so r FOM k isalso the oblique projection of r 0 onto the subspace range( P k ) ⊥ along range( Q k ) .
So r (1) k
= r FOM k ,which further indicates x (1) k
= x
FOM k .
Hence, the relation (iii) holds.
With relations (i)-(iv) being proved in the k -th iteration, we complete the induction.
B.3 Convergence analysis
Since the basic Min-AM is equivalent to FOM (or CG) method in the sense that ¯ x (1) k
= x
FOM k = x CG kfor strongly convex quadratic optimization, where x CG k is the k -th iterate of CG, we can obtain theconvergence result as a corollary of Theorem 1.
Corollary 1.
For solving the strongly convex quadratic problem (47) , let x ∗ be the exact solution,and { x k } is the sequence generated by the basic Min-AM described in Algorithm 2.
Deﬁne θ k =(cid:107)
I − β k A (cid:107) 2 .
Then
[Equation]
(59) where P k denotes the space of polynomials of degree not exceeding k .
Moreover, the algorithm ﬁndsthe exact solution in at most ( d + 1) iterations.
Proof.
From Theorem 1, the classical convergence analysis of the CG method [28, 52], and assumingthat CG and Min-AM are initialized from the same starting point, i
[Equation]
e., x CG 0 = x 0 , we have
[Equation]
Since x (2) k
= x (1) k
+ β k r (1) k
[Equation]
it follows that
[Equation]
Hence,
[Equation]
61) where we use the fact that
[Equation]
(cid:107) 2 .
Noting that x k +
= x (2) k − p k Γ (2) k , r k +
= r (2)
k
+ Ap k Γ (2) k ⊥ p k , it follows that
[Equation]
So
[Equation]
Combining (60), (61), and (64) yields (59).
Since CG iterates at most d steps to obtain x ∗ , the Min-AM will obtain the same solution in at most( d + 1) steps.
C Details of the restarted Min-AM
C.1 Proof of Theorem
For the proof of Theorem 2 about the restarted Min-AM, we use a local quadratic model inspired bythe techniques in [17, 18].
The second-order Taylor expansion ˆ f of f around x ∗ is
[Equation]
where A := ∇ 2 f ( x ∗ ) .
We compare the iterates { x k } generated by Min-AM to the iterates { ˆ x k } generated by applying 809 Min-AM to minimizing the quadratic model (65).
More precisely, we give the following deﬁnition.
Deﬁnition 1.
Let the mixing parameter { β k } be chosen to satisfy β ≤ | β k | ≤ β (cid:48) , where β and β (cid:48) are 811 two positive constants.
The sequences { x k } and { ˆ x } are generated by the following two processes:(1) Process I: Solve the optimization problem (1) with the restarted Min-AM (see Algorithm 1), andthe resulting sequence is { x k } .(2)
Process II: Apply the basic Min-AM to minimize ˆ f ( x ) in each interval between two successive restarts in Process I. Speciﬁcally, let m k , β
k be the same as those in Process I. Deﬁne
ˆ r k = −∇
ˆ f (ˆ x k ) .
Then, ˆ x k = x k , and ˆ p k
= ˆ q k =
∈ R d , if m k = 0 .
For m k ≥ 1 ,
[Equation]
ˆ
[Equation]
=
[Equation] [Equation]
, where ˆ ζ k = (ˆ p T k −
ˆ q k −
[Equation]
† ˆ p T k − 1 ∆ˆ r k − 1 ; deﬁning ˆ r (1) k
= ˆ r k − ˆ q k ˆΓ (1) k , the update of ˆ x k isˆ
[Equation]
, where ˆ Γ (1) k
= (ˆ p T k
ˆ q
[Equation]
† ˆ p T
[Equation]
ˆ r k , ˆ
[Equation]
, ˆ
[Equation]
where ˆΓ (2) k
= β k (ˆ p T k
ˆ q k ) † ˆ q T
k ˆ r (1) k .
[Equation]
By comparing { x k } and { ˆ x } , we ﬁrst have the following lemma.
Lemma 1.
Suppose that Assumption 1 holds for the optimization problem (1) .
For the sequences{ x k } and { ˆ x k } deﬁned in Deﬁnition 1, if (cid:107) x 0 − x ∗ (cid:107) 2 is sufﬁciently small and there exists a positive 818 constant η 0 such that (cid:107)∇ f ( x j ) (cid:107) 2 ≤ η 0 (cid:107)∇ f ( x 0 ) (cid:107) 2 for j = 0 , . .
, k , then
[Equation] [Equation]
Proof.
Besides (67) and (68), We will also prove the following relations.
[Equation]
ˆΓ
[Equation]
Here, for convenience, we deﬁne ζ k = ˆ ζ k
=
if m k = 0 .
We ﬁrst prove (69).
Recall that r k = −∇ f
( x k ) .
Due to (12a), we have
[Equation] [Equation]
Note that (cid:107) r k (cid:107) 2 ≤ η 0 (cid:107) r 0 (cid:107) 2 .
By choosing (cid:107) x 0 − x ∗ (cid:107)
[Equation]
≤ µ ˆ ρ η 0 L , we can ensure
[Equation]
Then (69) holds.
Since (cid:107) r j (cid:107) 2 ≤ η 0 (cid:107) r 0 (cid:107) 2 holds for j = 0 , . . .
, k , (78) implies that a sufﬁciently small(cid:107)
x 0 − x ∗ (cid:107) 2 ensures (cid:107) x k − m k − x ∗ (cid:107) 2 ≤ η 0 Lµ (cid:107) x 0 − x ∗ (cid:107) 2 is sufﬁciently small.
Next, we prove (67),(68), and (70)-(76) by induction.
For k = 0 , we have Γ (1) k = ˆΓ (1) k
= 0 , Γ (2) k = ˆΓ
[Equation] [Equation]
= 0 .
The relations (70)-(74), and (76) clearlyhold.
Also, due to (12b),
[Equation]
22 , namely (67).
Note that x k +
= x (2) k
= x k + β k r
[Equation]
and ˆ x k +
=
ˆ x (2) k
= ˆ x k + β k ˆ r k in this case.
Then(68) and (75) follow from
[Equation]
Suppose that k ≥ 1 , and as an inductive hypothesis, the relations (67), (68), and (70)-(76) hold forj = 0 , . .
. , k − 1 .
Consider the k -th iteration.
If m k = 0 , i.e., a condition in (11a)-(11c) is violated at the beginning of the k -th iteration, thenˆ x k = x k .
The same as the case that k = 0 , (67), (68), and (70)-(76) hold.
Consider the nontrivial case that m k ≥ 1 .
From (11c), it follows that
[Equation]
Therefore,
[Equation]
Since x k ∈ B ˆ ρ ( x ∗ ) , we have
[Equation]
where the second inequality is due to (12b) and (12a), and the last inequality is due to (79) and theinductive hypothesis (68).
Hence, the relation (67) holds.
Next, we prove the relation (70).
Due to the check (11b), with κ := τµ 3 β 2 , we have
[Equation]
for j = k − m k + 1 , . . .
, k .
If m k = 1 , then ζ k = ˆ ζ k = 0 , and (70) holds.
For m k > 1 , we have
[Equation] [Equation]
due to (71) and (cid:107) ∆ r k − 1 (cid:107) 2 ≤ (cid:107) r k (cid:107)
[Equation]
(cid:107) r k − 1 (cid:107) 2 ≤
[Equation]
(cid:107)
[Equation]
x
[Equation]
∗ (cid:107) 2 ) .
Next, if p T k − 1 ∆ r k − 1 (cid:54) = 0 , then
[Equation]
|
[Equation]
where a
[Equation]
=
[Equation]
We have
[Equation]
From (67) and (71),
[Equation]
We also have
[Equation]
Hence,
[Equation]
Combining (83) and (84), we have
[Equation] [Equation]
Note that
[Equation]
86) and
[Equation]
(87) where the existence of the constant c 1 is guaranteed by (86), and the last inequality holds by choosing(cid:107)
x 0 − x ∗ (cid:107) 2 ≤ µκ 2 η 0 L ˆ κc 1 , which ensures ˆ κc 1 (cid:107) x k − m k − x ∗ (cid:107) 2 ≤ ˆ κc 1 η
[Equation]
Lµ (cid:107) x
[Equation]
− x ∗ (cid:107) 2 ≤ 12 κ .
Then, we have
[Equation]
As a result, by (85), (88), (81), and (82), we have
[Equation]
Now consider the case that p T k − 1 ∆ r k − 1 = 0 .
It is clear that ζ k = 0 .
Then
[Equation]
where the inequality is due to (84) and (87).
Hence the (70) holds.
For the relation (71), it holds since
[Equation]
Next, we prove (72).
We have
[Equation] [Equation]
Therefore,
[Equation] [Equation]
The (72) is proved.
Now, we prove (73), which follows a very similar procedure in proving (70).
For the completeness,we still give the proof.
We have
[Equation]
due to (71) and (cid:107) r k (cid:107) 2 ≤ η (cid:107) r k − m k (cid:107) 2 =
O ( (cid:107) x k − m k − x ∗ (cid:107) 2 ) .
Next, if p T k r k (cid:54) = 0 , then
[Equation]
(95) where a 1 := 1 − ˆ p T k ˆ r k p T
k
r k , b 1 := 1 − p T k q k ˆ p T k ˆ q k .
We have
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12) (cid:12)(cid:12)(cid:12)
[Equation]
(cid:12) (cid:12)(cid:12)(cid:12)
[Equation]
(cid:12) (cid:12)(cid:12)(cid:12) .
From (67) and (71) which have been proved,
[Equation]
We also have
[Equation]
Hence,
[Equation]
Combining (96) and (97), we have
[Equation]
Note that
[Equation]
and
[Equation]
where the existence of the constant c 2 is guaranteed by (99)
[Equation]
and the last inequality holds by choosing
[Equation]
x
[Equation]
Then, we have
[Equation]
As a result, by (98), (101), (94), and (95), we have
[Equation]
Now consider the case that p T k r k = 0 .
It is clear that Γ (1) k = 0 .
Then
[Equation]
where the inequality is due to (97) and (100).
Hence the (73) holds.
Now, we prove (74).
We have
[Equation] [Equation]
Then, (75) holds as
[Equation]
is a consequence of (103) and (104)
[Equation]
Next, we prove (76), which follows a very similar procedure in proving (73).
For the completeness,we still give the proof.
We have
[Equation]
(105) due to (71) and (cid:107) r (1) k
(cid:107)
= (cid:107) r
[Equation]
− q
[Equation]
(cid:107)
[Equation]
≤
[Equation]
(cid:107)
[Equation]
+
[Equation]
q k (cid:107)
[Equation]
| Γ
[Equation] [Equation]
k | ≤ η (cid:107) r k − m k (cid:107)
+ O
[Equation]
(cid:107) x k
[Equation]
m k − 891 x ∗ (cid:107)
[Equation]
= O ( (cid:107) x k − m k − x ∗ (cid:107) 2 ) .
Next, if q T k r (1) k (cid:54) = 0 , then
[Equation]
ˆ
[Equation]
where
a 2 := 1 − ˆ q T k ˆ r (1) k q T k r (1) k , b 2 := 1 − p T k q k ˆ p T k ˆ q k .
We have
[Equation] [Equation]
From (71) and (74) which have been proved,
[Equation]
We also have
[Equation]
Hence,
[Equation]
Combining (107) and (108), we have
[Equation]
From (101),
[Equation]
As a result, by (109), (110), (105), and (106), we have
[Equation]
Now consider the case that q T k r (1) k = 0 .
It is clear that Γ (2) k = 0 .
Then
[Equation]
where the inequality is due to (108) and (100).
Hence the (76) holds.
Finally, we prove (cid:107) x k
+1 − ˆ x k +1 (cid:107)
[Equation]
= ˆ κ O ( (cid:107) x k − m k − x ∗ (cid:107) 22 ) .
It follows from (8c) that
[Equation]
where the last equality is due to (75), (71), (72), and (76) that have been proved.
Therefore, therelation (68) holds.
As a result, we complete the induction.
The relations (67) and (68) are proved.
Based on Lemma 1 and Corollary 1, we can obtain the convergence theorem of the restarted Min-AM.
Theorem 6.
Suppose that Assumption 1 holds for the optimization problem (1) .
Let { x k } denote thesequence of iterates generated by the restarted Min-AM, and deﬁne θ k = (cid:107) I − β k A (cid:107) 2 .
Assume β j ∈ 910 [ β, β (cid:48) ]
( j ≥ 0) for some positive constants β and β (cid:48) .
If (cid:107)∇ f ( x j ) (cid:107) 2 ≤ η 0 (cid:107)∇ f ( x 0 ) (cid:107) 2 (0 ≤ j ≤ k ) 911 for a constant η 0 > 0 , and x 0 is sufﬁciently close to x ∗ , then(cid:107)
[Equation]
θ k
[Equation]
where P m k is the space of polynomials of degree not exceeding m k .
As a result,
[Equation]
Proof.
From Corollary 1, the auxiliary sequence { ˆ x k } in Deﬁnition 1 satisﬁes
[Equation]
Then (112) follows from (114), (68), and the fact that (cid:107) x k +1 − ˆ x k +1 (cid:107) A ≤ √ L (cid:107) x k +1 − ˆ x k +1 (cid:107) 2 .
Since A is SPD, we can choose the Chebyshev polynomial for p ∈ P m k to obtain (e.g., see
Section 6.11.3 in [52])
[Equation]
Like (63) in the proof of Corollary 1, we also have (cid:107) p ( A ) (cid:107) A = (cid:107) p
[Equation]
A
[Equation]
(cid:107)
Then (cid:107) p ( A )( x k − m k −
(cid:107) A ≤ (cid:107) p ( A ) (cid:107)
A (cid:107) x
k − m k − x ∗ (cid:107)
A = (cid:107) p ( A ) (cid:107) 2 (cid:107) x
k − m k − x
(cid:107)
Remark 8.
In Theorem 6, we do not assume each β k is chosen such that θ k = (cid:107) I − β k A (cid:107) 2 < 1 .
It 921 is expected that when θ k is not too large, and m k is sufﬁciently large, the minimization problem onthe right-hand side of (112) can dominate the convergence rate, thus leading to fast convergence.
However, in the case that an improper choice of β k causes θ k (cid:29) 1 , the restarted Min-AM may havean erratic convergence behaviour or even diverge if m k is too small.
Remark 9.
From (112) , a large m k can make the ﬁrst-order term diminish signiﬁcantly.
However,the high-order terms of errors ˆ κ O ( (cid:107) x k − m k − x ∗ (cid:107) 22 ) can be large since they are accumulated from 927 the last restart.
In practice, we can ﬁrst choose large m , small τ , and large η in (11a) - (11c) so thatthe restarts do not occur too frequently.
When the restarted Min-AM has a problematic convergencebehaviour due to the high nonlinearity of ∇ f , more frequent restarts are necessary.
Now, we give the proof of Theorem 2, which can be obtained from Theorem 6.
Proof of Theorem 2 .
Under the assumptions of Theorem 2, we prove that
[Equation]
(116) and there exists a constant η 0 > 0 such that
[Equation]
We prove (116) and (117) by induction.
For k = 0 , (116) and (117) hold.
Suppose that k ≥ 0 , andthe results hold for k .
We establish the results for k + 1 .
From (114) in the proof of Theorem 6, itfollows that
[Equation]
Since θ k ≤ θ , we know β k ∈ [ 1 − θµ , 1+ θL ] .
With the inductive hypothesis (117), for sufﬁcientlysmall (cid:107) x 0 − x ∗ (cid:107) 2 , the relations (67) and (68) hold.
From (68), it follows that (cid:107) x k
+1 − ˆ x k +1 (cid:107)
A
=ˆ κ O ( (cid:107) x k − m k − x ∗ (cid:107) 2 A ) .
Then, there exists a constant c 1 such that (cid:107) x k
+1 − ˆ x k +1 (cid:107)
A ≤ ˆ κc 1 (cid:107) x k − m k − 939 x ∗ (cid:107) 2 A .
With (118), we have
[Equation]
Hence, by choosing (cid:107) x 0 − x ∗ (cid:107) 2 ≤ 1 − θ 2 √ L ˆ κc 1 , which ensures (cid:107) x k − m k − x ∗ (cid:107)
A ≤ (cid:107) x 0 − x ∗ (cid:107) A ≤√ L (cid:107) x 0 − x ∗ (cid:107) 2 ≤ 1 − θ 2ˆ κc 1 due to the inductive hypothesis (116)
, it follows that
[Equation]
A < (cid:107) x k − m k − x ∗ (cid:107)
A ≤ (cid:107) x 0 − x ∗ (cid:107) A , namely (116) for k +1 .
Also, (cid:107) x k +1 − x ∗ (cid:107) 2 ≤ 1 √ µ (cid:107) x k
+1 − x ∗ (cid:107)
A ≤ 1 √ µ (cid:107) x 0 − x ∗ (cid:107) A ≤ √ L √ µ (cid:107) x 0 − x ∗ (cid:107) 2 .
So we can impose (cid:107) x 0 − x ∗ (cid:107) 2 ≤ √ µ ˆ ρ √
L to ensure x k +1 ∈ B
ˆ ρ ( x ∗ ) , which further yields that (cid:107) r k +1 (cid:107) 2 ≤
L (cid:107) x k +1 − x ∗ (cid:107) 2 ≤ L √ L √ µ (cid:107) x 0 − x ∗ (cid:107) 2 ≤ L √ L µ √ µ (cid:107) r 0 (cid:107) 2 , namely (117) for k + 1 , and η 0 =
L √ L µ √ µ .
Hence,we complete the induction.
The convergence result (13) follows from (113) in Theorem 6.
If m k = d , then ˆ x k +1 = x ∗ from Corollary 1.
Therefore, (cid:107) x k +1 − x ∗ (cid:107)
=
ˆ κ O ( (cid:107) x
k − m k − x ∗ (cid:107) 22 ) .
D Details of the eigenvalue estimation procedure
D.1 Analysis of the quadratic case
The procedure of the basic Min-AM leads to three-term recurrences for P k and Q k .
Proposition 3.
Under the same assumptions of Theorem 1 for solving strongly convex quadraticoptimization (47) , we have
[Equation]
where the coefﬁcients are given by
[Equation]
and φ k :=
Γ (1) k
+ Γ (2) k
+ ζ k +
=
Γ (2) k .
Then there exists a tridiagonal matrix ¯ T k ∈ R ( k +1)
× k , suchthat
[Equation]
Proof.
The relations can be veriﬁed by direct computation.
In the ( k + 1) -th iteration, sincex
[Equation]
we have
[Equation]
where the last term vanishes due to
[Equation]
since r k ⊥ p k − 1 .
It also indicates that
[Equation]
Now
[Equation]
it follows from Equation (124) that
[Equation]
The second relation about Aq k can be obtained by noticing that q k = −
Ap k .
The tridiagonal matrix¯ T k is
[Equation]
Input : Starting vector v ∈ R d
Output : Approximate eigenvalues { θ i } ki = 1: r = v, v 0 =2: β 0 = (cid:107) r (cid:107) A 3: for j = 1 , 2 , . .
, until convergence do 4: v j = r/β j − 1 5: r =
Av j 6: r = r − v j − 1 β j −7: α j = v T j Ar 8: r = r − v j α j 9: β j = (cid:107) r (cid:107)
A 10: if j = k then 11: break 12: end if 13: end for 14: Construct tridiagonal T k ∈ R
× k : The main diagonal is { α i } ki
1 ; the subdiagonals are { β i } i = 15: Compute the eigenvalue decomposition T k = S k Θ (
) S T k 16: return Θ ( k )
D.2 Proof of Theorem 3
For solving general nonlinear optimization, the iterations between two successive restarts in therestarted Min-AM are the basic Min-AM iterations.
Hence, at the k -th iteration, a tridiagonal matrix
T k ∈ R m k × m k can be constructed based on the coefﬁcients starting from the ( k − m k ) -th iteration.
We formulate it in the following deﬁnition.
Deﬁnition 2.
For solving optimization problem (1) with the restarted Min-AM (see Algorithm 1), suppose that m k ≥ 1 , and t ( k − 1) k , t ( k ) k , t ( k +1) k are obtained by (121) .
Then T k ∈ R m k × m k is deﬁned as
T k = t k if m k = 1 , and T k =
( ¯ T k − 1 , t k ) if m k ≥ 2 .
Here, ¯ T k =
( T T k , t ( k
[Equation]
1) k
[Equation]
where e m k is the last column of I m k ; t k = t ( k ) k
if m k = 1 , and t k
= (0 , . . .
, 0 , t ( k − 1) k , t ( k ) k ) T ∈
R m kif m k ≥ 2 .
Now, we prove Theorem 3.
Proof of Theorem 3.
Without loss of generality, we consider the case that no restart happens duringthe iterations, i.e. m k
= k .
Let A :=
∇ 2 f ( x ∗ ) .
The A -norm based Lanczos algorithm with a starting vector v for computing eigenpairs of an SPDmatrix A is described in Algorithm 3, which is a modiﬁcation of the Algorithm 4.6 in [6] with
A -norm.
It forms an A -orthonormal basis V k of the Krylov subspace K k
( A, ˆ r 0 ) by the Lanczos
A -orthogonalization, i.e. V T k AV k
= I , where ˆ r
= A ( x ∗ − x 0 ) .
Then the A -norm based Lanczosalgorithm seeks to ﬁnd ˜ λ ∈ R and z ∈ R k such that
[Equation]
Then we have
[Equation]
[Equation]
The eigenvalues of ( AV k ) T AV k are the approximations to the true eigenvalues of A .
For Min-AM, we still use an auxiliary solving procedure of Min-AM on the local quadratic approximation (65), i.e. Process II in Deﬁnition 1.
The same as P k , T k , ¯ T k , t ( k − 1) k , t ( k ) k , t ( k +1) k in Process I,the notations ˆ P k , ˆ T k , ¯ ˆ T k , ˆ t ( k − 1) k , ˆ t ( k ) k , ˆ t ( k +1) k are deﬁned for Process II, correspondingly.
Then
the
Min-AM in Process II seeks ˆ λ ∈ R and y ∈ R k , such that
[Equation]
namely,
[Equation]
According to Proposition 3, we have A ˆ P k
= ˆ
P k +1 ¯ˆ T k .
Then
[Equation]
Then the eigenvalues of
[Equation]
are the approximations of the true eigenvalues of A .
ˆ T k is tridiagonal but generally not symmetric.
However, it is clear that ˆ T k is similar to
[Equation]
which is symmetric.
The columns of U k :
=
ˆ P k
( ˆ P T k
A ˆ P k ) − 1 / 2 are A -orthonormal and also span thesame Krylov subspace K k
( A, ˆ r 0 ) , which implies that there exists an orthonormal matrix S ∈ R k × k ,such that V k
= U k S .
Therefore,
[Equation]
Since S T S = I , the eigenvalues of (129) and (130) are identical.
Recall that we have the explicit forms of T k , ˆ T k by (121), and we restate it here for convenience.
[Equation]
With the already proved relations (69)-(76), we can compute the bounds of t ( j ) k − ˆ t ( j ) k , j = k −1 , k, k + 1 .
Let (cid:15) :
=
ˆ κ O ( (cid:107) x 0 − x ∗ (cid:107) 2 ) to simplify the notation.
Then
[Equation]
) where we use the fact that | Γ (1) k − ˆΓ (1) k | =
ˆ κ O (
[Equation]
x 0 − x ∗ (cid:107) 2 ) , and the assumption that | 1 − Γ (1) k | ≥ τ 0 .
Hence,
[Equation] [Equation]
[Equation]
| t
[Equation]
With m k ≤ m , we have that
[Equation]
0 − x ∗ (cid:107) 2 ) .
Deﬁne ˆ D k
= ˆ P T
k
A ˆ P k .
It is clear that ˆ D k is a diagonal matrix, and the i -th diagonal element is− ˆ p T
i ˆ q i .
From (100), (71), and (72), it follows that
[Equation] [Equation]
By (130), we know that ˆ T k can be diagonalized.
There exists orthonormal matrix W k ∈ R k × k anddiagonal matrix ∧ k ∈ R k × k , such that
[Equation]
Therefore, ˆ T k
= ˆ D − 1 / 2 k
W k ∧ k W T k ˆ D 1 / 2 k can be diagonalized.
Then with (137), (138), and applyingthe Bauer-Fike Theorem [32], we know that for an eigenvalue λ of T k , the following result holds: [Equation]
[Equation]
where σ ( ˆ T k ) denotes the spectrum of ˆ T k .
E Details of the stochastic Min-AM
Algorithm
Input : x 0 ∈ R d , β k >
[Equation]
[Equation] [Equation]
: p
[Equation]
2: for k = 0 , 1 , . . .
, until convergence, do
[Equation]
[Equation]
: if k > 0 then
[Equation] [Equation]
[Equation]
[Equation]
8: end if 9:
[Equation]
[Equation] [Equation]
[Equation]
Apply learning rate schedule of α k , β k 14: end for 15: return x k
Now, we turn to the proofs of the theorems about the stochastic Min-AM (sMin-AM) in Section 3.4.
The algorithm is given in Algorithm 4.
For brevity, we use δ k to denote δ (2) k , i.e. δ k ≡ δ (2) k .
From Assumption 3, for the mini-batch gradient f S k ( x k ) = 1 n k (cid:80) i ∈
S k f i ( x k ) , where n k = | S k | , wehave
[Equation]
141a
[Equation]
141b
[Equation]
Note that the update of sMin-AM (Line 10-12 in Algorithm 4) can be written as x k +
= x k + H k r k ,where r k = −∇ f S k ( x k ) , and for k ≥ 0 ,
[Equation]
To prove the theorems, we need H k to be positive deﬁnite, and show that the noise in gradientestimates is suppressed during iterations.
Lemma 2.
Suppose that the sequence { x k } is generated by sMin-AM.
If there are constantsµ ∈ (0 , 1) , C 2 > 0 such that α k ∈ [0 , 1 − µ ] , ρ k ≤ 0 and − ρ k
[Equation]
β k ρ 2 k
[Equation]
q k
[Equation]
C 2 , then it holds that
[Equation]
Proof.
Since
[Equation]
We have
[Equation]
Hence,
[Equation]
It follows that
[Equation]
Because ρ k ≤ 0 , it follows that H Ak (cid:23) 0 .
Hence H k (cid:23) (1 − α k )
β
k I (cid:23) µβ k I .
Remark 10.
The condition that ρ k ≤ 0 and − ρ k (cid:107) p k (cid:107) 22 − 2 β k ρ k (cid:107) p k (cid:107) 2 (cid:107) q k (cid:107) 2 + β k ρ 2 k (cid:107) p k (cid:107) 22 (cid:107) q k (cid:107) 22 ≤ 1027 β k C 2 for a positive constant C 2 can be fulﬁlled with some proper choice of δ k .
For example, if C 2satisﬁes (1 + 12 β k ) 2 ≤ C 2 and we choose δ k ≥ 12 + β − 1 k , then it can be proved that the conditionholds.
Moreover, in practice, δ k can be chosen as a constant, as shown in our experiments.
With Lemma 2, we can prove the convergence of sMin-AM for full-batch training.
Theorem 7.
Suppose that Assumption 2 holds and { x k } is the sequence generated by full-batch 1032 sMin-AM, i.e. n k = T .
Given constants µ ∈ (0 , 1) , C 2 > 0 , let β k = β ∈
0 , µ L (1+ C 2 ) 2 ] be aconstant, α k ∈ [0 , 1 − µ ] .
The δ (2) k is chosen such that ρ k ≤ 0 and − ρ k (cid:107) p k (cid:107) 22 − 2 β k ρ k (cid:107) p k (cid:107) 2 (cid:107) q k (cid:107) 2 +β k ρ 2 k (cid:107) p k (cid:107) 22 (cid:107) q k (cid:107) 22 ≤ β k C 2 .
Then
[Equation]
(149) in the N -th iterations.
To ensure 1 N (cid:80) N − 1 k =0 (cid:107)∇ f ( x k ) (cid:107) 22 < (cid:15) , the number of iterations is O (1 /(cid:15) ) .
Proof of Theorem 7 .
With Lemma 2, we have
[Equation]
and
[Equation]
Then, under Assumption 2, we have
[Equation]
where the last inequality is due to 0 < β ≤
[Equation] [Equation]
Thus, f
[Equation]
x
[Equation]
x k ) (cid:107) 22 .
Summing both sides of this inequality for k ∈ { 0 , . .
, N − 1 } and recalling f ( x ) >
f low in 1041 Assumption 2 gives
[Equation]
Rearranging and dividing further by N yields (149).
The next lemmas and proofs are about the stochastic case.
E.1 Some useful results
Lemma 3.
Suppose that Assumption 3 holds and { x k } is the sequence generated by sMin-AM withβ k > 0 .
Given constants µ ∈ (0 , 1) , C 2 > 0 , let α k ∈
0 , 1 − µ ] .
The δ (2) k is chosen such that ρ k ≤ 0and − ρ k
[Equation]
p
[Equation]
(cid:107)
[Equation] [Equation]
q k (cid:107) 22 ≤ β k C 2 .
Then
[Equation] [Equation]
151b)
Proof.
(i) From Lemma 2, we have
[Equation]
From Assumption 3, we have
[Equation]
With (152) and (153), we obtain (151a).(ii) Recalling that H
= β
I , the result holds for k = 0 .
Deﬁne (cid:15) k
= ∇ f S k ( x k ) − ∇ f ( x k ) =− r k − ∇ f ( x k ) , then H k r k = H k ( − (cid:15) k − ∇ f ( x k )) .
First, we have
[Equation]
which implies
[Equation]
Let M k
= α k
( H Ak − βI ) , then H k
= β
k
I + M k .
With the assumption (141a), i.e. E S k [ (cid:15) k ] = 0 , wehave
E
[Equation]
Using the Cauchy-Schwarz inequality with expectations , we obtain
[Equation]
We now bound (cid:107) M k
[Equation]
k
[Equation] [Equation]
With (156), we have (cid:107) M k (cid:15)
k
(cid:107) 2 ≤ C 2 α k β k (cid:107) (cid:15) k
(cid:107) 2 , which implies
[Equation]
where the last inequality is due to (141b).
Now we can obtain the bound of | E S k [ ∇ f ( x k ) T H k (cid:15) k ] | asfollows (cf.
[Equation]
With the inequality (154) and (158)
[Equation]
we obtain
[Equation]
By imposing one more restriction on α k , we can obtain a convenient corollary:
Corollary 2.
Suppose that Assumption 3 and Assumption 4 hold and { x k } is the sequence generated 1065 by sMin-AM.
Then
[Equation]
Proof.
The ﬁrst result (160a) is the same as (151a).
Since α k ≤ C 1 β 12 k , we have
[Equation]
Substituting it into (151b), we obtain (160b).
Using Corollary 2 we obtain the descent property of sMin-AM:
Lemma 4.
Suppose that Assumptions 2-4 hold, β k ∈ (0 , µ 2 L (1+ C 2 ) 2 ] , and { x k } is the sequencegenerated by sMin-AM.
Then
[Equation]
Proof.
Due to Assumption
[Equation]
we have
[Equation]
Taking expectation with respect to the mini-batch S k on both sides of (162) and using Corollary 2,we obtain
[Equation]
Then (163) combined with the assumption β k ≤
[Equation]
2 implies (161).
E.2 Proof of Theorem
Following the proofs in [60], we introduce the deﬁnition of a supermartingale .
Deﬁnition 3.
Let {F k } be an increasing sequence of σ -algebras.
If { X k } is a stochastic process 1079 satisfying (i) E
[ | X k | ] < ∞ , (ii) X k ∈ F k for all k , and (iii) E [ X k +1 |F k ] ≤ X k for all k , then { X k } 1080 is called a supermartingale.
Proposition 4 (Supermartingale convergence theorem, see, e.g., Theorem 4.2.12 in [20]) .
If { X k } isa nonnegative supermartingale, then lim k →∞ X k → X almost surely and E [ X ] ≤ E [ X 0 ] .
Now, we prove the convergence theory of sMin-AM in the nonconvex stochastic case.
Proof of Theorem 4 .
(i) Deﬁne φ k = β k µ 4 (cid:107)∇ f
( x k ) (cid:107) 22 and ˜ L = 12 (cid:0)
µ − 1 C 21 C 22 + L (1 + C 2 ) 2 (cid:1) ,γ k = f
( x k ) + ˜ L σ 2 n (cid:80) ∞ i = k β 2 i .
Let F k be the σ -algebra measuring φ k , γ k , and x k .
From (161) weknow that for any k ,
[Equation]
which implies that E
[ γ k +1 − f low |F k ] ≤ γ k − f low − φ k .
Since φ k ≥ 0 , we have 0 ≤ E
[ γ k − f low ] ≤γ 0 − f low < + ∞ .
As the diminishing condition (25) holds, we obtain E [ f ( x k )] ≤ M f forsome constant M f > 0 .
According to Deﬁnition 3, { γ k − f low } is a supermartingale.
Therefore, 1090 Proposition 4 indicates that there exists a constant γ such that lim k →∞ γ k
= γ with probability 1,and E
[ γ ] ≤ E [ γ 0 ] .
Note that from (164) we have E [ φ k ] ≤ E [ γ k ] − E
[ γ k +1 ] .
Thus,
[Equation]
which further yields that
[Equation]
Since (cid:80) ∞ k =0 β k = + ∞ , it follows that (26) holds.(ii)
If the noisy gradient is bounded, i.e.,
[Equation]
where M g > 0 is a constant, then a stronger result can be obtained.
For any give (cid:15) > 0 , according to (26), there exist inﬁnitely many iterates x
k such that (cid:107)∇ f ( x k ) (cid:107) 2 ≤ (cid:15) .
Then if (27) does not hold, there must exist two inﬁnite sequences of indices { s i } , { t i } with t i > s i , 1098 such that for i = 0 , 1 , . . .
, k = s i + 1 , . . .
, t i
[Equation] [Equation]
Then from (165) it follows that
[Equation]
k
[Equation]
β k with probability 1 , which implies that
[Equation]
According to (144), we have
[Equation]
where the last inequalities are due to Cauchy-Schwarz inequality and (166).
Then it follows from(169)
that
[Equation]
which together with (168) implies that (cid:107) x
t i − x s i (cid:107) 2 → 0 with probability 1, as i → + ∞ .
Hence, 1105 from the Lipschitz continuity of ∇ f , it follows that (cid:107)∇ f ( x t i ) − ∇ f
( x s i ) (cid:107) 2 → 0 with probability 1 1106 as i → + ∞ .
However, this contradicts (167).
Therefore, the assumption that (27) does not hold isnot true.
E.3 Proof of Theorem 5
Proof of Theorem 5 .
According to (163) in Lemma 4, we have
[Equation]
(170) where the expectation is taken with respect to { S j } N − 1 j =0 .
Deﬁne
[Equation]
(171) then
[Equation]
Let ˜ D be a problem-independent constant.
If we choose β k = β := min {
[Equation]
2 L (1+ C 2 ) 2 , ˜ Dσ √ N } , andn k ≡ n , then the deﬁnition of P R simpliﬁes to P R ( k ) = 1 /N .
From (172) we have
E [ (cid:107)∇ f
[Equation]
x R )
[Equation] [Equation]
Therefore, to ensure E [ (cid:107)∇ f ( x R ) (cid:107) 22 ] ≤ (cid:15) , the number of iterations is O (1 /(cid:15) 2 ) .
F Experimental details
Our main codes were written based on the PyTorch framework 1 and one GeForce RTX
Ti GPU was used for the tests in training neural networks (except that four GPUs were used fortraining ResNet50 on ImageNet).
We tested the basic Min-AM on solving strongly convex quadraticoptimization, the restarted Min-AM on solving regularized logistic regression, and the stochastic
Min-AM on training neural networks.
F.1 Strongly convex quadratic problem
The experiments on solving strongly convex quadratic optimization were conducted to verify Theorem 1 about the basic Min-AM.
The problem is
[Equation]
(173) where A ∈ R (cid:96) ×
d , b ∈ R (cid:96) .
For the test, we ﬁrst generated a random matrix A ∈ R 500 × 100 and arandom vector v ∈ R 100 following Gaussian distribution, then b ∈ R 500 was obtained as b
= Av .
The ﬁxed step size β for the gradient descent (GD) was chosen by a grid search in{ 0 . 001 , 0 . 002 , . . .
We set β = 0 .
001 that guarantees the convergence of GD.
The AM-I(1)and the full-memory AM-I (i.e. AM-I( ∞ )) also used the same mixing parameter setting β k = 0 . 001 .
For Min-AM, we set the initial mixing parameter β 0 = 1 , and the later mixing parameters { β k } wereadaptively determined based on the eigenvalue estimates (see Section 3.3).
Figure 4(a) compares the convergence behaviours of different methods in terms of relative residualnorm.
It can be observed that due to the improper initial setting of β 0 for Min-AM, Min-AM doesnot perform well in the beginning.
Nonetheless, as shown in Figure 4(b), the β k of Min-AM can bequickly adapted to the optimal value 2 / ( µ + L ) = 1 .
67 × 10 − 3 based on the eigenvalue estimates.
In Figure 4(c), we show the effects of β k on the full-memory AM-I and Min-AM, where bothmethods used ﬁxed β k chosen from { 0 . 01 , 0 . 1 , 1 } .
(For Min-AM, we disable the adaptive choice ofβ k .)
It should be noted that the GD method diverges when choosing the step size in { 0 . 01 , 0 . 1 , 1 } ,which suggests that θ k > 1 for the Min-AM in these settings.
Nevertheless, we ﬁnd that Min-AMstill converges to the tolerance of (cid:107) r k (cid:107) 2 / (cid:107) r 0 (cid:107) 2 ≤ 10 − 11 , which validates the convergence property
[Figure] [Figure]
[Figure]
(59) in Corollary 1, i.e., the minimization problem in (59) dominates the convergence when k islarge.
Note that AM-I fails to coincide with Min-AM in the later iterations.
It is due to the fact that
AM-I needs to solve ( X T k R k ) − 1 X T k r k to determine Γ k , where the matrix inverse operation can have
[Figure]
numerical weakness when k is large.
[Figure]
): relative residual norm, eigenvalue estimates, and β k of Min-AM, when L/µ = 10 8 .
In (b) and (c), “eigenvalue” denotes the exact eigenvalues, and “Min-AM” denotes the eigenvalue estimates computed by Min-AM.
Results about the problem with different condition numbers.
In Figure 5, we show the convergence of each method, the eigenvalue estimates from Min-AM, and the β k of Min-AM, in the testswith different condition numbers characterized by L/µ .
The eigenvalues of A ∈ R 100 × 100 werechosen to be in (0 , 1] with equal interval.
The results show that Min-AM is competitive with CG andthe full-memory AM-I. Min-AM also gives accurate enough estimates of the largest and the smallesteigenvalues, and the β k is quickly adapted to approximate the optimal value 2 / ( µ + L ) .
Cost of the eigenvalue estimation procedure.
In our implementation of the eigenvalue estimationprocedure in the Min-AM, we used the function “np.linalg.eigvals” in NumPy to compute theeigenvalues of T k constructed by (18), which needs O ( k 3 ) ﬂops.
Hence, the cost of the eigenvalueestimation increases with increasing k , and at the k -th iteration, the ratio of this cost to the total
[Figure]
cost is O ( k 3 ) /
( O ( k 3 ) + O ( d 2 )) , where O ( d 2 ) is due to the gradient evaluation.
To investigate thepractical performance of the eigenvalue estimation, we applied the basic Min-AM to solve problem(173) of dimension d = 1000 , 2000 , 4000 , where each A ∈ R d × d was generated following Gaussiandistribution, and the maximal iteration number is max _ iter = 1002 for Min-AM.
As can be foundin Figure 6(a), the additional computational time incurred by the eigenvalue estimations is marginalwhen d is large and k is small.
Figure 6(b) and Figure 6(c) show the case that d = 4000 .
(The casesd = 1000 and d = 2000 have similar results.)
We ﬁnd that β k is quickly adapted to the optimal value2 / ( µ + L ) within very small number of iterations.
So we can adaptively choose β k in the beginning,and ﬁx the obtained β k and disable the eigenvalue estimation procedure in the later iterations toreduce the computational cost.
Figure 6(c) shows that the eigenvalue estimates computed by Min-AM(at the last iteration) well approximate the exact eigenvalues.
F.2 Regularized logistic regression problem
The regularized logistic regression problem is formulated as
[Equation]
where ξ i ∈ R d is the i -th data sample and y i = ± 1 is the corresponding label.
The regularizationparameter w = 0 .
We used the datasets ‘madelon’ and ‘a9a’ from LIBSVM [14] that are two-classclassiﬁcations:
• madelon: training data size: 2000; feature size: 500;
• a9a: training data size: 32561; feature size: 123.
To set proper µ and L for Nesterov’s accelerated gradient (NAG) method and check the eigenvalueestimates from Min-AM, we applied the standard Lanczos algorithm [24] to compute 100 Ritz valuesof ∇ 2 f ( x ∗ ) as the estimates of the true eigenvalues of ∇ 2 f ( x ∗ ) , where the minima x ∗ was obtainedby solving (174).
For the test on madelon, Lanczos algorithm gave µ = 1 .
01 × 10 − 1 , L = 1 . 45 ; forthe test on a9a, Lanczos algorithm gave µ = 1 .
× 10 − 1 , L = 8 .
64 × 10 − 1 .
For the gradient descent (GD), the step size was tuned and set as 1, which was proper for bothdatasets.
We used the Polak-Ribière variant of nonlinear conjugate gradient (NCG), and the stepsize was determined by line search with cubic interpolation, where the strong Wolfe conditions werechecked.
The L-BFGS used the Barzilai–Borwein step size as the initial guess of the approximateinverse Hessian.
For AM and ST-AM, the mixing parameter was set as 1.
The AM-I performedsimilarly to AM in this test, so we only report the results of AM.
For the restarted Min-AM, we setτ = 10 − 16 for the madelon test and τ = 10 − 32 for the a9a test.
m
= 100 for both datasets.
η wasset as a large number since Min-AM converged for these tests.
Besides, we computed at most 20eigenvalue estimates in Min-AM (between two successive restarts) to determine β k .
Results about the choice of β k .
In Figure 7, we show additional results of the tests on madelondataset.
(1) To test the effect of β k on Min-AM, we disabled the adaptive choice of β k and used
[Figure]
ﬁxed β k chosen from { 1 , 2 , 5 } .
It is observed in Figure 7(a) that GD does not converge if the stepsize β k is chosen from { 2 , 5 } , which suggests that θ k > 1 in Theorem 2 and Theorem 6.
However,
Min-AM still converges in these cases, which validates our discussion in Remark 8 in Appendix C.1:
the minimization problem on the right-hand side of (112) in Theorem 6 dominates the convergencewhen m k is large.
It also indicates that Min-AM is less sensitive to β k than GD.
(2) When adaptivechoice of β k is used, we investigate the effect of the initialization β 0 on the convergence and thequality of eigenvalue estimates.
It is found in Figure 7(b) that even with the improper initializationβ
= 50 , 100 , 1000 , the iterations still ﬁnally converge to an acceptable precision.
In fact, GD,
AM, and Min-AM diverge if using ﬁxed β k chosen from { 50 , 100 , 1000 } .
Hence, Min-AM withadaptive choice of β k is easy to apply since it discards the requirement of manually tuning the mixingparameter.
In Figure 7(c), we ﬁnd that the eigenvalue estimates can still roughly approximate thelargest and the smallest Ritz values computed by the Lanczos algorithm, which veriﬁes Theorem 3.
[Figure]
Results of the problem with different regularization parameters.
We also tested Min-AM for theproblem (174) with different settings of the regularization parameter w .
The results in Figure 8 also show that Min-AM signiﬁcantly improves the convergence of AM(1) and is comparable to AM(20)when w is small.
The comparison between the Ritz values computed by Lanczos algorithm and theeigenvalue estimates computed by Min-AM validates Theorem 3.
Since the Ritz values approximatethe true eigenvalues of ∇ 2 f ( x ∗ ) , it is expected that Min-AM can give promising eigenvalue estimatesof ∇ 2 f ( x ∗ ) , which accounts for the efﬁciency of Min-AM.
[Figure]
Running time.
In Figure 9, we report the convergence of each method with respect to the running
[Figure]
time.
Min-AM has comparable performance to AM(20) in these tests, while using a smaller memory
[Figure]
size.
[Figure]
F.3 Deep neural network training problem
[Figure]
The experiments about the stochastic Min-AM (sMin-AM) focused on training deep neural networks.
Since Min-AM can be viewed as a special case of sMin-AM with δ (1) k
= δ (2) k
= 0 , α k = 1 , the basic
Min-AM was also covered.
F.3.1 Hyperparameter setting of the stochastic Min-AM
The main hyperparameters of sMin-AM are α k , β k , δ (1) k , δ (2) k .
We set δ (1) k
= δ (2) k
= 1 , which ensuresthat H Ak is positive deﬁnite.
We ﬁxed α k = 1 and only tuned β k ∈ (0 , 1] in the experiments.
F.3.2 Experiments on CIFAR-10 and CIFAR-100
We trained several networks on CIFAR-10 and CIFAR-100.
Either dataset contains 50K imagesfor training and 10K images for test.
CIFAR-10 has 10 classes and CIFAR-100 has 100 classesfor classiﬁcation.
The basic setting of network training followed the standard setting of training
ResNet [29]: The batch size was 128 as commonly suggested; when training with an optimizer for Niterations, the learning rate of the optimizer was decayed at the ( (cid:98) N 2 (cid:99) )
-th iteration and the ( (cid:98) 34 N (cid:99) ) -th 1223 iteration.
The experiments were run with 3 random seeds and the averaged results along with thestandard deviations were reported.
The ﬁnal accuracy on the test dataset was used as the evaluationmetric.
The accuracy reported in Table 1(a) is the ﬁnal test accuracy for training 160 epochs.
We compared sMin-AM with SGDM, Adam, AdaHessian, stochastic AM (SAM), and short-termrecurrence AM (ST-AM).
SGDM is the default optimizer for training many deep neural networks,such as ResNet [29], WideResNet [65], DenseNet [33], and ResNeXt [63].
Adam is an adaptivelearning rate method that uses diagonal approximation to the Hessian based on moving average.
AdaHessian also uses adaptive learning rates like Adam, but it relies on Hessian-vector products toobtain the diagonal approximation.
SAM and ST-AM are variants of AM.
Both can be used to trainneural networks.
The SAM with memory size m is denoted as SAM( m ).
ST-AM keeps two vectorpairs and has the same memory size as SAM(2).
We tuned the hyperparameters of all the optimizers (including sMin-AM) following the same way forfair comparison.
The tuning procedures were conducted on CIFAR-10/ResNet20.
For each optimizer,the hyperparameter setting that achieved the highest ﬁnal test accuracy on CIFAR-10/ResNet20 waskept unchanged and used for the other tests of training neural networks on CIFAR.
[Figure]
For SGDM, the momentum was set as 0.9, which is the default setting in the literature [29, 33].
Theinitial learning rate and weight decay were tuned and were set as 0.1, 0.0005, respectively.
Thelearning rate decay was 0.1.
This setting is also recommended in WideResNet [65].
For Adam, the initial learning rate and weight decay were tuned and set as 0.001, 0.0005, respectively.
The momentum terms were β 1 = 0 .
9 and β 2 = 0 .
999 as commonly suggested [64, 67].
The learningrate decay was 0.1.
For AdaHessian, the initial learning rate was 0.15, and the momentum terms β 1 = 0 .
9 , β 2 = 0 .
999 ;eps = 1 × 10 − 4 , and the hessian-power was 1 .
The weight decay was 0 .
0005 / 0 . 15 , and the learningrate decay was 0.1.
For SAM, the initial mixing parameter β 0 = 1 . 0 , the initial damping term α 0 = 1 . 0 , and theregularization parameter c 1 = 0 .
The weight decay was 0 . 0015 .
The decay rate for β k , α k was0.06.
For ST-AM, the initial mixing parameter β 0 = 1 . 0 , the initial damping term α 0 = 1 , and theregularization parameters were c 1 = 1 , c 2 = 1 × 10 − 7 .
The weight decay was 0 .
001 , and the decayrate for β k , α k was 0.1.
For sMin-AM, we set the initial mixing parameter β 0 = 0 .
2 , and the damping term α k = 1 .
Theregularization parameters were δ (1) k
= 1 , δ (2) k
= 1 .
The weight decay was 0 . 0015 , and the decay ratefor β k was 0.1.
Convergence behaviour in the training process.
In Figure 10, we plot the test accuracy of training
ResNet18 and VGG16 on CIFAR-10 and ResNeXt50 and DenseNet121 on CIFAR-100, for eachoptimizer.
It shows Adam has fast convergence in the beginning, but stagnates in the later training.
Compared with SGDM, SAM
[Figure]
and ST-AM, it is observed that sMin-AM converges faster to anacceptable accuracy.
The process of sMin-AM is also much more stable than SAM(1) and SAM(10)
[Figure]
Effect of weight decay.
Figure 11 shows the effect of weight decay on the training process.
Itsuggests that a larger weight decay can slow down the training in terms of training loss.
When usingthe same weight decay
[Figure]
sMin-AM often has faster convergence than SGDM.
The results also justifyour settings of the weight decay
[Figure]
Effect of learning rate/mixing parameter.
Figure 12 shows the behaviours of SGDM with differentlearning rate settings and sMin-AM with different mixing parameters.
Small learning rate or smallmixing parameter can lead to fast convergence in the beginning, but the training may stagnate earlyand the test performance is not satisfactory.
Effects of δ (1) k , δ (2) k , and α k .
In Figure 13, we show the results of sMin-AM with different settingsof other hyperparameters, i.e., δ (1) k , δ (2) k , and α k .
It is observed that sMin-AM is more sensitive toδ (2) k than δ (1) k
and α k , which may be due to the fact that δ (2) k directly affects the regularization in the
Min-AM update (Line 10 in Algorithm 4).
When choosing δ (2) k ≥ 1 , the effect of δ (2) k is also minor.
As a result, the most critical hyperparameters that affect the effectiveness of sMin-AM are the mixingparameter β k , the regularization parameter δ (2) k , and weight decay.
The weight decay is a commonhyperparameter that needs to be tuned for each optimizer; for δ (2) k , we can choose δ (2) k ≥ 1 to ensure
H Ak (cid:23) 0 .
So except for the weight decay, we only tuned the mixing parameter in the experiments.
Check of the choices of δ (1) k and δ (2) k .
From Lemma 2, we know (cid:107) H
Ak − β k
I (cid:107) 2 ≤ − ρ k (cid:107) p k (cid:107) 22 −2 β k ρ k (cid:107)
p k (cid:107) 2 (cid:107) q k (cid:107) 2 + β k ρ 2 k (cid:107)
p k
(cid:107) 22 (cid:107) q k (cid:107) 22 assuming ρ k ≤ 0 .
In Figure 14, we plot the value of thisbound.
It is found that (cid:107) H Ak − β k
I (cid:107) 2 ≤ β k C 2 for some constant C 2 , which justiﬁes our choice of 1280 the regularization parameters.
Train neural networks with fewer epochs and less time.
We also tested sMin-AM for trainingdifferent epochs.
Figure 15 shows the comparison between sMin-AM and SGDM for training 80,
[Figure] [Figure]
[Figure] [Figure]
120, and 160 epochs.
The results show that sMin-AM can attain comparable accuracy to SGDM infewer epochs, which infers the faster training process of sMin-AM.
In Table 2, by setting the ﬁnaltest accuracy (160 epochs) of SGDM as the baseline, we report the memory cost, per-epoch time,total training epochs, total running time of SAM(10), ST-AM
[Figure]
and sMin-AM to achieve a comparableaccuracy to SGDM (within 0 .
05% difference).
The attained ﬁnal accuracy is also shown.
It is foundthat sMin-AM largely reduces the memory cost of SAM(10) and can achieve comparable results tothose of SGDM using less training time.
Discussion about the computational cost and memory cost.
The per-epoch computational costand memory cost are closely related to three factors: the network architecture, the training data, andthe optimizer.
From Table 2, the effects of the network architecture and the optimizer are clear.
Forthe effect of training data, we consider the batch size.
Given a network, the cost comprises two parts:(i) updating network parameters by the optimizer; (ii) other necessary computations, such as theforward and back propagations of the neural network, data input and output, etc., where additionalmemory and processing time are required.
In Figure 16, we plot the ratios of memory/per-epochtime of SAM(10)/ST-AM/sMin-AM to that of SGDM, for training ResNeXt50 on CIFAR-100 withdifferent batch sizes (16, 32, 64, and 128).
When a smaller batch size is used, the Part (ii) cost
[Figure]
[Figure] [Figure]
[Table]
ST-AM
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
sMin
[Table]
AM
[Table] [Table]
[Table]
is reduced and the proportion of Part (i) cost in the total cost is increased, thus the performanceimprovement of sMin-AM over SAM(10)/ST-AM is more signiﬁcant.
The difference of CIFAR10 and CIFAR100 datasets has only a minor effect on the cost.
For a givennetwork architecture, e.g., VGG16, its implementations for CIFAR10 and CIFAR100 differ in thedimension of the last linear layer.
Since the parameters of this linear layer usually occupy a verysmall portion of the whole parameters, the costs (memory and per-epoch running time) on CIFAR10and CIFAR100 are roughly the same, as shown in Table 3.
[Figure]
[Table]
F.3.3 Experiment of training ResNet50 on ImageNet
We applied sMin-AM to train ResNet50 on the ImageNet dataset, which contains 1.2M images fortraining and 50K images for test.
The code was based on the example from PyTorch 2 .
We followedthe standard process of training ResNet [29].
Four GPUs were used to conduct the experiment.
SGDM was used for comparison.
The batch size was 256.
The weight decay for each method was0.0001.
We trained ResNet50 for 90 epochs.
The learning rate of SGDM and the mixing parameterof sMin-AM were divided by 10 at the 30th and 60th epochs.
For SGDM, the momentum was 0.9and the learning rate was 0.1, which was the standard setting [29, 65, 33, 63].
For sMin-AM, we setthe initial mixing parameter β 0 = 0 .
5 , and the other hyperparameters were kept unchanged.
In our test, we found the memory cost of sMin-AM was 1.04 times that of SGDM, and the per-epochtraining time of sMin-AM was 1.01 times that of SGDM.
The training processes of the two methodsare shown in Figure 2.
It is found that sMin-AM still improves the training process of SGDM.
[Figure]
Comparisons with SAM and ST-AM.
We conducted tests to compare the accuracy of sMin-AMwith that of SAM/ST-AM, and results in Figure 17 show that sMin-AM also improves the testaccuracy of SAM(10)/ST-AM.
F.3.4 An additional experiment: adversarial training
We conduct an additional experiment to further compare the effectiveness of SGDM, ST-AM, andsMin-AM for training deep neural networks.
The considered problem is adversarial training , whichtries to solve the following minimax problem
[Figure]
[Equation]
where ¯ ξ i is the adversarial data sample in the (cid:15) -ball centered at the data sample ξ i .
We used thestandard PGD adversarial training process [39]: using projection gradient descent to solve the maximization problem in (175), and applying SGDM/ST-AM/sMin-AM to solve the minimization problemin (175).
The tests were conducted on CIFAR-10/WideResNet34-10 and CIFAR-100/DenseNet121,where for the training dataset of CIFAR-10/CIFAR-100, 5K images were randomly selected as thevalidation dataset and the other 45K images were used as the new training dataset.
We trained theneural network for 200 epochs, where the learning rate/mixing parameter was decayed at the 100thand the 150th epochs.
The best checkpoint model on the validation dataset was chosen to be evaluatedon the test dataset.
For SGDM and ST-AM, we used the recommended setting of hyperparameters in[26, 62].
For sMin-AM, we set β 0 = 0 .
3 , and other hyperparameters were kept unchanged.
[Figure]
The training process of each optimizer is shown in Figure 18, where we plot the accuracy on validationdataset (called as validation accuracy), and the accuracy on adversarial data samples generated bythe PGD-10 attack [39] on validation dataset (called as robust validation accuracy).
It is observedthat the best robust validation accuracy of sMin-AM is higher than that of SGDM/ST-AM, whichindicates that better checkpoint model can be obtained by sMin-AM.
[Table]
In Table 4, we report two types of test accuracy: clean test accuracy, where the clean test data wasused for classiﬁcation; robust test accuracy, where corrupted test data was used for classiﬁcation.
Three attacking methods were used for the robust test accuracy evaluation: FGSM [25], PGD-20 [39],and C&W ∞ attack [13].
It shows that sMin-AM signiﬁcantly improves SGDM, and outperforms
ST-AM in terms of robust test accuracy, which is a desirable property in adversarial training.
G Limitations
We focus on developing the variant of AM with minimal memory size for solving optimizationproblems.
The properties of Min-AM for deterministic optimization are established in the smoothcase.
The non-smooth optimization is important and one possible direction is to adapt Min-AM tothe framework of proximal quasi-Newton methods [15, 7].
We leave it as an extension of Min-AM inthe future work.
Min-AM exploits the symmetry of Hessian in solving optimization problems.
For general ﬁxed-pointproblems, the Jacobian can be non-symmetric.
However, in this case, the short-term recurrencealgorithms that are equivalent to the full-memory methods generally do not exist even in solvinglinear systems [52].