A Variant of Anderson Mixing with Minimal Memory Size
Abstract
Anderson mixing (AM) is a useful method that can accelerate ﬁxed-point iterations by exploring the information from historical iterations.
Despite its numerical success in various applications, the memory requirement in AM remains a bottleneck when solving large-scale optimization problems in a resource-limited machine.
To address this problem, we propose a novel variant of AM method, called Min-AM, by storing only one vector pair, that is the minimal memory size requirement in AM.
Our method forms a symmetric approximation to the inverse Hessian matrix and is proved to be equivalent to the full-memory Type-I AM for solving strongly convex quadratic optimization.
Moreover, for general nonlinear optimization problems, we establish the convergence properties of Min-AM under reasonable assumptions and show that the mixing parameters can be adaptively chosen by estimating the eigenvalues of the Hessian.
Finally, we extend Min-AM to solve stochastic programming problems.
Experimental results on logistic regression and network training problems validate the effectiveness of the proposed Min-AM.
Introduction
Anderson mixing (AM) [2] is a powerful method for accelerating ﬁxed-point iterations, and has been widely used in scientiﬁc computing [1, 5, 47] and machine learning [31, 58, 29].
In each iteration, AM extrapolates a new iterate that satisﬁes certain optimality property by using the historical iterations and exhibits empirical acceleration over classical ﬁxed-point iterations [60].
Thus, AM can reduce the computation cost when evaluating the ﬁxed-point map is a time-consuming process.
Also, the inspiring connections of AM with GMRES [60, 48] and multisecant quasi-Newton methods [22] provide some explanations for the acceleration beneﬁt from AM.
In recent years, there have been many works applying AM to solve various optimization problems [56, 62, 41].
Their promising results in different tasks suggest that these AM-based methods are very competitive while being more economical than Newton’s method.
However, the existing convergence theory of AM is still unsatisfactory [3] and AM can diverge in some case [41].
Consequently, the applications of AM and its variants for machine learning problems and the corresponding theories deserve deep research.
One major concern of AM is the heavier memory overhead compared with the ﬁxed-point iteration.
In AM( m ), it has to store vector pairs about historical information from the m previous iterations, where m is the memory size.
Speciﬁcally, AM( m ) is called as the limited-memory AM if m < ∞ , and full-memory AM if m = ∞ .
The AM( 0 ) is the simple ﬁxed-point iteration.
In practice, choosing proper memory size m is heuristic and it is observed in [47] that a small m can deteriorate the efﬁcacy.
However, for practical applications, a large m can lead to memory issue for solving large-scale and high-dimensional problems when the memory resource is limited.
In this paper, we address the memory issue of AM for solving optimization problems by proposing
Min-AM, a novel variant of AM( 1 ) that has minimal memory size in AM.
Min-AM is built upon the Type-I AM [22, 60], a variant of AM.
Let AM-I( m ) denote the Type-I AM with memory size m .
Using a projection-mixing framework of AM-I, the Min-AM retains a recursively modiﬁed vector pair and adds one extra projection step in each step of AM-I( 1 ).
Compared to AM(1), Min-AM forms symmetric approximations to the Hessian and is equivalent to AM-I( ∞ ) when the objective function is strongly convex quadratic.
By incorporating the restarting strategy, the restarted Min-AM has faster convergence rate than AM( 1 ) for general nonlinear optimization and the spectrum of Hessian can be cheaply estimated that leads to adaptive choices of the mixing parameters.
Min-AM is also extended to stochastic optimization.
In summary, we highlight the main contributions of this work as follows.
• We propose Min-AM, a variant of AM(1) that achieves the minimal memory size in AM, for solving optimization problems.
Compared to AM( 1 ), Min-AM forms symmetric approximations to the Hessian and incorporates more historical information to obtain the update.
Thus, Min-AM can signiﬁcantly reduce the memory requirement and computational cost of AM while maintaining the fast convergence property.
• We show that the local convergence rate of the restarted Min-AM has optimal dependence on the condition number of the objective function.
We provide adaptive choices of the mixing parameters by estimating the spectrum of the Hessian economically in each iteration.
Also, we give the convergence and complexity analysis of a stochastic extension of Min-AM for solving nonconvex stochastic optimization.
• We verify the properties of Min-AM, and apply the restarted Min-AM to logistic regression and the stochastic Min-AM to train neural networks.
The experimental results are consistent with our theoretical analysis and show that the Min-AM is competitive with the limitedmemory quasi-Newton methods with large memory size in deterministic optimization and achieves promising results in training neural networks.
Related work.
Fixed-
point
problems can be recast as solving systems of (non)linear equations.
AM is a practical alternate for Newton’s method when the Jacobian is unavailable or difﬁcult to compute [35].
In the linear case, it is established in [60, 48] that the full-memory AM is equivalent to
GMRES [51], a classical Krylov subspace method for linear systems.
In the nonlinear case, AM is recognized as a multisecant quasi-Newton method and a variant, called Type-I AM, has been proposed in [22].
From the perspective of convergence analysis, it is proved that the local convergence rate of the limited-memory AM is no worse than that of ﬁxed-point iteration [59, 16] and the potential improvement depends on the quality of extrapolation [21, 46].
However, a counter example [41] shows the divergence behaviour of AM( 1 ) and the recent review [3] indicates the need of further research on the theoretical properties of AM.
The inspiring performance and the easy implementation of AM motivate many new variants of AM for various applications in machine learning.
In [55], a regularized variant of AM is proposed to accelerate gradient descent for unconstrained optimization, which is adapted in [23] for nonsmooth convex optimization and in [58] for reinforcement learning.
In [62], a stochastic version of AM is developed and has convergence guarantee for stochastic optimization.
The Type-I AM is modiﬁed in [67] to solve nonsmooth ﬁxed-point problems.
A more recent work, the short-term recurrence
AM (ST-AM) [63], stores two vector pairs to reduce the memory cost of AM.
Our Min-AM method achieves the minimal memory size, i.e. m = 1 , further reducing the memory size of ST-AM.
The memory overhead is a common issue of quasi-Newton methods.
It is desirable to keep memory size as small as possible without sacriﬁcing the efﬁcacy of the algorithm.
Some works [37, 9, 8] study the choice of memory size in L-BFGS [39], but the performance with minimal memory size is unsatisfactory.
In stochastic programming, the sublinear convergence rate is optimal when only stochastic gradients can be accessed [43].
Thus the heavier memory and computational cost makes quasi-Newton methods [12, 27, 61, 10] less appealing than the more economical ﬁrst-order methods [49, 36].
In this work, our proposed Min-AM method exhibits the fast convergence as quasi-Newton methods while the memory cost is close to ﬁrst-order methods.
Notations.
Let ∆ denote the forward difference operator, e.g. ∆ x k = x k +1 − x k .
range( X ) denotes the subspace spanned by the columns of a matrix X .
For every matrix A ∈ R d × d , the Krylov subspace K m ( A, v ) ≡ span { v, Av, . . .
A m − 1 v } , and (cid:107) x (cid:107)
A =
( x T Ax ) 1 / 2 denotes the A -norm when A is symmetric positive deﬁnite (SPD).
“ † ” is the Penrose-Moore inverse.
[Equation]
The Anderson mixing scheme
In this work, we consider the minimization problem
[Equation]
where f : R d → R is continously differentiable.
The ﬁrst-order optimality condition of (1) is equivalent to x = g ( x ) , where g ( x ) = x − ∇ f ( x ) .
We describe the Anderson mixing scheme with the projection-mixing framework [62].
Deﬁne r k = g
x k )
− x k = −∇ f
( x k ) to be the residual.
AM( m ) maintains m ( m ≤ min { d, k } ) vector pairs stored in X k , R k ∈
[Equation]
m :
[Equation]
Each update can be decoupled into two steps, namely the projection step and the mixing step :
[Equation]
where ¯ r k :=
r k − R k Γ k , β k > 0 is the mixing parameter, and Γ k = arg min Γ ∈
R m (cid:107) r k − R k Γ (cid:107) 2 .
It is easy to know that the Galerkin’s projection condition ¯ r k ⊥ range( R k ) holds.
Assuming R T k R k is nonsingular, the resulting one-step update of AM( m ) is x k +
= x k + H k r k where
[Equation]
The AM-I [22, 60] falls into the same projection-mixing framework, and the only difference lies in the choice of Γ k .
The Γ k of AM-I is determined by another Galerkin’s condition:
[Equation] [Equation]
Substituting the solution of (5) into (3), the approximated inverse Hessian matrix H k of AM-I is
[Equation]
by assuming X T k R k is nonsingular.
AM-I has a close relation to Newton’s method and an alternative interpretation of AM-I is given in Appendix A.1.
Remark 1.
It is established in [22] that the H k of AM solves min H k (cid:107)
H
k − β k
I (cid:107) F s.t.
H k R k = − X k and the H k of AM-I satisﬁes H k
= B − 1 k , where B k solves min B k (cid:107) B
k − β − 1 k I (cid:107) F s.t.
B
k X k = − R k .
Thus, both AM and AM-I are multisecant methods.
Besides, from (4) and (6) , the approximated inverse Hessian matrices H k are generally not symmetric.
The Min-AM methods
In each iteration, AM( m ) has to store two matrices X k ,
R k ∈ R d × m , which dramatically increases the memory burden in large-scale problems.
To reduce the memory requirement, we consider the minimal memory case, i.e. m = 1 .
The proposed Min-AM is a variant of AM( 1 ) and the convergence of Min-AM is established in this section.
3.1 The basic Min-AM
In this section, we consider the simple case of (1): the objective function is f ( x ) =
12 x T Ax − b T x where A is SPD.
Let s k :=
∆ x k − 1 , y k :
=
∆ r k − 1 , and choose m = 1 in (4).
The approximated inverse Hessian of AM(1) is
H k = β k
I − ( s k + β k y k )( y T k y k ) − 1 y T k , which is non-symmetric.
However, the Hessian matrix ∇ 2 f is naturally symmetric.
To achieve a better approximation of ∇ 2 f , we ﬁrst recursively modify the vector pair s k , y k to incorporate more information from the previous iterations.
Let p 1 = ∆ x 0 , q 1 = ∆ r 0 ∈ R d .
For k ≥ 2 , we construct p k and q k by
[Equation]
where ζ k =
( p T k − 1 q k − 1 ) − 1 ∆ r T k − 1 p k − 1 , assuming p T k − 1 q k − 1 (cid:54) = 0 .
Then, inspired by the Two-Grid Cycle method [53, Algorithm 13.2] in the multigrid techniques [11], we add an extra projection step to (3), and the resulting scheme is
[Equation]
where r (1) k :=
r k − q k Γ (1) k and β k > 0 .
Deﬁne r (2) k
= r (1)
k − β k Ar (1) k and r (3) k
= r (2) k − q k Γ (2) k .
It can be veriﬁed that r (1) k , r (2) k , and r (3) k are equal to the residuals at x (1) k , x (2) k , and x k +1 , respectively.
The coefﬁcients Γ (1) k and Γ (2) k are determined by applying the Galerkin’s condition of AM-I:
[Equation]
Assume p T k q k (cid:54) = 0 .
The scheme formulated by (8a)-(8c) gives the x k +1 by
[Equation]
leading to the basic Min-AM.
The derivation of (10) is in Appendix B.1.
It is worth mentioning that the approximated inverse Hessian H k in (10) is symmetric.
Due to the space limit, we give the details of the basic Min-AM in Algorithm 2 in Appendix B. Deﬁne P k
= ( p 1 , p 2 , . .
. , p k ) ,
Q k = ( q 1 , q 2 , . .
. , q k ) , we summarize the properties of the basic Min-AM in the next theorem.
Theorem 1.
Let f ( x ) =
12 x T Ax − b T x where A is SPD and { x k } be the sequence generated by the basic Min-AM for solving (1) .
x ∗ is the exact solution.
Then, the following properties
hold: (i) (cid:107) p k (cid:107) 2 > 0 , range( P k ) =
K k ( A, r 0 ) , range( Q k ) =
A K k ( A, r 0 ); (ii) Q k = − AP k , p i ⊥
q j (1 ≤ i (cid:54) =
j ≤ k ); (iii) r (1) k ⊥ range( P k ) and x (1) k
= x 0 + z k , where z k = arg min
z ∈K k
( A,r 0 ) (cid:107) x 0 + z − x ∗ (cid:107) A .
Moreover, if
(cid:107)
r
(1) k (cid:107)
2 = 0 , then x k +
= x ∗ .
The proof is deferred to Appendix B.2.
The property (i) ensures p T k q k (cid:54)
=
during the iterations, so ζ k and the update (10) are well deﬁned.
The property (iii) indicates x (1) k
= x CG k , where x CG k is the k -th iterate of the conjugate gradient (CG) method [32].
Also, due to Proposition 1 in Appendix B.2, the intermediate iterate ¯ x k of AM-I( ∞ ) satisﬁes ¯ x k = x CG k , so the basic Min-AM is essentially equivalent to the full-memory AM-I when solving unconstrained strongly convex quadratic problems.
In this sense, Min-AM can incorporate more historical information than AM(1).
Remark 2.
Although Min-AM and CG are equivalent for solving SPD linear systems, their numerical performance has a large difference for general nonlinear programming as shown in our experiments.
Remark 3.
The H k in (10) is reminiscent of the memoryless BFGS method [4], but the key difference lies in the construction of p k and q k in (7) .
In memoryless BFGS, there are no correction terms ( ζ k ≡ 0 ), and it is generally not equivalent to BFGS [45] and CG.
See Appendix A.3 for more details.
3.2 The restarted Min-AM
For general nonlinear optimization, since global convergence may be unavailable for Min-AM as a counter example exists for AM [41], we introduce the restarting strategy to the basic Min-AM and establish the local convergence analysis.
Our purpose is to prove the faster convergence of the restarted Min-AM than that of gradient descent (GD), and sharpen the existing results of AM [59, 21].
Restarting Strategy.
Let m k be the number of iterations from the last restart.
Min-AM restarts, i.e., setting m k = 0 and p k = q k
=
0 , when any of the following conditions is violated:
[Equation]
where m is a constant positive integer, and 0 < τ < 1 , η > 0 are constants.
In the interval between two successive restarts, the iterations are the basic Min-AM iterations.
The check (11a) is to limit the number of high-order terms of errors accumulated during iterations.
The check (11b) is for the numerical stability of Min-AM since p T k q k appears in the denominators in (10).
The check (11c) is to control divergence.
The algorithm is shown in Algorithm 1.
Let
B ρ ( x ) :=
{ y ∈ R d |(cid:107) y − x (cid:107) 2 ≤ ρ } denote the ball centered at x with radius ρ .
We impose the following assumptions on f that are standard in the convergence analysis of quasi-Newton methods [45] and the regularized nonlinear acceleration method [56].
Assumption 1.
f : R d → R is twice Lipschitz continuously differentiable in a local region of a local minimizer x ∗ , and the Hessian matrix A :
= ∇ 2 f ( x ∗ ) is SPD.
Input :
[Equation] [Equation]
[Equation] [Equation]
1: p
[Equation] [Equation]
[Equation]
=
[Equation]
2: for k = 0 , 1 , . . .
, until convergence do 3: r
[Equation] [Equation]
) 4: if m k > m or (cid:107) r k (cid:107)
[Equation]
> η (cid:107)
[Equation]
(cid:107) 2 then 5: m k =6: end if 7: if m k > 0 then 8:
[Equation]
[Equation] [Equation]
11: if | p T k q k | < τ | p T k − m k +
q k − m k +
| then 12: m k =13: end if 14: end if 15: if m k =
then 16: p
[Equation]
=
[Equation] [Equation]
d 17: end if 18:
[Equation]
19: m
[Equation]
m
[Equation]
+ 1 20: end for 21: return x k
From Assumption 1, there exist positive constants ˆ ρ, µ, L , and ˆ κ such that for all x ∈ B ˆ ρ ( x ∗ ) , µ ≤ y T ∇ 2 f ( x ) y y T y ≤ L, for all y ∈ R d and y (cid:54)
= 0 , (12a) (cid:107)∇ f ( x ) − ∇ 2 f ( x ∗ )( x − x ∗
[Equation]
Here, µ is the strong convexity constant, and L, ˆ κ are Lipschitz constants of ∇ f and ∇ 2 f respectively.
With the restarting strategy, we apply a multi-step analysis and has the following convergence theory:
Theorem 2.
Suppose that the Assumption 1 holds.
Let { x k } be the sequence generated by the restarted Min-AM for solving (1) and θ k = max {| 1 − β k L | , | 1 − β k µ |} ≤ θ for a constant θ ∈ (0 , 1) .
Then there exists a positive constant ρ ≤ ˆ ρ such that for any x 0 ∈ B ρ ( x ∗ ) , we have
[Equation]
Moreover, if m k = d , then (cid:107) x k +1 − x ∗ (cid:107) 2 =
[Equation]
x
[Equation]
The proof is in Appendix C.1, where a more general theorem (Theorem 6) is given.
The result from
Theorem 2 suggests that the restarted Min-AM method has a local linear-quadratic convergence rate, and the rate has optimal dependence on the condition number.
Compared to the existing results that show the local linear convergence rate of AM is no worse than that of gradient descent (GD) method [59, 16], the bound (13) clearly improves the convergence rate of GD.
Remark 4.
Classical nonlinear CG methods [17] are also memory-efﬁcient with recursively modiﬁed searching directions, but the step size relies on line-search or Hessian-vector products which can be time-consuming for solving large-scale optimization problems.
Remark 5.
The momentum based methods, e.g. Nesterov’s accelerated gradient (NAG) method [44], also achieve the optimal convergence rate if the parameters µ and L are known.
In restarted Min-AM, we give an economical method that approximates the best step size 2 / ( L + µ ) in the next subsection.
3.3 Eigenvalue estimates and the choice of mixing parameter
The mixing parameter β k is critical for AM and AM-I, and an improper choice of β k can lead to divergence [22].
Fortunately, the spectrum of the Hessian can be economically estimated as a by-product of Min-AM.
To better explain the idea, we consider min f ( x ) =
12 x T
Ax − b T x where
A is SPD.
We apply a projection method [54] to estimate the eigenvalues:
[Equation]
where v ∈ R d is the approximate eigenvector sought in the Krylov subspace K k
( A, r 0 ) , and λ ∈ R is an eigenvalue estimate, which is known as a generalized Ritz value [42].
Our goal is to compute λ .
Since range( P k ) =
K k ( A, r 0 ) as shown in Theorem 1, we can select v = P k y, y ∈ R k .
Then the projection condition (14) leads to
[Equation]
From the construction of P k and Q k , we can derive the three-term recurrence relations (see details in
Appendix D.1): there exists a tridiagonal matrix ¯ T k ∈ R ( k +1)
× k , such that
[Equation]
Then, we have P T k AAP k
= P T k
AP k
+1 ¯ T k .
By Theorem 1, P T k
Ap k +1 =
− Q T k p k +1 =
Thus,
[Equation]
where T k ∈ R k × k is obtained from ¯ T k by deleting its last row.
By (15) and (17), we obtain T k
y
= λy, which indicates that the generalized eigenvalue problem (15) is reduced to solving the eigenvalues of a tridiagonal matrix T k .
Moreover, the three elements of the k -th column of ¯ T k are easy to obtain from the coefﬁcients Γ (1) k , Γ (2) k , β k − 1 , and β k on the ﬂy (see details in Appendix D.1): t
[Equation]
where Γ (2)0 := 0 .
For solving general nonlinear optimization with the restarted Min-AM, suppose that m k ≥ 1 .
We can also construct a tridiagonal matrix T k ∈ R m k × m k starting from the ( k − m k ) -th iteration with the nonzero elements deﬁned using the coefﬁcients during the iterations (see Deﬁnitionin Appendix D.2).
We use the eigenvalues of T k as the eigenvalue estimates of the Hessian.
Theorem 3.
Suppose that Assumption 1 holds, and { x k } is the sequence generated by the restarted
Min-AM for solving (1) .
Assume that β j ∈ [ β, β (cid:48) ] ( ∀ j ≥ 0 ) for some positive constants β and β (cid:48) , and there are positive constants η 0 , τ 0 such that (cid:107)∇ f ( x j ) (cid:107) 2 ≤ η 0 (cid:107)∇ f ( x 0 ) (cid:107) 2 (0 ≤ j ≤ k + 1) , | 1 − Γ (1) j | ≥ τ 0 (1 ≤ j ≤ k ) ; λ is an eigenvalue of T k constructed by restarted Min-AM.
Let Θ ( m k ) denote the set of Ritz values computed by an m k -step
A -norm based Lanczos algorithm (Algorithmin Appendix D.2) for ∇ f 2 ( x ∗ ) with starting vector ∇ 2 f ( x ∗ )( x ∗ − x k − m k ) .
Then there is a positive constant ρ ≤ ˆ ρ such that for any x 0 ∈ B ρ ( x ∗ ) , we have
[Equation]
The proof is in Appendix D.2.
Since the Ritz value ˆ λ approximates a true eigenvalue of ∇ 2 f ( x ∗ ) [54],
Theorem 3 suggests that the restarted Min-AM can give reliable eigenvalue estimates if (cid:107) x 0 − x ∗ (cid:107) 2 is sufﬁciently small.
At the k -th iteration, where m k ≥ 2 , let ˜ µ and ˜ L be the eigenvalues of T k − 1 with the smallest absolute value and the largest absolute value, respectively.
We use | ˜ µ | and | ˜ L | to estimate µ and L .
Then we can take the mixing parameter as β k = 2 / ( | ˜ µ | + | ˜ L | ) to estimate the optimal value 2 / ( µ + L ) .
The total computational cost is O ( m 3 k − 1 ) ﬂops as it requires solving the eigenvalues of T k − 1 [24].
Since m k (cid:28) d in practice, this strategy can be a useful and economical option.
Remark 6.
The spectrum of Hessian is important for determining the step sizes.
Once we obtain the estimates of µ and L by Min-AM, we can apply them to many ﬁrst-order methods.
3.4 The stochastic Min-AM
For many applications in deep learning, the large data size prohibits the evaluation of the full gradient, and the stochastic optimization stands out as a solution.
Speciﬁcally, we consider developing a stochastic version of Min-AM to solve the nonconvex optimization problem:
[Equation]
where f i : R d → R may be nonconvex.
Since T can be extremely large, we sample a mini-batch
S k ⊆
[ T ] := { 1 , 2 , . .
, T } and use f S k ( x k ) :
= 1 n k (cid:80) i ∈ S k f i ( x k ) to estimate the full gradient ∇ f ( x k ) , where n k :=
| S k | is the batch size.
Then, r k := −∇ f S k ( x k ) is an estimate of −∇ f ( x k ) .
Due to the noise in the gradient estimates and the nonconvexity of f , a direct application of the deterministic Min-AM to (20) may be problematic.
For example, the searching direction may be not a descent direction since H k can be indeﬁnite.
Hence, we should introduce some fundamental modiﬁcations to the basic Min-AM.
Regularization.
Inspired by [23], we deﬁne the function
[Equation]
Let Φ k :=
Φ( p k , q k , δ (2) k ) for some δ (2) k > 0 .
We substitute Φ k for the p T k q k appeared in (10) and deﬁne ρ k = Φ † k , i.e. ρ k = 0 if Φ k = 0 and ρ k =
Φ − 1 k if Φ k (cid:54) = 0 .
The resulting regularized Min-AM update is x k +
= x k + H Ak r k , where
[Equation]
Damping.
We propose a specialized damping technique for Min-AM, which interpolates the updates of stochastic gradient descent [50] (SGD) and Min-AM, and is described as
[Equation]
where x Gk is the SGD update, x Ak is the Min-AM update, and α k is the damping parameter.
It can be seen that SGD and Min-AM serve as the predicted step, and x k +1 is a weighted average of them.
Moreover, the inverse Hessian approximation
[Equation]
provided that α k ∈ [0 , 1) and H Ak (cid:23) 0 .
By incorporating damping and regularization, we obtain the stochastic Min-AM (sMin-AM), and the algorithm is summarized in Algorithm 4 in Appendix E.
To analyze the convergence properties, we ﬁrst impose the following assumptions on the objective function f , which are the same as [61, 62].
Assumption 2.
f : R d → R is continuously differentiable, and f ( x ) is lower bounded by a real number f low for ∀ x ∈ R d . ∇ f is globally L -Lipschitz continuous.
Assumption 3.
At each iteration k , the gradient estimate ∇ f ξ k ( x k ) satisﬁes E ξ k [ ∇ f ξ k ( x k )]
= ∇ f ( x k ) , E ξ k [ (cid:107)∇ f ξ k ( x k ) − ∇ f ( x k ) (cid:107) 22 ] ≤ σ 2 , where σ > 0 , and ξ k ∈ [ T ] , k = 0 , 1 , . .
, are independent samples, and the random variable ξ k (for a given k ) is independent of { x j } kj =0 .
The diminishing condition about β k is
[Equation]
With chosen β k , we choose the α k and δ (2) k of sMin-AM to meet the following conditions:
Assumption 4.
Given constants µ ∈ (0 , 1) , C 1 > 0 , and C 2 > 0 , we have α k ∈ [0 , min { C 1 β 1 / 2 k , 1 − µ } ] , ρ k ≤ 0 , and − ( ρ k /β k ) (cid:107)
p k
(cid:107) 22 − 2 ρ k (cid:107)
p k
(cid:107) 2 (cid:107)
[Equation]
(cid:107)
[Equation]
(cid:107)
[Equation]
(cid:107)
[Equation]
We establish the convergence results of sMin-AM for nonconvex stochastic optimization and the proofs can be found in Appendix E.2 and Appendix E.3.
Theorem 4.
Suppose that Assumptions 2-4 hold and { x k } is the sequence generated by sMin-AM with batch size n k ≡ n ≤ T .
If β k ∈ (0 , µ 2 L
[Equation] [Equation]
2 ] and satisﬁes (25), then
[Equation]
and for all k , there exists M f > 0 such that E [ f ( x k )] ≤ M f .
If E ξ k
[ (cid:107)∇ f ξ k ( x k ) (cid:107) 22 ] ≤ M g , ∀ k where M g > 0 is a constant, we have
[Equation]
[Figure]
Theorem 5.
Suppose that Assumptions 2-4 hold.
{ x k } N − 1 k =0 are the ﬁrst N iterations generated by sMin-AM with batch size n k ≡ n ≤ T , and β k = min {
[Equation]
Dσ √ N } , where ˜ D is a problem-independent constant.
R is a random variable following P R ( k ) :
= Prob { R = k } = 1 /N , k = 0 , . . . , N − 1 .
Then
[Equation]
where D f := f
( x 0 ) − f low and the expectation is taken with respect to R and { S j } N − 1 j =0 .
To ensure
E [ (cid:107)∇ f ( x R ) (cid:107) 22 ] ≤ (cid:15) , the number of iterations is O (1 /(cid:15) 2 ) .
Remark 7.
The Theorem 4 establishes the global convergence of sMin-AM for solving the problem (20) , and Theorem 5 indicates that sMin-AM attains the O (1 /(cid:15) 2 ) iteration complexity which is asymptotically optimal for optimization methods based on a stochastic ﬁrst-order oracle [43].
Experiments
We ﬁrst tested the basic Min-AM in solving strongly convex quadratic optimization (cf.
Theorem 1) and the restarted Min-AM on logistic regression (cf.
Theorem 2).
Then, we applied the sMin-AM to train deep neural networks on CIFAR [38] and ImageNet [19].
Details about the experimental settings and more numerical results can be found in Appendix F.
Strongly convex quadratic optimization.
As shown in Figure 1(a), the ﬁrst intermediate residual r (1) k of Min-AM exactly matches the k -th residual of CG, which veriﬁes Theorem 1.
Since X T k R k can be ill-conditioned when k is large, it is found that AM-I( ∞ ) fails to coincide with CG and Min-AM in the later iterations.
The eigenvalue estimates computed by Min-AM also well approximate the true eigenvalues of the Hessian matrix, which is consistent with our analysis in Section 3.3.
In
Appendix F.1, we also give a discussion about the cost of the eigenvalue estimation procedure.
Regularized logistic regression.
We conducted the regularized logistic regression on the datasets “madelon” and “a9a” from LIBSVM [14].
The compared methods were GD, nonlinear CG (NCG) with line-search, Nesterov’s accelerated gradient (NAG), L-BFGS( m ), AM( m ), and the ST-AM.
We applied the standard k -step Lanczos algorithm
[24] to compute 100 Ritz values of ∇ 2 f ( x ∗ ) , in which the smallest and largest ones were used for the setting of µ, L in NAG.
The results in Figure 1 show that Min-AM is competitive with AM(20).
It converges much faster than AM(1), L-BFGS(1)
(i.e. memoryless BFGS), NCG, and NAG, and outperforms ST-AM.
Also, in both datasets, the eigenvalue estimates from Min-AM are accurate enough to approximate the largest and the smallest Ritz values, even though the objective function is non-quadratic.
More results including the effect of β k can be found in Appendix F.
[Table]
[Table] [Table]
[Table] [Table]
[Figure]
a i n
[Figure]
o
[Figure]
Experiments on CIFAR.
We applied sMin-AM to train VGG
[57], ResNet18/20/44/56 [30],
WideResNet16-4 [66] (abbr.
WRN16-4) on CIFAR-10, and train ResNet18, ResNeXt50 [64],
DenseNet121 [34] on CIFAR-100.
The compared optimizers were SGDM [49], Adam [36], AdaHessian [65], stochastic AM (SAM) [62], and ST-AM [63].
Since the experimental setting was the same as [62, 63], their results were used for reference.
Table 1(a) reports the ﬁnal test accuracy of each optimizer for training 160 epochs.
It shows sMin-AM improves SAM(1) and has comparable accuracy to SAM(10).
sMin-AM can also use fewer epochs for the training.
By setting the accuracy of SGDM as baseline, Table 1(b) shows the memory, per-epoch time, and the total time for an optimizer to achieve a comparable accuracy to SGDM (more details about the number of epochs and ﬁnal test accuracy can be found in Table 2 in Appendix F.3.2).
sMin-AM signiﬁcantly reduces the memory cost of SAM(10) and the total training time is less than SGDM.
Experiments on ImageNet.
We trained ResNet50 on ImageNet with SGDM and sMin-AM.
Figureshows that the training process of sMin-AM is faster than SGDM.
sMin-AM can achieve both higher train accuracy and higher test accuracy in fewer epochs.
In our test, we found the memory cost of sMin-AM was 1.04 times that of SGDM, and the per-epoch training time of sMin-AM was 1.
times that of SGDM.
Hence, for neural network training on a larger dataset, sMin-AM can still be competitive with SGDM.
Conclusion
In this paper we propose Min-AM, a variant of AM(1) with minimal memory size.
Min-AM only stores one recursively modiﬁed vector pair and is essentially equivalent to the full-memory Type-I
AM in strongly convex quadratic optimization.
We establish the convergence properties of Min-AM under deterministic and stochastic settings.
For deterministic optimization, we also propose an economical method to estimate the spectrum of Hessian, which can be used to adaptively choose the mixing parameters.
The experimental results validate the effectiveness of our methods.
Acknowledgments and Disclosure of Funding
This work was supported by the National Key R&D Program of China (No. 2021YFA1001300),
National Natural Science Foundation of China (No.61925601), National Key Basic Research Program of China (2020YFB0204800), Tsinghua University Initiative Scientiﬁc Research Program, National
Natural Science Foundation of China (No.11901338) and Huawei Noah’s Ark Lab.
We thank all anonymous reviewers for their valuable comments and suggestions on this work
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Appendix G. (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix B, Appendix C, Appendix D, and
Appendix E.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix F and the supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We reported standard deviation instead.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix F. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[No] The assets are publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Contents 1 Introduction
The Anderson mixing scheme
The Min-AM methods 3.
The basic Min-AM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The restarted Min-AM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Eigenvalue estimates and the choice of mixing parameter . . . . . . . . . . . . . .
The stochastic Min-AM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4 Experiments 5 Conclusion
A Comparisons with conventional methods
A.1 Relationship between AM-I, Min-AM, and Newton’s method . . . . . . . . . . . .
A.2 Comparison with the conjugate gradient/residual method . . . . . . . . . . . . . .
A.3 Comparison with the BFGS method . . . . . . . . . . . . . . . . . . . . . . . . .
A.4 Comparison with the momentum-based method . . . . . . . . . . . . . . . . . . .
A.5 Comparison with related variants of Anderson mixing . . . . . . . . . . . . . . . .
B Details of the basic Min-AM
B.1 Derivation of the basic Min-AM . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Convergence analysis . . .
C Details of
the restarted Min-AM
C.1 Proof of Theorem
D Details of the eigenvalue estimation procedure
D.1 Analysis of the quadratic case . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E Details of the stochastic Min-AM
E.1 Some useful results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.2 Proof of Theorem 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.
Proof
of
Theorem
F
Experimental details
F.
Strongly convex quadratic problem . .
F.
Regularized logistic regression problem . .
F.
Deep neural network training problem . . . . . . . . . . . . . . . . . . . . . . . .
F.3.
Hyperparameter setting of the stochastic Min-AM . . . . . . . . . . . . .
F.3.
Experiments on CIFAR-10 and CIFAR-100 . . . . . . . . . . . . . . . . .
F.3.
Experiment of training ResNet50 on ImageNet . . . . . . . . . . . . . . . .
F.3.
An additional experiment: adversarial training . . . . . . . . . . . . . . . .
G Limitations
A Comparisons with conventional methods
We compare the proposed Min-AM method with other related methods, including Newton’s method, the (nonlinear) conjugate gradient method, BFGS method, momentum-based method, and some other variants of Anderson mixing.
A.1 Relationship between AM-I, Min-AM, and Newton’s method
We ﬁrst give a new interpretation of the AM-I method here, which reveals the relationship between
AM-I and Newton’s method.
Recall that the historical information is stored in X k ,
R k ∈ R d
× m .
We have the decomposition of the solution space by R d = range( X k ) (cid:76) range( X k ) ⊥ .
Deﬁne V k ∈ R d × ( d − m ) whose columns form an orthonormal basis of range( X k ) ⊥ .
Then for any x ∈ R d , we have x = x k − X k γ − V k η , where γ ∈ R m , η ∈ R d − m .
As a result,
[Equation]
Suppose that f is twice differentiable and X T k ∇ 2 f ( x k )
X k is nonsingular.
We apply Newton’s method in the low dimensional subspace range( X k ) followed by a gradient descent with stepsize β k in the complementary subspace range( X k ) ⊥ .
Then
[Equation]
(30) where γ k =
( X T k ∇ 2 f ( x k )
X k ) − 1 X T k ∇ f ( x k ) , and η k = β k V T k ∇ f
(¯ x k ) .
When m (cid:28) d , the scheme (30) is more economical than applying the Newton’s method in the whole solution space R d .
Now, consider the simple case that f is a quadratic function: f ( x ) = 12 x T Ax − b T x .
The residual r k = −∇ f
( x k ) =
b − Ax k .
Then r k − r k − 1 = − A ( x k − x k − 1 ) , which implies R k = − AX k .
It follows that γ k
=
( X T k R k ) − 1 X T k r k .
Also, ¯ r k :=
r k − R k γ k
= −∇ f
(¯ x k ) .
So ¯ r k ⊥ range( X k ) due to the choice of γ k , i.e., a kind of the Galerkin’s projection condition [53].
Let U k be the matrix whose columns form an orthonormal basis of range( X k ) .
Then V k V T k
= I − U
k U T k
and U
T
k ¯ r k
=
For the gradient descent step, we have η k =
− β k V T
k ¯ r k , which implies
[Equation]
namely a mixing step.
Therefore, the AM-I method coincides with the scheme (30) for minimizing a quadratic function.
Since Min-AM is essentially equivalent to the full-memory AM-I when minimizing a strongly convex quadratic function, we know Min-AM is also closely related to the scheme (30).
In the general case, if the objective function f can be well approximated by a quadratic function in a local region around the optima, it is expected that AM-I and Min-AM can behave similarly to the scheme (30) when the iterates enter this local region.
A.2 Comparison with the conjugate gradient/residual method
The conjugate gradient (CG) method [32] and the conjugate residual (CR) method [53, Algorithm 6.20] are classical methods for solving SPD linear systems, where the CG method is extended to solving unconstrained optimization [45].
We ﬁrst discuss the connection between the proposed
Min-AM method and the CG/CR method.
Connection between Min-AM and CG/CR.
For solving linear systems, it has been established in [60] that the full-memory AM-I and AM are essentially equivalent to the full orthogonalization method (FOM) [52] and GMRES [51], respectively.
If the linear system is SPD, both FOM and
GMRES methods can be simpliﬁed to have short-term recurrences: FOM is equivalent to CG, and
GMRES is equivalent to CR.
Thus, the full-memory AM-I and AM are essentially equivalent to
CG and CR respectively, for solving SPD linear systems.
However, CG and CR are much more efﬁcient in terms of memory and per-step computational cost, which motivates the development of the proposed Min-AM method that also has short-term recurrences.
From Theorem 1, for solving
SPD linear systems, Min-AM is essentially equivalent to CG, FOM, and the full-memory AM-I.
Compared to CG, Min-AM has the advantage that it does not need explicit matrix-vector products to determine the step size.
The update scheme of Min-AM only depends on the historical iterations, which makes it easier to extend to the nonlinear case.
Min-AM can also have a similar convergence behaviour to CR for solving SPD linear systems, due to the close relationship between FOM and
GMRES [53, Section 6.5.7].
Now, we consider the unconstrained optimization problem
[Equation]
where f : R d → R is continuously differentiable and bounded from below.
The nonlinear CG (NCG) generates { x k } by the update scheme
[Equation]
where the step size α k is usually obtained by a line search, and the searching direction p k is constructed by
[Equation]
Here, r k :=
−∇ f ( x k ) and the β k is the momentum term.
There are several choices of β k , for example, the Fletcher-Reeves variant β k = (cid:107) r k +1 (cid:107) 22 (cid:107) r k (cid:107) 22 and the Polak-Ribière variant β k =
[Equation]
(See [45].)
However, the step size α k is not easy to obtain.
If f is a quadratic function, let A be the
Hessian of f .
The computation of α k involves a matrix-vector product, e.g.,
[Equation]
which ensures r k +1 ⊥ p k .
If ﬁnite difference is used to compute Ap k , it requires two gradient evaluations.
If f is a general nonlinear function, a line search is necessary to ensure the convergence of NCG: α k is chosen as an approximate solution to the problem
[Equation]
where p k is supposed to be a descent direction.
The d -step quadratic convergence proved in [17], which is similar to our result of the restarted Min-AM, was only established under the assumption of an exact line search, namely, (33) is exactly solved.
More practical choices of α k require some conditions to be satisﬁed.
For example, the strong Wolfe conditions are
[Equation]
where 0 < c 1 < c 2 < 1 .
Since the line search can incur many times of the backtracking procedure, the total number of gradient evaluations for NCG can be large.
In contrast, our proposed Min-AM method does not need explicit Hessian-vector products or line search to determine the step size, while the convergence results (cf.
Theorem 2) match those of
NCG with exact line search.
Therefore, Min-AM is more economical than NCG, and can have better convergence than the NCG with inexact line search.
Also, it should be pointed out that even in strongly convex quadratic optimization, Min-AM generally does not generate the same sequence of iterates as CG: They are equivalent in the sense that x (1) k = x CG k , where x (1) k is the intermediate step in (8a).
A.3 Comparison with the BFGS method
The BFGS constructs the approximate inverse Hessian by solving
[Equation]
[Equation]
where s k := x k − x k − 1 , y k := ∇ f ( x k ) − ∇ f ( x k − 1 ) .
The norm (cid:107) · (cid:107) F ( W ) is the weighted Frobenius norm (i.e., (cid:107) X (cid:107) 2 F ( W ) :
= (cid:107) W 1 / 2 XW 1 / 2 (cid:107) F for a matrix X ∈ R d × d ) and the weight matrix W satisﬁes
Ws k = y k .
The solution is
[Equation]
Note that the H k ∈ R d
×
d is recursively constructed and can be dense during the later iterations.
To reduce memory overhead, the limited-memory BFGS [39] is often used.
In memoryless BFGS, the previous approximate Hessian is always reset to I d , i.e.,
[Equation]
Memoryless BFGS has minimal memory size in L-BFGS, but has no equivalence to BFGS.
It is pointed in [45] that for the update x k +1 =
x
k − α k H
k ∇ f ( x k ) of memoryless BFGS, if α k is chosen by an exact line search, then memoryless BFGS is equivalent to CG for quadratic function.
However, an exact line search can incur prohibitive cost in practice.
The Min-AM method has a similar form to that of memoryless BFGS, but uses a recursively constructed vector pair.
Min-AM is essentially equivalent to the full-memory AM-I and CG for minimizing strongly convex quadratic functions.
In fact, let P k
=
( p 1 , . . .
, p k ) , Q k = ( q 1 , . . .
, q k ) , and deﬁne
[Equation]
assuming P T k Q k is nonsingular.
We call the iterations deﬁned by x k +
= x k + H Ak
r k as Scheme A.
Clearly, H Ak Q k = − P k holds.
Moreover, it can be proved by direct computation (using the properties in Theorem 1) that for strongly convex quadratic optimization, if ﬁxed β k is used, i.e., β k ≡ β , where β is a constant, we have
[Equation]
starting from H A 0 :=
βI .
Hence, the H Ak in (37) solves
[Equation]
where the weight matrix satisﬁes Wp k
= − q k .
The (39) is similar to (34), but based on modiﬁed vector pairs.
(Note that we use r k − r k − 1 = − ( ∇ f ( x k ) − ∇ f ( x k − 1 )) to construct q k , so there is a difference of sign.)
It can also be veriﬁed that for strongly convex quadratic optimization,
H k r k = H
Ak r k , where H k is deﬁned in (10).
So the iterations of Min-AM are identical to those of
Scheme A in this case.
In this sense, though only using one vector pair, Min-AM implicitly constructs
H Ak which can well approximate the inverse Hessian.
A.4 Comparison with the momentum-based method
The momentum-based method is similar to the scheme deﬁned in (31) and (32), but the momentum β k is different from that in CG.
We take the Nesterov’s accelerated gradient (NAG) method as an example.
The update scheme of NAG is deﬁned as
[Equation]
where L and µ are the Lipschitz constant and the strong convexity constant of ∇ f , respectively.
Unlike Min-AM, NAG generally does not form a symmetric approximation to the Hessian.
In
Figure 3, we show the convergence behaviours of NAG with different settings of µ and L .
It is found that the setting of L and µ can have a large effect on the convergence.
On the contrary, Min-AM forms symmetric Hessian approximations and can give useful information of L and µ via an economical eigenvalue estimation procedure.
[Figure]
A.5 Comparison with related variants of Anderson mixing
Compared with the original AM, AM-I, the regularized nonlinear acceleration scheme [55], and the stochastic AM [62], the proposed Min-AM has the minimal memory size, the same as that of
AM(1).
Nonetheless, Min-AM incorporates more historical information than AM(1) and AM-I(1), and is equivalent to the full-memory AM-I in solving strongly convex quadratic optimization.
The convergence analysis in Section 3.4 also shows that the stochastic Min-AM has similar convergence to that of the stochastic AM in theory.
The short-term recurrence AM (ST-AM) [63] stores two vector pairs and is equivalent to the fullmemory AM in solving strongly convex quadratic optimization, but the approximated Hessian from
ST-AM is generally not symmetric.
It is also worth pointing out that for AM and ST-AM, their existing theoretical results do not show the signiﬁcantly better convergence rate than the simple gradient descent method for unconstrained optimization.
Our proposed Min-AM further reduces the memory size of ST-AM and the faster convergence than gradient descent for general nonlinear optimization is rigorously justiﬁed in theory.
B Details of the basic Min-AM
The procedure of the basic Min-AM is described in Algorithm 2.
Next, we give the derivation of the basic Min-AM, and prove the related theoretical properties.
B.1 Derivation of the basic Min-AM
We give the derivation of the update scheme (10) for solving the quadratic optimization min x ∈ R d f ( x ) = 12 x T
Ax − b T x here.
Recall that the steps in one iteration of the basic Min-AM are:
[Equation] [Equation]
where r (1) k
= r k − q k Γ (1) k and β k > 0 .
Deﬁne r (2) k
= r (1)
k − β k Ar (1) k and r (3) k
= r (2) k − q k Γ (2) k .
The Γ (1) k and Γ (2) k are determined by imposing the projection conditions:
[Equation]
Here, we deﬁne p 1 = ∆ x 0 , q 1 = ∆ r 0 , and for k ≥ 2 , the construction of p k , q k at the beginning of the k -th iteration is
[Equation]
where ζ k =
( p T k − 1 q k − 1 ) − 1 ∆ r T k − 1 p k − 1 , assuming p T k − 1 q k − 1 (cid:54) = 0 .
[Equation]
[Equation] [Equation]
2: for k = 0 , 1 , . . .
, max _ iter do 3:
[Equation]
) 4: if k > 0 then 5:
[Equation]
Compute ∆
[Equation]
[Equation] [Equation]
8: end if 9:
[Equation] [Equation]
(Projection step:
[Equation] [Equation]
p
[Equation] [Equation]
(Mixing step) 12:
[Equation]
p
[Equation] [Equation]
(Projection step: r
[Equation]
p
[Equation]
) 14: if (cid:107) r (1) k (cid:107)
=
then 15: break 16: end if 17: end for 18: return x k
We ﬁrst prove that
[Equation]
(42) by induction.
By the deﬁnition of f and r k = −∇ f ( x k ) , it is clear that r k − r k − 1 = − A ( x k − x k − 1 ) .
So (42) holds for k = 1 .
For k ≥ 2 , suppose that (42) holds for k − 1 .
Then q k = ∆ r k −
[Equation]
− A ∆ x
[Equation]
+ Ap k − 1 ζ k = − Ap k , which completes the induction.
Suppose that p T k q k (cid:54) = 0 .
We have Γ (1) k
= ( p T k q k ) − 1 p T k r k .
For Γ (2) k , the exact solution is
[Equation]
due to p T k r (1) k
=
[Equation]
the symmetry of A
[Equation]
and (42).
As a result,
[Equation]
The corresponding inverse Hessian approximation is
[Equation]
The reason to use the projection condition of AM-I. Note that the Galerkin’s projection condition of AM-I is used to determine Γ (1) k and Γ (2) k .
On the other side, if the Galerkin’s condition of the original AM is used to determine Γ (1) k and Γ (2) k , the scheme (41) leads to
[Equation]
which is not a practical method since Aq k needs to be explicitly computed.
B.2 Proof of Theorem
Recall that the strongly convex quadratic optimization is formulated as
[Equation]
(47) where A ∈ R d × d is SPD, b ∈ R d .
Solving (47) is equivalent to solving the SPD linear system
Ax = b.
We ﬁrst state the relationship of the AM-I with the full orthogonalization method [52] (FOM) in the following propositions.
Let x FOM k and r FOM k :=
b − Ax FOM k denote the k -th FOM iterate and residual, respectively.
Deﬁne e j :=
, 1) T ∈ R j for j ≥ 1 .
The main results of the full-memory AM-I are stated in Proposition 1 and Proposition 2.
(Proposition 1 is a known result in [60].)
Proposition 1 (General linear system) .
For solving a general linear system Ax = b with the fullmemory AM-I, suppose that β k > 0 and the ﬁxed-point map is g ( x ) =
( I − A ) x + b .
If the initial point of AM-I is x 0 = x FOM, and X T j R j is nonsingular for j = 1 , . . .
, k , then the intermediate iterate ¯ x k satisﬁes ¯ x k = x FOM k .
We give the proof here, which is similar to [63, Proof of Proposition 1], but applies to the Type-I AM.
Proof.
The deﬁnition of the ﬁxed-point map suggests that the residual r k = g ( x k ) − x k =
b − Ax k .
From (5), for j = 1 , . . .
, k , the nonsingularity of X T j R j ensures that each Γ j is uniquely determined.
Thus the updates of AM-I are well deﬁned.
Since X T k R k is nonsingular, we have rank( X k )
=
rank( R k )
=
k .
We ﬁrst show range( X k ) =
K k ( A, r FOM) (49) by induction.
We abbreviate K k ( A, r FOM) as K k in this proof.
First, ∆ x 0 = β 0 r 0 = β 0 r
FOMsince x 1 = x 0 + β 0 r 0 .
If k = 1 , then the proof is complete.
Then, suppose that k > 1 and, as an inductive hypothesis, that range( X k − 1 ) =
K k − 1 .
From (3), we have x k +
= x k + β k r k −
X k + β k R k )Γ k .
Noting that R k
= − AX k , it follows that
[Equation] [Equation]
Since r 0 ∈ K k − 1 , and by the inductive hypothesis range( X k − 1 )
=
K k − 1 which also implies range( AX k − 1 ) ⊆
K k , we know ∆ x k − 1 ∈ K k .
Thus, range( X k ) ⊆
K k .
Since rank( X k ) =
k , namely dim(range( X k ))
= dim(
K k ) , we have range( X k ) =
K k , thus completing the induction.
As a result, we also have
[Equation] [Equation]
Recalling that to determine Γ k , we solve the projection condition:
[Equation]
The nonsingluarity of X T k R k ensures that Γ k is uniquely determined by
[Equation]
Also, since r k = b − Ax k = b − A ( x 0 + X k e k ) =
r 0 − AX k e k , we have r k − R k Γ =
r k + AX k Γ = r 0 − AX k e k + AX k Γ = r 0 − AX k ˜Γ , where ˜Γ =
e k − Γ , for ∀ Γ ∈ R k .
So Γ k solves (52) if and only if ˜Γ k =
[Equation]
Γ k solves
[Equation]
According to (49), the condition (54) is equivalent to
[Equation]
where range( X k ) =
K k .
Since the initializations are identical, the condition (55) for AM-I is the
Petrov-Galerkin condition for FOM.
Due to the nonsingularity of X T k R k , the solution of (54) is also unique.
Therefore, we have
[Equation]
In Proposition 1, the assumption that X T k R k is nonsingular is critical to ensure no stagnation occurs at the k -th iteration for solving a general linear system.
In fact, for SPD linear systems (48) or strongly convex quadratic optimization (47), when AM-I breaks down, i.e. X T k R k is singular, AM-I obtains the exact solution, as shown in the next proposition.
Proposition 2 (SPD) .
For applying the AM-I to minimize a strongly convex quadratic problem (47), or equivalently, solve an SPD linear system (48), suppose that β k > 0 and the ﬁxed-point map is g ( x )
=
( I − A ) x + b .
If the condition X T k R k is nonsingular holds for 1 ≤ k < s while failing to hold for k = s , where s ≥ 1 , then the residual of AM-I satisﬁes r s =
¯ r s − 1 = 0 .
We give the proof here, which is similar to [63, Proof of Proposition 2], but applies to the Type-I AM.
Proof.
The deﬁnition of g suggests that the residual r k = g
( x k )
− x k
= b − Ax k .
The relation
R k
= − AX k holds during the iterations and the nonsingularity of A implies rank( X k ) =
rank( R k ) .
Since A is SPD and X T k R k = − X T k AX k , it follows that X T k R k being nonsingular ⇔ rank( X k ) =
rank( R k )
=
k .
Hence rank( X k ) =
k holds for 1 ≤ k < s while failing to hold for k = s .
For s = 1 , since the ﬁrst step of AM is x 1 = x 0 + β 0 r 0 , the assumption rank( X 1 ) =
implies that rank( r 0 )
=
rank( X 1 ) = 0 , then r 1 = ¯ r 0 := 0 .
For s > 1 , ∆ x s − 1 = x s − x s − 1 = − X s − 1 Γ s − 1 + β s − 1 ¯ r s − 1 .
The rank deﬁciency of X s implies ∆ x s − 1 ∈ range( X s − 1 ) , which further implies ¯ r s − 1 ∈ range( X s − 1 ) .
So there exists ζ ∈ R s − 1 , such that ¯ r s − 1 =
X s − 1 ζ .
Due to ¯ r s − 1 ⊥ X s − 1 , we have
[Equation]
Because rank( X s − 1 ) = s − 1 , we have ζ = 0 , which implies ¯ r s − 1 = 0 .
Hence x s = ¯
x s − 1 and r s = ¯ r s − 1 = 0 .
It is also known that FOM method can be simpliﬁed to the conjugate gradient (CG) method [53] in this case.
Now we prove Theorem 1, following a similar procedure of [63, Proof of Theorem 1].
Proof of Theorem 1 .
The A -norm minimization problem in the property (iii) is equivalent to the
Galerkin condition [53], i.e. z k =
[Equation]
Also, r
[Equation]
=
[Equation]
Besides relations (i)-(iii), we add an auxiliary relation here:
[Equation]
( A, r 0 ) , where ¯Γ k ∈ R k .
We prove the relations (i)-(iv) by induction.
For k = 1 , since r 0 = r (1) (cid:54) = 0 , we have rank(∆ x 0 )
=
rank( X 1 ) = 1 and rank(∆ r 0 )
=
rank( R 1 )
= 1 .
So p k (cid:54) = 0 and p T k q k = − p T k
Ap k < 0 .
The relation (i) holds.
Since p 1 = ∆ x 0 , q 1 = ∆ r 0 , and ∆ r 0 = − A ∆ x 0 , the equality Q 1 = − AP 1 also holds.
Since r 1 = r 0 − β 0 Ar 0 and range( Q 1 ) = span { Ar 0 } , it is clear that r 1 = r 0 − Q 1 ¯Γ 1 ∈ K 2 ( A, r 0 ) , namely relation (iv).
Due to the projection step (Line 10 in Algorithm 2), r (1)⊥ range( P 1 ) and r (1)is unique, which is guaranteed by p T1 q 1 (cid:54) = 0 .
Also, r (1)1 = r 1 − q 1 Γ (1)1 = r 0 − β
Ar 0 − q 1 Γ (1)1 =
r
− Q 1 η 1 , where the last equality is due to span { Ar 0 } =
range( Q 1 ) .
For r FOM= r 0 − Az 1 , where z 1 ∈ K 1 ( A, r 0 ) , it holds r FOM⊥ K 1 ( A, r 0 )
=
range( P 1 ) .
As a result, both r (1)1 and r FOMare the oblique projections of r 0 onto the subspace range( P 1 ) ⊥ along range( Q 1 ) .
Since p T1 q 1 (cid:54) = 0 , the projection exists and is unique, which implies r (1) =
r FOM.
So x (1)1 = x FOM= x 0 + z 1 because their residuals are equal and A is nonsingular.
Hence the relation (iii) holds.
Suppose that k > 1 , and as an inductive hypothesis, the relations (i)-(iv) hold for j = 1 , . .
, k − 1 .
Consider the k -th iteration.
From Line 7 in Algorithm 2, q k ∈ span { ∆ r k − 1 , q k − 1 } , and p k ∈ span { ∆ x k − 1 , p k − 1 } .
Due to p T k − 1 q k − 1 = − p T k − 1 Ap k − 1 (cid:54) = 0 , we have q T k p k − 1 = 0 .
We ﬁrst prove that p k (cid:54) = 0 by contradiction.
If p k = 0 , then from Line 7 in Algorithm 2, ∆ x k − 1 ∈ span { p k − 1 } .
From Line 10, Line 11, and Line 13, we have
[Equation]
So r (1) k − 1 ∈ span { p k − 1 } since ∆ x k − 1 ∈ span { p k − 1 } .
Hence there exists ζ ∈ R , such that r (1) k − 1 =
p k − 1 ζ .
From the Line 10, we know r (1) k − 1 ⊥ span { p k − 1 } , so we have
[Equation]
Since p k − 1 (cid:54) = 0 , it follows that ζ = 0 which implies r (1) k − 1 = 0 .
It is impossible otherwise Algorithmhas terminated in the ( k − 1) -th iteration.
So p k (cid:54) = 0 .
Moreover p T k q k = − p T k
Ap k
(cid:54)
=
since A is SPD.
Since r (1) k − 1 =
r k − 1 − q k − 1 Γ (1) k − 1 , and r k − 1 ∈ K k
( A, r 0 ) , q k − 1 ∈ range( Q k − 1 ) =
A K k − 1 ( A, r 0 ) due to the inductive hypothesis, we have r (1) k − 1 ∈ K k
( A, r 0 ) , which together with (57) and p k − 1 ∈ range( P k − 1 ) =
K k − 1 ( A, r 0 ) infers ∆ x k − 1 ∈
K k ( A, r 0 ) .
As a result, p k ∈ span { ∆ x k − 1 , p k − 1 } ⊆
K k ( A, r 0 ) .
So range( P k ) =
range( P k − 1 , p k ) ⊆
K k ( A, r 0 ) .
Moreover, p k / ∈ range( P k − 1 ) .
We prove it by contradiction.
If p k ∈ range( P k − 1 ) , then from Line 7 in Algorithm 2, ∆ x k − 1 ∈ span { p k − 1 , p k } ⊆
range( P k − 1 ) , which together with (57) leads to r (1)
k − 1 ∈ range( P k − 1 ) .
Hence there exists ξ ∈ R k − 1 such that r (
[Equation]
k − 1 =
P k − 1 ξ .
From the inductive hypothesis, we know r (1) k − 1 ⊥ range( P k − 1 ) .
So we have
[Equation]
Since − P T k − 1 AP k − 1 = P T k − 1 Q k − 1 is nonsingular due to the inductive hypothesis p i ⊥ q j (1 ≤ i (cid:54) =
j ≤ k − 1) and p T i q i = − p T i Ap i (cid:54) =
0(1 ≤ i ≤ k − 1) as p i (cid:54) = 0 , it follows that rank( P k − 1 ) =
k − 1 .
So P T k − 1 P k − 1 is nonsingular which implies that ξ = 0 .
Then r (1) k − 1 =
P k − 1 ξ = 0 .
But it is impossible otherwise Algorithm 2 has terminated in the ( k − 1) -th iteration.
Hence p k / ∈ range( P k − 1 ) .
As a result, range( P k ) =
K k ( A, r 0 ) .
Because ∆ r k − 1 = − A ∆ x k − 1 and q k − 1 = − Ap k − 1 , Line 7 in Algorithm 2 infers q k = − Ap k .
So Q k = − AP k .
Hence range( Q k ) =
A K k ( A, r 0 ) .
To prove p i ⊥ q j for 1 ≤ i (cid:54)
=
j ≤ k , it sufﬁces to show q k ⊥ P
k − 1 .
From the construction of q k in
Line 7 in Algorithm 2 and p T k − 1 q k − 1 (cid:54) = 0 , we know q T k p k − 1 = 0 .
To further prove q k ⊥ range( P k − 2 ) ( k ≥ 3) , note that
[Equation]
where the second equality is a direct substitution with (57).
Therefore,
[Equation]
(58) where the second equality is due to q k − 1 ⊥ range( P k − 2 ) and A is SPD, and the third equality is due to r (1) k − 1 ⊥ range( P k − 1 )
=
K k − 1 ( A, r 0 ) and range( AP k − 2 )
=
A K k − 2 ( A, r 0 ) ⊆
K k − 1 ( A, r 0 ) .
As a result, we obtain
[Equation]
which is due to (58) and q k − 1 ⊥ range( P k − 2 ) .
Therefore, q k ⊥ range( P k − 1 ) .
Since A is SPD, we also have p T
[Equation]
Hence relation (ii) holds in the k -th iteration.
Next, we prove the relation (iv).
We have
[Equation]
where the last equality is due to r k −
= r
+ Q k − 1 ¯Γ k − 1 by the inductive hypothesis.
Since range( Q k
[Equation]
) =
[Equation] [Equation]
[Equation] [Equation]
A,
[Equation]
) , range( AQ
[Equation]
− 1 ) ⊆
A 2 K k − 1 ( A, r 0 ) ⊆
A K k
( A, r 0 ) , and Aq k − 1 ∈ range( AQ k − 1 ) , it is clear that r k
=
r 0 + Q k ¯Γ k ∈ K k
+
( A, r 0 ) for some ¯Γ k ∈ R k .
The relation (iv) is proved.
Finally, we prove the relation (iii).
For proving r (1) k ⊥ range( P k ) , note that r (1) k ⊥ span { p k } already holds due to the projection step (Line 9 and Line 10 in Algorithm 2) and p T k q k (cid:54) = 0 .
It sufﬁces to prove r (1) k ⊥ range( P k − 1 ) .
We ﬁrst prove r k ⊥ range( P k − 1 ) .
The projection step (Line 12 and Line 13 in Algorithm 2) at the ( k − 1) -iteration ensures r k ⊥ p
k − 1 .
It remains to prove r k ⊥ range( P k − 2 ) .
Because r k = r (2) k − 1 − q k − 1 Γ (2) k − 1 = r (1) k − 1 − β k − 1 Ar (1) k − 1 − q k − 1 Γ (2) k − 1 , and r (1) k − 1 ⊥ range( P k − 2 ) , q k − 1 ⊥ range( P k − 2 ) , ( Ar (1) k − 1 ) T P k − 2 =
( r (1) k − 1 ) T ( AP k − 2 ) =
as range( AP k − 2 ) ⊆
K k − 1 ( A, r 0 ) =
range( P k − 1 ) , it follows that r k ⊥ range( P k − 2 ) .
Therefore, r k ⊥ range( P k − 1 ) .
Also noting that q k ⊥ range( P k − 1 ) , we have r (1) k
= r k − q k Γ (1) k ⊥ range( P k − 1 ) .
So r (1) k ⊥ range( P k ) .
To prove x (1) k
= x FOM k := x 0 + z k , where z k ∈ K k
( A, r 0 ) should satisfy r 0 − Az k ⊥ K k
( A, r 0 ) , ﬁrst we have r k = r
+ Q k ¯Γ k , where ¯Γ k ∈ R k .
Hence r (1) k
= r k − q k Γ (1) k
= r 0 + Q k ¯Γ k − q k Γ (1) k
= r 0 − Q k η k , where η k ∈ R k .
Since P T k Q k is nonsingular, there exists a unique η k ∈ R k
such that r (1) k ⊥ range( P k ) , i.e., r (1) k is the oblique projection of r 0 onto the subspace range( P k ) ⊥ along range( Q k ) .
On the other side, for FOM, r FOM k
[Equation]
also the oblique projection of r 0 onto the subspace range( P k ) ⊥ along range( Q k ) .
So r (1) k
= r FOM k , which further indicates x (1) k
= x
FOM k .
Hence, the relation (iii) holds.
With relations (i)-(iv) being proved in the k -th iteration, we complete the induction.
B.3 Convergence analysis
Since the basic Min-AM is equivalent to FOM (or CG) method in the sense that ¯ x (1) k
= x
FOM k = x CG k for strongly convex quadratic optimization, where x CG k is the k -th iterate of CG, we can obtain the convergence result as a corollary of Theorem 1.
Corollary 1.
For solving the strongly convex quadratic problem (47) , let x ∗ be the exact solution, and { x k } is the sequence generated by the basic Min-AM described in Algorithm 2.
Deﬁne θ k = (cid:107) I − β k A (cid:107) 2 .
Then
[Equation] [Equation]
where P k denotes the space of polynomials of degree not exceeding k .
Moreover, the algorithm ﬁnds the exact solution in at most ( d + 1) iterations.
Proof.
From Theorem 1, the classical convergence analysis of the CG method [28, 53], and assuming that CG and Min-AM are initialized from the same starting point, i.e., x CG 0 = x 0 , we have
[Equation]
Since x (2) k
= x (1) k
+ β k r (1) k , it follows that
[Equation]
Hence,
[Equation]
61) where we use the fact that
[Equation] [Equation]
Noting that x k +
= x
[Equation]
− p
[Equation]
+
[Equation]
Γ
[Equation]
⊥ p k , it follows that
[Equation]
So
[Equation]
Combining (60), (61), and (64) yields (59).
Since CG iterates at most d steps to obtain x ∗ , the Min-AM will obtain the same solution in at most ( d + 1) steps.
C Details of the restarted Min-AM C.1 Proof of Theorem
For the proof of Theorem 2 about the restarted Min-AM, we use a local quadratic model inspired by the techniques in [17, 18].
The second-order Taylor expansion ˆ f of f around x ∗ is
[Equation]
where A := ∇ 2 f ( x ∗ ) .
We compare the iterates { x k } generated by Min-AM to the iterates { ˆ x k } generated by applying Min-AM to minimizing the quadratic model (65).
More precisely, we give the following deﬁnition.
Deﬁnition 1.
Let the mixing parameter { β k } be chosen to satisfy β ≤ | β k | ≤ β (cid:48) , where β and β (cid:48) are two positive constants.
The sequences { x k } and { ˆ x } are generated by the following two processes:
(1) Process I: Solve the optimization problem (1) with the restarted Min-AM (see Algorithm 1), and the resulting sequence is { x k } .
(2) Process II: Apply the basic Min-AM to minimize ˆ f ( x ) in each interval between two successive restarts in Process I. Speciﬁcally, let m k ,
β
k be the same as those in Process I. Deﬁne ˆ r
k =
−∇
ˆ f
(ˆ x k )
Then, ˆ x k = x k , and ˆ p k
= ˆ q k =
∈ R d , if m k = 0 .
For m k ≥ 1 ,
[Equation]
ζ k , where ˆ ζ
[Equation]
(ˆ p
[Equation] [Equation]
p T
[Equation]
−
[Equation]
; deﬁning ˆ r (1) k
= ˆ r k − ˆ q k ˆΓ (1) k , the update of ˆ x k is
[Equation]
, where ˆΓ (1) k
= (ˆ p T k
ˆ q
[Equation]
ˆ p T
k
ˆ r k ,
[Equation] [Equation]
[Equation]
where ˆΓ (2) k
=
[Equation] [Equation]
q
[Equation]
r
[Equation]
By comparing { x k } and { ˆ x } , we ﬁrst have the following lemma.
Lemma 1.
Suppose that Assumption 1 holds for the optimization problem (1) .
For the sequences { x k } and { ˆ x k } deﬁned in Deﬁnition 1, if (cid:107) x 0 − x ∗ (cid:107) 2 is sufﬁciently small and there exists a positive constant η 0 such that (cid:107)∇ f ( x j ) (cid:107) 2 ≤ η 0 (cid:107)∇ f
[Equation]
x
[Equation]
2 for j = 0 , . . .
, k , then
[Equation]
Proof.
Besides (67) and (68)
[Equation]
we will also prove the following relations.
[Equation]
Here, for convenience, we deﬁne ζ k = ˆ ζ k
=
if m k = 0 .
We ﬁrst prove (69).
Recall that r k = −∇ f
( x k ) .
Due to (12a), we have
[Equation] [Equation]
Note that (cid:107) r k (cid:107) 2 ≤ η 0 (cid:107) r 0 (cid:107) 2 .
By choosing (cid:107) x 0 − x ∗ (cid:107) 2 ≤ µ ˆ ρ η 0 L , we can ensure
[Equation]
k
[Equation]
x
[Equation] [Equation]
[Equation]
r k
[Equation]
Then (69) holds.
Since (cid:107) r j (cid:107) 2 ≤ η 0 (cid:107) r 0 (cid:107) 2 holds for j = 0 , . . .
, k , (78) implies that a sufﬁciently small (cid:107) x 0 − x ∗ (cid:107) 2 ensures (cid:107) x
k − m k − x ∗ (cid:107) 2 ≤ η 0 Lµ (cid:107) x 0 − x ∗ (cid:107) 2 is sufﬁciently small.
Next, we prove (67), (68), and (70)-(76) by induction.
For k = 0 , we have Γ (1) k
[Equation]
=
[Equation]
= 0 .
The relations (70)-(74), and (76) clearly hold.
Also, due to (12b),
[Equation]
namely (67).
Note that x k +
= x (2) k
= x k + β k r k
and ˆ x k +
=
ˆ x (2) k
= ˆ x k + β k ˆ r k in this case.
Then (68) and (75) follow from
[Equation]
Suppose that k ≥ 1 , and as an inductive hypothesis, the relations (67), (68), and (70)-(76) hold for j = 0 , . .
. , k − 1 .
Consider the k -th iteration.
If m k = 0 , i.e., a condition in (11a)-(11c) is violated at the beginning of the k -th iteration, then ˆ x k = x k .
The same as the case that k = 0 , (67), (68), and (70)-(76) hold.
Consider the nontrivial case that m k ≥ 1 .
From (11c), it follows that
[Equation]
Therefore,
[Equation]
Since x k ∈ B ˆ ρ ( x ∗ ) , we have
[Equation]
where the second inequality is due to (12b) and (12a), and the last inequality is due to (79) and the inductive hypothesis (68).
Hence, the relation (67) holds.
Next, we prove the relation (70).
Due to the check (11b), with κ := τµ 3 β 2 , we have
[Equation]
for j = k − m k + 1 , . . .
, k .
If m k = 1 , then ζ k = ˆ ζ k = 0 , and (70) holds.
For m k > 1 , we have
[Equation]
due to (71) and (cid:107) ∆ r k − 1 (cid:107)
[Equation]
≤ (cid:107)
[Equation]
x ∗ (cid:107) 2 ) .
Next, if p T k − 1 ∆ r k − 1 (cid:54) = 0 , then
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
82) where a := 1 − ˆ p T k − 1 ∆ˆ r k − 1 p T k − 1 ∆ r k − 1 , b := 1 − p T k − 1 q k − 1 ˆ p T
k
− 1 ˆ q k − 1 .
We have | ζ k | · | a | = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) p
[Equation]
k −
[Equation]
∆
[Equation]
p
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) · (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) p
[Equation]
k
[Equation]
ˆ
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
p T k
[Equation]
k
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
From (67) and (71),
[Equation]
We also have
[Equation]
Hence,
[Equation]
Combining (83) and (84), we have
[Equation]
Note that
[Equation]
and
[Equation]
where the existence of the constant c 1 is guaranteed by (86)
[Equation]
and the last inequality holds by choosing (cid:107) x
[Equation]
−
[Equation]
1 , which ensures
[Equation]
Then, we have
[Equation] [Equation]
As a result, by (85), (88), (81), and (82), we have
[Equation]
Now consider the case that p T k − 1 ∆ r k − 1 = 0 .
It is clear that ζ k = 0 .
Then
[Equation]
where the inequality is due to (84) and (87).
Hence the (70) holds.
For the relation (71), it holds since
[Equation]
Next
[Equation]
we prove (72)
[Equation]
We have
[Equation] [Equation]
Therefore
[Equation]
The (72) is proved.
Now
[Equation]
we prove (73)
[Equation]
which follows a very similar procedure in proving (70).
For the completeness, we still give the proof.
We have
[Equation]
[Table]
Next, if p T k r k (cid:54) =
[Table]
then | Γ (1) k − ˆΓ (1) k | =
| Γ (1) k | · (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 1 − ˆΓ (1) k Γ (1) k
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
| Γ (1) k | · (cid:12)(cid:12)(cid:12)(cid:12)
1 − ˆ p
T k ˆ r k ˆ p T k
ˆ q k · p T k q k p T k r k (cid:12)(cid:12)(cid:12)(cid:12) =
| Γ (1) k | · (cid:12)(cid:12)(cid:12)(cid:12)
1 − ˆ p
T k ˆ r k p T k r k · p T k q k ˆ p T k ˆ q k (cid:12)(cid:12)(cid:12)(cid:12) =
| Γ
[Equation]
|
[Equation]
where a 1 := 1 − ˆ p T k ˆ r k p T k r k , b 1 := 1 − p T k q k ˆ p T k ˆ q k .
We have
[Equation]
From (67) and (71) which have been proved,
[Equation]
We also have
[Equation]
Hence,
[Equation]
Combining (96) and (97), we have
[Equation]
Note that
[Equation]
and
[Equation]
where the existence of the constant c 2 is guaranteed by (99), and the last inequality holds by choosing
[Equation]
which
[Equation]
Then, we have
[Equation]
As a result, by (98), (101), (94), and (95), we have
[Equation]
Now consider the case that p T k r k = 0 .
It is clear that Γ (1) k = 0 .
Then
[Equation]
where the inequality is due to (97) and (100).
Hence the (73) holds.
Now
[Equation]
we prove (74).
We have
[Equation]
Then, (75) holds as
[Equation]
is a consequence of (103) and (104).
Next, we prove (76)
[Equation]
which follows a very similar procedure in proving (73).
For the completeness, we still give the proof.
We have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Next, if q T k
[Equation]
then
[Equation]
where
[Equation]
From (71) and (74) which have been proved,
[Equation]
We also have
[Equation]
Hence,
[Equation] [Equation]
Combining (107) and (108), we have
[Equation]
From (101),
[Equation] [Equation]
As a result
[Equation]
by (109), (110), (105)
[Equation]
and (106)
[Equation]
we have
[Equation]
Now consider the case that q T k r (1) k = 0 .
It is clear that Γ (2) k = 0 .
Then
[Equation]
where the inequality is due to (108) and (100).
Hence the (76) holds.
Finally, we prove (cid:107) x k
+1 − ˆ x k +1 (cid:107)
= ˆ κ O ( (cid:107) x k − m k − x ∗ (cid:107) 22 ) .
It follows from (8c) that
[Equation]
where the last equality is due to (75), (71), (72), and (76) that have been proved.
Therefore, the relation (68) holds.
As a result, we complete the induction.
The relations (67) and (68) are proved.
Based on Lemma 1 and Corollary 1, we can obtain the convergence theorem of the restarted Min-AM.
Theorem 6.
Suppose that Assumption 1 holds for the optimization problem (1) .
Let { x k } denote the sequence of iterates generated by the restarted Min-AM, and deﬁne θ k = (cid:107)
I − β k A (cid:107) 2 .
Assume β j ∈ [ β, β (cid:48) ]
( j ≥ 0) for some positive constants β and β (cid:48) .
If (cid:107)∇ f ( x j ) (cid:107) 2 ≤ η 0 (cid:107)∇ f ( x 0 ) (cid:107) 2 (0 ≤ j ≤ k ) for a constant η 0 > 0 , and x 0 is sufﬁciently close to x ∗ , then
[Equation]
where P m k is the space of polynomials of degree not exceeding m k .
As a result,
[Equation]
Proof.
From Corollary 1, the auxiliary sequence { ˆ x k } in Deﬁnition 1 satisﬁes
[Equation]
[Equation]
Then (112) follows from (114), (68), and the fact that (cid:107) x k
[Equation]
− ˆ x k
[Equation]
k
[Equation]
(cid:107) 2 .
Since A is SPD, we can choose the Chebyshev polynomial for p ∈ P m k to obtain (e.g., see
Section 6.11.3 in [53])
[Equation]
Like (63) in the proof of Corollary 1, we also have (cid:107) p ( A ) (cid:107)
A = (cid:107) p ( A ) (cid:107) 2 .
Then (cid:107) p ( A )( x k − m k − x ∗ ) (cid:107)
A ≤ (cid:107) p ( A ) (cid:107) A (cid:107) x k − m k − x
[Equation]
(cid:107)
A =
[Equation]
p
[Equation]
A )
[Equation]
2 (cid:107) x k
[Equation]
m k − x ∗ (cid:107) A .
With (115) and (112), the bound (113) holds.
Remark 8.
In Theorem 6, we do not assume each β k is chosen such that θ k = (cid:107) I − β k A (cid:107) 2 < 1 .
It is expected that when θ k is not too large, and m k is sufﬁciently large, the minimization problem on the right-hand side of (112) can dominate the convergence rate, thus leading to fast convergence.
However, in the case that an improper choice of β k causes θ k (cid:29) 1 , the restarted Min-AM may have an erratic convergence behaviour or even diverge if m k is too small.
Remark 9.
From (112) , a large m k can make the ﬁrst-order term diminish signiﬁcantly.
However, the high-order terms of errors ˆ κ O ( (cid:107) x k − m k − x ∗ (cid:107) 22 ) can be large since they are accumulated from the last restart.
In practice, we can ﬁrst choose large m , small τ , and large η in (11a) - (11c) so that the restarts do not occur too frequently.
When the restarted Min-AM has a problematic convergence behaviour due to the high nonlinearity of ∇ f , more frequent restarts are necessary.
Now, we give the proof of Theorem 2, which can be obtained from Theorem 6.
Proof of Theorem 2 .
Under the assumptions of Theorem 2, we prove that
[Equation]
and there exists a constant η 0 > 0 such that
[Equation]
We prove (116) and (117) by induction.
For k = 0 , (116) and (117) hold.
Suppose that k ≥ 0 , and the results hold for k .
We establish the results for k + 1 .
From (114) in the proof of Theorem 6, it follows that
[Equation] [Equation]
Since θ k ≤ θ , we know β k ∈ [ 1 − θ µ , 1+ θ L ] .
With the inductive hypothesis (117), for sufﬁciently small (cid:107) x 0 − x ∗ (cid:107) 2 , the relations (67) and (68) hold.
From (68), it follows that (cid:107) x k
+1 − ˆ x k +1 (cid:107)
A
=
ˆ κ
O ( (cid:107) x k − m k − x ∗ (cid:107) 2 A ) .
Then, there exists a constant c 1 such that (cid:107) x k
+1 − ˆ x k +1 (cid:107)
A ≤ ˆ κc 1 (cid:107) x k − m k − x ∗ (cid:107) 2 A .
With (118), we have
[Equation]
Hence, by choosing (cid:107) x 0 − x ∗ (cid:107) 2 ≤
[Equation]
1 , which ensures (cid:107)
[Equation]
√ L (cid:107) x 0 − x ∗ (cid:107)
[Equation]
≤
[Equation]
2ˆ κc 1 due to the inductive hypothesis (116), it follows that
[Equation]
namely (116) for k +1 .
Also, (cid:107)
[Equation]
So we can impose (cid:107) x 0 − x ∗ (cid:107) 2 ≤ √ µ ˆ
[Equation]
√
L to ensure x k +1 ∈ B
ˆ ρ ( x ∗ ) , which further yields that (cid:107) r k +1 (cid:107) 2 ≤
L (cid:107) x k +1 − x ∗ (cid:107)
[Equation]
≤ L
[Equation]
(cid:107)
[Equation]
(cid:107)
[Equation]
(cid:107) 2 , namely (117) for k + 1 , and η
[Equation]
=
L
[Equation]
µ √ µ .
Hence, we complete the induction.
The convergence result (13) follows from (113) in Theorem 6.
If m k = d , then ˆ x k +1 = x ∗ from Corollary 1.
Therefore, (cid:107) x k +1 − x ∗ (cid:107)
=
ˆ κ
[Equation]
D Details of the eigenvalue estimation procedure D.1 Analysis of the quadratic case
The procedure of the basic Min-AM leads to three-term recurrences for P k and Q k .
Proposition 3.
Under the same assumptions of Theorem 1 for solving strongly convex quadratic optimization (47) , we have
[Equation]
where the coefﬁcients are given by
[Equation] [Equation]
and φ k :=
Γ (1) k
+ Γ (2) k
+ ζ k +1 .
Then φ k = Γ (2) k
and there exists a tridiagonal matrix ¯ T k ∈ R ( k +1)
× k , such that
[Equation]
Proof.
The relations can be veriﬁed by direct computation.
In the ( k + 1) -th iteration, since x
[Equation]
123) we have
[Equation]
where the last term vanishes due to
[Equation]
since r k ⊥ p k − 1 .
It also indicates that
[Equation]
Now, it follows from Equation (124)
that
[Equation]
The second relation about Aq k can be obtained by noticing that q k = −
Ap k .
The tridiagonal matrix ¯
T k is
[Equation]
Input : Starting vector v ∈ R d
Output : Approximate eigenvalues { θ i } ki = 1: r = v, v 0 =2: β 0 = (cid:107) r (cid:107) A 3: for j = 1 , 2 , . .
, until convergence do 4: v j = r/β j − 1 5: r =
Av j 6: r = r − v j − 1 β j −7: α j = v T j Ar 8: r = r − v j α j 9: β j = (cid:107) r (cid:107)
A 10: if j = k then 11: break 12: end if 13: end for 14: Construct tridiagonal T k ∈ R k k : The main diagonal is { α i } ki =1 ; the subdiagonals are { β i } k − 1 i = 15: Compute the eigenvalue decomposition T k = S k Θ ( k
S T k 16: return Θ ( k )
D.2 Proof of Theorem
For solving general nonlinear optimization, the iterations between two successive restarts in the restarted Min-AM are the basic Min-AM iterations.
Hence, at the ( k + 1) -th iteration, a tridiagonal matrix T k ∈ R m k × m k can be constructed based on the coefﬁcients starting from the ( k − m k ) -th iteration.
We formulate it in the following deﬁnition.
Deﬁnition 2.
For solving optimization problem (1) with the restarted Min-AM (see Algorithm 1), suppose that m k ≥ 1 , and t ( k − 1) k , t ( k ) k , t ( k +1) k are obtained by (121) .
Then T k ∈ R m k × m k is deﬁned as
T k = t k if m k = 1 , and T k =
( ¯ T k − 1 , t k ) if m k ≥ 2 .
Here, ¯ T k =
( T T k , t ( k +1)
k
[Equation]
where e m k is the last column of I m k ; t k = t ( k ) k
if m k = 1 , and t k
= (0 , . . .
, 0 , t ( k − 1) k , t ( k ) k ) T ∈
R m k if m k ≥ 2 .
Now, we prove Theorem 3.
Proof of Theorem 3.
Without loss of generality, we consider the case that no restart happens during the iterations, i.e. m k
= k .
Let A :=
∇ 2 f ( x ∗ ) .
The A -norm based Lanczos algorithm with a starting vector v for computing eigenvalues of an SPD matrix A is described in Algorithm 3, which is a modiﬁcation of the Algorithm 4.6 in [6] with
A -norm.
It forms an A -orthonormal basis V k of the Krylov subspace K k
( A, ˆ r 0 ) by the Lanczos
A -orthogonalization, i.e. V T k AV k
= I , where ˆ r
= A ( x ∗ − x 0 ) .
Then the A -norm based Lanczos algorithm seeks to ﬁnd ˜ λ ∈ R and z ∈ R k such that
[Equation]
Then we have
[Equation]
[Equation]
The eigenvalues of ( AV k ) T AV k are the approximations to the true eigenvalues of A .
For Min-AM, we still use an auxiliary solving procedure of Min-AM on the local quadratic approximation (65), i.e. Process II in Deﬁnition 1.
The same as P k , T k , ¯ T k , t ( k − 1) k , t ( k ) k , t ( k +1) k in Process I, the notations ˆ P k , ˆ T k , ¯ˆ T k , ˆ t ( k − 1) k , ˆ t ( k ) k , ˆ t ( k +1) k are deﬁned for Process II, correspondingly.
Then
the
Min-AM in Process II seeks ˆ λ ∈ R and y ∈ R k , such that
[Equation]
namely,
[Equation]
According to Proposition 3, we have A ˆ P k
= ˆ
P k +1 ¯ˆ T k .
Then
[Equation]
Then the eigenvalues of
[Equation]
are the approximations of the true eigenvalues of A .
ˆ T k is tridiagonal but generally not symmetric.
However, it is clear that ˆ T k is similar to
[Equation]
which is symmetric.
The columns of U k :
=
ˆ P k
( ˆ P T k
A ˆ P k ) − 1 / 2 are A -orthonormal and also span the same Krylov subspace K k
( A, ˆ r 0 ) , which implies that there exists an orthonormal matrix S ∈ R k × k , such that V k
= U k S .
Therefore,
[Equation]
Since S T S = I , the eigenvalues of (129) and (130) are identical.
Recall that we have the explicit forms of T k , ˆ T k by (121), and we restate it here for convenience.
t
[Equation]
β k − 1 − ˆ φ k β k (cid:33) , ˆ t ( k +1) k
= − β k (1 − ˆΓ (1) k ) , (132b) and φ k :=
Γ (1) k
+Γ (2) k
+ ζ k
+1 , ˆ φ k := ˆΓ (1) k + ˆΓ (2) k
+ ˆ ζ k
+1 .
With the already proved relations (69)-(76), we can compute the bounds of t ( j ) k − ˆ t ( j ) k , j = k − 1 , k, k + 1 .
Let (cid:15) :
=
ˆ κ O ( (cid:107) x 0 − x ∗ (cid:107) 2 ) to simplify the notation.
Then
[Equation]
where we use the fact that | Γ (1) k − ˆΓ (1) k
| =
ˆ κ O ( (cid:107) x 0 − x ∗ (cid:107) 2 ) , and the assumption that | 1 − Γ (1) k | ≥ τ 0 .
We also have | φ k |
=
O (1) and | ˆ φ k − φ k | =
ˆ κ O ( (cid:107) x k − m k − x ∗ (cid:107) 2 ) .
Hence, | t ( k − 1) k
[Equation] [Equation]
With m k ≤ m , we have that
[Equation]
Deﬁne ˆ D k
= ˆ P T
k
A ˆ P k .
It is clear that ˆ D k is a diagonal matrix, and the i -th diagonal element is − ˆ
p T
i ˆ q i .
From (100), (71), and (72), it follows that
[Equation]
By (130), we know that ˆ T k can be diagonalized.
There exists orthonormal matrix W k ∈ R k × k and diagonal matrix ∧ k ∈ R k × k , such that
[Equation]
Therefore, ˆ T k
= ˆ D − 1 / 2 k
[Equation]
k ˆ D 1 / 2 k can be diagonalized.
Then with (137), (138), and applying the Bauer-Fike Theorem [33], we know that for an eigenvalue λ of T k , the following result holds:
[Equation]
where σ ( ˆ T k ) denotes the spectrum of ˆ T k .
E Details of the stochastic Min-AM
Now, we prove the theorems about the stochastic Min-AM (sMin-AM) in Section 3.4.
The algorithm is given in Algorithm 4.
For brevity, we use δ k to denote δ (2) k , i.e. δ k ≡ δ (2) k .
Our proofs follow those of SAM [62] and ST-AM [63].
From Assumption 3, for the mini-batch gradient f S k ( x k ) = 1 n k (cid:80) i ∈
S k f i ( x k ) , where n k = | S k | , we have
[Equation]
Note that the update of sMin-AM (Lines 10-12 in Algorithm 4) can be written as x k +
= x k + H k r k , where r k = −∇ f S k ( x k ) , and for k ≥ 0 ,
[Equation] [Equation]
To prove the theorems, we need H k to be positive deﬁnite, and show that the noise in gradient estimates is suppressed during iterations.
We ﬁrst state some useful results about sMin-AM, then we prove Theorem 4 and Theorem 5.
[Equation]
: for k = 0 , 1 , . . .
, until convergence, do
[Equation] [Equation]
: if k > 0 then
[Equation]
end if
[Equation] [Equation]
[Equation] [Equation]
Apply learning rate schedule of α k , β k 14: end for 15: return x k
E.1 Some useful results
Lemma 2.
Suppose that the sequence { x k } is generated by sMin-AM.
If there are constants µ ∈ (0 , 1) , C 2 > 0 such that α k ∈ [0 , 1 − µ ] , ρ k ≤ 0 and − ρ k (cid:107) p k (cid:107) 22 − 2 β k ρ k (cid:107) p k (cid:107) 2 (cid:107) q k (cid:107)
+ β k ρ 2 k (cid:107) p k (cid:107) 22 (cid:107) q k (cid:107) 22 ≤ β k C 2 , then we have that
[Equation] [Equation]
Proof.
Since
[Equation]
we have
[Equation]
Hence,
[Equation]
It follows that
[Equation]
Because ρ k ≤ 0 , it follows that H Ak (cid:23) 0 .
Hence H k (cid:23) (1 − α k )
β
k I (cid:23) µβ k I .
Remark 10.
The conditions that ρ k ≤ 0 and
[Equation]
≤ β k C 2 for a positive constant C 2 can be fulﬁlled with some proper choice of δ k .
For example, if Csatisﬁes (1 + 1 2 β k ) 2 ≤ C 2 and we choose δ k ≥ 1 2 + β − 1 k , then it can be proved that the conditions hold.
Moreover, in practice, δ k can be chosen as a constant, as shown in our experiments.
With Lemma 2, we can prove the convergence of sMin-AM for full-batch training.
Theorem 7.
Suppose that Assumption 2 holds and { x k } is the sequence generated by full-batch sMin-AM, i.e. n k = T .
Given constants µ ∈ (0 , 1) , C 2 > 0 , let β k = β ∈
0 , µ L (1+ C 2 ) 2 ] be a constant, α k ∈ [0 , 1 − µ ] .
The δ (2) k is chosen such that ρ k ≤ 0 and − ρ k (cid:107) p k (cid:107) 22 − 2 β k ρ k (cid:107) p k (cid:107) 2 (cid:107) q k (cid:107) 2 + β k ρ 2 k (cid:107) p k (cid:107) 22 (cid:107) q k (cid:107) 22 ≤ β k C 2 .
Then
[Equation]
in the N -th iterations.
To ensure 1 N (cid:80) N − 1 k =0 (cid:107)∇ f ( x k ) (cid:107) 22 < (cid:15) , the number of iterations is O (1 /(cid:15) ) .
Proof of Theorem 7 .
With Lemma 2, we have
[Equation]
and
[Equation]
Then, under Assumption 2, we have
[Equation]
where the last inequality is due to 0 < β ≤ µ L (1+ C 2 ) 2 .
Thus, f ( x k +1 ) − f ( x k ) ≤ − 12 βµ
(cid:107)∇ f ( x k ) (cid:107) 22 .
Summing both sides of this inequality for k ∈ { 0 , . .
, N − 1 } and recalling f ( x ) >
f low in Assumption 2 gives
[Equation]
Rearranging and dividing further by N yields (149).
The next lemmas are about the stochastic case.
Lemma 3.
Suppose that Assumption 3 and Assumption 4 hold and { x k } is the sequence generated by sMin-AM with β k > 0 .
Then
[Equation]
Proof.
(i) From Lemma 2, we have
[Equation]
From Assumption 3, we have
[Equation]
With (152) and (153), we obtain (151a).
ii)
Recalling that H
= β
I , the result holds for k = 0 .
Deﬁne (cid:15) k
= ∇ f S k ( x k ) − ∇ f ( x k ) = − r k − ∇ f ( x k ) , then H k r k = H k ( − (cid:15) k − ∇ f ( x k )) .
First, we have
[Equation]
which implies
[Equation]
Let M k :=
α k ( H Ak − βI ) , then H k = β
k
I + M k .
With the assumption (141a)
[Equation]
i.e. E S k [ (cid:15) k ]
= 0 , we have
[Equation]
Using the Cauchy-Schwarz inequality with expectations
[Equation]
we obtain
[Equation]
We now bound
[Equation]
With (156), we have (cid:107) M k (cid:15) k
[Equation]
≤
[Equation]
(cid:107) 2 , which implies
[Equation]
where the last inequality is due to (141b).
Now we can obtain the bound of | E S k [ ∇ f ( x k ) T H k (cid:15) k ] | as follows (cf.
[Equation]
With the inequality (154) and (158)
[Equation]
we obtain
[Equation]
where the last inequality is due to α k ≤ C 1 β 1 / 2 k .
Using Lemma 3 we obtain the descent property of sMin-AM:
Lemma 4.
Suppose that Assumptions 2-4 hold, β k ∈ (0 , µ 2 L (1+ C 2 ) 2 ] , and { x k } is the sequence generated by sMin-AM.
Then
[Equation]
Proof.
Due to Assumption 2, we have
[Equation] [Equation]
Taking expectation with respect to the mini-batch S k on both sides of (161) and using Lemma
[Equation]
we obtain
[Equation]
Then (162) combined with the assumption β k ≤
[Equation]
implies (160).
E.2 Proof of Theorem
Following the proofs in [61, 62, 63]
[Equation]
we introduce the deﬁnition of a supermartingale .
Deﬁnition 3.
Let {F k } be an increasing sequence of σ -algebras.
If { X k } is a stochastic process satisfying (
i) E
[ | X k | ] < ∞ , (ii) X k ∈ F k for all k , and (iii) E [ X k +1 |F k ] ≤ X k for all k , then { X k } is called a supermartingale.
Proposition 4 (Supermartingale convergence theorem, see, e.g., Theorem 4.2.12 in [20]) .
If { X k } is a nonnegative supermartingale, then lim k →∞ X k
→ X almost surely and E [ X ] ≤ E [ X 0 ] .
Now, we prove Theorem 4 of sMin-AM.
Proof of Theorem 4 .
(i) Deﬁne φ k = β k µ 4 (cid:107)∇ f
( x k ) (cid:107) 22 and ˜ L = 12 (cid:0)
µ − 1 C 21 C 22 + L (1 + C 2 ) 2 (cid:1) , γ k = f
( x k ) + ˜ L σ 2 n (cid:80) ∞ i = k β 2 i .
Let F k be the σ -algebra measuring φ k , γ k , and x k .
From (160) we know that for any k ,
[Equation]
which implies that E
[ γ k +1 − f low |F k ] ≤ γ k − f low − φ k .
Since φ k ≥ 0 , we have 0 ≤ E
[ γ k − f low ] ≤ γ 0 − f low < + ∞ .
As the diminishing condition (25) holds, we obtain E [ f ( x k )] ≤ M f for some constant M f > 0 .
According to Deﬁnition 3, { γ k − f low } is a supermartingale.
Therefore, Proposition 4 indicates that there exists a constant γ such that lim k →∞ γ k
= γ with probability 1, and E [ γ ] ≤ E [ γ 0 ] .
Note that from (163) we have E [ φ k ] ≤ E [ γ k ] − E
[ γ k +1 ] .
Thus,
[Equation]
which further yields that
[Equation]
Since (cid:80) ∞ k =0 β k = + ∞ , it follows that (26) holds.
(ii)
If the noisy gradient is bounded, i
e.,
[Equation]
where M g > 0 is a constant, then a stronger result can be obtained.
For any given (cid:15) > 0 , according to (26), there exist inﬁnitely many iterates x
k such that (cid:107)∇ f ( x k ) (cid:107) 2 ≤ (cid:15) .
Then if (27) does not hold, there must exist two inﬁnite sequences of indices { s i } , { t i } with t i >
s i , such that for i = 0 , 1 , . . .
, k = s i
[Equation] [Equation]
[Equation]
Then from (164) it follows that
[Equation]
probability
[Equation]
which implies that
[Equation]
According to (144), we have
[Equation]
168) where the last inequalities are due to Cauchy-Schwarz inequality and (165).
Then it follows from (168) that
[Equation]
which together with (167) implies that (cid:107) x
t i − x s i (cid:107) 2 → 0 with probability 1, as i → + ∞ .
Hence, from the Lipschitz continuity of ∇ f , it follows that (cid:107)∇ f ( x t i ) − ∇ f
( x s i ) (cid:107) 2 → 0 with probability 1 as i → + ∞ .
However, this contradicts (166).
Therefore, the assumption that (27) does not hold is not true.
E.3 Proof of Theorem
Proof of Theorem 5 .
According to (162) in Lemma 4, we have
[Equation]
where the expectation is taken with respect to { S j } N − 1 j =0 .
Deﬁne
P
[Equation]
(170) then
[Equation]
Let ˜ D be a problem-independent constant.
If we choose β k = β := min {
[Equation]
2 L (1+ C 2 ) 2 , ˜ Dσ √ N } , and n k ≡ n , then the deﬁnition of P R simpliﬁes to P R ( k ) = 1 /N .
From (171) we have
E [ (cid:107)∇ f
[Equation]
x
[Equation]
[Equation] [Equation]
+
[Equation]
Therefore, to ensure E [ (cid:107)∇ f ( x R ) (cid:107) 22 ] ≤ (cid:15) , the number of iterations is O (1 /(cid:15) 2 ) .
F Experimental details
We implemented the algorithms based on PyTorch 2 and used one GeForce RTX 2080 Ti GPU for training neural networks (except that four GPUs were used for training ResNet50 on ImageNet).
We tested the basic Min-AM on solving strongly convex quadratic optimization, the restarted Min-AM on solving regularized logistic regression, and the stochastic Min-AM on training neural networks.
F.1 Strongly convex quadratic problem
The experiments on solving strongly convex quadratic optimization were conducted to verify Theorem 1 about the basic Min-AM.
The problem is
[Equation]
where A ∈ R (cid:96) × d , b ∈ R (cid:96) .
For the test, we ﬁrst generated a random matrix A ∈ R 500 × 100 and a random vector v ∈ R 100 following Gaussian distribution, then b ∈ R 500 was obtained as b
= Av .
The ﬁxed step size β for the gradient descent (GD) was chosen by a grid search in { 0 . 001 , 0 . 002 , . . .
We set β = 0 .
001 that guarantees the convergence of GD.
The AM-I(1) and the full-memory AM-I (i.e. AM-I( ∞ )) also used the same mixing parameter setting β k = 0 . 001 .
For Min-AM, we set the initial mixing parameter β 0 = 1 , and the later mixing parameters { β k } were adaptively determined based on the eigenvalue estimates (see Section 3.3).
Figure 4(a) compares the convergence behaviours of different methods in terms of relative residual norm.
It can be observed that due to the improper initial setting of β 0 for Min-AM, Min-AM does not perform well in the beginning.
Nonetheless, as shown in Figure 4(b), the β k of Min-AM can be quickly adapted to the optimal value 2 / ( µ + L ) = 1 .
67 × 10 − 3 based on the eigenvalue estimates.
In Figure 4(c), we show the effects of β k on the full-memory AM-I and Min-AM, where both methods used ﬁxed β k chosen from { 0 . 01 , 0 . 1 , 1 } .
(For Min-AM, we disable the adaptive choice of β k .)
It should be noted that the GD method diverges when choosing the step size in { 0 . 01 , 0 . 1 , 1 } , which suggests that θ k > 1 for the Min-AM in these settings.
Nevertheless, we ﬁnd that Min-AM still converges to the tolerance of (cid:107) r k (cid:107) 2 / (cid:107) r 0 (cid:107) 2 ≤ 10 − 11 , which validates the convergence property (59) in Corollary 1, i.e., the minimization problem in (59) dominates the convergence when k is
[Figure]
[Figure] [Figure]
large.
Note that AM-I fails to coincide with Min-AM in the later iterations.
It is due to the fact that
AM-I needs to solve ( X T k R k ) − 1 X T k r k to determine Γ k , where the matrix inverse operation can have numerical weakness when k is large
[Figure] [Figure]
[Figure]
“eigenvalue” denotes the exact eigenvalues, and “Min-AM” denotes the eigenvalue estimates computed by Min-AM.
Results about the problem with different condition numbers.
In Figure 5, we show the convergence of each method, the eigenvalue estimates from Min-AM, and the β k of Min-AM, in the tests with different condition numbers characterized by L/µ .
The eigenvalues of A ∈ R 100 × 100 were chosen to be in (0 , 1] with equal interval.
The results show that Min-AM is competitive with CG and the full-memory AM-I. Min-AM also gives accurate enough estimates of the largest and the smallest eigenvalues, and the β k is quickly adapted to approximate the optimal value 2 / ( µ + L ) .
Cost of the eigenvalue estimation procedure.
In our implementation of the eigenvalue estimation procedure in the Min-AM, we used the function “np.linalg.eigvals” in NumPy to compute the eigenvalues of T k constructed by (18), which needs O ( k 3 ) ﬂops.
Hence, the cost of the eigenvalue estimation increases with increasing k , and at the k -th iteration, the ratio of this cost to the total cost is O ( k 3 ) /
( O ( k 3 ) + O ( d 2 )) , where O ( d 2 ) is due to the gradient evaluation.
To investigate the
[Figure]
practical performance of the eigenvalue estimation, we applied the basic Min-AM to solve problem (172) of dimension d = 1000 , 2000 , 4000 , where each A ∈ R d × d was generated following Gaussian distribution, and the maximal iteration number is max _ iter = 1001 for Min-AM.
As can be found in Figure 6(a), the additional computational time incurred by the eigenvalue estimations is marginal when d is large and k is small.
Figure 6(b) and Figure 6(c) show the case that d = 4000 .
(The cases d
= 1000 and d = 2000 have similar results.)
We ﬁnd that β k is quickly adapted to the optimal value 2 / ( µ + L ) within very small number of iterations.
So we can adaptively choose β k in the beginning, and ﬁx the obtained β k and disable the eigenvalue estimation procedure in the later iterations to reduce the computational cost.
Figure 6(c) shows that the eigenvalue estimates computed by Min-AM (at the last iteration) well approximate the exact eigenvalues.
F.2 Regularized logistic regression problem
The regularized logistic regression problem is formulated as
[Equation]
where ξ i ∈ R d is the i -th data sample and y i = ± 1 is the corresponding label.
The regularization parameter w = 0 .
We used the datasets “madelon” and “a9a” from LIBSVM [14] that are two-class classiﬁcations:
• madelon: training data size: 2000; feature size: 500; • a9a: training data size: 32561; feature size: 123.
To set proper µ and L for Nesterov’s accelerated gradient (NAG) method and check the eigenvalue estimates from Min-AM, we applied the standard Lanczos algorithm [24] to compute 100 Ritz values of ∇ 2 f ( x ∗ ) as the estimates of the true eigenvalues of ∇ 2 f ( x ∗ ) , where the minimizer x ∗ was obtained by solving (173).
For the test on madelon, Lanczos algorithm gave µ = 1 .
01 × 10 − 1 , L = 1 . 45 ; for the test on a9a, Lanczos algorithm gave µ = 1 .
× 10 − 1 , L = 8 .
64 × 10 − 1 .
For the gradient descent (GD), the step size was tuned and set as 1, which was proper for both datasets.
We used the Polak-Ribière variant of nonlinear conjugate gradient (NCG), and the step size was determined by line search with cubic interpolation, where the strong Wolfe conditions were checked.
The L-BFGS used the Barzilai–Borwein step size as the initial guess of the approximate inverse Hessian.
For AM and ST-AM, the mixing parameter was set as 1.
The AM-I performed similarly to AM in this test, so we only report the results of AM.
For the restarted Min-AM, we set τ = 10 − 16 for the madelon test and τ = 10 − 32 for the a9a test.
m
= 100 for both datasets.
η was set as a large number since Min-AM converged for these tests.
Besides, we computed at mosteigenvalue estimates in Min-AM (between two successive restarts) to determine β k .
Results about the choice of β k .
In Figure 7, we show additional results of the tests on madelon dataset.
(1) To test the effect of β k on Min-AM, we disabled the adaptive choice of β k and used
[Figure]
ﬁxed β k chosen from { 1 , 2 , 5 } .
It is observed in Figure 7(a) that GD does not converge if the step size β k is chosen from { 2 , 5 } , which suggests that θ k > 1 in Theorem 2 and Theorem 6.
However,
Min-AM still converges in these cases, which validates our discussion in Remark 8 in Appendix C.1:
the minimization problem on the right-hand side of (112) in Theorem 6 dominates the convergence when m k is large.
It also indicates that Min-AM is less sensitive to β k than GD.
(2) When adaptive choice of β k is used, we investigate the effect of the initialization β 0 on the convergence and the quality of eigenvalue estimates.
It is found in Figure 7(b) that even with the improper initialization β 0 = 50 , 100 , 1000 , the iterations still ﬁnally converge to an acceptable precision.
In fact, GD,
AM, and Min-AM diverge if using ﬁxed β k chosen from { 50 , 100 , 1000 } .
Hence, Min-AM with adaptive choice of β k is easy to apply since it discards the requirement of manually tuning the mixing parameter.
In Figure 7(c), we ﬁnd that the eigenvalue estimates can still roughly approximate the largest and the smallest Ritz values computed by the Lanczos algorithm, which veriﬁes Theorem 3.
[Figure]
Results of the problem with different regularization parameters.
We also tested Min-AM for the problem (173) with different settings of the regularization parameter w .
The results in Figure 8 also show that Min-AM signiﬁcantly improves the convergence of AM(1) and is comparable to AM(20) when w is small.
The comparison between the Ritz values computed by Lanczos algorithm and the eigenvalue estimates computed by Min-AM validates Theorem 3.
Since the Ritz values approximate the true eigenvalues of ∇ 2 f ( x ∗ ) , it is expected that Min-AM can give promising eigenvalue estimates of ∇ 2 f ( x ∗ ) , which accounts for the efﬁciency of Min-AM.
[Figure]
Running time.
In Figure 9, we report the convergence of each method with respect to the running time.
Min-AM has comparable performance to AM(20) in these tests, while using a smaller memory size.
F.3 Deep neural network training problem
The experiments about the stochastic Min-AM (sMin-AM) focused on training deep neural networks.
Since Min-AM can be viewed as a special case of sMin-AM with δ (1) k
= δ (2) k
= 0 , α k = 1 , the basic
Min-AM was also covered.
F.3.1 Hyperparameter setting of the stochastic Min-AM
The main hyperparameters of sMin-AM are α k , β k , δ (1) k , δ (2) k .
We set δ (1) k
= δ (2) k
= 1 , which ensures that H Ak is positive deﬁnite.
We ﬁxed α k = 1 and only tuned β k ∈ (0 , 1] in the experiments.
F.3.2 Experiments on CIFAR-10 and CIFAR-
The experiments on CIFAR-10 and CIFAR-100 followed the same settings of SAM [62] and ST-AM [63] for direct comparisons.
Both CIFAR-10 and CIFAR-100 contain a training dataset (50K images) and a test dataset (10K images), where CIFAR-10 has 10 classes and CIFAR-100 has 100 classes for classiﬁcation.
The basic setting of network training followed the standard setting of training
ResNet [30]: The batch size was 128 as commonly suggested; for training with N iterations, the learning rate of the optimizer was decayed at the ( (cid:98) N 2 (cid:99) ) -th iteration and the ( (cid:98) 34 N (cid:99) ) -th iteration.
The experiments were run with 3 random seeds and the averaged results along with the standard deviations were reported.
The ﬁnal accuracy on the test dataset was used as the evaluation metric.
The accuracy reported in Table 1(a) is the ﬁnal test accuracy of training with 160 epochs.
We compared sMin-AM with SGDM, Adam, AdaHessian, stochastic AM (SAM), and short-term recurrence AM (ST-AM).
SGDM is the default optimizer for training many deep neural networks, such as ResNet [30], WideResNet [66], DenseNet [34], and ResNeXt [64].
Adam is an adaptive learning rate method that uses diagonal approximation to the Hessian based on moving average.
AdaHessian also uses adaptive learning rates like Adam, but it relies on Hessian-vector products to obtain the diagonal approximation.
SAM and ST-AM are variants of AM.
Both can be used to train neural networks.
The SAM with memory size m is denoted as SAM( m ).
ST-AM keeps two vector pairs and has the same memory size as SAM(2).
We tuned the hyperparameters of all the optimizers (including sMin-AM) following the same way for fair comparison.
The tuning procedures were conducted on CIFAR-10/ResNet20.
For each optimizer, the hyperparameter setting that achieved the highest ﬁnal test accuracy for training ResNet20 on
CIFAR-10 was unchanged in the other tests of training neural networks on CIFAR.
[Figure]
For SGDM, the momentum was set as 0.9, which is the default setting in the literature [30, 34].
The initial learning rate and weight decay were tuned and were set as 0.1, 0.0005, respectively.
The learning rate decay was 0.1.
This setting is also recommended in WideResNet [66].
For Adam, the initial learning rate and weight decay were tuned and set as 0.001, 0.0005, respectively.
The momentum terms were β 1 = 0 .
9 and β 2 = 0 .
999 as commonly suggested [65,
The learning rate decay was 0.1.
For AdaHessian, the initial learning rate was 0.15, and the momentum terms β 1 = 0 .
9 , β 2 = 0 .
999 ; eps = 1 × 10 − 4 , and the hessian-power was 1 .
The weight decay was 0 .
0005 / 0 . 15 , and the learning rate decay was 0.1.
For SAM, the initial mixing parameter β 0 = 1 , the initial damping term α 0 = 1 , and the regularization parameter c 1 = 0 . 01 .
The weight decay was 0 . 0015 .
The decay rate for β k , α k was 0.06.
For ST-AM, the initial mixing parameter β 0 = 1 , the initial damping term α 0 = 1 , and the regularization parameters were c 1 = 1 , c 2 = 1 × 10 − 7 .
The weight decay was 0 .
001 , and the decay rate for β k , α k was 0.1.
For sMin-AM, the initial mixing parameter β 0 = 0 .
2 , the damping term α k = 1 , and the regularization parameters were δ (1) k
= 1 , δ (2) k
= 1 .
The weight decay was 0 . 0015 , and the decay rate for β k was 0.1.
Note that our hyperparameter settings of the baseline methods were the same as those in SAM [62] and ST-AM [63], so we used their results for reference.
Convergence behaviour in the training process.
In Figure 10, we plot the test accuracy of training four networks on CIFAR-10 and CIFAR-100, for each optimizer.
It shows Adam has fast convergence in the beginning, but stagnates in the later training.
Compared with SGDM, SAM, and ST-AM, it is observed that sMin-AM converges faster to an acceptable accuracy.
The process of sMin-AM is also much more stable than SAM(1) and SAM(10)
[Figure]
[Figure] [Figure]
Effect of weight decay.
Figure 11 shows the effect of weight decay on the training process.
It suggests that a larger weight decay can slow down the training in terms of training loss.
When using the same weight decay
[Figure]
sMin-AM often has faster convergence than SGDM.
The results also justify our settings of the weight decay
[Figure]
Effect of learning rate/mixing parameter.
Figure 12 shows the behaviours of SGDM with different learning rate settings and sMin-AM with different mixing parameters.
Small learning rate or small mixing parameter can lead to fast convergence in the beginning, but the training may stagnate early and the test performance is not satisfactory.
Effects of δ (1) k , δ (2) k , and α k .
In Figure 13, we show the results of sMin-AM with different settings of other hyperparameters, i.e., δ (1) k , δ (2) k , and α k .
It is observed that sMin-AM is more sensitive to δ (2) k
than δ
(1) k and α k , which may be due to the fact that δ (2) k directly affects the regularization in the
Min-AM update (Line 10 in Algorithm 4).
When choosing δ (2) k ≥ 1 , the effect of δ (2) k is also minor.
As a result, the most critical hyperparameters that affect the effectiveness of sMin-AM are the mixing parameter β k , the regularization parameter δ (2) k , and weight decay.
The weight decay is a common hyperparameter that needs to be tuned for each optimizer; for δ (2) k , we can choose δ (2) k ≥ 1 to ensure
H Ak (cid:23) 0 .
So except for the weight decay, we only tuned the mixing parameter in the experiments.
Check of the choices of δ (1) k and δ (2) k .
From Lemma 2, we know (cid:107) H
Ak − β k
I (cid:107) 2 ≤ − ρ k (cid:107) p k (cid:107) 22 − 2 β k ρ k (cid:107) p k (cid:107) 2 (cid:107) q k (cid:107)
+ β k ρ 2 k (cid:107) p k (cid:107) 22 (cid:107) q k (cid:107) 22 assuming ρ k ≤ 0 .
In Figure 14, we plot the value of this bound.
It is found that (cid:107) H Ak − β k
I (cid:107) 2 ≤ β k C 2 for some constant C 2 , which justiﬁes our choice of the regularization parameters.
Train neural networks with fewer epochs and less time.
We also tested sMin-AM for training different epochs.
Figure 15 shows the comparison between sMin-AM and SGDM for training 80,
[Figure] [Figure]
[Figure] [Figure]
120, and 160 epochs.
The results show that sMin-AM can attain comparable accuracy to SGDM in fewer epochs, which infers the faster training process of sMin-AM.
In Table
[Figure]
by setting the ﬁnal test accuracy (160 epochs) of SGDM as the baseline, we report the memory cost, per-epoch time, total training epochs, total training time of SAM(10), ST-AM, and sMin-AM to achieve a comparable accuracy to SGDM (within 0 .
05% difference).
The attained ﬁnal accuracy is also shown.
It is found that sMin-AM largely reduces the memory cost of SAM(10) and can achieve comparable results to those of SGDM using less training time.
Discussion about the computational cost and memory cost.
The per-epoch computational cost and memory cost are closely related to three factors: the network architecture, the training data, and the optimizer.
From Table 2, the effects of the network architecture and the optimizer are clear.
For the effect of training data, we consider the batch size.
Given a speciﬁc network, the cost is composed of two parts: (i) updating network parameters by the optimizer; (ii) other necessary computations, such as the forward and back propagations of the neural network, data transfers between memory and disks, etc., where additional memory and processing time are required.
If the cost of Part (ii) only occupies a small proportion of the total cost, the cost incurred by the optimizer will be of great importance.
In Figure 16, we plot the ratios of memory/per-epoch time of SAM(10)/ST-AM/sMin-AM to that of SGDM, for training ResNeXt50 on CIFAR-100 with different batch sizes (16, 32, 64, and 128).
[Figure]
[Figure] [Figure]
[Table]
When a smaller batch size is used, the Part (ii) cost reduces and the proportion of Part (i) cost in the total cost increases, thus the performance improvement of sMin-AM over SAM(10)/ST-AM is more signiﬁcant.
To investigate the effect of model size on the cost, we tested SAM(10)/ST-AM/sMin-AM on CIFAR-10/ResNet18 and CIFAR-10/ResNet50, where the batch size is 16 so that the Part (ii) cost is not large.
In Table 3, it is observed that for a network of larger scale (ResNet50), sMin-AM is more advantageous than SAM(10)/ST-AM in terms of memory cost.
The difference between CIFAR-10 and CIFAR-100 has only a minor effect on the cost.
For a given network architecture, e.g., VGG16, its implementations for CIFAR-10 and CIFAR-100 differ in the dimension of the last linear layer.
Since the parameters of this linear layer usually occupy a very small portion of the whole parameters, the costs (memory and per-epoch running time) on CIFARand CIFAR-100 are roughly the same, as shown in Table 4.
[Figure]
[Table]
[Figure]
[Table]
, where the batch size is 128.
The memory and per-epoch time of SGDM are set as the units; the ﬁrst element and the second element in ( · , · ) denote the memory and per-epoch time, respectively.
[Table]
F.3.3 Experiment of training ResNet50 on ImageNet
We applied sMin-AM to train ResNet50 on the ImageNet dataset, which contains 1.2M images for training and 50K images for test.
The code was based on the example from PyTorch 3 .
We followed the standard process of training ResNet [30].
Four GPUs were used to conduct the experiment.
SGDM was used for comparison.
The batch size was 256.
The weight decay for each method was 0.0001.
We trained ResNet50 for 90 epochs.
The learning rate of SGDM and the mixing parameter of sMin-AM were decayed by 0.1 at the 30th and 60th epochs.
For SGDM, the momentum was 0.
and the learning rate was 0.1, which was the standard setting [30, 66, 34, 64].
For sMin-AM, we set the initial mixing parameter β 0 = 0 .
5 , and the other hyperparameters were kept unchanged.
Comparisons with SAM and ST-AM.
We conducted tests to compare the accuracy of sMin-AM with that of SAM/ST-AM, and results in Figure 17 show that sMin-AM also improves the test accuracy of SAM(10)/ST-AM.
F.3.4 An additional experiment: adversarial training
We conduct an additional experiment to further compare the effectiveness of SGDM, ST-AM, and sMin-AM for training deep neural networks.
The considered problem is adversarial training , which
[Figure]
tries to solve the following minimax problem
[Equation]
(174) where ¯ ξ i is the adversarial data sample in the (cid:15) -ball centered at the data sample ξ i .
We used the standard PGD adversarial training process [40]: using projection gradient descent to solve the maximization problem in (174), and applying SGDM/ST-AM/sMin-AM to solve the minimization problem in (174).
The tests were conducted on CIFAR-10/WideResNet34-10 and CIFAR-100/DenseNet121, where for the training dataset of CIFAR-10/CIFAR-100, 5K images were randomly selected as the validation dataset and the other 45K images were used as the new training dataset.
We trained the neural network for 200 epochs, where the learning rate/mixing parameter was decayed at the 100th and the 150th epochs.
The best checkpoint model on the validation dataset was chosen to be evaluated on the test dataset.
For SGDM and ST-AM, we used the recommended setting of hyperparameters in [26, 63].
For sMin-AM, we set β 0 = 0 .
3 , and other hyperparameters were kept unchanged.
[Table]
The training process of each optimizer is shown in Figure 18, where we plot the accuracy on validation dataset (called as validation accuracy), and the accuracy on adversarial data samples generated by the PGD-10 attack [40] on validation dataset (called as robust validation accuracy).
It is observed that the best robust validation accuracy of sMin-AM is higher than that of SGDM/ST-AM, which indicates that better checkpoint model can be obtained by sMin-AM.
In Table 5, we report two types of test accuracy: clean test accuracy, where the clean test data was used for classiﬁcation; robust test accuracy, where corrupted test data was used for classiﬁcation.
Three attacking methods were used for the robust test accuracy evaluation: FGSM [25], PGD-20 [40], and C&W ∞ attack [13].
It shows that sMin-AM signiﬁcantly improves SGDM, and outperforms
ST-AM in terms of robust test accuracy, which is a desirable property in adversarial training.
G Limitations
We focus on developing the variant of AM with minimal memory size for solving optimization problems.
The properties of Min-AM for deterministic optimization are established in the smooth case.
The non-smooth optimization is important and one possible direction is to adapt Min-AM to the framework of proximal quasi-Newton methods [15, 7].
We leave it as an extension of Min-AM in the future work.
[Figure]
Min-AM exploits the symmetry of Hessian in solving optimization problems.
For general ﬁxed-point problems, the Jacobian can be non-symmetric.
However, in this case, the short-term recurrence algorithms that are equivalent to the full-memory methods generally do not exist even in solving linear systems [53]
[Figure]