Local Linear Convergence of Gradient Methods for Subspace Optimization via Strict Complementarity
Abstract
We consider optimization problems in which the goal is ﬁnd a k -dimensionalsubspace of R n , k << n , which minimizes a convex and smooth loss.
Suchproblems generalize the fundamental task of principal component analysis (PCA)to include robust and sparse counterparts, and logistic PCA for binary data, amongothers.
This problem could be approached either via nonconvex gradient methodswith highly-efﬁcient iterations, but for which arguing about fast convergenceto a global minimizer is difﬁcult or, via a convex relaxation for which arguingabout convergence to a global minimizer is straightforward, but the correspondingmethods are often inefﬁcient.
In this work we bridge these two approaches undera strict complementarity assumption, which in particular implies that the optimalsolution to the convex relaxation is unique and is also the optimal solution to theoriginal nonconvex problem.
Our main result is a proof that a natural nonconvexgradient method which is SVD-free and requires only a single QR-factorization ofan n ⇥ k matrix per iteration, converges locally with a linear rate.
We also establishlinear convergence results for the nonconvex projected gradient method, and the
Frank-Wolfe method when applied to the convex relaxation.
Introduction
We consider the problem of ﬁnding a k -dimensional subspace of R n k << n , which minimizes agiven objective function, where we identify a subspace with its corresponding projection matrix.
Thatis, we consider the following optimization problem:
[Equation]
Throughout this work and unless stated otherwise, we assume that f ( · ) is convex,  -smooth (gradient
Lipschitz) and, for ease of presentation, we also assume that the gradient r f ( · ) is a symmetric matrixover the space of n ⇥ n symmetric matrices S n 1 .
Problems of interest that fall into this model include among other robust counterparts of PCA whichare based on the smooth and convex
Huber loss (see concrete examples in Section 4), logistic PCA[15], and sparse PCA [26].
Note that, in problem (1) we are interested in the low-dimensionalsubspace itself (as opposed to the projection of the data onto it, as in many other formulations), whichis important for instance when the end goal is to perform dimension reduction, which is one of themost important applications of PCA-style methods.
Motivated by high-dimensional problems, we are interested in highly efﬁcient (in particular in termsof the dimension n ) ﬁrst-order methods for Problem (1).
Moreover, we are interested in establishing,at least locally, fast convergence to the global minimizer despite the fact that Problem (1) is nonconvex.
Subspace recovery/optimization problems similar to Problem (1) have received signiﬁcant interest inrecent years, see for instance [25, 28, 4, 17, 18, 11, 16, 22, 23] however, different from these works,our approach will not assume that f ( · ) admits a very speciﬁc structure (e.g., a linear or quadraticfunction), or will be based on a speciﬁc underlying statistical model.
Instead, we will be interested indeterministic conditions that may hold for quite general f ( · ) (which is convex and smooth), and mayrender quite a wide variety of problems “well-posed” for efﬁcient optimization.
We begin by brieﬂy describing two natural dimension-efﬁcient ﬁrst-order methods for tackling
Problem (1).
One such method is the nonconvex projected gradient method which follows thedynamics:
[Equation]
where ⇧ P n,k [ · ] denotes the Euclidean projection onto the set P n,k (note that since this set is nonconvex,in general, the projection need not be unique).
Given the gradient r f ( X t ) , the runtime to compute
X t +1 is dominated by the computation of the projection.
It is not hard to show that the Euclideanprojection is given by the projection matrix which corresponds to the span of the top k eigenvectors ofthe matrix X t  ⌘ r f ( X t ) .
While accurate computation of this projection requires a (thin) singularvalue decomposition (SVD) of a n ⇥ n matrix, which amounts to O ( n 3 ) runtime, it can also beapproximated up to sufﬁciently small error using fast iterative methods, such as the well-knownorthogonal iteration method [10] (aka subspace iteration method [24]) 2 , whose every iteration takesin worst case only O ( kn 2 ) time.
When the gradient r f ( X t ) admits a favorable structure such assparsity or low-rank factorization, this runtime could be signiﬁcantly improved.
Another natural approach to tackle Problem (1) is to exploit the fact that each X 2 P n,k could befactored as X
=
QQ > , Q 2 R n ⇥ k having orthonormal columns, and to apply gradient steps w.r.t.this factorization.
This leads to the following dynamics, which we refer to as Gradient Orthogonal
Iteration :
[Equation]
where QR- FACTORIZE ( · ) denotes the QR factorization operation, i.e., in the above
Q t +1 2 R n ⇥ k has orthonormal columns.
This step is required so Q t +1 has orthonormal columns(and so Q t +1 Q > t +1 remains a projection matrix) for all t  1 .
As opposed to the Dynamics (2), which as discussed, an efﬁcient implementation of will require torun a QR-based iterative method to compute the Euclidean projection onto P n,k on each iteration, the
Dynamics (3) only require a single QR factorization, and thus, given the gradient matrix r f ( Q t Q > t ) ,the next iterate Q t +1 can be computed in overall O ( n 2 k ) time.
As mentioned above, this runtimecould be further signiﬁcantly improved if the multiplication r f ( Q t Q t ) Q t could be carried out fasterthan O ( n 2 k ) (for instance when the gradient is sparse or admits low-rank factorization), since allother operations require only O ( k 2 n ) time (e.g., the QR-factorization of Z t +1 ).
Obtaining provable guarantees on the fast local convergence of the Dynamics (3) to a global optimalsolution of Problem (1) is the main contribution of this work.
While both Dynamics (2), (3) apply efﬁcient iterations, since they are inherently nonconvex, arguingabout their convergence to a global optimal solution of (1) is difﬁcult in general.
An alternative is toreplace Problem (1) with a convex counterpart, for which, arguing about the convergence of ﬁrst-ordermethods to a global optimal solution is well understood.
Consider the convex set F n,k = conv
( P n,k ) ,where conv ( · ) denotes the convex-hull operation.
F n,k is also called the Fantope and it is known toadmit the following important characterization: F n,k = { X 2 S n | I ⌫ X ⌫ 0 , Tr ( X ) =
k } , where
A ⌫ 0 denotes that A is a positive semideﬁnite (PSD) matrix, see for instance [20].
This leads to theconvex problem:min f ( X ) subject to X 2 F n,k = { X 2 S n | I ⌫ X ⌫ 0 , Tr ( X ) = k } .
A well known ﬁrst-order method applicable to (4) is the Frank-Wolfe method (aka conditionalgradient) [12], which for the convex Problem (4) follows the dynamics:
[Equation]
P n
[Equation]
It follows from Ky Fan’s maximum principle [7] that computing V t amounts to computing theprojection matrix onto the span of the k eigenvectors corresponding to the k smallest eigenvalues ofr f ( X t ) , and hence can be carried out efﬁciently using the orthogonal iterations method or similarmethods, similarly to the computation of the projection in (2)) discussed above.
3 Note however, thatthe Frank-Wolfe iterates will not be, in general, low rank, and only yield a O (1 /t ) convergence rate[12].
1.1 The eigengap assumption and strict complementarity
We now turn to discuss our only non-completely standard assumption on Problems (1), (4), whichwill underly all of our contributions and in particular facilitate our local linear convergence rates.
Assumption 1 (Main assumption) .
An optimal solution X ⇤ to the convex Problem (4) is said tosatisfy the eigen-gap assumption with parameter  > 0 , if  n  k ( r f ( X ⇤ ))
n  k +
( r f ( X ⇤ ))
.
Assumption 1 in particular implies the following theorem which states that the convex relaxation (4)exactly recovers the unique and optimal solution to the nonconvex Problem (1).
This is one aspect inwhich Assumption 1 captures “well-posed” instances of Problem (1).
The proof is in the appendix.
Theorem 1.
If an optimal solution X ⇤ to Problem (4) satisﬁes Assumption 1 with some parameter > 0 , then it has rank k , i.e., X ⇤ 2 P n,k , and it is the unique optimal solution to both Problem (4)and Problem (1) .
Assumption 1 is tightly related to the convex Problem (4) through the concept of strictcomplementarity which is a classical concept in constrained continuous optimization theory [1].
A similar connection between an eigengap in the gradient at an optimal solution and strict complementarity has been already established in [5] for low-rank matrix optimization problems, where theunderlying convex set is either the nuclear norm ball of matrices or the set of PSD matrices with unittrace.
Now we establish a similar relationship for the convex relaxation (4) and the Fantope, which isslightly more involved.
Let us write the Lagrangian of the convex Problem (4):
[Equation]
where the dual matrix variables Z 1 , Z 2 are constrained to be PSD, i.e., Z 1 ⌫ 0 , Z 2 ⌫ 0 .
The KKT conditions state that X ⇤ , ( Z ⇤ 1 , Z ⇤ 2 , s ⇤ ) are corresponding optimal primal-dual solutions ifand only if the following conditions hold:
[Equation]
Condition 3 is known as complementarity .
Since Z ⇤ 1 , Z ⇤ 2 are PSD and
X
⇤
I , this furtherimplies that Z ⇤ 1 X ⇤ = 0 , Z ⇤ 2 ( I  X ⇤ )
= 0 , which in turn implies thatrange ( X ⇤ )
✓ nullspace ( Z ⇤ 1 ) ^
range ( I  X ⇤ )
✓ nullspace ( Z ⇤ 2 ) .
Deﬁnition 1.
A pair of primal-dual solutions X ⇤ , ( Z ⇤ 1 , Z ⇤ 2 , s ⇤ ) for Problem (4) is said to satisfy strictcomplementarity if range ( X ⇤ )
=
nullspace ( Z ⇤ 1 ) _ range ( I  X ⇤ )
=
nullspace ( Z ⇤ 2 ) , which is thesame as: rank ( Z ⇤ 1 ) =
n  rank ( X ⇤ )
_ rank ( Z ⇤ 2 ) =
rank ( X ⇤ ) .
Theorem 2.
If an optimal solution X ⇤ for Problem (4) with rank ( X ⇤ )
=
k satisﬁes strict complementarity for some corresponding dual solution, then  n  k ( r f ( X ⇤ ))
111  n  k +1 ( r f ( X ⇤ ))
> 0 .
Conversely, if an optimal solution X ⇤ for Problem (4) satisﬁes 112  n  k ( r f ( X ⇤ ))
n  k +
( r f ( X ⇤ ))
> 0 , then it satisﬁes strict complementarity for every 113 corresponding dual solution.
The proof is given in the appendix.
Strict complementarity has played a central role in several recentworks, both for establishing linear convergence rates for ﬁrst-order methods, e.g., [29, 6, 8, 5], andimproving the complexity of projected gradient methods, due to SVD computations, for low-rankmatrix optimization problems, e.g., [9, 13].
1.2 Additional related work
Efﬁcient gradient methods for low-rank nonconvex optimization have received signiﬁcant interestin recent years, here we mention only a few.
[2, 21] gave deterministic guarantees on the localconvergence to a global minimizer of factorized gradient descent for certain low-rank optimizationproblems, under the mild assumption that a low-rank global minimizer exists.
However, theseresults cannot capture constraints such as those in our Problem (1) which encode projection matrices.[4] which considers nonconvex gradient methods for low-rank statistical estimation also considersconstraints that cannot capture projection matrices as in Problem (1).
An exception is a speciﬁc casethey consider of linear objective functions.
Moreover, even for linear functions such as the speciﬁcsparse PCA objective they consider, their analysis requires several non-trivial conditions to hold (e.g.local descent, local smoothness etc), which they only show to hold under Gaussian data.
1.3 Notation
Throughout this work we let k · k denote the Euclidean norm for vectors in R n and the spectral norm(largest singular value) for matrices in R m ⇥ n or S n .
We let k · k F denote the Frobenius (Euclidean)norm for matrices.
For a matrix X 2 S n , we let  i ( X ) denote the i th largest eigenvalue of X .
We leth · , · i denote the standard inner-product for both spaces R n and S n .
Overview of Results
2.1 Main result
Our main novel contribution is the proof of the following theorem regarding the local linear convergence of the gradient orthogonal iteration (3) to the optimal solution of Problems (4), (1).
Theorem 3.
[Local linear convergence of gradient orthogonal iteration] Suppose Assumption1 holds true for some optimal solution X ⇤ for Problem (4) with some parameter  > 0 .
Let
G  sup X 2 F n,k kr f ( X ) k .
Let Q 1 2 R n ⇥ k , rank ( Q 1 ) =
k and consider the sequence { Q t } t  1generated by Dynamics (3) with ﬁxed step-size ⌘ t = ⌘ = 1 5max {  ,G } for all t  1 , and wheninitialized with Q 1 2 P n,k such that k Q 1 Q > 1  X ⇤ k F  min { 1 ,
[Equation]
1+ ⌘ ) } .
Then, we have that
[Equation]
While, as stated above, this is not the ﬁrst work to consider strict complementarity conditionsfor bridging convex and nonconvex methods for low-rank optimization, previous works such as[8, 5, 9, 13] consider gradient methods that rely on (nearly) exact (low-rank)
SVD computations oneach iteration, whereas Theorem 3 considers the more efﬁcient SVD-free Dynamics (3), that requiresonly a single QR-factorization of an n ⇥ k matrix per iteration, which is much faster and simpler toimplement.
Accordingly, the proof is also considerably more challenging and requires new ideas.
2.2 Additional results
We also prove the following two theorems regarding the local linear convergence of the projectedgradient Dynamics (2) and the Frank-Wolfe Dynamics (5).
These extend the results in [9, 8] fromoptimization over the set of positive semideﬁnite matrices with unit trace to the Fantope.
Theorem 4.
[Local linear convergence of nonconvex PGD]
Suppose Assumption 1 holds true forsome optimal solution X ⇤ for Problem (4) with some parameter  > 0 .
Consider the sequence{ X t } t  1 generated by Dynamics (2) with a ﬁxed step-size ⌘ t = ⌘ = 1 /  for all t  1 , and wheninitialized with X 1 2 F n,k such that k X 1  X
⇤ k F   4  .
Then, for all t  1 it holds that1.
rank ( X t +1 )
=
k , and thus, given X t , r f ( X t ) , X t +1 can be computed via a rank- k SVD,2.
f ( X t )  f
( X ⇤ )
=
O (exp(  ⇥ (  /  )( t  1))) .
Theorem 5.
[Local linear convergence of Frank-Wolfe]
Suppose Assumption 1 holds true forsome optimal solution X ⇤ for Problem (4) with some parameter  > 0 .
Consider the sequences{ ( X t , V t ) } t  1 generated by Dynamics (5) when ⌘ t is chosen via line-search.
Then, there exists
T
= O  k (  /  ) such that,
[Equation]
Moreover, for all t  1 , the rank- k matrix V t satisﬁes k V t  X
⇤ k 2 F
[Equation]
X ⇤
[Equation]
⌘ .
What if Assumption 1 fails?
In case Assumption 1 does not hold or holds with negligible parameter  , not all is lost, since by considering weaker versions of Assumption 1, which consider eigen-gapsin lower eigenvalues, we can still guarantee X ⇤ has low rank, and that at least the projected gradientmethod, locally, will require only a low-rank SVD to compute the projection onto the Fantope, whileguaranteeing the standard convergence rate of O (1 /t )
(not linear rate as when Assumption 1 holds).
See the appendix for more details.
The complete proofs of Theorems 3, 4, 5, as well as additional results, are given in the appendix.
Below we give the main ideas in the proof of Theorem 3.
Proof Sketch of Theorem
3.1 Preliminaries
Lemma 1 (Euclidean projection onto the Fantope) .
Let X 2 R n ⇥ n and consider its eigen decomposition X =
P ni =1  i u i u > i .
The Euclidean projection ⇧ F n,k
[ X ] is given by: ⇧ F n,k
[ X ] =
P ni =1  + i ( ✓ ) u i u > i , where  + i ( ✓ ) = min(max
(  i  ✓ , 0) , 1) , and ✓ satisﬁes the equation
P ni =1  + i ( ✓ ) =
k .
Moreover, 8 r 2 { k, ..., n  1 } it holds that rank ( ⇧ F k ( X ))
 r
if and onlyif
[Equation]
Remark 1.
Lemma 1 implies that if rank ( X )  r ,only the top r component of the SVD of X areneeded to compute ⇧ F n,k [ X ] , i.e., a rank- r SVD of X .
Moreover, given the rank- ( r + 1) SVD, wecan check the condition
P ri =1 min(  i   r +1 , 1)  k , to verify whether the projection has rank  r .
The following lemma lower-bounds, under Assumption 1, the radius of the ball around the uniqueoptimal solution X ⇤ inside-which, the PGD mapping w.r.t.
the Fantope F n,k with a ﬁxed step-size, isguaranteed to produce rank- k matrices, i.e., matrices in P n,k , which means that it coincides preciselywith the PGD mapping w.r.t.
the nonconvex set P n,k given by the Dynamics (2).
Lemma 2.
Let X ⇤ 2 F n,k be an optimal solution to Problem (4) which satisﬁes Assumption 1 withsome parameter  > 0 , and let ⌘ > 0 .
For any X 2 F k which satisﬁes k X  X
⇤ k F  ⌘ 2(1 + ⌘ ) , itholds that rank ( ⇧ F n,k
[ X  ⌘ r f ( X )])
=
k .
The following lemma shows that under Assumption 1, Problem (4) has a quadratic growth property.
This property is well known to facilitate linear convergence rates of gradient methods [19, 14].
Lemma 3 (Quadratic Growth) .
Let X 2 F n,k be an optimal solution to Problem (4) for which
Assumption 1 holds with some  > 0 .
Then, 8 X 2 F n,k : k X  X
⇤ k 2 F  2  ( f ( X )  f
( X ⇤ )) .
3.2 Gradient Orthogonal Iteration Analysis
We outline the proof of our main algorithmic result — the local linear convergence result of thegradient orthogonal iteration (3) given in Theorem 3.
For convenience, we rewrite the Dynamics (3)as Algorithm 1 below which also introduces notation that will be helpful throughout the analysis.
Throughout this section we also introduce the auxiliary sequence { X t } t  1 ⇢ F n,k given by X 1 = Y 1 198 and X t +1 =
⇧ F n,k
[ Y t  ⌘ r f ( Y t )] .
At a very high-level our analysis of Algorithm 1 relies on the following two components:1.
Using Lemma 2 we can argue that, in the proximity of X ⇤ , rank ( X t ) =
k , i.e., X t 2 P n,k .
This implies that X t is the projection matrix onto the span of top k eigenvectors of W t .2.
We view Q t as the outcome of applying one iteration of the orthogonal iterations method[10, 24] to W t 4 .
Combined with the previous point, this allows to argue that Y t =
Q t Q > tis sufﬁciently close to the projected gradient update X t , which drives the convergence.
The following key lemma establishes the connection between the sequence { Y t } t  1 produced by 206 Algorithm 1 and the corresponding sequence of exact projected gradient steps { X t } t  1 .
The proof 207 relies on an original extension of the classical orthogonal iteration method (see [10]).
Lemma 4.
Fix some t  1 .
Suppose that ⌘ < 1 /G , X t +1 2 P n,k , and k X t +1  Y t k F < p 2 .
Itholds that, k X t +1  Y t
+1 k 2 F 
[Equation] [Equation]
X
[Equation]
⌘ G
[Equation]
k X t +1  Y t k 2 F .
Proof of Lemma 4.
Let us write the eigen decomposition of W t
+1 =
Y t  ⌘ r f ( Y t ) as:
[Equation]
where V 1 2 R n ⇥ k , ⇤ 1 2 R k ⇥ k correspond to the largest k eigenvalues.
The main part of the proof will be to prove that k V > 2 Q t +1 k 2 F 
[Equation] [Equation]
V
[Equation]
2 Q t k 2 F .
Note that by deﬁnition of X t
+1 we have that,
[Equation]
where (a) follows from the assumption of the lemma that X t +1 2 P n,k , and (b) follows since allmatrices in P n,k have the same Frobenius norm.
This further implies that
[Equation]
Thus, under the assumption that k X t +1  Y t k F < p 2 , we have that ( V > 1 Q t ) is invertible.
Since ( Q t +1 , R t +1 ) is the QR factorization of W t +1 Q t , using the eigen-decomposition of W t +1we can write Q t +1 R t
+1 =
V ⇤ V T Q t .
Multiplying both sides by V > we get
[Equation]
which leads to the two equations:
[Equation]
Under the assumption that ⌘ < 1 /G , using Weyl’s inequality we have that  k
( W t +1 )
k
( Y t )  ⌘ 1 ( r f ( Y t )) > 0 , and so ⇤ 1 is invertible.
Since following (6) we have that min ( V > 1 Q t ) > 0 , it follows that rank ( ⇤ 1 V > 1 Q t ) =
k and thus, from the equation (7) we have that
V > 1 Q t +1 and R t +1 are both invertible and we can write R t
+1 =
( V > 1 Q t +1 )
1 ⇤ 1 V > 1 Q t .
Multiplying equation (8) on both sides from the right with R  1 t
+1 , we get
[Equation]
Now we can use this to bound k V > 2 Q t +1 k 2 F :
[Equation] [Equation]
where (a) follows from the inequalities k AB k F  min { k A k F k B k 2 ,
k A k 2 k B k F } ,k AB k 2  k A
k 2 k B k 2 , and (b) follows from the eigen-decomposition of W t +1 and by noting thatsince V 1 , Q t +1 both have orthonormal columns, it holds that k V > 1 Q t +1 k 2  1 .
We upper-bound  k +1 ( W t +1 ) /
k
( W t +1 ) by using Weyl’s inequality as follows:
[Equation]
where we have used the fact that Y t 2 P n,k and
so  k
( Y t )
= 1 ,  k +1 ( Y t ) = 0 .
Plugging (10) into (9) we indeed obtain
[Equation]
V > 2 Q
[Equation]
Now, for the ﬁnal part of the proof, we note that k V > 2 Q t +1 k 2 F
=
Tr ( V 2 V > 2 Y t +1 ) =
Tr (( I  X t +1 )
Y t +1 ) =
k
Tr
( X t +1 Y t +1 )
= 12 k X t +1  Y t +1 k 2 F , and similarly, k V > 2 Q t k 2 F =12 k X t +1  Y t k 2 F .
Plugging these observations and (6) into (11), we obtain the lemma.
The following lemma is the main step in the proof of the convergence rate of Algorithm 1.
Lemma 5.
Let us denote h t
= f ( Y t )  f
( X ⇤ ) for all t  1 .
Suppose that ⌘  1 5max {  ,G } andsuppose that X t +1 2 P n,k and that k X t +1  Y t k F  1 .
Denote the constants C 0 =⇣ ⌘ G ⌘ G ⌘ 2 ,
[Equation]
It holds that, h t +1 
[Equation]
h t .
Proof.
Using the  -smoothness of f ( X ) , for any X 2 F n,k and ⌘  1  it holds that
[Equation]
where (a) follows since using the fact that Y t 2 P n,k , we have that for any X 2 F n,k it holds that
[Equation]
Since by the assumption of the lemma X t +1 = ⇧ F n,k
[ Y t  ⌘ r f ( Y t )] 2 P n,k , using the optimalitycondition in Lemma 6 we have that for all Z 2 F n,k it holds
that h X t +1  Z , Y t
⌘ t r f ( Y t )
i  0 .
This implies that for all Z 2 F n,k
[Equation] [Equation]
[Equation]
where the last inequality is due to Lemma 8 and recalling that W t
+1 =
Y t  ⌘ r f ( Y t ) .
Setting X
=
Y t +1 in (12) and plugging-in (13), we have that for any Z 2 F n,k it holds that
[Equation]
where the last inequality is due to the following upper-bound on k W t +1 k 2 :
[Equation]
In particular, setting Z = (1  ↵ ) Y t + ↵ X ⇤ for some ↵ 2 [0 , 1] , we get that
[Equation]
Subtracting f
( X ⇤ ) from both sides, using the convexity of f ( · )
[Equation]
and Lemma 3 gives
[Equation]
Setting ↵ = ⌘ / 2 (note that since ⌘  1 /G , we have that ↵ 2 [0 , 1] ), gives
[Equation]
We now continue to upper-bound the term k X t +1  Y t +1 k 2 F .
Using Lemma 9 we have that
[Equation]
Let us set Z = X t +1 in (14) to obtain that
[Equation]
where the last inequality is due to convexity of f ( · ) .
Rearranging and using Lemma 4 along with thenotation C 0 =⇣ ⌘ G ⌘ G ⌘ 2 , we have f ( X t +1 )
f ( Y t +1 )  1 ⌘ ⇣ 12 + C
(1 + ⌘ G ) ⌘ k
X t +1  Y t k 2 F .
Plugging into (16) we obtain
[Equation]
and rearranging we obtain
[Equation]
Using Lemma 4 again we have, k X t +1  Y t +1 k 2 F  2 ⌘ C 0 2 ⌘  2 C 0 (1+ ⌘ G )
[Equation]
h t  h t
[Equation]
Plugging back into (15) we obtain h t +1  ⇣ 1  ⌘⌘ h t + 2(1+ ⌘ G ) C 0 2 ⌘  2 C 0 (1+ ⌘ G ) ( h t  h t +1 ) .
Denoting
C 1 = 2(1+ ⌘ G ) C 0 2 ⌘  2 C 0 (1+ ⌘ G )
, we ﬁnally obtain h t +1  1 1+ C⇣ 1  ⌘ 4 + C⌘ h t = ⇣ 1  ⌘ 4(1+ C
[Equation]
⌘ h t
Numerical Simulations
We brieﬂy discuss our numerical simulations.
Additional information and results, including supportfor our warm-start assumption, are given in the appendix.
We consider two models for robust subspacerecovery from noisy samples: 1.
a spiked covariance model
[Table]
and 2.
a sparsely corrupted entriesmodel.
In both models we minimize a robust loss based on the Huber function, which is convex andsmooth, over the Fantope.
We generate random instances and solve them to high precision (dualitygap < 10  10 ) and take the resulting point X ⇤ as the optimal solution.
We measure the eigen-gap(as in Assumption 1) in r f ( X ⇤ ) and we compare the recovery error w.r.t.
the naive PCA solution
X PCA which simply computes the leading subspace of the empirical covariance.
The results aregiven in Table 1.
As can be seen, for both models the recovery error is signiﬁcantly lower than thatof naive PCA, which demonstrates the usefulness of the chosen models .
We see that the eigen-gapassumption indeed holds with substantial values of  .
[Table]
We additionally test the empirical convergence of nonconvex PGD (Dynamics (2)) and the gradientorthogonal iteration method (Dynamics (3)) on the above models.
We initialize both methods withthe PCA solution X PCA and use the same ﬁxed step-size for both.
We indeed observe the rapidconvergence of both methods.
In particular, we verify that the iterates of nonconvex PGD over P n,kcoincide precisely with those of PGD w.r.t.
the convex Fantope F n,k
(see Remark 1), which suggeststhat nonconvex PGD indeed converges to the global minimum.
Figure 1 (left panel)
also suggeststhat indeed both GD and GOI converge with a linear rate.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The details in
the experimental section allow to reproduce the results exactly
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
[Table]
e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]