U NDERSTANDING I NCREMENTAL L EARNING OF G RADIENT D ESCENT : A F INE - GRAINED A NALYSIS OF M ATRIX S ENSING
Anonymous authors
Paper under double-blind review
A BSTRACT
The implicit bias of optimization algorithms such as gradient descent (GD) is believed to play an important role in generalization of modern machine learning methods such as deep learning.
This paper provides a fine-grained analysis of the dynamics of GD for the matrix sensing problem, whose goal is to recover a low-rank ground-truth matrix from near-isotropic linear measurements.
With small initialization, we that GD behaves similarly to the greedy low-rank learning heuristics (Li et al., 2020) and follows an incremental learning procedure (Gissin et al., 2019).
That is, GD sequentially learns solutions with increasing ranks until it recovers the ground-truth matrix.
Compared to existing works which only analyze the first learning phase for rank-1 solutions, our result is stronger because it characterizes the whole learning process.
Moreover, our analysis of the incremental learning procedure applies to the under-parameterized regime as well.
As a key ingredient of our analysis, we observe that GD always follows an approximately low-rank trajectory and develops novel landscape properties for matrix sensing with low-rank parameterization.
Finally, we conduct numerical experiments which confirm our theoretical findings.
I NTRODUCTION
Understanding the optimization and generalization properties of optimization algorithms is one of the central topics in deep learning theory (Zhang et al., 2021; Sun, 2019).
It has long been a mystery why simple algorithms such as Gradient Descent (GD) or Stochastic Gradient Descent (SGD) can find global minima even for highly non-convex functions (Du et al., 2019), and why the global minima being found can generalize well (Hardt et al., 2016).
One influential line of works provides theoretical analysis of the implicit bias of GD/SGD.
These results typically exhibit theoretical settings where the low-loss solutions found by GD/SGD attain certain optimality conditions of a particular generalization metric, e.g. , the parameter norm (or the classifier margin) (Soudry et al., 2018; Gunasekar et al., 2018; Nacson et al., 2019; Lyu & Li, 2020;
Ji & Telgarsky, 2020), the sharpness of local loss landscape (Blanc et al., 2020; Damian et al., 2021; Li et al., 2022; Lyu et al., 2022). Among these works, a line of works seek to characterize the implicit bias even when the training is away from convergence. Kalimeris et al.
(2019) empirically observed that SGD learns model from simple ones, such as linear classifiers, to more complex ones. This behavior, usually referred to as the simplicity bias/incremental learning of GD/SGD, can help prevent overfitting for highly over-parameterized models since it tries to fit the training data with minimal complexity. Hu et al.
; Lyu et al. Frei et al.
(2021) theoretically establish that GD on two-layer nets learns linear classifiers first.
The goal of this paper is to demonstrate this simplicity bias/incremental learning in the matrix sensing problem, a non-convex optimization problem that arises in a wide range of real-world applications, e.g. , image reconstruction (Zhao et al., 2010; Peng et al., 2014), object detection (Shen & Wu, 2012; Zou et al., 2013) and array processing systems (Kalogerias & Petropulu, 2013). Moreover, this problem serves as a standard test-bed of the implicit bias of GD/SGD in deep learning theory, since it retains many of the key phenomena in deep learning while being simpler to analyze.
Formally, the matrix sensing problem asks for recovering a ground-truth matrix Z ∗ ∈ R
d × d given m observations y 1 , . . .
, y m .
Each observation y i here is resulted from a linear measurement y i = ⟨ A i , Z ∗ ⟩ , where { A i } 1 ≤ i ≤ m is a collection of symmetric measurement matrices.
In this paper, we focus on the case where Z ∗ is positive semi-definite (PSD) and is of low-rank: Z ∗ ⪰ 0 and rank ( Z ∗ ) =
r ∗ ≪ d .
An intriguing approach to solve this matrix sensing problem is to use the Burer-Monteiro type decomposition Z ∗ = UU ⊤ with U ∈ R d × ˆ r , and minimize the squared loss with GD:
[Equation]
In the ideal case, the number of columns of U , denoted as ˆ r above, should be set to r ∗ , but r ∗ may not be known in advance.
This leads to two training regimes that are more likely to happen: the under-parameterized regime where ˆ r < r ∗ , and the over-parameterized regime where ˆ r > r ∗ .
The over-parameterized regime may lead to overfitting at first glance, but surprisingly, with small initialization, GD induces a good implicit bias towards solutions with exact or approximate recovery of the ground truth.
It was first conjectured in Gunasekar et al.
(2017) that GD with small initialization finds the matrix with minimum nuclear norm.
However, a series of works point out that this nuclear norm minimization view cannot capture the simplicity bias/incremental learning behavior of GD.
In the matrix sensing setting, this term particularly refers to the phenomenon that GD tends learn solutions with rank gradually increasing with training steps.
Arora et al. (2019) exhibits this phenomenon when there is only one observation ( m = 1 ). Gissin et al.
; Jiang et al. (2022) study the full-observation case, where every entry of the ground truth is measured independently f ( U ) =
14 d 2 ∥
Z ∗ − UU ⊤ ∥ 2F , and GD is shown to sequentially recover singular components of the ground truth from the largest singular value to the smallest one. Li et al. (2020) provide theoretical evidence that the incremental learning behavior generally occurs for matrix sensing.
They also give a concrete counterexample for Gunasekar et al.
(2017)’s conjecture, where the [-gradient descent (GD) algorithm U α, 
-]simplicity bias drives GD to a rank-1 solution that has a large nuclear norm.
In spite of these progresses, theoretical understanding of the simplicity bias of GD remains limited.
Indeed, a vast majority of existing analysis only shows that GD is initially biased towards learning a rank- 1 solution, but their analysis cannot be generalized to higher ranks, unless additional assumptions on the GD dynamics are made (Li et al., 2020, Appendix H), (Belabbas, 2020;
[-α U ⊤ α,T α − Z ∗ (cid:13)(cid:13) F = 
-]
Jacot et al., 2021; Razin et al., 2021; 2022).
1. O UR C ONTRIBUTIONS
In this paper, we take a step towards understanding the generalization of GD with small initialization by firmly demonstrating the simplicity bias/incremental learning behavior in the matrix sensing setting, assuming the Restricted Isometry Property (RIP).
Our main result is informally stated below.
See Theorem 4.1 for the formal version.
Definition 1.1 (Best Rank- s Solution)
We define the best rank- s solution as the unique global minimizer Z ∗ s of the following constrained optimization problem:
[Equation]
Theorem 1.1 (Informal version of Theorem 4.1) Consider the matrix sensing problem (1) with rank- r ∗ ground-truth matrix Z ∗ and measurements { A i } mi =1 . Assume that the measurements satisfy the RIP condition (Definition 3.2).
With small learning rate µ > 0 and small initialization
U α, 0 = α U ∈ R d
× ˆ r , the trajectory of U α,t U ⊤ α,t during GD training enters an o (1) -neighbourhood of each of the best rank- s solutions in the order of s = 1 , 2 , . . , ˆ r ∧ r ∗ when α → 0 .
It is shown in Li et al. ; St¨oger & Soltanolkotabi (2021) that GD exactly recovers the ground truth under the RIP condition, but our theorem goes beyond this result in a number of ways.
First, in the over-parameterized regime (i.e., ˆ r ≥ r ∗ ), it implies that the trajectory of GD exhibits an incremental learning phenomenon: learning solutions with increasing ranks until it finds the ground truth.
Second, this result also shows that in the under-parameterized regime (i.e., ˆ r < r ∗ ), GD exhibits the same implicit bias, but finally it converges to the best low-rank solution of the matrix sensing loss.
By contrast, to the best of our knowledge, only the over-parameterized setting is analyzed in existing literature.
Theorem 1.1 can also be considered as a generalization of previous results in Gissin et al. Jiang et al.
(2022) which show that U α,t U ⊤ α,t passes by the best low-rank solutions one by one in the full observation case of matrix sensing f ( U ) = 14 d 2 ∥
Z ∗ − UU ⊤ ∥ 2F . However, our setting has two major challenges which significantly complicate our analysis.
First, since our setting only gives partial measurements, the decomposition of signal and error terms in Gissin et al. Jiang et al.
(2022) cannot be applied.
Instead, we adopt a different approach which is motivated by St¨oger & Soltanolkotabi (2021); intuitive explanations of our approach is discussed in Appendix B.
Second, it is well-known that the optimal rank- s solution of matrix factorization is X s (defined in
Section 3), but little is known for Z ∗ s .
In Section 5 we analyze the landscape of (2), establishing the uniqueness of Z ∗ s and local landscape properties under the RIP condition. We find that when
U α,t U ⊤ α,t ≈ Z ∗ s , GD follows an approximate low-rank trajectory, so that it behaves similarly to GD in the under-parameterized regime.
Using our landscape results, we can finally prove Theorem 1.1.
Organization.
We review additional related works in Section 2.
In Section 3, we provide an overview of necessary background and notations.
We then present our main results in Sectionwhere we also give a proof sketch.
In Section 5, we outline the key landscape results that we use to prove Theorem 4.1.
Experimental results are presented in Section 6 which verify our theoretical findings.
Finally, in Section 7, we summarize our main contributions and discuss some promising future directions.
Complete proofs of all results in this paper are given in the Appendix.
R ELATED WORK
Low-rank matrix recovery.
The goal of low-rank matrix recovery is to recover an unknown lowrank matrix from a finite number of (possibly noisy) measurements.
Examples include matrix sensing (Recht et al., 2010), matrix completion (Cand ` es & Recht, 2009; Candes & Plan, 2010) and robust PCA (Xu et al., 2010; Cand ` es et al., 2011).
Fornasier et al.
; Ngo & Saad (2012); Wei et al. ; Tong et al. (2021) study efficient optimization algorithms with convergence guarantees.
Interested readers can refer to Davenport & Romberg (2016) for an overview of low rank matrix recovery.
Simplicity bias/incremental learning of gradient descent.
Besides the works mentioned in the introduction, other works study the simplicity bias/incremental learning of GD/SGD on tensor factorization (Razin et al., 2021; 2022), deep linear networks (Gidel et al., 2019), two-layer nets with orthogonal inputs (Boursier et al., 2022).
Landscape analysis of non-convex low-rank problems.
The strict saddle property (Ge et al., 2016; 2015; Lee et al., 2016) was established for non-convex low-rank problems in a unified framework by Ge et al.
Tu et al.
(2016) proved a local PL property for matrix sensing with exact parameterization (i.e. the rank of parameterization and ground-truth matrix are the same).
The optimization geometry of general objective function with Burer-Monteiro type factorization is studied in Zhu et al.
; Li et al.
Zhu et al.
We provide a comprehensive analysis in this regime for matrix factorization as well as matrix sensing that improves over their results.
P RELIMINARIES
In this section, we first list the notations used in this paper, and then provide details of our theoretical setup and necessary preliminary results.
3. N OTATIONS
We write min { a, b } as a ∧ b for short.
For any matrix A , we use ∥ A ∥ F to denote the Frobenius norm of A , use ∥ A ∥ to denote the spectral norm ∥
A ∥ 2 , and use σ min ( A ) to denote the smallest singular value of A .
We use the following notation for Singular Value Decomposition (SVD): Definition 3.1 (Singular Value Decomposition) For any matrix A ∈ R d 1 × d 2 of rank r , we use A = V A Σ A W ⊤ A to denote a Singular Value Decomposition (SVD) of A , where V A ∈ R d 1 × r , W A ∈ R d 2 × r satisfy
V ⊤
A V A
= I ,
W A W ⊤ A
= I , and Σ A ∈ R r × r is diagonal.
For the matrix sensing problem (1), we write the ground-truth matrix as Z ∗ =
XX ⊤ for some
X =
[ v 1 , v 2 , · · · , v r ∗ ] ∈ R d × r ∗ with orthogonal columns.
We denote the singular values of X as σ 1 , σ 2 , . . .
, σ r ∗ , then the singular values of Z ∗ are σ 21 , σ 22 , . . .
, σ 2 r ∗ .
We set σ r ∗ +1 :
= 0 for convenience.
For simplicity, we only consider the case where Z ∗ has distinct singular values, i.e., σ 21 > σ 22 > · · · > σ 2 r ∗ > 0 .
We use κ :=
[Table]
s
[Table]
} to quantify the degeneracy of the singular values of Z ∗ .
We also use the notatiowrite X s =
[ v 1 , v 2 , · · · , v s ] for the matrix consisting of the first s columns of X .
Note that Z ∗ s (Definition 1.1) does not equal X s X ⊤ s in general.
We write the results of the measurements { A i } mi =1 as a linear mapping A : R d × d (cid:55)→ R m , where [ A ( Z )] i = 1 √ m ⟨ A i , Z ⟩ for all 1 ≤ i ≤ m .
We use A ∗ : R m → R d × d , A ∗ ( w ) = 1 √ m (cid:80) mi =1 w i
A
i to denote the adjoint operator of A .
Our loss function (1) can then be written as f [Equation]
(cid:1)(cid:13)(cid:13) 2 2 .
The gradient is given by ∇ f ( U ) =
A ∗ (cid:0) y − A ( UU ⊤ ) (cid:1) U = A ∗ A (cid:0) XX ⊤ − UU ⊤ (cid:1) U t .
In this paper, we consider GD with learning rate µ > 0 starting from U 0 .
The update rule is U t +1 = U t − µ ∇ f ( U t ) =:
( I + µ M t ) U t , (3) where M t =
A ∗ A (cid:0) XX T − U t
U Tt (cid:1) .
We specifically focus on GD with small initialization: letting
{+U 
+}
= α ¯ U for some matrix ¯ U ∈ R d × r with ∥ ¯ U ∥ = 1 , we are interested in the trajectory of GD when α → 0 .
Sometimes we write U t as U α,t to highlight the dependence of the trajectory on α .
3. A SSUMPTIONS
For our theoretical analysis of the matrix sensing problem, we make the following standard assumption in the matrix sensing literature:
Definition 3.2 (Restricted Isometry Property)
We say that a measurement operator A satisfies the ( δ, r ) -RIP condition if (1 − δ )
∥
Z ∥ 2F ≤ ∥A ( Z ) ∥ 22 ≤ (1 + δ ) ∥
Z ∥ 2F for all matrices Z ∈ R d × d with rank ( Z ) ≤ r .
Assumption 3.1 The measurement operator A satisfies the (2 r ∗ + 1 , δ ) -RIP property, where r ∗ = rank ( Z ∗ ) and δ ≤ 10 − 7 κ − 4 r − 1 ∗ .
The RIP condition is the key to ensure the ground truth to be recoverable with partial observations.
An important consequence of RIP is that it guarantees A ∗ A ( Z )
= 1 m (cid:80) mi =1 ⟨ A
i , Z ⟩ A i
≈ Z when Z is low-rank.
This is made rigorous in the following proposition.
Proposition 3.1 (St ¨ oger & Soltanolkotabi, 2021, Lemma 7.3) Suppose that A satisfies ( r, δ ) -RIP with r ≥ 2 , then for all symmetric Z ,
if rank ( Z ) ≤ r − 1 , we have ∥ ( A ∗ A − I )
Z ∥ 2 ≤ √ rδ ∥ Z ∥ .
∥ ( A ∗ A − I ) Z ∥ 2 ≤ δ ∥
Z ∥ ∗ , where ∥ · ∥ ∗ is the nuclear norm.
We also need the following regularity condition on the initialization.
Assumption 3.2 For all 1 ≤ s ≤ ˆ r ∧ r ∗ , σ min (cid:0) V ⊤ X
s ¯ U (cid:1) ≥ ρ for some positive constant ρ , where
[-In this paper, we consider GD with small initialization. Specifically, let U α, 
 = α U for some matrix U ∈ R d × r with ∥ U ∥ = 1 , we are interested in the trajectory of GD when α → 0 .-]V X s is defined as Definition 3.1.
The following proposition implies that Assumption 3.2 is satisfied with high probability with a Gaussian initialization.
Proposition 3.2 Suppose that all entries of U ∈ R d × ˆ r are independently drawn from N (cid:16) 0 , 1 √ ˆ r (cid:17) and ρ = ϵ √ ˆ r −√ ˆ r ∧ r ∗ − 1 √ ˆ r ≥ ϵ 2 r ∗ , then σ min (cid:0)
V ⊤ X s U (cid:1) ≥ ρ holds for all 1 ≤ s ≤ ˆ r ∧ r ∗ with probability at least 1 − ˆ r (cid:0) Cϵ + e − c
ˆ r (cid:1) , where c, C > 0 are universal constants.
M AIN RESULTS
In this section, we present our main theorem, following the theoretical setup in Section 3. Theorem 4.1 Under Assumptions 3.1 and 3.2, consider GD (3) with learning rate µ ≤ 1 10 3 ∥
Z ∗ ∥ and initialization U α, 0 = α ¯ U for solving the matrix sensing problem (1) .
There exists universal constants c, M > 0 , a constant C (depending on ˆ r and κ ) and a sequence of time points T 1 α < T 2 α < · · · < T ˆ r ∧ r ∗ α such that for all 1 ≤ s ≤ ˆ r ∧ r ∗ , the following holds when α = O (cid:16) ( ρr ∗ ) − cκ (cid:17) :
[Equation] where we recall that Z ∗ s is the best rank- s solution defined in Definition 1.1.
Moreover, GD follows an incremental learning procedure: we have lim α → 0 max 1 ≤ t ≤ {+T sα σ s +
+}
( U α,t ) = {+
+}
for all 1 ≤ s ≤ ˆ r ∧ r ∗ , where σ i ( A ) denotes the i -th largest singular value of a matrix A .
Compared with existing works (Li et al., 2018; St¨oger & Soltanolkotabi, 2021) in the same setting, our result characterizes the complete learning dynamics of GD and reveals an incremental learning mechanism, i.e. , GD starts from learning simple solutions and then gradually increasing the complexity of search space until it finds the ground truth. Now we outline the proof of our main theorem.
In the following, we fix an integer 1 ≤ s ≤ ˆ r ∧ r ∗ and show the existence of [-GD (3) can be written as U t +
-]T sα > 0 satisfying that lim
[Equation] F = 0 .
We then show that T sα is monotone increasing in s for any fixed α .
Our first result states that with small initialization, GD can get into a small neighbourhood of Z ∗ s .
Lemma 4.1 Under Assumptions 3.1 and 3.2, there exists ˆ T sα > 0 for all α > 0 and 1 ≤ s ≤ ˆ r ∧ r ∗ such that lim [Equation] t ) [Equation]
Furthermore, it holds that
[Equation]
Proof sketch : The proof is motivated by the three-phase analysis in St ¨ oger & Soltanolkotabi (2021) but has some technical modifications.
Starting from a small initialization, GD initially behaves similarly to power iteration since U t
+1 =
( I + µ M t ) U t ≈ ( I + µ M ) U t , where M : = A ∗ A ( XX ⊤ ) is a symmetric matrix.
Let M = (cid:80) dk
=1 ˆ σ 2 k ˆ v k ˆ v
⊤ k be the eigendecomposition of M . Then we have
[Equation]
where the last step holds because it can be shown that 1+ µ ˆ σ s > 1+ µ ˆ σ s +1 , causing an exponential separation between the magnitude of the top- s and the remaining components.
When (5) no longer holds, we enter a new phase which we call the parallel improvement phase .
We consider the decomposition U t =
U t W t W ⊤ t + U t W t, ⊥ W ⊤ t, ⊥ , where W t := W V ⊤ X s U t ∈ R ˆ r × s is the matrix consisting of the right singular vectors of V ⊤ X s U t (Definition 3.1) and W t, ⊥ ∈ R ˆ r × (ˆ r − s ) is an orthogonal complement of W t .
Assume X s = [ σ 1 e 1 , · · · , σ s e s ] without loss of generality. The columns of W t is an orthogonal basis of the subspace spanned by the first s rows of U t . Each vector in R r can be decomposed as a parallel component and an orthogonal component w.r.t.
this subspace. Intuitively, the row vectors of U t W t are the parallel components of the row vectors in U t ; we call U t W t the parallel component . For similar reasons, U t W t, ⊥ will be referred to as the orthogonal component .
More discussions for this decomposition are given in Appendix B.
By the end of the spectral phase we have σ min ( U t W t ) ≫ ∥
U t W t, ⊥ ∥ . We show in Appendix C. that afterwards, σ min ( U t W t )
∥ U t W t, ⊥ ∥ grows exponentially in t , until the former reaches a constant scale, while the latter stays o (1) ( α → 0 ).
After σ min ( U t W t ) = Θ(1) , we enter the refinement phase where we show in Appendix C. that (cid:13)(cid:13)
X s X ⊤ s − U t
U ⊤ t (cid:13)(cid:13) F keeps decreasing until it is O (cid:0) δκ 2 √ r ∗ ∥ X ∥ 2 (cid:1) (see Lemma 5.1).
On the other hand, we can show that best rank- s solution is close to the matrix factorization minimizer i.e. (cid:13)(cid:13) Z ∗ s − X s X ⊤ s (cid:13)(cid:13) F = O (cid:0) δκ √ r ∗ ∥ X ∥ 2 (cid:1) . We thus obtain that (cid:13)(cid:13) Z ∗ s −
U t U ⊤ t (cid:13)(cid:13) F = O (cid:0) δκ 2 √ r ∗ ∥ X ∥ 2 (cid:1) . Finally, since rank ( U t W t ) ≤ s , we have σ s +1 ( U t ) ≤ ∥ U t W t, ⊥ ∥ = o (1) , as desired.
□
Lemma 4.1 shows that U t U ⊤ t would enter a neighbourhood of Z ∗ s with constant radius.
However, there is still a gap between Lemma 4.1 and Theorem 4.1, since the latter states that U t U ⊤ t would actually get o (1) -close to Z ∗ s .
To illustrate our proof idea of this result, we first consider the simpler setting where the model is under-parameterized , i.e., ˆ r ≤ r ∗ and show that U t U ⊤ t would eventually converge to Z ∗ ˆ r .
Proposition 4.1 (Convergence in the under-parameterized regime)
Suppose that ˆ r ≤ r ∗ , then there exists a constant c
= c (ˆ r, κ ) > 0 such that when α < c , we have lim t → + ∞ U α,t U ⊤ α,t = Z ∗ ˆ r .
Proof sketch : We can deduce from Lemma 4.1 by taking s = ˆ r that there exists a global minimizer
U ∗ ˆ r of (1) (equivalently, a matrix in R d × s satisfying U [-
 =-]∗ ˆ r U ∗ ˆ r ⊤ =
Z ∗ ), such that (cid:13)(cid:13)(cid:13)
U α, ˆ T ˆ rα − U ∗ ˆ r (cid:13)(cid:13)(cid:13) =
O ( κ 3 √ r ∗ δ ∥ X ∥ ) (cf.
Corollary F.1). On the other hand, by taking s = ˆ r in Theorem 5.1, we can see that within a neighbourhood of U ∗ ˆ r with constant radius, f satisfies a Polyak-Łojasiewicz (PL) type condition with respect to the procrutes distance defined in Definition 5.1, and the global minimizer of f is unique up to orthogonal transformation. When δ is sufficiently small, U α,T ˆ rα lies in this neighbourhood, and the PL condition implies that GD converges linearly to the set of global minimizers, which yields the desired conclusion.
□
Now we turn to the over-parameterized regime, where f is not necessarily local PL, and thus we cannot directly derive convergence as in Proposition 4.1.
To proceed, we use a low-rank approximation for U t and associate the dynamics in this neighborhood with the GD dynamics of the following under-parameterized matrix sensing loss:
[Equation]
It can be shown that when δ is sufficiently small, the global minimizer of f s is unique up to rotation, i.e., if ˆ U ∗ s is a global minimizer of f s , then any other global minimizer can be written as ˆ U ∗ s R for some orthogonal matrix R ∈ R s × s .
The main observation is that GD follows [-directly by choosing (cid:101) U 
-]an approximate lowrank trajectory until it gets into a small neighbourhood of Z ∗ s , so that we can still use our landscape results in the low-rank regime.
Proof sketch of Theorem 4.1: For all t ≥ − ˆ T sα , define ˆ U α,t =
U α,t
+ ˆ T sα W α,t + ˆ T sα ∈ R d × s , where W α,t =
W V
⊤ X s U t as defined in the proof sketch of Lemma 4.1. Note that Lemma 4.1 implies that U α,T sα is approximately rank- s , so within a time period after ˆ T sα , the GD trajectory remains close to another GD initialized at ˆ U α, 0 for the rank- s matrix sensing loss (6) until t = Θ (cid:0) log 1 α (cid:1) , i.e.
[Equation]
Again by Lemma 4.1, the initialization ˆ U α, 0 is within a small neighbourhood of the global minima of f s ( U ) .
Furthermore, Theorem 5.1 implies that f s ( U ) satisfies a local PL-like condition, so that GD with good initialization would converge linearly to its global minima (Karimi et al., 2016).
We need to choose a time t such that (7) remains true while this linear convergence takes place for sufficiently many steps.
We can show that there always exists some t = t sα such that both (cid:13)(cid:13)(cid:13) ˆ U α,t ˆ U ⊤ α,t − U α, ˆ T sα + t U
⊤ α, ˆ T sα + t (cid:13)(cid:13)(cid:13) F and (cid:13)(cid:13)(cid:13)
ˆ U α,t − U ∗ s (cid:13)(cid:13)(cid:13) F are O (cid:16)
α 1 Mκ 2 (cid:17) .
Hence (cid:13) (cid:13)
U α,t U ⊤ α,t − Z ∗ s (cid:13) (cid:13) F ≲ α 1 Mκ 2 for t =
T s α :
=
ˆ T s α + t s α . For all 1 ≤ s < ˆ r ∧ r ∗ , since (7) always holds for t ≤ t sα and rank (cid:16) ˆ U α,t (cid:17) ≤ s , we must have max 1 ≤ {+t ≤ T sα σ s +
+}
( U α,t ) =
o (1) as α → 0 .
Finally, (cid:13)(cid:13)
Z ∗ s +1 − X s +1 X ⊤ s +1 (cid:13)(cid:13)
=
O ( δ ) (cf. Lemma E.5), so σ s +1 (cid:0) Z ∗ s +1 (cid:1) = Θ(1) . Therefore, U α,t U ⊤ α,t cannot be close to Z ∗ s +1 when t ≤ T sα , so we must have T s +1 α > T sα .
This completes the proof of Theorem 4.1.
C ONVERGENCE IN U NDER -P ARAMETERIZED M ATRIX S ENSING
In this section, we analyze the properties of the landscape of the matrix sensing loss (1), which plays a crucial role in proving results in Section 4.
While the landscape of over-parameterized matrix sensing (i.e. ˆ r ≥ r ∗ ) is well-studied (Tu et al., 2016;
Ge et al., 2017), few results are known for the under-parameterized case.
Our results provide useful tools for analyzing the convergence of gradient-based algorithms for solving problems like low-rank matrix approximation and might have independent interests.
To prove convergence results like Proposition 4.1, we first study the landscape of f s defined in (6) near the set of global minimizers. One major difficulty here is that f s is not locally stronglyconvex,
because if U ∈ R
d × s is a global minimizer of f s , then UR is also a global minimizer, for any orthogonal matrix R ∈ R s × s .
Nonetheless, we can establish a Polyak-Łojasiewicz (PL) like condition, Theorem 5.1, which is the main result of this section.
Definition 5.1 (Procrutes distance) For any d, s ∈ N + and U 1 , U 2 ∈ R d × s , we define dist( U 1 , U 2 ) =
min {∥ U 1 − U 2 R ∥
F : R ∈ R s × s is orthogonal } .
We note that the procrutes distance is well-defined because the s × s orthogonal matrices are a compact set and thus ∥ U 1 − U 2 R ∥ F is continuous in R . It can be verified that procrutes distance is a pseudo metric, i.e. , it is symmetric and satisfies the triangle inequality.
Theorem 5.1 (Landscape of under-parameterized matrix sensing)
The global minimizer of f s is unique up to an orthogonal transformation, i.e. the set of global minimizers of f s is { U ∗ s R : R ∈ R s × s is orthogonal } where U ∗ s is an arbitrary global minimizer. Moreover, let
U ∈ R
d
× s and U ∗ s be a global minimizer of f s such that ∥ U − U ∗ s ∥ F = dist( U , U ∗ s ) . Suppose that Assumption 3.1 holds and ∥ U − U ∗ s ∥ ≤ 10 − 2 κ − 1 ∥ X ∥ , then ⟨∇ f s ( U ) , U − U ∗ s ⟩ ≥ 0 . 1 τ dist 2 ( U , U ∗ s ) .
Remark 5.1 Recall PL condition means there exists some constant µ > 0 , such that ∥∇ g ( x ) ∥ 2 ≥ 2 µ ( g ( x )
− min y ∈ R n g ( y )) holds for all x . Since f s is locally smooth around U ∗ s , there exists a constant c 1 > 0 such that f s ( U ) − f s ( U ∗ s ) ≤ c 1 ∥ U − U ∗ s ∥ 2 F . Moreover, Theorem 5.1 implies that ∥ U − U ∗ s ∥ 2 F ≤ 100 τ − 2 ∥∇ f s ( U ) ∥ 2 F , so we have ∥∇ f s ( U ) ∥ 2 F ≥ 10 − 2 τ 2 c − 1 1 ( f s ( U ) − f s ( U ∗ s )) .
In other words, Theorem 5.1 implies that the matrix sensing loss (1) is locally PL.
When δ = 0 , f s reduces to the matrix factorization loss F s : R d × s → R , F s ( U ) = 14 d 2 (cid:13)(cid:13) UU ⊤ − Z ∗ (cid:13)(cid:13) 2 F .
The following corollary immediately follows from Theorem 5.1.
Corollary 5.1 (Landscape of under-parameterized matrix factorization)
The set of global minimizers of F s is { X s R : R ∈
R s × s is orthogonal } .
Moreover, under Assumption 3.1, given
U ∈ R d × s and let R be an orthogonal matrix such that ∥ U − X s R ∥ F = dist( U , X s )
If dist( U , X s ) ≤ 10 − 2 κ − 1 ∥ X ∥ , then ⟨∇ F ( U ) , U − X s R ⟩ ≥ 0 .
1 τ dist 2 ( U , X s ) .
We end this section with the following lemma that formalizes the intuition that all global minimizers of the f s must be close to X s under the procrutes distance, which is used in the proof sketch of
Theorem 4.1 in Section 4.
Lemma 5.1 Under Assumption 3.1, we have dist( U ∗ s , X s ) ≤ 40 δκ ∥ X ∥ F for any global minimizer
U ∗ s of f s .
Moreover, (cid:13)(cid:13)
Z ∗ s − X s X ⊤ s (cid:13)(cid:13) F ≤ 80 δκ √ r ∗ ∥ X ∥ 2 .
E XPERIMENTS
In this section, we perform some numerical experiments to illustrate our theoretical findings.
Experimental setup.
Consistent with our theory, we consider the matrix sensing problem (1) with d = 50 , r ∗ = 5 , α ∈ { 1 , 0 . 1 , 0 . 01 , 0 .
001 } , m ∈ { 1000 , 2000 , 5000 } .
We will consider different choices for ˆ r in the experiments.
The ground-truth Z ∗ =
XX ⊤ is generated such that the entries [Figure] of X are i.i.d.
standard Gaussian variables.
Throughout our experiments, we use the same groundtruth to compare different choices of parameters.
For i = 1 , 2 , · · · , m , all entries of the measurement A i ∈ R
d × d are chosen i.i.d.
from the standard Gaussian N (0 , 1) .
When m ≳ dr ∗ δ − 2 , this set of measurements satisfies the RIP with high probability (Recht et al., 2010, Theorem 4.2).
We solve the problem (1) via running GD for T = 10 4 iterations starting with small initialization with scale α .
Specifically, we choose U
= α U where the entries of U ∈ R
d × ˆ r are drawn i.i.d.
from standard Gaussian distribution.
We consider both the over-parameterized and the exact/under-parameterized regime.
The learning rate of GD is set to be µ = 0 .
6. I MPLICIT LOW - RANK BIAS
In this subsection, we consider the over-parameterized setting with r[-∗ i =
 σ 2 i v i v ⊤ i and X s X ⊤ s-] = 50 .
For each iteration t ∈ [ T ] and rank s ∈ [ r ∗ ] , we define the relative error E s ( t ) =
∥
[Equation]
to measure the proximity of the GD iterates to X s .
We plot the relative error in Figure 1 for different choices of α and m (which affects the measurement error δ ).
Small initialization.
The implicit low-rank bias of GD is evident when the initialization scale α is small.
Indeed, one can observe that GD first visits a small neighbourhood of X 1 , spends a long period of time near it, and then moves towards X 2 .
It then proceeds to learn X 3 , X 4 , · · · in a similar way, until it finally fits the ground truth.
This is in align with Theorem 4.1.
By contrast, for large initialization we do not have this implicit bias.
[Figure]
The effect of measurement error.
For fixed α , one can observe the the relative error becomes smaller when the number of measurement increases.
This is in align with Lemma 4.1 in which the bound depends on δ .
In particular, for the case s = r ∗ , although GD with fixed initialization does not converge to global minima, but the distance to the set of global minima scales as poly( α ) .
6. M ATRIX SENSING WITH EXACT PARAMETERIZATION
Now we study the behavior of GD in the exact parameterization regime ( r = r ∗ ).
We fix m =and r = r ∗ = 5 and run GD for T = 500 iterations.
We plot the relative error in Figure 2.
We can see that GD exhibits an implicit low-rank bias when α is small.
However, choosing a very small α would slow down the convergence speed.
This is because GD would get into a poly( α ) - neighbourhood of the saddle point Z s and spend a long time escaping the saddle.
Also, convergence to global minimizers is guaranteed as long as α is below a certain threshold (see Proposition 4.1).
C ONCLUSION
In this paper, we study the matrix sensing problem with RIP measurements and show that GD with small initialization follows an incremental learning procedure, where GD finds near-optimal solutions with increasing ranks until it finds the ground-truth.
We take a step towards understanding the optimization and generalization aspects of simple optimization methods, thereby providing insights into their success in modern applications such as deep learning (Goodfellow et al., 2016).
Also, we provide a detailed landscape analysis in the under-parameterized regime, which to the best of our knowledge is the first analysis of this kind.
Although we focus on matrix sensing in this paper, it has been revealed in a line of works that the implicit regularization effect may vary for different models, including deep matrix factorization (Arora et al., 2019) and nonlinear ReLU/LeakyReLU networks (Lyu et al., 2021; Timor et al., 2022).
Also, it is shown in Woodworth et al. (2020) that different initialization scales can lead to distinct inductive bias and affect the generalization and optimization behaviors.
All these results indicate that we need further studies to comprehensively understand gradient-based optimization methods from the generalization aspect. Appendix Table of Contents A Preliminaries
A.1 The RIP condition and its properties . . . . . . . . . . . . . . . . . . . . . . . . A.2 Matrix analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.3 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.4 Proof of Proposition 3.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Main idea for the proof of Theorem 4.
B.1 Heuristic explanations of the decomposition . . .
B.2 Proof outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C Proof of Lemma B.
{+C.
+}
The spectral phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
{+C.
+}
The parallel improvement phase . . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 Induction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
{+C.
+}
The refinement phase and concluding the proof of Lemma B.1 . . . . . . . . . .
D Auxiliary results for proving Lemma B.
D.1 The spectral phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
{+D.
+}
The parallel improvement phase . . . . . . . . . . . . . . . . . . . . . . . . . .
The appendix is organized as follows: in Appendix A we present a number of results that will be used for later proof.
Appendix B sketches the main idea for proving our main results.
Appendix C is devoted to a rigorous proof of Lemma B.1 ,with some auxiliary lemmas proved in Appendix D.
In
Appendix E we analyze the landscape of low-rank matrix sensing and prove our results in Section 5.
These results are then used in Appendix F to prove Theorem 4.1.
Finally, Appendix G studies the landscape of rank- 1 matrix sensing, which enjoys a strongly convex property, as we mentioned in
Section 5 without proof.
A P RELIMINARIES
In this section, we present some useful results that is needed in subsequent analysis.
A. T HE RIP CONDITION AND ITS PROPERTIES
In this subsection, we collect a few useful properties of the RIP condition, which we recall below:
Definition A.1 We say that the measurement A satisfies the ( δ, r ) -RIP condition if for all matrices
Z ∈ R d × d with rank ( Z ) ≤ r , we have
[Equation]
The key intuition behind RIP is that A ∗ A ≈ I , where A ∗ : v (cid:55)→ 1 √ m (cid:80) mi
=1 v i
A
i is the adjoint of
A .
This intuition is made rigorous by the following proposition:
Proposition A.1 (St¨oger & Soltanolkotabi, 2021, Lemma 7.3) Suppose that A satisfies ( r, δ ) -RIP, then for all symmetric matrix Z of rank ≤ r − 1 , we have
[Equation]
A. M ATRIX ANALYSIS
The following lemma is a direct corollary of Proposition A.1 and will be frequently used in our proof.
Lemma A.1 Suppose that the measurement A satisfies ( δ, 2 r ∗ + 1) -RIP condition, then for all matrices
U ∈ R
d × r such that rank ( U ) ≤ r ∗ , we have
[Equation]
In our proof we will frequently make use of the Weyl’s inequality for singular values:
Lemma A.2 (Weyl’s inequality) Let A ,
∆ ∈ R d
× d be two matrices, then for all 1 ≤ k ≤ d , we have
[Equation]
We will also need the Wedin’s sin theorem for singular value decomposition:
Lemma A.3 (Wedin, 1972, Section 3)
Define R ( · ) to be the column space of a matrix.
Suppose that matrices B = A + T , A 1 , B 1 are the top- s components in the SVD of A and B respectively, and
A
=
A − A 1 , B 0 = B − B 1 .
If δ = σ min ( B 1 ) − σ max ( A 0 ) > 0 , then we have
[Equation]
where Θ( · , · ) denotes the angle between two subspaces.
Equipped with Lemma A.1, we can have the following characterization of the eigenvalues of M (recall that M =
A ∗ A ( XX ⊤ ) ):
Lemma [-A.
-]A.4 Let M : = A ∗ A ( XX ⊤ ) and M = (cid:80) dk =1 ˆ σ 2 k ˆ v k ˆ v ⊤ k be the eigen-decomposition of
M .
For 1 ≤ i ≤ d we have
[Equation]
Proof : By Weyl’s inequality we have
[Equation]
as desired.
□
A. O PTIMIZATION
Lemma A.5 Suppose that a smooth function f ∈ R m (cid:55)→ R with minimum value f ∗ > −∞ satisfies the following conditions with some ϵ > 0 :
lim
∥ x ∥→ + ∞ f ( x ) = + ∞ . There exists an open subset S ⊂ R m such that the set S ∗ of global minima of f is contained in S , and for all stationary points x of f in R m − S , we have f ( x ) − f ∗ ≥ 2 ϵ .
Moreover, we also have f ( x ) − f ∗ ≥ 2 ϵ on ∂S .
Then we have
[Equation]
Proof : Let x ∗ be the minimizer of f on R m − S .
By condition (1) we can deduce that x ∗ always exists.
Moreover, since any local minimizer of a function defined on a compact set must either be a stationary point or lie on the boundary of its domain, we can see that either x ∗ ∈ ∂S or ∇ f ( x ∗ ) =holds.
By condition (2), either cases would imply that f ( x ∗ ) − f ∗ ≥
2 ϵ , as desired.
□
Lemma A.6 Let { x k } , { y k } ⊂ R n be two sequences generated by x k +1 =
x k − µ ∇ f ( x k ) and y k +
=
y k − µ ∇ f ( y k ) .
Suppose that ∥ x k ∥ ≤ B and ∥ y k ∥ ≤ B for all k and f is L -smooth in { x ∈ R n : ∥ x ∥ ≤ B } , then we have
[Equation]
Proof : The update rule implies that
[Equation]
which yields the desired inequality.
□
A. P ROOF OF P ROPOSITION 3.
Proposition 3.2 immediately follows from the following result:
Proposition A.2 (Rudelson & Vershynin, 2009) Suppose that all entries of U ∈ R d × r are independently drawn from N (cid:16) 0 , 1 √ r (cid:17) and ρ = ϵ √ r −√ s − 1 √ r , then σ min (cid:0)
V ⊤ X s U (cid:1) ≥ ρ with probability at least 1 − e − cr − ( Cϵ ) r − s +1 .
Here c, C > 0 are universal constants.
By Proposition A.2, we have
[Equation]
which concludes the proof of Proposition 3.2. B M AIN IDEA FOR THE PROOF OF T HEOREM 4.
In this section, we briefly introduce our main ideas for proving Theorem 4.1.
Motivated by St¨oger & Soltanolkotabi (2021), we decompose the matrix U t into a parallel component and an orthogonal component.
Specifically, we write
[Equation]
where W t :=
W V
⊤ X s U t ∈ R ˆ r × s is the matrix consisting of the right singular vectors of V ⊤ X s U t (Definition 3.1) and W t,
⊥ ∈ R ˆ r × (ˆ r − s ) is an orthogonal complement of W t .
Our goal is to prove that at some time t , we have V ⊤ X s (cid:0)
U t
U ⊤ t − X s X ⊤ s (cid:1)
≈ 0 and ∥
U t W t, ⊥ ∥
≈ 0 .
As we will see later, these imply that (cid:13)(cid:13)
U t U ⊤ t − X s X ⊤ s (cid:13)(cid:13)
≈ 0 .
The remaining part of this section is organized as follows: in Appendix B.1 we give a heuristic explanation for considering (8), and in Appendix B.2, we present our proof outline.
Additional Notations.
[-≈ 
-]Let V X s , ⊥ ∈ R d × ( d − s ) be an orthogonal complement of V X s ∈ R d × s . Let Σ s = diag( σ 1 , . . .
, σ s ) and Σ s, ⊥ = diag( σ s +1 , . . . , σ d ) . We use ∆ t := ( A ∗ A − I )( XX ⊤ −
U t U ⊤ t ) to denote the vector consisting of measurement errors for XX ⊤ − U t
U ⊤ t .
B. H EURISTIC EXPLANATIONS OF THE DECOMPOSITION
A simple and intuitive approach for showing the implicit low rank bias is to directly analyze the growth of V ⊤ X s U t versus V ⊤ X s , ⊥ U t .
Ideally, the former grows faster than the latter, so that GD only learns the components in X s .
By the update rule of GD (3),
[Equation]
For the first term G t, 1 , we have [Equation] where the last term O ( µ 2 ) is negligible when µ is sufficiently small.
Since ∥ Σ s, ⊥ ∥
=
σ s +1 , the spectral norm of G t, 1 can be bounded by
[Equation]
However, the main difference with the full-observation case (Jiang et al., 2022) is the second term
G t, 2 := V ⊤ X s , ⊥ ∆ t U t . Since the measurement errors ∆ t are small but arbitrary, it is hard to compare this term with V ⊤ X s , ⊥ U t +1 .
As a result, we cannot directly bound the growth of ∥ V ⊤ X s , ⊥ U t ∥ .
However, the aforementioned problem disappears if we turn to bound the growth of ∥ V ⊤ X s , ⊥ U t +1 W t, ⊥ ∥ . To see this, first we deduce the following by repeatedly using V ⊤ X s U t W t, ⊥ = {+
 due to the definition of W t, ⊥ . [Equation]+}
[Equation]
So we have the following recursion:
[Equation]
We further note that
[Equation]
which establishes the relationship between V ⊤ X s , ⊥ U t +1 W t, ⊥ and V ⊤ X s , ⊥ U t +1 W t +1 , ⊥
To complete the proof we need to prove the following:
• The minimal eigenvalue of the parallel component U t W t W ⊤ t grows at a linear rate with speed strictly faster than σ s +1 .
• The term (cid:13)(cid:13)(cid:13) V ⊤ X s , ⊥ V U t W t (cid:13)(cid:13)(cid:13) ≪ 1 , which implies that the first term in (9) is negligible.
B. P ROOF OUTLINE
An important intermediate step of our proof is the following result:
Lemma B.1 Under Assumptions 3.1 and 3.2, if the initialization scale α is sufficiently small, then for all 1 ≤ s ≤ ˆ r ∧ r ∗ there exists a time T
sα ∈ Z + such that
[Equation]
We begin with the spectral alignment phase , where we can make the following approximation
[Equation]
since U t is initially small.
At some time t = T 0 , U t would become approximately aligned with the first s components, as long as there is a positive gap between the s -th and ( s + 1) -th largest eigenvalues of M .
The choice of T 0 is subject to a trade-off such that the alignment takes effect while (10) does not induce large error.
We then enter the second phase which we call the parallel matching phase, in which the parallel components grow to a constant magnitude and is well-matched with the ground-truth (the orthogonal components remain small).
Specifically, for small constants c i , 1 ≤ i ≤ 3 , we show that the followings are true in this phase:
σ min ( V ⊤ X s U t ) grows exponentially fast until it reaches c · σ s for some constant c at some time T s .
Specifically, we have
[Equation]
When t ≤ T s , the growth speed of ∥ U t W t, ⊥ ∥ is slower than σ 2min ( V ⊤ X s U t ) :
[Equation]
(cid:13)(cid:13)(cid:13) V ⊤ X s , ⊥ V U t W t (cid:13)(cid:13)(cid:13) ≤ c 3 remains true until t = T s
These statements will be proven by induction.
At t =
T s , we have σ min ( V ⊤ X s U t )
= Θ(1) , and we enter the refinement phase in which we show that the quantity (cid:13)(cid:13) [Equation]
decrease exponentially until it reaches O ( δ ) .
Note that in general GD would not converge to an o (1) -neighbourhood of X s X ⊤ s as the initialization scale α → 0 , because X s X ⊤ s is not the rank- s minima of the RIP loss.
As a result, in the refinement phase we can only expect to obtain (cid:13)(cid:13)
[Equation]
To conclude the proof of Theorem 4.1, we prove in Section
that the landscape of the matrix sensing loss with rank- s parameterization, though non-convex, satisfies a local Polyak-Lojasiewicz (PL) condition within a neighborhood of constant radius.
As a result, for sufficiently small δ , GD converges linearly to global minima with good initialization.
We then show that within a time period after the refinement phase, the GD trajectory is close to the trajectory of well-initialized GD for ranks parameterized matrix sensing.
The length of this period goes to infinity when α → 0 , thereby implying that GD finds the rank- s minimizer with o (1) error.
The details are given in Appendix F.
C P ROOF OF L EMMA B.
In this section, we give the full proof of Lemma B.1, with some additional technical lemmas left to Appendix D. Appendices C.1 and C.2 are devoted to analyzing the spectral phase and parallel improvement phase, respectively.
Appendix C.3 uses induction to characterize the low-rank GD trajectory in the parallel improvement phase.
In Appendix C.4 we study the refinement phase, which allows us to derive Lemma B.1.
C. T HE SPECTRAL PHASE
Starting from a small U
= α U , α ≪ 1 , we first enter the spectral phase where GD behaves similar to power iteration.
As in St¨oger & Soltanolkotabi (2021), we refer to this phase as the spectral phase.
Specifically, we have in the spectral phase that
[Equation]
The approximation holds with high accuracy as long as ∥ U t ∥ ≪ 1 .
Moreover we have M :=
( A ∗ A ) ( XX ⊤ )
≈ XX ⊤ by the RIP condition; when δ is sufficiently small, we can still ensure a positive eigen-gap of M .
As a result, with small initialization U t would become approximately aligned with the top eigenvector u 1 of M .
Since ∥ M − XX ⊤ ∥
=
O ( δ √ r ∗ ) by Proposition A.1, we have ∥ u 1 − v 1 ∥ =
O ( δ √ r ∗ ) so that ∥ V ⊤ X s V
U t W t ∥ =
O ( δ √ r ∗ ) .
This proves the base case for the induction.
Formally, we define M =
A ∗ A ( XX ⊤ ) , Z t =
( I + µ M ) t and (cid:101) U t
=
Z t U 0 .
Suppose that [Equation]
ˆ σ 2 i ˆ v
i ˆ v ⊤ i is the spectral decomposition of M .
We additionally define M s = (cid:80) min { s, rank( M ) } i
= ˆ σ 2 i ˆ v i ˆ v ⊤ i .
By Lemma A.4 and δ √ r ∗ ≤ 10 − 3 κ as stated in Lemma B.1, we have ˆ σ {+s ≥ σ s − 0 . 01 τ and ˆ σ s +1 ≥ σ s +
 + 0 . 01 τ , where τ+}
= σ s − σ s +1 > 0 .
Additionally, let L t be the span of the top- s left singular vectors of U t .
We make the following assumption on the initialization, which holds with high probability when it is i.i.d.
Gaussian:
Assumption C.
The matrix V ⊤ M s U ∈ R s × r has full row-rank i.e. ρ = σ min (cid:0) V ⊤ M s U (cid:1) > 0 .
Let
[Equation]
the following lemma bounds the error of approximating U t via (cid:101)
U t :
Lemma C.1 (St¨oger & Soltanolkotabi, 2021, Lemma 8.1) Suppose that A satisfies the rank-1 RIP with constant δ 1 .
For all integers t such that 1 ≤ t ≤ t ⋆ it holds that
[Equation]
Corollary C.
We have
[Equation]
Proof : By Lemma C.1 we have
[Equation]
for all t ≤ t ∗ .
On the other hand, we have
[Equation]
Thus, it follows from ∥ E t ∗ ∥ ≥ ∥ (cid:101)
U t ∗ ∥ that
[Equation]
≥
[Equation]
as desired.
□
Lemma C.2 There exists a time
[Equation]
such that
[Equation]
where γ = 1 −
[Equation]
Proof : It’s easy to check that T 0 ≤ t ∗ by applying Corollary C.1.
We consider the following decomposition:
[Equation]
When t ≤ t ∗ , the first term can be bounded as
[Equation]
For the second term we have
[Equation]
In particular, the definition of T 0 implies that
[Equation]
as desired.
□
We conclude this section with the following lemma, which states that initially the parallel component
U t W t would grow much faster than the noise term, and would become well-aligned with X s .
Lemma C.
The following inequalities hold for t =
T 0 when α ≲ ρ − 4 κ is sufficiently small:
[Equation]
Proof : We prove this lemma by applying Corollary D.1 to t =
T 0 defined in the previous lemma.
The inequality (12a) can be directly verified by using Lemma C.2:
[Equation]
For the remaining inequalities, we first verify that the assumption in Corollary D.1:
[Equation]
[Equation]
[Equation]
By definition of Z t , we can see that
[Equation]
when α ≤ poly( r ∗ ) − 1 , so that (13) holds.
As a result, we have
[Equation]
The conclusion follows.
□
C. T HE PARALLEL IMPROVEMENT PHASE C.2. T HE SIGNAL TERM
In the following we estimate σ min (cid:0)
V ⊤ X
s U t +1 W t (cid:1) .
We state our main result of this section in the lemma below.
Lemma C.4 Suppose that V ⊤ X s U t ∈ R s × r is of full rank, µ < 10 − 4 ∥
X ∥ − 2 , c 3 < 10 − 3 κ − 1 , (cid:13)(cid:13)
( A ∗ A − I ) ( XX ⊤ − U t
U ⊤ t ) (cid:13)(cid:13) <
10 − 3 κ − 1 ∥ X ∥ 2 and ∥ V X ⊥ s V U t W t ∥ ≤ c 3 , then we have σ min ( V ⊤ X s U t +1 ) ≥ σ min ( V ⊤ X s U t +1 W t ) ≥ (cid:0) 1 + µ (cid:0) σ 2 s − (5 c 3 + 20 δ )
∥
X ∥ 2 (cid:1) − 500 µ 2 ∥ X ∥ 4 (cid:1) (cid:0) 1 − µσ 2min
( V ⊤ X s U t ) (cid:1) σ min ( V ⊤ X s U t ) .
Proof : The update rule of GD implies that
[Equation]
15a)
[Equation]
15b)
[Equation]
W
[Equation]
15c) where (15a) follows from V ⊤ X s XX ⊤ =
V ⊤ X s X s X ⊤ s + V ⊤ X
s X s, ⊥ X ⊤ s, ⊥ and V ⊤ X s X s, ⊥ = 0 ; (15b) follows from V ⊤ X s X s X ⊤ s = V ⊤ X
s V X
s Σ s V ⊤ X s = Σ s V ⊤ X s , and (15c) follows from
V ⊤ X s U t =
V ⊤ X s U t W t W ⊤ t +
[Equation]
⊤ X
[Equation]
, ⊥
[Equation]
⊤ t, ⊥ = V ⊤ [Equation] s U t
[Equation] t [Equation] ⊤ t by definition of W t and W t, ⊥ .
We now relate the last three terms in (15c) to V ⊤ X s U t W t .
Since V ⊤ X s U t W t is assumed to be invertible, so is V ⊤ X
s V U t W t , Σ U t W t and W U t W t , thus we have
[Equation]
Plugging (16) into the second and third terms of (15) and re-arranging, we deduce that
[Equation]
where we use the equation A
= ( I − µ AA ⊤ ) − 1 A
( I − µ A ⊤ A ) with A = V ⊤ X s U t W t (when µ < 1 9 ∥ X ∥ 2 , I − µ AA ⊤ is invertible)
[Equation]
and
[Equation]
By assumption we have
[Equation]
so that
[Equation]
19) and by our assumption we have
[Equation]
Moreover, note that
[Equation]
, and when µ
[Equation]
1 [-
-]10 ∥
[Equation]
we have
[Equation]
(cid:13)(cid:13)(cid:13) < 2 . Thus
[Equation]
The equation (18) implies that
[Equation]
Recall that P 1 and P 2 are bounded in (19) and (20) respectively, so we have that
[Equation]
The conclusion follows.
□
The corollaries below immediately follow from Lemma C.4.
Corollary C.2 Under the conditions in Lemma C.4, if σ 2min ( V ⊤ X s U t ) < 0 .
3( σ 2 s − σ 2 s +1 )
= 0 .
3 κ − 1 ∥ X ∥ 2 , then we have
[Equation]
s +1 ) (cid:1)
[Equation]
min ( V ⊤ X s U t ) .
Corollary C.3 Under the conditions in Lemma C.4, if
[Equation]
then we have that σ min ( V ⊤ X s U t +1 ) ≥ σ min ( V ⊤ X s U t ) .
C.2. T HE NOISE TERM
In this section we turn to analyze the noise term.
The main result of this section is presented in the following:
Lemma C.5 Suppose that V ⊤ X s U t +1 W t ∈ R s × r is of full rank,
[Equation]
, ⊥
[Equation]
U t
[Equation]
<
[Equation]
κ
[Equation]
and [Equation]
then we have [Equation]
Proof : By the definition of W t, ⊥ , we have V ⊤ X s U
t W t, ⊥ = 0 , thus ∥ U t W t, ⊥ ∥ =
(cid:13)(cid:13)(cid:13) V ⊤ X s , ⊥ U t W t, ⊥ (cid:13)(cid:13)(cid:13) .
The latter can be decomposed as follows:
[Equation]
In the following, we are going to show that the term (a) is bounded by c · µ where c is a small constant, while (b) grows linearly with a slow speed.
Bounding summand (a).
Since
[Equation]
by definition
[Equation]
we have
[Equation]
[Equation]
Thus the summand (a) can be rewritten as follows:
[Equation]
[Equation]
=
[Equation]
[Equation]
, where M 1 = V ⊤ X s (cid:2)
U t U ⊤ t V X s , ⊥ − ∆ t V X s , ⊥ (cid:3) .
In (22), (22a) follows from (21), (22b) holds since Σ U t +1 W t W ⊤ U t +1 W t ∈
R s × s is invertible, and in (22c)
we use V ⊤ X s U t W t, ⊥ = 0 .
It follows that ∥
[Equation]
By Lemma D.4 we have (cid:13)(cid:13)(cid:13) V ⊤ X s , ⊥ V U t +1 W t (cid:13)(cid:13)(cid:13) ≤ 0 . 01 , which implies that (cid:13)(cid:13) (cid:13) (cid:0)
V ⊤ X s V
U t +1 W t (cid:1)
[Equation]
Lastly, we bound M 1 as follows:
[Equation]
where the second inequality follows from our assumption on (cid:13)(cid:13) (
[Equation]
Combining (23) [Equation] (24) and (25) yields
[Equation]
Bounding summand (b).
This is the main component in the error term.
We’ll see that although this term can grow exponentially fast, the growth speed is slower than the minimal eigenvalue of the parallel component.
We have
[Equation]
26d [Equation] where we recall that Σ 2 s, ⊥ = diag (cid:0) σ 2 s +1 , · · · , σ 2 r , 0 , · · · , 0 (cid:1) ∈ R ( d − s )
× ( d − s ) .
In (26), (26a) follows from the update rule of GD, (26b) is obtained from V ⊤ X s , ⊥ XX ⊤ =
Σ 2 s, ⊥ V ⊤ X s , ⊥ and U t W t, ⊥ =
V X
[Equation] [Equation]
, ⊥ U
[Equation]
t, ⊥ , and lastly in (26d) we use
[Equation]
U t W t, ⊥ .
It follows that
[Equation]
≤
[Equation]
I
[Equation]
To summarize, we have
[Equation]
as desired.
□
To bound the growth speed of the orthogonal component, we need to show that the quantity (cid:13)(cid:13) (cid:13) V ⊤ X s , ⊥ V U t W t (cid:13)(cid:13) (cid:13) remains small.
The following lemma serves to complete an induction step from t to t + 1 :
Lemma C.6 Suppose that ∥ V X s , ⊥ V U t W t ∥ ≤ cand ∥
U t W t, ⊥ ∥ ≤ c 4 , with max (cid:8) c 3 , c
∥
X ∥ − 1 (cid:9) ≤ 10 − 3 κ − 1 and (cid:13)(cid:13)
( A ∗ A − I )( XX ⊤ − U t
U ⊤ t ) (cid:13)(cid:13)
≤ 10 − 3 κ − 1 c 3 ∥ X ∥(where κ is the condition number defined in Section 3.1) and µ ≤ 10 − 4 κ − 1 ∥ X ∥ − 2 c 3 , then we have ∥ V X s , ⊥ V U t +1 W t +1 ∥ ≤ c 3 .
Proof : Let M t =
A ∗ A ( XX ⊤ − U t
U ⊤ t ) , so the update rule of GD implies that
[Equation]
where
[Equation]
U
[Equation]
[Equation]
and V ⊤ U t W t U t W t W ⊤ t W t +1 is invertible since V ⊤ U t W t U t W t is invertible by our assumption that V ⊤ X s U t is of full rank and rank ( U t W t )
[Equation]
= rank (cid:0) V
⊤ X s U t (cid:1)
=
s , and W ⊤ t W t +1 is invertible by Lemma D.6 and the assumptions on µ and (cid:13) (cid:13)
[Equation]
(cid:13) (cid:13) [Equation]
The key observation here is that because the (square) matrix V ⊤ U t W t U t W t W ⊤ t W t +1 is invertible, so that the column space of U t +1 W t +1 is the same as that of Z .
Following the line of proof of (St ¨ oger & Soltanolkotabi, 2021, Lemma 9.3) (for completeness, we provide details in Lemma D.7), we deduce that
[Equation]
[Equation]
where B =
( I + µ M t )
( I + P ) − I and ∥ D ∥ ≤
100 ∥ B ∥ 2 .
By assumption we have
[Equation]
so that
[Equation]
where we use Lemma D.6 to bound (cid:13)(cid:13)(cid:13) W ⊤ t, ⊥ W t +1 (cid:13)(cid:13)(cid:13) .
Let B 1 = µ ( XX ⊤ − U t
U ⊤ t ) and R 1 =
V ⊤ X s , ⊥ (cid:0) I + B 1 − V U t W t V ⊤ U t W t B 1 (cid:1)
V U t W t , then we have
[Equation]
[Equation]
By Weyl’s inequality (cf.
Lemma A.2) and our assumption on c 3 ,
[Equation]
So we have
[Equation]
It thus follows from (27) that
[Equation]
+ 0 . 02 µκ − 1 c 3 ∥ X ∥ 2 + 10 3 µ 2 ∥ X ∥ 4 . Since (cid:13)(cid:13)(cid:13) V ⊤ X s , ⊥ V U t W t (cid:13)(cid:13)(cid:13) ≤ c 3 , it follows from our assumption on c 3 , c 4 and µ that (cid:13)(cid:13)(cid:13) V ⊤
X ⊥ s V
U t +1 W t +1 (cid:13)(cid:13)(cid:13) ≤ c 3 as well, which concludes the proof.
□
C. I NDUCTION
Let
[Equation]
In this section, we show that when T 0 ≤ t < (cid:101) T sα , the parallel component grows exponentially faster than the orthogonal component.
We prove this via induction and the base case is already shown in
Lemma C.3.
Lemma C.7 Let max { c 3 , c 4 ∥ X ∥ − 1 } ≤ 10 − 3 κ − 1 , δ ≤ 10 − 4 κ − 1 r − 12 ∗ c 3 and µ ≤ 10 − 4 κ − 1 ∥ X ∥ − 2 .
Then the following holds for all T 0 ≤ t < (cid:101) T α,s as long as α ≤ poly( r ) − 1 :
[Equation]
t
[Equation]
(cid:1) ≥
[Equation]
min
[Equation]
V ⊤ X s U t [Equation] W [Equation]
(cid:1) ≥ (cid:0)
[Equation]
Proof : The base case t = T 0 is already proved in (12).
Now suppose that the lemma holds for t , we now show that it holds for t
+ 1 as well.
To begin with
[Equation]
we bound the term ∥ ∆ t ∥ as follows:
[Equation]
By induction hypothesis
[Equation]
it’s easy to see that
[Equation]
where
[Equation]
Since we must have σ min (cid:0) V ⊤ X s U
t (cid:1) ≤ 0 .
3 τ = O (1) by definition of (cid:101) T α,s , it follows that ∥ U t W t, ⊥ ∥ ≤ poly( r ) α 14 κ , so for sufficiently small α ≤ poly( r ) − 1 , ∥ ∆ t ∥ ≤ 11 δ √ r ∥ X ∥ 2 holds.
The above inequality combined with our assumption on δ implies that the conditions on ∥ ∆ t ∥ in
Lemmas C.4 to C.6 hold.
We now show that (30a) to (30d) hold for t + 1 , which completes the induction step.
First, since t < (cid:101) T sα , we have σ min (cid:0) V ⊤ X s U
t
+1 (cid:1) ≤ τ .
Moreover, the induction hypothesis implies that
[Equation] 1 (cid:13)(cid:13)(cid:13) ≤ c 3 and that V
⊤ X s U α,t is of full rank.
Thus the conditions of Corollary C. are all satisfied, and we deduce that (30a) holds.
Second, the assumptions on c 3 , c 4 and δ , combined with Lemma C.5, immediately implies
[Equation]
As a result, similar to (31) we observe that
[Equation]
Since σ min (cid:0) V ⊤ X s U t +1 (cid:1) ≤ ∥ X ∥ , when α < poly( r ) − 1 we must have that ∥ U t +1 W t +1 , ⊥ ∥ ≤ c 4 .
Finally, Lemma C.6 implies that (30c) is true, and (30d) follows from our application of Lemma C.4.
This concludes the proof.
□
C. T HE REFINEMENT PHASE AND CONCLUDING THE PROOF OF L EMMA B.
We have shown that the parallel component σ min (cid:0)
V ⊤ X s U t +1 (cid:1) grows exponentially faster than the orthogonal component ∥
U t W t, ⊥ ∥ .
In this section, we characterize the GD dynamics after (cid:101) T sα .
We begin with the following lemma, which is straightforward from the proof of Lemma C.7.
Lemma C.8 The following inequality holds when α ≤ poly( r ) − 1 is sufficiently small:
[Equation]
The following lemma states that in a certain time period after (cid:101) T sα , the parallel and orthogonal components still behave similarly to the second (parallel improvement) phase.
Lemma C.
There exists (cid:101) t sα = Θ (cid:0) log 1 α (cid:1) when α → 0 (here we omit the dependence of (cid:101) T sα on α for simplicity) such that when 0 ≤ t − (cid:101) T sα ≤ ˜ t sα , we have [Equation] 32b [Equation]
32c
[Equation]
Proof : We choose (cid:101) t sα = min (cid:8) t ≥ 0 :
∥ U t +1 W t +1 , ⊥ ∥ 2 ≤ c 5 (cid:9) where
[Equation]
We prove (32) by induction.
The proof follows the idea of Lemma C.7, except that we need to bound ∥ ∆ t ∥ in each induction step.
Concretely, suppose that (32) holds at time t , then
[Equation]
where we used the definition of c 5 in the last step.
As a result, we can apply the conclusion of
Lemmas C.4 to C.6 which implies that (32) holds for t
+ 1 .
Finally, combining Lemma C.8 and (32b) yields (cid:101) T sα = ω (1) .
□
We now present the main result of this section:
Lemma C.10 Suppose that 0 ≤ t − (cid:101) T sα ≤ (cid:101) t sα , (cid:13)(cid:13)(cid:13) V ⊤ X s , ⊥ V U t W t (cid:13)(cid:13)(cid:13) ≤ c 3 and the conditions on c 3 , c 4 , δ and µ in Lemma C.7 hold, then we have
[Equation]
where we recall that τ = min 1 ≤ s ≤ ˆ r ∧ r ∗ σ 2 s − σ 2 s +1 > 0 .
Proof : The update of GD implies that
[Equation]
where
[Equation]
and
[Equation]
Note that we would like to bound (cid:13)(cid:13)
V ⊤ X s (cid:0) XX ⊤ − U t
+1 U ⊤ t
+1 (cid:1)(cid:13)(cid:13) .
We deal with the above three terms separately.
For the first term, we have
[Equation]
35c
[Equation]
where in (35a) we use (cid:13)(cid:13) I − µ U t U ⊤ t (cid:13)(cid:13) ≤ 1 , (35b) follows from σ min (cid:0)
[Equation]
min
[Equation]
and (cid:13)(cid:13)
[Equation]
⊤
[Equation]
U
[Equation]
U
[Equation]
≤
[Equation]
U [Equation] and lastly (35c) is obtained from
[Equation]
For the second and the third terms, we have
[Equation]
36) where we use the estimate in (34).
Combining (35) and (36) yields
[Equation]
□ To apply the result of Lemma C.10, we need to verify that ∥ V X s , ⊥ V U t W t ∥ ≤ c 3 still holds when t ≥ (cid:101) T s α .
In fact, this is true as long as t − (cid:101) T s α ≤ O (cid:0) log 1 α (cid:1)
[Equation]
We are now ready to present our first main result, which states that with small initialization, GD would visit the O ( δ ) -neighbourhood of the rank- s minimizer of the full observation loss i.e. X s X ⊤ s .
Theorem C.
When α < poly( r ∗ ) − 1 and δ = 10 − 4 κ − 1 c 3 with c 3 < 10 − 3 κ − 1 r − 12 ∗ , there exists a time t = ˆ T sα ∈ Z + such that
[Equation]
Proof : First, observe that for all t ≥ 0 ,
[Equation]
⊥ V U t W t ∥ 2 + √ d ∥
U t W t, ⊥ ∥ 2 .
We set δ = 10 − 3 ∥ X ∥ − 2 τc 3 and
[Equation]
, (38) then for small α we have ˆ T sα ≤ (cid:101) T sα + (cid:101) T sα (defined in Lemma C.9).
Hence for (cid:101) T sα ≤ t < ˆ T sα we always have ∥ V X s , ⊥ V U t W t ∥ ≤ c 3 .
By Lemma C.10 and the choice of c 3 and δ , we have for (cid:101) T sα ≤ t < T s that
[Equation]
which implies that
[Equation]
Meanwhile, by Lemma C.9 we have ∥ U t W t, ⊥ ∥ ≤ c 5 and (cid:13)(cid:13)(cid:13) V ⊤ X s , ⊥ V U t W t (cid:13)(cid:13)(cid:13) ≤ c 3 at t =
ˆ T sα .
Plugging into (37) yields
[Equation]
T sα U ⊤ ˆ T
[Equation]
(cid:13)(cid:13)(cid:13) F ≤ 400 τ −
[Equation]
√
[Equation]
c [Equation] √ d. By definition of c 3 and c 5 we deduce that (cid:13)(cid:13)(cid:13)
X s X ⊤ s − U ˆ T
sα U ⊤ ˆ T sα (cid:13)(cid:13)(cid:13) F ≤ 10 3 τ − 2 ∥
X ∥ √ r ∗ c 3 , as desired.
□
Corollary C.4 There exists a constant C 1 = C 1 ( κ, r ∗ ) such that
[Equation]
Proof : The case of (cid:101) T sα directly follows from Lemma C.8.
For t > (cid:101) T sα , we know from Lemma C. that
[Equation] By (38), the second term is a constant independent of α , so the conclusion follows. □ D A UXILIARY RESULTS FOR PROVING L EMMA B.
This section contains a collection of auxiliary results that are used in the previous section.
D. T HE SPECTRAL PHASE
In the section, we provide auxiliary results for the analysis in the spectral phase.
Recall that N t =
( I + µ M ) t and U t
= (cid:101)
U t + E t
=
N t U 0 + E t
and U
= α U .
Also recall that M = (cid:80) rank( M ) i
=
ˆ λ
i ˆ v
i ˆ v ⊤ i ; we additionally define M s = (cid:80) min { s, rank( M ) } i
=
ˆ λ
i ˆ v
i ˆ v ⊤ i .
Similarly, let L t be the span of the top- s left singular vectors of U t .
The following lemma shows that power iteration would result in large eigengap of U t .
Lemma D.1 Let ρ = σ min (cid:0) V ⊤ M s U (cid:1) > 0 , then the following three inequalities hold, given that the denominator of the third is positive.
[Equation]
Proof : By Weyl’s inequality we have
[Equation]
Thus (39b) holds.
Similarly,
[Equation]
Finally, note that we can write
[Equation]
so that the subspace spanned by the left singular vectors of α (1 + µ M s ) t U coincides with the column span of V M s .
Since L t is the span of top- s left singular vectors of U t , we apply Wedin’s sin theorem (Wedin, 1972) and deduce (39c).
□
The next lemma relates the quantities studied in Lemma D.1 with those that are needed in the induction.
The proof is the same as (St¨oger & Soltanolkotabi, 2021, Lemma 8.4), so we omit it here.
Lemma D.2 Suppose that (cid:13)(cid:13)(cid:13) V ⊤ X s , ⊥ V L t (cid:13)(cid:13)(cid:13) ≤ 0 .
for some t ≥ 1 .
Then it holds that
[Equation]
Combining the above two lemmas, we directly obtain the following corollary:
Corollary D.1 Suppose that ασ s ( N t ) > 10 ( ασ s +1 ( N t ) + ∥ E t ∥ ) , then we have that
[Equation]
D. T HE PARALLEL IMPROVEMENT PHASE
In the section, we provide auxiliary results for the analysis in the parallel improvement phase.
Lemma D.3 (St¨oger & Soltanolkotabi, 2021, Lemma 9.4)
For sufficiently small µ and δ , suppose that ∥ U t ∥ ≤ 3 ∥ X ∥ , then we also have ∥ U t +1 ∥ ≤ 3 ∥ X ∥ .
Lemma D.4 Under the assumptions in Lemma C.5, we have
[Equation]
Proof : The proof of this lemma is essentially the same as (St ¨ oger & Soltanolkotabi, 2021, Lemma
B.1), and we omit it here.
□
Lemma D.
Under the assumptions in Lemma C.6, we have
[Equation]
Proof : We have
[Equation]
where the last step follows from
[Equation]
The conclusion follows.
□
Lemma D.6 Under the assumptions in Lemma C.6, we have
[Equation]
Proof : The proof roughly follows (St¨oger & Soltanolkotabi, 2021, Lemma B.3), but we include it here for completeness.
Since V ⊤ X s U t +1 =
V t +1 Σ t +1 W t +1 and V t +1 Σ t
+1 ∈
R s × s is invertible, we have
[Equation]
Since
[Equation]
where (42a) follows from V ⊤ X s XX ⊤ U t W t, ⊥
= Σ s V ⊤ X s U t W t, ⊥ = 0 , and in (42b) and (42c) we use V ⊤
[Equation]
= 0 .
[-+1 V X s (cid:1) − 12 V ⊤ X s U t +
-]
For K 1 , note that
[Table]
so that
[Table]
Plugging into (42), we deduce that
[Table]
[Equation]
[Table]
[-∥ 2 + c 4 (cid:1) c 
 ∥ X ∥ + µ (cid:13)(cid:13) ( A ∗ A − I )( XX ⊤ − U t U ⊤ t ) (cid:13)(cid:13) .-][Equation]
where in the last step we use Lemma D.5 and the induction hypothesis which implies that σ min ( U t W t ) ≥
∥
U t W t, ⊥ ∥ .
□
Lemma D.
The matrix ˆ Z defined in the proof of Lemma C.6 satisfies the following:
[Equation]
where ∥ D ∥ ≤ 30 ∥ B ∥ 2 .
Proof : By definition of ˆ Z we have
[Equation]
V U t W t (cid:1) −= ( I + B ) V U
t W t (cid:2) V ⊤ U t W t (cid:0) I + B ⊤ + B + B ⊤ B (cid:1)
V U t W t (cid:3) −= ( I + B )
V U
t W t   I + V ⊤ U t W t (cid:0) B ⊤ + B + B ⊤ B (cid:1) V U t W t (cid:124) (cid:123)(cid:122) (cid:125) =:∆ 
 −.
It follows from (28) and our assumptions on c 3 and c 4 that
[Equation]
(note that this step is independent and does not rely on earlier derivations in the proof of Lemma C.6), so by Taylor’s formula [Equation] we have
[Equation]
Hence,
[Equation]
as desired
[Equation]
□
E P ROOF OF RESULTS IN S ECTION
Theorem 1.1 states that GD approximately learns the rank- s constrained minimizer of the matrix sensing loss.
However, Theorem C.1 only implies that GD would get into an O ( δ ) neighborhood of
X s X ⊤ s .
As a result, a more fine-grained analysis is needed.
Note that it is not even clear whether the rank- s minimizer of is unique.
If it is not, then we may naturally ask which minimizer it converges to.
In this section, we study the landscape of under-parameterized matrix sensing problem
[Equation]
and establish local convergence of gradient descent.
Our key result in this section is Lemma E.6, which states a local PL condition for the matrix sensing loss.
Most existing results only study the landscape of (1) in the exact- and over-parameterized case.
Zhu et al.
(2021) studies the landscape of under-parameterized matrix factorization problem, but they only prove a strict saddle property without asymptotic convergence rate of GD.
When the measurement satisfies the RIP condition, we can expect that the landscape of f looks similar to that of the (under-parameterized) matrix factorization loss:
[Equation]
for some s < r .
Recall that XX ⊤ =
(cid:80) r ∗ i =1 σ
i
v
i v ⊤ i .
The critical points of F s ( U ) is characterized by the following lemma:
Lemma E.1 U ∈ R
d × s is a critical point of F s ( U ) if and only if there exists an orthogonal matrix
R ∈ R s × s , such that all columns of UR are in { σ i v i : 1 ≤ i ≤ r ∗ } .
Proof : Assume WLOG that XX ⊤ = diag( σ 21 , σ 22 , · · · , σ 2 r , 0 , · · · , 0)
=: Σ .
Let U be a critical point of F s , then we have that (cid:0) UU ⊤ − XX ⊤ (cid:1) U = 0 .
Let W = UU ⊤ , then (
Σ − W ) W = 0 .
Since W is symmetric, so is W 2 , and we obtain that Σ W is also symmetric.
It’s then easy to see that
that if Σ = diag ( λ 1 I m 1 , · · · , λ t I m t ) with λ 1 > λ 2 > · · · > λ t ≥ 0 , then W is also in block-diagonal form: W = diag ( W 1 , W 2 , · · · , W t )
where
W i ∈
R m i × m i .
For each 1 ≤ i ≤ t , we then have the equation ( λ i
I m i − W i )
W i = 0 .
Hence, there exists an orthogonal matrix R i
such that R ⊤ i
W i
R i is a diagonal matrix where the diagonal entries are either 0 or √ λ i =
σ i .
Let
R = diag ( R 1 , R 2 , · · · , R t ) , then R ⊤ W R is diagonal and its nonzero diagonal entries form an s -subset of the multi-set { σ i : 1 ≤ i ≤ r ∗ } .
The conclusion follows.
□
In the case of s = 1 , the global minimizers of F s are ± σ 1 v 1 , and we can show that F s is locally strongly convex around these minimizers.
Therefore, we can deduce that f is locally stronglyconvex as well.
Since our main focus is on s > 1 , we put these details in Appendix G. When s > 1 ,
F s ( U ) is not locally strongly-convex due to rotational invariance: if U is a global minima, then so is UR for any orthogonal matrix R ∈ R s × s .
Instead, we will establish a local PL property w.r.t the procrustes distance:
Definition E.1 Let U 1 , U 2 ∈ R d × s , we define
[Equation]
The following characterization of the optimal R in Definition 5.1 is known in the literature (see e.g.
(Tu et al., 2016, Section 5.2.1))
but we provide a proof of it for completeness.
Lemma E.2 Let R be the orthogonal matrix which minimizes ∥ U 1 − U 2 R ∥ F , then U ⊤ 1 U 2 R is positive semi-definite.
Proof : Observe that
[Equation]
Let U ⊤ 2 U 1 =
A Σ B ⊤ be its SVD, then
[Equation]
, where the final step is due to orthogonality of B ⊤ R ⊤ A ∈ R s × s , and equality holds if and only if
B ⊤ R ⊤ A
= I i.e. R = AB ⊤ .
Thus U ⊤ 1 U 2 R =
B Σ B ⊤ is positive semi-definite.
□
The following lemma states that the minimizer of matrix sensing loss is also near-optimal for the matrix factorization loss.
The main idea for proving this result is to utilize
Lemma E.3 Let Z ∗ s be a best rank- s solution as defined in (1) , then we have
[Equation]
Proof : Since U ∗ s is the global minimizer of f , by the RIP property we have
[Equation]
as desired.
□
The next lemma show that F s satisfies a local PL property:
Lemma E.
Given U ∈ R d × s
and let R be an orthogonal matrix such that ∥ U − X s R ∥ F = dist( U , X s ) .
Suppose that dist( U , X s ) ≤ 0 .
1 ∥ X ∥ − 1 τ (where we recall that τ = min s ∈ [ r ∗ ] (cid:0) σ 2 s − σ 2 s +1 (cid:1) is the eigengap of XX ⊤ ), then we have ⟨∇ F s ( U ) , U − X s R ⟩ ≥ 0 .
1 τ dist 2 ( U , X s ) .
Proof : Assume WLOG that R
=
I , then U ⊤ X s is positive semi-definite.
Let H =
U − X s , then
[Equation]
so that
[Equation]
where in (43a) we use tr (cid:0) ( H ⊤ X s ) 2 (cid:1) ≥ 0 (since H ⊤ X s is symmetric as noticed in the beginning of the proof)
[Equation]
and (43b) is because of
[Equation]
□
Corollary E.1 Under the conditions of Lemma E.4, we have ∥∇ F s ( U ) ∥
F ≥ 0 .
1 τ dist( U , X s ) .
The following lemma shows that the rank- s global minima of matrix sensing must lie in an O ( δ ) - neighbourhood of the minima of F s .
Lemma E.5 Suppose that Assumption 3.1 holds.
Let U ∗ s be a global minimizer of f s , then we have
[Equation]
where we recall that κ = τ − 1 ∥ X ∥ is the condition number of XX ⊤ .
Proof : Define
[Equation]
First we can show that U ∗ s ∈ S .
The main idea is to apply Lemma A.5.
Indeed, it’s easy to see that
[Equation]
Let U ∈ ∂S i.e. dist 2 ( U , X s )
= 0 .
1 ∥ X ∥ − 1 τ .
Assume WLOG that dist( U , X s ) =
∥
U − X s ∥ F , then by Lemma E.
we have
[Equation]
Recall that all the stationary points of F s are characterized in Lemma E.1, so that for all U / ∈ S with ∇ F s ( U ) =
0 , we have
[Equation]
On the other hand, we know from Lemma E.
that
[Equation]
[Equation]
so Lemma A.5 implies that U ∗ s ∈ S .
Inside S , we can apply the local PL property that we previously derived.
Indeed, note that
[Equation]
Hence we have that
[Equation]
□
Corollary E.2 Suppose that Assumption 3.1 holds, then we have σ min (cid:16) ( U ∗ s )
⊤ U ∗ s (cid:17) ≥ σ 2 s − 80 δκ ∥
X ∥∥ X ∥ F .
Proof : We assume WLOG that ∥
U ∗ s − X s ∥ F = dist( U ∗ s , X s )
i.e. R
=
I in Definition 5.1.
By
Lemma E.5, we have that
[Equation]
≤ 80 δκ ∥
X ∥∥ X ∥ F .
□
Lemma E.6 Suppose that Assumption 3.1 holds.
Given U ∈ R d × s , let U ∗ s ∈ R d × s be a minimizer of f s , and U ∗ s R be the rank- s minimizer which is closest to U ( R ∈ R s × s is orthogonal).
When ∥ U − U ∗ s R ∥ ≤ 10 − 2 κ − 1 ∥ X ∥ , we have ⟨∇ f [Equation] U [Equation]
U [Equation] U [Equation]
[Equation] dist [Equation]
Proof : We assume WLOG that R
=
I , then U ⊤ U ∗ s is positive semi-definite.
Let H =
U − U ∗ s , then
[Equation]
where we use the first-order optimality condition
[Equation]
Since ∥ U ∗ s ∥ ≤ 2 ∥
X
∥ by Lemma E.5, we may thus deduce that
[Equation]
[Table]
By
Corollary
E. we have
[Equation]
and
[Equation]
[Equation]
X ∥ 2 F , so that
[Equation]
When δ ≤ 10 − 3 r − 12 ∗ κ − 2 and ∥ H ∥ ≤ 10 − 2 τ ∥ X ∥ − 1 , the above implies that ⟨∇ f s ( U ) , U − U ∗ s ⟩ ≥ 0 .
5 τ ∥ H ∥ 2 F , as desired.
□
F P ROOF OF T HEOREM 4.
With the landscape results in hand, we are now ready to characterize the saddle-to-saddle dynamics of GD.
We first note the following proposition, with is straightforward from Lemma C.9 and
Theorem C.1.
In the following we use U α,t to denote the t -th iteration of GD when initialized at
U
= α U .
Proposition F.1 There exists matrices ¯ U α,t , − ˆ T sα ≤ t ≤ 0 with rank ≤ s such that
[Equation]
where ˆ T sα is defined in Theorem C.1 (we omit the dependence on α there) and moreover
[Equation]
where Z s =
U ∗ s ( U ∗ s )
⊤ is the rank- s minimizer of the matrix sensing loss i.e.
Z s = arg
[Equation]
is positive semi-definite and rank
[Equation]
Remark F.
We omit the dependence on α for simplicity of notations, when it is clear from context.
Proof : It follows from Lemma C.9 that max 1 ≤ t ≤ ˆ T sα ∥ U t W t, ⊥ ∥
= O (cid:16) α 14 κ (cid:17) .
We choose ¯ U [-
-]t =
U ˆ T sα + t W ˆ T sα + t W ⊤ ˆ T sα + t , then rank (cid:0) ¯ U t (cid:1) ≤ s and moreover by Theorem C.
we have (cid:13)(cid:13)
X s X ⊤ s − ¯ U
¯ U ⊤
(cid:13)(cid:13) F ≤ 2 κ 2 ∥ X ∥ 2 √ r ∗ δ .
On the other hand, similar to Corollary E.2 we have that (cid:13)(cid:13)
Z s − X s X ⊤ s (cid:13)(cid:13)
F ≤ 80 δκ ∥ X ∥ 2 F .
Thus (cid:13)(cid:13) ¯ U
¯ U ⊤ 0 − Z s (cid:13)(cid:13) ≤ 100 κ 2 ∥ X ∥ 2 r ∗ δ as desired.
□
Let ˆ U α,t =
U α,t W α,t ∈ R d × s , then it satisfies ˆ U {+
 ˆ+} U ⊤
= ¯ U
¯ U ⊤ 0 .
The following corollary shows that ˆ U α, 0 is close to U ∗ s in terms of the procrutes distance.
Corollary F.
We have dist( ˆ U 0 , U ∗ s ) ≤ 80 κ 3 r 12 ∗ ∥ X ∥ δ .
Proof : We know from Lemma E.5 that dist( U ∗ s , X s ) ≤ 40 δκ ∥ X ∥ F , so it remains to bound dist( ˆ U 0 , X s ) .
The proof idea is the same as that of Lemma E.5, so we only provide a proof sketch here.
It has been shown in the proof of Proposition F.1 that
[Equation]
Note that F s is the matrix factorization loss with X s X ⊤ s as the ground-truth, so the local PL property (cf.
Lemma E.4) still holds here, and by the same reason as (44), we deduce that dist( ˆ U 0 , X s ) ≤ 0 .
1 ∥ X ∥ − 1 τ i.e. ˆ U 0 is in the local PL region around X s .
Finally, it follows from the PL property that
[Equation]
The conclusion follows.
□
Recall that matrix sensing loss satisfies a local PL property (cf.
Lemma E.6).
As a result, when δ is small, we can show that GD initialized at ˆ U 0 converges linearly to the ground-truth.
Lemma F.1 Let ˆ U t be the t -th iteration of GD initialized at ˆ U 0 .
Suppose that δ ≤ 10 − 2 r − 12 ∗ κ −and µ ≤ 10 − 3 ∥ X ∥ − 2 , then we have that
[Equation]
Proof : We know from Corollary F.1 that
[Equation]
We will prove the following result, which immediately implies Lemma F.1: suppose that dist
[Equation]
ˆ U
[Equation] , U ∗ [Equation] ) ≤ dist( ˆ U 0 , U
[Equation]
) , then
[Equation]
Let R be the orthogonal matrix satisfying ∥ U t − U ∗ s R ∥ F = dist
( U t , U ∗ s ) .
Assume WLOG that
R =
I .
We first bound the gradient ∇ f ( ˆ U t ) as follows:
[Equation]
(47) where we use (cid:13)(cid:13)(cid:13) ˆ U t (cid:13)(cid:13)(cid:13) ≤ ∥ X ∥ + 40 κ 3 r 12 ∗ ∥
X ∥ δ ≤ 2 ∥ X ∥ and the RIP property.
It follows that
[Equation]
(48b) where (48a) follows from the definition of dist [Equation] and (48b) is due to Lemma E.6 and (47).
Finally, (46) follows from the condition µ ≤ 10 − 3 κ − 2 .
□
We are now ready to complete the proof of Theorem 4.1.
Theorem F.1 (Restatement of Theorem 4.1) Under Assumptions 3.1 and 3.2, consider GD (3) with learning rate µ ≤ 1 10 3 ∥ Z ∗ ∥ and initialization U α, 0 =
α ¯ U for solving the matrix sensing problem (1) .
There exists a universal constant c > 0 , a constant C (depending on ˆ r and κ ) and a sequence of time points T 1 α < T 2 α < · · · < T ˆ r ∧ r ∗ α such that for all 1 ≤ s ≤ ˆ r ∧ r ∗ , the following holds when α =
O (cid:16)
[Equation]
[Equation] where we recall that Z ∗ s is the best rank- s solution defined in Definition 1.1. Moreover, GD [-initialized at U 
-]follows an incremental learning procedure: for all 1 ≤ s ≤ ˆ r ∧ r ∗ , we have lim
[Equation] max
[Equation] sα σ s +1 ( U α,t ) = 0 , where σ i ( A ) denotes the ( s + 1) -th largest singular value of a matrix A . Proof : Recall that (cid:13) (cid:13) (cid:13)
U ˆ T sα − ¯[-U 
-]
U 0 (cid:13) (cid:13) (cid:13) F = o (1) ( α → 0 ) where ˆ T s α is defined in Proposition F.1; we omit the dependence on α to simplify notations.
We also note that by the update of GD, we have ¯ U t ¯ U ⊤ t
= ˆ U t ˆ U ⊤ t for all t ≥ 0 .
By Lemma F.1, we have that dist 2 ( ˆ U t , U ∗ s ) ≤ (1 − 0 .
05 τµ ) t dist 2 ( ˆ U 0 , U ∗ s ) and, in particular, (cid:13)(cid:13)(cid:13)
ˆ U t (cid:13)(cid:13)(cid:13) ≤ 2 ∥
X ∥ for all t .
Thus (cid:13)(cid:13)
¯ U t (cid:13)(cid:13) ≤ 2 ∥ X ∥ as well.
Moreover, recall that ∥ U t ∥ ≤ 3 ∥ X ∥ for all
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
Since when α → 0 , (cid:13)(cid:13)(cid:13) U ˆ T sα − ¯ U 0 (cid:13)(cid:13)(cid:13) F = O ( α 14 κ ) , it’s easy to see that there exists a time t =
t
sα so that we have max − ˆ T sα ≤
[Table]
≤
[Table]
sα (cid:13)(cid:13)
[Table]
U ˆ T sα +
[Table]
− ¯ U t (cid:13)(cid:13) (cid:13) F = O (cid:16) α
[Table] [Table]
and (cid:13)(cid:13) (cid:13) U ˆ
[Table]
sα +
[Table]
U ⊤ ˆ
[Table]
sα +
[Table]
− Z s (cid:13)(cid:13) (cid:13) F = O (cid:16) α c 1 κ 2 (cid:17) as well, where c 1 is a universal constant.
Let T sα
= ˆ T sα + t
sα , then (cid:13) (cid:13)(cid:13)
U T sα U ⊤ T sα − Z s (cid:13)(cid:13)(cid:13) F = o (1) holds. Recall that rank ( U t ) ≤ s , so that max 0 ≤ t ≤ T sα σ s +1 ( U t )
= o (1) .
Finally, for all 0 ≤ s < ˆ r ∧ r ∗ , we need to show that T sα < T s +1 α . Indeed, by Corollary E.2 and the Assumption 3.1 we have {+σ 2 s +1 (cid:0) U T sα (cid:1) ≥ σ s +
+}
( Z s +1 )
− o (1) {+≥ 0 . 5 σ 2 s +1 , so that T s +
 α > T sα ,+} as desired.
□
G T HE LANDSCAPE OF MATRIX SENSING WITH RANK - 1 PARAMETERIZATION
In this section, we establish a local strong-convexity result Lemma G.2 for rank- 1 parameterized matrix sensing.
This result is stronger than the PL-condition we established for general ranks, though the latter is sufficient for our analysis.
Lemma G.1 Define the full-observation loss with rank- 1 parameterization
[Equation]
Then the global minima of g 1 are u ∗ =
σ 1 v 1 and − u ∗ .
Moreover, suppose that g ( u ) − g ( u ∗ ) ≤ 0 .
5 τ 1 where τ 1 = σ 21 − σ 22 is the eigengap, then we must have [Equation]
Proof : We can assume WLOG that XX T = diag (cid:0) σ 21 , · · · , σ 2 r ∗ , 0 , · · · , 0 (cid:1) .
Then
[Equation]
where equality holds if and only if u 2 = · · · = u d = 0 and ∥ u ∥ 2 = σ 21 i.e. u = ±
σ 1 e 1 .
Moreover, suppose that g 1 ( u ) − g 1 ( u ∗ ) ≤ 0 .
[Equation] 1 , it follows from (50b) that τ 1 (cid:80) di
=2 u 2 i ≤ 2( g 1 ( u ) − g 1 ( u ∗ )) which implies that (cid:80) di =
u 2 i ≤ 2 τ − 1 1 ( g 1 ( u ) − g 1 ( u ∗ )) .
Also (50c) yields (cid:12)(cid:12) ∥
u ∥ 2 − σ 21 (cid:12)(cid:12) ≤ 4 (cid:112) g 1 ( u ) − g 1 ( u ∗ ) .
Assume WLOG that u 1 > 0 , then we have
[Equation]
□
Lemma G.2 Let
[Equation]
Suppose that δ ≤ 10 − 3 ∥ X ∥ − 2 τ 1 , then there exists constants a 1 and ι , such that f 1 is locally ιstrongly convex in B
= B ( σ 1 v 1 , a 1 ) ⊂ R d .
Furthermore, there is a unique global minima of finside B 1 .
Proof : Recall that we defined the full observation loss g 1 ( u ) = 14 (cid:13)(cid:13) uu T − XX T (cid:13)(cid:13) 2 F .
Let h 1 = f 1 − g 1 , then
[Equation]
≤ δ (cid:0) 2 ∥ u ∥
[Equation]
+ ∥ X ∥ 2 (cid:1) . When ∥ u − σ 1 v 1 ∥ 2 ≤ 0 . 1 min (cid:8) σ 21 , τ 1 (cid:9) (recall τ 1 = σ 21 − σ 22 ),
[Equation]
4 τ 1 .
Hence we have
[Equation]
= 0 .
2 τ 1 .
Let u ∗ be a global minima of f 1 , then we must have ∥ u ∗ ∥ ≤ 2 ∥ X ∥ (otherwise f 1 ( u ) > f 1 (0) ).
We can thus deduce that
[Equation]
It follows from Lemma G.1 and our assumption on δ that min (cid:110) ∥ u [Equation]
12 a 21 . Moreover, by strong convexity, there exists only one global minima in B 1 , which concludes the proof.
□