L ARGE L EARNING R ATE T AMES H OMOGENEITY : C ONVERGENCE AND B ALANCING E FFECT
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent empirical advances show that training deep models with large learning rate often improves generalization performance.
However, theoretical justiﬁcations on the beneﬁts of large learning rate are highly limited, due to challenges in analysis.
In this paper, we consider using Gradient Descent (GD) with a large learning rate on a homogeneous matrix factorization problem, i e., min X,Y (cid:107) A − XY (cid:62) (cid:107) 2 F .
We prove a convergence theory for constant large learning rates well beyond 2 /L , where L is the largest eigenvalue of Hessian at the initialization.
Moreover, we rigorously establish an implicit bias of GD induced by such a large learning rate, termed ‘balancing’, meaning that magnitudes of X and Y at the limit of GD iterations will be close even if their initialization is signiﬁcantly unbalanced.
Numerical experiments are provided to support our theory
I NTRODUCTION
Training machine learning models such as deep neural networks involves optimizing highly nonconvexfunctions.
Empirical results indicate an intimate connection between training algorithms and theperformance of trained models (Le et al., 2011; Bottou et al., 2018; Zhang et al., 2021;
Soydaner,2020; Zhou et al., 2020).
Especially for widely used ﬁrst-order training algorithms
(e.g., GD and
SGD), the learning rate is of essential importance and has received extensive focus from researchers(Smith, 2017; Jastrzebski et al., 2017; Smith, 2018; Gotmare et al., 2018; Liu et al., 2019; Li & Arora,2019).
A recent perspective is that large learning rates often lead to improved testing performancecompared to the counterpart trained with small learning rates (Smith & Topin, 2019;
Yue et al., 2020).
Towards explaining the better performance, a common belief is that large learning rates encouragethe algorithm to search for ﬂat minima, which often generalize better and are more robust than sharpones (Seong et al., 2018; Lewkowycz et al., 2020).
Despite abundant empirical observations, theoretical understandings of the beneﬁts of large learningrate are still limited for non-convex functions, partly due to challenges in analysis.
For example, theconvergence (of GD or SGD) under large learning rate is not guaranteed.
Even for globally smoothfunctions, very few general results exist if the learning rate exceeds certain threshold (Kong & Tao,2020).
Besides, popular regularity assumptions such as global smoothness for simpliﬁed analyses areoften absent in homogeneous models, including commonly used ReLU neural networks.
This paper theoretically studies the beneﬁts of large learning rate in a matrix factorization problem
[Equation]
where A ∈ R n × n , X, Y ∈ R n × d .
We consider Gradient Descent (GD) for solving (1): at the k -th iteration, we have
[Equation]
where h is the learning rate.
Despite its simple formula, problem (1) serves as an important foundationof a variety of problems, including matrix sensing (Chen & Wainwright, 2015; Bhojanapalli et al.
,2016; Tu et al., 2016), matrix completion (Keshavan et al., 2010; Hardt, 2014), and linear neuralnetworks (Ji & Telgarsky, 2018; Gunasekar et al., 2018).
Problem (1) possesses several intriguing properties.
Firstly, the objective function is non-convex, andcritical points are either global minima or saddles (see e.g., Baldi & Hornik (1989);
Li et al.
(2019b);
Valavi et al.
(2020a)).
Secondly, problem (1) is homogeneous in X and Y , meaning that rescaling
X, Y to aX, a − 1 Y for any a (cid:54) = 0 will not change the objective’s value.
This property is sharedby commonly used ReLU neural networks.
A direct consequence of homogeneity is that globalminima of (1) are non-isolated and can be unbounded.
The curvatures at these global minima arehighly dependent on the magnitudes of X, Y .
When X, Y have comparable magnitudes, the largesteigenvalue of Hessian is small, and this corresponds to a ﬂat minimum; on the contrary, unbalanced Xand Y give a sharp minimum.
Last but not the least, the homogeneity impairs smoothness conditionsof (1), rendering the gradient being not Lipschitz continuous unless X, Y are bounded.
See a formaldiscussion in Section 2.
Existing approaches for solving (1) often uses explicit regularization (Ge et al., 2017; Tu et al., 2016;
Cabral et al., 2013; Li et al., 2019a), or inﬁnitesimal (or diminishing) learning rates for controllingthe magnitudes of X, Y (Du et al., 2018;
Ye & Du, 2021).
In this paper, we go beyond the scope ofaforementioned works, and analyze GD with a large learning rate for solving (1).
In particular, weallow the learning rate h to be as large as approximately 4 /L , where L denotes the largest eigenvalueof Hessian at GD initialization.
In connection to empirical observations, we provide positive answersto the following two questions:
Does GD with large learning rate converge at least for some cases of (1) ?
Does larger learning rate biases toward ﬂatter minima (i.e., X, Y with comparable magnitudes)?
We theoretically show the convergence of GD with large learning rate for the two situations n =1 , d ∈ N + or d = 1 , n ∈ N + with isotropic A .
We also observe a, perhaps surprising, “balancingeffect” for general matrix factorization (i.e., any d , n , and A ), meaning that when h is sufﬁcientlylarge, the difference between X and Y shrinks signiﬁcantly at the convergence of GD compared toits initial, even if the initial point is close to an unbalanced global minimum.
In fact, with a properlarge learning rate h , (cid:107) X k − Y k (cid:107) 2 F may decrease by an arbitrary factor at its limit.
The following is asimple example of our theory for n = 1 (i.e. scalar factorization), and more general results will bepresented later with a precise bound for h depending on the initial condition and A .
Theorem 1.1 (Informal version of Thm.3.1 & 3.2) .
Given scalar A and initial condition X 0 , Y 0 [-
-]∈ 62 R 1 × d chosen almost everywhere, with learning rate h (cid:46) 4 /L , GD converges to a global minimum( X, Y ) satisfying (cid:107) X (cid:107) 2 F + (cid:107) Y (cid:107) 2 F ≤ 2 h .
Consequently, its extent of balancing is quantiﬁed by(cid:107) X − Y (cid:107) 2 F ≤ 2 h − 2 A .
We remark that having a learning rate h ≈ 4 /L is far beyond the commonly analyzed regime inoptimization.
Even for globally L -smooth objective, traditional theory requires h < 2 /L for GDconvergence and h = 1 /L is optimal for convex functions (Boyd et al., 2004), not to mention thatour problem (1) is never globally L -smooth due to homogeneity.
Modiﬁed equation provides a toolfor probing intermediate learning rates (see Hairer et al.
(2006, Chapter 9) for a general review, and
Kong & Tao (2020, Appendix A) for the speciﬁc setup of GD), but the learning rate here is too largefor modiﬁed equation to work (see Appendix C).
In fact, besides blowing up, GD with large learningrate may have a zoology of limiting behaviors (see e.g., Appendix B for convergence to periodicorbits under our setup, and Kong & Tao (2020) for convergence to chaotic attractors).
Our analyses (of convergence and balancing) leverage various mathematical tools, including aproper partition of state space and its dynamical transition (speciﬁcally invented for this problem),stability theory of discrete time dynamical systems (Alligood et al., 1996), and geometric measuretheory (Federer, 2014).
The rest of the paper is organized as: Section 2 provides the background of studying (1) and discussesrelated works; Section 3 presents convergence and balancing results for scalar factorization problems;
Section 4 generalizes the theory to rank- 1 matrix approximation; Section 5 studies problem (1) witharbitrary A and its arbitrary-rank approximation; Section 6 summarizes the paper and discussesbroadly related topics and future directions.
B ACKGROUND AND R ELATED W ORK
Notations .
(cid:107) v (cid:107) is (cid:96) 2 norm of a column or row vector v .
(cid:107) M (cid:107) F is the Frobenius norm of a matrix M .
Sharp and ﬂat minima in (1) We discuss the curvatures at global minima of (1).
To ease thepresentation, consider simpliﬁed versions of (1) with either n = 1 or d = 1 .
In this case, X and
Y become vectors and we denote them as x, y , respectively.
We show the following propositioncharacterizing the spectrum of Hessian at a global minimum.
Proposition 2.1.
When n = 1 , d ∈ N + or d = 1 , n ∈ N + in (1) , the largest eigenvalue of Hessianat a global minimum ( x, y ) is (cid:107) x {+(cid:107) 
+} + (cid:107) y (cid:107) 2 and the smallest eigenvalue is 0 .
Homogenity implies global minimizers of (1) are not isolated, which is consistent with the 0 eigenvalue.
On the other hand, if the largest eigenvalue (cid:107) x (cid:107) 2 + (cid:107) y (cid:107) 2 is large (or small), then the curvatureat such a global minimum is sharp (or ﬂat), in the direction of the leading eigenvector.
Meanwhile,note this sharpness/ﬂatness is an indication of the balancedness between magnitudes of x, y at aglobal minimum.
To see this, singular value decomposition (SVD) yields that at a global minimum,( x, y ) satisﬁes (cid:13)(cid:13) xy (cid:62) (cid:13)(cid:13) 2 F = σ 2max ( A ) .
Therefore, large (cid:107) x (cid:107) 2 + (cid:107) y
(cid:107) 2 is obtained when |(cid:107) x (cid:107) − (cid:107) y (cid:107)| islarge, i.e., x and y magnitudes are unbalanced, and small (cid:107) x (cid:107) 2 + (cid:107) y (cid:107) 2 is obtained when balanced.
Large learning rate We study smoothness properties of (1) and demonstrate that our learning rateis well beyond conventional optimization theory.
We ﬁrst deﬁne the smoothness of a function.
Deﬁnition 2.2 ( L -smooth) .
A function f ∈ C 1 deﬁned on R N is L -smooth if for all u 1 , u 2 ∈ R N ,
[Equation]
If further f ∈ C 2 , then ∇ 2 f (cid:22) LI .
Moreover, if we have (2) for u 1 , u 2 ∈ X ⊆ R N , we call it locally L -smooth.
In traditional optimization (Nesterov, 2003; Polyak, 1987; Nesterov, 1983; Polyak, 1964; Beck &
Teboulle, 2009), most analyzed objective functions often satisfy (i) (some relaxed form of) convexityor strong convexity, and (ii) L -smoothness.
Choosing a step size h < 2 /L guarantees the convergenceof GD to a minimum by the existing theory (reviewed in Appendix F.3).
Our choice of learning rateh ≈ 4 /L (more precisely, 4 /L 0 ; see below) goes beyond the classical analyses.
Besides, in our problem (1), the regularity is very different.
Even simpliﬁed versions of (1), i.e., witheither n = 1 or d = 1 , suffer from (i) non-convexity and (ii) unbounded eigenvalues of Hessian,i.e., no global smoothness (see Appendix F.2 for more details).
As shown in Du et al.
; Ye &
Du (2021); Ma et al.
(2021), decaying or inﬁnitesimal learning rate ensures that the GD trajectorystays in a locally smooth region.
However, the gap between the magnitudes of X, Y can only bemaintained in that case.
We show, however, that larger learning rate can shrink this gap.
Related work Matrix factorization problems in various forms have been extensively studied inthe literature.
The version of (1) is commonly known as the low-rank factorization, although here dcan arbitrary and we consider both d ≤ rank ( A ) and d > rank ( A ) cases.
Baldi & Hornik (1989);
Li et al.
(2019b)
; Valavi et al.
(2020b) provide landscape analysis of (1).
Ge et al.
; Tu et al.(2016) propose to penalize the Frobenius norm of (cid:13)(cid:13) X (cid:62) X − Y (cid:62) Y (cid:13)(cid:13) 2 F to mitigate the homogeneityand establish global convergence guarantees of GD for solving (1).
Cabral et al.
; Li et al.(2019a) instead penalize individual Frobenius norms of (cid:107) X (cid:107) 2 F + (cid:107) Y (cid:107) 2 F .
We remark that penalizing(cid:107) X (cid:107) 2 F + (cid:107) Y {+(cid:107) 
+} F is closely related to nuclear norm regularization, since the variational formula fornuclear norm (cid:107) Z (cid:107) 2 ∗
= min Z = XY (cid:62) (cid:107) X (cid:107) 2 F + (cid:107) Y (cid:107) 2 F .
More recently, Liu et al.
(2021) consider usinginjected noise as regularization to GD and establish a global convergence.
More speciﬁcally, byperturbing the GD iterate ( X k , Y k ) with Gaussian noise, GD will converge to a ﬂat global optimum.
On the other hand, Du et al.
; Ye & Du (2021); Ma et al.
(2021) show that even without explicitregularization, when the learning rate of GD is inﬁnitesimal, i.e., GD approximating gradient ﬂow, Xand Y maintain the gap in their magnitudes.
Such an effect is more broadly recognized as implicitbias of learning algorithms (Neyshabur et al., 2014; Gunasekar et al., 2018; Soudry et al., 2018).
Built upon this implicit bias, Du et al.
(2018) further prove that GD with diminishing learning ratesconverges to a bounded global minimum of (1), and this conclusion is recently extended to the case ofa constant small learning rate (Ye & Du, 2021).
Our work goes beyond the scopes of these milestonesand considers matrix factorization with much larger learning rates.
Additional results exist that demonstrate large learning rate can improve performance in variouslearning problems.
Most of them involve non-constant learn rates.
Speciﬁcally, Li et al. (2019c) consider a two-layer neural network setting, where using learning rate annealing (initially large,followed by small ones) can improve classiﬁcation accuracy compared to training with small learningrates.
Nakkiran (2020) shows that the observation in Li et al.
(2019c) even exists in convex problems.
Lewkowycz et al.
(2020) study constant large learning rates, and demonstrate distinct algorithmicbehaviors of large and small learning rates, as well as empirically illustrate large learning rate yieldsbetter testing performance on neural networks.
Their analysis is built upon the neural tangent kernelperspective (Jacot et al., 2018), with a focus on the kernel spectrum evolution under large learningrate.
Worth noting is, Kong & Tao (2020) also study constant large learning rates, and show that largelearning rate provides a mechanism for GD to escape local minima, alternative to noisy escapes dueto stochastic gradients.
O VERPARAMETERIZED SCALAR FACTORIZATION
In order to provide intuition before directly studying the most general problem, we begin with asimple special case, namely factorizing a scalar by two vectors.
It corresponds to (1) with n = 1 andd ∈ N + , and this overparameterized problem is written as
[Equation]
where µ is assumed without loss of generality to be a positive scalar.
Problem (3) can be viewed asunivariate regression using a linear two-layer neural network with the quadratic loss, which is studiedin Lewkowycz et al.
(2020) with atomic data distribution.
Yet our analysis in the sequel can be usedto study arbitrary univariate data distributions; see details in Section 6.
Although simpliﬁed, problem (3) is still nonconvex and exhibits the same homogeneity as (1).
Theconvergence of its large learning rate GD optimization was previously not understood, let alonebalancing.
Many results that we will obtain for this problem will remain true for more generalproblems.
We ﬁrst prove that GD converges despite of > 2 /L learning rate and for almost all initial conditions:
Theorem 3.1 (Convergence) .
Given ( x (cid:62) 0 , y (cid:62) 0 ) ∈ ( R d × R d ) \B where B is some Lebesgue measureset, when the learning rate h satisﬁes
[Equation]
GD converges to a global minimum.
Theorem 3.1 says that choosing a constant learning rate depending on GD initialization guaranteesthe convergence for almost every starting point in the whole space.
This result is even strongerthan the already nontrivial convergence under small learning rate with high probability over randominitialization (Ye & Du, 2021).
Furthermore, the upper bound on h is sufﬁciently large: on the onehand, suppose GD initialization ( x 0 , y 0 ) is close to an unbalanced global minimum.
By Proposition 2.1, we can check that the largest eigenvalue L ( x 0 , y 0 ) of Hessian ∇ 2 f ( x 0 , y 0 ) is approximately(cid:107) x 0 (cid:107)
+ (cid:107) y
(cid:107) 2 .
Consequently, our upper bound of h is almost 4 /L , which is beyond 2 /L
(see Section 2 for more details).
On the other hand, we observe numerically that the 4 /L upper boundis actually very close to the stability limit of GD when initialized away from the origin (see moredetails in Appendix E).
The convergence in Theorem 3.1 has an interesting searching-to-converging transition as depicted in
Figure 1, where we observe two phases.
In Phase 1 , large learning rate drives GD to search for ﬂatregions, escaping from the attraction of sharp minima.
After some iterations, the algorithm enters thevicinity of a global minimum with more balanced magnitudes in x, y .
Then in Phase 2 , GD convergesto the found balanced global minimum.
We remark that the searching-to-converging transition alsoappears in Lewkowycz et al.
However, the algorithmic behaviors are not the same.
In fact, inour searching phase (phase 1), the objective function does not exhibit the blow-up phenomenon.
Inour convergence phase (phase 2), the analysis relies on a detailed state space partition (see Line 192)due to nonconvex nature of (3), while the analysis in Lewkowycz et al.
(2020) is akin to monotoneconvergence in a convex problem.
In comparison with the dynamics of small learning rate, we note that the searching phase (Phase 1 ) isvital to the convergence analysis.
Meanwhile, the searching phase induces a balancing effect of largelearning rate.
The following theorem explicitly quantiﬁes the extent of balancing.
Theorem 3.2 (Balancing) .
Under the same initial condition and learning rate h as Theorem 3.1, GDfor (3) converges to a global minimizer ( x, y ) satisfying
[Equation]
Consequently, its extent of balancing is quantiﬁed by
[Equation]
One special case of Theorem 3.2 is the following theorem, which states that no matter how close to aglobal minimum does GD start, if this minimum does not correspond to well-balanced norms, a largelearning rate will take the iteration to a more balanced limit.
We also demonstrate a sharp shrinkagein the distance between x and y .
Corollary 3.3 (From ‘unbalanced’ to ‘balanced’) .
For any δ ∈ (0 , µ ) , let the GD initialization satisfy
[Equation]
When the learning rate h satisﬁes h
= [Equation]
, the extent of balancing at the limiting point( x, y ) of GD obeys
[Equation]
Both Theorem 3.2 and Corollary 3.3 suggest that larger learning rate yields better balancing effect, as(cid:107) x − y (cid:107) 2 at the limit of GD may decrease a lot and is controlled by the learning rate.
We remark thatthe balancing effect is a consequence of large learning rate, as small learning rate can only maintainthe difference in magnitudes of x, y (Du et al., 2018).
In addition, the actual balancing effect can be quite strong with (cid:107) x k − y k (cid:107) 2 decreasing to be almost 0at its limit under a proper choice of large learning rate.
Figure 2 illustrates an almost perfect balancingcase when h =
[Equation]
+4 ≈ 0 . 0122 is chosen as the upper bound.
The difference (cid:107) x k − y k (cid:107)decreases from approximately 17 . 9986 to 0 . 0154 at its limit.
Additional experiments with variouslearning rates and initializations can be found in Appendix A.
Technical Overview We sketch the main ideas behind Theorem 3.1, which lead to the balancingeffect in Theorem 3.2.
Full proof is deferred to Appendix G.
The convergence is proved by handling Phase 1 and 2 separately (see Fig.1).
In Phase 1 , we prove that(cid:107) x k [-k 
-](cid:107) 2 + (cid:107) y k (cid:107) 2 has a decreasing trend as GD searches for ﬂat minimum.
We show that (cid:107) x k [-k 
-](cid:107) 2 + (cid:107) y k (cid:107) 2may not be monotone, i.e., it either decreases every iteration or decreases every other iteration.
In Phase 2 , we carefully partition the state space and show GD at each partition will eventually entera monotone convergence region.
Note that the partition is based on detailed understanding of thedynamics and is highly nontrivial.
Attentive readers may refer to Appendix G for more details.
Thecombination of Phase 1 & 2 is brieﬂy summarized as a proof ﬂow chart in Figure 3. [Figure]
R ANK -1 APPROXIMATION OF ISOTROPIC A ( AN UNDER - PARAMETERIZEDCASE )
Given the insights from scalar factorization we consider rank- 1 factorization of an isotropic matrix
A , i e., A = µI n × n with µ > 0 , d = 1 , and n ∈ N + .
The corresponding optimization problem is written as
[Equation]
Although similar at an uncareful glance, Problems (4) and (3) are rather different unless n
= d = 1 .
First of all, Problem (4) is under-parameterized for n > 1 , while (3) is overparameterized.
Moreimportantly, we’ll show that, when ( x, y ) is a global minimum of (4), x, y must be aligned, i.e., x = (cid:96)yfor some (cid:96) > 0 .
In the scalar factorization problem, however, no such alignment is required.
As aresult, the set of global minima of (4) is an n -dimensional submanifold embedded in a 2 n -dimensional space in the scalar factorization problem the set of global minimum is a (2 d − 1) -dimensional submanifold one rank deﬁcient — in a 2 d -dimensional space expect the convergence in (4) is more complicated than that in (3), since searching in (4) is demanding
To prove the convergence of large learning rate GD for (4), our theory consists of two steps: (i) showthe convergence of the alignment between x and y (this is new); (ii) use that to prove the convergenceof the full iterates (i.e., x & y ).
Step (i) ﬁrst:
Theorem 4.1 (Alignment) .
Given ( x 0 , y 0 ) ∈ ( R n × R n ) \B , where B is some Lebesgue measure-0set, when learning rate h ≤ min (cid:110)
[Equation]
µ [Equation] , the iterator ( x k , y k ) of GD at thek -th iteration satisﬁes | cos( ∠ ( x k , y k )) | → 1 as k → ∞ .
Proof sketch.
A sufﬁcient condition for the convergence of | cos( ∠ ( x k , y k )) | is (cid:107) x k [-k 
 k-](cid:107) 2 (cid:107) y k [-k 
 -](cid:107) 2 − ( x (cid:62) k y k ) 2 → 0 .
To ease the presentation, let U k
= x (cid:62) k y k , V k = x (cid:62) k x k , and W k = y
(cid:62) k y k .
By the GD update and some algebraic manipulation, we derive
[Equation]
where r k = (1 − h ( V k + W k ) + h 2 ( V k W k − µ 2 )) 2 .
When k is sufﬁciently large, we can show auniform upper bound on r k < 1 − c for some constant c > 0 .
In this way, we deduce that V k W k − U 2 kwill exponentially decay and converge to 0 .
More details are provided in Appendix H.
Theorem 4.1 indicates that GD iterations will converge to the neighbourhood of { ( x, y ) : x =(cid:96)y, for some (cid:96)
∈ R \{ 0 }} .
This helps establish the global convergence as stated in Step (ii).
Theorem 4.2 (Convergence) .
Under the same initial conditions and learning rate h as Theorem 4.1,
GD for (4) converges to a global minimum.
Similar to the over-parametrized scalar case, this convergence can also be split into two phases wherephase 1 motivates the balancing behavior with the decrease of (cid:107) x k (cid:107)
+ (cid:107) y k (cid:107) 2 , and phase 2 ensuresthe convergence.
The following balancing theorem is thus obtained.
Theorem 4.3 (Balancing) .
Under the same initial conditions and learning rate h as Theorem 4.1,
GD for (4) converges to a global minimizer that obeys
[Equation]
and its extent of balancing is quantiﬁed by
[Equation]
This conclusion is the same as the one in Section 3.
A quantitatively similar corollary like Corollary 3.3 can also be obtained from the above theorem, namely, if x 0 and y 0 start from an unbalancedpoint near a minimum, the limit will be a more balanced one.
G ENERAL MATRIX FACTORIZATION
In this section, we consider problem (1) with an arbitrary matrix A ∈ R n × n .
We replicate the problem formulation here for convenience, [Equation]
[Equation]
Note this is the most general case with n, d ∈ N + and any square matrix A .
Due to this generalization,we no longer utilize the convergence analysis and instead, establish the balancing theory via stabilityanalysis of GD as a discrete time dynamical system.
Let µ 1 ≥ µ 2 ≥ · · · ≥ µ n ≥ 0 be the singular values of A .
Assume for technical convenience(cid:107)
A (cid:107) 2 F = (cid:80) ni
=1 µ 2 i being independent of d and n .
We denote the singular value decomposition of Aas A = UDV (cid:62) , where U, D, V ∈ R n × n , U, V are orthogonal matrices and D is diagonal.
Then weestablish the following balancing effect.
Theorem 5.1.
Given almost all the initial conditions, for any learning rate h such that GD for (5) converges to a point ( X, Y ) , there exists c
= c ( n, d ) > c 0 with constant c 0 > 0 independent of h , n , and d , such that ( X, Y ) satisﬁes
[Equation]
and the extent of balancing is quantiﬁed by (cid:13)(cid:13) X − ( UV [Equation] Y (cid:13)(cid:13)
[Equation]
< 2 ch − 2 (cid:80)
[Equation]
,n } i = µ i , which means
[Equation]
min [Equation] In particular, when d = 1 , i.e., rank- 1 factorization of an arbitrary A , the constant c equals 1 .
We observe that the extent of balancing can be quantiﬁed under some rotation of Y .
This is necessary,since for factorizing a general matrix A (which can be asymmetric), at a global minimum, X, Y mayonly align after a rotation (which is however ﬁxed by A , independent of initial or ﬁnal conditions).
Figure 4 illustrates an example of the balancing effect under different learning rates.
Evidently,larger learning rate leads to a more balanced global minimizer.
Additional experiments with variousdimensions, learning rates, and initializations can be found in Appendix A; a similar balancing effectis also shown for additional problems including matrix sensing and matrix completion there.
[Figure]
Different from previous sections, Theorem 5.1 builds on stability analysis by viewing GD as adynamical system in discrete time.
More precisely, the proof of Theorem 5.1 (see Appendix I fordetails) consists of two parts: (i) the establishment of an easier but equivalent problem via the rotationof X and Y , (ii) stability analysis of the equivalent problem.
For (i), by singular value decomposition (SVD), A = UDV (cid:62) , where U, D, V ∈ R n × n , U and V areorthogonal matrices, and D is a non-negative diagonal matrix.
Let X
k
=
UR
k , Y k
=
V S k .
Then(cid:26) X k +
[Equation]
X k
[Equation]
k
[Equation]
(cid:62) k
[Equation] [Equation] [Equation] Therefore, GD for problem (5) is equivalent to GD for the following problem
[Equation]
and it thus sufﬁces to work with diagonal non-negative A .
For (ii), here is a brief description of the idea of stability analysis: consider each iteration of GD as a mapping ψ from u k ∼ ( X k , Y K ) to u k +1 ∼ ( X k +1 ,
Y k +1 ) , where matrices X and Y are ﬂattened and concatenated into a vector so that ψ is a closed map on vector space R 2 dn .
GD iteration is thus a discrete time dynamical system on state space R 2 dn given by
[Equation]
where f is the objective function f ( u k )
=
12 (cid:13)(cid:13) A − X k Y (cid:62) k (cid:13)(cid:13) 2 F , and gradient returns a vector thatcollects all component-wise partial derivatives.
It’s easy to see that any stationary point of f , denoted by u ∗ , is a ﬁxed point of ψ , i.e., u ∗ = ψ ( u ∗ ) .
We are interested in what ﬁxed point will the iterations of ψ converge to.
For this, the followingnotions are helpful:
Proposition 5.2.
Consider a ﬁxed point u ∗ of ψ .
If all the eigenvalues of Jacobian matrix ∇ ψ ( u ∗ )are of complex modulus less than 1 , it is a stable ﬁxed point.
Proposition 5.3. Consider a ﬁxed point u ∗ of ψ .
If at least one eigenvalue of Jacobian matrix∇ ψ ( u ∗ ) is of complex modulus greater than 1 , it is an unstable ﬁxed point.
Roughly put, the stable set of an unstable ﬁxed point is of negligible size when compared to thatof a stable ﬁxed point, and thus what GD converges to is a stable ﬁxed point for almost all initialconditions (Alligood et al., 1996).
Thus, we investigate the stability of each global minimum of f(each saddle of f is an unstable ﬁxed point of ψ and thus is irrelevant).
By a detailed evaluation of∇ ψ ’s eigenvalues (Appendix I), we see that a global minimum ( X, Y ) of f corresponds to a stableﬁxed point of GD iteration if (cid:12)(cid:12)(cid:12) 1 − ch (cid:16) (cid:107) X (cid:107) 2 F + (cid:107) Y (cid:107)
[Table]
< 1 , i.e., it is balanced as in Thm. 5.1.
C ONCLUSION AND D ISCUSSION
In this paper, we demonstrate an implicit regularization effect of large learning rate on the homogeneous matrix factorization problem solved by GD.
More precisely, a phenomenon termed as“balancing” is theoretically illustrated, which says the difference between the two factors X and Ymay decrease signiﬁcantly at the limit of GD, and the extent of balancing can increase as learningrate increases.
In addition, we provide theoretical analysis of the convergence of GD to the globalminimum, and this is with large learning rate that can exceed the typical limit of
/L , where L is thelargest eigenvalue of Hessian at GD initialization.
For the matrix factorization problem analyzed here, large learning rate avoids bad regularities inducedby the homogeneity between X and Y .
We feel it is possible that such balancing behavior can also beseen in problems with similar homogeneous properties, for example, in tensor decomposition (Kolda& Bader, 2009), matrix completion
(Keshavan et al., 2010; Hardt, 2014), generalized phase retrieval(Candes et al., 2015; Sun et al., 2018), and neural networks with homogeneous activation functions(e.g.
, ReLU).
Besides the balancing effect, the convergence analysis under large learning rate may betransplanted to other non-convex problems and help discover more implicit regularization effects.
In addition, factorization problems studied here are closely related to two-layer linear neural networks.
For example, one-dimensional regression via a two-layer linear neural network can be formulated as the scalar factorization problem (3): Suppose we have a collection of data points ( x i , y i ) ∈ R × R for i = 1 , . . . , n .
We aim to train a linear neural network y
=
( u (cid:62) v ) x with u, v ∈ R d for ﬁtting the data.
We optimize u, v by minimizing the quadratic loss, ( u
[Equation]
[Equation]
As can be seen, taking µ = (cid:80) ni
=1 x i y i
(cid:80) ni
=1 x 2 i recovers (3).
In this regard, our theory indicates that trainingof u, v by GD with large learning rate automatically balances u, v , and the obtained minimum is ﬂat.
This may provide some initial understanding of the improved performance brought by large learningrates in practice.
Note that (6) generalizes to arbitrary data distribution of training a two-layer linearnetwork with atomic data (i.e., x = 1 and y = 0 ) in Lewkowycz et al.
It is important to clarify, however, that there is a substantial gap between this demonstration andextensions to general neural networks, including deep linear and nonlinear networks.
Although wesuspect that large learning rate leads to similar balancing effect of weight matrices in the network,rigorous theoretical analysis is left as a future direction.