Paper: Gradient descent for matrix factorization: Understanding large initialization (ICLR-2024)

Reviewer comment (Presentation):
The theoretical results in this paper are presented in a somewhat isolated manner, and it seems that the result for rank-$2$ is not even stated as an independent theorem. I suggest that the authors can briefly summarize the main results (for rank-$2$ and general ranks) in the introduction, before going into technical details.

Author response:
Thanks for the reviewer’s suggestion. In our updated version, we summarize our results in a main theorem, Theorem 6, and an intuitive theorem, Theorem 1. We put the intuitive theorem in the introduction. We also move the visualization results and the intuitive ideas to the introduction. We move the analysis of rank-2 results to a later section called sketch of proof.

Reviewer comment (Novelty):
The "signal-noise-ratio" argument in this work looks different from existing works that also decompose the GD trajectory in the signal and noise term (e.g. [1,2]), but the decomposition seems to be the same. I suggest that the authors can add more discussions about this difference to highlight the contribution of this paper.

Author response:
Thanks for the reviewer’s suggestion. We have added more discussions in our updated version. Specifically, to employ the SNR analysis, we need a lower bound of the signal and an upper bound on the noise, both in terms of their previous iterations. These two bounds need to be related so that the ratio of SNR$_{t+1}$  by SNR$_t$ can be analyzed. The challenging part is to obtain two related upper and lower bounds. Previous literature does not provide SNR considerations so their upper and lower bounds are not applicable for our purpose. In addition, for the large initialization setting, our SNR analysis is even more novel, because we introduce new signal and noise terms.

Reviewer comment (Theory):
I am confused about Assumption 8 and cannot see how it is related to the arguments at the beginning of Sec. 4.2.3. According to my understanding, it says that the norm of the $2$-to-$d$ components is larger than its inner product with the first component. Can you explain this assumption in more details?

Author response:
Thanks for the question. First, the reviewer may misunderstand the meaning of this assumption. In this assumption, we assume the existence of $t_1^*$ such that
        
        $r(1-\eta\Delta/6)^{t_{1}^*}\leq\sqrt{\frac{\Delta}{8}}\min\{\sigma_1(u_{2,t_1+t_1^*}),\sqrt{\frac{\Delta}{2}}\}$
        
        Recall that for all $t\geq t_1$, we have shown that 
        $\sigma_1(u_{1,t}K_{1,t}^\top)\leq(1-\eta\Delta/6)^{t-t_{1}}$
        
        Therefore, the condition for $t_1^*$ means the second signal $\sigma_1(u_{2,t_1+t_1^*})$ is larger than the noise term $\sigma_1(u_{1,t}K_{1,t}^\top).$ Hence, when we analyze the second signal $\sigma_1(u_{2,\cdot})$, we could disregard the effect of the noise term $u_{1,t}K_{1,t}^\top$. This allows us to use the inductive argument even when the noise term is not exactly zero.  We have also added this explanation to the paper in our updated version.

Reviewer comment (Theory):
The authors seem to employ a successive argument for general rank matrix. However, the definition of the benign set $R$ in Sec. 3.1 would change for higher ranks. Is there any arguments in your proof verifying that GD remains in the high-rank $R$? Probably you did it in Theorem 13, but I cannot understand how they are related.

Author response:
Thanks for the question. We are afraid that the reviewer may misunderstand some points here. Let us clarify for the reviewer and answer all the questions. 
    - When we want to study rank-r approximation, we will fix the benign set in Sec 3.1 as a rank-r set and we never change it. To apply this result in Theorem, we need to prove that after r inductive arguments, the GD sequence will enter the benign set defined in Sec 3.1. This is because when all r signals are strong enough and noise are weak, the sequence will enter the desired strong SNR region.
    - Consider the heat plot of $XX^\top$ and the first $r\times r$ block. Our argument is to show that the first r diagonal elements will increase above $\Delta/2$ and the norm of all off-diagonal elements in this block decrease fast to zero. Combining, we know this block’s smallest singular value is larger than  $\Delta/4$.
    - The challenge is to verify that all r signals can be strong enough and noise are weak. We prove this using our novel inductive argument.
    - In our updated version, we have clearly discussed the relationship between local linear convergence theorem and the theorem using large initialization. Hope this has addressed the reviewer's question.

Reviewer comment (Theory):
In existing works it is commonly the case that convergence/incremental learning hold with high probability (e.g. [1] Theorem 3.3), since the initialization has to be aligned with each component, otherwise it cannot make progress in some direction. Does this paper need to impose similar requirements for initialization?

Author response:
Yes, if we employ random initialization, our results apply to them with high probability. This is the best outcome we can hope because it is always possible that the initialization is pretty bad so that the algorithm can diverge. For example, a extremely large $\sigma_1(X_0)$ could result in the divergence of GD.
    - In addition, we would like to mention the different requirements between ours and [1]’s. The main difference is the initialization magnitude. They require small initialization, and specifically $\varpi \leq  \min(d^{-1/2},d^{-3\kappa^2})$, where $\kappa>1$ is the condition number. In contrast, we only require $\varpi=O(1/\sqrt{\eta})$ where $\eta$ is the step size. Notably, our requirement is independent of d. Also, our requirement is rate-optimal.

Reviewer comment (Experiments):
only one experiment, on rank 2 matrix approximation

Author response:
- only one experiment, on rank 2 matrix approximation  
   - we have added more experiments in the appendix; due to the limitation of pages, we remain a single experiment in the main text for illustrating our ideas.

Reviewer comment (Experiments):
unclear dependency on initialization: $\alpha$ needs to be small

Author response:
- unclear dependency on initialization: $\varpi$ needs to be small;  
   - we have demonstrated that our initialization condition is dimension-independent and rate-optimal; in particular, we only require $\varpi\lesssim 1/\sqrt{\eta}$, which is in sharp contrast to the previous 'small' requirements $\varpi\lesssim \min(d^{-1/2},d^{-3\kappa^2})$.

Reviewer comment (Theory):
unclear it it is required in the proof that $X_0 \in S$ or not (contradictory statements in the paper)

Author response:
- unclear it it is required in the proof that $X_0\in S$  
   - we have made it clear that we do not need this condition; if this is the point that you are concerned, then we have demonstrated that such condition can be completely removed; hope this could address the reviewer's concern.

Reviewer comment (Theory):
The paper claims to study large initialization, and that the first result with initialization in the region of Eq 10 is not satisfying. However, the proof of the main result states "This is achievable if we use random initialization X0 = αN0 with a reasonably small constant α", which is not "large initialization". $\alpha$ must be small enough such that $\sigma^2_1(K_r) \leq \lambda_r - 3 \Delta/4$, so $\alpha$ depends on $r$, and it's not so wild to think that $r$ is a fraction of $d$. Remark 4 seems to contradict that, highlighting that the theorem does not require $X_0 \in S$, but the exposition is very confusing here (eg, Lemma 5 assumes that $X_0 \in S$)

Author response:
- The paper claims to study large initialization, and that the first result with initialization in the region of Eq 10 is not satisfying…. However, the proof of the main result states "This is achievable if we use random initialization X0 = αN0 with a reasonably small constant α", which is not "large initialization".  must be small enough such that , so  depends on , and it's not so wild to think that  is a fraction of . Remark 4 seems to contradict that, highlighting that the theorem does not require , but the exposition is very confusing here (eg, Lemma 5 assumes that )  
    - Sorry for the confusion here. In our updated version, we have removed all the confusion part. Specifically, for every results we display, we only require $\sigma_1(X_0)\leq1/\sqrt{3\eta}$ and correspondingly $\varpi\lesssim 1/\sqrt{\eta}$. All the subsequent results are then proved, rather than assumed. In addition, we emphasize that this requirement is rate-optimal because if $\sigma_1(X_0)$ is too large, then the GD sequence can simply diverge. A simple counterexample is when $\Sigma=0$ and $\sigma_1^2(X_0)\geq 3/\eta$. An inductive argument implies that $\sigma_1(X_{t+1})\geq 2\sigma_1(X_t)$  for all t, and thus $X_t$ diverges. We have added this example and explanation below Theorem 6 in our updated version.  
    - We would also like to emphasize the difference between large initialization and small initialization. Large initialization means the norm of $X_0$ converges to a positive constant while small initialization means the norm of $X_0$ converges to zero. By concentration results, the norm of $X_0=\varpi N_0$ is O($\varpi$). Therefore, taking a positive constant $\varpi$ means the large initialization. In contrast, in previous literature that assumes small initialization, they require $\varpi$ to be as small as $\min(d^{-1/2},d^{-3\kappa^2})$, where $\kappa>1$ is the condition number. Therefore, there should be a significant difference between two settings.

Reviewer comment (Writing):
It is clear that Sigma can be assumed diagonal in problem 1 up to an orthogonal change of variable in $X$ (because GD is equivariant to such a change of variable), but this deserves to be stated explicitly the first time this assumption is made (P2).

Author response:
- It is clear that Sigma can be assumed diagonal in problem 1 up to an orthogonal change of variable in  (because GD is equivariant to such a change of variable), but this deserves to be stated explicitly the first time this assumption is made (P2).  
    - Yes. In our revised version, we have mentioned that such assumption is made without loss of generality, and explained the reason.

Reviewer comment (Writing):
$\sigma_1(u_{k,t})$ is just its norm, right? $u_{k,t}$ is a vector. Same for $u_{k,t} K_{k, t}^\top$

Author response:
- $\sigma_1(u_{1,t})$ is just its norm, right? Same for $u_{k,t}K_{k,t}^\top$.  
    - Yes. Both the norm notation and the singular value notation can be used. We chose the singular value notation for its consistency with other results in the paper.

Reviewer comment (Experiments):
In the experimental example, how do we know if 0.5 N(0, 1/d) is a large initialization? Since $d$ does not vary, what can we say? It would be better to have the experiment for several values of $d$ and fixed $\bar \omega$

Author response:
- In the experimental example, how do we know if 0.5 N(0, 1/d) is a large initialization? Since d does not vary, what can we say? It would be better to have the experiment for several values of d and $\varpi$  
    - We appreciate your attention to the details of our matrix factorization experiment. Theoretically, by the concentration results, the norm of $X_t$ is around 0.5. This is comparable to the largest eigenvalue 1 of $\Sigma$. Consequently, the higher order term $X_tX_t^\top X_t$ cannot be disregarded in the GD iterations. In this context, the theory of small initialization fails to account for the convergence behavior observed in GD.  
    - Additionally, we add new experiments with varying d and $\varpi$ in the appendix. In particular, we choose d from 1000, 2000, 4000, and we take $\varpi$ from 0.001, 0.5, 2, and 10. Our findings show that if $\varpi$ is as large as 10 in this setting, the GD sequence will diverge. For the remaining three choices of $\varpi$, we can observe similar convergence behaviors of GD.

Reviewer comment (Experiments):
Please provide more than one experiment in dimension 2. The analysis is incomplete without more proof that the results hold for varying r and d, fixed $\bar \omega$.

Author response:
- Please provide more than one experiment in dimension 2. The analysis is incomplete without more proof that the results hold for varying r and d, fixed $\varpi$.  
    - Thanks for the reviewer’s suggestion. In our updated version, we have provided additional experiments in the appendix. For rank-two matrix approximation, we vary both dimension d and the initial magnitude $\varpi$. Our results are consistent with the results we have presented.

Reviewer comment (Writing):
Incremental learning has many possible meanings, it would be nice to clarify it here. Same for "spectral method", I think the amount of details given in section 3.2 should be slightly increased.

Author response:
- Incremental learning has many possible meanings, it would be nice to clarify it here.  
    - Thank you for the suggestions. We have explained that in our context, incremental learning refers to the process where eigenvectors associated with larger eigenvalues are learned first.  
- Same for "spectral method", I think the amount of details given in section 3.2 should be slightly increased.  
    - We have explained the spectral method. It also means the power method. For these kinds of methods, the eigenvectors related to larger eigenvalues are learned first. Notably, $\sigma_r(U)$ increases faster than $\sigma_1(J)$.  
    - We have extended the content of that section. We add more details on two aspects: the theoretical intuition and their assumptions on small initialization.

Reviewer comment (Theory):
P3 "because at the global minima"/"the set R contains all the global minima": it seems to me that there is a single global minima, the rank-$r$ truncated SVD of $\Sigma$. can the authors clarify?

Author response:
- P3 "because at the global minima"/"the set R contains all the global minima": it seems to me that there is a single global minima, the rank-truncated SVD of $\Sigma$. can the authors clarify?  
    - The global minima refer to all $X$ such that $XX^\top=\Sigma_r.$  While  $\Sigma_r$ is unique, there are multiple $X$ satisfying the equality. Since R is a set of X, it is more accurate to say that the set R contains all the global minima.

Reviewer comment (Writing):
First sentence of Section 3.3 is a repetition from above.

Author response:
- First sentence of Section 3.3 is a repetition from above. Paragraph 4.1 consider replacing rank- by rank-2 for clarity (and all instances of  in that paragraph)  
    - Yes, we have made all the modifications.

Reviewer comment (Writing):
Paragraph 4.1 consider replacing rank-$r$ by rank-2 for clarity (and all instances of $r$ in that paragraph)

Author response:
- Paragraph 4.1 consider replacing rank-$r$ by rank-2 for clarity (and all instances of $r$ in that paragraph)  
    - Yes, we have made all the modifications.

Reviewer comment (Presentation):
Legend of Figure 1: "top three rows" are the first three rows?

Author response:
- Legend of Figure 1: "top three rows" are the first three rows?  
    - Yes.

Reviewer comment (Presentation):
The presentation could be further improved. The authors claim that they aim to understand the convergence of GD with large random initialization on simple matrix factorization problems, but as a reader, I could not find a main theorem or corollary that clearly states that with large random initialization, GD converges with certain rates under appropriate parameter settings and assumptions.

Author response:
In our revised version, we present an improved version of the main theorem, Theorem 6, in Section 4.2. We also provide an intuitive version of the theorem in the introduction. Our theorem is deterministic and applicable to large random initialization. In our theorem, we prove that all the quantities beyond $t^*_k$ are bounded by either a constant or a logarithmic term. Hence, we claim that we provide a detailed trajectory analysis of GD. If we assume $t_k^*=O(\log(d))$, then GD achieves $\epsilon$-accuracy in $O(\log(d)+\log(1/\epsilon))$ iterations. This is probably the desired result the reviewer want to see. However, due to the challenges posed by large initialization, we are currently unable to verify such assumption. This is one of the limitation of the paper. In some sense, this also reflects the difficulty of the problem we are studying. On the other hand, even without this assumption, our results still reveal the incremental learning behavior of GD and many other characteristics of GD. We hope the reviewer could appreciate our attempts to tackle the challenging and meaningful problem.

Reviewer comment (Writing):
The presentation of the theorem involves many notations, which on the other hand, looks could not be uninvolved.

Author response:
Second, the reviewer suggests us to present a more concise and clear theorem. 
   - Thanks for the suggestion! We have carefully addressed this point. We have presented a clearer theorem that emphasizes our assumptions and results. Specifically, when using random initialization, our trajectory analysis holds almost surely. Moreover, if an additional assumption is made, we may obtain the $O(\log(d)+\log(1/\epsilon))$ convergence rate of GD. We have also included detailed discussions in the paper. Hope this revision can address the reviewer's concern.

Reviewer comment (Theory):
The 'large initialization' is in fact the `large random initialization'. The authors consider GD with large random initialization, but the variance still depends on the dimension of the problem under consideration. Therefore, when the dimension d is relatively large, it reduces to the ``small" random initialization setting.

Author response:
To clarify for the reviewer, our perspective on large initialization refers to the scenario where the norm of $X_0$ remains a positive constant, even as the dimension approaches infinity. By statistical concentration results, the norm of $N_0$ approaches to a finite positive constant when dimension d tends to infinity. Therefore, if $X_0=\varpi N_0$ with a positive constant $\varpi$ independent of d, then it belongs to the large initialization regime. On the other hand, if $\varpi$ tends to zero as d increases, then it belongs to the small initialization regime.
    - In previous literature, $\varpi$ is as small as $\min(d^{-1/2},d^{-3\kappa^2})$, where $\kappa>1$ is the conditional number. In our case, $\varpi$ is a constant independent of d. As we discussed on page 6 below Theorem 6, we only require $\varpi=O(1/\sqrt{\eta})$ where $\eta$  is the step size. We also demonstrate that this is rate optimal. If we further increase it, the GD algorithm will simply diverge.

Reviewer comment (Novelty):
The authors consider a simple matrix factorization problem, but as claimed in the main text, the motivation of this paper is to better understand GD with large random initialization in training neural networks.

Author response:
Yes. Even for matrix factorization, the understanding of large initialization is rather limited. We hope our paper could bring people to notice this gap between theory and practice. We also hope our technique and results can inspire future research in related fields, including deep learning theory.

Reviewer comment (Novelty):
Up to minor clarity points below, the paper is understandable, but dense. My main concern regarding the submission is its intended audience might be limited to the people looking to build upon those results to get towards a better understanding of symmetric matrix factorization. But my perspective is likely limited and there might be a wider applicability to the presented results.

Author response:
Symmetric matrix factorization intersects with a diverse array of problems such as compressed sensing, phase retrieval, matrix completion, PCA, quadratic regression, and neural networks, among others. It serves as a fundamental model that facilitates theoretical analysis. In addition, our work is the first one (except for the rank one case) to study large initialization. Such initialization is commonly used in practice, including neural networks or simply phase retrieval and PCA. Therefore, we expect our theoretical investigation will spark further research across these related fields.

Reviewer comment (Theory):
The introduction states that the focus of the submission is on the implicit bias of gradient descent, but I do not see how studying the dynamics of Fig. 1 connects to the implicit bias (which, in my understanding, refers to the limit point the optimizer converges to)?

Author response:
We would like to offer two perspectives on the implicit bias of the gradient descent algorithm. Firstly, from the viewpoint of the limit point, the algorithm almost surely  converges to the global minimum. Although this is an optimization result, one may also view it as certain implicit bias, because the optimization problem is non-convex. Secondly, our trajectory analysis reveals incremental learning behaviors of GD during the training process. This second viewpoint, focusing on the training process, is typically useful when early stopping is utilized.

Reviewer comment (Theory):
Remark 2 states that the results hold for PSD $\Sigma$, but the results seem to also assume that $\Sigma$ is diagonal. Is this assumption necessary, or is it presented for the diagonal case without loss of generality? What is the key difficulty in generalizing it to non-diagonal matrices?

Author response:
There is no loss of generality to assume that $\Sigma$ is diagonal, because the analysis of GD is invariant to the orthogonal transformation. In our updated version, we have mentioned this point when we make the assumption. Moreover, our research can be easily extended to general symmetric (not necessarily positive semi-definite) cases.

Reviewer comment (Writing):
Please define incremental to avoid confusion with the alternative use of incremental learning as a synonym for seeing-one-example-at-a-time learning. I realise in post that this is what the second-to-last paragraph in §1 is doing, but I didn't read it as such. A more explicit phrasing the first time the term appears might help, e.g., in the 4th paragraph "Jin et al. demonstrate an incremental learning phenomenon with small initialization; Eigenvectors associated with large eigenvalues are learned first". A sentence as to how this differs qualitatively from a similar observation on linear regression might help contextualize too.

Author response:
Thank you for the suggestions. In our revised version, we have explained that in our context, incremental learning refers to the process where eigenvectors associated with larger eigenvalues are learned first.

Reviewer comment (Theory):
The introduction describes matrix factorization as "mirroring the training of a two-layer linear network", but this doesn't hold for the symmetric matrix factorization studied here, which seems more similar to quadratic regression.

Author response:
The training of a two-layer linear network can be interpreted as asymmetric matrix factorization (MF). See Section 2.2 in Sun et al. (2019). While there are distinctions between asymmetric and symmetric MF, the study of symmetric MF can be seen as a vital preliminary step towards analyzing asymmetric MF. Moreover, we would like to mention that asymmetric MF with a suitable regularizer
     $$\min_{X,Y}|\Sigma-XY^T|_F^2+\frac{1}{4}|X^T X-Y^T Y|_F^2$$
    can be directly analyzed using the symmetric results.