Paper: A LOCAL POLYAK-ŁOJASIEWICZ AND DESCENT LEMMA OF GRADIENT DESCENT FOR OVERPARAMETERIZED LINEAR MODELS (ICLR-2024)

Reviewer comment (Novelty):
Literature review misses some references of closely-related (and even possibly overlapping works), e.g [Nguyen-Mondelli, 2020], [Radhakrishnan-Belkin-Uhler, 2021]. One really needs to clarify the novelty compared to existing results.

Author response:
We thank the reviewer for pointing us to these results (Nguyen et al., 2020; Radhakrishanan et al., 2020). We have added (Nguyen et al., 2020) in the "Related Work" section. In (Radhakrishanan et al., 2020), the authors study the convergence of autoencoder which is less relevant to our work. Thus, we won't cite it in the paper. We clarify the novelty of this work compared to existing results as follows. Existing results that study the convergence of deep linear networks (Arora et al., 2018; Du et al., 2018a) or deep nonlinear networks (Nguyen et al., 2020; Du et al., 2018; Jacot et al., 2018; Liu et al., 2022; Lee et al., 2019) all make unrealistic assumptions. Specifically, the NTK analysis[1,5,6,7,8] is done under rather unrealistic assumptions such as extremely large width and large Gaussian initialization. Further, (Chizat et al., 2019) show that the performance of neural networks degrades when trained in the NTK regime, thus the NTK analysis does not capture the behavior of neural networks used in practice. On the other hand, (Arora et al., 2018; Du et al., 2018a) focus on deep linear networks under the assumption that there is sufficient margin at initialization. This requires the initialization to be close to optimal, which is rarely seen in practice. Therefore, both lines of work make some unrealistic assumptions. The goal of our work is precisely to relax these unrealistic assumptions by considering general initializations with either sufficient margin or sufficient imbalance, which subsumes NTK initialization, mean-field initialization and spectral initialization. Moreover, our work assumes the width is larger than or equal to the input and output dimensions, which can be considered as a "mild-overparameterization" compared with the NTK analysis. Therefore, our work is an important stepping stone to analyzing deep networks of finite width under general initialization, which is not covered by existing results.

Reviewer comment (Theory):
How could we estimate \eta_\max? How small could it be? What does it depend on?

Author response:
In Appendix E, we show $\eta_\max=\min(\eta_0^{(1)}, \eta_0^{(2)}, \log(1+\frac{\alpha_1}{2\alpha_2}))^{1/c}$. Thus, $\eta_\max$ depends on initial weights, $c, d, \mu, K$, and $L(0)$. One can see that if $L(0), K, \beta_2 $and $c, d$ increase or $\mu, \frac{\alpha_1}{\alpha_2}$ decrease, then $\eta_\max$ will decrease accordingly. To numerically solve for $\eta_\max$, one can use existing equation solver to solve eq(87) and eq(88) to get $\eta_0^{(1)}, \eta_0^{(2)}$. Then, $\eta_\max$ can be estimated by the minimum value by its definition.

Reviewer comment (Novelty):
A work which "revived" the interest in PL from the Optimization community is "Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition", Karimi 2016, which the authors could cite.

Author response:
3. Missing reference: we thank the reviewer for pointing us to this work. We have added the reference to Karimi et. al. 2016 in the "Related Work" section.

Reviewer comment (Writing):
The way the authors cite the Descent Lemma is broken: "where Descent lemma is" should be "where the Descent Lemma is", same for "to derive Descent lemma," etc.

Author response:
4. Cosmetics: we thank the reviewer for the suggestions. We have modified the paper accordingly. For Descent lemma, we treated it as a noun and that's why we cite it as "Descent lemma" instead of "the Descent lemma".

Reviewer comment (Writing):
P6 "In §2.1, we show that as": we showed.

Author response:
4. Cosmetics: we thank the reviewer for the suggestions. We have modified the paper accordingly. For Descent lemma, we treated it as a noun and that's why we cite it as "Descent lemma" instead of "the Descent lemma".

Reviewer comment (Writing):
"if the $\lim_{t \to \infty}": extra "the", this time.

Author response:
4. Cosmetics: we thank the reviewer for the suggestions. We have modified the paper accordingly. For Descent lemma, we treated it as a noun and that's why we cite it as "Descent lemma" instead of "the Descent lemma".

Reviewer comment (Novelty):
$\textbf{(1) Incremental contribution}$: Arora et al. in Ref [1] studied linear convergence of gradient descent for multi-layer neural networks. While Arora et al. assumed balanced weights and a deficiency margin, these conditions were proven by them in the context of gradient descent. In this work, although the authors only focus on general loss, they just study two-layer linear networks. Moreover, their convergence rate also depends on margin and imbalance. The contribution of this work is very incremental in terms of Ref [1].

Author response:
1. Incremental contribution compared with (Arora et al., 2018). 
We respectfully disagree that our work is an incremental contribution compared with (Arora et al., 2018). In the work of (Arora et al., 2018), the authors provide the convergence rate of deep linear networks trained with gradient descent. While deep linear networks are indeed more general, (Arora et al., 2018) focus on deep linear networks under the assumption that there is sufficient margin and small imbalance at initialization. This requires the initialization to be close to optimal, which is rarely seen in practice. Moreover, standard Gaussian initialization, Xavier initialization, and He initialization all have large imbalance at initialization. Therefore, the convergence in (Arora et al., 2018) is proved under some unrealistic assumptions that cannot be satisfied by practical initialization schemes. The goal of our work is precisely to relax these unrealistic assumptions by considering general initialization with either sufficient margin or sufficient imbalance. Therefore, our work is an important stepping stone to analyzing deep networks of finite width under general initialization.

Reviewer comment (Theory):
$\textbf{(2) Limited generalizability to deep linear networks}$: It seems that the authors don't mention how to generalize their results to deep linear networks. It is believed that deep networks are more commonly used in applications. The paper leaves a significant gap in its discussion by omitting details on the generalization of their findings to deep linear networks.

Author response:
2. Generalizability to deep linear networks.
Extending the results of this work to deep linear networks trained via gradient descent is the next step, and the analysis can be done in the same spirit as it was done in this work. In the case of two-layer linear networks, we present a framework showing that one can use the imbalance and margin to bound the singular values of \mathcal{T} during the training. In the case of deep linear networks, $\mathcal{T}$ takes a different form depending on all layer weights which is more complicated than the two-layer case. On the other hand, we have $L-1$ imbalances for L-layer deep linear networks. As long as one can bound the $\mathcal{T}$ for deep linear networks using $L-1$ imbalances and the margin, the convergence rate of deep linear networks is derived. We refer the reviewer to (Min et al., 2023) which proves the convergence of deep linear networks trained via gradient flow in a similar spirit as described above.

Reviewer comment (Theory):
$\textbf{Q1.}$ Is it possible to generalize the current analysis to deep linear networks or deep nonlinear networks?

Author response:
2. Generalizability to deep linear networks.
Extending the results of this work to deep linear networks trained via gradient descent is the next step, and the analysis can be done in the same spirit as it was done in this work. In the case of two-layer linear networks, we present a framework showing that one can use the imbalance and margin to bound the singular values of \mathcal{T} during the training. In the case of deep linear networks, $\mathcal{T}$ takes a different form depending on all layer weights which is more complicated than the two-layer case. On the other hand, we have $L-1$ imbalances for L-layer deep linear networks. As long as one can bound the $\mathcal{T}$ for deep linear networks using $L-1$ imbalances and the margin, the convergence rate of deep linear networks is derived. We refer the reviewer to (Min et al., 2023) which proves the convergence of deep linear networks trained via gradient flow in a similar spirit as described above.

Reviewer comment (Theory):
$\textbf{Q2.}$ In Theorem 3.2, it is assumed that $\alpha_1 > 0$. Can the authors verify this condition?

Author response:
3. Verification of $\alpha_1>0$. 
Please see our response to all reviewers.

Reviewer comment (Novelty):
The prior work this works improves on already has entire sections on adaptive step sizes: It is in a small sentence on page 8 we discover that [1] already proposes adaptive stepsizes when throughout it is presented as only having fixed stepsize schemes. The presentation of the prior work needs to include this fact.

Author response:
We thank the reviewer for the suggestion. We modified the "Related Work" section to highlight that (Xu et al., 2023) also designed an adaptive step size scheme to accelerate the convergence, and the convergence rate of the adaptive step size scheme heavily relies on two auxiliary constants which is a slower rate compared with this work.