Paper: On the Disconnect Between Theory and Practice of Overparametrized Neural Networks (ICLR-2024)

Reviewer comment (Presentation):
While each of the authors' claims is clear, the connections between them are somewhat disjointed. Specifically, it is challenging to perceive Section 3.1-3.2's issues as problems since, in reality, wide NNs experience less catastrophic forgetting, as suggested in Section 3.3. Therefore, combining Sections 3.1 and 3.2 into one section and separating Section 3.3 might help avoid this confusion.

Author response:
Could you clarify what you mean by "[...] it is challenging to perceive Section 3.1-3.2's issues as problems since, in reality, wide NNs experience less catastrophic forgetting, as suggested in Section 3.3."?
While the issues described in Sections 3.1 and 3.2 have the same underlying assumptions (that the networks behave according to the kernel regime), they all consider different domains (training, uncertainty quantification and continual learning). Only Section 3.3 concerns itself with catastrophic forgetting.

All three sections (training, uncertainty quantification, and continual learning) make assumptions based on the kernel regime. For provably faster optimization via natural gradient descent, we find that practical architectures are not wide enough to be close to the kernel regime and thus satisfy Jacobian stability. In the neural bandit setting, we find that sequential actions taken based on the uncertainty as given by the empirical NTK perform no better than random. In theory if the network was sufficiently close to the kernel regime, it should achieve near-optimal regret (Zhou et al., 2020). Finally, in the continual learning setting catastrophic forgetting can be predicted and mitigated if the network is near the kernel regime. In practice, at first glance wider networks seem to mitigate forgetting (as observed by Mirzadeh et al., 2022), but based on our experiments only if the networks are not trained to high accuracy, which is clearly desirable in practice. Therefore all of these sections show that drawing conclusions from the kernel regime about the behavior of neural networks used in practice can be problematic. This is the observed disconnect we describe.

Reviewer comment (Presentation):
Have you considered restructuring the sections to more coherently present the content, possibly merging Sections 3.1 and 3.2, and separating Section 3.3?

Author response:
We believe all three sections 3.1-3.3 fit semantically under one heading, since they all assume the networks behave according to the kernel regime, while in practice with realistic widths they behave very differently. For the final version we will improve the introduction to Section 3 to make that very explicit. Would a different title for Section 3 rather than "Connecting Theory and Practice" help make it more clear what we are describing in Section 3? 

First of all, even though Sec. 3.1-3.3 fall into the same theme in that they based their assumptions in the kernel regime, we take your point that they can potentially be clearer as separate sections.
Our main goal is indeed to show independent examples of where the kernel-regime assumption fails when used to explain practical phenomena (Sec. 3.1, 3.3) or to develop a practical method (Sec. 3.2).
We are happy to separate them into three sections as you suggested.

Reviewer comment (Theory):
Second-order optimization can exhibit issues like slow convergence and poor generalization. Could you specify which of these issues the unstable Jacobian directly contributes to, based on your research?

Author response:
You are completely correct that second-order methods may not only converge slowly, but also demonstrate poor generalization or other issues.
We only consider the _speed of convergence_ of the training loss in our work. As we describe on page 4 in the paragraph "Fast Convergence of NGD in the Kernel Regime", a network in the kernel regime, if trained with natural gradient descent (an approximate second-order method), provably exhibits fast convergence (Theorem 3, Zhang et al. 2019). 
The proof by Zhang et al. (2019) assumes conditions (7) and (8), whereby (8), the Jacobian stability, is the condition which is satisfied if the network is sufficiently close to the kernel regime. We experimentally demonstrate that this assumption is not satisfied for practical architectures (see Figure 2b). 
While second-order methods may perform well for deep learning, our findings show this cannot be explained via the kernel regime. To reflect better what we describe in Section 3.1, we changed the title to "Training: Conditions for Fast Convergence of Second-Order Optimization".

Reviewer comment (Novelty):
**Novelty**: It was hard for me to understand how novel the submitted paper is. I would appreciate a comparison to "Empirical Limitations of the NTK for Understanding Scaling Laws in Deep Learning".

Author response:
While Vyas et al. (2023) briefly discuss generalization as a function of width, they primarily study the NTK in the context of scaling laws, i.e. how quickly generalization error decreases with more training data. In contrast, our paper primarily studies predicted phenomena from the kernel regime as a function of width. Our work focuses on domains where the kernel regime promises to inform better algorithms in practice. More concretely, how to train them efficiently, how to obtain reliable and cheap uncertainty quantification and how to deploy them in a continual learning setting. In that sense our paper complements the findings by Vyas et al. (2023). Not only is the NTK limit not predictive of finite-width NN generalization, but predictions from the kernel regime should also not be used without scrutiny as the basis for better algorithms (training, uncertainty quantification) or guiding principles in practice (continual learning). We've added a citation to Vyas et al. (2023) to the revised version of the paper.

Reviewer comment (Theory):
In Section 3.1, can the authors explain in what sense does NGD have "favorable convergence over GD" in theory? Does the paper refer to an appropriate source for this statement? Is this “strict” in some way? (Otherwise it's unclear why testing only NGD is sufficient to draw conclusions on GD).

Author response:
We apologize for not clearly citing this statement, which is detailed in [1] ([paragraph below Equation (11)](https://arxiv.org/pdf/1905.10961.pdf), $n$ is the number of data points and $\lambda_0$ is defined in footnote 6 of our paper):

> Compared to analogous bounds for gradient descent [Du et al., 2018a, Oymak and Soltanolkotabi, 2019, Wu et al., 2019], we improve the maximum allowable learning rate from $\mathcal{O}(1/n)$ to $\mathcal{O}(1)$ and also get rid of the dependency on $\lambda_0$. Overall, NGD (Theorem 3) gives an $\mathcal{O}(\lambda_0 / n)$ improvement over gradient descent.

We will make this more clear in the final version of the paper.

Reviewer comment (Experiments):
In Section 3.1 (Page 6), why do the authors use only a subset of the data for the CNN regression experiment? (This should be explained in the paper as well).

Author response:
We use only a subset of the data for the regression experiment in Section 3.1. due to computational constraints at large widths. For example, for the WRN computing one data point for large widths and $N=2000$ requires roughly 3 GPU days. 

More specifically, the JVPs and VJPs required to multiply with the NTK matrix require looping over the data in batches, performing forward and backward passes. For NNs with very large widths those operations quickly become expensive (note that for fully-connected NNs the number of parameters scale as $O(P)=O(m^L)$ with width, and JVP and VJP costs scale with number of parameters). Since we want to explore large widths, our only choice to reduce cost is to use less data. To make sure that this does not corrupt our experiments, we compared the conditions for two different sub-set sizes in Fig. 3 which confirms that using more data leads to weaker linearity (as predicted by the theory too).

Reviewer comment (Experiments):
Still in Section 3.1, does it make sense to somehow plot in Figure 2 (e.g., with a horizontal line) the $\lambda_{\min}$ and $C'$ in the limit where the width $\to\infty$ (i.e., in the NTK regime)? Is there perhaps a way to compute that theoretically rather than numerically? I believe it may complete the picture for this experiment.

Author response:
We can compute $\lambda_{\text{min}}$ theoretically for the toy architectures, since the limiting kernel is known. 
The Gram matrix converges to $\mathbf{G}^\infty_{ij} = \mathbf{x}_i^T \mathbf{x}_j\frac{\pi - \arccos(\mathbf{x}_i^T \mathbf{x}_j)}{2\pi}$ (see Zhang et al. (2019), Definition 1). We've computed the minimum eigenvalue of this matrix and added it as a dashed horizontal line to the plot (see Figure 2a in the revised paper). For Figure 2b, we can draw $C'$ in the limit of infinite width, since as we write in the paper below eqn. 8, $C \to 0$. We've drawn $C'=0.5$ as a black dashed line in Figure 2 to indicate the threshold for which the Jacobian is sufficiently stable for rapid convergence of NGD.