Paper: More is Better: when Infinite Overparameterization is Optimal and Overfitting is Obligatory (ICLR-2024)

Reviewer comment (Theory):
How are the approximations of the train and test errors on page 4 derived. I think it is crucial to at least provide intuitions about them and quantify the approximation errors. This is because much of the results of the paper rely on those approximations.

Author response:
We thank the reviewer for their evaluation, noting in particular the clear writing and effort to crystallize common wisdom. To the first weakness: we opted for a streamlined, nonrigorous approach for our derivation of the RF eigenframework, adopting the philosophy typical in statistical physics where one makes justified approximations and then checks against numerics at the end of the calculation. As we discuss at various points, we expect our RF eigenframework to become exact in a proportional limit (e.g. precisely the limit taken by Bach et al (2023)). Extracting approximation errors for finite $n, k$ is not a trivial task, but we know of other work in progress that intends to do this.

Reviewer comment (Theory):
I'm unsure about the significance of the theoretical results. Mainly because the authors claim that the test error becomes better as $n$, $k$ increase because the RF model converges to the kernel regression limit. However, it is a bit hard to believe since kernel machines (RFs, GPs) are in general worse than their finite-width NNs counterpart (kernel machines = linear models => no representation learning). As the authors mentioned on page 5, Thm. 1 works since intuitively RF -> kernel machine; the latter is the ideal limiting process. If this is the case, doesn't this imply that GPs should be better than standard NNs?

Author response:
To the question about significance: we find that [the limiting KRR] is better than [finite RF regression]. However, this does not mean that KRR is arbitrarily good! Some other model type (e.g., a neural network) can certainly be better, and we make no claims about performance relative to any model outside of RF regression. To further clarify the relationship between these model classes: RF regression trains only the last layer, does not learn features, and becomes KRR in the limit of infinitely-many features. Infinite-width NNs are a special case of KRR and do not learn features, so long as one scales init + lr in the NTK fashion. Finite-width NNs learn features and are not simply related to RF regression or KRR. Heuristically speaking, we show that infinite-width NNs are better than RF regression, but whether infinite-width NNs are better than finite-width NNs remains an open problem.

Reviewer comment (Theory):
The authors mentioned that "we do not want to take a fixed noise variance" and then argued that it is better to scale it proportional to the uncaptured signal. I'm not sure if this is realistic since $\sigma^2$ is the aleatoric (irreducible) uncertainty inherent in the data (think about the noise resulting from some measurement error in a sensor). How do you handle this, then? Maybe you should instead argue using the language of GPs, and thus give you the freedom to talk about _epistemic uncertainty_, which decreases as $n$ increases?

Author response:
2. We refer the reviewer to Appendix E discussing the motivation for scaling down noise with test error. Ultimately, this is just a theoretical convenience: in practice, $\sigma^2$ will indeed be whatever finite quantity it actually is, and $n$ will be finite. For the purposes of theoretical calculation, we want to take $n$ to infinity, and so we’re left with the question of what to do with $\sigma^2$ to preserve behavior similar to what we started with. We argue that keeping $\sigma^2$ unchanged as $n \rightarrow \infty$ actually changes the behavior in undesirable ways, and one instead ought to scale it down in the manner described. This is just a trick for writing down a better model for our learning task and does not actually amount to an assumption that the noise is really $n$-dependent.

Reviewer comment (Theory):
How should one relate Thm. 1 and Fig. 1? In the former, the test error should always decrease, but in the latter, the test error saturates. Note that, I think Fig. 1 is the "correct" behavior: kernel machines are not powerful and thus the test error is bounded away from zero, higher than DNN's. (As for the train error, this seems fine since infinite-width => linearity => quadratic loss => global optimum is achievable.)

Author response:
3. Ah, there is no conflict: in Figure 1, the test error decreases continuously, and it *also* approaches an asymptote. Consider, for example, the function $x^{-1} + 1$ as $x$ grows: it always decreases, yet never decreases below its asymptote of $1$. Theorem 1 only promises that the gains from increasing $k$ are positive, but they will become smaller and smaller as $k$ grows.

Reviewer comment (Novelty):
How does this paper relate to the work of [Bowman & Montufar (NeurIPS 2022)](https://arxiv.org/abs/2206.02927), which also studies the large-width large-data regime?

Author response:
4. Like us, Bowman and Montufár (2022) compare the learning of models with finite sample size and width to those of infinite sample size and width. They focus on training dynamics: their claim is of the flavor that “even at finite sample size and width, the evolution of the learned function looks like the infinite case early in training.” This is a similar message to the (empirical) message of the “[deep bootstrap](https://arxiv.org/abs/2010.08127).” By contrast, we study generalization rather than the training trajectory, focus on the learned function at the end of training rather than early in training.

Reviewer comment (Novelty):
While the analysis for the power-law eigenstructure is solid and concrete, the other part seems not so surprising as the theory. The heuristic derivation is rather straightforward, and it seems that the authors just substituted heuristically-derived terms for the kernel matrix of RF (eqs. 26-30) into a well-known expression of the generalization error (eqs. 13, 14). Although providing a unified formulation itself is a very curious and interesting attempt, this part feels less novel. Please refer to my Question (i) for more details.

Author response:
(i) We understand the reviewer’s comments to mean that, while they find our study of overfitting for powerlaw tasks (Section 5) solid and concrete, they find the RF eigenframework in Section 4 to be less surprising. This is fair: we are generally less interested in the RF eigenframework itself than in what new things we can conclude from it (namely the “more is better” result of Theorem 1). If some prior work had given a simple derivation and statement of this general result, we would have been happy just citing it. As nobody had done that to the best of our knowledge and it seems generally useful, we decided to. We also note that the simplicity with which a result may be derived is not necessarily indicative of the result’s importance, and the result reached here is indeed fairly general :)

Reviewer comment (Theory):
The definition of overfitting in this paper is unique, and its relation to the ridgeless case seems unclear. Considering a ridgeless case seems appropriate when talking about overfitting to training samples. Indeed, Mei & Montanari 2019 have reported a certain condition where generalization error is minimized in a ridgeless setting, which was one of their main contributions. In contrast, the current study does not directly analyze the ridgeless case but rather allows a non-zero fitting ratio in its analysis. For example, while Theorem 2 contemplates the optimal ridge, it is not explicitly stated when the optimal ridge becomes zero. It is also unclear whether the noise condition in Corollary 1 works as a necessary and sufficient condition where the ridgeless case gives the best performance (i.e., where the generalization error is minimized).

Author response:
(ii) We understand that the reviewer finds unclear the relation of our “fitting-ratio-based” notion of overfitting to the simpler, more common approach of simply studying the ridgeless case (and perhaps comparing to the optimally-regularized case). To clear this up: the model is ridgeless ($\delta = 0$) precisely when the fitting ratio is zero ($\mathcal{R}\_{tr/te} = 0$). When we find that $\mathcal{R}_{tr/te} = 0$ is optimal, we are precisely finding that zero ridge is optimal. We realize that we missed saying this explicitly, and so we have added some appropriate text where we introduce the fitting ratio. The reviewer’s interpretation of Corollary 1 is correct (and indeed answers the reviewer’s concern regarding Theorem 2), and the text will say so.

Reviewer comment (Theory):
Regarding Weakness (i), eq. (13) seems to be a starting point to derive the omniscient risk estimate. However, the authors provide insufficient explanation on how and which paper has derived this equation. As far as I know, Sollich, 2001, and Bordelon et al., 2020 obtained this expression for KRR using heuristic derivations. Did the other cited works — Jacot et al., 2020a; Simon et al., 2021; Loureiro et al., 2021; Dobriban & Wager, 2018; Wu & Xu, 2020; Hastie et al., 2020; Richards et al., 2021 — explicitly give this expression (13) as well? If so, under what conditions does each study derive it? Are they mathematically rigorous? Because this equation is the starting point of all other analyses, it should be clearly and carefully described.

Author response:
(Qi): All the cited works derive the same eigenframework. Right now, the best (i.e. most general while still rigorous) source for this eigenframework is probably [Cheng and Montanari (2022)](https://arxiv.org/abs/2210.08571), and this could be taken as the canonical starting point if one is required. We have edited the text to say this.

Reviewer comment (Evaluation):
There is a lack of clarity regarding what the authors are trying to claim in Figure 2. Do you intend to claim that overfitting is obligatory only for the noiseless case ($\sigma_{rel}^2=0$)? For $\sigma_{rel}^2>0$, the minimal test MSE is obtained for non-zero fitting ratios, which implies that the best generalization occurs in non-overfitting situations, making overfitting seem *not* obligatory. This point is unclear.

Author response:
(Qii): Yes, this is the intended claim. The most important curve is the one with zero noise, because this is the true, unmodified task. The point of adding noise is that Corollary 1 predicts that adding noise should make overfitting no longer obligatory, and we are testing that. Comparing to noisy curves is also a way of stressing that the unmodified tasks seem to not have significant noise.

Reviewer comment (Theory):
In Section 5, the authors focus on the KRR limit (k → ∞). It is unclear at what point the finite-k case becomes hard to analyze.

Author response:
(Qiii): This is a good question. The finite-k case is algebraically harder to handle: from the start, one has the two coupled equations (2) defining implicit constants instead of just one. To undertake a similar powerlaw analysis, one might write test error in terms of $\gamma$, then write $\gamma$ in terms of the train-test ratio, and finally optimize over the train-test ratio. The test error in the finite-k case would be significantly more complicated than in the infinite-k case (Lemmas 4 and 10), but this might work. It may be a doable thing for a future paper.

Reviewer comment (Theory):
On Page 6, the definition of the fitting ratio seems to be incorrect. It should not be \( R_{tr/te} := E_{te} / E_{tr} \) but rather \( E_{tr} / E_{te} \).

Author response:
(Qiv): Thanks for catching this. We’ve fixed it.