Paper: An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regression (ICLR-2024)

Reviewer comment (Theory):
The only potential flaw that I can see is the use of the result from [Simon2021]. Although the Gaussian Design Ansatz is widely accepted in machine learning theory, the equation (10) comes from [Simon2021] which used the rotational invariance of Gaussian, which seems to be too strong for a realistic case. I wonder if the argument of this paper can still be applicable in a more realistic setup.

Author response:
Regarding the reliance on the results of Simon et al. (2021): We note that the framework we take as our starting point is not only due to them: the KRR generalization equations we use (Eqs. 8-10 in our work) are obtained also in the seven other works cited at the top of Section 2.3, all of which have appeared in major peer-reviewed venues. Some of these, including Hastie et al. (2019), permit the data to be slightly different from Gaussian (e.g., covariates independent and *sub*-Gaussian), which is a bit weaker than a strict Gaussian design ansatz, but still a strong assumption. We will reword to make it clearer that these works are also sources for the framework.

Regarding the use of the Gaussian design ansatz (GDA) at all (or perhaps some slightly weaker cousin): while we agree that one would ideally avoid its use, the GDA has become a standard part of the ML theory toolkit because (a) predictions generated using the GDA often match real, non-Gaussian experiment remarkably well, and (b) the GDA permits one to obtain sharp closed-form expressions in many cases. To our knowledge, no starting assumption for the feature distribution of KRR which is both more realistic and equally tractable is known.

Reviewer comment (Theory):
As stated in the abstract: "We analyze the cost of overfitting under a Gaussian universality ansatz using recently derived (non-rigorous) risk estimates in terms of the task eigenstructure." I guess that this is addressing the use of the result from [Simon2021], which was rejected by a conference before.

Author response:
Regarding the reliance on the results of Simon et al. (2021): We note that the framework we take as our starting point is not only due to them: the KRR generalization equations we use (Eqs. 8-10 in our work) are obtained also in the seven other works cited at the top of Section 2.3, all of which have appeared in major peer-reviewed venues. Some of these, including Hastie et al. (2019), permit the data to be slightly different from Gaussian (e.g., covariates independent and *sub*-Gaussian), which is a bit weaker than a strict Gaussian design ansatz, but still a strong assumption. We will reword to make it clearer that these works are also sources for the framework.

Regarding the use of the Gaussian design ansatz (GDA) at all (or perhaps some slightly weaker cousin): while we agree that one would ideally avoid its use, the GDA has become a standard part of the ML theory toolkit because (a) predictions generated using the GDA often match real, non-Gaussian experiment remarkably well, and (b) the GDA permits one to obtain sharp closed-form expressions in many cases. To our knowledge, no starting assumption for the feature distribution of KRR which is both more realistic and equally tractable is known.

Reviewer comment (Theory):
The worst-case bounds achieved in this agnostic model are tight only in unrealistic cases. Specifically, the worst-case ratio $\mathcal{E}_0$ should only be representative of the $\text{ratio}$ for a real ML task if the Bayes-Optimal estimator is the always-zero function. While it is reasonable to give guarantees in the worst case, the fact that the worst case is achieved by an unrealistic setting makes it questionable whether this theory can precisely characterize when overfitting is benign or catastrophic. Practical takeaways will require understanding the gap between $\mathcal{E}_0$ and $\text{ratio}$, which harms significance, though it is not a deal breaker.

Author response:
*Criticism*: The worst-case bounds achieved in this agnostic model are tight only in unrealistic cases.

*Our Response* (elaborated below): This is not the case. They are per-instance tight much more generally, qualitatively per-instance tight in broad generality, and since they are worst-case tight they also shed light on what can be said only based on the data distribution.

*Elaboration*:
It is true that in the paragraph you mention, at the top of page 5 (we assume you meant before eq 12, rather than before eq 11), we give the signal-less case ($v_i=0$) to argue tightness.  But this is not *necessary* for tightness, it is merely the easiest argument to show the worst-case tightness in (12).  We regret that this created an impression as if this is the only situation where $\tilde{C} \geq \mathcal{E_0}$.  In fact, in the paragraph following (12) we provide a lower bound we should have emphasized much more:
$\tilde{C} \geq \mathcal{E_0} \frac{\sigma^2}{\tilde{R}(\hat{f}_{\delta^*})}$.

This shows that $\tilde{C} \rightarrow \mathcal{E_0}$ (i.e. $\mathcal{E_0}$ precisely captures the cost of overfitting) whenever $\tilde{R}(\hat{f}_{\delta^*}) \rightarrow \sigma^2$, i.e. whenever optimal tuned ridge is consistent (approaches the Bayes error).  Thus, the upper bound is tight much more generally (recall that much of the prior literature focuses on such a consistent setting, and that this is also generically the case when $n\rightarrow\infty$ while the source distribution is fixed).  We should have said this explicitly and emphasized this.

It is true that when $\tilde{R}(\hat{f}_{\delta^*}) > \sigma^2$, i.e. optimal tuned ridge is far from Bayes optimality, we could have $\mathcal{E_0}$ be a loose upper bound on $\tilde{C}$.  But even in this case, $\mathcal{E_0}$ still captures the *qualitative* (if not quantitative) noisy overfitting behavior in the following way:
* If $\mathcal{E_0} = 1$ we have benign overfitting ($\tilde{C}=1$) regardless of the target, since $\tilde{C} \leq \mathcal{E_0}$ for any target.
* If $\mathcal{E_0} \to \infty$ and $\sigma^2>0$ then we have catastrophic overfitting ($\tilde{C}\to\infty$) regardless of the target, since $R(\hat{f}_{\delta^*}) < \infty$ and 

$\tilde{C} \geq \mathcal{E_0} \frac{\sigma^2}{\tilde{R}(\hat{f}_{\delta^*})}$.  

* If $1 < \mathcal{E_0} < \infty$ and $\sigma^2>0$, then overfitting is either benign or tempered ($1\le\tilde{C}<\infty$), since $\tilde{C} \leq \mathcal{E_0}<\infty$.

All the above is independent of the target, as long as $\sigma^2>0$ (ie we are in a noisy setting, which is where we typically talk about overfitting).  Thus, $\mathcal{E_0}$ *does* capture the *qualitative* overfitting behavior (up to the indistinguishability of benign and tempered fitting in the third case).  This discussion should of course be much more explicit in the paper.

Finally, according to eq (12), $\mathcal{E_0}$ is the tightest possible “agnostic” bound on $\tilde{C}$. i.e. that does not depend on the target $f_*$ but only on the data distribution. Part of the goal of the paper is to study just how much such an agnostic view can characterize overfitting behavior, and this is the answer—it’s tight in some senses (discussed above), but is inherently not tight in a generic quantitative sense (and this is just reality, it is not due to loose analysis).

Disclaimer: all the above discussion is of course based on the eigenframework and valid under it.

Reviewer comment (Theory):
The paper lacks a clear discussion of the mathematics involved, with no proof sketches or intuitive justifications for why the theorems should be true, and no good intuitions given for key mathematical properties of a kernel used in the theorems. The absence of proof sketches slightly harms confidence in the correctness of the results, even though they feel believable.

Author response:
*Lack of clear discussion of the mathematics involved.*
The formal proofs are indeed a bit technical, and follow by analyzing the estimate from Eq. (10). We will try to add more intuitive explanations in the main text.

Reviewer comment (Theory):
There is a particularly problematic lack of intuitive explanation regarding the **Effective Rank** of a matrix. On page 5, Definition 2 introduces $r_k$ and $R_k$, both called the effective rank, which can differ by up to a quadratic factor, without providing any intuitive understanding of the difference between them or justification for calling them "ranks" when rank usually does not involve counting from the $k^{th}$ eigenvalue.

Author response:
*Regarding the comment on the effective ranks.* 
In our work, we used the definitions from the seminal work of Bartlett et al. (2020), which were also used in many other works on benign overfitting. The term “effective ranks” is used in the literature to describe these quantities. We understand your criticism regarding this term, but we preferred to be consistent with prior works.  Perhaps a better way to write the limit $\lim_{k \to \infty} \frac{k}{r_k}$ that avoids $r_k$ altogether is:  
$$\lim_{k \to \infty} \frac{k \lambda_k}{\sum_{i>k} \lambda_i}$$  
This perhaps gives a better intuitive sense of how spectral decay controls $\mathcal{E_0}$ and thus overfitting behavior.

Reviewer comment (Theory):
Theorems throughout the paper depend on both $r_k$ and $R_k$, especially $r_k$, but their real meaning is unclear. For example, Theorem 6 on page 8 classifies the kernel matrix into benign, tempered, or catastrophic settings depending on the limit $\lim_{k\rightarrow\infty} \frac{k}{r_k}$. It is unclear if this limit characterizes spectrum decay rate, or what kinds of spectra make the limit large or small. This lack of discussion significantly reduces clarity of the takeaways from the paper.

Author response:
*Regarding the comment on the effective ranks.* 
In our work, we used the definitions from the seminal work of Bartlett et al. (2020), which were also used in many other works on benign overfitting. The term “effective ranks” is used in the literature to describe these quantities. We understand your criticism regarding this term, but we preferred to be consistent with prior works.  Perhaps a better way to write the limit $\lim_{k \to \infty} \frac{k}{r_k}$ that avoids $r_k$ altogether is:  
$$\lim_{k \to \infty} \frac{k \lambda_k}{\sum_{i>k} \lambda_i}$$  
This perhaps gives a better intuitive sense of how spectral decay controls $\mathcal{E_0}$ and thus overfitting behavior.

Reviewer comment (Theory):
$\mathcal{E}_0$ only matches $\text{ratio}$ if the optimal KRR estimator is the always zero estimator. This is explained at the top of page 5 before equation (11), where $\text{ratio} = \mathcal{E}_0$ if $v_i = 0$ for all $i = 1,...,\infty$ in the expansion $f^*(x) = \sum_{i=1}^\infty v_i \phi_i(x)$, implying $f^*(x) = 0$ everywhere.

Author response:
*Criticism*: The worst-case bounds achieved in this agnostic model are tight only in unrealistic cases.

*Our Response* (elaborated below): This is not the case. They are per-instance tight much more generally, qualitatively per-instance tight in broad generality, and since they are worst-case tight they also shed light on what can be said only based on the data distribution.

*Elaboration*:
It is true that in the paragraph you mention, at the top of page 5 (we assume you meant before eq 12, rather than before eq 11), we give the signal-less case ($v_i=0$) to argue tightness.  But this is not *necessary* for tightness, it is merely the easiest argument to show the worst-case tightness in (12).  We regret that this created an impression as if this is the only situation where $\tilde{C} \geq \mathcal{E_0}$.  In fact, in the paragraph following (12) we provide a lower bound we should have emphasized much more:
$\tilde{C} \geq \mathcal{E_0} \frac{\sigma^2}{\tilde{R}(\hat{f}_{\delta^*})}$.

This shows that $\tilde{C} \rightarrow \mathcal{E_0}$ (i.e. $\mathcal{E_0}$ precisely captures the cost of overfitting) whenever $\tilde{R}(\hat{f}_{\delta^*}) \rightarrow \sigma^2$, i.e. whenever optimal tuned ridge is consistent (approaches the Bayes error).  Thus, the upper bound is tight much more generally (recall that much of the prior literature focuses on such a consistent setting, and that this is also generically the case when $n\rightarrow\infty$ while the source distribution is fixed).  We should have said this explicitly and emphasized this.

It is true that when $\tilde{R}(\hat{f}_{\delta^*}) > \sigma^2$, i.e. optimal tuned ridge is far from Bayes optimality, we could have $\mathcal{E_0}$ be a loose upper bound on $\tilde{C}$.  But even in this case, $\mathcal{E_0}$ still captures the *qualitative* (if not quantitative) noisy overfitting behavior in the following way:
* If $\mathcal{E_0} = 1$ we have benign overfitting ($\tilde{C}=1$) regardless of the target, since $\tilde{C} \leq \mathcal{E_0}$ for any target.
* If $\mathcal{E_0} \to \infty$ and $\sigma^2>0$ then we have catastrophic overfitting ($\tilde{C}\to\infty$) regardless of the target, since $R(\hat{f}_{\delta^*}) < \infty$ and 

$\tilde{C} \geq \mathcal{E_0} \frac{\sigma^2}{\tilde{R}(\hat{f}_{\delta^*})}$.  

* If $1 < \mathcal{E_0} < \infty$ and $\sigma^2>0$, then overfitting is either benign or tempered ($1\le\tilde{C}<\infty$), since $\tilde{C} \leq \mathcal{E_0}<\infty$.

All the above is independent of the target, as long as $\sigma^2>0$ (ie we are in a noisy setting, which is where we typically talk about overfitting).  Thus, $\mathcal{E_0}$ *does* capture the *qualitative* overfitting behavior (up to the indistinguishability of benign and tempered fitting in the third case).  This discussion should of course be much more explicit in the paper.

Finally, according to eq (12), $\mathcal{E_0}$ is the tightest possible “agnostic” bound on $\tilde{C}$. i.e. that does not depend on the target $f_*$ but only on the data distribution. Part of the goal of the paper is to study just how much such an agnostic view can characterize overfitting behavior, and this is the answer—it’s tight in some senses (discussed above), but is inherently not tight in a generic quantitative sense (and this is just reality, it is not due to loose analysis).

Disclaimer: all the above discussion is of course based on the eigenframework and valid under it.

Reviewer comment (Theory):
$r_k$ and $R_k$ can differ by up to a quadratic factor, as noted on page 5 just below equation (13).

Author response:
*Regarding the comment on the effective ranks.* 
In our work, we used the definitions from the seminal work of Bartlett et al. (2020), which were also used in many other works on benign overfitting. The term “effective ranks” is used in the literature to describe these quantities. We understand your criticism regarding this term, but we preferred to be consistent with prior works.  Perhaps a better way to write the limit $\lim_{k \to \infty} \frac{k}{r_k}$ that avoids $r_k$ altogether is:  
$$\lim_{k \to \infty} \frac{k \lambda_k}{\sum_{i>k} \lambda_i}$$  
This perhaps gives a better intuitive sense of how spectral decay controls $\mathcal{E_0}$ and thus overfitting behavior.

Reviewer comment (Theory):
Usually, the rank of a matrix has no notion of starting to count only from the $k^{th}$ eigenvalue, even for "smooth" notions of rank such as the intrinsic dimension $tr(K(K+\lambda I)^{-1})$ or stable rank $\frac{\tr(A)}{\|\|A\|\|_2}$.

Author response:
*Regarding the comment on the effective ranks.* 
In our work, we used the definitions from the seminal work of Bartlett et al. (2020), which were also used in many other works on benign overfitting. The term “effective ranks” is used in the literature to describe these quantities. We understand your criticism regarding this term, but we preferred to be consistent with prior works.  Perhaps a better way to write the limit $\lim_{k \to \infty} \frac{k}{r_k}$ that avoids $r_k$ altogether is:  
$$\lim_{k \to \infty} \frac{k \lambda_k}{\sum_{i>k} \lambda_i}$$  
This perhaps gives a better intuitive sense of how spectral decay controls $\mathcal{E_0}$ and thus overfitting behavior.

Reviewer comment (Writing):
For people unfamiliar with the literature, some concepts are hard to understand, for example, “omniscient risk estimate” and the “cost of overfitting.” Can you provide more intuition about these parts?

Author response:
Regarding giving more intuition on terms used: As we mention in footnote 2, “omniscient risk estimate” is just a term that Wei et al. (2022) gave to the estimate which we provide in Eq. (10). The “cost of overfitting” is the main new concept we introduce: it is defined formally in Definition 1, and intuitively it compares a predictor which is obtained as a result of extreme overfitting and interpolation to the “optimal” non-overfitting predictor, and thus captures the “cost” (in terms of test error blow-up) of this overfitting.

Reviewer comment (Theory):
For a purely theoretical paper, this work provides its main results directly, without mentioning the techniques or the difficulty of the proof. Highlighting the techniques used can help the readers understand this paper's technical contribution. For example, when comparing with Bartlett et al. (2020), the paper says, "Our proof technique is completely different and simpler since we have a closed-form expression." This, however, is not helpful enough because it does not provide any equation or concrete comparison to show what the closed-form expression is and where the key difference is.

Author response:
The proofs follow by analyzing the estimate from Eq. (10), which gives a closed-form expression for the risk. We will add a short discussion about it.

Reviewer comment (Theory):
Theorem 2 and 3 depend on $R_k$, while Theorem 4, 5, 6 depend on $r_k$. What’s the connection of these results, or are they independent?

Author response:
These results are not directly comparable, except for the fact that we can use the relationship $r_k \leq R_k \leq r_k^2$ which we mention in page 5 after Eq. (13).

Reviewer comment (Theory):
Although assumption 1 is commonly used in the literature, I’m still curious whether the analysis can work for some different assumptions. What key property of the Gaussian distribution is used in the proof?

Author response:
Strict Gaussianity isn’t actually required – the same eigenframework holds for a variety of distributions that are “close enough to Gaussian” (e.g. sub-Gaussian covariates, or “random-looking” feature vectors that satisfy some delocalization property). The key intuition is that these distributions form an equivalence class in that all are asymptotically described by the same eigenframework. The Gaussian distribution just happens to be the member of this class which is easiest to work with, so it is canonically chosen as the defining member. Empirically, it appears that feature distributions in real kernel tasks also lie in this Gaussian equivalence class (or at least close to it). This seems reasonable, but it is not yet understood why this is.