Paper: Noisy Interpolation Learning with Shallow Univariate ReLU Networks (ICLR-2024)

Reviewer comment (Novelty):
Comparison with (Kornowski et al. 2023) requires more discussion, especially in terms of the technical tools.

Author response:
We thank the reviewer for the positive feedback on the work! We will add a more detailed comparison with Kornowski et. al. [1] and a discussion on the topics raised by the reviewer.

Reviewer comment (Novelty):
Some references on the following topics in the related work are missing:  
- Neural networks, the representer theorem, splines, Banach spaces  
- Benign overfitting papers

Author response:
We thank the reviewer for the positive feedback on the work! We will add a more detailed comparison with Kornowski et. al. [1] and a discussion on the topics raised by the reviewer.

Reviewer comment (Novelty):
It appears that this paper can be regarded as a subsequent work to Boursier & Flammarion (2023). The connection is evident as the pivotal lemma (Lemma 2.1) employed in this paper is directly drawn from Boursier & Flammarion (2023). Furthermore, the neural network model studied in this paper aligns with the one extensively examined in Boursier & Flammarion (2023). Consequently, the technical innovation in this paper seems somewhat limited in this regard.

Author response:
* **Novelty relative to Boursier and Flammarion [1]:** Indeed, we study the same model as in Boursier and Flammarion [1].  This is also the same model studied in several other papers (e.g. Severese et. al. [2], Ergen and Pilanci [4], Hanin [5], and many more). The main novelty is in the question we ask, which is completely different: none of the previous papers, including Boursier and Flammarion [1], asked how such a model overfits, i.e. how it behaves when interpolating noisy data. In particular, our main results do not appear in any way in previous papers. To answer this novel question we indeed build on a characterization given by Boursier and Flammarion [1], but this is just a component of our analysis.

Reviewer comment (Novelty):
It's important to note that this paper adopts a one-dimensional perspective, assuming $x$ to be a single dimension, confined within the range $x\sim[0,1]$. In contrast, Kornowski et al. (2023) considered high-dimensional $x$. Another differentiating factor is that this paper primarily focuses on characterizing the asymptotic behavior of population error and reconstruction error, while Kornowski et al. (2023) delved into an analysis that extends beyond the asymptotic realm.

Author response:
* **Limiting to one dimension:** This is indeed a limitation, which is shared also by many other papers studying low-norm ReLU networks (e.g. Boursier and Flammarion [1], Hanin [5], Severevse et. al. [2], Debarre et. al. [3] and others mentioned above) and in-fact also the Kornowski et. al. [6] paper mentioned.  Kornowski et. al.’s treatment of higher dimensions is limited to simulations or the extreme case $d \gg n^2$ (in which case learning is essentially linear and the overfitting is benign—see paragraph 2 on page 1 and also the footnote)--they also do not have theoretical analysis for *fixed* dimensions beyond $d=1$.  Going beyond one dimension is very interesting, and we believe it is possible, but has proven to be difficult not only for us but also for the entire community.

Reviewer comment (Theory):
In Theorem 4, which addresses catastrophic overfitting for $L_p$ with $p\geq 2$, it is essential to note that only a specific case with $f^*=0$ is considered. To enhance the depth and relevance of the analysis, it would be particularly intriguing to explore this phenomenon with a more general $f^*$.

Author response:
* **Considering** $f^* \equiv 0$ **for catastrophic overfitting:** As this is a lower bound, we find it strongest to show that overfitting is catastrophic even in the easiest case, i.e. $f^* \equiv 0$. Our result should be interpreted as: "Even for such a simple target function $f^* \equiv 0$, in the presence of noise, the interpolating predictor exhibits catastrophic behavior."  Overfitting will also be catastrophic with any Lipschitz target and non-zero noise.

Reviewer comment (Theory):
The results in this paper seem to consider the asymptotic regime that sample size $n$ goes to infinity. I was wondering if there is a rate for this asymptotic convergence.

Author response:
We seriously thank the reviewer for their constructive feedback! Below we address the questions.

* In line with other papers on interpolation learning, we indeed provide an asymptotic characterization (benign/tempered/catastrophic).  Obtaining a finite sample bound on the convergence (or divergence) would require additional assumptions (which we preferred not to get into in order to keep the paper focused—e.g. our results do not depend on the Lipschitz constant, only on it being finite) but should be possible using standard concentration inequalities and a more elaborate analysis.

Reviewer comment (Theory):
I was wondering if the min-norm interpolation considered in the paper can be naturally reached based on the implicit bias of some simple algorithms. For example, the authors mentioned that Shevchenko et al. showed similar spikes as the current paper. Does their algorithm exactly lead to the same implicit bias here?

Author response:
As we explain on page 2, the precise implicit bias of GD (and even Shevchenko et. al. [1]’s analysis) does not guarantee that min-norm interpolators are reached. However, we note that the main property of min-norm interpolators that determines the overfitting behavior in our analysis is the formation of “spikes”, and such spikes are empirically observed while interpolation with GD in Shevchenko et. al. [1]. Thus, studying min-norm interpolators with these properties is a good starting point. Moreover, considering the min-norm interpolator is natural when training with small weight decay.