Original: S TRONG INDUCTIVE BIASES PROVABLY PREVENT HARMLESS INTERPOLATION
Revised: S TRONG INDUCTIVE BIASES PROVABLY PREVENT HARMLESS INTERPOLATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Classical wisdom suggests that estimators should avoid fitting noise to achievegood generalization.
Revised: Classical wisdom suggests that estimators should avoid fitting noise to achievegood generalization.

Original: In contrast, modern overparameterized models can yieldsmall test error despite interpolating noise — a phenomenon often called “benignoverfitting” or “harmless interpolation”.
Revised: In contrast, modern overparameterized models can yieldsmall test error despite interpolating noise — a phenomenon often called “benignoverfitting” or “harmless interpolation”.

Original: This paper argues that the degree to whichinterpolation is harmless hinges upon the strength of an estimator’s inductive bias,i.e., how heavily the estimator favors solutions with a certain structure: whilestrong inductive biases prevent harmless interpolation, weak inductive biases caneven require fitting noise to generalize well.
Revised: This paper argues that the degree to whichinterpolation is harmless hinges upon the strength of an estimator’s inductive bias,i.e., how heavily the estimator favors solutions with a certain structure: whilestrong inductive biases prevent harmless interpolation, weak inductive biases caneven require fitting noise to generalize well.

Original: Our main theoretical result establishestight non-asymptotic bounds for high-dimensional kernel regression that reflect thisphenomenon for convolutional kernels, where the filter size regulates the strengthof the inductive bias.
Revised: Our main theoretical result establishestight non-asymptotic bounds for high-dimensional kernel regression that reflect thisphenomenon for convolutional kernels, where the filter size regulates the strengthof the inductive bias.

Original: We further provide empirical evidence of the same behaviorfor deep neural networks with varying filter sizes and rotational invariance.
Revised: We further provide empirical evidence of the same behaviorfor deep neural networks with varying filter sizes and rotational invariance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: According to classical wisdom (see e.g., Hastie et al.
Revised: According to classical wisdom (see, e.g., Hastie et al.

Original: , an estimator that fits noise suffersfrom “overfitting”, and cannot generalize well.
Revised: , an estimator that fits noise suffersfrom “overfitting”, and cannot generalize well.

Original: A typical solution is to prevent interpolation, that is,stopping the estimator from achieving zero training error and thereby fitting less noise.
Revised: A typical solution is to prevent interpolation, that is,stopping the estimator from achieving zero training error and thereby fitting less noise.

Original: For example,one can use ridge regularization or early stopping for iterative algorithms to obtain a model thathas training error close to the noise level.
Revised: For example,one can use ridge regularization or early stopping for iterative algorithms to obtain a model thathas training error close to the noise level.

Original: However, large overparameterized models such as neuralnetworks seem to behave differently: even on noisy data, they may achieve optimal test performanceat convergence after interpolating the training data (Nakkiran et al., 2021; Belkin et al., 2019a) — aphenomenon referred to as harmless interpolation (Muthukumar et al., 2020) or benign overfitting(Bartlett et al., 2020) and often discussed in the context of double descent (Belkin et al., 2019a).
Revised: However, large overparameterized models such as neuralnetworks seem to behave differently: even on noisy data, they may achieve optimal test performanceat convergence after interpolating the training data (Nakkiran et al., 2021; Belkin et al., 2019a) — aphenomenon referred to as harmless interpolation (Muthukumar et al., 2020) or benign overfitting(Bartlett et al., 2020) and often discussed in the context of double descent (Belkin et al., 2019a).

Original: To date, we lack a general understanding of when interpolation is harmless for overparameterizedmodels.
Revised: To date, we lack a general understanding of when interpolation is harmless for overparameterizedmodels.

Original: In this paper, we argue that the strength of an inductive bias critically influences whetheran estimator exhibits harmless interpolation.
Revised: In this paper, we argue that the strength of an inductive bias critically influences whetheran estimator exhibits harmless interpolation.

Original: An estimator with a strong inductive bias heavilyfavors “simple” solutions that structurally align with the ground truth (such as sparsity or rotationalinvariance).
Revised: An estimator with a strong inductive bias heavilyfavors “simple” solutions that structurally align with the ground truth (such as sparsity or rotationalinvariance).

Original: Based on well-established high-probability recovery results of sparse linear regression(Tibshirani, 1996; Candes, 2008; Donoho & Elad, 2006), we expect that models with strongerinductive biases generalize better than ones with weaker inductive biases, particularly from noiselessdata.
Revised: Based on well-established high-probability recovery results of sparse linear regression(Tibshirani, 1996; Candes, 2008; Donoho & Elad, 2006), we expect that models with strongerinductive biases generalize better than ones with weaker inductive biases, particularly from noiselessdata.

Original: In contrast, the effects of inductive biases are much less studied for interpolators of noisy data.
Revised: In contrast, the effects of inductive biases are much less studied for interpolators of noisy data.

Original: Recently, Donhauser et al.
Revised: Recently, Donhauser et al.

Original: (2022) provided a first rigorous analysis of the effects of inductivebias strength on the generalization performance of linear max- ℓ p -margin/min- ℓ p -norm interpolatorsinterpolators.
Revised: (2022) provided a first rigorous analysis of the effects of inductive biasstrength on the generalization performance of linear max- ℓ p -margin/min- ℓ p -norm interpolators.

Original: In particular, the authors prove that a stronger inductive bias (small p → 1 ) not onlyenhances a model’s ability to generalize on noiseless data, but also increases a model’s sensitivity tonoise — eventually harming generalization when interpolating noisy data.
Revised: Inparticular, the authors prove that a stronger inductive bias (small p → 1 ) not solely enhances amodel’s ability to generalize on noiseless data, but also increases a model’s sensitivity to noise —eventually harming generalization when interpolating noisy data.

Original: As a consequence, theirresult suggests that interpolation might not be harmless when the inductive bias is too strong.
Revised: As a consequence, their resultsuggests that interpolation might not be harmless when the inductive bias is too strong.

Original: In this paper, we confirm the hypothesis and show that strong inductive biases indeed prevent harmlessinterpolation, while also moving away from sparse linear models.
Revised: In this paper, we confirm the hypothesis and show that strong inductive biases indeed prevent harmlessinterpolation, while also moving away from sparse linear models.

Original: As one example, we consider datawhere the true labels nonlinearly only depend on input features in a local neighborhood, and vary thestrength of the inductive biases via the filter size of convolutional kernels or shallow convolutionalneural networks — small filter sizes encourage functions that depend nonlinearly only on localneighborhoods of the input features.
Revised: As one example, we consider datawhere the true labels nonlinearly only depend on input features in a local neighborhood, and vary thestrength of the inductive biases via the filter size of convolutional kernels or shallow convolutionalneural networks — small filter sizes encourage functions that depend nonlinearly only on localneighborhoods of the input features.

Original: As a second example, we also investigate classification for
Revised: As a second example, we also investigate classification for

Original: rotationally invariant data, where we encourage different degrees of rotational invariance for neural
Revised: rotationally invariant data, where we encourage different degrees of rotational invariance for neural

Original: networks.
Revised: networks.

Original: In particular,
Revised: In particular,

Original: • we prove a phase transition between harmless and harmful interpolation that occurs byvarying the strength of the inductive bias via the filter size of convolutional kernels for kernelregression in the high-dimensional setting (Theorem 1).• we further show that not only is interpolation harmless for weak inductive biases, but thatpartially fitting the observation noise is in fact necessary (Theorem 2). • we show the same phase transition experimentally for neural networks with two commoninductive biases: varying convolution filter size, and rotational invariance enforced via data augmentation (Section 4).
Revised: • we prove a phase transition between harmless and harmful interpolation that occurs byvarying the strength of the inductive bias via the filter size of convolutional kernels for kernel regression in the high-dimensional setting (Theorem 1).• we further show that, for weak inductive biases, not only is interpolation harmless butpartially fitting the observation noise is in fact necessary (Theorem 2).• we show the same phase transition experimentally for neural networks with two commoninductive biases: varying convolution filter size, and rotational invariance enforced via dataaugmentation (Section 4).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We now discuss three related lines of  work and explain how the theoretical results cannot reflect thephase transition between harmless and harmful interpolation for high-dimensional kernel learning.
Revised: We now discuss three groups of related work and explain how their theoretical results cannot reflectthe phase transition between harmless and harmful interpolation for high-dimensional kernel learning.

Original: Low-dimensional kernel learning: Many recent papers (Bietti et al., 2021;Favero et al., 2021;
Revised: Low-dimensional kernel learning: Many recent works (Bietti et al., 2021;

Original: Bietti, 2022; Cagnetta et al., 2022) prove statistical rates for kernel regression with convolutionalkernels in low-dimensional regimes - however, these bounds crucially rely on ridge regularization.
Revised: Favero et al., 2021; Bietti,2022; Cagnetta et al., 2022) prove statistical rates for kernel regression with convolutional kernels inlow-dimensional settings, but crucially rely on ridge regularization.

Original: In general, one cannot expect harmless interpolation forkernel regression for such kernels in thelow-dimensional setting (Rakhlin & Zhai, 2019; Mallinar et al., 2022; Buchholz, 2022) - positiveresults only exist  for very specific adaptive spiked kernels (Belkin et al., 2019b).
Revised: In general, one cannot expectharmless interpolation for such kernels in the low-dimensional regime (Rakhlin & Zhai, 2019; Mallinar et al., 2022; Buchholz, 2022); positive results exist only for very specific adaptive spikedkernels (Belkin et al., 2019b).

Original: Furthermore,techniques developed for low dimensional regimes (see e.g., Schölkopf et al.
Revised: Furthermore, techniques developed for low-dimensional settings (see,e.g., Schölkopf et al.

Original:  usually sufferfrom a curse of dimensionality; that is, become vacuous in high-dimensional settings when thedimension is allowed to grow with n .
Revised: (2018)) usually suffer from a curse of dimensionality, that is, the bounds becomevacuous in high-dimensional settings where the input dimension grows with the number of samples.

Original: High-dimensional kernel learning: Several papers in the literature (Liang et al., 2020; McRae et al.
Revised: High-dimensional kernel learning: One line of research (Liang et al., 2020; McRae et al., 2022; Liang& Rakhlin, 2020;

Original: ,2022; Liang & Rakhlin, 2020; Liu et al., 2021) tackle high-dimensional kernel learning and provenon-asymptotic bounds using advanced high-dimensional random matrix concentration tools in(El Karoui, 2010).
Revised: Liu et al., 2021) tackles high-dimensional kernel learning and proves non-asymptoticbounds using advanced high-dimensional random matrix concentration tools from El Karoui (2010).

Original: However, their results heavily rely on a bounded Hilbert norm assumption, that isa natural in the low-dimensional regime.
Revised: However, those results heavily rely on a bounded Hilbert norm assumption.

Original: However, as pointed out in (Donhauser et al., 2021b), suchan assumption is misleading in the high-dimensional regime.
Revised: This assumption isnatural in the low-dimensional regime, but misleading in the high-dimensional regime, as pointed outin Donhauser et al. (2021b).

Original: Another line of research (Ghorbani et al., 2021; 2020; Mei et al., 2021; Ghosh et al., 2022; Misiakiewicz & Mei, 2021; Mei et al., 2022) gives precise asymptotic characterizations of the risk of thekernel regression estimator in specific settings where we have access to the kernel eigenfunctions andeigenvalues.
Revised: Another line of research (Ghorbani et al., 2021; 2020; Mei et al., 2021; Ghosh et al., 2022; Misiakiewicz & Mei, 2021; Mei et al., 2022) asymptotically characterizes theprecise risk of kernel regression estimators in specific settings with access to a kernel’s eigenfunctionsand eigenvalues.

Original: However,  to investigate how varying the filter size of the convolution kernel affectstherisk of the  kernel regression estimator, these asymptotic results are insufficient.
Revised: However, these asymptotic results are insufficient to investigate how varying the filtersize of a convolutional kernel affects the risk of a kernel regression estimator.

Original: In contrast to theabove papers, we prove tight non-asymptotic matching upper and lower bounds for high-dimensionalkernel learning which precisely capture the phase transition described in Section 3.2.
Revised: In contrast to both linesof research, we prove tight non-asymptotic matching upper and lower bounds for high-dimensionalkernel learning which precisely capture the phase transition described in Section 3.2.

Original: Related work on overfitting of structured interpolators: Several papers question the generality ofharmless interpolation for models which incorporate strong structural assumptions.
Revised: Overfitting of structured interpolators: Several works question the generality of harmless interpolationfor models that incorporate strong structural assumptions.

Original: Examples includestructural invariances enforced via data augmentation (Nishi et al., 2021), adversarial training (Riceet al., 2020; Kamath et al., 2021; Sanyal et al., 2021;
Revised: Examples include structures enforcedvia data augmentation (Nishi et al., 2021), adversarial training Rice et al., 2020; Kamath et al.

Original: Donhauser et al., 2021a; Sanyal et al. , 2021),neural network architectures (Li et al., 2021), pruning-based sparsity (Chang et al., 2021) and sparselinear models (Wang et al., 2022; Muthukumar et al., 2020; Chatterji & Long, 2022).
Revised: ,2021; Sanyal et al., 2021; Donhauser et al., 2021a), neural network architectures (Li et al., 2021),pruning-based sparsity (Chang et al., 2021), and sparse linear models (Wang et al., 2022; Muthukumaret al., 2020; Chatterji & Long, 2022).

Original: In this paper,we continue this line of research and offer a new theoretical perspective that characterizes wheninterpolation is expected to be harmless.
Revised: In this paper, we continue that line of research and offer a newtheoretical perspective to characterize when interpolation is expected to be harmless.

Original:  T HEORETICAL RESULTS
Revised:  T HEORETICAL RESULTS

Original: For convolutional kernels, a small filter size induces a strong bias towards estimators that dependnonlinearly on the input features only via a small patch in the input.
Revised: For convolutional kernels, a small filter size induces a strong bias towards estimators that dependnonlinearly on the input features only via a small patch in the input.

Original: This section analyzes the effectofthe filter size (as an examplefor inductive bias) on the degree of harmless interpolation for kernelridge regression.
Revised: This section analyzes the effectof filter size (as an example inductive bias) on the degree of harmless interpolation for kernel ridgeregression.

Original: For this purpose, we derive and compare tight non-asymptotic bias and variancebounds for optimally ridge-regularized and minimum-norm interpolators as a function of the filtersize (Theorem 1).
Revised: For this purpose, we derive and compare tight non-asymptotic bias and variance boundsas a function of filter size for min-norm interpolators and optimally ridge-regularized estimators(Theorem 1).

Original: Furthermore,  for small filter sizes, we not only prove that harmless interpolation occurs (Theorem 1, butthat fitting the noise(at lest partially) is even necessary to some degree toachieve optimal test performance (Theorem 2).
Revised: Furthermore, we prove for large filter sizes that not only does harmless interpolationoccur (Theorem 1), but fitting some degree of noise is even necessary to achieve optimal testperformance (Theorem 2).

Original: 3. S ETTING
Revised: 3. S ETTING

Original: We study kernel regression with a (cyclic) convolutional kernel in a high-dimensional setting wherethe number of training samples n scales with the dimension of the input data d as n ∈ Θ( d ℓ ) .
Revised: We study kernel regression with a (cyclic) convolutional kernel in a high-dimensional setting wherethe number of training samples n scales with the dimension of the input data d as n ∈ Θ( d ℓ ) .

Original: Weuse the same setting as in previous works on high-dimensional kernel learning, e.g. (Misiakiewicz &
Revised: We usethe same setting as in previous works on high-dimensional kernel learning such as Misiakiewicz &

Original: Mei, 2021) and assume that the training samples { x i , y i } ni =1 are i.i.d.
Revised: Mei (2021): we assume that the training samples { x i , y i } ni =1 are i.i.d.

Original: draws from a distribution withx i ∼ U ( {− 1 , 1 } d )  and y i 
Revised: draws from the distributionsx i ∼ U ( {− 1 , 1 } d ) , and y i =

Original: = f ∗ ( x i )
Revised: f ⋆ ( x i )

Original: + ϵ i with ground truth 
Revised: + ϵ i with ground truth f

Original: f ∗ and  ϵ ∼ N (0 , σ 2 ) .
Revised: ⋆ and noise ϵ ∼ N (0 , σ 2 ) .

Original: For simplicity ofexposition, we further assume that f ∗ ( x ) =
Revised: For simplicityof exposition, we further assume that f ⋆ ( x ) =

Original: x 1 x 2 · · · x L ∗ , with L ∗ specified in Theorem 1.
Revised: x 1 x 2 · · · x L ∗ , with L ∗ specified in Theorem 1.

Original: While the assumptions on the noise and ground truth can be easily extended by following the proofsteps in Section 5, generalizing the input data distribution is a difficult task.
Revised: While the assumptions on the noise and ground truth can be easily extended by following the proofsteps in Section 5, generalizing the feature distribution is challenging.

Original: Indeed, existing resultsthat establish precisecharacterizations for the risk  (see Section 2) crucially rely on the assumption thatthe input data distribution is hypercontractive – so far only shown to hold for few high-dimensionaldistributions, including the hypersphere (Beckner, 1992) and the discrete hypercube (Beckner, 1975)that we use in this paper.
Revised: Indeed, existing results thatestablish precise risk characterizations (see Section 2) crucially rely on hypercontractivity of thefeature distribution — an assumption so far only proven for few high-dimensional distributions,including the hypersphere (Beckner, 1992), and the discrete hypercube (Beckner, 1975) which weuse in this paper.

Original: It is essential to tightly control the empirical kernel matrix in Lemma 1 in
Revised: Hypercontractivity is essential to tightly control the empirical kernel matrix within

Original:  Section 5.
Revised: Lemma 3 in Section 5.

Original: Generalizations beyond this assumptionwould require the development of new tools inrandom matrix theory that we consider important future work.
Revised: Generalizations beyond this assumption require the development of new toolsin random matrix theory, which we consider important future work.

Original: We consider (cyclic) convolutional kernels with filter size q ∈ [1 , d ] of the form
Revised: We consider (cyclic) convolutional kernels with filter size q ∈ { 1 , . . . , d } of the form

Original: [Equation]
Revised: [Equation]

Original: where x ( k,q ) :=
Revised: where x ( k,q ) :=

Original: [ x mod( k,d ) · · · x mod( k + q − 1 ,d ) ] . andwe assume that κ : [ − 1 , 1] → R satisfies 111 standard regularity assumptions (seedetailed Assumption 3 in Appendix B)) that includes forinstance the exponential function.
Revised: [ x mod( k,d ) · · · x mod( k + q − 1 ,d ) ] , and κ : [ − 1 , 1] → R is a nonlinear function thatsatisfies standard regularity assumptions (see Assumption 1 in Appendix B) that hold for instance forthe exponential function.

Original: By decreasing the filter size q , we encourage the KRR solution todepend nonlinearly only on local neighborhoods instead of the entire input x .
Revised: Decreasing the filter size q restricts kernel regression solutions to dependnonlinearly only on local neighborhoods instead of the entire input x .

Original: We analyze the kernel ridge regression (KRR) estimator, which is the minimizer of the followingconvex optimization problem:
Revised: We analyze the kernel ridge regression (KRR) estimator, which is the minimizer of the followingconvex optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where H is the Reproducing Kernel Hilbert space (RKHS) over {− 1 , 1 } d generated by the convolutional kernel K in Equation (1), ∥·∥ H the corresponding norm and λ > [-
-]
Revised: where H is the Reproducing Kernel Hilbert space (RKHS) over {− 1 , 1 } d generated by the convolutional kernel K in Equation (1), ∥·∥ H the corresponding norm, and λ > [-
-]0 the ridge regularizationpenalty.

Original: the ridge regularizationpenalty.

Original: In the interpolation limit  λ → 0 , we obtain the minimum-RKHS-norm interpolator
Revised: In the interpolation limit ( λ → 0 ), we obtain the min-RKHS-norm interpolator

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we refer to ˆ f 0 as the kernel ridge regression estimator with λ = 0 .
Revised: For simplicity, we refer to ˆ f 0 as the kernel ridge regression estimator with λ = 0 .

Original: We evaluate all estimators with the expected population risk over the noise, defined by:
Revised: We evaluate all estimators with the expected population risk over the noise, defined as

Original: [Equation]
Revised: [Equation]

Original: := Variance

Original: [Equation]

Original: λ ) .

Original: 3. M AIN R ESULT
Revised: 3. M AIN R ESULT

Original: We now present tight upper and lower bounds for the prediction error of kernel regression estimatorsin the setting of Section 3.1.
Revised: We now present tight upper and lower bounds for the prediction error of kernel regression estimatorsin the setting from Section 3.1.

Original: The resulting rates hold for the high-dimensional regime, that is whenboth the ambient dimension d and filter size q scale with n .
Revised: The resulting rates hold for the high-dimensional regime, that is,when both the ambient dimension d and filter size q scale with n .

Original: We refer to Section 5 for the proof.
Revised: We defer the proof to Section 5.

Original: Theorem 1 (Non-asymptotic prediction error rates) .
Revised: Theorem 1 (Non-asymptotic prediction error rates) .

Original:  Assume a dataset and  kernel as described in Section 3.1 withnoise variance σ 2 and let ˆ f λ be the KRR estimate in Eq.
Revised: Let ℓ > 0 , β ∈ (0 , 1) , ℓ σ ∈ R . Assume a dataset and a kernel as described in Section 3.1, with the kernel satisfying Assumption 1.

Original: Further, let n ∈ Θ( d ℓ ) ,  filter size q ∈ Θ 
Revised: Assume further n ∈ Θ( d ℓ ) , the filter size q ∈ Θ (cid:0) d β (cid:1) , and σ 2 ∈ Θ( d − ℓ σ ) . Lastly, define ¯ δ :=

Original: [Equation]
Revised: ℓ − 1 β − ⌊ ℓ − 1 β ⌋ and δ :=

Original: , max { λ, 1 } ∈ Θ( d ℓλ ) and σ 2 ∈ Θ( d − ℓσ ) for some ℓ, β, ℓ λ , ℓ σ ∈ R and ℓ λ [-≥ 
 and define δ =-] ℓ  −  ℓ λ  −  1 β −⌊
Revised: ℓ − ℓ λ − 1 β − ⌊ ℓ − ℓ λ [-≥ 
 and define δ =-]− 1 β ⌋ for any ℓ λ . Then, with probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } uniformly over all ℓ λ ∈ [0 , ℓ − 1) , the KRR estimate ˆ f λ in Equation (2) with max { λ, 1 } ∈ Θ( d ℓ λ ) satisfies

Original: [Equation]
Revised: [Equation]

Revised: Further, for a ground truth f

Original: [Equation]
Revised: ⋆ ( x ) =

Original: ⌋ and ¯ δ = ℓ − 1 β −⌊ ℓ − 1 β ⌋ .
Revised: x 1 x 2 · · · x L ∗ with L ∗ ≤

Original: Then, for some universal constant c , we have with probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } , that

Original: [Equation]
Revised: [Equation]

Original: Further, for a ground truth f ∗ ( x ) = x 1 x 2 · · · x L ∗ , with L ∗ ≤ 1 + ⌊ ℓ − ℓ λ − 1 β ⌋ , with probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } , it also holds that:
Revised: , with probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } , we have

Original: [Equation]
Revised: [Equation]

Original: Finally, the ratesfor bias and variance are also valid for  ˆ f 0 in Eq.(3) by setting ℓ λ = 0 .
Revised: Finally, by setting ℓ λ = 0 , both rates also hold for the min-RKHS-norm interpolator ˆ f 0 in Eq.

Original: First note how the theorem statement reflects the usual intuition for the effect of noise and regularization strength on bias and variance via the parameters ℓ λ : with increasing ridge regularization ℓ λ (andthus increasing λ ), the bias increases and the variance decreases.
Revised: Note how the theorem statement reflects the usual intuition for the effect of noise and regularizationstrength on bias and variance via the parameter ℓ λ : With increasing ridge regularization ℓ λ (and thusincreasing λ ), the bias increases and the variance decreases.

Original: Similarly, as noise increases (thusdecreasing ℓ σ ), the variance increases.
Revised: Similarly, as noise increases (and thusdecreasing ℓ σ ), the variance increases.

Original: Phase transition as a function of β : In our discussion, wewould like to focus on the impact of the filtersize q ∈ Θ (cid:0) d β (cid:1) on the risk (sum of bias and variance), entering via the growth rate β .
Revised: Phase transition as a function of β : In the following, we focus on the impact of the filter sizeq ∈ Θ (cid:0) d β (cid:1) on the risk (sum of bias and variance) via the growth rate β .

Original: Recalling thata small filter size (small values of β ) corresponds to a strong inductive bias and vice versa, Figure 1illustrates how the strength of inductive bias affects generalization. 
Revised: Recalling that a small filtersize (small values of β ) corresponds to a strong inductive bias, and vice versa, Figure 1 demonstrateshow the strength of inductive bias affects generalization. For illustration, we choose the ground truthf ⋆ ( x ) =

Original:  Specifically, Figure 1a shows therates for the minimum-RKHS-norm interpolator ˆ f 0 and the optimally ridge-regularized estimator ˆ f λ opt ,where  λ optis chosen to minimize the expected population risk Risk ( ˆ f λ opt ) .
Revised: x 1 x 2 so that the assumption on L ∗ is satisfied for all β . Specifically, Figure 1a shows therates for the min-RKHS-norm interpolator ˆ f 0 and the optimally ridge-regularized estimator ˆ f λ opt ,where we choose λ opt to minimize the expected population risk Risk ( ˆ f λ opt ) .

Original: Furthermore, Figure 1bdepicts the (statistical) bias and variance of the interpolator ˆ f 0 .
Revised: Furthermore, Figure 1bdepicts the (statistical) bias and variance of the interpolator ˆ f 0 .

Original: The ground truth f

Original: ⋆ ( x ) = x 1 x 2 usedfor the corresponding experiments is chosen such that the assumption on L ∗ is satisfied for all β .

Original: Thefollowing phase transition occurs at the threshold β ∗ ∈ (0 , 1)  at which the rates ofthe statistical biasand variance in Theorem 1 match:
Revised: At the threshold β ∗ ∈ (0 , 1) , implicitlydefined as the β at which the rates of statistical bias and variance in Theorem 1 match, we can observethe following phase transition:

Original: • For β < β ∗ , that is, for strong inductive biases, the rates in Figure 1a for the optimally ridge-
Revised: • For β < β ∗ , that is, for strong inductive biases, the rates in Figure 1a for the optimally ridge-

Original: regularized estimator ˆ f λ opt are strictly better than the ones for the corresponding interpolator
Revised: regularized estimator ˆ f λ opt are strictly better than the ones for the corresponding interpolator

Original: ˆ f 0 .
Revised: ˆ f 0 .

Original: In other words, we are observing harmful interpolation .
Revised: In other words, we are observing harmful interpolation .

Original: • For β > β ∗ , that is, for weak inductive biases, the rates in Figure 1a of the optimally ridgeregularized estimator ˆ f λ opt and the minimum-RKHS-norm interpolator ˆ f 0 match.
Revised: • For β > β ∗ , that is, for weak inductive biases, the rates in Figure 1a of the optimally ridge-regularized estimator ˆ f λ opt and the min-RKHS-norm interpolator ˆ f 0 match.

Original: Hence, we
Revised: Hence, we

Original: observe harmless interpolation .
Revised: observe harmless interpolation .

Original: In the following theorem, we  show that interpolation is not only harmless for β > β ∗ , but theoptimally ridge-regularized estimator ˆ f λ optin fact necessarily fits part of the noise; that is, it has atraining error strictly below the noise level.
Revised: In the following theorem, we additionally show that interpolation is not only harmless for β > β ∗ , butthe optimally ridge-regularized estimator ˆ f λ opt necessarily fits part of the noise and has a training errorstrictly below the noise level.

Original: Further, we show that when interpolation is harmful in Figure 1a, i.e., when β < β ∗ , the training error of the optimally ridge-regularized model approachesthe noise level.
Revised: In contrast, we show that when interpolation is harmful in Figure 1a,that is, when β < β ∗ , the training error of the optimally ridge-regularized model approaches thenoise level.

Original: Theorem 2 (Informal - training error) .
Revised: Theorem 2 (Training error (informal)) .

Original: Let λ opt be such that the expected population risk Risk ( ˆ f λ opt ) is minimal and let β ∗ be the threshold where the bias and variance bounds in Theorem 1 are of the same order for the interpolator ˆ f 0 (setting ℓ λ = 0 ).
Revised: Let λ opt be such that the expected population risk Risk ( ˆ f λ opt ) is minimal, and let β ∗ be the threshold where the bias and variance bounds in Theorem 1 are of the same order for the interpolator ˆ f 0 (setting ℓ λ = 0 ).

Original: Then, the expected training error converges in probability to
Revised: Then, the expected training error converges in probability:

Original: [Equation]
Revised: [Equation]

Original: with c β < 1 for any β > β ∗ .
Revised: where c β < 1 for any β > β ∗ .

Original: We refer to Appendix D for the proof and a more general statement.
Revised: We refer to Appendix D for the proof and a more general statement.

Original: Bias-variance trade-off: We now discuss how the phase transition arises from a (statistical) biasand variance trade-off of the minimum-RKHS-norm interpolator as a function of β , reflected in 
Revised: Bias-variance trade-off: We conclude by discussing how the phase transition arises from a (statistical)bias and variance trade-off for the min-RKHS-norm interpolator as a function of β , reflected in interpolating optimally regularized 0. β ∗ [Figure] ﬁlter

Original: [Figure]
Revised: [Figure]

Revised: (cid:0) d

Revised: [Figure]

Revised: (cid:1) -0.

Revised: [Figure]

Original:  e rr o r r a t e e x p o n e n t α ↓
Revised: − ‘ σ ‘ e rr o r r a t e e x p o n e n t α ↓

Original: [Figure]
Revised: [Figure]

Revised: variance biasvar.

Revised: lower bound 0.

Revised: β ∗ β s.t. ﬁlter

Revised: [Figure]

Revised: Θ (cid:0) d β (cid:1)

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: a

Original:  Theorem 1 when setting ℓ λ = 0 and illustrated in Figure 1b.
Revised: [Figure] n [Figure] Theorem 1 when setting ℓ λ = 0 and illustrated in Figure 1b.

Original: While the statistical bias monotonicallydecreases with decreasing β (i.e. increasing strength of the inductive bias), the variance follows amultiple descent curve with increasing minima as β decreases.
Revised: While the statistical bias monotonicallydecreases with decreasing β (i.e., increasing strength of the inductive bias), the variance follows amultiple descent curve with increasing minima as β decreases.

Original: Hence, analogous to the observationsin (Donhauser et al., 2022) for linear max- ℓ p -margin/min- ℓ p -norm interpolators, the interpolatorachieves the optimal performance at a β ∈ (0 , 1) , and therefore at a moderate inductive bias.
Revised: Hence, analogous to the observationsin Donhauser et al. for linear max- ℓ p -margin/min- ℓ p -norm interpolators, the interpolatorachieves the optimal performance at a β ∈ (0 , 1) , and therefore at a moderate inductive bias.

Original: Finally,we note that a multiple descent curve for the variance has been previously observed in (Liang et al.
Revised: Finally,we note that Liang et al.

Original: ,2020) that arises with varying input dimension with no connection to structural biases.
Revised: (2020) previously observed a multiple descent curve for the variance, but asa function of input dimension and without any connection to structural biases.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now empirically study whether the phase transition phenomenon that we prove for kernelregression persists for deep neural networks with feature learning.
Revised: We now empirically study whether the phase transition phenomenon that we prove for kernelregression persists for deep neural networks with feature learning.

Original: More precisely, we presentcontrolled experiments to investigate how the strength of a CNN’s inductive bias influences ifinterpolating noisy data is harmless.
Revised: More precisely, we presentcontrolled experiments to investigate how the strength of a CNN’s inductive bias influences ifinterpolating noisy data is harmless.

Original: In practice, the inductive bias of a neural network varies by wayof design choices such as the architecture (e.g., convolutional vs. fully-connected vs. graph neuralnetworks) or the training procedure (e.g., data augmentation, adversarial training).
Revised: In practice, the inductive bias of a neural network varies byway of design choices such as the architecture (e.g., convolutional vs. fully-connected vs. graphnetworks) or the training procedure (e.g., data augmentation, adversarial training).

Original: We focus on twoexamples: convolutional filter size that we vary via the architecture, and rotational invariance viadata augmentation.
Revised: We focus on twoexamples: convolutional filter size that we vary via the architecture, and rotational invariance viadata augmentation.

Original: To isolate the effects of filter size and provide conclusive results, we use datasetswhere we know a priori that the ground truth exhibits a simple structure that matches the networks’inductive bias.
Revised: To isolate the effects of inductive biases and provide conclusive results, we usedatasets where we know a priori that the ground truth exhibits a simple structure that matches thenetworks’ inductive bias.

Original: See Appendix E for experimental details.
Revised: See Appendix E for experimental details.

Original: Analogous to ridge regularization for kernels, we use early stopping as a mechanism to prevent noisefitting.
Revised: Analogous to ridge regularization for kernels, we use early stopping as a mechanism to prevent noisefitting.

Original: Our experiments compare optimally early-stopped CNNs to their interpolating versions.
Revised: Our experiments compare optimally early-stopped CNNs to their interpolating versions.

Original: Thishighlights a trend that mirrors our theoretical results: the stronger the inductive bias of a neuralnetwork, the more harmful interpolation becomes.
Revised: Thishighlights a trend that mirrors our theoretical results: the stronger the inductive bias of a neuralnetwork grows, the more harmful interpolation becomes.

Original: These results suggest exciting future work: toprove this trend for models with feature learning.
Revised: These results suggest exciting future work:proving this trend for models with feature learning.

Original: 4. E FFECT OF FILTER SIZE ON CNN S FOR SYNTHETIC IMAGE DATASET
Revised: 4. F ILTER SIZE OF CNN S ON SYNTHETIC IMAGES

Original: In a first experiment, we study the impact of filter size on the generalization of interpolating CNNs.
Revised: In a first experiment, we study the impact of filter size on the generalization of interpolating CNNs.

Original: As a reminder, small filter sizes yield functions that depend nonlinearly only on local neighborhoodsof the input features.
Revised: As a reminder, small filter sizes yield functions that depend nonlinearly only on local neighborhoodsof the input features.

Original: To clearly isolate the effects of filter size, we choose a special architectureon a synthetic classification problem such that the true label function is indeed a CNN with smallfilter size.
Revised: To clearly isolate the effects of filter size, we choose a special architectureon a synthetic classification problem such that the true label function is indeed a CNN with smallfilter size.

Original: In particular, we generate images of size 32 × 32 containing scattered circles (negativeclass) and crosses (positive class) with size at most 5 × 5 .[-
-]
Revised: In particular, we generate images of size 32 × 32 containing scattered circles (negativeclass) and crosses (positive class) with size at most 5 × 5 .[-
-]

Original: Thus, decreasing filter size up to 5 × [-
-]
Revised: Thus, decreasing filter size down to 5 × [-
-]5corresponds to a stronger inductive bias.

Original: See Figure
Revised: Motivated by our theory, we hypothesize that interpolatingnoisy data is harmful with small filter size, but harmless when using a large filter size.

Original: [Figure]
Revised: [Figure]

Original: t e s t
Revised: t e s t

Original: e rr o r
Revised: e rr o r

Original: [Figure]
Revised: [Figure]

Original: s
Revised: noisy [Figure] clean

Original: [Figure]
Revised: [Figure]

Original: s  e t t r a i n e rr o r
Revised: s ub s e t t r a i n e rr o r

Original: ( o p t i m a l e a r l y s t
Revised: ( o p t . e a r l y s t

Original: o pp i n g )
Revised: o pp i n g )

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: corresponds to a stronger inductive bias.

Original: Motivated by our theory, we hypothesize that interpolatingnoisy data is harmful with small filter size but harmless when using a large filter size.

Original: Training setup: In the experiments, we use CNNs with a single convolution layer, followed byglobal max poolingover the spatial dimensions and two dense layers.
Revised: Training setup: In the experiments, we use CNNs with a single convolutional layer, followed by globalspatial max pooling and two dense layers.

Original: We train those CNNs withdifferent filter sizes on 200 training samples (either noiseless or with 20% label flips) to minimize thelogistic loss and achieve zero training error.
Revised: We train those CNNs with different filter sizes on 200training samples (either noiseless or with 20% label flips) to minimize the logistic loss and achievezero training error.

Original: We repeat all experiments over 5 random datasets with15 optimizations per dataset and filter size, and report the average 0 - 1 -error for 100 k test samplesper dataset.
Revised: We repeat all experiments over 5 random datasets with 15 optimizations perdataset and filter size, and report the average 0 - 1 -error for 100 k test samples per dataset.

Original: Lastly, we provide a detailed discussion on the choice of hyperparameters and moreexperimental details in Appendix E.1.
Revised: For a detaileddiscussion on the choice of hyperparameters and more experimental details, see Appendix E.1.

Original: Results: First, the noiseless error curves (blue) in Figure 2a confirm the common intuition thatthe strongest inductive bias (matching the ground truth) at size 5 yields the lowest test error.
Revised: Results: First, the noiseless error curves (blue) in Figure 2a confirm the common intuition thatthe strongest inductive bias (matching the ground truth) at size 5 yields the lowest test error.

Original: Moreinterestingly, for 20% training noise (yellow), Figure 2a reveals a similar phase transition as Theorem 1and confirms our hypothesis: Models with weak inductive biases (large filter sizes) exhibit harmlessinterpolation, as indicated by the matching test error of interpolating (yellow solid) and optimallyearly-stopped (yellow dashed) models.
Revised: Moreinterestingly, for 20% training noise (yellow), Figure 2a reveals a similar phase transition as Theorem 1and confirms our hypothesis: Models with weak inductive biases (large filter sizes) exhibit harmlessinterpolation, as indicated by the matching test error of interpolating (yellow solid) and optimallyearly-stopped (yellow dashed) models.

Original: In contrast,  models with strong inductive biases (small filtersizes) suffer from an increasing gap in test errors when interpolating versus using optimal earlystopping as filter size decreases.
Revised: In contrast, as filter size decreases, models with a stronginductive bias (small filter sizes) suffer from an increasing gap in test errors when interpolatingversus using optimal early stopping.

Original:  Figure 2b, on the other hand, reflects the dual perspective of thephase transition as presented in Theorem 2: for small filter size, optimally early-stopped modelsentirely avoid fitting training noise such that the training error on the noisy training subset equals100% , while  large filter sizes require interpolation of the noise.
Revised: Furthermore, Figure 2b reflects the dual perspective of the phasetransition as presented in Theorem 2 under optimal early stopping: models with a small filter sizeentirely avoid fitting training noise, such that the training error on the noisy training subset equals100% , while models with a large filter size interpolate the noise.

Original: Difference to double descent: One might suspect that our empirical observations simply reflectanother form of double descent (Belkin et al., 2019a).
Revised: Difference to double descent: One might suspect that our empirical observations simply reflectanother form of double descent (Belkin et al., 2019a).

Original: As a CNN’s filter size increases (inductive biasbecomes weaker), so does its number of parameters andhence the degree of overparameterization.
Revised: As a CNN’s filter size increases (inductivebias becomes weaker), so does the number of parameters and degree of overparameterization.

Original: Hence, double descentpredicts harmless interpolation due to model size for weak inductive biases.
Revised: Thus,double descent predicts vanishing benefits of regularization due to model size for weak inductivebiases.

Original: Nevertheless, we argue that the phenomenon we observe here is distinct, and discuss two forms ofevidence in Appendix E.3:
Revised: Nevertheless, we argue that the phenomenon we observe here is distinct, and provide anextended discussion in Appendix E.3.

Original: For one, we choose sufficiently large models and tune hyperparametersto ensure that all models interpolate and yield small training loss, even for the minimum filter sizeand 20% training noise.
Revised: In short, we choose sufficiently large networks and tune theirhyperparameters to ensure that all models interpolate and yield small training loss, even for filtersize 5 and 20% training noise.

Original: Additionally, we repeat  the experiments with two and four times as manyconvolutional filters.
Revised: To justify this approach, we repeat a subset of the experiments whilesignificantly increasing the convolutional layer width.

Original: Even for significantly larger models, overfitting still increases with strongerinductive biases, and the phase transition between harmless and harmful interpolation persists.
Revised: As the number of parameters increases fora fixed filter size, double descent would predict that the benefits of optimal early stopping vanish. However, we observe that our phenomenon persists. In particular, for filter size 5 (strongest inductivebias), the test error gap between interpolating and optimally early-stopped models remains large.

Original: 4. R OTATIONAL INVARIANCE OF W IDE R ESNET ON SATELLITE IMAGES
Revised: 4. R OTATIONAL INVARIANCE OF W IDE R ESIDUAL N ETWORKS ON SATELLITE IMAGES

Original: In a second experiment, we investigate rotational invariance as an inductive bias for CNNs whenevertrue labels are independent of an image’s rotation.
Revised: In a second experiment, we investigate rotational invariance as an inductive bias for CNNs whenevertrue labels are independent of an image’s rotation.

Original: Our experiments enforce inductive biases byfitting models on multiple rotated versions of an original training set, effectively performing data
Revised: Our experiments enforce inductive biases by fittingmodels on multiple rotated versions of an original training dataset, effectively performing varying

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: t e s t
Revised: t e s t

Original: e rr o r
Revised: e rr o r

Original: [Figure]
Revised: [Figure]

Revised: noisy

Original: [Figure]
Revised: [Figure]

Original: error
Revised: clean

Original: [Figure]
Revised: [Figure]

Original: 1 4# rotations 0% 38% 100% s ub s e t t r a i n e rr o r
Revised: s ub s e t t r a i n e rr o r

Original: ( o p t i m a l e a r l y s t
Revised: ( o p t . e a r l y s t

Original: o pp i n g )
Revised: o pp i n g )

Original: [Figure]
Revised: [Figure]

Original:  augmentation.
Revised: degrees of data augmentation.

Original: As an example dataset with a rotationally invariant ground truth, we classify satelliteimages from the EuroSAT dataset (Helber et al., 2018) into 10 types of land usage.
Revised: As an example dataset with a rotationally invariant ground truth, weclassify satellite images from the EuroSAT dataset (Helber et al., 2018) into 10 types of land usage.

Original: Because the truelabels are independent of image orientation, we expect rotational invariance to be a particularly fittinginductive bias for this task.
Revised: Because the true labels are independent of image orientation, we expect rotational invariance to be aparticularly fitting inductive bias for this task.

Original: Training and test setup: For computational reasons, we subsample the original EuroSAT training setto 7680 raw training and 10 k raw test samples.
Revised: Training and test setup: For computational reasons, we subsample the original EuroSAT training setto 7680 raw training and 10 k raw test samples.

Original: In the noisy case, we replace 20% of the raw traininglabels by a different label chosen uniformly at random.
Revised: In the noisy case, we replace 20% of the raw traininglabels with a wrong label chosen uniformly at random.

Original: We then vary the strength of the inductivebias towards rotational invariance by augmenting the dataset with an increasing numberk of  rotatedversions of  each training sample of equal-spaced angles spanning 360 ◦ , plus a random offset.
Revised: We then vary the strength of the inductivebias towards rotational invariance by augmenting the dataset with an increasing number of k rotatedversions of itself. For each sample, we use k equal-spaced angles spanning 360 ◦ , plus a random offset.

Original: Notethat training noise applies before rotations, so that all rotated versions of the same image share thesame label.
Revised: Note that training noise applies before rotations, so that all rotated versions of the same image sharethe same label.

Original: We then center-crop all rotated images such that they only contain valid pixels.
Revised: We then center-crop all rotated images such that they only contain valid pixels.

Original: In allexperiments, we fit Wide Residual Networks (Zagoruyko & Komodakis, 2016) on the augmentedtraining set for 5 different network initializations.
Revised: In allexperiments, we fit Wide Residual Networks (Zagoruyko & Komodakis, 2016) on the augmentedtraining set for 5 different network initializations.

Original: We evaluate the 0 - 1 -error on the randomly rotatedtest samples to avoid distribution shift effects from image interpolation.
Revised: We evaluate the 0 - 1 -error on the randomly rotatedtest samples to avoid distribution shift effects from image interpolation.

Original: All random rotations are thesame for all experiments and stay fixed throughout training.
Revised: All random rotations are thesame for all experiments and stay fixed throughout training.

Original: See Appendix E.2 for more experimentaldetails.
Revised: See Appendix E.2 for more experimentaldetails.

Revised: Lastly, we perform additional experiments with larger models to differentiate from doubledescent; see Appendix E.3 for the results and further discussion.

Original: Results: Similar to the previous section
Revised: Results: Similar to the previous subsection

Original: , Figure 3a corroborates our hypothesis under rotationalinvariance: stronger inductive biases result in lower test errors on noiseless data, but an increasedgap between the test errors of interpolating and optimally early-stopped models.
Revised: , Figure 3a corroborates our hypothesis under rotationalinvariance: stronger inductive biases result in lower test errors on noiseless data, but an increasedgap between the test errors of interpolating and optimally early-stopped models.

Original: In contrast to filtersize, the phase transition is more abrupt; invariance to 180 ◦ rotations already prevents harmlessinterpolation.
Revised: In contrast to filtersize, the phase transition is more abrupt; invariance to 180 ◦ rotations already prevents harmlessinterpolation.

Original: Figure 3b confirms this from a dual perspective, since all rotationally invariant modelscannot fit noisy samples for optimal generalization.
Revised: Figure 3b confirms this from a dual perspective, since all models with some rotationalinvariance cannot fit noisy samples for optimal generalization.

Original:  P ROOF OF THE MAIN RESULT
Revised:  P ROOF OF THE MAIN RESULT

Original: The proof of the main result, Theorem 1, is divided into two steps. First,we present with Theorem 3in Section 5.1  a  resultfor fixed-design that gives tight non-asymptotic matching upper and lowerbounds for the prediction error of general kernels.
Revised: The proof of the main result, Theorem 1, proceeds in two steps: First, Section 5.1 presents a fixeddesign result that yields matching upper and lower bounds for the prediction error of general kernelsunder additional conditions.

Original: Then, in Section 5.2, we proceed by showing thatthe assumptions in Theorem 3 hold.
Revised: Second, in Section 5.2, we show that the setting of Theorem 1 satisfiesthose conditions with high probability over dataset draws.

Original: Notation Assuming thatthe inputs are sampled i.i.d. from a data distribution ν , i.e. x i iid ∼ ν , we can express any continuous, positive semi-definite kernel function asthe following sum:
Revised: Notation Assuming that inputs are draws from a data distribution ν (i.e., x, x ′ ∼ ν ), we candecompose and divide any continuous, positive semi-definite kernel function as

Original: [Equation]
Revised: [Equation]

Original: where { ψ k } k> 0 is an orthonormal basis of the RKHSwith respect to the inner kernel induced byν : ⟨ f, g ⟩ ν :=
Revised: where { ψ k } k ≥ 1 is an orthonormal basis of the RKHS induced by ⟨ f, g ⟩ ν :=

Original: E x ∼ ν [ f ( x ) g ( x )] . Eigenfunction are sorted in descending order λ k ≥ λ k +1 .
Revised: E x ∼ ν [ f ( x ) g ( x )]and the eigenvalues λ k are sorted in descending order. In the following, we write [ · ] i,j to referto the entry in row i and column j of a matrix.

Original: Definethe empirical kernel matrix K ∈ R n × n , [ K ] ij = K
Revised: Then, we define the empirical kernel matrixfor K as K ∈ R n × n with [ K ] i,j :=

Original:  ( x i , x j ) , and analogously  K ≤ m and K >m .
Revised: K ( x i , x j ) , and analogously the truncated versions K ≤ mand K >m for K ≤ m and K >m , respectively.

Original: Furthermore, we make use of the matrices Ψ ≤ m ∈ R n × m , [ Ψ ≤ m ] il :=
Revised: Next, we utilize the matrices Ψ ≤ m ∈ R n × m with[ Ψ ≤ m ] i,l :=

Original: ψ l ( x i ) , and D ≤ m :=diag ( λ 1 , . . . , λ m ) .
Revised: ψ l ( x i ) , and D ≤ m := diag ( λ 1 , . . . , λ m ) .

Original: Define the squared kernel S1 ( x, x ′ ) 
Revised: We further use the squared kernel S ( x, x ′ ) :

Original: =  E z ∼
Revised: = 251 E z ∼

Original: ν
Revised: ν

Original: [ K ( x, z ) K ( z, x ′ )] , the truncatedversions of it S 1 >m , S1 ≤ m and  the corresponding empirical kernel matrices S , S ≤ m and S >m ∈251 R n × n .
Revised: [ K ( x, z ) K ( z, x ′ )] , its truncated versions S ≤ m and S >m , as well as the corresponding empiricalkernel matrices S , S ≤ m , S >m ∈ R n × n .

Original: We use µ min ( · ) and µ max ( · ) (or ∥·∥ ) to indicate respectively the smallest and the largesteigenvalue of a symmetric positive definite matrix. We use ⟨· , ·⟩ to indicate the euclidean innerproduct in R d .
Revised: Finally, we write µ min ( · ) and µ max ( · ) (or ∥·∥ ) to indicatethe min and max eigenvalue of a symmetric positive-definite matrix, respectively, and use ⟨· , ·⟩ toindicate the Euclidean inner product in R d .

Original: 5. G ENERALIZATION BOUND FOR FIXED - DESIGN
Revised: 5. G ENERALIZATION BOUND FOR FIXED - DESIGN

Original: We begin with a fixed design bound for the population error based on the following assumptions.
Revised: First, Theorem 3 provides tight fixed-design bounds for the prediction error.

Original: Assumption 1 (on the eigenfunctions) .
Revised: Theorem 3 (Generalization bound for fixed-design) .

Original: Assume that
Revised: Let K : X × X

Original: [Equation] Assumption 2 (on the ground truth) .
Revised: → R be a kernel, and{ ( x i , y i ) } ni =1 a dataset where y i =

Original: The ground truth is contained in the span of the first meigenfunctions. We now present in Theorem 3 the main result of this section, where we make use of the ratio
Revised: f ⋆ ( x i )

Original: [Equation]
Revised: + ϵ

Original: Theorem 3 (Generalization bound for fixed-design) . Given a kernel K : X × X → R , and a dataset { ( x i , y i ) } ni =1 , and y i = f ∗ ( x i ) + ϵ i with ground truth f ∗ .
Revised: i for zero-mean i.i.d.

Original: Let ˆ f λ be the KRR estimate in Eq. (2) for any λ ≥ 0 .
Revised: noise ϵ

Original: Let m ∈ Nbe such thatAssumption 1 holds. Then:
Revised: i with variance σ 2and a ground truth f ⋆ . For any m ∈ N such that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if we assume that, with the same m , also Assumption 2 is respected. We can write ˆ f λ ( x ) = (cid:80)
Revised: the KRR estimate ˆ f λ in Equation (2) for any λ ≥ 0 has a variance upper and lower bounded by

Original: [Equation]
Revised: [Equation]

Original: k ( x ) , where ψ k are the eigenfunctions of K . Then:
Revised: where τ 1 = min

Original: [Equation]
Revised: [Equation]

Original: where C = 8 max { 1 . 5 r, 1 } and τ 1 = min
Revised: Furthermore, for any ground truth which can be expressed as f ⋆ ( x ) = (cid:80) mk =1 a k ψ k ( x ) with a ∈ R m + and ψ k as defined via Equation (4) , the bias is upper and lower bounded by

Original: [Equation]
Revised: [Equation]

Original: We refer the reader to Appendix A for the proof, which is mostly algebra and essentially consists ofmultiple applications of the Woodbury matrix identity.
Revised: See Appendix A for the proof. Note that this result holds for any fixed-design dataset.

Original: The main ideasare derived from the proofsin (Bartlett et al., 2020; Tsigler & Bartlett, 2020) where the authors establish tight bounds for theminimum- ℓ 2 -norm interpolator on independent sub-Gaussian features.
Revised: We derivethe main ideas from the proofs in Bartlett et al. ; Tsigler & Bartlett (2020), where the authorsestablish tight bounds for the min- ℓ 2 -norm interpolator on independent sub-Gaussian features.

Original: Remark 1. (Comparison with (McRae et al., 2022))
Revised: Remark 1 (Comparison with McRae et al.

Original: The upper bound  in Theorem 1 in (McRae et al.,2022) for the bias depends on the sub-optimal term ∥ D − 1 / 2 ≤ m a ∥ /n , but also applies to more generalground truths.
Revised: The upper bound for the bias in Theorem 1 from McRae et al. (2022) depends on the suboptimal term ∥ D − 1 / 2 ≤ m a ∥ /n , but also applies to more generalground truths.

Original: We improve this bound in Theorem 3 and present a matching lower bound.
Revised: We improve that bound in Theorem 3 and present a matching lower bound.

Original: 5. P ROOF OF T HEOREM 1
Revised: 5. P ROOF OF T HEOREM 1

Original: Throughout the  proof, all statements hold for the settingdescribed in Section 3.1 and under theassumptions in Theorem 1, where we recall that n ∈ Θ( d ℓ )  and max { λ, 1 } ∈ Θ( d ℓ λ ) .
Revised: Throughout the remainder of the proof, all statements hold for the setting in Section 3.1 and under theassumptions of Theorem 1; in particular, Assumption 1 on κ , n ∈ Θ( d ℓ ) , and max { λ, 1 } ∈ Θ( d ℓ λ ) .

Original: Furthermore,as in Theorem 1,  δ, ¯ δ > 0 with δ :=
Revised: Furthermore, as in Theorem 1, we use δ, ¯ δ > 0 where δ :=

Original: ℓ − ℓ λ − 1 β − (cid:106) ℓ − ℓ λ − 1 β (cid:107) and ¯ δ :=
Revised: ℓ − ℓ λ − 1 β − (cid:106) ℓ − ℓ λ − 1 β (cid:107) and ¯ δ :=

Original: ℓ − 1 β − (cid:106) ℓ − 1 β (cid:107) .
Revised: ℓ − 1 β − (cid:106) ℓ − 1 β (cid:107) .

Original: To prove the Theorem 1, all we need to do is to show that there exists an m such that Theorem 3 applies withhigh-probability.
Revised: We show that there exists a particular m for which the conditions in Theorem 3 hold and we cancontrol the terms in the bias and variance bounds.

Original: Step 1: Bounding the variance.
Revised: Step 1: Conditions for the bounds in Theorem 3.

Original: We begin with a lemma showing the concentration of the spectrumof the late part of the kernel matrix K ≤ m and squared kernel matrix S ≤ m , where we make heavy useof the data distribution.
Revised: We first derive sufficient conditions on m suchthat the conditions on Ψ ≤ m and f ⋆ in Theorem 3 hold with high probability. The following resultfrom standard random matrix concentration shows that any m ≪ n satisfies Equation (5) with highprobability.

Original: Lemma 1 (Tight bound conditions) .
Revised: Lemma 1 (Corollary of Theorem 5.44 in Vershynin (2012)) .

Original: There exists m such that nλ m ∈ Θ(max { λ, 1 } ) , and, for such m , with probability at least 1 − cd −2 βmin { ¯ δ, 1 − ¯ δ} , r ∈ Θ(1) and [Equation]
Revised: For d large enough, with probability atleast 1 − cd − β ¯ δ , all m ∈ O ( n · d − β ¯ δ ) satisfy Equation (5) .

Original: We refer to Appendix C for the proof.
Revised: See Appendix C.1 for the proof.

Original: Note that for the choice of m made inLemma 1, also C, τ 1 , τ 2 ∈ Θ(1) . Assume for the moment that Assumption 1 holds for the choice of  m presented in
Revised: Simultaneously, to ensure that f ⋆ is contained in the span of the firstm eigenfunctions, m must be sufficiently large.

Original: Lemma 1, then we can apply Theorem 3 to get the following tight high-probability bound for thevariance: [Equation]
Revised: We formalize this in the following lemma.

Original: Step 2: Bounding the bias.
Revised: Lemma 2 (Condition for the bias bound) .

Original: To bound the bias, we need to show that Assumption 2 on the groundtruth holds and bound the term
Revised: Consider a kernel as in Theorem 1 satisfying Assumption 1,and a ground truth

Original: [Equation] , as done in the following lemma. Lemma 2 (Ground truth assumption) . Let the ground truth f∗ ( x ) =
Revised: f

Original:  (cid:81)
Revised: ⋆ ( x ) = (cid:81)

Original: L ∗ i =1 x i be as in Theorem 1with L ∗ > 0 .
Revised: L ∗ j =1 x j with L ∗ ≥ 1 .

Original: Then, as long as λ m ∈ o (cid:16) 1 d · q L ∗−(cid:17) , Assumption 2 is respected. Furthermore, if wewrite f ∗ ( x ) = [Equation]
Revised: Then, for any m such that λ m ∈ o (cid:16) 1 dq L ∗−(cid:17) andd sufficiently large, f ⋆ is in the span of the first m eigenfunctions and (cid:13)(cid:13)(cid:13)

Original: mk = [Equation] x ) , then [Equation]
Revised: D − 1 ≤ m a (cid:13)(cid:13)(cid:13) ∈ Θ (cid:0) dq L ∗ −(cid:1) .

Original: We refer to Appendix B for the proof.
Revised: See Appendix B.3 for the proof.

Original: Assume again for the moment that, for  the choice of m in
Revised: Step 2: Concentration of the (squared) kernel matrix. In a second step, we show existence ofan m that satisfies the sufficient conditions, and for which the spectrum of the kernel matrix K >mand squared kernel matrix S >m concentrates.

Original:  Lemma1, Assumption 1 and Lemma 2 hold. We would then again obtain from Theorem 3 thefollowing tight high-probability  bound  for the bias:
Revised: First, Lemma 3 yields high-probability bounds for aspecific set of m . Lemma 3 (Tight bound conditions) . With probability at least 1 − cd − 2 β min { ¯ δ, 1 − ¯ δ } uniformly over all ℓ λ ∈ [0 , ℓ − 1) , for any λ ∈ R and m ∈ N with max { λ, 1 } ∈ Θ( d ℓ λ ) and nλ m ∈ Θ(max { λ, 1 } ) , we have

Original: [Equation]
Revised: [Equation]

Original: Step 3: Proving the Theorem. We begin with the following lemma, which is a direct consequenceof well-known matrix concentration results.
Revised: and

Original: Lemma 3 (Corollary of Theorem 5.44 in (Vershynin, 2012)) . For any m ∈ O ( n · d − βδ ) , Assumption 1holds with probability 1 − cd − βδ .
Revised: [Equation]

Original: We refer to Appendix C for the proof.
Revised: We refer to Appendix C.2 for the proof, which heavily relies on the feature distribution.

Original: As a result, we are only left to show that the m such thatnλ m∈ in Lemma  1 satisfies the assumptions in Lemma 2 and 3.
Revised: Note that theresults for m in Lemma 3 also imply τ 1 , τ 2 ∈ Θ(1) .

Original: This requires a careful analysis ofthe eigenvalues of the convolutional kernel presented in (Misiakiewicz & Mei, 2021) and summarizedin Appendix B.
Revised: Step 3: Completing the proof.

Original: We collected what is useful for thepurpose of this proof in the following Lemma.
Revised: Finally, we complete the proof by showing the existence of aparticular m that simultaneously satisfies all conditions on m in Lemmas 1 to 3.

Original: Lemma 4 (Eigendecay) .
Revised: Lemma 4 (Eigendecay) .

Original: Let mbe such that nλ m ∈ Θ(max { λ, 1 } ) .
Revised: There exists an m such that [Equation]

Original: Then, assuming L ∗ ≤(cid:106) ℓ − ℓ λ − 1 β (cid:107) + 1 , we havethat λ m ∈
Revised: and [Equation] Furthermore, assuming L ∗ ≤ (cid:108) ℓ − ℓ λ − 1 β (cid:109) , we have λ m ∈

Original: o (cid:16) 1 d · q L ∗−(cid:17)and m ∈ O ( n · d − δ ) . 
Revised: o (cid:16) 1 d · q L ∗−(cid:17) . We refer to Appendix B.4 for the proof.

Original: Finally, we can conclude the proof by taking the union bound to show that Equation (5) and (6) holdwith probability 1 − cd − 2 β min { ¯ δ, 1 − ¯ δ } .
Revised: As a result, we can use Lemmas 1 to 4 to instantiate Theorem 3 for the setting in Theorem 1, resulting in the following tight high-probability bounds for variance and bias: [Equation] Reformulating the bounds in terms of n then concludes the proof of Theorem 1.

Original:  S UMMARY AND FUTURE WORK
Revised:  S UMMARY AND FUTURE WORK

Original: In this paper, we highlight  theimpact of strength of an inductive bias on the generalization performance of the resulting trained model.
Revised: In this paper, we highlight how the strength of an inductive bias impacts generalization.

Original: In particular we study when the gap  between optimallyridge-regularized or early-stopped models with their interpolating counterparts is zero (harmlessinterpolation).
Revised: Moreprecisely, we study when the gap in test error between optimally ridge-regularized or early-stoppedmodels and their interpolating counterparts is zero, that is, when interpolation is harmless.

Original: In particular, we prove a phase transition for kernel learning with convolutional kernelsof different filter sizes: weak inductive bias (large filter size) yields harmless interpolation whereasforstrong inductive bias (small filter size), interpolating noise leads to suboptimal rates.
Revised: In particular, we prove a phase transition for kernel regression using convolutional kernels with different filtersizes: a weak inductive bias (large filter size) yields harmless interpolation, and even requires fittingnoise for optimal test performance, whereas interpolating noise leads to suboptimal generalization fora strong inductive bias (small filter size).

Original: Intuitively, thisphenomenon can be explained by a bias-variance trade-off captured by our main result, Theorem 1:with increasing inductive bias the risk on noiseless training samples (bias) decreases, while thesensitivity to noise in the training data (variance) increases.
Revised: Intuitively, this phenomenon arises from a bias-variancetrade-off, captured by our main result in Theorem 1: with increasing inductive bias, the risk onnoiseless training samples (bias) decreases, while the sensitivity to noise in the training data (variance)increases.

Original: Our empirical results on neural networkssuggest that this phenomenon might extend to feature learning models, which opens up an avenue forexciting future work.
Revised: Our empirical results on neural networks suggest that this phenomenon might extend tomodels with feature learning, which opens up an avenue for exciting future work.

Original:  A P ROOF OF T HEOREM 3: G ENERALIZATION BOUND FOR FIXED - DESIGN
Revised: R A P ROOF OF T HEOREM 3: G ENERALIZATION BOUND FOR FIXED - DESIGN

Original: Since we have a closed formula for  bias and variance,  the proof of this result boils down to an accurateapplication of Woodbury identity.
Revised: We proof Theorem 3 by deriving a closed-form expression for the bias and variance, and boundingeach term individually. Due to the closed form, the proof does not rely on any matrix concentrationresults.

Original: Proof.
Revised: Proof.

Original: Thanks to the representer theorem we know that the optimization problemin kernel ridge regression defined in Equation 2 is solved by the following estimator:
Revised: It is well-known that the KRR problem defined in Equation (2) is solved by the estimator

Original: [Equation]
Revised: [Equation]

Original: where  y :
Revised: where x i ∼ ν , y :

Original: = 
Revised: = f + ϵ =

Original: [ y 1 , . . .
Revised: [ f ⋆ ( x 1 ) + ϵ 1 , . . .

Original: , y n ] ⊺ and k ( x ) : =
Revised: , f ⋆ ( x n ) + ϵ n ] ⊺ , k ( x ) :=

Original: [ K ( x 1 , x ) , . . .
Revised: [ K ( x 1 , x ) , . . .

Original: , K ( x n , x )] ⊺ ,  H =
Revised: , K ( x n , x )] ⊺ , and H :=

Original: K + λ I n .
Revised: K + λ I n .

Original: Thus, we can get a closed formula for bias and variance:
Revised: For this estimator, both bias and variance exhibit a closed-form expression:

Original: [Equation] 
Revised: [Equation] [Equation]

Original: Where in ( i ) we use the definition of the S 1 kernel 
Revised: In ( i ) we use the definition of the squared kernel S , and that f ⋆ ( x ) = (cid:80)

Original: [Equation]
Revised: [Equation]

Original:  as well as the following simple consequences of the eigenfunctions orthonormality:
Revised: k ( x ) , since we are assuming that f ⋆ is in the span of the first m eigenfunctions, as well as the following simple consequences of the eigenfunctions’ orthonormality:

Original: [Equation]
Revised: [Equation]

Original: Now we decompose the proof of the theorem in two parts, one for variance and one for bias.
Revised: We now bound the closed-form expressions of bias and variance individually.

Original: BiasWe start by applying Lemma 5 to decompose S in Ψ ≤ m D ≤ m Ψ ⊺ ≤ m + S >m :
Revised: Bias Lemma 5 below yields S = Ψ ≤ m D ≤ m Ψ ⊺ ≤ m + S >m . Hence, the bias decomposes as

Original: Bias
Revised: Bias

Original: [Equation]
Revised: [Equation]

Original: Now, we first notice that B 1 can be written as:
Revised: First, we rewrite B 1 as

Original: [Equation]
Revised: [Equation]

Original: Where in ( i ) we decomposed H in K ≤ m + H >m , where H >m = K >m + λ I n .
Revised: where ( i ) uses the decomposition H = K ≤ m + H >m with H >m :

Original: In ( ii ) we applied Woodbury identity.
Revised: = K >m + λ I n , and ( ii ) appliesthe Woodbury identity.

Original: We bound now B 2 :
Revised: Second, we can upper-bound B 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: = 
Revised: = B

Original: [Equation]
Revised: [Equation]

Original: Where in ( i ) we use Lemma 7.
Revised: where ( i ) uses Lemma 6.

Original: Thus, the bias can be bounded by B 1(1 + C 1) .
Revised: Thus, the bias can be bounded by B 1 (1 + C 1 ) with C 1 ≥ [Equation]

Original: We now start by upper bounding C 1 :
Revised: We proceed by upper bounding C 1 :

Original: [Equation]
Revised: [Equation]

Original: Where in ( i ) we used Assumption 3 to bound ∥ Ψ ⊺ ≤ m Ψ ≤ m /n ∥ , and Lemma 6 to bound ∥ S >m ∥ .Toconclude
Revised: where ( i ) uses Equation (5) to bound ∥ Ψ ⊺ ≤ m Ψ ≤ m /n ∥ , and Lemma 7 to bound ∥ S >m ∥ . In ( ii ) weuse the bound: cx + 1 ≤ ( c + 1) max { x, 1 } . In ( iii ) we just see that the terms matches the definitionof τ 2 .

Original:  the bias bound, we need toupper and lower bound B 1 .
Revised: Hence, to conclude the bias bound, we need to bound B 1 from above and below.

Original: Upper bound:
Revised: Upper bound:

Original: [Equation]
Revised: [Equation]

Revised: where ( i ) follows from Equation (5).

Original: n.
Revised: This concludes the upper bound on the bias since we can compute: [Equation] where in ( i ) we use the bound in Equation (10), and in ( ii ) we used Equation (11). This concludesthe upper bound on the bias.

Original: Lower bound:
Revised: Lower bound:

Original: [Equation]
Revised: [Equation]

Original: Where ( i ) follows from Assumption 1. Finally, the bound for the bias follows directly by taking C 3 = 8 max { 1 . 5 r, 1 }  . 
Revised: where ( i ) follows from Equation (5), ( ii ) from the fact that 1 1+ cx ≤ 11+ c min { x, 1 } , and ( iii ) thedefinition of τ 1 . Since Bias 2 ≥ B 1 , this concludes the lower bound for the bias.

Original: Variance: We start by applying Lemma 5 to decompose S in Ψ ≤ m D ≤ m Ψ ⊺ ≤ m + S >m , hence wedecompose the variance as:
Revised: Variance: As for the bias bound, we first apply Lemma 5 to write S = Ψ ≤ m D ≤ m Ψ ⊺ ≤ m + S >m ,and decompose the variance in Equation (7) as [Equation] (cid:17) (cid:124) (cid:123)(cid:122) (cid:125)

Original: [Equation]
Revised: [Equation]

Original: We then start by bounding V 1 .
Revised: (cid:0) H

Original: [Equation]
Revised: [Equation]

Original: Where in ( i ) we used Lemma 7. We now use the fact that the trace is the sum of the eigenvalues, hence, it’s bounded from above by the size of the matrix times the largest eigenvalue. And from below by the size of the matrix times the smallest eigenvalue.
Revised: (cid:1) (cid:124) (cid:123)(cid:122) (cid:125) V.

Original: We then get:
Revised: Next, we rewrite V 1 as follows:

Original: [Equation]
Revised: [Equation]

Original: Where in ( i )  and ( ii ) we bound these terms analogously to the upper and lower bound of B 1 . Therest is algebra and usage of Assumption 1.
Revised: Step ( i ) follows from Lemma 6. To bound V 1 and V 2 , we use the fact that the trace is the sum of all eigenvalues. Therefore, the trace is bounded from above and below by the size of the matrix times the largest and smallest eigenvalue, respectively.

Original: Now we move to the  bound of V 2 , here we get that:
Revised: This yields the following bound for V 1 :

Original: [Equation] 
Revised: [Equation] τ

Revised: For ( i ) and ( iii ) , we bound the terms analogously to the upper and lower bound of B 1 , and use

Original: Finally, the bound for the variance follows directly by taking C 1 = 6 r 2 , C 2 = 1 / (6 r 2 ) .
Revised: Equation (5) in ( ii ) and ( iv ) . Next, we bound V 2 as [Equation] This concludes the bound for the variance, and hence the proof.

Original: Lemma 5 (Squared kernel decomposition) .
Revised: Lemma 5 (Squared kernel decomposition) .

Original: Let K be a kernel function that under a distribution ν can be decomposed as K ( x, x ′ )
Revised: Let K be a kernel function that under a distribution ν can be decomposed as K ( x, x ′ )

Original: = (cid:80)k
Revised: = (cid:80)

Original: λ k ψ k ( x ) ψ k ( x ′ ) , where E x ∼ ν [ ψ k ( x ) ψ k ′ ( x )] 
Revised: [Equation] x ) ψ k ( x ′ ) , where E x ∼ ν [ ψ k ( x ) ψ k ′ ( x )] =

Original: = δ k,k ′ .
Revised: δ k,k ′ .

Original: Then the squared kernel defined as S1 ( x, x ′ )
Revised: Then the squared kernel defined as S ( x, x ′ )

Original: =
Revised: =

Original: E
Revised: E

Original: z ∼
Revised: z ∼

Original: ν
Revised: ν

Original: [ K ( x, z ) K ( x ′ , z )] can be written as:
Revised: [ K ( x, z ) K ( x ′ , z )] can be written as S ( x, x ′ ) = (cid:88) k λ

Original: [Equation]
Revised: [Equation]

Original: And the kernel matrix S can be written as Ψ ≤ m D  ≤ m Ψ ⊺ ≤ m + S >m .
Revised: k ψ k ( x ) ψ k ( x ′ ) , and the kernel matrix S can be written as Ψ ≤ m D 2 ≤ m Ψ ⊺ ≤ m + S >m .

Original: Proof.
Revised: Proof.

Original: The proof simply follows from
Revised: The statement simply follows from

Original: [Equation]
Revised: [Equation]

Original: Lemma 6 (Squared kernel tail) .
Revised: Lemma 6 (Corollary of Lemma 20 in Bartlett et al. [Equation] Proof. [Equation]

Revised: where ( i ) applies Lemma 20 from Bartlett et al.

Original:  Let S >m be the kernel matrix of the squared kernel S 1 , then
Revised: Lemma 7 (Squared kernel tail) . Let S >m be the kernel matrix of the squared kernel S . Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We show that for any vector v , v ⊺ S >m v ≤ λ m +1 v ⊺ K >m v , which is sufficient to prove the Lemma.
Revised: We show that for any vector v , v ⊺ S >m v ≤ λ m +1 v ⊺ K >m v , which is sufficient to prove the lemma.

Original: To do so, let us define Ψ k ∈ R n × n , [ Ψ k ] ij = ψ k ( x i ) ψ k ( x j ) .
Revised: To do so, we define Ψ k ∈ R n × n with [ Ψ k ] i,j = ψ k ( x i ) ψ k ( x j ) .

Original: Then we can write
Revised: Then we can write

Original: K >m =
Revised: K >m =

Original: (cid:80) k>m λ k Ψ k andS >m = (cid:80) k>
Revised: (cid:80) [Equation] and

Original: [Equation]
Revised: [Equation]

Original:  λ 2 k Ψ k .
Revised: k>m λ 2 k Ψ k .

Original: Since the eigenvalues are ina decreasing order, we havethat λ k ≤ λ m +1 for any k > m , hence:
Revised: Since the eigenvalues are in decreasing order, we have λ k ≤ λ m +1 for any k > m , and thus

Original: [Equation]
Revised: [Equation]

Original: ⊺
Revised: B C ONVOLUTIONAL KERNELS ON THE HYPERCUBE

Original: [Equation] m [Equation]
Revised: First, Appendix B.1 provides a way to decompose general functions for features distributed uniformlyon the hypercube.

Original: Concluding the proof. Lemma 7 (Lemma 20 from Bartlett et al. Let Ψ ≤ m , D ≤ m , H , H ≤ m be defined as above, then: [Equation]
Revised: Next, Appendix B.2 uses those results to characterize the eigenfunctions and eigenvalues of cyclic convolutional kernels.

Original: Proof. The proof is simple Woodbury identity with some added algebra. [Equation] Where in ( i ) we used Lemma 20 from Bartlett et al.
Revised: Finally, Appendices B.3 and B.4 apply this characterization toprove Lemmas 2 and 4, respectively.

Original: B T HE H YPERCUBEB. F UNCTIONS ON THE HYPERCUBE
Revised: B. G ENERAL FUNCTIONS ON THE HYPERCUBE

Original: In this section we focus on the specific hypercube domain {− [Equation] 1 } d together with the uniform probability density,which has been previously studied in Misiakiewicz & Mei (2021).
Revised: This subsection focuses on the main setting in our paper: the hypercube domain {− 1 , 1 } d togetherwith the uniform probability density, previously studied in Misiakiewicz & Mei (2021).

Original: Define the following inner product:
Revised: For any S ⊆ { 1 , . . . , d } , we define the polynomial

Original: [Equation]
Revised: [Equation]

Original: We can now define, for any S ⊆ { 1 , . . . , d } : [Equation]
Revised: of degree | S | , where [ x ] j is the j -th entry of x ∈ {− 1 , 1 } d .

Original: [Equation] It’s easy to see that {Y S } S ⊆{ 1 ,...,d } isan orthonormal set of  functions with respect to ⟨· , ·⟩ {− 1 , 1 } d  .
Revised: It is easy to see that {Y S } S ⊆{ 1 ,...,d } is set of orthonormal functions with respect to the inner product ⟨ f, g ⟩ {− 1 , 1 } d := E x ∼U ( {− 1 , 1 } d ) [ f ( x ) g ( x )] .

Original: We will later show that such functions  are the eigenfunctions of the kernel of our interest.
Revised: Those functions play a key role in the remainder of our proof; as it turns out, they are the eigenfunctions of the kernel in Section 3.1.

Original: Furthermore, let us define the polynomials:
Revised: Towards a formal statement, define the polynomials [Equation] where B ( l, d ) := |{ S ⊆ { 1 , . . . , d } | | S | = l

Original: [Equation]
Revised: [Equation]

Original: Notice that G ( d ) ldepends only  on the  inner product of x and y .
Revised: Notice that G ( d ) l only depends on the (Euclidean) inner product of x and x ′ .

Original: Furthermore, note that the G ( d ) l area set of orthonormal polynomial with respect to the distribution of ⟨ x, x ′ ⟩ / √ d .
Revised: Furthermore, {G ( d ) l } dl =0is a set of orthonormal polynomials with respect to the distribution of ⟨ x, x ′ ⟩ / √ d .

Original: Now this generallemma shows howwe can decompose any rotational invariant kernel in such eigenbases.
Revised: The followinglemma shows how such polynomials form an eigenbasis for functions that only depend on the innerproduct between points in the unit hypercube.

Original: Lemma 8 (Local kernel decomposition) .
Revised: Lemma 8 (Local kernel decomposition) .

Original: Let κ : R → R be any function, then we can writeκ ( ⟨ x, x ′ ⟩ /d ) as:
Revised: Let κ : R → R be any function and d ∈ N > 0 . Then, forany x, x ′ ∈ {− 1 , 1 } d , we can decompose κ ( ⟨ x, x ′ ⟩ /d ) as

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For the proof of this lemma we are only interested at the evaluation of κ in the values that ⟨ x,x ′ ⟩ d can take,given that x, x ∈ {− 1 , 1 } d . Hence we are interestend only in κ computed in {− 1 , − 1 + 2 /d, . . 
Revised: Note that the decomposition only needs to hold at the evaluation of κ in the values that ⟨ x, x ′ ⟩ /d can take, that is, κ computed in {− 1 , − 1 + 2 /d, . . .

Original:  − 2 /d + 1 , 1 } .
Revised: , − 2 /d + 1 , 1 } .

Original: Since this set has a cardinalityof d + 1 , we can write κ as a linear combination of d + 1 linearly independent functions.
Revised: Since that set has a cardinality d + 1 , we can write κ as a linear combination of d + 1 uncorrelated functions.

Original: This is the case for {G ( d ) l } dl =0 since such function are uncorrelated.
Revised: This is the case for {G ( d ) l } dl =0 with respect to the distribution of ⟨ x, x ′ ⟩ / √ d by construction.

Original: Hence we can write:
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original:  Finally, the proof follows by  the expansion of G ( d ) l inEquation 10.
Revised: for some (unknown) coefficients c l . Finally, the proof follows by expanding the definition of G ( d ) l in

Original:  While this lemma is very general, we need some regularity conditions on ξ ( d ) l , for our theory to gothrough.
Revised: Equation (14) and choosing ξ ( d ) l = c l . While this lemma is very general, the proof of Theorem 1 requires a mild regularity assumption on κ .

Original: Assumption 3 (Regularity of κ ) .
Revised: Assumption 1 (Regularity of κ ) .

Original: In the setting presented in Section 3.1, we say that the κ satisfy the genericity assumption if, writing κ as in Equation 12, there exist constants c 1 , c 2 , c 3 , c 4 > 0  such that:
Revised: A function κ : [ − 1 , 1] → R in the setting of Section 3.1 satisfies the regularity assumption if there exist constants c 1 , c 2 > 0 and a series of constants { c l > 0 } l ≥ 0 such that, for any q ≥ c 1 , the decomposition κ ( ⟨ x, x ′ ⟩ /q ) = (cid:80) ql =0 ξ ( q ) l 1 B ( l,q ) (cid:80) | S | = l Y S ( x ) Y S ( x ′ ) from

Revised: Lemma 8 over inputs x, x ′ ∈ {− 1 , 1 } q satisfies

Original: [Equation]
Revised: [Equation]

Original: for any q ≥ c 4 . This condition is quite mild, and it only asks for the tail of the polynomial expansion to not vanish.
Revised: For sufficiently high-dimensional inputs x, x ′ , Assumption 1 ensures that the convolutional kernel

Original: This allows our model to have the so-called implicit regularization. Furthermore, the expansion ofthe polynomial degree that we have to learn also to be non-vanishing, this makes the task to learnpolynomial of degree L possible.
Revised: K ( x, x ′ ) in Equation (1) is a valid kernel, that it possesses the so-called “implicit regularization”property , and enables learning of polynomials of any degree. This assumption resembles Assumption 1 from Misiakiewicz & Mei (2021). However, our version is more general: we enable learningof polynomials of any degree, while Assumption 1 from Misiakiewicz & Mei (2021) only coversthe degrees of interest.

Original: Now we move to the study of the convolutional kernel to build the tools to prove Lemma 4.
Revised: Still, our assumption holds for standard kernels such as the Gaussian kernel. Finally, our assumption does not depend on ℓ . Thus, a kernel satisfying our assumption will alsosatisfy Assumption 1 from Misiakiewicz & Mei (2021) for any choice of ℓ .

Original: B. C ONVOLUTIONAL KERNEL
Revised: B. C ONVOLUTIONAL KERNELS ON THE HYPERCUBE

Original: In this section we propose tool to analyze convolutions in the setting proposed in Section 3.1.
Revised: While the previous subsection considers general functions on the hypercube, we now focus onconvolutional kernels and their eigenvalues.

Original: Then,we use them to prove Lemma 4 and Lemma 2.
Revised: This yields the tools to prove Lemmas 2 and 4.

Revised: In orderto characterize eigenvalues, we first introduce the following quantities, also used e.g. in Misiakiewicz& Mei (2021).

Original: Definition 1 (Diameter of a subset) .
Revised: Definition 1 (Diameter) .

Original: Let S be a subset of { 1 , . . .
Revised: Let S ⊆ { 1 , . . .

Original: , d } , then we call diameter of S :
Revised: , d } . The diameter of S is

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: j [Equation] S }  .
Revised: 1 } for S ̸ = ∅ , and γ ( ∅ ) = 0 .

Original: Furthermore, we defineB to be:
Revised: Furthermore, we define

Original: [Equation]
Revised: [Equation]

Original: The diameter of S can be understood as the size of the smallest patch that fully contains S .
Revised: Intuitively, the diameter of S is the smallest number of contiguous feature indices that fully contain S .

Revised: The following lemma yields an explicit formula for C ( l, q, d ) , that is, the number of sets of size lwith diameter at most q .

Revised: Lemma 9 (Number of overlapping sets) .

Revised: Let l, q, d ∈ N with q ≥ 0 and l ≤ q < d/ 2 .

Original: Proposition 1 (Proposition 1 from Misiakiewicz & Mei (2021)) .
Revised: Then, C ( l, q, d ) = (cid:40) d (cid:0) q − 1 l −(cid:1) l > 0 , l = 0 or q = 0 . Proof. Since the result holds trivially for q = 0 , l = 0 , and l = 1 , we henceforth focus on l ≥ 2 .

Revised: Let ˜ C

Revised: ( l, γ, d ) be the number of subsets S ⊆ { 1 , . . .

Revised: , d } of cardinality | S | =

Revised: l with diameter exactly γ ( S ) =

Revised: γ .

Revised: First, consider ˜ C (2 , γ, d ) .

Revised: For each set, we can choose the first element i from d different values, and the second as mod (( i − 1) ± ( γ − 1) , d ) + 1 .

Revised: In this way, since q < d/ 2 , we count each set exactly twice.

Revised: Thus,

Revised: [Equation]

Revised: Next, consider l > 2 .

Revised: We can build all the possible sets by starting with one of the ˜ C (2 , γ, d ) =

Revised: d sets, and adding the remaining l − 2 elements from γ − 2 possible indices.

Revised: Hence, every fixed set of size 2 and diameter γ yields

Revised: [Equation]

Revised: different sets of size l .

Revised: Furthermore, by construction, every set of size l and diameter γ results from exactly on set of size 2 and diameter γ .

Revised: Therefore,

Revised: [Equation]

Revised: The result for l ≥ 2 then follows from summing ˜ C ( l, γ, d ) over all diameters γ ≤ q :

Revised: [Equation]

Revised: where ( i ) follows from the hockey-stick identity.

Original: Let Kbe a kernel defined as in 
Revised: Now, we focus on cyclic convolutional kernels K as in Equation (1). First, we restate Proposition 1from Misiakiewicz & Mei (2021). This proposition establishes that Y S for S ⊆ { 1 , . . . , d } are indeedthe eigenfunctions of K , and yields closed-form eigenvalues λ S up to factors ξ ( q ) | S | that depend onthe inner nonlinearity κ .

Original:  Equation 1.
Revised: Then, Lemma 10 uses the regularity assumptions on κ to eliminate thedependency on ξ ( q ) | S | . This characterization of the eigenvalues enables the proof Lemmas 2 and 4. Proposition 1 (Proposition 1 from Misiakiewicz & Mei (2021)) . Let K be a cyclic convolutional kernel over the unit hypercube as defined in Equation (1) .

Original: Then it can be written as:
Revised: Then we can write

Original: [Equation]
Revised: [Equation]

Original: x ( k ) ′ ⟩ q
Revised: with

Original: [Equation]
Revised: [Equation]

Original: l
Revised: where ξ ( q ) | S | are the coefficients of the κ -decomposition (Equation (16) ) over {− 1 , 1 } | S | . Alternatively,

Original: [Equation]
Revised: [Equation]

Original: The proof of Proposition 1 follows by applying Lemma 8 once on each of the patches, and then sumeverything.
Revised: where we order all subsets S k ⊆ { 1 , . .

Original: The factor q + 1 − γ ( S  ) stems from the fact that the eigenfunction Y S appears as manytimes as the number of patches of size q , that cover S  .
Revised: , d } with γ ( S k ) ≤ q such that λ S k ≥ λ S k +1 . In particular,λ S k = λ k .

Original: For a formal proof look at Proposition 1 from Misiakiewicz & Mei (2021).
Revised: We refer to Proposition 1 from Misiakiewicz & Mei (2021) for a formal proof.

Original: Now we present two lemmas that help us to better characterize the eigenvalues.

Original:  Lemma 9 (Number of overlapping sets) . Assume q < d/ 2 , then: [Equation]
Revised: Intuitively, the resultfollows from applying Lemma 8 for each subset S ⊆ { 1 , . . .

Original: Proof.
Revised: , d } .

Original: Let ˜ C ( l, γ, d ) be the number of subset of { 1 , . . . , d } , of cardinality l , where the diameter is exactly γ . To compute such value start by computing ˜ C
Revised: This is possible because crucially,any subset of q features is again distributed uniformly on the q -dimensional unit hypercube.

Original:  ( l, 2 , d ) . We can choose the first of the two elements to be in d different position, and the second either. q after it, or before it. Finally, in this way, we count each set exactly twice.
Revised: Lastly,the factor q + 1 − γ ( S ) stems from the fact that each eigenfunction Y S appears as many times asthere are contiguous index sets of size γ ( S ) supported in a fixed contiguous index set of size q .

Original: Hence: [Equation] Now, if l > 2 . We can build all the possible sets by starting with one of the d sets of diameter γ and size 2 , and then adding the leftovers l − 2 elements.
Revised: Inother words, the term is the number of shifted instances of S supported in a contiguous subset of qfeatures.

Original: Note that if we start with different sets of size 2 , we must achieve different sets of size l thanks to the diameter condition. Finally, we need to add those l − 2 elements in any of  the  q − 2 elements between the two in the set of size 2 : this can be done in (cid:0) γ − 2 l −(cid:1) .Finally, combining the two ideas we get that: [Equation]
Revised: While Proposition 1 yields eigenfunctions and eigenvalues of cyclic convolutional kernels, it does notdescribe the factors ξ ( q ) | S | which depend on the inner nonlinearity κ .

Original: Hence, we can also compute: [Equation] Where ( i ) follows from the hockey-stick identity.
Revised: The final lemma of this sectionis a corollary that characterizes the eigenvalues solely in terms of | S | , and further shows that, for dlarge enough, the eigenvalues decay as | S | grows.

Original: Lemma 10 (Corollary of Proposition 1) .
Revised: Lemma 10 (Corollary of Proposition 1) .

Original: Assume that q ∈ Θ( d β ) . Let { S d } d ≥ l be a sequence of sets such that: S d ⊆ { 1 ,  . . .
Revised: Consider a cyclic convolutional kernel from Propositionwith q ∈ Θ( d β ) for some β ∈ (0 , 1) . Assume that κ satisfies Assumption 1 with c 1 ≤ q .

Original: , d } , such that | S d | = l for any d ≥ l , and such that γ ( Sd ) ≤ q . Let K be a kernel respecting Assumption 3 for l .
Revised: Then, for any S ⊆ { 1 , . . . , d } such that γ ( S ) ≤ q , the eigenvalue λ S corresponding to the eigenfunction Y S ( x ) satisfies

Original: Then, writing our kernel as:

Original: [Equation]
Revised: [Equation]

Original: we have that:
Revised: where ≲ hides positive constants that depend on | S | .

Original: [Equation]

Original: In particular, there exists a sequence { S  d } d ≥ l such that  λ Sd ∈ Θ 
Revised: Furthermore, there exists an S ⊆ { 1 , . . . , d }such that γ ( S ) ≤ q for which λ S ∈ Θ (cid:0)

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: 1+ β | S | (cid:1) .

Original: Proof.
Revised: Proof.

Original: Letus define the sequence { r d } d ≥ l , where r d = q + 1 − γ ( Sd ) . 
Revised: Let l := | S | and r := q + 1 − γ ( S ) . Since l ≤ γ ( S ) ≤ q , we have [Equation]

Original: Now, since l ≤ γ ( S d ) ≤ q  , we have that:
Revised: Furthermore, because B ( l, q ) = (cid:0) ql (cid:1) , the classical bound on the binomial coefficient yields [Equation] Hence,

Original: [Equation]
Revised: [Equation]

Original: In particular, we can take the sequence { S d } d ≥ l such that  γ ( Sd ) = q , for any d : in this case r d =for all d .
Revised: where Assumption 1 yields in ( i ) that ξ ( q ) l ≤ c 2 , and in ( ii ) that ξ ( q ) l ≥ c l . Since c l, 1 and c l, 2 only 511 depend on l , this concludes the first part of the proof. For the second part, any S with γ ( S ) = q yields r = 1 .

Original: Now, thanks to Assumption 3, can write λ S d as:
Revised: Hence, the previous upper bound becomes [Equation] where c l, 3 only depends on l . This yields the second statement, since dq l ∈ Θ (cid:0) d 1+ β | S | (cid:1) .

Original: [Equation]
Revised: Finally, using Proposition 1 and Lemma 10, we are able to prove Lemmas 2 and 4. B. P ROOF OF L EMMA 2 First, notice that f ⋆ = Y S ∗ for S ∗ = { 1 , . . . , L ∗ } .

Original: In particular, we can take the sequence { S d } d ≥ l such that γ ( S d ) = q , for any d : in this case r d = 1 ,and thus λ S d ∈ Θ(1 /dq l ) .
Revised: Since | S ∗ | = γ ( S ∗ ) = L ∗ , applying Propositionyields [Equation] where in ( i ) we used assumption Assumption 1 to get that ξ ( q ) L ∗ ∈ Θ(1) , and in ( ii ) we used the fact that λ m ∈ o (cid:16) 1 dq L ∗−

Original: The proof follows remembering that q ∈ Θ( d β ) .
Revised: [Equation] (cid:17) . This implies that for d sufficiently large, λ S ∗ > λ m .

Original: Finally, we move to the proof of the two lemmas of interest:
Revised: Thus, since the eigenvalues are sorted, f ⋆ is in the span of the first m eigenfunctions. Finally, [Equation] since in ( i ) , the entry of a corresponding to Y S ∗ is 1 , and all others are 0 .

Original: Proof of Lemma 4.
Revised: Before proving Lemma 4, we introduce the following quantity: [Equation]

Original: We show that m is in O ( nd − δ ) . 
Revised: Intuitively, L corresponds to the degree of the largest polynomial that a cyclic convolutional kernel asdefined in Equation (1) can learn. This quantity plays a key role throughout the proof of Lemma 4,and Lemma 3 later. All eigenfunctions of degree at most L will have an index smaller than m , whileall eigenfunctions of degree at least L + 2 will have an index larger than m . The proof of Lemma 4shows this statement formally.

Original: Thanks to Proposition 1, we can write the  convolutional kernel as:
Revised: Proof of Lemma 4. First, we use Proposition 1 to write the cyclic convolutional kernel as

Original: [Equation]
Revised: [Equation]

Original: Where S i is an ordering such that λ S i +1 ≤ λ S 
Revised: where S i are ordered such that λ S k +1 ≤ λ S k .

Original: i .
Revised: Here we need to show that we can pick m ∈ N such that nλ S m ∈ Θ (max { λ, 1 } ) , and m ∈ Θ [Equation] nq − δ max { λ, 1 } (cid:17) .

Original: We now show that  mis such that | S m| = 
Revised: Since n ∈ Θ( d ℓ ) and max { λ, 1 } ∈ Θ( d ℓ λ ) , it suffices to show that there exists a m such that [Equation] By definition of λ m = λ S m , this corresponds to showing that there exists S m ⊆ { 1 , . . . , d } withγ ( S m ) ≤ q such that the eigenvalue corresponding to Y S m is λ S m ∈ Θ (cid:16)

Original: L + 1for d sufficiently large.
Revised: 1 d · q L + δ (cid:17) , and m ∈ Θ (cid:0) dq L (cid:1) . Choice of m .

Original: By contradiction, assume that | S m | = ˆ L < L + 1 , then:
Revised: The way we find such S m is different in two cases:

Original: [Equation]
Revised: [Equation]

Original: But this contradicts the assumption that nλ m  ∈ Θ(max { λ, 1 } ) .
Revised: First, in both cases, since δ < 1 , S m is well-defined for d — and hence q ∈ Θ( d β ) — sufficiently large.

Original: Analogously, if we assume that| S m | 
Revised: Assume that this is the case. Then, | S m | =

Original: = ˆ L > L + 1 , it is easy to see that nλ m ∈
Revised: L + 1 in both cases and [Equation]

Original: o (max { λ, 1 } ) , which again contradicts the sameassumption.
Revised: by definition.

Original: Now, thanks to what we have just found and Proposition 1,  we can write λ m as:
Revised: Using Proposition 1, since we know that | S m | = L + 1 , we can write the eigenvalue corresponding to Y S m as

Original: [Equation]
Revised: [Equation]

Original: Since we want that nλ m ∈ Θ( d ℓ λ ) , we need q+ 1 − γ (
Revised: Numerator: Now we show that in both cases, the numerator is in Θ (cid:0) q 1 − δ (cid:1) . In the case where δ ∈ (0 , 1) :

Original: S m ) ∈ Θ( q − ( δ − 1) ) .
Revised: [Equation] where ( i ) follows from δ < 1 and q sufficiently large. In the case where δ = 0 :

Original: Hence, γ ( S m ) ∈ Θ( q ) .
Revised: [Equation] Thus, the numerator is in Θ( q ) =

Original: Finally, we need to find m itself.
Revised: Θ( q 1 − δ ) , since in this setting δ = 0 . Denominator: Notice that the denominator does not depend on δ , thus we use the same technique for both δ = 0 , δ ∈ (0 , 1) . Indeed, we use the classical bound on B ( L + 1 , q ) = [Equation]

Original: Hence, we need to count how many S are there such thatλ S< λ m .
Revised: Rate of λ S m : Using both parts, together with ξ ( q ) L +1 ∈ Θ(1) by Assumption 1, we can conclude that S m satisfies [Equation]

Original: Thanks to Lemma 10, we have that, for d sufficiently large, for all S of cardinality smaller than L ,λ S < λ m ; and for all S of cardinality larger than L , λ S > λ m .
Revised: Hence, thanks to Equation (19), we have shown that there exists an m for which nλ m ∈Θ(max { λ, 1 } ) . We now need to show that for this choice of m , we also have that m ∈Θ [Equation] =

Original: Finally, if S ofcardinality L , than λ S < λ m if and only ofγ ( S ) <
Revised: [Equation] Rate of m . Upper bound: From the construction of S m in Equation (21) it directly follows that| S m | =

Original: γ ( S m ) .
Revised: L + 1 .

Original: The number of such S , is exactly C ( L, γ ( S m) , d ) .
Revised: Furthermore, Lemma 10 shows that, for d sufficiently large, all S k with | S k | > L + 1 correspond to eigenvalues λ S k < λ S m .

Original: Hence, we can compute m as:
Revised: Thus, m is at most the number of polynomials with degree no larger than L + 1 :

Original: [Equation]
Revised: [Equation]

Original: Where in ( i ) we used that
Revised: where ( i ) uses Lemma 9 for d large enough with

Original: [Equation]
Revised: [Equation]

Original: ql
Revised: Lower bound: From the definition of S m in Equation (21), we have that γ ( S m ) ≥ ⌊ q/ 2 ⌋ . This, combined with Proposition 1, yield that all the indexes of polynomials of degree L + 1 , but diameter ≤ ⌊ q/ 2 ⌋ − 1 appear before m . Hence, thanks to Lemma 9,

Original: [Equation]
Revised: [Equation]

Original: ∈ Θ
Revised: ∈ Ω( dq L ) , concluding the proof of the lower bound. Thus, m ∈ Θ( d L ) , and, thanks to Equation (20), m ∈Θ

Original: [Equation]
Revised: [Equation]

Revised: (cid:17) .

Revised: This concludes the proof of the existence.

Revised: To conclude the proof we only need toshow additional formulas for the rates.

Revised: Rate of m with respect to n .

Original: q l ) , as well as γ ( S m )  ∈ Θ( q ) .
Revised: Finally, we only need to show that Θ( dq L ) ⊆ O ( nq − δ ) . For this wenotice that, since ¯ L := (cid:106) ℓ − 1 β (cid:107) ≥ L , we have that m ∈ O ( dq ¯ L ) .

Original: Finally, we can see n as
Revised: Then, we can write n as

Original: [Equation]
Revised: [Equation]

Original: Which, plugging in, yields that m ∈ Θ( n · d − ℓ λ · q − δ ) =
Revised: and hence m ∈ O ( dq ¯ L ) =

Original: O ( n · q −  δ ) .
Revised: O ( nq − ¯ δ ) .

Original: Now we need to find the rate of λ m  .
Revised: Rate of λ m with respect to L ∗ .

Original: But this is not hard since we know that nλ m ∈ Θ(max { λ, 1 } ) :
Revised: By assumption, nλ m ∈ Θ(max { λ, 1 } ) . Therefore,

Original: [Equation]
Revised: [Equation]

Original: Where ( i ) follows by the assumption that ℓ λ ≤ L + 1 .
Revised: where ( i ) uses the identity ℓ = 1 + ℓ λ + β ( L + δ ) .

Original: Proof of Lemma 2. It is easy to see that f ∗ =
Revised: For the remainder, we need to consider two cases depending on δ .

Original: Y S ∗ , where S ∗ = { 1 , . . .
Revised: If δ > 0 , then L ∗ ≤ L + 1 , and we have

Original: , L ∗ } .
Revised: [Equation] ∈

Original: Now, notice that | S ∗ | = γ ( S ∗ ) =
Revised: O

Original:  L ∗ . Hence, we havea formula for λ − 1 S ∗ in Proposition 1:
Revised: [Equation] [Equation] If δ = 0 , then L ∗ ≤ L , and we have

Original: [Equation]
Revised: [Equation]

Original: Finally, as long as λ m ∈o (cid:0) 1 /dq
Revised: In both cases, λ m ∈

Original: L ∗ −(cid:1) , we have that, since eigenvalues are sorted, the index of the eigenfunction Y S ∗ is less than m . Hence, for d sufficiently large, f ∗ is in the span of the first m eigenfunctions. Finally, if this is the case:
Revised: o

Original: [Equation]
Revised: [Equation]

Original: Concluding the proof.
Revised: , concluding the proof.

Original: C M ATRIX CONCENTRATION
Revised: C M ATRIX CONCENTRATION

Original: The proof of Lemma 3 is a quite direct consequence of Theorem 5.44 from Vershynin (2012).{+
 Lastly, Appendix C.4 establishes some technical tools that weuse throughout the proofs.+}
Revised: This section focuses on random matrix theory. First, Appendix C.1 focuses on the large eigenvaluesof our kernel, and proves Lemma 1. Next, Appendix C.2 focuses on the tail of the eigenvalues,culminating in the proof of Lemma {+
 Lastly, Appendix C.4 establishes some technical tools that weuse throughout the proofs.+}

Revised: C. P ROOF OF L EMMA 1

Original: Westate the  proof here.
Revised: In this proof, we show that the matrix Ψ ⊺ ≤ m Ψ ≤ m /n concentrates around the identity matrix underappropriate assumptions, thereby establishing Equation (5). Let us define ˆ m to be the largest m such that m ∈ O ( nd − β ¯ δ ) . The proof consists of applying

Revised: Theorem 5.44 from Vershynin (2012) to the matrix Ψ ≤ ˆ m .

Original: Proof of Lemma 3.
Revised: Then a simple argument extend it to anyother choice of m . Bound for ˆ m . To apply Theorem 5.44 from Vershynin (2012) we need to verify the theorem’s conditions on the rows of Ψ ≤ ˆ m . In particular, we show that the rows are independent, have a common second moment matrix, and that their norm is bounded.

Original: Letus use [ Ψ ≤  m ] i, :to indicate the i -th row of Ψ ≤  m .
Revised: Let [ Ψ ≤ ˆ m ] i, : indicate the i -th row of Ψ ≤ ˆ m ∈ R n × ˆ m .

Original: Then we can see that:
Revised: We may write each row entry-wise as

Original: [Equation]
Revised: [Equation]

Original: Where ( i ) follows from the fact that Y i ( x) 2 = 1 for any i for any x .
Revised: First, the rows of Ψ ≤ ˆ m are independent since each row depends on a different x i , and we assumedata to be i.i.d..

Original: To see this, it suffices to look at the eigenfunctions, as presented in Equation 9. Then:
Revised: Second, since the eigenfunctions are orthonormal w.r.t. the data distribution, the second moment ofthe rows is E

Original: [Equation]
Revised: [Equation]

Original: Note the rows are isotropic since the eigenfunctions form an orthonormal basis.
Revised: for all rows. Third, to show that each row has bounded norm, we use the fact that the eigenfunctions Y k in

Revised: Equation (13) over {− 1 , 1 } d satisfy Y k

Original: Since the rows of Ψ ≤ m are independent (each row depends on a different sample, and samples are assumed to be i.i.d.), we can  apply Theorem 5.44 from Vershynin (2012).
Revised: ( x i ) 2 = 1 for all k . Thus, the norm of each row is [Equation] We can now apply Theorem 5.44 from Vershynin (2012).

Original: This yields  with probability 1 −  m exp {− ct 2 } :(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: For any t ≥ 0 , this yields the following inequality with probability 1 − ˆ m exp {− ct 2 } :

Original: [Equation]
Revised: [Equation]

Original: n
Revised: where ∆ = t (cid:114) ˆ m

Original: [Equation]
Revised: [Equation]

Original: As long as t (cid:112) mn < 1 .
Revised: Choosing t =(cid:112) n ˆ m yields ∆ = 1 / 2 < 1 , and hence max [Equation]

Original: We then take t =(cid:112) nm to achieve the desired bound.
Revised: = [Equation] Lastly, for d large enough, the error probability is [Equation]

Original: This yields a statementwhich is true with probability 1 − m exp {− cd δ } > 1 − d − δ for d sufficiently large.
Revised: where ( i ) follows from the fact that m ∈ O ( n β ¯ δ ) . Bound for any m < ˆ m . For this it suffices to notice that the matrix Ψ ⊺ ≤ m Ψ ≤ m is a sub-matrix of Ψ ⊺ ≤ ˆ m Ψ ≤ ˆ m . Hence, also Ψ ⊺ ≤ m Ψ ≤ m n − I m is a sub-matrix of Ψ ⊺ ≤ ˆ m Ψ ≤ ˆ m n − I ˆ m .

Revised: Thus, uniformly over all m ≤ ˆ m ,

Original: The proof Lemma 1 is quite tricky since we need to combine diverse strategies for different parts ofthe proof.
Revised: [Equation] Ψ [Equation] m n − I m (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)

Revised: Ψ ⊺ ≤ ˆ m Ψ ≤ ˆ m n − I ˆ m (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) ≤ 1 / 2 w.p. ≥ 1 − cd − β ¯ δ , concluding the proof.

Revised: C. C ONCENTRATION OF THE TERMS AFTER m AND PROOF OF L EMMA 3

Original: For this reason, we  first introduce some definitions.
Revised: In the following, we focus on the concentration of the smallest and largest eigenvalue of the kernelmatrix K >m to prove Lemma 3. However, this proof is involved, and requires some additional tools. In particular, we further decompose K >m into two kernels K 1 and K 2 . We first introduce the twoadditional kernels, and then show in Lemma 11 that K >m = K 1 + K 2 .

Original: Definition 2 (Of useful quantities) .
Revised: Definition 2 ( 1 - 2 decomposition) .

Original: In thesame setting  as in Section 3.1, let n ∈ Θ( d ℓ ) , q∈ Θ (cid:0) d β (cid:1) , max { λ, 1 } ∈ Θ( d ℓ λ ) .
Revised: Consider the setting of Theorem 1, and decompose the convolutional kernel K via Proposition 1 as [Equation] where { S k } k> 0 is a sequence of all subsets S k ⊆ { 1 , . . , d } with γ ( S k ) ≤ q , ordered such thatλ S k ≥ λ S k +1 .

Original: Let m  be such that nλ  m ∈ Θ(max { λ, 1 } ) 
Revised: Lastly, let m ∈ N be such that nλ S m ∈ Θ(max { λ, 1 } ) . We define the index sets [Equation]

Original: Then we define:
Revised: and the corresponding kernels [Equation] where S 1 and S 2 are the squared kernels corresponding to K 1 and K 2 , respectively. Furthermore, as in the original kernel decomposition

Original: [Equation]
Revised: [Equation]

Original: We then define:
Revised: we define the matrices

Original: [Equation]
Revised: [Equation]

Original: Now, let S 1 :=
Revised: and the empirical kernel matrices K 1 , K 2 ∈ R n × n with [ K 1 ] i,j = K 1 ( x i , x j ) and [ K 2 ] i,j = K 2 ( x i , x j ) .

Original: { S i : i > m and | Si | ≤ ¯ L + 1 } , and let us define:
Revised: We further define ¯ m as the largest eigenvalue corresponding to an eigenfunction Y S of degree | S | ≥ ¯ L + 2 , that is, [Equation] Finally, we define

Original: [Equation]
Revised: [Equation]

Original: where S l is the  l -th element of S 1  in a fixed order.
Revised: with the corresponding kernel matrix Q ( d,q ) l . Intuitively, the quantities Ψ 1 , D 1 are the equivalent to Ψ ≤ m , D ≤ m in the original decomposition K = K ≤ m + K >m .

Original: Finally, let us naturally define the respectiveempirical kernel matrices K 1 and K 2  .
Revised: The following lemma establishes that K 1 and K 2 indeed constitute a decompositionof K >m .

Original: Lemma 11 (K 1 , K 2 decomposition) .
Revised: Lemma 11 ( 1 - 2 decomposition) .

Original: In the same setting as Definition 2, for d sufficiently large, we have that:
Revised: In the same setting as Definition 2, for d sufficiently large, we have K >m ( x, x ′ ) = K 1 ( x, x ′ )

Original: [Equation]
Revised: + K 2 ( x, x ′ ) and S >m ( x, x ′ ) = S 1 ( x, x ′ )

Revised: + S 2 ( x, x ′ ) .

Original: Proof.
Revised: Proof.

Original: While it is clear that every eigenfunction beyond m appears exactly once among K 1 and K 2 , we need to guarantee that we are not considering any of the eigenfunctions before m . In K 1 there are no eigenfunction before m by definition.
Revised: For the decomposition of K >m , we have to show that exactly the eigenfunctions with indexlarger than m appear in either K 1 or K 2 , that is, I 1 ∪ I 2 = { k > m } , and that no eigenfunctionappears in both K 1 or K 2 , that is, I 1 ∩ I 2 = ∅ . Furthermore, since we can write S >m ( x, x ′ ) =(

Original: In K 2 all eigenfunctions are polynomials of degree at least ¯
Revised: cid:80) k>m λ

Original: L + 2 .
Revised: 2 S k Y S

Original: Hence, thanks to Lemma 10, we have that all the eigenvalues are such that:
Revised: k ( x ) Y S k ( x ′ ) by Lemma 5, the same argument also yields the 1 - 2 decomposition of

Original: [Equation]{+S >m . First, from the definition of I 
 and I 2 , it follows directly that I 1 ∩ I 2 = ∅ , that I 1 ∪ I 2 ⊇ { k > m } ,and that I 1 ⊆ { k > m } . Hence, to conclude the proof, we only need to show that I 2 ⊆ { k > m } .+}
Revised: {+S >m . First, from the definition of I 
 and I 2 , it follows directly that I 1 ∩ I 2 = ∅ , that I 1 ∪ I 2 ⊇ { k > m } ,and that I 1 ⊆ { k > m } . Hence, to conclude the proof, we only need to show that I 2 ⊆ { k > m } .+}

Original: Hence, for d sufficiently large all eigenvalues are smaller than λ  m , which yields the result. Lemma 12 (Bound on K 1  ) . In the setting defined in Section 3.1, we have that:
Revised: Since the indices are sorted in decreasing order, we equivalently show that, for d sufficiently large, alleigenvalues λ S k with k ∈ I 2 are smaller than λ S m ∈ Θ(max { λ, 1 } /n ) . More precisely, we show that nλ S k ∈ o ( nλ S m ) for any k ∈

Original: [Equation] with probability at least 1 − cd − ¯ δ .Proof.
Revised: I 2 .

Original: The proof follows directly from Lemma 3. We only need to show that |S 1 | , which is just the number of columns of Ψ 1 , is in O ( d · q ¯ L ) . But this follows simply my noticing that in S 1 we include only polynomials of degree no larger than ¯ L + 1 . Hence:
Revised: Applying Lemma 10 yields

Original: [Equation]
Revised: [Equation]

Original: Since C ( l, q, d ) = d (cid:0) q − 1 l −(cid:1) ≤ d (cid:16) e q − 1 l −(cid:17) l − 1 ∈ O ( d · q l − 1 )  .
Revised: where ( i ) follows from | S k | ≥ ¯ L + 2 , and ( ii ) uses ¯ L ≥ L and δ < 1 .

Original: Lemma 13 (Bound on K 2) . In the setting defined in Section 3.1, we have that: [Equation] with probability at least 1 − cq
Revised: Hence, for d sufficiently large,all k ∈ I 2 have λ S k < λ m and therefore k > m by definition.

Original: [Equation] (1 − ¯ δ ) [Equation] Proof.
Revised: Using the 1 - 2 decomposition, we now prove Lemma 3.

Original: We start by decomposing as: [Equation] Where we ( i ) follows from the definition of Q ( d,q ) l ( x, x [Equation]
Revised: We defer the auxiliary Lemmas 12 to 15 afterthe proof, and more technical tools to Appendix C.4.

Original: [Equation] Finally, let us define Q ( d,q ) l to be the kernel matrix with respect to Q ( d,q ) l . Now, we can finally bound: [Equation]
Revised: Proof of Lemma 3.

Original: Where ( i ) follows from Lemma 14 with probability at least 1  − q − ¯ δ , and ( ii ) follows from Assumption 3. For the lower bound we simply look at this inequality:
Revised: We address the bound r 1 and r 2 , and finally bound Tr ( S >m ) . Throughout this proof, we write “with high probability” to indicate with probability at least 1 −cq − min { ¯ δ, 1 − ¯ δ } for some context-dependent constant c .

Original: [Equation] Where ( i ) follows from Lemma 14 with probability at least 1 − q − ¯ δ . Finally, we get to the proof of Lemma 1. Proof of Lemma 1.
Revised: Furthermore, this proof does not establish therelevant matrix concentration results directly, but combines them from auxiliary lemmas.

Original: We prove the Lemma in two parts. First, we address the bound on r , and then wecover the bounds on Tr ( S >m ) . 
Revised: Furthermore,we use the m corresponding to S m , and instantiate Definition 2 with that m . Then, Lemma 11 yieldsthe 1 - 2 decomposition K >m = K 1 + K 2 , which we will henceforth use.

Original: Bound onr
Revised: Bound on

Revised: r

Original: Step 1: rephrase the problem To upper andlower bound r , we notice that it suffices to show that:
Revised: Remember the definition of r 1 and r 2 :

Original: [Equation]
Revised: [Equation]

Original: From this we would get that:
Revised: { λ, I } . To bound those quantities, we have to bound µ min ( K >m ) and ∥ K >m ∥ .

Original: [Equation]
Revised: For the upper bound on∥ K >m ∥ , we use the triangle inequality on ∥ K >m ∥ = ∥ K 1 + K 2 ∥ , and then bound ∥ K 1 ∥ and ∥ K 2 ∥individually. Note that we can write K 1 =

Original: This then implies that:
Revised: Ψ 1 D 1 Ψ ⊺ 1 by definition. Then,

Original: [Equation]
Revised: [Equation]

Original: Hence, we must now show that Equation (18) holds withhigh probability in the setting of interest. Throughout this proof we use multiple times the fact that  K >m = K [Equation]
Revised: where ( i ) follows from Lemma 12 with probability at least 1 − c 3 q − ¯ δ , ( ii ) uses that all eigenvaluesof K 1 are at most λ m by definition, and ( iii ) follows from nλ m ∈ Θ(max { λ, 1 } ) . Finally, Lemma 13 directly yields with probability at least 1 − c 4 q − (1 − ¯ δ ) that ∥ K 2 ∥ ∈ Θ(1) and µ min ( K 2 ) ∈ Θ(1) .

Original: K 2 .
Revised: Hence, for d large enough, with probability at least 1 − ( c 3 q − ¯ δ + c 4 q − (1 − ¯ δ ) ) ≥ 1 − cq − min { ¯ δ, 1 − ¯ δ } , we have [Equation]

Original: Step 2: upper bound
Revised: This implies that with high probability:

Original: [Equation]
Revised: [Equation]

Original: Where in ( i ) we used Lemma 13, in ( ii ) Lemma 12, and in ( iii ) the fact that nλ m ∈ Θ(max { λ, 1 } ) . Step 3: lower bound
Revised: This last equation directly implies that, with high probability:

Original: [Equation]
Revised: [Equation]

Original: Where in ( i ) we used that K 1is postive-semidefinite, and in ( ii ) we used Lemma 13.
Revised: But from the definition of r 1 and r 2 , it follows that r 1 ≤ r 2 . Thus, r 1 , r 2 ∈ Θ(1) .

Original: Bound on Tr ( S m ) : We start by dividing the squared kernel as:
Revised: Bound on Tr ( S >m ) . Finally, we need to show that

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: with high probability.

Original: S
Revised: To do so, we differentiate whether ¯ L = L and ¯ L > L . Intuitively, the case¯ L = L corresponds to interpolation or mild regularization, because the maximum degree of learnablepolynomials with regularization L equals the one without regularization ¯ L . Conversely, ¯ L > Lcorresponds to strong regularization.

Revised: Case ¯ L = L (interpolation or mild regularization):

Revised: In this setting,

Original: [Equation]
Revised: [Equation]

Original: Note that we are dividing the kernel as in Definition 2.
Revised: First, Lemma 14 yields

Original: Let S 1 and S 2 be the kernel matrix ofrespectively S 11 and S 12 . 
Revised: Tr ( S 2 ) ∈ Θ( q − (1 − ¯ δ ) ) with probability at least 1 − c 1 q − (1 − ¯ δ ) . Therefore, [Equation]

Revised: where ( i ) follows from Equation (25).

Original: Upper bound: here we  need to upper-bound the Trace of S >m :
Revised: This directly yields the lower bound [Equation] For the upper bound, we also need to consider Tr ( S 1 ) :

Original: [Equation]
Revised: [Equation]

Revised: where ( i ) follows from Lemma 12 with probability at least 1 − c 2 q − ¯ δ .

Original: To bound Tr ( S 1 ) , notice that S 1 =
Revised: Then, Tr [Equation] S 1 ) ≲ n

Revised: [Equation]

Revised: where ( i ) uses λ k ≤ λ m for all k ∈

Revised: I 1 by definition, ( ii ) uses the assumption nλ m ∈ Θ(max { λ, 1 } ) , and ( iv ) follows from Equation (25).

Revised: Furthermore, ( iii ) uses the following bound of |I 1 | :

Revised: [Equation]

Revised: where C ( l, q, d ) is defined in Equation (17), ( i ) follows from Lemma 9 from d large enough, and ( ii ) is a classical bound on the binomial coefficient.

Revised: Finally, collecting the upper bounds on Tr ( S 1 ) and

Revised: Tr ( S 2 ) yields the desired upper bound

Revised: [Equation]

Revised: with high probability, and hence the overall bound on Tr ( S ≥ m ) with probability at least 1 −( c 1 q −

Revised: [Equation]

Revised: 1 − ¯ δ ) + c 2 q − ¯ δ ) ≥ 1 − cq − min {

Revised: ¯ δ, 1 − ¯ δ } for the case where ¯ L = L .

Revised: Case ¯ L > L (strong regularization):

Revised: In this setting, the dominating rate will arise from Tr ( S 1 ) .

Revised: Westart by linking ¯ δ and δ analogous to Equation (25):

Revised: [Equation]

Revised: Next, as in the previous case, Lemma 14 yields Tr ( S 2 )

Original: Ψ 1 D 21 Ψ ⊺ 1  , hence:
Revised: ∈ Θ( q − (1 − ¯ δ ) ) with probability at least 1 − c 1 q − (1 − ¯ δ ) , and therefore [Equation] where ( i ) follows from Equation (26), and ( ii ) from ¯ L ≥ L + 1 .

Revised: Due to the slower growth in this setting, we consider Tr ( S 2 ) only for the upper bound of Tr ( S ≥ m ) , but not for the lower bound.

Revised: To bound Tr ( S 1 ) , we first use

Original: [Equation]
Revised: [Equation]

Original: Where ( i ) follows from Lemma 12.
Revised: where ( i ) and ( ii ) follow from Lemma 12 simultaneously with probability at least 1 − c 2 q − ¯ δ . We then decompose the sum over all squared eigenvalues with index in I 1 as n [Equation] + n (cid:88) k ∈I 1 | S k | =

Revised: L + λ 2 S k

Revised: (cid:124) (cid:123)(cid:122) (cid:125)

Revised: T+ n ¯ L + (cid:88) l = L +3 (cid:88) k ∈I 1 | S k | =

Original: Now we move our attention to Tr ( S  2 ) .
Revised: l λ 2 S k (cid:124) (cid:123)(cid:122) (cid:125) T. In the following, we analyze T 1 , T 2 , and T 3 separately.

Revised: Upper bound For the upper bound of Tr ( S ≥ m ) , we bound all three terms from above.

Revised: First, we upper-bound T 1 as follows:

Revised: [Equation]

Revised: where ( i ) follows from λ k ≤ λ m for all k > m due to the decreasing order of eigenvalues.

Revised: Furthermore, ( ii ) uses nλ m ∈ Θ(max { λ, 1 } ) and the fact that C ( l, q, d ) ∈ O ( dq l − 1 ) , which follows from

Revised: Lemma 9 together with a classical bound on the binomial coefficient: for d large enough,

Revised: [Equation]

Revised: dq l −(cid:1) .

Revised: Second, the upper bound of T 2 arises as follows:

Revised: [Equation]

Revised: where ( i ) follows from the eigenvalue formula in Proposition 1, and ( ii ) uses q + 1 − γ ( S ) ≤ q .

Revised: Next, ( iii ) uses Assumption 1 and applies the bound B ( L + 2 , q )

Revised: = (cid:0) qL + (cid:1) ≤ ( eq/ ( L + 2))

Revised: L +2 .

Revised: Step( iv ) uses Y S ( x ) Y S ( x ) = 1 for all S and x ∈ {− 1 , 1 } d , together with the definition of Q ( d,q ) L +2 .

Revised: Lastly,( v ) applies Lemma 15.

Revised: Third, we upper-bound T 3 .

Revised: For this, let ˜ k be the smallest index that corresponds to a set S ˜ k with | S ˜ k

Revised: | =

Revised: L + 3 .

Revised: Since the eigenvalues are indexed in decreasing order, λ S ˜ k is the largest eigenvalue corresponding to a set of size L + 3 .

Revised: Furthermore, for d large enough, Lemma 10 yields that λ S ˜

Original: Let ¯ m be the largest eigenvalue  of a polynomial of degree ¯ L + 2 . 
Revised: k is the maximum eigenvalue for all sets of size at least L + 3 . Then, for d large enough, [Equation]

Revised: where ( i ) follows from the fact that the sum of all the eigenvalues of the kernel must be bounded bya constant, thus the sum of a subset of the eigenvalues must be bounded as well.

Original: Then:
Revised: Furthermore, ( ii )follows from Lemma 10 with | S ˜ k | = L + 3 . [Table]

Revised: where ( i ) follows from Proposition 1.

Revised: Step ( ii ) uses that all k ∈ I 1 have | S k | ≤ q by definition; hence the k ∈ I 1 with | S k | ≤

Revised: ⌊ q/ 2 ⌋ are a subset of I 1 .

Revised: Next, ( iii ) follows since q ≥ ⌊ q/ 2 ⌋ and q + 1 − γ ( S ) ≥ q/ 2 .

Revised: Step ( v ) uses Y S (

Revised: x ) Y S ( x ) = 1 for all S and x ∈ {− 1 , 1 } d , and ( vi ) from Lemma 15 with ⌊ q/ 2 ⌋ as filter size.

Revised: Finally, ( iv ) follows from the fact that B ( L + 2 , q ) and

Revised: B ( L +2 , ⌊ q/ 2 ⌋ ) are of the same order; this follows from a classical bound on the binomial coefficient:

Revised: [Equation]

Revised: Ultimately, we may summarize all individual bounds for the case ¯ L > L as follows:

Revised: [Equation]

Revised: with prob.

Revised: at least 1 − c 1 q − (1 − ¯ δ )

Revised: Tr

Revised: [Equation]

Revised: S

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: 1 + T 2 + Twith prob.

Revised: at least 1 − c 2 q − ¯ δ

Revised: T

Revised: [Equation]

Revised: O

Revised: [Equation]

Revised: Combined, those bounds yield Equation (24) with probability at least 1 − ( c 1 q − (1 − ¯ δ ) + c 2 q − ¯ δ )

Revised: ≥1 − cq − min { ¯ δ, 1 − ¯ δ } for the case where ¯ L > L .

Revised: Since the cases are exhaustive, and hold both withsame probability, this concludes the proof of the bound on Tr ( S >m ) .

Revised: The statement of Lemma 3 then follows from taking the union bound on the probabilities over theindividual proof parts.

Original: [Equation]
Revised: [Equation]

Revised: Finally, we use the following technical lemmas in the proof of Lemma 3.

Revised: Lemma 12 (Bound on K 1 ) .

Revised: In the setting of Definition 2, uniformly over all choices of λ , we have (cid:13)(cid:13)(cid:13)(cid:13) Ψ ⊺ 1 Ψn − I (cid:13)(cid:13)(cid:13)(cid:13) ≤ 1 / 2 , with probability at least 1 − cq − ¯ δ .

Revised: Proof.

Revised: The proof follows a very similar argument to Lemma 1.

Revised: Still, minor differences prevent usfrom using that result directly.

Revised: We start by defining

Revised: [Equation]

Revised: Notice that the definition of ˆ I 1 does not depend on m , and hence it does not depend on λ .

Original: Where ( i ) follows from Lemma 5, and ( ii
Revised: Furthermore,notice that for any m , I 1 ⊆ ˆ I 1 . We start by computing the

Revised: [Equation]

Revised: where ( i ) follows from Lemma 9 for d large enough, ( ii ) is a classical bound on the binomialcoefficient, and ( iii ) follows from the fact that the largest degree monomial dominates the others.

Revised: Now we define ˆ Ψ 1 ∈ R n ×| ˆ I 1 | in analogous way to how Ψ 1 is defined in Definition 2.

Revised: Using the same arguments used already in the proof of Lemma 1, we can show that the rows of ˆ Ψ 1 are independent, with norm bounded by | ˆ I 1 | , and with expectation of outer product equal to

Revised: I | ˆ I 1 | .

Revised: Again as in Lemma 1, we can apply Theorem 5.44 from Vershynin (2012), and, choosing t = 1 n (cid:113) n | ˆ I 1 | , we get that: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Revised: [Equation]

Revised: To conclude the proof, it suffices to remember that, uniformly over all choices of λ , and consequential choice of m , I 1 ⊆

Revised: ˆ

Revised: I 1 .

Revised: Thus:

Revised: [Equation]

Revised: concluding the proof.

Revised: Lemma 13 (Bound on K 2 ) .

Revised: In the setting of Definition 2, assuming that the kernel satisfies Assumption 1, with probability at least 1 − cq − (1 − ¯ δ ) , we have

Revised: [Equation]

Original: Lower move on similar
Revised: for some positive constants c 1 , c 2 . Proof. For the upper bound on the largest eigenvalue, Proposition 1 yields the following decomposition of K 2 ( x, x ′ ) : [Equation]

Revised: where Q ( d,q ) l ( x, x ′ ) is defined in Definition 2.

Original: For
Revised: Then, using the triangle inequality and positivity of the ξ ( q ) l from Assumption [Equation] we have [Equation]

Revised: where Q ( d,q ) l is the kernel matrix corresponding to Q ( d,q ) l ( x, x ′ ) , ( i ) uses Lemma 16, and ( ii ) followsfrom Assumption 1.

Revised: The lower bound follows from

Original: [Equation]
Revised: [Equation]

Original: Where ( i ) follows from Lemma 12.
Revised: where ( i ) follows from Lemma 16.

Revised: Since Lemma 16 yields both the upper and lower bound at once with probability 1 − cq − (1 − ¯ δ ) , thisconcludes the proof.

Revised: Lemma 14 (Bound on Tr ( S 2 ) ) .

Revised: In the setting defined in Section 3.1, we have that:

Revised: [Equation]

Original: For S 2 we need to be a bit more precise than in the upper bound.
Revised: with probability at least 1 − cq − (1 − ¯ δ ) . Proof. We start by proofing two technicalities. Proof that ∥ S 2 ∥ ≤ λ ¯ m ∥ K 2 ∥ .

Revised: Let v be an arbitrary vector in R n , then:

Original: [Equation]
Revised: [Equation]

Original: Where in ( i ) we use the fact that ¯ m is the index of the smallest eigenvalue of a degree ¯  L +polynomial.
Revised: where in ( i ) we used the definition of m together with the fact that the eigenvalues are sorted indescending order. Proof that λ ¯ m ∈ Θ [Equation] L +

Revised: [Equation]

Revised: To show this, we show that for any k in I 2 , λ k ∈ O (cid:16) 1 dq ¯ L + (cid:17) and thatthere exists a ˜ m such that λ ˜ m ∈ Ω (cid:16) 1 dq ¯ L + (cid:17) .

Original: Q ( d,q ) ¯ L +2 is defined in Equation 10.
Revised: Since λ ¯ m is the largest of the eigenvalues λ S k , k ∈ I 2 ,those two fact imply that λ ¯ m ∈ Θ [Equation] Let k be an arbitrary index in I 2 , then | S k | ≥ ¯ L + 2 . Lemma 10 yields that:

Original: Let Q ( d,q ) ¯ L +2 be the kernel matrix of Q ( d,q )
Revised: [Equation] This concludes the proof that for any k in I 2 , λ k ∈ O (cid:16) [Equation] dq ¯ L + (cid:17) .

Revised: Now we show that there exist ˜ m such that λ ˜ m ∈ Ω (cid:16) 1 dq ¯ L + (cid:17) .

Revised: To do so, we take ˜ m be such that

Original: ¯ L +2 . 
Revised: S ˜ m = { 1 , 2 , . . . , L + 2 } . Note that ˜ m ∈

Revised: I 2 for d sufficiently large, since | S ˜ m | =

Revised: L + 2 .

Original: Then,
Revised: Then,

Original: Lemma 14 yields that µ min (cid:16)
Revised: Proposition 1 yields the following formula for λ S ˜ m : [Equation] Upper bound. The upper bound follows by simply applying the last two results:

Original: Q ( d,q ) ¯ L + (cid:17) ≥ 1 / 2 with probability at least 1 − cq 1 − ¯ δ .
Revised: [Equation] where ( i ) follows with probability at least 1 − cq − (1 − ¯ δ ) from the Lemma 13. Lower bound. For the lower bound we need a more refined argument.

Original: From this it follows that:
Revised: [Equation] where ( i ) follows Proposition 1. In ( ii ) we used that as long as γ ( S ) ≤ ⌊ q/ 2 ⌋ , q + 1 − γ ( S ) ≥ qand q ≥ ⌊ q/ 2 ⌋ . In ( iii ) we use the bound on the binomial coefficient (cid:16)

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: q

Revised: [Equation]

Revised: ≤ (cid:16) eq ¯ L + (cid:17) ¯ L +2 to compute:

Original: [Equation]
Revised: [Equation]

Original: Cases distinction: now, there are two possible cases: ¯ L = L (Interpolation or mild regularization)
Revised: where c ′ ¯ L is a constant that depends on ¯ L . Finally, in ( iv ) , we used the definition of Q ( d, ⌊ q/ 2 ⌋ ) ¯ L + in Definition 2.

Original: In this setting:
Revised: This implies that:

Original: [Equation]
Revised: [Equation]

Original: Where in ( i ) we use Lemma 10. Furthermore,
Revised: Hence, we finally bound Tr ( S 2 ) :

Original: [Equation]
Revised: [Equation]

Original: Where ( i ) follows from thefact that nλ S i ≤ c max { λ, 1 } . The proof follows by combiningthese rates with the upper and lower bounds on Tr ( S >m ) found before and by noticing that max { λ, 1 } ∈ Θ( d ℓ λ ) . ¯ L ≥ L + 1 (Strong regularization)
Revised: where ( i ) follows from the lower bound on µ min

Original: In this settings, it is easy to

Original: [Equation]
Revised: [Equation]

Original: T 1 behaves at Tr ( S 1 ) in the case where b L = L , hence T 1 ∈ O ( q − δ d ℓ λ ) . It is easy to see that the eigenvalues in T 3 are small enough that T 3 ∈ o ( q − 1 ) . Hence, the dominating factor will be T 2 . Now we bound it:
Revised: L +

Original: [Equation]
Revised: [Equation]

Original: Where the proof of (cid:80)
Revised: found in Lemma 16, and on the lowerbound on ξ ( q ) ¯ L +2 is from Assumption 1. This concludes the proof. Lemma 15 (Diagonal elements of Q ( d,q ) ) . For any x ∈ {− 1 , 1 } d , for any d ∈ N , q ∈ N , q < d/ 2 , for any l ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: S
Revised: Proof. We compute:

Original: [Equation]
Revised: [Equation]

Original: 2 ,q ) = 1 can be found in the beginning of the proof of Lemma 6, when addressing the diagonal elements of Q ( d,q ) l . Finally, we show a lower-bound for T 2 :
Revised: where ( i ) is the definition of Q ( d,q ) l ( x, x ) itself. ( ii ) follows noticing that the hypercube eigenfunctions squared are always equal to 1 , as clear from their definition in Equation (13).

Revised: ( iv ) follows from

Revised: Lemma 9, ( v ) from the hockey-stick identity and ( vi ) from the formula to compute B in Equation (15).

Revised: The last thing to justify is ( iii ) .

Revised: For this we notice that

Revised: [Equation]

Revised: matches the definition of

Original: [Equation]
Revised: [Equation]

Revised: found in the proof of Lemma 9.

Original: Finally, we combine the upper and lower bounds with the rate just found.
Revised: We then show the following recurrence: [Equation] where in ( i ) we use the fact that C ( l, γ, d ) = (cid:80) qγ = l ˜ C ( l, γ, d ) , which follows from the definition. Furthermore, the second sum only reaches q − 1 since the last element would be 0 .

Original: Noticing thatmax { λ, 1 } ∈ Θ( d ℓ λ )  the proof is concluded.
Revised: Applying recursively this formula q − l times, we get: [Equation] which explains ( iii ) in Equation (27). This concludes the proof.

Original: C. R ANDOM MATRIX THEORY LEMMAS
Revised: C. R ANDOM MATRIX THEORY LEMMAS

Original: In this subsection are contained the key random matrix concentration needed for the proof to gothrough.
Revised: In this subsection are contained the key random matrix concentration needed for the proof to gothrough.

Original: Lemma 14 (Hypercube bound on the tail kernel) .
Revised: Lemma 16 (Bound on the kernel’s tail) .

Original: In the same setting as Definition 2, with probability at least 1 − cq (1 − ¯ δ ) :
Revised: In the same setting as Definition 2:

Original: [Equation]
Revised: [Equation]

Original: Where Q ( d,q ) l is defined in Equation 10.
Revised: with probability at least 1 − cq − (1 − ¯ δ ) .

Original: Notice that, from this lemma, it naturally follows that ∥ Q ( d,q ) l ∥ ≤ 1 . 5 , and µ min (cid:16)
Revised: Notice that, from this lemma, it naturally follows that ∥ Q ( d,q ) l ∥ ≤ 1 . 5 , and µ min (cid:16)

Original: Q ( d,q ) l (cid:17) ≥ 1 / 2 .
Revised: Q ( d,q ) l (cid:17) ≥ 1 / 2 .

Original: Now we present the proof:
Revised: Now we present the proof:

Original: Proof of Lemma 14.
Revised: Proof of Lemma 16.

Original: First, we show that the diagonal elements of Q ( d,q ) l are just 1 .
Revised: Thanks to Lemma 15, we know that the diagonal elements of Q ( d,q ) l are just 1 .

Original: Hence, we compute: [Equation] q
Revised: Hence, we define W ( d,q ) l := Q ( d,q ) l − I n .

Original: [Equation]
Revised: Our objective is now to show that ∥ W ( d,q ) l ∥ ≤ 1 / 2 at the same time for every l > ¯ L + 1 with high probability. We will however use two different strategies for small l and large l .

Original: d
Revised: In particular, let us define

Original: [Equation]
Revised: [Equation]

Original: Where ( i ) follows from the fact Y S ( x ) 2 = 1 , and ( ii ) follows noticing that we are counting a set

Original: S exactly ( q + 1 γ ( S )) times.

Original: And, by summing all C ( l, γ, d ) over γ , we count S again exactly( q + 1 γ ( S )) times.

Original: Now, let W ( d,q ) l := Q ( d,q ) l − I n . Our objective is now to show that∥ W ( d,q ) l ∥ ≤ 1 / 2 at the same time for every l > ¯ L + 1 with highprobability. We will however use two different strategies for l ≤ T , and l > T . We then show howto pick T  .
Revised: Notice that T does not depend on d nor ℓ λ .

Original: Terms l < T : here we use Lemma 15.
Revised: Terms l < T : here we use Lemma 17.

Original: To successfully apply it,however, we need to find an appropriate M  , and show that the assumptions of the theorem holds.
Revised: To successfully apply it, we need to find an appropriate M ( l,q ) , and show that the assumptions of the Lemma holds.

Original: The first assumption has already been checked, since we showed
Revised: The first assumption has already been checked, since we showed

Original: that diag( Q ( d,q ) l ) =
Revised: that diag( Q ( d,q ) l ) =

Original: I n .
Revised: I n .

Original: For the second one we can compute:
Revised: For the second one we can compute, for any i ̸ = j : [Equation] ≤ [Equation]

Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Revised: (cid:12)(cid:12)(cid:12)(cid:12) ≤

Original: [Equation]
Revised: [Equation]

Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:104)

Revised: W ( d

Revised: [Equation]

Revised: (cid:105)

Original: Hence we  chose M ( l,q ) =
Revised: i [Equation] (cid:12)(cid:12)(cid:12)(cid:12) , where in ( i ) we use that the eigenfunctions are orthonormal. We then chose M ( l,q ) =

Original: d B ( l, q ) /lq .
Revised: d B ( l, q ) /lq .

Original: The  l factor will be needed for the third assumption to hold.
Revised: The extra l factor will be needed for the third assumption to hold.

Original: For the third assumption we compute:
Revised: We now move to the third assumption. As in Lemma 17, let p ∈ N , p ≥ 2 , then:

Original: [Equation]
Revised: [Equation]

Original: Where ( i ) follows from hypercontractivity as in Lemma 16, and ( ii )follows from the formula(cid:0) q −
Revised: where ( i ) follows from hypercontractivity as in Lemma 18, ( ii ) from the definition of C ( l, q, d ) . ( iii )follows from, Lemma 9, the formula of B ( l, q ) in Equation (15), the formula (cid:0)

Original: [Equation]
Revised: [Equation]

Original: −(cid:1)
Revised: (cid:1) =

Original: = ql (cid:0) ql (cid:1) , and the definition ofC ( l, q, d ) , B ( l, q ) and M ( l,q ) .
Revised: ql (cid:0) ql (cid:1) , and thedefinition of M ( l,q ) .

Original: Finally, we can apply Lemma 15 and we get that for any p  > 2 :
Revised: Finally, we can apply Lemma 17 and we get that for any p ∈ N , p > 2 :

Original: [Equation]
Revised: [Equation]

Original: In particular, if we take p ≥ 2 β +
Revised: In particular, if we take p ≥ 2 β +

Original: [Equation]
Revised: [Equation]

Original:  ¯ δ ) , then we get that:
Revised: 1 − ¯ δ ) β − (¯ L + 2 − ¯ δ ) , then we get that:

Original: [Equation]
Revised: [Equation]

Original: We can than take the union bound for all l = ¯ L + 1 , . . .
Revised: We can than take the union bound for all l = ¯ L + 1 , . . .

Original: , T , and we get: 
Revised: , T , and we get: Pr

Original: [Equation]
Revised: [Equation]

Original: Where C ¯ L depends only  ¯ L and T , which are fixed in our setting. 
Revised: where C ¯ L depends only on ¯ L and T , which are fixed in our setting. We remark that ¯ L does notdepend on ℓ λ .

Original: We later show how to pick T .

Original: Since we cannot control the dependency of c l on l , with this strategy, we cannot bound ∥ W ( d,q ) l ∥ forall l at the same time, but only  a fixed amount of it.
Revised: Since we cannot control the dependency of c l on l , with this strategy, we cannot bound ∥ W ( d,q ) l ∥ forall l at the same time, but only for a finite number of different values of l , i.e. all l ≤ T .

Original: Hence, we present another technique for the restof the terms:
Revised: We presentanother technique for the rest of the terms:

Original: Terms l ≥ T : for this section we can just bound the off-diagonal part using the Frobenius norm:
Revised: Terms l ≥ T : for this section we can just bound the off-diagonal part using the Frobenius norm:

Original: [Equation]
Revised: [Equation]

Original: Where in ( i ) we assume that max i,j [ W ( d,q ) l ] 2 ij ≤ 1 / 4 n 2 .
Revised: where in ( i ) we use that max i,j [ W ( d,q ) l ] 2 i,j ≤ 1 / 4 n 2 .

Original: Now we compute the probability that this is  the case:
Revised: Now we compute the probability that this is indeed the case:

Original: Pr

Original: [Equation]
Revised: [Equation]

Original: Where in ( i ) we used Markov inequality (aslo known as Chebychev for second moments), and in ( i ) we used rate of n and the bound B ( l, q ) = (cid:0) ql (cid:1) ≥ ( q/l ) l .
Revised: where in ( i ) we used Markov inequality (also known as Chebychev for second moments), and in ( ii ) we used rate of n and the bound B ( l, q ) = (cid:0) ql (cid:1) ≥ ( q/l ) l .

Original: Notice that the constant C does not depend on l .
Revised: Notice that the constant C does not depend on l .

Original: Now,since we assume l > T , we can compute the probability that max i,j [ W ( d,q ) l [Equation] ij
Revised: Now, we can compute the probability that max i,j [ W ( d,q ) l ] 2 i,j ≤ 1 / 4 n 2 for all l ≥ T at the same time.

Original: [Equation]

Original: nfor all l > T , which indeed implies that ∥ W ( d,q ) l ∥ ≤ 1 / 2 for all l > T .

Original: Thanks to union bound this happens with probability 1 − △ , where:
Revised: Thanks to union bound this happens with probability 1 − ∆ , where:

Original: [Equation]
Revised: [Equation]

Original:  To conclude the proof, we just take T > 4 β + 4(¯ L + ¯ δ ) . Notice that T is fixed, and it does not depend on d  .
Revised: where in ( i ) we used the convergence of a classical series. To conclude the proof, we remember that T > 4 β + 4(¯ L + ¯ δ ) , hence, it is easy to verify that ∆ ∈ O ( d − 1 ) .

Original: Finally, it is easy to verify that
Revised: Which implies that:

Original: △ ∈
Revised: [Equation]

Original: O ( d − 1  ) . Hence:
Revised: But thanks to Equation (28), the fact that max i,j [ W ( d,q ) l ] 2 i,j ≤ 1 / 4 n 2 implies that ∥ W ( d,q ) l ∥ ≤ 1 / 2 , hence:

Original: [Equation]
Revised: [Equation]

Original:  The next statement is a non-asymptotic version of Proposition 3 from Ghorbani et al.
Revised: Concluding the proof. The next statement is a non-asymptotic version of Proposition 3 from Ghorbani et al.

Original: Lemma 15 (Graph argument) .
Revised: Lemma 17 (Graph argument) .

Original: Let W ∈ R n × n be a random matrix such that:
Revised: Let W ∈ R n × n be a random matrix such that:

Original: • W i,i = 0 , ∀ i ∈ [ n] .
Revised: • W i,i = 0 , ∀ i ∈ 1 , . . . , n .

Original: • There exists a M > 0 , such that 
Revised: • There exists a M > 0 , such that for any i ̸ = j ∈ 1 , . . . , n :

Original: [Equation]
Revised: [Equation]

Original: • There exists a l such that, for any p  :
Revised: • There exists a l such that, for any p ∈ N , p ≥ 2 :

Original: [Equation]
Revised: [Equation]

Original: Then, we have that, for any p ∈ N , p > 2 :
Revised: Then, we have that, for any p ∈ N , p > 2 :

Original: [Equation]
Revised: [Equation]

Original: Where c 
Revised: where c 1 and c 2 depend on l .

Original: Proof.
Revised: Proof.

Original: Doing thesame stepsdone in the proof of Proposition 3 from Ghorbani et al.
Revised: Repeating the steps in the proof of Proposition 3 from Ghorbani et al.

Original: (2021), we getthat:
Revised: , we get

Original: [Equation]
Revised: [Equation]

Original: Note that for Ghorbani, M = d l , but we move in a slightly more general setting.
Revised: Note that the proof in Ghorbani et al. (2021) assumes M to be in the order of d l . We get rid of thisassumption and keep M explicit.

Original: Furthermore,instead of Lemma 4 from 15 within the proof, we use  Lemma 16.
Revised: Furthermore, Ghorbani et al. (2021) use their Lemma 4 during theirproof, but we use our Lemma 18 instead.

Original: Now, we use Markov inequality to get a high probability bound:
Revised: Now, we use Markov inequality to get a high probability bound:

Original: [Equation]
Revised: [Equation]

Original: Which, renaming the constants, yields the thesis.
Revised: Which, renaming the constants, yields the thesis.

Original: Lemma 16 (Hypercontractivity) .
Revised: Lemma 18 (Hypercontractivity) .

Original: For any l ∈ N , for any p ≥ 2 : 
Revised: For any l ∈ N , for any p ≥ 2 : [Equation] , x

Original: [Equation]
Revised: [Equation]

Original: Where Q ( d,q ) l ( x, x ′ ) is defined in Equation 10.
Revised: where Q ( d,q ) l ( x, x ′ ) is defined in Definition 2.

Original: Proof.
Revised: Proof.

Original: Let z be the entry-wise product between x and y .
Revised: Let z be the entry-wise product between x and y .

Original: For any S , Y S ( x ) Y S ( x ′ ) depends only on z :
Revised: For any S , Y S ( x ) Y S ( x ′ ) depends only on z :

Original: [Equation]
Revised: [Equation]

Original: Hence, given the definition of Q ( d,q ) l ( x, x ′ ) , also Q ( d,q ) l ( x, x ′ ) depends only on z .
Revised: Hence, given the definition of Q ( d,q ) l ( x, x ′ ) , also Q ( d,q ) l ( x, x ′ ) depends only on z .

Original: Finally, note thatz ∼ U ( {− 1 , 1 } d ) .
Revised: Finally, note thatz ∼ U ( {− 1 , 1 } d ) .

Original: Hence, we can use Hypercontractivity Beckner (1975), as for instance in Lemma 4from
Revised: Hence, we can use hypercontractivity (Beckner, 1975), as for instance in Lemma 4from

Original: Misiakiewicz & Mei (2021) to conclude the proof.
Revised: Misiakiewicz & Mei (2021), to conclude the proof.

Revised: D O PTIMAL R EGULARIZATION AND T RAINING E RRORD. O PTIMAL R EGULARIZATION

Revised: In the main text we often refer to the optimal regularization λ opt , however it is not clear how thisregularization term can be computed.

Revised: While we do not expect to compute it exactly, we show a wayto compute the rate of it.

Revised: First we introduce some additional notation:

Revised: [Equation]

Original: D P ROOF OF T HEOREM 2: T RAINING ERROR
Revised: where the dependency with ℓ λ is also through δ = ℓ − ℓ λ − 1 β [Equation] Note that Intuitively, in the setting of Theorem 1, we have that with high probability, [Equation]

Revised: Risk

Revised: [Equation]

Revised: Hence, those quantities represent the exponent of the rate of variance, bias, and risk.

Revised: We then define:

Revised: [Equation]

Revised: Note that ℓ λ min is a set that might have cardinality larger than one, but it cannot be empty.

Revised: This isbecause η is a continuous function, and [0 , ¯ ℓ ] is a closed set.

Revised: The quantity ¯ ℓ defines the scope of our minimization searching domain, guaranteeing that theconstraint on L ∗ in Theorem 1 is respected.

Revised: For this reason, we are not considering the case wherethe optimal regularization parameter is larger than ¯ ℓ .

Revised: Yet, simple intuition reveals that such a strongregularization parameter yields models where the risk does not even vanish.

Revised: Thus, we still claim thatlooking for the optimum within [0 , ¯ ℓ ] is sufficient.

Revised: The next Lemma goes more in detail about the properties of η, η b , η v .

Revised: Lemma 19 (Properties of η ) .

Revised: Assuming that ℓ σ > 0 , β ∈ (0 , 1) , L ∗ ∈ (cid:104) 1 , ⌈ ℓ − 1 β ⌉ (cid:105) ∩ N .

Revised: Let¯ ℓ = ℓ − 1 − β

Revised: ( L ∗ − 1) , then the following holds:

Revised: η v ( l ; ℓ, ℓ σ , β ) is a monotonically non-increasing function with respect to l , and η b ( l ; ℓ, L ∗ , β )is a monotonically increasing one.2.

Revised: If there exists ˜ l ∈ (cid:2) 0 , ¯ ℓ (cid:3) such that η v (˜ l ; ℓ, ℓ σ , β ) =

Revised: η b (˜ l ; ℓ, L ∗ , β ) , then η (˜ l ; ℓ, ℓ σ , L ∗ , β ) =

Revised: η min , otherwise, η (0; ℓ, ℓ σ , L ∗ , β ) =

Revised: η min .

Revised: Let l be larger than l for any l ∈ ℓ λ min , then:

Revised: [Equation]

Revised: Let l be smaller than l for any l ∈ ℓ λ min , let c ℓ σ be a constant that depends only on ℓ σ , ℓ, β .

Revised: Then, if η ( l ; ℓ, ℓ σ , β ) ≤ η min + c ℓ σ :

Revised: [Equation]

Revised: Rate of the optimal regularization parameter vs. optimal rate of the regularization parameter: for theoptimal regularization parameter let ℓ λ opt be such that λ opt ∈ Θ( d ℓ λ opt ) .

Revised: Then it is not clear a priorithat ℓ λ opt minimizes η .

Revised: However, Lemma 20 bridges the two terms and guarantees that this is the casewith high probability.

Revised: Lemma 20 (Optimal regularization) .

Revised: In the same setting as in Theorem 1, for d sufficiently large, with probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } , there exists l ∈ ℓ λ min such that

Revised: [Equation]

Revised: Finding optimal parameters: now that we understand the importance of finding the minimal rate,we notice that Lemma 19 suggests a simple strategy to find a l ∈ ℓ λ min .

Revised: We start by searching theintersection of η v and η b in [0 , ¯ ℓ ] : if it is found, then the intersection point is optimal, otherwise justchoose l = 0 .

Revised: Note that the intersection is unique and, if exists, it is easy to compute since η v ismonotonically non-decreasing, and η b is monotonically increasing.

Revised: Multiple minimizer issue: the last caveat here is that ℓ λ min might contain multiple values, and, in thatcase we are not guaranteed that our algorithm retrieves the one which is the rate of λ opt .

Revised: It mightinstead find a different l ∈ ℓ λ min .

Revised: Yet, Theorem 1 guarantees that both the optimally regularizedestimator and any estimator regularized with λ ∈ d l , l ∈ ℓ λ min , have a risk vanishing with the samerate n η min with high probability.

Revised: This made it possible to plot the rate of the optimally regularizedestimator in Figure 1a.

Revised: Closed formula solutions: because of the phenomenon of multiple descent, it is hard to write bothℓ λ opt or β ∗ in an easily readable closed form.

Revised: Still, simple optimization algorithms make it easy toretrieve such quantities.

Revised: Proof of Lemma 20.

Revised: Let ℓ λ opt be defined such that:

Revised: [Equation]

Revised: Combining the bound on the bias and the variance in Theorem 1, we get that for any ℓ λ , λ ∈ Θ( ℓ λ ) ,

Revised: [Equation]

Revised: Throughout the rest of the proof we will write η ( ℓ λ ) to indicate η ( ℓ λ ; ℓ, ℓ σ , L ∗ , β ) , for simplicity.

Revised: We will also not repeat that each step true with probability at least 1 − cd − min { ¯ δ, 1 − ¯ δ } , but we are implying it.

Revised: Now we want to prove that η ( ℓ λ opt ) is close to η min .

Revised: Hence, we notice that Equation (29) applied to ℓ λ opt and any l ∈ ℓ λ min implies that:

Revised: [Equation]

Revised: where 0 < c 1 < c 2 are the constants hidden by Θ in Equation (29).

Revised: Now, we use the optimality of λ opt :

Revised: [Equation]

Revised: To conclude the proof we need to somehow show that there exists l ∈ ℓ λ min sufficiently close to ℓ λ opt .

Revised: Formally we need to show that:

Revised: [Equation]

Revised: [Equation]

Revised: Once we have shown that this is the case, we can simply use the definition of ℓ λ opt and compute:

Revised: [Equation]

Revised: where in ( i ) we use Equation (30).

Revised: This concludes the proof.

Revised: Now we need to show that Equation (30)actually holds.

Revised: To do so, we distinguish between three cases:

Revised: ( ℓ

Revised: λ

Revised: opt ∈ ℓ λ min ): in this

Revised: case we simply notice that we can take l

Revised: ∈

Revised: ℓ λ

Revised: min to

Revised: be exactly

Revised: ℓ λ opt

Revised: , hence:

Revised: [Equation]

Revised: ( ℓ λ opt < l , for any l ∈ ℓ λ min ):

Revised: let l be min

Revised: ( ℓ

Revised: λ min ) .

Revised: Then, thanks to Lemma 19, we have that,

Revised: as long

Revised: as

Revised: η ( ℓ λ opt ) − η

Revised: [Equation]

Revised: Notice that for n sufficiently large, it is always the case that η ( ℓ λ opt ) − η min ≤ c .

Revised: Hence, we have that:

Revised: [Equation]

Revised: Now, notice that since ℓ λ opt ≤ l , d ℓ λ opt − l ∈ O (1) , furthermore:

Revised: [Equation]

Revised: where in ( i ) we used that n ∈ Θ( d ℓ ) .

Revised: Since both d l − ℓ λ opt and its inverse are ∈ O (1) , and l ∈ ℓ λ min , we can conclude that there exists a l ∈ ℓ λ min , such that:

Revised: [Equation]

Revised: ( ℓ λ opt > l , for any l ∈ ℓ λ min ): let l be max( ℓ λ min ) .

Revised: Then, thanks to Lemma 19, we have that:

Revised: [Equation]

Revised: Hence, we have that:

Revised: [Equation]

Revised: Analogously to the previous case, this implies that d ℓ λ opt − l ∈ O (1) , and the fact that ℓ λ opt > l implies that d ℓ λ opt − l ∈ Ω(1) .

Revised: Together, this means that there exists a l ∈ ℓ λ min such that:

Revised: [Equation]

Revised: Proof of Lemma 19.

Revised: Throughout this proof we will drop the dependencies ℓ, ℓ σ , L ∗ , β in the notationof η, η v , η b , simply writing η ( l ) , η v ( l ) , η b ( l ) .

Revised: Continuity: the proof start by showing that η v ( l ) and η b ( l ) , are continuous functions of l .

Revised: η b ( l ) is an affine function of l , hence it is also continuous.

Revised: η v ( l ) , however, has a hidden dependency with l through δ , which is not linear.

Revised: Hence, to show that η v ( l ) is continuous, we need to show that min { δ, 1 − δ } is continuous.

Revised: This is equivalent to show that:

Revised: [Equation]

Revised: is continuous.

Revised: This is because ϖ

Revised: [Equation]

Revised: β is a linear function of l .

Revised: Finally, it is easy to see that ϖ is the triangular wave, and hence, it is indeed continuous.

Revised: Both η v ( l )and η b ( l ) are then continuous functions, implying that η ( l ) is alsoncontinuous.

Revised: Derivative: now we can move our attention to the derivative of η v ( l ) and η b ( l ) .

Revised: Again, it is easy to see that:

Revised: [Equation]

Revised: since η b is just an affine function.

Revised: This also implies that η b ( l ) is a monotonically increasing function.

Revised: Again, the reasoning is a bit more complicated for η v .

Revised: Here there are two cases:

Revised: [Equation]

Revised: Since η v is continuous, and the derivative is always ≤ 0 , then η v is monotonically non-increasingfunction.

Revised: Concluding the proof of Item 1.

Revised: Since η b is monotonically increasing, and η v monotonically non-increasing: if there exist ˜ l ∈ (cid:2) 0 , ¯ ℓ (cid:3) such that for any η b (˜ l )

Revised: =

Revised: η b (˜ l ) , then, for any l > ˜ l, η b ( l ) >

Revised: η v ( l ) , and for any l < ˜ l, η b ( l ) <

Revised: η v ( l ) .

Revised: Notice that such ˜ l might not exist since it might be the case that η b ( l ) >

Revised: η v ( l ) for any l ∈ (cid:2) 0 , ¯ ℓ (cid:3) .

Revised: η function decomposition: Notice now that η is defined as the maximum between η v and η b .

Revised: Hence, if ˜ l exists, then:

Revised: [Equation]

Revised: otherwise.

Revised: If however, ˜ l does not exist, it means that η v ( l ) does not intersect η b ( l ) .

Revised: Thanks to continuity, either η b ( l ) >

Revised: η v ( l ) for any l ∈ (cid:2) 0 , ¯ ℓ (cid:3) , or η v ( l ) > η b ( l ) .

Revised: A more accurate analysis, however, shows that the latter is not possible.

Revised: To see this, we simply compute both η v (¯ ℓ ) and η b (¯ ℓ ) :

Revised: [Equation]

Revised: where ( i ) follows from the positivity of ℓ σ , and the positivity of min { δ, 1 − δ } .

Revised: Step ( ii ) follows from the assumption on L ∗ .

Revised: Hence, in the case ˜ l does not exist, since the two functions are continuous η v ( l ) ≤ η b ( l ) , ∀ l ∈ [0 , ¯ ℓ ] .

Revised: We can conclude that η ( l ) can be written as:

Revised: [Equation]

Revised: exists, and l < ˜ l, η

Revised: [Equation]

Revised: l ) otherwise.

Revised: Furthermore, since η v is non-increasing, and η b is decreasing, we have that, in the case where ˜ l existsfor any l , η (˜ l ) ≤ η ( l ) for any other l ∈ (cid:2) 0 , ¯ ℓ (cid:3) .

Revised: Hence, ˜ l ∈ ℓ λ min .

Revised: In the case where ˜ l does not exist, wehave that η ( l ) =

Revised: η b ( l ) , which is a monotonically increasing function.

Revised: Hence, it is minimal in 0 .

Revised: Thisproves Item 2.

Revised: Proof of Item 3 and Item 4: Item 3 follows directly from the fact that η ( l ) =

Revised: η b ( l ) if ˜ l does notexist, or for any l ≥ ˜ l , and ˜ l ∈ ℓ λ min .

Revised: Item 4 requires some additional steps.

Revised: In the case where ˜ l does not exist, we have that the ℓ λ min = { 0 } ,hence the result follows directly.

Revised: Otherwise, let l min =

Revised: min ℓ λ min : with the same argument as for Item 3, we have that η ( l ) =

Revised: η v ( l ) forany l ≤ l min .

Revised: The function η v ( l ) alternates steps with derivative 0 , with steps with derivative − 2 /ℓ .

Revised: We argue that there exists a left neighborhood of l min where the derivative is − 2 /ℓ .

Revised: By contradiction, assume that this neighborhood does not exist: since the function η v ( l ) alternates between intervals of fixed size with derivative − 2 /ℓ and 0 , it would imply that there exists a left neighborhood where the derivative is 0 .

Revised: But this would hence mean that within this neighborhood, the function is constant, and all the values η v ( l ) are equal to η min , which contradicts the hypothesis that l min = min ℓ λ min .

Revised: Hence, there exist such left neighborhood.

Revised: Let ε ℓ σ be the diameter of such interval, then, for any l ∈ [ l min − ε ℓ σ , l min ]

Revised: [Equation]

Revised: Finally, since the function is non-decreasing, as long as η v ( l ) <

Revised: η min + 2 ε ℓσ ℓ , we have that l must belarger than l min − ε ℓ σ .

Original:  The formal versionof Theorem 2 provides rates for the average training errorover noise, in twosettings: when variance domiantes, and when bias dominates.
Revised: Taking c ℓ σ = 2 ε ℓσ ℓ , concludes the proof of Item 4. D. P ROOF OF T HEOREM 2: T RAINING ERROR The formal version Theorem 2 states a bound for the expected training error in two settings: • The optimally regularized model achieves a better risk rate than the interpolating one.

Revised: • The optimally regularized model achieves the same risk rate as the interpolating one.

Original: The informal statement follows simplynoticing that for β < β ∗ , the  variance dominates (hence v 0 > b 0 in the notation ofTheorem 4), andfor β > β ∗ , the bias dominates [-b 0 > v 
-]
Revised: The quantity β ∗ is defined in the main text as the intersection between variance and bias. Sinceregularization benefits the variance and hurts the bias, it is the case that only for values of β > β ∗ , i.e.,where variance dominates [-b 0 > v 
-]bias, that the regularization helps. This how we state the formal version of

Original: Theorem  4 (Formal version of Theorem 2) .
Revised: Theorem 2 as in Theorem 4. Theorem 4 (Formal version of Theorem 2) .

Original: In the same setting as Theorem 1 assume that for two constants b 0 , v 0 ≥ 0 : Bias
Revised: In the same setting as Theorem 1, let us fix ℓ > 0 , β ∈(0 , 1) , ℓ σ ≥ 0 , ¯ ℓ = ℓ − 1 − β

Original: [Equation]
Revised: ( L ∗ − 1) .

Original: Furthermore, let λ opt be such that Risk ( ˆ f λ opt ) is minimal. 
Revised: Finally, let λ opt be such that Risk ( ˆ f λ opt ) is minimal. If l ∈ [0 , ¯ ℓ

Original: Then, if v 0 > b 0 , with probability at least 1 − cd − β min { δ, 1 − δ } , λ opt is Θ( d ℓ λ ) and (cid:12)(cid:12) (cid:12)(cid:12)(cid:12)
Revised: that minimizes η ( l ; ℓ ℓ σ , L ∗ , β ) is such that η l

Revised: ; ℓ, ℓ σ ,

Revised: L

Revised: ∗ , β ) <

Original:  E ϵ (cid:34) n (cid:88) i( ˆ f λ opt ( x i ) − y i )(cid:35) − σ(cid:12)(cid:12) (
Revised: η (0; ℓ, ℓ σ , L ∗ , β ) , then: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) E ϵ (cid:34) n (cid:88) i

Revised: [Equation]

Revised: [Equation]

Revised: (cid:35)

Revised: [Equation]

Original: cid:12)(cid:12)(cid:12) ∈
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) [Equation] ( d [Equation]

Original: O ( d − ℓ λ opt + q − ¯ δ + d − b ) .
Revised: ( ˆ f [Equation] opt )) , w

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: [Equation]

Revised: min

Revised: [Equation]

Original: If instead v 0 < b 0 ,with probability at least 1 − cd − β min { δ, 1 − δ } , λ opt ≤ Λ for a constant Λ , and
Revised: If, for any l ∈ (0 , ¯ ℓ ] , η ( l ; ℓ, ℓ σ , L ∗ , β ) > η (0; ℓ, ℓ σ , L ∗ , β ) , then: E ϵ (cid:34) n (cid:88) i ( x

Original: [Equation]
Revised: [Equation]

Original: For some constant c < 1 .
Revised: y )(cid:35) [Equation] O (cid:16)

Original: Proof.
Revised: Risk [Equation] ( ˆ f [Equation]

Original: Case ( v 0 > b 0 ) .
Revised: opt ) (cid:17) , w. [Equation]

Original:  For  the proof of this part, we just show that, as ℓ λ increases, the rate ofbiasmonotonically increases (slower rate), and the one of the variance decreases (faster rate).
Revised: 1 − δ } . For some constant c < 1 . This theorem covers the case where either regularization is harmful, or it is beneficial. There arestill some edge cases where, for small ℓ λ , the rate of the risk is the same as for the interpolatingmodel, and for large ℓ λ , the performance deteriorate.

Original: Hence, there exists a ℓ λ opt such that the two rates match.
Revised: In this case our theory does not allow discerningweather the expected training matches the noise level or not.

Original: Furthermore, for such ℓ λ opt the maximum betweenthe rate of  the variance and the bias is minimized, hence, such ℓ λ opt is the rate of λ opt .
Revised: Prood of Theorem 4. Throughout the proof we will use with high probability to mean with probabilityat least 1 − cd − β min { ¯ δ, 1 − ¯ δ } .

Original: Looking at the rates from Theorem 1 it is clear that the exponent of bias decreases, with ℓ λ increasing. For the variance we need to be more careful, since also δ depends on ℓ λ : exponent of the variance
Revised: Thanks to Lemma 20, we know that max { λ opt , 1 } ∈ Θ( d l ) , where lminimizes η ( l ; ℓ, ℓ σ , L ∗ , β ) . The condition in Item 1 guarantees that the optimal l is positive, and thecondition in Item 1 guarantees that the optimal l is 0 .

Original: [Equation]
Revised: Harmful interpolation setting.

Original: Finally, taking the derivative withrespect to ℓ λ we get:
Revised: In this setting we have that λ opt ∈ Θ( d l ) , l > 0 . We start by applying Theorem 1 with ℓ λ =

Original: [Equation]
Revised: l in this setting.

Original: Finally, since the exponent of the variance is continuous, and the derivative withrespect to ℓ λ is neverpositive, we have that the rate of the variance is non-increasing with ℓ λ . Hence, the optimal ℓ λ opt iswell defined. Further, since v 0 is strictly larger than b 0 , it is easy to see that such ℓ λ opt is strictly largerthan 0 . Now we move to the bound of the training error.
Revised: This confirms that, with the same m chosen for the proof, Equation (5) holds with high probability, and the ground truth is in the span of the first m eigenfunctions.

Original: We start by using Lemma 
Revised: Hence, we apply Lemma 22:

Original: [Equation]
Revised: [Equation]

Original: E ϵ [Equation] (cid:12)(cid:12)(cid:12)(cid:12)
Revised: Then, by subtracting σ 2 and taking the absolute value we get:

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12) . The second term is proportional to the bias, and hence it vanishes at a rate d − b .
Revised: ) − y i )(cid:35) − σ(cid:12) (cid:12)(cid:12)(cid:12) (cid:12) ≤ σ(cid:12) (cid:12)(cid:12)(cid:12) (cid:12) λ 2 opt n Tr (cid:0)

Original: We now focus on the first term. For this we use Lemma 17.
Revised: H −(cid:1) −(cid:12)

Original: n − ¯
Revised: (cid:12)(cid:12)(cid:12) (cid:12) (cid:124) (cid:123)(cid:122) (cid:125) : = T+(cid:12)(cid:12)(cid:12) (cid:12) λ opt 2 ∥ D ≤ m − 1 a ∥n(cid:12)(cid:12)(cid:12) (cid:12) (cid:124) (cid:123)(cid:122) (cid:125) : = T.

Revised: To bound T 2 we do:

Original: [Equation] 
Revised: [Equation] [Equation]

Original: Where ¯ m , defined within Lemma 17 itself, is such that ¯ m/n ∈ O ( q − ¯ δ ) , and hence A ∈ O ( q − δ ) .
Revised: iii

Original: B, C ∈ O (1 /λ )
Revised: [Equation] (cid:16) Bias 2 ( ˆ f λ

Original: =
Revised: [Equation]

Original: O ( d − ℓ λ opt ) .
Revised: ) (cid:17) ∈ O (cid:16)

Original: Concluding the proof for this case.
Revised: Risk 2 ( ˆ f λ opt ) (cid:17) , where in ( i ) we used that n ∈ Θ( d ℓ ) , λ opt ∈ Θ( d l ) . In step ( ii ) we used Lemma 2 to have the rate of∥ D ≤ m − 1 a ∥ and Lemma 3 for r 22 , r 21 ∈ Θ(1) .

Original: Case ( v 0 < b 0 ) . The idea of this part of the proof is that regularization benefits variance and hurtsbias.
Revised: In ( iii ) we see that this rate matches the rate of thebias found in Theorem 1. To bound T 1 we apply Lemma 21, which yields with high probability:

Original: However, when variance vanishes at a faster rate, it makes no sense to regularize because wecan only reduce an already negligible variance.
Revised: [Equation]

Original: The reason why we cannot prove a tight bound, hencethat λ → 0 , is that it is not clear that even a small λ hurts the bias. To formalize this we write:
Revised: Since c 1 and c 2 are constants, we can bound the absolute value of the inner part as:

Original: [Equation]
Revised: [Equation]

Original: Where ( i ) follows on the hypothesis on the bound of  the bias of ˆ f 0 .
Revised: where in ( i ) we use the rate of | N \ I 2 | computed in Lemma 21 itself. Concluding the proof for thiscase.

Original: Furthermore: [Equation]
Revised: Harmless interpolation setting.

Original: So finally, we want to check for which value ofλ the risk decreases:
Revised: In this setting we know that the minimizer of the rate is 0 , and hence, thanks to Lemma 20, we know that:

Original: [Equation]
Revised: [Equation]

Original: Hence, note that if λ is such that
Revised: with high prob. Like in the other setting, we start by applying Theorem 1 with ℓ λ = l = 0 . Hence, we apply Lemma 22 and look at the upper bound:

Original: [Equation]
Revised: [Equation]

Original: 0 with high probability, then we are guaranteed that regularization hurts for d sufficiently large.
Revised: For T 2 we move analogously to the previous case (Equation (31)) in the specific case where l = 0 .

Original: Hence:
Revised: This yields:

Original: [Equation]
Revised: [Equation]

Original: Finally, since ∥ K >m ∥ ≤ c 2 , µ min ( K >m ) ≥ c 1with high probability, we have that the optimalλ ≤ Λ with high probability. Now we move to the bound of the training error:
Revised: To bound T 1 we apply Lemma 21, which yields with high probability:

Original: [Equation]
Revised: [Equation]

Original: Where, thanks to Lemma 13, µ min ( K 2 ) ≥ c 1 with high probability.Hence, Λ 2 / ( µ min ( K 2 ) + Λ) 2 ≤c 2 < 1 .
Revised: where in ( i ) we used that with high probability λ opt ∈ Θ(1) , hence there exists a C > 0 such thatλ opt ≤ C with high probability.

Original:  Concluding the proof. Here, we present the final two technical lemmas.
Revised: Finally, it is clear that C ( c 1 + C ) 2 < 1 , Concluding the proof. D. T ECHNICAL L EMMAS

Original: Lemma 17 (Trace of the inverse) .
Revised: Lemma 21 (Trace of the inverse) .

Original: In the setting defined in Section 3.1.
Revised: In the setting defined in Section 3.1.

Original: Let ¯ m be the number ofeigenfunctions which are a polynomial of degree at most ¯ L + 1 .

Original: Then, with probability at least1 − cd − ¯ δ ,
Revised: Then, with high probability,

Original: [Equation]
Revised: [Equation]

Original: for two positive constants c 1 , c 2 .
Revised: for two positive constants c 1 , c 2 .

Original: Furthermore, ¯ m ∈ Θ( nq − ¯ δ ) . ¯ m, ¯ L as in Definition [-
-]
Revised: Furthermore, | N \ I 2 | , I 2 , ¯ δ defined as in Definition [-
-]2.

Original: Proof.
Revised: Proof.

Original: We start by computing ¯ m using Lemma 9.
Revised: We start by computing | N \ I 2 | . Computing | N \

Revised: I 2 | corresponds to counting how many indexes do not appear in I 2 .

Revised: Give the definition of the latter, for an index not to be in I 2 , it means that | S k | ≤ ¯ L + 1 .

Revised: Hence, the total number of such index can be at most:

Original: [Equation]
Revised: [Equation]

Original: Where in ( i ) we use the bound the binomial coefficient
Revised: where in ( i ) we use the bound the binomial coefficient

Original: [Equation]
Revised: [Equation]

Original: l − 1 .
Revised: l − 1 .

Original: Finally, since n ∈Θ( d · q ¯ L +¯ δ ) , we have that ¯ m ∈
Revised: Finally, since n ∈Θ( d · q ¯ L +¯ δ ) , we have that | N \ I 2 | ∈

Original: O ( nq − ¯ δ ) .
Revised: O ( nq − ¯ δ ) .

Original: Now we move to the proof of Eq.
Revised: Now we move to the proof of Equation (32).

Original: We divide H = Ψ ≤ m D ≤ m Ψ ⊺ ≤ m + K 1 + K 2{+
+} + λ I as in
Revised: We divide H = K ≤ m + K 1 + K {+
+} + λ I , where K 1 and

Original:  Definition 2.
Revised: K 2 are defined as in Definition 2.

Original: Now, we define H 2 =
Revised: Now, we define H 2 =

Original: K 

Revised: K

Original: + λ I .
Revised: + λ I .

Original: For the lower bound it sufficies to see:
Revised: For the lower bound it suffices to see:

Original: [Equation]
Revised: [Equation]

Original: For some constant c 1 .
Revised: with prob. ≥ 1 − [Equation] , for some constant c 1 .

Original: In ( i ) we used Lemma 13 to bound µ min ( K 2 ) .
Revised: In ( i ) we used Lemma 13 to bound µ min ( K 2 ) .

Original: For the upper bound we need a more refined analysis.
Revised: For the upper bound we need a more refined analysis.

Original: Let us define the following matrixes:
Revised: Let us define the following matrix:

Original: [Equation]
Revised: [Equation]

Original: [Equation] Then it is easy to see thatwe can write ¯ K = ¯
Revised: Notice that K = K − 2 + K 2 , which gives an intuition for the notation.

Original: Ψ ¯ D ¯ Ψ ⊺ . Finally, note that  ¯ Ψ ∈ R n × ¯ m .
Revised: Thanks to Lemma 13, we know that K 2 is non-singular wit probability at least 1 − cd − β (1 − ¯ δ ) .

Original: Now, H = ¯ K + H 2 , and  we  apply Woodbury identity:
Revised: Furthermore, since K − 2 is still positive semi-definite, also K and H and H 2 are non-singular. Thus, we can apply the classical linear algebra formula

Original: [Equation]
Revised: [Equation]

Original: Now, it’s easy to see that A has rank at most ¯ m , this is because ¯ Ψ ∈ R n × ¯ m , hence it has rank ¯ m .
Revised: Since K − 2 is the kernel matrix that stems from a kernel with exactly | N \ I 2 | features, it has rank atmost | N \ I 2 | .

Original: The product of  matrices has rank no bigger than the smallest rank among the multiplicands, hence also A has rank at most ¯ m .
Revised: Since A is defined as the product of some matrices including K − 2 , thus, it also hasrank at most | N \ I 2 | .

Original: Notice furthermore, that since ¯ m ∈  o ( n ) , ¯ m < n for d sufficiently large.
Revised: Notice furthermore, that since | N \ I 2 | ∈ O ( nd − β ¯ δ ) ⊆ o ( n ) , we have that | N \ I 2 | < n for d sufficiently large.

Original: Let now { v 1 , . .
Revised: Let now { v 1 , . .

Original: . , v ¯ m } be an orthonormal basis of col ( A ) , and { v ¯ m +1 , . .
Revised: . , v | N \I 2 | } be an orthonormal basis of col ( A ) , and { v | N \I 2 | +1 , . .

Original: . , v n } an orthonormal basis of col ( A ) ⊥ .
Revised: . , v n } an orthonormal basis of col ( A ) ⊥ .

Original: This makes { v 1 , . . .
Revised: This makes { v 1 , . . .

Original: , v n } a full orthonormal basis of R n .
Revised: , v n } a full orthonormal basis of R n .

Original: Now, since the trace is invariant to similarities, we get that: 
Revised: Now, since the trace is invariant to similarities, we get that: [Equation] i [Equation]

Revised: (cid:124)(cid:123)(cid:122)(cid:125) =

Original: [Equation]
Revised: [Equation]

Revised: (cid:124)(cid:123)(cid:122)(cid:125)

Revised: [Equation]

Revised: (cid:124)

Revised: [Equation]

Revised: 

Revised: [Equation]

Revised: [Equation]

Revised: c

Revised: [Equation]

Original: Where in ( i ) we use the fact that, as long as i > ¯ m , we have thatAv i = v ⊺ i A = 0 .
Revised: + λ [Equation] with prob. ≥ 1 − cd − β (1 − ¯ δ ) , where in ( i ) we use the fact that, as long as i > | N \ I 2 | , we have that

Original: Since v i isorthogonal to the column space.
Revised: Av i = v ⊺ i A = 0 ,since v i isorthogonal to the column space.

Original: In ( ii ) we used  that  ∥ K 2 ∥ ≤ c 2 with high probability thanks to Lemma 13.
Revised: In ( ii ) we used the fact that the vectors v i in the orthonormal basishave norm 1 , and in ( iii ) we used Lemma 13 to bound ∥ K 2 ∥ .

Original: Lemma 18 (Fixed design training error) .
Revised: Lemma 22 (Fixed design training error) .

Original: In the same setting as Theorem 3, let m be such that
Revised: In the same setting as Theorem 3, let m be such that

Original: Assumption 1 and 2 hold.
Revised: Equation (5) holds and the ground truth is in the span of the first m eigenfunctions.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In kernel ridge regression, our estimator is:
Revised: In kernel ridge regression, our estimator is:

Original: [Equation]
Revised: [Equation]

Original: where y :
Revised: where y :

Original: =
Revised: =

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y n ] ⊺ and k ( x ∗ ) :
Revised: , y n ] ⊺ and k ( x ∗ ) :

Original: =
Revised: =

Original: [ K ( x 1 , x ∗ ) , . .
Revised: [ K ( x 1 , x ∗ ) , . .

Original: , K ( x n , x ∗ )] ⊺ , H =
Revised: , K ( x n , x ∗ )] ⊺ , H =

Original: K
Revised: K

Original: + λ I n .
Revised: + λ I n .

Original: Hence, our estimator, evaluated at x 1 , . . .
Revised: Hence, our estimator, evaluated at x 1 , . . .

Original: , x n is KH − 1 y .
Revised: , x n is KH − 1 y .

Original: Hence, the training error is:
Revised: Hence, the training error is:

Original: [Equation]
Revised: [Equation]

Original: Where ( i ) follows from the simple calculation: 
Revised: Where ( i ) follows from the simple calculation: I [Equation] [Equation]

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: + λ I n − λ I n ) =

Revised: λ ( K + λ I ) − 1 .

Original: Now, y =
Revised: Now, y =

Original: ϵ + Ψ ≤ m a .
Revised: ϵ + Ψ ≤ m a .

Original: And we are interested in the expectation with regard to noise, hence:
Revised: And we are interested in the expectation with regard to noise, hence:

Original: [Equation]
Revised: [Equation]

Original: While T 1 is already the desired term
Revised: While T 1 is already the desired term

Original: [Equation]
Revised: [Equation]

Original: for T 2 we do the following:
Revised: for T 2 we do the following:

Original: [Equation]
Revised: [Equation]

Original: n [Equation] Where in ( i ) we use Lemma 7.
Revised: where in ( i ) we use Lemma 6.

Original: In ( ii ) we use Assumption 1 as well as the fact that we recognize term
Revised: In ( ii ) we use Equation (5) as well as the fact that we recognize term

Original: B 1 from the proof of Theorem 3.
Revised: B 1 from the proof of Theorem 3.

Original: In ( iii ) we then use the same bound on B 1 achieved in Theorem 3.
Revised: In ( iii ) we then use the same bound on B 1 achieved in Theorem 3.

Original: This concludes the proof.
Revised: This concludes the proof.

Original: E E XPERIMENTAL DETAILS
Revised: E E XPERIMENTAL DETAILS

Original: This section describes our experimental setup and provides additional details.
Revised: This section describes our experimental setup and provides additional details.

Original: Our submission alreadycontains the code required to replicate all experiments and plots.
Revised: Our submission alreadycontains the code required to replicate all experiments and plots.

Original: We will release a documented andcleaned version of the code with the camera-ready version of the paper.
Revised: We will release a documented andcleaned version of the code with the camera-ready version of the paper.

Original: E. S ETUP FOR FILTER SIZE EXPERIMENTS
Revised: E. S ETUP FOR FILTER SIZE EXPERIMENTS

Original: The following describes the main filter size experiments presented in Section 4.1.
Revised: The following describes the main filter size experiments presented in Section 4.1.

Original: Network architecture We use a special CNN architecture that amplifies the role of filter size asan inductive bias.
Revised: Network architecture We use a special CNN architecture that amplifies the role of filter size asan inductive bias.

Original: Each model of the main filter size experiments in Figure 2 has the followingarchitecture:
Revised: Each model of the main filter size experiments in Figure 2 has the followingarchitecture:

Original: convolutional layer with 128 filters of varying size and no padding2.
Revised: Convolutional layer with 128 filters of varying size and no padding2.

Original: global max pooling over the spatial feature dimensions3.
Revised: Global max pooling over the spatial feature dimensions3.

Original: ReLU activation4.
Revised: ReLU activation4.

Original: linear layer with 256 output features5.
Revised: Linear layer with 256 output features5.

Original: ReLU activation6.
Revised: ReLU activation

Original: linear layer with 1 output feature
Revised: Linear layer with 1 output feature

Original: All convolutional and linear layers use a bias term.
Revised: All convolutional and linear layers use a bias term.

Original: Since we employ a single convolutional layerbefore global max pooling, the convolutional filter size directly determines the maximum size of aninput patch that can influence the CNN’s output.
Revised: Since we employ a single convolutional layerbefore global max pooling, the convolutional filter size directly determines the maximum size of aninput patch that can influence the CNN’s output.

Original: Note that this architecture reduces to an MLP if thefilter size equals the input image size.
Revised: Note that this architecture reduces to an MLP if thefilter size equals the input image size.

Original: Optimization procedure We use the same training procedure for all settings in Figure 2.
Revised: Optimization procedure We use the same training procedure for all settings in Figure 2.

Original: Optimization minimizes the logistic loss for 300 epochs of mini-batch SGD with momentum 0 .
Revised: Optimization minimizes the logistic loss for 300 epochs of mini-batch SGD with momentum 0 .

Original: 9 and batchsize 100 .
Revised: 9 and batchsize 100 .

Original: We linearly increase the learning rate from 10 − 6 to a peak value of 0 . 2 during the first 50epochs, and then reduce the learning rate according to an inverse square-root decay every 20 epochs.
Revised: We linearly increase the learning rate from 10 − 6 to a peak value of 0 . 2 during the first 50epochs, and then reduce the learning rate according to an inverse square-root decay every 20 epochs.

Original: For peak learning rate γ 0 , a decay rate L , the inverse square-root decay schedule at epoch t ≥ 0 is
Revised: For peak learning rate γ 0 , a decay rate L , the inverse square-root decay schedule at epoch t ≥ 0 is

Original: [Equation]
Revised: [Equation]

Original: /L ⌋ .

Original: [Figure]

Original: Learning rate warm-up helps to capture the early-stopped test error more precisely.
Revised: Learning rate warm-up helps to capture the early-stopped test error more precisely.

Original: Wheneverpossible, we use deterministic training algorithms, so that our results are as reproducible as possible.
Revised: Wheneverpossible, we use deterministic training algorithms, so that our results are as reproducible as possible.

Original: We selected all hyperparameters to minimize the training loss of the strongest inductive bias (filtersize 5 ) on noisy training data, with the constraint that all other settings still converge and interpolate.
Revised: We selected all hyperparameters to minimize the training loss of the strongest inductive bias (filtersize 5 ) on noisy training data, with the constraint that all other settings still converge and interpolate.

Original: Note that we do not use data augmentation, dropout, or weight decay.
Revised: Note that we do not use data augmentation, dropout, or weight decay.

Original: Evaluation We observed that all models achieved their minimum test error either at the beginningor very end of training.
Revised: Evaluation We observed that all models achieved their minimum test error either at the beginningor very end of training.

Original: Hence, our experiments evaluate the test error every 2 epochs during thefirst 150 epochs, and every 10 epochs afterwards to save computation time.
Revised: Hence, our experiments evaluate the test error every 2 epochs during thefirst 150 epochs, and every 10 epochs afterwards to save computation time.

Original: We use an oracle, thatis, the true test error, to determine the optimal early stopping epoch in retrospective.
Revised: We use an oracle, thatis, the true test error, to determine the optimal early stopping epoch in retrospective.

Original: The optimalearly stopping training error is always over the entire training set (including potential noise) for afixed model, not an average over mini-batches.
Revised: The optimalearly stopping training error is always over the entire training set (including potential noise) for afixed model, not an average over mini-batches.

Original: To mitigate randomness in both the training data andoptimization procedure, we average over multiple dataset and training seeds.
Revised: To mitigate randomness in both the training data andoptimization procedure, we average over multiple dataset and training seeds.

Original: More precisely, wesample 5 different pairs of training and test datasets.
Revised: More precisely, wesample 5 different pairs of training and test datasets.

Original: For each dataset, we fit 15 randomly initializedmodels per filter size on the same dataset, and calculate average metrics.
Revised: For each dataset, we fit 15 randomly initializedmodels per filter size on the same dataset, and calculate average metrics.

Original: The plots then display themean and standard error over the 5 datasets.
Revised: The plots then display themean and standard error over the 5 datasets.

Original: Dataset All filter size experiments use synthetic images.
Revised: Dataset All filter size experiments use synthetic images.

Original: For a fixed seed, the experiments generate200 training and 100 k test images, both having an equal amount of positive and negative classes.
Revised: For a fixed seed, the experiments generate200 training and 100 k test images, both having an equal amount of positive and negative classes.

Original: Given a class, the sampling procedure iteratively scatters 10 shapes on an initially black 32 × 32image.
Revised: Given a class, the sampling procedure iteratively scatters 10 shapes on an initially black 32 × 32image.

Original: A single shape is either a circle (negative class) or a cross (positive class), has a uniformlyrandom size in [3 , 5] , and a uniformly random center such that all shapes end up completely insidethe target image.
Revised: A single shape is either a circle (negative class) or a cross (positive class), has a uniformlyrandom size in [3 , 5] , and a uniformly random center such that all shapes end up completely insidethe target image.

Original: We use a conceptual line width of 0 .
Revised: We use a conceptual line width of 0 .

Original: 5 pixels, but discretize the shapes into a grid.
Revised: 5 pixels, but discretize the shapes into a grid.

Original: See Figure 4 for examples.
Revised: See Figure 4 for examples.

Original: A single dataset seed fully determines the training data, test data, and allscattered shapes.
Revised: A single dataset seed fully determines the training data, test data, and allscattered shapes.

Original: Noise model
Revised: Noise model

Original: In the noisy case, we select 20% of all training samples uniformly at random withoutreplacement, and flip their label.
Revised: In the noisy case, we select 20% of all training samples uniformly at random withoutreplacement, and flip their label.

Original: The noise is deterministic per dataset seed and does not changebetween different optimization runs.
Revised: The noise is deterministic per dataset seed and does not changebetween different optimization runs.

Original: Note that we never apply noise to the test data. 
Revised: Note that we never apply noise to the test data. [Figure]

Original: E. S ETUP FOR ROTATIONAL INVARIANCE EXPERIMENTS
Revised: E. S ETUP FOR ROTATIONAL INVARIANCE EXPERIMENTS

Original: The following describes the rotational invariance experiments presented in Section 4.2.
Revised: The following describes the rotational invariance experiments presented in Section 4.2.

Original: Dataset We use the EuroSAT (Helber et al., 2018) training split and subsample it into 7680 rawtraining and 10 k raw test samples in a stratified way.
Revised: Dataset We use the EuroSAT (Helber et al., 2018) training split and subsample it into 7680 rawtraining and 10 k raw test samples in a stratified way.

Original: For a fixed number of rotations k , we generate atraining dataset as follows:
Revised: For a fixed number of rotations k , we generate atraining dataset as follows:

Original: In the noisy case, select a random 20% subset of all training samples without replacement;for each, change the label to one of the other 9 classes uniformly at random.2.
Revised: In the noisy case, select a random 20% subset of all training samples without replacement; for each, change the label to one of the other 9 classes uniformly at random.

Original: For the i -th training sample:(a)
Revised: For the i -th training sample ( i ∈ { 1 , . . . , 7680 } ):

Original:  Determine a random offset angle α i .
Revised: (a) Determine a random offset angle α i .

Original: (b) Rotate the original image by each angle in {  i +  · (360 ◦ /k )
Revised: (b) Rotate the original image by each angle in { α i + j · (360 ◦ /k )

Original: | j = 0 , . .
Revised: | j = 0 , . .

Original: , k − 1 } .(c)
Revised: , k − 1 } .(c)

Original: Crop each of the k rotated 64 × 64 images to 44 × 44 so that no network sees blackborders from image interpolation.3.
Revised: Crop each of the k rotated 64 × 64 images to 44 × 44 so that no network sees black borders from image interpolation.

Original: Concatenate all k × 7680 samples into a single training dataset.
Revised: Concatenate all k × 7680 samples into a single training dataset.

Original: Shuffle this final dataset (at the beginning of training and every epoch).
Revised: Shuffle this final dataset (at the beginning of training and every epoch).

Original: To generate the actual test dataset, wesimply apply a random rotation to each raw test sampleindependently, and crop the rotated images to the same size as the training samples.
Revised: To generate the actual test dataset, we apply a random rotation to each raw test sample independently,and crop the rotated images to the same size as the training samples.

Original: This procedurerotates every image exactly once, and uses random angle offsets to avoid distribution shift effectsfrom image interpolation.
Revised: This procedure rotates everyimage exactly once, and uses random angle offsets to avoid distribution shift effects from imageinterpolation.

Original: Note that all random rotations are independent of the label noise and thenumber of training rotations.
Revised: Note that all random rotations are independent of the label noise and the number oftraining rotations.

Original: Hence, all experiments share the same test dataset.
Revised: Hence, all experiments share the same test dataset.

Original: Furthermore, sincewe apply label noise before rotating images, all rotations of an image consistently share the samelabel.
Revised: Furthermore, since we applylabel noise before rotating images, all rotations of an image consistently share the same label.

Original: Network architecture All experiments use a Wide Residual Network (Zagoruyko & Komodakis,2016) with 16 layers, widen factor 6 , and default PyTorch weight initialization.
Revised: Network architecture All experiments use a Wide Residual Network (Zagoruyko & Komodakis,2016) with 16 layers, widen factor 6 , and default PyTorch weight initialization.

Original: We chose the widthand depth such that all networks are sufficiently overparameterized while still being manageable interms of computational cost.
Revised: We chose the widthand depth such that all networks are sufficiently overparameterized while still being manageable interms of computational cost.

Original: Optimization procedure We use the same training procedure for all settings in Figure 3.
Revised: Optimization procedure We use the same training procedure for all settings in Figure 3.

Original: Optimization minimizes the softmax cross-entropy loss using mini-batch SGD with momentum 0 .
Revised: Optimization minimizes the softmax cross-entropy loss using mini-batch SGD with momentum 0 .

Original: 9 and batchsize 128 .
Revised: 9 and batchsize 128 .

Original: Since the training set size grows in the number of rotations, all experiments fix the numberof gradient updates to 144 k.
Revised: Since the training set size grows in the number of rotations, all experiments fix the numberof gradient updates to 144 k.

Original: This corresponds to 150 epochs over a dataset with 16 rotations.
Revised: This corresponds to 200 epochs over a dataset with 12 rotations.

Original: Similarto the filter size experiments, we linearly increase the learning rate from zero to a peak value of 0 .
Revised: Similarto the filter size experiments, we linearly increase the learning rate from zero to a peak value of 0 .

Original: 15during the first 4800 steps, and then reduce the learning rate according to an inverse square-rootdecay (Equation (20))
Revised: 15during the first 4800 steps, and then reduce the learning rate according to an inverse square-rootdecay (Equation (33))

Original: every 960 steps.
Revised: every 960 steps.

Original: Whenever possible, we use deterministic training algorithms,so that our results are as reproducible as possible.
Revised: Whenever possible, we use deterministic training algorithms,so that our results are as reproducible as possible.

Original: We selected all hyperparameters to minimize thetraining loss of the strongest inductive bias ( 16 rotations) on noisy training data, with the constraintthat all other settings still converge and interpolate.
Revised: We selected all hyperparameters to minimize thetraining loss of the strongest inductive bias ( 12 rotations) on noisy training data, with the constraintthat all other settings still converge and interpolate.

Original: As for all experiments in this paper, we do notuse additional data augmentation, dropout, or weight decay.
Revised: As for all experiments in this paper, we do notuse additional data augmentation, dropout, or weight decay.

Original: Evaluation Similar to the filter size experiments, we evaluate the test error more frequently duringearly training iterations: every 480 steps for the first 9600 steps, every 1920 steps afterwards.
Revised: Evaluation Similar to the filter size experiments, we evaluate the test error more frequently duringearly training iterations: every 480 steps for the first 9600 steps, every 1920 steps afterwards.

Original: Theexperiments again use the actual test error to determine the best step for early-stopping, and calculatethe corresponding training error over the entire training dataset, including all rotations and potentialnoise.
Revised: Theexperiments again use the actual test error to determine the best step for early-stopping, and calculatethe corresponding training error over the entire training dataset, including all rotations and potentialnoise.

Original: Due to the larger training set size and increased computational costs, we only sample a singletraining and test dataset, and report the mean and standard error of all metrics over five training seeds.
Revised: Due to the larger training set size and increased computational costs, we only sample a singletraining and test dataset, and report the mean and standard error of all metrics over five training seeds.

Original: E. D IFFERENCE TO DOUBLE DESCENT
Revised: E. D IFFERENCE TO DOUBLE DESCENT

Original: As mentioned in Section 4.1, our empirical observations resemble the double descent phenomenon.
Revised: As mentioned in Section 4.1, our empirical observations resemble the double descent phenomenon.

Original: This subsection expands on the discussion and provides additional details about how this paper’sphenomenon differs from double descent.
Revised: This subsection expands on the discussion and provides additional details about how this paper’sphenomenon differs from double descent.

Original: While all models in all experiments interpolate the training data, we observe that both noisy labelsand stronger inductive biases increase the final training loss of an interpolating model: Smaller filtersize results in a decreasing number of model parameters.
Revised: While all models in all experiments interpolate the training data, we observe that both noisy labelsand stronger inductive biases increase the final training loss of an interpolating model: Smaller filtersize results in a decreasing number of model parameters.

Original: Enforcing invariance to more rotationsrequires a model to interpolate more (correlated) training samples.
Revised: Enforcing invariance to more rotationsrequires a model to interpolate more (correlated) training samples.

Original: Hence, in both cases, increasinginductive bias strength decreases a model’s overparameterization in relation to the number of trainingsamples — shifting the setting closer to the interpolation threshold.
Revised: Thus, in both cases, increasinginductive bias strength decreases a model’s overparameterization in relation to the number of trainingsamples — shifting the setting closer to the interpolation threshold.

Original: In the following, we argue that our method of  hyperparameter tuning ensures that all models in allexperiments lie far in  the interpolating regime.
Revised: We argue that our choice of architecture and hyperparameter tuning ensures that no model in anyexperiment is close to the corresponding interpolation threshold.

Original: Hence, our empirical observations are not a specialcase of double descent, but an unrelated phenomenon.
Revised: If that is the case, then doubledescent predicts that increasing the number of model parameters has a negligible effect on whetherregularization benefits generalization, and does therefore not explain our observations.

Original: We further provide  experimental evidence byre-running a subset of thefilter size experimentsusing significantly larger models, and observing thatthe phase transition between harmless and harmful interpolation persists. Hyperparameter tuning We mitigate differences in model complexity for different strengths ofinductive biases by tuning all hyperparameters on worst-case settings, that is, maximum inductive [Figure]
Revised: In the following, we first describe how our hyperparameter and model selection procedure ensuresthat all models in all experiments are sufficiently overparameterized, so that double descent predictsnegligible effects from increasing the number of parameters. Then, we provide additional experimental evidence that supports our argument: We repeat a subset of the experiments in Section 4 whileupscaling the number of parameters in all models.

Original: width: [Figure]
Revised: For a fixed model scale and varying inductive bias,we observe that all phenomena in Section 4 persist.

Original: [Figure] t r a i n i n g l o ss [Figure] 0% noise 20% noise 1 4# rotations
Revised: For a fixed inductive bias strength, we furthersee that the test error of interpolating models saturates at a value that matches our hypothesis.

Original: [Figure] t r a i n i n g l
Revised: Inparticular, for strong inductive biases, the gap in test error between interpolating models and theiroptimally early-stopped version — harmful interpolation — persists.

Original: [Figure] ss [Figure] bias with noisy training samples.
Revised: Hyperparameter tuning We mitigate differences in model complexity for different strengths ofinductive biases by tuning all hyperparameters on worst-case settings, that is, maximum inductive biaswith noisy training samples.

Original: To avoid optimizing on test data, we tune on dataset seeds  that differfrom the ones used in actual experiments.
Revised: To avoid optimizing on test data, we tune on dataset seeds and networkinitializations that differ from the ones used in actual experiments.

Original: Figure 5 displays the final training loss for all empiricalsettings in this paper.
Revised: Figure 5 displays the final trainingloss for all empirical settings in this paper.

Original: While  stronger inductive biases yield increasing training losses, all valuesare close to zero, and the numerical difference is small.
Revised: While models with stronger inductive biases exhibit largertraining losses, all values are close to zero, and the numerical difference is small.

Original: Finally, we want to stress again that thisdiscussion is only about the training loss ; all models in all experiments have zero training error , i.e.,perfectly fit the corresponding training data.
Revised: Finally, we want tostress again that this discussion is only about the training loss ; all models in all experiments havezero training error and perfectly fit the corresponding training data.

Original: Increasing model complexity For additional evidence, wefurther repeat the main filter sizeexperiments from Figure 2 in Section 4.1 using the same setup as before (see Appendix E.1), butincrease the convolutional layer width to 256  and 512 , respectively.
Revised: Increasing model complexity for varying filter size As additional evidence, we repeat the mainfilter size experiments from Figure 2 in Section 4.1 using the same setup as before (see Appendix E.1),but increase the convolutional layer width to 256 , 512 , 1024 , and 2048 . For computational reasons,we evaluate a reduced number of filter sizes for widths 256 and 512 , and only the smallest filter size5 for widths 1024 and 2048 .

Original: We also use a decreased peaklearning rate 0 . 13 , since we found the original learning rate 0 .
Revised: Since we found the original learning rate 0 .

Original: 2 to be too unstable for the largermodel sizes.
Revised: 2 to be too unstable for thelarger model sizes, we use a decreased peak learning rate 0 .

Original: For computational reasons, we only evaluate a reduced number of filter sizes.
Revised: 13 for widths 256 and 512 , and 0 .

Original:  Figures 6aand 6b show the test errors for 20% and 0% training noise, respectively.
Revised: 1 forwidths 1024 and 2048 . Figures 6a and 6b show the test errors for 20% and 0% training noise, respectively.

Original: With noisy training data(Figure 6a), larger interpolating models yield a slightly smaller test error, but the overall trendsremain: overfitting increases with inductive bias strength, and the phase transition between harmlessand harmful interpolation persists.{+interpolationpersists. In particular, Figure 6a shows strong evidence that the number of model parameters doesnot influence our phenomenon: for example, models with filter size 5 (strong inductive bias) andwidth 
 (red line) have more parameters than models with filter size 27 (weak inductive bias) andwidth 
 (blue line).+}
Revised: With noisytraining data (Figure 6a), larger interpolating models yield a slightly smaller test error, but the overalltrends remain: the gap in test error between converged and optimally early-stopped models increaseswith inductive bias strength, and the phase transition between harmless and harmful {+interpolationpersists. In particular, Figure 6a shows strong evidence that the number of model parameters doesnot influence our phenomenon: for example, models with filter size 5 (strong inductive bias) andwidth 
 (red line) have more parameters than models with filter size 27 (weak inductive bias) andwidth 
 (blue line).+}

Original: In the noiseless case (Figure 6b), increasing model complexitydoes neither harm nor improve generalization, and all models achieve their optimal performance afterinterpolating the entire training dataset.
Revised: Nevertheless, models with filter size 5 benefit significantly from early stopping, [Figure] t r a i n i n g l o [Figure]

Original: Lastly, Figure 6c reveals that the training error of optimallyearly-stopped models stays the same despite overparameterization.
Revised: interpolating, width [Figure] t e s t e rr o r

Original: [Figure]
Revised: [Figure]

Revised: while interpolation for models with filter size 27 is harmless.

Revised: In the noiseless case (Figure 6b),increasing model complexity does neither harm nor improve generalization, and all models achievetheir optimal performance after interpolating the entire training dataset.

Revised: Similarly, Figure 6c revealsthat the fraction of training noise that optimally early-stopped models fit stays the same for largermodels.

Original: t e s t
Revised: Finally, for a fixed inductive bias strength, the test errors saturate as model size increases,making a different trend for models with more than 2048 filters unlikely. To increase legibility, wepresent the numerical results for the largest two filter sizes in Table 1. Increasing model capacity for varying rotational invariance For completeness, we also repeatthe rotation invariance experiments from Figure 3 in Section 4.2 with twice as wide Wide Residual Networks on a reduced number of rotations.

Revised: More precisely, we increase the network widen-factorfrom 6 to 12 , and otherwise use the same setting as the main experiments (see Appendix E.2).

Revised: Notethat this corresponds to a parameter increase from around 6 million to around 24 million parameters.

Revised: [Table]

Original:  e rr o r
Revised: t e s t e rr o r

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Table]

Revised: t e s t

Revised: e rr o r

Revised: [Table]

Revised: [Table]

Revised: s u b s e t t

Revised: r a i n e rr o r

Revised: ( o p t i m a l e a r l y s t

Revised: o

Revised: pp i n g ) (c) Opt

Revised: [Figure]

Revised: [Figure]

Revised: The results in Figure 7 provide additional evidence that our phenomenon is distinct from doubledescent: both the test error (Figures 7a and 7b) and fraction of fitted noise under optimal early stopping(Figure 7c) exhibit the same trend, despite the significant difference in number of parameters.