Original: S TRONG INDUCTIVE BIASES PROVABLY PREVENT HARMLESS INTERPOLATION
Revised: S TRONG INDUCTIVE BIASES PROVABLY PREVENT HARMLESS INTERPOLATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Classical wisdom suggests that estimators should avoid fitting noise to achievegood generalization.
Revised: Classical wisdom suggests that estimators should avoid fitting noise to achieve good generalization.

Original: In contrast, modern overparameterized models can yieldsmall test error despite interpolating noise — a phenomenon often called “benignoverfitting” or “harmless interpolation”.
Revised: In contrast, modern overparameterized models can yield small test error despite interpolating noise — a phenomenon often called “benign overfitting” or “harmless interpolation”.

Original: This paper argues that the degree to whichinterpolation is harmless hinges upon the strength of an estimator’s inductive bias,i.e., how heavily the estimator favors solutions with a certain structure: whilestrong inductive biases prevent harmless interpolation, weak inductive biases caneven require fitting noise to generalize well.
Revised: This paper argues that the degree to which interpolation is harmless hinges upon the strength of an estimator’s inductive bias, i e., how heavily the estimator favors solutions with a certain structure: while strong inductive biases prevent harmless interpolation, weak inductive biases can even require fitting noise to generalize well.

Original: Our main theoretical result establishestight non-asymptotic bounds for high-dimensional kernel regression that reflect thisphenomenon for convolutional kernels, where the filter size regulates the strengthof the inductive bias.
Revised: Our main theoretical result establishes tight non-asymptotic bounds for high-dimensional kernel regression that reflect this phenomenon for convolutional kernels, where the filter size regulates the strength of the inductive bias.

Original: We further provide empirical evidence of the same behaviorfor deep neural networks with varying filter sizes and rotational invariance.
Revised: We further provide empirical evidence of the same behavior for deep neural networks with varying filter sizes and rotational invariance

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: According to classical wisdom (see, e.g., Hastie et al.
Revised: According to classical wisdom (see, e.g., Hastie et al.

Original: , an estimator that fits noise suffersfrom “overfitting”, and cannot generalize well.
Revised: , an estimator that fits noise suffers from “overfitting” and cannot generalize well.

Original: A typical solution is to prevent interpolation, that is,stopping the estimator from achieving zero training error and thereby fitting less noise.
Revised: A typical solution is to prevent interpolation, that is, stopping the estimator from achieving zero training error and thereby fitting less noise.

Original: For example,one can use ridge regularization or early stopping for iterative algorithms to obtain a model thathas training error close to the noise level.
Revised: For example, one can use ridge regularization or early stopping for iterative algorithms to obtain a model that has training error close to the noise level.

Original: However, large overparameterized models such as neuralnetworks seem to behave differently: even on noisy data, they may achieve optimal test performanceat convergence after interpolating the training data (Nakkiran et al., 2021; Belkin et al., 2019a) — aphenomenon referred to as harmless interpolation (Muthukumar et al., 2020) or benign overfitting(Bartlett et al., 2020) and often discussed in the context of double descent (Belkin et al., 2019a).
Revised: However, large overparameterized models such as neural networks seem to behave differently: even on noisy data, they may achieve optimal test performance at convergence after interpolating the training data (Nakkiran et al., 2021; Belkin et al., 2019a) — a phenomenon referred to as harmless interpolation (Muthukumar et al., 2020) or benign overfitting (Bartlett et al., 2020) and often discussed in the context of double descent (Belkin et al., 2019a).

Original: To date, we lack a general understanding of when interpolation is harmless for overparameterizedmodels.
Revised: To date, we lack a general understanding of when interpolation is harmless for overparameterized models.

Original: In this paper, we argue that the strength of an inductive bias critically influences whetheran estimator exhibits harmless interpolation.
Revised: In this paper, we argue that the strength of an inductive bias critically influences whether an estimator exhibits harmless interpolation.

Original: An estimator with a strong inductive bias heavilyfavors “simple” solutions that structurally align with the ground truth (such as sparsity or rotationalinvariance).
Revised: An estimator with a strong inductive bias heavily favors “simple” solutions that structurally align with the ground truth (such as sparsity or rotational invariance).

Original: Based on well-established high-probability recovery results of sparse linear regression(Tibshirani, 1996; Candes, 2008; Donoho & Elad, 2006), we expect that models with strongerinductive biases generalize better than ones with  weaker inductive biases, particularly from noiselessdata.
Revised: Based on well-established high-probability recovery results of sparse linear regression (Tibshirani, 1996; Candes, 2008; Donoho & Elad, 2006), we expect that models with a stronger inductive bias generalize better than ones with a weaker inductive bias, particularly from noiseless data.

Original: In contrast, the effects of inductive biases are much less studied for interpolators of noisy data.
Revised: In contrast, the effects of inductive bias are much less studied for interpolators of noisy data.

Original: Recently, Donhauser et al.
Revised: Recently, Donhauser et al.

Original: (2022) provided a first rigorous analysis of the effects of inductive biasstrength on the generalization performance of linear max- ℓ p -margin/min- ℓ p -norm interpolators.
Revised: (2022) provided a first rigorous analysis of the effects of inductive bias strength on the generalization performance of linear max- ℓ p -margin/min- ℓ p -norm interpolators.

Original: Inparticular, the authors prove that a stronger inductive bias (small p → 1 ) not solely enhances amodel’s ability to generalize on noiseless data, but also increases a model’s sensitivity to noise —eventually harming generalization when interpolating noisy data.
Revised: In particular, the authors prove that a stronger inductive bias (small p → 1 ) not solely enhances a model’s ability to generalize on noiseless data, but also increases a model’s sensitivity to noise — eventually harming generalization when interpolating noisy data.

Original: As a consequence, their resultsuggests that interpolation might not be harmless when the inductive bias is too strong.
Revised: As a consequence, their result suggests that interpolation might not be harmless when the inductive bias is too strong.

Original: In this paper, we confirm the hypothesis and show that strong inductive biases indeed prevent harmlessinterpolation, while also moving away from sparse linear models.
Revised: In this paper, we confirm the hypothesis and show that strong inductive biases indeed prevent harmless interpolation, while also moving away from sparse linear models.

Original: As one example, we consider datawhere the true labels nonlinearly only depend on input features in a local neighborhood, and vary thestrength of the inductive biases via the filter size of convolutional kernels or shallow convolutionalneural networks — small filter sizes encourage functions that depend nonlinearly only on localneighborhoods of the input features.
Revised: As one example, we consider data where the true labels nonlinearly only depend on input features in a local neighborhood, and vary the strength of the inductive bias via the filter size of convolutional kernels or shallow convolutional neural networks — small filter sizes encourage functions that depend nonlinearly only on local neighborhoods of the input features.

Original: As a second example, we also investigate classification for rotationally invariant data, where we encourage different degrees of rotational invariance for neural
Revised: As a second example we also investigate classification for rotationally invariant data , where we encourage different degrees of rotational invariance for neural

Original: networks.
Revised: networks

Original: In particular, • we prove a phase transition between harmless and harmful interpolation that occurs byvarying the strength of the inductive bias via the filter size of convolutional kernels for kernel 
Revised: In particular, • we prove a phase transition between harmless and harmful interpolation that occurs by varying the strength of the inductive bias via the filter size of convolutional kernels for kernel regression in the high-dimensional setting (Theorem 1).

Original: regression in the high-dimensional setting (Theorem 1).• we further show that, for  weak inductive biases, not only is interpolation harmless butpartially fitting the observation noise is in fact necessary (Theorem 2).• we show the same phase transition experimentally for neural networks with two commoninductive biases: varying convolution filter size, and rotational invariance enforced via dataaugmentation (Section 4). 
Revised: • we further show that, for a weak inductive bias, not only is interpolation harmless but partially fitting the observation noise is in fact necessary (Theorem 2). • we show the same phase transition experimentally for neural networks with two common inductive biases: varying convolution filter size, and rotational invariance enforced via data augmentation (Section 4). From a practical perspective, empirical evidence suggests that large neural networks not necessarily benefit from early stopping. Our results match those observations for typical networks with a weak inductive bias; however, we caution that strongly structured models must avoid interpolation, even if they are highly overparameterized.

Original:  R ELATED W ORK
Revised:  R ELATED WORK

Original: We now discuss three groups of related work and explain how their theoretical results cannot reflectthe phase transition between harmless and harmful interpolation for high-dimensional kernel learning.
Revised: We now discuss three groups of related work and explain how their theoretical results cannot reflect the phase transition between harmless and harmful interpolation for high-dimensional kernel learning.

Original: Low-dimensional kernel learning: Many recent works (Bietti et al., 2021;
Revised: Low-dimensional kernel learning: Many recent works (Bietti et al., 2021;

Original: Favero et al., 2021; Bietti,2022; Cagnetta et al., 2022) prove statistical rates for kernel regression with convolutional kernels inlow-dimensional settings, but crucially rely on ridge regularization.
Revised: Favero et al., 2021; Bietti, 2022; Cagnetta et al., 2022) prove statistical rates for kernel regression with convolutional kernels in low-dimensional settings, but crucially rely on ridge regularization.

Original: In general, one cannot expectharmless interpolation for such kernels in the low-dimensional regime (Rakhlin & Zhai, 2019;
Revised: In general, one cannot expect harmless interpolation for such kernels in the low-dimensional regime (Rakhlin & Zhai, 2019;

Original: Mallinar et al., 2022; Buchholz, 2022); positive results exist only for very specific adaptive spikedkernels (Belkin et al., 2019b).
Revised: Mallinar et al., 2022; Buchholz, 2022); positive results exist only for very specific adaptive spiked kernels (Belkin et al., 2019b).

Original: Furthermore, techniques developed for low-dimensional settings (see,e.g., Schölkopf et al.
Revised: Furthermore, techniques developed for low-dimensional settings (see, e.g., Schölkopf et al.

Original: (2018)) usually suffer from a curse of dimensionality, that is, the bounds becomevacuous in high-dimensional settings where the input dimension grows with the number of samples.
Revised: (2018)) usually suffer from a curse of dimensionality, that is, the bounds become vacuous in high-dimensional settings where the input dimension grows with the number of samples.

Original: High-dimensional kernel learning: One line of research (Liang et al., 2020;
Revised: High-dimensional kernel learning: One line of research (Liang et al., 2020;

Original: McRae et al., 2022; Liang& Rakhlin, 2020;
Revised: McRae et al., 2022; Liang & Rakhlin, 2020;

Original: Liu et al., 2021) tackles high-dimensional kernel learning and proves non-asymptoticbounds using advanced high-dimensional random matrix concentration tools from El Karoui (2010).
Revised: Liu et al., 2021) tackles high-dimensional kernel learning and proves non-asymptotic bounds using advanced high-dimensional random matrix concentration tools from El Karoui (2010).

Original: However, those results heavily rely on a bounded Hilbert norm assumption.
Revised: However, those results heavily rely on a bounded Hilbert norm assumption.

Original: This assumption isnatural in the low-dimensional regime, but misleading in the high-dimensional regime, as pointed outin Donhauser et al.
Revised: This assumption is natural in the low-dimensional regime, but misleading in the high-dimensional regime, as pointed out in Donhauser et al.

Original: (2021b).
Revised: (2021b).

Original: Another line of research (Ghorbani et al., 2021; 2020; Mei et al., 2021;
Revised: Another line of research (Ghorbani et al., 2021; 2020; Mei et al., 2021;

Original: Ghosh et al., 2022; Misiakiewicz & Mei, 2021; Mei et al., 2022) asymptotically characterizes theprecise risk of kernel regression estimators in specific settings with access to a kernel’s eigenfunctionsand eigenvalues.
Revised: Ghosh et al., 2022; Misiakiewicz & Mei, 2021; Mei et al., 2022) asymptotically characterizes the precise risk of kernel regression estimators in specific settings with access to a kernel’s eigenfunctions and eigenvalues.

Original: However, these asymptotic results are insufficient to investigate how varying the filtersize of a convolutional kernel affects the risk of a kernel regression estimator.
Revised: However, these asymptotic results are insufficient to investigate how varying the filter size of a convolutional kernel affects the risk of a kernel regression estimator.

Original: In contrast to both linesof research, we prove tight non-asymptotic matching upper and lower bounds for high-dimensionalkernel learning which precisely capture the phase transition described in Section 3.2.
Revised: In contrast to both lines of research, we prove tight non-asymptotic matching upper and lower bounds for high-dimensional kernel learning which precisely capture the phase transition described in Section 3.2.

Original: Overfitting of structured interpolators: Several works question the generality of harmless interpolationfor models that incorporate strong structural assumptions.
Revised: Overfitting of structured interpolators: Several works question the generality of harmless interpolation for models that incorporate strong structural assumptions.

Original: Examples include structures enforcedvia data augmentation (Nishi et al., 2021), adversarial training Rice et al., 2020; Kamath et al.
Revised: Examples include structures enforced via data augmentation (Nishi et al., 2021), adversarial training (Rice et al., 2020; Kamath et al., 2021;

Original: ,2021; Sanyal et al., 2021; Donhauser et al., 2021a), neural network architectures (Li et al., 2021),pruning-based sparsity (Chang et al., 2021), and sparse linear models (Wang et al., 2022; Muthukumaret al., 2020; Chatterji & Long, 2022).
Revised: Sanyal et al., 2021; Donhauser et al., 2021a), neural network architectures (Li et al., 2021), pruning-based sparsity (Chang et al., 2021), and sparse linear models (Wang et al., 2022; Muthukumar et al., 2020; Chatterji & Long, 2022).

Original: In this paper, we continue that line of research and offer a newtheoretical perspective to characterize when interpolation is expected to be harmless.
Revised: In this paper, we continue that line of research and offer a new theoretical perspective to characterize when interpolation is expected to be harmless.

Original:  T HEORETICAL RESULTS
Revised:  T HEORETICAL RESULTS

Original: For convolutional kernels, a small filter size induces a strong bias towards estimators that dependnonlinearly on the input features only viaa small patch in the input.
Revised: For convolutional kernels, a small filter size induces a strong bias towards estimators that depend nonlinearly on the input features only via small patches.

Original: This section analyzes the effectof filter size (as an example inductive bias) on the degree of harmless interpolation for kernel ridgeregression.
Revised: This section analyzes the effect of filter size (as an example inductive bias) on the degree of harmless interpolation for kernel ridge regression.

Original: For this purpose, we derive and compare tight non-asymptotic bias and variance boundsas a function of filter size for min-norm interpolators and optimally ridge-regularized estimators(Theorem 1).
Revised: For this purpose, we derive and compare tight non-asymptotic bias and variance bounds as a function of filter size for min-norm interpolators and optimally ridge-regularized estimators (Theorem 1).

Original: Furthermore, we prove for large filter sizes that not only does harmless interpolationoccur (Theorem 1), but fitting some degree of noise is even necessary to achieve optimal testperformance (Theorem 2).
Revised: Furthermore, we prove for large filter sizes that not only does harmless interpolation occur (Theorem 1), but fitting some degree of noise is even necessary to achieve optimal test performance (Theorem 2).

Original: 3. S ETTING
Revised: 3. S ETTING

Original: We study kernel regression with a (cyclic) convolutional kernel in a high-dimensional setting wherethe number of training samples n scales with the dimension of the input data d as n ∈ Θ( d ℓ ) .
Revised: We study kernel regression with a (cyclic) convolutional kernel in a high-dimensional setting where the number of training samples n scales with the dimension of the input data d as n ∈ Θ( d ℓ ) .

Original: We usethe same setting as in previous works on high-dimensional kernel learning such as Misiakiewicz &
Revised: We use the same setting as in previous works on high-dimensional kernel learning such as Misiakiewicz &

Original: Mei (2021): we assume that the training samples { x i , y i } ni =1 are i.i.d.
Revised: Mei (2021): we assume that the training samples { x i , y i } ni =1 are i.i.d.

Original: draws from the distributionsx i ∼ U ( {− 1 , 1 } d ) , and y i =
Revised: draws from the distributions x i ∼ U ( {− 1 , 1 } d ) , and y i =

Original: f ⋆ ( x i )
Revised: f ⋆ ( x i )

Original: + ϵ i with ground truth f
Revised: + ϵ i with ground truth f

Original: ⋆ and noise ϵ ∼ N (0 , σ 2 ) .
Revised: ⋆ and noise ϵ ∼ N (0 , σ 2 ) .

Original: For simplicityof exposition, we further assume that f ⋆ ( x ) =
Revised: For simplicity of exposition, we further assume that f ⋆ ( x ) =

Original: x 1 x 2 · · · x L ∗ , with L ∗ specified in Theorem 1.
Revised: x 1 x 2 · · · x L ∗ , with L ∗ specified in Theorem 1.

Original: While the assumptions on the noise and ground truth can be easily extended by following the proofsteps in Section 5, generalizing the feature distribution is challenging.
Revised: While the assumptions on the noise and ground truth can be easily extended by following the proof steps in Section 5, generalizing the feature distribution is challenging.

Original: Indeed, existing results thatestablish precise risk characterizations (see Section 2) crucially rely on hypercontractivity of thefeature distribution — an assumption so far only proven for few high-dimensional distributions,including the hypersphere (Beckner, 1992), and the discrete hypercube (Beckner, 1975) which weuse in this paper.
Revised: Indeed, existing results that establish precise risk characterizations (see Section 2) crucially rely on hypercontractivity of the feature distribution — an assumption so far only proven for few high-dimensional distributions, including the hypersphere (Beckner, 1992), and the discrete hypercube (Beckner, 1975) which we use in this paper.

Original: Hypercontractivity is essential to tightly control the empirical kernel matrix within
Revised: Hypercontractivity is essential to tightly control the empirical kernel matrix within

Original: Lemma 3 in Section 5.
Revised: Lemma 3 in Section 5.

Original: Generalizations beyond this assumption require the development of new toolsin random matrix theory, which we consider important future work.
Revised: Generalizations beyond this assumption require the development of new tools in random matrix theory, which we consider important future work.

Original: We consider (cyclic) convolutional kernels with filter size q ∈ { 1 , . . .
Revised: We consider (cyclic) convolutional kernels with filter size q ∈ { 1 , . . .

Original: , d } of the form
Revised: , d } of the form

Original: [Equation]
Revised: [Equation]

Original: where x ( k,q ) :=
Revised: where x ( k,q ) :=

Original: [ x mod( k,d ) · · · x mod( k + q − 1 ,d ) ] , and κ : [ − 1 , 1] → R is a nonlinear function thatsatisfies standard regularity assumptions (see Assumption 1 in Appendix B) that hold for instance forthe exponential function.
Revised: [ x mod( k,d ) · · · x mod( k + q − 1 ,d ) ] , and κ : [ − 1 , 1] → R is a nonlinear function that implies standard regularity assumptions (see Assumption 1 in Appendix B) that hold for instance for the exponential function.

Original: Decreasing the filter size q restricts kernel regression solutions to dependnonlinearly only on local neighborhoods instead of the entire input x .
Revised: Decreasing the filter size q restricts kernel regression solutions to depend nonlinearly only on local neighborhoods instead of the entire input x .

Original: We analyze the kernel ridge regression (KRR) estimator, which is the minimizer of the followingconvex optimization problem:
Revised: We analyze the kernel ridge regression (KRR) estimator, which is the minimizer of the following convex optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where H is the Reproducing Kernel Hilbert space (RKHS) over {− 1 , 1 } d generated by the convolutional kernel K in Equation (1), ∥·∥ H the corresponding norm, and λ > 0 the ridge regularizationpenalty.
Revised: where H is the Reproducing Kernel Hilbert space (RKHS) over {− 1 , 1 } d generated by the convolutional kernel K in Equation (1), ∥·∥ H the corresponding norm, and λ > 0 the ridge regularization penalty.

Original: In the interpolation limit ( λ → 0 ), we obtain the min-RKHS-norm interpolator
Revised: In the interpolation limit ( λ → 0 ), we obtain the min-RKHS-norm interpolator

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we refer to ˆ f 0 as the kernel ridge regression estimator with λ = 0 .
Revised: For simplicity, we refer to ˆ f 0 as the kernel ridge regression estimator with λ = 0 .

Original: We evaluate all estimators with the expected population risk over the noise, defined as
Revised: We evaluate all estimators with the expected population risk over the noise, defined as

Original: [Equation]
Revised: [Equation]

Revised: (cid:124) (cid:123)(cid:122) (cid:125) :

Revised: =

Revised: Variance ( ˆ f λ ) .

Original: 3. M AIN R ESULT
Revised: 3. M AIN RESULT

Original: We now present tight upper and lower bounds for the prediction error of kernel regression estimatorsin the setting from Section 3.1.
Revised: We now present tight upper and lower bounds for the prediction error of kernel regression estimators in the setting from Section 3.1.

Original: The resulting rates hold for the high-dimensional regime, that is,when both the ambient dimension d and filter size q scale with n .
Revised: The resulting rates hold for the high-dimensional regime, that is, when both the ambient dimension d and filter size q scale with n .

Original: We defer the proof to Section 5.
Revised: We defer the proof to Section 5.

Original: Theorem 1 (Non-asymptotic prediction error rates) .
Revised: Theorem 1 (Non-asymptotic prediction error rates) .

Original: Let ℓ > 0 , β ∈ (0 , 1) , ℓ σ ∈ R .
Revised: Let ℓ > 0 , β ∈ (0 , 1) , ℓ σ ∈ R .

Original: Assume a dataset and a kernel as described in Section 3.1, with the kernel satisfying Assumption 1.
Revised: Assume a dataset and a kernel as described in Section 3.1, with the kernel satisfying Assumption 1.

Original: Assume further n ∈ Θ( d ℓ ) , the filter size q ∈ Θ (cid:0) d β (cid:1) , and σ 2 ∈ Θ( d − ℓ σ ) .
Revised: Assume further n ∈ Θ( d ℓ ) , the filter size q ∈ Θ (cid:0) d β (cid:1) , and σ 2 ∈ Θ( d − ℓ σ ) .

Original: Lastly, define ¯ δ :=
Revised: Lastly, define ¯ δ :=

Original: ℓ − 1 β − ⌊ ℓ − 1 β ⌋ and δ :=
Revised: ℓ − 1 β − ⌊ ℓ − 1 β ⌋ and δ :=

Original: ℓ − ℓ λ − 1 β − ⌊ ℓ − ℓ λ − 1 β ⌋ for any ℓ λ .
Revised: ℓ − ℓ λ − 1 β − ⌊ ℓ − ℓ λ − 1 β ⌋ for any ℓ λ .

Original: Then, with probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } uniformly over all ℓ λ ∈ [0 , ℓ − 1) , the KRR estimate ˆ f λ in Equation (2) with max { λ, 1 } ∈ Θ( d ℓ λ ) satisfies
Revised: Then, with probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } uniformly over all ℓ λ ∈ [0 , ℓ − 1) , the KRR estimate ˆ f λ in Equation (2) with max { λ, 1 } ∈ Θ( d ℓ λ ) satisfies

Original: [Equation]
Revised: [Equation]

Original: Further, for a ground truth f
Revised: Further, for a ground truth f

Original: ⋆ ( x ) =
Revised: ⋆ ( x ) =

Original: x 1 x 2 · · · x L ∗ with L ∗ ≤
Revised: x 1 x 2 · · · x L ∗ with L ∗ ≤

Original: [Equation]
Revised: [Equation]

Original: , with probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } , we have
Revised: , with probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } , we have

Original: [Equation]
Revised: [Equation]

Original: Finally, by setting ℓ λ = 0 , both rates also hold for the min-RKHS-norm interpolator ˆ f 0 in Eq.
Revised: Finally, by setting ℓ λ = 0 , both rates also hold for the min-RKHS-norm interpolator ˆ f 0 in Eq.

Original: Note how the theoremstatement reflects the usual intuition for the effect of noise and regularizationstrength on bias and variance via the parameter ℓ λ : With increasing ridge regularization ℓ λ (and thusincreasing λ ), the bias increases and the variance decreases.
Revised: Note how the theorem reflects the usual intuition for the effects of noise and ridge regularization strength on bias and variance via the parameter ℓ λ : With increasing ridge regularization ℓ λ (and thus increasing λ ), the bias increases and the variance decreases.

Original: Similarly, as noise increases (and thusdecreasing ℓ σ ), the variance increases.
Revised: Similarly, as noise increases (and thus ℓ σ decreases), the variance increases.

Original: Phase transition as a function of β : In the following, we focus on the impact of the filter sizeq ∈ Θ (cid:0) d β (cid:1) on the risk (sum of bias and variance) via the growth rate β .
Revised: Phase transition as a function of β : In the following, we focus on the impact of the filter size q ∈ Θ (cid:0) d β (cid:1) on the risk (sum of bias and variance) via the growth rate β .

Original: Recalling that a small filtersize (smallvalues of β ) corresponds to a strong inductive bias, and vice versa, Figure 1 demonstrateshow the strength of  inductive bias affects generalization.
Revised: Recalling that a small filter size (small β ) corresponds to a strong inductive bias, and vice versa, Figure 1 demonstrates how the strength of the inductive bias affects generalization.

Original: For illustration, we choose the ground truthf ⋆ ( x ) =
Revised: For illustration, we choose the ground truth f

Original:  x 1 x 2 so that the assumption on L ∗ is satisfied for all β .
Revised: ⋆ ( x ) = x 1 x 2 so that the assumption on L ∗ is satisfied for all β .

Original: Specifically, Figure 1a shows therates for the min-RKHS-norm interpolator ˆ f 0 and the optimally ridge-regularized estimator ˆ f λ opt ,where we choose λ opt to minimize the expected population risk Risk ( ˆ f λ opt ) .
Revised: Specifically, Figure 1a shows the rates for the min-RKHS-norm interpolator ˆ f 0 and the optimally ridge-regularized estimator ˆ f λ opt , where we choose λ opt to minimize the expected population risk Risk ( ˆ f λ opt ) .

Original: Furthermore, Figure 1bdepicts the (statistical) bias and variance of the interpolator ˆ f 0 .
Revised: Furthermore, Figure 1b depicts the (statistical) bias and variance of the interpolator ˆ f 0 .

Original: At the threshold β ∗ ∈ (0 , 1) , implicitlydefined as the β at which the rates of statistical bias and variance in Theorem 1 match, we can observethe following phase transition:
Revised: At the threshold β ∗ ∈ (0 , 1) , implicitly defined as the β at which the rates of statistical bias and variance in Theorem 1 match, we can observe the following phase transition:

Original: • For β < β ∗ , that is, for  strong inductive biases, the rates in Figure 1a for the optimally ridge- regularized estimator ˆ f λ opt are strictly better than the ones for the corresponding interpolator ˆ f 0 .
Revised: • For β < β ∗ , that is, for a strong inductive bias, the rates in Figure 1a for the optimally ridgeregularized estimator ˆ f λ opt are strictly better than the ones for the corresponding interpolator ˆ f 0 .

Original: In other words, we are observing harmful interpolation .
Revised: In other words, we are observing harmful interpolation .

Original: • For β > β ∗ , that is, for  weak inductive biases, the rates in Figure 1a of the optimally ridge-regularized estimator ˆ f λ opt and the min-RKHS-norm interpolator ˆ f 0 match.
Revised: • For β > β ∗ , that is, for a weak inductive bias, the rates in Figure 1a of the optimally ridge-regularized estimator ˆ f λ opt and the min-RKHS-norm interpolator ˆ f 0 match.

Original: Hence, we observe harmless interpolation .
Revised: Hence, we observe harmless interpolation .

Original: In the following theorem, we additionally show that interpolation is not only harmless for β > β ∗ , butthe optimally ridge-regularized estimator ˆ f λ opt necessarily fits part of the noise and has a training errorstrictly below the noise level.
Revised: In the following theorem, we additionally show that interpolation is not only harmless for β > β ∗ , but the optimally ridge-regularized estimator ˆ f λ opt necessarily fits part of the noise and has a training error strictly below the noise level.

Original: In contrast, we show that when interpolation is harmful in Figure 1a,that is, when β < β ∗ , the training error of the optimally ridge-regularized model approaches thenoise level.
Revised: In contrast, we show that when interpolation is harmful in Figure 1a, that is, when β < β ∗ , the training error of the optimally ridge-regularized model approaches the noise level.

Original: Theorem 2 (Training error (informal)) .
Revised: Theorem 2 (Training error (informal)) .

Original: Let λ opt be such that the expected population risk Risk ( ˆ f λ opt ) is minimal, and let β ∗ be the  threshold  where the bias and variance bounds in Theorem 1 are of the same order for the interpolator ˆ f 0 (setting ℓ λ = 0 ).
Revised: Let λ opt be such that the expected population risk Risk ( ˆ f λ opt ) is minimal, and let β ∗ be the unique threshold 3 where the bias and variance bounds in Theoremare of the same order for the interpolator ˆ f 0 (setting ℓ λ = 0 ).

Original: Then, the expected training error converges in probability:
Revised: Then, the expected training error converges in probability:

Original: [Equation]
Revised: [Equation]

Original: where c β < 1 for any β > β ∗ .
Revised: where c β < 1 for any β > β ∗ .

Original: We refer to Appendix D for the proof and a more general statement.
Revised: We refer to Appendix D.2 for the proof and a more general statement.

Original: Bias-variance trade-off: We conclude by discussing how the phase transition arises from a (statistical)bias and variance trade-off for the min-RKHS-norm interpolator as a function of β , reflected in interpolating optimally regularized0. β ∗ [Figure] ﬁlter
Revised: interpolating optimally regularized

Original: [Figure]

Original: (cid:0) d

Original: [Figure]

Original: (cid:1) -0.

Original: [Figure]
Revised: [Figure]

Original: − ‘ σ ‘ e rr o r r a t e e x p o n e n t α ↓
Revised: e rr o r r a t e e x p o n e n t α ↓

Original: [Figure]
Revised: [Figure]

Original: variance biasvar.
Revised: variance biasvar.

Original: lower bound0.
Revised: lower bound

Original: β ∗ β s.t. ﬁlter

Original: [Figure]
Revised: [Figure]

Original: Θ (cid:0) d β (cid:1) [Figure] [Figure] [Figure]
Revised: Bias-variance trade-off: We conclude by discussing how the phase transition arises from a (statistical) bias and variance trade-off for the min-RKHS-norm interpolator as a function of β , reflected in

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: a

Original: [Figure] n [Figure] Theorem 1 when setting ℓ λ = 0 and illustrated in Figure 1b.
Revised: Theorem 1 when setting ℓ λ = 0 and illustrated in Figure 1b.

Original: While the statistical bias monotonicallydecreases with decreasing β (i.e., increasing strength of the inductive bias), the variance follows amultiple descent curve with increasing minima as β decreases.
Revised: While the statistical bias monotonically decreases with decreasing β (i.e., increasing strength of the inductive bias), the variance follows a multiple descent curve with increasing minima as β decreases.

Original: Hence, analogous to the observationsin Donhauser et al.
Revised: Hence, analogous to the observations in Donhauser et al.

Original: for linear max- ℓ p -margin/min- ℓ p -norm interpolators, the interpolatorachieves the optimal performance at a β ∈ (0 , 1) , and therefore at a moderate inductive bias.
Revised: for linear max- ℓ p -margin/min- ℓ p -norm interpolators, the interpolator achieves its optimal performance at a β ∈ (0 , 1) , and therefore at a moderate inductive bias.

Original: Finally,we note that Liang et al.
Revised: Finally, we note that Liang et al.

Original: (2020) previously observed a multiple descent curve for the variance, but asa function of input dimension and without any connection to structural biases.
Revised: (2020) previously observed a multiple descent curve for the variance, but as a function of input dimension and without any connection to structural biases.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now empirically study whether the phase transition phenomenon that we prove for kernelregression persists for deep neural networks with feature learning.
Revised: We now empirically study whether the phase transition phenomenon that we prove for kernel regression persists for deep neural networks with feature learning.

Original: More precisely, we presentcontrolled experiments to investigate how the strength of a CNN’s inductive bias influences ifinterpolating noisy data is harmless.
Revised: More precisely, we present controlled experiments to investigate how the strength of a CNN’s inductive bias influences if interpolating noisy data is harmless.

Original: In practice, the inductive bias of a neural network varies byway of design choices such as the architecture (e.g., convolutional vs. fully-connected vs. graphnetworks) or the training procedure (e.g., data augmentation, adversarial training).
Revised: In practice, the inductive bias of a neural network varies by way of design choices such as the architecture (e.g., convolutional vs. fully-connected vs. graph networks) or the training procedure (e.g., data augmentation, adversarial training).

Original: We focus on twoexamples: convolutional filter size that we vary via the architecture, and rotational invariance viadata augmentation.
Revised: We focus on two examples: convolutional filter size that we vary via the architecture, and rotational invariance via data augmentation.

Original: To isolate the effects of inductive biases and provide conclusive results, we usedatasets where we know a priori that the ground truth exhibits a simple structure that matches thenetworks’ inductive bias.
Revised: To isolate the effects of inductive bias and provide conclusive results, we use datasets where we know a priori that the ground truth exhibits a simple structure that matches the networks’ inductive bias.

Original: See Appendix E for experimental details.
Revised: See Appendix E for experimental details.

Original: Analogous to ridge regularization for kernels, we use early stopping as a mechanism to prevent noisefitting.
Revised: Analogous to ridge regularization for kernels, we use early stopping as a mechanism to prevent noise fitting.

Original: Our experiments compare optimally early-stopped CNNs to their interpolating versions.
Revised: Our experiments compare optimally early-stopped CNNs to their interpolating versions.

Original: Thishighlights a trend that mirrors our theoretical results: the stronger the inductive bias of a neuralnetwork grows, the more harmful interpolation becomes.
Revised: This highlights a trend that mirrors our theoretical results: the stronger the inductive bias of a neural network grows, the more harmful interpolation becomes.

Original: These results suggest exciting future work:proving this trend for models with feature learning.
Revised: These results suggest exciting future work: proving this trend for models with feature learning.

Original: 4. F ILTER SIZE OF CNN S ON SYNTHETIC IMAGES
Revised: 4. F ILTER SIZE OF CNN S ON SYNTHETIC IMAGES

Original: In a first experiment, we study the impact of filter size on the generalization of interpolating CNNs.
Revised: In a first experiment, we study the impact of filter size on the generalization of interpolating CNNs.

Original: As a reminder, small filter sizes yield functions that depend nonlinearly only on local neighborhoodsof the input features.
Revised: As a reminder, small filter sizes yield functions that depend nonlinearly only on local neighborhoods of the input features.

Original: To clearly isolate the effects of filter size, we choose a special architectureon a synthetic classification problem such that the true label function is indeed a CNN with smallfilter size.
Revised: To clearly isolate the effects of filter size, we choose a special architecture on a synthetic classification problem such that the true label function is indeed a CNN with small filter size.

Original: In particular, we generate images of size 32 × 32 containing scattered circles (negativeclass) and crosses (positive class) with size at most 5 × 5 .
Revised: Concretely, we generate images of size 32 × 32 containing scattered circles (negative class) and crosses (positive class) with size at most 5 × 5 .

Original: Thus, decreasing filter size down to 5 × 5corresponds to a stronger inductive bias.
Revised: Thus, decreasing filter size down to 5 × 5 corresponds to a stronger inductive bias.

Original: Motivated by our theory, we hypothesize that interpolatingnoisy data is harmful with  small filter size, but harmless when using a large filter size.
Revised: Motivated by our theory, we hypothesize that interpolating noisy data is harmful with a small filter size, but harmless when using a large filter size.

Original: [Figure]
Revised: [Figure]

Original: t e s t
Revised: t e s t

Original: e rr o r
Revised: e rr o r

Original: [Figure]

Original: noisy

Original: [Figure]
Revised: [Figure]

Original: clean
Revised: clean

Original: [Figure]
Revised: [Figure]

Original: s ub s e t t r a i n e rr o r
Revised: s ub s e t t r a i n e rr o r

Original: ( o p t .
Revised: ( o p t .

Original: e a r l y s t
Revised: e a r l y s t

Original: o pp i n g )
Revised: o pp i n g )

Original: [Figure]
Revised: [Figure]

Original: Training setup: In the experiments, we use CNNs with a single convolutional layer, followed by globalspatial max pooling and two dense layers.
Revised: Training setup: In the experiments, we use CNNs with a single convolutional layer, followed by global spatial max pooling and two dense layers.

Original: We train those CNNs with different filter sizes on 200training samples (either noiseless or with 20% label flips) to minimize the logistic loss and achievezero training error.
Revised: We train those CNNs with different filter sizes ontraining samples (either noiseless or with 20% label flips) to minimize the logistic loss and achieve zero training error.

Original: We repeat all experiments over 5 random datasets with 15 optimizations perdataset and filter size, and report the average 0 - 1 -error for 100 k test samples per dataset.
Revised: We repeat all experiments over 5 random datasets with 15 optimizations per dataset and filter size, and report the average 0 - 1 -error for 100 k test samples per dataset.

Original: For a detaileddiscussion on the choice of hyperparameters and more experimental details, see Appendix E.1.
Revised: For a detailed discussion on the choice of hyperparameters and more experimental details, see Appendix E.1.

Original: Results: First, the noiseless error curves (blue) in Figure 2a confirm the common intuition thatthe strongest inductive bias (matching the ground truth) at size 5 yields the lowest test error.
Revised: Results: First, the noiseless error curves (dashed) in Figure 2a confirm the common intuition that the strongest inductive bias (matching the ground truth) at size 5 yields the lowest test error.

Original: Moreinterestingly, for 20% training noise (yellow), Figure 2a reveals a similar phase transition as Theorem 1and confirms our hypothesis: Models with weak inductive biases (large filter sizes) exhibit harmlessinterpolation, as indicated by the matching test error of interpolating (yellow solid) and optimallyearly-stopped (yellow dashed) models.
Revised: More interestingly, for 20% training noise (solid), Figure 2a reveals a similar phase transition as Theoremand confirms our hypothesis: Models with weak inductive biases (large filter sizes) exhibit harmless interpolation, as indicated by the matching test error of interpolating (blue) and optimally earlystopped (yellow) models.

Original: In contrast, as filter size decreases, models with a stronginductive bias (small filter sizes) suffer from an increasing gap in test errors when interpolatingversus using optimal early stopping.
Revised: In contrast, as filter size decreases, models with a strong inductive bias (small filter sizes) suffer from an increasing gap in test errors when interpolating versus using optimal early stopping.

Original: Furthermore, Figure 2b reflects the dual perspective of the phasetransition as presented in Theorem 2 under optimal early stopping: models with a small filter sizeentirely avoid fitting training noise, such that the training error on the noisy training subset equals100% , while models with a large filter size interpolate the noise.
Revised: Furthermore, Figure 2b reflects the dual perspective of the phase transition as presented in Theorem 2 under optimal early stopping: models with a small filter size entirely avoid fitting training noise, such that the training error on the noisy training subset equals 100% , while models with a large filter size interpolate the noise.

Original: Difference to double descent: One might suspect that our empirical observations simply reflectanother form of double descent (Belkin et al., 2019a).
Revised: Difference to double descent: One might suspect that our empirical observations simply reflect another form of double descent (Belkin et al., 2019a).

Original: As a CNN’s filter size increases (inductivebias becomes weaker), so does the number of parameters and degree of overparameterization.
Revised: As a CNN’s filter size increases (inductive bias becomes weaker), so does the number of parameters and degree of overparameterization.

Original: Thus,double descent predicts vanishing benefits of regularization due to model size for weak inductivebiases.
Revised: Thus, double descent predicts vanishing benefits of regularization due to model size for weak inductive biases.

Original: Nevertheless, we argue that the phenomenon we observe here is distinct, and provide anextended discussion in Appendix E.3.
Revised: Nevertheless, we argue that the phenomenon we observe here is distinct, and provide an extended discussion in Appendix E.3.

Original: In short, we choose sufficiently large networks and tune theirhyperparameters to ensure that all models interpolate and yield small training loss, even for filtersize 5 and 20% training noise.
Revised: In short, we choose sufficiently large networks and tune their hyperparameters to ensure that all models interpolate and yield small training loss, even for filter size 5 and 20% training noise.

Original: To justify this approach, we repeat a subset of the experiments whilesignificantly increasing the convolutional layer width.
Revised: To justify this approach, we repeat a subset of the experiments while significantly increasing the convolutional layer width.

Original: As the number of parameters increases fora fixed filter size, double descent would predict that the benefits of optimal early stopping vanish.
Revised: As the number of parameters increases for a fixed filter size, double descent would predict that the benefits of optimal early stopping vanish.

Original: However, we observe that our phenomenon persists.
Revised: However, we observe that our phenomenon persists.

Original: In particular, for filter size 5 (strongest inductivebias), the test error gap between interpolating and optimally early-stopped models remains large.
Revised: In particular, for filter size 5 (strongest inductive bias), the test error gap between interpolating and optimally early-stopped models remains large.

Original: 4. R OTATIONAL INVARIANCE OF W IDE R ESIDUAL N ETWORKS ON SATELLITE IMAGES
Revised: 4. R OTATIONAL INVARIANCE OF W IDE R ESIDUAL N ETWORKS ON SATELLITE IMAGES

Original: In a second experiment, we investigate rotational invariance as an inductive bias for CNNs whenevertrue labels are independent of an image’s rotation.
Revised: In a second experiment, we investigate rotational invariance as an inductive bias for CNNs whenever true labels are independent of an image’s rotation.

Original: Our experiments enforce inductive biases by fittingmodels on multiple rotated versions of an original training dataset, effectively performingvarying
Revised: Our experiments control inductive bias strength by fitting models on multiple rotated versions of an original training dataset, effectively performing

Original: [Figure]
Revised: [Figure]

Original: t e s t
Revised: t e s t

Original: e rr o r
Revised: e rr o r

Original: [Figure]
Revised: [Figure]

Original: noisy [Figure] clean 
Revised: noisy subset clean subset

Original: [Figure]
Revised: [Figure]

Original: s ub s e t t r a i n e rr o r
Revised: s ub s e t t r a i n e rr o r

Original: ( o p t .
Revised: ( o p t .

Original: e a r l y s t
Revised: e a r l y s t

Original: o pp i n g )
Revised: o pp i n g )

Original: [Figure]
Revised: [Figure]

Original:  degrees of data augmentation.
Revised: varying degrees of data augmentation.

Original: As an example dataset with a rotationally invariant ground truth, weclassify satellite images from the EuroSAT dataset (Helber et al., 2018) into 10 types of land usage.
Revised: As an example dataset with a rotationally invariant ground truth, we classify satellite images from the EuroSAT dataset (Helber et al., 2018) into 10 types of land usage.

Original: Because the true labels are independent of image orientation, we expect rotational invariance to be aparticularly fitting inductive bias for this task.
Revised: Because the true labels are independent of image orientation, we expect rotational invariance to be a particularly fitting inductive bias for this task.

Original: Training and test setup: For computational reasons, we subsample the original EuroSAT training setto 7680 raw training and 10 k raw test samples.
Revised: Training and test setup: For computational reasons, we subsample the original EuroSAT training set to 7680 raw training and 10 k raw test samples.

Original: In the noisy case, we replace 20% of the raw traininglabels with a wrong label chosen uniformly at random.
Revised: In the noisy case, we replace 20% of the raw training labels with a wrong label chosen uniformly at random.

Original: We then vary the strength of the inductivebias towards rotational invariance by augmenting the dataset with an increasing number of k rotatedversions of itself.
Revised: We then vary the strength of the inductive bias towards rotational invariance by augmenting the dataset with an increasing number of k rotated versions of itself.

Original: For each sample, we use k equal-spaced angles spanning 360 ◦ , plus a random offset.
Revised: For each sample, we use k equal-spaced angles spanning 360 ◦ , plus a random offset.

Original: Note that training noise applies before rotations, so that all rotated versions of the same image sharethe same label.
Revised: Note that training noise applies before rotations, so that all rotated versions of the same image share the same label.

Original: We then center-crop all rotated images such that they only contain valid pixels.
Revised: We then center-crop all rotated images such that they only contain valid pixels.

Original: In allexperiments, we fit Wide Residual Networks (Zagoruyko & Komodakis, 2016) on the augmentedtraining set for 5 different network initializations.
Revised: In all experiments, we fit Wide Residual Networks (Zagoruyko & Komodakis, 2016) on the augmented training set for 5 different network initializations.

Original: We evaluate the 0 - 1 -error on the randomly rotatedtest samples to avoid distribution shift effects from image interpolation.
Revised: We evaluate the 0 - 1 -error on the randomly rotated test samples to avoid distribution shift effects from image interpolation.

Original: All random rotations are thesame for all experiments and stay fixed throughout training.
Revised: All random rotations are the same for all experiments and stay fixed throughout training.

Original: See Appendix E.2 for more experimentaldetails.
Revised: See Appendix E.2 for more experimental details.

Original: Lastly, we perform additional experiments with larger models to differentiate from doubledescent; see Appendix E.3 for the results and further discussion.
Revised: Lastly, we perform additional experiments with larger models to differentiate from double descent; see Appendix E.3 for the results and further discussions.

Original: Results: Similar to the previous subsection
Revised: Results: Similar to the previous subsection

Original: , Figure 3a corroborates our hypothesis under rotationalinvariance: stronger inductive biases result in lower test errors on noiseless data, but an increasedgap between the test errors of interpolating and optimally early-stopped models.
Revised: , Figure 3a corroborates our hypothesis under rotational invariance: stronger inductive biases result in lower test errors on noiseless data, but an increased gap between the test errors of interpolating and optimally early-stopped models.

Original: In contrast to filtersize, the phase transition is more abrupt; invariance to 180 ◦ rotations already prevents harmlessinterpolation.
Revised: In contrast to filter size, the phase transition is more abrupt; invariance to 180 ◦ rotations already prevents harmless interpolation.

Original: Figure 3b confirms this from a dual perspective, since all models with some rotationalinvariance cannot fit noisy samples for optimal generalization.
Revised: Figure 3b confirms this from a dual perspective, since all models with some rotational invariance cannot fit noisy samples for optimal generalization.

Original:  P ROOF OF THE MAIN RESULT
Revised:  P ROOF OF THE MAIN RESULT

Original: The proof of the main result, Theorem 1, proceeds in two steps: First, Section 5.1 presents a fixeddesign result that yields matching upper and lower bounds for the prediction error of general kernelsunder additional conditions.
Revised: The proof of the main result, Theorem 1, proceeds in two steps: First, Section 5.1 presents a fixeddesign result that yields matching upper and lower bounds for the prediction error of general kernels under additional conditions.

Original: Second, in Section 5.2, we show that the setting of Theorem 1 satisfiesthose conditions with high probability over dataset draws.
Revised: Second, in Section 5.2, we show that the setting of Theorem 1 satisfies those conditions with high probability over dataset draws.

Original: Notation Assuming that inputs are draws from a data distribution ν (i.e., x, x ′ ∼ ν ), we candecompose and divide any continuous, positive semi-definite kernel function as
Revised: Notation Assuming that inputs are draws from a data distribution ν (i.e., x, x ′ ∼ ν ), we can decompose and divide any continuous, positive semi-definite kernel function as

Original: [Equation]
Revised: [Equation]

Original: where { ψ k } k ≥ 1 is an orthonormal basis of the RKHS induced by ⟨ f, g ⟩ ν :=
Revised: where { ψ k } k ≥ 1 is an orthonormal basis of the RKHS induced by ⟨ f, g ⟩ ν :=

Original: E x ∼ ν [ f ( x ) g ( x )]and the eigenvalues λ k are sorted in descending order.
Revised: E x ∼ ν [ f ( x ) g ( x )] and the eigenvalues λ k are sorted in descending order.

Original: In the following, we write [ · ] i,j to referto the entry in row i and column j of a matrix.
Revised: In the following, we write [ · ] i,j to refer to the entry in row i and column j of a matrix.

Original: Then, we define the empirical kernel matrixfor K as K ∈ R n × n with [ K ] i,j :=
Revised: Then, we define the empirical kernel matrix for K as K ∈ R n × n with [ K ] i,j :=

Original: K ( x i , x j ) , and analogously the truncated versions K ≤ mand K >m for K ≤ m and K >m , respectively.
Revised: K ( x i , x j ) , and analogously the truncated versions K ≤ m and K >m for K ≤ m and K >m , respectively.

Original: Next, we utilize the matrices Ψ ≤ m ∈ R n × m with[ Ψ ≤ m ] i,l :=
Revised: Next, we utilize the matrices Ψ ≤ m ∈ R n × m with [ Ψ ≤ m ] i,l :=

Original: ψ l ( x i ) , and D ≤ m := diag ( λ 1 , . . . , λ m ) .
Revised: ψ l ( x i ) , and D ≤ m := diag ( λ 1 , . . . , λ m ) .

Original: We further use the squared kernel S ( x, x ′ ) :
Revised: We further use the squared kernel S ( x, x ′ ) :

Original: =251 E z ∼ ν
Revised: = E z ∼ ν

Original: [ K ( x, z ) K ( z, x ′ )] , its truncated versions S ≤ m and S >m , as well as the corresponding empiricalkernel matrices S , S ≤ m , S >m ∈ R n × n .
Revised: [ K ( x, z ) K ( z, x ′ )] , its truncated versions S ≤ m and S >m , as well as the corresponding empirical kernel matrices S , S ≤ m , S >m ∈ R n × n .

Original: Finally, we write µ min ( · ) and µ max ( · ) (or ∥·∥ ) to indicatethe min and max eigenvalue of a symmetric positive-definite matrix, respectively, and  use ⟨· , ·⟩ toindicate the Euclidean inner product in R d .
Revised: Next, for a symmetric positive-definite matrix, we write µ min ( · ) and µ max ( · ) (or ∥·∥ ) to indicate the min and max eigenvalue, respectively, and µ i ( · ) for the i -th eigenvalue in decreasing order. Finally, we use ⟨· , ·⟩ for the Euclidean inner product in R d .

Original: 5. G ENERALIZATION BOUND FOR FIXED - DESIGN
Revised: 5. G ENERALIZATION BOUND FOR FIXED - DESIGN

Original: First, Theorem 3 provides tight fixed-design bounds for the prediction error.
Revised: First, Theorem 3 provides tight fixed-design bounds for the prediction error.

Original: Theorem 3 (Generalization bound for fixed-design) .
Revised: Theorem 3 (Generalization bound for fixed-design) .

Original: Let K: X × X → R be a kernel, and{ (  x i , y i ) } ni
Revised: Let K be a kernel that under a distribution ν decomposes as K ( x, x ′ ) =

Original: =1 a dataset where y i =
Revised: (cid:80) k λ k ψ k ( x ) ψ k ( x ′ ) with E x ∼ ν [ ψ k ( x ) ψ k ′ ( x )] = δ k,k ′ , and { ( x i , y i ) } ni = be a dataset with y i =

Original: f ⋆ ( x i )
Revised: f ⋆ ( x i )

Original: + ϵ
Revised: + ϵ

Original: i for zero-mean  i.i.d.
Revised: i for zero-mean σ 2 -variance i.i.d.

Original: noise ϵ i with variance σ 2and a ground truth f ⋆ .
Revised: noise ϵ i and ground truth f ⋆ .

Original: For any m ∈ N such that
Revised: Define τ 1 := min

Original: [Equation]
Revised: [Equation]

Original: the KRR estimate ˆ f λ in Equation (2) for any λ ≥ 0has a variance upper andlower bounded by
Revised: Then, for any m ∈ N such that r 1 > 0 and

Original: [Equation]
Revised: [Equation]

Original: where τ 1 = min
Revised: the KRR estimate ˆ f λ in Equation (2) for any λ ≥ 0 has a variance upper and lower bounded by

Original: [Equation]
Revised: [Equation]

Original: Furthermore, for any ground truth which can be expressed as f ⋆ ( x ) = (cid:80) mk =1 a k ψ k ( x ) with a ∈ R m + and ψ k as defined via Equation (4) , the bias is upper and lower bounded by
Revised: Furthermore, for any ground truth that can be expressed as f ⋆ ( x ) = (cid:80) mk =1 a k ψ k ( x ) with a ∈ R m + and ψ k as defined in Equation (4) , the bias is upper and lower bounded by

Original: [Equation]
Revised: [Equation]

Original: See Appendix A for the proof.
Revised: See Appendix A.1 for the proof.

Original: Note that this result holds for any fixed-design dataset.
Revised: Note that this result holds for any fixed-design dataset.

Original: We derivethe main ideas from the proofs in Bartlett et al.
Revised: We derive the main ideas from the proofs in Bartlett et al.

Original: ; Tsigler & Bartlett (2020), where the authorsestablish tight bounds for the min- ℓ 2 -norm interpolator on independent sub-Gaussian features.
Revised: ; Tsigler & Bartlett (2020), where the authors establish tight bounds for the min- ℓ 2 -norm interpolator on independent sub-Gaussian features.

Original: Remark 1 (Comparison with McRae et al.
Revised: Remark 1 (Comparison with McRae et al.

Original: The upper bound for the bias in Theorem 1 from
Revised: The upper bound for the bias in Theorem 1 from

Original: McRae et al.
Revised: McRae et al.

Original: (2022) depends on the suboptimal term ∥ D − 1 / 2 ≤ m a ∥ /n , but also applies to more generalground truths.
Revised: (2022) depends on the suboptimal term ∥ D − 1 / 2 ≤ m a ∥ /n , but also applies to more general ground truths.

Original: We improve that  bound in Theorem 3 and present a matching lower bound.
Revised: We improve that upper bound in Theorem 3 and present a matching lower bound.

Original: 5. P ROOF OF T HEOREM1
Revised: 5. P ROOF OF T HEOREM

Original: Throughout the remainder of the proof, all statements hold for the setting in Section 3.1 and under theassumptions of Theorem 1; in particular, Assumption 1 on κ , n ∈ Θ( d ℓ ) , and max { λ, 1 } ∈ Θ( d ℓ λ ) .
Revised: Throughout the remainder of the proof, all statements hold for the setting in Section 3.1 and under the assumptions of Theorem 1, especially Assumption 1 on K , n ∈ Θ( d ℓ ) , and max { λ, 1 } ∈ Θ( d ℓ λ ) .

Original: Furthermore, as in Theorem 1, we use δ, ¯ δ > 0 where δ :=
Revised: Furthermore, as in Theorem 1, we use δ, ¯ δ > 0 with δ :=

Original: ℓ − ℓ λ − 1 β − (cid:106) ℓ − ℓ λ − 1 β (cid:107) and ¯ δ := 
Revised: ℓ − ℓ λ − 1 β − (cid:106) [Equation] (cid:107) and ¯ δ := ℓ

Original: ℓ − 1 β − (cid:106) ℓ − 1 β (cid:107) .
Revised: [Equation] − (cid:106) ℓ − 1 β (cid:107) .

Original: We show that there exists a particular m for which the conditions in Theorem 3 hold and we cancontrol the terms in the bias and variance bounds.
Revised: We show that there exists a particular m for which the conditions in Theorem 3 hold and we can control the terms in the bias and variance bounds.

Original: Step 1: Conditions for the bounds in Theorem 3. We first derive sufficient conditions on m suchthat the conditions on Ψ ≤ m and f ⋆ in Theorem 3 hold with high probability.
Revised: Step 1: Conditions for the bounds in Theorem 3 We first derive sufficient conditions on m such that the conditions on Ψ ≤ m and f ⋆ in Theorem 3 hold with high probability.

Original: The followingresultfrom standardrandom matrix concentration  shows that any m ≪ n satisfies Equation (5) with highprobability.
Revised: The following standard concentration result shows that all m ≪ n satisfy Equation (5) with high probability.

Original: Lemma 1 (Corollary of Theorem 5.44 in Vershynin (2012)) .
Revised: Lemma 1 (Corollary of Theorem 5.44 in Vershynin (2012)) .

Original: For d large enough, with probability atleast 1 − cd − β ¯ δ , all m ∈ O ( n · d −β ¯ δ ) satisfy Equation (5) .
Revised: For d large enough, with probability at least 1 − cd − β ¯ δ , all m ∈ O ( n · q − ¯ δ ) satisfy Equation (5) .

Original: See Appendix C.1 for the proof.
Revised: See Appendix C.1 for the proof.

Original: Simultaneously, to ensure that f ⋆ is contained in the span of the firstm eigenfunctions, m must be sufficiently large.
Revised: Simultaneously, to ensure that f ⋆ is contained in the span of the first m eigenfunctions, m must be sufficiently large.

Original: We formalize this in the following lemma.
Revised: We formalize this in the following lemma.

Original: Lemma 2 (Condition for the bias bound) .
Revised: Lemma 2 (Bias bound condition) .

Original: Consider a kernel as in Theorem 1 satisfying Assumption 1,and a ground truth f
Revised: Consider a kernel as in Theorem 1 satisfying Assumption 1, and a ground truth f

Original: ⋆ ( x ) = (cid:81)
Revised: ⋆ ( x ) = (cid:81)

Original: L ∗ j =1 x j withL ∗ ≥ 1 .
Revised: [Equation] with 1 ≤ [Equation] ≤

Original:  Then, for any m such that λ m ∈ o (cid:16) 1 dq L ∗−(cid:17) andd sufficiently large, f ⋆ is in the span of the first m eigenfunctions and(cid:13)(cid:13)(cid:13)
Revised: [Equation] Then, for any m with λ m ∈ o (cid:16) 1 dq L ∗− [Equation] (cid:17) and d sufficiently large, f ⋆ is in the span of the first m eigenfunctions and

Original: D − 1 ≤ m a (cid:13)(cid:13)(cid:13) ∈ Θ (cid:0) dq L ∗ −(cid:1) .
Revised: [Equation] (cid:0) [Equation] (cid:1) .

Original: See Appendix B.3 for the proof.
Revised: See Appendix B.3 for the proof.

Original:  Step 2: Concentration of the (squared) kernel matrix. In a second step, we show existence ofan m that satisfies the sufficient conditions, and for which the spectrum of the kernel matrix K >mand squared kernel matrix S >m concentrates. First, Lemma 3 yields high-probability bounds for aspecific set of m .
Revised: Note that L ∗ ≥ 1 follows from ℓ λ < ℓ − 1 in Theorem 1, and allows us to focus on non-trivial ground truth functions. Step 2: Concentration of the (squared) kernel matrix In a second step, we show that there exists a set of m that satisfy the sufficient conditions, and for which the spectra of the kernel matrix K >m and squared kernel matrix S >m concentrate.

Original: Lemma 3 (Tight bound conditions) .
Revised: Lemma 3 (Tight bound conditions) .

Original: With probability at least 1 − cd −2 β min { ¯ δ, 1 − ¯ δ } uniformly over all ℓ λ ∈ [0 , ℓ − 1) , for  any λ ∈ R and m ∈ N with max { λ, 1 } ∈ Θ( d ℓ λ ) and nλ m ∈ Θ(max { λ, 1 } ) ,  we have 
Revised: With probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } uniformly over all ℓ λ ∈ [0 , ℓ − 1) , for d sufficiently large and any λ ∈ R , m ∈ N with max { λ, 1 } ∈ Θ( d ℓ λ ) , nλ m ∈ Θ(max { λ, 1 } ) , and m ∈ O (cid:16) nq − δ max { λ, 1 } (cid:17) , we have r 1 , r 2 ∈ Θ(1) , Tr

Original: [Equation]
Revised: [Equation]

Original: and
Revised: m

Original: [Equation]
Revised: [Equation]

Original: We refer to Appendix C.2 for the proof, which heavily relies on the feature distribution.
Revised: We refer to Appendix C.3 for the proof, which heavily relies on the feature distribution.

Original: Note that theresults for m in Lemma 3 also imply τ 1 , τ 2 ∈ Θ(1) .
Revised: Note that the results for m in Lemma 3 also imply τ 1 , τ 2 ∈ Θ(1) .

Original: Step 3:Completing the proof.
Revised: Step 3:

Original:  Finally, we complete the proof by showing the existence of aparticular m that simultaneously satisfies all conditions on m in Lemmas 1 to 3.
Revised: Completing the proof Finally, we complete the proof by showing the existence of a particular m that simultaneously satisfies all conditions of Lemmas 1 to 3.

Original: Lemma 4 (Eigendecay) .
Revised: Lemma 4 (Eigendecay) .

Original: There exists an m such that
Revised: There exists an m such that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Furthermore, assuming L ∗ ≤ (cid:108) ℓ − ℓ λ − 1 β (cid:109) , we have λ m ∈
Revised: Furthermore, assuming L ∗ ≤ (cid:108) [Equation] λ − [Equation]

Original:  o (cid:16) 1 d · q L ∗−(cid:17) .
Revised: β (cid:109) , we have λ m ∈ o [Equation]

Original: We refer to Appendix B.4 for the proof.
Revised: We refer to Appendix B.4 for the proof.

Original: As a result, we can use Lemmas 1 to 4 to instantiate
Revised: As a result, we can use Lemmas 1 to 4 to instantiate

Original: Theorem 3 for the setting in Theorem 1, resulting in the following tight high-probability bounds for variance and bias:
Revised: Theorem 3 for the setting in Theorem 1, resulting in the following tight high-probability bounds for variance and bias:

Original: [Equation]
Revised: [Equation]

Original: Reformulating the bounds in terms of n then concludes the proof of Theorem 1.
Revised: Reformulating the bounds in terms of n then concludes the proof of Theorem 1.

Original:  S UMMARY AND FUTURE WORK
Revised:  S UMMARY AND OUTLOOK

Original: In this paper, we highlight how the strength of an inductive bias impacts generalization.
Revised: In this paper, we highlight how the strength of an inductive bias impacts generalization.

Original: Moreprecisely, we study when the gap in test error between  optimally ridge-regularized or early-stoppedmodels and their interpolating counterparts is zero, that is, when interpolation is harmless.
Revised: Concretely, we study when the gap in test error between interpolating models and their optimally ridge-regularized or early-stopped counterparts is zero, that is, when interpolation is harmless.

Original: In particular, we prove a phase transition for kernel regression using convolutional kernels with different filtersizes: a weak inductive bias (large filter size) yields harmless interpolation, and even requires fittingnoise for optimal test performance, whereas interpolating noise leads to suboptimal generalization fora strong inductive bias (small filter size).
Revised: In particular, we prove a phase transition for kernel regression using convolutional kernels with different filter sizes: a weak inductive bias (large filter size) yields harmless interpolation, and even requires fitting noise for optimal test performance, whereas a strong inductive bias (small filter size) suffers from suboptimal generalization when interpolating noise.

Original: Intuitively, this phenomenon arises from a bias-variancetrade-off, captured by our main result in Theorem 1: with increasing inductive bias, the risk onnoiseless training samples (bias) decreases, while the sensitivity to noise in the training data (variance)increases.
Revised: Intuitively, this phenomenon arises from a bias-variance trade-off, captured by our main result in Theorem 1: with increasing inductive bias, the risk on noiseless training samples (bias) decreases, while the sensitivity to noise in the training data (variance) increases.

Original: Our empirical results on neural networks suggest that this phenomenon might extend tomodels with feature learning, which opens up an avenue for exciting future work.
Revised: Our empirical results on neural networks suggest that this phenomenon extends to models with feature learning, which opens up an avenue for exciting future work.

Original: R
Revised: K.D. is supported by the ETH AI Center and the ETH Foundations of Data Science. We would further like to thank Afonso Bandeira for insightful discussions

Original: AP ROOF OF T HEOREM 3: G ENERALIZATION BOUND FOR FIXED - DESIGN
Revised: A G ENERALIZATION BOUND FOR FIXED - DESIGN

Original: We proof Theorem 3 by deriving a closed-form expression for the bias and variance, and boundingeach term individually.
Revised: We prove Theorem 3 by deriving a closed-form expression for the bias and variance, and bounding them individually.

Original: Due to the closed form, the proof does not rely on any matrix concentrationresults. Proof.
Revised: Hence, the proof does not rely on any matrix concentration results. A. P ROOF OF T HEOREM

Original: It is well-known that the KRR problem defined in Equation (2) is solved by the estimator
Revised: It is well-known that the KRR problem defined in Equation (2) yields the estimator

Original: [Equation]
Revised: [Equation]

Original: where  x i ∼ν , y :
Revised: where x, x [Equation] ∼

Original: = f + ϵ =
Revised: [Equation] [Equation] :=

Original: [ f ⋆ ( x 1 ) + ϵ 1 , . . .
Revised: [Equation] = [Equation]

Original: , f ⋆ ( x n ) +ϵ n ] ⊺ , k ( x ) :=
Revised: +

Original: [ K ( x 1 , x ) , . . .
Revised: [Equation] [Equation] [Equation] [Equation]

Original: , K ( x n , x )] ⊺ , and H :=
Revised: K [Equation] n , x )] ⊺ , and H : =

Original: K + λ I n .
Revised: K + λ I n .

Original: For this estimator, both bias and variance exhibit a closed-form expression: 
Revised: For this estimator, both bias and variance exhibit a closed-form expression: Bias

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: In ( i ) we use the definition of the squared kernel S ,and that f ⋆ ( x ) =(cid:80)
Revised: Step ( i ) uses the definition of the squared kernel S , that f ⋆ ( x ) =

Original: [Equation]
Revised: [Equation]

Original:  k ( x ) , since we are assuming that f ⋆ is in the span of the first m eigenfunctions, as well as the followingsimple consequences of the eigenfunctions’ orthonormality:
Revised: mk [Equation] [Equation] k ψ k ( x ) as f ⋆ is in the span of the first m eigenfunctions, and the following consequences of the eigenfunctions’ orthonormality:

Original: [Equation]
Revised: [Equation]

Original: We now bound the closed-form expressions of bias and variance individually.
Revised: We now bound the closed-form expressions of bias and variance individually.

Original: Bias Lemma 5 below yields S =
Revised: Bias Lemma 5 below yields S =

Original: Ψ ≤ m D  ≤ m Ψ ⊺ ≤ m + S >m .
Revised: Ψ ≤ m D 2 ≤ m Ψ ⊺ ≤ m + S >m .

Original: Hence, the bias decomposes as
Revised: Hence, the bias decomposes into

Original: Bias
Revised: Bias

Original: [Equation]
Revised: [Equation]

Revised: ≤ m a = (cid:13) (cid:13)(cid:13) (cid:16)

Revised: I m − D ≤ m Ψ ⊺

Revised: ≤ m H − 1 Ψ ≤ m (cid:17) a (cid:13) (cid:13)(cid:13) (cid:124) (cid:123)(cid:122) (cid:125) :

Revised: = B+ a ⊺ Ψ ⊺ ≤ m H − 1 S >m H − 1 Ψ ≤ m a (cid:124) (cid:123)(cid:122) (cid:125) :

Revised: =

Original:  First, we rewrite B 1 as
Revised: B. First, we rewrite B 1 as

Revised: B 1 = (cid:13)(cid:13)(cid:13)(cid:16)

Original: [Equation]
Revised: I m − D ≤ m Ψ ⊺ ≤ m H − 1 Ψ ≤ m (cid:17) a (cid:13)(cid:13)(cid:13) ( i ) = (cid:13)(cid:13)(cid:13)(cid:13)(cid:18) D ≤ m − D ≤ m Ψ ⊺ ≤ m (cid:16) Ψ ≤ m D ≤ m Ψ ⊺ ≤ m + H >m (cid:17) − 1 Ψ ≤ m D ≤ m (cid:19)

Original:  where ( i ) uses the decomposition H = K ≤ m + H >m with H >m :
Revised: D − 1 ≤ m a (cid:13)(cid:13)(cid:13)(cid:13) ( ii ) = (cid:13)(cid:13)(cid:13)(cid:13)(cid:16) D − 1 ≤ m + Ψ ⊺ ≤ m H − 1 >m Ψ ≤ m (cid:17) − 1 D − 1 ≤ m a (cid:13)(cid:13)(cid:13)(cid:13) , (8) where ( i ) uses the decomposition H = K ≤ m + H >m with H >m :=

Original: = K >m + λ I n , and ( ii ) appliesthe Woodbury  identity.
Revised: K >m + λ I n , and ( ii ) applies the Woodbury matrix identity.

Original: Second, we can upper-bound B 2 as follows:
Revised: Second, we can upper-bound B 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: = B

Original: [Equation]

Original: where ( i ) uses Lemma 6.
Revised: where ( i ) uses Lemma 6.

Original: Thus, the bias can be bounded by B 1 (1 + C 1 ) with C 1 ≥ 
Revised: Thus, the bias can be bounded by B 1 (1 + C 1 ) with C 1 ≥ 0 .

Original: [Equation]

Original: We proceed by upper bounding C 1 :
Revised: We proceed by upper bounding C 1 :

Original: [Equation]
Revised: [Equation]

Original: where ( i ) uses Equation (5) to bound ∥ Ψ ⊺ ≤ m Ψ ≤ m /n ∥, and Lemma 7 to bound ∥ S >m ∥ . In ( ii ) weuse the bound: cx + 1 ≤ ( c + 1) max { x, 1 }  .
Revised: where ( i ) uses Equation (5) to bound ∥ Ψ ⊺ ≤ m Ψ ≤ m /n ∥ and Lemma 7 to bound ∥ S >m ∥ , and ( ii ) follows from cx + 1 ≤ ( c + 1) max { x, 1 } for c ≥ 0 .

Original: In ( iii ) we just see that the terms matches the definitionof τ 2 .

Original: Hence, to conclude the bias bound, we need to bound B 1  from above and below.
Revised: Hence, to conclude the bias bound, we need to bound B 1 in Equation (8) from above and below.

Original: Upper bound:
Revised: Upper bound:

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from Equation (5).
Revised: where ( i ) follows from Equation (5)

Original: This concludes the upper bound on the bias since we can compute:

Original: [Equation]
Revised: [Equation]

Original: where in ( i )we use the bound in Equation (10), and  in ( ii ) we used Equation (11). This concludesthe upper bound on the bias.
Revised: Combining Equation (9) in ( i ) and Equation (10) in ( ii ) yields the desired upper bound on the bias: Bias [Equation]

Original: Lower bound:
Revised: Lower bound:

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from Equation (5),  ( ii ) from the fact that 1 1+ cx ≤ 11+ c min { x, 1 } , and ( iii ) thedefinition of τ 1 .
Revised: where ( i ) follows from Equation (5), and ( ii ) from the fact that [Equation] { x [Equation]

Original:  Since Bias 2 ≥ B 1 , this concludes the lower bound for the bias.
Revised: 1 } for x, c ≥ [Equation] Since Bias 2 ≥ B 1 , this concludes the lower bound for the bias.

Original: Variance: As for the bias bound, we first apply Lemma 5 to write S
Revised: Variance As for the bias bound, we first apply Lemma 5 to write S

Original: = Ψ ≤ m D  ≤ m Ψ ⊺ ≤ m + S >m ,and decompose the variance in Equation (7) as
Revised: = Ψ ≤ m D 2 ≤ m Ψ ⊺ ≤ m + S >m and decompose the variance in Equation (7) into

Original: [Equation]
Revised: [Equation]

Original: (cid:17) (cid:124) (cid:123)(cid:122) (cid:125) [Equation] (cid:0) H
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: (cid:1) (cid:124) (cid:123)(cid:122) (cid:125)

Original: V. Next, we rewrite V 1 as follows:
Revised: we rewrite V 1 as follows:

Original: [Equation]
Revised: [Equation]

Original: Step ( i ) follows from Lemma6.
Revised: where ( i ) follows from Lemma

Original:  To bound V 1 and V 2 , we use the fact that the trace is the sum of all eigenvalues.
Revised: [Equation] To bound V 1 and V [Equation] we use the fact that the trace is the sum of all eigenvalues.

Original: Therefore, the trace is bounded from above and below by the size of the matrix times the largest and smallest eigenvalue, respectively.
Revised: Therefore, the trace is bounded from above and below by the size of the matrix times the largest and smallest eigenvalue, respectively.

Original: This yields the following bound for V 1 : 
Revised: This yields the following bounds for V 1 : [Equation] n

Original: [Equation]
Revised: [Equation]

Original: τ
Revised: and

Original:  For ( i ) and ( iii ) , we bound the terms analogously to the upper and lower bound of B1 , and use
Revised: [Equation] For ( i ) and ( iii ) [Equation] we bound the terms analogously to the upper and lower bound of B

Revised: [Equation]

Original:  Equation (5) in ( ii ) and ( iv ) .
Revised: and use Equation (5) in ( ii ) and ( iv ) .

Revised: Next, the upper bound on V 2 follows from a special case of Hölder’s inequality as follows:

Revised: [Equation]

Revised: For the lower bound

Revised: [Equation]

Revised: we need a more accurate analysis.

Original: Next, we bound V 2 as
Revised: First [Equation] we apply the identity [Equation]

Revised: which is valid since H and H >m are full rank.

Revised: Next, note that the rank of A is bounded by the rank of K ≤ m , which can be written as Ψ ≤ m D ≤ m Ψ ⊺ ≤ m and therefore has itself rank at most m .

Revised: Furthermore, Equation (5) implies that Ψ ⊺ ≤ m Ψ ≤ m has full rank, and hence m ≤ n .

Revised: Let now { v 1 , . .

Revised: . , v rank( A ) } be an orthonormal basis of col( A ) , and let { v rank

Revised: ( A )+1 , . .

Revised: . , v n } be an orthonormal basis of col( A ) ⊥ .

Revised: Hence, { v 1 , . .

Revised: . , v n } is an orthonormal basis of R n , and similarity invariance of the trace yields

Revised: [Equation]

Revised: where P is the projection matrix of R n onto col( A ) ⊥ , and ( i ) follows from Equation (12).

Revised: Step ( ii ) uses that, for all i > rank( A ) , v i is orthogonal to the column space of A , and hence

Revised: [Equation]

Revised: (cid:124)(cid:123)(cid:122)(cid:125)

Revised: [Equation]

Revised: Finally, let µ i ( · ) be the i -th eigenvalue of its argument with respect to a decreasing order.

Revised: Then, the Cauchy interlacing theorem yields µ rank( A )+ i ( S ≤ m ) ≤ µ i ( P ⊺ S >m P ) for all i = 1 , . . .

Revised: , n − rank( A ) .

Revised: This implies

Original: [Equation]
Revised: [Equation]

Revised: where ( i ) uses that the rank of A is bounded by m .

Original: This concludes the  bound for the variance, and hence the proof.
Revised: This concludes the lower bound on V 2 as follows: [Equation]

Original: A. A DDITIONAL LEMMAS
Revised: A. T ECHNICAL LEMMAS

Original: Lemma 5 (Squared kernel decomposition) .
Revised: Lemma 5 (Squared kernel decomposition) .

Original: Let K be a kernel function that under a distribution ν can be decomposed as K ( x, x ′ )
Revised: Let K be a kernel function that under a distribution ν can be decomposed as K ( x, x ′ )

Original: = (cid:80) 
Revised: = (cid:80) k

Original: [Equation] x ) ψ k ( x ′ ) , where E x ∼ ν [ ψ k ( x ) ψ k ′ ( x )]=
Revised: λ k ψ k ( x ) ψ k ( x ′ ) , where E x ∼ ν [ ψ k ( x ) ψ k ′ ( x )]

Original:  δ k,k ′ .
Revised: = δ k,k ′ .

Original: Then the squared kerneldefined as S ( x, x ′ ) =
Revised: Then, the squared kernel S ( x, x ′ ) =

Original: E
Revised: E

Original: z ∼
Revised: z ∼

Original: ν
Revised: ν

Original: [ K ( x, z ) K ( x ′ , z )] can be written as S ( x, x ′ ) =
Revised: [ K ( x, z ) K ( x ′ , z )] can be written as [Equation]

Original: (cid:88)
Revised: [Equation]

Original: k λ
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: k ψ k ( x ) ψ k ( x ′ ) , and  the  kernel matrixS can be written as  Ψ ≤ m D 2 ≤ m Ψ ⊺ ≤ m + S >m .
Revised: ) , and for any m > 0 , the corresponding kernel matrix can be written as S = Ψ ≤ m D 2 ≤ m Ψ ⊺ ≤ m + S >m .

Original: Proof.
Revised: Proof.

Original: The statement simply follows from
Revised: The statement simply follows from

Original: [Equation]
Revised: [Equation]

Original: Lemma 6 (Corollary of Lemma 20 in Bartlett et al.
Revised: Lemma 6 (Corollary of Lemma 20 in Bartlett et al.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original:  where ( i ) applies Lemma 20 from Bartlett et al.
Revised: − 1 , where ( i ) applies Lemma 20 from Bartlett et al.

Original: Lemma 7 (Squared kernel tail) .
Revised: Lemma 7 (Squared kernel tail) .

Original: Let S >m be the kernel matrix of the  squared kernel S  .
Revised: For m > 0 , let S >m be the kernel matrix of the truncated squared kernel S >m = (cid:80) k>m λ 2 k ψ k ( x ) ψ k ( x ′ ) , and let K >m be the kernel matrix of the truncated original kernel K >m = (cid:80) k>m λ k ψ k ( x ) ψ k ( x ′ ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We show that for any vector v , v ⊺ S >m v ≤ λ m +1 v ⊺ K >m v , which is sufficient to prove the lemma.
Revised: We show that for any vector v , v ⊺ S >m v ≤ λ m +1 v ⊺ K >m v , which implies the claim.

Original: To do so, we define Ψ k ∈ R n × n with [ Ψ k ] i,j = ψ k ( x i )
Revised: To do so, we define Ψ k ∈ R n × n with [ Ψ k ] i,j = ψ k ( x i )

Original: ψ k ( x j )  .
Revised: ψ k ( x j ) for all k > m .

Original: Then we can write
Revised: Then we can write

Original: K >m =
Revised: K >m =

Original: (cid:80) [Equation] and [Equation]
Revised: (cid:80) k>m λ k Ψ k and S >m = (cid:80)

Original: k>m λ 2 k Ψ k .
Revised: k >m λ 2 k Ψ k .

Original: Since the eigenvalues are in decreasing order, we have λ k ≤ λ m +1 for any k > m , and thus
Revised: Since the eigenvalues are in decreasing order, we have λ k ≤ λ m +1 for any k > m , and thus

Original: [Equation]
Revised: [Equation]

Original: B C ONVOLUTIONAL KERNELS ON THE HYPERCUBE
Revised: B C ONVOLUTIONAL KERNELS ON THE HYPERCUBE

Original: First, Appendix B.1 provides a way to decompose general functions for features distributed uniformlyon the hypercube.
Revised: First, Appendix B.1 provides a way to decompose general functions for features distributed uniformly on the hypercube.

Original: Next, Appendix B.2 uses those results to characterize the eigenfunctions and eigenvalues of cyclic convolutional kernels.
Revised: Next, Appendix B.2 uses those results to characterize the eigenfunctions and eigenvalues of cyclic convolutional kernels.

Original: Finally, Appendices B.3 and B.4 apply this characterization toprove Lemmas 2 and 4, respectively.
Revised: Finally, Appendices B.3 and B.4 apply this characterization to prove Lemmas 2 and 4, respectively.

Original: B. G ENERAL FUNCTIONS ON THE HYPERCUBE
Revised: B. G ENERAL FUNCTIONS ON THE HYPERCUBE

Original: This subsection focuses on the main setting in our paper: the hypercube domain {− 1 , 1 } d togetherwith the uniform probability density, previously studied in Misiakiewicz & Mei (2021).
Revised: This subsection focuses on the main setting in our paper: the hypercube domain {− 1 , 1 } d together with the uniform probability distribution, previously studied in Misiakiewicz & Mei (2021).

Original: For any
Revised: For any

Original: S ⊆ { 1 , . . .
Revised: S ⊆ { 1 , . . .

Original: , d } , we define the polynomial
Revised: , d } , we define the polynomial

Revised: Y S

Original: [Equation]
Revised: [Equation]

Original:  of degree | S | , where [ x ] j is the j -th entry of x ∈ {− 1 , 1 } d .
Revised: x [Equation] [Equation] (cid:89) j ∈ S [ x ] j (13) of degree | S | , where [ x ] j is the j -th entry of x ∈ {− 1 , 1 } d .

Original: It is easy to see that {Y S } S ⊆{ 1 ,...,d } is set of orthonormal functions with respect to the inner product ⟨ f, g ⟩ {− 1 , 1 } d :=
Revised: It is easy to see that {Y S } S ⊆{ 1 ,...,d } is set of orthonormal functions with respect to the inner product ⟨ f, g ⟩ {− 1 , 1 } d :=

Original: E x ∼U ({− 1 , 1 } d ) [ f ( x ) g
Revised: E x ∼U (

Original: ( x )] .
Revised: [Equation]

Original: Those functions play a key role in the remainder of our proof; as it turns out, they are the eigenfunctions of the kernel in Section 3.1.
Revised: Those functions play a key role in the remainder of our proof; as it turns out, they are the eigenfunctions of the kernel in Section 3.1.

Original: Towards a formal statement, define the polynomials
Revised: Towards a formal statement, define the polynomials

Original: [Equation]
Revised: [Equation]

Original: where B ( l, d ) := |{ S ⊆ { 1 , . . .
Revised: where B ( l, d ) := |{ S [Equation] , d } | | S | [Equation]

Original: , d } | | S | = l 
Revised: l [Equation] (cid:18) d

Original: [Equation]
Revised: [Equation]

Original: Notice that G ( d ) l only depends on the (Euclidean) inner product of x and x ′ .
Revised: (cid:19) . [Equation] Note that G ( d ) l only depends on the (Euclidean) inner product of x and x ′ .

Original: Furthermore, {G ( d ) l } dl =0is a set of orthonormal polynomials with respect to the distribution of ⟨ x, x ′ ⟩ / √ d .
Revised: Furthermore, {G ( d ) l } dl = is a set of orthonormal polynomials with respect to the distribution of ⟨ x, x ′ ⟩ / √ d .

Original: The followinglemma shows how such polynomials form an eigenbasis for functions that only depend on the innerproduct between points in the unit hypercube.
Revised: The following lemma shows how such polynomials form an eigenbasis for functions that only depend on the inner product between points in the unit hypercube.

Original: Lemma 8 (Local kernel decomposition) .
Revised: Lemma 8 (Local kernel decomposition) .

Original: Let κ : R → R be any function and d ∈ N > 0 .
Revised: Let κ : R → R be any function and d ∈ N > 0 .

Original: Then, forany x, x ′ ∈ {− 1 , 1 } d , we can decompose κ ( ⟨ x, x ′ ⟩ /d ) as 
Revised: Then, for any x, x ′ ∈ {− 1 , 1 } d , we can decompose κ ( ⟨ x, x ′ ⟩ /d ) as [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that the decomposition only needs to hold at the evaluation of κ in the values that ⟨ x, x ′ ⟩ /d can take, that is, κ computed in {− 1 , − 1 + 2 /d, . . .
Revised: Note that the decomposition only needs to hold at the evaluation of κ in the values that ⟨ x, x ′ ⟩ /d can take, that is, κ computed in {− 1 , − 1 + 2 /d, . . .

Original: , − 2 /d + 1 , 1 } .
Revised: , − 2 /d + 1 , 1 } .

Original: Since that set has a cardinality d + 1 , we can write κ as a linear combination of d + 1 uncorrelated functions.
Revised: Since that set has a cardinality d +1 , we can write κ as a linear combination of d +1 uncorrelated functions.

Original: This is the case for {G ( d ) l } dl =0  with respect to the distribution of ⟨ x, x ′ ⟩ / √ d by construction. Hence,
Revised: In particular, {G ( d ) l } dl =0 is a set of such functions with respect to the distribution of ⟨ x, x ′ ⟩ / √ d , and hence

Original: [Equation]
Revised: [Equation]

Original: for some (unknown) coefficients c l .
Revised: for some (unknown) coefficients c l .

Original: Finally, the proof follows by expanding the definition of G ( d ) l in
Revised: Finally, the proof follows by expanding the definition of G ( d ) l in

Original: Equation (14) and choosing ξ ( d ) l = c l .
Revised: Equation (14) and choosing ξ ( d ) l = c l .

Original: While this lemma is very general, the proof of Theorem 1 requires a mild regularity assumption on κ .

Original: Assumption 1 (Regularity of κ ) .

Original: A function κ : [ − 1 , 1] → R in the setting of Section 3.1 satisfies the regularity assumption if there exist constants c 1 , c 2 > 0 and a series of constants { c l > 0 } l ≥ 0 such that, for any q ≥ c 1 , the decomposition κ ( ⟨ x, x ′ ⟩ /q )

Original: = (cid:80) ql =0 ξ ( q ) l 1 B ( l,q ) (cid:80)

Original: | S | =

Original: l Y S ( x ) Y S ( x ′ ) from

Original: Lemma 8 over inputs x, x ′ ∈ {− 1 , 1 } q satisfies

Original: [Equation]

Original: For sufficiently high-dimensional inputs x, x ′ , Assumption 1 ensures that the convolutional kernel

Original: K ( x, x ′ ) in Equation (1) is a valid kernel, that it possesses the so-called “implicit regularization”property , and enables learning of polynomials of any degree.

Original: This assumption resembles Assumption 1 from Misiakiewicz & Mei (2021).

Original: However, our version is more general: we enable learningof polynomials of any degree, while Assumption 1 from Misiakiewicz & Mei (2021) only coversthe degrees of interest.

Original: Still, our assumption holds for standard kernels such as the Gaussian kernel.

Original: Finally, our assumption does not depend on ℓ .

Original: Thus, a kernel satisfying our assumption will alsosatisfy Assumption 1 from Misiakiewicz & Mei (2021) for any choice of ℓ .

Original: B. C ONVOLUTIONAL KERNELS ON THE HYPERCUBE
Revised: B. C ONVOLUTIONAL KERNELS ON THE HYPERCUBE

Original: While the previous subsection considers general functions on the hypercube, we now focus onconvolutional kernels and their eigenvalues.
Revised: While the previous subsection considers general functions on the hypercube, we now focus on convolutional kernels and their eigenvalues.

Original: This yields the tools to prove Lemmas 2 and 4.
Revised: This yields the tools to prove Lemmas 2 and 4.

Original: In orderto characterize eigenvalues, we first introduce the following quantities, also used e.g. in Misiakiewicz& Mei (2021).
Revised: In order to characterize eigenvalues, we first follow existing literature such as Misiakiewicz & Mei (2021) and introduce useful quantities.

Original: Definition 1 (Diameter) .

Original: Let S ⊆ { 1 , . . .
Revised: Let S ⊆ { 1 , . . .

Original: , d } .
Revised: , d } .

Original: The diameter of S is
Revised: The diameter of S is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: 1 } for S ̸ = ∅ , and γ ( ∅ ) = 0 .
Revised: for S ̸ = ∅ , and γ ( ∅ ) = 0 .

Original: Furthermore, we define
Revised: Furthermore, we define

Original: [Equation]
Revised: [Equation]

Original: Intuitively, the diameter of S is the smallest number of contiguous feature indices that fully contain
Revised: Intuitively, the diameter of S is the smallest number of contiguous feature indices that fully contain

Original: S .
Revised: S .

Original: The following lemma yields an explicit formula for C ( l, q, d ) , that is, the number of sets of size lwith diameter at most q .
Revised: The following lemma yields an explicit formula for C ( l, q, d ) , that is, the number of sets of size l with diameter at most q .

Original: Lemma 9 (Number of overlapping sets) .
Revised: Lemma 9 (Number of overlapping sets) .

Original: Let l, q, d ∈ N withq ≥ 0 and l ≤ q < d/ 2 .
Revised: Let l, q, d ∈ N with l ≤ q < d/ 2 .

Original: Then,
Revised: Then,

Original: C ( l, q, d ) = (cid:40) d (cid:0) q − 1 l −(cid:1) l > 0 , l = 0 or q = 0 .
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Since the result holds trivially forq = 0 , l = 0, and l = 1 , we henceforth focus on l ≥ 2 .
Revised: Since the result holds trivially for l = 0 and l = 1 , we henceforth focus on l ≥ 2 .

Original: Let ˜C
Revised: Let ˜

Original:  ( l, γ, d ) be the number of subsets S ⊆ { 1 , . . .
Revised: C ( l, γ, d ) be the number of subsets S ⊆ { 1 , . . .

Original: , d } of cardinality | S | =
Revised: , d } of cardinality | S | =

Original: l with diameter exactly γ ( S ) =
Revised: l with diameter exactly γ ( S ) =

Original: γ .
Revised: γ .

Original: First, consider ˜ C (2 , γ, d ) .
Revised: First, consider ˜ C (2 , γ, d ) .

Original: For each set, we can choose the first element i from d different values, and the second as mod (( i − 1) ± ( γ − 1) , d ) + 1 .
Revised: For each set, we can choose the first element i from d different values, and the second as mod (( i − 1) ± ( γ − 1) , d ) + 1 .

Original: In this way, since q < d/ 2 , we count each set exactly twice.
Revised: In this way, since q < d/ 2 , we count each set exactly twice.

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Next, consider l > 2 .
Revised: Next, consider l > 2 .

Original: We can build all the possible sets by starting with one of the ˜ C (2 , γ, d ) =
Revised: We can build all the possible sets by starting with one of the ˜ C (2 , γ, d ) =

Original: d sets, and adding the remaining l − 2 elements from γ − 2 possible indices.
Revised: d sets, and adding the remaining l − 2 elements from γ − 2 possible indices.

Original: Hence, every fixed set of size 2 and diameter γ yields
Revised: Hence, every fixed set of size 2 and diameter γ yields

Original: [Equation]
Revised: [Equation]

Original: different sets of size l .
Revised: different sets of size l .

Original: Furthermore, by construction, every set of size l and diameter γ results from exactly on set of size 2 and diameter γ .
Revised: Furthermore, by construction, every set of size l and diameter γ results from exactly one set of size 2 and diameter γ .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: The result for l ≥ 2 then follows from summing ˜ C ( l, γ, d ) over all diameters γ ≤ q :
Revised: The result for l ≥ 2 then follows from summing ˜ C ( l, γ, d ) over all diameters γ ≤ q :

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from the hockey-stick identity.
Revised: where ( i ) follows from the hockey-stick identity.

Original: Now, we focus on cyclic convolutional kernels K as in Equation (1).
Revised: Now, we focus on cyclic convolutional kernels K as in Equation (1).

Original: First, we restate Proposition 1from Misiakiewicz & Mei (2021).
Revised: First, we restate Propositionfrom Misiakiewicz & Mei (2021).

Original: This proposition establishes that Y S for S ⊆ { 1 , . . .
Revised: This proposition establishes that Y S for S ⊆ { 1 , . . .

Original: , d } are indeedthe eigenfunctions of K , and yields closed-form eigenvalues λ S up to factors ξ ( q ) | S | that depend onthe inner nonlinearity κ .
Revised: , d } are indeed the eigenfunctions of K , and yields closed-form eigenvalues λ S up to factors ξ ( q ) | S | that depend on the inner nonlinearity κ .

Original: Then, Lemma 10 uses the regularity assumptions on κ to eliminate thedependency on ξ ( q ) | S | .
Revised: Next, Lemma 10 uses additional regularity assumptions on κ to eliminate the dependency on ξ ( q ) | S | .

Original: This characterization of the eigenvalues  enables the proof Lemmas 2 and 4.
Revised: This characterization of the eigenvalues then enables the proof Lemmas 2 and 4.

Original: Proposition 1 (Proposition 1 from Misiakiewicz & Mei (2021)) .
Revised: Proposition 1 (Proposition 1 from Misiakiewicz & Mei (2021)) .

Original: Let K be a cyclic convolutional kernel over the unit hypercube as defined in Equation (1) .
Revised: Let K be a cyclic convolutional kernel over the unit hypercube as defined in Equation (1) .

Original: Then we can write
Revised: Then, [Equation] k [Equation]

Revised: q ) ⟩

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: where ξ ( q ) | S | are the coefficients of the κ -decomposition (Equation (16) ) over {− 1 , 1 } | S | .
Revised: where ξ ( q ) | S | are the coefficients of the κ -decomposition (Equation (16) ) over {− 1 , 1 } q .

Original: Alternatively,
Revised: Alternatively,

Original: [Equation]
Revised: [Equation]

Original: where we order allsubsets S k ⊆ { 1 , . .
Revised: where we order all S k ⊆ { 1 , . .

Original: , d } with γ ( S k ) ≤ q such that λ S k ≥ λ S k +1 .
Revised: , d } with γ ( S k ) ≤ q such that λ S k ≥ λ S k +1 .

Original: In particular,λ S k = λ k .
Revised: In particular, λ S k = λ k .

Original: We refer to Proposition 1 from Misiakiewicz & Mei (2021) for a formal proof.
Revised: We refer to Proposition 1 from Misiakiewicz & Mei (2021) for a formal proof.

Original: Intuitively, the resultfollows from applying Lemma 8 for each subset S ⊆ { 1 , . . .
Revised: Intuitively, the result follows from applying Lemma 8 for each subset S ⊆ { 1 , . . .

Original: , d } .
Revised: , d } of contiguous elements.

Original: This is possible because crucially,any subset of q features is again distributed uniformly on the q -dimensional unit hypercube.
Revised: This is possible because crucially, any subset of q features is again distributed uniformly on the q -dimensional unit hypercube.

Original: Lastly,the factor q + 1 − γ ( S ) stems from the fact that each eigenfunction Y S appears as many times asthere are contiguous index sets of size γ ( S ) supported in a fixed contiguous index set of size q .
Revised: Lastly, the factor q + 1 − γ ( S ) stems from the fact that each eigenfunction Y S appears as many times as there are contiguous index sets of size γ ( S ) supported in a fixed contiguous index set of size q .

Original: Inother words, the term is the number of shifted instances of S supported in a contiguous subset of qfeatures.
Revised: In other words, the term is the number of shifted instances of S supported in a contiguous subset of q features.

Original: While Proposition 1 yields eigenfunctions and eigenvalues of cyclic convolutional kernels, it does notdescribe the factors ξ ( q ) | S | which depend on the inner nonlinearity κ . 
Revised: As mentioned before, Proposition 1 characterizes the eigenvalues of cyclic convolutional kernels K up to factors ξ ( q ) | S | that depend on the inner nonlinearity κ . To avoid the additional factors, we require the following regularity assumptions: Assumption 1 (Regularity) . Let T :=

Original: The final lemma of this sectionis a corollary that characterizes the eigenvalues solely in terms of | S | ,  and further shows that, for dlarge enough, the eigenvalues decay as | S | grows.
Revised: (cid:108) 4 + 4 ℓβ (cid:109) . A cyclic convolutional kernel K ( x, x ′ ) = [Equation] from the setting of Section 3.1 with inner function κ satisfies the regularity assumption if there exist constants c ≥ T , c ′ , c ′′ > 0 and a series of constants { c l > 0 } Tl =0 such that, for any q ≥ c , the decomposition

Revised: [Equation]

Revised: from Proposition 1 over inputs x, x ′ ∈ {− 1 , 1 } d satisfies

Revised: [Equation]

Revised: For sufficiently high-dimensional inputs x, x ′ , Equations (18) and (19) ensure that the convolutional kernel K ( x, x ′ ) in Equation (1) is a valid kernel, and that it can learn polynomials of degree up to T .

Revised: Indeed, if ξ ( q ) l = 0 for some l , then there are no polynomials of degree l among the eigenfunctions of K .

Revised: Furthermore, Equations (18), (20) and (21) guarantee that the eigenvalue tail is sufficiently bounded.

Revised: This allows us to bound ∥ K >m ∥ and µ min ( K >m ) in Appendix C.

Revised: Our assumption resembles Assumption 1 by Misiakiewicz & Mei (2021):

Revised: For one, Equations (20) and (21) are equivalent to Equations 43 and 44 in Misiakiewicz & Mei (2021).

Revised: Furthermore,

Revised: Equation (18) above is a slightly stronger version of Equation 42, where strengthening is necessary due to the non-asymptotic nature of our results.

Revised: We still argue that many standard κ , for example, the Gaussian kernel, satisfy Assumption 1 with our convolution kernel K .

Revised: Because such κ satisfy Assumption 1 in Misiakiewicz & Mei (2021), we only need to check that they additionally satisfy our Equation (18).

Revised: If κ is a smooth function, we have ξ ( q ) l = κ ( l ) (0)

Revised: + o (1) for all l ≤ T , where κ ( l ) is the l -th derivative of κ .

Revised: In particular, all derivatives of the exponential function at 0 are strictly positive, implying Equation (18) for the Gaussian kernel if d is large enough.

Revised: The final lemma of this section is a corollary that characterizes the eigenvalues solely in terms of | S | , and further shows that, for d large enough, the eigenvalues decay as | S | grows.

Original: Lemma 10 (Corollary of Proposition 1) .
Revised: Lemma 10 (Corollary of Proposition 1) .

Original: Consider a cyclic convolutional kernel from Propositionwith q ∈ Θ( d β ) for some β ∈ (0 , 1) .Assume that κ satisfies Assumption 1 with c 1 ≤ q .
Revised: Consider a cyclic convolutional kernel as in Propositionthat satisfies Assumption 1 with q ∈ Θ( d β ) for some β ∈ (0 , 1) .

Original: Then, for any S ⊆ { 1 , . ..
Revised: Then, for any S ⊆ { 1 , . .

Original: , d } such that γ ( S ) ≤ q  , the eigenvalue λ S corresponding to the eigenfunction Y S ( x ) satisfies
Revised: , d } such that γ ( S ) ≤ q and | S | < T , the eigenvalue λ S corresponding to the eigenfunction Y S ( x ) satisfies

Original: [Equation]
Revised: [Equation]

Original: where ≲ hides positive constants that depend on | S | .
Revised: and

Original: Furthermore, there exists an S ⊆ { 1 , . . .
Revised: [Equation]

Original: , d }such that γ ( S ) ≤ q for which λ S ∈ Θ (cid:0)
Revised: O

Original: [Equation] 
Revised: [Equation] Furthermore,

Original: [Equation]
Revised: [Equation]

Original: 1+ β | S | (cid:1) .

Original: Proof.
Revised: Proof.

Original:  Let  l := | S | and r :=
Revised: Without loss of generality, assume d is large enough such that d > q/ 2 ≥ c/ 2 , where c is a constant from Assumption 1. Let S ⊆ { 1 , . . . , d } with γ ( S ) ≤ q be arbitrary and define l := | S | and r :=

Original: q + 1 − γ ( S ) .
Revised: q + 1 − γ ( S ) .

Original: Since l ≤ γ ( S ) ≤ q , we have
Revised: Since l ≤ γ ( S ) ≤ q , we have

Original: [Equation]
Revised: [Equation]

Original: Furthermore, because B ( l, q ) = (cid:0) ql (cid:1) ,  the  classical bound on the binomial coefficient yields
Revised: Furthermore, since B ( l, q ) = (cid:0) ql (cid:1) , we use the following classical bound on the binomial coefficient throughout the proof:

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: For the first part of the lemma, assume | S | < T . Then, using Assumption 1, we have

Original: [Equation]
Revised: [Equation]

Original: where Assumption 1 yields in ( i ) that ξ ( q ) l ≤ c 2 , and in ( ii ) that ξ ( q ) l ≥ c l .
Revised: for some positive constants c l, 1 , c l, 2 that depend on l , c T, 1 := max l ∈{ 0 ,...,T − 1 } c l, 1 , and c T, 2 := min l ∈{ 0 ,...,T − 1 } c l, 2 . Step ( i ) follows from the upper bound in Equation (21) with non-negativity in Equations (18) and (19), and ( ii ) follows from the lower bound in Equation (18).

Original: Since c l, 1 and c l, 2 only 511 depend on l , this concludes the first part of the proof.
Revised: Since c T, 1 and c T,do not depend on l , this concludes the first part of the proof.

Original: For the second part, any S with γ ( S ) = q yields r = 1 .
Revised: For the second part of the proof, we consider two cases depending on whether | S | ∈ [ T, q − T ] or | S | > q − T .

Original: Hence, the previous upper bound becomes
Revised: Hence, first assume T ≤ | S | ≤ q − T . Then,

Original: [Equation]
Revised: [Equation]

Original: where c l, 3 only depends on l .
Revised: where ( i ) follows from Proposition 1. In step ( ii ) , we use that (cid:0) q | S | (cid:1) is minimized when | S | is has the largest difference to q/ 2 . Lastly, step ( iii ) applies the upper bound from Equation (21) together with non-negativity of ξ | S | in Assumption 1, and the classical bound on the binomial coefficient. Now assume | S | > q − T .

Original: This yields the second statement, since dq l ∈ Θ (cid:0) d 1+ β | S | (cid:1) .
Revised: Then, [Equation] where ( i ) follows from Proposition 1, ( ii ) from the fact that (cid:0) nn

Revised: [Equation]

Revised: (cid:1) = (cid:0) nk (cid:1) , ( iii ) from the classical bound for the binomial coefficient, ( iv ) from Equation (20) in Assumption 1, and ( v ) from the fact that q − | S | < T in the current case.

Original: Finally, using Proposition 1 and Lemma 10, we are able to prove Lemmas 2 and 4.
Revised: Combining Equations (24) and (25) from the two cases finally yields [Equation] which concludes the second part of the proof.

Original: B. P ROOF OF L EMMA2
Revised: B. P ROOF OF L EMMA

Original: First, notice that f ⋆ = Y S ∗ for S ∗ = { 1 , . . .
Revised: First, note that f ⋆ = Y S ∗ for S ∗ = { 1 , . . .

Original: , L ∗ } .
Revised: , L ∗ } .

Original: Since | S ∗ | = γ ( S ∗ ) =
Revised: Since | S ∗ | = γ ( S ∗ ) =

Original: L ∗ , applying Propositionyields
Revised: L ∗ , Proposition 1 yields

Original: [Equation]
Revised: [Equation]

Original: wherein ( i ) we used assumption Assumption 1  to getthat ξ ( q ) L ∗ ∈ Θ(1) , andin ( ii ) we used the fact that λm ∈ o (cid:16) 1 dq L ∗−
Revised: where ( i ) uses Equations (18) and (21) in Assumption 1 for L ∗ ≤ T and d large enough to get ξ ( q ) L ∗ ∈ Θ(1) , and ( ii ) uses λ

Original: [Equation] (cid:17) .
Revised: m ∈ o (cid:16) 1 dq L ∗−(cid:17) .

Original: This implies that for d sufficiently large, λ S ∗ > λ m .
Revised: Hence, for d sufficiently large, λ S ∗ > λ m .

Original: Thus, since the eigenvalues are sorted, f ⋆ is in the span of the first m eigenfunctions.
Revised: Since the eigenvalues are in decreasing order, this implies that f ⋆ is in the span of the first m eigenfunctions.

Original: Finally,
Revised: This further yields

Original: [Equation]
Revised: [Equation]

Original: sincein ( i ) , the entry of a corresponding to Y S ∗ is 1 , and all others are 0 .
Revised: since the entry of a corresponding to Y S ∗ is 1 while all others are 0 .

Original: B. P ROOF OF L EMMA4
Revised: B. P ROOF OF L EMMA

Original: Before proving Lemma 4, we introduce the following quantity:
Revised: Before proving Lemma 4, we introduce the following quantity:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, L corresponds to the degree of the largest polynomial that a cyclic convolutional kernel asdefined in Equation (1) can learn.
Revised: Intuitively, L corresponds to the degree of the largest polynomial that a cyclic convolutional kernel as defined in Equation (1) can learn.

Original: This quantity plays a key role throughout the proof of Lemma 4,and Lemma 3 later.
Revised: This quantity plays a key role throughout the proof of Lemma 4, and Lemma 3 later.

Original: All eigenfunctions of degree at most L will have an index smaller than m , whileall eigenfunctions of degree at least L+ 2 will have an index larger than m .The proof of Lemma 4shows this statement formally.
Revised: Finally, note that δ as defined in Theorem 1 can be written as δ = ℓ − ℓ λ − 1 β − L .

Original: Proof of Lemma 4.
Revised: Proof of Lemma 4.

Original: First, we use Proposition 1 to write the cyclic convolutional kernel as
Revised: First, we use Proposition 1 to write the cyclic convolutional kernel as

Original: [Equation]
Revised: [Equation]

Original: where  S i are ordered such that λ S k +1 ≤ λ S k .
Revised: where the λ S k are ordered such that λ S k +1 ≤ λ S k .

Original: Here we need toshow that we can pick  m ∈ N such that nλS m ∈ Θ (max { λ, 1 } ), and m ∈ Θ 
Revised: For the first part of the proof, we need to pick an m ∈ N such that nλ m ∈ Θ (max { λ, 1 } ) and m ∈ Θ (cid:16)

Original: [Equation]
Revised: [Equation]

Original: nq − δ max { λ, 1 } (cid:17) .
Revised: { λ, [Equation] (cid:17) .

Original: Since n ∈ Θ( d ℓ ) and max { λ, 1 } ∈ Θ( d ℓ λ ) , it suffices to show that there exists a m such that
Revised: We will equivalently choose m ∈ Θ( dq L ) with λ m ∈ Θ (cid:16) 1 d · q L + δ (cid:17) ; since n ∈ Θ( d ℓ ) and max { λ, 1 } ∈ Θ( d ℓ λ ) , we have

Original: [Equation]
Revised: [Equation]

Original: By definition of λ m = λ S m , this corresponds to showing that there exists S m ⊆ { 1 , . . .
Revised: The remainder of the proof proceeds in five steps: we first construct a candidate S m ⊆ { 1 , . . .

Original: , d } withγ ( S m ) ≤ q such that the  eigenvalue corresponding to Y S m is λ S m  ∈ Θ (cid:16) 
Revised: , d } with γ ( S m ) ≤ q , show that the rate of the eigenvalue corresponding to Y S m satisfies λ S m = λ m ∈ Θ (cid:16) 1 d · q L + δ (cid:17) , show that the rate of m ∈ Θ( dq L ) , establish Θ [Equation]

Original: 1 d · q L + δ (cid:17) , and  m ∈ Θ (cid:0) dq L (cid:1) .
Revised: O [Equation] n · q − ¯ δ ) , and finally show that λ m ∈ o (cid:16) 1 d · q L ∗−(cid:17) for appropriate L ∗ .

Original: Choice of m.
Revised: Construction of m

Original: The way we find such S m is different in two cases:
Revised: We consider two different S m depending on δ :

Original: [Equation]
Revised: [Equation]

Original: First, in both cases, since δ < 1 , S m is well-defined for d — and hence q ∈ Θ( d β ) — sufficiently large. Assume that this is the case. Then, | S m | =
Revised: For d —and hence q ∈ Θ( d β ) —large enough, S m is well-defined, | S m | =

Original: L + 1 in both cases and 
Revised: L + 1 , and the diameter is

Original: [Equation]
Revised: [Equation]

Original: by definition.
Revised: For the rest of the proof, assume that d is sufficiently large.

Original: Rate of λ S m
Revised: Rate of λ S m

Original: Using Proposition 1, since we know that | S m | =
Revised: Using Proposition 1 and | S m | =

Original: L + 1 , we can write  the eigenvalue corresponding to Y S m as
Revised: L + 1 , we can write [Equation] First, we show that the numerator is in Θ

Original: [Equation]
Revised: [Equation]

Original: Numerator: Now we show that in both cases, the numerator is in Θ (cid:0) q 1 − δ (cid:1) .
Revised: for both definitions of S m .

Original: In the case where δ ∈ (0 , 1) :
Revised: In the case where δ ∈ (0 , 1) , we have

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from δ < 1 and q sufficiently large.
Revised: where ( i ) follows from δ < 1 and q sufficiently large.

Original: In the case where δ = 0 :
Revised: In the case where δ = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Thus,  the numerator is in Θ( q ) = Θ( q 1 − δ ), since in this setting δ = 0 .
Revised: Thus, since δ = 0 in this case, the numerator is in Θ( q ) = Θ( q 1 − δ ) .

Original: Denominator: Notice that the denominator does not depend on δ ,thus we use the same technique for both δ = 0 , δ ∈ (0 , 1) .
Revised: As the denominator does not depend on δ , we use the same technique for both δ = 0 and δ ∈ (0 , 1) .

Original: Indeed, we use the classical bound on B ( L + 1 , q ) = 
Revised: The classical bound on B ( L + 1 , q ) = (cid:0) qL + (cid:1) yields q L +1 ≲ (cid:18) [Equation] ≲ q

Original: [Equation]
Revised: [Equation]

Original: Rate of λ S m : Using both parts, together with ξ ( q ) L +1 ∈ Θ(1) by Assumption 1, we can conclude that
Revised: + [Equation] Therefore, d B ( L + 1 , q ) ∈ Θ (cid:0) dq L + (cid:1) . Finally, since L + 1 ≤ T = ⌈ 4 + 4 ℓ/β ⌉ , we have ξ ( q ) L +1 ∈ Θ(1) by Equations (18) and (21) in

Original:  S m satisfies
Revised: Assumption 1 for d sufficiently large. Combining all results then yields the desired rate of λ S m as follows:

Original: [Equation]
Revised: [Equation]

Original: Hence, thanks to Equation (19), we have shown that there exists an m for which nλ m ∈Θ(max { λ, 1 } )  .
Revised: Rate of m To establish m ∈ Θ( dq L ) , we bound m individually from above and below. Upper bound: Since the eigenvalues are in decreasing order, we can bound m from above by counting how many eigenvalues are larger than λ m .

Original: We now need to show that for this choice of m  , we also have that  m ∈Θ
Revised: To do so, we use | S m | = L + 1 , and show that for d sufficiently large, all S k with | S k | > L + 1 correspond to eigenvalues λ S k < λ S m . We first decompose

Original: [Equation]
Revised: [Equation]

Original: =
Revised: 

Original: [Equation]
Revised: [Equation]

Original: Rate of m . Upper bound: From the construction of S m in Equation (21) it directly follows that| S m | = L + 1 .
Revised: For M 1 , let k with L + 1 < | S k | < T be arbitrary.

Original: Furthermore, Lemma 10 shows that, for d sufficiently large, all S k with | S k | > L + 1 correspond to eigenvalues λ S k < λ S m .Thus, m is at most the number of polynomials with degree no larger than
Revised: Then, [Equation] where we apply Equation (22) from Lemma 10 in ( i ) , and use Equation (28) with δ < 1 in ( ii ) .

Revised: This implies M 1 ∈ o ( λ S m ) .

Revised: For M 2 , we directly get

Revised: [Equation]

Revised: where we apply Equation (23) from Lemma 10 in ( i ) , ( ii ) follows from L + 2 ≤ T , and step ( iii ) uses Equation (28) with δ < 1 .

Revised: Combined, we have max k : | S k | >

Original: L  + 1 :
Revised: L +1 ,γ ( S k ) ≤ q λ S k = max { M 1 , M 2 } ∈ o ( λ S m ) . Thus, for d sufficiently large and | S k | > L + 1 , we have λ S k < λ S m . For this reason, m is at most the number of eigenfunctions with degree no larger than L + 1 :

Original: [Equation]
Revised: [Equation]

Original: where ( i ) uses Lemma 9 for d large enough with 
Revised: where ( i ) uses Lemma 9 for d large enough with [Equation] ∈ Θ [Equation]

Revised: q l

Original: [Equation]
Revised: [Equation]

Original: Lower bound:
Revised: Lower bound:

Original: From the definition of S m in Equation (21), we havethat γ ( S m ) ≥ ⌊ q/ 2 ⌋ .
Revised: By construction of S m in Equation (27), we have γ ( S m ) ≥ ⌊ q/ 2 ⌋ .

Original: This, combined with Proposition 1, yield thatall the indexes of  polynomials of degree L + 1, but diameter ≤ ⌊ q/ 2 ⌋ − 1 appear before m .
Revised: This, combined with Proposition 1, implies that the indices of all polynomials with degree L + 1 but diameter at most ⌊ q/ 2 ⌋ − 1 are smaller than m .

Original: Hence, thanks to Lemma 9,
Revised: Hence, for large enough d , Lemma 9 yields the following lower bound:

Original: [Equation]
Revised: [Equation]

Original:  ∈ Ω( dq L ) , concluding the proof of the lower bound. Thus, m ∈ Θ( d L ) , and, thanks to Equation (20), m ∈Θ
Revised: The upper and lower bound together then imply m ∈ Θ( dq L ) . This concludes the existence of an m ∈ N such that λ m and m exhibit the desired rates. Rate of m with respect to n We can write n as

Original: [Equation]
Revised: [Equation]

Original: (cid:17) . This concludes the proof of the existence.
Revised: Combining this with L ≤ (cid:106)

Original: To conclude the proof we only need toshow additional formulas for the rates. Rate of m with respect to n . Finally, we only need to show that Θ( dq L ) ⊆ O ( nq − δ ) .
Revised: [Equation]

Original: For this wenotice that, since ¯ L := (cid:106) ℓ − 1 β (cid:107)≥ L , we have that m ∈
Revised: (cid:107) , we directly get Θ( dq L ) ⊆

Original: O ( dq ¯ L ) .
Revised: O ( dq ⌊ ℓ

Original: Then, we can write n as

Original: [Equation]
Revised: [Equation]

Original: and hence m ∈ O ( dq ¯ L ) =
Revised: [Equation]

Original: O ( nq − ¯ δ ) .
Revised: nq − ¯ δ ) .

Original: Rate of λ m with respect to L ∗.
Revised: Rate of λ m for appropriate L ∗

Original: By assumption, nλ m ∈ Θ(max { λ, 1 } ) . Therefore,
Revised: Since nλ m ∈ Θ(max { λ, 1 } ) , we have

Original: [Equation]
Revised: [Equation]

Original: where ( i ) uses the identity ℓ = 1 + ℓ λ + β ( L + δ ) . 
Revised: where ( i ) uses the identity ℓ = 1 + ℓ λ + β ( L + δ ) . Assume now L ∗ ≤ (cid:108) ℓ − [Equation] λ − 1 β (cid:109) .

Original: For the remainder, we need to consider two cases depending on δ .
Revised: For the remainder, we need to consider two cases depending on δ .

Original: If δ > 0 , then L ∗ ≤ L + 1 , and we have
Revised: If δ > 0 , then L ∗ ≤ L + 1 , and we have

Original: [Equation]
Revised: [Equation]

Original: ∈

Original: O

Original: [Equation]

Original: [Equation]

Original: If δ = 0 , then L ∗ ≤ L , and we have
Revised: If δ = 0 , then L ∗ ≤ L , and we have

Original: [Equation]
Revised: [Equation]

Original: In both cases, λ m ∈
Revised: In both cases, λ m ∈

Original: o
Revised: o

Original: [Equation]
Revised: [Equation]

Original: , concluding the proof.
Revised: , concluding the proof.

Original: C M ATRIX CONCENTRATION
Revised: C M ATRIX CONCENTRATION

Original: This section focuses on random matrix theory.
Revised: This section considers the random matrix theory part of our main result.

Original: First, Appendix C.1 focuses on the large eigenvaluesof our kernel, and proves Lemma 1.
Revised: First, Appendix C.1 focuses on the large eigenvalues of our kernel, and proves Lemma 1.

Original: Next, Appendix C.2 focuses on the tail of the eigenvalues,culminating in the proof of Lemma [-
-]
Revised: Next, Appendices C.2 and C.3 focus on the tail of the eigenvalues, culminating in the proof of Lemma [-
-]3.

Original: Lastly, Appendix C.4  establishes some technical tools that weuse throughout the proofs.
Revised: Lastly, Appendices C.4 and C. establishes some technical tools that we use throughout the proofs.

Original: C. P ROOF OF L EMMA1
Revised: C. P ROOF OF L EMMA

Original: In this proof, we show that the matrix Ψ ⊺
Revised: In this proof, we show that the matrix Ψ ⊺

Original: ≤ m Ψ ≤ m /n concentrates around the identity matrix underappropriate assumptions, thereby establishing Equation (5).
Revised: ≤ m Ψ ≤ m /n concentrates around the identity matrix for all m ∈ O ( nd − β ¯ δ ) , thereby establishing Equation (5).

Original: Letus define ˆ mto be the largest msuch that m ∈ O ( nd −β ¯ δ ) .
Revised: Let ˆ m be the largest m ∈ O ( nq − ¯ δ ) .

Original: The proof consists of applying Theorem 5.44 from Vershynin (2012) to the matrix Ψ ≤ ˆ m . Then a simple argument extend it to anyother choice of m .
Revised: The proof consists of applying Theorem 5.44 from Vershynin (2012) to the matrix Ψ ≤ ˆ m , and extending the result to all suitable choices of m simultaneously.

Original: Bound for ˆ m  . 
Revised: More precisely, let ˜ c be the implicit constant of the O ( nq − ¯ δ ) -notation, and define ˆ m to be the largest m ∈ N with m ≤ ˜ c · nq − ¯ δ . Note that ˆ m exists, because d is large enough and fixed.

Original:  To apply Theorem 5.44 from Vershynin (2012) we need to verify the theorem’s conditions on the rows of Ψ ≤ ˆ m .
Revised: Bound for ˆ m To apply Theorem 5.44 from Vershynin (2012), we need to verify the theorem’s conditions on the rows of Ψ ≤ ˆ m .

Original: In particular, we show that the rows are independent, have a common second moment matrix, and that their norm is bounded.
Revised: In particular, we show that the rows are independent, have a common second moment matrix, and that their norm is bounded.

Original: Let [ Ψ ≤ ˆ m ] i, : indicate the i -th row of Ψ ≤ ˆ m ∈ R n × ˆ m .
Revised: Let [ Ψ ≤ ˆ m ] i, : indicate the i -th row of Ψ ≤ ˆ m ∈ R n × ˆ m .

Original: We may write each row entry-wise as
Revised: We may write each row entry-wise as

Original: [Equation]
Revised: [Equation]

Original: First, the rows of Ψ ≤ ˆ m are independent since each row depends on a different x i , and we assumedata to be i.i.d..
Revised: First, the rows of Ψ ≤ ˆ m are independent, since each row depends on a different x i , and we assume the data to be i.i.d..

Original: Second, since the eigenfunctions are orthonormal w.r.t.
Revised: Second, since the eigenfunctions are orthonormal w.r.t.

Original: the data distribution, the second moment ofthe rows is E
Revised: the data distribution, the second moment of the rows is E

Original: [Equation]
Revised: [Equation]

Original: for all rows.
Revised: for all rows i ∈ { 1 , . . . , n } .

Original: Third, to show that each row has  bounded norm, we use the fact that the eigenfunctions Y k in
Revised: Third, to show that each row has a bounded norm, we use the fact that the eigenfunctions Y k in

Original: Equation (13) over {− 1 , 1 } d satisfy Y k
Revised: Equation (13) over {− 1 , 1 } d satisfy Y k

Original: ( x i ) 2 = 1 for all k .
Revised: ( x i ) 2 = 1 for all k .

Original: Thus, the norm of each row is
Revised: Thus, the norm of each row is

Original: [Equation]
Revised: [Equation]

Original: We can now apply Theorem 5.44 from Vershynin (2012).
Revised: We can now apply Theorem 5.44 from Vershynin (2012).

Original: For any t ≥ 0 , this yields the following inequality with probability 1 − ˆ m exp {− ct 2 } :
Revised: For any t ≥ 0 , this yields the following inequality with probability 1 − ˆ m exp {− ct 2 } , where c is an absolute constant:

Original: [Equation]
Revised: [Equation]

Original: where ∆ = t (cid:114) ˆ m

Original: [Equation]
Revised: [Equation]

Original: Choosing t =(cid:112) n ˆ m yields∆ = 1 / 2 < 1 , and hence max
Revised: , where ∆ = t (cid:114) ˆ m n . The choice t =(cid:112) n ˆ m yields max

Original: [Equation]
Revised: [Equation]

Original: = [Equation] Lastly, for d large enough, the  error probability is
Revised: = 1 / 2 , and the following error probability for large enough d :

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from the fact that m ∈ O ( n β ¯ δ ) .
Revised: where ( i ) follows from ˆ m ∈ O ( nq − ¯ δ ) .

Original: Bound for any m <ˆ m .
Revised: Bound for any m <

Original: For this it suffices to notice thatthe matrix Ψ ⊺ ≤ m Ψ ≤ m is a sub-matrix of Ψ ⊺ ≤ ˆ m Ψ ≤ ˆ m .
Revised: ˆ m Note that Ψ ⊺ ≤ m Ψ ≤ m is a submatrix of Ψ ⊺ ≤ ˆ m Ψ ≤ ˆ m .

Original: Hence, also Ψ ⊺ ≤ m Ψ ≤ m n − I m is  a sub-matrix of Ψ ⊺ ≤ ˆ m Ψ ≤ ˆ m n − I ˆ m .
Revised: Thus, Ψ ⊺ ≤ m Ψ ≤ m n − I m is also a submatrix of Ψ ⊺ ≤ ˆ m Ψ ≤ ˆ m n − I ˆ m .

Original: Thus, uniformly over all m ≤ ˆ m ,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original:  Ψ [Equation] m n − I m (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)
Revised: (cid:13)(cid:13) Ψ ⊺ ≤ m Ψ ≤ m n − I m (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)

Original: Ψ ⊺ ≤ ˆ m Ψ ≤ ˆ m n − I ˆ m (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) ≤ 1 / 2 w.p. ≥ 1 − cd − β ¯ δ , concluding the proof.
Revised: Ψ ⊺ ≤ ˆ m Ψ ≤ ˆ m n − I ˆ m (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) ≤ 1 /with probability at least 1 − cd − β ¯ δ uniformly over all m ≤ ˆ m .

Original: C. C ONCENTRATION OF THE TERMS AFTER mAND PROOF OF L EMMA 3
Revised: C. F URTHER DECOMPOSITION OF THE TERMS AFTER m

Original: In the following, we focus on the concentration of the smallest and largest eigenvalue of the kernelmatrix K >m to prove Lemma 3.
Revised: In this section, we focus on the concentration of the smallest and largest eigenvalue of the kernel matrix K >m to prove Lemma 3.

Original: However, this proof is involved, and requiressome additional tools.
Revised: However, this proof is involved, and requires additional tools.

Original: In particular, we further decompose K >m into two kernels K 1 and K 2 .
Revised: In particular, we further decompose K >m into two kernels K 1 and K 2 .

Original: We first introduce the twoadditional kernels, and then show in Lemma 11 that K >m = K 1 + K 2 .
Revised: In the following, we consider the setting of Theorem 1 with a convolutional kernel K that satisfies Assumption 1. We define the additional notation

Original: Definition 2 ( 1 - 2 decomposition) .
Revised: [Equation] Intuitively, L is the maximum polynomial degree that K can learn with regularization, and ¯ L is the analogue without regularization. Finally, note that δ and ¯ δ as defined in Theorem 1 can be written as δ [Equation]

Original: Consider the setting of Theorem 1, and decompose the convolutional kernel K via Proposition 1 as
Revised: and [Equation] We now introduce the two additional kernels, and then show in Lemma 11 that K >m = K 1 + K 2 . First, applying Proposition 1 to K yields

Original: [Equation]
Revised: [Equation]

Original: where { S k } k> 0 is a sequence of all subsets S k ⊆ { 1 , . .
Revised: where { S k } k> 0 is a sequence of all subsets S k ⊆ { 1 , . .

Original: , d } with γ ( S k ) ≤ q , ordered such thatλ S k ≥ λ S k +1 .
Revised: , d } with γ ( S k ) ≤ q , ordered such that λ S k ≥ λ S k +1 .

Original: Lastly, let m ∈ N be such that nλ S m ∈ Θ(max { λ, 1 } ) . We define the index sets
Revised: Next, let m ∈ N be such that nλ S m ∈ Θ(max { λ, 1 } ) , and define the index sets

Original: [Equation]
Revised: [Equation]

Original: and the corresponding kernels
Revised: Those sets induce the following kernels:

Original: [Equation]
Revised: [Equation]

Original: where S 1 and S 2 are the squared kernels corresponding to K 1 and K 2 , respectively.
Revised: where S 1 and S 2 are the squared kernels corresponding to K 1 and K 2 , respectively.

Original: Furthermore, as in the original kernel decomposition
Revised: The empirical kernel matrices K 1 , K 2 , S 1 , S 2 ∈ R n × n are [ K 1 ] i,j = K 1 ( x i , x j ) , [ K [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: we define the matrices
Revised: = K

Original: [Equation]
Revised: [Equation]

Original: and the empirical kernel matrices K 1 , K 2 ∈ R n × n with [ K 1 ] i,j = K 1 ( x i , x j ) and [ K 2 ] i,j = K 2 ( x i , xj ) . We further define ¯ m as the largest eigenvalue corresponding to an eigenfunction Y S of degree | S | ≥ ¯ L + 2 , that is,
Revised: x [Equation] , x

Original: [Equation] 
Revised: [Equation] [ S 1 ] i,j = S 1 ( x i , x j ) , and [ S 2 ] i,j = S 2 ( x i , x j ) .

Original: Finally, we define 
Revised: Furthermore, as in the original kernel decomposition, we define the matrices

Original: [Equation]
Revised: [Equation]

Original: with the corresponding kernel matrix Q ( d,q ) l .
Revised: where { k j } |I 1 | j =1 is a sequence of all indices in I 1 ordered such that λ S kj ≥ λ S kj +1 .

Original: Intuitively,the quantities Ψ 1 , D 1 are the equivalent to Ψ ≤ m , D ≤ m in the original decomposition K = 
Revised: Intuitively, Ψ 1 , D 1 are the analogue to Ψ ≤ m , D ≤ m in the original decomposition K = K ≤ m + K >m .

Revised: Lastly, we define ¯ m as the largest eigenvalue corresponding to an eigenfunction Y S of degree | S | ≥ ¯ L + 2 , that is,

Original: K ≤ m + K >m .
Revised: [Equation] min

Original: The following lemma establishes that K 1 and K 2 indeed constitute a decompositionof K >m .
Revised: I 2 . Using the previous definitions, the following lemma establishes that K 1 and K 2 indeed constitute a decomposition of K >m .

Original: Lemma 11 ( 1 - 2 decomposition) .
Revised: Lemma 11 ( 1 - 2 decomposition) .

Original: In the same setting as Definition 2, for d sufficiently large, we have
Revised: For d sufficiently large, we have

Original: K >m ( x, x ′ )
Revised: K >m ( x, x ′ )

Original: =
Revised: =

Original: K 1 ( x, x ′ )
Revised: K 1 ( x, x ′ )

Original: + K 2 ( x, x ′ )
Revised: + K 2 ( x, x ′ )

Original: and S >m ( x, x ′ )
Revised: and S >m ( x, x ′ )

Original: =
Revised: =

Original: S 1 ( x, x ′ )
Revised: S 1 ( x, x ′ )

Original: + S 2 ( x, x ′ ) .
Revised: + S 2 ( x, x ′ ) .

Original: Proof.
Revised: Proof.

Original: For the decomposition of K >m , we have to show that exactly the eigenfunctions with indexlarger than m appear in either K 1 or K 2 , that is, I 1 ∪ I 2 = { k > m } , and that no eigenfunctionappears in both K 1 or K 2 , that is, I 1 ∩ I 2 = ∅ .
Revised: For the decomposition of K >m , we have to show that exactly the eigenfunctions with index larger than m appear in either K 1 or K 2 , that is, I 1 ∪ I 2 = { k > m } , and that no eigenfunction appears in both K 1 or K 2 , that is, I 1 ∩ I 2 = ∅ .

Original: Furthermore, since we can write S >m ( x, x ′ )=(
Revised: Furthermore, since we can write S >m ( x, x ′ )

Original: cid:80) k>m λ
Revised: = (cid:80) k

Original:  2 S k Y S
Revised: >m λ 2 S k Y S

Original: k
Revised: k

Original: ( x ) Y S k ( x ′ ) by Lemma 5, the same argument also yields the 1 - 2 decomposition of S >m .
Revised: ( x ) Y S k ( x ′ ) by Lemma 5, the same argument implies the 1 - 2 decomposition of S >m .

Original: First, from the definition of I 

Revised: First, from the definition of I

Original: and I 2 , it follows directly that I 1 ∩ I 2 = ∅ , that I 1 ∪ I 2 ⊇ { k > m } ,and that I 1 ⊆ { k > m } .
Revised: and I 2 , it follows directly that I 1 ∩ I 2 = ∅ , that I 1 ∪ I 2 ⊇ { k > m } , and that I 1 ⊆ { k > m } .

Original: Hence, to conclude the proof, we only need to show that I 2 ⊆ { k > m } .
Revised: Hence, to conclude the proof, we only need to show that I 2 ⊆ { k > m } .

Original: Since the indices are sorted in decreasing order, we equivalently show that, for d sufficiently large, alleigenvalues λ S k with k ∈ I 2 are smaller than λ S m ∈ Θ(max { λ, 1 } /n ) .
Revised: Since the eigenvalues are sorted in decreasing order, we equivalently show that, for d sufficiently large, all eigenvalues λ S k with k ∈ I 2 are smaller than λ S m ∈ Θ(max { λ, 1 } /n ) .

Original: More precisely, we show that  nλ S k ∈ o ( nλ S m ) for any k ∈
Revised: More precisely, we show that max k ∈I 2 nλ S k ∈ o ( nλ S m ) = o (max { λ, 1 } ) .

Original:  I 2 .
Revised: Using T from Assumption 1, we have [Equation] For M 1 , we bound a generic k ∈ I 2 with | S k | < T as follows: [Equation]

Original: Applying Lemma 10 yields
Revised: where ( i ) applies Equation (22) from Lemma 10, and ( ii ) uses ¯ L ≥ L and δ < 1 . In particular, this implies M 1 ∈ o ( nλ m ) . For M 2 we have

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from | S k | ≥ ¯ L + 2  , and ( ii ) uses ¯ L ≥ L and δ < 1 . 
Revised: where ( i ) applies Equation (23) from Lemma 10, ( ii ) uses that ¯ L + 2 ≤ T , and ( iii ) follows from ¯ L ≥ L and δ < 1 . Combining the bounds on M 1 and M 2 , we have max k ∈I 2 nλ S k ∈ o ( nλ S m ) =

Original:  Hence, for d sufficiently large,all k ∈ I 2 have λ S k < λ  m and therefore k > m by definition.
Revised: o (max { λ, 1 } ) . Hence, for d sufficiently large, all k ∈ I 2 yield λ S k < λ S m and consequently k > m .

Original: Using the 1 - 2 decomposition, we now prove Lemma 3.
Revised: Using the 1 - 2 decomposition, we now prove Lemma 3.

Original: We defer the auxiliary Lemmas 12 to 15 afterthe proof, and more technical tools to Appendix C.4.
Revised: We defer the auxiliary Lemmas 12 to 15 to Appendix C.4, and concentration-results to Appendix C.5. C. P ROOF OF L EMMA

Revised: Throughout the proof, we assume d to be large enough such that all quantities are well-defined and all necessary lemmas apply.

Original: Proof of Lemma 3.
Revised: In particular, we assume the conditions of Lemma 11 to be satisfied, and that c < ⌊ q/ 2 ⌋ < q < d/ 2 for c in Assumption 1. Hence, L + 2 ≤ ¯ L + 2 < T < ⌊ q/ 2 ⌋ , and we can apply Lemmas 9 to 15, the setting of Appendix C.2, as well as Assumption 1 throughout the proof. We will mention additional implicit lower bounds on d as they arise.

Original: We address the bound r 1 and r 2 , and finally bound Tr ( S >m )  .
Revised: The proof proceeds in three steps: we first bound r 1 and r 2 , then bound Tr ( S >m ) , and finally (cid:80) ni =1+ m µ i ( S >m ) .

Original: Throughout this proof, we write “with high probability” to indicate with probability at least 1−cq −  min { ¯ δ, 1 − ¯ δ }for some context-dependent constant c .Furthermore, this proof does not establish therelevant matrix concentration results directly, but combines them from auxiliary lemmas.
Revised: We do not establish the required matrix concentration results directly, but apply various auxiliary lemmas. All corresponding statements hold with either probability at least 1 − ˜ cq − ¯ δ or at least 1 − ˜ cq − (1 − ¯ δ ) for context-dependent constants ˜ c . We hence implicitly choose a c >such that collecting all error probabilities yields the statement of Lemma 3 with probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } .

Original: Furthermore,we use the m corresponding to S m , and instantiate Definition 2 with that m .
Revised: To start, let m ∈ N as in the statement of Lemma 3, and instantiate Appendix C.2 with that m .

Original: Then, Lemma 11 yieldsthe 1 - 2 decomposition K >m = K 1 + K 2  , which we will henceforth use. 
Revised: In particular, Lemma 11 yields the 1 - 2 decomposition K >m = K 1 + K 2 and S >m = S 1 + S 2 , which we will henceforth use. Finally, we define [Equation] with the corresponding kernel matrix Q ( d,q ) l

Revised: [Equation]

Original:  r 1 and r
Revised: Bound on r 1 and r

Original: Remember the definition of r 1 and r 2 :
Revised: Remember the definition of r 1 and r 2 :

Original: [Equation]
Revised: [Equation]

Original: { λ, I } . To bound those quantities, we have to bound µ min ( K >m ) and ∥ K >m ∥ .
Revised: To bound those quantities, we have to bound µ min ( K >m ) and ∥ K >m ∥ .

Original: For the upper bound on∥ K >m ∥ , we use the triangle inequality on ∥ K >m ∥ =
Revised: For the upper bound on ∥ K >m ∥ , we use the triangle inequality on ∥ K >m ∥ =

Original: ∥ K 1 + K 2 ∥ , and then bound ∥ K 1 ∥ and ∥ K 2 ∥individually.
Revised: ∥ K 1 + K 2 ∥ , and then bound ∥ K 1 ∥ and ∥ K 2 ∥ individually.

Original: Note that we can write K 1 =
Revised: Note that we can write K 1 =

Original: Ψ 1 D 1 Ψ ⊺ 1 by definition.
Revised: Ψ 1 D 1 Ψ ⊺ 1 by definition.

Original: Then,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from Lemma 12 with probability at least 1 −  c 3 q − ¯ δ , ( ii ) uses that all eigenvaluesof K 1 are at most λ m by definition, and ( iii ) follows from nλ m ∈ Θ(max { λ, 1 } ) .
Revised: where ( i ) follows from Lemma 12 with probability at least 1 − ˜ c 1 q − ¯ δ , ( ii ) uses that all eigenvalues of K 1 are at most λ m by definition, and ( iii ) follows from nλ m ∈ Θ(max { λ, 1 } ) .

Original: Finally, Lemma 13 directly yields with probability at least 1 −  c 4 q − (1 − ¯ δ ) that ∥ K 2 ∥ ∈ Θ(1) and µ min ( K 2 ) ∈ Θ(1) .
Revised: Next, Lemma 13 directly yields with probability at least 1 − ˜ c 2 q − (1 − ¯ δ ) that ∥ K 2 ∥ ∈ Θ(1) and µ min ( K 2 ) ∈ Θ(1) .

Original: Hence,for d large enough, with probability at least 1 − ( c 3 q − ¯ δ +  c 4 q − (1 − ¯ δ ) ) ≥ 1 − cq − min { ¯ δ, 1 − ¯ δ } , we have
Revised: Hence, with probability at least 1 − (˜ c 1 q − ¯ δ + ˜ c 2 q − (1 − ¯ δ ) ) ≥ 1 − cq − min { ¯ δ, 1 − ¯ δ } , we have

Original: [Equation]
Revised: [Equation]

Original: This impliesthat with high probability:
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: This last equation directly implies that, with high probability:
Revised: + λ ≥ µ min ( K 2 ) + λ ∈ Ω

Original: [Equation]
Revised: [Equation]

Original: But from the definition of r 1 and r 2 , it follows that r 1 ≤ r 2 . Thus, r 1 , r 2 ∈ Θ(1) .
Revised: { λ, 1 } ) , and subsequently [Equation] Finally, since r 1 ≤ r 2 , this yields r 1 , r 2 ∈ Θ(1) .

Original: Bound on Tr ( S >m ).
Revised: Bound on Tr ( S >m )

Original: Finally, we need to show that
Revised: We need to show that

Original: [Equation]
Revised: [Equation]

Revised: with high probability, where the two terms correspond to the 1 - 2 decomposition

Original: [Equation]
Revised: Tr ( S >m ) =

Original: with high probability.
Revised: Tr ( S 1 ) + Tr ( S 2 ) .

Original: To do so, we differentiate whether ¯ L = L and ¯ L > L .
Revised: We differentiate between ¯ L = L and ¯ L > L .

Original: Intuitively, the case¯ L =
Revised: Intuitively, the case ¯ L =

Original: L corresponds to interpolation or mild regularization, because the maximum degree of learnablepolynomials with regularizationL equals the one without regularization ¯ L .
Revised: L corresponds to interpolation or weak regularization, because the maximum degree of learnable polynomials with regularization equals the one without regularization.

Original: Conversely, ¯ L > Lcorresponds to strong regularization.
Revised: Conversely, ¯ L > L corresponds to strong regularization.

Original: Case ¯ L = L (interpolation or mild regularization):
Revised: Case ¯ L = L (interpolation or weak regularization):

Original: In this setting,
Revised: In this setting,

Original: [Equation]
Revised: [Equation]

Original: First, Lemma 14 yields
Revised: First, Lemma 14 yields

Original: Tr ( S 2 ) ∈ Θ( q − (1 − ¯ δ ) )
Revised: Tr ( S 2 ) ∈ Θ( q − (1 − ¯ δ ) )

Original: with probability at least 1 −  c 1 q − (1 − ¯ δ ) .
Revised: with probability at least 1 − ˜ c 3 q − (1 − ¯ δ ) .

Original: Therefore,
Revised: Therefore,

Revised: Tr ( S 2 ) ∈ Θ( q

Original: [Equation]
Revised: [Equation]

Revised: − ¯

Original: where ( i ) follows from Equation (25).
Revised: [Equation] [Equation] [Equation]

Original: This directly yields the lower bound
Revised: −

Original: [Equation]
Revised: [Equation]

Revised: −

Original: For the upper bound, we also need to consider Tr ( S 1 ) :
Revised: [Equation] [Equation] [Equation] −

Original: [Equation]
Revised: [Equation]

Original:  where ( i ) follows from Lemma 12 with probability at least 1 − c 2 q − ¯ δ . Then,
Revised: (cid:17) , where ( i ) follows from Equation (32). We now consider Tr ( S 1 ) : [Equation]

Original: Tr
Revised: Tr

Revised: [Equation]

Revised: [Equation]

Revised: i

Original: [Equation]
Revised: [Equation]

Original: S 1 ) ≲ n 
Revised: 1 . 5 n (cid:88) [Equation] where ( i ) follows from Lemma 12 with probability at least 1 − ˜ c 1 q − ¯ δ .

Revised: The bound continues as

Original: [Equation]
Revised: [Equation]

Original: where ( i ) uses λ k ≤ λ m for all k ∈
Revised: where ( i ) uses λ k ≤ λ m for all k ∈

Original: I 1 by definition, ( ii ) usesthe assumption nλ m ∈ Θ(max { λ, 1 } ) , and ( iv ) follows from Equation (25).
Revised: I 1 by definition, ( ii ) uses nλ m ∈ Θ(max { λ, 1 } ) , and ( iv ) follows from Equation (32).

Original: Furthermore, ( iii ) uses the following bound of |I 1 | :
Revised: Furthermore, ( iii ) uses the following bound of |I 1 | :

Original: [Equation]
Revised: [Equation]

Original: where C ( l, q, d ) is defined in Equation (17), ( i ) follows from Lemma 9 from d large enough, and ( ii ) is a classical bound on the binomial coefficient.
Revised: where C ( l, q, d ) is defined in Equation (17), ( i ) follows from Lemma 9, ( ii ) is a classical bound on the binomial coefficient, and ( iii ) follows from the fact that the term corresponding to ¯ L +dominates the polynomial.

Original: Finally, collecting the upper bounds on Tr ( S 1 ) and Tr ( S 2 ) yieldsthe desired upper bound
Revised: Finally, collecting the upper bounds on Tr ( S 1 ) and Tr ( S 2 ) yields

Original: [Equation]
Revised: [Equation]

Original: withhigh probability, and hence the overall bound on Tr ( S ≥ m ) with probability at least 1 −( c 1 q − 
Revised: with probability at least 1 − (˜ c 3 q − (1 − ¯ δ ) + ˜ c

Original: [Equation] 1− ¯ δ ) + c 2 q − ¯ δ ) ≥ 1 − cq − min { ¯ δ, 1 − ¯ δ }for the case where ¯ L = L .
Revised: 1 q − ¯ δ ) ≥ 1 − cq − min { ¯ δ, 1 − ¯ δ } .

Original: Case ¯ L > L (strong regularization):
Revised: Case ¯ L > L (strong regularization):

Original: In this setting, the dominating rate will arise from Tr ( S 1 ) .
Revised: In this setting, the dominating rate will arise from Tr ( S 1 ) .

Original: Westart by linking ¯ δ and δ analogous to Equation (25):
Revised: We start by linking ¯ δ and δ in analogy to Equation (32):

Original: [Equation]
Revised: [Equation]

Original: Next, as in the previous case, Lemma 14 yields Tr ( S 2 )
Revised: Next, as in the previous case, Lemma 14 yields Tr ( S 2 )

Original: ∈ Θ( q − (1 − ¯ δ ) )
Revised: ∈ Θ( q − (1 − ¯ δ ) )

Original: with probability at least 1 −  c 1 q − (1 − ¯ δ ) , and therefore
Revised: with probability at least 1 − ˜ c 3 q − (1 − ¯ δ ) , and therefore

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from Equation (26), and ( ii ) from ¯ L ≥ L+ 1 .
Revised: where ( i ) follows from Equation (33), and ( ii ) from ¯ L > L .

Original: Due to the slower growth in this setting, we consider Tr ( S 2 ) only for the upper bound of Tr ( S ≥ m ) , but not for the lower bound.

Original: To bound Tr ( S 1 ) , we first use
Revised: To bound Tr ( S 1 ) , we start as in the previous case:

Original: [Equation]
Revised: [Equation]

Original: where ( i ) and ( ii ) follow from Lemma 12simultaneously with probability at least 1 −  c 2 q − ¯ δ .
Revised: where ( i ) follows from Lemma 12 with probability at least 1 − ˜ c 1 q − ¯ δ .

Original: We then decompose the sum over all squared eigenvalues with index in I 1 asn
Revised: We then decompose the sum over all squared eigenvalues with index in I 1 as

Original: [Equation]
Revised: [Equation]

Original: + n (cid:88) k ∈I 1 | S k | =

Original:  L + λ 2 S k
Revised: S k |≤ L + λ 2 S k

Original: (cid:124) (cid:123)(cid:122) (cid:125) 
Revised: (cid:124) (cid:123)(cid:122) (cid:125) =:

Original: T+ n¯ L + (cid:88) l = L +3 (cid:88) k ∈I 1 | S k | =
Revised: E+ n (cid:88) k ∈I 1 | S k | =

Original: l λ 2 S k
Revised: L + λ 2 S k

Original: (cid:124) (cid:123)(cid:122) (cid:125) 
Revised: (cid:124) (cid:123)(cid:122) (cid:125) =:

Original: T. In the following, we analyze T 1 , T 2 , andT 3 separately. Upper boundFor theupper bound of Tr ( S ≥ m ) , we bound all three terms from above.
Revised: E+ n (cid:88) k ∈I 1 | S k |≥ L + λ 2 S k (cid:124) (cid:123)(cid:122) (cid:125) =: E, and bound the three terms individually.

Original: First, we upper-bound T 1 as follows:
Revised: First, we upper-bound E 1 as follows:

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from λ k ≤ λ m for all k > m due to the decreasing order of eigenvalues.
Revised: where ( i ) follows from λ k ≤ λ m for all k > m due to the decreasing order of eigenvalues.

Original: Furthermore, ( ii ) uses nλ m ∈ Θ(max { λ, 1 } )and the fact that C ( l, q, d ) ∈ O ( dq l − 1 ) , which follows from
Revised: Step ( ii ) applies nλ m ∈ Θ(max { λ, 1 } ) , as well as (cid:80)

Original: Lemma 9 together with a classical bound on the binomial coefficient: for d large enough,
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: dq l −(cid:1) .
Revised: 0 C ( l, q, d ) ∈ O ( d · q L ) , which follows as in the other case from Lemma 9 and the classical bound on the binomial coefficient.

Original: Second, the upper bound of T 2 arises as follows:
Revised: Second, the upper bound of E 2 arises as follows:

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows fromthe eigenvalue formula in Proposition 1, and ( ii ) uses q + 1 − γ ( S  ) ≤ q .
Revised: where ( i ) follows from Proposition 1, and ( ii ) uses q + 1 − γ ( S k ) ≤ q .

Original: Next, ( iii ) uses  Assumption 1  and applies the bound B ( L + 2 , q ) 
Revised: Next, ( iii ) uses that Equations (18) and (21) in Assumption 1 imply ξ ( q ) L +2 ∈ Θ(1) , and applies the bound B ( L + 2 , q ) =

Original: = (cid:0) qL + (cid:1) ≤ ( eq/ ( L + 2))
Revised: (cid:0) qL + (cid:1) ≤ ( eq/ ( L + 2))

Original: L +2 .
Revised: L +2 .

Original: Step( iv ) uses Y S ( x ) Y S ( x ) = 1 for all S and x ∈ {− 1 , 1 } d , together with the definition of Q ( d,q ) L +2 .
Revised: Step ( iv ) uses Y S ( x ) Y S ( x ) = 1 for all S and x ∈ {− 1 , 1 } d , together with the definition of Q ( d,q ) L +2 .

Original: Lastly,( v ) applies Lemma 15.
Revised: Lastly, ( v ) applies Lemma 15 and n ∈ Θ( d ℓ ) =

Revised: Θ( d · d ℓ λ q L + δ ) .

Original: Third, we upper-bound T 3 .
Revised: Third, we upper-bound E 3 :

Original: For this, let ˜ k be the smallest index that corresponds to a set S ˜ k with | S ˜ k
Revised: [Equation] The last step follows from

Original: | =
Revised: [Equation]

Original: L + 3 .
Revised: where ( i ) follows from Proposition 1, ( ii ) uses Y S ( x ) Y S ( x ) = 1 for all S and x ∈ {− 1 , 1 } d , ( iii ) applies the definition of Q ( d,q ) l and Lemma 15, and ( iv ) follows from Equations (18) and (21) in

Original: Since the eigenvalues are indexed in decreasing order, λ S ˜ k is the largest eigenvalue corresponding to a set of size L + 3 . 
Revised: Assumption 1 since ¯ L + 1 ≤ T . For max k ∈I 1 , | S k |≥ L +3 ( λ S k ) , we bound each element individually: [Equation]

Original: Furthermore, for d large enough, Lemma 10 yields that λ S ˜ k 
Revised: where ( i ) uses Equation (22) in Lemma 10 since k ≤ ¯ L + 1 < T by definition of I 1 .

Original: is the maximum eigenvalue for all sets of size at least L + 3 .
Revised: Hence, we obtain the following bound on E 3 : [Equation] Finally, we can bound Tr ( S 1 ) as

Original: Then, for d large enough,
Revised: [Equation] which yields [Equation] as desired with probability at least

Original: [Equation]
Revised: [Equation]

Revised: Bound on (cid:80)

Revised: ni =1+ m µ i ( S >m )

Revised: As before, we differentiate between no/weak and strong regularization, that is, between ¯ L = L and ¯ L > L :

Revised: Case ¯ L = L (interpolation or weak regularization):

Revised: In this case, we start by directly bounding n

Original:  where ( i ) follows from the fact that the sum ofall the  eigenvalues of  the kernel must be bounded bya constant, thus the sum of a subset of the eigenvalues must be bounded as well.
Revised: [Equation] where ( i ) bounds each of the first m eigenvalues of S 2 with the largest one, ( ii ) follows from Lemma 14 with probability at least 1 − ˜ c 3 q − (1 − ¯ δ ) , and ( iii ) from Equation (32) since ¯ L = L . To conclude the lower bound, it suffices to show that

Revised: [Equation]

Original: Furthermore, ( ii )follows from Lemma 10 with | S ˜ k
Revised: where ( i ) follows from m ∈ O (cid:16) nq − δ max { λ, 1 } (cid:17) , and ( ii ) from Equation (32). For ( iii ) , note that ¯ δ =for a sufficiently large c yields a vacuous result. We hence assume without loss of generality that ¯ δ > 0 , which justifies the step.

Revised: This concludes the proof for the current case with probability at least

Original: | =
Revised: [Equation] ˜ [Equation] − min { ¯ δ, 1 − ¯ δ } .

Original:  L + 3  .
Revised: Case ¯ L > L (strong regularization): In this case, we define the additional index set [Equation] with S 3 , Ψ 3 , D 3 analogously to S 1 , Ψ 1 , D 1 in Appendix C.2, but using I 3 instead of I 1 .

Revised: Since

Revised: I 3 ⊆

Revised: I 1 , it follows that Ψ ⊺ 3 Ψ 3 is a submatrix of Ψ ⊺ 1 Ψ 1 , and thus Ψ ⊺ 3 Ψn

Original: [Table]
Revised: [Equation]

Revised: I |I 3 | is a submatrix of Ψ ⊺ 1 Ψn

Revised: [Equation]

Original:  where ( i ) follows from Proposition 1.
Revised: |I 1 | . This particularly implies [Equation] where ( i ) follows from Lemma 12 with probability at least 1 − ˜ c

Revised: 1 q − ¯ δ .

Revised: We now move our focus back to the lower bound of

Original: Step ( ii ) uses that all k ∈ I 1 have | S k | ≤ q by definition; hence the k ∈ I 1 with | S k | ≤
Revised: [Equation] 36) where ( i ) follows from the 1 - 2 decomposition S >m = S 1 + S 2 , ( ii ) from the fact that I 3 ⊆ I 1 , and ( iii ) analogously to Equation (34). Similar to the previous case, we conclude the proof by first showing that Tr ( S 3 ) ∈

Revised: Ω

Revised: [Equation]

Revised: d

Revised: [Equation]

Original: ⌊ q/ 2 ⌋ are a subset of I 1 .
Revised: q − [Equation] 1 − δ [Equation]

Revised: , and then m ∥ S 3 ∥ ∈ o ( Tr ( S 3 )) .

Revised: For the lower bound of Tr ( S 3 ) , we start with

Revised: [Equation]

Original: Next, ( iii ) follows since q ≥ ⌊ q/ 2 ⌋ and q + 1 − γ ( S ) ≥ q/ 2 .
Revised: where ( i ) follows with high probability from Equation (35). Step ( ii ) applies Proposition 1, and the fact that I 3 = { k ∈ N | | S k | = L + 2 and γ ( S ) ≤ q } for d sufficiently large. To show this, we use λ m ∈ Θ (cid:18)

Revised: [Equation]

Revised: Since the eigenvalues are in decreasing order and L + 2 ≤ ¯ L + 1 in the current case, we only need to show that λ S <

Revised: λ m for all S ⊆ { 1 , . . .

Revised: , d } with | S | =

Original: Step (v ) uses Y S  ( 
Revised: L + 2 and γ ( S ) ≤ q : [Equation] where ( i ) applies Equation (22) in Lemma 10 since L +2 < T . Thus, λ S < λ m for all S ⊆ { 1 , . . .

Original: x ) Y S ( x ) = 1 for all S and x ∈ {− 1 ,1 } d , and  ( vi )  from Lemma 15 with ⌊ q/ 2 ⌋ as filter size.
Revised: , d } with | S | = L + 2 and γ ( S ) ≤ q if d is sufficiently large, which we additionally assume from now on. The lower bound of Tr ( S 3 ) continues as follows:

Original: Finally, ( iv ) follows from the fact thatB ( L + 2 , q ) and
Revised: [Equation] where ( iii ) follows from q ≥ ⌊ q/ 2 ⌋ and q + 1 − γ ( S ) ≥ q/ 2 . Step ( iv ) follows from the fact that

Original: B ( L +2 , ⌊ q/ 2 ⌋ ) are of the same order; this follows from a classical bound on the binomial coefficient:
Revised: B ( L + 2 , q ) and B ( L + 2 , ⌊ q/ 2 ⌋ ) are of the same order; this follows from a classical bound on the binomial coefficient:

Original: [Equation]
Revised: [Equation]

Original: Ultimately, we may summarize all individual bounds for the case ¯ L > L as follows:
Revised: We conclude the lower bound on Tr ( S 3 ) as follows:

Original: [Equation]
Revised: [Equation]

Original: with prob. at least 1 − c 1 q − (1 − ¯ δ )
Revised: where ( v ) uses Y S ( x ) Y S ( x ) = 1 for all S and x ∈ {− 1 , 1 } d with the definition of Q ( d, ⌊ q/ 2 ⌋ )

Original: Tr
Revised: L + in Equation (30), ( vi ) applies Lemma 15 with ⌊ q/ 2 ⌋ as filter size, ( vii ) follows from Equation (18) in Assumption 1, and ( viii ) uses the classical bound on the binomial coefficient. Finally, for the upper bound on m ∥ S 3 ∥ , we have

Original: [Equation]
Revised: [Equation]

Original: S
Revised: where ( i ) follows with high probability from Equation (35), and ( ii ) from Proposition 1. Step ( iii ) uses that Equations (18) and (21) in Assumption 1 yield ξ ( q ) L +2 ∈ Θ(1) , and B ( L + 2 , q ) = (cid:0) qL + (cid:1) ∈

Original: [Equation]
Revised: O ( q L +2 ) .

Original: = [Equation] 1 + T 2 + Twith prob. at least 1  − c 2 q − ¯ δ 
Revised: Furthermore, ( iv ) follows from m ∈ O (cid:16) nq − δ max { λ, 1 } (cid:17) , ( v ) from q − δ q − (1 − δ )

Original: T
Revised: ∈

Original: [Equation]
Revised: o (1) , and ( vi ) from the lower bound on Tr ( S 3 ) in Equation (37).

Original: O
Revised: Finally, combining this result with Equations (36) and (37), we have

Original: [Equation]
Revised: [Equation]

Original: Combined, those bounds yield Equation (24) with probability at least 1 − ( c 1 q − (1 −¯ δ ) + c 2 q − ¯ δ )
Revised: (cid:16) d ℓ λ q − (1 − δ ) (cid:17) with probability at least 1 − ˜ c

Original: ≥1 − cq − min { ¯ δ, 1 − ¯ δ }for the case where ¯ L > L .
Revised: 1 q − ¯ δ ≥ 1 − cq − min { ¯ δ, 1 − ¯ δ } .

Original: Since the cases are exhaustive, and hold both withsame probability, this concludes the proof of the bound on Tr ( S >m ) . The statement of Lemma 3 then follows from taking the union bound on the probabilities over theindividual proof parts. [Equation]
Revised: C. T ECHNICAL LEMMAS

Original: Finally, we use the following technical lemmas in the proof of Lemma 3. 
Revised: We use the following technical lemmas in the proof of Lemma 3. All results assume the setting of

Revised: Appendix C.2, particularly a kernel as in Theorem 1 that satisfies Assumption 1.

Original: Lemma 12 (Bound on K 1 ) .
Revised: Lemma 12 (Bound on K 1 ) .

Original: In the setting of Definition 2, uniformly over all choices of λ , we have (cid:13)(cid:13)(cid:13)(cid:13) Ψ ⊺ 1 Ψn − I  (cid:13)(cid:13)(cid:13)(cid:13) ≤ 1 / 2 , with probability at least 1 − cq − ¯ δ  .
Revised: In the setting of Appendix C.2, for d/ 2 > q ≥ ¯ L + 1 , we have (cid:13)(cid:13)(cid:13)(cid:13) Ψ ⊺ 1 Ψn − I |I 1 | (cid:13)(cid:13)(cid:13)(cid:13) ≤ 1 /with probability at least 1 − cq − ¯ δ uniformly over all choices of m and λ .

Original: Proof.
Revised: Proof.

Original: The proof follows a very similar argument to Lemma 1.
Revised: The proof follows a very similar argument to Lemma 1 with minor modifications.

Original: Still, minor differences prevent usfrom using that result directly.

Original: We start by defining
Revised: First, define ˆ I 1 : =

Original: [Equation] Notice thatthe definition of ˆ I 1 does not depend on m , and hence it does not depend on λ. Furthermore,notice that for any m ,  I 1 ⊆
Revised: { k ∈ N | | S k | ≤ ¯ L + 1 } . Note that ˆ I 1 does not depend on m or λ , and I 1 ⊆

Original: ˆ
Revised: ˆ

Original: I 1  . We start by computing the
Revised: I 1 for any m . Furthermore, | ˆ I 1 | =

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from Lemma 9 for d large enough, ( ii ) is a classical bound on the binomialcoefficient, and ( iii ) follows from the fact that the largest degree monomial dominates the others.
Revised: where ( i ) follows from Lemma 9, ( ii ) is a classical bound on the binomial coefficient, and ( iii ) follows from the fact that the largest degree monomial dominates the others.

Original: Now we define ˆ Ψ 1 ∈ R n ×| ˆ I 1 | in analogous way to how Ψ 1  is defined in Definition 2.
Revised: Next, we define ˆ Ψ 1 ∈ R n ×| ˆ I 1 | with [ ˆ Ψ 1 ] i,j = Y S kj ( x i ) where k j is the j -th element in ˆ I 1 .

Original: Using the same arguments used already in the proof of Lemma 1, we can show that the rows of ˆ Ψ 1 are independent, with norm  bounded by | ˆ I 1 | , and with expectation of outer product equal to I | ˆ I 1 | .
Revised: Using the same arguments as in the proof of Lemma 1, it follows that the rows of ˆ Ψ 1 are independent, that their norm is bounded by | ˆ I 1 | , and that they have an expected outer product equal to I | ˆ I 1 | .

Original: Again as in  Lemma 1, we can apply Theorem 5.44 from Vershynin (2012), and, choosing t = 1 n (cid:113) n | ˆ I 1 | , we get that: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: Hence, as in the proof of Lemma 1, we can apply Theorem 5.44 from Vershynin (2012). Choosing t =(cid:113) n | ˆ I 1 | yields (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: ˆ Ψ ⊺ 1 ˆ Ψn − I | ˆ I 1 | (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ 1 /with probability at least 1 − cq − ¯ δ for some absolute constant c .

Original: To conclude the proof, it suffices to remember that, uniformly over all choices of λ, andconsequential choice of m , I 1 ⊆
Revised: Finally, since I 1 ⊆ ˆ I 1 for all choices of λ and m , (cid:13)(cid:13)(cid:13)(cid:13)

Original: ˆ
Revised: Ψ ⊺ 1 Ψn − I |I 1 | (cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original:  I 1  .Thus: [Equation] concluding the proof.
Revised: ˆ Ψ ⊺ 1 ˆ Ψn − I | ˆ I 1 | (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ 1 /with probability at least 1 − cq − ¯ δ uniformly over all λ and m .

Original: Lemma 13 (Bound on K 2 ) .
Revised: Lemma 13 (Bound on K 2 ) .

Original: In the setting of Definition 2, assuming that the kernel satisfies Assumption 1,with probability at least 1 − cq − (1 − ¯ δ ) , we have
Revised: In the setting of Appendix C.2, if c < ⌊ q/ 2 ⌋ < q < d/ 2 for c as in Assumption 1, we have

Original: [Equation]
Revised: [Equation]

Original: for some positive constants c 1 , c 2  .
Revised: for some positive constants c 1 , c 2 with probability at least 1 − ˜ cq − (1 − ¯ δ ) .

Original: Proof.
Revised: Proof.

Original: For the upper bound on the largest eigenvalue, Proposition 1 yields the following decomposition of K 2 ( x, x ′ ) :
Revised: First, the condition on d implies q > T and ensures that we can apply Lemmas 16 and 17, and Assumption 1. Furthermore, note that T − 1 > ¯ L + 2 . Proposition 1 and the definition of K 2 yield the following decomposition:

Original: [Equation]
Revised: [Equation]

Original: where Q ( d,q ) l ( x, x ′ ) is defined in Definition 2.
Revised: where Q ( d,q ) l ( x, x ′ ) is defined in Equation (30).

Original: Then, using the triangle inequality and positivity of the ξ ( q ) l from  Assumption 
Revised: Then, using the triangle inequality with non-negativity of the ξ ( q ) l from Equations (18) and (19) in Assumption 1, we have

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: (cid:13)(cid:13)(cid:13) (cid:13) = (cid:13)(cid:13) (cid:13)(cid:13)(cid:13) (cid:13) T −(cid:88) l =¯ L + ξ ( q ) l Q ( d,q ) l + q (cid:88) l = T ξ ( q ) l Q ( d,q ) l (cid:13)(cid:13) (cid:13)(cid:13)(cid:13) (cid:13) ≤

Original: [Equation]
Revised: T −(cid:88) l =¯ L + ξ ( q ) l (cid:13)(cid:13) (cid:13) Q ( d,q ) l (cid:13)(cid:13) (cid:13) + (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) q (cid:88) l = T ξ ( q ) l Q ( d,q ) l (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) ( i ) ≤ 1 . 5 T −(cid:88) l =¯ L + ξ ( q ) l + c 3 ( ii ) ≤ cwith probability ≥ 1 − cq − (1 − ¯ δ ) .

Original: where Q( d,q ) l is the kernel matrix corresponding to Q ( d,q ) l ( x, x ′ ) , ( i ) uses Lemma 16, and ( ii ) followsfrom Assumption 1.
Revised: Q d,q ) l is the kernel matrix corresponding to Q ( d,q ) l ( x, x ′ ) , ( i ) uses Lemmas 16 and 17, and ( ii ) uses non-negativity and additionally Equation (21) in Assumption 1.

Original: The lower bound follows  from
Revised: The lower bound follows similarly from

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from Lemma 16.
Revised: where ( i ) follows from Lemma 16 with probability at least 1 − cq − (1 − ¯ δ ) , and ( ii ) from Equation (18) in Assumption 1.

Original: Since Lemma 16 yields both the upper and lower bound at once with probability 1 − cq − (1 − ¯ δ ) , thisconcludes the proof.
Revised: Since Lemma 16 yields both the upper and lower bound for all l uniformly with probability 1 − cq − (1 − ¯ δ ) , this concludes the proof.

Original: Lemma 14 (Bound on Tr ( S 2 ) ) .
Revised: Lemma 14 (Bound on Tr ( S 2 ) ) .

Original: In the setting defined in Section 3.1, we have that:
Revised: In the setting of Appendix C.2, if c < ⌊ q/ 2 ⌋ < q < d/ 2 for c as in Assumption 1, we have with probability at least 1 − ˜ cq − (1 − ¯ δ ) that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − cq − (1 − ¯ δ ) .
Revised: where S 2 is defined in Appendix C.2.

Original: Proof.
Revised: Proof.

Revised: Throughout the proof, the conditions on d and hence q ∈ Θ( d β ) ensure that we can apply

Original: We start by proofing two technicalities.
Revised: Assumption 1 and Lemma 16, as well as ¯ L + 2 ≤ T < ⌊ q/ 2 ⌋ . First, Lemma 13 yields ∥ K 2 ∥ ∈ Θ(1) with probability at least 1 − c ′ q − (1 − ¯ δ ) , which we will use throughout the proof. Next, we bound ∥ S 2 ∥ in two steps.

Revised: For this, remember that λ ¯ m is the largest eigenvalue corresponding to an eigenfunction Y S of degree | S | ≥ ¯ L + 2 .

Original: Proof that ∥ S 2 ∥ ≤ λ ¯ m ∥ K 2 ∥.
Revised: Proof that ∥ S 2 ∥ ≤ λ ¯ m ∥ K 2 ∥

Revised: Define ˜ Ψ k ∈ R n × n with [ ˜ Ψ k ] i,j :=

Revised: Y S k

Revised: ( x i )

Revised: Y S k

Revised: ( x j ) for all i, j ∈ { 1 , . . . , n } , k ∈

Original: Let v be an arbitrary vector in R n , then:
Revised: I 2 , and let v be any vector in R n . Then,

Original: [Equation]
Revised: [Equation]

Original: wherein ( i ) we used the definition of m together with the fact that the eigenvalues are sorted indescending order.
Revised: where ( i ) follows from the definition of S 2 and ( ii ) from the definition of ¯ m .

Original: Proof that λ ¯ m ∈ Θ [Equation] L + 
Revised: Proof that λ ¯ m ∈ Θ (cid:16) 1 dq ¯ L + (cid:17)

Original: [Equation] To show this, we show thatfor any k in I 2 , λ k ∈ O (cid:16) 1 dq ¯ L + (cid:17)  and thatthere existsa ˜ m such that λ ˜ m ∈ Ω (cid:16) 1 dq ¯ L + (cid:17) .
Revised: We show that λ k ∈ O (cid:16) 1 dq ¯ L + (cid:17) for all k ∈ I 2 , and that there exists ˜ m ∈ I 2 with λ ˜ m ∈ Ω (cid:16) 1 dq ¯ L + (cid:17) .

Original: Since λ ¯ m is the largest of the eigenvalues λ S k , k ∈ I 2 ,those two fact implythat λ ¯ m ∈ Θ 
Revised: Since λ ¯ m = max k ∈I 2 λ S k , those two facts imply λ ¯ m ∈ Θ (cid:16) 1 dq ¯ L + (cid:17) . Let k ∈

Original: [Equation] Let k be an arbitrary index in I 2 , then | S k | ≥ ¯ L + 2 .
Revised: I 2 be arbitrary.

Original: Lemma 10 yieldsthat:
Revised: Lemma 10 yields

Original: [Equation]
Revised: [Equation]

Original: This concludes the proof that for any k in I 2 , λ k ∈ O (cid:16) [Equation] dq ¯ L + (cid:17) .
Revised: where ( i ) follows from | S k | ≥ ¯ L + 2 and T ≥ ¯ L + 2 .

Original: Now we show that there exist ˜ m such that λ ˜ m ∈ Ω (cid:16) 1 dq ¯ L + (cid:17) .
Revised: Now we show that there exists ˜ m with λ ˜ m ∈ Ω (cid:16) 1 dq ¯ L + (cid:17) .

Original: To do so, we take ˜ m be such that S ˜ m
Revised: We choose ˜ m with S ˜ m

Original: = { 1 , 2 , . ..
Revised: = { 1 , 2 , . .

Original: ,  L + 2 } .
Revised: , ¯ L +2 } .

Original: Note that ˜ m ∈ I 2 for d sufficiently large, since | S ˜ m | =
Revised: Note that ¯ L + 2 = γ ( S ˜ m ) ≤ q and thus ˜ m ∈

Original: L + 2 .
Revised: I 2 .

Original: Then,
Revised: Next, Proposition 1 yields [Equation] where ( i ) follows from Equations (18) and (21) in Assumption 1.

Original: Proposition 1 yields the following formula for λ S ˜ m :
Revised: Finally, combining the previous two results and ∥ K 2 ∥ ∈ Θ(1) [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Upper bound.
Revised: Upper bound of Tr ( S 2 )

Original: The upper bound  follows by simply applying the last two results:
Revised: The upper bound also follows directly from the last two results:

Original: [Equation]
Revised: [Equation]

Original: where ( i )follows with probability at least 1 − cq − (1 − ¯ δ ) from the Lemma 13. Lower bound.
Revised: Lower bound of Tr ( S 2 )

Original: For the lower bound we need a more refined argument. 
Revised: The lower bound requires a more refined argument. [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows  Proposition 1.
Revised: where ( i ) follows from Proposition 1.

Original: In ( ii )  we used that as long as γ ( S ) ≤ ⌊ q/ 2 ⌋ ,  q + 1 − γ ( S ) ≥ qand q ≥ ⌊ q/ 2 ⌋ .
Revised: In ( ii ) , we use that, as long as γ ( S ) ≤ ⌊ q/ 2 ⌋ , we have q + 1 − γ ( S ) ≥ q 2 and q ≥ ⌊ q/ 2 ⌋ .

Original:  In ( iii )we use the bound on the binomial coefficient (cid:16)
Revised: Continuing the bound, we have [Equation] In ( iii )

Original: [Equation]
Revised: [Equation]

Original: =
Revised: we use the classical bound on the binomial coefficient

Original: [Equation]
Revised: [Equation]

Original: q
Revised: B

Original: [Equation]
Revised: [Equation]

Original: ≤ (cid:16) eq ¯ L + (cid:17) ¯ L +2 to compute:
Revised: ¯ L [Equation] 2 , q ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where c ′ ¯ L is a constant that depends  on ¯ L .
Revised: where c ′ ¯ L is a constant that depends only on ¯ L .

Original: Finally,in ( iv ) , we used the definition of Q ( d, ⌊ q/ 2 ⌋ ) ¯ L + in 
Revised: Finally, ( iv ) follows from the definition of Q ( d, ⌊ q/ 2 ⌋ ) ¯ L + in Equation (30).

Original: Definition 2. This impliesthat:
Revised: The bound on S 2 ( x, x ′ ) implies

Original: [Equation]
Revised: [Equation]

Original: Hence, we finally bound Tr ( S 2 ) :
Revised: and allows us to ultimately lower-bound Tr ( S 2 ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from the lower bound on µ min
Revised: where ( i ) follows from the lower bound on µ min

Original: [Equation]
Revised: [Equation]

Original: L + [Equation] found in Lemma 16, andon the lowerbound on ξ ( q ) ¯ L +2 is from Assumption 1.
Revised: in Lemma 16 with probability at least 1 − c ′′ ⌊ q/ 2 ⌋ − (1 − ¯ δ ) ≥ 1 − c ′′ q − (1 − ¯ δ ) , and ( ii ) follows from Equation (18) in Assumptionand the classical lower bound on the binomial coefficient.

Original: This  concludes the proof.
Revised: This yields the desired lower bound Tr ( S 2 ) ∈ Ω( q − (1 − ¯ δ ) ) . Finally, collecting all error probabilities concludes the proof.

Original: Lemma 15 (Diagonal elements of Q ( d,q ) ) .
Revised: Lemma 15 (Diagonal elements of Q ( d,q ) ) .

Original: For any x ∈ {− 1 , 1 } d, for any d ∈ N , q ∈ N , q < d/ 2 , for any l ∈ N , 
Revised: Let l, q, d ∈ N with 0 < l ≤ q < d/ 2 and x ∈ {− 1 , 1 } d . Then, Q ( d

Original: [Equation]
Revised: [Equation]

Original: Proof. We compute:
Revised: q

Original: [Equation]
Revised: [Equation]

Original: where (i ) is the definition of Q ( d,q ) l ( x, x) itself.
Revised: ( x

Original: ( ii ) follows noticing that the hypercube eigenfunctions squared are always equal to 1 , as clear from their definition in Equation (13). ( iv ) follows from
Revised: [Equation]

Original: Lemma 9, ( v ) from the hockey-stick identity and ( vi ) from the formula to compute B in Equation (15). The last thing to justify is ( iii ) . For this we notice that
Revised: x

Original: [Equation]
Revised: [Equation]

Original: matches the definition of
Revised: = 1 . Proof. First,

Original: [Equation]
Revised: [Equation]

Revised: where ( i ) follows from the fact that Y S ( x ) 2 = 1 .

Original: found in the proof of Lemma 9.
Revised: Note that (cid:80) γ ( S ) = γ | S | = l 1 matches the definition of ˜ C ( l, γ, d ) in the proof of Lemma 9.

Original: We then show the following recurrence:
Revised: Next, we use the following recurrence:

Original: [Equation]
Revised: [Equation]

Original: where in ( i ) we use the fact that C ( l, γ, d ) = (cid:80) qγ = l ˜ C ( l, γ, d ) , which follows from the definition.
Revised: where the last step uses the fact that C ( l, q, d ) = [Equation] qγ = l ˜ C ( l, γ, d ) by definition, and that the term corresponding to γ

Original: Furthermore, the second sum only reaches q − 1 since the last element would be 0 .
Revised: = q in the second sum is zero.

Original: Applying recursively this formula q − l times, we get:
Revised: Recursively applying this formula q − l times yields [Equation]

Revised: Using this identity, we finally get

Original: [Equation]
Revised: [Equation]

Original: which explains ( iii )  in Equation (27). This concludes the proof.
Revised: where ( i ) follows from Lemma 9, ( ii ) from the hockey-stick identity, and ( iii ) from the definition of B ( l, q ) in Equation (15).

Original: C. R ANDOM MATRIX THEORY LEMMAS
Revised: C. R ANDOM MATRIX THEORY LEMMAS

Original: In this subsection are contained the key random matrix concentration needed for the proof to gothrough.
Revised: We use the following lemmas related to random matrix theory in the proof of Lemma 3. The first two results bound the kernel’s intermediate and late eigenvalues.

Original: Lemma 16 (Bound on the kernel’s  tail) .
Revised: Lemma 16 (Bound on the kernel’s intermediate tail) .

Original: In thesame setting as Definition 2: [Equation] with probability at least 1 − cq − (1 − ¯ δ )  .
Revised: In the setting of Appendix C.2, for T − 1 ≤ q < d/ 2 , with probability at least 1 − cq − (1 − ¯ δ ) , all l ∈ N with ¯ L + 2 ≤ l < T satisfy ∥ Q ( d,q ) l − I n ∥ ≤ 1 2 , where T is defined in Assumption 1 and Q in Equation (30) .

Original: Notice that, from this lemma, it naturally follows that ∥ Q ( d,q ) l ∥ ≤ 1 . 5, and µ min (cid:16)
Revised: This lemma particularly implies ∥ Q ( d,q ) l ∥ ≤ 1 . 5 and µ min (cid:16)

Original: Q ( d,q ) l (cid:17) ≥ 1 / 2 .
Revised: Q ( d,q ) l (cid:17) ≥ 1 / 2 for all ¯ L + 2 ≤ l < T with high probability.

Original: Now we present the proof:
Revised: Lemma 17 (Bound on the kernel’s late tail) . In the setting of Appendix C.2, if c ≤ ⌊ q/ 2 ⌋ − 1 < q < d/ 2 , for c and T as in Assumption 1, we have (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) q (cid:88) l = T ξ ( q ) l Q ( d,q ) l (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≲with probability at least 1 − ˜ cd − 1 .

Original: Proof of Lemma 16.
Revised: Proof of Lemma 16.

Original: Thanks to Lemma 15, we know that the diagonal elements of Q ( d,q ) l are just 1 .
Revised: First, Lemma 15 yields that the diagonal elements of Q ( d,q ) l are just 1 .

Original: Hence, we define W ( d,q ) l := Q ( d,q ) l − I n . Our objective is now to show that ∥ W ( d,q ) l ∥ ≤ 1 / 2at the same time for every l > ¯ L + 1 with high probability. We will however use two different strategies for small l and large l . In particular, let us define
Revised: Hence, we define W ( d,q ) l := Q ( d,q ) l − I n , and want to show that, with high probability, ∥ W ( d,q ) l ∥ ≤ 1 / 2 for all ¯ L + 2 ≤ l < T at the same time.

Original: [Equation]

Original: Notice that T does not depend on d nor ℓ λ .

Original: Terms l < T : here we use  Lemma 17.
Revised: The proof makes use of Lemma 18.

Original: To successfully apply it, we need to find an appropriate M ( l,q )  , and show that the assumptions of the Lemma holds.
Revised: Therefore, we need to find an appropriate M ( l,q ) for each considered l , and show that the conditions of the lemma hold.

Original: The first assumption has already been checked, since we showed that diag( Q ( d,q ) l ) =
Revised: The first condition follows directly from the construction of W ( d,q ) l and diag( Q ( d,q ) l ) =

Original: I n .
Revised: I n .

Original: For the second one we can compute, for any i ̸ =  j :
Revised: To establish the second condition, we have for all i ̸ = j, k ̸ = j (cid:12)(cid:12)(cid:12) E x j (cid:104)

Revised: [ W

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: d,q [Equation] l [Equation]

Revised: i

Original: [Equation]
Revised: [Equation]

Revised: =

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: [Equation] [Equation] (cid:12)(cid:12)(cid:12)(cid:12) [Equation]

Revised: [Equation]

Revised: (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) ≤ q

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12) ≤ 
Revised: (cid:12)(cid:12)(cid:12) (cid:12)(cid:12) (cid:88) γ ( S ) ≤ q | S | = l ( [Equation] +

Original: [Equation]
Revised: [Equation]

Revised: γ

Revised: [Equation]

Revised: S )

Revised: [Equation]

Revised: S (

Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:104)
Revised: i ) [Equation] S ( x k ) (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) (cid:12)(cid:12) = (cid:18)

Original: W ( d 
Revised: d [Equation] l [Equation]

Revised: q ) q (cid:19)

Original: [Equation]
Revised: [Equation]

Original:  (cid:105) 
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:104) W ( [Equation] q ) l (cid:105) i,k (cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:18)

Original: i
Revised: d

Original: [Equation]
Revised: [Equation]

Original:  (cid:12)(cid:12)(cid:12)(cid:12) , wherein ( i ) we use that the eigenfunctions are orthonormal.
Revised: ( l, q ) lq (cid:19) −(cid:12)(cid:12)(cid:12)(cid:12)(cid:104) W ( d,q ) l (cid:105) i,k (cid:12)(cid:12)(cid:12)(cid:12) , where ( i ) follows from orthogonality of the eigenfunctions.

Original: We then chose M ( l,q ) = d B ( l, q ) /lq .
Revised: Hence, [Equation] satisfies the second condition in Lemma 18 for all ¯ L + 2 ≤ l < T .

Original: The extra l factor will be needed for the third assumption to hold.
Revised: The extra l factor is necessary for the third condition to hold.

Original: We now move to the third assumption.

Original: As in Lemma 17, let p ∈ N , p ≥ 2  ,then:
Revised: As in Lemma 18, let p ∈ N , p ≥ 2 . Then, for all i ̸ = j ,

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from hypercontractivityas in Lemma 18, ( ii ) from  the definition of C ( l, q, d )  .
Revised: where ( i ) follows from hypercontractivity in Lemma 19, ( ii ) from orthogonality of the eigenfunctions, and ( iii ) from the definition of C ( l, q, d ) as well as 1 ≤ γ ( S ) ≤ q .

Original:  ( iii )follows from, Lemma 9, the formula of B ( l, q ) in Equation (15),the formula (cid:0)
Revised: Step ( iv ) follows from Lemma 9, the definition of B ( l, q ) in Equation (15),

Original: [Equation]
Revised: [Equation]

Original: (cid:1)= ql (cid:0) ql (cid:1) , and thedefinition of M ( l,q ) .
Revised: (cid:1) , and the definition of M ( l,q ) .

Original: Finally, we can apply Lemma 17 and we get that for any p ∈ N , p >2 :
Revised: Since all conditions are satisfied, Lemma 18 yields for all p ∈ N , p >

Original: [Equation]
Revised: [Equation]

Original:  In particular, ifwe take p ≥ 2β +[Equation] 1− ¯ δ ) β − (¯ L + 2 − ¯ δ) , then we getthat:
Revised: , where c l, 1 , c l, 2 are positive constants that depend on l . In particular, if p ≥ 2 + 1 β + ¯ L + ¯ δ 1 − ¯ δ , then we get

Original: [Equation]
Revised: [Equation]

Revised: where c l is a positive constant that depends on l .

Original: We canthan take the union bound for all l = ¯ L + 1 , . ..
Revised: To avoid this dependence, we can take the union bound over all l = ¯ L + 2 , . .

Original: , T , and we get: Pr
Revised: , T − 1 :

Original: [Equation]
Revised: [Equation]

Original: where C ¯ Ldepends only  on ¯ L and T , which are fixed in our setting.
Revised: where c ′ ¯ L only depends on ¯ L and T , which are fixed in our setting.

Original: We remark that  ¯ L does notdepend on ℓ λ . 
Revised: Finally, additionally note that neither ¯ L nor T depend on ℓ λ . Proof of Lemma 17.

Original: Since we cannot control the dependency of c l on l  , with this strategy, we cannot bound ∥ W ( d,q ) l ∥ forall l at the same time, but only for a finite number of different values of l , i.e. all l ≤ T .
Revised: First, Lemma 15 yields that the diagonal elements of Q ( d,q ) l are just 1 for all l ∈ { T, . . . , q } . Hence, we define W ( d,q ) l [Equation]

Original:  We presentanother technique for the rest of the terms:
Revised: − I n , and decompose the kernel matrix as [Equation] We can hence apply the triangle inequality to bound the norm as follows: [Equation]

Original: Terms l ≥ T : for this section we can just bound the off-diagonal part using the Frobenius norm:
Revised: ,q ) l (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) ≤ (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) q (cid:88) l = T [Equation] q

Original: [Equation]
Revised: [Equation]

Revised: l (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original:  wherein ( i ) we use that max i,j [ W ( d,q ) l ] 2 i,j ≤ 1 / 4 n 2 .
Revised: [Equation] with probability ≥ 1 − cd − [Equation] where ( i ) uses non-negativity of the ξ ( q ) l from Equations (18) and (19) in Assumption 1, ( ii ) bounds the operator norm with the Frobenius norm, and ( iii ) additionally bounds the sum of the ξ ( q ) l using

Original: Now we compute the probability that this is indeed the case:
Revised: Equation (21) in Assumption 1. Step ( iv ) use a bound that we show in the remainder of the proof: with probability at least 1 − cd − 1 , we have ϖ l ≤ c ′ /q uniformly over all l ∈ { T, . . . , q } . We first bound ϖ l for a fixed T ≤ l ≤ q as follows:

Original: [Equation]
Revised: [Equation]

Original: wherein ( i ) we used Markov inequality (also known as Chebychev for second moments), and  in  ( ii )  we used rate ofn and the bound B ( l, q ) = (cid:0) ql (cid:1) ≥ ( q/l ) l  . 
Revised: where ( i ) follows from the union bound and the distribution of the off-diagonal entries in W ( d,q ) , and ( ii ) from the Markov inequality. In step ( iii ) , we use orthogonality of the eigenfunctions, as well as the fact that W ( d,q ) l and Q ( d,q ) l coincide on off-diagonal entries by construction. Step ( iv ) follows from Y S ( x ) 2 = 1 for all S and x ∈ {− 1 , 1 } d . Finally, step ( v ) applies Lemma 15.

Revised: Next, we use the union bound over all ϖ l of interest:

Revised: [Equation]

Revised: d

Revised: [Equation]

Original: Notice that the constant C does not depend on l .
Revised:  [Equation] (cid:0) ql [Equation]

Revised: (cid:123)(cid:122) (cid:125)

Original: Now, we can compute the probability that max i,j [ W ( d,q )  l ] 2 i,j ≤ 1 / 4 n 2 for all l ≥ T at the same time.
Revised: [Equation]   , where ( i ) substitutes l ′ = q − l , and ( ii ) uses q − ⌈ q/ 2 ⌉ − 1 ≤ ⌈ q/ 2 ⌉ as well as the fact that (cid:0) q q − l ′ (cid:1) = (cid:0) ql ′ (cid:1) .

Original: Thanks to union bound this happens with probability 1 − ∆ , where:
Revised: We bound both E 1 and E 2 using Assumption 1. For E 1 in particular, Equations (18), (19) and (21) imply that all ξ ( q ) l ≲ 1 . Hence, E

Original: [Equation]
Revised: [Equation]

Original:  wherein ( i ) we used the convergence of a classical series.
Revised: = [Equation] where ( i ) exploits that T is the value in { T, . .

Original: To conclude the proof, we remember that
Revised: . , ⌈ q/ 2 ⌉} the furthest away from q/ [Equation] and thus [Equation]

Original: T > 4 β + 4(¯ L + ¯ δ ) , hence, it is easy to verify that ∆ ∈ O ( d − 1 ) . Which implies that:
Revised: and ( ii ) follows from the classical lower bound on the binomial coefficient [Equation] For E 2 , we have

Original: [Equation]
Revised: [Equation]

Original: But thanks to Equation (28), the fact that max i,j [ W ( d,q ) l ] 2 i,j ≤ 1 / 4 n 2 implies that ∥ W ( d,q ) l ∥ ≤ 1 / 2 , hence:
Revised: 2 = [Equation] , where ( i ) uses the classical bound on the binomial coefficient, and ( ii ) Equation (20) in Assumption 1. Combining the bounds on E 1 and E 2 finally yields

Original: [Equation]
Revised: [Equation]

Original: Concluding the proof.
Revised: where ( i ) follows from the definition of T = ⌈ 4 + 4 ℓβ ⌉ in Assumption 1.

Original: The next statement is a non-asymptotic version of Proposition 3 from Ghorbani et al.
Revised: The next statement is a non-asymptotic version of Proposition 3 from Ghorbani et al.

Original: Lemma 17 (Graph argument) .
Revised: Lemma 18 (Graph argument) .

Original: Let W ∈ R n × n be a random matrix such that:
Revised: Let W ∈ R n × n be a random matrix that satisfies the following conditions:

Original: • W  i,i = 0 , ∀ i ∈  1 , . . . , n  .
Revised: [ W ] i,i = 0 , ∀ i ∈ { 1 , . . . , n } .

Original: • There existsa M > 0, such that for any i ̸=
Revised: There exists M > 0 such that, for all i, j, k ∈ { 1 , . . . , n } with i ̸

Original:  j ∈ 1 , . .
Revised: = j and j ̸

Original: . , n :
Revised: = k , we have

Original: [Equation]
Revised: [Equation]

Original: • There existsa l  such that, for any p ∈ N , p ≥ 2 :
Revised: There exists l ∈ N such that, for all p ∈ N , p ≥ 2 and all i, j ∈ { 1 , . . . , n } , i ̸ = j , we have

Original: [Equation]
Revised: [Equation]

Original: Then,we have that, for any p ∈ N , p > 2 :
Revised: Then, for all p ∈ N , p > 2 ,

Original: [Equation]
Revised: [Equation]

Original: where c 1 and c 2  depend on l .
Revised: where c 1 and c 2 are positive constants that depend on l .

Original: Proof.
Revised: Proof.

Original: Repeating the steps in the proof of Proposition 3 from Ghorbani et al.
Revised: Repeating the steps in the proof of Proposition 3 from Ghorbani et al.

Original: , we get
Revised: , we get

Original: [Equation]
Revised: [Equation]

Original: Note that the proof in Ghorbani et al.
Revised: Note that the proof in Ghorbani et al.

Original: (2021) assumes M to be in the order of d l .
Revised: (2021) assumes M to be in the order of d l .

Original: We get rid of thisassumption and keep M explicit.
Revised: We get rid of this assumption and keep M explicit.

Original: Furthermore, Ghorbani et al.
Revised: Furthermore, Ghorbani et al.

Original: (2021) use their Lemma 4 during theirproof, but we use our Lemma 18 instead.
Revised: (2021) use their Lemma 4 during their proof, but we use our Lemma 19 instead.

Original: Now, we use Markov inequality to get a high probability bound:
Revised: Ultimately, we apply the Markov inequality to get a high-probability bound:

Original: [Equation]
Revised: [Equation]

Original: Which, renaming the constants, yields the thesis.
Revised: Renaming the constants concludes the proof.

Original: Lemma 18 (Hypercontractivity) .
Revised: Lemma 19 (Hypercontractivity) .

Original: For any l ∈ N , for any p ≥ 2: [Equation] , x
Revised: For all l, q, d ∈ N and p ≥ 2 , we have

Original: [Equation]
Revised: [Equation]

Original: where Q ( d,q ) l ( x, x ′ ) is defined in Definition 2.
Revised: where Q ( d,q ) l ( x, x ′ ) is defined in Equation (30) .

Original: Proof.
Revised: Proof.

Original: Let  z be the entry-wise product between x and y . 
Revised: Let x, x ′ ∼ U ( {− 1 , 1 } d ) and let z be the entry-wise product of x and x ′ . Then, for all

Original: For any S  , Y S ( x ) Y S ( x ′ ) depends only on z :
Revised: S ⊆ { 1 , . . . , d } , Y S ( x ) Y S ( x ′ ) depends only on z :

Original: [Equation]
Revised: [Equation]

Original: Hence,given the definition of Q ( d,q ) l ( x, x ′ ), alsoQ ( d,q ) l ( x, x ′ ) depends only  on  z .
Revised: Hence, Q ( d,q ) l ( x, x ′ ) also only depends on x and x ′ via z .

Original: Finally, note thatz ∼ U ( {− 1 , 1 } d ) .
Revised: Furthermore, note that z ∼ U ( {− 1 , 1 } d ) .

Original: Hence, we can use hypercontractivity (Beckner, 1975), as for instance in Lemma 4from Misiakiewicz & Mei (2021), to conclude the proof.
Revised: Therefore, we can use hypercontractivity (Beckner, 1975) as for instance in Lemma 4 from Misiakiewicz & Mei (2021) to conclude the proof.

Original: D O PTIMAL R EGULARIZATION AND T RAINING E RRORD. O PTIMAL R EGULARIZATION
Revised: D O PTIMAL REGULARIZATION AND TRAINING ERROR D. O PTIMAL REGULARIZATION

Original: In the main text we often refer to the optimal regularization λ opt , however it is not clear how thisregularization term can be computed.
Revised: In the main text we often refer to the optimal regularization λ opt , defined as the minimizer of the risk Risk ( ˆ f λ ) .

Original: While we do not expect to compute it exactly, we show a wayto compute the rate of it. First we introduce some additional notation:
Revised: While we cannot calculate λ opt directly, we only need the rate ℓ λ opt such that max { λ opt , 1 } ∈ Θ

Original: [Equation]
Revised: [Equation]

Original: where the dependency with ℓ λ  is also through δ = ℓ − ℓ λ − 1 β
Revised: Furthermore, it is not a priori clear that such a ℓ λ opt minimizes the rate exponent of the risk in Theorem 1. The current subsection establishes that this is indeed the case, and provides a way to determine ℓ λ opt . We introduce some shorthand notation for the rate exponents in Theorem 1:

Original: [Equation]
Revised: [Equation]

Revised: We highlight that η b and η depend on ℓ λ also through δ =

Revised: ℓ − ℓ λ − 1 β − (cid:106) ℓ − ℓ λ − 1 β (cid:107) .

Original: Note that Intuitively, in the setting of Theorem 1, we havethat with high probability,
Revised: Hence, in the setting of Theorem 1, we have with high probability that

Original: [Equation]
Revised: [Equation]

Original: Risk [Equation] Hence, those quantities represent the exponent of the rate of variance, bias, and risk.
Revised: In the following, we view those quantities as functions of ℓ λ , with all other parameters fixed.

Original: We then define:
Revised: Next, we additionally define

Original: [Equation]
Revised: [Equation]

Original: Note that  ℓ λ min is a set that might have cardinality larger than one, but it cannot be empty.
Revised: First, we remark that Second, ¯ ℓ defines the scope of the minimization domain, guaranteeing that the constraint on L ∗ in

Original: This isbecause η is a continuous function, and [0 , ¯ ℓ ] is a closed set.
Revised: Theorem 1 holds for all candidate ℓ λ . Rate of optimal regularization ℓ λ opt vs. optimal rate ℓ λ min : Let ℓ λ opt be the rate of the optimal regularization strength such that max { λ opt , 1 } ∈ Θ( d ℓ λ opt ) . It is a priori not clear that ℓ λ opt minimizes η .

Original: The quantity ¯ ℓ defines the scope of our minimization searching domain, guaranteeing that theconstraint on L ∗ in Theorem 1 is respected.
Revised: However, Lemma 20 bridges the two quantities, and guarantees with high probability that the rate of λ opt minimizes the rate of the risk.

Original: For this reason, we are not considering the case wherethe optimal regularization parameter is larger than ¯ ℓ .
Revised: Lemma 20 (Optimal regularization and optimal rate) . In the setting of Theorem 1, assume ℓ > 0 , β ∈ (0 , 1) , ℓ σ ≥ − ¯ ℓ , L [Equation] ∈

Original: Yet, simple intuition reveals that such a strongregularization parameter yields models where the risk does not even vanish.
Revised: [Equation] ∩ N . Then, for d sufficiently large, with probability at least [Equation]

Original: Thus, we still claim thatlooking for the optimum within [0 , ¯ ℓ ] is sufficient.
Revised: [Equation] } there exists l ∈ ℓ λ min such that [Equation] Hence, we only need to obtain a minimum rate l ∈ ℓ λ min instead of ℓ λ opt .

Original: The next Lemma goes more in detail about the properties of η, η b , η v .
Revised: In order to propose a method for this, we first establish properties of η, η b , η v in the following lemma.

Original: Lemma 19 (Properties of η ) .
Revised: Lemma 21 (Properties of η ) .

Original: Assuming that ℓσ > 0 , β ∈ (0 , 1) ,L ∗ ∈ (cid:104) 1 , ⌈ ℓ  − 1 β ⌉ (cid:105) ∩ N . Let¯ ℓ = ℓ − 1 − β
Revised: Assume ℓ > 0 , β ∈ (0 , 1) , ℓ σ ≥ − ¯ [Equation]

Original: ( L ∗ − 1) , then the following holds:
Revised: , L ∗ [Equation]

Original:  η v ( l ; ℓ, ℓ σ , β ) is a monotonically non-increasing function with respect to l , and η b ( l ; ℓ, L ∗ , β )is a monotonically increasing one.2.
Revised: Over [0 , ¯ ℓ ] , η v ( · ; ℓ, ℓ σ , β ) is continuous and non-increasing, and η b ( · ; ℓ, L ∗ , β ) is continuous and strictly increasing.

Original: If there exists ˜ l ∈ (cid:2) 0 , ¯ ℓ (cid:3) such that η v (˜ l ; ℓ, ℓ σ ,β ) = η b (˜ l ; ℓ, L ∗ , β )  , then η  (˜ l ; ℓ, ℓ σ ,L ∗ , β ) =
Revised: ℓ λ min := arg min ℓ λ ∈ [ 0 , ¯ ℓ ] η ( ℓ λ ; ℓ, ℓ σ , L ∗ , β ) is a closed interval. If there exists ˜ l ∈ (cid:2) 0 , ¯ ℓ (cid:3) with η v (˜ l ; ℓ, ℓ σ , β ) =

Original: η min , otherwise, η (0; ℓ,ℓ σ , L ∗ , β )  =
Revised: η b (˜ l ; ℓ, L ∗ , β ) , then η min =

Original: ηmin .
Revised: η

Original: Let l be larger than l for any l ∈ ℓ λ min , then: [Equation]
Revised: (˜ l ; ℓ, ℓ σ , L ∗ , β ) .

Original: Let l be smaller than l for any l ∈ ℓ λ min , let c ℓ σ be a constant that depends only on ℓ σ , ℓ, β . Then, if η ( l ; ℓ, ℓ σ ,  β ) ≤ η min + c ℓ σ : [Equation]
Revised: Otherwise, η min = η (0; ℓ, ℓ σ , L ∗ , β ) . Every l ∈ [0 , ¯ ℓ ] with l ≥ l for all l ∈ ℓ λ min satisfies η ( l ; ℓ, ℓ σ , L ∗ , β ) =

Original: Rate of the optimal regularization parameter vs. optimal rate of the regularization parameter: for theoptimal regularization parameter let ℓ λ opt be such that λ opt ∈ Θ( d ℓ λ opt ) . Then it is not clear a priorithat ℓ λ opt minimizes η . However, Lemma 20 bridges the two terms and guarantees that this is the casewith high probability. Lemma 20 (Optimal regularization) .
Revised: η b ( l ; ℓ, L ∗ , β ) .

Original: In the same setting as in Theorem 1, for d sufficiently large, with probability at least 1 − cd − β min { ¯ δ, 1 − ¯ δ } , there exists l ∈ ℓ λ min such that
Revised: Let l ∈ [0 , ¯ ℓ ] with l ≤ l for all l ∈ ℓ λ min . If η ( l ; ℓ, ℓ σ , L ∗ , β ) − η min ≤ c where c > 0 is constant 5 and depends only on ℓ, ℓ σ , L ∗ , β , then

Original: [Equation]
Revised: [Equation]

Original: Finding  optimal parameters: now that we understand the importance of finding the minimal rate,we notice that Lemma 19 suggests a simple strategy to find a l ∈ ℓ λ min . We start by searching theintersection of η v and η b in [0 , ¯ ℓ ] : ifit is found, then the intersection point is optimal, otherwisejustchoose l = 0 .
Revised: Finding an optimal rate: Lemma 21 suggests a simple strategy to find a l ∈ ℓ λ min numerically: search the intersection of η v and η b in [0 , ¯ ℓ ] ; if found, then the intersection point is optimal, otherwise l =is optimal.

Original: Note that the intersection is unique and, if exists, it is  easy to compute since η v ismonotonically non-decreasing, and η b is monotonically increasing.
Revised: Note that, if the intersection point exists, it is unique and easy to numerically approximate, since η v is non-increasing, and η b is strictly increasing.

Original: Multiple minimizer issue: the last caveat here is that ℓ λ min  might contain multiple values, and, in thatcase we are not guaranteed that our algorithm retrieves the one which is the rate of λ opt . It mightinstead find a different l ∈ ℓ λ min .
Revised: Calculating numerical solutions: However, Lemma 21 also shows that ℓ λ min is an interval and thus might contain multiple values. In that case, the proposed strategy might not necessarily retrieve the rate of λ opt , but a different l ∈ ℓ λ min .

Original: Yet, Theorem 1 guarantees that both the optimally regularizedestimator and any estimator regularized with λ ∈ d l , l ∈ ℓ λ min, have a risk vanishing with the samerate n η min with high probability.
Revised: Yet, Theorem 1 guarantees that both the optimally regularized estimator and any estimator regularized with max { λ, 1 } ∈ d l for any l ∈ ℓ λ min have a risk vanishing with the same rate n η min with high probability.

Original: This made it possible to plot the rate of the optimally regularizedestimator in Figure 1a.
Revised: In particular, this allows us to exhibit the rate of the optimally regularized estimator in Figure 1a.

Original: Closed formula solutions: because of thephenomenon of multiple descent, it is hard to write bothℓ λ opt or β ∗ in an easily readable closed form.
Revised: Finally, because of the multiple descent phenomenon (see, for example, Figure 1), we do not expect either ℓ λ opt or β ∗ to attain an easily readable closed-form expression.

Original: Still, simple optimization algorithms make it easy toretrieve such quantities.
Revised: Nevertheless, simple optimization procedures allow us to calculate accurate numerical approximations.

Revised: Finally, we prove Lemmas 20 and 21.

Original: Proof of Lemma 20.
Revised: Proof of Lemma 20.

Original: Let ℓ λ opt bedefined such that:
Revised: Let ℓ λ opt be such that

Original: [Equation]
Revised: [Equation]

Original: Combining the bound onthe bias andthe variance in Theorem 1, we get that for any ℓ λ , λ ∈ Θ( ℓ λ ) ,
Revised: Combining the bounds on bias and variance from Theorem 1, we have [Equation] with probability ≥ 1 − cd − min {

Original: [Equation]
Revised: [Equation]

Revised: ) uniformly for all ℓ λ ∈ (cid:2) 0 , ¯ ℓ (cid:3) with max { λ, 1 } ∈ Θ( d ℓ λ ) .

Original: Throughout the rest of the proof we will write η ( ℓ λ ) to indicate η ( ℓ λ ; ℓ, ℓ σ , L ∗ , β  ) , for simplicity.
Revised: Throughout the remainder of this proof, we drop the dependencies on ℓ, ℓ σ , L ∗ , β in the notation of η and simply write η ( ℓ λ ) .

Original: We will also not repeat that each step  true with probability at least 1 − cd − min { ¯ δ, 1 − ¯ δ } , but we are implying it.
Revised: We further omit repeating that each step is true with probability at least 1 − cd − min { ¯ δ, 1 − ¯ δ } , but imply it throughout.

Original: Now we want to prove that η ( ℓ λopt ) is close to η min . Hence, we notice that Equation (29) applied to ℓ λ opt and any l ∈ ℓ λ min implies that:
Revised: The goal of the proof is to show that there exists l ∈ ℓ λ min sufficiently close to ℓ λ opt ; formally, we need to show that

Original: [Equation]
Revised: [Equation]

Original: where 0 < c 1 < c 2 are the constants hidden by Θ in Equation (29). Now, we use the optimality of  λ opt :
Revised: If this is the case, then the definition of ℓ λ opt yields the conclusion as follows:

Original: [Equation]
Revised: [Equation]

Original: To conclude the proof we need to somehow show that there exists l ∈ ℓ λ min sufficiently close to ℓ λ opt . Formally we need to show that: [Equation] [Equation]
Revised: where ( i ) uses Equation (39).

Original: Once we have shown that this is the case, we can simply use the definition of ℓ λ opt and compute:
Revised: Towards an auxiliary result, we first apply Equation (38) to ℓ λ opt and l ∈ ℓ λ min with max { λ, 1 } ∈ Θ( d l ) :

Original: [Equation]
Revised: [Equation]

Original: where  in( i ) we use Equation (30).
Revised: where c 1 , c 2 > 0 are the constants hidden by the Θ -notation in Equation (38). Next, the optimality of λ opt yields c 1 < c 2 , and [Equation]

Original: This concludes the proof.
Revised: This implies

Original: Now we need to show that  Equation (30)actually holds.
Revised: [Equation] where the second implication uses that η ( ℓ λ opt ) − η min ≥ 0 , since η min is the minimum rate. With this result, we finally focus on establishing Equation (39) which yields the claim of this lemma.

Original: To do so, we distinguishbetween three cases:
Revised: Lemma 21 shows that ℓ λ min is an interval; hence, we distinguish three cases:

Original:  ( ℓ 
Revised: Case ℓ λ opt ∈ ℓ λ min : Picking ℓ λ opt = l ∈ ℓ λ min directly yields d ℓ λ opt − ℓ λ opt = d 0 ∈ Θ(1) . Case ℓ λ opt < min ( ℓ λ min ) : Let l := min( ℓ λ min ) . Equation (40) yields η ( ℓ λ opt ) − η min ≤ c for any c >if d is large enough.

Original: λ
Revised: Hence, for d fixed but sufficiently large, Lemma 21 yields [Equation] Applying Equation (40), we get [Equation]

Original:  opt ∈ ℓ λ min ): in this case we simply notice that we can take l 
Revised: Now, since ℓ λ opt ≤ l , d ℓ λ opt − l ∈ O (1) . Furthermore,

Original:  ∈ 
Revised: [Equation] ) , where ( i ) follows from n ∈ Θ( d ℓ ) . Since both d l − ℓ λ opt and d [Equation]

Original: ℓ λ
Revised: d [Equation]

Original: min to be exactly ℓ λ opt ,hence:
Revised: l are in O (1) ,

Original: [Equation]
Revised: [Equation]

Original: ( ℓ λ opt < l , forany l ∈ ℓ λ min ): let l be min
Revised: which establishes Equation (39) for l ∈ ℓ λ min and thereby concludes this case.

Original: ( ℓ
Revised: Case ℓ

Original:  λmin ) .
Revised: [Equation] λ

Original: Then, thanks to Lemma 19, we have that,
Revised: [Equation]

Original: as long as η ( ℓ λ opt ) − η
Revised: : Let l :

Original: [Equation]
Revised: [Equation]

Original: Notice that for n sufficiently large, it is always the case that η ( ℓ λopt ) − η min ≤ c . Hence, we have that:
Revised: ( ℓ λ

Original: [Equation]
Revised: [Equation]

Original: Now, notice that since ℓ λ opt ≤ l , d ℓ λ opt − l ∈ O (1) , furthermore:
Revised: Then, Lemma 21 yields

Original: [Equation]
Revised: [Equation]

Original: wherein ( i ) we used that n ∈Θ( d ℓ ) .
Revised: where ( i ) uses that l ∈ ℓ λ min .

Original: Since both d l − ℓ λ opt and its inverse are ∈ O (1) , and l ∈ ℓ λ min , we can conclude that there exists a l ∈ ℓ λ min , such that:
Revised: Applying Equation (40), we get

Original: [Equation]
Revised: [Equation]

Original: ( ℓ λ opt > l , for any l ∈  ℓ λ min ): let l be max( ℓ λ min ) .Then, thanks to Lemma 19, we have that: [Equation]
Revised: Analogously to the previous case, this implies d ℓ λ opt − l ∈ O (1) , and the fact that ℓ λ opt > l implies d ℓ λ opt − l ∈ Ω(1) .

Original: Hence, we have that:
Revised: Together, this yields

Original: [Equation]
Revised: [Equation]

Original: Analogously to the previous case, this implies that d ℓ λ opt − l ∈ O (1) , and the fact that ℓ λ opt > l implies that d ℓ λ opt − l ∈ Ω(1) .

Original: Together, this means that there exists a l ∈ ℓ λ min such that: [Equation]
Revised: which establishes Equation (39) for l ∈ ℓ λ min and thereby concludes this case.

Original: Proof of Lemma 19.
Revised: Proof of Lemma 21.

Original: Throughout this proof wewill drop the dependencies  ℓ, ℓ σ , L ∗ , β in the notationof η, η v , η b , simply writing η ( l ) , η v ( l ) , η b ( l ) .
Revised: Throughout this proof, we drop the dependencies on ℓ, ℓ σ , L ∗ , β in the notation of η, η v , η b and simply write η ( l ) , η v ( l ) , η b ( l ) .

Original: Continuity: the proof start by showing that η v ( l ) and η b ( l ), are continuous functions of l .
Revised: Continuity and monotonicity (Item 1) We first show that η v ( l ) and η b ( l ) are continuous functions.

Original: η b ( l ) is an affine function of l , henceit is also continuous.
Revised: η b ( l ) is an affine function of l , hence continuous.

Original: η v ( l ) , however, has a hidden dependency with l through δ , which is not linear.
Revised: η v ( l ) , however, additionally depends on l via δ , which is not linear.

Original: Hence, to show that η v ( l ) is continuous, we need to show that min { δ, 1 − δ } is continuous. This is equivalent to show that:
Revised: Hence, to show that η v ( l ) is continuous, we need to show that min { δ, 1 − δ } is continuous. Consider the triangle wave function

Original: [Equation]
Revised: [Equation]

Original:  is  continuous.
Revised: which is well-known to be continuous.

Original: This is because ϖ
Revised: Because min { δ, 1 − δ } = ϖ

Original: [Equation]
Revised: [Equation]

Original:  β is a linear function of l . Finally, it is easy to see thatϖ is the triangular wave, and hence, it is indeed continuous. Both η v ( l)and η b ( l ) are then continuous functions, implying that η ( l ) is alsoncontinuous.
Revised: ℓ − l − 1 β (cid:17) , and ℓ − l − 1 β is a linear function of l , we get that η v ( l ) is also continuous.

Original: Derivative: now we can move our attention to the derivative of η v ( l ) and η b ( l ) . Again, it is easy to see that:
Revised: For monotonicity, we consider the derivatives of η v ( l ) and η b ( l ) . For η b ( l ) , we have

Original: [Equation]
Revised: [Equation]

Original: since η b isjust an affine function. This also implies that η b( l ) is a monotonically increasing function.
Revised: Since η b is an affine function, and 2 ℓ > 0 , this also implies that η b is strictly increasing.

Original: Again, the reasoning is a bit more complicated for η v . Here there are two cases:
Revised: For η v , we need to distinguish two cases:

Original: [Equation]
Revised: [Equation]

Original: Since η v is continuous, and the derivative is always ≤ 0 , then η v is monotonically non-increasingfunction.
Revised: Since η v is a continuous function with non-positive derivatives, this implies that η v is non-increasing.

Original: Concluding the proof of Item 1.
Revised: η decomposition (Item 3)

Original: Since η b is monotonically increasing, and η v monotonically non-increasing: if there exist ˜ l ∈ (cid:2) 0 , ¯ ℓ (cid:3) such that for any η b (˜ l ) =η b (˜ l ) , then, for any l > ˜ l, η b ( l ) > η v ( l ) , and for any l < ˜ l, η b ( l ) <
Revised: First assume that there exists ˜ l ∈ (cid:2) 0 , ¯ ℓ (cid:3) with η v (˜ l ) =

Original: η v ( l ) .
Revised: η b (˜ l ) .

Original: Notice that such ˜ l might not exist since it might be the case that η b ( l ) > η v( l ) for any l ∈ (cid:2) 0 , ¯ ℓ (cid:3) . η function decomposition: Notice now that η is defined as the maximum between η v and η b . Hence, if ˜ l exists, then:
Revised: Since η v is non-increasing and η b is strictly increasing,

Original: [Equation]
Revised: [Equation]

Original: otherwise.
Revised: otherwise.

Original: If however, ˜ l does not exist, it means that ηv ( l ) does not intersect ηb (  l  ) . 
Revised: In particular, η ( l ) > η (˜ l ) for all l > ˜ l as η b is strictly increasing, and η ( l ) ≥ η (˜ l ) for all l < ˜ l since η v is non-increasing. Combined, this yields η min = η ( ˜ l ; ℓ, ℓ σ , L ∗ , β ) . Next, assume ˜ l does not exist,

Revised: that is, η v ( l ) ̸ = η b ( l ) for all l ∈ (cid:2) 0 , ¯ ℓ (cid:3) .

Original: Thanks to continuity, either η b ( l ) >
Revised: Then, due to continuity, either η b ( l ) >

Original: η v ( l ) for any l ∈ (cid:2) 0 , ¯ ℓ (cid:3) , or η v ( l ) > η b ( l )  .
Revised: η v ( l ) for all l ∈ (cid:2) 0 , ¯ ℓ (cid:3) , or η v ( l ) > η b ( l ) for all l ∈ (cid:2) 0 , ¯ ℓ (cid:3) .

Original: A more accurate analysis, however, shows that the latter is not possible. To see this, we simply compute both η v (¯ ℓ ) and η b (¯ ℓ ) :
Revised: However, a closer analysis shows that the latter is not possible: the rates at the boundary ¯ ℓ are

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from the positivity of ℓ σ , and the positivity of min { δ, 1 − δ }  .
Revised: where ( i ) follows from the assumption ℓ σ ≥ − ¯ ℓ and the fact that min { δ, 1 − δ } ≥ 0 .

Original: Step ( ii ) follows from the assumption on L ∗ . Hence, in the case ˜ l does not exist,since the two functions are continuous η v ( l ) ≤ η b ( l ) , ∀ l ∈ [0 , ¯ ℓ ] . 
Revised: Hence, if ˜ l does not exist, η v ( l ) < η b ( l ) , ∀ l ∈ [0 , ¯ ℓ ] . In particular, η min is the minimum of the strictly increasing function η b ( l ) over [ (cid:2) 0 , ¯ ℓ (cid:3) , and therefore attained only at 0 .

Original: We can conclude that η ( l ) can be written as:
Revised: Lastly, combining both cases of ˜ l yields the following convenient expression:

Original: [Equation]
Revised: [Equation]