Original: L EARNING WITH CONVOLUTION AND POOLING OPERATIONS IN KERNEL METHODS
Revised: L EARNING WITH CONVOLUTION AND POOLING OPERATIONS IN KERNEL METHODS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent empirical work has shown that hierarchical convolutional kernels inspired by convolutional neural networks (CNNs) signiﬁcantly improve the performance of kernel methods in image classiﬁcation tasks.
Revised: Recent empirical work has shown that hierarchical convolutional kernels inspired by convolutional neural networks (CNNs) signiﬁcantly improve the performance of kernel methods in image classiﬁcation tasks.

Original: A widely accepted explanation for the success of these architectures is that they encode hypothesis classes that are suitable for natural images.
Revised: A widely accepted explanation for the success of these architectures is that they encode hypothesis classes that are suitable for natural images.

Original: However, understanding the precise interplay between approximation and generalization in convolutional architectures remains a challenge.
Revised: However, understanding the precise interplay between approximation and generalization in convolutional architectures remains a challenge.

Original: In this paper, we consider the stylized setting of covariates (image pixels) uniformly distributed on the hypercube and fully characterize the RKHS of kernels composed of single layers of convolution, pooling, and downsampling operations.
Revised: In this paper, we consider the stylized setting of covariates (image pixels) uniformly distributed on the hypercube and fully characterize the RKHS of kernels composed of single layers of convolution, pooling, and downsampling operations.

Original: We then study the gain in sample efﬁciency of kernel methods using these kernels over standard inner-product kernels.
Revised: We then study the gain in sample efﬁciency of kernel methods using these kernels over standard inner-product kernels.

Original: In particular, we show that 1) the convolution layer breaks the curse of dimensionality by restricting the RKHS to ‘local’ functions; 2) local pooling biases learning towards low-frequency functions, which are stable by small translations; 3) downsampling may modify the high-frequency eigenspaces but leaves the low-frequency part approximately unchanged.
Revised: In particular, we show that 1) the convolution layer breaks the curse of dimensionality by restricting the RKHS to ‘local’ functions; 2) local pooling biases learning towards low-frequency functions, which are stable by small translations; 3) downsampling may modify the high-frequency eigenspaces but leaves the low-frequency part approximately unchanged.

Original: Notably, our results quantify how choosing an architecture adapted to the target function leads to a large improvement in the sample complexity
Revised: Notably, our results quantify how choosing an architecture adapted to the target function leads to a large improvement in the sample complexity

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Convolutional neural networks (CNNs) have become essential elements of the deep learning toolbox, achieving state-of-the-art performance in many computer vision tasks (Krizhevsky et al., 2012;
Revised: Convolutional neural networks (CNNs) have become essential elements of the deep learning toolbox, achieving state-of-the-art performance in many computer vision tasks (Krizhevsky et al., 2012;

Original: He et al., 2016).
Revised: He et al., 2016).

Original: CNNs are constructed by stacking convolution and pooling layers, which were shown to be paramount to their empirical success (LeCun et al., 2015).
Revised: CNNs are constructed by stacking convolution and pooling layers, which were shown to be paramount to their empirical success (LeCun et al., 2015).

Original: A widely accepted hypothesis to explain their favorable properties is that these architectures successfully encode useful properties of natural images: locality and compositionality of the data, stability by local deformations, and translation invariance.
Revised: A widely accepted hypothesis to explain their favorable properties is that these architectures successfully encode useful properties of natural images: locality and compositionality of the data, stability by local deformations, and translation invariance.

Original: While some theoretical progress has been made in studying the approximation and generalization beneﬁts brought by convolution and pooling operations (Cohen &
Revised: While some theoretical progress has been made in studying the approximation and generalization beneﬁts brought by convolution and pooling operations (Cohen &

Original: Shashua, 2016a;b; Bietti, 2021), our mathematical understanding of the interaction between network architecture, image distribution, and efﬁcient learning remains limited.
Revised: Shashua, 2016a;b; Bietti, 2021), our mathematical understanding of the interaction between network architecture, image distribution, and efﬁcient learning remains limited.

Original: Consider x ∈ R d an input signal, which we can think of as a grayscale pixel representation of an image.
Revised: Consider x ∈ R d an input signal, which we can think of as a grayscale pixel representation of an image.

Original: For mathematical convenience, we will consider one-dimensional images with cyclic convention x d
Revised: For mathematical convenience, we will consider one-dimensional images with cyclic convention x d

Original: + i := x i , and denote x ( k ) =
Revised: + i := x i , and denote x ( k ) =

Original: ( x k , x k +1 , . . .
Revised: ( x k , x k +1 , . . .

Original: , x k + q − 1 ) the k -th patch of the signal x , k ∈ [ d ] , with patch size q ≤ d .
Revised: , x k + q − 1 ) the k -th patch of the signal x , k ∈ [ d ] , with patch size q ≤ d .

Original: Most of our results can be extended to two-dimensional images.
Revised: Most of our results can be extended to two-dimensional images.

Original: We further consider a simple convolutional neural network composed of a single convolution layer followed by local average pooling and downsampling.
Revised: We further consider a simple convolutional neural network composed of a single convolution layer followed by local average pooling and downsampling.

Original: The network ﬁrst computes the nonlinear convolution of N ﬁlters w 1 , . .
Revised: The network ﬁrst computes the nonlinear convolution of N ﬁlters w 1 , . .

Original: w N ∈ R q with the image patches x ( k ) .
Revised: w N ∈ R q with the image patches x ( k ) .

Original: The outputs of the convolution operation σ ( (cid:104) w i , x ( k ) (cid:105) ) are then averaged locally over segments of length ω (local average pooling).
Revised: The outputs of the convolution operation σ ( (cid:104) w i , x ( k ) (cid:105) ) are then averaged locally over segments of length ω (local average pooling).

Original: This pooling operation is followed by downsampling which extracts one out of every ∆ output coordinates (for simplicity, ∆ is assumed to be a divisor of d ).
Revised: This pooling operation is followed by downsampling which extracts one out of every ∆ output coordinates (for simplicity, ∆ is assumed to be a divisor of d ).

Original: Finally, the results are combined linearly using coefﬁcients ( a ik ) i ∈ [ N ] ,k ∈ [ d/ ∆] : f CNN ( x ; a , Θ ) =
Revised: Finally, the results are combined linearly using coefﬁcients ( a ik ) i ∈ [ N ] ,k ∈ [ d/ ∆] : f CNN ( x ; a , Θ ) =

Original: (cid:114) ∆
Revised: (cid:114) ∆

Original: Nωd (cid:88) i ∈
Revised: Nωd (cid:88) i ∈

Original: [ N ] (cid:88) k ∈ [ d/ ∆] a ik (cid:88) s ∈ [ ω ] σ (cid:0) (cid:104) w i , x ( k ∆+ s ) (cid:105) (cid:1) .
Revised: [ N ] (cid:88) k ∈ [ d/ ∆] a ik (cid:88) s ∈ [ ω ] σ (cid:0) (cid:104) w i , x ( k ∆+ s ) (cid:105) (cid:1) .

Original: (CNN-AP-DS)
Revised: (CNN-AP-DS)

Original: Note that pooling and downsampling operations are often tied together in the literature.
Revised: Note that pooling and downsampling operations are often tied together in the literature.

Original: However in this work we will treat these two operations separately.
Revised: However in this work we will treat these two operations separately.

Original: In the formula above, different values for q, ω, ∆ lead to different architectures with vastly different behaviors.
Revised: In the formula above, different values for q, ω, ∆ lead to different architectures with vastly different behaviors.

Original: For example, when q = ∆ =
Revised: For example, when q = ∆ =

Original: d and ω = 1 , we recover a two-layer fully-connected neural network f FC ( x ; a , Θ )
Revised: d and ω = 1 , we recover a two-layer fully-connected neural network f FC ( x ; a , Θ )

Original: =
Revised: =

Original: N − 1 / 2 (cid:80) i ∈
Revised: N − 1 / 2 (cid:80) i ∈

Original: [ N ] a i σ ( (cid:104) w i , x (cid:105) ) which has the universal approximation property at large N .
Revised: [ N ] a i σ ( (cid:104) w i , x (cid:105) ) which has the universal approximation property at large N .

Original: When ω = ∆ = 1 and q < d , the network is “locally connected” f LC ( x ; a , Θ ) =
Revised: When ω = ∆ = 1 and q < d , the network is “locally connected” f LC ( x ; a , Θ ) =

Original: N − 1 / 2 (cid:80) i ∈
Revised: N − 1 / 2 (cid:80) i ∈

Original: [ N ] ,k ∈ [ d ] a ik σ ( (cid:104) w i , x ( k ) (cid:105) ) , and not a universal approximator anymore: however, f LC vastly outperforms f FC in some cases (Li et al., 2020).
Revised: [ N ] ,k ∈ [ d ] a ik σ ( (cid:104) w i , x ( k ) (cid:105) ) , and not a universal approximator anymore: however, f LC vastly outperforms f FC in some cases (Li et al., 2020).

Original: For ω > 1 , local pooling enables learning functions that are locally invariant by translations more efﬁciently than without pooling.
Revised: For ω > 1 , local pooling enables learning functions that are locally invariant by translations more efﬁciently than without pooling.

Original: For ω = d (global pooling), the network only ﬁts functions fully invariant by cyclic translations.
Revised: For ω = d (global pooling), the network only ﬁts functions fully invariant by cyclic translations.

Original: The aim of this paper is to formalize and quantify the trade-off between the target function class and the statistical efﬁciency brought by these different architectures.
Revised: The aim of this paper is to formalize and quantify the trade-off between the target function class and the statistical efﬁciency brought by these different architectures.

Original: As a concrete ﬁrst step in this direction, we consider kernel models that are naturally associated with the convolutional neural networks (CNN-AP-DS) through the neural tangent kernel perspective (Daniely et al., 2016; Jacot et al., 2018).
Revised: As a concrete ﬁrst step in this direction, we consider kernel models that are naturally associated with the convolutional neural networks (CNN-AP-DS) through the neural tangent kernel perspective (Daniely et al., 2016; Jacot et al., 2018).

Original: Kernel methods have the advantage of 1) being tractable—leaving the computational issue of learning CNNs aside; 2) having well-understood approximation and generalization properties, which depends on the eigendecomposition of the kernel and the alignment between the target function and associated RKHS (Caponnetto & De Vito, 2007;
Revised: Kernel methods have the advantage of 1) being tractable—leaving the computational issue of learning CNNs aside; 2) having well-understood approximation and generalization properties, which depends on the eigendecomposition of the kernel and the alignment between the target function and associated RKHS (Caponnetto & De Vito, 2007;

Original: Wainwright, 2019) (see Appendices
Revised: Wainwright, 2019) (see Appendices

Original: B and C for background).
Revised: B and C for background).

Original: While kernel models only describe neural networks in the lazy training regime (Chizat et al., 2019; Du et al., 2019b;a; Allen-Zhu et al., 2019; Zou et al., 2018) and miss important properties of deep learning, such as feature learning, we see that architecture choice is already crucial to enable efﬁcient learning of ‘image-like’ functions in the ﬁxed-feature regime.
Revised: While kernel models only describe neural networks in the lazy training regime (Chizat et al., 2019; Du et al., 2019b;a; Allen-Zhu et al., 2019; Zou et al., 2018) and miss important properties of deep learning, such as feature learning, we see that architecture choice is already crucial to enable efﬁcient learning of ‘image-like’ functions in the ﬁxed-feature regime.

Original: Neural tangent kernels are obtained by linearizing the associated neural networks.
Revised: Neural tangent kernels are obtained by linearizing the associated neural networks.

Original: Here we consider the tangent kernel associated to the network f CNN (c.f. Appendix A.2 for a detailed derivation):
Revised: Here we consider the tangent kernel associated to the network f CNN (c.f. Appendix A.2 for a detailed derivation):

Original: [Equation]
Revised: [Equation]

Original: where h : R → R is related to the activation function σ in (CNN-AP-DS).
Revised: where h : R → R is related to the activation function σ in (CNN-AP-DS).

Original: As a linearization of
Revised: As a linearization of

Original: CNNs, the kernel (CK-AP-DS) inherits some of the favorable properties of convolution, pooling, and downsampling operations.
Revised: CNNs, the kernel (CK-AP-DS) inherits some of the favorable properties of convolution, pooling, and downsampling operations.

Original: Indeed, a line of work (Mairal et al., 2014; Mairal, 2016; Arora et al., 2019;
Revised: Indeed, a line of work (Mairal et al., 2014; Mairal, 2016; Arora et al., 2019;

Original: Li et al., 2019; Shankar et al., 2020) showed that, though performing slightly worse than CNNs, such (hierarchical) convolutional kernels have empirically outperformed the former state-of-the-art kernels.
Revised: Li et al., 2019; Shankar et al., 2020) showed that, though performing slightly worse than CNNs, such (hierarchical) convolutional kernels have empirically outperformed the former state-of-the-art kernels.

Original: For instance, these kernels achieved test accuracy around 87% − 90% on
Revised: For instance, these kernels achieved test accuracy around 87% − 90% on

Original: CIFAR-10, against 79 .
Revised: CIFAR-10, against 79 .

Original: 6% for the best former unsupervised feature-extraction method (Coates et al., 2011) (currently, the state-of-the-art CNNs can achieve test accuracy 99% ).
Revised: 6% for the best former unsupervised feature-extraction method (Coates et al., 2011) (currently, the state-of-the-art CNNs can achieve test accuracy 99% ).

Original: In this paper, we will further consider a stylized setting with input signal distribution x ∼ Unif( Q d ) (uniform distribution over Q d := {− 1 , +1 } d the discrete hypercube in d dimensions).
Revised: In this paper, we will further consider a stylized setting with input signal distribution x ∼ Unif( Q d ) (uniform distribution over Q d := {− 1 , +1 } d the discrete hypercube in d dimensions).

Original: This simple choice allows for a complete characterization of the eigendecomposition of H CK ω, ∆ , thanks to all patches having same marginal distribution x ( k ) ∼ Unif( Q q ) .
Revised: This simple choice allows for a complete characterization of the eigendecomposition of H CK ω, ∆ , thanks to all patches having same marginal distribution x ( k ) ∼ Unif( Q q ) .

Original: We will be particularly interested in four speciﬁc choices of ( q, ω, ∆) in (CK-AP-DS):
Revised: We will be particularly interested in four speciﬁc choices of ( q, ω, ∆) in (CK-AP-DS):

Original: [Equation]
Revised: [Equation]

Original: These kernels are respectively the neural tangent kernels of a fully-connected network f FC (FC), a convolutional network f LC (CK), a convolutional network followed by local average pooling (CKAP) and a convolutional network followed by global pooling (CK-GP).
Revised: These kernels are respectively the neural tangent kernels of a fully-connected network f FC (FC), a convolutional network f LC (CK), a convolutional network followed by local average pooling (CKAP) and a convolutional network followed by global pooling (CK-GP).

Original: We will further be interested in (CK-GP) with patch size q = d , which we denote H FCGP :
Revised: We will further be interested in (CK-GP) with patch size q = d , which we denote H FCGP :

Original: this corresponds to a convolutional kernel with full-size patches q = d , followed by global pooling.
Revised: this corresponds to a convolutional kernel with full-size patches q = d , followed by global pooling.

Original: In this paper, we ﬁrst characterize the reproducing kernel Hilbert space (RKHS) of these convolutional kernels, and then investigate their generalization properties in the regression setup.
Revised: In this paper, we ﬁrst characterize the reproducing kernel Hilbert space (RKHS) of these convolutional kernels, and then investigate their generalization properties in the regression setup.

Original: More speciﬁcally, assume { ( x i , y i ) } i ≤ n are n i.i.d.
Revised: More speciﬁcally, assume { ( x i , y i ) } i ≤ n are n i.i.d.

Original: samples with x i ∼ Unif( Q d ) and y i =
Revised: samples with x i ∼ Unif( Q d ) and y i =

Original: f (cid:63) ( x i )
Revised: f (cid:63) ( x i )

Original: + ε i .
Revised: + ε i .

Original: Here f (cid:63) ∈
Revised: Here f (cid:63) ∈

Original: L 2 ( Q d ) and ( ε i ) i ≤ n are independent errors with mean zero and variance bounded by σ 2 ε .
Revised: L 2 ( Q d ) and ( ε i ) i ≤ n are independent errors with mean zero and variance bounded by σ 2 ε .

Original: We will focus on the generalization error of kernel ridge regression (KRR) (see Appendix B. for general kernel methods).
Revised: We will focus on the generalization error of kernel ridge regression (KRR) (see Appendix B. for general kernel methods).

Original: In particular, given a kernel function H : Q d × Q d → R and a regularization parameter λ ≥ 0 , the KRR estimator is the solution of the tractable convex problem
Revised: In particular, given a kernel function H : Q d × Q d → R and a regularization parameter λ ≥ 0 , the KRR estimator is the solution of the tractable convex problem

Original: [Equation]
Revised: [Equation]

Original: where H is the RKHS associated to H with RKHS norm (cid:107)·(cid:107) H .
Revised: where H is the RKHS associated to H with RKHS norm (cid:107)·(cid:107) H .

Original: We denote the test error with square loss by R ( f (cid:63) , ˆ f λ ) =
Revised: We denote the test error with square loss by R ( f (cid:63) , ˆ f λ ) =

Original: E x { ( f (cid:63) ( x ) − ˆ f λ ( x )) 2 } .
Revised: E x { ( f (cid:63) ( x ) − ˆ f λ ( x )) 2 } .

Original: We will sometimes consider the expected test error
Revised: We will sometimes consider the expected test error

Original: E ε { R ( f (cid:63) , ˆ f λ ) } , where expectation is taken with respect to noise ε = ( ε i ) i ≤ n in the training data.
Revised: E ε { R ( f (cid:63) , ˆ f λ ) } , where expectation is taken with respect to noise ε = ( ε i ) i ≤ n in the training data.

Original: The generalization properties of the kernels H FC and H FCGP were recently studied in Mei et al.
Revised: The generalization properties of the kernels H FC and H FCGP were recently studied in Mei et al.

Original: (2021b);
Revised: (2021b);

Original: Bietti et al.
Revised: Bietti et al.

Original: In particular, they showed that global pooling (kernel H FCGP ) leads to a gain of a factor d in sample complexity when ﬁtting cyclic invariant functions, but still suffers from the curse of dimensionality ( H FC GP only ﬁts very smooth functions in high-dimension).
Revised: In particular, they showed that global pooling (kernel H FCGP ) leads to a gain of a factor d in sample complexity when ﬁtting cyclic invariant functions, but still suffers from the curse of dimensionality ( H FC GP only ﬁts very smooth functions in high-dimension).

Original: More precisely,
Revised: More precisely,

Original: Mei et al.
Revised: Mei et al.

Original: (2021b)
Revised: (2021b)

Original: considered the high-dimensional framework of Mei et al.
Revised: considered the high-dimensional framework of Mei et al.

Original: (2021a)
Revised: (2021a)

Original: and showed the following: KRR with H FC requires n ≈ d (cid:96) samples to ﬁt degree- (cid:96) cyclic polynomials, while KRR with H FCGP only needs n ≈ d (cid:96) − 1 .
Revised: and showed the following: KRR with H FC requires n ≈ d (cid:96) samples to ﬁt degree- (cid:96) cyclic polynomials, while KRR with H FCGP only needs n ≈ d (cid:96) − 1 .

Original: To enable milder dependence on the dimension d , further structural assumptions on the kernel and the target function should be considered (for instance, in this paper, we use the kernel H CK and consider ‘local’ functions).
Revised: To enable milder dependence on the dimension d , further structural assumptions on the kernel and the target function should be considered (for instance, in this paper, we use the kernel H CK and consider ‘local’ functions).

Original: 1. S UMMARY OF MAIN RESULTS
Revised: 1. S UMMARY OF MAIN RESULTS

Original: Our contributions are two-fold.
Revised: Our contributions are two-fold.

Original: First, we describe the RKHS associated to the convolutional kernel (CK-AP-DS) in the stylized setting x ∼ Unif( Q d ) , which provides a fully explicit illustration of the roles of convolution, pooling and downsampling operations in learning speciﬁc classes of functions.
Revised: First, we describe the RKHS associated to the convolutional kernel (CK-AP-DS) in the stylized setting x ∼ Unif( Q d ) , which provides a fully explicit illustration of the roles of convolution, pooling and downsampling operations in learning speciﬁc classes of functions.

Original: Second, based on the eigendecomposition of the kernels, we provide generalization bounds on kernel ridge regression both in the ﬁxed and high-dimension settings.
Revised: Second, based on the eigendecomposition of the kernels, we provide generalization bounds on kernel ridge regression both in the ﬁxed and high-dimension settings.

Original: In particular, we quantify the gain in sample complexity achieved by different architectures when learning target functions with corresponding structures.
Revised: In particular, we quantify the gain in sample complexity achieved by different architectures when learning target functions with corresponding structures.

Original: Let us deﬁne the q -local function class L 2 ( Q d , Loc q ) and the cyclic q -local function class
Revised: Let us deﬁne the q -local function class L 2 ( Q d , Loc q ) and the cyclic q -local function class

Original: L 2 ( Q d , CycLoc q ) (subspace of L 2 ( Q d , Loc q ) consisting of cyclic-invariant functions) as follow:
Revised: L 2 ( Q d , CycLoc q ) (subspace of L 2 ( Q d , Loc q ) consisting of cyclic-invariant functions) as follow:

Original: [Equation]
Revised: [Equation]

Original: CycLoc
Revised: CycLoc

Original: [Equation]
Revised: [Equation]

Original: k ∈ [ d ]
Revised: k ∈ [ d ]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (CYC-LOC)
Revised: (CYC-LOC)

Original: We summarize below some of the insights that follow from our results.
Revised: We summarize below some of the insights that follow from our results.

Original: One-layer convolutional layer.
Revised: One-layer convolutional layer.

Original: The RKHS of H CK is constituted of q -local functions
Revised: The RKHS of H CK is constituted of q -local functions

Original: L 2 ( Q d , Loc q ) .
Revised: L 2 ( Q d , Loc q ) .

Original: Furthermore, the decay of the eigenvalues of H CK is controlled by a q -dimensional kernel, instead of a d -dimensional kernel for H FC .
Revised: Furthermore, the decay of the eigenvalues of H CK is controlled by a q -dimensional kernel, instead of a d -dimensional kernel for H FC .

Original: In particular, this implies that the test error of kernel ridge regression decays at rate n − O
Revised: In particular, this implies that the test error of kernel ridge regression decays at rate n − O

Original: (1 /q ) instead of n − O (1 /d ) , when the target function is in L 2 ( Q d , Loc q ) .
Revised: (1 /q ) instead of n − O (1 /d ) , when the target function is in L 2 ( Q d , Loc q ) .

Original: Hence, when q (cid:28) d , kernel methods with convolutional kernel H CK breaks the curse of dimensionality .
Revised: Hence, when q (cid:28) d , kernel methods with convolutional kernel H CK breaks the curse of dimensionality .

Original: Average pooling.
Revised: Average pooling.

Original: The RKHS of H CK ω is still constituted of q -local functions f ∈ L 2 ( Q d , Loc q ) , but penalizes differently the frequency components f j ( x ) by reweighting their eigenspaces by a factor κ j , where f j ( x )
Revised: The RKHS of H CK ω is still constituted of q -local functions f ∈ L 2 ( Q d , Loc q ) , but penalizes differently the frequency components f j ( x ) by reweighting their eigenspaces by a factor κ j , where f j ( x )

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: k ∈ [ d ] ρ kj f ( t k · x ) with ρ j = e 2 iπj d and t k · x = ( x k +1 , . . .
Revised: k ∈ [ d ] ρ kj f ( t k · x ) with ρ j = e 2 iπj d and t k · x = ( x k +1 , . . .

Original: , x d , x 1 , . . .
Revised: , x d , x 1 , . . .

Original: , x k ) denotes the k -shift.
Revised: , x k ) denotes the k -shift.

Original: As ω increases, local pooling penalizes more and more heavily the high-frequency components ( κ j (cid:28) 1 ), while making lowfrequency components statistically easier to learn ( κ j (cid:29) 1 ).
Revised: As ω increases, local pooling penalizes more and more heavily the high-frequency components ( κ j (cid:28) 1 ), while making lowfrequency components statistically easier to learn ( κ j (cid:29) 1 ).

Original: For global pooling ω
Revised: For global pooling ω

Original: = d ,
Revised: = d ,

Original: H CKGP only learns cyclic local functions L 2 ( Q d , CycLoc q ) and enjoy a factor d gain in statistical complexity compared to H CK .
Revised: H CKGP only learns cyclic local functions L 2 ( Q d , CycLoc q ) and enjoy a factor d gain in statistical complexity compared to H CK .

Original: To ﬁt a degree (cid:96) polynomial
Revised: To ﬁt a degree (cid:96) polynomial

Original: H FC
Revised: H FC

Original: H FCGP
Revised: H FCGP

Original: H CK
Revised: H CK

Original: H CK ω
Revised: H CK ω

Original: H CKGP
Revised: H CKGP

Original: Sample complexity d (cid:96) d (cid:96) −dq (cid:96) −dq (cid:96) − 1 /ω q (cid:96) −
Revised: Sample complexity d (cid:96) d (cid:96) −dq (cid:96) −dq (cid:96) − 1 /ω q (cid:96) −

Original: Downsampling.
Revised: Downsampling.

Original: When ∆ ≤ ω , downsampling after average pooling leaves the low-frequency eigenspaces of H CK ω stable.
Revised: When ∆ ≤ ω , downsampling after average pooling leaves the low-frequency eigenspaces of H CK ω stable.

Original: In particular, the downsampling operation does not modify the statistical complexity of learning low-frequency functions in one-layer kernels, while being potentially beneﬁcial in further layers in deep convolutional kernels.
Revised: In particular, the downsampling operation does not modify the statistical complexity of learning low-frequency functions in one-layer kernels, while being potentially beneﬁcial in further layers in deep convolutional kernels.

Original: In Table 1, we report our high-dimensional predictions for the sample complexity of learning q -local cyclic-invariant polynomials of degree (cid:96) .
Revised: In Table 1, we report our high-dimensional predictions for the sample complexity of learning q -local cyclic-invariant polynomials of degree (cid:96) .

Original: These results are proven within the framework of Mei et al.
Revised: These results are proven within the framework of Mei et al.

Original: (2021a) (see Appendix C for background).
Revised: (2021a) (see Appendix C for background).

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: We discuss related work in Section 1.2.
Revised: We discuss related work in Section 1.2.

Original: In Section 2, we present our main results on convolutional kernels and describe precisely the roles of convolution, pooling and downsampling operations.
Revised: In Section 2, we present our main results on convolutional kernels and describe precisely the roles of convolution, pooling and downsampling operations.

Original: We brieﬂy mention the multilayer case in Section 2.4.
Revised: We brieﬂy mention the multilayer case in Section 2.4.

Original: Finally, we present a numerical simulation on synthetic data in Section 3 and conclude in Section 4.
Revised: Finally, we present a numerical simulation on synthetic data in Section 3 and conclude in Section 4.

Original: Some details and discussions are deferred to Appendix A.
Revised: Some details and discussions are deferred to Appendix A.

Original: 1. R ELATED WORK
Revised: 1. R ELATED WORK

Original: Convolutional kernels have been considered in Mairal et al.
Revised: Convolutional kernels have been considered in Mairal et al.

Original: (2014); Mairal (2016);
Revised: (2014); Mairal (2016);

Original: Li et al.
Revised: Li et al.

Original: Shankar et al.
Revised: Shankar et al.

Original: Bietti (2021); Thiry et al. (2021).
Revised: Bietti (2021); Thiry et al. (2021).

Original: In particular, they showed that these architectures achieve good results in image classiﬁcation ( 90% accuracy on Cifar10) and that pooling and downsampling were necessary for their good performance (Li et al., 2019).
Revised: In particular, they showed that these architectures achieve good results in image classiﬁcation ( 90% accuracy on Cifar10) and that pooling and downsampling were necessary for their good performance (Li et al., 2019).

Original: The generalization error of kernel ridge regression (KRR) has been well-studied in both the ﬁxed dimension regime (Wainwright, 2019, Chap. 13), Caponnetto & De Vito (2007) and the highdimensional regimes El Karoui (2010);
Revised: The generalization error of kernel ridge regression (KRR) has been well-studied in both the ﬁxed dimension regime (Wainwright, 2019, Chap. 13), Caponnetto & De Vito (2007) and the highdimensional regimes El Karoui (2010);

Original: Liang et al.
Revised: Liang et al.

Original: ; Ghorbani et al.
Revised: ; Ghorbani et al.

Original: (2020; 2021); Mei et al.
Revised: (2020; 2021); Mei et al.

Original: (2021b).
Revised: (2021b).

Original: These results show that the generalization error depends on the eigenvalues and eigenfunctions of the kernel, and the alignment of the kernel with the target function.
Revised: These results show that the generalization error depends on the eigenvalues and eigenfunctions of the kernel, and the alignment of the kernel with the target function.

Original: Recently, a few theoretical work have considered the generalization properties of invariant kernels and convolutional kernels (Scetbon & Harchaoui, 2020; Mei et al., 2021b;
Revised: Recently, a few theoretical work have considered the generalization properties of invariant kernels and convolutional kernels (Scetbon & Harchaoui, 2020; Mei et al., 2021b;

Original: Bietti et al., 2021; Favero et al., 2021).
Revised: Bietti et al., 2021; Favero et al., 2021).

Original: In particular, Mei et al.
Revised: Favero et al.

Original: (2021b) consider  convolutional kernel with  global pooling and full-size patches q = d , and show a gain of factor d in sample complexity when learning cyclic functions, compared to inner-product kernels.
Revised: (2021) consider a one-layer convolutional kernel with and without global pooling and derived asymptotic rates in n , the number of samples, in a student-teacher scenario using statistical physics heuristics and a Gaussian equivalence conjecture.

Original: Bietti et al.
Revised: In particular, they show that locality rather than translation-invariance breaks the curse of dimensionality.

Original: (2021) considers additionally kernels that are stable with respect to local deformations, and similarly quantify the sample complexity gain.
Revised: Here, our goal is different: we derive mathematically rigorous quantitative bounds that give separation in generalization power between different architectures.

Original: See Malach & Shalev-Shwartz (2020); Li et al.
Revised: See Malach & Shalev-Shwartz (2020); Li et al.

Original: (2020) for more theoretical results on the separation between convolutional and fully connected neural networks, and Boureau et al.
Revised: (2020) for more theoretical results on the separation between convolutional and fully connected neural networks, and Boureau et al.

Original: ; Cohen &
Revised: ; Cohen &

Original: Shashua (2016b) for the inductive bias of pooling operations in convolutional neural networks.
Revised: Shashua (2016b) for the inductive bias of pooling operations in convolutional neural networks.

Original:  M AIN RESULTS
Revised:  M AIN RESULTS

Original: We start by introducing some basic background on functions on the hypercube and eigendecomposition of kernel operators in Section 2.1.
Revised: We start by introducing some basic background on functions on the hypercube and eigendecomposition of kernel operators in Section 2.1.

Original: We ﬁrst consider a kernel with a single convolution layer in
Revised: We ﬁrst consider a kernel with a single convolution layer in

Original: Section 2.2, and characterize its eigendecomposition and generalization properties.
Revised: Section 2.2, and characterize its eigendecomposition and generalization properties.

Original: We then show how these results are modiﬁed when applying local average pooling and downsampling in Section 2.3.
Revised: We then show how these results are modiﬁed when applying local average pooling and downsampling in Section 2.3.

Original: Finally, we brieﬂy discuss multilayer convolutional kernels in Section 2.4.
Revised: Finally, we brieﬂy discuss multilayer convolutional kernels in Section 2.4.

Original: 2. F UNCTIONS ON THE HYPERCUBE AND EIGENDECOMPOSITION 
Revised: 2. F UNCTIONS ON THE HYPERCUBE AND EIGENDECOMPOSITION OF KERNEL OPERATORS

Original: Recall that we work on the d -dimensional hypercube Q d := {− 1 , +1 } d .
Revised: Recall that we work on the d -dimensional hypercube Q d := {− 1 , +1 } d .

Original: Let L 2 ( Q d ) =
Revised: Let L 2 ( Q d ) =

Original: L 2 ( Q d , Unif) be the 2 d -dimensional vector space of all functions f : Q d → R , with scalar product (cid:104) f, g (cid:105)
Revised: L 2 ( Q d , Unif) be the 2 d -dimensional vector space of all functions f : Q d → R , with scalar product (cid:104) f, g (cid:105)

Original: L 2 :=
Revised: L 2 :=

Original: E x ∼ Unif( Q d ) [ f ( x ) g ( x )] .
Revised: E x ∼ Unif( Q d ) [ f ( x ) g ( x )] .

Original: Let (cid:107)·(cid:107)
Revised: Let (cid:107)·(cid:107)

Original: L 2 be the norm associated with the scaler product.
Revised: L 2 be the norm associated with the scaler product.

Original: We introduce the set of Fourier functions { Y ( d ) S ( x ) } S ⊆
Revised: We introduce the set of Fourier functions { Y ( d ) S ( x ) } S ⊆

Original: [ d ] which forms an orthonormal basis of L 2 ( Q d ) .
Revised: [ d ] which forms an orthonormal basis of L 2 ( Q d ) .

Original: For any subset S ⊆
Revised: For any subset S ⊆

Original: [ d ] , the Fourier function is deﬁned as Y ( d ) S ( x ) :=
Revised: [ d ] , the Fourier function is deﬁned as Y ( d ) S ( x ) :=

Original: (cid:81)
Revised: (cid:81)

Original: i ∈
Revised: i ∈

Original: S x i with the convention that Y ( d ) ∅ :
Revised: S x i with the convention that Y ( d ) ∅ :

Original: = 1 (it is easy to verify that (cid:104) Y ( d ) S , Y ( d ) S (cid:48) (cid:105)
Revised: = 1 (it is easy to verify that (cid:104) Y ( d ) S , Y ( d ) S (cid:48) (cid:105)

Original: L 2 = 1 S = S (cid:48) ).
Revised: L 2 = 1 S = S (cid:48) ).

Original: We will omit the superscript ( d ) which will be clear from context and write Y S :
Revised: We will omit the superscript ( d ) which will be clear from context and write Y S :

Original: = Y ( d ) S .
Revised: = Y ( d ) S .

Original: Consider a nonnegative deﬁnite kernel function H : Q p × Q p → R ( p = d or q in this paper) with associated integral operator H :
Revised: Consider a nonnegative deﬁnite kernel function H : Q p × Q p → R ( p = d or q in this paper) with associated integral operator H :

Original: L 2 ( Q p ) → L 2 ( Q p ) deﬁned as H f ( u ) =
Revised: L 2 ( Q p ) → L 2 ( Q p ) deﬁned as H f ( u ) =

Original: E v { h ( u , v )
Revised: E v { h ( u , v )

Original: f ( v ) } with v ∼ Unif( Q p ) .
Revised: f ( v ) } with v ∼ Unif( Q p ) .

Original: By spectral theorem of compact operators, there exists an orthonormal basis { ψ j } j ≥ 1 of L 2 ( Q p ) and nonnegative eigenvalues ( λ j ) j ≥ 1 such that H = (cid:80) j ≥ 1 λ j ψ j ψ ∗ j (i.e.,
Revised: By spectral theorem of compact operators, there exists an orthonormal basis { ψ j } j ≥ 1 of L 2 ( Q p ) and nonnegative eigenvalues ( λ j ) j ≥ 1 such that H = (cid:80) j ≥ 1 λ j ψ j ψ ∗ j (i.e.,

Original: H ( u , v ) = (cid:80) j ≥ 1 λ j ψ j ( u ) ψ
Revised: H ( u , v ) = (cid:80) j ≥ 1 λ j ψ j ( u ) ψ

Original: j ( v ) for any u , v ∈ L 2 ( Q p ) ).
Revised: j ( v ) for any u , v ∈ L 2 ( Q p ) ).

Original: The most widespread example are inner-product kernels deﬁned as H ( u , v ) :
Revised: The most widespread example are inner-product kernels deﬁned as H ( u , v ) :

Original: = h ( (cid:104) u , v (cid:105) /p ) for some function h : R → R .
Revised: = h ( (cid:104) u , v (cid:105) /p ) for some function h : R → R .

Original: Inner-product kernels have the following simple eigendecomposition in
Revised: Inner-product kernels have the following simple eigendecomposition in

Original: L 2 ( Q p ) (taking here u , v ∈ Q p ):
Revised: L 2 ( Q p ) (taking here u , v ∈ Q p ):

Original: [Equation]
Revised: [Equation]

Original: where ξ p,(cid:96) ( h ) is the (cid:96) -th
Revised: where ξ p,(cid:96) ( h ) is the (cid:96) -th

Original: Gegenbauer coefﬁcient of h ( · / √ p ) in dimension p , i.e.,
Revised: Gegenbauer coefﬁcient of h ( · / √ p ) in dimension p , i.e.,

Original: [Equation]
Revised: [Equation]

Original: for e ∈ Q p arbitrary and Q ( p ) (cid:96) the degree- (cid:96) Gegenbauer polynomial on Q p (see Appendix D for details).
Revised: for e ∈ Q p arbitrary and Q ( p ) (cid:96) the degree- (cid:96) Gegenbauer polynomial on Q p (see Appendix D for details).

Original: Note that ( ξ p,(cid:96) ) 

Revised: Note that ( ξ p,(cid:96) )

Original: ≤ (cid:96) ≤ q are non-negative by positive semideﬁniteness of the kernel.
Revised: ≤ (cid:96) ≤ q are non-negative by positive semideﬁniteness of the kernel.

Original: We will write ξ p,(cid:96) :
Revised: We will write ξ p,(cid:96) :

Original: = ξ p,(cid:96) ( h ) and use extensively the decomposition identity (1) in the rest of the paper.
Revised: = ξ p,(cid:96) ( h ) and use extensively the decomposition identity (1) in the rest of the paper.

Original: 2. O NE - LAYER CONVOLUTIONAL KERNEL
Revised: 2. O NE - LAYER CONVOLUTIONAL KERNEL

Original: We ﬁrst consider the convolutional kernel H CK (CK) given by a one-layer convolution layer with patch size q and inner-product kernel function h : R → R :
Revised: We ﬁrst consider the convolutional kernel H CK (CK) given by a one-layer convolution layer with patch size q and inner-product kernel function h : R → R :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ) where we recall that x ( k ) =
Revised: where we recall that x ( k ) =

Original: ( x k , . . .
Revised: ( x k , . . .

Original: , x k + q − 

Revised: , x k + q −

Original: ) ∈ Q q is the k ’th patch of the image with size q .
Revised: ) ∈ Q q is the k ’th patch of the image with size q .

Original: Before stating the eigendecomposition of H CK , we introduce some notations.
Revised: Before stating the eigendecomposition of H CK , we introduce some notations.

Original: For any subset S ⊆
Revised: For any subset S ⊆

Original: [ d ] , denote γ ( S ) the diameter of S with cyclic convention, i.e., γ ( S )
Revised: [ d ] , denote γ ( S ) the diameter of S with cyclic convention, i.e., γ ( S )

Original: =
Revised: =

Original: max { min { mod ( j − i, d ) + 1 , mod ( i − j, d ) + 1 } : i, j ∈ S } (e.g., γ ( { 2 , d } ) = 3 ).
Revised: max { min { mod ( j − i, d ) + 1 , mod ( i − j, d ) + 1 } : i, j ∈ S } (e.g., γ ( { 2 , d } ) = 3 ).

Original: For any integer (cid:96) ≤ q , consider the set
Revised: For any integer (cid:96) ≤ q , consider the set

Original: E (cid:96) =
Revised: E (cid:96) =

Original: { S ⊆
Revised: { S ⊆

Original: [ d ] : | S | = (cid:96), γ ( S ) ≤ q } of all subsets of [ d ] of size (cid:96) with diameter less or equal to q .
Revised: [ d ] : | S | = (cid:96), γ ( S ) ≤ q } of all subsets of [ d ] of size (cid:96) with diameter less or equal to q .

Original: We will assume throughout this paper that q ≤ d/ 2 to avoid additional overlap between sets.
Revised: We will assume throughout this paper that q ≤ d/ 2 to avoid additional overlap between sets.

Original: Proposition 1 (Eigendecomposition of H CK ) .
Revised: Proposition 1 (Eigendecomposition of H CK ) .

Original: Let H CK be a convolutional kernel as deﬁned in
Revised: Let H CK be a convolutional kernel as deﬁned in

Original: Eq.
Revised: Eq.

Original: Then H CK admits the following eigendecomposition:
Revised: Then H CK admits the following eigendecomposition:

Original: [Equation]
Revised: [Equation]

Original: where r ( S ) =
Revised: where r ( S ) =

Original: q + 1 − γ ( S ) and ξ q,(cid:96) ≥ 0 is deﬁned in Eq.
Revised: q + 1 − γ ( S ) and ξ q,(cid:96) ≥ 0 is deﬁned in Eq.

Original: Notice that Y S with γ ( S ) > q (monomials with support not contained in a segment of size q ) are in the null space of H CK .
Revised: Notice that Y S with γ ( S ) > q (monomials with support not contained in a segment of size q ) are in the null space of H CK .

Original: Hence (as long as ξ q,(cid:96) > 0 for all 0 ≤ (cid:96) ≤ q ), the RKHS associated to
Revised: Hence (as long as ξ q,(cid:96) > 0 for all 0 ≤ (cid:96) ≤ q ), the RKHS associated to

Original: H CK exactly contains all the functions in the q -local function class L 2 ( Q d , Loc q ) (c.f. Eq.
Revised: H CK exactly contains all the functions in the q -local function class L 2 ( Q d , Loc q ) (c.f. Eq.

Original: (LOC)).
Revised: (LOC)).

Original: In words, L 2 ( Q d , Loc q ) consists of functions that are localized on patches, with no long-range interactions between different parts of the image.
Revised: In words, L 2 ( Q d , Loc q ) consists of functions that are localized on patches, with no long-range interactions between different parts of the image.

Original: An example of local function with q = 3 is given by f ( x ) =
Revised: An example of local function with q = 3 is given by f ( x ) =

Original: x 1 x 2 x 3 + x 4 x 6 + x 5 .
Revised: x 1 x 2 x 3 + x 4 x 6 + x 5 .

Original: On the other hand, the RKHS associated to the fully-connected kernel H FC (FC) typically contains all the functions  dim( L 2 ( Q d , Loc q ))
Revised: On the other hand, the RKHS associated to the fully-connected kernel H FC (FC) typically contains all the functions in L 2 ( Q d ) (under genericity assumptions on h ). The RKHS with convolution dim( L 2 ( Q d , Loc q ))

Original: =
Revised: =

Original: d 2 q − 1 + 1 is signiﬁcantly smaller than dim( L 2 ( Q d ))
Revised: d 2 q − 1 + 1 is signiﬁcantly smaller than dim( L 2 ( Q d ))

Original: = 2 d , which prompts the following question: what is the statistical advantage of using H CK over H FC when learning functions in L 2 ( Q d , Loc q ) ?
Revised: = 2 d , which prompts the following question: what is the statistical advantage of using H CK over H FC when learning functions in L 2 ( Q d , Loc q ) ?

Original: We ﬁrst consider the classical approach to bounding the test error of Caponnetto & De Vito (2007);
Revised: We ﬁrst consider the classical approach to bounding the test error of Caponnetto & De Vito (2007);

Original: Wainwright (2019); Bach (2021) which relies on the following two standard assumptions:
Revised: Wainwright (2019); Bach (2021) which relies on the following two standard assumptions:

Original: (A1) Capacity condition: we assume N ( h, λ ) :=
Revised: (A1) Capacity condition: we assume N ( h, λ ) :=

Original: Tr[ h/ ( h + λ I ) − 1 ] ≤ C h λ − 1 /α with 1 α > 1 .
Revised: Tr[ h/ ( h + λ I ) − 1 ] ≤ C h λ − 1 /α with 1 α > 1 .

Original: (A2) Source condition: (cid:107) h − β/ 2 g (cid:107) L 2 ≤ B with 2 β > α − 1 α and B ≥ 0 .
Revised: (A2) Source condition: (cid:107) h − β/ 2 g (cid:107) L 2 ≤ B with 2 β > α − 1 α and B ≥ 0 .

Original: The capacity condition (A1) characterizes the size of the RKHS: for increasing α , the RKHS contains less and less functions.
Revised: The capacity condition (A1) characterizes the size of the RKHS: for increasing α , the RKHS contains less and less functions.

Original: The source condition (A2) characterizes the regularity of the target function (the ‘source’) with respect to the kernel: increasing β corresponds to smoother and smoother functions.
Revised: The source condition (A2) characterizes the regularity of the target function (the ‘source’) with respect to the kernel: increasing β corresponds to smoother and smoother functions.

Original: See Appendix B.2 for more discussions.
Revised: See Appendix B.2 for more discussions.

Original: Based on these two assumptions, we can apply standard bounds on the KRR test error and obtain:
Revised: Based on these two assumptions, we can apply standard bounds on the KRR test error and obtain:

Original: Theorem 1 (Generalization error of KRR with H CK ) .
Revised: Theorem 1 (Generalization error of KRR with H CK ) .

Original: Let h : R → R be an inner-product kernel satisfying (A1).
Revised: Let h : R → R be an inner-product kernel satisfying (A1).

Original: Let f (cid:63)
Revised: Let f (cid:63)

Original: ∈
Revised: ∈

Original: L 2 ( Q d , Loc q ) with f ( x ) =
Revised: L 2 ( Q d , Loc q ) with f ( x ) =

Original: (cid:80)
Revised: (cid:80)

Original: k ∈
Revised: k ∈

Original: [ d ] g k ( x ( k ) ) satisfying (A2) with (cid:80) k ∈ [ d ] (cid:107) h −
Revised: [ d ] g k ( x ( k ) ) satisfying (A2) with (cid:80) k ∈ [ d ] (cid:107) h −

Original: β/ 2 g k (cid:107) 2 L 2 ≤ B 2 .
Revised: β/ 2 g k (cid:107) 2 L 2 ≤ B 2 .

Original: Then there exists C 1 , C 2 , C 3 > 0 constants that only depend on (A1) and (A2) (and independent of d ), such that for n ≥ C 1 max( (cid:107) f
Revised: Then there exists C 1 , C 2 , C 3 > 0 constants that only depend on (A1) and (A2) (and independent of d ), such that for n ≥ C 1 max( (cid:107) f

Original: (cid:63) (cid:107) 2 L ∞ , d ) and
Revised: (cid:63) (cid:107) 2 L ∞ , d ) and

Original: [Equation]
Revised: [Equation]

Original: Note that the exponent βαβα +1 only depends on the q -dimensional kernel h .
Revised: Note that the exponent βαβα +1 only depends on the q -dimensional kernel h .

Original: Hence, the generalization bound with respect to ( n/d ) is independent of the dimension d of the image.
Revised: Hence, the generalization bound with respect to ( n/d ) is independent of the dimension d of the image.

Original: Let’s compare to
Revised: Let’s compare to

Original: KRR with inner-product kernel H FC (FC): from Caponnetto & De Vito (2007), we have the minmax rate E ε { R ( f (cid:63) , ˆ f λ ) } (cid:16) n − ˜
Revised: KRR with inner-product kernel H FC (FC): from Caponnetto & De Vito (2007), we have the minmax rate E ε { R ( f (cid:63) , ˆ f λ ) } (cid:16) n − ˜

Original: α
Revised: α

Original: ˜
Revised: ˜

Original: β
Revised: β

Original: ˜
Revised: ˜

Original: α ˜
Revised: α ˜

Original: β
Revised: β

Original: +1 where h is now deﬁned in d dimension and veriﬁes (A1) and (A2) with constants ˜ α, ˜ β .
Revised: +1 where h is now deﬁned in d dimension and veriﬁes (A1) and (A2) with constants ˜ α, ˜ β .

Original: Typically, if f (cid:63) is only assumed Lipschitz, then ˜ β
Revised: Typically, if f (cid:63) is only assumed Lipschitz, then ˜ β

Original: ˜ α
Revised: ˜ α

Original: = O (1 /d ) , which leads to a minmax rate n − O (1 /d ) for H FC , while for H CK , βα = O (1 /q ) , which leads to a minmax rate n − O (1 /q ) .
Revised: = O (1 /d ) , which leads to a minmax rate n − O (1 /d ) for H FC , while for H CK , βα = O (1 /q ) , which leads to a minmax rate n − O (1 /q ) .

Original: Hence, for q (cid:28) d , H CK breaks the curse of dimensionality by restricting the RKHS to ‘local’ functions.
Revised: Hence, for q (cid:28) d , H CK breaks the curse of dimensionality by restricting the RKHS to ‘local’ functions.

Original: This was already noticed in Scetbon & Harchaoui (2020); Favero et al. 
Revised: Similarly, Favero et al. (2021) derived a decay rates in n that do not depend on d for a one-layer convolutional kernel. The key difference between Theorem 1 and Favero et al. (2021) is that we obtain a non-asymptotic bound that is minmax optimal up to a constant multiplicative factor in both d and n (this can be showed for example by adapting the proof in Appendix B.6 in Bietti et al.

Original: (2021) for slightly simpliﬁed architectures.
Revised: (2021)) using a rigorous framework of source and capacity condition.

Original: Theorem 1 and results of this type suffers from several limitations: 1) they are tight only in terms of the exponent of n in a minmax sense; 2) they do not provide comparisons for speciﬁc subclasses of functions; 3) in order to obtain the minmax rate, the regularization parameter λ has to be carefully tuned to balance the bias and variance terms, which is in contrast to modern practice where often the model is trained until interpolation.
Revised: Theorem 1 and results of this type suffers from several limitations: 1) they are tight only in terms of the exponent of n in a minmax sense; 2) they do not provide comparisons for speciﬁc subclasses of functions; 3) in order to obtain the minmax rate, the regularization parameter λ has to be carefully tuned to balance the bias and variance terms, which is in contrast to modern practice where often the model is trained until interpolation.

Original: This led several groups to consider instead the test error of
Revised: This led several groups to consider instead the test error of

Original: KRR in a high-dimensional limit Ghorbani et al.
Revised: KRR in a high-dimensional limit Ghorbani et al.

Original: (2021); Mei et al.
Revised: (2021); Mei et al.

Original: (2021a)
Revised: (2021a)

Original: ; Canatar et al.
Revised: ; Canatar et al.

Original: and derive exact asymptotic predictions correct up to an additive vanishing constant for any f (cid:63) ∈
Revised: and derive exact asymptotic predictions correct up to an additive vanishing constant for any f (cid:63) ∈

Original: L(see Appendix C for more details).
Revised: L(see Appendix C for more details).

Original: Using the general framework in Mei et al.
Revised: Using the general framework in Mei et al.

Original: (2021a), we get the following result for q, d large:
Revised: (2021a), we get the following result for q, d large:

Original: Theorem 2 (Generalization error of KRR with H CK in high-dimension (informal)) .
Revised: Theorem 2 (Generalization error of KRR with H CK in high-dimension (informal)) .

Original: Let f (cid:63) ∈
Revised: Let f (cid:63) ∈

Original: L 2 ( Q d , Loc q ) and h : R → R verifying some ‘genericity condition’.
Revised: L 2 ( Q d , Loc q ) and h : R → R verifying some ‘genericity condition’.

Original: Then for n = dq s − 1+ ν with 0 < ν < 1 , and λ
Revised: Then for n = dq s − 1+ ν with 0 < ν < 1 , and λ

Original: = O (1) (in particular λ = 0 works), we have
Revised: = O (1) (in particular λ = 0 works), we have

Original: [Equation]
Revised: [Equation]

Original: where P E ≤ s ,ν is the projection on the span of Y S with either | S | < s and S ∈ E
Revised: where P E ≤ s ,ν is the projection on the span of Y S with either | S | < s and S ∈ E

Original: | S | or | S | =
Revised: | S | or | S | =

Original: s and γ ( S ) ≤ q (1 − q − ν ) .
Revised: s and γ ( S ) ≤ q (1 − q − ν ) .

Original: See Appendix C.1 for a rigorous statement.
Revised: See Appendix C.1 for a rigorous statement.

Original: In words, when dq s − 1 (cid:28) n (cid:28) dq s , KRR with H CK only learns a degree- s polynomial approximation to f (cid:63) .
Revised: In words, when dq s − 1 (cid:28) n (cid:28) dq s , KRR with H CK only learns a degree- s polynomial approximation to f (cid:63) .

Original: On the other hand, when considering the standard inner-product kernel H FC (FC) we get:
Revised: On the other hand, when considering the standard inner-product kernel H FC (FC) we get:

Original: Theorem 3 (Generalization error of KRR with H FC in high-dimension (informal)) .
Revised: Theorem 3 (Generalization error of KRR with H FC in high-dimension (informal)) .

Original: Let f (cid:63) ∈
Revised: Let f (cid:63) ∈

Original: L 2 ( Q d ) and ˜ h : R → R with some ‘genericity condition’.
Revised: L 2 ( Q d ) and ˜ h : R → R with some ‘genericity condition’.

Original: Then for d s (cid:28) n (cid:28) d s +1 and λ
Revised: Then for d s (cid:28) n (cid:28) d s +1 and λ

Original: = O (1) ,
Revised: = O (1) ,

Original: [Equation]
Revised: [Equation]

Original:  where P ≤ s is the projection on the subspace of degree- s polynomials.
Revised: (7) where P ≤ s is the projection on the subspace of degree- s polynomials.

Original: This theorem was proved in Ghorbani et al.
Revised: This theorem was proved in Ghorbani et al.

Original: (2021); Mei et al.
Revised: (2021); Mei et al.

Original: (2021a).
Revised: (2021a).

Original: Notice that Eq.
Revised: Notice that Eq.

Original: (7) does not depend on the structure of f (cid:63) .
Revised: (7) does not depend on the structure of f (cid:63) .

Original: Hence, when f (cid:63) ∈ L 2 ( Q d , Loc q ) , Theorems 2 and 3 shows a clear statistical advantage of H CK over H FC when q (cid:28) d (and therefore of one-layer CNNs over fully-connected neural networks in the kernel regime).
Revised: Hence, when f (cid:63) ∈ L 2 ( Q d , Loc q ) , Theorems 2 and 3 shows a clear statistical advantage of H CK over H FC when q (cid:28) d (and therefore of one-layer CNNs over fully-connected neural networks in the kernel regime).

Original: 2. L OCAL AVERAGE POOLING AND DOWNSAMPLING
Revised: 2. L OCAL AVERAGE POOLING AND DOWNSAMPLING

Original: In many applications such as object recognition, we expect the target function to depend mildly on the absolute spatial position of an object and to be stable under small shifts of the input.
Revised: In many applications such as object recognition, we expect the target function to depend mildly on the absolute spatial position of an object and to be stable under small shifts of the input.

Original: To take this local invariance into account, convolution layers are often followed by a pooling operation.
Revised: To take this local invariance into account, convolution layers are often followed by a pooling operation.

Original: Here we consider local average pooling on a segment of length ω and obtain the kernel
Revised: Here we consider local average pooling on a segment of length ω and obtain the kernel

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Deﬁne S (cid:96) =
Revised: Deﬁne S (cid:96) =

Original: { S ⊆ [ q ] : | S | = (cid:96) } as the collection of sets of size (cid:96) .
Revised: { S ⊆ [ q ] : | S | = (cid:96) } as the collection of sets of size (cid:96) .

Original: We further deﬁne an equivalence relation ∼ on S (cid:96) : S ∼ S (cid:48) if S (cid:48) is a translated subset of S in [ q ] (without cyclic convention).
Revised: We further deﬁne an equivalence relation ∼ on S (cid:96) : S ∼ S (cid:48) if S (cid:48) is a translated subset of S in [ q ] (without cyclic convention).

Original: We denote C (cid:96) the quotient set of A (cid:96) under the equivalence relation ∼ .
Revised: We denote C (cid:96) the quotient set of A (cid:96) under the equivalence relation ∼ .

Original: Proposition 2 (Eigendecomposition of H CK ω ) .
Revised: Proposition 2 (Eigendecomposition of H CK ω ) .

Original: Let H CK ω be a convolutional kernel with local average pooling as deﬁned in Eq.
Revised: Let H CK ω be a convolutional kernel with local average pooling as deﬁned in Eq.

Original: Then H CK ω admits the following eigendecomposition:
Revised: Then H CK ω admits the following eigendecomposition:

Original: [Equation]
Revised: [Equation]

Original: where (denoting k + S the translated set S by k positions with cyclic convention in [ d ] ) 
Revised: where (denoting k + S the translated set S by k positions with cyclic convention in [ d ] ) κ

Original: [Equation]
Revised: [Equation]

Original: First notice that, as long as gcd ( ω, d ) = 1 , the RKHS associated to H CK contains the same set of functions as the RKHS of H CK , i.e., all local functions L 2 ( Q d , Loc q ) .
Revised: First notice that, as long as gcd ( ω, d ) = 1 , the RKHS associated to H CK contains the same set of functions as the RKHS of H CK , i.e., all local functions L 2 ( Q d , Loc q ) .

Original: (There are gcd ( ω, d ) −number of zero weights: κ j = 0 for all j ∈
Revised: (There are gcd ( ω, d ) −number of zero weights: κ j = 0 for all j ∈

Original: [ d − 1] such that d is a divisor of jω .
Revised: [ d − 1] such that d is a divisor of jω .

Original: See Appendix A. for details.)
Revised: See Appendix A. for details.)

Original: However H CK will penalize different frequency components of the functions differently.
Revised: However H CK will penalize different frequency components of the functions differently.

Original: Denote f j ( x ) the j -th component of the discrete Fourier transform of the function, i.e., f j ( x ) =√ d (cid:80) k ∈ [ d ] ρ kj f ( t k · x ) where ρ j = e 2 iπj/d and t k · x = ( x k +1 , . . .
Revised: Denote f j ( x ) the j -th component of the discrete Fourier transform of the function, i.e., f j ( x ) =√ d (cid:80) k ∈ [ d ] ρ kj f ( t k · x ) where ρ j = e 2 iπj/d and t k · x = ( x k +1 , . . .

Original: , x d , x 1 , . . .
Revised: , x d , x 1 , . . .

Original: , x k ) is the cyclic shift by k pixels.
Revised: , x k ) is the cyclic shift by k pixels.

Original: Then H CK reweights the eigenspaces associated with f j ( x ) by a factor κ j , promoting low-frequency components ( κ j > 1 ) and penalizing the high-frequencies ( κ j < 1 ).
Revised: Then H CK reweights the eigenspaces associated with f j ( x ) by a factor κ j , promoting low-frequency components ( κ j > 1 ) and penalizing the high-frequencies ( κ j < 1 ).

Original: In words, pooling biases the learning towards low-frequency functions , which are stable by small shifts.
Revised: In words, pooling biases the learning towards low-frequency functions , which are stable by small shifts.

Original: Let us focus on two special choices here: the pooling parameter ω = 1 and ω = d .
Revised: Let us focus on two special choices here: the pooling parameter ω = 1 and ω = d .

Original: When ω = 1 ,
Revised: When ω = 1 ,

Original: H CK ω reduces to H CK ( κ j = 1 for all j ∈ [ d ] ) which does not bias towards either low or high frequency components.
Revised: H CK ω reduces to H CK ( κ j = 1 for all j ∈ [ d ] ) which does not bias towards either low or high frequency components.

Original: When ω =
Revised: When ω =

Original: d , we denote such kernel H CK ω
Revised: d , we denote such kernel H CK ω

Original: =
Revised: =

Original: d by H CKGP which corresponds to global average pooling.
Revised: d by H CK GP which corresponds to global average pooling.

Original: In this case, we have κ d = d and κ j = 0 for j < d which enforces exact invariance under the group of cyclic translations.
Revised: In this case, we have κ d = d and κ j = 0 for j < d which enforces exact invariance under the group of cyclic translations.

Original: More precisely, H CKGP has RKHS that contains all cyclic q-local functions f ( x ) = (cid:80) k ∈
Revised: More precisely, H CKGP has RKHS that contains all cyclic q-local functions f ( x ) = (cid:80) k ∈

Original: [ d ] g ( x ( k ) ) ∈
Revised: [ d ] g ( x ( k ) ) ∈

Original: L 2 ( Q d , CycLoc q ) (c.f. Eq.
Revised: L 2 ( Q d , CycLoc q ) (c.f. Eq.

Original: (CYC-LOC)).
Revised: (CYC-LOC)).

Original: We obtain a bound on the test error of KRR with H CK ω similar to Theorem 1, but with d replaced by an effective dimension d eff .
Revised: We obtain a bound on the test error of KRR with H CK ω similar to Theorem 1, but with d replaced by an effective dimension d eff .

Original: Theorem 4 (Generalization of KRR with average pooling (ﬁxed d, q )) .
Revised: Theorem 4 (Generalization of KRR with average pooling (ﬁxed d, q )) .

Original: Assume that h : R → R has ξ q, 0 = 0 and satisﬁes (A1).
Revised: Assume that h : R → R has ξ q, 0 = 0 and satisﬁes (A1).

Original: Further assume ( A 2 (cid:48) )
Revised: Further assume ( A 2 (cid:48) )

Original: that (cid:107) ( H CK ω /ω )
Revised: that (cid:107) ( H CK ω /ω )

Original: − β/ 

Revised: − β/

Original: f (cid:63) (cid:107) L 2 ≤ B .
Revised: f (cid:63) (cid:107) L 2 ≤ B .

Original: Deﬁne d eff = (cid:80) j ∈
Revised: Deﬁne d eff = (cid:80) j ∈

Original: [ d ]: κ j > 0 ( κ j /ω ) 1 /α .
Revised: [ d ]: κ j > 0 ( κ j /ω ) 1 /α .

Original: Then there exists C 1 , C 2 , C 3 > 0 constants independent of d , such that for n ≥ C 1 max( (cid:107)
Revised: Then there exists C 1 , C 2 , C 3 > 0 constants independent of d , such that for n ≥ C 1 max( (cid:107)

Original: f
Revised: f

Original: (cid:63) (cid:107) 2 L ∞ , d eff ) and setting λ ∗ =
Revised: (cid:63) (cid:107) 2 L ∞ , d eff ) and setting λ ∗ =

Original: C 2 ( d eff /n )
Revised: C 2 ( d eff /n )

Original: α αβ +1 , we get[Equation]
Revised: α [Equation] +1 , we get

Original: [Equation]
Revised: [Equation]

Original: By Jensen’s inequality, we have d eff ≤ d/ω 1 /α .
Revised: By Jensen’s inequality, we have d eff ≤ d/ω 1 /α .

Original: In particular, for global pooling, d eff = 1 and the bound (11) does not depend on d at all.
Revised: In particular, for global pooling, d eff = 1 and the bound (11) does not depend on d at all.

Original: Adding average pooling improve by a factor ω 1 /α
Revised: Adding average pooling improve by a factor ω 1 /α

Original: the upper bound on the sample complexity for ﬁtting low-frequency functions.
Revised: the upper bound on the sample complexity for ﬁtting low-frequency functions.

Original: Can we conﬁrm this statistical advantage using the predictions for KRR in high dimension?
Revised: Can we conﬁrm this statistical advantage using the predictions for KRR in high dimension?

Original: Consider ﬁrst the case of global pooling:
Revised: Consider ﬁrst the case of global pooling:

Original: Theorem 5 (Generalization of KRR with H CKGP in high-dimension (informal)) .
Revised: Theorem 5 (Generalization of KRR with H CKGP in high-dimension (informal)) .

Original: Let f (cid:63) ∈
Revised: Let f (cid:63) ∈

Original: L 2 ( Q d , CycLoc q ) and h : R → R verifying some ‘genericity condition’.
Revised: L 2 ( Q d , CycLoc q ) and h : R → R verifying some ‘genericity condition’.

Original: Then for n = q s − 1+ ν with 0 < ν < 1 , and λ
Revised: Then for n = q s − 1+ ν with 0 < ν < 1 , and λ

Original: = O (1) , we have ( P E ≤ s ,ν is deﬁned as in Theorem 2)
Revised: = O (1) , we have ( P E ≤ s ,ν is deﬁned as in Theorem 2)

Original: [Equation]
Revised: [Equation]

Original: Hence, global average pooling results in an improvement by a factor d in statistical efﬁciency when ﬁtting cyclic local functions, compared to H CK .
Revised: Hence, global average pooling results in an improvement by a factor d in statistical efﬁciency when ﬁtting cyclic local functions, compared to H CK .

Original: This improvement was already noticed in Mei et al.
Revised: This improvement was already noticed in Mei et al.

Original: (2021b)
Revised: (2021b)

Original: ; Bietti et al.
Revised: ; Bietti et al.

Original: but in the case of q = d (fully connected neural networks).
Revised: but in the case of q = d (fully connected neural networks).

Original: For ω < d , the asymptotic framework Mei et al.
Revised: For ω < d , the asymptotic framework Mei et al.

Original: (2021a) is more challenging to implement.
Revised: (2021a) is more challenging to implement.

Original: However, we present in Appendix C.1 a simpliﬁed kernel with non-overlapping local pooling which we believe captures the statistical behavior of local pooling.
Revised: However, we present in Appendix C.1 a simpliﬁed kernel with non-overlapping local pooling which we believe captures the statistical behavior of local pooling.

Original: In this case, we show that Theorem 5 holds with n = ( d/ω ) · q s − 1+ ν , which interpolates between Theorem 2 ( ω = 1 ) and Theorem 5 ( ω = d ).
Revised: In this case, we show that Theorem 5 holds with n = ( d/ω ) · q s − 1+ ν , which interpolates between Theorem 2 ( ω = 1 ) and Theorem 5 ( ω = d ).

Original: Downsampling: Often pooling is associated with a downsampling operation, which subsample one every ∆ output coordinates.
Revised: Downsampling: Often pooling is associated with a downsampling operation, which subsample one every ∆ output coordinates.

Original: In Appendix A.4, we characterize the eigendecomposition of H CK ω, ∆ (Proposition 4) and prove for the popular choice ω = ∆ , that downsampling does not modify the cyclic invariant subspace j = d (Proposition 5).
Revised: In Appendix A.4, we characterize the eigendecomposition of H CK ω, ∆ (Proposition 4) and prove for the popular choice ω = ∆ , that downsampling does not modify the cyclic invariant subspace j = d (Proposition 5).

Original: More generally, we conjecture and check numerically that downsampling with ∆ ≤ ω leaves the low-frequency eigenspaces approximately unchanged.
Revised: More generally, we conjecture and check numerically that downsampling with ∆ ≤ ω leaves the low-frequency eigenspaces approximately unchanged.

Original: In particular, the statistical complexity of learning low-frequency functions is not modiﬁed by downsampling operation in the one-layer case (while downsampling is potentially beneﬁcial in further layers).
Revised: In particular, the statistical complexity of learning low-frequency functions is not modiﬁed by downsampling operation in the one-layer case (while downsampling is potentially beneﬁcial in further layers).

Original: 2. M ULTILAYER CONVOLUTIONAL KERNELS
Revised: 2. M ULTILAYER CONVOLUTIONAL KERNELS

Original: For completeness, we brieﬂy discuss here some intuitions of multilayer convolutional kernels.
Revised: For completeness, we brieﬂy discuss here some intuitions of multilayer convolutional kernels.

Original: The beneﬁt of depth in convolutional kernels has been investigated in Cohen & Shashua (2016b);
Revised: The beneﬁt of depth in convolutional kernels has been investigated in Cohen & Shashua (2016b);

Original: Mhaskar & Poggio (2016); Scetbon & Harchaoui (2020); Bietti (2021).
Revised: Mhaskar & Poggio (2016); Scetbon & Harchaoui (2020); Bietti (2021).

Original: In particular, Bietti (2021) observed that the top layer operation of a two-layers convolutional kernel can be replaced by a low-degree polynomial without a performance change.
Revised: In particular, Bietti (2021) observed that the top layer operation of a two-layers convolutional kernel can be replaced by a low-degree polynomial without a performance change.

Original: Here, as a concrete example, we consider a two-layers convolutional kernel with ω -local pooling on ﬁrst layer, quadratic kernel and global pooling on the second layer (see Appendix A.5 for details):H 2 CK ω ( x , y ) = (cid:88) [Equation]
Revised: Here, as a concrete example, we consider a two-layers convolutional kernel with ω -local pooling on ﬁrst layer, quadratic kernel and global pooling on the second layer (see Appendix A.5 for details):

Original: q

Original: [Equation]
Revised: [Equation]

Original: While we believe techniques contained in this paper could be used to study kernels of the type (13), we leave it to future work.
Revised: While we believe techniques contained in this paper could be used to study kernels of the type (13), we leave it to future work.

Original: Here we only comment on the structure of H 2 CK ω : 1) Including a second convolutional layer allows interactions between patches: the RKHS of (13) contains functions of the form f ( x )
Revised: Here we only comment on the structure of H 2 CK ω : 1) Including a second convolutional layer allows interactions between patches: the RKHS of (13) contains functions of the form f ( x )

Original: = (cid:80) | k − l |≤ q g kl ( x ( k ) , x ( l ) ) .
Revised: = (cid:80) | k − l |≤ q g kl ( x ( k ) , x ( l ) ) .

Original: Pooling on the ﬁrst layer encourages interactions that do not rely too much on the relative position of the two patches.
Revised: Pooling on the ﬁrst layer encourages interactions that do not rely too much on the relative position of the two patches.

Original: Pooling on the second layer penalizes functions that depend on the absolute position of the interaction.
Revised: Pooling on the second layer penalizes functions that depend on the absolute position of the interaction.

Original: For more layers and higher degree kernels, one obtain hierarchical interactions of higher-order, with multi-scale absolute and relative local invariances brought by pooling layers.
Revised: For more layers and higher degree kernels, one obtain hierarchical interactions of higher-order, with multi-scale absolute and relative local invariances brought by pooling layers.

Original: [Figure]
Revised: [Figure]

Original:  N UMERICAL SIMULATIONS
Revised:  N UMERICAL SIMULATIONS

Original: In order to check our theoretical predictions
Revised: In order to check our theoretical predictions

Original: [Figure]
Revised: [Figure]

Original: we perform a simple numerical experiment on simulated data.
Revised: we perform a simple numerical experiment on simulated data.

Original: We take x ∼ Unif( Q d ) with d =
Revised: We take x ∼ Unif( Q d ) with d =

Original: [Figure]
Revised: [Figure]

Original: and consider two target functions:
Revised: and consider two target functions:

Original: [Equation]
Revised: [Equation]

Original: Here f LF , 3 is a cyclic-invariant local polynomial ( f LF , 3 is ‘low-frequency’).
Revised: Here f LF , 3 is a cyclic-invariant local polynomial ( f LF , 3 is ‘low-frequency’).

Original: The function f HF , 3 is a high-frequency local polynomial, and is orthogonal to the space of cyclic invariant functions.
Revised: The function f HF , 3 is a high-frequency local polynomial, and is orthogonal to the space of cyclic invariant functions.

Original: On these target functions, we compare the test error of kernel ridge regression with 5 different kernels: a standard inner-product kernel H FC ( x , y )
Revised: On these target functions, we compare the test error of kernel ridge regression with 5 different kernels: a standard inner-product kernel H FC ( x , y )

Original: =
Revised: =

Original: h ( (cid:104) x , y (cid:105) /d ) ; a cyclic invariant kernel H FCGP ( x , y ) (convolutional kernel with global pooling and full-size patches q = d ); a convolutional kernel H CK with patch size q = 10 ; a convolutional kernel with local pooling H CK ω with q = 10 and ω = 5 ; and a convolutional kernel with global pooling H CKGP with q = 10 .
Revised: h ( (cid:104) x , y (cid:105) /d ) ; a cyclic invariant kernel H FCGP ( x , y ) (convolutional kernel with global pooling and full-size patches q = d ); a convolutional kernel H CK with patch size q = 10 ; a convolutional kernel with local pooling H CK ω with q = 10 and ω = 5 ; and a convolutional kernel with global pooling H CKGP with q = 10 .

Original: In all these kernels, we choose a common h ( t ) =
Revised: In all these kernels, we choose a common h ( t ) =

Original: (cid:80) i ∈ [5] 0 .
Revised: (cid:80) i ∈ [5] 0 .

Original: 2 ∗ t i which is a degree 5 -polynomial.
Revised: 2 ∗ t i which is a degree 5 -polynomial.

Original: In Figure 1, we report the test errors of ﬁtting f LF , 3 (left) and f HF , 3 (right) using kernel ridge regression with these 5 kernels.
Revised: In Figure 1, we report the test errors of ﬁtting f LF , 3 (left) and f HF , 3 (right) using kernel ridge regression with these 5 kernels.

Original: We choose a small regularization parameter λ
Revised: We choose a small regularization parameter λ

Original: = 10 − 6 , and the noise level σ ε = 0 .
Revised: = 10 − 6 , and the noise level σ ε = 0 .

Original: The curves are averaged over 5 independent instances and the error bar stands for the standard deviation of these instances.
Revised: The curves are averaged over 5 independent instances and the error bar stands for the standard deviation of these instances.

Original: The results match well our theoretical predictions.
Revised: The results match well our theoretical predictions.

Original: For the function f LF , 3 , the sample sizes required to achieve vanishing test errors are ordered as H CKGP <
Revised: For the function f LF , 3 , the sample sizes required to achieve vanishing test errors are ordered as H CKGP <

Original: H CK ω <
Revised: H CK ω <

Original: H CK < H FCGP <
Revised: H CK < H FCGP <

Original: H FC and are around the predicted thresholds
Revised: H FC and are around the predicted thresholds

Original: q 2 < dq 

Revised: q 2 < dq

Original: /ω
Revised: /ω

Original: <
Revised: <

Original: d 2 < dq 2 < d 3 respectively.
Revised: d 2 < dq 2 < d 3 respectively.

Original: Next we look at the test error of ﬁtting the high frequency local function f HF , 3 .
Revised: Next we look at the test error of ﬁtting the high frequency local function f HF , 3 .

Original: The test errors of H CK and H FC are the same for f HF , 3 and f LF , 3 : this is because these kernels do not have bias towards either high-frequency or low-frequency functions.
Revised: The test errors of H CK and H FC are the same for f HF , 3 and f LF , 3 : this is because these kernels do not have bias towards either high-frequency or low-frequency functions.

Original: The kernel H CK ω perform worse on f HF , 3 than on f LF , 3 : this is because the eigenspaces of H CK ω are biased towards low-frequency polynomials.
Revised: The kernel H CK ω perform worse on f HF , 3 than on f LF , 3 : this is because the eigenspaces of H CK ω are biased towards low-frequency polynomials.

Original: The kernels H CKGP and H FCGP do not ﬁt f HF , 3 at all (test error greater than or equal to 1 ): this is because the RKHS of these two kernels only contain cyclic polynomials, but f HF , 3 is orthogonal to the space of cyclic polynomials.
Revised: The kernels H CKGP and H FCGP do not ﬁt f HF , 3 at all (test error greater than or equal to 1 ): this is because the RKHS of these two kernels only contain cyclic polynomials, but f HF , 3 is orthogonal to the space of cyclic polynomials.

Original:  D ISCUSSION AND F UTURE W ORK
Revised:  D ISCUSSION AND F UTURE W ORK

Original: In this paper, we characterized in a stylized setting how convolution, average pooling and downsampling operations modify the RKHS, by restricting it to q -local functions and then biasing the RKHS towards low-frequency components.
Revised: In this paper, we characterized in a stylized setting how convolution, average pooling and downsampling operations modify the RKHS, by restricting it to q -local functions and then biasing the RKHS towards low-frequency components.

Original: We quantiﬁed precisely the gain in statistical efﬁciency of KRR using these operations.
Revised: We quantiﬁed precisely the gain in statistical efﬁciency of KRR using these operations.

Original: Beyond illustrating the ‘RKHS engineering’ of image-like function classes, these results can further provide intuition and a rigorous foundation for convolution and pooling operations in kernels and CNNs.
Revised: Beyond illustrating the ‘RKHS engineering’ of image-like function classes, these results can further provide intuition and a rigorous foundation for convolution and pooling operations in kernels and CNNs.

Original: A natural extension would be to study the multilayer convolutional kernels in details and consider other pooling operations such as max-pooling.
Revised: A natural extension would be to study the multilayer convolutional kernels in details and consider other pooling operations such as max-pooling.

Original: Another important question is how anisotropy of the data impacts the results of this paper: in particular, it was shown that pre-processing (whitening of the patches) greatly improves the performance of convolutional kernels Thiry et al.
Revised: Another important question is how anisotropy of the data impacts the results of this paper: in particular, it was shown that pre-processing (whitening of the patches) greatly improves the performance of convolutional kernels Thiry et al.

Original: ; Bietti (2021).
Revised: ; Bietti (2021).

Original: A more challenging question is to study how training and feature learning can further improve the performance of CNNs outside the kernel regime.
Revised: A more challenging question is to study how training and feature learning can further improve the performance of CNNs outside the kernel regime.

Original: C ONTENTS 1 Introduction
Revised: C ONTENTS 1 Introduction

Original: Summary of main results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Summary of main results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 2 Main results
Revised: 2 Main results

Original: Functions on the hypercube and eigendecomposition of kernel operators . . . . . .
Revised: Functions on the hypercube and eigendecomposition of kernel operators . . . . . .

Original: One-layer convolutional kernel . .
Revised: One-layer convolutional kernel . .

Original: Local average pooling and downsampling . .
Revised: Local average pooling and downsampling . .

Original: Multilayer convolutional kernels . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Multilayer convolutional kernels . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 3 Numerical simulations 4 Discussion and Future Work
Revised: 3 Numerical simulations 4 Discussion and Future Work

Original: A Details from the main text
Revised: A Details from the main text

Original: A.1 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.1 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.2 Convolutional neural tangent kernel . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.2 Convolutional neural tangent kernel . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.3 Local average pooling operation . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.3 Local average pooling operation . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.4 Downsampling operation . . .
Revised: A.4 Downsampling operation . . .

Original: A.5 Multilayer convolutional kernels . .
Revised: A.5 Multilayer convolutional kernels . .

Original: A.6 Proofs diagonalization of convolutional kernels . . . . . . . . . . . . . . . . . . .
Revised: A.6 Proofs diagonalization of convolutional kernels . . . . . . . . . . . . . . . . . . .

Original: A
Revised: A

Original: .7 Additional numerical
Revised: .7 Additional numerical

Original: simulations . . .
Revised: simulations . . .

Original: B Generalization error of kernel methods in ﬁxed dimension
Revised: B Generalization error of kernel methods in ﬁxed dimension

Original: B.1 Bound
Revised: B.1 Bound

Original: on kernel
Revised: on kernel

Original: methods using Rademacher complexities . .
Revised: methods using Rademacher complexities . .

Original: B.2 Generalization error of KRR in the classical regime . . . . . . . . . . . . . . . . .
Revised: B.2 Generalization error of KRR in the classical regime . . . . . . . . . . . . . . . . .

Original: C
Revised: C

Original: Generalization error of KRR in high dimension
Revised: Generalization error of KRR in high dimension

Original: C.1 Generalization error of convolutional kernels in high dimension . .
Revised: C.1 Generalization error of convolutional kernels in high dimension . .

Original: C.2 Checking the assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.2 Checking the assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.3 Proof of Proposition 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.3 Proof of Proposition 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.4 Proof of Theorem 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.4 Proof of Theorem 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.5 Proof of Theorem 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.5 Proof of Theorem 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.6 Auxiliary results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.6 Auxiliary results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D Technical background of function spaces on the hypercube
Revised: D Technical background of function spaces on the hypercube

Original: D.1 Fourier basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.1 Fourier basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.2 Hypercubic Gegenbauer . .
Revised: D.2 Hypercubic Gegenbauer . .

Original: D.3 Hermite polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.3 Hermite polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.4 Hypercontractivity of uniform distributions on the hypercube
Revised: D.4 Hypercontractivity of uniform distributions on the hypercube

Original: A D ETAILS FROM THE MAIN TEXT A. N OTATIONS
Revised: A D ETAILS FROM THE MAIN TEXT A. N OTATIONS

Original: For a positive integer, we denote by [ n ] the set { 1 , 2 , . . . , n } .
Revised: For a positive integer, we denote by [ n ] the set { 1 , 2 , . . . , n } .

Original: For vectors u , v ∈ R d , we denote (cid:104) u , v (cid:105) =
Revised: For vectors u , v ∈ R d , we denote (cid:104) u , v (cid:105) =

Original: u 1 v 1 + . . .
Revised: u 1 v 1 + . . .

Original: + u d v d their scalar product, and (cid:107) u (cid:107) 2 = (cid:104) u , u (cid:105) 1 / 2 the (cid:96) 2 norm.
Revised: + u d v d their scalar product, and (cid:107) u (cid:107) 2 = (cid:104) u , u (cid:105) 1 / 2 the (cid:96) 2 norm.

Original: Given a matrix A ∈ R n × m , we denote (cid:107) A (cid:107) op = max (cid:107) u (cid:107) 2 =1 (cid:107)
Revised: Given a matrix A ∈ R n × m , we denote (cid:107) A (cid:107) op = max (cid:107) u (cid:107) 2 =1 (cid:107)

Original: Au (cid:107) 

Revised: Au (cid:107)

Original: its operator norm and by (cid:107) A (cid:107) F = (cid:0) (cid:80) i,j A 2 ij (cid:1) 1 / 2 its Frobenius norm.
Revised: its operator norm and by (cid:107) A (cid:107) F = (cid:0) (cid:80) i,j A 2 ij (cid:1) 1 / 2 its Frobenius norm.

Original: If A ∈ R n × n is a square matrix, the trace of A is denoted by
Revised: If A ∈ R n × n is a square matrix, the trace of A is denoted by

Original: Tr( A ) =
Revised: Tr( A ) =

Original: (cid:80) i ∈ [ n ] A ii .
Revised: (cid:80) i ∈ [ n ] A ii .

Original: We use O d ( · ) (resp.
Revised: We use O d ( · ) (resp.

Original: o d ( · ) ) for the standard big-O (resp.
Revised: o d ( · ) ) for the standard big-O (resp.

Original: little-o) relations, where the subscript d emphasizes the asymptotic variable.
Revised: little-o) relations, where the subscript d emphasizes the asymptotic variable.

Original: Furthermore, we write f = Ω d ( g ) if g ( d ) =
Revised: Furthermore, we write f = Ω d ( g ) if g ( d ) =

Original: O d ( f ( d )) , and f = ω
Revised: O d ( f ( d )) , and f = ω

Original: d ( g ) if g ( d ) =
Revised: d ( g ) if g ( d ) =

Original: o d ( f ( d )) .
Revised: o d ( f ( d )) .

Original: Finally, f = Θ d ( g ) if we have both f = O d ( g ) and f = Ω d ( g ) .
Revised: Finally, f = Θ d ( g ) if we have both f = O d ( g ) and f = Ω d ( g ) .

Original: We use O d, P ( · ) (resp.
Revised: We use O d, P ( · ) (resp.

Original: o
Revised: o

Original: d, P ( · ) )
Revised: d, P ( · ) )

Original: the big-O (resp.
Revised: the big-O (resp.

Original: little-o) in probability relations.
Revised: little-o) in probability relations.

Original: Namely, for h 1 ( d ) and h 2 ( d ) two sequences of random variables, h 1 ( d ) =
Revised: Namely, for h 1 ( d ) and h 2 ( d ) two sequences of random variables, h 1 ( d ) =

Original: O d, P ( h 2 ( d ))
Revised: O d, P ( h 2 ( d ))

Original: if for any ε > 0 , there exists
Revised: if for any ε > 0 , there exists

Original: C ε > 0 and d ε ∈ Z > 0 , such that
Revised: C ε > 0 and d ε ∈ Z > 0 , such that

Original: [Equation]
Revised: [Equation]

Original: and respectively: h 1 ( d ) =
Revised: and respectively: h 1 ( d ) =

Original: o d, P ( h 2 ( d )) , if h 1 ( d ) /h 2 ( d ) converges to 0 in probability.
Revised: o d, P ( h 2 ( d )) , if h 1 ( d ) /h 2 ( d ) converges to 0 in probability.

Original: Similarly, we will denote h 1 ( d ) =
Revised: Similarly, we will denote h 1 ( d ) =

Original: Ω d, P ( h 2 ( d )) if h 2 ( d ) =
Revised: Ω d, P ( h 2 ( d )) if h 2 ( d ) =

Original: O d, P ( h 1 ( d )) , and h 1 ( d ) =
Revised: O d, P ( h 1 ( d )) , and h 1 ( d ) =

Original: ω d, P ( h 2 ( d )) if h 2 ( d ) =
Revised: ω d, P ( h 2 ( d )) if h 2 ( d ) =

Original: o d, P ( h 1 ( d )) .
Revised: o d, P ( h 1 ( d )) .

Original: Finally, h 1 ( d ) =
Revised: Finally, h 1 ( d ) =

Original: Θ d, P ( h 2 ( d )) if we have both h 1 ( d ) =
Revised: Θ d, P ( h 2 ( d )) if we have both h 1 ( d ) =

Original: O d, P ( h 2 ( d )) and h 1 ( d ) =
Revised: O d, P ( h 2 ( d )) and h 1 ( d ) =

Original: Ω d, P ( h 2 ( d )) .
Revised: Ω d, P ( h 2 ( d )) .

Original: A. C ONVOLUTIONAL NEURAL TANGENT KERNEL
Revised: A. C ONVOLUTIONAL NEURAL TANGENT KERNEL

Original: In this section, we justify the expression of the convolutional neural tangent kernel H CK w , ∆ (CKAP-DS), obtained as the tangent kernel of a neural network composed of a one convolution layer followed by local average pooling and downsampling (CNN-AP-DS).
Revised: In this section, we justify the expression of the convolutional neural tangent kernel H CK w , ∆ (CKAP-DS), obtained as the tangent kernel of a neural network composed of a one convolution layer followed by local average pooling and downsampling (CNN-AP-DS).

Original: Proposition 3.
Revised: Proposition 3.

Original: Let σ ∈ C 1 ( R ) be an activation function.
Revised: Let σ ∈ C 1 ( R ) be an activation function.

Original: Consider the following one-layer convolutional neural network with ω -local average pooling and ∆ -downsampling:
Revised: Consider the following one-layer convolutional neural network with ω -local average pooling and ∆ -downsampling:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Let a 0 ik ∼ i.i.d.
Revised: Let a 0 ik ∼ i.i.d.

Original: N (0 , 1) and √ q w 0 i ∼ i.i.d.
Revised: N (0 , 1) and √ q w 0 i ∼ i.i.d.

Original: Unif( Q q ) independently, and Θ= { ( a 0 ik ) i ∈
Revised: Unif( Q q ) independently, and Θ= { ( a 0 ik ) i ∈

Original: [ N ] ,k ∈ [ d/ ∆] , ( w 0 i )
Revised: [ N ] ,k ∈ [ d/ ∆] , ( w 0 i )

Original: i ∈
Revised: i ∈

Original: [ N ] } .
Revised: [ N ] } .

Original: Then there exists h : [ − 1 , 1] → R , such that for any x , y ∈ Q d , we have almost surely
Revised: Then there exists h : [ − 1 , 1] → R , such that for any x , y ∈ Q d , we have almost surely

Original: [Equation]
Revised: [Equation]

Original: Proof of Proposition 3.
Revised: Proof of Proposition 3.

Original: For u , v ∈ Q q , deﬁne
Revised: For u , v ∈ Q q , deﬁne

Original: [Equation]
Revised: [Equation]

Original: The functions h (1) , h (2) are well deﬁned (the RHS only depend on the inner product (cid:104) u , v (cid:105) ) and can be extended to functions h (1) , h (2) :
Revised: The functions h (1) , h (2) are well deﬁned (the RHS only depend on the inner product (cid:104) u , v (cid:105) ) and can be extended to functions h (1) , h (2) :

Original: [ − 1 , 1] → R .
Revised: [ − 1 , 1] → R .

Original: Computing the derivative of the convolutional neural network with respect to a = ( a 0 ik ) i ∈
Revised: Computing the derivative of the convolutional neural network with respect to a = ( a 0 ik ) i ∈

Original: [ N ] ,k ∈ [ d/ ∆] , we have
Revised: [ N ] ,k ∈ [ d/ ∆] , we have

Original: [Equation]
Revised: [Equation]

Original: Hence by law of large number, we have almost surely
Revised: Hence by law of large number, we have almost surely

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: computing the derivative with respect to √ q W = ( √ q w 0 i )
Revised: computing the derivative with respect to √ q W = ( √ q w 0 i )

Original: i ∈
Revised: i ∈

Original: [ N ] gives
Revised: [ N ] gives

Original: [Equation]
Revised: [Equation]

Original: By law of large number, using that a ik and a ik (cid:48) are independent of mean zero and variance
Revised: By law of large number, using that a ik and a ik (cid:48) are independent of mean zero and variance

Original: [Equation]
Revised: [Equation]

Original: we get almost surely
Revised: we get almost surely

Original: [Equation]
Revised: [Equation]

Original: Taking h = h (1) + h (2) concludes the proof.
Revised: Taking h = h (1) + h (2) concludes the proof.

Original: A. L OCAL AVERAGE POOLING OPERATION
Revised: A. L OCAL AVERAGE POOLING OPERATION

Original: Consider a function f ∈ L 2 ( Q d ) : we can decompose it as
Revised: Consider a function f ∈ L 2 ( Q d ) : we can decompose it as

Original: [Equation]
Revised: [Equation]

Original: where ρ j = e 2 iπjd and t k · x = ( x k +1 , . .
Revised: where ρ j = e 2 iπjd and t k · x = ( x k +1 , . .

Original: , x d , x 1 , . . .
Revised: , x d , x 1 , . . .

Original: , x k ) is the cyclic shift of x by k pixels.
Revised: , x k ) is the cyclic shift of x by k pixels.

Original: We can think about f j ( x ) as the j -th component of the discrete Fourier transform of the function f ( x ) seen as a d -dimensional vector { f ( t k · x ) } k ∈ [ d ] for any x ∈ Q d .
Revised: We can think about f j ( x ) as the j -th component of the discrete Fourier transform of the function f ( x ) seen as a d -dimensional vector { f ( t k · x ) } k ∈ [ d ] for any x ∈ Q d .

Original: Notice furthermore that if f is a local function, i.e., f can be decomposed as a sum of functions on patches f ( x ) =
Revised: Notice furthermore that if f is a local function, i.e., f can be decomposed as a sum of functions on patches f ( x ) =

Original: (cid:80)
Revised: (cid:80)

Original: k ∈
Revised: k ∈

Original: [ d ] g k ( x ( k ) ) , then we can write
Revised: [ d ] g k ( x ( k ) ) , then we can write

Original: [Equation]
Revised: [Equation]

Original: where we denoted ( v ∈ Q q
Revised: where we denoted ( v ∈ Q q

Original: ) ˜
Revised: ) ˜

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In particular, decomposing ˜ g j in the Fourier basis, we get (denoting c S = (cid:104)
Revised: In particular, decomposing ˜ g j in the Fourier basis, we get (denoting c S = (cid:104)

Original: ˜
Revised: ˜

Original: g j , Y S (cid:105) L 2 ), f j
Revised: g j , Y S (cid:105) L 2 ), f j

Original: [Equation]
Revised: [Equation]

Original: which shows that the j -th frequency component f j is in the span of { Y j,S } S ⊆
Revised: which shows that the j -th frequency component f j is in the span of { Y j,S } S ⊆

Original: [ q ] .
Revised: [ q ] .

Original: In particular, applying average pooling operation in the kernel will reweight this eigenspace by a factor κ j .
Revised: In particular, applying average pooling operation in the kernel will reweight this eigenspace by a factor κ j .

Original: Let us further comment on the values of κ j .
Revised: Let us further comment on the values of κ j .

Original: First, we have
Revised: First, we have

Original: [Equation]
Revised: [Equation]

Original: In particular, the maximal eigenvalue is attained at j = d with κ d = ω , which corresponds to the subspace of cyclic invariant functions.
Revised: In particular, the maximal eigenvalue is attained at j = d with κ d = ω , which corresponds to the subspace of cyclic invariant functions.

Original: Furthermore, κ j = 

Revised: Furthermore, κ j =

Original: if and only if d is a divisor of jω for j ≤ d − 1 , i.e., j is a multilple of gcd ( ω, d ) .
Revised: if and only if d is a divisor of jω for j ≤ d − 1 , i.e., j is a multilple of gcd ( ω, d ) .

Original: There are gcd ( ω, d ) − 1 such zero eigenvalues.
Revised: There are gcd ( ω, d ) − 1 such zero eigenvalues.

Original: In convolutional kernels, a weighted average is often preferred to local average pooling (Mairal et al., 2014; Mairal, 2016; Bietti, 2021): in that case we consider τ : R → R and obtain the kernel
Revised: In convolutional kernels, a weighted average is often preferred to local average pooling (Mairal et al., 2014; Mairal, 2016; Bietti, 2021): in that case we consider τ : R → R and obtain the kernel

Original: [Equation]
Revised: [Equation]

Original: where d ( s ) =
Revised: where d ( s ) =

Original: min( s, d − s ) (the distance between k + s and k on [ d ] with cyclic convention).
Revised: min( s, d − s ) (the distance between k + s and k on [ d ] with cyclic convention).

Original: Note that H CK τ has the same eigendecomposition as H CK ω but with different weights κ j .
Revised: Note that H CK τ has the same eigendecomposition as H CK ω but with different weights κ j .

Original: A popular choice for τ is the Gaussian ﬁlter τ ( x ) =
Revised: A popular choice for τ is the Gaussian ﬁlter τ ( x ) =

Original: [Figure]
Revised: [Figure]

Original: In Figure 2, we compare the eigenvalues κ j for local average pooling and Gaussian ﬁlter with different value of ω and σ 2 .
Revised: In Figure 2, we compare the eigenvalues κ j for local average pooling and Gaussian ﬁlter with different value of ω and σ 2 .

Original: Note that the eigenvalue decay controls how much high-frequencies are penalized: faster decay induces heavier penalty on the high-frequency components.
Revised: Note that the eigenvalue decay controls how much high-frequencies are penalized: faster decay induces heavier penalty on the high-frequency components.

Original: [Figure]
Revised: [Figure]

Original: A. D OWNSAMPLING OPERATION
Revised: A. D OWNSAMPLING OPERATION

Original: As mentioned in the main text, a downsampling operation is often added after pooling.
Revised: As mentioned in the main text, a downsampling operation is often added after pooling.

Original: The kernel is given by
Revised: The kernel is given by

Original: [Equation]
Revised: [Equation]

Original: Let us introduce the family { M r } r ∈
Revised: Let us introduce the family { M r } r ∈

Original: [ q ] of block-circulant matrices deﬁned by
Revised: [ q ] of block-circulant matrices deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where we introduced the set of indices
Revised: where we introduced the set of indices

Original: [Equation]
Revised: [Equation]

Original: We can now state the eigendecomposition of H CK ω, ∆ in terms of the eigenvalues and eigenvectors of the matrices { M r } r ∈ [ q ] .
Revised: We can now state the eigendecomposition of H CK ω, ∆ in terms of the eigenvalues and eigenvectors of the matrices { M r } r ∈ [ q ] .

Original: Proposition 4 (Eigendecomposition of H CK ω, ∆ ) .
Revised: Proposition 4 (Eigendecomposition of H CK ω, ∆ ) .

Original: Let H CK ω, ∆ be a convolutional kernel with local average pooling and downsampling, as deﬁned in Eq.
Revised: Let H CK ω, ∆ be a convolutional kernel with local average pooling and downsampling, as deﬁned in Eq.

Original: Then H CK ω, ∆ admits the following eigendecomposition:
Revised: Then H CK ω, ∆ admits the following eigendecomposition:

Original: [Equation]
Revised: [Equation]

Original: where ψ ∆ j,S ( x ) = (cid:80) dk
Revised: where ψ ∆ j,S ( x ) = (cid:80) dk

Original: =1 v S j,k Y k + S ( x ) with { κ S j , v S j } j ∈ [ d ] eigenvalues and eigenvectors of M γ ( S ) .
Revised: =1 v S j,k Y k + S ( x ) with { κ S j , v S j } j ∈ [ d ] eigenvalues and eigenvectors of M γ ( S ) .

Original: Let us make a few comments on these matrices M γ ( S ) .
Revised: Let us make a few comments on these matrices M γ ( S ) .

Original: First because they only depend on S through the diameter γ ( S ) , the eigenvalues and eigenvectors { κ Sj , v Sj } j ∈ [ d ] only depend on γ ( S ) .
Revised: First because they only depend on S through the diameter γ ( S ) , the eigenvalues and eigenvectors { κ Sj , v Sj } j ∈ [ d ] only depend on γ ( S ) .

Original: Second, we see that M γ ( S ) ( i +∆)( j +∆) =
Revised: Second, we see that M γ ( S ) ( i +∆)( j +∆) =

Original: M γ ( S ) ij and M γ ( S ) ij = 0 if d ( i, j ) ≥ ω , where d ( i, j ) =
Revised: M γ ( S ) ij and M γ ( S ) ij = 0 if d ( i, j ) ≥ ω , where d ( i, j ) =

Original: min( | i − j | , d − | i − j | ) (i.e., the distance between i and j on the torus [ d ] ).
Revised: min( | i − j | , d − | i − j | ) (i.e., the distance between i and j on the torus [ d ] ).

Original: In words M γ ( S ) is a symmetric block-circulant matrix with non-zero elements on a band of size ω − 1 on the left and right of the diagonal, and on the upper-right and lower-left corners.
Revised: In words M γ ( S ) is a symmetric block-circulant matrix with non-zero elements on a band of size ω − 1 on the left and right of the diagonal, and on the upper-right and lower-left corners.

Original: Furthermore, notice that
Revised: Furthermore, notice that

Original: [Equation]
Revised: [Equation]

Original: 1 , which is independent of ω, ∆ , γ ( S ) and justify the chosen normalization.
Revised: 1 , which is independent of ω, ∆ , γ ( S ) and justify the chosen normalization.

Original: In particular, this implies that (for ξ q, 0 = 0 )
Revised: In particular, this implies that (for ξ q, 0 = 0 )

Original: [Equation]
Revised: [Equation]

Original: is also independent of the parameters ( q, ω, ∆) .
Revised: is also independent of the parameters ( q, ω, ∆) .

Original: Example 1.
Revised: Example 1.

Original: Take ∆ = 3 , ω = 5 , q = 11 , then
Revised: Take ∆ = 3 , ω = 5 , q = 11 , then

Original: [Equation]
Revised: [Equation]

Original: = 3  18 15   15 19 15   . . .
Revised: = 3  18 15   15 19 15   . . .

Original:   , and
Revised:   , and

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original:   13    11 14  3 11 13 

Revised:   13    11 14  3 11 13

Original: [Equation]
Revised: [Equation]

Original:   .
Revised:   .

Original: Remark A.1.
Revised: Remark A.1.

Original: Symmetric block-circulant matrices can be easily diagonalized as follows.
Revised: Symmetric block-circulant matrices can be easily diagonalized as follows.

Original: Consider
Revised: Consider

Original: M =
Revised: M =

Original: Circulant ( B 1 , B 2 , . .
Revised: Circulant ( B 1 , B 2 , . .

Original: B m ) where B k ∈ R ∆ × ∆ , B T 1 = B 1 and B 2+ k =
Revised: B m ) where B k ∈ R ∆ × ∆ , B T 1 = B 1 and B 2+ k =

Original: B T m − k for k = 0 , . . .
Revised: B T m − k for k = 0 , . . .

Original: , m − 2 .
Revised: , m − 2 .

Original: Denote ρ j = e 2 iπj/m and γ j ( v ) =
Revised: Denote ρ j = e 2 iπj/m and γ j ( v ) =

Original: [ v , ρ j v , · · · , ρ m − 1 j v ] / √ m ∈ R m ∆ for any v ∈ R ∆ .
Revised: [ v , ρ j v , · · · , ρ m − 1 j v ] / √ m ∈ R m ∆ for any v ∈ R ∆ .

Original: Introduce for j = 0 , . . .
Revised: Introduce for j = 0 , . . .

Original: , m − 1 , the matrix H j ∈ R ∆ × ∆ given by
Revised: , m − 1 , the matrix H j ∈ R ∆ × ∆ given by

Original: [Equation]
Revised: [Equation]

Original: The matrix H j is Hermitian
Revised: The matrix H j is Hermitian

Original: and we denote ( λ j,s ) s ∈ [∆] and ( v j,s ) s ∈ [∆] its eigenvalues and eigenvectors.
Revised: and we denote ( λ j,s ) s ∈ [∆] and ( v j,s ) s ∈ [∆] its eigenvalues and eigenvectors.

Original: Then the eigenvalues and eigenvectors of M are given by { λ j,s } j ∈ [ m ] ,s ∈
Revised: Then the eigenvalues and eigenvectors of M are given by { λ j,s } j ∈ [ m ] ,s ∈

Original: [∆] and { γ j ( v j,s ) } j ∈ [ m ] ,s ∈ [∆] .
Revised: [∆] and { γ j ( v j,s ) } j ∈ [ m ] ,s ∈ [∆] .

Original: In particular, if ∆ = 1 and M = Circulant ( b 1 , b 2 , . . .
Revised: In particular, if ∆ = 1 and M = Circulant ( b 1 , b 2 , . . .

Original: , b m ) is a circulant matrix, then the eigenvalues are simply given by
Revised: , b m ) is a circulant matrix, then the eigenvalues are simply given by

Original: [Equation]
Revised: [Equation]

Original: and eigenvectors
Revised: and eigenvectors

Original: [Equation]
Revised: [Equation]

Original: Here we will focus on the impact of downsampling for single-layer convolutional kernels.
Revised: Here we will focus on the impact of downsampling for single-layer convolutional kernels.

Original: We expect the downsampling operation to have a much more important role for the next layers: for example, increasing the scale of interactions or reducing the dimensionality of the pixel space.
Revised: We expect the downsampling operation to have a much more important role for the next layers: for example, increasing the scale of interactions or reducing the dimensionality of the pixel space.

Original: We will argue below that adding a downsampling operation after local pooling leaves the lowfrequency components approximately unchanged (while potentially modifying the high-frequency eigenspaces).
Revised: We will argue below that adding a downsampling operation after local pooling leaves the lowfrequency components approximately unchanged (while potentially modifying the high-frequency eigenspaces).

Original: We consider ∆ ≤ ω : for ∆ > ω , some basis functions Y S with S ∈ E (cid:96) are in the null space of H CK ω, ∆ , which impact all frequencies.
Revised: We consider ∆ ≤ ω : for ∆ > ω , some basis functions Y S with S ∈ E (cid:96) are in the null space of H CK ω, ∆ , which impact all frequencies.

Original: To emphasize the dependency on ω, ∆ , denote M rω, ∆ the matrix (20).
Revised: To emphasize the dependency on ω, ∆ , denote M rω, ∆ the matrix (20).

Original: We will study the change in the matrix M rω, 1 when adding downsampling ∆ , and consider
Revised: We will study the change in the matrix M rω, 1 when adding downsampling ∆ , and consider

Original: [Equation]
Revised: [Equation]

Original: where we denote A rω, ∆ =
Revised: where we denote A rω, ∆ =

Original: M rω, ∆ − M rω, 1 .
Revised: M rω, ∆ − M rω, 1 .

Original: Notice that A rω, ∆ is a symmetric block-circulant matrix.
Revised: Notice that A rω, ∆ is a symmetric block-circulant matrix.

Original: Therefore, from Remark A.1, the eigenvectors of A rω, ∆ are given by { γ j ( v j,s ) } j ∈ [ m ] ,s ∈
Revised: Therefore, from Remark A.1, the eigenvectors of A rω, ∆ are given by { γ j ( v j,s ) } j ∈ [ m ] ,s ∈

Original: [∆] where d = m ∆ and γ j ( v j,s ) =
Revised: [∆] where d = m ∆ and γ j ( v j,s ) =

Original: [ v j,s , ρ m,j v j,s , . . .
Revised: [ v j,s , ρ m,j v j,s , . . .

Original: , ρ m − 1 m,j v j,s ] with ρ m,j = e 2 iπjm and ( v j,s ) s ∈
Revised: , ρ m − 1 m,j v j,s ] with ρ m,j = e 2 iπjm and ( v j,s ) s ∈

Original: [∆] eigenvectors of H j (24).
Revised: [∆] eigenvectors of H j (24).

Original: The eigenvectors of M rω, 1 are given by u t = [1 , ρ d,t , · · · , ρ d − 1 d,t ] / √ d with ρ d,t = e 2 iπtd .
Revised: The eigenvectors of M rω, 1 are given by u t = [1 , ρ d,t , · · · , ρ d − 1 d,t ] / √ d with ρ d,t = e 2 iπtd .

Original: Notice that
Revised: Notice that

Original: [Equation]
Revised: [Equation]

Original: which is 0 except when t ≡ j [ m ] .
Revised: which is 0 except when t ≡ j [ m ] .

Original: Hence, we see that A rω, ∆ in Eq.
Revised: Hence, we see that A rω, ∆ in Eq.

Original: (25) only modify the eigenspaces of M rω, 1 as follows: the eigendirections { γ j ( v j,s ) } j ∈ [ m ] ,s ∈ [∆] coming from H j (24) only modify the eigenspaces of M rω, 1 spanned by { u am + j } a =0 ,..., ∆ − 1 .
Revised: (25) only modify the eigenspaces of M rω, 1 as follows: the eigendirections { γ j ( v j,s ) } j ∈ [ m ] ,s ∈ [∆] coming from H j (24) only modify the eigenspaces of M rω, 1 spanned by { u am + j } a =0 ,..., ∆ − 1 .

Original: For simplicity, we will focus on the popular choice ∆ = ω .
Revised: For simplicity, we will focus on the popular choice ∆ = ω .

Original: Furthermore, we will only look at the impact of the eigenvalues H 0 on the eigenspace spanned by { u am } a =0 ,..., ∆ − 1 , which contain the cyclic invariant direction.
Revised: Furthermore, we will only look at the impact of the eigenvalues H 0 on the eigenspace spanned by { u am } a =0 ,..., ∆ − 1 , which contain the cyclic invariant direction.

Original: We show below that H 

Revised: We show below that H

Original: = 0 and therefore A rω,ω does not modify the cyclic invariant eigenspace of M rω, 1 :
Revised: = 0 and therefore A rω,ω does not modify the cyclic invariant eigenspace of M rω, 1 :

Original: Proposition 5.
Revised: Proposition 5.

Original: Consider d = mω and the symmetric block-circulant matrix
Revised: Consider d = mω and the symmetric block-circulant matrix

Original: A rω,ω = M rω,ω − M rω, 1 .
Revised: A rω,ω = M rω,ω − M rω, 1 .

Original: Denote A rω,ω =
Revised: Denote A rω,ω =

Original: Circulant ( B 1 , B 2 , . .
Revised: Circulant ( B 1 , B 2 , . .

Original: B m ) and
Revised: B m ) and

Original: H 

Revised: H

Original: = B 1 + . . .
Revised: = B 1 + . . .

Original: + B m .
Revised: + B m .

Original: We have the following properties:
Revised: We have the following properties:

Original: (a) If q + 1 − r ≡ 

Revised: (a) If q + 1 − r ≡

Original: [ ω ] , then A rω,ω = 0 , and downsampling does not modify the matrix M rω,
Revised: [ ω ] , then A rω,ω = 0 , and downsampling does not modify the matrix M rω,

Original: ω =
Revised: ω =

Original: M rω, 1 .
Revised: M rω, 1 .

Original: (b) We have H 

Revised: (b) We have H

Original: = 

Revised: =

Original: and downsampling does not modify the cyclic invariant eigenspace
Revised: and downsampling does not modify the cyclic invariant eigenspace

Original: A rω,ω 1 = 0 .
Revised: A rω,ω 1 = 0 .

Original: Proof of Proposition 5.
Revised: Proof of Proposition 5.

Original: Let us ﬁrst start by proving point (a).
Revised: Let us ﬁrst start by proving point (a).

Original: Consider q + 1 − r = pω .
Revised: Consider q + 1 − r = pω .

Original: Fix i ∈ { 0 , . . .
Revised: Fix i ∈ { 0 , . . .

Original: , ∆ − 1 } and κ ∈ { 0 , . . .
Revised: , ∆ − 1 } and κ ∈ { 0 , . . .

Original: , ω − 1 } .
Revised: , ω − 1 } .

Original: Let us compute the entry ( i, i + κ ) of the matrix
Revised: Let us compute the entry ( i, i + κ ) of the matrix

Original: M rω,ω : this amounts to counting the number of quadruples ( k, s, s (cid:48) , t ) with k ∈ [ d/ω ] , s, s (cid:48) ∈
Revised: M rω,ω : this amounts to counting the number of quadruples ( k, s, s (cid:48) , t ) with k ∈ [ d/ω ] , s, s (cid:48) ∈

Original: [ ω ] and 0 ≤ t ≤ pω − 1 , satisfying ( kω + s + t, kω + s (cid:48) + t ) ≡ ( i, i + κ )[ d ] .
Revised: [ ω ] and 0 ≤ t ≤ pω − 1 , satisfying ( kω + s + t, kω + s (cid:48) + t ) ≡ ( i, i + κ )[ d ] .

Original: Notice that we must have s (cid:48)
Revised: Notice that we must have s (cid:48)

Original: = s + κ and therefore s ∈ { 0 , . . .
Revised: = s + κ and therefore s ∈ { 0 , . . .

Original: , ω − 1 − κ } .
Revised: , ω − 1 − κ } .

Original: Notice that for each interval uω ≤ t < ( u + 1) ω with u ∈ { 0 , . .
Revised: Notice that for each interval uω ≤ t < ( u + 1) ω with u ∈ { 0 , . .

Original: . , p − 1 } , there are exactly ω − κ ways of choosing s and then t and k to satisfy the equality.
Revised: . , p − 1 } , there are exactly ω − κ ways of choosing s and then t and k to satisfy the equality.

Original: We deduce that
Revised: We deduce that

Original: [Equation]
Revised: [Equation]

Original: By symmetry of M rω,ω , this concludes the proof of point (a).
Revised: By symmetry of M rω,ω , this concludes the proof of point (a).

Original: Consider now point (b).
Revised: Consider now point (b).

Original: First notice, because M rω,ω has zero entries for min( | i − j | , d −| i − j | ) ≥ ω , the only non-zero blocks are B 1 , B 2 and B m .
Revised: First notice, because M rω,ω has zero entries for min( | i − j | , d −| i − j | ) ≥ ω , the only non-zero blocks are B 1 , B 2 and B m .

Original: Furthermore, when computing H 0 , the diagonal entries only have one contribution from the diagonal elements of B 1 .
Revised: Furthermore, when computing H 0 , the diagonal entries only have one contribution from the diagonal elements of B 1 .

Original: The off-diagonal elements of
Revised: The off-diagonal elements of

Original: H 0 have two contribution: one from B 1 and one from B 2 (if below the diagonal) or B m (if above the diagonal), i.e.,
Revised: H 0 have two contribution: one from B 1 and one from B 2 (if below the diagonal) or B m (if above the diagonal), i.e.,

Original: [Equation]
Revised: [Equation]

Original: Let us compute ﬁrst the diagonal elements: we have easily, by a similar argument as above ( M rω,ω ) ii = 1 =
Revised: Let us compute ﬁrst the diagonal elements: we have easily, by a similar argument as above ( M rω,ω ) ii = 1 =

Original: ( M rω, 1 ) ii , and therefore H 0 has zero zero diagonal entries.
Revised: ( M rω, 1 ) ii , and therefore H 0 has zero zero diagonal entries.

Original: For off-diagonal elements, ﬁrst notice that ( M rω,ω ) i
Revised: For off-diagonal elements, ﬁrst notice that ( M rω,ω ) i

Original: ( i + κ − ω )
Revised: ( i + κ − ω )

Original: =
Revised: =

Original: ( M rω,ω ) i ( i + ω − κ ) .
Revised: ( M rω,ω ) i ( i + ω − κ ) .

Original: Then for q + 1 − r =
Revised: Then for q + 1 − r =

Original: pω + v , we can consider each subsegment uω ≤ t < ( u + 1) ω separately, and by a simple counting argument, get ( M rω,ω ) i
Revised: pω + v , we can consider each subsegment uω ≤ t < ( u + 1) ω separately, and by a simple counting argument, get ( M rω,ω ) i

Original: ( i + ω − κ )
Revised: ( i + ω − κ )

Original: +( M rω,ω )
Revised: +( M rω,ω )

Original: i ( i + κ ) = 1 − κω .
Revised: i ( i + κ ) = 1 − κω .

Original: We deduce that ( H 0 )
Revised: We deduce that ( H 0 )

Original: i ( i + κ ) = 0 , which by symmetry implies H 

Revised: i ( i + κ ) = 0 , which by symmetry implies H

Original: = 

Revised: =

Original: and concludes the proof.
Revised: and concludes the proof.

Original: From the above result, we conjecture that more generally, for ∆ ≤ ω , the low-frequency eigenspaces of H CK ω remain approximately unchanged when applying a downsampling operation.
Revised: From the above result, we conjecture that more generally, for ∆ ≤ ω , the low-frequency eigenspaces of H CK ω remain approximately unchanged when applying a downsampling operation.

Original: We verify this conjecture numerically in several examples.
Revised: We verify this conjecture numerically in several examples.

Original: In Figure 3, we plot the eigenvalues κ j with and without downsampling.
Revised: In Figure 3, we plot the eigenvalues κ j with and without downsampling.

Original: On the left, we compare κ j for ﬁxed ω = 25 and increasing ∆ .
Revised: On the left, we compare κ j for ﬁxed ω = 25 and increasing ∆ .

Original: We notice that the eigenvalues do not change much for ∆ ≤ ω , and for ∆ > ω , some κ j become null, as discussed above.
Revised: We notice that the eigenvalues do not change much for ∆ ≤ ω , and for ∆ > ω , some κ j become null, as discussed above.

Original: On the right, we plot κ j for ∆ = 1 (continuous line) and ∆ = ω
Revised: On the right, we plot κ j for ∆ = 1 (continuous line) and ∆ = ω

Original: (dashed lines) for several ω .
Revised: (dashed lines) for several ω .

Original: As conjectured, the top eigenvalues (low-frequency) are left approximately unchanged.
Revised: As conjectured, the top eigenvalues (low-frequency) are left approximately unchanged.

Original: In Figure 4, we plot a heatmap of the eigenvectors ordered vertically from highest associated eigenvalue (bottom) to lowest (top) for a ﬁxed ω = 25 and increasing downsampling ∆ ∈ { 1 , 25 , 40 } .
Revised: In Figure 4, we plot a heatmap of the eigenvectors ordered vertically from highest associated eigenvalue (bottom) to lowest (top) for a ﬁxed ω = 25 and increasing downsampling ∆ ∈ { 1 , 25 , 40 } .

Original: First indeed check that the top eigenvectors correspond to low-frequency functions and the bottom eigenvectors correspond to high-frequency functions.
Revised: First indeed check that the top eigenvectors correspond to low-frequency functions and the bottom eigenvectors correspond to high-frequency functions.

Original: Second, most eigenvectors are not much modiﬁed between ∆ = 1 and ∆ = ω
Revised: Second, most eigenvectors are not much modiﬁed between ∆ = 1 and ∆ = ω

Original: = 25 .
Revised: = 25 .

Original: For the case, ∆ > ω , the top eigenvectors corresponds still low-frequency functions.
Revised: For the case, ∆ > ω , the top eigenvectors corresponds still low-frequency functions.

Original: [Figure]
Revised: [Figure]

Original: From these observations, we expect H CK ω, ∆ to have the same statistical properties as H CK ω when learning low-frequency functions.
Revised: From these observations, we expect H CK ω, ∆ to have the same statistical properties as H CK ω when learning low-frequency functions.

Original: In Figure 5, we plot the test error of kernel ridge regression for ﬁtting cyclic q -local polynomials (see Section A.7) on the hypercube of dimension d = 30 .
Revised: In Figure 5, we plot the test error of kernel ridge regression for ﬁtting cyclic q -local polynomials (see Section A.7) on the hypercube of dimension d = 30 .

Original: We report the test error of one realization, against the sample size n , and choose regularization λ = 10 − 6 and noise σ ε = 0 .
Revised: We report the test error of one realization, against the sample size n , and choose regularization λ = 10 − 6 and noise σ ε = 0 .

Original: We compare kernels with and without downsampling.
Revised: We compare kernels with and without downsampling.

Original: On the left, we consider q =and ω =
Revised: On the left, we consider q =and ω =

Original: ∆ = 5 , and compare the test error with H CK ω (continous line) and with H CK ω, ∆ (dashed line) when learning degree 2 , 3 and 4 polynomials.
Revised: ∆ = 5 , and compare the test error with H CK ω (continous line) and with H CK ω, ∆ (dashed line) when learning degree 2 , 3 and 4 polynomials.

Original: On the right, we ﬁx the target function to be the cubic local cyclic polynomial and consider the test error of learning with H CK ω, ∆ for q = 10 , ω = 10 , and ∆ ∈ { 1 , 3 , 6 , 10 } .
Revised: On the right, we ﬁx the target function to be the cubic local cyclic polynomial and consider the test error of learning with H CK ω, ∆ for q = 10 , ω = 10 , and ∆ ∈ { 1 , 3 , 6 , 10 } .

Original: As expected, we observe in both simulations that the test error is almost identical between the kernels with and without downsampling, when learning cyclic invariant functions.
Revised: As expected, we observe in both simulations that the test error is almost identical between the kernels with and without downsampling, when learning cyclic invariant functions.

Original: In Section C.1, we further check that downsampling with ∆ > ω does not improve the high
Revised: In Section C.1, we further check that downsampling with ∆ > ω does not improve the high

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A.
Revised: A.

Original: M ULTILAYER CONVOLUTIONAL KERNELS
Revised: M ULTILAYER CONVOLUTIONAL KERNELS

Original: As an example
Revised: As an example

Original: [Figure]
Revised: [Figure]

Original: we will consider a two layers convolutional kernel with patch and local average pooling sizes ( q 1 , ω 1 ) on the ﬁrst layer and ( q 2 , ω 2 ) on the second layer.
Revised: we will consider a two layers convolutional kernel with patch and local average pooling sizes ( q 1 , ω 1 ) on the ﬁrst layer and ( q 2 , ω 2 ) on the second layer.

Original: We consider a general inner-product kernel for the ﬁrst layer:
Revised: We consider a general inner-product kernel for the ﬁrst layer:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the feature map is given explicitly ψ ( u ) = { ξ q 1 , | S | Y S ( u ) } S ⊆
Revised: where the feature map is given explicitly ψ ( u ) = { ξ q 1 , | S | Y S ( u ) } S ⊆

Original: [ q 1 ] ∈ R 2 q 1 .
Revised: [ q 1 ] ∈ R 2 q 1 .

Original: Following the work
Revised: Following the work

Original: Bietti (2021), we consider a degree- 2 polynomial kernel on the second layer, i.e., h 2 ( (cid:104) φ, φ (cid:48) (cid:105) )
Revised: Bietti (2021), we consider a degree- 2 polynomial kernel on the second layer, i.e., h 2 ( (cid:104) φ, φ (cid:48) (cid:105) )

Original: = (cid:104) φ, φ (cid:48) (cid:105) 2 .
Revised: = (cid:104) φ, φ (cid:48) (cid:105) 2 .

Original: Let us decompose this two-layers convolutional kernel in the Fourier basis.
Revised: Let us decompose this two-layers convolutional kernel in the Fourier basis.

Original: Let Ψ( x ) =
Revised: Let Ψ( x ) =

Original: { Ψ k ( x ) } k ∈ [ d ] be the output of the ﬁrst layer, with
Revised: { Ψ k ( x ) } k ∈ [ d ] be the output of the ﬁrst layer, with

Original: [Equation]
Revised: [Equation]

Original: Then denoting Ψ ( k ) ( x )
Revised: Then denoting Ψ ( k ) ( x )

Original: =
Revised: =

Original: (Ψ k +1 ( x ) , . . .
Revised: (Ψ k +1 ( x ) , . . .

Original: , Ψ k + q 2 ( x )) , the two-layers convolutional kernel is given by
Revised: , Ψ k + q 2 ( x )) , the two-layers convolutional kernel is given by

Original: [Equation]
Revised: [Equation]

Original: Let us comment the structure of H 2 CK ω 1 ,ω 2 :
Revised: Let us comment the structure of H 2 CK ω 1 ,ω 2 :

Original: The associated RKHS, which we will denote H 2 CK , contains all the homogeneous polynomials Y S with S = S 1 ∪ S 2 with S 1 , S 2 contained on segments of size q 1 , with the two segments separated by at most q 2 + ω 2 − 2 .
Revised: The associated RKHS, which we will denote H 2 CK , contains all the homogeneous polynomials Y S with S = S 1 ∪ S 2 with S 1 , S 2 contained on segments of size q 1 , with the two segments separated by at most q 2 + ω 2 − 2 .

Original: In words, the RKHS contains interaction between patches x ( k ) and x ( k (cid:48) )
Revised: In words, the RKHS contains interaction between patches x ( k ) and x ( k (cid:48) )

Original: that are within some distance.
Revised: that are within some distance.

Original: The eigenvalue associated to a degree- k homogeneous polynomials is still of order q − k in high-dimension.
Revised: The eigenvalue associated to a degree- k homogeneous polynomials is still of order q − k in high-dimension.

Original: To learn functions restricted to L 2 ( Q 2 , Loc q ) , it is statistically more efﬁcient to use H CK (smaller degeneracy of eigenvalues).
Revised: To learn functions restricted to L 2 ( Q 2 , Loc q ) , it is statistically more efﬁcient to use H CK (smaller degeneracy of eigenvalues).

Original: However H 2 CK will ﬁt a richer class of functions with two-patch interactions, while still not being plagued by dimensionality: dim( H 2 CK ) ≤ q 2 d
Revised: However H 2 CK will ﬁt a richer class of functions with two-patch interactions, while still not being plagued by dimensionality: dim( H 2 CK ) ≤ q 2 d

Original: 2 2 q 1 .
Revised: 2 2 q 1 .

Original: Hence we still expect H 2 CK to be much more statistically efﬁcient than a standard inner-product kernel.
Revised: Hence we still expect H 2 CK to be much more statistically efﬁcient than a standard inner-product kernel.

Original: Local pooling on the two layers plays different roles: pooling on the ﬁrst layer encourages the interactions to not depend strongly on the relative positions of the patches, while pooling on the second layer penalizes functions that depend on the global position of these interactions.
Revised: Local pooling on the two layers plays different roles: pooling on the ﬁrst layer encourages the interactions to not depend strongly on the relative positions of the patches, while pooling on the second layer penalizes functions that depend on the global position of these interactions.

Original: We believe that Eq.
Revised: We believe that Eq.

Original: (28) can be studied in more details, by a careful combinatorial argument and a 2-dimensional Fourier transform on the second layer (see Bietti (2021)).
Revised: (28) can be studied in more details, by a careful combinatorial argument and a 2-dimensional Fourier transform on the second layer (see Bietti (2021)).

Original: We leave this problem to future work.
Revised: We leave this problem to future work.

Original: Similarly, we can consider instead a degree- k kernel on the second layer (which would include interactions between k patches), or three layers and deeper networks.
Revised: Similarly, we can consider instead a degree- k kernel on the second layer (which would include interactions between k patches), or three layers and deeper networks.

Original: A. P ROOFS DIAGONALIZATION OF CONVOLUTIONAL KERNELS
Revised: A. P ROOFS DIAGONALIZATION OF CONVOLUTIONAL KERNELS

Original: In this section, we prove the diagonalization of the kernels H CK , H CK ω and H CK ω, ∆ introduced in
Revised: In this section, we prove the diagonalization of the kernels H CK , H CK ω and H CK ω, ∆ introduced in

Original: Propositions 1, 2 and 4 respectively.
Revised: Propositions 1, 2 and 4 respectively.

Original: Recall that we can associate to a kernel function H :
Revised: Recall that we can associate to a kernel function H :

Original: X × X → R deﬁned on a probability space ( X , τ ) (assume x (cid:55)→ H ( x , x ) square integrable), the integral operator H : L 2 ( X , τ ) →
Revised: X × X → R deﬁned on a probability space ( X , τ ) (assume x (cid:55)→ H ( x , x ) square integrable), the integral operator H : L 2 ( X , τ ) →

Original: L 2 ( X , τ )
Revised: L 2 ( X , τ )

Original: [Equation]
Revised: [Equation]

Original: By the spectral theorem of compact operators, there exists an orthonormal basis ( ψ j ) j ≥ 1 of L 2 ( X , τ ) and eigenvalues ( λ j ) j ≥ 1 , with nonincreasing values λ 1 ≥ λ 2 ≥ · · · ≥ 0 and (cid:80) j ≥ 1 λ j < ∞ , such that
Revised: By the spectral theorem of compact operators, there exists an orthonormal basis ( ψ j ) j ≥ 1 of L 2 ( X , τ ) and eigenvalues ( λ j ) j ≥ 1 , with nonincreasing values λ 1 ≥ λ 2 ≥ · · · ≥ 0 and (cid:80) j ≥ 1 λ j < ∞ , such that

Original: H
Revised: H

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: j
Revised: j

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst prove the diagonalization of H CK ω, ∆ in Proposition 4.
Revised: We ﬁrst prove the diagonalization of H CK ω, ∆ in Proposition 4.

Original: The case of H CK ω and H CK then follows by setting ∆ = 1 , and ∆ = ω
Revised: The case of H CK ω and H CK then follows by setting ∆ = 1 , and ∆ = ω

Original: = 1 respectively.
Revised: = 1 respectively.

Original: Proof of Proposition 4.
Revised: Proof of Proposition 4.

Original: Consider the inner-product kernel function h : R → R deﬁned on the hypercube Q q .
Revised: Consider the inner-product kernel function h : R → R deﬁned on the hypercube Q q .

Original: By rotational symmetry (see Section 2.1 and Appendix D), h admits the following diagonalization: for any u , v ∈ Q q ,
Revised: By rotational symmetry (see Section 2.1 and Appendix D), h admits the following diagonalization: for any u , v ∈ Q q ,

Original: [Equation]
Revised: [Equation]

Original: where ( Y S ) S ⊆
Revised: where ( Y S ) S ⊆

Original: [ q ] is the Fourier basis on Q q , and ξ d,(cid:96) ( h ) is the (cid:96) -th
Revised: [ q ] is the Fourier basis on Q q , and ξ d,(cid:96) ( h ) is the (cid:96) -th

Original: Gegenbauer coefﬁcient of h in dimension q (see Sections 2.1 or D for background).
Revised: Gegenbauer coefﬁcient of h in dimension q (see Sections 2.1 or D for background).

Original: Recall that we deﬁned S (cid:96) =
Revised: Recall that we deﬁned S (cid:96) =

Original: { S ⊆ [ q ] : | S | = (cid:96) } , the equivalence relation S ∼ S (cid:48) if S (cid:48) is a translated subset of S in [ q ] (without cyclic convention), and C (cid:96) the quotient set of A (cid:96) by ∼ .
Revised: { S ⊆ [ q ] : | S | = (cid:96) } , the equivalence relation S ∼ S (cid:48) if S (cid:48) is a translated subset of S in [ q ] (without cyclic convention), and C (cid:96) the quotient set of A (cid:96) by ∼ .

Original: For each equivalence class S ∈ C (cid:96) , consider S the unique subset in S that contains 1 .
Revised: For each equivalence class S ∈ C (cid:96) , consider S the unique subset in S that contains 1 .

Original: Then the equivalence class S contains the subsets u + S
Revised: Then the equivalence class S contains the subsets u + S

Original: = { u + k : k ∈ S } ⊆
Revised: = { u + k : k ∈ S } ⊆

Original: [ q ] with u = 0 , . .
Revised: [ q ] with u = 0 , . .

Original: , q − γ ( S ) .
Revised: , q − γ ( S ) .

Original: By a slight abuse of notations, we will identify S and this subset S .
Revised: By a slight abuse of notations, we will identify S and this subset S .

Original: Below we will denote u + S the translated subset with cyclic convention on [ d ] (e.g., 2 + { 1 , 3 , d − 1 } = { 3 , 5 , 1 } ).
Revised: Below we will denote u + S the translated subset with cyclic convention on [ d ] (e.g., 2 + { 1 , 3 , d − 1 } = { 3 , 5 , 1 } ).

Original: Using Eq.
Revised: Using Eq.

Original: (30) and that Y S ( x ( k ) )
Revised: (30) and that Y S ( x ( k ) )

Original: =
Revised: =

Original: Y k + S ( x ) , we have the following decomposition of H CK ω, ∆ in the
Revised: Y k + S ( x ) , we have the following decomposition of H CK ω, ∆ in the

Original: Fourier basis
Revised: Fourier basis

Original: [Equation]
Revised: [Equation]

Original: where we recall the deﬁnition of the set of indices
Revised: where we recall the deﬁnition of the set of indices

Original: [Equation]
Revised: [Equation]

Original: Note that the diagonalization of the kernel H can be obtained by computing the matrix M =
Revised: Note that the diagonalization of the kernel H can be obtained by computing the matrix M =

Original: ( M SS (cid:48) ) S,S (cid:48) ⊆
Revised: ( M SS (cid:48) ) S,S (cid:48) ⊆

Original: [ d ] ∈ R 2 d × 2 d with M = E x , y [ Y S ( x ) H ( x , y )
Revised: [ d ] ∈ R 2 d × 2 d with M = E x , y [ Y S ( x ) H ( x , y )

Original: Y S (cid:48)
Revised: Y S (cid:48)

Original: ( y )] : if λ j and v j ∈ R 2 d are the eigenvalues and eigenvectors of M , then λ j and ψ j ( x )
Revised: ( y )] : if λ j and v j ∈ R 2 d are the eigenvalues and eigenvectors of M , then λ j and ψ j ( x )

Original: = (cid:80) S ⊆
Revised: = (cid:80) S ⊆

Original: [ d ] v j,S Y S ( x ) are the eigenvalues and eigenvectors of H .
Revised: [ d ] v j,S Y S ( x ) are the eigenvalues and eigenvectors of H .

Original: From Eq.
Revised: From Eq.

Original: (31), we see 1) the basis functions Y S with γ ( S ) >
Revised: (31), we see 1) the basis functions Y S with γ ( S ) >

Original: q (subset S not contained in a segment of size q ) are in the null space of H CK ω, ∆ , 2) for S, S (cid:48) ⊆
Revised: q (subset S not contained in a segment of size q ) are in the null space of H CK ω, ∆ , 2) for S, S (cid:48) ⊆

Original: [ d ] with S and S (cid:48) not translations of each other, then E x , y [ Y S ( x ) H CK ω, ∆ ( x , y )
Revised: [ d ] with S and S (cid:48) not translations of each other, then E x , y [ Y S ( x ) H CK ω, ∆ ( x , y )

Original: Y S (cid:48)
Revised: Y S (cid:48)

Original: ( y )]
Revised: ( y )]

Original: = 0 , and Y S and Y S (cid:48) are contained in orthogonal eigenspaces.
Revised: = 0 , and Y S and Y S (cid:48) are contained in orthogonal eigenspaces.

Original: We deduce that it is sufﬁcient to diagonalize H CK ω, ∆ on each of the (orthogonal) subspaces V S :
Revised: We deduce that it is sufﬁcient to diagonalize H CK ω, ∆ on each of the (orthogonal) subspaces V S :

Original: = span { Y k + S : k ∈
Revised: = span { Y k + S : k ∈

Original: [ d ] } for 0 ≤ (cid:96) ≤ q and S ∈ C (cid:96) .
Revised: [ d ] } for 0 ≤ (cid:96) ≤ q and S ∈ C (cid:96) .

Original: For each S ∈
Revised: For each S ∈

Original: C (cid:96) , deﬁne M
Revised: C (cid:96) , deﬁne M

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) ∈
Revised: ) ∈

Original: R d × d the matrix with entries M γ ( S ) ij
Revised: R d × d the matrix with entries M γ ( S ) ij

Original: [Equation]
Revised: [Equation]

Original: 1 r ( S ) E x , y
Revised: 1 r ( S ) E x , y

Original: [ Y i + S ( x ) H CK ω, ∆ ( x , y )
Revised: [ Y i + S ( x ) H CK ω, ∆ ( x , y )

Original: Y j + S ( y )] .
Revised: Y j + S ( y )] .

Original: From Eq.
Revised: From Eq.

Original: (31), we get
Revised: (31), we get

Original: [Equation]
Revised: [Equation]

Original: 33) which concludes the proof of Proposition 4.
Revised: 33) which concludes the proof of Proposition 4.

Original: We can now prove Propositions 1 and 2 by taking ω
Revised: We can now prove Propositions 1 and 2 by taking ω

Original: =
Revised: =

Original: ∆ = 1 and ∆ = 1 respectively.
Revised: ∆ = 1 and ∆ = 1 respectively.

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: Set ∆ = ω = 

Revised: Set ∆ = ω =

Original: in Proposition 4.
Revised: in Proposition 4.

Original: We get
Revised: We get

Original: [Equation]
Revised: [Equation]

Original: In this case, M γ ( S ) is simply equal to identity, which concludes the proof.
Revised: In this case, M γ ( S ) is simply equal to identity, which concludes the proof.

Original: Proof of Proposition 2.
Revised: Proof of Proposition 2.

Original: Set ∆ = 1 in Proposition 4.
Revised: Set ∆ = 1 in Proposition 4.

Original: We get
Revised: We get

Original: [Equation]
Revised: [Equation]

Original: + , where d ( i, j ) is the distance between i and j on the torus [ d ] (i.e., if i > j , d ( i, j )
Revised: + , where d ( i, j ) is the distance between i and j on the torus [ d ] (i.e., if i > j , d ( i, j )

Original: =
Revised: =

Original: min( i − j, d + j − i ) ).
Revised: min( i − j, d + j − i ) ).

Original: Hence, M γ ( S ) is a circulant matrix independent of γ ( S ) , which has well known explicit formula for eigenvalues and eigenvectors (see for example Remark A.1).
Revised: Hence, M γ ( S ) is a circulant matrix independent of γ ( S ) , which has well known explicit formula for eigenvalues and eigenvectors (see for example Remark A.1).

Original: A. A DDITIONAL NUMERICAL SIMULATIONS
Revised: A. A DDITIONAL NUMERICAL SIMULATIONS

Original: Here, we consider a numerical experiment similar to Figure 1.
Revised: Here, we consider a numerical experiment similar to Figure 1.

Original: We consider x ∼ Unif( Q d ) with d = 30 and consider three cyclic invariant target functions:
Revised: We consider x ∼ Unif( Q d ) with d = 30 and consider three cyclic invariant target functions:

Original: [Equation]
Revised: [Equation]

Original: We consider a higher order polynomial kernel h ( x ) = (cid:80) k ∈
Revised: We consider a higher order polynomial kernel h ( x ) = (cid:80) k ∈

Original: 2 · x k than in Figure 1, which should lead to higher self-induced regularization.
Revised: 2 · x k than in Figure 1, which should lead to higher self-induced regularization.

Original: We consider the same kernels as before, with q = 10 and ω = 5 .
Revised: We consider the same kernels as before, with q = 10 and ω = 5 .

Original: In Figure 6, we report the test errors of ﬁtting f 2 (top), f 3 (middle) and f 4 (bottom) using kernel ridge regression with the 5 kernels of interests in the main text.
Revised: In Figure 6, we report the test errors of ﬁtting f 2 (top), f 3 (middle) and f 4 (bottom) using kernel ridge regression with the 5 kernels of interests in the main text.

Original: We choose a small regularization parameter λ
Revised: We choose a small regularization parameter λ

Original: = 10 − 6 , and the noise level σ ε = 0 .
Revised: = 10 − 6 , and the noise level σ ε = 0 .

Original: The curves are averaged over 5 independent instances and the error bar stands for the standard deviation of these instances.
Revised: The curves are averaged over 5 independent instances and the error bar stands for the standard deviation of these instances.

Original: The results again match with our overall theoretical predictions.
Revised: The results again match with our overall theoretical predictions.

Original: We report the predicted thresholds for the three functions:
Revised: We report the predicted thresholds for the three functions:

Original: For f 2 target: q < d < dq/ω < dq < d 2 for H CKGP <
Revised: For f 2 target: q < d < dq/ω < dq < d 2 for H CKGP <

Original: H FCGP <
Revised: H FCGP <

Original: H CK ω <
Revised: H CK ω <

Original: H CK < H FC .
Revised: H CK < H FC .

Original: For f 3 target:
Revised: For f 3 target:

Original: q 2 < dq 2 /ω <
Revised: q 2 < dq 2 /ω <

Original: d 2 < dq 2 < d 3 for H CKGP <
Revised: d 2 < dq 2 < d 3 for H CKGP <

Original: H CK ω <
Revised: H CK ω <

Original: H CK < H FCGP <
Revised: H CK < H FCGP <

Original: H FC .
Revised: H FC .

Original: For f 4 target:
Revised: For f 4 target:

Original: q 3 < dq 3 /ω <
Revised: q 3 < dq 3 /ω <

Original: d 3 < dq 3 < d 4 for H CKGP <
Revised: d 3 < dq 3 < d 4 for H CKGP <

Original: H CK ω <
Revised: H CK ω <

Original: H CK < H FCGP <
Revised: H CK < H FCGP <

Original: H FC .
Revised: H FC .

Original: We see that the kernels, especially for f 4 , perform much better than their theoretical high-dimension predictions: this can be explained by the low-dimensionality of the experiment where q = 10 .
Revised: We see that the kernels, especially for f 4 , perform much better than their theoretical high-dimension predictions: this can be explained by the low-dimensionality of the experiment where q = 10 .

Original: [Figure]
Revised: [Figure]

Original: B G ENERALIZATION ERROR OF KERNEL METHODS IN FIXED DIMENSION B. B OUND ON KERNEL METHODS USING R ADEMACHER COMPLEXITIES
Revised: B G ENERALIZATION ERROR OF KERNEL METHODS IN FIXED DIMENSION B. B OUND ON KERNEL METHODS USING R ADEMACHER COMPLEXITIES

Original: We ﬁrst consider the case of a Lipschitz bounded loss and uniform convergence, and make a few simple remarks on the connection between generalization error and eigendecomposition in kernel methods.
Revised: We ﬁrst consider the case of a Lipschitz bounded loss and uniform convergence, and make a few simple remarks on the connection between generalization error and eigendecomposition in kernel methods.

Original: Consider i.i.d data ( x i , y i )
Revised: Consider i.i.d data ( x i , y i )

Original: ∈
Revised: ∈

Original: X × R with ( x , y ) ∼ P and a loss function (cid:96) : R × R → R that we take 1 -Lipschitz w.r.t second argument and bounded by 1 .
Revised: X × R with ( x , y ) ∼ P and a loss function (cid:96) : R × R → R that we take 1 -Lipschitz w.r.t second argument and bounded by 1 .

Original: The goal is to minimize the expected loss
Revised: The goal is to minimize the expected loss

Original: L ( ˆ f ) =
Revised: L ( ˆ f ) =

Original: E y , x { (cid:96) ( y, ˆ f ( x )) } .
Revised: E y , x { (cid:96) ( y, ˆ f ( x )) } .

Original: Take a RKHS H with kernel function H :
Revised: Take a RKHS H with kernel function H :

Original: X × X → R and consider following constrained empirical risk minimizer:
Revised: X × X → R and consider following constrained empirical risk minimizer:

Original: [Equation]
Revised: [Equation]

Original: The generalization error of ˆ f B has the following standard bound on the Rademacher complexity of the kernel class { f : (cid:107) f (cid:107) H ≤ B } (Boucheron et al., 2005; Shalev-Shwartz & Ben-David, 2014): with probability 1 − δ ,
Revised: The generalization error of ˆ f B has the following standard bound on the Rademacher complexity of the kernel class { f : (cid:107) f (cid:107) H ≤ B } (Boucheron et al., 2005; Shalev-Shwartz & Ben-David, 2014): with probability 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Note that instead of a constraint on the norm in Eq.
Revised: Note that instead of a constraint on the norm in Eq.

Original: (34), one might ﬁnd more convenient to use a penalty.
Revised: (34), one might ﬁnd more convenient to use a penalty.

Original: In that case, there exists an equivalent to the bound (35) (Wainwright, 2019; Bach, 2021), but we focus here on the constrained formulation for simplicity.
Revised: In that case, there exists an equivalent to the bound (35) (Wainwright, 2019; Bach, 2021), but we focus here on the constrained formulation for simplicity.

Original: From the bound (35), we see that the generalization error depends crucially on the choice of B .
Revised: From the bound (35), we see that the generalization error depends crucially on the choice of B .

Original: For simplicity, let us forget about the approximation error and take (cid:107) f (cid:63) (cid:107) H ≤ B where f (cid:63) =
Revised: For simplicity, let us forget about the approximation error and take (cid:107) f (cid:63) (cid:107) H ≤ B where f (cid:63) =

Original: E { y | x } .
Revised: E { y | x } .

Original: Recall that for a kernel H with eigenvalues { λ j } j ≥ 1 and eigenvectors { ψ j } j ≥ 1 , we have
Revised: Recall that for a kernel H with eigenvalues { λ j } j ≥ 1 and eigenvectors { ψ j } j ≥ 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Consider H CK ω, ∆ as in Eq.
Revised: Consider H CK ω, ∆ as in Eq.

Original: (8) and assume ξ q, 0 = 0 .
Revised: (8) and assume ξ q, 0 = 0 .

Original: From the normalization choice of the kernel (see
Revised: From the normalization choice of the kernel (see

Original: Eq.
Revised: Eq.

Original: (23)), we have
Revised: (23)), we have

Original: [Equation]
Revised: [Equation]

Original: Consider now for simplicity ∆ = 1 .
Revised: Consider now for simplicity ∆ = 1 .

Original: From the eigendecomposition in Proposition 2, the RKHS norm of f ∈ L 2 ( Q d , Loc q ) is given by
Revised: From the eigendecomposition in Proposition 2, the RKHS norm of f ∈ L 2 ( Q d , Loc q ) is given by

Original: [Equation]
Revised: [Equation]

Original: Consider the case where f ∈ L 2 ( Q d , Loc q ) has a unique non-zero component in its discrete Fourier transform,
Revised: Consider the case where f ∈ L 2 ( Q d , Loc q ) has a unique non-zero component in its discrete Fourier transform,

Original: i.e., f ( x ) = 1 √ d (cid:80) k ∈ [ d ] ρ k j g ( x ( k ) ) with E { g ( x ) } = 0 and ρ j = e 2 iπj/d (see Section
Revised: i.e., f ( x ) = 1 √ d (cid:80) k ∈ [ d ] ρ k j g ( x ( k ) ) with E { g ( x ) } = 0 and ρ j = e 2 iπj/d (see Section

Original: A.3).
Revised: A.3).

Original: Note that, denoting c S =
Revised: Note that, denoting c S =

Original: (cid:104) Y S , g (cid:105)
Revised: (cid:104) Y S , g (cid:105)

Original: L 2 ( Q q ) :
Revised: L 2 ( Q q ) :

Original: [Equation]
Revised: [Equation]

Original: ρ − u j c u + S   ψ j,S .
Revised: ρ − u j c u + S   ψ j,S .

Original: Hence, (cid:107) f (cid:107)
Revised: Hence, (cid:107) f (cid:107)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (cid:96) =
Revised: (cid:96) =

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]
Revised: [Equation]

Original: d (cid:107) g (cid:107)
Revised: d (cid:107) g (cid:107)

Original: [Equation]
Revised: [Equation]

Original: h κ
Revised: h κ

Original: [Equation]
Revised: [Equation]

Original: , where (cid:107) g (cid:107) 2 h is the RKHS norm associated to the inner-product kernel h : R → R in Q q , i.e., (cid:107) g (cid:107) 

Revised: , where (cid:107) g (cid:107) 2 h is the RKHS norm associated to the inner-product kernel h : R → R in Q q , i.e., (cid:107) g (cid:107)

Original: h
Revised: h

Original: =
Revised: =

Original: (cid:80) S ⊆
Revised: (cid:80) S ⊆

Original: [ q ] c S ξ q, | S | .
Revised: [ q ] c S ξ q, | S | .

Original: From the bound (35), we deduce the ﬁrst generalization bound using a convolutional kernel: with probability at least 1 − δ ,
Revised: From the bound (35), we deduce the ﬁrst generalization bound using a convolutional kernel: with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: We make the following two remarks on this bound:
Revised: We make the following two remarks on this bound:

Original: It depends on (cid:107) g (cid:107) h , which is a RKHS norm on Q q instead of Q d , which has potentially much lower dimension and contain less smooth function for balls of same radius.
Revised: It depends on (cid:107) g (cid:107) h , which is a RKHS norm on Q q instead of Q d , which has potentially much lower dimension and contain less smooth function for balls of same radius.

Original: There is a factor κ j gain in sample complexity when learning functions that have j -th frequency with κ j > 1 .
Revised: There is a factor κ j gain in sample complexity when learning functions that have j -th frequency with κ j > 1 .

Original: In particular, for j = d (cyclic invariant functions), κ j = ω , and we need ω less samples to get the same (upper) bound on the generalization error.
Revised: In particular, for j = d (cyclic invariant functions), κ j = ω , and we need ω less samples to get the same (upper) bound on the generalization error.

Original: On the contrary, when κ j < 1 , i.e., high-frequency oscillatory functions, the generalization bound becomes worse.
Revised: On the contrary, when κ j < 1 , i.e., high-frequency oscillatory functions, the generalization bound becomes worse.

Original: B. G ENERALIZATION ERROR OF KRR IN THE CLASSICAL REGIME
Revised: B. G ENERALIZATION ERROR OF KRR IN THE CLASSICAL REGIME

Original: We consider here the regression setting which allows for ﬁner results.
Revised: We consider here the regression setting which allows for ﬁner results.

Original: Several works have considered bounding the generalization error of kernel ridge regression (KRR) Caponnetto & De Vito (2007);
Revised: Several works have considered bounding the generalization error of kernel ridge regression (KRR) Caponnetto & De Vito (2007);

Original: Jacot et al.
Revised: Jacot et al.

Original: (2020), (Wainwright, 2019, Theorem 13.17).
Revised: (2020), (Wainwright, 2019, Theorem 13.17).

Original: In this section, we consider the following fully-explicit upper bound from Bach (2021).
Revised: In this section, we consider the following fully-explicit upper bound from Bach (2021).

Original: Consider i.i.d data ( x i , y i )
Revised: Consider i.i.d data ( x i , y i )

Original: ∈
Revised: ∈

Original: X × R with x i ∼ P , and y i =
Revised: X × R with x i ∼ P , and y i =

Original: f (cid:63) ( x i )
Revised: f (cid:63) ( x i )

Original: + ε i .
Revised: + ε i .

Original: Assume the noise E [ ε i | x i ] = 0 and E [ ε 2 i | x i ] ≤ σ 2 ε , and denote ε = ( ε 1 , . .
Revised: Assume the noise E [ ε i | x i ] = 0 and E [ ε 2 i | x i ] ≤ σ 2 ε , and denote ε = ( ε 1 , . .

Original: . , ε n ) .
Revised: . , ε n ) .

Original: Let H be a RKHS with reproducing kernel H :
Revised: Let H be a RKHS with reproducing kernel H :

Original: X × X → R .
Revised: X × X → R .

Original: The KRR solution with regularization parameter λ ≥ 0 is given by
Revised: The KRR solution with regularization parameter λ ≥ 0 is given by

Original: [Equation]
Revised: [Equation]

Original: which has the following analytical formula:
Revised: which has the following analytical formula:

Original: [Equation]
Revised: [Equation]

Original: where H =
Revised: where H =

Original: ( H ( x i , x j )) ij ∈ [ n ] is the empirical kernel matrix, h ( x ) =
Revised: ( H ( x i , x j )) ij ∈ [ n ] is the empirical kernel matrix, h ( x ) =

Original: [ H ( x , x 1 ) , . . .
Revised: [ H ( x , x 1 ) , . . .

Original: , H ( x , x n )] and y =
Revised: , H ( x , x n )] and y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y n ) .
Revised: , y n ) .

Original: The risk is taken to be the test error with squared error loss
Revised: The risk is taken to be the test error with squared error loss

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Below, we give an upper bound on the expected risk over the noise ε in the training data, i.e.,
Revised: Below, we give an upper bound on the expected risk over the noise ε in the training data, i.e.,

Original: E ε { R ( f (cid:63) , ˆ f λ ) } (it is also possible to give high probability bounds by concentration arguments, but we restrict ourselves to bounding the expected risk).
Revised: E ε { R ( f (cid:63) , ˆ f λ ) } (it is also possible to give high probability bounds by concentration arguments, but we restrict ourselves to bounding the expected risk).

Original: Theorem 6.
Revised: Theorem 6.

Original: (Bach, 2021, Theorem 7.2) Assume H ( x , x ) ≤ R 2 almost surely and let the regularization parameter λ ≤ R 2 .
Revised: (Bach, 2021, Theorem 7.2) Assume H ( x , x ) ≤ R 2 almost surely and let the regularization parameter λ ≤ R 2 .

Original: If n ≥ 5 R 2 λ (cid:16) 1 + log R 2 λ (cid:17) , then
Revised: If n ≥ 5 R 2 λ (cid:16) 1 + log R 2 λ (cid:17) , then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where N ( H, λ )
Revised: where N ( H, λ )

Original: =
Revised: =

Original: Tr[( H + λ I ) − 1 H ] .
Revised: Tr[( H + λ I ) − 1 H ] .

Original: Let us comment on the upper-bound in Eq.
Revised: Let us comment on the upper-bound in Eq.

Original: The ﬁrst term corresponds to an upper bound on the variance: N ( H, λ ) is sometimes called the degrees of freedom or the effective dimension of the kernel H .
Revised: The ﬁrst term corresponds to an upper bound on the variance: N ( H, λ ) is sometimes called the degrees of freedom or the effective dimension of the kernel H .

Original: The second term bounds the bias term and corresponds to an approximation error.
Revised: The second term bounds the bias term and corresponds to an approximation error.

Original: In particular, for any r > 0 ,
Revised: In particular, for any r > 0 ,

Original: [Equation]
Revised: [Equation]

Original: 38) where we recall that H is the integral operator associated to H (see Eq.
Revised: 38) where we recall that H is the integral operator associated to H (see Eq.

Original: The third term can be removed by a more intricate analysis.
Revised: The third term can be removed by a more intricate analysis.

Original: From the above discussion, it is natural to consider the following two assumptions on H and f (cid:63) , that are standard in the kernel literature:
Revised: From the above discussion, it is natural to consider the following two assumptions on H and f (cid:63) , that are standard in the kernel literature:

Original: (B1)
Revised: (B1)

Original: Capacity condition: N ( H, λ ) ≤ C H λ − 1 /α with α > 1 . (B2)
Revised: Capacity condition: N ( H, λ ) ≤ C H λ − 1 /α with α > 1 . (B2)

Original: Source condition: there exists β > 0 such that (cid:107) H − β/ 2 f (cid:63) (cid:107) 2 L 2 =:
Revised: Source condition: there exists β > 0 such that (cid:107) H − β/ 2 f (cid:63) (cid:107) 2 L 2 =:

Original: B 2 f (cid:63) < ∞ .
Revised: B 2 f (cid:63) < ∞ .

Original: Intuitively, the capacity condition (B1) characterizes the size of the RKHS: for increasing α , the
Revised: Intuitively, the capacity condition (B1) characterizes the size of the RKHS: for increasing α , the

Original: RKHS contains less and less functions.
Revised: RKHS contains less and less functions.

Original: It is veriﬁed when the eigenvalues
Revised: It is veriﬁed when the eigenvalues

Original: λ j ’s of H decay at the rate j − α .
Revised: λ j ’s of H decay at the rate j − α .

Original: For example, taking the Matern kernel of order s > d/ 2 , whose RKHS is the Sobolev space of order s (i.e., functions with bounded s -order derivatives), we have α = 2 s/d (e.g., see
Revised: For example, taking the Matern kernel of order s > d/ 2 , whose RKHS is the Sobolev space of order s (i.e., functions with bounded s -order derivatives), we have α = 2 s/d (e.g., see

Original: Harchaoui et al.
Revised: Harchaoui et al.

Original: The source condition (B2) characterizes the regularity of the target function (the ‘source’) with respect to the kernel: β = 1 is equivalent to f (cid:63) ∈ H , while β > 1 corresponds to f (cid:63) more smooth (and β < 1 less smooth f (cid:63) ).
Revised: The source condition (B2) characterizes the regularity of the target function (the ‘source’) with respect to the kernel: β = 1 is equivalent to f (cid:63) ∈ H , while β > 1 corresponds to f (cid:63) more smooth (and β < 1 less smooth f (cid:63) ).

Original: Assuming (B1) and (B2) in Theorem 6, we get the bound
Revised: Assuming (B1) and (B2) in Theorem 6, we get the bound

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where in the second line, we balanced the two terms by taking λ ∗ :
Revised: where in the second line, we balanced the two terms by taking λ ∗ :

Original: = (cid:16) C H
Revised: = (cid:16) C H

Original: [Equation]
Revised: [Equation]

Original: B 2 f(cid:63) n (cid:17)
Revised: B 2 f(cid:63) n (cid:17)

Original: α αβ +1 .
Revised: α αβ +1 .

Original: Note that in order to use Theorem 6, we need further to constrain n ≥ 5 R 2 λ (cid:16) 

Revised: Note that in order to use Theorem 6, we need further to constrain n ≥ 5 R 2 λ (cid:16)

Original: + log R 2 λ (cid:17) .
Revised: + log R 2 λ (cid:17) .

Original: For simplicity, we will choose r > α − 1 α , so that this condition is veriﬁed for n sufﬁciently large.
Revised: For simplicity, we will choose r > α − 1 α , so that this condition is veriﬁed for n sufﬁciently large.

Original: Remark B.1.
Revised: Remark B.1.

Original: The rate in n in Eq.
Revised: The rate in n in Eq.

Original: (39) is minmax optimal over all functions that verify assumptions (A1) and (A2) Caponnetto & De Vito (2007).
Revised: (39) is minmax optimal over all functions that verify assumptions (A1) and (A2) Caponnetto & De Vito (2007).

Original: However, for large d , the RKHS is composed of very smooth functions (e.g., Sobolev spaces of order s are RKHS if and only if s > d/ 2 , i.e., if the order of the bounded derivatives grows with the dimension d ) and β will be small,
Revised: However, for large d , the RKHS is composed of very smooth functions (e.g., Sobolev spaces of order s are RKHS if and only if s > d/ 2 , i.e., if the order of the bounded derivatives grows with the dimension d ) and β will be small,

Original: such that βα ≈ κ/d for functions with bounded derivatives up to order κ .
Revised: such that βα ≈ κ/d for functions with bounded derivatives up to order κ .

Original: In that case, the risk decreases at the rate n − O ( κd ) :
Revised: In that case, the risk decreases at the rate n − O ( κd ) :

Original: KRR suffers from the curse of dimensionality when κ does not scale with d .
Revised: KRR suffers from the curse of dimensionality when κ does not scale with d .

Original: As a consequence, the bound (39) is vacuous when n does not scale exponentially in d , which led several groups to derive ﬁner bounds on KRR in the high dimensional regime (see Section C).
Revised: As a consequence, the bound (39) is vacuous when n does not scale exponentially in d , which led several groups to derive ﬁner bounds on KRR in the high dimensional regime (see Section C).

Original: Let us now apply Theorem 6 and Eq.
Revised: Let us now apply Theorem 6 and Eq.

Original: (39) to our convolutional kernels to show Theorems 1 and 4.
Revised: (39) to our convolutional kernels to show Theorems 1 and 4.

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: First notice that H CK ( x , x ) = h (1)
Revised: First notice that H CK ( x , x ) = h (1)

Original: =:
Revised: =:

Original: R 2 and we can therefore apply Theorem 6.
Revised: R 2 and we can therefore apply Theorem 6.

Original: The effective dimension of H CK is bounded by
Revised: The effective dimension of H CK is bounded by

Original: [Equation]
Revised: [Equation]

Original: where we used that r ( S ) ≥ 1 in the second line and N ( h, λ ) is the effective dimension of the innerproduct kernel h on Q q .
Revised: where we used that r ( S ) ≥ 1 in the second line and N ( h, λ ) is the effective dimension of the innerproduct kernel h on Q q .

Original: We deduce from (A1) that N ( H CK , λ ) ≤ C h
Revised: We deduce from (A1) that N ( H CK , λ ) ≤ C h

Original: d 1 − 1 /α λ − 1 /α .
Revised: d 1 − 1 /α λ − 1 /α .

Original: Furthermore, from (A2) and the assumption that E { g k ( x ) } = 0 , we have
Revised: Furthermore, from (A2) and the assumption that E { g k ( x ) } = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Injecting the two above bounds in Eq.
Revised: Injecting the two above bounds in Eq.

Original: (39), we deduce that there exists constants C 1 , C 2 , C 3 that only depends on the constants in (A1) and (A2), and h (1) , σ 2 ε (but independent of d ), such that taking n ≥ C 1 max( (cid:107) f
Revised: (39), we deduce that there exists constants C 1 , C 2 , C 3 that only depends on the constants in (A1) and (A2), and h (1) , σ 2 ε (but independent of d ), such that taking n ≥ C 1 max( (cid:107) f

Original: (cid:63) (cid:107) 2 L ∞ , d ) and λ ∗ =
Revised: (cid:63) (cid:107) 2 L ∞ , d ) and λ ∗ =

Original: C 2 d ( d/n ) α αβ +1 , we get
Revised: C 2 d ( d/n ) α αβ +1 , we get

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 4.
Revised: Proof of Theorem 4.

Original: The proof is similar to the proof of Theorem 1.
Revised: The proof is similar to the proof of Theorem 1.

Original: Notice that H CK ω ( x , x ) ≤ h (1) , and that the effective dimension of H CK ω is bounded by
Revised: Notice that H CK ω ( x , x ) ≤ h (1) , and that the effective dimension of H CK ω is bounded by

Original: [Equation]
Revised: [Equation]

Original: where we used condition (A1).
Revised: where we used condition (A1).

Original: Denoting d eff = (cid:80) d j =1 ( κ j /ω ) 1 /α , the rest of the proof follows from the proof of Theorem 1 with d replaced by d eff ω 1 /α and B 2 replaced by ω β B 2 .
Revised: Denoting d eff = (cid:80) d j =1 ( κ j /ω ) 1 /α , the rest of the proof follows from the proof of Theorem 1 with d replaced by d eff ω 1 /α and B 2 replaced by ω β B 2 .

Original: Remark B.2.
Revised: Remark B.2.

Original: Note that the requirement (cid:107) ( H CK ω /ω ) − β/ 

Revised: Note that the requirement (cid:107) ( H CK ω /ω ) − β/

Original: f (cid:63) (cid:107) L 2 ≤ B is to make the result comparable to the other theorems when we consider target functions with low-frequencies.
Revised: f (cid:63) (cid:107) L 2 ≤ B is to make the result comparable to the other theorems when we consider target functions with low-frequencies.

Original: For a cyclic invariant function
Revised: For a cyclic invariant function

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: C G ENERALIZATION ERROR OF KRR IN HIGH DIMENSION
Revised: C G ENERALIZATION ERROR OF KRR IN HIGH DIMENSION

Original: In Section B.2, we considered upper bounds on the test error of KRR using the standard capacity and source conditions .
Revised: In Section B.2, we considered upper bounds on the test error of KRR using the standard capacity and source conditions .

Original: However, these results suffer from several limitations:
Revised: However, these results suffer from several limitations:

Original: They only provide an upper bound on the test error.
Revised: They only provide an upper bound on the test error.

Original: While the decay rate with respect to n is minmax optimal (see Caponnetto & De Vito (2007)), this is not strong enough to show, for example, a statistical advantage of using local average pooling, which appears as a prefactor d eff , and which would require a lower bound matching the upper bound within a constant factor.
Revised: While the decay rate with respect to n is minmax optimal (see Caponnetto & De Vito (2007)), this is not strong enough to show, for example, a statistical advantage of using local average pooling, which appears as a prefactor d eff , and which would require a lower bound matching the upper bound within a constant factor.

Original: As mentioned in Remark B.1, the bound is of order n − 1 /O
Revised: As mentioned in Remark B.1, the bound is of order n − 1 /O

Original: ( d ) , except when the target function has smoothness order increasing with d .
Revised: ( d ) , except when the target function has smoothness order increasing with d .

Original: This bound is non-vacuous only if n = exp
Revised: This bound is non-vacuous only if n = exp

Original: ( O ( d )) which is impractical in modern image datasets where typically d ≥ 100 .
Revised: ( O ( d )) which is impractical in modern image datasets where typically d ≥ 100 .

Original: This motivates a new type of question: given n (cid:16) d α , what is the prediction error achieved by KRR for a given function?
Revised: This motivates a new type of question: given n (cid:16) d α , what is the prediction error achieved by KRR for a given function?

Original: In order to achieve the bound Eq.
Revised: In order to achieve the bound Eq.

Original: (39), one need to carefully balance the bias and the variance terms by setting the regularization parameter.
Revised: (39), one need to carefully balance the bias and the variance terms by setting the regularization parameter.

Original: This is in contrast with modern practice which usually train until interpolation (which corresponds to setting λ → 0 ).
Revised: This is in contrast with modern practice which usually train until interpolation (which corresponds to setting λ → 0 ).

Original: Given the above limitations, several recent works have instead considered a high-dimensional setting where the number of samples scales with d , and derived asymptotic test errors, exact up to a vanishing additive error Ghorbani et al.
Revised: Given the above limitations, several recent works have instead considered a high-dimensional setting where the number of samples scales with d , and derived asymptotic test errors, exact up to a vanishing additive error Ghorbani et al.

Original: (2021; 2020); Mei et al.
Revised: (2021; 2020); Mei et al.

Original: (2021a).
Revised: (2021a).

Original: In addition to these works, several papers have derived general estimates for the test error using non-rigorous methods
Revised: In addition to these works, several papers have derived general estimates for the test error using non-rigorous methods

Original: Jacot et al.
Revised: Jacot et al.

Original: Canatar et al.
Revised: Canatar et al.

Original: ; Cui et al.
Revised: ; Cui et al.

Original: (2021) that are believe to be correct in the high dimensional limit and which show great agreement with numerical experiments.
Revised: (2021) that are believe to be correct in the high dimensional limit and which show great agreement with numerical experiments.

Original: The picture that emerges in this regime is much more precise than in the classical regime: KRR approximately acts as a shrinkage operator on the target function (not assumed to be in a particular space anymore), with shrinkage parameter that scales as a self-induced regularization parameter over the number of samples.
Revised: The picture that emerges in this regime is much more precise than in the classical regime: KRR approximately acts as a shrinkage operator on the target function (not assumed to be in a particular space anymore), with shrinkage parameter that scales as a self-induced regularization parameter over the number of samples.

Original: More precisely, Mei et al.
Revised: More precisely, Mei et al.

Original: (2021a) shows the following: considers a kernel H d : R d × R d → R with eigenvalues ( λ d,j ) j ≥ 1 in nonincreasing order and n ≡ n ( d ) the number of samples.
Revised: (2021a) shows the following: considers a kernel H d : R d × R d → R with eigenvalues ( λ d,j ) j ≥ 1 in nonincreasing order and n ≡ n ( d ) the number of samples.

Original: Let m ≡ m ( d ) be an integer such that m ≤ n 1 − δ and
Revised: Let m ≡ m ( d ) be an integer such that m ≤ n 1 − δ and

Original: [Equation]
Revised: [Equation]

Original: for some δ > 0 .
Revised: for some δ > 0 .

Original: Then, assuming some additional conditions insuring that the kernel H d is ‘spreadout’ and well behaved, the KRR solution
Revised: Then, assuming some additional conditions insuring that the kernel H d is ‘spreadout’ and well behaved, the KRR solution

Original: [Equation]
Revised: [Equation]

Original: is equal up to a vanishing additive L 2 -error (as d → ∞ ) to the following effective ridge regression estimator
Revised: is equal up to a vanishing additive L 2 -error (as d → ∞ ) to the following effective ridge regression estimator

Original: [Equation]
Revised: [Equation]

Original: where λ eff = λ + (cid:80) ∞
Revised: where λ eff = λ + (cid:80) ∞

Original: j = m +1 λ d,j .
Revised: j = m +1 λ d,j .

Original: The effective estimator (41) amounts to replacing the empirical risk in Eq.
Revised: The effective estimator (41) amounts to replacing the empirical risk in Eq.

Original: (40) by its population counterpart (cid:107) f (cid:63) − f (cid:107) 2 L 2 =
Revised: (40) by its population counterpart (cid:107) f (cid:63) − f (cid:107) 2 L 2 =

Original: E x { ( f (cid:63) ( x ) − f ( x )) 2 } .
Revised: E x { ( f (cid:63) ( x ) − f ( x )) 2 } .

Original: In words, in high dimension, KRR with a ﬁnite number of samples is the same as KRR with inﬁnite number of samples but with a larger ridge regularization.
Revised: In words, in high dimension, KRR with a ﬁnite number of samples is the same as KRR with inﬁnite number of samples but with a larger ridge regularization.

Original: The solution of Eq.
Revised: The solution of Eq.

Original: (41) admits an explicit solution in terms of a shrinkage operator in the basis ( ψ d,j ) j ≥ 1 of eigenfunctions of H d :
Revised: (41) admits an explicit solution in terms of a shrinkage operator in the basis ( ψ d,j ) j ≥ 1 of eigenfunctions of H d :

Original: [Equation]
Revised: [Equation]

Original: Hence, KRR will ﬁt better the target function along eigendirections associated to larger eigenvalues of H .
Revised: Hence, KRR will ﬁt better the target function along eigendirections associated to larger eigenvalues of H .

Original: If λ d,j (cid:29) λ eff /n , KRR ﬁts perfectly f (cid:63) along the eigendirection ψ d,j , while if λ d,j (cid:28) λ eff /n ,
Revised: If λ d,j (cid:29) λ eff /n , KRR ﬁts perfectly f (cid:63) along the eigendirection ψ d,j , while if λ d,j (cid:28) λ eff /n ,

Original: KRR does not ﬁt this eigendirection at all.
Revised: KRR does not ﬁt this eigendirection at all.

Original: This phenomena has been referred as the spectral bias and task-kernel alignment of kernel ridge regression in several works.
Revised: This phenomena has been referred as the spectral bias and task-kernel alignment of kernel ridge regression in several works.

Original: Finally, notice from Eq.
Revised: Finally, notice from Eq.

Original: (42) that the minimum test error is achieved for the regularization parameter λ = 0 , which corresponds to the KRR estimator ﬁtting perfectly the training data.
Revised: (42) that the minimum test error is achieved for the regularization parameter λ = 0 , which corresponds to the KRR estimator ﬁtting perfectly the training data.

Original: In other words, the interpolating solution is optimal for kernel ridge regression in high dimension.
Revised: In other words, the interpolating solution is optimal for kernel ridge regression in high dimension.

Original: C. G ENERALIZATION ERROR OF CONVOLUTIONAL KERNELS IN HIGH DIMENSION
Revised: C. G ENERALIZATION ERROR OF CONVOLUTIONAL KERNELS IN HIGH DIMENSION

Original: Consider a sequence of integers { d ( q ) } q ≥ 1 which corresponds to a sequence of image spaces x ∈ Q d of increasing dimension, and assume d ( q ) / 2 ≥ q ≥ d ( q ) δ for some constant δ > 0 .
Revised: Consider a sequence of integers { d ( q ) } q ≥ 1 which corresponds to a sequence of image spaces x ∈ Q d of increasing dimension, and assume d ( q ) / 2 ≥ q ≥ d ( q ) δ for some constant δ > 0 .

Original: For ease of notations, we will keep the dependency on q implicit, i.e., d := d ( q ) .
Revised: For ease of notations, we will keep the dependency on q implicit, i.e., d := d ( q ) .

Original: Let { h q } q ≥ 1 be a sequence of inner-product kernels
Revised: Let { h q } q ≥ 1 be a sequence of inner-product kernels

Original: h q : R → R .
Revised: h q : R → R .

Original: Test error with one-layer convolutional kernel: we ﬁrst consider a vanilla one-layer convolutional kernel H CK as deﬁned in Eq.
Revised: Test error with one-layer convolutional kernel: we ﬁrst consider a vanilla one-layer convolutional kernel H CK as deﬁned in Eq.

Original: We will assume that the kernels { h q } q ≥ 1 verify the following ‘genericity’ condition.
Revised: We will assume that the kernels { h q } q ≥ 1 verify the following ‘genericity’ condition.

Original: Assumption 1 (Generecity assumption on { h q } q ≥ 1 at level s ∈ N ) .
Revised: Assumption 1 (Generecity assumption on { h q } q ≥ 1 at level s ∈ N ) .

Original: For { h q } q ≥ 1 a sequence of inner-product kernels h q : R → R , we assume the following conditions to hold.
Revised: For { h q } q ≥ 1 a sequence of inner-product kernels h q : R → R , we assume the following conditions to hold.

Original: There exists s (cid:48) ≥ 1 /δ
Revised: There exists s (cid:48) ≥ 1 /δ

Original: + 2 s + 3 where δ > 0 veriﬁes q ≥ d δ and a constant C such that h q (1) ≤ C , and min
Revised: + 2 s + 3 where δ > 0 veriﬁes q ≥ d δ and a constant C such that h q (1) ≤ C , and min

Original: [Equation]
Revised: [Equation]

Original: Assumption 1 will be veriﬁed by standard kernels, e.g., the Gaussian kernel.
Revised: Assumption 1 will be veriﬁed by standard kernels, e.g., the Gaussian kernel.

Original: We discuss this assumption in Section C.2 and present sufﬁcient conditions on the activation function σ for its associated
Revised: We discuss this assumption in Section C.2 and present sufﬁcient conditions on the activation function σ for its associated

Original: CNTK to verify Assumption 1.
Revised: CNTK to verify Assumption 1.

Original: Recall that we denoted L 2 ( Q d , Loc q ) the space of local functions, i.e., that can be decomposed as f ( x ) = (cid:80)
Revised: Recall that we denoted L 2 ( Q d , Loc q ) the space of local functions, i.e., that can be decomposed as f ( x ) = (cid:80)

Original: k ∈ [ d ] f k ( x ( k ) ) .
Revised: k ∈ [ d ] f k ( x ( k ) ) .

Original: Denote h q,>(cid:96)
Revised: Denote h q,>(cid:96)

Original: the inner-product kernel h q with its ( (cid:96) +1)
Revised: the inner-product kernel h q with its ( (cid:96) +1)

Original: -ﬁrst Gegenbauer coefﬁcients set to 0 , i.e.,
Revised: -ﬁrst Gegenbauer coefﬁcients set to 0 , i.e.,

Original: [Equation]
Revised: [Equation]

Original: for any u , v ∈ Q q .
Revised: for any u , v ∈ Q q .

Original: The following result is a consequence of the general theorem on the generalization error of KRR in Mei et al.
Revised: The following result is a consequence of the general theorem on the generalization error of KRR in Mei et al.

Original: (2021a).
Revised: (2021a).

Original: Theorem 7 (Test error of CK in high dimension) .
Revised: Theorem 7 (Test error of CK in high dimension) .

Original: Let { f d ∈
Revised: Let { f d ∈

Original: L 2 ( Q d , Loc q ) } q ≥ 1 be a sequence of local functions.
Revised: L 2 ( Q d , Loc q ) } q ≥ 1 be a sequence of local functions.

Original: Let ( x i ) i ∈
Revised: Let ( x i ) i ∈

Original: [ n ( d )] ∼ i.i.d.
Revised: [ n ( d )] ∼ i.i.d.

Original: Unif( Q d ) and y i =
Revised: Unif( Q d ) and y i =

Original: f d ( x i ) + ε i with ε
Revised: f d ( x i ) + ε i with ε

Original: i ∼ i.i.d.
Revised: i ∼ i.i.d.

Original: N (0 , σ 2 ε ) .
Revised: N (0 , σ 2 ε ) .

Original: Assume d · q s − 1+ δ ≤ n ≤ d · q s − δ for some δ > 0 and let { h q } q ≥ 1 be a sequence of activation functions satisfying Assumption 1 at level s .
Revised: Assume d · q s − 1+ δ ≤ n ≤ d · q s − δ for some δ > 0 and let { h q } q ≥ 1 be a sequence of activation functions satisfying Assumption 1 at level s .

Original: Consider { H CK ,d } q ≥ 1 the sequence of convolutional kernels associated to { h q } q ≥ 1 as deﬁned in Eq.
Revised: Consider { H CK ,d } q ≥ 1 the sequence of convolutional kernels associated to { h q } q ≥ 1 as deﬁned in Eq.

Original: Then the following holds for the solution ˆ f λ of
Revised: Then the following holds for the solution ˆ f λ of

Original: KRR with kernels { H CK ,d } q ≥ 1 .
Revised: KRR with kernels { H CK ,d } q ≥ 1 .

Original: For any regularization parameter λ ≥ 0 , deﬁne the effective regularization λ eff :=
Revised: For any regularization parameter λ ≥ 0 , deﬁne the effective regularization λ eff :=

Original: λ + h q,> s (1) .
Revised: λ + h q,> s (1) .

Original: Then for any η > 0 , we have
Revised: Then for any η > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 7 is deferred to Section C.4.
Revised: The proof of Theorem 7 is deferred to Section C.4.

Original: Let us expound on the predictions of Theorem 7.
Revised: Let us expound on the predictions of Theorem 7.

Original: First, recall that ˆ f eff λ eff is given explicitly in Eq.
Revised: First, recall that ˆ f eff λ eff is given explicitly in Eq.

Original: (42) by a shrinkage operator with parameter λ eff .
Revised: (42) by a shrinkage operator with parameter λ eff .

Original: From Assumption 1 and taking λ = 0 , the shrinkage operator is of order
Revised: From Assumption 1 and taking λ = 0 , the shrinkage operator is of order

Original: [Equation]
Revised: [Equation]

Original: From the eigendecomposition of H CK introduced in Proposition 1, KRR ﬁts perfectly f (cid:63) along the eigendirection Y S with | S | = (cid:96) if n · ξ d,(cid:96) r ( S ) /d
Revised: From the eigendecomposition of H CK introduced in Proposition 1, KRR ﬁts perfectly f (cid:63) along the eigendirection Y S with | S | = (cid:96) if n · ξ d,(cid:96) r ( S ) /d

Original: (cid:29) λ eff , while it does not ﬁt this eigendirection at all if n · ξ d,(cid:96)
Revised: (cid:29) λ eff , while it does not ﬁt this eigendirection at all if n · ξ d,(cid:96)

Original: r ( S ) /d ≤ λ eff .
Revised: r ( S ) /d ≤ λ eff .

Original: Consider n = d · q s − 1+ α :
Revised: Consider n = d · q s − 1+ α :

Original: • KRR ﬁts the eigendirections corresponding to the homogeneous polynomials of degree s − 1 and less, and of degree s for subsets S such that γ ( S ) (cid:28) q − q 1 − α .
Revised: • KRR ﬁts the eigendirections corresponding to the homogeneous polynomials of degree s − 1 and less, and of degree s for subsets S such that γ ( S ) (cid:28) q − q 1 − α .

Original: • KRR does not ﬁt at all the eigendirections correpsonding to homogeneous polynomials of degree s + 1 and larger, and degree s for subsets S such that γ ( S ) (cid:29) q − q 1 − α .
Revised: • KRR does not ﬁt at all the eigendirections correpsonding to homogeneous polynomials of degree s + 1 and larger, and degree s for subsets S such that γ ( S ) (cid:29) q − q 1 − α .

Original: In words, for d · q s − 1 (cid:28) n (cid:28) d · q s , KRR ﬁts at least a degree- ( s − 1) polynomial approximation to f (cid:63) and at most a degree- s polynomial approximation.
Revised: In words, for d · q s − 1 (cid:28) n (cid:28) d · q s , KRR ﬁts at least a degree- ( s − 1) polynomial approximation to f (cid:63) and at most a degree- s polynomial approximation.

Original: As n increases from d · q s − 1 to d · q s , KRR ﬁrst ﬁts degree- s homogeneous polynomials that have smaller diameter γ ( S ) (i.e., ‘more localized’).
Revised: As n increases from d · q s − 1 to d · q s , KRR ﬁrst ﬁts degree- s homogeneous polynomials that have smaller diameter γ ( S ) (i.e., ‘more localized’).

Original: Test error of CK with global average pooling: we consider the kernel H CKGP given by a convolutional layer followed by global average pooling:
Revised: Test error of CK with global average pooling: we consider the kernel H CKGP given by a convolutional layer followed by global average pooling:

Original: [Equation]
Revised: [Equation]

Original: In addition to the genericity condition, we will assume that the kernels { h q } q ≥ 1 verify the following differentiability condition.
Revised: In addition to the genericity condition, we will assume that the kernels { h q } q ≥ 1 verify the following differentiability condition.

Original: Assumption 2 (Differentiability assumption on { h q } q ≥ 1 at level s ∈ N ) .
Revised: Assumption 2 (Differentiability assumption on { h q } q ≥ 1 at level s ∈ N ) .

Original: For { h q } q ≥ 1 a sequence of inner-product kernels h q : R → R , we assume the following conditions to hold.
Revised: For { h q } q ≥ 1 a sequence of inner-product kernels h q : R → R , we assume the following conditions to hold.

Original: There exists v ≥ max(2 /δ, s ) where δ > 0 veriﬁes q ≥ d δ such that h q is ( v + 1) -differentiable
Revised: There exists v ≥ max(2 /δ, s ) where δ > 0 veriﬁes q ≥ d δ such that h q is ( v + 1) -differentiable

Original: and for k ≤ v ,
Revised: and for k ≤ v ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: where we denoted h q,>v the truncated inner-product kernel h q as in Eq.
Revised: where we denoted h q,>v the truncated inner-product kernel h q as in Eq.

Original: Assumption 2 is used to extend the following theorem to non-polynomial kernel h q (in particular, it is trivially veriﬁed for polynomial kernels by taking v larger than the degree of h q ).
Revised: Assumption 2 is used to extend the following theorem to non-polynomial kernel h q (in particular, it is trivially veriﬁed for polynomial kernels by taking v larger than the degree of h q ).

Original: This assumption is difﬁcult to check in practice, however we provide some examples where it holds in Appendix C.2.
Revised: This assumption is difﬁcult to check in practice, however we provide some examples where it holds in Appendix C.2.

Original: Recall that we denoted L 2 ( Q d , CycLoc q ) the space of functions that are given by the convolution of a function g : R q → R with the image x ∈ Q d , i.e., f ( x ) =
Revised: Recall that we denoted L 2 ( Q d , CycLoc q ) the space of functions that are given by the convolution of a function g : R q → R with the image x ∈ Q d , i.e., f ( x ) =

Original: (cid:80)
Revised: (cid:80)

Original: k ∈
Revised: k ∈

Original: [ d ] g ( x ( k ) ) .
Revised: [ d ] g ( x ( k ) ) .

Original: Theorem 8 (Test error of CK with GP in high dimension) .
Revised: Theorem 8 (Test error of CK with GP in high dimension) .

Original: Let { f d ∈
Revised: Let { f d ∈

Original: L 2 ( Q d , CycLoc q ) } q ≥be a sequence of convolutional functions.
Revised: L 2 ( Q d , CycLoc q ) } q ≥be a sequence of convolutional functions.

Original: Assume q s − 1+ δ ≤ n ≤ q s − δ for some δ > 0 and let { h q } q ≥ 1 be a sequence of activation functions satisfying Assumptions 1 and 2 at level s .
Revised: Assume q s − 1+ δ ≤ n ≤ q s − δ for some δ > 0 and let { h q } q ≥ 1 be a sequence of activation functions satisfying Assumptions 1 and 2 at level s .

Original: Consider { H CK ,d GP } q ≥ 1 the sequence of convolutional kernels with global pooling associated to { h q } q ≥ 1 as deﬁned in Eq.
Revised: Consider { H CK ,d GP } q ≥ 1 the sequence of convolutional kernels with global pooling associated to { h q } q ≥ 1 as deﬁned in Eq.

Original: Then the solution ˆ f λ of KRR with kernels { H CK ,d GP } q ≥ 1 veriﬁes Eq.
Revised: Then the solution ˆ f λ of KRR with kernels { H CK ,d GP } q ≥ 1 veriﬁes Eq.

Original: (47) with λ eff :=
Revised: (47) with λ eff :=

Original: λ + h q,> s (1) .
Revised: λ + h q,> s (1) .

Original: The proof of Theorem 8 is deferred to Section C.5.
Revised: The proof of Theorem 8 is deferred to Section C.5.

Original: The predictions of Theorem 8 are similar to the ones of Theorem 7 but with a factor d gain in statistical efﬁciency: this is due to the eigenvalues of H CKGP being a factor d larger than for H CK .
Revised: The predictions of Theorem 8 are similar to the ones of Theorem 7 but with a factor d gain in statistical efﬁciency: this is due to the eigenvalues of H CKGP being a factor d larger than for H CK .

Original: Therefore, with global average pooling, for q s − 1 (cid:28) n (cid:28) q s , KRR ﬁts at least a degree- ( s − 1) invariant polynomial approximation to f (cid:63) and at most a degree- s invariant polynomial approximation.
Revised: Therefore, with global average pooling, for q s − 1 (cid:28) n (cid:28) q s , KRR ﬁts at least a degree- ( s − 1) invariant polynomial approximation to f (cid:63) and at most a degree- s invariant polynomial approximation.

Original: As n increases from q s − 1 to q s , KRR ﬁrst degree- s invariant homogeneous polynomials with increasing diameter γ ( S ) .
Revised: As n increases from q s − 1 to q s , KRR ﬁrst degree- s invariant homogeneous polynomials with increasing diameter γ ( S ) .

Original: Test error of CK with local average pooling: In the case of local average pooling with ω < d , the eigenvalues are harder to control.
Revised: Test error of CK with local average pooling: In the case of local average pooling with ω < d , the eigenvalues are harder to control.

Original: Indeed, we have mixing of the eigenvalues between polynomials of different degree: there exists j, j (cid:48) ∈
Revised: Indeed, we have mixing of the eigenvalues between polynomials of different degree: there exists j, j (cid:48) ∈

Original: [ d ] such that ξ q,(cid:96) κ j (cid:28) ξ q,(cid:96) +1 κ j (cid:48) .
Revised: [ d ] such that ξ q,(cid:96) κ j (cid:28) ξ q,(cid:96) +1 κ j (cid:48) .

Original: The eigenvalues are not ordered in increasing degree of their associated eigenfunctions anymore.
Revised: The eigenvalues are not ordered in increasing degree of their associated eigenfunctions anymore.

Original: While this case is potentially tractable with a more careful analysis, we instead introduce a simpliﬁed kernel which we believe qualitatively captures the statistical behavior of local average pooling.
Revised: While this case is potentially tractable with a more careful analysis, we instead introduce a simpliﬁed kernel which we believe qualitatively captures the statistical behavior of local average pooling.

Original: Assume q ≤ ω/ 2 and ω is a divisor of d .
Revised: Assume q ≤ ω/ 2 and ω is a divisor of d .

Original: Denote x ( kω ) =
Revised: Denote x ( kω ) =

Original: ( x kω +1 , . . .
Revised: ( x kω +1 , . . .

Original: , x kω + ω ) the k -th segment of length ω in [ d ] and x ( kω ) ( i ) =
Revised: , x kω + ω ) the k -th segment of length ω in [ d ] and x ( kω ) ( i ) =

Original: ( x kω + i , . . .
Revised: ( x kω + i , . . .

Original: , x kω + q + i ) the patch of size q with cyclic convention in { kω +1 , . . .
Revised: , x kω + q + i ) the patch of size q with cyclic convention in { kω +1 , . . .

Original: , kω + ω } .
Revised: , kω + ω } .

Original: Consider the following convolutional kernel with ‘non-overlapping’ average pooling:
Revised: Consider the following convolutional kernel with ‘non-overlapping’ average pooling:

Original: [Equation]
Revised: [Equation]

Original: In words, H CK , NO ω is the combination of d/ω non-overlapping convolutional kernels with global average pooling on images of size ω :
Revised: In words, H CK , NO ω is the combination of d/ω non-overlapping convolutional kernels with global average pooling on images of size ω :

Original: [Equation]
Revised: [Equation]

Original: where ψ k,S ( x ) = 1 √ ω (cid:80) i ∈ [ ω ] Y i + S ( x ( kω ) ) where i + S is the translated set with cyclic convention in [ ω ] .
Revised: where ψ k,S ( x ) = 1 √ ω (cid:80) i ∈ [ ω ] Y i + S ( x ( kω ) ) where i + S is the translated set with cyclic convention in [ ω ] .

Original: Denote L 2 ( Q d , LocCycLoc q ) the RKHS associated to H CK , NO ω , which contains functions that are locally convolutions on segments of size ω .
Revised: Denote L 2 ( Q d , LocCycLoc q ) the RKHS associated to H CK , NO ω , which contains functions that are locally convolutions on segments of size ω .

Original: For this simpliﬁed model, the proof of Theorem 8 can be easily adapted and we obtain the following result:
Revised: For this simpliﬁed model, the proof of Theorem 8 can be easily adapted and we obtain the following result:

Original: Corollary 1 (Test error of CK with NO pooling in high dimension) .
Revised: Corollary 1 (Test error of CK with NO pooling in high dimension) .

Original: Let { f d ∈
Revised: Let { f d ∈

Original: L 2 ( Q d , LocCycLoc q ) } q ≥ 1 be a sequence of local convolutional functions.
Revised: L 2 ( Q d , LocCycLoc q ) } q ≥ 1 be a sequence of local convolutional functions.

Original: Assume ( d/ω ) · q s − 1+ δ ≤ n ≤ ( d/ω ) · q s − δ for some δ > 0 and let { h q } q ≥ 1 be a sequence of activation functions satisfying Assumptions 1 and 2 at level s .
Revised: Assume ( d/ω ) · q s − 1+ δ ≤ n ≤ ( d/ω ) · q s − δ for some δ > 0 and let { h q } q ≥ 1 be a sequence of activation functions satisfying Assumptions 1 and 2 at level s .

Original: Consider { H CK , NO ,d ω } q ≥ 1 the sequence of convolutional kernels with non-overlapping pooling associated to { h q } q ≥ 1 as deﬁned in Eq.
Revised: Consider { H CK , NO ,d ω } q ≥ 1 the sequence of convolutional kernels with non-overlapping pooling associated to { h q } q ≥ 1 as deﬁned in Eq.

Original: Then the solution ˆ f λ of KRR with kernels { H CK , NO ,d ω } q ≥ 1 veriﬁes Eq.
Revised: Then the solution ˆ f λ of KRR with kernels { H CK , NO ,d ω } q ≥ 1 veriﬁes Eq.

Original: (47) with λ eff :=
Revised: (47) with λ eff :=

Original: λ
Revised: λ

Original: + dω h q,> s (1) .
Revised: + dω h q,> s (1) .

Original: Corollary 1 shows that H CK , NO ω enjoys a factor ω gain in statistical efﬁciency compared to H CK , due to a factor ω smaller effective ridge regularization.
Revised: Corollary 1 shows that H CK , NO ω enjoys a factor ω gain in statistical efﬁciency compared to H CK , due to a factor ω smaller effective ridge regularization.

Original: Therefore, with (non-overlapping) local average pooling, for ( d/ω ) · q s − 1 (cid:28) n (cid:28) ( d/ω ) · q s , KRR ﬁts degree- ( s − 1) locally invariant polynomials and none of the polynomials of degree- ( s + 1) and larger.
Revised: Therefore, with (non-overlapping) local average pooling, for ( d/ω ) · q s − 1 (cid:28) n (cid:28) ( d/ω ) · q s , KRR ﬁts degree- ( s − 1) locally invariant polynomials and none of the polynomials of degree- ( s + 1) and larger.

Original: Heuristically, we see that this yields the same statistical efﬁciency than H CK for ω = 1 and H CKGP for ω
Revised: Heuristically, we see that this yields the same statistical efﬁciency than H CK for ω = 1 and H CKGP for ω

Original: = d , and interpolates between the two cases for 1 < ω < d .
Revised: = d , and interpolates between the two cases for 1 < ω < d .

Original: Test error of convolutional kernels with downsampling: We consider adding a downsampling operation to the previous kernels.
Revised: Test error of convolutional kernels with downsampling: We consider adding a downsampling operation to the previous kernels.

Original: Let ∆ be a constant and a divisor of d and ω and consider the following ‘downsampled’ kernels:
Revised: Let ∆ be a constant and a divisor of d and ω and consider the following ‘downsampled’ kernels:

Original: [Equation]
Revised: [Equation]

Original: We can easily adapt the proofs of Theorems 7 and 8, and Corollary 1 to these kernels.
Revised: We can easily adapt the proofs of Theorems 7 and 8, and Corollary 1 to these kernels.

Original: In particular, their conclusions do not change (for any constant ∆ ) and downsampling do not provide a statistical advantage.
Revised: In particular, their conclusions do not change (for any constant ∆ ) and downsampling do not provide a statistical advantage.

Original: C. C HECKING THE ASSUMPTIONS
Revised: C. C HECKING THE ASSUMPTIONS

Original: In this section, we discuss Assumptions 1 and 2 and present sufﬁcient conditions for them to be veriﬁed.
Revised: In this section, we discuss Assumptions 1 and 2 and present sufﬁcient conditions for them to be veriﬁed.

Original: Genericity assumption: Recall that the inner-product kernel h q : R → R has the following eigendecomposition on Q q as
Revised: Genericity assumption: Recall that the inner-product kernel h q : R → R has the following eigendecomposition on Q q as

Original: [Equation]
Revised: [Equation]

Original: The genericity assumption amounts to: 1)
Revised: The genericity assumption amounts to: 1)

Original: A universality condition in Eqs.
Revised: A universality condition in Eqs.

Original: (43) and (44): if
Revised: (43) and (44): if

Original: P k h ( (cid:104) 1 , ·(cid:105) /q )
Revised: P k h ( (cid:104) 1 , ·(cid:105) /q )

Original: = 0 , then h does not learn degree- k homogeneous polynomials; 2) A constant order scaling of the self-induced regularization h q,> s (1) , from h q (1) ≤ C and Eq.
Revised: = 0 , then h does not learn degree- k homogeneous polynomials; 2) A constant order scaling of the self-induced regularization h q,> s (1) , from h q (1) ≤ C and Eq.

Original: (44) with s (cid:48) , i.e., h q,> s (1) ≤ h q (1) =
Revised: (44) with s (cid:48) , i.e., h q,> s (1) ≤ h q (1) =

Original: O q (1) and h q,> s (1) ≥ ξ q, s (cid:48) B ( q, s (cid:48) )
Revised: O q (1) and h q,> s (1) ≥ ξ q, s (cid:48) B ( q, s (cid:48) )

Original: =
Revised: =

Original: Ω q (1) ; 3)
Revised: Ω q (1) ; 3)

Original: The last eigenvalues decay sufﬁciently fast in Eq.
Revised: The last eigenvalues decay sufﬁciently fast in Eq.

Original: (45) in order to avoid pathological cases.
Revised: (45) in order to avoid pathological cases.

Original: For generic kernels, we have typically ξ q,(cid:96) (cid:16) q − (cid:96) (for ﬁx (cid:96) ).
Revised: For generic kernels, we have typically ξ q,(cid:96) (cid:16) q − (cid:96) (for ﬁx (cid:96) ).

Original: For example, if h is smooth, ξ q,(cid:96) =
Revised: For example, if h is smooth, ξ q,(cid:96) =

Original: q − (cid:96) ( h ( k ) (0)+ o q (1)) and it is sufﬁcient to have h ( k ) (0) > 0 .
Revised: q − (cid:96) ( h ( k ) (0)+ o q (1)) and it is sufﬁcient to have h ( k ) (0) > 0 .

Original: See Appendix D.2 in Mei et al.
Revised: See Appendix D.2 in Mei et al.

Original: (2021a)
Revised: (2021a)

Original: for a proof of Eq.
Revised: for a proof of Eq.

Original: (45) when h is sufﬁciently smooth.
Revised: (45) when h is sufﬁciently smooth.

Original: Below, we present instead sufﬁcient conditions on the activation σ such that the induced neural tangent kernel veriﬁes the ‘genericity’ assumption.
Revised: Below, we present instead sufﬁcient conditions on the activation σ such that the induced neural tangent kernel veriﬁes the ‘genericity’ assumption.

Original: More precisely, we display sufﬁcient conditions on the sequence { σ q } q ≥ 1 of activation functions σ q : R → R , such that the induced neural tangent kernels { h q } q ≥ 1 veriﬁes Assumption 1, where h q was derived in Section A.2 and is given by ( u , v ∈ Q q )
Revised: More precisely, we display sufﬁcient conditions on the sequence { σ q } q ≥ 1 of activation functions σ q : R → R , such that the induced neural tangent kernels { h q } q ≥ 1 veriﬁes Assumption 1, where h q was derived in Section A.2 and is given by ( u , v ∈ Q q )

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Assumption 3 (Assumptions on { σ q } q ≥ 1 at level s ∈ N ) .
Revised: Assumption 3 (Assumptions on { σ q } q ≥ 1 at level s ∈ N ) .

Original: For { σ q } q ≥ 1 a sequence of functions σ q : R → R , we assume the following conditions to hold.
Revised: For { σ q } q ≥ 1 a sequence of functions σ q : R → R , we assume the following conditions to hold.

Original: There exists s (cid:48) ≥ 1 /δ
Revised: There exists s (cid:48) ≥ 1 /δ

Original: + 2 s + 3 where δ > 0 veriﬁes q ≥ d δ , such that
Revised: + 2 s + 3 where δ > 0 veriﬁes q ≥ d δ , such that

Original: (a) The function σ q is differentiable and there exists c 0 > 0 and c 1 < 1 independent of q , such that | σ q ( x ) | , | σ (cid:48) q ( x ) | ≤ c 0 exp( c 1 x 2 / 2) .
Revised: (a) The function σ q is differentiable and there exists c 0 > 0 and c 1 < 1 independent of q , such that | σ q ( x ) | , | σ (cid:48) q ( x ) | ≤ c 0 exp( c 1 x 2 / 2) .

Original: (b) We have
Revised: (b) We have

Original: [Equation]
Revised: [Equation]

Original: where e ∈ Q q is arbitrary.
Revised: where e ∈ Q q is arbitrary.

Original: (c) We have for a ﬁxed δ >
Revised: (c) We have for a ﬁxed δ >

Original: [Equation]
Revised: [Equation]

Original: Proposition 6.
Revised: Proposition 6.

Original: Consider a sequence { σ q } q ≥ 1 of activation functions σ q : R → R that satisﬁes
Revised: Consider a sequence { σ q } q ≥ 1 of activation functions σ q : R → R that satisﬁes

Original: Assumption 3.
Revised: Assumption 3.

Original: Let { h q } q ≥ 1 be the sequence of neural tangent kernels associated to { σ q } q ≥ 1 as deﬁned in Eq.
Revised: Let { h q } q ≥ 1 be the sequence of neural tangent kernels associated to { σ q } q ≥ 1 as deﬁned in Eq.

Original: Then the sequence { h q } q ≥ 1 satisﬁes the ‘genericity’ Assumption 1.
Revised: Then the sequence { h q } q ≥ 1 satisﬁes the ‘genericity’ Assumption 1.

Original: Differentiability assumption: As mentioned in the previous section, this condition is required in our proof technique to extend Theorem 8 to non-polynomial kernel functions.
Revised: Differentiability assumption: As mentioned in the previous section, this condition is required in our proof technique to extend Theorem 8 to non-polynomial kernel functions.

Original: While we believe that weaker conditions should be sufﬁcient, we leave checking them to future work.
Revised: While we believe that weaker conditions should be sufﬁcient, we leave checking them to future work.

Original: Note that Assumption 2 was proved for x ∼ Unif( S
Revised: Note that Assumption 2 was proved for x ∼ Unif( S

Original: C. P ROOF OF P ROPOSITION
Revised: C. P ROOF OF P ROPOSITION

Original: Proof of Proposition 6.
Revised: Proof of Proposition 6.

Original: Step 1.
Revised: Step 1.

Original: Effective activation function.
Revised: Effective activation function.

Original: Let us decompose both functions σ q and σ (cid:48)
Revised: Let us decompose both functions σ q and σ (cid:48)

Original: q in the Gegenbauer polynomial on the hypercube basis:
Revised: q in the Gegenbauer polynomial on the hypercube basis:

Original: [Equation]
Revised: [Equation]

Original: where we recall B ( Q q ; (cid:96) )
Revised: where we recall B ( Q q ; (cid:96) )

Original: = (cid:0) d(cid:96) (cid:1) and (for e ∈ Q q arbitrary)
Revised: = (cid:0) d(cid:96) (cid:1) and (for e ∈ Q q arbitrary)

Original: [Equation]
Revised: [Equation]

Original: From the deﬁnition of h (1) q in Eq.
Revised: From the deﬁnition of h (1) q in Eq.

Original: (55) and the eigendecomposition (61), we have
Revised: (55) and the eigendecomposition (61), we have

Original: [Equation]
Revised: [Equation]

Original: Similarly, from the deﬁnition of h (2) q in Eq.
Revised: Similarly, from the deﬁnition of h (2) q in Eq.

Original: (56), the eigendecomposition (62) and using Lemmastated below, we get
Revised: (56), the eigendecomposition (62) and using Lemmastated below, we get

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We can therefore deﬁne π
Revised: We can therefore deﬁne π

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: : Q q × Q q → R by
Revised: : Q q × Q q → R by

Original: [Equation]
Revised: [Equation]

Original: such that the NT kernel (54) can be written as the kernel of the effective activation σ eﬀ ,q :
Revised: such that the NT kernel (54) can be written as the kernel of the effective activation σ eﬀ ,q :

Original: [Equation]
Revised: [Equation]

Original: We will show that h q with Gegenbauer coefﬁcients ξ q,(cid:96) :
Revised: We will show that h q with Gegenbauer coefﬁcients ξ q,(cid:96) :

Original: = π 2 q,(cid:96) veriﬁes Assumption 1.
Revised: = π 2 q,(cid:96) veriﬁes Assumption 1.

Original: Step 2.
Revised: Step 2.

Original: Decay of the eigenvalues.
Revised: Decay of the eigenvalues.

Original: Recall that the sequence { σ q } q ≥ 1 satisﬁes Assumption 3 at level s .
Revised: Recall that the sequence { σ q } q ≥ 1 satisﬁes Assumption 3 at level s .

Original: From Assumption 3.
Revised: From Assumption 3.

Original: ( a ) (for example by adapting the proof of Lemma C.1 in Ghorbani et al. (2021) to the hypercube), there exists C > 0 such that
Revised: ( a ) (for example by adapting the proof of Lemma C.1 in Ghorbani et al. (2021) to the hypercube), there exists C > 0 such that

Original: [Equation]
Revised: [Equation]

Original: and we deduce that χ 2 q,(cid:96) , κ 2 q,(cid:96) , π 2 q,(cid:96) =
Revised: and we deduce that χ 2 q,(cid:96) , κ 2 q,(cid:96) , π 2 q,(cid:96) =

Original: O q ( B ( Q q ; (cid:96) ) − 1 ) .
Revised: O q ( B ( Q q ; (cid:96) ) − 1 ) .

Original: Using that B ( Q q ; (cid:96) )
Revised: Using that B ( Q q ; (cid:96) )

Original: = (cid:0) q(cid:96) (cid:1) , we deduce that for any ﬁxed (cid:96) , χ 2 q,(cid:96) , κ 2 q,(cid:96) , π 2 q,(cid:96) =
Revised: = (cid:0) q(cid:96) (cid:1) , we deduce that for any ﬁxed (cid:96) , χ 2 q,(cid:96) , κ 2 q,(cid:96) , π 2 q,(cid:96) =

Original: O q ( q − (cid:96) ) .
Revised: O q ( q − (cid:96) ) .

Original: Furthermore, from Assumption
Revised: Furthermore, from Assumption

Original: [Equation]
Revised: [Equation]

Original: By Eq.
Revised: By Eq.

Original: (63) and the deﬁnition of π 2 q,(cid:96) , we have π 2 q,q − k
Revised: (63) and the deﬁnition of π 2 q,(cid:96) , we have π 2 q,q − k

Original: = O d ( q − s (cid:48) − 1 ) for any k ≤ s (cid:48) , which veriﬁes
Revised: = O d ( q − s (cid:48) − 1 ) for any k ≤ s (cid:48) , which veriﬁes

Original: Eq.
Revised: Eq.

Original: (45) in Assumption 1.
Revised: (45) in Assumption 1.

Original: Furthermore, by Assumption 3.
Revised: Furthermore, by Assumption 3.

Original: ( b ) , using that χ 2 q,k = B ( Q q ; k ) − 1 (cid:107) P k σ q (cid:107) 2 L 2 ( Q q ) and ξ 2 q,k ≥ χ 2 q,k , we get
Revised: ( b ) , using that χ 2 q,k = B ( Q q ; k ) − 1 (cid:107) P k σ q (cid:107) 2 L 2 ( Q q ) and ξ 2 q,k ≥ χ 2 q,k , we get

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: In particular, this implies that (cid:107) σ eﬀ ,d,> s (cid:107) 2 L
Revised: In particular, this implies that (cid:107) σ eﬀ ,d,> s (cid:107) 2 L

Original: [Equation]
Revised: [Equation]

Original: ≥
Revised: ≥

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: 2 L 2 ( Q q )
Revised: 2 L 2 ( Q q )

Original: =
Revised: =

Original: Ω q (1) .
Revised: Ω q (1) .

Original: Lemma 1.
Revised: Lemma 1.

Original: Let (cid:96) be an integer such that 0 ≤ (cid:96) ≤ q .
Revised: Let (cid:96) be an integer such that 0 ≤ (cid:96) ≤ q .

Original: Consider the following Gegenbauer polynomial deﬁned on the q -dimensional hypercube (see Section D): for x , y ∈ Q q ,
Revised: Consider the following Gegenbauer polynomial deﬁned on the q -dimensional hypercube (see Section D): for x , y ∈ Q q ,

Original: Q ( q
Revised: Q ( q

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where we recall the deﬁnition of the homogeneous polynomial Y S ( x ) = x S =
Revised: where we recall the deﬁnition of the homogeneous polynomial Y S ( x ) = x S =

Original: (cid:81)
Revised: (cid:81)

Original: i ∈
Revised: i ∈

Original: S x i .
Revised: S x i .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: with the convention Q ( q ) − 1 = Q ( q ) q +1 = 0 .
Revised: with the convention Q ( q ) − 1 = Q ( q ) q +1 = 0 .

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: Consider 1 ≤ (cid:96) ≤ q − 1 .
Revised: Consider 1 ≤ (cid:96) ≤ q − 1 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: We have Y S ( x ) x i =
Revised: We have Y S ( x ) x i =

Original: Y S ∪{ i } ( x ) if i (cid:54)∈ S , and Y S ( x ) x i = Y S \{ i } ( x ) if i ∈ S .
Revised: Y S ∪{ i } ( x ) if i (cid:54)∈ S , and Y S ( x ) x i = Y S \{ i } ( x ) if i ∈ S .