Original: A PPROXIMATION AND L EARNING WITH D EEP C ONVOLUTIONAL M ODELS : A K ERNEL P ERSPECTIVE
Revised: A PPROXIMATION AND L EARNING WITH D EEP C ONVOLUTIONAL M ODELS : A K ERNEL P ERSPECTIVE

Original: A BSTRACT
Revised: A BSTRACT

Original: The empirical success of deep convolutional networks on tasks involving highdimensional data such as images or audio suggests that they can efﬁciently approximate certain functions that are well-suited for such tasks.
Revised: The empirical success of deep convolutional networks on tasks involving highdimensional data such as images or audio suggests that they can efﬁciently approximate certain functions that are well-suited for such tasks.

Original: In this paper, we study this through the lens of kernel methods, by considering simple hierarchical kernels with two or three convolution and pooling layers, inspired by convolutional kernel networks.
Revised: In this paper, we study this through the lens of kernel methods, by considering simple hierarchical kernels with two or three convolution and pooling layers, inspired by convolutional kernel networks.

Original: These achieve good empirical performance on standard vision datasets, while providing a precise description of their functional space that yields new insights on their inductive bias.
Revised: These achieve good empirical performance on standard vision datasets, while providing a precise description of their functional space that yields new insights on their inductive bias.

Original: We show that the RKHS consists of additive models of interaction terms between patches, and that its norm encourages spatial similarities between these terms through pooling layers.
Revised: We show that the RKHS consists of additive models of interaction terms between patches, and that its norm encourages spatial similarities between these terms through pooling layers.

Original: We then provide generalization bounds which illustrate how pooling and patches yield improved sample complexity guarantees when the target function presents such regularities
Revised: We then provide generalization bounds which illustrate how pooling and patches yield improved sample complexity guarantees when the target function presents such regularities

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep convolutional models have been at the heart of the recent successes of deep learning in problems where the data consists of high-dimensional signals, such as image classiﬁcation or speech recognition.
Revised: Deep convolutional models have been at the heart of the recent successes of deep learning in problems where the data consists of high-dimensional signals, such as image classiﬁcation or speech recognition.

Original: Convolution and pooling operations have notably contributed to the practical success of these models, yet our theoretical understanding of how they enable efﬁcient learning is still limited.
Revised: Convolution and pooling operations have notably contributed to the practical success of these models, yet our theoretical understanding of how they enable efﬁcient learning is still limited.

Original: One key difﬁculty for understanding such models is the curse of dimensionality: due to the highdimensionality of the input data, it is hopeless to learn arbitrary functions from samples.
Revised: One key difﬁculty for understanding such models is the curse of dimensionality: due to the highdimensionality of the input data, it is hopeless to learn arbitrary functions from samples.

Original: For instance, classical non-parametric regression techniques for learning generic target functions typically require either low dimension or very high degrees of smoothness in order to obtain good generalization ( e.g. ,
Revised: For instance, classical non-parametric regression techniques for learning generic target functions typically require either low dimension or very high degrees of smoothness in order to obtain good generalization ( e.g. ,

Original: Wainwright, 2019), which makes them impractical for dealing with high-dimensional signals.
Revised: Wainwright, 2019), which makes them impractical for dealing with high-dimensional signals.

Original: Thus, further assumptions on the target function are needed to make the problem tractable, and we seek assumptions that make convolutions a useful modeling tool.
Revised: Thus, further assumptions on the target function are needed to make the problem tractable, and we seek assumptions that make convolutions a useful modeling tool.

Original: Various works have studied approximation beneﬁts of depth with models that resemble deep convolutional architectures (Cohen & Shashua, 2017; Mhaskar & Poggio, 2016; Schmidt-Hieber et al., 2020).
Revised: Various works have studied approximation beneﬁts of depth with models that resemble deep convolutional architectures (Cohen & Shashua, 2017; Mhaskar & Poggio, 2016; Schmidt-Hieber et al., 2020).

Original: Nevertheless, while such function classes may provide improved statistical efﬁciency in theory, it is unclear if there exist efﬁcient algorithms to learn such models, and hence, whether they might correspond to what convolutional networks learn in practice.
Revised: Nevertheless, while such function classes may provide improved statistical efﬁciency in theory, it is unclear if there exist efﬁcient algorithms to learn such models, and hence, whether they might correspond to what convolutional networks learn in practice.

Original: To overcome this issue, we consider instead function classes based on kernel methods (Schölkopf & Smola, 2001;
Revised: To overcome this issue, we consider instead function classes based on kernel methods (Schölkopf & Smola, 2001;

Original: Wahba, 1990), which are known to be learnable with efﬁcient (polynomial-time) algorithms, such as kernel ridge regression or gradient descent.
Revised: Wahba, 1990), which are known to be learnable with efﬁcient (polynomial-time) algorithms, such as kernel ridge regression or gradient descent.

Original: We consider “deep” structured kernels known as convolutional kernels, which yield good empirical performance on standard computer vision benchmarks (Li et al., 2019; Mairal, 2016; Mairal et al., 2014; Shankar et al., 2020), and are related to over-parameterized convolutional networks (CNNs) in so-called “kernel regimes” (Arora et al., 2019; Bietti & Mairal, 2019b; Daniely et al., 2016; GarrigaAlonso et al., 2019;
Revised: We consider “deep” structured kernels known as convolutional kernels, which yield good empirical performance on standard computer vision benchmarks (Li et al., 2019; Mairal, 2016; Mairal et al., 2014; Shankar et al., 2020), and are related to over-parameterized convolutional networks (CNNs) in so-called “kernel regimes” (Arora et al., 2019; Bietti & Mairal, 2019b; Daniely et al., 2016; GarrigaAlonso et al., 2019;

Original: Jacot et al., 2018; Novak et al., 2019; Yang, 2019).
Revised: Jacot et al., 2018; Novak et al., 2019; Yang, 2019).

Original: Such regimes may be seen as providing a ﬁrst-order description of what common deep models trained with gradient methods may learn.
Revised: Such regimes may be seen as providing a ﬁrst-order description of what common deep models trained with gradient methods may learn.

Original: Studying the corresponding function spaces (reproducing kernel Hilbert spaces, or RKHS) may then provide insight into the beneﬁts of various architectural choices.
Revised: Studying the corresponding function spaces (reproducing kernel Hilbert spaces, or RKHS) may then provide insight into the beneﬁts of various architectural choices.

Original: For fully-connected architectures, such kernels are rotation-invariant, and the corresponding RKHSs are well understood in terms of regularity properties on the sphere (Bach, 2017a; Smola et al., 2001), but do not show any major differences between deep and shallow kernels (Bietti & Bach, 2021;
Revised: For fully-connected architectures, such kernels are rotation-invariant, and the corresponding RKHSs are well understood in terms of regularity properties on the sphere (Bach, 2017a; Smola et al., 2001), but do not show any major differences between deep and shallow kernels (Bietti & Bach, 2021;

Original: Chen & Xu, 2021;
Revised: Chen & Xu, 2021;

Original: Geifman et al., 2020).
Revised: Geifman et al., 2020).

Original: In contrast, in this work we show that even in the kernel setting, multiple layers of convolution and pooling operations can be crucial for efﬁcient learning of functions with speciﬁc structures that are well-suited for natural signals.
Revised: In contrast, in this work we show that even in the kernel setting, multiple layers of convolution and pooling operations can be crucial for efﬁcient learning of functions with speciﬁc structures that are well-suited for natural signals.

Original: Our work paves the way for further studies of the inductive bias of optimization algorithms on deep convolutional networks beyond kernel regimes, for instance by incorporating adaptivity to low-dimensional structure (Bach, 2017a; Chizat & Bach, 2020; Wei et al., 2019) or hierarchical learning (Allen-Zhu & Li, 2020).
Revised: Our work paves the way for further studies of the inductive bias of optimization algorithms on deep convolutional networks beyond kernel regimes, for instance by incorporating adaptivity to low-dimensional structure (Bach, 2017a; Chizat & Bach, 2020; Wei et al., 2019) or hierarchical learning (Allen-Zhu & Li, 2020).

Original: We make the following contributions:
Revised: We make the following contributions:

Original: • We revisit convolutional kernel networks (Mairal, 2016), ﬁnding that simple two or three layers models with Gaussian pooling and polynomial kernels of degree 2-4 at higher layers provide competitive performance with state-of-the-art convolutional kernels such as Myrtle kernels (Shankar et al., 2020) on Cifar10.
Revised: • We revisit convolutional kernel networks (Mairal, 2016), ﬁnding that simple two or three layers models with Gaussian pooling and polynomial kernels of degree 2-4 at higher layers provide competitive performance with state-of-the-art convolutional kernels such as Myrtle kernels (Shankar et al., 2020) on Cifar10.

Original: • For such kernels, we provide an exact description of the RKHS functions and their norm, illustrating representation beneﬁts of multiple convolutional and pooling layers for capturing additive and interaction models on patches with certain spatial regularities among interaction terms.
Revised: • For such kernels, we provide an exact description of the RKHS functions and their norm, illustrating representation beneﬁts of multiple convolutional and pooling layers for capturing additive and interaction models on patches with certain spatial regularities among interaction terms.

Original: • We provide generalization bounds that illustrate the beneﬁts of architectural choices such as pooling and patches for learning additive interaction models with spatial invariance in the interaction terms, namely, improvements in sample complexity by polynomial factors in the size of the input signal.
Revised: • We provide generalization bounds that illustrate the beneﬁts of architectural choices such as pooling and patches for learning additive interaction models with spatial invariance in the interaction terms, namely, improvements in sample complexity by polynomial factors in the size of the input signal.

Original: Related work.
Revised: Related work.

Original: Convolutional kernel networks were introduced by Mairal et al.
Revised: Convolutional kernel networks were introduced by Mairal et al.

Original: (2014); Mairal (2016).
Revised: (2014); Mairal (2016).

Original: Empirically, they used kernel approximations to improve computational efﬁciency, while we evaluate the exact kernels in order to assess their best performance, as in (Arora et al., 2019; Li et al., 2019; Shankar et al., 2020).
Revised: Empirically, they used kernel approximations to improve computational efﬁciency, while we evaluate the exact kernels in order to assess their best performance, as in (Arora et al., 2019; Li et al., 2019; Shankar et al., 2020).

Original: Bietti & Mairal (2019a;b) show invariance and stability properties of its RKHS functions, and provide upper bounds on the RKHS norm for some speciﬁc functions (see also Zhang et al., 2017); in contrast, we provide exact characterizations of the RKHS norm, and study generalization beneﬁts of certain architectures.
Revised: Bietti & Mairal (2019a;b) show invariance and stability properties of its RKHS functions, and provide upper bounds on the RKHS norm for some speciﬁc functions (see also Zhang et al., 2017); in contrast, we provide exact characterizations of the RKHS norm, and study generalization beneﬁts of certain architectures.

Original: Scetbon & Harchaoui (2020) study statistical properties of simple convolutional kernels without pooling, while we focus on the role of architecture choices with an emphasis on pooling.
Revised: Scetbon & Harchaoui (2020) study statistical properties of simple convolutional kernels without pooling, while we focus on the role of architecture choices with an emphasis on pooling.

Original: Cohen & Shashua (2016; 2017); Mhaskar & Poggio (2016);
Revised: Cohen & Shashua (2016; 2017); Mhaskar & Poggio (2016);

Original: Poggio et al.
Revised: Poggio et al.

Original: (2017) study expressivity and approximation with models that resemble CNNs, showing beneﬁts thanks to hierarchy or local interactions, but such models are not known to be learnable with tractable algorithms, while we focus on (tractable) kernels.
Revised: (2017) study expressivity and approximation with models that resemble CNNs, showing beneﬁts thanks to hierarchy or local interactions, but such models are not known to be learnable with tractable algorithms, while we focus on (tractable) kernels.

Original: Regularization properties of convolutional models were also considered in (Gunasekar et al., 2018; Heckel & Soltanolkotabi, 2020), but in different regimes or architectures than ours.
Revised: Regularization properties of convolutional models were also considered in (Gunasekar et al., 2018; Heckel & Soltanolkotabi, 2020), but in different regimes or architectures than ours.

Original: Li et al.
Revised: Li et al.

Original: Malach & Shalev-Shwartz (2021) study beneﬁts of convolutional networks with efﬁcient algorithms, but do not study the gains of pooling.
Revised: Malach & Shalev-Shwartz (2021) study beneﬁts of convolutional networks with efﬁcient algorithms, but do not study the gains of pooling.

Original: Du et al.
Revised: Du et al.

Original: (2018) study sample complexity of learning CNNs, focusing on parametric rather than non-parametric models.
Revised: (2018) study sample complexity of learning CNNs, focusing on parametric rather than nonparametric models.

Original: Mei et al.
Revised: Mei et al.

Original: (2021) study statistical beneﬁts of global pooling for learning invariant functions, but only consider one layer with full-size patches.
Revised: (2021) study statistical beneﬁts of global pooling for learning invariant functions, but only consider one layer with full-size patches.

Original: Concurrently to our work, Favero et al.
Revised: Concurrently to our work, Favero et al.

Original:  (2021) study beneﬁts of local patches, but focus on one-layer models.
Revised: ; Misiakiewicz & Mei (2021) study beneﬁts of local patches, but focus on one-layer models.

Original:  D EEP C ONVOLUTIONAL K ERNELS
Revised:  D EEP C ONVOLUTIONAL K ERNELS

Original: In this section, we recall the construction of multi-layer convolutional kernels on discrete signals, following most closely the convolutional kernel network (CKN) architectures studied by Mairal (2016);
Revised: In this section, we recall the construction of multi-layer convolutional kernels on discrete signals, following most closely the convolutional kernel network (CKN) architectures studied by Mairal (2016);

Original: Bietti & Mairal (2019a).
Revised: Bietti & Mairal (2019a).

Original: These architectures rely crucially on pooling layers, typically with
Revised: These architectures rely crucially on pooling layers, typically with

Original: Gaussian ﬁlters, which make them empirically effective even with just two convolutional layers.
Revised: Gaussian ﬁlters, which make them empirically effective even with just two convolutional layers.

Original: These kernels deﬁne function spaces that will be the main focus of our theoretical study of approximation and generalization in the next sections.
Revised: These kernels deﬁne function spaces that will be the main focus of our theoretical study of approximation and generalization in the next sections.

Original: In particular, when learning a target function of the form f ∗ ( x ) = (cid:80) i f i ( x ) , we will show that they are able to efﬁciently exploit two useful properties of f ∗ : (locality) each f i may depend on only one or a few small localized patches of the signal; (invariance) many different terms f i may involve the same function applied to different input patches.
Revised: In particular, when learning a target function of the form f ∗ ( x ) = (cid:80) i f i ( x ) , we will show that they are able to efﬁciently exploit two useful properties of f ∗ : (locality) each f i may depend on only one or a few small localized patches of the signal; (invariance) many different terms f i may involve the same function applied to different input patches.

Original: We provide further background and motivation in Appendix A.
Revised: We provide further background and motivation in Appendix A.

Original: For simplicity, we will focus on discrete 1D input signals, though one may easily extend our results to 2D or higher-dimensional signals.
Revised: For simplicity, we will focus on discrete 1D input signals, though one may easily extend our results to 2D or higher-dimensional signals.

Original: We will assume periodic signals in order to avoid difﬁculties with border effects, or alternatively, a cyclic domain Ω = Z / | Ω | Z .
Revised: We will assume periodic signals in order to avoid difﬁculties with border effects, or alternatively, a cyclic domain Ω = Z / | Ω | Z .

Original: A convolutional kernel of depth L may then be deﬁned for input signals x, x
Revised: A convolutional kernel of depth L may then be deﬁned for input signals x, x

Original: (cid:48) ∈
Revised: (cid:48) ∈

Original: L 2 (Ω , R p ) by K L ( x, x (cid:48) )
Revised: L 2 (Ω , R p ) by K L ( x, x (cid:48) )

Original: = (cid:104) Ψ
Revised: = (cid:104) Ψ

Original: ( x ) , Ψ( x (cid:48) )
Revised: ( x ) , Ψ( x (cid:48) )

Original: (cid:105) , through the explicit feature map
Revised: (cid:105) , through the explicit feature map

Original: [Equation]
Revised: [Equation]

Original: Here, P (cid:96) , M (cid:96) and A (cid:96) are linear or non-linear operators corresponding to patch extraction , kernel mapping and pooling , respectively, and are described below.
Revised: Here, P (cid:96) , M (cid:96) and A (cid:96) are linear or non-linear operators corresponding to patch extraction , kernel mapping and pooling , respectively, and are described below.

Original: They operate on feature maps in L 2 (Ω (cid:96) )
Revised: They operate on feature maps in L 2 (Ω (cid:96) )

Original: (with Ω 0 =Ω ) with values in different Hilbert spaces, starting from H 0 :
Revised: (with Ω 0 =Ω ) with values in different Hilbert spaces, starting from H 0 :

Original: = R p , and are deﬁned below.
Revised: = R p , and are deﬁned below.

Original: An illustration of this construction is given in Figure 1.
Revised: An illustration of this construction is given in Figure 1.

Original: [Figure]
Revised: [Figure]

Original: Patch extraction.
Revised: Patch extraction.

Original: Given a patch shape S (cid:96) ⊂ Ω (cid:96) − 1 , such as S (cid:96) =
Revised: Given a patch shape S (cid:96) ⊂ Ω (cid:96) − 1 , such as S (cid:96) =

Original: [ − 1 , 0 , 1] for one-dimensional patches of size 3 , the operator P (cid:96) is deﬁned for x ∈ L 2 (Ω (cid:96)
Revised: [ − 1 , 0 , 1] for one-dimensional patches of size 3 , the operator P (cid:96) is deﬁned for x ∈ L 2 (Ω (cid:96)

Original: [Equation]
Revised: [Equation]

Original: 1 , H (cid:96)
Revised: 1 , H (cid:96)

Original: [Equation]
Revised: [Equation]

Original: 1 ) by
Revised: 1 ) by

Original: [Equation]
Revised: [Equation]

Original: Kernel mapping.
Revised: Kernel mapping.

Original: The operators M (cid:96) perform a nonlinear embedding of patches into a new Hilbert space using dot-product kernels.
Revised: The operators M (cid:96) perform a nonlinear embedding of patches into a new Hilbert space using dot-product kernels.

Original: We consider homogeneous dot-product kernels given for z, z (cid:48) ∈ H | S (cid:96) | (cid:96) − 1 by k
Revised: We consider homogeneous dot-product kernels given for z, z (cid:48) ∈ H | S (cid:96) | (cid:96) − 1 by k

Original: [Equation]
Revised: [Equation]

Original: ( z, z (cid:48) )
Revised: ( z, z (cid:48) )

Original: = (cid:107) z (cid:107)(cid:107) z (cid:48) (cid:107)
Revised: = (cid:107) z (cid:107)(cid:107) z (cid:48) (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:96) (cid:18) (
Revised: (cid:96) (cid:18) (

Original: cid:104) z, z (cid:48) (cid:105) (cid:107) z
Revised: cid:104) z, z (cid:48) (cid:105) (cid:107) z

Original: (cid:107)(cid:107) z (cid:48) (cid:107) (cid:19) =
Revised: (cid:107)(cid:107) z (cid:48) (cid:107) (cid:19) =

Original: [Equation]
Revised: [Equation]

Original: where ϕ (cid:96) : H | S (cid:96) | (cid:96) − 1 → H (cid:96) is a feature map for the kernel.
Revised: where ϕ (cid:96) : H | S (cid:96) | (cid:96) − 1 → H (cid:96) is a feature map for the kernel.

Original: The kernel functions take the form κ (cid:96) ( u )
Revised: The kernel functions take the form κ (cid:96) ( u )

Original: =
Revised: =

Original: (cid:80) j ≥ 

Revised: (cid:80) j ≥

Original: b j u j with b j ≥ 0 .
Revised: b j u j with b j ≥ 0 .

Original: This includes the exponential kernel κ ( u ) =
Revised: This includes the exponential kernel κ ( u ) =

Original: e α ( u − 1) ( i.e. , Gaussian kernel on the sphere) and the arc-cosine kernel arising from random ReLU features (Cho & Saul, 2009), for which our construction is equivalent to that of the conjugate or NNGP kernel for an inﬁnite-width random ReLU network with the same architecture.
Revised: e α ( u − 1) ( i.e. , Gaussian kernel on the sphere) and the arc-cosine kernel arising from random ReLU features (Cho & Saul, 2009), for which our construction is equivalent to that of the conjugate or NNGP kernel for an inﬁnite-width random ReLU network with the same architecture.

Original: The operator M (cid:96) is then deﬁned pointwise by
Revised: The operator M (cid:96) is then deﬁned pointwise by

Original: [Equation]
Revised: [Equation]

Original: At the ﬁrst layer on image patches, these kernels lead to functional spaces consisting of homogeneous functions with varying degrees of smoothness on the sphere, depending on the properties of the kernel (Bach, 2017a; Smola et al., 2001).
Revised: At the ﬁrst layer on image patches, these kernels lead to functional spaces consisting of homogeneous functions with varying degrees of smoothness on the sphere, depending on the properties of the kernel (Bach, 2017a; Smola et al., 2001).

Original: At higher layers, our theoretical analysis will also consider simple polynomial kernels such as k (cid:96) ( z, z (cid:48) )
Revised: At higher layers, our theoretical analysis will also consider simple polynomial kernels such as k (cid:96) ( z, z (cid:48) )

Original: =
Revised: =

Original: ( (cid:104) z, z (cid:48) (cid:105) )
Revised: ( (cid:104) z, z (cid:48) (cid:105) )

Original: r , in which case the feature map may be explicitly written in terms of tensor products.
Revised: r , in which case the feature map may be explicitly written in terms of tensor products.

Original: For instance, r = 2 gives ϕ (cid:96) ( z ) =
Revised: For instance, r = 2 gives ϕ (cid:96) ( z ) =

Original: z ⊗ z and H (cid:96) =
Revised: z ⊗ z and H (cid:96) =

Original: ( H | S (cid:96) | (cid:96) − 1 ) ⊗ 

Revised: ( H | S (cid:96) | (cid:96) − 1 ) ⊗

Original: =
Revised: =

Original: ( H (cid:96) − 1 ⊗ H (cid:96) − 1 ) | S (cid:96) |×| S (cid:96) | .
Revised: ( H (cid:96) − 1 ⊗ H (cid:96) − 1 ) | S (cid:96) |×| S (cid:96) | .

Original: See Appendix A.2 for more background on dot-product kernels, their tensor products, and their regularization properties.
Revised: See Appendix A.2 for more background on dot-product kernels, their tensor products, and their regularization properties.

Original: Pooling.
Revised: Pooling.

Original: Finally, the pooling operators A (cid:96) perform local averaging through convolution with a ﬁlter h (cid:96) [ u ] , which we may consider to be symmetric ( h (cid:96) [ − u ]=:¯ h (cid:96) [ u ]= h (cid:96) [ u ] ).
Revised: Finally, the pooling operators A (cid:96) perform local averaging through convolution with a ﬁlter h (cid:96) [ u ] , which we may consider to be symmetric ( h (cid:96) [ − u ]=:¯ h (cid:96) [ u ]= h (cid:96) [ u ] ).

Original: In practice, the pooling operation is often followed by downsampling by a factor s (cid:96) , in which case the new signal A (cid:96) x is deﬁned on a new domain Ω (cid:96) with | Ω (cid:96) | = | Ω (cid:96) − 1 | /s (cid:96) , and we may write for x ∈ L 2 (Ω (cid:96) − 1 ) and u ∈ Ω (cid:96) ,
Revised: In practice, the pooling operation is often followed by downsampling by a factor s (cid:96) , in which case the new signal A (cid:96) x is deﬁned on a new domain Ω (cid:96) with | Ω (cid:96) | = | Ω (cid:96) − 1 | /s (cid:96) , and we may write for x ∈ L 2 (Ω (cid:96) − 1 ) and u ∈ Ω (cid:96) ,

Original: [Equation]
Revised: [Equation]

Original: Our experiments consider Gaussian pooling ﬁlters with a size and bandwidth proportional to the downsampling factor s (cid:96) , following Mairal (2016), namely, size 2 s (cid:96) + 1 and bandwidth √ 2 s (cid:96) .
Revised: Our experiments consider Gaussian pooling ﬁlters with a size and bandwidth proportional to the downsampling factor s (cid:96) , following Mairal (2016), namely, size 2 s (cid:96) + 1 and bandwidth √ 2 s (cid:96) .

Original: In
Revised: In

Original: Section 3, we will often assume no downsampling for simplicity, in which case we may see the ﬁlter bandwidth as increasing with the layers.
Revised: Section 3, we will often assume no downsampling for simplicity, in which case we may see the ﬁlter bandwidth as increasing with the layers.

Original: Links with other convolutional kernels.
Revised: Links with other convolutional kernels.

Original: We note that our construction closely resembles kernels derived from inﬁnitely wide convolutional networks, known as conjugate or NNGP kernels (GarrigaAlonso et al., 2019; Novak et al., 2019), and is also related to convolutional neural tangent kernels (Arora et al., 2019; Bietti & Mairal, 2019b; Yang, 2019).
Revised: We note that our construction closely resembles kernels derived from inﬁnitely wide convolutional networks, known as conjugate or NNGP kernels (GarrigaAlonso et al., 2019; Novak et al., 2019), and is also related to convolutional neural tangent kernels (Arora et al., 2019; Bietti & Mairal, 2019b; Yang, 2019).

Original: The Myrtle family of kernels (Shankar et al., 2020) also resembles our models, but they use small average pooling ﬁlters instead of Gaussian ﬁlters, which leads to deeper architectures due to smaller receptive ﬁelds.
Revised: The Myrtle family of kernels (Shankar et al., 2020) also resembles our models, but they use small average pooling ﬁlters instead of Gaussian ﬁlters, which leads to deeper architectures due to smaller receptive ﬁelds.

Original:  A PPROXIMATION WITH (D EEP ) C ONVOLUTIONAL K ERNELS
Revised:  A PPROXIMATION WITH (D EEP ) C ONVOLUTIONAL K ERNELS

Original: In this section, we present our main results on the approximation properties of convolutional kernels, by characterizing functions in the RKHS as well as their norms.
Revised: In this section, we present our main results on the approximation properties of convolutional kernels, by characterizing functions in the RKHS as well as their norms.

Original: We begin with the one-layer case, which does not capture interactions between patches but highlights the role of pooling, before moving multiple layers, where interaction terms play an important role.
Revised: We begin with the one-layer case, which does not capture interactions between patches but highlights the role of pooling, before moving multiple layers, where interaction terms play an important role.

Original: Proofs are given in Appendix E.
Revised: Proofs are given in Appendix E.

Original: 3. T HE O NE -L AYER C ASE
Revised: 3. T HE O NE -L AYER C ASE

Original: We begin by considering the case of a single convolutional layer, which can already help us illustrate the role of patches and pooling.
Revised: We begin by considering the case of a single convolutional layer, which can already help us illustrate the role of patches and pooling.

Original: Here, the kernel is given by
Revised: Here, the kernel is given by

Original: [Equation]
Revised: [Equation]

Original: with Φ( x )[ u ] = ϕ
Revised: with Φ( x )[ u ] = ϕ

Original: ( x u ) , where we use the shorthand x u
Revised: ( x u ) , where we use the shorthand x u

Original: = Px [ u ] for the patch at position u .
Revised: = Px [ u ] for the patch at position u .

Original: We now characterize the RKHS of K 1 , showing that it consists of additive models of functions in H deﬁned on patches, with spatial regularities among the terms, induced by the pooling operator A .
Revised: We now characterize the RKHS of K 1 , showing that it consists of additive models of functions in H deﬁned on patches, with spatial regularities among the terms, induced by the pooling operator A .

Original: ( Notation :
Revised: ( Notation :

Original: A ∗ and A † denote the adjoint and pseudo-inverse of an operator A , respectively.)
Revised: A ∗ and A † denote the adjoint and pseudo-inverse of an operator A , respectively.)

Original: Proposition 1 (RKHS for 1-layer CKN.) .
Revised: Proposition 1 (RKHS for 1-layer CKN.) .

Original: The RKHS of K 1 consists of functions f ( x ) =
Revised: The RKHS of K 1 consists of functions f ( x ) =

Original: (cid:104) G, Φ( x ) (cid:105)
Revised: (cid:104) G, Φ( x ) (cid:105)

Original: L 2 (Ω , H ) =
Revised: L 2 (Ω , H ) =

Original: (cid:80) u ∈ Ω G [ u ]( x u ) , with G ∈ Range ( A ∗ ) , and with RKHS norm
Revised: (cid:80) u ∈ Ω G [ u ]( x u ) , with G ∈ Range ( A ∗ ) , and with RKHS norm

Original: [Equation]
Revised: [Equation]

Original: Note that if A ∗ is not invertible (for instance in the presence of downsampling), the constraint G ∈
Revised: Note that if A ∗ is not invertible (for instance in the presence of downsampling), the constraint G ∈

Original: Range( A ∗ ) is active and A †∗ is its pseudo-inverse.
Revised: Range( A ∗ ) is active and A †∗ is its pseudo-inverse.

Original: In the extreme case of global average pooling, we have A = (1 , . . .
Revised: In the extreme case of global average pooling, we have A = (1 , . . .

Original: , 1) ⊗ Id : L 2 (Ω , H )
Revised: , 1) ⊗ Id : L 2 (Ω , H )

Original: → H , so that G ∈ Range( A ∗ ) is equivalent to G [ u ]
Revised: → H , so that G ∈ Range( A ∗ ) is equivalent to G [ u ]

Original: = g for all u , for some ﬁxed g ∈ H .
Revised: = g for all u , for some ﬁxed g ∈ H .

Original: In this case, the penalty in (5) is simply the squared RKHS norm (cid:107) g (cid:107) 2 H .
Revised: In this case, the penalty in (5) is simply the squared RKHS norm (cid:107) g (cid:107) 2 H .

Original: In order to understand the norm (5) for general pooling, recall that A is a convolution operator with ﬁlter h 1 , hence its inverse (which we now assume exists for simplicity) may be easily computed in the Fourier basis.
Revised: In order to understand the norm (5) for general pooling, recall that A is a convolution operator with ﬁlter h 1 , hence its inverse (which we now assume exists for simplicity) may be easily computed in the Fourier basis.

Original: In particular, for a patch z ∈ R p | S 1 | , deﬁning the scalar signal g z [ u ] =
Revised: In particular, for a patch z ∈ R p | S 1 | , deﬁning the scalar signal g z [ u ] =

Original: G [ u ]( z ) , we may write the following using the reproducing property and linearity:
Revised: G [ u ]( z ) , we may write the following using the reproducing property and linearity:

Original: [Equation]
Revised: [Equation]

Original: where ¯ h 1 [ u ] :
Revised: where ¯ h 1 [ u ] :

Original: = h 1 [ − u ] arises from transposition, F is the discrete Fourier transform, and both F and A are viewed here as | Ω | × | Ω | matrices.
Revised: = h 1 [ − u ] arises from transposition, F is the discrete Fourier transform, and both F and A are viewed here as | Ω | × | Ω | matrices.

Original: From this expression, we see that by penalizing the RKHS norm of f , we are implicitly penalizing the high frequencies of the signals g z
Revised: From this expression, we see that by penalizing the RKHS norm of f , we are implicitly penalizing the high frequencies of the signals g z

Original: [ u ] for any z , and this regularization is stronger when the pooling ﬁlter h 1 has a fast spectral decay.
Revised: [ u ] for any z , and this regularization is stronger when the pooling ﬁlter h 1 has a fast spectral decay.

Original: For instance, as the spatial bandwidth of h 1 increases (approaching a global pooling operation), F h 1 decreases more rapidly, which encourages g z [ u ] to be more smooth as a function of u , and thus prevents f from relying too much on the location of patches.
Revised: For instance, as the spatial bandwidth of h 1 increases (approaching a global pooling operation), F h 1 decreases more rapidly, which encourages g z [ u ] to be more smooth as a function of u , and thus prevents f from relying too much on the location of patches.

Original: If instead h 1 is very localized in space ( e.g. , a
Revised: If instead h 1 is very localized in space ( e.g. , a

Original: Dirac ﬁlter, which corresponds to no pooling), g z [ u ] may vary much more rapidly as a function of u , which then allows f to discriminate differently depending on the spatial location.
Revised: Dirac ﬁlter, which corresponds to no pooling), g z [ u ] may vary much more rapidly as a function of u , which then allows f to discriminate differently depending on the spatial location.

Original: This provides a different perspective on the invariance properties induced by pooling.
Revised: This provides a different perspective on the invariance properties induced by pooling.

Original: If we denote ˜ G [ u ] =
Revised: If we denote ˜ G [ u ] =

Original: A †∗ G [ u ] , the penalty writes
Revised: A †∗ G [ u ] , the penalty writes

Original: [Equation]
Revised: [Equation]

Original: Here, the RKHS norm (cid:107) · (cid:107) H also controls smoothness, but this time for functions
Revised: Here, the RKHS norm (cid:107) · (cid:107) H also controls smoothness, but this time for functions

Original: ˜ G
Revised: ˜ G

Original: [ u ]( · ) deﬁned on input patches.
Revised: [ u ]( · ) deﬁned on input patches.

Original: For homogeneous dot-product kernels of the form (2), the norm takes the form
Revised: For homogeneous dot-product kernels of the form (2), the norm takes the form

Original: [Equation]
Revised: [Equation]

Original: (cid:107) T − 12 g (cid:107) L 2 ( S d − 1 ) , where the regularization operator T − 12 is the self-adjoint inverse square root of the integral operator for the patch kernel restricted to L 2 ( S d − 1 ) .
Revised: (cid:107) T − 12 g (cid:107) L 2 ( S d − 1 ) , where the regularization operator T − 12 is the self-adjoint inverse square root of the integral operator for the patch kernel restricted to L 2 ( S d − 1 ) .

Original: For instance, when the eigenvalues of T decay polynomially, as for arc-cosine kernels, T − 12 behaves like a power of the spherical
Revised: For instance, when the eigenvalues of T decay polynomially, as for arc-cosine kernels, T − 12 behaves like a power of the spherical

Original: Laplacian (see Bach, 2017a, and Appendix A.2).
Revised: Laplacian (see Bach, 2017a, and Appendix A.2).

Original: Then we may write
Revised: Then we may write

Original: [Equation]
Revised: [Equation]

Original: which highlights that the norm applies two regularization operators ( A − 1 ) (cid:62) and T − 12 independently on the spatial variable and the patch variable of ( u, z )
Revised: which highlights that the norm applies two regularization operators ( A − 1 ) (cid:62) and T − 12 independently on the spatial variable and the patch variable of ( u, z )

Original: (cid:55)→ G [ u ]( z ) , viewed here as an element of L 2 (Ω) ⊗ L 2 ( S d − 1 ) .
Revised: (cid:55)→ G [ u ]( z ) , viewed here as an element of L 2 (Ω) ⊗ L 2 ( S d − 1 ) .

Original: [Table]
Revised: [Table]

Original: We now study the case of convolutional kernels with more than one convolutional layer.
Revised: We now study the case of convolutional kernels with more than one convolutional layer.

Original: While the patch kernels used at higher layers are typically similar to the ones from the ﬁrst layer, we show empirically on Cifar10 that they may be replaced by simple polynomial kernels with little loss in accuracy.
Revised: While the patch kernels used at higher layers are typically similar to the ones from the ﬁrst layer, we show empirically on Cifar10 that they may be replaced by simple polynomial kernels with little loss in accuracy.

Original: We then proceed by studying the RKHS of such simpliﬁed models, highlighting the role of depth for capturing interactions between different patches via kernel tensor products.
Revised: We then proceed by studying the RKHS of such simpliﬁed models, highlighting the role of depth for capturing interactions between different patches via kernel tensor products.

Original: An empirical study.
Revised: An empirical study.

Original: Table 1 shows the performance of a given 2-layer convolutional kernel architecture, with different choices of patch kernels κ 1 and κ 2 .
Revised: Table 1 shows the performance of a given 2-layer convolutional kernel architecture, with different choices of patch kernels κ 1 and κ 2 .

Original: The reference model uses exponential kernels in both layers, following the construction in Mairal (2016).
Revised: The reference model uses exponential kernels in both layers, following the construction in Mairal (2016).

Original: We ﬁnd that replacing the second layer kernel by a simple polynomial kernel of degree 3, κ 2 ( u ) =
Revised: We ﬁnd that replacing the second layer kernel by a simple polynomial kernel of degree 3, κ 2 ( u ) =

Original: u 3 , leads to roughly the same test accuracy.
Revised: u 3 , leads to roughly the same test accuracy.

Original: By changing κ 2 to κ 2 ( u ) = u 2 , the test accuracy is only about 1% lower, while doing the same for the ﬁrst layer decreases it by about 3%.
Revised: By changing κ 2 to κ 2 ( u ) = u 2 , the test accuracy is only about 1% lower, while doing the same for the ﬁrst layer decreases it by about 3%.

Original: The shallow kernel with a single non-linear convolutional layer (shown in the last line of Table 1) performs signiﬁcantly worse.
Revised: The shallow kernel with a single non-linear convolutional layer (shown in the last line of Table 1) performs signiﬁcantly worse.

Original: This suggests that the approximation properties described in Section 3.1 may not be sufﬁcient for this task, while even a simple polynomial kernel of order 2 at the second layer may substantially improve things by capturing interactions, in a way that we describe below.
Revised: This suggests that the approximation properties described in Section 3.1 may not be sufﬁcient for this task, while even a simple polynomial kernel of order 2 at the second layer may substantially improve things by capturing interactions, in a way that we describe below.

Original: Two-layers with a quadratic kernel.
Revised: Two-layers with a quadratic kernel.

Original: Motivated by the above experiments, we now study the
Revised: Motivated by the above experiments, we now study the

Original: RKHS of a two-layer kernel K 2 ( x, x (cid:48) )
Revised: RKHS of a two-layer kernel K 2 ( x, x (cid:48) )

Original: = (cid:104) Ψ
Revised: = (cid:104) Ψ

Original: ( x ) , Ψ( x (cid:48) )
Revised: ( x ) , Ψ( x (cid:48) )

Original: (cid:105)
Revised: (cid:105)

Original: L 2 (Ω 2 , H 2 ) with Ψ as in (1) with L = 2 , where the second-layer uses a quadratic kernel 2 on patches k 2 ( z, z (cid:48) )
Revised: L 2 (Ω 2 , H 2 ) with Ψ as in (1) with L = 2 , where the second-layer uses a quadratic kernel 2 on patches k 2 ( z, z (cid:48) )

Original: =
Revised: =

Original: ( (cid:104) z, z (cid:48) (cid:105) )
Revised: ( (cid:104) z, z (cid:48) (cid:105) )

Original: An explicit feature map for k 2 is given by ϕ 2 ( z ) =
Revised: An explicit feature map for k 2 is given by ϕ 2 ( z ) =

Original: z ⊗ z .
Revised: z ⊗ z .

Original: Denoting by H the RKHS of k 1 , the patches z lie in H | S 2 | , thus we may view ϕ 2 as a feature map into a Hilbert space H 

Revised: Denoting by H the RKHS of k 1 , the patches z lie in H | S 2 | , thus we may view ϕ 2 as a feature map into a Hilbert space H

Original: = ( H ⊗ H ) | S 2 |×| S 2 | (by isomorphism to H | S 2 | ⊗ H | S 2 | ).
Revised: = ( H ⊗ H ) | S 2 |×| S 2 | (by isomorphism to H | S 2 | ⊗ H | S 2 | ).

Original: The following result characterizes the RKHS of such a 2-layer convolutional kernel, showing that it consists of additive models of interaction functions in H ⊗ H on pairs of patches, with different spatial regularities on the interaction terms induced by the two pooling operations.
Revised: The following result characterizes the RKHS of such a 2-layer convolutional kernel, showing that it consists of additive models of interaction functions in H ⊗ H on pairs of patches, with different spatial regularities on the interaction terms induced by the two pooling operations.

Original: ( Notation : we use the notations diag( M )[ u ] =
Revised: ( Notation : we use the notations diag( M )[ u ] =

Original: M [ u, u ] for M ∈ L 2 (Ω 2 ) , diag( x )[ u, v ] = 1 { u = v } x [ u ] for x ∈ L 2 (Ω) , and L c is the translation operator L c x
Revised: M [ u, u ] for M ∈ L 2 (Ω 2 ) , diag( x )[ u, v ] = 1 { u = v } x [ u ] for x ∈ L 2 (Ω) , and L c is the translation operator L c x

Original: [ u ] = x [ u − c ] .)
Revised: [ u ] = x [ u − c ] .)

Original: Proposition 2 (RKHS of 2-layer CKN with quadratic k 2 ) .
Revised: Proposition 2 (RKHS of 2-layer CKN with quadratic k 2 ) .

Original: Let Φ( x ) =
Revised: Let Φ( x ) =

Original: ( ϕ 1 ( x u ) ⊗ ϕ 1 ( x v ))
Revised: ( ϕ 1 ( x u ) ⊗ ϕ 1 ( x v ))

Original: u,v ∈ Ω ∈
Revised: u,v ∈ Ω ∈

Original: L 2 (Ω 2 , H ⊗ H ) .
Revised: L 2 (Ω 2 , H ⊗ H ) .

Original: The RKHS of K 2 when k 2 ( z, z (cid:48) )
Revised: The RKHS of K 2 when k 2 ( z, z (cid:48) )

Original: =
Revised: =

Original: ( (cid:104) z, z (cid:48) (cid:105) )
Revised: ( (cid:104) z, z (cid:48) (cid:105) )

Original: 2 consists of functions of the form f ( x ) =
Revised: 2 consists of functions of the form f ( x ) =

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where G pq ∈ L 2 (Ω 2 , H ⊗ H )
Revised: where G pq ∈ L 2 (Ω 2 , H ⊗ H )

Original: obeys the constraints G pq ∈ Range
Revised: obeys the constraints G pq ∈ Range

Original: ( E pq ) and diag(( L p A 1 ⊗
Revised: ( E pq ) and diag(( L p A 1 ⊗

Original: L q A 1 ) †∗ G pq ) ∈
Revised: L q A 1 ) †∗ G pq ) ∈

Original: Range( A ∗ 2 ) .
Revised: Range( A ∗ 2 ) .

Original: Here, E pq : L 2 (Ω 1 ) →
Revised: Here, E pq : L 2 (Ω 1 ) →

Original: L 2 (Ω 2 ) is a linear operator given by
Revised: L 2 (Ω 2 ) is a linear operator given by

Original: [Equation]
Revised: [Equation]

Original: The squared RKHS norm (cid:107) f (cid:107) 2 H K 2 is then equal to the minimum over such decompositions of the quantity (cid:88)
Revised: The squared RKHS norm (cid:107) f (cid:107) 2 H K 2 is then equal to the minimum over such decompositions of the quantity (cid:88)

Original: [Equation]
Revised: [Equation]

Original: As discussed in the one-layer case, the inverses should be replaced by pseudo-inverses if needed, e.g. , when using downsampling.
Revised: As discussed in the one-layer case, the inverses should be replaced by pseudo-inverses if needed, e.g. , when using downsampling.

Original: In particular, if A ∗ 2 is singular, the second constraint plays a similar role to the one-layer case.
Revised: In particular, if A ∗ 2 is singular, the second constraint plays a similar role to the one-layer case.

Original: In order to understand the ﬁrst constraint, we show in Figure 2 the outputs
Revised: In order to understand the ﬁrst constraint, we show in Figure 2 the outputs

Original: The penalty then involves operators L p
Revised: The penalty then involves operators L p

Original: A 1 ⊗ L q A 1 , which may be seen as separable 2D convolutions on the “images” G pq [ u, v ] .
Revised: A 1 ⊗ L q A 1 , which may be seen as separable 2D convolutions on the “images” G pq [ u, v ] .

Original: Then, if z, z (cid:48) ∈
Revised: Then, if z, z (cid:48) ∈

Original: R p | S 1 | are two ﬁxed patches, deﬁning g z,z (cid:48)
Revised: R p | S 1 | are two ﬁxed patches, deﬁning g z,z (cid:48)

Original: [ u, v ] =
Revised: [ u, v ] =

Original: G pq [ u, v ]( z, z (cid:48) ) , we have, assuming A 1 is invertible and symmetric,
Revised: G pq [ u, v ]( z, z (cid:48) ) , we have, assuming A 1 is invertible and symmetric,

Original: [Equation]
Revised: [Equation]

Original: where F 2 = F ⊗ F is the 2D discrete Fourier transform.
Revised: where F 2 = F ⊗ F is the 2D discrete Fourier transform.

Original: Thus, this penalizes the variations of g z,z (cid:48) in both dimensions, encouraging the interaction functions to not rely too strongly on the speciﬁc positions of the two patches.
Revised: Thus, this penalizes the variations of g z,z (cid:48) in both dimensions, encouraging the interaction functions to not rely too strongly on the speciﬁc positions of the two patches.

Original: This regularization is stronger when the spatial bandwidth of h 1 is large, since this leads to a more localized ﬁlter in the frequency domain, with stronger penalties on high frequencies.
Revised: This regularization is stronger when the spatial bandwidth of h 1 is large, since this leads to a more localized ﬁlter in the frequency domain, with stronger penalties on high frequencies.

Original: In addition to this 2D smoothness, the penalty in Proposition 2 also encourages smoothness along the p − q diagonal of this resulting 2D image using the pooling operator A 2 .
Revised: In addition to this 2D smoothness, the penalty in Proposition 2 also encourages smoothness along the p − q diagonal of this resulting 2D image using the pooling operator A 2 .

Original: This has a similar behavior to the one-layer case, where the penalty prevents the functions from relying too much on the absolute position of the patches.
Revised: This has a similar behavior to the one-layer case, where the penalty prevents the functions from relying too much on the absolute position of the patches.

Original: Since A 2 typically has a larger bandwidth than A 1 , interaction functions G pq [ u, u + r ] are allowed to vary with r more rapidly than with u .
Revised: Since A 2 typically has a larger bandwidth than A 1 , interaction functions G pq [ u, u + r ] are allowed to vary with r more rapidly than with u .

Original: The regularity of the resulting “smoothed” interaction terms as a function of the input patches is controlled by the
Revised: The regularity of the resulting “smoothed” interaction terms as a function of the input patches is controlled by the

Original: RKHS norm of the tensor product kernel k 1 ⊗ k 1 as described in Appendix A.2.
Revised: RKHS norm of the tensor product kernel k 1 ⊗ k 1 as described in Appendix A.2.

Original: Extensions.
Revised: Extensions.

Original: When using a polynomial kernel k 2 ( z, z (cid:48) )
Revised: When using a polynomial kernel k 2 ( z, z (cid:48) )

Original: =
Revised: =

Original: ( (cid:104) z, z (cid:48) (cid:105) )
Revised: ( (cid:104) z, z (cid:48) (cid:105) )

Original: α with α > 2 , we obtain a similar picture as above, with higher-order interaction terms.
Revised: α with α > 2 , we obtain a similar picture as above, with higher-order interaction terms.

Original: For example, if α = 3 , the RKHS contains functions with interaction terms of the form G pqr [ u, v, w ]( x u , x v , x w ) , with a penalty (cid:88)
Revised: For example, if α = 3 , the RKHS contains functions with interaction terms of the form G pqr [ u, v, w ]( x u , x v , x w ) , with a penalty (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where A 1 c =
Revised: where A 1 c =

Original: L c A 1 .
Revised: L c A 1 .

Original: Similarly to the quadratic case, the ﬁrst-layer pooling operator encourages smoothness with respect to relative positions between patches, while the second-layer pooling penalizes dependence on the global location.
Revised: Similarly to the quadratic case, the ﬁrst-layer pooling operator encourages smoothness with respect to relative positions between patches, while the second-layer pooling penalizes dependence on the global location.

Original: One may extend this further to higher orders to capture more complex interactions, and our experiments suggest that a two-layer kernel of this form with a degree-4 polynomial at the second layer may achieve state-of-the-art accuracy for kernel methods on Cifar10 (see Table 2).
Revised: One may extend this further to higher orders to capture more complex interactions, and our experiments suggest that a two-layer kernel of this form with a degree-4 polynomial at the second layer may achieve state-of-the-art accuracy for kernel methods on Cifar10 (see Table 2).

Original: We note that such ﬁxed-order choices for κ 2 lead to convolutional kernels that lower-bound richer kernels with, e.g. , an exponential kernel at the second layer, in the Loewner order on positive-deﬁnite kernels.
Revised: We note that such ﬁxed-order choices for κ 2 lead to convolutional kernels that lower-bound richer kernels with, e.g. , an exponential kernel at the second layer, in the Loewner order on positive-deﬁnite kernels.

Original: This imples in particular that the RKHS of these “richer” kernels also contains the functions described above.
Revised: This imples in particular that the RKHS of these “richer” kernels also contains the functions described above.

Original: For more than two layers with polynomial kernels, one similarly obtains higher-order interactions, but with different regularization properties (see
Revised: For more than two layers with polynomial kernels, one similarly obtains higher-order interactions, but with different regularization properties (see

Original: Appendix D).
Revised: Appendix D).

Original:  G ENERALIZATION P ROPERTIES
Revised:  G ENERALIZATION P ROPERTIES

Original: In this section, we study generalization properties of the convolutional kernels studied in Section 3, and show improved sample complexity guarantees for architectures with pooling and small patches when the problem exhibits certain invariance properties.
Revised: In this section, we study generalization properties of the convolutional kernels studied in Section 3, and show improved sample complexity guarantees for architectures with pooling and small patches when the problem exhibits certain invariance properties.

Original: Learning setting.
Revised: Learning setting.

Original: We consider a non-parametric regression setting with data distribution ρ over ( x, y ) , where the goal is to minimize R ( f ) =
Revised: We consider a non-parametric regression setting with data distribution ρ over ( x, y ) , where the goal is to minimize R ( f ) =

Original: E ( x,y ) ∼ ρ
Revised: E ( x,y ) ∼ ρ

Original: [( y − f ( x )) 2 ] .
Revised: [( y − f ( x )) 2 ] .

Original: We denote by f ∗ =
Revised: We denote by f ∗ =

Original: E ρ [ y | x ] = arg min f R ( f ) the regression function, and assume f ∗ ∈ H for some RKHS H with kernel K .
Revised: E ρ [ y | x ] = arg min f R ( f ) the regression function, and assume f ∗ ∈ H for some RKHS H with kernel K .

Original: Without any further assumptions on the kernel, we have the following generalization bound on the excess risk for the kernel ridge regression (KRR) estimator, denoted ˆ f n (see Proposition 7 in
Revised: Without any further assumptions on the kernel, we have the following generalization bound on the excess risk for the kernel ridge regression (KRR) estimator, denoted ˆ f n (see Proposition 7 in

Original: Appendix E):
Revised: Appendix E):

Original: [Equation]
Revised: [Equation]

Original: where ρ X is the marginal distribution of ρ on inputs x , C is an absolute constant, and τ 2 ρ is an upper bound on the conditional noise variance Var[ y | x ] .
Revised: where ρ X is the marginal distribution of ρ on inputs x , C is an absolute constant, and τ 2 ρ is an upper bound on the conditional noise variance Var[ y | x ] .

Original: We note that this 1 / √ n rate is optimal if no further assumptions are made on the kernel (Caponnetto & De Vito, 2007).
Revised: We note that this 1 / √ n rate is optimal if no further assumptions are made on the kernel (Caponnetto & De Vito, 2007).

Original: The quantity E x ∼ ρ X
Revised: The quantity E x ∼ ρ X

Original: [ K ( x, x )] corresponds to the trace of the covariance operator, and thus provides a global control of eigenvalues through their sum, which will already highlight the gains that pooling can achieve.
Revised: [ K ( x, x )] corresponds to the trace of the covariance operator, and thus provides a global control of eigenvalues through their sum, which will already highlight the gains that pooling can achieve.

Original: Faster rates can be achieved, e.g. , when assuming certain eigenvalue decays on the covariance operator, or when further restricting f ∗ .
Revised: Faster rates can be achieved, e.g. , when assuming certain eigenvalue decays on the covariance operator, or when further restricting f ∗ .

Original: We discuss in Appendix F.2 how similar gains to those described in this section can extend to fast rate settings under speciﬁc scenarios.
Revised: We discuss in Appendix F.2 how similar gains to those described in this section can extend to fast rate settings under speciﬁc scenarios.

Original: One-layer CKN with invariance.
Revised: One-layer CKN with invariance.

Original: As discussed in Section 3, the RKHS of 1-layer CKNs consists of sums of functions that are localized on patches, each belonging to the RKHS H of the patch kernel k 1 .
Revised: As discussed in Section 3, the RKHS of 1-layer CKNs consists of sums of functions that are localized on patches, each belonging to the RKHS H of the patch kernel k 1 .

Original: The next result illustrates the beneﬁts of pooling when f ∗ is translation invariant.
Revised: The next result illustrates the beneﬁts of pooling when f ∗ is translation invariant.

Original: Proposition 3 (Generalization for 1-layer CKN.) .
Revised: Proposition 3 (Generalization for 1-layer CKN.) .

Original: Assume f ∗ ( x )
Revised: Assume f ∗ ( x )

Original: =
Revised: =

Original: (cid:80) u ∈ Ω g ( x u ) with g ∈ H of minimal norm, and assume E x ∼ ρ X
Revised: (cid:80) u ∈ Ω g ( x u ) with g ∈ H of minimal norm, and assume E x ∼ ρ X

Original: [ k 1 ( x u , x v )] ≤ σ 2 u − v for some ( σ 2 r ) r ∈ Ω .
Revised: [ k 1 ( x u , x v )] ≤ σ 2 u − v for some ( σ 2 r ) r ∈ Ω .

Original: For a 1-layer CKN Kwith any pooling ﬁlter h ≥ 0 with (cid:107) h (cid:107) 1 = 1 , we have (cid:107) f ∗ (cid:107) H K 1 = (cid:112) | Ω |(cid:107) g (cid:107) H , KRR satisﬁes
Revised: For a 1-layer CKN Kwith any pooling ﬁlter h ≥ 0 with (cid:107) h (cid:107) 1 = 1 , we have (cid:107) f ∗ (cid:107) H K 1 = (cid:112) | Ω |(cid:107) g (cid:107) H , KRR satisﬁes

Original: [Equation]
Revised: [Equation]

Original: The quantities σ 2 r can be interpreted as auto-correlations between patches at distance r from each other.
Revised: The quantities σ 2 r can be interpreted as auto-correlations between patches at distance r from each other.

Original: Note that if h is a Dirac ﬁlter, then (cid:104) h, L r h (cid:105) =
Revised: Note that if h is a Dirac ﬁlter, then (cid:104) h, L r h (cid:105) =

Original: 1 { r = 0 } (recall L r h
Revised: 1 { r = 0 } (recall L r h

Original: [ u ] = h [ u − r ] ), thus only σ 20 plays a role in the bound, while if h is an average pooling ﬁlter, we have (cid:104) h, L r h (cid:105) = 1 / | Ω | , so that σ 20 is replaced by the average ¯ σ 2 :
Revised: [ u ] = h [ u − r ] ), thus only σ 20 plays a role in the bound, while if h is an average pooling ﬁlter, we have (cid:104) h, L r h (cid:105) = 1 / | Ω | , so that σ 20 is replaced by the average ¯ σ 2 :

Original: = (cid:80) r
Revised: = (cid:80) r

Original: σ 2 r / | Ω | .
Revised: σ 2 r / | Ω | .

Original: Natural signals commonly display a decay in their auto-correlation functions, suggesting that a similar decay may be present in σ 2 r as a function of r .
Revised: Natural signals commonly display a decay in their auto-correlation functions, suggesting that a similar decay may be present in σ 2 r as a function of r .

Original: In this case, ¯ σ 2 may be much smaller than σ 20 , which in turn yields an improved sample complexity guarantee for learning such an f ∗ with global pooling, by a factor up to | Ω | in the extreme case where σ 2 r vanishes for r ≥ 1 .
Revised: In this case, ¯ σ 2 may be much smaller than σ 20 , which in turn yields an improved sample complexity guarantee for learning such an f ∗ with global pooling, by a factor up to | Ω | in the extreme case where σ 2 r vanishes for r ≥ 1 .

Original: In Appendix F.1, we provide simple models where this can be quantiﬁed.
Revised: In Appendix F.1, we provide simple models where this can be quantiﬁed.

Original: For more general ﬁlters, such as local averaging or Gaussian ﬁlters, and assuming σ 2 r ≈ 0 for r (cid:54) = 0 , the bound interpolates between no pooling and global pooling through the quantity (cid:107) h (cid:107) 22 .
Revised: For more general ﬁlters, such as local averaging or Gaussian ﬁlters, and assuming σ 2 r ≈ 0 for r (cid:54) = 0 , the bound interpolates between no pooling and global pooling through the quantity (cid:107) h (cid:107) 22 .

Original: While this yields a worse bound than global pooling on invariant functions, such ﬁlters enable learning functions that are not fully invariant, but exhibit some smoothness along the translation group, more efﬁciently than with no pooling.
Revised: While this yields a worse bound than global pooling on invariant functions, such ﬁlters enable learning functions that are not fully invariant, but exhibit some smoothness along the translation group, more efﬁciently than with no pooling.

Original: It should also be noted that the requirement that g belongs to an RKHS H is much weaker when the patches are small, as this typically implies that g admits more than p | S
Revised: It should also be noted that the requirement that g belongs to an RKHS H is much weaker when the patches are small, as this typically implies that g admits more than p | S

Original: | /derivatives, a condition which becomes much stronger as the patch size grows.
Revised: | /derivatives, a condition which becomes much stronger as the patch size grows.

Original: In the fast rate setting that we study in Appendix F.2, this also leads to better rates that only depend on the dimension of the patch instead of the full dimension.
Revised: In the fast rate setting that we study in Appendix F.2, this also leads to better rates that only depend on the dimension of the patch instead of the full dimension (see Theorem 8).

Original: Two layers.
Revised: Two layers.

Original: When using two layers with polynomial kernels at the second layer, we saw in Sectionthat the RKHS of CKNs consists of additive models of interaction terms of the order of the polynomial kernel used.
Revised: When using two layers with polynomial kernels at the second layer, we saw in Sectionthat the RKHS of CKNs consists of additive models of interaction terms of the order of the polynomial kernel used.

Original: The next proposition illustrates how pooling ﬁlters and patch sizes at the second layer may affect generalization on a simple target function consisting of order-2 interactions.
Revised: The next proposition illustrates how pooling ﬁlters and patch sizes at the second layer may affect generalization on a simple target function consisting of order-2 interactions.

Original: Proposition 4 (Generalization for 2-layer CKN.) .
Revised: Proposition 4 (Generalization for 2-layer CKN.) .

Original: Consider a 2-layer CKN K 2 with quadratic k 2 , as in Proposition 2, and pooling ﬁlters h 1 , h 2 with (cid:107) h 1 (cid:107) 1 = (cid:107) h 2 (cid:107) 1 = 1 .
Revised: Consider a 2-layer CKN K 2 with quadratic k 2 , as in Proposition 2, and pooling ﬁlters h 1 , h 2 with (cid:107) h 1 (cid:107) 1 = (cid:107) h 2 (cid:107) 1 = 1 .

Original: Assume that ρ X satis- ﬁes E x ∼
Revised: Assume that ρ X satis- ﬁes E x ∼

Original: [Equation]
Revised: [Equation]

Original: [Equation] ( x u , x u (cid:48) )
Revised: X [ k 1 ( x u , x u (cid:48) )

Original: k 1 ( x v , x v (cid:48) )] ≤ (cid:15) if u (cid:54) = u (cid:48) or v (cid:54) = v (cid:48) , and ≤ 1 otherwise.
Revised: k 1 ( x v , x v (cid:48) )] ≤ (cid:15) if u (cid:54) = u (cid:48) or v (cid:54) = v (cid:48) , and ≤ 1 otherwise.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Revised: (cid:33) .

Original: As an example, consider f ∗ ( x )
Revised: As an example, consider f ∗ ( x )

Original: =
Revised: =

Original: (cid:80) u,v g ( x u , x v ) for g ∈ H ⊗ H of minimal norm.
Revised: (cid:80) u,v g ( x u , x v ) for g ∈ H ⊗ H of minimal norm.

Original: The following table illustrates the obtained generalization bounds R ( ˆ f n ) − R ( f ∗ ) for KRR with various two-layer architectures ( δ : Dirac ﬁlter; 1 : global average pooling):
Revised: The following table illustrates the obtained generalization bounds R ( ˆ f n ) − R ( f ∗ ) for KRR with various two-layer architectures ( δ : Dirac ﬁlter; 1 : global average pooling):

Original: [Equation]
Revised: [Equation]

Original: The above result shows that the two-layer model allows for a much wider range of behaviors than the one-layer case, between approximation (through the norm (cid:107) f ∗ (cid:107) K 2 ) and estimation (through E x ∼ ρ X
Revised: The above result shows that the two-layer model allows for a much wider range of behaviors than the one-layer case, between approximation (through the norm (cid:107) f ∗ (cid:107) K 2 ) and estimation (through E x ∼ ρ X

Original: [ K 2 ( x, x )] ), depending on the choice of architecture.
Revised: [ K 2 ( x, x )] ), depending on the choice of architecture.

Original: Choosing the right architecture may lead to large improvements in sample complexity when the target functions has a speciﬁc structure, for instance here by a factor up to | Ω | 2 . 5 .
Revised: Choosing the right architecture may lead to large improvements in sample complexity when the target functions has a speciﬁc structure, for instance here by a factor up to | Ω | 2 . 5 .

Original: In Appendix F.1, we discuss simple possible models where we may have a small (cid:15) .
Revised: In Appendix F.1, we discuss simple possible models where we may have a small (cid:15) .

Original: Note that choosing ﬁlters that are less localized than Dirac impulses, but more than global average pooling, will again lead to different “variance” terms (10), while providing more ﬂexibility in terms of approximation compared to global pooling.
Revised: Note that choosing ﬁlters that are less localized than Dirac impulses, but more than global average pooling, will again lead to different “variance” terms (10), while providing more ﬂexibility in terms of approximation compared to global pooling.

Original: This result may be easily extended to higher-order polynomials at the second layer, by increasing the exponents on | S 2 | and (cid:104)
Revised: This result may be easily extended to higher-order polynomials at the second layer, by increasing the exponents on | S 2 | and (cid:104)

Original: h 1 , L v h 1 (cid:105) to the degree of the polynomial.
Revised: h 1 , L v h 1 (cid:105) to the degree of the polynomial.

Original: Other than the gains in sample complexity due to pooling, the bound also presents large gains compared to a “fully-connected” architecture, as in the one-layer case, since it only grows with the norm of a local interaction function in H ⊗ H that depends on two patches, which may then be small even when this function has low smoothness.
Revised: Other than the gains in sample complexity due to pooling, the bound also presents large gains compared to a “fully-connected” architecture, as in the one-layer case, since it only grows with the norm of a local interaction function in H ⊗ H that depends on two patches, which may then be small even when this function has low smoothness.

Original:  N UMERICAL E XPERIMENTS
Revised:  N UMERICAL E XPERIMENTS

Original: In this section, we provide additional experiments illustrating numerical properties of the convolutional kernels considered in this paper.
Revised: In this section, we provide additional experiments illustrating numerical properties of the convolutional kernels considered in this paper.

Original: We focus here on the Cifar10 dataset, and on CKN architectures based on the exponential kernel.
Revised: We focus here on the Cifar10 dataset, and on CKN architectures based on the exponential kernel.

Original: Additional results are given in Appendix B.
Revised: Additional results are given in Appendix B.

Original: Experimental setup on Cifar10.
Revised: Experimental setup on Cifar10.

Original: We consider classiﬁcation on Cifar10 dataset, which consists of 50k training images and 10k test images with 10 different output categories.
Revised: We consider classiﬁcation on Cifar10 dataset, which consists of 50k training images and 10k test images with 10 different output categories.

Original: We pre-process the images using a whitening/ZCA step at the patch level, which is commonly used for such kernels on images (Mairal, 2016; Shankar et al., 2020; Thiry et al., 2021).
Revised: We pre-process the images using a whitening/ZCA step at the patch level, which is commonly used for such kernels on images (Mairal, 2016; Shankar et al., 2020; Thiry et al., 2021).

Original: This may help reduce the effective dimensionality of patches, and better align the dominant eigen directions to the target function, a property which may help kernel methods (Ghorbani et al., 2020).
Revised: This may help reduce the effective dimensionality of patches, and better align the dominant eigen directions to the target function, a property which may help kernel methods (Ghorbani et al., 2020).

Original: Our convolutional kernel evaluation code is written in C++ and leverages the Eigen library for hardware-accelerated numerical computations.
Revised: Our convolutional kernel evaluation code is written in C++ and leverages the Eigen library for hardware-accelerated numerical computations.

Original: The computation of kernel matrices is distributed on up to 1000 cores on a cluster consisting of Intel Xeon processors.
Revised: The computation of kernel matrices is distributed on up to 1000 cores on a cluster consisting of Intel Xeon processors.

Original: Computing the full Cifar10 kernel matrix typically takes around 10 hours when running on all 1000 cores.
Revised: Computing the full Cifar10 kernel matrix typically takes around 10 hours when running on all 1000 cores.

Original: Our results use kernel ridge regression in a one-versus-all approach, where each labels
Revised: Our results use kernel ridge regression in a one-versus-all approach, where each labels

Original: Varying the kernel architecture.
Revised: Varying the kernel architecture.

Original: Table 2 shows test accuracies for different architectures compared to Table 1, including 3-layer models and 2-layer models with larger patches.
Revised: Table 2 shows test accuracies for different architectures compared to Table 1, including 3-layer models and 2-layer models with larger patches.

Original: In both cases, the full models with exponential kernels outperform the 2-layer architecture of Table 1, and provide comparable accuracy to the Myrtle10 kernel of Shankar et al.
Revised: In both cases, the full models with exponential kernels outperform the 2-layer architecture of Table 1, and provide comparable accuracy to the Myrtle10 kernel of Shankar et al.

Original: (2020), with an arguably simpler architecture.
Revised: (2020), with an arguably simpler architecture.

Original: We also see that using degree-3 or 4 polynomial kernels at the second second layer of the two-layer model essentially provides the same performance to the exponential kernel, and that degree-2 at the second and third layer of the 3-layer model only results in a 0.3% accuracy drop.
Revised: We also see that using degree-3 or 4 polynomial kernels at the second second layer of the two-layer model essentially provides the same performance to the exponential kernel, and that degree-2 at the second and third layer of the 3-layer model only results in a 0.3% accuracy drop.

Original: The two-layer model with degree-2 at the second layer loses about 1% accuracy, suggesting that certain
Revised: The two-layer model with degree-2 at the second layer loses about 1% accuracy, suggesting that certain

Original: Cifar10 images may require capturing interactions between at least 3 different patches in the image for good classiﬁcation, though even with only second-order interactions, these models signiﬁcantly outperform single-layer models.
Revised: Cifar10 images may require capturing interactions between at least 3 different patches in the image for good classiﬁcation, though even with only second-order interactions, these models signiﬁcantly outperform single-layer models.

Original: While these results are encouraging, computing such kernels is prohibitively costly, and we found that applying the Nyström approach of Mairal (2016) to these
Revised: While these results are encouraging, computing such kernels is prohibitively costly, and we found that applying the Nyström approach of Mairal (2016) to these

Original: [Figure]
Revised: [Figure]

Revised: The plots illustrate that pooling is essential for reducing effective dimensionality.

Original: kernels with more layers or larger patches requires larger models than for the architecture of Tablefor a similar accuracy.
Revised: kernels with more layers or larger patches requires larger models than for the architecture of Tablefor a similar accuracy.

Original: Figure 3(left) shows learning curves for different architectures, with slightly better convergence rates for more expressive models involving higher-order kernels or more layers; this suggests that their approximation properties may be better suited for these datasets [Figure]
Revised: Figure 3(left) shows learning curves for different architectures, with slightly better convergence rates for more expressive models involving higher-order kernels or more layers; this suggests that their approximation properties may be better suited for these datasets.

Original: Role of pooling.
Revised: Role of pooling.

Original: Figure 3 shows the spectral decays of the empirical kernel matrix on
Revised: Figure 3 shows the spectral decays of the empirical kernel matrix on

Original: Cifar images, which may help assess the “effective dimensionality” of the data, and are related to generalization properties (Caponnetto & De Vito, 2007).
Revised: Cifar images, which may help assess the “effective dimensionality” of the data, and are related to generalization properties (Caponnetto & De Vito, 2007).

Original: While multi-layer architectures with pooling seem to provide comparable decays for various depths, removing pooling leads to signiﬁcantly slower decays, and hence much larger RKHSs.
Revised: While multi-layer architectures with pooling seem to provide comparable decays for various depths, removing pooling leads to signiﬁcantly slower decays, and hence much larger RKHSs.

Original: In particular, the “strided pooling” architecture ( i.e. , with Dirac pooling ﬁlters and downsampling) shown in Figure 3(right), which resembles the kernel considered in (Scetbon & Harchaoui, 2020), obtains less than 40% accuracy on 10k examples.
Revised: In particular, the “strided pooling” architecture ( i.e. , with Dirac pooling ﬁlters and downsampling) shown in Figure 3(right), which resembles the kernel considered in (Scetbon & Harchaoui, 2020), obtains less than 40% accuracy on 10k examples.

Original: This suggests that the regularization properties induced by pooling, studied in Section 3, are crucial for efﬁcient learning on these problems, as shown in Section 4.
Revised: This suggests that the regularization properties induced by pooling, studied in Section 3, are crucial for efﬁcient learning on these problems, as shown in Section 4.

Original: Appendix B provides more empirics on different pooling conﬁgurations.
Revised: Appendix B provides more empirics on different pooling conﬁgurations.

Original:  D ISCUSSION AND C ONCLUDING R EMARKS
Revised:  D ISCUSSION AND C ONCLUDING R EMARKS

Original: In this paper, we studied approximation and generalization properties of convolutional kernels, showing how multi-layer models with convolutional architectures may effectively break the curse of dimensionality on problems where the input consists of high-dimensional natural signals, by modeling localized functions on patches and interactions thereof.
Revised: In this paper, we studied approximation and generalization properties of convolutional kernels, showing how multi-layer models with convolutional architectures may effectively break the curse of dimensionality on problems where the input consists of high-dimensional natural signals, by modeling localized functions on patches and interactions thereof.

Original: We also show how pooling induces additional smoothness constraints on how interaction terms may or may not vary with global and relative spatial locations.
Revised: We also show how pooling induces additional smoothness constraints on how interaction terms may or may not vary with global and relative spatial locations.

Original: An important question for future work is how optimization of deep convolutional networks may further improve approximation properties compared to what is captured by the kernel regime presented here, for instance by selecting well-chosen convolution ﬁlters at the ﬁrst layer, or interaction patterns in subsequent layers, perhaps in a hierarchical manner.
Revised: An important question for future work is how optimization of deep convolutional networks may further improve approximation properties compared to what is captured by the kernel regime presented here, for instance by selecting well-chosen convolution ﬁlters at the ﬁrst layer, or interaction patterns in subsequent layers, perhaps in a hierarchical manner.

Original: κ conv pool
Revised: κ conv pool

Original: Test
Revised: Test

Original: [Table]
Revised: [Table]

Original: (10k)
Revised: [Table] Test

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (Exp,Exp) (3,5) (2,5)
Revised: (Exp,Exp) (3,5) (2,5)

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: % (Exp,Poly4) (3,5) (2,5) 81.3% 88.
Revised: % (Exp,Poly4) (3,5) (2,5) 81.

Original: [Table]
Revised: [Table]

Original: % (Exp,Poly3)

Original: [Table]
Revised: [Table]

Original: Myrtle10 Shankar et al.
Revised: Myrtle10 Shankar et al.

Original: [Table]
Revised: [Table]

Original: A F URTHER B ACKGROUND
Revised: A F URTHER B ACKGROUND

Original: This section provides further background on the problem of approximation of functions deﬁned on signals, as well as on the kernels considered in the paper.
Revised: This section provides further background on the problem of approximation of functions deﬁned on signals, as well as on the kernels considered in the paper.

Original: We begin by introducing and motivating the problem of learning functions deﬁned on signals such as images which captures tasks such as image classiﬁcation where deep convolutional networks are predominant.
Revised: We begin by introducing and motivating the problem of learning functions deﬁned on signals such as images, which captures tasks such as image classiﬁcation where deep convolutional networks are predominant.

Original: We then recall properties of dot-product kernels and kernel tensor products, which are key to our study of approximation.
Revised: We then recall properties of dot-product kernels and kernel tensor products, which are key to our study of approximation.

Original: A. N ATURAL S IGNALS AND C URSE OF D IMENSIONALITY
Revised: A. N ATURAL S IGNALS AND C URSE OF D IMENSIONALITY

Original: We consider learning problems consisting of labeled examples ( x, y ) ∼ ρ from a data distribution ρ , where x is a discrete signal x [ u ] with u ∈ Ω denoting the position ( e g. , pixel location in an image) in a domain Ω , x [ u ] ∈ R p ( e.g. , p = 3 for RGB pixels), and y ∈ R is a target label.
Revised: We consider learning problems consisting of labeled examples ( x, y ) ∼ ρ from a data distribution ρ , where x is a discrete signal x [ u ] with u ∈ Ω denoting the position ( e.g. , pixel location in an image) in a domain Ω , x [ u ] ∈ R p ( e.g. , p = 3 for RGB pixels), and y ∈ R is a target label.

Original: In a non-parametric setup, statistical learning may be framed as trying to approximate the regression function
Revised: In a non-parametric setup, statistical learning may be framed as trying to approximate the regression function

Original: [Equation]
Revised: [Equation]

Original: using samples from the data distribution ρ .
Revised: using samples from the data distribution ρ .

Original: If f ∗ is only assumed to be Lipschitz, learning requires a number of samples that scales exponentially in the dimension (see, e.g. , von Luxburg & Bousquet (2004); Wainwright (2019)), a phenomenon known as the curse of dimensionality .
Revised: If f ∗ is only assumed to be Lipschitz, learning requires a number of samples that scales exponentially in the dimension (see, e.g. , von Luxburg & Bousquet (2004); Wainwright (2019)), a phenomenon known as the curse of dimensionality .

Original: In the case of natural signals, the dimension d = p | Ω | scales with the size of the domain | Ω | ( e.g. , the number of pixels), which is typically very large and thus makes this intractable.
Revised: In the case of natural signals, the dimension d = p | Ω | scales with the size of the domain | Ω | ( e.g. , the number of pixels), which is typically very large and thus makes this intractable.

Original: One common way to alleviate this is to assume that f ∗ is smooth, however the order of smoothness typically needs to be of the order of the dimension in order for the problem to become tractable, which is a very strong assumption here when d is very large.
Revised: One common way to alleviate this is to assume that f ∗ is smooth, however the order of smoothness typically needs to be of the order of the dimension in order for the problem to become tractable, which is a very strong assumption here when d is very large.

Original: This highlights the need for more structured assumptions on f ∗ which may help overcome the curse of dimensionality.
Revised: This highlights the need for more structured assumptions on f ∗ which may help overcome the curse of dimensionality.

Original: Insufﬁciency of invariance and stability.
Revised: Insufﬁciency of invariance and stability.

Original: Two geometric properties that have been successful for studying the beneﬁts of convolutional architectures are (near-)translation invariance and stability to deformations.
Revised: Two geometric properties that have been successful for studying the beneﬁts of convolutional architectures are (near-)translation invariance and stability to deformations.

Original: Various works have shown that certain convolutional models f yield good invariance and stability Mallat (2012); Bruna & Mallat (2013); Bietti & Mairal (2019a), in the sense that when ˜ x is a translation or a small deformation of x , then | f (˜ x ) − f ( x ) | is small.
Revised: Various works have shown that certain convolutional models f yield good invariance and stability Mallat (2012); Bruna & Mallat (2013); Bietti & Mairal (2019a), in the sense that when ˜ x is a translation or a small deformation of x , then | f (˜ x ) − f ( x ) | is small.

Original: Nevertheless, one can show that for band-limited signals (such as discrete signals), (cid:107) ˜ x − x (cid:107) 2 can be controlled in a similar way (though with worse constants, see (Wiatowski & Bölcskei, 2018, Proposition 5)), so that
Revised: Nevertheless, one can show that for band-limited signals (such as discrete signals), (cid:107) ˜ x − x (cid:107) 2 can be controlled in a similar way (though with worse constants, see (Wiatowski & Bölcskei, 2018, Proposition 5)), so that

Original: Lipschitz functions on such signals obey such stability properties.
Revised: Lipschitz functions on such signals obey such stability properties.

Original: Thus, deformation stability is not a much stronger assumption than Lipschitzness, and is insufﬁcient by itself to escape the curse of dimensionality.
Revised: Thus, deformation stability is not a much stronger assumption than Lipschitzness, and is insufﬁcient by itself to escape the curse of dimensionality.

Original: Spatial localization.
Revised: Spatial localization.

Original: One successful strategy for learning image recognition models which predates deep learning is to rely on simple aggregations of local features.
Revised: One successful strategy for learning image recognition models which predates deep learning is to rely on simple aggregations of local features.

Original: These may be extracted using hand-crafted procedures (Lowe, 1999; Sánchez et al., 2013; Jégou et al., 2011), or using learned feature extractors, either through learned ﬁlters in the early layers of a CNN (Zeiler & Fergus, 2014), or other procedures ( e.g. , Thiry et al.
Revised: These may be extracted using hand-crafted procedures (Lowe, 1999; Sánchez et al., 2013; Jégou et al., 2011), or using learned feature extractors, either through learned ﬁlters in the early layers of a CNN (Zeiler & Fergus, 2014), or other procedures ( e.g. , Thiry et al.

Original: One simpliﬁed example that encodes such a prior is if the target function
Revised: One simpliﬁed example that encodes such a prior is if the target function

Original: f ∗ only depends on the input image through a localized part of the input such as a patch x u
Revised: f ∗ only depends on the input image through a localized part of the input such as a patch x u

Original: = ( x [ u + v ]) v ∈ S ∈ R p | S | , where S is a small box centered around 0 , that is, f ∗ ( x ) =
Revised: = ( x [ u + v ]) v ∈ S ∈ R p | S | , where S is a small box centered around 0 , that is, f ∗ ( x ) =

Original: g ∗ ( x u ) .
Revised: g ∗ ( x u ) .

Original: Then, if g ∗ is assumed to be Lipschitz, we would like a sample complexity that only scales exponentially in the dimension of a patch p | S | , which is much smaller than dimension of the entire image p | Ω | .
Revised: Then, if g ∗ is assumed to be Lipschitz, we would like a sample complexity that only scales exponentially in the dimension of a patch p | S | , which is much smaller than dimension of the entire image p | Ω | .

Original: This is indeed the case if we use a kernel deﬁned on such patches, such as
Revised: This is indeed the case if we use a kernel deﬁned on such patches, such as

Original: [Equation]
Revised: [Equation]

Original: where k is a “simple” kernel such as a dot-product kernel, as discussed in Appendix C.
Revised: where k is a “simple” kernel such as a dot-product kernel, as discussed in Appendix C.

Original: In contrast, if K is a dot-product kernel on the entire image, corresponding to an inﬁnite-width limit of a fullyconnected network, then approximation is more difﬁcult and is generally cursed by the full dimension (see Appendix C).
Revised: In contrast, if K is a dot-product kernel on the entire image, corresponding to an inﬁnite-width limit of a fullyconnected network, then approximation is more difﬁcult and is generally cursed by the full dimension (see Appendix C).

Original: While some models of wide fully-connected networks provide some adaptivity to low-dimensional structures such as the variables in a patch (Bach, 2017a), no tractable algorithms are currently known to achieve such behavior provably, and it is reasonable to instead encode such prior information in a convolutional architecture.
Revised: While some models of wide fully-connected networks provide some adaptivity to low-dimensional structures such as the variables in a patch (Bach, 2017a), no tractable algorithms are currently known to achieve such behavior provably, and it is reasonable to instead encode such prior information in a convolutional architecture.

Original: Modeling interactions.
Revised: Modeling interactions.

Original: Modeling interactions between elements of a system at different scales, possibly hierarchically, is important in physics and complex systems, in order to efﬁciently handle systems with large numbers of variables (Beylkin & Mohlenkamp, 2002; Hackbusch & Kühn, 2009).
Revised: Modeling interactions between elements of a system at different scales, possibly hierarchically, is important in physics and complex systems, in order to efﬁciently handle systems with large numbers of variables (Beylkin & Mohlenkamp, 2002; Hackbusch & Kühn, 2009).

Original: As an example, one may consider target functions f ∗ ( x ) that consist of interaction functions of the form g ( x p , x q ) , where p, q denote locations of the corresponding patches, and higher-order interactions may also be considered.
Revised: As an example, one may consider target functions f ∗ ( x ) that consist of interaction functions of the form g ( x p , x q ) , where p, q denote locations of the corresponding patches, and higher-order interactions may also be considered.

Original: In the context of image recognition, while functions of a single patch may capture local texture information such as edges or color, such an interaction function may also respond to speciﬁc spatial conﬁgurations of relevant patches, which could perhaps help identify properties related to the “shape” of an object, for instance.
Revised: In the context of image recognition, while functions of a single patch may capture local texture information such as edges or color, such an interaction function may also respond to speciﬁc spatial conﬁgurations of relevant patches, which could perhaps help identify properties related to the “shape” of an object, for instance.

Original: If such functions g are too general, then the curse of dimensionality may kick in again when one considers more than a handful of patches.
Revised: If such functions g are too general, then the curse of dimensionality may kick in again when one considers more than a handful of patches.

Original: Certain idealized models of approximation may model such interactions more efﬁciently through hierarchical compositions ( e.g. , Poggio et al.
Revised: Certain idealized models of approximation may model such interactions more efﬁciently through hierarchical compositions ( e.g. , Poggio et al.

Original: or tensor decompositions (Cohen &
Revised: or tensor decompositions (Cohen &

Original: Shashua, 2016; 2017), though no tractable algorithms are known to ﬁnd such models.
Revised: Shashua, 2016; 2017), though no tractable algorithms are known to ﬁnd such models.

Original: In this work, we tackle this in a tractable way using multi-layer convolutional kernels.
Revised: In this work, we tackle this in a tractable way using multi-layer convolutional kernels.

Original: We show that they can model interactions through kernel tensor products, which deﬁne functional spaces that are typically much smaller and more structured than for a generic kernel on the full vector ( x p , x q ) .
Revised: We show that they can model interactions through kernel tensor products, which deﬁne functional spaces that are typically much smaller and more structured than for a generic kernel on the full vector ( x p , x q ) .

Original: A. D OT -P RODUCT K ERNELS AND THEIR T ENSOR P RODUCTS
Revised: A. D OT -P RODUCT K ERNELS AND THEIR T ENSOR P RODUCTS

Original: In this section, we review some properties of dot-product kernels, their induced RKHS and regularization properties.
Revised: In this section, we review some properties of dot-product kernels, their induced RKHS and regularization properties.

Original: We then recall the notion of tensor product of kernels, which allows us to describe the RKHS of products of kernels in terms of that of individual kernels.
Revised: We then recall the notion of tensor product of kernels, which allows us to describe the RKHS of products of kernels in terms of that of individual kernels.

Original: Dot-product kernels.
Revised: Dot-product kernels.

Original: The rotation-invariance of dot-product kernels provides a natural description of their RKHS in terms of harmonic decompositions of functions on the sphere using spherical harmonics (Smola et al., 2001; Bach, 2017a).
Revised: The rotation-invariance of dot-product kernels provides a natural description of their RKHS in terms of harmonic decompositions of functions on the sphere using spherical harmonics (Smola et al., 2001; Bach, 2017a).

Original: This leads to natural connections with regularity properties of functions deﬁned on the sphere.
Revised: This leads to natural connections with regularity properties of functions deﬁned on the sphere.

Original: For instance, if the kernel integral operator on L 2 ( S d − 1 ) has a polynomially decaying spectral decay, as is the case for kernels arising from the ReLU activation (Bach, 2017a; Bietti & Mairal, 2019b), then the RKHS contains functions g ∈ L 2 ( S [Equation] 1 ) with an RKHS norm equivalent to
Revised: For instance, if the kernel integral operator on L 2 ( S d − 1 ) has a polynomially decaying spectral decay, as is the case for kernels arising from the ReLU activation (Bach, 2017a; Bietti & Mairal, 2019b), then the RKHS contains functions g ∈ L 2 ( S d − 1 ) with an RKHS norm equivalent to

Original: [Equation]
Revised: [Equation]

Original:  for some β that depends on the decay exponent and must be larger than ( d − 1) / 2 , with ∆ S [Equation] the Laplace-Beltrami operator on the sphere.
Revised: ) for some β that depends on the decay exponent and must be larger than ( d − 1) / 2 , with ∆ S d −the Laplace-Beltrami operator on the sphere.

Original: This resembles a Sobolev norm of order β , and the
Revised: This resembles a Sobolev norm of order β , and the

Original: RKHS contains functions with bounded derivatives up to order β .
Revised: RKHS contains functions with bounded derivatives up to order β .

Original: When d is small ( e.g. , at the ﬁrst layer with small images patches), the space contains functions that need not be too regular, and may thus be quite discriminative, while for large d ( e.g. , for a fully-connected network), the functions must be highly smooth in order to be in the RKHS, and large norms are necessary to approach 
Revised: When d is small ( e.g. , at the ﬁrst layer with small images patches), the space contains functions that need not be too regular, and may thus be quite discriminative, while for large d ( e.g. , for a fully-connected network), the functions must be highly smooth in order to be in the RKHS, and large norms are necessary to approach non-smooth functions.

Original: [Table]
Revised: For kernels with decays faster than polynomial, such as the Gaussian kernel, the RKHS contains smooth functions, but may still provide good approximation to non-smooth functions, particularly with small d and when using small bandwidth parameters. The homogeneous case (2) leads to functions f ( x ) = (cid:107) x (cid:107) g

Original: [Table] functions kernel
Revised: ( x (cid:107) x (cid:107) ) with g deﬁned on the sphere, with a norm given by the same penalty (11) on the function g (Bietti & Mairal, 2019b).

Original: Kernel tensor products.
Revised: Kernel tensor products.

Original: For more than one layer, the convolutional kernels we study in Sectioncan be expressed in terms of products of kernels on patches, of the form
Revised: For more than one layer, the convolutional kernels we study in Sectioncan be expressed in terms of products of kernels on patches, of the form

Original: [Equation]
Revised: [Equation]

Original: where x 1 , . .
Revised: where x 1 , . .

Original: x m , x (cid:48) 1 , . .
Revised: x m , x (cid:48) 1 , . .

Original: , x (cid:48) m ∈ R
Revised: , x (cid:48) m ∈ R

Original: d are patches which may come from different signal locations.
Revised: d are patches which may come from different signal locations.

Original: If ϕ : R d → H is the feature map into the RKHS H of k , thenψ ( x 1 , . . .
Revised: If ϕ : R d → H is the feature map into the RKHS H of k , then

Original: , x m ) =
Revised: [Equation]

Original: ϕ ( x 1 ) ⊗ · · · ⊗ ϕ ( x m ) is a feature map for K , and the corresponding RKHS, denoted H ⊗ m=
Revised: is a feature map for K , and the corresponding RKHS, denoted H ⊗ m

Original:  H ⊗ · · · ⊗ H , contains all functionsf ( x 1 , . . .
Revised: = H ⊗ · · · ⊗ H , contains all functions

Original: , x m ) = n (cid:88) i = g i, 1 ( x 1 ) . . .
Revised: [Equation]

Original: g i,m ( x m ) , for some n , with g i,m ∈ H for i ∈ [ n ] and j ∈ [ m ] (see, e.g. , (Wainwright, 2019, Section 12.4.2) for a precise construction).
Revised: , for some n , with g i,m ∈ H for i ∈ [ n ] and j ∈ [ m ] (see, e.g. , (Wainwright, 2019, Section 12.4.2) for a precise construction).

Original: The resulting RKHS is often much smaller than for a more generic kernel on R d × m ; for instance, if H is a Sobolev space of order β in dimension d , then H ⊗ m is much smaller than the Sobolev space of order β in d × m dimensions, and corresponds to stronger, mixed regularity conditions (see, e.g. , Bach, 2017b; Sickel & Ullrich, 2009, for the d = 1 case).
Revised: The resulting RKHS is often much smaller than for a more generic kernel on R d × m ; for instance, if H is a Sobolev space of order β in dimension d , then H ⊗ m is much smaller than the Sobolev space of order β in d × m dimensions, and corresponds to stronger, mixed regularity conditions (see, e.g. , Bach, 2017b; Sickel & Ullrich, 2009, for the d = 1 case).

Original: This can yield improved generalization properties if the target function has such a structure (Lin, 2000).
Revised: This can yield improved generalization properties if the target function has such a structure (Lin, 2000).

Original: Kernels of the form (12) and sums of such kernels have been useful tools for avoiding the curse of dimensionality by encoding interactions between variables that are relevant to the problem at hand (Wahba, 1990, Chapter 10).
Revised: Kernels of the form (12) and sums of such kernels have been useful tools for avoiding the curse of dimensionality by encoding interactions between variables that are relevant to the problem at hand (Wahba, 1990, Chapter 10).

Original: In what follows, we show how patch extraction and pooling operations shape the properties of such interactions between patches in convolutional kernels through additional spatial regularities.
Revised: In what follows, we show how patch extraction and pooling operations shape the properties of such interactions between patches in convolutional kernels through additional spatial regularities.

Original: B A DDITIONAL E XPERIMENTS
Revised: B A DDITIONAL E XPERIMENTS

Original: In this section, we provide additional experiments to those presented in Section 5, using different patch kernels, patch sizes, pooling ﬁlters, preprocessings, and datasets.
Revised: In this section, we provide additional experiments to those presented in Section 5, using different patch kernels, patch sizes, pooling ﬁlters, preprocessings, and datasets.

Original: Three-layer architectures with different patch kernels.
Revised: Three-layer architectures with different patch kernels.

Original: Table 3 provides more results on 3-layer architectures compared to Table 2, including different changes in the degrees of polynomial kernels at the second and third layer.
Revised: Table 3 provides more results on 3-layer architectures compared to Table 2, including different changes in the degrees of polynomial kernels at the second and third layer.

Original: In particular we see that the architecture with degree-2 kernels at both layers, which captures interactions of order 4, also outperforms the simpler ones using degreekernels at either layer, suggesting that a deeper architecture may better model relevant interactions terms on this problem. 
Revised: In particular we see that the architecture with degree-2 kernels at both layers, which captures interactions of order 4, also outperforms the simpler ones using degreekernels at either layer, suggesting that a deeper architecture may better model relevant interactions terms on this problem. [Table]

Original: [Table]
Revised: [Table]

Original: Varying the second layer patch size.
Revised: Varying the second layer patch size.

Original: Table 4 shows the variations in test performance when changing the size of the second patches at the second layer.
Revised: Table 4 shows the variations in test performance when changing the size of the second patches at the second layer.

Original: We see that intermediate sizes between 3x3 and 9x9 work best, but that performance degrades when using patches that are too large or too small.
Revised: We see that intermediate sizes between 3x3 and 9x9 work best, but that performance degrades when using patches that are too large or too small.

Original: For very large patches, this may be due to the large variance in (10), or perhaps instability (Bietti & Mairal, 2019a).
Revised: For very large patches, this may be due to the large variance in (10), or perhaps instability (Bietti & Mairal, 2019a).

Original: For | S 2 | = 1x1, note that while pooling after the ﬁrst layer allows even 1x1 patches to capture interactions across different input image patches, these may be limited to short range interactions when the pooling ﬁlter is localized (see Proposition 2), which may limit the expressivity of the 
Revised: For | S 2 | = 1x1, note that while pooling after the ﬁrst layer allows even 1x1 patches to capture interactions across different input image patches, these may be limited to short range interactions when the pooling ﬁlter is localized (see Proposition 2), which may limit the expressivity of the model.

Original: [Table]
Revised: [Table]

Original: Arc-cosine kernel.
Revised: Arc-cosine kernel.

Original: In Table 5, we consider 2-layer convolutional kernels with a similar architecture to those considered in Table 1, but where we use arc-cosine kernels arising from ReLU activations instead of the exponential kernel used in Section 5, given by
Revised: In Table 5, we consider 2-layer convolutional kernels with a similar architecture to those considered in Table 1, but where we use arc-cosine kernels arising from ReLU activations instead of the exponential kernel used in Section 5, given by

Original: [Equation]
Revised: [Equation]

Original: The obtained convolutional kernel then corresponds to the conjugate kernel or NNGP kernel arising from an inﬁnite-width convolutional network with the ReLU activation (Daniely et al., 2016; GarrigaAlonso et al., 2019; Novak et al., 2019).
Revised: The obtained convolutional kernel then corresponds to the conjugate kernel or NNGP kernel arising from an inﬁnite-width convolutional network with the ReLU activation (Daniely et al., 2016; GarrigaAlonso et al., 2019; Novak et al., 2019).

Original: We may also consider the neural tangent kernel (NTK) for the same architecture, which additionally involves arc-cosine kernels of degree 0, which correspond to random feature kernels for step activations
Revised: We may also consider the neural tangent kernel (NTK) for the same architecture, which additionally involves arc-cosine kernels of degree 0, which correspond to random feature kernels for step activations

Original: u (cid:55)→ 1 { u ≥ 0 } .
Revised: u (cid:55)→ 1 { u ≥ 0 } .

Original: We ﬁnd that the NTK performs slightly better than the conjugate kernel, but both kernels achieve lower accuracy compared to the Exponential kernel shown in Table 1.
Revised: We ﬁnd that the NTK performs slightly better than the conjugate kernel, but both kernels achieve lower accuracy compared to the Exponential kernel shown in Table 1.

Original: Nevertheless, we observe a similar pattern regarding the use of polynomial kernels at the second layer, namely, the drop in accuracy is much smaller when using a quadratic kernel compared to a linear kernel, suggesting that non-linear kernels on top of the ﬁrst layer, and the interactions they may capture, are crucial on this dataset for good accuracy.
Revised: Nevertheless, we observe a similar pattern regarding the use of polynomial kernels at the second layer, namely, the drop in accuracy is much smaller when using a quadratic kernel compared to a linear kernel, suggesting that non-linear kernels on top of the ﬁrst layer, and the interactions they may capture, are crucial on this dataset for good accuracy.

Original: [Table]
Revised: [Table]

Original: One-layer architectures and larger initial patches.
Revised: One-layer architectures and larger initial patches.

Original: Table 6 shows the accuracy for one-layer convolutional kernels with 6x6 patches 3 and various pooling sizes, with a highest accuracy of 75.8% for a pooling size of 8 .
Revised: Table 6 shows the accuracy for one-layer convolutional kernels with 6x6 patches 3 and various pooling sizes, with a highest accuracy of 75.8% for a pooling size of 8 .

Original: While this improves on the accuracy obtained with 3x3 patches (slightly above 74% for the architectures in Tables 1 and 3 with a single non-linear kernel at the ﬁrst layer), these accuracies remain much lower than those achieved by two-layer architectures with even quadratic kernels at the second layer.
Revised: While this improves on the accuracy obtained with 3x3 patches (slightly above 74% for the architectures in Tables 1 and 3 with a single non-linear kernel at the ﬁrst layer), these accuracies remain much lower than those achieved by two-layer architectures with even quadratic kernels at the second layer.

Original: While using larger patches may allow capturing patterns that are less localized compared to small 3x3 patches, the neighborhoods that they model need to remain small in order to avoid the curse of dimensionality when using dot-product kernels, as discussed in
Revised: While using larger patches may allow capturing patterns that are less localized compared to small 3x3 patches, the neighborhoods that they model need to remain small in order to avoid the curse of dimensionality when using dot-product kernels, as discussed in

Original: Section 2.
Revised: Section 2.

Original: Instead, the multi-layer architecture may model information at larger scales with a much milder dependence on the size of the neighborhood, thanks to the structure imposed by tensor product kernels (see Section A.2) and the additional regularities induced by pooling.
Revised: Instead, the multi-layer architecture may model information at larger scales with a much milder dependence on the size of the neighborhood, thanks to the structure imposed by tensor product kernels (see Section A.2) and the additional regularities induced by pooling.

Original: We also found that larger patches at the ﬁrst layer may hurt performance in multi-layer models: when considering the architecture of Table 1 with exponential kernels, using 5x5 patches instead of 3x3 at the ﬁrst layer yields an accuracy of 79.6% instead of 80.5% on Cifar10 when training on the same 10k images.
Revised: We also found that larger patches at the ﬁrst layer may hurt performance in multi-layer models: when considering the architecture of Table 1 with exponential kernels, using 5x5 patches instead of 3x3 at the ﬁrst layer yields an accuracy of 79.6% instead of 80.5% on Cifar10 when training on the same 10k images.

Original: This again reﬂects the beneﬁts of using small patches at the ﬁrst layer for allowing better approximation on small neighborhoods, while modeling larger scales using interaction models according to the structure of the architecture.
Revised: This again reﬂects the beneﬁts of using small patches at the ﬁrst layer for allowing better approximation on small neighborhoods, while modeling larger scales using interaction models according to the structure of the architecture.

Original: We note nevertheless that for standard deep networks, larger patches are often used at the ﬁrst layer ( e.g. , He et al., 2016), as the feature selection capabilities of SGD may alleviate the dependence on dimension, e.g. , by ﬁnding Gabor-like ﬁlters.
Revised: We note nevertheless that for standard deep networks, larger patches are often used at the ﬁrst layer ( e.g. , He et al., 2016), as the feature selection capabilities of SGD may alleviate the dependence on dimension, e.g. , by ﬁnding Gabor-like ﬁlters.

Original: Gaussian vs average pooling.
Revised: Gaussian vs average pooling.

Original: Table 7 shows the differences in performance between two or three layer architectures considered in Table 2, when Gaussian pooling ﬁlters are replaced by average pooling ﬁlters.
Revised: Table 7 shows the differences in performance between two or three layer architectures considered in Table 2, when Gaussian pooling ﬁlters are replaced by average pooling ﬁlters.

Original: For both architectures considered, average pooling leads to a signiﬁcant performance drop.
Revised: For both architectures considered, average pooling leads to a signiﬁcant performance drop.

Original: This suggests that one may need deeper architectures in order for such average pooling ﬁlters to work well, as in (Shankar et al., 2020), either with multiple 3x3 convolutional layers before applyingpooling, or by applying multiple average pooling layers in a row as in certain Myrtle kernels.
Revised: This suggests that one may need deeper architectures in order for such average pooling ﬁlters to work well, as in (Shankar et al., 2020) [Table] either with multiple 3x3 convolutional layers before applying pooling [Table]

Revised: or by applying multiple average pooling layers in a row as in certain Myrtle kernels.

Original: Note that iterating multiple average pooling layers in a row is equivalent to using a larger and more smooth pooling ﬁlter (with one more order of smoothness at each layer), which may then be more comparable to our Gaussian pooling ﬁlters
Revised: Note that iterating multiple average pooling layers in a row is equivalent to using a larger and more smooth pooling ﬁlter (with one more order of smoothness at each layer), which may then be more comparable to our Gaussian pooling ﬁlters

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: SVHN dataset.
Revised: SVHN dataset.

Original: We now consider the SVHN dataset, which consists of 32x32 images of digits from
Revised: We now consider the SVHN dataset, which consists of 32x32 images of digits from

Original: Google Street View images, 73 257 for training and 26 032 for testing.
Revised: Google Street View images, 73 257 for training and 26 032 for testing.

Original: Due to the larger dataset size, we only consider the kernel approximation approach of Mairal (2016) based on the Nyström method, which projects the patch kernel feature maps at each layer to ﬁnite-dimensional subspaces generated by a set of anchor points (playing the role of convolutional ﬁlters), themselves computed via a
Revised: Due to the larger dataset size, we only consider the kernel approximation approach of Mairal (2016) based on the Nyström method, which projects the patch kernel feature maps at each layer to ﬁnite-dimensional subspaces generated by a set of anchor points (playing the role of convolutional ﬁlters), themselves computed via a

Original: K-means clustering of patches.
Revised: K-means clustering of patches.

Original: 4 We train one-versus-all classiﬁers on the resulting ﬁnite-dimensional representations using regularized ERM with the squared hinge loss [Table] and simply report the best test accuracy over a logarithmic grid of choices for the regularization parameter, ignoring model selection issues in order to assess approximation properties.
Revised: 4 We train one-versus-all classiﬁers on the resulting ﬁnite-dimensional representations using regularized ERM with the squared hinge loss, and simply report the best test accuracy over a logarithmic grid of choices for the regularization parameter, ignoring model selection issues in order to assess approximation properties.

Original: We use the same ZCA preprocessing as on Cifar and the same architecture as in Table 1, with a relatively small number of ﬁlters (256 at the ﬁrst layer, 4096 at the second layer, leading to representations of dimension 65 536), noting that the accuracy can further improve when increasing this number.
Revised: We use the same ZCA preprocessing as on Cifar and the same architecture as in Table 1, with a relatively small number of ﬁlters (256 at the ﬁrst layer, 4096 at the second layer, leading to representations of dimension 65 536), noting that the accuracy can further improve when increasing this number.

Original: Our observations are similar to those for the Cifar dataset: using a degree-3 polynomial kernel at the second layer reaches very similar accuracy to the exponential kernel; using a degree-2 polynomial leads to a slight drop, but a smaller drop than when making this same change at the ﬁrst layer; using a linear kernel at the second layer leads to a much larger drop.
Revised: Our observations are similar to those for the Cifar dataset: using a degree-3 polynomial kernel at the second layer reaches very similar accuracy to the exponential kernel; using a degree-2 polynomial leads to a slight drop, but a smaller drop than when making this same change at the ﬁrst layer; using a linear kernel at the second layer leads to a much larger drop.

Original: This again highlights the importance of using non-linear kernels on top of the ﬁrst layer in order to capture interactions at larger scales than the scale of a single patch.
Revised: This again highlights the importance of using non-linear kernels on top of the ﬁrst layer in order to capture interactions at larger scales than the scale of a single patch.

Original: Local versus global whitening.
Revised: Local versus global whitening.

Original: Recall that our pre-processing is based on a patch-level whitening or ZCA on each image, following Mairal (2016).
Revised: Recall that our pre-processing is based on a patch-level whitening or ZCA on each image, following Mairal (2016).

Original: In practice, this is achieved by whitening extracted patches from each image, and reconstructing the image from whitened patches via averaging.
Revised: In practice, this is achieved by whitening extracted patches from each image, and reconstructing the image from whitened patches via averaging.

Original: In contrast, other approaches use global whitening of the entire image Lee et al.
Revised: In contrast, other approaches use global whitening of the entire image Lee et al.

Original: ; Shankar et al.
Revised: ; Shankar et al.

Original: For the 2-layer model shown in Table 2 with 5x5 patches at the second layer, we found global
Revised: For the 2-layer model shown in Table 2 with 5x5 patches at the second layer, we found global

Original: ZCA to provide signiﬁcantly worse performance, with a drop from 88.3% to about 80%.
Revised: ZCA to provide signiﬁcantly worse performance, with a drop from 88.3% to about 80%.

Original: Finite networks and comparison to Shankar et al.
Revised: Finite networks and comparison to Shankar et al.

Original: The work Shankar et al.
Revised: The work Shankar et al.

Original: (2020) introduces Myrtle kernels but also consider similar architectures for usual CNNs with ﬁnite-width, trained with stochastic gradient descent.
Revised: (2020) introduces Myrtle kernels but also consider similar architectures for usual CNNs with ﬁnite-width, trained with stochastic gradient descent.

Original: Obtaining competitive architectures for the ﬁnite-width case is not the goal of our work, which focuses on good architectures for the kernel setup, yet it remains interesting to consider this question.
Revised: Obtaining competitive architectures for the ﬁnite-width case is not the goal of our work, which focuses on good architectures for the kernel setup, yet it remains interesting to consider this question.

Original: In the case of Shankar et al.
Revised: In the case of Shankar et al.

Original: (2020), training the ﬁnite-width networks yields better accuracy compared to their “inﬁnite-width” kernel counterparts, a commonly observed phenomenon which may be due to better “adaptivity” of optimization algorithms compared to kernel methods, which have a ﬁxed representation and thus may not learn representations adapted to the data (see, e.g. , Allen-Zhu & Li, 2020; Bach, 2017a;
Revised: (2020), training the ﬁnite-width networks yields better accuracy compared to their “inﬁnite-width” kernel counterparts, a commonly observed phenomenon which may be due to better “adaptivity” of optimization algorithms compared to kernel methods, which have a ﬁxed representation and thus may not learn representations adapted to the data (see, e.g. , Allen-Zhu & Li, 2020; Bach, 2017a;

Original: Chizat et al., 2019).
Revised: Chizat et al., 2019).

Original: Nevertheless, we found that for the two-layer architecture considered in Table 1, which has many fewer layers compared to the Myrtle architectures of Shankar et al.
Revised: Nevertheless, we found that for the two-layer architecture considered in Table 1, which has many fewer layers compared to the Myrtle architectures of Shankar et al.

Original: (2020), using a ﬁnite-width ReLU network yields poorer performance compared to the kernel (around 83% at best, compared to 87.9%).
Revised: (2020), using a ﬁnite-width ReLU network yields poorer performance compared to the kernel (around 83% at best, compared to 87.9%).

Original: This may suggest that for convolutional networks, deeper networks may have additional advantages when using optimization algorithms, in terms of adapting to possibly relevant structure of the problem, such as hierarchical representations (see, e.g. , Allen-Zhu & Li (2020);
Revised: This may suggest that for convolutional networks, deeper networks may have additional advantages when using optimization algorithms, in terms of adapting to possibly relevant structure of the problem, such as hierarchical representations (see, e.g. , Allen-Zhu & Li (2020);

Original: Chen et al.
Revised: Chen et al.

Original: ; Poggio et al.
Revised: ; Poggio et al.

Original: (2017) for theoretical justiﬁcations of the beneﬁts of depth in non-kernel regimes).
Revised: (2017) for theoretical justiﬁcations of the beneﬁts of depth in non-kernel regimes).

Original: C C OMPLEXITY OF S PATIALLY L OCALIZED F UNCTIONS
Revised: C C OMPLEXITY OF S PATIALLY L OCALIZED F UNCTIONS

Original: In this section, we brieﬂy elaborate on our discussion in Section A.1 on how simple convolutional structure may improve complexity when target functions are spatially localized.
Revised: In this section, we brieﬂy elaborate on our discussion in Section A.1 on how simple convolutional structure may improve complexity when target functions are spatially localized.

Original: We assume f ∗ ( x ) =
Revised: We assume f ∗ ( x ) =

Original: g ∗ ( x u ) with x u = ( x [ u + v ]) v ∈ S ∈ R p | S | a patch of size | S | , where g ∗ is a Lipschitz function.
Revised: g ∗ ( x u ) with x u = ( x [ u + v ]) v ∈ S ∈ R p | S | a patch of size | S | , where g ∗ is a Lipschitz function.

Original: If we deﬁne the kernel K u ( x, x (cid:48) )
Revised: If we deﬁne the kernel K u ( x, x (cid:48) )

Original: =
Revised: =

Original: k ( x u , x (cid:48) u ) , where k is a dot-product kernel arising from a one-hidden layer network with positively-homogeneous activation such as the ReLU, and further assume patches to be bounded and g ∗ to be bounded, then the uniform approximation error bound of Bach (2017a, Proposition 6) together with a simple O (1 / √ n ) Rademacher complexity bound on estimation error shows that we may achieve a generalization bound with a rate that only depends on the patch dimension p | S | rather than p | Ω | in this setup ( i.e. , a sample complexity that is exponential in p | S | , which is much smaller than p | Ω | ).
Revised: k ( x u , x (cid:48) u ) , where k is a dot-product kernel arising from a one-hidden layer network with positively-homogeneous activation such as the ReLU, and further assume patches to be bounded and g ∗ to be bounded, then the uniform approximation error bound of Bach (2017a, Proposition 6) together with a simple O (1 / √ n ) Rademacher complexity bound on estimation error shows that we may achieve a generalization bound with a rate that only depends on the patch dimension p | S | rather than p | Ω | in this setup ( i.e. , a sample complexity that is exponential in p | S | , which is much smaller than p | Ω | ).

Original: If we consider the kernel K ( x, x (cid:48) )
Revised: If we consider the kernel K ( x, x (cid:48) )

Original: = (cid:80) u ∈
Revised: = (cid:80) u ∈

Original: Ω
Revised: Ω

Original: k
Revised: k

Original: ( x u , x (cid:48) u ) , the RKHS contains all functions in the
Revised: ( x u , x (cid:48) u ) , the RKHS contains all functions in the

Original: RKHS of K u for all u ∈ Ω , with the same norm (this may be seen as an application of Theoremwith a feature map given by concatenating the kernel maps of each K u ), so that we may achieve the same approximation error as above, and thus a similar generalization bound that is not cursed by dimension.
Revised: RKHS of K u for all u ∈ Ω , with the same norm (this may be seen as an application of Theoremwith a feature map given by concatenating the kernel maps of each K u ), so that we may achieve the same approximation error as above, and thus a similar generalization bound that is not cursed by dimension.

Original: This kernel also allows us to obtain similar generalization guarantees when f ∗ consists of linear combinations of such spatially localized functions on different patches within the image.
Revised: This kernel also allows us to obtain similar generalization guarantees when f ∗ consists of linear combinations of such spatially localized functions on different patches within the image.

Original: In contrast, when using a similar dot-product kernel on the full signal, corresponding to using a fully-connected network in a kernel regime, one may construct functions f ∗ ( x ) =
Revised: In contrast, when using a similar dot-product kernel on the full signal, corresponding to using a fully-connected network in a kernel regime, one may construct functions f ∗ ( x ) =

Original: g ∗ ( x u ) with g ∗
Revised: g ∗ ( x u ) with g ∗

Original: Lipschitz where an RKHS norm that is exponentially large in the (full) dimension p | Ω | is needed for a small approximation error (see Bach, 2017a, Appendix D.5).
Revised: Lipschitz where an RKHS norm that is exponentially large in the (full) dimension p | Ω | is needed for a small approximation error (see Bach, 2017a, Appendix D.5).

Original: Related to this, Malach & Shalev-Shwartz (2021) show a separation in the different setting of learning certain parity functions on the hypercube using gradient methods; their upper bound for convolutional networks is based on a similar kernel regime as above.
Revised: Related to this, Malach & Shalev-Shwartz (2021) show a separation in the different setting of learning certain parity functions on the hypercube using gradient methods; their upper bound for convolutional networks is based on a similar kernel regime as above.

Original: We note that kernels that exploit such a localized structure have also been considered in the context of structured prediction for improved statistical guarantees (Ciliberto et al., 2019).
Revised: We note that kernels that exploit such a localized structure have also been considered in the context of structured prediction for improved statistical guarantees (Ciliberto et al., 2019).

Original: D E XTENSIONS TO M ORE L AYERS
Revised: D E XTENSIONS TO M ORE L AYERS

Original: In this section, we study the RKHS for convolutional kernels with more than 2 convolutional layers, by considering the simple example of a 3-layer convolutional kernel K 3 deﬁned by the feature map
Revised: In this section, we study the RKHS for convolutional kernels with more than 2 convolutional layers, by considering the simple example of a 3-layer convolutional kernel K 3 deﬁned by the feature map

Original: [Equation]
Revised: [Equation]

Original: with quadratic kernels at the second and third layer, i.e. ,k 2 ( z, z (cid:48) )
Revised: with quadratic kernels at the second and third layer, i.e. ,

Original: = ( (cid:104) z, z (cid:48) (cid:105) )
Revised: [Equation]

Original: 2 and k 3 ( z, z (cid:48) )
Revised: and k 3 ( z, z (cid:48) )

Original: =
Revised: =

Original: ( (cid:104) z, z (cid:48) (cid:105) )
Revised: ( (cid:104) z, z (cid:48) (cid:105) )

Original: By isomorphism, we may consider the sequence of Hilbert spaces H (cid:96) to be H 

Revised: By isomorphism, we may consider the sequence of Hilbert spaces H (cid:96) to be H

Original: = H ,
Revised: = H ,

Original: H 

Revised: H

Original: = ( H ⊗ H ) | S 2 |×| S 2 | , and H 

Revised: = ( H ⊗ H ) | S 2 |×| S 2 | , and H

Original: =
Revised: =

Original: ( H ⊗ 4 ) ( | S 3 |×| S 2 |×| S 2 | ) 2 .
Revised: ( H ⊗ 4 ) ( | S 3 |×| S 2 |×| S 2 | ) 2 .

Original: For some domain Ω , we deﬁne the operators diag 2 : L 2 (Ω 4 )
Revised: For some domain Ω , we deﬁne the operators diag 2 : L 2 (Ω 4 )

Original: → L 2 (Ω 2 ) and its adjoint diag 2 : L 2 (Ω 2 ) →
Revised: → L 2 (Ω 2 ) and its adjoint diag 2 : L 2 (Ω 2 ) →

Original: L 2 (Ω 4 ) bydiag 2 ( M )[ u, v ] = M [ u, u, v, v ] for M ∈ L 2 (Ω 4 ) diag 2 ( M )[ u 1 , u 2 , u 3 , u 4 ] = [Equation] { u
Revised: L 2 (Ω 4 ) by

Original: [Equation]
Revised: [Equation]

Original: = u [Equation] [Equation] = u
Revised: diag

Original: [Equation]

Original: } M [ u 1 , u

Original: [Equation]
Revised: [Equation]

Original: ] for M ∈ L 2 (Ω 2 ) .

Original: We may then describe the RKHS as follows.
Revised: We may then describe the RKHS as follows.

Original: Proposition 5 (RKHS of 3-layer CKN with quadratic k 2 / 3 ) .
Revised: Proposition 5 (RKHS of 3-layer CKN with quadratic k 2 / 3 ) .

Original: The RKHS of K 3 when k 2 and k 3 are quadratic kernels ( (cid:104)· , ·(cid:105) ) 2 consists of functions of the form
Revised: The RKHS of K 3 when k 2 and k 3 are quadratic kernels ( (cid:104)· , ·(cid:105) ) 2 consists of functions of the form

Original: [Equation]
Revised: [Equation]

Original: where G α ∈ L 2 (Ω 4 , H ⊗ 4 )
Revised: where G α ∈ L 2 (Ω 4 , H ⊗ 4 )

Original: obeys the constraint
Revised: obeys the constraint

Original: G

Original: [Equation]

Original: Range

Original: [Equation]
Revised: [Equation]

Original: , (14) where the linear operator E α : L 2 (Ω 3 )
Revised: (14) where the linear operator E α : L 2 (Ω 3 )

Original: → L 2 (Ω 4 ) for α = ( p, q, r, p (cid:48) , q (cid:48) , r (cid:48) ) (with p, p (cid:48) ∈ Sand q, r, q (cid:48) , r (cid:48)
Revised: → L 2 (Ω 4 ) for α = ( p, q, r, p (cid:48) , q (cid:48) , r (cid:48) ) (with p, p (cid:48) ∈ Sand q, r, q (cid:48) , r (cid:48)

Original: ∈ S 2 ) is deﬁned by
Revised: ∈ S 2 ) is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: The operators A 1 ,α and A 2 ,α denote:
Revised: The operators A 1 ,α and A 2 ,α denote:

Original: [Equation]
Revised: [Equation]

Original: The squared RKHS norm (cid:107) f (cid:107) 2 H K 3 is then equal to the minimum over decompositions (13) of the quantity
Revised: The squared RKHS norm (cid:107) f (cid:107) 2 H K 3 is then equal to the minimum over decompositions (13) of the quantity

Original: [Equation]
Revised: [Equation]

Original: The constraint (14) and penalty (15) resemble the corresponding constraint/penalty in the twolayer case for an order-4 polynomial kernel at the second layer, but provide more structure on the interactions, using a multi-scale structure that may model interactions between certain pairs of patches ( ( x u 1 , x u 2 ) and ( x u 3 , x u 4 ) in (13)) more strongly than those between all four patches.
Revised: The constraint (14) and penalty (15) resemble the corresponding constraint/penalty in the twolayer case for an order-4 polynomial kernel at the second layer, but provide more structure on the interactions, using a multi-scale structure that may model interactions between certain pairs of patches ( ( x u 1 , x u 2 ) and ( x u 3 , x u 4 ) in (13)) more strongly than those between all four patches.

Original: In addition to localizing the interactions G α around certain diagonals, the kernel also promotes spatial regularities: assuming that the spatial bandwidths of A (cid:96) increase with (cid:96) , the functions ( u, v, w 1 , w 2 ) (cid:55)→ G α [ u, u + w 1 , u + v, u + v + w 2 ] may vary quickly with w 1 or w 2 (distances between patches in each of the two pairs), but should vary more slowly with v (distance between the two pairs) and even more slowly with u (a global position).
Revised: In addition to localizing the interactions G α around certain diagonals, the kernel also promotes spatial regularities: assuming that the spatial bandwidths of A (cid:96) increase with (cid:96) , the functions ( u, v, w 1 , w 2 ) (cid:55)→ G α [ u, u + w 1 , u + v, u + v + w 2 ] may vary quickly with w 1 or w 2 (distances between patches in each of the two pairs), but should vary more slowly with v (distance between the two pairs) and even more slowly with u (a global position).

Original: E P ROOFS
Revised: E P ROOFS

Original: We recall the following result about reproducing kernel Hilbert spaces, which characterizes the RKHS of kernels deﬁned by explicit Hilbert space features maps (see, e.g. , Saitoh, 1997, §2.1).
Revised: We recall the following result about reproducing kernel Hilbert spaces, which characterizes the RKHS of kernels deﬁned by explicit Hilbert space features maps (see, e.g. , Saitoh, 1997, §2.1).

Original: Theorem 6 (RKHS from explicit feature map) .
Revised: Theorem 6 (RKHS from explicit feature map) .

Original: Let H be some Hilbert space, ψ : X → H a feature map, and K ( x, x (cid:48) )
Revised: Let H be some Hilbert space, ψ : X → H a feature map, and K ( x, x (cid:48) )

Original: = (cid:104) ψ ( x ) , ψ ( x (cid:48) )
Revised: = (cid:104) ψ ( x ) , ψ ( x (cid:48) )

Original: (cid:105) H a kernel on X .
Revised: (cid:105) H a kernel on X .

Original: The RKHS H of K consists of functions f = [Equation] g, ψ ( · ) (cid:105) H , with norm
Revised: The RKHS H of K consists of functions f = (cid:104) g, ψ ( · ) (cid:105) H , with norm

Original: [Equation]
Revised: [Equation]

Original: We also state here the generalization bound for kernel ridge regression used in Section 4, adapted from Bach (2021, Proposition 7.1).
Revised: We also state here the generalization bound for kernel ridge regression used in Section 4, adapted from Bach (2021, Proposition 7.1).

Original: Proposition 7 (Generalization bound for kernel ridge regression) .
Revised: Proposition 7 (Generalization bound for kernel ridge regression) .

Original: Denote f ∗ ( x ) =
Revised: Denote f ∗ ( x ) =

Original: E ρ [ y | x ] .
Revised: E ρ [ y | x ] .

Original: Assume f ∗ ∈ H , Var ρ
Revised: Assume f ∗ ∈ H , Var ρ

Original: [ y | x ] ≤ σ 2 , K ( x, x ) ≤ 1 a.s., and deﬁne 
Revised: [ y | x ] ≤ σ 2 , K ( x, x ) ≤ 1 a.s., and deﬁne [Equation]

Original: [Equation]
Revised: [Equation]

Original: 17) for i.i.d.
Revised: for i.i.d.

Original: data ( x i , y i ) ∼ ρ , i = 1 , . . . , n .
Revised: data ( x i , y i ) ∼ ρ , i = 1 , . . . , n .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: For λ = (cid:112) σ 2 E ρ X [ K ( x, x )] /n (cid:107)
Revised: For λ = (cid:112) σ [Equation] E [Equation]

Revised: [ K

Revised: [Equation]

Original: f ∗ (cid:107) 2 H , we have
Revised: x, [Equation] ] /n (cid:107) f

Original: [Equation]
Revised: [Equation]

Revised: (cid:107)

Revised: [Equation]

Revised: , we have

Revised: [Equation]

Revised: [Equation]

Original: where C is an absolute constant.
Revised: where C is an absolute constant.

Original: Proof.
Revised: Proof.

Original: Under the conditions of the theorem, we may apply (Bach, 2021, Proposition 7.1), which states that for λ ≤ 1 and n ≥ 5 λ
Revised: Under the conditions of the theorem, we may apply (Bach, 2021, Proposition 7.1), which states that for λ ≤ 1 and n ≥ 5 λ

Original: 1 + log(1 /λ )) , we have
Revised: 1 + log(1 /λ )) , we have

Original: [Equation]
Revised: [Equation]

Original: where Σ = E ρ X
Revised: where Σ = E ρ X

Original: [ K ( x, · ) ⊗ K ( x, · )] is the covariance operator.
Revised: [ K ( x, · ) ⊗ K ( x, · )] is the covariance operator.

Original: We conclude by using the inequalities
Revised: We conclude by using the inequalities

Original: [Equation]
Revised: [Equation]

Original: and optimizing for λ .
Revised: and optimizing for λ .

Original: E. P ROOF OF P ROPOSITION 1 (RKHS OF O NE -L AYER C ONVOLUTIONAL K ERNEL )
Revised: E. P ROOF OF P ROPOSITION 1 (RKHS OF O NE -L AYER C ONVOLUTIONAL K ERNEL )

Original: Proof.
Revised: Proof.

Original: From Theorem 6, the RKHS contains functions of the form 
Revised: From Theorem 6, the RKHS contains functions of the form f

Original: [Equation]
Revised: [Equation]

Revised: ) =

Revised: [Equation]

Original:  with RKHS norm equal to the minimum of (cid:107) F (cid:107) L 2 (Ω 1 , H ) over such decompositions.
Revised: , A [Equation] [Equation] (Ω 1 , H ) , with RKHS norm equal to the minimum of (cid:107) F (cid:107) L 2 (Ω 1 , H ) over such decompositions.

Original: We may alternatively write f ( x ) = (cid:104) G, Φ( x ) (cid:105) L 2 (Ω , H ) with G =
Revised: We may alternatively write f ( x ) = (cid:104) G, Φ( x ) (cid:105) L 2 (Ω , H ) with G =

Original: A ∗ F .
Revised: A ∗ F .

Original: The mapping from F to G is one-to-one if G ∈ Range
Revised: The mapping from F to G is one-to-one if G ∈ Range

Original: ( A ∗ ) .
Revised: ( A ∗ ) .

Original: Then, we obtain that equivalently, the RKHS contains functions of this form, with G ∈ Range( A ∗ ) , and with RKHS norm equal to the minimum of (cid:107) A †∗ G (cid:107) L 2 (Ω 1 , H ) over such decompositions.
Revised: Then, we obtain that equivalently, the RKHS contains functions of this form, with G ∈ Range( A ∗ ) , and with RKHS norm equal to the minimum of (cid:107) A †∗ G (cid:107) L 2 (Ω 1 , H ) over such decompositions.

Original: E. P ROOF OF P ROPOSITION 2 (RKHS OF 2- LAYER CKN WITH QUADRATIC k 2 )
Revised: E. P ROOF OF P ROPOSITION 2 (RKHS OF 2- LAYER CKN WITH QUADRATIC k 2 )

Original: Proof.
Revised: Proof.

Original: From Theorem 6, the RKHS contains functions of the form
Revised: From Theorem 6, the RKHS contains functions of the form

Original: [Equation]
Revised: [Equation]

Original: with RKHS norm equal to the minimum of (cid:107) F (cid:107) L 2 (Ω 2 , H 2 ) over such decompositions.
Revised: with RKHS norm equal to the minimum of (cid:107) F (cid:107) L 2 (Ω 2 , H 2 ) over such decompositions.

Original: Here, Φ 1 ( x ) ∈
Revised: Here, Φ 1 ( x ) ∈

Original: L 2 (Ω , H ) is given by Φ 1 ( x )[ u ] = ϕ 1 ( x u ) , so that Φ( x ) in the statement is given by Φ( x ) =
Revised: L 2 (Ω , H ) is given by Φ 1 ( x )[ u ] = ϕ 1 ( x u ) , so that Φ( x ) in the statement is given by Φ( x ) =

Original: Φ 1 ( x ) ⊗ Φ 1 ( x ) .
Revised: Φ 1 ( x ) ⊗ Φ 1 ( x ) .

Original: We also have that H 

Revised: We also have that H

Original: =
Revised: =

Original: ( H ⊗ H ) | S 2 |×| S 2 | , so that we may write F = ( F pq ) p,q ∈ Swith F pq ∈ L 2 (Ω 2 , H ⊗ H ) . 
Revised: ( H ⊗ H ) | S 2 |×| S 2 | , so that we may write F = ( F pq ) p,q ∈ Swith F pq ∈ L 2 (Ω 2 , H ⊗ H ) . For p

Revised: [Equation]

Original: For p, q ∈ S 2 , denoting by L c the translation operator L c x [ u ] = x 
Revised: q ∈ S 2 , denoting by L c the translation operator L c x [ u ] = x [ u − c ]

Original: [ u − c ] , we have
Revised: [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: We may then write this as (cid:104) G pq , Φ( x ) (cid:105)
Revised: We may then write this as (cid:104) G pq , Φ( x ) (cid:105)

Original: L 2 (Ω 2 , H⊗H ) with
Revised: L 2 (Ω 2 , H⊗H ) with

Original: [Equation]
Revised: [Equation]

Original:  and the mapping between F pq ∈ L 2 (Ω 2 , H ⊗ H ) and
Revised: , and the mapping between F pq ∈ L 2 (Ω 2 , H ⊗ H ) and

Original: G pq ∈ L 2 (Ω 2 , H ⊗ H ) is one-to-one if G pq ∈ Range(( L p A 1 ⊗ L q A 1 ) ∗ ) , and diag(( L p A 1 ⊗ L q A 1 ) †∗ G pq
Revised: G pq ∈ L 2 (Ω 2 , H ⊗ H ) is one-to-one if G pq ∈ Range(( L p A 1 ⊗ L q A 1 ) ∗ ) [Equation] and diag(( L p A 1 ⊗ L q A 1 ) †∗ G pq

Original: ) ∈
Revised: ) ∈

Original: Range( A ∗ 2 ) .
Revised: Range( A ∗ 2 ) .

Original: We may then equivalently write the RKHS norm as the minimum over G pq satisfying such constraints for all p, q ∈ S 2 , of the quantity
Revised: We may then equivalently write the RKHS norm as the minimum over G pq satisfying such constraints for all p, q ∈ S 2 , of the quantity

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF OF P ROPOSITION 5 (RKHS OF 3- LAYER CKN WITH QUADRATIC k 2 / 3 )
Revised: E. P ROOF OF P ROPOSITION 5 (RKHS OF 3- LAYER CKN WITH QUADRATIC k 2 / 3 )

Original: Proof.
Revised: Proof.

Original: Let Φ( x ) =
Revised: Let Φ( x ) =

Original: ( ϕ 1 ( x u ))
Revised: ( ϕ 1 ( x u ))

Original: u ∈ L 2 (Ω , H ) , so that we may write 
Revised: u ∈ L 2 (Ω , H ) , so that we may write (cid:88)

Original: [Equation]
Revised: [Equation]

Original: for some G ∈ L 2 (Ω 4 , H ⊗ 4 ) .
Revised: for some G ∈ L 2 (Ω 4 , H ⊗ 4 ) .

Original: From Theorem 6, the RKHS contains functions of the form
Revised: From Theorem 6, the RKHS contains functions of the form

Original: [Equation]
Revised: [Equation]

Original: (19) with RKHS norm equal to the minimum of (cid:107) F (cid:107) L 2 (Ω 3 , H 3 ) over such decompositions.
Revised: with RKHS norm equal to the minimum of (cid:107) F (cid:107) L 2 (Ω 3 , H 3 ) over such decompositions.

Original: Here, Φ 2 ( x ) ∈
Revised: Here, Φ 2 ( x ) ∈

Original: L 2 (Ω 1 , H 2 ) =
Revised: L 2 (Ω 1 , H 2 ) =

Original: L 2 (Ω 1 , ( H ⊗ H ) | S 2 |×| S 2 | ) is given as in the proof of Proposition 2, by[Equation] [Equation] Φ
Revised: L 2 (Ω 1 , ( H ⊗ H ) | S 2 |×| S 2 | ) is given as in the proof of Proposition 2, by

Original: [Equation]

Original: x )

Original: [Equation]

Original: Φ

Original: [Equation]
Revised: [Equation]

Original: x )))
Revised: x )))

Original: [ u ] , for q, r ∈ S 2 .
Revised: [ u ] , for q, r ∈ S 2 .

Original: A patch P 

Revised: A patch P

Original: A 2 Φ 2 ( x )[ u ] is then given by
Revised: A 2 Φ 2 ( x )[ u ] is then given by

Original: [Equation]
Revised: [Equation]

Original: Applying the quadratic feature map given by ϕ 3 ( z ) =
Revised: Applying the quadratic feature map given by ϕ 3 ( z ) =

Original: z ⊗ z ∈
Revised: z ⊗ z ∈

Original: ( H ⊗ 4 ) ( | S 3 |×| S 2 |×| S 2 | ) 2 for z ∈ ( H ⊗ H ) | S 3 |×| S 2 |×| S 2 | , we obtain for α = ( p, q, r, p (cid:48) , q (cid:48) , r (cid:48) )
Revised: ( H ⊗ 4 ) ( | S 3 |×| S 2 |×| S 2 | ) 2 for z ∈ ( H ⊗ H ) | S 3 |×| S 2 |×| S 2 | , we obtain for α = ( p, q, r, p (cid:48) , q (cid:48) , r (cid:48) )

Original: ∈ ( S 3 × S 2 × S 2 ) 2 , 
Revised: ∈ ( S 3 × S 2 × S 2 ) 2 , [Equation] [Equation] [Equation]

Revised: x )

Original: [Equation]
Revised: [Equation]

Revised: Φ

Revised: [Equation]

Original:  where
Revised: r [Equation] x ))) [ u ] , where

Original: [Equation]
Revised: [Equation]

Original: Now, one can check that we have the following relation: 
Revised: Now, one can check that we have the following relation: Φ

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: Since H 3 = ( H ⊗ 4 ) ( | S 3 |×| S 2 |×| S 2 | )
Revised: Since H 3 = ( H ⊗ 4 ) ( | S 3 |×| S 2 |×| S 2 | )

Original: [Equation]
Revised: [Equation]

Original: we may write F =
Revised: we may write F =

Original: ( F α ) α ∈
Revised: ( F α ) α ∈

Original: ( S 3 × S 2 × S 2 ) 2 , with each F α ∈
Revised: ( S 3 × S 2 × S 2 ) 2 , with each F α ∈

Original: L 2 (Ω 3 , H ⊗ 4 ) .
Revised: L 2 (Ω 3 , H ⊗ 4 ) .

Original: We then have
Revised: We then have

Original: [Equation]
Revised: [Equation]

Original: We may write this as (cid:104) G α , Φ( x ) ⊗ 4 (cid:105)
Revised: We may write this as (cid:104) G α , Φ( x ) ⊗ 4 (cid:105)

Original: L 2 (Ω 4 , H ⊗ 4 ) , with
Revised: L 2 (Ω 4 , H ⊗ 4 ) , with

Original: G 
Revised: G α =

Original: [Equation]
Revised: A ∗ 1 ,α diag 2 ( A ∗ 2 ,α diag( A ∗ 3 F α )) .

Original: The mapping from F α to G α is bijective if G α is constrained to the lie in the range of the operator E α .
Revised: The mapping from F α to G α is bijective if G α is constrained to the lie in the range of the operator E α .

Original: If G α satisﬁes this constraint, we may write
Revised: If G α satisﬁes this constraint, we may write

Original: [Equation]
Revised: [Equation]

Original: Then, the resulting penalty on G α is as desired.
Revised: Then, the resulting penalty on G α is as desired.

Original: E. P ROOF OF P ROPOSITION 3 ( GENERALIZATION FOR ONE - LAYER CKN)
Revised: E. P ROOF OF P ROPOSITION 3 ( GENERALIZATION FOR ONE - LAYER CKN)

Original: Proof.
Revised: Proof.

Original: Note that we have
Revised: Note that we have

Original: [Equation]
Revised: [Equation]

Original: It remains to verify that (cid:107) f ∗ (cid:107) H K 1 = (cid:112) | Ω |(cid:107) g (cid:107) H .
Revised: It remains to verify that (cid:107) f ∗ (cid:107) H K 1 = (cid:112) | Ω |(cid:107) g (cid:107) H .

Original: Note that if we denote G = ( g ) u ∈ Ω ∈
Revised: Note that if we denote G = ( g ) u ∈ Ω ∈

Original: L 2 (Ω , H ) , then we have A ∗ G =
Revised: L 2 (Ω , H ) , then we have A ∗ G =

Original: G , since (cid:80) v h [ v − u ] G [ v ] =
Revised: G , since (cid:80) v h [ v − u ] G [ v ] =

Original: ( (cid:80) v h [ v − u ]) g
Revised: ( (cid:80) v h [ v − u ]) g

Original: = g .
Revised: = g .

Original: This implies that A ∗† G = G , regardless of which pooling ﬁlter is used.
Revised: This implies that A ∗† G = G , regardless of which pooling ﬁlter is used.

Original: Then we have, by (5)
Revised: Then we have, by (5)

Original: that (cid:107) f ∗ (cid:107) 2 ≤ | Ω |(cid:107) g (cid:107) 2 .
Revised: that (cid:107) f ∗ (cid:107) 2 ≤ | Ω |(cid:107) g (cid:107) 2 .

Original: Further, since g is of minimal norm, no other G ∈ L 2 (Ω , H ) may lead to a smaller norm, so that we can conclude (cid:107) f ∗ (cid:107) 2 = | Ω |(cid:107) g (cid:107) 2 .
Revised: Further, since g is of minimal norm, no other G ∈ L 2 (Ω , H ) may lead to a smaller norm, so that we can conclude (cid:107) f ∗ (cid:107) 2 = | Ω | [Equation]

Original: E. 
Revised: E. P ROOF OF P ROPOSITION 4 ( GENERALIZATION FOR TWO - LAYER CKN WITH QUADRATIC k 2 )

Original: QUADRATIC k 2 )

Original: Proof.
Revised: Proof.

Original: We begin by studying the “variance” quantity E x
Revised: We begin by studying the “variance” quantity E x

Original: [ K 2 ( x, x )] .
Revised: [ K 2 ( x, x )] .

Original: By expanding the construction of the kernel K 2 , we may write
Revised: By expanding the construction of the kernel K [Equation] we may write

Original: [Equation]
Revised: [Equation]

Original: Upper bounding the quantity E [ k 1 ( x w 1 − p , x w (cid:48) 1 − p ) k 1 ( x w 2 − q , x w (cid:48) 2 − q )] by 

Revised: Upper bounding the quantity E [ k 1 ( x w 1 − p , x w (cid:48) 1 − p ) k 1 ( x w 2 − q , x w (cid:48) 2 − q )] by

Original: when w 1 = w (cid:48) 1 and w 2 = w (cid:48) 2 , and by (cid:15) otherwise, the sum of the coefﬁcients in front of 1 can be bounded as follows:
Revised: when w 1 = w (cid:48) 1 and w 2 = w (cid:48) 2 , and by (cid:15) otherwise, the sum of the coefﬁcients in front of 1 can be bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: while the sum of coefﬁcients bounded by (cid:15) is upper bounded by | Ω || S 2 | 2 .
Revised: while the sum of coefﬁcients bounded by (cid:15) is upper bounded by | Ω || S 2 | 2 .

Original: Overall, this yields
Revised: Overall, this yields

Original: [Equation]
Revised: [Equation]

Original: The bounds obtained for the example function f ∗ ( x ) = (cid:80) u,v g ( x u , x v ) rely on plugging in the values of the Dirac ﬁlter h
Revised: The bounds obtained for the example function f ∗ ( x ) = (cid:80) u,v g ( x u , x v ) rely on plugging in the values of the Dirac ﬁlter h

Original: [ u ] = δ ( u = 0) or average pooling ﬁlters
Revised: [ u ] = δ ( u = 0) or average pooling ﬁlters

Original: h [ u ] = 1 / | Ω | in the expression of E x
Revised: h [ u ] = 1 / | Ω | in the expression of E x

Original: [ K 2 ( x, x )] , and on computing the norm (cid:107) f ∗ (cid:107) K 2 for different architectures using Eq.
Revised: [ K 2 ( x, x )] , and on computing the norm (cid:107) f ∗ (cid:107) K 2 for different architectures using Eq.

Original: (7) in Proposition 2.
Revised: (7) in Proposition 2.

Original: Computing E
Revised: Computing E

Original: x [ K 2 ( x, x )] is immediate using the expression above.
Revised: x [ K 2 ( x, x )] is immediate using the expression above.

Original: Computing (cid:107) f ∗ (cid:107) K 2 is more involved, and requires ﬁnding appropriate decompositions of the form f ∗ ( x )
Revised: Computing (cid:107) f ∗ (cid:107) K 2 is more involved, and requires ﬁnding appropriate decompositions of the form f ∗ ( x )

Original: = (cid:80) p,q (cid:80) u,v G pq [ u, v ]( x u , x v ) in order to leverage Proposition 2:
Revised: = (cid:80) p,q (cid:80) u,v G pq [ u, v ]( x u , x v ) in order to leverage Proposition 2:

Original: • When using a Dirac ﬁlter at the ﬁrst layer, we need | S 2 | = | Ω | in order to capture log- range interaction terms, and we represent f ∗ as a sum of G p that are non-zero and equal to g only on the p -th diagonal,
Revised: • When using a Dirac ﬁlter at the ﬁrst layer, we need | S 2 | = | Ω | in order to capture log- range interaction terms, and we represent f ∗ as a sum of G p that are non-zero and equal to g only on the p -th diagonal,

Original: i
Revised: i

Original: e. , G p [ u, v ] = g when v = u + p , and zero otherwise.
Revised: e. , G p [ u, v ] = g when v = u + p , and zero otherwise.

Original: We then verify
Revised: We then verify

Original: (cid:80) p (cid:80)
Revised: (cid:80) p (cid:80)

Original: u,v G p [ u, v ]( x u , x v )
Revised: u,v G p [ u, v ]( x u , x v )

Original: =
Revised: =

Original: (cid:80) p
Revised: (cid:80) p

Original: (cid:80)
Revised: (cid:80)

Original: u g ( x u , x u + p ) =
Revised: u g ( x u , x u + p ) =

Original: f ∗ ( x ) .
Revised: f ∗ ( x ) .

Original: Then, the expression (7) on this decomposition yields | S 2 || Ω |(cid:107)
Revised: Then, the expression (7) on this decomposition yields | S 2 || Ω |(cid:107)

Original: g
Revised: g

Original: (cid:107)
Revised: (cid:107)

Original: H⊗H =
Revised: H⊗H =

Original: | Ω | 2 (cid:107) g (cid:107)
Revised: | Ω | 2 (cid:107) g (cid:107)

Original: H⊗H
Revised: H⊗H

Original: , for any choice of pooling
Revised: , for any choice of pooling

Original: h
Revised: h

Original: 2 such that (cid:107) h 2 (cid:107) 1 = 1 (using similar arguments to the proof of Proposition 3).
Revised: 2 such that (cid:107) h 2 (cid:107) 1 = 1 (using similar arguments to the proof of Proposition 3).

Original: This is then equal to the squared norm of f ∗ due to the minimality of
Revised: This is then equal to the squared norm of f ∗ due to the minimality of

Original: [Equation]
Revised: [Equation]

Original: H⊗H
Revised: H⊗H

Original: • When using average pooling at the ﬁrst layer and | S 2 | = | Ω | , we may use a single term G [ u, v ] with all entries equal to g , i
Revised: • When using average pooling at the ﬁrst layer and | S 2 | = | Ω | , we may use a single term G [ u, v ] with all entries equal to g , i

Original: e. , a decomposition f ∗ ( x ) =
Revised: e. , a decomposition f ∗ ( x ) =

Original: (cid:80) u,v G [ u, v ]( x u , x v ) .
Revised: (cid:80) u,v G [ u, v ]( x u , x v ) .

Original: Using (7), we obtain | Ω |(cid:107) g (cid:107) 2 for the squared norm.
Revised: Using (7), we obtain | Ω |(cid:107) g (cid:107) 2 for the squared norm.

Original: The same decomposition can be used when | S 2 | = 1 , leading to the same norm.
Revised: The same decomposition can be used when | S 2 | = 1 , leading to the same norm.

Original: F G ENERALIZATION G AINS UNDER S PECIFIC D ATA M ODELS
Revised: F G ENERALIZATION G AINS UNDER S PECIFIC D ATA M ODELS

Original: In this section, we consider simple models of architectures and data distribution where we may quantify more precisely the improvements in sample complexity guarantees thanks to pooling.
Revised: In this section, we consider simple models of architectures and data distribution where we may quantify more precisely the improvements in sample complexity guarantees thanks to pooling.

Original: We consider architectures with non-overlapping patches, and a data distribution where the patches are independent and uniformly distributed on the sphere S d − 1 sphere in d := p | S 1 | dimensions.
Revised: We consider architectures with non-overlapping patches, and a data distribution where the patches are independent and uniformly distributed on the sphere S d − 1 sphere in d := p | S 1 | dimensions.

Original: Further, we consider a dot-product kernel on patches of the form k 1 ( z, z (cid:48) )
Revised: Further, we consider a dot-product kernel on patches of the form k 1 ( z, z (cid:48) )

Original: = κ ( (cid:104) z, z (cid:48) (cid:105) ) for z, z (cid:48)
Revised: = κ ( (cid:104) z, z (cid:48) (cid:105) ) for z, z (cid:48)

Original: ∈ S
Revised: ∈ S

Original: d − 1 , with the common normalization κ (1) = 1 .
Revised: d − 1 , with the common normalization κ (1) = 1 .

Original: In the construction of Section 2, using non-overlapping patches corresponds to taking a downsampling factor s 1 =
Revised: In the construction of Section 2, using non-overlapping patches corresponds to taking a downsampling factor s 1 =

Original: | S 1 | at the ﬁrst layer, and a corresponding pooling ﬁlter such that h 1 [ u ] = 0 for u (cid:54) =
Revised: | S 1 | at the ﬁrst layer, and a corresponding pooling ﬁlter such that h 1 [ u ] = 0 for u (cid:54) =

Original: mod | S 1 | .
Revised: mod | S 1 | .

Original: Alternatively, we may more simply denote patches by x u with u ∈ Ω by considering a modiﬁed signal with more channels ( x u = x
Revised: Alternatively, we may more simply denote patches by x u with u ∈ Ω by considering a modiﬁed signal with more channels ( x u = x

Original: [ u ] of dimension pe instead of p , where e is the patch size) so that extracting patches of size 1 actually corresponds to a patch of size e of the underlying signal.
Revised: [ u ] of dimension pe instead of p , where e is the patch size) so that extracting patches of size 1 actually corresponds to a patch of size e of the underlying signal.

Original: We then have that the patches x u in a signal x are i.i.d., uniformly distributed on the sphere S d − 1 .
Revised: We then have that the patches x u in a signal x are i.i.d., uniformly distributed on the sphere S d − 1 .

Original: We denote the uniform measure on S d − 1 by dτ .
Revised: We denote the uniform measure on S d − 1 by dτ .

Original: We note that when patches are in high dimension, overlapping patches may become near-orthogonal, which could allow extensions of our arguments below to the case with overlap, yet this may require different tools similar to Mei et al.
Revised: We note that when patches are in high dimension, overlapping patches may become near-orthogonal, which could allow extensions of our arguments below to the case with overlap, yet this may require different tools similar to Mei et al.

Original: We leave these questions to future work.
Revised: We leave these questions to future work.

Original: F. Q UANTIFYING THE TRACE OF THE COVARIANCE OPERATOR E [ K ( x, x )]
Revised: F. Q UANTIFYING THE TRACE OF THE COVARIANCE OPERATOR E [ K ( x, x )]

Original: In this section, we focus on the “variance” term E [ K ( x, x )] which is used in the generalization results of Section 4.
Revised: In this section, we focus on the “variance” term E [ K ( x, x )] which is used in the generalization results of Section 4.

Original: One layer.
Revised: One layer.

Original: In the one-layer case, we clearly have σ 20 :=
Revised: In the one-layer case, we clearly have σ 20 :=

Original: E [ κ ( (cid:104) x u , x u (cid:105) )]
Revised: E [ κ ( (cid:104) x u , x u (cid:105) )]

Original: = κ (1) = 1 .
Revised: = κ (1) = 1 .

Original: For u (cid:54) = v , since patches x u and x v are independent and i.i.d., σ 2 u − v is a constant independent of u, v , which may be computed by integration on the sphere as:
Revised: For u (cid:54) = v , since patches x u and x v are independent and i.i.d., σ 2 u − v is a constant independent of u, v , which may be computed by integration on the sphere as:

Original: [Equation]
Revised: [Equation]

Original: where we used a standard change of variable t = (cid:104) x u , x v (cid:105) when integrating x v over S d − 1 (see, e.g. , Efthimiou & Frye, 2014), with ω p − 1 = 2 π p/ 2 / Γ( p/ 2) the surface measure of the sphere S p − 1 .
Revised: where we used a standard change of variable t = (cid:104) x u , x v (cid:105) when integrating x v over S d − 1 (see, e.g. , Efthimiou & Frye, 2014), with ω p − 1 = 2 π p/ 2 / Γ( p/ 2) the surface measure of the sphere S p − 1 .

Original: Note that the integral in (21) corresponds to the constant component in the Legendre decomposition of κ , which is known for common kernels as consider in various works studying spectral properties of dot-product kernels (Bach, 2017a; Minh et al., 2006).
Revised: Note that the integral in (21) corresponds to the constant component in the Legendre decomposition of κ , which is known for common kernels as consider in various works studying spectral properties of dot-product kernels (Bach, 2017a; Minh et al., 2006).

Original: For instance, for the exponential kernel (or Gaussian on the sphere) κ ( (cid:104) x, y (cid:105)
Revised: For instance, for the exponential kernel (or Gaussian on the sphere) κ ( (cid:104) x, y (cid:105)

Original: [Equation]
Revised: [Equation]

Original: 1) , which is used in most of our experiments with σ = 0 .
Revised: 1) , which is used in most of our experiments with σ = 0 .

Original: 6 , we have (Minh et al., 2006, Theorem 2):
Revised: 6 , we have (Minh et al., 2006, Theorem 2):

Original: [Equation]
Revised: [Equation]

Original: where I denotes the modiﬁed Bessel function of the ﬁrst kind.
Revised: where I denotes the modiﬁed Bessel function of the ﬁrst kind.

Original: For arc-cosine kernels, it may be obtained by leveraging the random feature expansion of the kernel (Bach, 2017a).
Revised: For arc-cosine kernels, it may be obtained by leveraging the random feature expansion of the kernel (Bach, 2017a).

Original: More generally, we also note that when the patch dimension d is large, we have
Revised: More generally, we also note that when the patch dimension d is large, we have

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: ∞
Revised: ∞

Original: [Equation]
Revised: [Equation]

Original: since ω d − 2 ω d − 1 (1 − t 2 ) d − 32 is a probability density that converges weakly to a Dirac mass at 0.
Revised: since ω d − 2 ω d − 1 (1 − t 2 ) d − 32 is a probability density that converges weakly to a Dirac mass at 0.

Original: In particular, for the exponential kernel with σ = 0 .
Revised: In particular, for the exponential kernel with σ = 0 .

Original: 6 , we have κ (0) =
Revised: 6 , we have κ (0) =

Original: e − 1 /σ 2 ≈ 0 . 06 .
Revised: e − 1 /σ 2 ≈ 0 . 06 .

Original: When learning a translation-invariant function, the bound in Prop. 3 then shows that global average pooling yields an improvement w.r.t.
Revised: When learning a translation-invariant function, the bound in Prop. 3 then shows that global average pooling yields an improvement w.r.t.

Original: no pooling of order | Ω | / (1+0 . 

Revised: no pooling of order | Ω | / (1+0 .

Original: | Ω | ) .
Revised: | Ω | ) .

Original: Note that removing the constant component of κ , i.e. , using the kernel κ ( u ) − κ (0) κ (1) − κ (0) , may further improve this bound, leading to a denominator very close to 1 when d is large, and hence an improvement in sample complexity of order | Ω | .
Revised: Note that removing the constant component of κ , i.e. , using the kernel κ ( u ) − κ (0) κ (1) − κ (0) , may further improve this bound, leading to a denominator very close to 1 when d is large, and hence an improvement in sample complexity of order | Ω | .

Original: We also remark that the dependence on κ (0) may be removed by using a ﬁner generalization analysis beyond uniform convergence that leverages spectral properties of the kernel (see Section F.2).
Revised: We also remark that the dependence on κ (0) may be removed by using a ﬁner generalization analysis beyond uniform convergence that leverages spectral properties of the kernel (see Section F.2).

Original: Two layers with quadratic k 2 .
Revised: Two layers with quadratic k 2 .

Original: For the two-layer case, we may obtain expressions of E [ k 1 ( x u , x u (cid:48) )
Revised: For the two-layer case, we may obtain expressions of E [ k 1 ( x u , x u (cid:48) )

Original: k 1 ( x v , x v (cid:48) )] as above.
Revised: k 1 ( x v , x v (cid:48) )] as above.

Original: Denote (cid:15) :
Revised: Denote (cid:15) :

Original: = E z,z (cid:48) ∼ τ
Revised: = E z,z (cid:48) ∼ τ

Original: [ k 1 ( z, z (cid:48) )] where z, z (cid:48) are independent, which is given in (21).
Revised: [ k 1 ( z, z (cid:48) )] where z, z (cid:48) are independent, which is given in (21).

Original: We may have the following cases
Revised: We may have the following cases

Original: : • If u = u (cid:48) and v = v (cid:48) , we have, trivially, E [ k 1 ( x u , x u (cid:48) )
Revised: : • If u = u (cid:48) and v = v (cid:48) , we have, trivially, E [ k 1 ( x u , x u (cid:48) )

Original: k 1 ( x v , x v (cid:48) )]
Revised: k 1 ( x v , x v (cid:48) )]

Original: = 1 .
Revised: = 1 .

Original: • If u = u (cid:48) and v (cid:54) = v (cid:48) , we have E [ k 1 ( x u , x u (cid:48) )
Revised: • If u = u (cid:48) and v (cid:54) = v (cid:48) , we have E [ k 1 ( x u , x u (cid:48) )

Original: k 1 ( x v , x v (cid:48) )]
Revised: k 1 ( x v , x v (cid:48) )]

Original: =
Revised: =

Original: E [ k 1 ( x v , x v (cid:48) )]
Revised: E [ k 1 ( x v , x v (cid:48) )]

Original: = (cid:15) , since x v and x v (cid:48) are independent.
Revised: = (cid:15) , since x v and x v (cid:48) are independent.

Original: The same holds if u (cid:54) =
Revised: The same holds if u (cid:54) =

Original: u (cid:48) and v = v (cid:48) .
Revised: u (cid:48) and v = v (cid:48) .

Original: • If |{ u, v, u (cid:48) , v (cid:48) }| = 4 , we have E [ k 1 ( x u , x u (cid:48) )
Revised: • If |{ u, v, u (cid:48) , v (cid:48) }| = 4 , we have E [ k 1 ( x u , x u (cid:48) )

Original: k 1 ( x v , x v (cid:48) )]
Revised: k 1 ( x v , x v (cid:48) )]

Original: =
Revised: =

Original: E [ k 1 ( x u , x u (cid:48) )] E [ k 1 ( x v , x v (cid:48) )]
Revised: E [ k 1 ( x u , x u (cid:48) )] E [ k 1 ( x v , x v (cid:48) )]

Original: = (cid:15) 2 .
Revised: = (cid:15) 2 .

Original: • If u = v and |{ u, u (cid:48) , v (cid:48) }| = 3 , then we have
Revised: • If u = v and |{ u, u (cid:48) , v (cid:48) }| = 3 , then we have

Original: [Equation]
Revised: [Equation]

Original: ) | x u ]] = (cid:15) 2 , by using E x u (cid:48) [ k 1 ( x u , x u (cid:48) )
Revised: ) | x u ]] = (cid:15) 2 , by using E x u (cid:48) [ k 1 ( x u , x u (cid:48) )

Original: | x u ] =
Revised: | x u ] =

Original: E z,z (cid:48) ∼ τ
Revised: E z,z (cid:48) ∼ τ

Original: [ k 1 ( z, z (cid:48) )] , which holds by rotational invariance.
Revised: [ k 1 ( z, z (cid:48) )] , which holds by rotational invariance.

Original: • If u = v (cid:48) and u (cid:48)
Revised: • If u = v (cid:48) and u (cid:48)

Original: = v , we have E [ k 1 ( x u , x u (cid:48) )
Revised: = v , we have E [ k 1 ( x u , x u (cid:48) )

Original: k 1 ( x v , x v (cid:48) )]
Revised: k 1 ( x v , x v (cid:48) )]

Original: =
Revised: =

Original: E z,z (cid:48) ∼ τ
Revised: E z,z (cid:48) ∼ τ

Original: [ k 1 ( z, z (cid:48) ) 2 ]
Revised: [ k 1 ( z, z (cid:48) ) 2 ]

Original: =:
Revised: =:

Original: ˜ (cid:15) .
Revised: ˜ (cid:15) .

Original: This takes the same form as (21), but with a different kernel function κ 2 instead of κ .
Revised: This takes the same form as (21), but with a different kernel function κ 2 instead of κ .

Original: Note that in the case of the Exponential kernel, κ 2 is also an exponential kernel with different bandwidth.
Revised: Note that in the case of the Exponential kernel, κ 2 is also an exponential kernel with different bandwidth.

Original: Overall, when (cid:15) and ˜ (cid:15) are small compared to 1, we can see that the quantity E [ k 1 ( x u , x u (cid:48) )
Revised: Overall, when (cid:15) and ˜ (cid:15) are small compared to 1, we can see that the quantity E [ k 1 ( x u , x u (cid:48) )

Original: k 1 ( x v , x v (cid:48) )] is small compared to 1 unless u = u (cid:48) and v = v (cid:48) , thus satisfying the assumptions in Prop. 4.
Revised: k 1 ( x v , x v (cid:48) )] is small compared to 1 unless u = u (cid:48) and v = v (cid:48) , thus satisfying the assumptions in Prop. 4.

Original: As described above, we may obtain expressions of (cid:15) and ˜ (cid:15) in various cases, and in particular these vanish in high dimension when using a kernel with κ (0)
Revised: As described above, we may obtain expressions of (cid:15) and ˜ (cid:15) in various cases, and in particular these vanish in high dimension when using a kernel with κ (0)

Original: = 0 .
Revised: = 0 .

Original: F. F AST RATES
Revised: F. F AST RATES

Original: In this section, we derive spectral compositions of 1-layer CKN architectures with non-overlapping patches under the product of spheres distribution described above.
Revised: In this section, we derive spectral decompositions of 1-layer CKN architectures with non-overlapping patches under the product of spheres distribution described in the previous section.

Original: This allows us to derive fast rates that depend on the complexity of the target functions on patches, and shows similar improvement factors to those derived in Section F.1, without the κ (0) term, which in fact turns out to only be due to a single eigenspace, namely constant functions.
Revised: This allows us to derive fast rates that depend on the complexity of the target functions on patches, and shows similar improvement factors to those derived in Section F.1, without the κ (0) term, which in fact turns out to only be due to a single eigenspace, namely constant functions.

Original: We note that our derivation extends (Favero et al., 2021) to the case of generic pooling ﬁlters, and considers a different data distribution.
Revised: We note that our derivation extends (Favero et al., 2021) to the case of generic pooling ﬁlters, and considers a different data distribution.

Original: Before studying the 1-layer case, we remark that while it may seem natural to extend such decompositions to the 2-layer case using tensor products of spherical harmonics, as done by Scetbon &
Revised: Before studying the 1-layer case, we remark that while it may seem natural to extend such decompositions to the 2-layer case using tensor products of spherical harmonics, as done by Scetbon &

Original: Harchaoui (2020) in the case without pooling, it appears that pooling may make it more challenging to ﬁnd an eigenbasis since subspaces consisting of tensor products of spherical harmonics with ﬁxed total degree are no longer left stable by the kernel.
Revised: Harchaoui (2020) in the case without pooling, it appears that pooling may make it more challenging to ﬁnd an eigenbasis since subspaces consisting of tensor products of spherical harmonics with ﬁxed total degree are no longer left stable by the kernel.

Original: 5 We thus leave such a study to future work.
Revised: 5 We thus leave such a study to future work.

Original: We begin by considering the following Mercer decomposition of the patch kernel
Revised: We begin by considering the following Mercer decomposition of the patch kernel

Original: [Equation]
Revised: [Equation]

Original: where Y k,j for k ≥ 0 and j = 1 , . .
Revised: where Y k,j for k ≥ 0 and j = 1 , . .

Original: , N ( d, k ) are spherical harmonic polynomials of degree k forming an orthonormal basis of L 2 ( dτ ) .
Revised: , N ( d, k ) are spherical harmonic polynomials of degree k forming an orthonormal basis of L 2 ( dτ ) .

Original: The µ k here are Legendre/Gegenbauer coefﬁcients of the function κ (see, e.g. , Bach, 2017a; Smola et al., 2001).
Revised: The µ k here are Legendre/Gegenbauer coefﬁcients of the function κ (see, e.g. , Bach, 2017a; Smola et al., 2001).

Original: Note that the 1-layer kernel may be written as
Revised: Note that the 1-layer kernel may be written as

Original: [Equation]
Revised: [Equation]

Original: where ∗ denotes convolution and we denote ¯ h [ u ] = h [ − u ] .
Revised: where ∗ denotes convolution and we denote ¯ h [ u ] = h [ − u ] .

Original: Denoting by e w [ u ] , k = 0 , . .
Revised: Denoting by e w [ u ] , k = 0 , . .

Original: . , | Ω | − 1 the orthonormal DFT basis vectors, normalized such that 
Revised: . , | Ω | − 1 the orthonormal DFT basis vectors, normalized such that (cid:104) e

Original: [Equation]
Revised: [Equation]

Original:  e
Revised: , ¯ e

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: = 1 , we have
Revised: = 1 , we have

Original: [Equation]
Revised: [Equation]

Original: with λ w :=
Revised: [ u ]¯ [Equation] [Equation]

Original:  (cid:91) h ∗ ¯ h [ w ] = | ˆ h [ w ] | 2 .
Revised: ] , with λ w := (cid:91) h ∗ ¯ h [ w ] = | ˆ h [ w ] | 2 .

Original: Then, we have
Revised: Then, we have

Original: [Equation]

Original: k

Original: [Equation]
Revised: [Equation]

Original: Note that when k = 0 , we have Y0 , 1 ( x u ) = 1 for all u , hence (cid:32)(cid:88)
Revised: k = 0 , we have Y 1 ( x u ) = 1 for all u , hence (cid:32)(cid:88)

Original: [Equation]
Revised: [Equation]

Original: since e 0 = | Ω | − 1 / 2 (1 , . .
Revised: since e 0 = | Ω | − 1 / 2 (1 , . .

Original: . , 1) and (cid:80)
Revised: . , 1) and (cid:80)

Original: u
Revised: u

Original: e w [ u ] = 0 for w > 0 .
Revised: e w [ u ] = 0 for w > 0 .

Original: Then, we may write
Revised: Then, we may write

Original: [Equation]
Revised: [Equation]

Original:  with φ 0 ( x ) = 1 , φ w,k,j ( x ) = (cid:80) u
Revised: , with φ 0 ( x ) = 1 , φ w,k,j ( x ) = (cid:80) u

Original: e w [ u ] Y k,j ( x u ) , and ¯ φ denotes the complex conjugate of φ .
Revised: e w [ u ] Y k,j ( x u ) , and ¯ φ denotes the complex conjugate of φ .

Original: It is then easy to check that the φ 0 and φ w,k,j form an orthonormal basis of L 2 ( dτ ⊗| Ω | ) .
Revised: It is then easy to check that the φ 0 and φ w,k,j form an orthonormal basis of L 2 ( dτ ⊗| Ω | ) .

Original: We thus have obtained a Mercer decomposition of the kernel K w.r.t.
Revised: We thus have obtained a Mercer decomposition of the kernel K w.r.t.

Original: the data distribution, so that its eigenvalues are also the eigenvalues of the covariance operator (Caponnetto & De Vito, 2007), and control generalization performance, typically  the degrees of freedom
Revised: the data distribution, so that its eigenvalues are also the eigenvalues of the covariance operator (Caponnetto & De Vito, 2007), and control generalization performance, typically through the degrees of freedom

Original: [Equation]
Revised: [Equation]

Original: where Σ is the covariance operator, and ( ξ m ) m is the collection of its eigenvalues.
Revised: where Σ is the covariance operator, and ( ξ m ) m is the collection of its eigenvalues.

Original: In particular, if we have a decay ξ m (cid:16) m − α (with α < 1 ), then we have N ( λ )
Revised: In particular, if we have a decay ξ m (cid:16) m − α (with α < 1 ), then we have N ( λ )

Original: ≤
Revised: ≤

Original: O ( λ − 1 /α ) , which then leads to a fast rate of n − α/ ( α +1) on the excess risk when optimizing for λ in kernel ridge regression (Bach, 2021;
Revised: O ( λ − 1 /α ) , which then leads to a fast rate of n − α/ ( α +1) on the excess risk when optimizing for λ in kernel ridge regression (Bach, 2021;

Original: Caponnetto & De Vito, 2007).
Revised: Caponnetto & De Vito, 2007).

Original: In our case, we have
Revised: In our case, the degrees of freedom takes the form

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: We make a few remarks:
Revised: We make a few remarks:

Original: • Given that the number of spatial frequencies w is ﬁxed, the asymptotic decay rate of eigenvalues associated to the φ w,k,j (that is, λ w µ k , each with multiplicity N ( d, k ) ) is the same as that of the eigenvalues associated to φ w 0 ,k,j for some ﬁxed w 0 , which in turn corresponds to the decay for the corresponding dot-product kernel on the sphere.
Revised: • Given that the number of spatial frequencies w is ﬁxed, the asymptotic decay rate of eigenvalues associated to the φ w,k,j (that is, λ w µ k , each with multiplicity N ( d, k ) ) is the same as that of the eigenvalues associated to φ w 0 ,k,j for some ﬁxed w 0 , which in turn corresponds to the decay for the corresponding dot-product kernel on the sphere.

Original: For instance, if κ is the arc-cosine kernel, we have ξ m (cid:16) m − α with α
Revised: For instance, if κ is the arc-cosine kernel, we have ξ m (cid:16) m − α with α

Original: = d +2 d − 1 , and more generally α = 2 s d −for a kernel resembling a Sobolev space with s bounded derivatives.
Revised: = d +2 d − 1 , and more generally α = 2 s d −for a kernel resembling a Sobolev space with s bounded derivatives.

Original: This then leads to a rate n 2 s/ (2 s +( d − 1)) , which only depends on the dimension d of patches, rather than the full dimension d | Ω | .
Revised: This then leads to a rate n 2 s/ (2 s +( d − 1)) , which only depends on the dimension d of patches, rather than the full dimension d | Ω | .

Original: One may also add more general assumption on the smoothness of the localized components of f ∗ (such as g in Proposition 3) in order to get rates that depend explicitly on the order of smoothness s of such components (as in Caponnetto & De Vito, 2007).
Revised: One may also add more general assumption on the smoothness of the localized components of f ∗ (such as g in Proposition 3) in order to get rates that depend explicitly on the order of smoothness s of such components (as in Caponnetto & De Vito, 2007).

Original: • The eigenvalue associated to φ 0 plays a minor role as by itself as it only contributes at most τ 2 ρ /n to the excess risk, which is negligible compared to the rest of the eigenvalues which lead to a slower n − α/ ( α +1) rate.
Revised: • The eigenvalue associated to φ 0 plays a minor role as by itself as it only contributes at most τ 2 ρ /n to the excess risk, which is negligible compared to the rest of the eigenvalues which lead to a slower n − α/ ( α +1) rate.

Original: • With no pooling ( h is a Dirac delta), we have λ w =
Revised: • With no pooling ( h is a Dirac delta), we have λ w =

Original: | ˆ h [ w ] | 2 = 1 / | Ω | for all w .
Revised: | ˆ h [ w ] | 2 = 1 / | Ω | for all w .

Original: We then have
Revised: We then have

Original: [Equation]
Revised: [Equation]

Original: where we deﬁned N κ ( λ ) :
Revised: where we deﬁned N κ ( λ ) :

Original: = (cid:80) k ≥ 1 N ( d, k ) µ k / [Equation] λ + µ k ) .
Revised: = (cid:80) k ≥ 1 N ( d, k ) µ k / ( λ + µ k ) .

Original: • With global pooling ( h = 1 / | Ω | is constant), we have λ 0 = 1 / | Ω | , and λ w = 0 for w > 0 .
Revised: • With global pooling ( h = 1 / | Ω | is constant), we have λ 0 = 1 / | Ω | , and λ w = 0 for w > 0 .

Original: This yields
Revised: This yields

Original: [Equation]
Revised: [Equation]

Original: This then yields an improvement by a factor | Ω | in sample complexity guarantees compared to the scenario above with no pooling, namely the dominant term in the excess risk bound will be C (1 /n ) α/ ( α +1) compared to C
Revised: This then yields an improvement by a factor | Ω | in sample complexity guarantees compared to the scenario above with no pooling, namely the dominant term in the excess risk bound will be C (1 /n ) α/ ( α +1) compared to C

Original: ( | Ω | /n ) α/ ( α +1) .
Revised: ( | Ω | /n ) α/ ( α +1) .

Original: • For more general pooling, one may exploit speciﬁc decays of λ w to obtain ﬁner bounds.
Revised: • For more general pooling, one may exploit speciﬁc decays of λ w to obtain ﬁner bounds.

Original: We may also obtain the following bound by Jensen’s inequality
Revised: We may also obtain the following bound by Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: where we used that ¯ λ
Revised: where we used that ¯ λ

Original: = ( (cid:80) w λ w ) / | Ω | = (cid:107) h (cid:107) 22 / | Ω | .
Revised: = ( (cid:80) w λ w ) / | Ω | = (cid:107) h (cid:107) 22 / | Ω | .

Revised: When N κ ( λ ) ≤ C κ λ − 1 /α , we get a bound

Revised: [Equation]

Original: When N κ ( λ ) ≤ Cλ − 1 /α , we get a bound N ( λ ) ≤ 1 + C | Ω | 1 − 1 /α(cid:107) h (cid:107) 2 /α 2 λ − 1 /α instead of a  bound N ( λ ) ≤ 1 +
Revised: 24) instead of a bound N ( λ ) ≤ 1 + C | Ω | 1 − 1 /α λ − 1 /α for the case of no pooling, that is, the improvement is again controlled by (cid:107) h (cid:107) 22 , which goes from 1 / | Ω | for global pooling, to 1 for no pooling. The following result provides an example of a generalization bound for invariant target functions, which illustrates that there is no curse of dimensionality in the rate if the patch dimension is much smaller than the full dimension ( i.e. , d (cid:28) d | Ω | ), as well as the beneﬁts of pooling. Theorem 8 (Fast rates for one-layer CKN on invariant targets) . Consider an invariant target of the form f ∗ ( x ) = (cid:80) u ∈ Ω g ∗ ( x u ) , and assume:

Original:  C | Ω | 1 − 1 /α λ − 1 /α for the case of no pooling, that is, the improvement is again controlled by (cid:107) h (cid:107) 22 , which goes from 1 / | Ω | for global pooling, to 1 for no pooling.
Revised: • (capacity condition) N κ ( λ ) ≤ C κ λ − 1 /α , • (source condition) g ∗ = T rκ g 0 and (cid:107) g 0 (cid:107) L 2 ( dτ ) ≤ C ∗

Revised: where T

Revised: κ

Revised: is the integral operator of

Revised: the kernel κ on L 2 ( dτ ) , with r > α − 1 2 α

Revised: Then, kernel ridge regression with the one-layer CKN kernel K h with pooling ﬁlter h (with (cid:107) h (cid:107) 1 = 1 ) satisﬁes, for n large enough,

Revised: [Equation]

Revised: (25) where C is independent of | Ω | and h .

Revised: For global pooling, the factor | Ω |(cid:107) h (cid:107)

Revised: [Equation]

Revised: = | Ω | 1 − 1 /α can be improved to 1 , leading to a bound independent of | Ω | .

Revised: In contrast, with no pooling we have | Ω |(cid:107) h (cid:107) 2 /α= |

Revised: [Equation]

Revised: | .

Revised: We note that if g ∗ is assumed to be s -smooth on the sphere, the source condition with 2 αr

Revised: = 2 s d −corresponds to a Sobolev condition of order s , and leads to the bound

Revised: [Equation]

Revised: which highlights that the rate only depends on the patch dimension

Revised: d instead of the full dimension d | Ω | .

Revised: Proof.

Revised: Under the conditions of the theorem, we may apply (Bach, 2021, Proposition 7.2), which states that for λ ≤ 1 and n ≥ 5 λ

Revised: 1 + log(1 /λ )) , we have

Revised: [Equation]

Revised: [Equation]

Revised: where the degrees of freedom N h ( λ ) is given in (22) and satisﬁes the upper bound (24), and the approximation error is given by

Revised: [Equation]

Revised: where H K h denotes the RKHS of K h .

Revised: If we denote by

Revised: [Equation]

Revised: the decompositions of f and f ∗ in the orthonormal basis deﬁned above, note that we have a w,k,j =for any w (cid:54)

Original: {+= 
+}
Revised: {+= 
+}

Revised: by the invariance of f ∗ .

Revised: This yields

Revised: [Equation]

Revised: Note that we have λ 0 = 1 / | Ω | regardless of the choice of pooling ﬁlter h , so that A h ( λ, f ∗ ) does not depend on h .

Revised: Under the source condition, we have, by (Cucker & Smale, 2002, Theorem 3, p.33),

Revised: [Equation]

Revised: with a ∗ 0 ,k,j = a ∗ 0 .

Revised: Combining with (24) and plugging this into (27), we obtain

Revised: [Equation]

Revised: For the choice

Revised: [Equation]

Revised: we have

Revised: [Equation]

Revised: In the case of global pooling, the ﬁrst term scales as ( | Ω | 1 − 1 /α /n ) 2 αr 2 αr +1 , but can be improved to (1 /n )

Revised: 2 αr 2 αr +1 by using the bound (23) on N h ( λ ) instead of (24).

Revised: When r > α − 1 2 α , we can choose n large enough so that n ≥ 5 λ n (1 + log(1 /λ n )) is satisﬁed, and the higher order terms in n are negligible.