Learning with convolution and pooling operations in kernel methods
Abstract
Recent empirical work has shown that hierarchical convolutional kernels inspired by convolutional neural networks (CNNs) signiﬁcantly improve the performance of kernel methods in image classiﬁcation tasks.
A widely accepted explanation for their success is that these architectures encode hypothesis classes that are suitable for natural images.
However, understanding the precise interplay between approximation and generalization in convolutional architectures remains a challenge.
In this paper, we consider the stylized setting of covariates (image pixels) uniformly distributed on the hypercube, and characterize exactly the RKHS of kernels composed of single layers of convolution, pooling, and downsampling operations.
We use this characterization to compute sharp asymptotics of the generalization error for any given function in high-dimension.
In particular, we quantify the gain in sample complexity brought by enforcing locality with the convolution operation and approximate translation invariance with average pooling.
Notably, these results provide a precise description of how convolution and pooling operations trade off approximation with generalization power in one layer convolutional kernels.
Introduction
Convolutional neural networks (CNNs) have become essential elements of the deep learning toolbox, achieving state-of-the-art performance in many computer vision tasks [27, 30].
CNNs are constructed by stacking convolution and pooling layers, which were shown to be paramount to their empirical success [31].
A widely accepted hypothesis to explain their favorable properties is that these architectures successfully encode useful properties of natural images: locality and compositionality of the data, stability by local deformations, and translation invariance.
While some theoretical progress has been made in studying the approximation and generalization beneﬁts brought by convolution and pooling operations [6, 15, 16], our mathematical understanding of the interaction between network architecture, image distribution, and efﬁcient learning remains limited.
Consider x ∈ R d an input signal, which we can think of as a grayscale pixel representation of an image.
For mathematical convenience, we will consider one-dimensional images with cyclic convention x d
+ i := x i , and denote x ( k ) =
( x k , x k +1 , . . .
, x k + q − 1 ) the k -th patch of the signal x , k ∈ [ d ] , with patch size q ≤ d .
Most of our results can be extended to two-dimensional images.
We further consider a simple convolutional neural network composed of a single convolution layer followed by local average pooling and downsampling.
The network ﬁrst computes the nonlinear convolution of N ﬁlters w 1 , . .
w N ∈ R q with the image patches x ( k ) .
The outputs of the convolution operation σ ( (cid:104) w i , x ( k ) (cid:105) ) are then averaged locally over segments of length ω (local average pooling).
This pooling operation is followed by downsampling which extracts one out of every ∆ output coordinates (for simplicity, ∆ is assumed to be a divisor of d ).
Finally, the results are combined linearly using coefﬁcients ( a ik ) i ∈ [ N ] ,k ∈ [ d/ ∆] : f
[Equation]
(CNN-AP-DS)
Note that pooling and downsampling operations are often tied together in the literature.
However in this work we will treat these two operations separately.
In the formula above, different values for q, ω, ∆ lead to different architectures with vastly different behaviors.
For example, when q = ∆ =
d and ω = 1 , we recover a two-layer fully-connected neural network f FC ( x ; a , Θ )
=
N − 1 / 2 (cid:80) i ∈
[ N ] a i σ ( (cid:104) w i , x (cid:105) ) which has the universal approximation property at large N .
When ω = ∆ = 1 and q < d , the network is locally connected f LC ( x ; a , Θ ) =
N − 1 / 2 (cid:80) i ∈
[ N ] ,k ∈ [ d ] a ik σ ( (cid:104) w i , x ( k ) (cid:105) ) , and not a universal approximator anymore: however, f LC vastly outperforms f FC in some cases [33].
For ω > 1 , local pooling enables learning functions that are locally invariant by translations more efﬁciently than without pooling.
For ω = d ( global pooling ), the network only ﬁts functions fully invariant by cyclic translations.
The aim of this paper is to formalize and quantify the interplay between the target function class and the statistical efﬁciency brought by these different architectures .
As a concrete ﬁrst step in this direction, we consider kernel models that are naturally associated to (CNN-AP-DS) through the neural tangent kernel perspective [18, 28].
Kernel methods have the advantage of 1) being tractable— leaving the computational issue of learning CNNs aside; 2) having well-understood approximation and generalization properties, which depends on the eigendecomposition of the kernel and the alignment between the target function and the RKHS [12, 47] (see Appendices B and C for background).
While kernel models only describe neural networks in the lazy training regime [1, 13, 19, 20, 49] and miss important properties of deep learning, such as feature learning, architecture choice already plays a crucial role to learn efﬁciently ‘image-like’ functions in the ﬁxed-feature regime.
Neural tangent kernels are obtained by linearizing the associated neural networks.
Here we consider the tangent kernel associated to the network f CNN (c.f. Appendix A.2 for a detailed derivation):
[Equation]
where h : R → R is related to the activation function σ in (CNN-AP-DS).
As a linearization of
CNNs, the kernel (CK-AP-DS) inherits some of the favorable properties of convolution, pooling, and downsampling operations.
Indeed, a line of work [2, 32, 35, 36, 45] showed that, though performing slightly worse than CNNs, such (hierarchical) convolutional kernels have empirically outperformed the former state-of-the-art kernels.
For instance, these kernels achieved test accuracy around 87% − 90% on CIFAR-10 (the state-of-the-art CNNs can achieve test accuracy 99% ), against 79 .
6% for the best former unsupervised feature-extraction method [14].
In this paper, we will further consider a stylized setting with input signal distribution x ∼ Unif( Q d ) (uniform distribution over Q d := {− 1 , +1 } d the discrete hypercube in d dimensions).
This simple choice allows for a complete characterization of the eigendecomposition of H CK ω, ∆ , thanks to all patches having same marginal distribution x ( k ) ∼ Unif( Q q ) .
We will be particularly interested in four speciﬁc choices of ( q, ω, ∆) in (CK-AP-DS):
[Equation]
These kernels are respectively the neural tangent kernels of a fully-connected network f FC (FC), a convolutional network f LC (CK), a convolutional network followed by local average pooling (CK-AP) and a convolutional network followed by global pooling (CK-GP).
We will further be interested in (CK-GP) with patch size q = d , which we denote H FCGP :
this corresponds to a convolutional kernel with full-size patches q = d , followed by global pooling.
In this paper, we ﬁrst characterize the reproducing kernel Hilbert space (RKHS) of these convolutional kernels, and then investigate their generalization properties in the regression setup.
More speciﬁcally, assume { ( x i , y i ) } i ≤ n are n i.i.d.
samples with x i ∼ Unif( Q d ) and y i =
f (cid:63) ( x i )
+ ε i .
Here f (cid:63) ∈
L 2 ( Q d ) and ( ε i ) i ≤ n are independent errors with mean zero and variance bounded by σ 2 ε .
We will focus on the generalization error of kernel ridge regression (KRR) (see Appendix B.1 for general kernel methods).
In particular, given a kernel function H : Q d × Q d → R and a regularization parameter λ ≥ 0 , the KRR estimator is the solution of the tractable convex problem
[Equation]
where H is the RKHS associated to H with RKHS norm (cid:107) · (cid:107) H .
We denote the test error with square loss by R ( f (cid:63) , ˆ f λ ) =
E x { ( f (cid:63) ( x ) − ˆ f λ ( x )) 2 } .
We will sometimes consider the expected test error
E ε { R ( f (cid:63) , ˆ f λ ) } , where expectation is taken with respect to noise ε = ( ε i ) i ≤ n in the training data.
The generalization properties of kernels H FC and H FC GP were recently studied in [7, 39].
In particular, they showed that global pooling (kernel H FCGP ) leads to a gain of a factor d in sample complexity when ﬁtting cyclic invariant functions, but still suffers from the curse of dimensionality ( H FCGP only ﬁts very smooth functions in high-dimension).
More precisely, Mei et al.
[39] considered the high-dimensional framework of [38] and showed the following: KRR with H FC requires n ≈ d (cid:96) samples to ﬁt degree- (cid:96) cyclic polynomials, while KRR with H FCGP only needs n ≈ d (cid:96) − 1 .
To enable milder dependence on the dimension d , further structural assumptions on the kernel and the target function should be considered (for instance, in this paper, we use the kernel H CK and consider ‘local’ functions).
1.1 Summary of main results
Our contributions are two-fold.
First, we describe the RKHS associated with the convolutional kernel (CK-AP-DS) in the stylized setting x ∼ Unif( Q d ) , which provides a fully explicit picture of the roles of convolution, pooling and downsampling operations in approximating speciﬁc classes of functions.
Second, we provide sharp asymptotics for the generalization error of KRR in highdimension, given any target function and one of the kernels described in the introduction 1 .
These asymptotics are obtained rigorously using the framework of [38] (see Appendix C for background).
For completeness, we also include bounds on the KRR test error in the classical ﬁxed-dimension setting with capacity/source assumptions (see Appendix C for limitations of this classical approach).
We summarize our results below.
Deﬁne the q -local function class L 2 ( Q d , Loc q ) and the cyclic q -local function class L 2 ( Q d , CycLoc q ) (subspace of L 2 ( Q d , Loc q ) consisting of cyclic-invariant functions) as follows:
[Equation]
k ( x ( k ) ) (cid:111) , (LOC)
L 2 ( Q d , CycLoc q ) =
(cid:110)
f ∈ L 2 ( Q d ) : ∃ g ∈ L 2 ( Q q ) , f ( x ) = (cid:88) k ∈
[ d ] g ( x ( k ) ) (cid:111) .
(CYC-LOC)
One-layer convolutional layer.
The RKHS of H CK is equal to L 2 ( Q d , Loc q ) : kernel methods with
H CK can only ﬁt the projection P Loc q f ∗ of the target function onto L 2 ( Q d , Loc q ) .
For a sample size n (cid:16) dq (cid:96) − 1 , KRR ﬁts exactly a degree- (cid:96) polynomial approximation to P Loc q f ∗ .
In particular, for q (cid:28) d , the convolution kernel H CK is much more sample efﬁcient than the standard inner-product kernel H FC for ﬁtting functions in L 2 ( Q d , Loc q ) (sample sizes dq (cid:96) − 1 (cid:28) d (cid:96) for ﬁtting a degree- (cid:96) polynomials).
The convolution operation breaks the curse of dimensionality by restricting the RKHS to local functions.
To ﬁt a degree (cid:96) polynomial
H FC
H FCGP
H CK
H CK ω
H CKGP
Sample complexity d (cid:96) d (cid:96) −dq (cid:96) −dq (cid:96) − 1 /ω q (cid:96) −
Average pooling.
The RKHS of H CK ω is still constituted of q -local functions f ∗ ∈
L 2 ( Q d , Loc q ) , but penalizes differently the frequency components f ∗ ,j ( x ) by reweighting their eigenspaces by a factor κ j , where f ∗ ,j ( x ) =
(cid:80)
k ∈ [ d ] ρ kj f ∗ ( t k · x ) with ρ j = e 2 iπjd and t k · x = ( x k +1 , . . .
, x d , x 1 , . . .
, x k ) denotes the k -shift.
As ω increases, local pooling penalizes more and more heavily the high-frequency components ( κ j (cid:28) 1 ), while making lowfrequency components statistically easier to learn ( κ j (cid:29) 1 ).
For global pooling ω
= d , H CKGP only learns cyclic local functions L 2 ( Q d , CycLoc q ) and enjoy a factor d gain in statistical complexity compared to H CK (sample sizes
(cid:96) − (cid:28) dq − to learn a degree- (cid:96) polynomial)
Local pooling biases learning towards functions that are stable by small translations.
Downsampling.
When ∆ ≤ ω , downsampling after average pooling leaves the low-frequency eigenspaces of H CK ω stable.
In particular, the downsampling operation does not modify the statistical complexity of learning low-frequency functions in one-layer kernels, while being potentially beneﬁcial in further layers in deep convolutional kernels.
These theoretical results answer the following question: given a target function and a sample size n , what is the impact of the architecture on the test error?
For example, Table 1 shows how the architecture modify the sample size required to achieve small test error when learning a degree- (cid:96) polynomial in L 2 ( Q d , CycLoc q ) .
There are two important model assumptions in this paper, which deserve some discussions:
One-layer convolutional kernel (CK): extra layers allow for hierarchical interactions between the patches (see for example [6]).
However, we believe that the main insights on the approximation and statistical trade-off are already captured in the one-layer case (see [48] for multi-layer but independent patches).
Note that depth might be less important for CKs than for CNNs: the one-layer
CK considered in this paper achieves 80 .
9% accuracy on CIFAR-10 [6] (versus 79 .
6% in [14]) and 3-layers CK achieves 88 .
2% accuracy [6] (versus 90% for the best multi-layer CK [45]).
See
Appendix A.5 for a discussion on how our results could be extended to 2-layers.
Data uniform on the hypercube: this choice is motivated by our goal of deriving rigorous ﬁne-grained approximation and generalization errors, which requires to diagonalize the kernel (CK-AP-DS).
More general data distributions either require strong assumptions (independent patches [43, 48]), loose minmax bounds on the generalization error (e.g., classical source/capacity assumptions) or non-rigorous statistical physics heuristics [22].
The rest of the paper is organized as follows.
We discuss related work in Section 1.2.
In Section 2, we present our main results on convolutional kernels and describe precisely the roles of convolution, pooling and downsampling operations.
Finally, we present a numerical simulation on synthetic data in Section 3 and conclude in Section 4.
Some details and discussions are deferred to Appendix A.
1.2 Related work
Convolutional kernels have been considered in [6, 32, 35, 36, 45, 46].
In particular, they showed that these architectures achieve good results in image classiﬁcation ( 90% accuracy on Cifar10) and that pooling and downsampling were necessary for their good performance [32].
The generalization error of kernel ridge regression (KRR) has been well-studied in both the ﬁxed dimension regime [47, Chap. 13], [12] and the high-dimensional regimes [21, 23, 24, 34, 39, 48].
These results show that the generalization error depends on the eigenvalues and eigenfunctions of the kernel, and the alignment of the kernel with the target function.
Recently, a few theoretical work have considered the generalization properties of invariant kernels and convolutional kernels [7, 22, 39, 43, 48].
In particular, a concurrent work [48] considers sharp asymptotics of the KRR test error using the framework of [38] for certain hierarchical convolutional kernels under the strong assumption of non-overlapping patches (whereas we consider the more natural architecture of overlapping patches).
They arrive at a similar trade-off between approximation and generalization power in convolutional kernels, which they call ‘eigenspace restructuring principle’: given a ﬁnite statistical budget (i.e., a sample size n ), convolutional architectures allocate the ‘eigenvalue mass’ by weighting differently the eigenspaces.
Favero et al.
[22] considers a one-layer convolutional kernel with and without global pooling and derive asymptotic rates in sample size n in a student-teacher scenario using statistical physics heuristics and a Gaussian equivalence conjecture.
In particular, they show that locality rather than translation-invariance breaks the curse of dimensionality.
Our goal in this paper is different: we derive rigorous quantitative bounds that give separation in generalization power between different architectures.
See [33, 37] for more theoretical results on the separation between convolutional and fully connected neural networks, and [10, 16] for the inductive bias of pooling operations in convolutional neural networks.
Main results
We start by introducing some background on functions on the hypercube and eigendecomposition of kernel operators in Section 2.1.
We ﬁrst consider a kernel with a single convolution layer in Section 2.2, and characterize its eigendecomposition and generalization properties.
We then show how these results are modiﬁed when applying local average pooling and downsampling in Section 2.3.
2.1 Functions on the hypercube and eigendecomposition of kernel operators
Recall that we work on the d -dimensional hypercube Q d := {− 1 , +1 } d .
Let L 2 ( Q d ) =
L 2 ( Q d , Unif) be the 2 d -dimensional vector space of all functions f : Q d → R , with scalar product (cid:104) f, g (cid:105)
L 2 :=
E x ∼ Unif( Q d ) [ f ( x ) g ( x )] .
Let (cid:107) · (cid:107) L 2 be the norm associated with the scaler product.
We introduce the set of Fourier functions { Y ( d ) S ( x ) } S ⊆
[ d ] which forms an orthonormal basis of L 2 ( Q d ) .
For any subset S ⊆
[ d ] , the Fourier function is deﬁned as Y ( d ) S ( x ) :=
(cid:81)
i ∈
S x i with the convention that Y ( d ) ∅ : = 1 (it is easy to verify that (cid:104) Y ( d ) S , Y ( d ) S (cid:48) (cid:105) L 2 = 1 S = S (cid:48) ).
We will omit the superscript ( d ) which will be clear from context and write Y S :
= Y ( d ) S .
Consider a nonnegative deﬁnite kernel function H : Q p × Q p → R ( p = d or q in this paper) with associated integral operator H :
L 2 ( Q p ) → L 2 ( Q p ) deﬁned as H f ( u ) =
E v { h ( u , v )
f ( v ) } with v ∼ Unif( Q p ) .
By spectral theorem of compact operators, there exists an orthonormal basis { ψ j } j ≥ 1 of L 2 ( Q p ) and nonnegative eigenvalues ( λ j ) j ≥ 1 such that H = (cid:80) j ≥ [-
-]1 λ j ψ j ψ ∗ j (i.e.,
H ( u , v ) = (cid:80) j ≥ 1 λ j ψ j ( u ) ψ
j ( v ) for any u , v ∈ L 2 ( Q p ) ).
The most widespread example are inner-product kernels deﬁned as H ( u , v ) :
= h ( (cid:104) u , v (cid:105) /p ) for some function h : R → R .
Inner-product kernels have the following simple eigendecomposition in
L 2 ( Q p ) (taking here u , v ∈ Q p ): [Equation] p [Equation]
ξ
[Equation]
where ξ p,(cid:96) ( h ) is the (cid:96) -th
Gegenbauer coefﬁcient of h ( · / √ p ) in dimension p , i.e.,
[Equation]
for e ∈ Q p arbitrary and Q ( p ) (cid:96) the degree- (cid:96) Gegenbauer polynomial on Q p (see Appendix D for details).
Note that ( ξ p,(cid:96) )
≤ (cid:96) ≤ q are non-negative by positive semideﬁniteness of the kernel.
We will write ξ p,(cid:96) :
= ξ p,(cid:96) ( h ) and use extensively the decomposition identity (1) in the rest of the paper.
2.2 One-layer convolutional kernel
We ﬁrst consider the convolutional kernel H CK (CK) given by a one-layer convolution layer with patch size q and inner-product kernel function h : R → R :
[Equation]
where we recall that x ( k )
=
( x k , . . .
, x k + q −
) ∈ Q q is the k ’th patch of the image with size q .
Before stating the eigendecomposition of H CK , we introduce some notations.
For any subset S ⊆
[ d ] , denote γ ( S ) the diameter of S with cyclic convention, i.e., γ ( S )
= max { min { mod ( j − i, d ) + 1 , mod ( i − j, d ) + 1 } : i, j ∈ S } (e.g., γ ( { 2 , d } ) = 3 ).
For any integer (cid:96) ≤ q , consider the set
E (cid:96) =
{ S ⊆
[ d ] : | S | = (cid:96), γ ( S ) ≤ q } of all subsets of [ d ] of size (cid:96) with diameter less or equal to q .
We will assume throughout this paper that q ≤ d/ 2 to avoid additional overlap between sets.
Proposition 1 (Eigendecomposition of H CK ) .
Let H CK be a convolutional kernel as deﬁned in Eq.
Then H CK admits the following eigendecomposition:
[Equation]
where r ( S ) =
q + 1 − γ ( S ) and ξ q,(cid:96) ≥ 0 is deﬁned in Eq.
Notice that Y S with γ ( S ) > q (monomials with support not contained in a segment of size q ) are in the null space of H CK .
Hence (as long as ξ q,(cid:96) > 0 for all 0 ≤ (cid:96) ≤ q ), the RKHS associated to
H CK exactly contains all the functions in the q -local function class L 2 ( Q d , Loc q ) (c.f. Eq.
(LOC)).
In words, L 2 ( Q d , Loc q ) consists of functions that are localized on patches, with no long-range interactions between different parts of the image.
An example of local function with q = 3 is given by f ( x ) =
x 1 x 2 x 3 + x 4 x 6 + x 5 .
On the other hand, the RKHS associated to the fully-connected kernel H FC (FC) typically contains all the functions in L 2 ( Q d ) (under genericity assumptions on h ).
The RKHS with convolution dim( L 2 ( Q d , Loc q )) =
d 2 q − 1 + 1 is signiﬁcantly smaller than dim( L 2 ( Q d ))
= 2 d , which prompts the following question: what is the statistical advantage of using H CK over H FC when learning functions in L 2 ( Q d , Loc q ) ?
We ﬁrst consider the classical approach to bounding the test error of [3, 12, 47] which relies on the following two standard assumptions:
(A1) Capacity condition: we assume N ( h, λ ) :=
Tr[ h/ ( h + λ I ) − 1 ] ≤ C h λ − 1 /α with 2 α > 1 .
(A2) Source condition: (cid:107) h − β/ 2 g (cid:107) L 2 ≤ B with 3 β > α − 1 α and B ≥ 0 .
The capacity condition (A1) characterizes the size of the RKHS: for increasing α , the RKHS contains less and less functions.
The source condition (A2) characterizes the regularity of the target function (the ‘source’) with respect to the kernel: increasing β corresponds to smoother and smoother functions
See Appendix B.2 for more discussions.
Based on these two assumptions, we can apply standard bounds on the KRR test error and obtain:
Theorem 1 (Generalization error of KRR with H CK ) .
Let h : R → R be an inner-product kernel satisfying (A1).
Let f (cid:63)
∈
L 2 ( Q d , Loc q ) with f (cid:63) ( x ) =
(cid:80)
k ∈
[ d ] g k ( x ( k ) ) satisfying source condition (cid:80)
k ∈ [ d ] (cid:107) h − β/ 2 g k (cid:107) 2 L 2 ≤ B 2 .
Then there exists C 1 , C 2 , C 3 > 0 constants that only depend on (A1) and B (and independent of d ), such that for n ≥ C 1 max( (cid:107) f (cid:63) (cid:107) 2 L ∞ , d ) and λ ∗ = C 2 d ( d
[Equation]
Note that the exponent βαβα +1 only depends on the q -dimensional kernel h .
Hence, the generalization bound with respect to ( n/d ) is independent of the dimension d of the image.
Let’s compare to KRR with inner-product kernel H FC (FC): from [12], we have the minmax rate E ε { R ( f (cid:63) , ˆ f λ ) } (cid:16) n − ˜
α ˜ β ˜ α ˜ β
+ where h is now deﬁned in d dimension and veriﬁes (A1) and (A2) with constants ˜ α, ˜ β .
Typically, if f (cid:63) is only assumed Lipschitz, then ˜ β
˜ α
= O (1 /d ) , which leads to a minmax rate n − O (1 /d ) for
H FC , while for H CK , βα = O (1 /q ) , which leads to a minmax rate n − O (1 /q ) .
Hence, for q (cid:28) d , 2 Here, h is the integral operator and Tr[ h/ ( h + λ I ) − 1 ] = (cid:80) j ≥ 1 λ j λ j + λ with { λ j } j ≥ 1 eigenvalues of h .
Again, h is the operator with h − κ g
= (cid:80) j ≥ 1 λ − κ j (cid:104) f, ψ j (cid:105) ψ j , where { ψ j } j ≥ 1 are the eigenvectors h .
H CK breaks the curse of dimensionality by restricting the RKHS to ‘local’ functions.
Similarly, [22] derived a decay rates in n that do not depend on d for a one-layer convolutional kernel.
The key difference between Theorem 1 and [22] is that we obtain a non-asymptotic bound that is minmax optimal up to a constant multiplicative factor in both d and n (this can be showed for example by adapting the proof in Appendix B.6 in [7]) using a rigorous framework of source and capacity condition.
Theorem 1 and results of this type suffers from several limitations: 1) they are tight only in a minmax sense; 2) they do not provide comparisons for speciﬁc subclasses of functions; 3) in order to obtain the minmax rate, the regularization parameter λ has to be carefully tuned to balance the bias and variance terms, which is in contrast to modern practice where often the model is trained until interpolation.
This led several groups to consider instead the test error of KRR in a high-dimensional limit [11, 24, 38] and derive exact asymptotic predictions correct up to an additive vanishing constant for any f (cid:63)
∈
L 2 (see Appendix C for more details).
Using the general framework in [38], we get the following result for q, d large:
Theorem 2 (Generalization error of KRR with H CK in high-dimension (informal)) .
Let f (cid:63) ∈
L 2 ( Q d , Loc q ) and h : R → R verifying some ‘genericity condition’.
Then for n = dq s − 1+ ν with 0 < ν < 1 , and λ = O (1) (in particular λ = 0 works), we have
[Equation]
(6) where P E ≤ s ,ν is the projection on the span of Y S with either | S | < s and S ∈ E | S | or | S | =
s and γ ( S ) ≤ q (1 − q − ν ) .
See Appendix C.1 for a rigorous statement.
In words, when dq s − 1 (cid:28) n (cid:28) dq s , KRR with H CK only learns a degree- s polynomial approximation to f (cid:63) .
On the other hand, when considering the standard inner-product kernel H FC (FC) we get:
Theorem 3 (Generalization error of KRR with H FC in high-dimension (informal)) .
Let f (cid:63)
∈
L 2 ( Q d ) and ˜
h : R → R with some ‘genericity condition’.
Then for d s (cid:28) n (cid:28) d s +1 and λ
= O (1) ,
[Equation]
[Equation]
) where P ≤ s is the projection on the subspace of degree- s polynomials.
This theorem was proved in [24, 38].
Notice that Eq.
(7) does not depend on the structure of f (cid:63) .
Hence, when f (cid:63) ∈ L 2 ( Q d , Loc q ) , Theorems 2 and 3 shows a clear statistical advantage of H CK over
H FC when q (cid:28) d (and therefore of one-layer CNNs over fully-connected neural networks in the kernel regime).
2.3 Local average pooling and downsampling
In many applications such as object recognition, we expect the target function to depend mildly on the absolute spatial position of an object and to be stable under small shifts of the input.
To take this local invariance into account, convolution layers are often followed by a pooling operation.
Here we consider local average pooling on a segment of length ω and obtain the kernel
[Equation] [Equation]
Deﬁne S (cid:96) = { S ⊆ [ q ] : | S | = (cid:96) } as the collection of sets of size (cid:96) .
We further deﬁne an equivalence relation ∼ on S (cid:96) : S ∼ S (cid:48) if S (cid:48) is a translated subset of S in [ q ] (without cyclic convention).
We denote C (cid:96) the quotient set of A (cid:96) under the equivalence relation ∼ .
Proposition 2 (Eigendecomposition of H CK ω ) .
Let H CK ω be a convolutional kernel with local average pooling as deﬁned in Eq.
Then H CK ω admits the following eigendecomposition:
[Equation]
where (denoting k + S the translated set S by k positions with cyclic convention in [ d ] )
[Equation]
First notice that, as long as gcd ( ω, d ) = 1 , the RKHS associated to H CK contains the same set of functions as the RKHS of H CK , i.e., all local functions L 2 ( Q d , Loc q ) .
(There are gcd ( ω, d ) −number of zero weights: κ j = 0 for all j ∈
[ d − 1] such that d is a divisor of jω .
See Appendix
A.3 for details.)
However H CK will penalize different frequency components of the functions differently.
Denote f j ( x ) the j -th component of the discrete Fourier transform of the function, i.e., f j ( x ) = 1 √ d (cid:80) k ∈ [ d ] ρ kj f ( t k · x ) where ρ j = e 2 iπj/d and t k · x = ( x k +1 , . . .
, x d , x 1 , . . .
, x k ) is the cyclic shift by k pixels.
Then H CK reweights the eigenspaces associated with f j ( x ) by a factor κ j , promoting low-frequency components ( κ j > 1 ) and penalizing the high-frequencies ( κ j < 1 ).
In words, pooling biases the learning towards low-frequency functions , which are stable by small shifts.
Let us focus on two special choices here: the pooling parameter ω = 1 and ω = d .
When ω = 1 ,
H CK ω reduces to H CK ( κ j = 1 for all j ∈ [ d ] ) which does not bias towards either low or high frequency components.
When ω =
d , we denote such kernel H CK ω
=
d by H CKGP which corresponds to global average pooling.
In this case, we have κ d = d and κ j = 0 for j < d which enforces exact invariance under the group of cyclic translations.
More precisely, H CKGP has RKHS that contains all cyclic q-local functions f ( x ) = (cid:80) k ∈
[ d ] g ( x ( k ) ) ∈
L 2 ( Q d , CycLoc q ) (c.f. Eq.
(CYC-LOC)).
We obtain a bound on the test error of KRR with H CK ω similar to Theorem 1, but with d replaced by an effective dimension d eff .
Theorem 4 (Generalization of KRR with average pooling (ﬁxed d, q )) .
Assume that h : R → R has ξ q, 0 = 0 and satisﬁes (A1).
Further assume ( A 2 (cid:48) )
that (cid:107) ( H CK ω /ω )
− β/
f (cid:63) (cid:107) L 2 ≤ B .
Deﬁne d eff = (cid:80) j ∈
[ d ]: κ j > 0 ( κ j /ω ) 1 /α .
Then there exists C 1 , C 2 , C 3 > 0 constants independent of d , such that for n ≥ C 1 max( (cid:107)
f (cid:63) (cid:107) 2 L ∞ , d eff ) and setting λ ∗ =
C 2 ( d eff /n )
α αβ +1 , we get
[Equation]
By Jensen’s inequality, we have d eff ≤ d/ω 1 /α .
In particular, for global pooling, d eff = 1 and the bound (11) does not depend on d at all.
Adding average pooling improve by a factor ω 1 /α the upper bound on the sample complexity for ﬁtting low-frequency functions.
Can we conﬁrm this statistical advantage using the predictions for KRR in high dimension?
Consider ﬁrst the case of global pooling:
Theorem 5 (Generalization of KRR with H CKGP in high-dimension (informal)) .
Let f (cid:63) ∈
L 2 ( Q d , CycLoc q ) and h : R → R verifying some ‘genericity condition’.
Then for n = q s − 1+ ν with 0 < ν < 1 , and λ
= O (1) , we have ( P E ≤ s ,ν is deﬁned as in Theorem 2)
[Equation]
Hence, global average pooling results in an improvement by a factor d in statistical efﬁciency when ﬁtting cyclic local functions, compared to H CK .
This improvement was already noticed in [7, 39] but in the case of q = d (fully connected neural networks).
For ω < d , a direct application of the theorems in [38] is more challenging because of the mixing of eigenvalues.
In this case, a modiﬁcation of [38], where eigenvalues are not necessary ordered anymore would apply.
However, for simplicity, we present in Appendix C.1 a simpliﬁed kernel with non-overlapping local pooling which we believe captures the statistical behavior of local pooling.
In this case, we show that Theorem 5 holds with n = ( d/ω ) ·
q s − 1+ ν , which interpolates between
Theorem 2 ( ω = 1 ) and Theorem 5 ( ω = d ).
Downsampling: Often pooling is associated with a downsampling operation, which subsample one every ∆ output coordinates.
In Appendix A.4, we characterize the eigendecomposition of H CK ω, ∆ (Proposition 4) and prove for the popular choice ω = ∆ , that downsampling does not modify the cyclic invariant subspace j = d (Proposition 5).
More generally, we conjecture and check numerically that downsampling with ∆ ≤ ω leaves the low-frequency eigenspaces approximately unchanged.
In particular, the statistical complexity of learning low-frequency functions is not modiﬁed by downsampling operation in the one-layer case (while downsampling is potentially beneﬁcial in further layers).
[Figure]
Numerical simulations
In order to check our theoretical predictions, we perform a simple numerical experiment on simulated data.
We take x ∼ Unif( Q d ) with d = 30 , and consider two target functions:
[Equation]
Here f LF , 3 is a cyclic-invariant local polynomial ( f LF , 3 is ‘low-frequency’).
The function f HF , 3 is a high-frequency local polynomial, and is orthogonal to the space of cyclic invariant functions.
On these target functions, we compare the test error of kernel ridge regression with 5 different kernels: a standard inner-product kernel H [Equation]
[Equation]
, y ) = h ( (cid:104) x , y (cid:105) /d ) ; a cyclic invariant kernel H FCGP ( x , y ) (convolutional kernel with global pooling and full-size patches q = d ); a convolutional kernel H CK with patch size q = 10 ; a convolutional kernel with local pooling H CK ω with q = 10 and ω = 5 ; and a convolutional kernel with global pooling H CKGP with q = 10 .
In all these kernels, we choose a common h ( t ) =
(cid:80) i ∈ [5] 0 .
2 ∗ t i which is a degree 5 -polynomial.
In Figure 1, we report the test errors of ﬁtting f LF , 3 (left) and f HF , 3 (right) using kernel ridge regression with these 5 kernels.
We choose a small regularization parameter λ
= 10 − 6 , and the noise level σ ε = 0 .
The curves are averaged over 5 independent instances and the error bar stands for the standard deviation of these instances.
The results match well our theoretical predictions.
For the function f LF , 3 , the sample sizes required to achieve vanishing test errors are ordered as H CKGP <
H CK ω <
H CK <
H FCGP <
H FC and are around the predicted thresholds
q 2 < dq
/ω
<
d 2 < dq 2 < d 3 respectively.
Next we look at the test error of ﬁtting the high frequency local function f HF , 3 .
The test errors of H CK and H FC are the same for f HF , 3 and f LF , 3 : this is because these kernels do not have bias towards either high-frequency or low-frequency functions.
The kernel H CK ω perform worse on f HF , 3 than on f LF , 3 : this is because the eigenspaces of H CK ω are biased towards low-frequency polynomials.
The kernels
H CKGP and H FCGP do not ﬁt f HF , 3 at all (test error greater than or equal to 1 ): this is because the RKHS of these two kernels only contain cyclic polynomials, but f HF , 3 is orthogonal to the space of cyclic polynomials.
Discussion and Future Work
In this paper, we characterized in a stylized setting how convolution, average pooling and downsampling operations modify the RKHS, by restricting it to q -local functions and then biasing the RKHS towards low-frequency components.
We quantiﬁed precisely the gain in statistical efﬁciency of KRR using these operations.
Beyond illustrating the ‘RKHS engineering’ of image-like function classes, these results can further provide intuition and a rigorous foundation for convolution and pooling operations in kernels and CNNs.
A natural extension would be to study the multilayer convolutional kernels in details and consider other pooling operations such as max-pooling.
Another important question is how anisotropy of the data impacts the results of this paper: in particular, it was shown that pre-processing (whitening of the patches) greatly improves the performance of convolutional kernels [6, 46].
A more challenging question is to study how training and feature learning can further improve the performance of CNNs outside the kernel regime.
Acknowledgement
S.M. is supported by NSF grant DMS-2210827.
T.M. is supported by NSF through award DMS2031883 and the Simons Foundation through Award 814639 for the Collaboration on the Theoretical
Foundations of Deep Learning.
T.M. also acknowledge the NSF grant CCF-2006489 and the ONR grant N00014-18-1-2729.
Checklist
For all authors..
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] All mathematical statements are proved in the Appendix.
(b) Did you describe the limitations of your work?
[Yes] See Sections 1.1 and 4.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This is a theoretical work with no societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See main text and Appendix.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A]
Our paper is theoretical and we only provide small numerical illustrations (which are fully mathematically explicit and only require matrix inversions).
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Contents 1 Introduction 1.
Summary of main results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2 Main results 2.
Functions on the hypercube and eigendecomposition of kernel operators . . . . . .
One-layer convolutional kernel . .
Local average pooling and downsampling . .
3 Numerical simulations 4 Discussion and Future Work[-
-]
A Details from the main text[-
-]
A.1 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Convolutional neural tangent kernel . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Local average pooling operation . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.4 Downsampling operation . . .
A.5 Multilayer convolutional kernels . .
A.6 Proofs diagonalization of convolutional kernels . . . . . . . . . . . . . . . . . . .
A.7 Additional numerical simulations . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Generalization error of kernel methods in ﬁxed dimension
1 Bound on kernel methods using Rademacher complexities
.2 Generalization error of KRR
regime
C Generalization error of KRR in high dimension
C
.1 Generalization error of convolutional
C. the assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 Proof of Proposition 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.4 Proof of Theorem 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.5 Proof of Theorem
C.6 Auxiliary results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D Technical background of function spaces on the hypercube
D.1 Fourier basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.
Hypercubic Gegenbauer . .
D.3 Hermite polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.4 Hypercontractivity of uniform distributions on the hypercube
A Details from the main text A.1 Notations
For a positive integer, we denote by [ n ] the set { 1 , 2 , . . . , n } .
For vectors u , v ∈ R d , we denote (cid:104) u , v (cid:105) =
u 1 v 1 + . . .
+ u d v d their scalar product, and (cid:107) u (cid:107) 2 = (cid:104) u , u (cid:105) 1 / 2 the (cid:96) 2 norm.
Given a matrix A ∈ R n × m , we denote (cid:107) A (cid:107) op = max (cid:107) u (cid:107) 2 =1 (cid:107)
Au (cid:107)
its operator norm and by (cid:107) A (cid:107) F = (cid:0) (cid:80) i,j A 2 ij (cid:1) 1 / 2 its Frobenius norm.
If A ∈ R n × n is a square matrix, the trace of A is denoted by
Tr( A ) =
(cid:80) i ∈ [ n ] A ii .
We use O d ( · ) (resp.
o d ( · ) ) for the standard big-O (resp.
little-o) relations, where the subscript d emphasizes the asymptotic variable.
Furthermore, we write f = Ω d ( g ) if g ( d ) =
O d ( f ( d )) , and f = ω
d ( g ) if g ( d ) =
o d ( f ( d )) .
Finally, f = Θ d ( g ) if we have both f = O d ( g ) and f = Ω d ( g ) .
We use O d, P ( · ) (resp.
o
d, P ( · ) )
the big-O (resp.
little-o) in probability relations.
Namely, for h 1 ( d ) and h 2 ( d ) two sequences of random variables, h 1 ( d ) =
O d, P ( h 2 ( d ))
if for any ε > 0 , there exists
C ε > 0 and d ε ∈ Z > 0 , such that
[Equation] ≤ ε, ∀ d ≥ d ε , and respectively: h 1 ( d ) =
o d, P ( h 2 ( d )) , if h 1 ( d ) /h 2 ( d ) converges to 0 in probability.
Similarly, we will denote h 1 ( d ) =
Ω d, P ( h 2 ( d )) if h 2 ( d ) =
O d, P ( h 1 ( d )) , and h 1 ( d ) =
ω d, P ( h 2 ( d )) if h 2 ( d ) =
o d, P ( h 1 ( d )) .
Finally, h 1 ( d ) =
Θ d, P ( h 2 ( d )) if we have both h 1 ( d ) =
O d, P ( h 2 ( d )) and h 1 ( d ) =
Ω d, P ( h 2 ( d )) .
A.2 Convolutional neural tangent kernel
In this section, we justify the expression of the convolutional neural tangent kernel H CK w , ∆ (CK-AP-DS) , obtained as the tangent kernel of a neural network composed of a one convolution layer followed by local average pooling and downsampling (CNN-AP-DS).
Proposition 3.
Let σ ∈ C 1 ( R ) be an activation function.
Consider the following one-layer convolutional neural network with ω -local average pooling and ∆ -downsampling:
[Equation]
Let a 0 ik ∼ i.i.d.
N (0 , 1) and √ q w 0 i ∼ i.i.d.
Unif( Q q ) independently, and Θ= { ( a 0 ik ) i ∈
[ N ] ,k ∈ [ d/ ∆] , ( w 0 i )
i ∈
[ N ] } .
Then there exists h : [ − 1 , 1] → R , such that for any x , y ∈ Q d , we have almost surely
[Equation]
Proof of Proposition 3.
For u [Equation] v ∈ Q q
[Equation]
deﬁne [Equation] The functions h (1) , h (2) are well deﬁned (the RHS only depend on the inner product (cid:104) u , v (cid:105) ) and can be extended to functions h (1) , h (2) : [ − 1 , 1] → R .
Computing the derivative of the convolutional neural network with respect to a = ( a 0 ik ) i ∈
[ N ]
[Equation]
k ∈ [ d/ ∆] , we have
[Equation]
Hence by law of large number, we have almost surely lim N →∞
[Equation]
Similarly, computing the derivative with respect to √ q W = ( √ q w 0 i ) i ∈
[ N ] gives
[Equation]
By law of large number, using that a ik and a ik (cid:48) are independent of mean zero and variance 1 , we get almost surely lim N →∞
[Equation]
Taking h = h (1) + h (2) concludes the proof.
A.3 Local average pooling operation
Consider a function f ∈ L 2 ( Q d ) : we can decompose it as
[Equation] [Equation]
where ρ j = e 2 iπjd and t k · x = ( x k +1 , . .
, x d , x 1 , . . .
, x k ) is the cyclic shift of x by k pixels.
We can think about f j ( x ) as the j -th component of the discrete Fourier transform of the function f ( x ) seen as a d -dimensional vector { f ( t k · x ) } k ∈ [ d ] for any x ∈ Q d .
Notice furthermore that if f is a local function, i.e., f can be decomposed as a sum of functions on patches f ( x ) = (cid:80)
k ∈ [ d ] g k ( x ( k ) ) , then we can write
[Equation]
where we denoted ( v ∈ Q q )
[Equation]
In particular, decomposing ˜ g j in the Fourier basis, we get (denoting c S = (cid:104)
˜
g j , Y S (cid:105) L 2 ),
[Equation]
which shows that the j -th frequency component f j is in the span of { Y j,S } S ⊆
[ q ] .
In particular, applying average pooling operation in the kernel will reweight this eigenspace by a factor κ j .
Let us further comment on the values of κ j .
First, we have
[Equation]
In particular, the maximal eigenvalue is attained at j = d with κ d = ω , which corresponds to the subspace of cyclic invariant functions.
Furthermore, κ j =
if and only if d is a divisor of jω for j ≤ d − 1 , i.e., j is a multilple of gcd ( ω, d ) .
There are gcd ( ω, d ) − 1 such zero eigenvalues.
In convolutional kernels, a weighted average is often preferred to local average pooling [6, 35, 36]: in that case we consider τ : R → R and obtain the kernel
[Equation]
where d ( s ) =
min( s, d − s ) (the distance between k + s and k on [ d ] with cyclic convention).
Note that H CK τ has the same eigendecomposition as H CK ω but with different weights κ j .
A popular choice for τ is the Gaussian ﬁlter τ ( x ) = 1 √ 2 πσ e − x 2 2 σ 2 .
In Figure 2, we compare the eigenvalues κ j for local average pooling and Gaussian ﬁlter with different value of ω and σ 2 .
Note that the eigenvalue decay controls how much high-frequencies are penalized: faster decay induces heavier penalty on the high-frequency components.
[Figure]
A.4 Downsampling operation
As mentioned in the main text, a downsampling operation is often added after pooling.
The kernel is given by
[Equation]
Let us introduce the family { M r } r ∈
[ q ] of block-circulant matrices deﬁned by
[Equation]
where we introduced the set of indices
[Equation]
We can now state the eigendecomposition of H CK ω, ∆ in terms of the eigenvalues and eigenvectors of the matrices { M r } r ∈ [ q ] .
Proposition 4 (Eigendecomposition of H CK ω, ∆ ) .
Let H CK ω, ∆ be a convolutional kernel with local average pooling and downsampling, as deﬁned in Eq.
Then H CK ω, ∆ admits the following eigendecomposition:
[Equation]
where ψ ∆ j,S ( x ) = (cid:80) dk
=1 v Sj,k Y k + S ( x ) with { κ Sj , v Sj } j ∈ [ d ] eigenvalues and eigenvectors of M γ ( S ) .
Let us make a few comments on these matrices M γ ( S ) .
First because they only depend on S through the diameter γ ( S ) , the eigenvalues and eigenvectors { κ Sj , v Sj } j ∈ [ d ] only depend on γ ( S ) .
Second, we see that M γ ( S ) ( i +∆)( j +∆) =
M γ ( S ) ij and M γ ( S ) ij = 0 if d ( i, j ) ≥ ω , where d ( i, j ) =
min( | i − j | , d − | i − j | ) (i.e., the distance between i and j on the torus [ d ] ).
In words M γ ( S ) is a symmetric block-circulant matrix with non-zero elements on a band of size ω − 1 on the left and right of the diagonal, and on the upper-right and lower-left corners.
Furthermore, notice that
[Equation]
, which is independent of ω, ∆ , γ ( S ) and justify the chosen normalization.
In particular, this implies that (for ξ q, 0 = 0 )
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
is also independent of the parameters ( q, ω, ∆) .
Example 1.
Take ∆ = 3 , ω = 5 , q = 11 , then
M 1 = 3     18 15   15 19 15   . . .
     , and
M 4 = 3  13    11 14  3 11 13 5       . . .
  .
Remark 1.
Symmetric block-circulant matrices can be easily diagonalized as follows.
Consider
M =
Circulant ( B 1 , B 2 , . .
B m ) where B k ∈ R ∆ × ∆ , B T 1 = B 1 and B 2+ k =
B T m − k for k = 0 , . . .
, m − 2 .
Denote ρ j = e 2 iπj/m and γ j ( v ) =
[ v , ρ j v , · · · , ρ m − 1 j v ] / √ m ∈ R m ∆ for any v ∈ R ∆ .
Introduce for j = 0 , . . .
, m − 1 , the matrix H j ∈ R ∆ × ∆ given by
[Equation]
The matrix H j is Hermitian
and we denote ( λ j,s ) s ∈ [∆] and ( v j,s ) s ∈ [∆] its eigenvalues and eigenvectors.
Then the eigenvalues and eigenvectors of M are given by { λ j,s } j ∈ [ m ] ,s ∈
[∆] and { γ j ( v j,s ) } j ∈ [ m ] ,s ∈ [∆] .
In particular, if ∆ = 1 and M = Circulant ( b 1 , b 2 , . . .
, b m ) is a circulant matrix, then the eigenvalues are simply given by
[Equation]
and eigenvectors v j = [1 , ρ j , · · · ,
[Equation]
Here we will focus on the impact of downsampling for single-layer convolutional kernels.
We expect the downsampling operation to have a much more important role for the next layers: for example, increasing the scale of interactions or reducing the dimensionality of the pixel space.
We will argue below that adding a downsampling operation after local pooling leaves the lowfrequency components approximately unchanged (while potentially modifying the high-frequency eigenspaces).
We consider ∆ ≤ ω : for ∆ > ω , some basis functions Y S with S ∈ E (cid:96) are in the null space of H CK ω, ∆ , which impact all frequencies.
To emphasize the dependency on ω, ∆ , denote M rω, ∆ the matrix (19).
We will study the change in the matrix M rω, 1 when adding downsampling ∆ , and consider
[Equation]
24) where we denote A rω, ∆ =
M rω, ∆ − M rω, 1 .
Notice that A rω, ∆ is a symmetric block-circulant matrix.
Therefore, from Remark 1, the eigenvectors of A rω, ∆ are given by { γ j ( v j,s ) } j ∈ [ m ] ,s ∈
[∆] where d = m ∆ and γ j ( v j,s ) =
[ v j,s , ρ m,j v j,s , . . .
, ρ m − 1 m,j v j,s ] with ρ m,j = e 2 iπjm and ( v j,s ) s ∈
[∆] eigenvectors of H j (23).
The eigenvectors of M rω, 1 are given by u t = [1 , ρ d,t , · · · , ρ d − 1 d,t ] / √ d with ρ d,t = e 2 iπtd .
Notice that
[Equation]
which is 0 except when t ≡ j [ m ] .
Hence, we see that A rω, ∆ in Eq.
(24) only modify the eigenspaces of M r ω, 1 as follows: the eigendirections { γ j ( v j,s ) } j ∈ [ m ] ,s ∈ [∆] coming from H j (23) only modify the eigenspaces of M rω, 1 spanned by { u am + j } a =0 ,..., ∆ − 1 .
For simplicity, we will focus on the popular choice ∆ = ω .
Furthermore, we will only look at the impact of the eigenvalues H 0 on the eigenspace spanned by { u am } a =0 ,..., ∆ − 1 , which contain the cyclic invariant direction.
We show below that H
= 0 and therefore A rω,ω does not modify the cyclic invariant eigenspace of M rω, 1 :
Proposition 5.
Consider d = mω and the symmetric block-circulant matrix
A rω,ω = M rω,ω − M rω, 1 .
Denote A rω,ω =
Circulant ( B 1 , B 2 , . .
B m ) and
H
= B 1 + . . .
+ B m .
We have the following properties:
(a) If q + 1 − r ≡
[ ω ] , then A rω,ω = 0 , and downsampling does not modify the matrix M rω, ω =
M rω, 1 . (b) We have H
=
and downsampling does not modify the cyclic invariant eigenspace A rω,ω 1 = 0 .
Proof of Proposition 5.
Let us ﬁrst start by proving point (a).
Consider q + 1 − r = pω .
Fix i ∈ { 0 , . . .
, ∆ − 1 } and κ ∈ { 0 , . . .
, ω − 1 } .
Let us compute the entry ( i, i + κ ) of the matrix
M rω,ω : this amounts to counting the number of quadruples ( k, s, s (cid:48) , t ) with k ∈ [ d/ω ] , s, s (cid:48) ∈
[ ω ] and 0 ≤ t ≤ pω − 1 , satisfying ( kω + s + t, kω + s (cid:48) + t ) ≡ ( i, i + κ )[ d ] .
Notice that we must have s (cid:48)
= s + κ and therefore s ∈ { 0 , . . .
, ω − 1 − κ } .
Notice that for each interval uω ≤ t < ( u + 1) ω with u ∈ { 0 , . .
. , p − 1 } , there are exactly ω − κ ways of choosing s and then t and k to satisfy the equality.
We deduce that
[Equation]
By symmetry of M rω,ω , this concludes the proof of point (a).
Consider now point (b).
First notice, because M rω,ω has zero entries for min( | i − j | , d −| i − j | ) ≥ ω , the only non-zero blocks are B 1 , B 2 and B m .
Furthermore, when computing H 0 , the diagonal entries only have one contribution from the diagonal elements of B 1 .
The off-diagonal elements of
H 0 have two contribution: one from B 1 and one from B 2 (if below the diagonal) or B m (if above the diagonal), i.e.,
[Equation]
Let us compute ﬁrst the diagonal elements: we have easily, by a similar argument as above ( M rω,ω ) ii = 1 =
( M rω, 1 ) ii , and therefore H 0 has zero zero diagonal entries.
For off-diagonal elements, ﬁrst notice that ( M rω,ω ) i
( i + κ − ω )
=
( M rω,ω ) i ( i + ω − κ ) .
Then for q + 1 − r =
pω + v , we can consider each subsegment uω ≤ t < ( u +1) ω separately, and by a simple counting argument, get ( M rω,ω ) i
( i + ω − κ )
+ ( M rω,ω )
i ( i + κ ) = 1 − κω .
We deduce that ( H 0 )
i ( i + κ ) = 0 , which by symmetry implies H
=
and concludes the proof.
From the above result, we conjecture that more generally, for ∆ ≤ ω , the low-frequency eigenspaces of H CK ω remain approximately unchanged when applying a downsampling operation.
We verify this conjecture numerically in several examples.
In Figure 3, we plot the eigenvalues κ j with and without downsampling.
On the left, we compare κ j for ﬁxed ω = 25 and increasing ∆ .
We notice that the eigenvalues do not change much for ∆ ≤ ω , and for ∆ > ω , some κ j become null, as discussed above.
On the right, we plot κ j for ∆ = 1 (continuous line) and ∆ = ω
(dashed lines) for several ω .
As conjectured, the top eigenvalues (low-frequency) are left approximately unchanged.
In Figure 4, we plot a heatmap of the eigenvectors ordered vertically from highest associated eigenvalue (bottom) to lowest (top) for a ﬁxed ω = 25 and increasing downsampling ∆ ∈ { 1 , 25 , 40 } .
First indeed check that the top eigenvectors correspond to low-frequency functions and the bottom eigenvectors correspond to high-frequency functions.
Second, most eigenvectors are not much modiﬁed between ∆ = 1 and ∆ = ω = 25 .
For the case, ∆ > ω , the top eigenvectors corresponds still low-frequency functions.
[Figure] [Figure]
From these observations, we expect H CK ω, ∆ to have the same statistical properties as H CK ω when learning low-frequency functions.
In Figure 5, we plot the test error of kernel ridge regression for ﬁtting cyclic q -local polynomials (see Section A.7) on the hypercube of dimension d = 30 .
We report the test error of one realization, against the sample size n , and choose regularization λ = 10 − 6 and noise σ ε = 0 .
We compare kernels with and without downsampling.
On the left, we consider q =and ω =
∆ = 5 , and compare the test error with H CK ω (continous line) and with H CK ω, ∆ (dashed line) when learning degree 2 , 3 and 4 polynomials.
On the right, we ﬁx the target function to be the cubic local cyclic polynomial and consider the test error of learning with H CK ω, ∆ for q = 10 , ω = 10 , and ∆ ∈ { 1 , 3 , 6 , 10 } .
As expected, we observe in both simulations that the test error is almost identical between the kernels with and without downsampling, when learning cyclic invariant functions.
In Section C.1, we further check that downsampling with ∆ > ω does not improve the highdimensional predictions for the test error of KRR.
[Figure]
A.5 Multilayer convolutional kernels
For completeness, we brieﬂy discuss here some intuitions of multilayer convolutional kernels.
The beneﬁt of depth in convolutional kernels has been investigated in [6, 16, 40, 43].
In particular, [6] observed that the top layer operation of a two-layers convolutional kernel can be replaced by a low-degree polynomial without a performance change.
As an example, we will consider a two layers convolutional kernel with patch and local average pooling sizes ( q 1 , ω 1 ) on the ﬁrst layer and ( q 2 , ω 2 ) on the second layer.
We consider a general inner-product kernel for the ﬁrst layer:
[Equation]
where the feature map is given explicitly ψ ( u ) = { ξ q 1 , | S | Y S ( u ) } S ⊆
[ q 1 ] ∈ R 2 q 1 .
Following the work [6], we consider a degree- 2 polynomial kernel on the second layer, i.e., h 2 ( (cid:104) φ, φ (cid:48) (cid:105) )
= (cid:104) φ, φ (cid:48) (cid:105) 2 .
Let us decompose this two-layers convolutional kernel in the Fourier basis.
Let Ψ( x ) =
{ Ψ k ( x ) } k ∈ [ d ] be the output of the ﬁrst layer, with [Equation] ψ
[Equation]
Then denoting Ψ ( k ) ( x )
=
(Ψ k +1 ( x ) , . . .
, Ψ k + q 2 ( x )) , the two-layers convolutional kernel is given by
[Equation]
We believe that techniques contained in this paper can be used to study kernels of the type ( ?? )
by a careful combinatorial argument and a 2-dimensional Fourier transform on the second layer (see [6]).
We leave this problem to future work.
Here we only comment on the structure of H 2 CK ω 1 ,ω 2 : 1.
Including a second convolutional layer allows interactions between patches
The associated RKHS, which we will denote H 2 CK
contains all the homogeneous polynomials Y S with S = S 1 ∪ S 2 with S 1 , S 2 contained on segments of size q 1 , with the two segments separated by at most q 2 + ω 2 − 2 .
In words, the RKHS contains interaction between patches x ( k ) and x ( k (cid:48) ) that are within some distance.
The eigenvalue associated to a degree- k homogeneous polynomials is still of order q − k in high-dimension.
To learn functions restricted to L 2 ( Q 2 , Loc q ) , it is statistically more efﬁcient to use H CK (smaller degeneracy of eigenvalues).
However H 2 CK will ﬁt a richer class of functions with two-patch interactions, while still not being plagued by dimensionality: dim( H 2 CK ) ≤ q 2 d
2 2 q 1 .
Hence we still expect H 2 CK to be much more statistically efﬁcient than a standard inner-product kernel.
Local pooling on the two layers plays different roles: pooling on the ﬁrst layer encourages the interactions to not depend strongly on the relative positions of the patches, while pooling on the second layer penalizes functions that depend on the global position of these interactions.
For more layers and higher degree kernels, one obtain hierarchical interactions of higher-order, with multi-scale absolute and relative local invariances brought by pooling layers
A.6 Proofs diagonalization of convolutional kernels
In this section, we prove the diagonalization of the kernels H CK , H CK ω and H CK ω, ∆ introduced in
Propositions 1, 2 and 4 respectively.
Recall that we can associate to a kernel function H :
X × X → R deﬁned on a probability space ( X , τ ) (assume x (cid:55)→ H ( x , x ) square integrable), the integral operator H : L 2 ( X , τ ) →
L 2 ( X , τ )
[Equation]
By the spectral theorem of compact operators, there exists an orthonormal basis ( ψ j ) j ≥ 1 of L 2 ( X , τ ) and eigenvalues ( λ j ) j ≥ 1 , with nonincreasing values λ 1 ≥ λ 2 ≥ · · · ≥ 0 and (cid:80) j ≥ 1 λ j < ∞ , such that
[Equation]
We ﬁrst prove the diagonalization of H CK ω, ∆ in Proposition 4.
The case of H CK ω and H CK then follows by setting ∆ = 1 , and ∆ = ω = 1 respectively.
Proof of Proposition 4.
Consider the inner-product kernel function h : R → R deﬁned on the hypercube Q q .
By rotational symmetry (see Section 2.1 and Appendix D), h admits the following diagonalization: for any u , v ∈ Q q ,
[Equation]
where ( Y S ) S ⊆
[ q ] is the Fourier basis on Q q , and ξ d,(cid:96) ( h ) is the (cid:96) -th
Gegenbauer coefﬁcient of h in dimension q (see Sections 2.1 or D for background).
Recall that we deﬁned S (cid:96) =
{ S ⊆ [ q ] : | S | = (cid:96) } , the equivalence relation S ∼ S (cid:48) if S (cid:48) is a translated subset of S in [ q ] (without cyclic convention), and C (cid:96) the quotient set of A (cid:96) by ∼ .
For each equivalence class S ∈ C (cid:96) , consider S the unique subset in S that contains 1 .
Then the equivalence class S contains the subsets u + S
= { u + k : k ∈ S } ⊆
[ q ] with u = 0 , . .
, q − γ ( S ) .
By a slight abuse of notations, we will identify S and this subset S .
Below we will denote u + S the translated subset with cyclic convention on [ d ] (e.g., 2 + { 1 , 3 , d − 1 } = { 3 , 5 , 1 } ).
Using Eq.
(29) and that Y S ( x ( k ) )
=
Y k + S ( x ) , we have the following decomposition of H CK ω, ∆ in the
Fourier basis
[Equation]
where we recall the deﬁnition of the set of indices
[Equation]
Note that the diagonalization of the kernel H can be obtained by computing the matrix M =
( M SS (cid:48) ) S,S (cid:48) ⊆
[ d ] ∈ R 2 d × 2 d with M = E x , y [ Y S ( x ) H ( x , y )
Y S (cid:48)
( y )] : if λ j and v j ∈ R 2 d are the eigenvalues and eigenvectors of M , then λ j and ψ j ( x )
= (cid:80) S ⊆
[ d ] v j,S Y S ( x ) are the eigenvalues and eigenvectors of H .
From Eq.
(30), we see 1) the basis functions Y S with γ ( S ) >
q (subset S not contained in a segment of size q ) are in the null space of H CK ω, ∆ , 2) for S, S (cid:48) ⊆
[ d ] with S and S (cid:48) not translations of each other, then E x , y [ Y S ( x ) H CK ω, ∆ ( x , y )
Y S (cid:48)
( y )]
= 0 , and Y S and Y S (cid:48) are contained in orthogonal eigenspaces.
We deduce that it is sufﬁcient to diagonalize H CK ω, ∆ on each of the (orthogonal) subspaces
V S :
= span { Y k + S : k ∈
[ d ] } for 0 ≤ (cid:96) ≤ q and S ∈ C (cid:96) .
For each S
[Equation] , deﬁne M γ ( S ) ∈ R d
[Equation] d the matrix with entries M γ ( [Equation] 1 r ( S ) E x , y
[ Y i + S ( x ) H CK ω, ∆ ( x , y )
Y j + S ( y )] .
From Eq.
(30), we get
[Equation]
32) which concludes the proof of Proposition 4.
We can now prove Propositions 1 and 2 by taking ω
=
∆ = 1 and ∆ = 1 respectively.
Proof of Proposition 1.
Set ∆ = ω =
in Proposition 4.
We get
[Equation]
In this case, M γ ( S ) is simply equal to identity, which concludes the proof.
Proof of Proposition 2.
Set ∆ = 1 in Proposition 4.
We get
[Equation]
where d ( i, j ) is the distance between i and j on the torus [ d ] (i.e., if i > j , d ( i, j )
=
min( i − j, d + j − i ) ).
Hence, M γ ( S ) is a circulant matrix independent of γ ( S ) , which has well known explicit formula for eigenvalues and eigenvectors (see for example Remark 1).
A.7 Additional numerical simulations
Here, we consider a numerical experiment similar to Figure 1.
We consider x ∼ Unif( Q d ) with d = 30 and consider three cyclic invariant target functions:
[Equation]
We consider a higher order polynomial kernel h ( x ) = (cid:80) k ∈
2 · x k than in Figure 1, which should lead to higher self-induced regularization.
We consider the same kernels as before, with q = 10 and ω = 5 .
In Figure 6, we report the test errors of ﬁtting f 2 (top), f 3 (middle) and f 4 (bottom) using kernel ridge regression with the 5 kernels of interests in the main text.
We choose a small regularization parameter λ = 10 − 6 , and the noise level σ ε = 0 .
The curves are averaged over 5 independent instances and the error bar stands for the standard deviation of these instances.
The results again match with our overall theoretical predictions.
We report the predicted thresholds for the three functions:
For f 2 target: q < d < dq/ω < dq < d 2 for H CKGP <
H FCGP <
H CK ω <
H CK < H FC .
For f 3 target:
q 2 < dq 2 /ω <
d 2 < dq 2 < d 3 for H CKGP <
H CK ω <
H CK < H FCGP <
H FC .
For f 4 target
: q 3 < dq 3 /ω <
d 3 < dq 3 < d 4 for H CKGP <
H CK ω <
H CK < H FCGP <
H FC .
We see that the kernels, especially for f 4 , perform much better than their theoretical high-dimension predictions: this can be explained by the low-dimensionality of the experiment where q = 10 .
[Figure]
B Generalization error of kernel methods in ﬁxed dimension B.1 Bound on kernel methods using Rademacher complexities
We ﬁrst consider the case of a Lipschitz bounded loss and uniform convergence, and make a few simple remarks on the connection between generalization error and eigendecomposition in kernel methods.
Consider i.i.d data ( x i , y i )
∈
X × R with ( x , y ) ∼ P and a loss function (cid:96) : R × R → R that we take 1 -Lipschitz w.r.t second argument and bounded by 1 .
The goal is to minimize the expected loss
L ( ˆ f ) =
E y , x { (cid:96) ( y, ˆ f ( x )) } .
Take a RKHS H with kernel function H :
X × X → R and consider following constrained empirical risk minimizer:
[Equation]
The generalization error of ˆ f B has the following standard bound on the Rademacher complexity of the kernel class { f : (cid:107) f (cid:107) H ≤ B } [9, 44]: with probability 1 − δ ,
[Equation]
Note that instead of a constraint on the norm in Eq.
(33), one might ﬁnd more convenient to use a penalty.
In that case, there exists an equivalent to the bound (34) [3, 47], but we focus here on the constrained formulation for simplicity.
From the bound (34), we see that the generalization error depends crucially on the choice of B .
For simplicity, let us forget about the approximation error and take (cid:107) f (cid:63) (cid:107) H ≤ B where f (cid:63) =
E { y | x } .
Recall that for a kernel H with eigenvalues { λ j } j ≥ 1 and eigenvectors { ψ j } j ≥ 1 , we have
[Equation]
Consider H CK ω, ∆ as in Eq.
(8) and assume ξ q, 0 = 0 .
From the normalization choice of the kernel (see
Eq.
(22)), we have
[Equation]
Consider now for simplicity ∆ = 1 .
From the eigendecomposition in Proposition 2, the RKHS norm of f ∈ L 2 ( Q d , Loc q ) is given by
[Equation]
Consider the case where f ∈ L 2 ( Q d , Loc q ) has a unique non-zero component in its discrete Fourier transform,
i.e., f ( x ) = 1 √ d (cid:80) k ∈ [ d ] ρ kj g ( x ( k ) ) with E { g ( x ) } = 0 and ρ j = e 2 iπj/d (see Section A.3).
Note that, denoting c S = (cid:104) Y S , g (cid:105)
L 2 ( Q q ) :
[Equation]
Hence,
[Equation]
(cid:88) (cid:96) = (cid:88) S ∈C (cid:96) (cid:104)
[Equation]
(cid:105)
[Equation]
(cid:96)
[Equation]
(cid:88) (cid:96)
[Equation]
(cid:88)
S
[Equation]
(cid:96) r ( S
[Equation]
(cid:88) u
=
[Equation]
+
[Equation] q [Equation] (cid:96) r ( S )
[Equation]
{+d (cid:107) g (cid:107) 
 h κ j , where (cid:107) g+} (cid:107) 2 h is the RKHS norm associated to the inner-product kernel h : R → R in Q q , i.e., (cid:107) g (cid:107) {+
+} h
= (cid:80) S ⊆
[ q ] c S ξ q, | S | .
From the bound (34), we deduce the ﬁrst generalization bound using a convolutional kernel: with probability at least 1 − δ ,
[Equation]
We make the following two remarks on this bound:
It depends on (cid:107) g (cid:107) h , which is a RKHS norm on Q q instead of Q d , which has potentially much lower dimension and contain less smooth function for balls of same radius.
There is a factor κ j gain in sample complexity when learning functions that have j -th frequency with κ j > 1 .
In particular, for j = d (cyclic invariant functions), κ j = ω , and we need ω less samples to get the same (upper) bound on the generalization error.
On the contrary, when κ j < 1 , i.e., high-frequency oscillatory functions, the generalization bound becomes worse.
B.2 Generalization error of KRR in the classical regime
We consider here the regression setting which allows for ﬁner results.
Several works have considered bounding the generalization error of kernel ridge regression (KRR) [12, 29], [47, Theorem 13.17].
In this section, we consider the following fully-explicit upper bound from [3].
Consider i.i.d data ( x i , y i )
∈
X × R with x i ∼ P , and y i =
f (cid:63) ( x i )
+ ε i .
Assume the noise E [ ε i | x i ] = 0 and E [ ε 2 i | x i ] ≤ σ 2 ε , and denote ε = ( ε 1 , . .
. , ε n ) .
Let H be a RKHS with reproducing kernel H :
X × X → R .
The KRR solution with regularization parameter λ ≥ 0 is given by
[Equation]
which has the following analytical formula:
[Equation]
where H =
( H ( x i , x j )) ij ∈ [ n ] is the empirical kernel matrix, h ( x ) =
[ H ( x , x 1 ) , . . .
, H ( x , x n )] and y =
( y 1 , . . .
, y n ) .
The risk is taken to be the test error with squared error loss
[Equation]
Below, we give an upper bound on the expected risk over the noise ε in the training data, i.e.,
E ε { R ( f (cid:63) , ˆ f λ ) } (it is also possible to give high probability bounds by concentration arguments, but we restrict ourselves to bounding the expected risk).
Theorem 6.
[3, Theorem 7.2] Assume H ( x , x ) ≤ R 2 almost surely and let the regularization parameter λ ≤ R 2 .
If n ≥ 5 R 2 λ (cid:16) 1 + log R 2 λ (cid:17) , then
[Equation]
[Equation]
where N ( H, λ )
=
Tr[( H + λ I ) − 1 H ] .
Let us comment on the upper-bound in Eq.
The ﬁrst term corresponds to an upper bound on the variance: N ( H, λ ) is sometimes called the degrees of freedom or the effective dimension of the kernel
H .
The second term bounds the bias term and corresponds to an approximation error.
In particular, for any r > 0 ,
[Equation]
(37) where we recall that H is the integral operator associated to H (see Eq.
The third term can be removed by a more intricate analysis.
From the above discussion, it is natural to consider the following two assumptions on H and f (cid:63) , that are standard in the kernel literature:
(B1)
Capacity condition: N ( H, λ ) ≤ C H λ − 1 /α with α > 1 . (B2)
Source condition: there exists β > 0 such that (cid:107) H − β/ 2 f (cid:63) (cid:107) 2 L 2 =:
B 2 f (cid:63) < ∞ .
Intuitively, the capacity condition (B1) characterizes the size of the RKHS: for increasing α , the
RKHS contains less and less functions.
It is veriﬁed when the eigenvalues
λ j ’s of H decay at the rate j − α .
For example, taking the Matern kernel of order s > d/ 2 , whose RKHS is the Sobolev space of order s (i.e., functions with bounded s -order derivatives), we have α = 2 s/d (e.g., see [26]).
The source condition (B2) characterizes the regularity of the target function (the ‘source’) with respect to the kernel: β = 1 is equivalent to f (cid:63) ∈ H , while β > 1 corresponds to f (cid:63) more smooth (and β <less smooth f (cid:63) ).
Assuming (B1) and (B2) in Theorem 6, we get the bound
[Equation]
where in the second line, we balanced the two terms by taking λ ∗ :
= (cid:16)
[Equation]
f(cid:63) n (cid:17) α αβ
+1 .
Note that in order to use Theorem 6, we need further to constrain n ≥ 5 R 2 λ (cid:16)
+ log R 2 λ (cid:17) .
For simplicity, we will choose r > α − 1 α , so that this condition is veriﬁed for n sufﬁciently large.
Remark 2.
The rate in n in Eq.
(38) is minmax optimal over all functions that verify assumptions (A1) and (A2) [12].
However, for large d , the RKHS is composed of very smooth functions (e.g.,
Sobolev spaces of order s are RKHS if and only if s > d/ 2 , i.e., if the order of the bounded derivatives grows with the dimension d ) and β will be small, such that βα
≈ κ/d for functions with bounded derivatives up to order κ .
In that case, the risk decreases at the rate n − O ( κd ) : KRR suffers from the curse of dimensionality when κ does not scale with d .
As a consequence, the bound (38) is vacuous when n does not scale exponentially in d , which led several groups to derive ﬁner bounds on KRR in the high dimensional regime (see Section C).
Let us now apply Theorem 6 and Eq.
(38) to our convolutional kernels to show Theorems 1 and 4.
Proof of Theorem 1.
First notice that H CK ( x , x ) = h (1)
=:
R 2 and we can therefore apply Theorem 6.
The effective dimension of H CK is bounded by
[Equation]
where we used that r ( S ) ≥ 1 in the second line and N ( h, λ ) is the effective dimension of the innerproduct kernel h on Q q .
We deduce from (A1) that N ( H CK , λ ) ≤ C h
d 1 − 1 /α λ − 1 /α .
Furthermore, from (A2) and the assumption that E { g k ( x ) } = 0 , we have
[Equation]
Injecting the two above bounds in Eq.
(38), we deduce that there exists constants C 1 , C 2 , C 3 that only depends on the constants in (A1) and (A2), and h (1) , σ 2 ε (but independent of d ), such that taking n ≥ [Equation] max
[Equation]
f
[Equation]
, d ) and λ ∗ =
C 2 d ( d/n ) α αβ +1 , we get
[Equation]
Proof of Theorem 4.
The proof is similar to the proof of Theorem 1.
Notice that H CK ω ( x , x ) ≤ h (1) , and that the effective dimension of H CK ω is bounded by
[Equation]
where we used condition (A1).
Denoting d eff = (cid:80) dj
=1 ( κ j /ω ) 1 /α , the rest of the proof follows from the proof of Theorem 1 with d replaced by d eff ω 1 /α and B 2 replaced by ω β B 2 .
Remark 3.
Note that the requirement (cid:107) ( H CK ω /ω ) − β/
f (cid:63) (cid:107) L 2 ≤ B is to make the result comparable to the other theorems when we consider target functions with low-frequencies.
For a cyclic invariant function, we get exactly (cid:107) ( H CK
[Table]
C Generalization error of KRR in high dimension
In Section B.2, we considered upper bounds on the test error of KRR using the standard capacity and source conditions .
However, these results suffer from several limitations:
They only provide an upper bound on the test error.
While the decay rate with respect to n is minmax optimal (see [12]), this is not strong enough to show, for example, a statistical advantage of using local average pooling, which appears as a prefactor d eff , and which would require a lower bound matching the upper bound within a constant factor.
As mentioned in Remark 2, the bound is of order n − 1 /O
( d ) , except when the target function has smoothness order increasing with d .
This bound is non-vacuous only if n = exp
( O ( d )) which is impractical in modern image datasets where typically d ≥ 100 .
This motivates a new type of question: given n (cid:16) d α , what is the prediction error achieved by KRR for a given function?
In order to achieve the bound Eq.
(38), one need to carefully balance the bias and the variance terms by setting the regularization parameter.
This is in contrast with modern practice which usually train until interpolation (which corresponds to setting λ → 0 ).
Given the above limitations, several recent works have instead considered a high-dimensional setting where the number of samples scales with d , and derived asymptotic test errors, exact up to a vanishing additive error [23, 24, 38].
In addition to these works, several papers have derived general estimates for the test error using non-rigorous methods [11, 17, 29] that are believe to be correct in the high dimensional limit and which show great agreement with numerical experiments.
The picture that emerges in this regime is much more precise than in the classical regime: KRR approximately acts as a shrinkage operator on the target function (not assumed to be in a particular space anymore), with shrinkage parameter that scales as a self-induced regularization parameter over the number of samples.
More precisely, [38] shows the following: considers a kernel H d : R d × R d → R with eigenvalues ( λ d,j ) j ≥ 1 in nonincreasing order and n ≡ n ( d ) the number of samples.
Let m ≡ m ( d ) be an integer such that m ≤ n 1 − δ and
[Equation]
for some δ > 0 .
Then, assuming some additional conditions insuring that the kernel H d is ‘spread-out’ and well behaved, the KRR solution
[Equation]
is equal up to a vanishing additive L 2 -error (as d → ∞ ) to the following effective ridge regression estimator
[Equation]
where λ eff = λ + (cid:80) ∞
j = m +1 λ d,j .
The effective estimator (40) amounts to replacing the empirical risk in Eq.
(39) by its population counterpart (cid:107) f
[Equation]
− f
[Equation]
2 L 2 = E x { ( f
[Equation]
( x ) − f ( x )) 2 } .
In words, in high dimension, KRR with a ﬁnite number of samples is the same as KRR with inﬁnite number of samples but with a larger ridge regularization.
The solution of Eq.
(40) admits an explicit solution in terms of a shrinkage operator in the basis ( ψ d,j ) j ≥ 1 of eigenfunctions of H d :
[Equation]
Hence, KRR will ﬁt better the target function along eigendirections associated to larger eigenvalues of H .
If λ d,j (cid:29) λ eff /n , KRR ﬁts perfectly f (cid:63) along the eigendirection ψ d,j , while if λ d,j (cid:28) λ eff /n ,
KRR does not ﬁt this eigendirection at all.
This phenomena has been referred as the spectral bias and task-kernel alignment of kernel ridge regression in several works.
Finally, notice from Eq.
(41) that the minimum test error is achieved for the regularization parameter λ = 0 , which corresponds to the KRR estimator ﬁtting perfectly the training data.
In other words, the interpolating solution is optimal for kernel ridge regression in high dimension.
C.1 Generalization error of convolutional kernels in high dimension
Consider a sequence of integers { d ( q ) } q ≥ 1 which corresponds to a sequence of image spaces x ∈ Q d of increasing dimension, and assume d ( q ) / 2 ≥ q ≥ d ( q ) δ for some constant δ > 0 .
For ease of notations, we will keep the dependency on q implicit, i.e., d := d ( q ) .
Let { h q } q ≥ 1 be a sequence of inner-product kernels
h q : R → R .
Test error with one-layer convolutional kernel: we ﬁrst consider a vanilla one-layer convolutional kernel H CK as deﬁned in Eq.
We will assume that the kernels { h q } q ≥ 1 verify the following ‘genericity’ condition.
Assumption 1 (Generecity assumption on { h q } q ≥ 1 at level s ∈ N ) .
For { h q } q ≥ 1 a sequence of inner-product kernels h q : R → R , we assume the following conditions to hold.
There exists s (cid:48) ≥ 1 /δ + 2 s + 3 where δ > 0 veriﬁes q ≥ d δ and a constant C such that h q (1) ≤ C , and
[Equation] [Equation]
Assumption 1 will be veriﬁed by standard kernels, e.g., the Gaussian kernel.
We discuss this assumption in Section C.2 and present sufﬁcient conditions on the activation function σ for its associated CNTK to verify Assumption 1.
Recall that we denoted L 2 ( Q d , Loc q ) the space of local functions, i.e., that can be decomposed as f ( x ) = (cid:80)
k ∈ [ d ] f k ( x ( k ) ) .
Denote h q,>(cid:96)
the inner-product kernel h q with its ( (cid:96) + 1) -ﬁrst Gegenbauer coefﬁcients set to 0 , i.e.,
[Equation]
for any u , v ∈ Q q .
The following result is a consequence of the general theorem on the generalization error of KRR in [38].
Theorem 7 (Test error of CK in high dimension) .
Let { f d ∈
L 2 ( Q d , Loc q ) } q ≥ 1 be a sequence of local functions.
Let ( x i ) i ∈
[ n ( d )] ∼ i.i.d.
Unif( Q d ) and y i =
f d ( x i ) + ε i with ε
i ∼ i.i.d.
N (0 , σ 2 ε ) .
Assume d · q s − 1+ δ ≤ n ≤ d · q s − δ for some δ > 0 and let { h q } q ≥ 1 be a sequence of activation functions satisfying Assumption 1 at level s .
Consider { H CK ,d } q ≥ 1 the sequence of convolutional kernels associated to { h q } q ≥ 1 as deﬁned in Eq.
Then the following holds for the solution ˆ f λ of
KRR with kernels { H CK ,d } q ≥ 1 .
For any regularization parameter λ ≥ 0 , deﬁne the effective regularization λ eff :=
λ + h q,> s (1) .
Then for any η > 0 , we have
[Equation]
The proof of Theorem 7 is deferred to Section C.4.
Let us expound on the predictions of Theorem 7.
First, recall that ˆ f eff λ eff is given explicitly in Eq.
(41) by a shrinkage operator with parameter λ eff .
From Assumption 1 and taking λ = 0 , the shrinkage operator is of order[-
-]
[Equation]
From the eigendecomposition of H CK introduced in Proposition 1, KRR ﬁts perfectly f (cid:63) along the eigendirection Y S with | S | = (cid:96) if n · ξ d,(cid:96) r ( S ) /d
(cid:29) λ eff , while it does not ﬁt this eigendirection at all if n · ξ d,(cid:96)
r ( S ) /d ≤ λ eff .
Consider n = d · q s − 1+ α :
• KRR ﬁts the eigendirections corresponding to the homogeneous polynomials of degree s −and less, and of degree s for subsets S such that γ ( S ) (cid:28) q − q 1 − α .
• KRR does not ﬁt at all the eigendirections correpsonding to homogeneous polynomials of degree s + 1 and larger, and degree s for subsets S such that γ ( S ) (cid:29) q − q 1 − α .
In words, for d · q s − 1 (cid:28) n (cid:28) d · q s , KRR ﬁts at least a degree- ( s − 1) polynomial approximation to f (cid:63) and at most a degree- s polynomial approximation.
As n increases from d · q s − 1 to d · q s , KRR ﬁrst ﬁts degree- s homogeneous polynomials that have smaller diameter γ ( S ) (i.e., ‘more localized’).
Test error of CK with global average pooling: we consider the kernel H CKGP given by a convolutional layer followed by global average pooling:
[Equation]
In addition to the genericity condition, we will assume that the kernels { h q } q ≥ 1 verify the following differentiability condition.
Assumption 2 (Differentiability assumption on { h q } q ≥ 1 at level s ∈ N ) .
For { h q } q ≥ 1 a sequence of inner-product kernels h q : R → R , we assume the following conditions to hold.
There exists v ≥ max(2 /δ, s ) where δ > 0 veriﬁes q ≥ d δ such that h q is ( v + 1) -differentiable
and for k ≤ v ,
[Equation]
where we denoted h q,>v the truncated inner-product kernel h q as in Eq.
Assumption 2 is used to extend the following theorem to non-polynomial kernel h q (in particular, it is trivially veriﬁed for polynomial kernels by taking v larger than the degree of h q ).
This assumption is difﬁcult to check in practice, however we provide some examples where it holds in Appendix C.2.
Recall that we denoted L 2 ( Q d , CycLoc q ) the space of functions that are given by the convolution of a function g : R q → R with the image x ∈ Q d , i.e., f ( x ) =
(cid:80)
k ∈
[ d ] g ( x ( k ) ) .
Theorem 8 (Test error of CK with GP in high dimension) .
Let { f d ∈
L 2 ( Q d , CycLoc q ) } q ≥ 1 be a sequence of convolutional functions.
Assume q s − 1+ δ ≤ n ≤ q s − δ for some δ > 0 and let { h q } q ≥ 1 be a sequence of activation functions satisfying Assumptions 1 and 2 at level s .
Consider { H CK ,d GP } q ≥ 1 the sequence of convolutional kernels with global pooling associated to { h q } q ≥ 1 as deﬁned in Eq.
Then the solution ˆ f λ of KRR with kernels { H CK ,d GP } q ≥ 1 veriﬁes Eq.
(46) with λ eff :=
λ + h q,> s (1) .
The proof of Theorem 8 is deferred to Section C.5.
The predictions of Theorem 8 are similar to the ones of Theorem 7 but with a factor d gain in statistical efﬁciency: this is due to the eigenvalues of H CKGP being a factor d larger than for H CK .
Therefore, with global average pooling, for q s − 1 (cid:28) n (cid:28) q s , KRR ﬁts at least a degree- ( s − 1) invariant polynomial approximation to f (cid:63) and at most a degree- s invariant polynomial approximation.
As n increases from q s − 1 to q s , KRR ﬁrst degree- s invariant homogeneous polynomials with increasing diameter γ ( S ) .
Test error of CK with local average pooling: In the case of local average pooling with ω < d , the eigenvalues are harder to control.
Indeed, we have mixing of the eigenvalues between polynomials of different degree: there exists j, j (cid:48) ∈
[ d ] such that ξ q,(cid:96) κ j (cid:28) ξ q,(cid:96) +1 κ j (cid:48) .
The eigenvalues are not ordered in increasing degree of their associated eigenfunctions anymore.
While this case is potentially tractable with a more careful analysis, we instead introduce a simpliﬁed kernel which we believe qualitatively captures the statistical behavior of local average pooling.
Assume q ≤ ω/ 2 and ω is a divisor of d .
Denote x ( kω ) =
( x kω +1 , . . .
, x kω + ω ) the k -th segment of length ω in [ d ] and x ( kω ) ( i ) =
( x kω + i , . . .
, x kω + q + i ) the patch of size q with cyclic convention in { kω + 1 , . . .
, kω + ω } .
Consider the following convolutional kernel with ‘non-overlapping’ average pooling:
[Equation]
In words, H CK , NO ω is the combination of d/ω non-overlapping convolutional kernels with global average pooling on images of size ω :
[Equation]
where ψ k,S ( x ) = 1 √ ω (cid:80) i ∈ [ ω ] Y i + S ( x ( kω ) ) where i + S is the translated set with cyclic convention in [ ω ] .
Denote L 2 ( Q d , LocCycLoc q ) the RKHS associated to H CK , NO ω , which contains functions that are locally convolutions on segments of size ω .
For this simpliﬁed model, the proof of Theorem 8 can be easily adapted and we obtain the following result:
Corollary 1 (Test error of CK with NO pooling in high dimension) .
Let { f d ∈
L 2 ( Q d , LocCycLoc q ) } q ≥ 1 be a sequence of local convolutional functions.
Assume ( d/ω ) · q s − 1+ δ ≤ n ≤ ( d/ω ) · q s − δ for some δ > 0 and let { h q } q ≥ 1 be a sequence of activation functions satisfying
Assumptions 1 and 2 at level s .
Consider { H CK , NO ,d ω } q ≥ 1 the sequence of convolutional kernels with non-overlapping pooling associated to { h q } q ≥ 1 as deﬁned in Eq.
Then the solution ˆ f λ of KRR with kernels { H CK , NO ,d ω } q ≥ 1 veriﬁes Eq.
(46) with λ eff :=
λ
+ dω h q,> s (1) .
Corollary 1 shows that H CK , NO ω enjoys a factor ω gain in statistical efﬁciency compared to H CK , due to a factor ω smaller effective ridge regularization.
Therefore, with (non-overlapping) local average pooling, for ( d/ω ) · q s − 1 (cid:28) n (cid:28) ( d/ω ) · q s , KRR ﬁts degree- ( s − 1) locally invariant polynomials and none of the polynomials of degree- ( s + 1) and larger.
Heuristically, we see that this yields the same statistical efﬁciency than H CK for ω = 1 and H CKGP for ω
= d , and interpolates between the two cases for 1 < ω < d .
Test error of convolutional kernels with downsampling: We consider adding a downsampling operation to the previous kernels.
Let ∆ be a constant and a divisor of d and ω and consider the following ‘downsampled’ kernels:
[Equation]
We can easily adapt the proofs of Theorems 7 and 8, and Corollary 1 to these kernels.
In particular, their conclusions do not change (for any constant ∆ ) and downsampling do not provide a statistical advantage.
C.2 Checking the assumptions
In this section, we discuss Assumptions 1 and 2 and present sufﬁcient conditions for them to be veriﬁed.
Genericity assumption: Recall that the inner-product kernel h q : R → R has the following eigendecomposition on Q q as
[Equation]
The genericity assumption amounts to: 1)
A universality condition in Eqs.
(42) and (43): if
P k h ( (cid:104) 1 , ·(cid:105) /q )
= 0 , then h does not learn degree- k homogeneous polynomials; 2) A constant order scaling of the self-induced regularization h q,> s (1) , from h q (1) ≤ C and Eq.
(43) with s (cid:48) , i.e., h q,> s (1) ≤ h q (1) =
O q (1) and h q,> s (1) ≥ ξ q, s (cid:48) B ( q, s (cid:48) )
=
Ω q (1) ; 3)
The last eigenvalues decay sufﬁciently fast in Eq.
(44) in order to avoid pathological cases.
For generic kernels, we have typically ξ q,(cid:96) (cid:16) q − (cid:96) (for ﬁx (cid:96) ).
For example, if h is smooth, ξ q,(cid:96) =
q − (cid:96) ( h ( k ) (0) + o q (1)) and it is sufﬁcient to have h ( k ) (0) > 0 .
See Appendix D.2 in [38] for a proof of Eq.
(44) when h is sufﬁciently smooth.
Below, we present instead sufﬁcient conditions on the activation σ such that the induced neural tangent kernel veriﬁes the ‘genericity’ assumption.
More precisely, we display sufﬁcient conditions on the sequence { σ q } q ≥ 1 of activation functions σ q : R → R , such that the induced neural tangent kernels { h q } q ≥ 1 veriﬁes Assumption 1, where h q was derived in Section A.2 and is given by ( u , v ∈ Q q )
[Equation]
(53) where
[Equation]
Assumption 3 (Assumptions on { σ q } q ≥ 1 at level s ∈ N ) .
For { σ q } q ≥ 1 a sequence of functions σ q : R → R , we assume the following conditions to hold.
There exists s (cid:48) ≥ 1 /δ
+ 2 s + 3 where δ > 0 veriﬁes q ≥ d δ , such that
(a) The function σ q is differentiable and there exists c 0 > 0 and c 1 < 1 independent of q , such that | σ q ( x ) | , | σ (cid:48) q ( x ) | ≤ c 0 exp( c 1 x 2 / 2) .
(b) We have
[Equation]
where e ∈ Q q is arbitrary.
(c) We have for a ﬁxed δ >
[Equation]
Proposition 6.
Consider a sequence { σ q } q ≥ 1 of activation functions σ q : R → R that satisﬁes
Assumption 3.
Let { h q } q ≥ 1 be the sequence of neural tangent kernels associated to { σ q } q ≥ 1 as deﬁned in Eq.
Then the sequence { h q } q ≥ 1 satisﬁes the ‘genericity’ Assumption 1.
Differentiability assumption: As mentioned in the previous section, this condition is required in our proof technique to extend Theorem 8 to non-polynomial kernel functions.
While we believe that weaker conditions should be sufﬁcient, we leave checking them to future work.
Note that
Assumption 2 was proved for x ∼ Unif( S d − 1 ( √ d )) and h q ( (cid:104) x , y (cid:105) /q )
=
E w { σ ( (cid:104) x , w (cid:105) )
σ ( (cid:104) y , w (cid:105) ) } for w ∼ Unif( S d − 1 (1)) , given that σ satisﬁes some differentiability conditions, in [39].
C.3 Proof of Proposition
Proof of Proposition 6.
Step 1.
Effective activation function.
Let us decompose both functions σ q and σ (cid:48)
q in the Gegenbauer polynomial on the hypercube basis:
[Equation]
) where (cid:96) ) [Table]
(cid:1)
[Equation]
From the deﬁnition of h (1) q in Eq.
(54) and the eigendecomposition (60), we have
[Equation]
Similarly, from the deﬁnition of h (2) q in Eq.
(55), the eigendecomposition (61) and using Lemmastated below, we get
[Equation]
where
[Equation]
[Equation]
We can therefore deﬁne π q,(cid:96)
= (cid:113) χ 2 q,(cid:96)
+ ζ 2 q,(cid:96) and σ eﬀ ,q ( (cid:104)· , ·(cid:105) / √ q ) : Q q × Q q → R by
[Equation]
, such that the NT kernel (53) can be written as the kernel of the effective activation σ eﬀ ,q :
[Equation]
We will show that h q with Gegenbauer coefﬁcients ξ q,(cid:96) :
= π 2 q,(cid:96) veriﬁes Assumption 1.
Step 2.
Decay of the eigenvalues.
Recall that the sequence { σ q } q ≥ 1 satisﬁes Assumption 3 at level s .
From Assumption 3.
( a ) (for example by adapting the proof of Lemma C.1 in [24] to the hypercube), there exists C > 0 such that
[Equation]
and we deduce that χ 2 q,(cid:96) , κ 2 q,(cid:96) , π 2 q,(cid:96) =
O q ( B ( Q q ; (cid:96) ) − 1 ) .
Using that B ( Q q ; (cid:96) )
= (cid:0) q(cid:96) (cid:1) , we deduce that for any ﬁxed (cid:96) , χ 2 q,(cid:96) , κ 2 q,(cid:96) , π 2 q,(cid:96) =
O q ( q − (cid:96) ) .
Furthermore, from Assumption 3.
( c ) , we have for k = 0 , . . .
, s (cid:48)
+ 1 ,
[Equation]
By Eq.
(62) and the deﬁnition of π 2 q,(cid:96) , we have π 2 q,q − k
= O d ( q − s (cid:48) − 1 ) for any k ≤ s (cid:48) , which veriﬁes
Eq.
(44) in Assumption 1.
Furthermore, by Assumption 3.
( b ) , using that χ 2 q,k = B ( Q q ; k ) − 1 (cid:107) P k σ q (cid:107) 2 L 2 ( Q q ) and ξ 2 q,k ≥ χ 2 q,k , we get
[Equation]
and
[Equation]
In particular, this implies that (cid:107) σ eﬀ ,d,> s (cid:107) 2 L 2 ( Q q ) ≥ (cid:107) P s (cid:48)
σ q (cid:107) 2 L 2 ( Q q ) =
Ω q (1) .
Lemma 1.
Let (cid:96) be an integer such that 0 ≤ (cid:96) ≤ q .
Consider the following Gegenbauer polynomial deﬁned on the q -dimensional hypercube (see Section D): for x , y ∈ Q q ,
[Equation]
where we recall the deﬁnition of the homogeneous polynomial Y S ( x ) = x S =
(cid:81)
i ∈
S x i .
We have
[Equation]
y (cid:105) ) , with the convention Q ( q ) − 1 = Q ( q ) q +1 = 0 .
Proof of Lemma 1.
Consider 1 ≤ (cid:96) ≤ q − 1 .
We have
[Equation]
We have Y S ( x ) x i =
Y S ∪{ i } ( x ) if i (cid:54)∈ S , and Y S ( x ) x i = Y S \{ i } ( x ) if i ∈ S .
Hence, the above sum contains sets of size (cid:96) − 1 and (cid:96) + 1 .
For each set S ⊂ [ q ] with | S | = (cid:96) − 1 , there q + 1 − (cid:96) sets | ˜ S | =
(cid:96) , such that by removing one element we can obtain S .
For each set S ⊂ [ q ] with | S | = (cid:96) + 1 , there (cid:96) + 1 sets | ˜ S
| =
(cid:96) , such that by adding one element we can obtain S .
We deduce that
[Equation]
+
[Equation] qB
[Equation]
Using B ( Q q ; (cid:96) )
= (cid:0) q(cid:96) (cid:1) , we obtain
[Equation]
The cases (cid:96) = 0 and (cid:96) =
q are straightforward.
C.4 Proof of Theorem
Let { d ( q ) } q ≥ 1 be a sequence of integers with 2 q ≤ d ( q ) ≤ q 1 /δ for some δ > 0 .
We will denote d = d ( q ) for simplicity.
Consider x ∼ Unif( Q d ) , dq s − 1+ δ ≤ n ≤ dq s − δ for some δ > 0 and a sequence of inner-product kernels { h q } q ≥ 1 that satisﬁes Assumption 1 at level s .
We consider the vanilla one-layer convolutional kernel
[Equation]
Theorem 7 is a consequence of Theorem 4 in [38] where we take X d = Q d , ν d = Unif( X d ) and
D d = L 2 ( Q d , Loc q ) ⊂
L 2 ( Q d ) .
The proof amounts to checking that { H CK ,d } q ≥ 1 veriﬁes the kernel concentration properties and eigenvalue condition (see Section 3.2 in [38]).
We borrow some of the notations introduced in [38] and we refer the reader to their Section 2.1.
Proof of Theorem 7.
Step 1.
Diagonalization of the kernel and choosing m = m ( q ) .
From Proposition 1, we have the following diagonalization of H CK ,d :
[Equation]
where r ( ∅ ) =
d and r ( S ) =
q + 1 − γ ( S ) for S ⊂ [ q ] \ {∅} , and we recall E (cid:96) =
{ S ⊆
[ d ] : | S | = (cid:96), γ ( S ) ≤ q } .
Using that B ( Q q ; (cid:96) )
=
Θ q ( q (cid:96) ) , ξ q,
[Equation]
B ( Q q ; (cid:96) ) ≤ h q (1) and Assumption 1, we have
[Equation]
sup
[Equation]
Further deﬁne E (cid:96),h = { S ∈ E (cid:96) : γ ( S )
= h } for h
= (cid:96), . . .
, q .
It is easy to check that |E (cid:96),h | =
d (cid:0) h − 2 (cid:96) − 2 (cid:1) and |E (cid:96) | =
q (cid:88) h
= (cid:96) |E (cid:96),h |
=
d q (cid:88) h
= (cid:96) (cid:18) h −(cid:96) −(cid:19)
=
d (cid:18) q −(cid:96) −(cid:19) , and therefore |E (cid:96) | =
Θ q ( d · q (cid:96) − 1 ) .
Denote { λ q,j } j ≥ 1 the eigenvalues { ξ q,(cid:96) r ( S ) /d } (cid:96) =0 ,...,q ; S ∈E (cid:96) in nonincreasing order, and { ψ q,j } j ≥ 1 the reordered eigenfunctions.
Set m to be the number of eigenvalues such that λ q,j > qξ q, s +1 /d
(recall qξ q, s +1 =
Θ d ( q − s ) ).
Denote α =
qξ q, s
+1 /ξ
q, s .
From the bounds (64) on ξ q, s +1 and ξ q, s , we have α = Θ q (1) .
Denote ˜
α
= q + 1 − α and E s , ≥ ˜
α = { S ∈ E s : γ ( S ) ≥ ˜ α } and E s ,< ˜ α
= E s \ E s , ≥ ˜ α .
Using Eq.
(64) and that 1 ≤ r ( S ) ≤ q , we have { λ d,j } j ∈ [ m ] that contains exactly the eigenvalues associated to homogeneous polynomials of degree less or equal to s − 1 and of degree s with S ∈ E s ,< ˜ α (which corresponds to the sets S such that r ( S ) > α , i.e., ξ q, s r ( S ) >
qξ q, s +1 ).
In particular, if α < 1 , then { λ d,j } j ∈ [ m ] contains exactly the eigenvalues associated to all homogeneous polynomials of degree less or equal to s .
Note that we have
[Equation]
Step 2.
Diagonal elements of the truncated kernel.
Deﬁne the truncated kernel H d,> m to be
[Equation]
The diagonal elements of the truncated kernel are given by: for any x ∈ Q d ,
[Equation]
Notice that
[Equation]
Hence using that ξ q, s = O d ( q − s ) , we have
[Equation]
where h q,> s is the inner-product kernel with the ( s + 1) -ﬁrst Gegenbauer coefﬁcients set to zero, i.e., h q,> s ( (cid:104) u , v (cid:105) /q )
= (cid:80) q(cid:96)
=
s +1 ξ q,(cid:96) B
( Q q ; (cid:96) ) Q ( q ) (cid:96) ( (cid:104) u , v (cid:105) ) , for any u , v ∈ Q q .
From [-Assumption
-]Assumption 1 at level s , we have Ω q (1) = ξ q,(cid:96) (cid:48)
B ( Q q ; (cid:96) (cid:48) ) ≤ h q,> s (1) ≤
h q (1) =
O q (1) .
Hence, Tr( H d,> m ) =
Θ d (1) .
Similarly,
[Equation]
[Equation]
Step 3.
Choosing the sequence u
= u ( d ) .
Let s (cid:48) be chosen as in Assumption 1, i.e., such that ξ q, s (cid:48) B
( Q q ; s (cid:48) )
=
Ω q (1) .
We have
[Equation]
Set u
= u ( d ) to be the number of eigenvalues such that λ q,j > qξ q, s (cid:48) /d
=
Θ q ( q − s (cid:48) +1 /d ) .
From
Eqs.
(64) and (68), and recalling that 1 ≤ r ( S ) ≤ q , we deduce that { λ d,j } j ∈ [ u ] must contain all the eigenvalues associated to homogeneous polynomials of degree less or equal to (cid:96) and does not contain any of the eigenvalues associated to homogeneous polynomials of degree larger or equal to s (cid:48) .
We have
[Equation]
Ω q
[Equation]
Similarly, we have
Tr( H 2 d
[Equation]
>u ) =
(cid:88)
[Equation]
Tr( H 2 d
[Equation]
>u ) ≥ ξ
[Equation]
d
[Equation]
(cid:88)
S
[Equation]
(cid:48) r ( S )
[Equation]
Finally,
Tr( H 4 d,>u ) =
(cid:88)
j>u λ
[Equation]
d
[Equation]
d
[Equation]
q
[Equation]
Tr( H d
[Equation]
>
[Equation]
d
[Equation]
+
[Equation]
Step 4.
Checking the kernel concentration property at level { ( n ( q ) , m ( q ))
} q ≥ 1 .
Let us check the kernel concentration property at level ( n, m ) with the sequence of integers { u ( q ) } q ≥deﬁned in the previous step (Assumption 4 in [38]):
(a) (Hypercontractivity of ﬁnite eigenspaces)
The subspace spanned by the top eigenvectors { ψ q,j } j ∈ [ u ] is contained in the subspace of polynomials of degree less or equal to s (cid:48) − 1 on the hypercube.
The hypercontractivity of this subspace is a consequence of a classical result due to Beckner, Bonami and Gross (see Lemma 4 in Section D). (b) (Properly decaying eigenvalues.)
From step 3 and recalling that s (cid:48) ≥ 1 /δ + 2 s + 3 where δ > 0 veriﬁes q ≥ d δ , we have
[Equation]
for δ (cid:48) > 0 sufﬁciently small.
Similarly,
[Equation]
for δ (cid:48) > 0 chosen sufﬁciently small. (c) (Concentration of the diagonal elements of the kernel) From Eqs.
(66) and (67), the diagonal elements of the kernel are constant and the assumption is automatically veriﬁed.
Step 5.
Checking the eigenvalue condition at level { ( n ( q ) , m ( q ))
} q ≥ 1 .
Let us now check the eigenvalue condition at level { ( n ( q ) , m ( q )) } q ≥ 1 which corresponds to Assumption 5 in [38]):
(a) First notice that
[Equation]
Hence
Tr( H 2 d,> m )
[Equation]
for δ > 0 sufﬁciently small.
Similarly,
[Equation]
(b) This is a direct consequence of Eq.
We can therefore apply Theorem 4 in [38], which concludes the proof.
C.5 Proof of Theorem
Consider q s − 1+ δ ≤ n ≤ q s − δ for some δ > 0 and a sequence of inner-product kernels { h q } q ≥ 1 that satisﬁes Assumptions 1 and 2 at level s .
We consider the one-layer convolutional kernel with global average pooling
[Equation]
Again, the proof of Theorem 8 will amount to checking that the conditions of Theorem 4 in [38] hold.
For the sake of simplicity, we will further assume that ξ q, s > qξ q, s +1 , which simpliﬁes some of the computation.
This condition can be removed as in Theorem 7, by considering the set C s ,< ˜ α
= { S ∈ C s : γ ( S ) <
˜ α } and showing that the extra terms corresponding to these eigenfunctions are negligible.
Proof of Theorem 8.
Step 1.
Diagonalization of the kernel and choosing m = m ( q ) .
From Proposition 2 with ω = d , we have the following diagonalization of H dd,q :
[Equation]
where we recall ψ S ( x )
= 1 √ d (cid:80) k ∈ [ d ] Y k + S ( x ) and that C (cid:96) is the quotient space of E (cid:96) with the translation equivalence relation.
It is easy to check that |C (cid:96) | = (cid:0) q − 1 (cid:96) − 1 (cid:1) .
From Assumption 1, we get the same bounds on the Gegenbauer coefﬁcients ξ q,(cid:96) as Eq.
(64) in the proof of Theorem 7.
Denote { λ q,j } j ≥ 1 the eigenvalues { ξ q,(cid:96) r ( S ) } (cid:96) =0 ,...,q ; S ∈E (cid:96) in nonincreasing order, and { ψ q,j } j ≥ 1 the reordered eigenfunctions.
Set m to be the number of eigenvalues such that λ q,j > qξ q, s +1 (recall qξ q, s +1 =
Θ d ( q − s ) ).
From the bounds (64) and our simplifying assumption that ξ q, s > qξ q, s +1 , we have { λ d,j } j ∈ [ m ] that contains exactly the eigenvalues associated to homogeneous polynomials of degree less or equal to s .
Note that we have
[Equation]
Step 2.
Diagonal elements of the truncated kernel.
Deﬁne the truncated kernel H d,> m to be
[Equation]
The diagonal elements of the truncated kernel are given by: for any x ∈ Q d ,
[Equation]
where
[Equation]
Notice that we have now
[Equation]
Therefore E x [Υ ( q ) (cid:96) ( x )]
= 1 and
Tr( H d,> m ) =
E x
[ H d,> m ( x , x )] = q
[Equation]
s +
[Equation]
,> s (1) .
From Proposition 7 with (cid:96) =
s , we have
[Equation]
Step 3.
Choosing the sequence u
= u ( d ) .
Let s (cid:48) be chosen as in Assumption 1.
Similarly to step 3 in the proof of Theorem 7, take u = u ( d ) to be the number of eigenvalues such that λ q,j > qξ q,
[Equation]
We get
[Equation]
Step 4.
Checking the kernel concentration property at level { ( n ( q ) , m ( q ))
} q ≥ 1 .
The kernel concentration property at level ( n, m ) hold with the sequence { u ( q ) } q ≥ 1 as deﬁned in step 3.
The hypercontractivity of ﬁnite eigenspaces and the properly decaying eigenvalues are obtained as in step 4 of the proof of Theorem 7, while the concentration of the diagonal elements of the kernel is given by Eq.
Step 5.
Checking the eigenvalue condition at level { ( n ( q ) , m ( q ))
} q ≥ 1 .
This is obtained similarly as in step 5 of the proof of Theorem 7.
C.6 Auxiliary results
Proposition 7.
Let s ≥ 1 be a ﬁxed integer.
Assume that the sequence of inner-product kernels { h q } q ≥ 1 satisﬁes Assumptions 1 and 2 at level s .
Deﬁne H > s d : Q d × Q d → R as the convolutional kernel with global average pooling
[Equation]
where h q,> s is the inner-product kernel where the s + 1 ﬁrst Gegenbauer coefﬁcients are set to 0 .
Then for n =
O q ( q p ) for some ﬁxed p , letting ( x i ) i ∈ [ n ] ∼ Unif( Q d ) , we have
[Equation]
[Equation]
Proof of Proposition 7.
Step 1.
Bounding
[Equation]
Recall that we deﬁned
[Equation]
Following the same proof as Proposition 8 in [39]
[Equation]
notice that for the integer v in Assumption
[Equation]
by
Lemma 2 stated below
[Equation]
we have
[Equation]
[Equation]
By Assumption 2, there exists C > 0 such that for any γ ∈ [ − 1 , 1] ,
[Equation]
and | h ( r ) q,>v (0) | ≤ Cq − ( v +1 − r ) / 2 for r ≤ v .
Moreover, by Hanson-Wright inequality as in Lemma 3, using n = O q ( q p ) (at most polynomial in q ) and a union bound, we have for any η > 0 ,
[Equation]
Therefore
[Equation]
injecting these bounds in Eq.
(74), we get
[Equation]
Hence
[Equation]
we deduce that
[Equation]
Furthermore, recall that by Assumption 1, we have E [ H >(cid:96)d ( x , x )] ≥ ξ q, s (cid:48)
B ( Q q ; s (cid:48) )
=
Ω q (1) .
We get sup
[Equation]
[Equation]
which concludes the proof of the ﬁrst bound.
Step 2.
Bounding sup
[Equation]
Notice that we can write,
[Equation]
x
[Equation]
where we denoted R (cid:96) =
(cid:80) S ∈C (cid:96) r ( S ) 2 and Ξ ( d ) (cid:96) ( x ) = 1 R (cid:96) (cid:88)
S ∈C (cid:96) r ( S ) 2 ψ S ( x ) 2 .
Then, by Lemma 2, we get for any u ≥ s , sup i ∈ [ n ] (cid:12)(cid:12)(cid:12) E
[Equation]
(cid:48)
[ H > s d ( x i ,
[Equation]
E
[Equation]
H
[Equation]
(cid:48)
[Equation]
≤ sup i ∈ [ n ] (cid:12)
[Equation]
[Equation]
We conclude following the same argument as in the proof of Proposition 9 in [39].
Lemma 2.
Let (cid:96) ≥ 2 be an integer.
Deﬁne Υ ( d ) (cid:96) : Q d → R and Ξ ( d )
[Equation]
[Equation] r ( S ) [Equation] ψ S ( x )
[Equation]
where R (cid:96) =
(cid:80) S ∈C (cid:96) r ( S ) 2 .
Let n ≤ q p for some ﬁxed p .
Then, for ( x i ) i ∈ [ n ] i.i.d.
∼ Unif( Q d ) , we have
[Equation]
where E θ [Υ ( d ) (cid:96) ( θ )]
=
E x [Ξ ( d ) (cid:96) ( x )] = 1 .
Proof of Lemma 2.
Step 1.
Bounding max
[Equation]
Deﬁne F (cid:96) : Q d → R to be F
[Equation]
Notice that F (cid:96) ( x ) is a degree 2 (cid:96) polynomial and therefore satisﬁes the hypercontractivity property.
For any m ≥ 1 , there exists C > 0 such that
[Equation]
Let us bound the right hand side.
We have
[Equation]
B
[Equation]
, B [Equation] ) , where B 1 = i + S , B 2 = j + S , B 3 = i (cid:48) + S (cid:48) and B 4 = j (cid:48)
+ S (cid:48) , and we denoted
[Equation]
Notice that ω ( B 1 , B 2 , B 3 , B 4 ) =
if B 1 ∆ B 2 = B 3 ∆ B 4 (the symmetric difference) and 0 otherwise.
In other words, every elements in B 1 ∪ B 2 ∪ B 3 ∪ B 4 appears exactly in 2 or 4 of these sets.
Let us ﬁx i ∈
[ d ] and S ∈ C (cid:96) , and bound (cid:88)
[Equation]
Denote | B 1 ∆ B 2 | = 2 k with 1 ≤ k ≤ (cid:96) .
In order for ω ( B 1 , B 2 , B 3 , B 4 ) = 1 , B 3 must contain exactly k points in
B 1 ∆ B 2 while B 4 must contain the remaining k points.
• Case k < (cid:96) .
There are at most (cid:96) 2 ways of choosing j such that B 1 ∩ B 2 (cid:54) = ∅ .
Fixing j (i
[Equation] e., B 1 and B 2 ) and S (cid:48) , then there are 2 k(cid:96) ways of choosing i (cid:48) and 2 k(cid:96) ways of choosing j (cid:48) such that B 3 ∩ ( B 1 ∆ B 2 ) (cid:54) = ∅ and B 4 ∩ ( B 1 ∆ B 2 ) (cid:54) = ∅ .
Hence the contribution of these terms in Eq.
(81) is upper bounded by
[Equation]
• Case k = (cid:96) .
There are at most d ways of choosing j .
Furthermore, for j ﬁxed, there are at most (cid:0) 2 (cid:96)(cid:96) (cid:1) ways of choosing B 3 and B 4 such that B 3 ∪ B {+
+}
= B 1 ∪ B 2 (note that B 1 ∩ B 2 = ∅ and therefore B 3 ∩ B 4 = ∅ ).
Hence the contribution of these terms in Eq.
(81) is upper bounded by
[Equation]
where we
Combining Eqs.
(82) and (83) and using there are dB ( Q q ; (cid:96) ) choices for i and S 1 , we get E [ F [Equation] ≤
[Equation]
where we used that (cid:96) ≥ 2 and B ( Q q ; (cid:96) )
=
Ω q ( q (cid:96) ) .
Using Eq.
(80), we deduce
[Equation]
Using Markov’s inequality and taking m sufﬁciently small yield Eq.
[Equation]
The second bound (78) is obtained very similarly.
Deﬁne G (cid:96) : Q d → R to be G (cid:96)
[Equation]
Then
[Equation]
we have
[Equation]
Further notice that following the same computation as in Eq.
[Equation]
we get
[Equation]
Hence, the same computation as for F (cid:96) in step 1 yields
[Equation]
where we used that (cid:96) ≥ 2 .
We deduce Eq.
(78) similarly to step
[Equation]
Lemma 3 (Hanson-Wright inequality) .
There exists a universal constant c > 0 , such that for any t > 0 and q 1 /δ ≥ d ≥ q ∈ N for some δ > 0 , when x ∈ Unif( Q d ) , we have
P (cid:32) sup k [Equation]
[Equation]
[Equation] q > t (cid:33) ≤ [Equation] q 2 /δ exp {− cq · min( t 2 , t ) } , where we recall that x ( k ) =
( x k , . . .
, x k + q − 1 ) .
Proof of Lemma 3.
For any k (cid:54) = l , denote A =
( a ij )
i,j ∈ [ d ] the matrix with a ( k + i ) , ( l + i ) = 1 for i = 0 , . . .
, q − 1 and a ij = 0 otherwise, such that (cid:104) x , Ax (cid:105) =
(cid:104) x ( k ) , x ( l ) (cid:105) .
Note that we have (cid:107) A (cid:107) F = √ q , (cid:107) A (cid:107) op ≤ 1 and E [ (cid:104) x , Ax (cid:105) ]
= 0 .
By Hanson-Wright inequality of vectors with independent sub-Gaussian entries (for example, see Theorem 1.1 in [42]), we have
P ( |(cid:104) x , Ax (cid:105)| /q > t ) ≤ 2 exp {− cq · min( t 2 , t ) } .
Taking the union bound over k (cid:54)
=
l concludes the proof.
D Technical background of function spaces on the hypercube
Fourier analysis on the hypercube is a well studied subject [41].
The purpose of this section is to introduce some notations and objects that are useful in the statement and proofs in the main text.
D.1 Fourier basis
Denote Q d = {− 1 , +1 } d the hypercube in d dimension, and τ d to the uniform probability measure on Q d .
All the functions will be assumed to be elements of L 2 ( Q d , τ d ) (which contains all the bounded functions f : Q d → R ), with scalar product and norm denoted as (cid:104)· , ·(cid:105) L 2 and (cid:107) · (cid:107) L 2 : (cid:104) f, g (cid:105)
L 2 ≡ (cid:90)
[Equation]
x [Equation] [Equation] Notice that L 2 ( Q d , τ d ) is a 2 n dimensional linear space.
By analogy with the spherical case we decompose L 2 ( Q d , τ d ) as a direct sum of d + 1 linear spaces obtained from polynomials of degree (cid:96) = 0 , . . .
, d
L 2 ( Q d ,
[Equation]
= d (cid:77) (cid:96) =
V d,(cid:96) .
For each (cid:96) ∈ { 0 , . . .
, d } , consider the Fourier basis { Y ( d ) (cid:96),S } S ⊆
[ d ] , | S | = (cid:96) of degree (cid:96) , where for a set
S ⊆
[ d ] , the basis is given by
[Equation]
x
[Equation]
It is easy to verify that (notice that x ki = x i if k is odd and x ki
= 1 if k is even)
[Equation]
Hence { Y ( d ) (cid:96),S } S ⊆
[ d ] , | S | = (cid:96) form an orthonormal basis of V d,(cid:96) and dim( V d,(cid:96) )
=
B ( Q d ; (cid:96) ) =
(cid:18) d (cid:96) (cid:19) .
We will omit the superscript ( d ) in Y ( d ) (cid:96),S when clear from the context and write Y S :
= Y ( d ) (cid:96),S .
We denote by P (cid:96) the orthogonal projections to V d,(cid:96) in L 2 ( Q d ) .
This can be written in terms of the
Fourier basis as
[Equation]
[Equation]
We also deﬁne P ≤ (cid:96) ≡ (cid:80) (cid:96)k =
P k , P >(cid:96) ≡ I − P ≤ (cid:96) =
(cid:80) ∞
k = (cid:96) +1 P k , and P <(cid:96) ≡ P ≤ (cid:96) − 1 , P ≥ (cid:96) ≡ P >(cid:96) − 1 .
D.2 Hypercubic Gegenbauer
We consider the following family of polynomials { Q ( d ) (cid:96) } (cid:96) =0 ,...,d that we will call hypercubic Gegenbauer , or G egenbauer on the d -dimensional hypercube , deﬁned as
[Equation]
[Equation]
Notice that the right hand side only depends on (cid:104) x , y (cid:105) and therefore these polynomials are well deﬁned.
In particular,
[Equation]
Hence { Q ( d ) (cid:96) } (cid:96) =0 ,...,d form an orthogonal basis of L 2 ( {− d, − d + 2 , . . .
, d − 2 , d } , ˜ τ 1 d ) where ˜ τ 1 d is the distribution of (cid:104) 1 , x (cid:105) when x ∼ τ d , i.e., ˜ τ 1 d ∼ 2 Bin ( d, 1 / 2) − d/ 2 .
It is easy to check more generally that
[Equation]
Furthermore, Eq.
(86) imply that —up to
a constant— Q ( d ) k
( (cid:104) x , y (cid:105) ) is a representation of the projector onto the subspace of degree- k polynomials
[Equation]
[Equation]
For a function σ ( · / √ d ) ∈
L 2 ( {− d, − d + 2 , . . .
, d − 2 , d } , ˜ τ 1 d ) , denote its hypercubic Gegenbauer coefﬁcients ξ d,k ( σ ) to be [Equation]
[Equation]
To any inner-product kernel H d ( x 1 , x 2 ) =
h d ( (cid:104) x 1 , x 2 (cid:105) /d ) , with h d ( · / √ d ) ∈
L 2 ( {− d, − d + 2 , . . .
, d − 2 , d } , ˜ τ 1 d ) , we can associate a self adjoint operator H d : L 2 ( Q d ) →
L 2 ( Q d ) via
[Equation]
By permutation invariance, the space V k of homogeneous polynomials of degree k is an eigenspace of H d , and we will denote the corresponding eigenvalue by ξ d,k ( h d ) .
In other words H d f ( x ) ≡ (cid:80) q k =0 ξ d,k ( h d ) P k f .
The eigenvalues can be computed via
[Equation]
D.3 Hermite polynomials
The Hermite polynomials { He k } k ≥ 0 form an orthogonal basis of L 2 ( R , γ ) , where γ (d x ) =
e − x 2 / 2 d x/ √ 2 π is the standard Gaussian measure, and He k has degree k .
We will follow the classical normalization (here and below, expectation is with respect to G ∼ N (0 , 1) ):
[Equation]
As a consequence, for any function g ∈ L 2 ( R , γ ) , we have the decomposition
[Equation]
The Hermite polynomials can be obtained as high-dimensional limits of the Gegenbauer polynomials introduced in the previous section.
Indeed, the Gegenbauer polynomials (up to a √ d scaling in domain) are constructed by Gram-Schmidt orthogonalization of the monomials { x k } k ≥ 0 with respect to the measure ˜ τ 1 d , while Hermite polynomial are obtained by Gram-Schmidt orthogonalization with respect to γ .
Since ˜ τ 1 d ⇒ γ (here ⇒ denotes weak convergence), it is immediate to show that, for any ﬁxed integer k ,
[Equation]
Here and below, for P a polynomial, Coeﬀ { P ( x ) } is the vector of the coefﬁcients of P .
As a consequence, for any ﬁxed integer k , we have
[Equation]
where µ k ( σ ) and ξ d,k ( σ ) are given in Eq.
(92) and (88).
D.4 Hypercontractivity of uniform distributions on the hypercube
By Holder’s inequality, we have (cid:107) f (cid:107) L p ≤ [Equation] f [Equation]
L q for any f and any p ≤ q .
The reverse inequality does not hold in general, even up to a constant.
However, for some measures, the reverse inequality will hold for some sufﬁciently nice functions.
These measures satisfy the celebrated hypercontractivity properties [4, 5, 8, 25].
Lemma 4 (Hypercube hypercontractivity [4]) .
For any (cid:96) = { 0 , . . .
, d } and f d ∈ L 2 ( Q d ) to be a degree (cid:96) polynomial, then for any integer q ≥ 2 , we have
[Equation]