W HAT CAN BE LEARNT WITH WIDE CONVOLUTIONAL NEURAL NETWORKS ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Understanding how convolutional neural networks (CNNs) can efficiently learn high-dimensional functions remains a fundamental challenge.
A popular belief is that these models harness the local and hierarchical structure of natural data such as images.
Yet, we lack a quantitative understanding of how such structure affects performance, e.g. the rate of decay of the generalisation error with the number of training samples.
In this paper, we study deep CNNs in the kernel regime.
First , we show that the spectrum of the corresponding kernel inherits the hierarchical structure of the network, and we characterise its asymptotics.
Then, we use this result together with generalisation bounds to prove that deep CNNs adapt to the spatial scale of the target function.
In particular, we find that if the target function depends on low-dimensional subsets of adjacent input variables, then the rate of decay of the error is controlled by the effective dimensionality of these subsets.
Conversely, if the target function depends on the full set of input variables, then the error rate is inversely proportional to the input dimension.
We conclude by computing the rate when a deep CNN is trained on the output of another deep
CNN with randomly-initialised parameters.
Interestingly, we find that, despite their hierarchical structure, the functions generated by deep CNNs are too rich to be efficiently learnable in high dimension.
I NTRODUCTION
Deep convolutional neural networks (CNNs) are particularly successful in certain tasks such as image classification.
Such tasks generally entail the approximation of functions of a large number of variables, for instance the number of pixels which determine the content of an image.
Learning a generic high-dimensional function is plagued by the curse of dimensionality : the rate at which the generalisation error ϵ decays with the number of training samples n vanishes as the dimensionality d of the input space grows, i.e. ϵ ( n ) ∼ n − β with β =
O (1 /d )
(Wainwright, 2019).
Therefore, the success of CNNs in classifying data whose dimension can be in the hundreds or more (Hestness et al., 2017; Spigler et al., 2020) points to the existence of some underlying structure in the task that CNNs can leverage.
Understanding the structure of learnable tasks is arguably one of the most fundamental problems in deep learning, and also one of central practical importance—as it determines how many examples are required to learn up to a certain error.
A popular hypothesis is that learnable tasks are local and hierarchical: features at any scale are made of sub-features of smaller scales.
Although many works have investigated this hypothesis (Biederman, 1987; Poggio et al., 2017; Kondor & Trivedi, 2018;
Zhou et al., 2018; Deza et al., 2020; Kohler et al., 2020; Poggio et al., 2020; Schmidt-Hieber, 2020; Finocchio & Schmidt-Hieber, 2021; Giordano et al., 2022), there are no available predictions for the exponent β for deep CNNs trained on tasks with a varying degree of locality or a truly hierarchical structure.
In this paper we perform such a computation in the overparameterised regime, where the width of the hidden layer of the neural networks diverges and the network output is rescaled so as to converge to that of a kernel method (Jacot et al., 2018; Lee et al., 2019).
Although the deep networks deployed in real scenarios do not generally operate in such regime, the connection with the theory of kernel regression provides a recipe for computing the decay of the generalisation error with the number of training examples.
Namely, given an infinitely wide neural network, its generalisation abilities depend on the spectrum of the corresponding kernel (Caponnetto & De Vito, 2007; Bordelon et al.,
2020): the main challenge is then to characterise this spectrum, especially for deep CNNs whose kernels are rather cumbersome and defined recursively (Arora et al., 2019).
This characterisation is the main result of our paper, together with the ensuing study of generalisation in deep CNNs.
1. O UR CONTRIBUTIONS
More specifically, this paper studies the generalisation properties of deep CNNs with nonoverlapping patches and no pooling (defined in Sec. 2, see Fig.
1 for an illustration), trained on a target function f ∗ by empirical minimisation of the mean squared loss.
We consider the infinitewidth limit (Sec. 3) where the model parameters change infinitesimally over training, thus the trained network coincides with the predictor of kernel regression with the Neural Tangent Kernel (NTK) of the network.
Due to the equivalence with kernel methods, generalisation is fully characterised by the spectrum of the integral operator of the kernel: in simple terms, the projections on the eigenfunctions with larger eigenvalues can be learnt (up to a fixed generalisation error) with fewer training points (see, e.g., Bach (2021)).
Spectrum of deep hierarchical kernels (Thm. 3.1).
Due to the network architecture, the hidden neurons of each layer depend only on a subset of the input variables, known as the receptive field of that neuron (highlighted by coloured boxes in Fig.
1, left panel).
We find that the eigenfunctions of the NTK of a hierarchical CNN of depth L + 1 can be organised into sectors l = 1 , . . .
, L associated with the hidden layers of the network (Thm. 3.1).
The eigenfunctions of each sector depend only on the receptive fields of the neurons of the corresponding hidden layer: if we denote with d eff ( l ) the size of the receptive fields of neurons in the l -th hidden layer, then the eigenfunctions of the l -th sector are effectively functions of d eff ( l ) variables.
We characterise the asymptotic behaviour of the
NTK eigenvalues with the degree of the corresponding eigenfunctions (Thm. 3.1) and find that it is controlled by d eff ( l ) .
As a consequence, the eigenfunctions with the largest eigenvalues—the easiest to learn—are those which depend on small subsets of the input variables and have low polynomial degree.
This is our main technical contribution and all of our conclusions follow from it.
Adaptivity to the spatial structure of the target (Cor. 4.1).
We use the above result to prove that deep CNNs can adapt to the spatial scale of the target function (Sec. 4).
More specifically, by using rigorous bounds from the theory of kernel ridge regression (Caponnetto & De Vito, 2007) (reviewed in the first paragraph of Sec. 4), we show that when learning with the kernel of a CNN and optimal regularisation, the decay of the error depends on the effective dimensionality of the target f ∗ —i.e., if f ∗ only depends on d eff adjacent coordinates of the d -dimensional input, then ϵ ∼ n − β with β ≥ O (1 /d eff )
(Cor. 4.1, see Fig.
1 for a pictorial representation).
We find a similar picture in ridgeless regression by using non-rigorous results derived with the replica method (Bordelon et al., 2020; Loureiro et al., 2021)
(Sec. 5).
Notice that for targets which are spatially localised (or sums of spatially localised functions), the rates achieved with deep CNNs are much closer to the
Bayes-optimal rates—realised when the architecture is fine-tuned to the structure of the target— than β
= O (1 /d )
obtained with the kernel of a fully-connected network.
Moreover, we find that hierarchical functions generated by the output of deep CNNs are too rich to be efficiently learnable in high dimensions (Lemma 5.1).
We confirm these results through extensive numerical studies and find them to hold even if the nonoverlapping patches assumption is relaxed (Subsec. G.4).
1. R ELATED WORK
The benefits of shallow CNNs in the kernel regime have been investigated by Bietti (2022);
Favero et al.
; Misiakiewicz & Mei (2021); Xiao & Pennington (2022); Xiao (2022); Geifman et al.
Favero et al.
(2021), and later (Misiakiewicz & Mei, 2021; Xiao & Pennington, 2022), studied generalisation properties of shallow CNNs, finding that they are able to beat the curse of dimensionality on local target functions.
However, these architectures can only approximate functions of single input patches or linear combinations thereof.
Bietti (2022), in addition, includes generic pooling layers and begins considering the role of depth by studying the approximation properties of kernels which are integer powers of other kernels.
We generalise this line of work by studying CNNs of any depth with nonanalytic (ReLU) activations: we find that the depth and nonanalyticity of the resulting kernel are crucial for understanding the inductive bias of deep CNNs.
This result should also be contrasted with the spectrum of the kernels of deep fully-connected networks, whose asymptotics log n l og ε n − O (1 /d eff ) n − O
(1 /d eff )
n − O (1 /d eff ) do not depend on depth (Bietti & Bach, 2021).
Furthermore, we extend the analysis of generalisation to target functions that have a hierarchical structure similar to that of the networks themselves.
Geifman et al.
(2022) derive bounds on the spectrum of the kernels of deep CNNs.
However, they consider only filters of size one in the first layer and do not include a theoretical analysis of generalisation.
Instead, we allow filters of general dimension and give tight estimates of the asymptotic behaviour of eigenvalues, which allow us to predict generalisation properties.
Xiao (2022) is the closest to our work, as it also investigates the spectral bias of deep CNNs in the kernel regime.
However, it considers a different limit where both the input dimension and the number of training points diverge and does not characterise the asymptotic decay of generalisation error with the number of training samples.
Paccolat et al.
; Malach & Shalev-Shwartz (2021); Abbe et al.
(2022) use sparse target functions which depend only on a few of the input variables to prove sample complexity separation results between networks operating in the kernel regime and in the feature regime—where the change in parameters during training can be arbitrarily large.
In this respect, our work shows that when the few relevant input variables are adjacent, i.e. the target function is spatially localised, deep CNNs achieve near-optimal performances even in the kernel regime.
N OTATION AND SETUP
Our work considers CNNs with nonoverlapping patches and no pooling layers.
These networks are fully characterised by the depth L + 1 (or number of hidden layers L ) and a set of filters sizes { s l } l (one per hidden layer).
We call such networks hierarchical CNNs.
Definition 2.1 ( L -hidden-layers hierarchical CNN)
Denote by σ the normalised ReLU function, σ ( x ) = √ 2 max(0 , x ) .
For each input x ∈ R d 1 and s a divisor of d , denote by x i the i -th s -dimensional patch of x , x i =
( x ( i − 1) × s +1 , . . .
, x i × s ) for all i = 1 , . . .
, d/s .
The output of a Lhidden-layers hierarchical neural network can be defined recursively as follows.
[Equation]
H l denotes the width of the l -th layer, s l the filter size ( s 1 = s ), p l the number of patches ( p 1 ≡ p = d/s )
[Equation]
Hierarchical CNNs are best visualised by considering their computational skeleton, i.e. the directed acyclic graph obtained by setting H l = 1 ∀ l (example in Fig.
1, left, with L = 3 hidden layers and filter sizes ( s 1 , s 2 , s 3 )
= (3 , 2 , 2) ).
Having nonoverlapping patches, the computational skeleton is an ordered tree, whose root is the output (empty circle at the top of the figure) and the leaves are the input coordinates (squares at the bottom).
All the other nodes represent neurons and all the neurons belonging to the same hidden layer have the same distance from the input nodes.
The tree structure highlights that the post-activations f li of the l -th layer depend only on a subset of the input variables, also known as the receptive field .
Since the first layer of a hierarchical CNN acts on s 1 -dimensional patches of the input, it is convenient to consider each d -dimensional input signal as the concatenation of p s -dimensional patches, with s =
s 1 and p × s =
d .
We assume that each patch is normalised to 1 2 , so that the input space is a product of p s -dimensional unit spheres (called multisphere in Geifman et al.
[Equation]
Notice that the s -dimensional patches are also the receptive fields of the first-hidden-layer neurons (as in the blue rectangle in Fig.
1 for s = 3 ).
In general, the receptive field of a neuron in the l -th hidden layer with l > 1 is a group of (cid:81) ll ′
=
2 s l ′ adjacent patches (as in the orange rectangle of Fig.
1 for l = 2 , s
= 2 or the green rectangle for l = 3 , s
= s 2 = 2 ), which we refer to as a meta-patch .
Due to the correspondence with the receptive fields, each meta-patch is identified with one path on the computational skeleton: the path which connects the output node to the hidden neuron whose receptive field coincides with the meta-patch.
If such hidden neuron belongs to the l -th hidden layer, the path is specified by a tuple of L − l + 1 indices, i l +1 → L +1 := i
L +1 . . .
i l +1 , where each index indicates which branch to select when descending from the root to the neuron node.
With this notation, x i l +1 → i L +1 denotes one of the p l meta-patches of size (cid:81)
l ′≤ l s l ′ .
Because of the normalisation of the s 1 -dimensional patches, each meta-patch has an effective dimensionality which is lower than its size:
[Equation]
H IERARCHICAL KERNELS AND THEIR SPECTRA
We turn now to the infinite-width limit H l → ∞ : because of the aforementioned equivalence with kernel methods, this limit allows us to deduce the generalisation properties of the network from the spectrum of a kernel.
In this section, we present the kernels corresponding to the hierarchical models of Def. 2.1 and characterise the spectra of the associated integral operators.
We consider specifically two kernels: the Neural Tangent Kernel (NTK), corresponding to training all the network parameters (Jacot et al., 2018); and the Random Feature Kernel (RFK), corresponding to training only the weights of the linear output layer (Rahimi & Recht, 2007; Daniely et al.,
In both cases, the kernel reads:
[Equation]
The NTK and RFK of deep CNNs have been derived previously by Arora et al.
In App.
B we report the functional forms of these kernels in the case of hierarchical CNNs.
These kernels inherit the hierarchical structure of the original architecture and their operations can be visualised again via the tree graph of Fig.
In this case, the leaves represent products between the corresponding elements of two inputs x and y .
, i.e. x 1 y 1 to x d y d , and the root the kernel output K ( x , y ) .
The output can be built layer by layer by following the same recipe for each node: first sum the outputs of the previous layer which are connected to the present node, then apply some nonlinear function which depends on the activation function of the network.
In particular, for each couple of inputs x and y on the multisphere M p S s − 1 , hierarchical kernels depend on x and y via the p dot products between corresponding s -dimensional patches of x and y .
As a comparison, Bietti & Bach (2021) showed that the NTK and RFK of a fully-connected network of any depth depend on the full dot product x · y , whereas those of a shallow CNN can be written as the sum of p kernels, each depending on only one of the patch dot products (Favero et al., 2021).
Given the kernel, the associated integral operator reads
[Equation]
with dp ( x ) denoting the uniform distribution of input points on the multisphere.
The spectrum of this operator provides, via Mercer’s theorem (Mercer, 1909), an alternative representation of the kernel K ( x , y ) and a basis for the space of functions that the kernel can approximate.
The asymptotic decay of the eigenvalues, in particular, is crucial for the generalisation properties of the kernel, as it will be clarified at in Sec.
Since the input space is a product of s -dimensional unit spheres and the kernel depends on the p scalar products between corresponding s -dimensional patches of x and y , the eigenfunctions of T K are products of spherical harmonics acting on the patches (see App.
A for definitions and the relevant background).
For the sake of clarity, we limit the discussion in the main paper to the case s = 2 , where, since each patch x i is entirely determined by an angle θ i , the multisphere M p S s − 1 reduces to the p -dimensional torus and the eigenfunctions to p -dimensional plane waves: e i k · θ with θ :=
( θ 1 , . .
, θ p ) and label k :=
( k 1 , . . .
, k p ) .
In this case the eigenvalues coincide with the p -dimensional Fourier transform of the kernel K (cos θ 1 , . . .
, cos θ p ) and the large- k asymptotics are controlled by the nonanalyticities of the kernel (Widom, 1963).
The general case with patches of arbitrary dimension is presented in the appendix.
Theorem 3.1 (Spectrum of hierarchical kernels) Let T K be the integral operator associated with a d -dimensional hierarchical kernel of depth L + 1 , L > 1 and filter sizes ( s 1 , . . .
, s L ) with s 1 = 2 .
Eigenvalues and eigenfunctions of T K can be organised into L sectors associated with the hidden layers of the kernel/network.
For each 1 ≤ l ≤ L , the l -th sector consists of ( (cid:81) l l ′ =
1 s l ′ ) - local eigenfunctions: functions of a single meta-patch x i l +1 → L +1 which cannot be written as linear com- binations of functions of smaller meta-patches.
The labels k of these eigenfunctions are such that there is a meta-patch k i l
+
→ L +1 of k with no vanishing sub-meta-patches and all the k
i ’s outside of k
i l +
→ L +1 are 0 (because the eigenfunction is constant outside of x i
l +1 → L +1 ).
The corresponding eigenvalue is degenerate with respect to the location of the meta-patch: we call it Λ ( l ) k il +1 → iL +1 .
When ∥ k
i
l
{++
+}
→ L +1 ∥ → ∞ , with k = ∥
k i l +
→ L +1 ∥ ,
[Equation]
(6) with ν NTK = 1 / 2 , ν RFK = 3 / 2 and d eff the effective dimensionality of the meta-patches defined in Eq.
C 2 ,l is a strictly positive constant for l ≥ 2 whereas for l = 1 it can take two distinct strictly positive values depending on the parity of k i
→ L +1 .
The proof is in App.
C, together with the extension to the s ≥ 3 case (Thm.
C.1).
It is useful to compare the spectrum in the theorem with the limiting cases of a deep fully-connected network and a shallow CNN.
In the former case, the spectrum consists only of the L -th sector with p L = 1 —the global sector.
The eigenvalues decay as ∥ k
∥ − 2 ν − p , with ν depending ultimately on the nonanalyticity of the network activation function (see Bietti & Bach (2021) or App. C) and p =
d eff ( L ) the effective dimensionality of the input.
As a result, all eigenfunctions with the same ∥ k ∥ have the same eigenvalue, even those depending on a subset of the input coordinates.
For example, assume that all the components of k are zero but k 1 , i.e. the eigenfunction depends only on the first 2 -dimensional patch: the eigenvalue is O ( k − 2 ν − p) .
By contrast, for a hierarchical kernel, the eigenvalue is O ( k − 2 ν − 1) , much larger than the former as p > 1 .
In the case of a shallow CNN the spectrum consists only of the first sector, so that each eigenfunction depends only on one of the input patches.
In this case only one of the k can be non-zero, say k 1 , and the eigenvalue is O ( k − 2 ν − 1) .
However, from (Favero et al., 2021), a kernel of this kind is only able to approximate functions which depend on one of the input patches or linear combinations of such functions.
Instead, for a hierarchical kernel with p L = 1 , the eigenfunctions of the L -th sector are supported on the full input space.
Then, if Λ k > 0 for all k , hierarchical kernels are able to approximate any function on the multisphere, dispensing with the need for fine-tuning the kernel to the structure of the target function.
Overall, given an eigenfunction of a hierarchical kernel, the asymptotic scaling of the corresponding eigenvalue depends on the spatial structure of the eigenfunction support.
More specifically, the effective dimensionality of the smallest meta-patch which contains all the variables that the eigenfunction depends on.
In simple terms, the decay of an eigenvalue with k is slower if the associated eigenfunction depends on a few adjacent patches—but not if the patches are far apart!
This is a property of hierarchical architectures which use nonlinear activation functions at all layers.
Such a feature disappears if all hidden layers apart from the first have polynomial (Bietti, 2022) or infinitely smooth (Azevedo & Menegatto
G ENERALISATION PROPERTIES AND ADAPTIVITY TO SPATIAL STRUCTURE
In this section, we study the implications of the peculiar spectra of hierarchical NTKs and RFKs on the generalisation properties of and prove a form of adaptivity to the spatial structure of the target function.
We follow the classical analysis of Caponnetto & De Vito (2007) for kernel ridge regression (see Bach (2021);
Bietti (2022) for a modern treatment) and employ a spectral bias ansatz for the ridgeless limit (Bordelon et al., 2020; Spigler et al., 2020).
Theory of kernel ridge regression and source-capacity conditions.
Given a set of n training points { ( x µ , y µ ) } nµ =1 i .
i .
d . ∼ p ( x , y ) for some probability density function p ( x , y ) and a regularisation parameter λ > 0 , the kernel ridge regression estimate of the functional relation between x ’s and y ’s, or predictor , is
[Equation]
where H is the Reproducing Kernel Hilbert Space (RKHS) of a (hierarchical) kernel K .
If f ( x ) denotes the model from which the kernel was obtained via Eq.
4, the space H is contained in the span of the network features { ∂ θ f ( x ) } θ in the infinite-width limit.
Alternatively, H can be defined via the kernel’s eigenvalues Λ k and eigenfunctions Y k : denoting with f k the projections of a function f onto the kernel eigenfunctions, then f belongs to H if it belongs to the span of the eigenfunctions and
[Equation]
The performance of the kernel is measured by the generalisation error and its expectation over training sets of fixed size n (denoted with E n )
[Equation]
or the excess generalisation error, obtained by subtracting from ϵ ( λ, n ) the error of the optimal predictor f ∗ ( x )
= (cid:82) dy p ( x , y ) y .
The decay of the error with n can be controlled via two exponents, depending on the details of the kernel and the target function.
Specifically, if α ≥ 1 and r ≥ 1 − 1 /α satisfy the following conditions,
[Equation]
then, by choosing a n -dependent regularisation parameter λ n ∼ n − α / ( αr + 1) , one gets the following bound on generalisation (Caponnetto & De Vito, 2007):
[Equation]
Spectral bias ansatz for ridgeless regression.
The bound above is actually tight in the noisy setting, for instance when having labels y µ = f ∗ ( x µ ) + ξ µ with ξ µ Gaussian.
In a noiseless problem where y µ = f ∗ ( x µ ) one expects to find the best performances in the ridgeless limit λ → 0 , so that the rate of Eq.
11 is only an upper bound.
In the ridgeless case—where the correspondence between kernel methods and infinitely-wide neural networks actually holds—there are unfortunately no rigorous results for the decay of the generalisation error.
Therefore, we provide a heuristic derivation of the error decay based on a spectral bias ansatz.
Consider the projections of the target function f ∗ on the eigenfunctions of the student kernel Y k
( f ∗ k ) 3 and assume that kernel methods learn only the n projections corresponding to the highest eigenvalues.
Then, if the decay of f ∗ k with k is sufficiently slow, one has (recall that both λ and ϵ ( f ∗ ) vanish in this setting)
[Equation]
with Λ( n ) the value of the n -th largest eigenvalue of the kernel.
This result can be derived using the replica method of statistical physics (see Canatar et al.
; Loureiro et al. (2021); Tomasini et al.
(2022) and
App. E) or by assuming that input points lie on a lattice (Spigler et al., 2020).
These two approaches rely on the very same features of the problem, namely the asymptotic decay of Λ k and | f ∗ k | 2 —see also Cui et al.
For instance, the capacity condition depends only on the kernel spectrum: α ≥ 1 since Tr ( T K ) is finite (Sch ¨ olkopf et al., 2002); the specific value is determined by the decay of the ordered eigenvalues with their rank, which in turn depends on the scaling of Λ k with k .
Similarly, the power-law decay of the ordered eigenvalues with the rank determines the scaling of the n -th largest eigenvalue, Λ( n ) ∼ n − α .
The source condition characterises the regularity of the target function relative to the kernel and depends explicitly of the decay of | f ∗ k |with k , as does the right-hand side of Eq.
This condition was used by Bach (2021) to prove that kernel methods are adaptive to the smoothness of the target function: the projections of smoother targets on the eigenfunctions display a faster decay with k , thus allowing to choose a larger r and leading to better generalisation performances.
The following corollary of Thm.
3.1 (proof and extension to s 1 ≥ 3 presented in App.
D, Cor.
D.1) shows that, since the spectrum can be partitioned as in Thm.
3.1, hierarchical kernels display adaptivity to targets which depend only on a subset of the input variables.
Specific examples of bounds are considered explicitly in Sec. 5.
Corollary 4.1 (Adaptivity to spatial structure) Let T K be the integral operator of the kernel of a hierarchical deep CNN as in Thm.
3.1 with s = 2 .
Then: i) the capacity exponent α is controlled by the largest sector of the spectrum, i.e. [Equation]
ii) the source exponent r is controlled by the structure of the target function f ∗ , i.e., if there is l ≤ L such that f ∗ depends only on some meta-patch x i
l +1 → L +1 , then only the first l sectors of the spectrum contribute to the source condition,
[Equation]
The same holds if f ∗ is a linear combination of such functions.
As a result, when d eff ( L ) is large and α → 1 , the decay of the error is controlled by the effective dimensionality of the target d eff ( l ) .
E XAMPLES AND EXPERIMENTS
Source-capacity bound for functions of controlled smoothness and d eff .
Consider a target function f ∗ which only depends on the meta-patch x i {+l +
+}
→ L +1 as in Cor. 4.1.
Combining the source condition (Eq. 14) with the asymptotic scaling of eigenvalues (Eq. 6), we get
[Equation]
15) where ν = 1 / 2 ( 3 / 2 ) for the NTK (RFK) and k denotes the meta-patch k
i l {++
+} → L +1 without the subscript to ease notation.
Since the eigenvalues depend on the norm of k , Eq. 15 is equivalent to a finite-norm condition for all the derivatives of f ∗ up to order m < r (2 ν + d eff ( l )) / 2 , ∥ ∆ m/ 2 f ∗ ∥ 2 = (cid:80) k
∥ k
∥ 2 m | f ∗ k | 2 < + ∞ with ∆ denoting the Laplace operator.
As a result, if f ∗ has derivatives of finite norm up to the m -th, then the source exponent can be tuned to r = 2 m/ (2 ν + d eff ( l )) , inversely proportional to the effective dimensionality of f ∗ .
Since the exponent on the right-hand side of Eq.
11 is an increasing function of r , the smaller the effective dimensionality of f ∗ the faster the decay of the error—hence hierarchical kernels are adaptive to the spatial structure of f ∗ .
[-To fully determine the error bound, we need to find the largest α satisfying the capacity condition, which coincides with the exponent describing the decay of eigenvalues sorted in descending order with the rank. This exponent is controlled by the L -th sector, which is the most numerous, thus α = 
-]In particular, from Eq.
11 with α = 1 + 2 ν/d eff ( L ) ,
[Equation]
For instance, if p L = 1 then d eff ( L ) = p = d/ 2 (the number of 2 -dim.
patches).
Even when p ≫ 1 , if f ∗ depends only on a finite-dimensional meta-patch (or is a sum of such functions) the exponent β converges to the finite value 2 m/ (2( m + ν ) + d eff ( l )) .
In stark contrast, using a fully-connected kernel to learn the same target results in r = 2 m/ (2 ν + p ) , thus β = 2 m/ (2 m + p ) —vanishing as 1 /p when p ≫ 1 , thus cursed by dimensionality.
Rates from spectral bias ansatz.
The same picture emerges when estimating the actual decay of the error from Eq.
Λ( n ) ∼ n − α , whereas (cid:80) k
∥ k
∥ 2 m | f ∗ k | 2 < + ∞ implies | f ∗ k | 2 ≲ ∥ k
∥ − 2 m − d eff ( l ) for a target supported on a d eff ( l ) -dimensional meta-patch.
Plugging such decays in Eq.
12 we obtain (details in Subsec. F.1)
[Equation]
Again, with p L = 1 and d eff ( L ) = p , the exponent remains finite for p ≫ 1 .
Notice that we recover the results of Favero et al.
by using a shallow local kernel if the target is supported on sdimensional patches.
These results show that hierarchical kernels play significantly better with the approximation-estimation trade-off than shallow local kernels, as they are able to approximate global functions of the input while not being cursed when the target function has a local structure.
Numerical experiments.
We test our predictions by training a hierarchical kernel ( student ) on a random Gaussian function with zero mean and covariance given by another hierarchical kernel ( teacher ).
A learning problem is fully specified by the depths, sets of filter sizes, and smoothness exponents ν of teacher and student kernels.
In particular, the depth and the set of filter sizes of the teacher kernel control the effective dimension of the target function.
Fig.
2 shows the learning curves (solid lines) together with the predictions from Eq.
17 (dashed lines), confirming the picture emerging from our calculations.
Panel (a) of Fig.
2 shows a depth-four student learning depthtwo, depth-three, and depth-four teachers.
This student is not cursed in the first two cases and is cursed in the third one, which corresponds to a global target function.
Panel (b) illustrates the curse of dimensionality with the effective input dimension d eff ( L ) by comparing the learning curves of depth-three students learning global target functions with an increasing number of variables.
All our simulations are in excellent agreement with the predictions of Eq.
The bounds coming from Eq.
16 would display a slightly slower decay, as sketched in Fig.
1, right panel.
All the details of numerical experiments are reported in App.
G, together with a comparison between the ridgeless and optimally-regularised cases (Fig. S3) and additional results for: s 1 ≥ 3 (Fig. S2); kernels with overlapping patches (Fig. S5); different input spaces (Fig. S4) and the CIFAR-10 dataset (Fig. S6).
Notice that when the teacher kernel is a hierarchical RFK, the target is equivalent to the output of a randomly-initialised, infinitely-wide CNN Novak et al. Although this target is highly structured, it leads to the same rate that we would have obtained for a global non-hierarchical target:
[Figure]
Lemma 5.1 (Curse of dimensionality for hierarchical targets)
The problem of regression of the output of a randomly-initialised and infinitely-wide hierarchical network suffers from the curse of dimensionality, in the sense that no methods using n examples can achieve a generalisation error decaying faster than n − β with β = 3 /d eff
( L ) .
This lemma is a simple consequence of the equivalence of the predictors of kernel ridgeless regression and Bayesian inference, which implies that the rate achieved when learning a Gaussian random function with a kernel matching the covariance kernel of the function is Bayes-optimal (Kanagawa et al., 2018). The optimal rate n − 3 /d eff
( L ) comes from Eq. 17 with l = L and m = ν = 3 / 2 .
We conclude that, despite their intrinsically hierarchical structure, these targets cannot be good models of learnable tasks.
C ONCLUSIONS AND OUTLOOK
We have proved that deep CNNs can adapt to the spatial scale of the target function, thus beating the curse of dimensionality if the target depends only on local groups of variables.
Yet, if considered as ‘teachers’, they generate functions that cannot be learnt efficiently in high dimensions, even in the
Bayes-optimal setting where the student is matched to the teacher.
Thus, the architectures we considered are not good models of the hierarchical structure of real data which are efficiently learnable.
Enforcing a stronger notion of compositionality is an interesting endeavour for the future.
Following Poggio et al.
(2017), one may consider a much smaller family of functions of the form
[Equation]
From an information theory viewpoint, Schmidt-Hieber (2020); Finocchio & Schmidt-Hieber (2021) showed that it is possible to learn such functions efficiently.
However, these arguments do not provide guarantees for any practical algorithm such as stochastic gradient descent.
Moreover, preliminary results (not shown) assuming that the functions f ’s are random Gaussian functions suggest that these tasks are not learnable efficiently by a hierarchical CNN in the kernel regime—see also Giordano et al.
It is unclear whether this remains true when the networks closely resemble the structure of Eq.
18 as in Poggio et al.
(2017), or when the networks are trained in a regime where features can be learnt from data.
Recently, for instance, Ingrosso & Goldt (2022) have observed that under certain conditions locality can be learnt from scratch.
It is not clear whether compositionality can also be learnt, beyond some very stylised settings (Abbe et al., 2022).
Finally, another direction to explore is the stability of the task toward smooth transformations or diffeomorphisms.
This form of stability has been proposed as a key element to understanding how the curse of dimensionality is beaten for image datasets (Bruna & Mallat, 2013; Petrini et al., 2021).
Such a property can be enforced with pooling operations (Bietti & Mairal, 2019; Bietti et al., 2021); therefore diagonalising the NTK in this case as well would be of high interest.
S UPPLEMENTARY MATERIAL A Harmonic analysis on the sphere
A
Dot-product kernels on the sphere
A.2 Multi-dot-product kernels on the multi-sphere
B RFK and NTK of deep convolutional networks
C Spectra of deep convolutional kernels
C.1 Singular expansion of hierarchical kernels . . . . . . . . . . . . . . . . . . . . . .
C.2 Patches on the ring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 Patches on the s dimensional hypersphere . . . . . . . . . . . . . . . . . . . . . .
C.4 General depth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D Generalisation bounds for kernel regression and spatial adaptivity
D.1 Classical generalisation bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Extension of Cor.
4.1 to patches on the hypersphere . . . . . . . . . . . . . . . . .
D.3 Decay of
eigenvalues
with the rank E Statistical mechanics of generalisation in kernel regression
F Examples
F.
Rates from spectral bias ansatz . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G Numerical experiments
G.1 Experimental setup . .
[Table]
[Table]
[Table]
[Table]
[Table]
A H ARMONIC ANALYSIS ON THE SPHERE
This appendix collects some introductory background on spherical harmonics and dot-product kernels on the sphere Smola et al.
See Efthimiou & Frye (2014);
Atkinson & Han (2012); Bach (2017) for a complete description.
Spherical harmonics are homogeneous polynomials on the sphere S s − 1 = { x ∈ R s | ∥ x ∥ = 1 } , with ∥·∥ denoting the L2 norm.
Given the polynomial degree k ∈ N , there are N k,s linearly independent spherical harmonics of degree k on S s − 1 , with
[Equation]
for
[Equation]
S [Equation]
Thus, we can introduce a set of N k,s spherical harmonics Y k,ℓ for each k , with ℓ ranging in 1 , . .
. , N k,s , which are orthonormal with respect to the uniform measure on the sphere dτ ( x ) ,
[Equation]
Because of the orthogonality of homogeneous polynomials with a different degree, the set { Y k,ℓ } k,ℓ is a complete orthonormal basis for the space of square-integrable functions on the s -dimensional unit sphere.
Furthermore, spherical harmonics are eigenfunctions of the Laplace-Beltrami operator ∆ , which is nothing but the restriction of the standard Laplace operator to S s − 1 .
[Equation]
The Laplace-Beltrami operator ∆ can also be used to characterise the differentiability of functions f on the sphere via the L2 norm of some power of ∆ applied to f .
By fixing a direction y in S d − 1 one can select, for each k , the only spherical harmonic of degree k which is invariant for rotations that leave y unchanged.
This particular spherical harmonic is, in fact, a function of x · y and is called the Legendre polynomial of degree k , P k,s ( x · y ) (also referred to as
Gegenbauer polynomial).
Legendre polynomials can be written as a combination of the orthonormal spherical harmonics Y k,ℓ via the addition formula (Atkinson & Han, 2012, Thm. 2.9),
[Equation]
Alternatively, P k,s is given explicitly as a function of t = x · y ∈ [ − 1 , +1] via the Rodrigues formula (Atkinson & Han, 2012, Thm. 2.23),
[Equation]
Legendre polynomials are orthogonal on [ − 1 , +1] with respect to the measure with density (1 − t 2 ) ( s − 3) / 2 , which is the probability density function of the scalar product between two points on S s −.
[Equation]
with | S s − 1 | denoting the surface area of the s -dimensional unit sphere.
To sum up, given x , y ∈ S s − 1 , functions of x or y can be expressed as a sum of projections on the orthonormal spherical harmonics { Y k,ℓ } k,ℓ , whereas functions of x · y can be expressed as a sum of projections on the Legendre polynomials { P k,s ( x · y ) } k .
The relationship between the two expansions is elucidated in the Funk-Hecke formula (Atkinson & Han, 2012, Thm. 2.22),
[Equation]
If the function f has continuous derivatives up to the k -th order in [ − 1 , +1] , then one can plug
Rodrigues’ formula in the right-hand side of Funk-Hecke formula and get, after k integrations by parts,
[Equation]
with f ( k ) ( t ) denoting the k -th order derivative of f in t .
This trick also applies to functions which are not k times differentiable at ± 1 , provided the boundary terms due to integration by parts vanish.
A. D OT - PRODUCT KERNELS ON THE SPHERE
Dot-product kernels are kernels which depend on the two inputs x and y via their scalar product x · y .
When the inputs lie on the unit sphere S s − 1 , one can use the machinery introduced in the previous section to arrive immediately at the Mercer’s decomposition of the kernel Smola et al.
[Equation]
In the first line we have just decomposed K into projections onto the Legendre polynomials [Equation] the second line follows immediately from the addition formula [Equation]
and the third is just a definition of the eigenvalues Λ k .
Notice that the eigenfunctions of the kernel are orthonormal spherical harmonics and the eigenvalues are degenerate with respect to the index ℓ .
The Reproducing Kernel Hilbert
Space (RKHS) of K can be characterised as follows,
[Equation]
A. M ULTI - DOT - PRODUCT KERNELS ON THE MULTI - SPHERE
Mercer’s decomposition of dot-product kernels extends naturally to the case considered in this paper, where the input space is the Cartesian product of p s -dimensional unit sphere, M
[Equation]
(S11) which we refer to as the multi-sphere following the notation of Geifman et al.
After defining a scalar product between functions on M p S s − 1 by direct extension of Eq.
S2, one can immediately find a set of orthonormal polynomials by taking products of spherical harmonics.
With the multiindex notation k = ( k 1 , . . .
, k p ) , ℓ = ( ℓ 1 , . . . , ℓ p ) , for all x ∈ M p S s −˜ Y k , ℓ ( x ) =
p (cid:89) i =
Y k i ,ℓ i ( x i ) , with k i ≥ 0 , ℓ i = 1 , . . . , N k i ,s = 2 k i + s − 2 k i (cid:18) [Equation] (cid:19)
[Equation]
(S12)
These product spherical harmonics ˜
Y k , ℓ ( x ) span the space of square-integrable functions on
M p S s − 1 .
Furthermore, as each spherical harmonic is an eigenfunction of the Laplace-Beltrami operator, ˜ Y k , ℓ is an eigenfunction of the sum of Laplace-Beltrami operators on the p unit spheres, ∆ p,s ˜ Y k , ℓ := (cid:32) p (cid:88) i = ∆ i (cid:33) p (cid:89) i = Y k i ,ℓ i = (cid:32) p (cid:88)
i = (( − k i ) ( k i + s − 2)) (cid:33)
˜ Y k , ℓ . (S13) We can thus characterise the differentiability of functions of the multi-sphere X s,p via finiteness in
L2 norm of some power of ∆ p,s .
Similarly, we can consider products of Legendre polynomials to obtain a set of orthogonal polynomials on [ − 1 , 1] p (see Geifman et al.
(2022), appendix A).
Then, any function f on M p S s − 1 × M p S s −which depends only on the p scalar products between patches,
[Equation]
can be written as a sum of projections on products of Legendre polynomials
[Equation]
Following Geifman et al.
, we call such functions multi-dot-product kernels.
When fixing one of the two arguments of f (say x ), f becomes a function on M p S s − 1 × M p S s − 1 and can be written as a sum of projections on the ˜ Y k , ℓ ’s. The two expansions are related by the following generalised
Funk-Hecke formula,
[Equation]
Having introduced the product spherical harmonics
˜
Y k , ℓ as basis of M p S s − 1 and the product Legendre polynomials ˜
P k ,s ( t ) as basis of [ − 1 , +1] p , the Mercer’s decomposition of multi-dot-product kernels follows immediately.
[Equation]
B RFK AND NTK OF DEEP CONVOLUTIONAL NETWORKS
This appendix gives the functional forms of the RFK and NTK of hierarchical CNNs.
We refer the reader to Arora et al.
for the derivation.
Definition B.1 (RFK and NTK of hierarchical CNNs) Let x ,
y ∈
M p S s − 1 = (cid:81) pi
=1 S s − 1 .
Denote tuples of the kind i l i l +1 . . .
i m with i l → m for m ≥ l .
For m < l , i l → m denotes the empty tuple.
For each tuple i 2 → L +1 , denote with t i 2 → L +1 the scalar product between the s -dimensional patches of x and y identified by the same tuple, i.e.
[Equation]
For 1 ≤ l ≤ L + 1 , denote with (cid:8) t i 2 → L +1 (cid:9) i 2 → l the sequence of t ’s obtained by letting the indices of the tuple i 2 → l vary in their respective range.
Consider a hierarchical CNN with L hidden layers, filter sizes ( s 1 , . . .
, s L ) , p L ≥ 1 and all the weights w (1) h,i , w ( l ) h,h ′ ,i , w ( L +1) h,i initialised as Gaussian random numbers with zero mean and unit variance.
RFK.
The corresponding RFK (or covariance kernel) is a function K ( L +1) RFK of the p
= d/s 1 scalar products t i L ...
i 1 which can be obtained recursively as follows.
With
[Equation]
NTK.
The NTK of the same hierarchical CNN is also a function of the p 1 =
d/s 1 scalar products t i L ...
i 2 which can be obtained recursively as follows.
With κ 0 ( t ) =
( π − arccos t )
/π ,
[Equation]
C S PECTRA OF DEEP CONVOLUTIONAL KERNELS
In this section we state and prove a generalised version of Thm. 3.1 which includes non-binary patches.
Our proof strategy is to relate the asymptotic decay of eigenvalues to the singular behaviour of the kernel, as it is customary in Fourier analysis and was done in Bietti & Bach (2021) for standard dot-product kernel.
In Subsec.
C.
we perform the singular expansion of hierarchical kernels, in Subsec.
C.2 we use this expansion to prove Thm.
3.1 with L = 2 ( 2 hidden layers) and s 1 =(patches on the ring), which we then generalise to general s 1 in Subsec.
C.3 and to general depth in Subsec.
C.4.
Theorem C.1 (Spectrum of hierarchical kernels) Let T K be the integral operator associated with a d -dimensional hierarchical kernel of depth L + 1 , L > 1 and filter sizes ( s 1 , . . .
, s L ).
Eigenvalues and eigenfunctions of T K can be organised into L sectors associated with the hidden layers of the kernel/network.
For each 1 ≤ l ≤ L , the l -th sector consists of ( (cid:81) l l ′ =
1 s l ′ ) - local eigenfunctions: functions of a single meta-patch x i l +1 → L +1 which cannot be written as linear combinations of functions of smaller meta-patches.
The labels k of these eigenfunctions are such that there is a meta-patch k i l
+
→ L +1 of k with no vanishing sub-meta-patches and all the k
i ’s outside of k
i l +
→ L +1 are 0 (be- cause the eigenfunction is constant outside of x i
l +1 → L +1 ).
The corresponding eigenvalue is degenerate with respect to the location of the meta-patch: we call it Λ ( l ) k il +1 → iL +1 .
When ∥ k
i
l
+
→ L +1 ∥ → ∞ , with k = ∥
k i l +
→ L +1 ∥ , i. if s 1 = 2 , then
[Equation]
with ν NTK = 1 / 2 , ν RFK = 3 / 2 and d eff the effective dimensionality of the meta-patches defined in Eq.
C 2 ,l is a strictly positive constant for l ≥ 2 whereas for l = 1 it can take two distinct strictly positive values depending on the parity of k i
→ L +1 .
ii.
if s 1 ≥ 3 , then for fixed non-zero angles k /k ,
[Equation]
where C s 1 ,l is a positive function for l ≥ whereas for l = 1 it is a strictly positive constant which depends on the parity of k i [-
-]2 → L +
C. S INGULAR EXPANSION OF HIERARCHICAL KERNELS
Both the RFK and NTK of ReLU networks, whether deep or shallow, are built by applying the two functions κ 0 and κ 1 Cho & Saul (2009) (see also Def. B.1),
[Equation]
The functions κ 0 and κ 1 are non-analytic in t = ± 1 , with the following singular expansion Bietti &
Bach (2021).
Near t = 1 , with u = 1 − t
[Equation]
Near t = − 1 , with u = 1 + t ,
[Equation]
As a result, hierarchical kernels have a singular expansion when the t i 2 → L +1 ’s are close to ± 1 .
In particular, the following expansions are relevant for computing the asymptotic scaling of eigenvalues.
Proposition C.1 (RFK when x = y )
The RFK of a hierarchical network of depth L + 1 , filter sizes ( s 1 , . . .
, s L ) and p L ≥ 1 has the following singular expansion when all t i 2 → L +1 → 1 .
With u i 2 → L +1 = 1 − t i 2 → L +1 , c =√ 2 / (3 π ) , and (cid:81) l ∈
I s l := 1 if I is the empty set,
[Equation]
[Equation]
Proof.
With L = 1 one has (recall that i 2 → 1+1 = i 2 → 2 reduces to a single index)
[Equation]
With L = 2 , K ( [Equation] i
[Equation]
O
[Equation]
u
[Equation]
[Equation]
(S
[Equation]
therefore
K (2+
[Equation]
RFK (cid:16) { 1 − u i 2 ,i 3 } i 2 ,i 3 (cid:17) =1 − s 2 p(cid:88)
i 2 , i [Equation] The proof of the general case follows by induction by applying the function κ 1 to the singular expansion of the kernel with L − 1 hidden layers, then using Eq.
S24.
Proposition C.2 (RFK when x = − y )
The RFK of a hierarchical network of depth L + 1 , filter sizes ( s 1 , . . .
, s L ) and p L ≥ 1 has the following singular expansion when all t i 2 → L +1 → − 1 .
With u i 2 → L +1 = 1 + t i 2 → L +1 , c =√ 2 / (3 π ) and (cid:81) l ∈
I s l := 1 if I is the empty set,
[Equation]
with b L = κ 1 ( b L − 1 ) , b 1 = 0 ; and c L
= c L − 1 κ ′ 1 ( b L − 1 ) , c 1 = c .
Proof.
This can be proved again by induction.
For L = 1 ,
[Equation]
Thus, for L = 2 ,
[Equation]
so that
[Equation]
The proof is completed by applying the function κ 1 to the singular expansion of the kernel with
L − 1 hidden layers.
Proposition C.3 (NTK when x = y )
The NTK of a hierarchical network of depth L + 1 , filter sizes ( s 1 , . . .
, s L ) and p L ≥ 1 has the following singular expansion when all t i 2 → L +1 → 1 .
With u i 2 → L +1 = 1 − t i 2 → L +1 , c = √ 2 π , and (cid:81) l ∈
I s l := 1 if I is the empty set,
K
[Equation]
Proposition C.4 (NTK when x = − y )
The NTK of a hierarchical network of depth L + 1 , filter sizes ( s 1 , . . .
, s L ) and p L ≥ 1 has the following singular expansion when all t i 2 → L +1 → − 1 .
With u i 2 → L +1 = 1 + t i 2 → L +1 , c = √ 2 /π and (cid:81) l ∈
I s l := 1 if I is the empty set,
[Equation]
with a L = b L + b L − 1 κ 0 ( b L − 1 ) , b L = κ 1 ( b L − 1 ) , b 1 = 0 ; and
c L = c L − 1 κ 0 ( b L − 1 ) , c 1 = c .
Notice that both κ 1 and κ 0 are positive and strictly increasing in [0 , 1] and κ 1 (1) = κ 0 (1)
= 1 ,
thus b L ∈ (0 , 1) and c L < c L − 1 .
The proofs of the two propositions above are omitted, as they follow the exact same steps as the previous two proofs.
C. P ATCHES ON THE RING
In this section, we prove a restricted version of Thm.
3.1 for the case of 2 -dimensional input patches, since the reduction of spherical harmonics to the Fourier basis simplifies the proof significantly.
We also consider, for convenience, hierarchical kernels of depth 3 with the filter size of the second hidden layer set to p = d/ 2 , the total number of 2 -patches of the input.
Once this case is understood, extension to arbitrary filter size and arbitrary depth is trivial.
Theorem C.2 (Spectrum of depth- 3 kernels on 2 -patches) Let T K be the integral operator associated with a d -dimensional hierarchical kernel of depth 3 , ( 2 hidden layers), with filter sizes ( s 1 = 2 , s 2 ) and
p 2 = 1 , such that 2 s 2 =
d and s 2 = p (the number of 2 -patches).
Eigenvalues and eigenfunctions of T K can be organised into 2 sectors associated with the hidden layers of the kernel/network.
i.
The first sector consists of s 1 - local eigenfunctions, which are functions of a single patch x i for i = 1 , . . .
, p .
The labels k , ℓ of local eigenfunctions are such that all the k j ’s with j ̸
= i are zero (because the eigenfunction is constant outside x i ).
The corresponding eigenvalue is degenerate with respect to the location of the patch: we call it Λ (1) k i .
When k i → ∞
[Equation]
with ν NTK = 1 / 2 , ν RFK = 3 / 2 .
C 2 ,l can take two distinct strictly positive values depending on the parity of k i ; ii.
The second sector consists of global eigenfunctions, which are functions of the whole input x .
The labels k , ℓ of global eigenfunctions are such that at least two of the k
i ’s are nonzero.
We call the corresponding eigenvalue Λ (2) k .
When ∥ k ∥ → ∞ , with k = ∥
k ∥ ,
[Equation]
Proof.
If we consider binary patches in the first layer, the input space becomes the Cartesian product of two-dimensional unit spheres, i.e. circles, X = (cid:81) di
=1 S 1 .
Then, each patch x i corresponds to an angle θ i and the spherical harmonics are equivalent to Fourier atoms,
[Equation]
Therefore, solving the eigenvalue problem for a dot-product kernel K ( x · y ) =
K (cos( θ x − θ y )) with x , y ∈ S 1 reduces to computing its Fourier transform.
With | S 0 | = 2 and | S 1 | = 2 π , 2 π
[Equation]
where we denoted with θ the difference between the two angles.
Similarly, for a multi-dot-product kernel, the eigenvalues coincide with the p -dimensional Fourier transform of the kernel, where p is the number of patches,
[Equation]
with k = ( k 1 , . . .
, k p )
⊤ the vector of the patch wavevectors and θ =
( θ 1 , . .
. , θ p ) ⊤ the vector of the patch angle differences θ i = θ x,i − θ y,i .
The nonanaliticity of the kernel at t i = 1 for all i moves to θ i = 0 for all i , whereas those in t i = −move to θ i = π and − π .
The corresponding singular expansion is obtained from Eq.
S26 after replacing t i with cos ( θ i ) and expanding cos ( θ i ) as 1 − θ 2 i / 2 , resulting in
[Equation]
The first nonanalytic terms are 13 πp (cid:80) [-p i =
-]pi
= | θ i | 3 and 2 √ 2 3 π (cid:16) 1 p (cid:80) {+pi =
+}
θ 2 i 2 (cid:17) 3 / 2 .
After recalling that
the
Fourier transform of ∥ θ ∥ 2 ν with θ ∈ R p decays asymptotically as ∥ k
∥ − 2 ν − p Widom (1963), one has ( ν = 3 / 2 )
[Equation]
for
[Equation]
and
[Equation]
for
[Equation]
All the other terms in the kernel expansion will result in subleading contributions in the Fourier transform.
Therefore, the former of the two equations above yields the asymptotic scaling of eigenvalues of the local sector, whereas the latter yields the asymptotic scaling of the global sector.
The proof for the NTK case is analogous to the RFK case, except that the singular expansion near θ i = 0 is given by
[Equation]
C. P ATCHES ON THE s - DIMENSIONAL HYPERSPHERE
In this section, we make an additional step towards Thm. 3.1 by extending Thm.
C.2 to the case of s -dimensional input patches.
We still consider hierarchical kernels of depth 3 with the filter size of the second hidden layer set to p =
d/s (the total number of s -patches of the input) so as to ease the presentation.
The extension to general depth and filter sizes is presented in Subsec.
C.4.
Theorem C.3 (Spectrum of depth- 3 kernels on s -patches) Let T K be the integral operator associated with a d -dimensional hierarchical kernel of depth 3 , ( 2 hidden layers), with filter sizes ( s 1 = s, s 2 ) and p 2 = 1 , such that 2 s 2 =
d
and s 2 = p (the number of s -patches).
Eigenvalues and eigenfunctions of T K can be organised into 2 sectors associated with the hidden layers of the kernel/network.
i.
The first sector consists of s 1 - local eigenfunctions, which are functions of a single patch x i for i = 1 , . . .
, p .
The labels k , ℓ of local eigenfunctions are such that all the k j ’s with j ̸
= i are zero (because the eigenfunction is constant outside of x i ).
The corresponding eigenvalue is degenerate with respect to the location of the patch: we call it Λ (1) k i .
When k i → ∞ ,
[Equation]
with ν NTK = 1 / 2 , ν RFK = 3 / 2 .
C s, 1 can take two distinct strictly positive values depending on the parity of k i ; ii.
The second sector consists of global eigenfunctions, which are functions of the whole input x .
The labels k , ℓ of global eigenfunctions are such that at least two of the k
i ’s are nonzero.
We call the corresponding eigenvalue Λ (2) k .
When k ≡ ∥ k
∥ → ∞ , for fixed non-zero angles k /k ,
[Equation]
where C s, 2 is a positive function.
Proof.
A hierarchical RFK/NTK is a multi-dot-product kernel, therefore its eigenfunctions are products of spherical harmonics ˜
Y k , ℓ ( x ) =
(cid:81) pi
=1 Y k
i ,ℓ i ( x i ) and the eigenvalues of K are given by Eq.
S17,
[Equation]
The proof follows the following strategy: first, we show that the infinitely differentiable part of K results in eigenvalues which decay faster than any polynomial of the degrees k i .
We then show that the decay is controlled by the most singular term of the singular expansion of the kernel and finally compute such decay by relating it to the number of derivatives of the kernel having a finite l2 norm.
When K is infinitely differentiable in [ − 1 , +1] p , we can plug Rodrigues’ formula Eq.
S5 for each
P k i ,s ( t i ) and get
[Equation]
with (cid:82) +1 − 1 d t denoting integration over the p -dimensional hypercube [ − 1 , +1] p .
We can simplify the integral further via integration by parts, so as to obtain
[Equation]
where K ( k ) denotes the partial derivative of order k 1 with respect to t 1 , k 2 with respect to t 2 and so on until k p with respect to t p .
Notice that the function (1 − t 2 ) d − 32 is proportional to the probability measure of the scalar product t between two points sampled uniformly at random on the unit sphere (Atkinson & Han, 2012, Sec. 1.3),
[Equation]
This probability measure converges weakly to a Dirac mass δ ( t ) when d → ∞ .
Recall, in addition, that | S d − 1 | = 2 π d/ 2 / Γ( d/ 2) , where Γ denotes the Gamma function Γ( x ) =
(cid:82) ∞
dx x z − 1 e − x .
Thus, with converges weakly to a Dirac measure δ ( t ) as c → ∞ , once properly rescaled.
In particular, choosing k i such that k
i
+ ( s {+− 3) / 
+}
= ( d − 3) / 2 , one has
[Equation]
As a result, when K is infinitely differentiable, one has the following equivalence in the limit where all k
i ’s are large,
[Equation]
which implies that, when K is infinitely differentiable, the eigenvalues decay exponentially or faster with the k i .
Let us now consider the nonanalytic part of K .
There are three kinds of terms appearing in the singular expansion of depth- 3 kernels (cf. Subsec.
C.1):
ia) c
+ (cid:80) i (1 − t i )
ν near t i = + 1 ; ib)
c − (cid:80) i
(1 + t i )
ν near t i = − 1 ; ii) c
+ , all ( (cid:80) i (1 − t i ) /p )
ν near t i =
+ 1 for all i ;
where the exponent ν is 1 / 2 for the NTK and 3 / 2 for the RFK.
We will not consider terms of the kind ib) explicitly, as the analysis is equivalent to that of terms of the kind ia) .
After replacing t i with cos( θ i ) , as in Subsec.
C.2, we get again (cid:80) i | θ i | 2 ν and ∥
θ ∥ 2 ν as leading nonanalytic terms.
Therefore, we can rewrite the nonanalytic part of the kernel as follows,
[Equation]
where f 1 , f 2 are single-variable functions which behave as θ 2 ν near zero and have compact support, whereas ˜ K has a singular expansion near θ i
=
analogous to that of K but with leading nonanalyticities controlled by an exponent ν ′ ≥ ν
+ 1 .
Let us look at the contribution to the eigenvalue Λ k due to the term f 1 ( | θ i | ) :
[Equation]
where we have introduced ( f 1 ) k as the projection of f 1 ( θ ) on the k -th Legendre polynomial.
The asymptotic decay of ( f 1 ) k is strictly related to the differentiability of f 1 , which is in turn controlled by action of the Laplace-Beltrami operator ∆ on f 1 .
As a function on the sphere S s − 1 , f 1 depends only on one angle, therefore the Laplace-Beltrami operator acts as follows,
[Equation]
In terms of singular behaviour near θ = 0 , f 1 ( θ ) ∼ | θ | 2 ν implies ∆ f 1 ( θ ) ∼ | θ | 2 ν − 2 , thus ∆ m f 1 ( θ ) ∼ | θ | 2( ν − m ) .
Given ν , repeated applications of ∆ eventually result in a function whose l norm on the sphere diverges.
On the one hand,
[Equation]
The integrand behaves as | θ | d − 2+4 ν − 2 m near 0 , thus the integral diverges for m ≥ 2 ν
+ ( d − 1) / 2 .
On the other hand, from Eq.
S3,
[Equation]
As N k,s ∼ k s − 2 and the sum must converge for m < 2 ν
+( d − 1) / 2 and diverge otherwise, ( f 1 )
k ∼ k − 2 ν − ( s − 1) .
The projections of all the other terms in K on Legendre polynomials of one of the p angles θ i display a faster decay with k , therefore the above results imply the asymptotic scaling of local eigenvalues.
Notice that such scaling matches with the result of Bietti & Bach (2021), which was obtained with a different argument.
Finally, let us look at the contribution to the eigenvalue Λ k due to the term f 2 ( ∥ θ ∥ ) :
[Equation]
where we have introduced ( f 2 ) k as the projection of f 2 ( ∥ θ ∥ ) on the multi-Legendre polynomial with multi-degree k .
The asymptotic decay of ( f 2 ) k is again related to the differentiability of f 2 , controlled by action of the multi-sphere Laplace-Beltrami operator ∆ p,s in Eq.
S13.
As f 2 depends only on one angle per sphere,
[Equation]
Further simplifications occur since f 2 depends only on the norm of θ .
In terms of the singular behaviour near ∥ θ ∥ = 0 , f 2 ∼ ∥ θ ∥ 2 ν implies ∆ mp,s f 2 ∼ ∥ θ ∥ 2( ν − m ) , thus
[Equation]
requires m < 2 ν + p
( s − 1) / 2 (compare with m < 2 ν
+ ( s − 1) / 2 for the local contributions).
Therefore, one has
[Equation]
while the sum diverges for m ≥ 2 ν + p ( s − 1) / 2 .
In addition, since f 2 is a radial function of θ which is homogeneous (or scale-invariant) near ∥ θ ∥ = 0 , ( f 2 ) k can be factorised in the large- ∥ k
∥ limit into a power of the norm ∥ k
∥ α and a finite angular part C ( k / ∥ k ∥ ) .
By plugging the factorisation into Eq.
S61, we get
[Equation]
The projections of all the other terms in K on multi-Legendre polynomials display a faster decay with ∥ k ∥ , therefore the above results imply the asymptotic scaling of global eigenvalues.
C. G ENERAL DEPTH
The generalisation to arbitrary depth is trivial once the depth- 3 case is understood.
For global and s 1 -local eigenvalues, the analysis of the previous section carries over unaltered.
All the other intermediate sectors correspond to the other terms singular expansion of the kernel: from Subsec.
C.1, these terms can be written as
[Equation]
 
[Equation]
S
[Equation]
for some l ′ = 2 , . . .
, L − 1 and fractional ν .
In practice, this term is a sum over the p l ′ =
p L [Equation] l ′ <l ′′ ≤ L s l ′′ meta-patches of t having size s 2 → l ′ :=
(cid:81) 2 ≤ l ′′ ≤ l ′ s l ′′ .
Each summand is the fractional power ν of the average of the t
i ’s within a meta-patch.
When plugging such term into Eq.
S47, the integrals over the t i ’s which do not belong to that meta-patch yield Kronecker deltas for the corresponding k i ’s.
The integrals over the t i ’s within the meta-patch, instead, can be written as in Eq.
S58 with the product and the norm restricted over the elements of that meta-patch, i.e., ∥ θ ∥ → (cid:16)(cid:80) i 2 → l ′ θ 2 i 2 → L +1 (cid:17) 1 / 2 .
Therefore, the scaling of the eigenvalue with k is given again by Eq.
S63, but with p replaced by the size of the meta-patch (cid:81) 2 ≤ l ′′ ≤ l ′ s l ′′ , so that the effective dimension of Eq.
3 appears at the exponent.
D G ENERALISATION BOUNDS FOR KERNEL REGRESSION AND SPATIAL ADAPTIVITY
This appendix provides an introduction to classical generalisation bounds for kernel regression and extends Cor.
4.1 to patches on the hypersphere.
D. C LASSICAL GENERALISATION BOUNDS
Consider the regression setting detailed in Sec.
4 of the main text.
First, assume that the target function f ∗ belongs to the RKHS H of the kernel K .
Then, without further assumptions on K , we have the following dimension-free bound on the excess risk, based on Rademacher complexity (Bach, 2021, Chs. 4, 7), Bietti (2022),
[Equation]
where T K is the integral operator associated to K .
For a hierarchical kernel, having a target with more power in the local sectors can result in a smaller ∥ f ∗ ∥ H , hence a smaller excess risk.
However, this gain is only a constant factor in terms of sample complexity and, more importantly, being in the
RKHS requires an order of smoothness which typically is of the order of the dimension, which is a very-restrictive assumption in high-dimensional settings.
This result can be extended by including more details about the kernel and the target function.
In particular, (Bach, 2021, Prop. 7.2) states that, for f ∗ in the closure of H , regularisation λ ≤and n ≥ 5 λ (1 + log(1 /λ )) , one has
[Equation]
where σ 2 bounds the conditional variance of the labels, i.e. E ( x ,y ) ∼ p (cid:104) ( [Equation] |
[Equation]
Then, let us consider the following standard assumptions in the kernel literature Caponnetto &
De Vito (2007),
[Equation]
In short, the first assumption characterises the ‘size’ of the RKHS (the larger α , the smaller the number of functions in the RKHS), while the second assumption defines the regularity of the target function relative to that of the kernel (when r = 1 , f ∗ ∈ H ; when r < 1 , f ∗ is less smooth; when r > 1 , f ∗ is smoother).
Combining these assumptions with Eq.
S65, one gets
[Equation]
Optimising for λ results in
[Equation]
and the bound becomes
[Equation]
Finally, when r > ( α − 1) /α , n ≥ 5 λ n (1 + log(1 /λ n )) is always satisfied for n large enough.
D. E XTENSION OF C OR . 4.1 TO PATCHES ON THE HYPERSPHERE
Corollary D.1 (Adaptivity to spatial structure) Let T K be the integral operator of the kernel of a hierarchical deep CNN as in Thm. 3.1.
Then: i) the capacity exponent α is controlled by the largest sector of the spectrum, i.e.
[Equation] ii) the source exponent r is controlled by the structure of the target function f ∗ , i.e., if there is l ≤ L such that f ∗ depends only on some meta-patch x i
l +1 → L +1 , then only the first l sectors of the spectrum contribute to the source condition,
[Equation]
The same holds if f ∗ is a linear combination of such functions.
As a result, when d eff ( L ) is large and α → 1 , the decay of the error is controlled by the effective dimensionality of the target d eff ( l ) .
D. D ECAY OF EIGENVALUES WITH THE RANK
Shallow kernels.
Consider a depth-two kernel with filters of size s . Our goal is to compute the scaling of the eigenvalues of the kernel Λ k with their rank ρ . The eigenvalues decay with k as [Equation]
In order to take into account their algebraic multiplicity, we introduce the eigenvalue density D (Λ) , whose asymptotic form for small eigenvalues is
[Equation]
Thus, the scaling of Λ( ρ ) can be determined self-consistently,
[Equation]
Deep kernels.
Consider a kernel of depth L + 1 with filter sizes ( s 1 , . .
, s L ) and p L = 1 .
For each sector l , one can compute the density of eigenvalues D ( l ) (Λ) .
Depending on s 1 , there are two different cases.
If s 1 = 2 ,
[Equation]
If s 1 ≥ 3 ,
[Equation]
When summing over all layers l ’s, the asymptotic behaviour of the total density of eigenvalues
D (Λ) =
(cid:80) l D ( l ) (Λ) is dictated by the density of the sector with the slowest decay, i.e. the last one.
Hence,
[Equation]
Therefore, similarly to the shallow case, one finds self-consistently that the ρ -th eigenvalue of the kernel decays as
[Equation]
E S TATISTICAL MECHANICS OF GENERALISATION IN KERNEL REGRESSION
In Bordelon et al.
Canatar et al.
(2021), the authors derived a heuristic expression for the average-case mean-squared error of kernel (ridge) regression with the replica method of statistical physics M´ezard et al.
Denoting with { ϕ ρ ( x ) , Λ ρ } ρ ≥ 1 the eigenfunctions and eigenvalues of the kernel and with c ρ the coefficients of the target function in this basis, i.e. f ∗ ( x ) =
(cid:80) ρ ≥ 1 c
ρ
ϕ
ρ ( x ) , one has
[Equation]
where λ is the ridge and κ ( n ) satisfies the implicit equation
[Equation]
In short, the replica calculation used to obtain these equations consists in defining an energy functional E ( f ) related to the empirical MSE and assigning to the predictor f a Boltzmann measure, i.e. P ( f ) ∝ e − βE ( f ) .
When β → ∞ , the measure concentrates around the minimum of E ( f ) , which coincides with the minimiser of the empirical MSE.
Then, since E ( f ) depends only quadratically on the projections c ρ , computing the average over data that appears in the definition of the generalisation error, reduces to computing Gaussian integrals.
While non-rigorous, this method has been successfully used in physics—to study disordered systems—and in machine learning theory.
In particular, the predictions obtained with Eq.
S79 and Eq.
S80 have been validated numerically for both synthetic and real datasets.
In Eq.
S79, κ λ ( n ) /n plays the role of a threshold: the modal contributions to the error tend tofor ρ such that Λ
ρ ≫ κ λ ( n ) /n , and to E [ c 2 ρ ] for ρ such that Λ ρ ≪ κ λ ( n ) /n .
This is equivalent to saying that kernel regression can capture only the modes corresponding to the eigenvalues larger than κ λ ( n ) /n
(see also Jacot et al.
(2020a;b)).
In the ridgeless limit λ → 0 + , this threshold asymptotically tends to the n -th eigenvalue of the student, resulting in the intuitive picture presented in the main text.
Namely, given n training points, ridgeless regression learns the n projections corresponding to the highest eigenvalues.
In particular, assume that the kernel spectrum and the target function projections decay as power laws.
Namely, Λ ρ ∼ ρ − a and E [ c ρ 2 ] ∼ ρ − b , with 2 a > b − 1 .
Furthermore, we can approximate the summations over modes with an integral by using the Euler-MacLaurin formula.
Hence, we substitute the eigenvalues with their asymptotic limit Λ ρ =
Aρ − a .
Since, κ 0 ( n ) /n
→ 0 as n → ∞ , these two operations result in an error which is asymptotically independent of n .
In particular,
[Equation]
Since the integration over σ is finite and independent of n , we obtain that κ 0 ( n ) /n
=
O ( n − a ) .
Similarly, we find that the mode-independent prefactor ∂ λ ( κ λ ( n ) /n )
| λ =
= O (1) .
As a result, we have
[Equation]
Following the intuitive argument about the thresholding action of κ 0 ( n ) /n ∼ n − a , we can split the summation in Eq.
S82 into modes where Λ ρ ≫ κ 0 ( P ) /n , Λ ρ ∼ κ 0 ( n ) /n and Λ ρ ≪ κ 0 ( n ) /n ,
[Equation]
Finally, Eq. 12 is obtained by noticing that, under the assumption on the decay of E [ c 2 ρ ] , the contribution of the summation over ρ ≪ n is subleading in n , whereas the other two can be merged together.
F E XAMPLES F. R ATES FROM SPECTRAL BIAS ANSATZ
Consider a target function f ∗ which only depends on the meta-patch x i l +
→ L +1 and with squareintegrable derivatives up to order m , i.e. ∥ ∆ m/ 2 f ∗ ∥ 2 < + ∞ , with ∆ denoting the Laplace operator.
Moreover, consider a hierarchical kernel of depth L + 1 with filter sizes ( s 1 , . . .
, s L ) and p L = 1 .
We want to compute the asymptotic scaling of the error by using Eq.
12, i.e.
[Equation]
In the previous section, we showed that the n -th eigenvalue of the kernel Λ( n ) decays as
[Equation]
Since by construction the target function depends only on a meta-patch of the l -th sector, the only non-zero projections will be the ones on eigenfunctions of the first l sectors.
Thus, all the k ’s corresponding to the sectors of layers with l ′ >
l do not contribute to the sum.
In particular, the sum is dominated by the k ’s of the largest sector and the set { k s.t.
Λ k < Λ( n ) } is the set of k
i l
+
→ L +1 ’s with norm larger than n
[Equation]
Finally, we notice that the finite-norm condition on the derivatives,
[Equation]
i ,s (cid:33) (cid:32) p (cid:88) i
= k
i ( k i + s − 2) (cid:33) m | f ∗ k , ℓ | 2 < + ∞ , (S86) implies | f ∗ k , ℓ
| 2 ≲ ∥ k ∥ − 2 m − d eff ( L ) (see Subsec.
C.3).
Hence, plugging everything in Eq.
S84 we find
[Equation]
(S87)
G N UMERICAL EXPERIMENTS G. E XPERIMENTAL SETUP
Experiments were run on a high-performance computing cluster with nodes having Intel Xeon Gold processors with 20 cores and 192 GB of DDR4 RAM.
All codes are written in PyTorch Paszke et al.
G. T EACHER - STUDENT LEARNING CURVES
In order to obtain the learning curves, we generate n + n test random points uniformly distributed on the product of hyperspheres over the patches.
We use n ∈ { 128 , 256 , 512 , 1024 , 2048 , 4096 , 8192 } and n test = 8192 .
For each value of n , we sample a Gaussian random field with zero mean and covariance given by the teacher kernel.
Then, we compute the kernel regression predictor of the student kernel, and we estimate the generalisation error as the mean squared error of the obtained predictor on the n test unseen example.
The expectation over the teacher randomness is obtained by averaging over 16 independent sets of random input points and realisations of the Gaussian random fields.
As teacher and student kernels, we use the analytical forms of the neural tangent kernels of hierarchical convolutional networks, with different combinations of depths and filter sizes. {+[Figure] T: (2), S: (2) = 1 s 
+}
T: (2), S: (2, 2) = 1+ d eff s 1 d eff a 1010 4 n [Figure] T: (2), S: (2, 2) = 1+ d eff s 1 d eff T, S: (2, 2) =
1/ d eff
[Figure]
Depth-two and depth-three architectures.
Fig.
S1 reports the learning curves of depth-two and depth-three kernels with binary filters at all layers.
Depth-three students defeat the curse of dimensionality when learning depth-two teachers, achieving a similar performance of depth-two students matched to the teacher’s structure.
However, as we predict, these students encounter the curse of dimensionality when learning depth-three teachers.
Ternary filters.
Fig.
S2 reports the learning curves for kernels with 3-dimensional filters and confirms our predictions in the s 1 ≥ 3 case.
Comparison with the noisy and optimally-regularised case.
Panel (a) of Fig.
S3 compares the learning curves obtained in the optimally-regularised and ridgeless cases for noisy and noiseless data, respectively.
The first case corresponds to the setting studied in Caponnetto & De Vito (2007), in which the source-capacity formalism applies.
In contrast with the second setting—which is the one used in the teacher-student scenarios and where it holds the correspondence between kernel methods and neural networks— i) we add to the labels a Gaussian random noise with standard deviation σ = 0 .
1 , ii) for each n , we select the ridge resulting in the best generalisation performance.
We observe that the decay obtained in the bound derived from the source-capacity conditions is exactly the one found numerically, i.e. the rate of the bound is tight.
As a further check, panel (b) shows that the optimal ridge decays as prescribed.
G. I LLUSTRATION OF DIFFERENT TEACHER - STUDENT SCENARIOS
In this subsection, we comment on the results obtained in the different teacher-student scenarios of
Fig. 2, panel (a), and Fig.
S1, panel (a).
To ease notation, in the following we always consider the
NTK for both teacher and student kernels, i.e. smoothness exponent ν T = ν S
= 1 / 2 .
However, we point out that when the teacher kernel is a hierarchical RFK ( ν T = 3 / 2 ), the target function corresponds to the output of an infinitely-wide, deep hierarchical network at initialisation 4 .
The error rates are obtained from Eq.
17, after setting the smoothness exponent m =
ν T (the smoothness exponent of the teacher covariance kernel). [Figure] [Figure]
The first case we consider consists of one-hidden-layer convolutional teacher ( and student ) kernels.
[Equation]
As highlighted in blue, the output of the teacher is a linear combination (dashed lines indicate the linear output weights) of s 1 -dimensional functions of the input patches.
If the structure of the student is matched to the one of the teacher, the learning problem becomes effectively ( s 1 − 1) -dimensional
and the error decays as n − 1 / ( s 1 − 1) , instead of n − 1 /d eff , with d eff the total input dimension with the number of spherical constraints subtracted (one per patch).
Notice that the role of the student’s structure, i.e. the algorithm, is as crucial as the role of the teacher, i.e. the task.
Indeed, using a fully-connected student with no prior on the task’s locality would result in an error’s decay cursed by dimensionality.
However, in contrast to fully-connected students, shallow convolutional students are only able to learn tasks with the same structure.
In particular, any task entailing non-linear interactions between patches—which are arguably crucial in order to learn image data—belongs to their null space.
As we illustrated in the main text, to solve this strong constraint on the hypothesis space, one has to consider deep convolutional architectures.
In particular, consider the same shallow teacher of the previous paragraph (left) learnt by a depth-four convolutional student (right).
[Equation]
Remarkably, this student is able to learn the teacher without being cursed by input dimensionality.
Indeed, as the number of patches diverges, the error decay asymptotes to n − 1 /s 1 .
This rate is slightly worse than the one obtained by the student matched with the teacher, which is proven to be the
Bayes-optimal case, but far from being cursed.
Intuitively, this fast rate is obtained because the student eigenfunctions of the first sector, i.e. constant outside a single patch, correspond to large eigenvalues and bias the learning dynamics towards s 1 -local functions.
Yet, this student is also able to represent functions which are considerably more complex.
Now consider a depth-three teacher (left) learned by a depth-four student (right).
[Equation]
As highlighted in orange, the output of the teacher is a linear combination of a composition of nonlinear functions acting on patches and coupling them.
In this setting, the error decay is controlled by the effective dimension of the second layer.
In fact, when the number of patches diverges, the error decay asymptotes to n − 1 /d eff (2) .
In general, this behaviour is a result of what we called ‘adaptivity to the spatial structure’ of the target.
Finally, consider both teacher and student with the complete hierarchy, i.e. the receptive fields of the neurons in the penultimate layers coincide with the full input.
[Equation]
In this case, we show that the error decays as n − 1 /d eff (3) , i.e. the rate is cursed by the input dimension.
The physical meaning of this result is that the hierarchical structure we are considering is still too complex and cannot be learnt efficiently.
In other words, these hierarchical convolutional networks are excellent students, since they can adapt to the spatial structure of the task, but bad teachers, since they generate global functions which are too complex to be learnt efficiently.
G. E XTENSIONS TO DIFFERENT NORMALISATIONS AND OVERLAPPING PATCHES
This section investigates the robustness of our results to changes in the input distribution, i.e., for data outside the multisphere M p S s − 1 , and relaxes the non-overlapping patches assumption.
Inputs in R d .
While our analysis requires that each patch of the input data is normalised to lie on a unit sphere, this normalisation is not the standard one used for neural networks.
Therefore, in this section we investigate the robustness of our predictions to the data distribution.
In particular, we consider data uniformly distributed in the unit hypercube, i.e., x ∈ [0 , 1] d , and data with standard Gaussian distribution, i.e., x ∼ N (0 , I d ) . First, we extend the definition of the RFK and NTK to inputs in R d . Definition G.1 (RFK and NTK of hierarchical CNNs for inputs in R d ) Let x , y ∈ R d .
Denote tuples of the kind i l i l +1 . . .
i m with i l → m for m ≥ l .
For m < l , i l → m denotes the empty tuple.
For each tuple i 2 → L +1 and s a divisor of d , denote with t i 2 → L +1 the angle between the s -dimensional patches of x and y identified by the same tuple, i.e.
[Equation]
For 1 ≤ l ≤ L + 1 , denote with (cid:8) x i 2 → L +1 , y i 2 → L +1 (cid:9) i 2 → l the sequence of patches obtained by letting the indices of the tuple i 2 → l vary in their respective range.
Consider a hierarchical CNN with filter sizes ( s 1 , . . .
, s L ) , p L ≥ 1 and all the weights w (1) h,i , w ( l ) h,h ′ ,i , w ( L +1) h,i initialised as Gaussian random numbers with zero mean and unit variance.
RFK.
The corresponding RFK (or covariance kernel) can be obtained recursively as follows.
With κ
[Equation]
[Table]
[Figure]
NTK.
The NTK of the same hierarchical CNN can be obtained recursively as follows.
With κ 0 ( t ) =
( π − arccos t )
/π ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Fig.
S4 reports the learning curve of different teacher-student scenarios with the kernels defined in
Def. G.1 and inputs i) on the multisphere M p S s − 1 , ii) uniformly-distributed in the unit d -hypercube [0 , 1] d , and iii) with standard Gaussian distribution N (0 , I d ) .
Remarkably, our predictions are in excellent agreement with the different input normalisations.
Overlapping patches.
Fig.
S5 shows the comparison between convolutional kernels with nonoverlapping patches, i.e., stride corresponding to the filter size, and overlapping patches, i.e., stride 1, for inputs uniform in the d -dimensional hypercube.
Despite our theoretical analysis requires the patches to be non-overlapping, our predictions are still confirmed for architectures with overlapping patches.
G. CIFAR-2 LEARNING CURVES
Fig.
S6 shows the learning curves of the neural tangent kernels of different architectures applied to pairs of classes of the CIFAR-10 dataset.
In particular, the task is built by selecting two CIFARclasses, e.g. plane and car, and assigning label +1 to the elements belonging to one class and label − 1 to the remaining ones.
Learning is again achieved by minimising the empirical mean squared error using a ‘student’ kernel.
We find that the kernels with the worst performance are the ones corresponding to shallow fully-connected and convolutional architectures.
Instead, for all the pairs of classes considered here, deep hierarchical convolutional kernels achieve the best performance. [Figure]
[Figure]