A SYMPTOTIC LEARNING CURVES OF KERNEL METHODS : EMPIRICAL DATA
v.s.
T EACHER -S TUDENT
PARADIGM
Anonymous authors
Paper under double-blind review
A BSTRACT
How many training data are needed to learn a supervised task?
It is often observed that the generalization error decreases as n − β where n is the number of training examples and β an exponent that depends on both data and algorithm.
In this work we measure β when applying kernel methods to real datasets.
For MNIST we ﬁnd β ≈ 0 .
4 and for CIFAR10 β ≈ 0 .
Remarkably, β is the same for regression and classiﬁcation tasks, and for Gaussian or Laplace kernels.
To rationalize the existence of non-trivial exponents that can be independent of the speciﬁc kernel used, we introduce the Teacher-Student framework for kernels.
In this scheme, a
Teacher generates data according to a Gaussian random ﬁeld, and a Student learns them via kernel regression.
With a simplifying assumption — namely that the data are sampled from a regular lattice — we derive analytically β for translation invariant kernels, using previous results from the kriging literature.
Provided that the Student is not too sensitive to high frequencies, β depends only on the training data and their dimension.
We conﬁrm numerically that these predictions hold when the training points are sampled at random on a hypersphere.
Overall, our results quantify how smooth Gaussian data should be to avoid the curse of dimensionality, and indicate that for kernel learning the relevant dimension of the data should be deﬁned in terms of how the distance between nearest data points depends on n .
With this deﬁnition one obtains reasonable effective smoothness estimates for
MNIST and CIFAR10.
I NTRODUCTION
In supervised learning machines learn from a ﬁnite collection of n training data, and their generalization error is then evaluated on unseen data drawn from the same distribution.
How many data are needed to learn a task is characterized by the learning curve relating generalization error to n .
In various cases, the generalization error decays as a power law n − β , with an exponent β that depends on both the data and the algorithm.
In (Hestness et al., 2017) β is reported for state-of-the-art (SOTA) deep neural networks for various tasks: in for neural-machine translation β ≈ 0 . 3 – 0 .
36 (for ﬁxed model size) or β ≈ 0 .
13 (for best-ﬁt models at any n ); language modeling shows β ≈ 0 .
06 – 0 . 09 ; in speech recognition
β ≈ 0 . 3 ; SOTA models for image classiﬁcation (on ImageNet) have exponents β ≈ 0 .
Currently there is no available theory of deep learning to rationalize these observations.
Recently it was shown that for a proper initialization of the weights, deep learning in the inﬁnite-width limit (Jacot et al., 2018) converges to kernel learning.
Moreover, it is nowadays part of the lore that there exist kernels whose performance is nearly comparable to deep networks (Bruna and Mallat, 2013; Arora et al., 2019), at least for some tasks.
It is thus of great interest to understand the learning curves of kernels.
For regression, if the target function being learned is simply assumed to be Lipschitz, then the best guarantee is β = 1 / d (Luxburg and Bousquet, 2004; Bach, 2017) where d is the data dimension.
Thus for large d , β is very small: learning is completely inefﬁcient, a phenomenon referred to as the curse of dimensionality .
As a result, various works on kernel regression make the much stronger assumption that the training points are sampled from a target function that belongs to the reproducing kernel Hilbert space (RKHS) of the kernel (see for example (Smola et al., 1998)).
With this assumption β does not depend on d (for instance in (Rudi and Rosasco, 2017) β
= 1 /is guaranteed).
Yet, RKHS is a very strong assumption which requires the smoothness of the target function to increase with d (Bach, 2017)
(see more on this point below), which may not be realistic in large dimensions.
In this work we compute β empirically for kernel methods applied on MNIST and CIFAR10 datasets.
We ﬁnd β MNIST ≈ 0 . 4 and β CIFAR10 ≈ 0 .
1 respectively.
Quite remarkably, we observe essentially the same exponents for regression and classiﬁcation tasks, using either a Gaussian or a Laplace kernel.
Thus the exponents are not as small as 1 / d ( d = 784 for MNIST, d = 3072 for CIFAR10), but neither are they 1 / 2 as one would expect under the RKHS assumption.
These facts call for frameworks in which assumptions on the smoothness of the data can be intermediary between Lipschitz and RKHS.
Here we propose such a framework for regression, in which the target function is assumed to be a
Gaussian random ﬁeld of zero mean with translation-invariant isotropic covariance K T ( x ) .
The data can equivalently be thought as being synthesized by a “Teacher” kernel K T ( x ) .
Learning is performed with a “Student” kernel K S ( x ) that minimizes the mean-square error.
In general K T ( x ) (cid:54) =
K S ( x ) .
In this set-up learning is very similar to a technique referred to as kriging , or Gaussian process regression, originally developed in the geostatistics community (Matheron, 1963; Stein, 1999b).
To quantify learning, we ﬁrst perform numerical experiments for data points distributed uniformly at random on a hypersphere of varying dimension d , focusing on a Laplace kernel for the Student, and considering a Laplace or Gaussian kernel for the Teacher.
We observe that in both cases β ( d ) is a decreasing function.
To derive β ( d ) we consider the simpliﬁed situation where the Gaussian random ﬁeld is sampled at training points lying on a regular lattice.
Building on the kriging literature (Stein, 1999b), we show that β is controlled by the high-frequency scaling of both the Teacher and Student kernels: assuming that the Fourier transforms of the kernels decay as ˜ K T ( w ) =
c T || w || − α T
+ o ( || w
|| − α T ) and ˜ K S ( w ) =
c S || w || − α S
+ o ( || w
|| − α S ) , we obtain
[Equation]
Importantly (i) Eq.
(1) leads to a prediction for β ( d ) that accurately matches our numerical study for random training data points, leading to the conjecture that Eq.
(1) holds in that case as well.
We offer the following interpretation: ultimately, kernel methods are performing a local interpolation whose quality depends on the distance δ ( n ) between adjacent data points.
δ ( n ) is asymptotically similar for random data or data sitting on a lattice.
(ii)
If the kernel K S is not too sensitive to high-frequencies, then learning is optimal as far as scaling is concerned and β =
( α T − d ) /d .
We will argue that the smoothness index s ≡ [( α T − d ) / 2] characterizes the number of derivatives of the target function that are continuous.
We thus recover the curse of dimensionality: s needs to be of order d to have non-vanishing β in large dimensions.
Point (ii) leads to an apparent paradox: β is signiﬁcant for
MNIST and CIFAR10, for which d is a priori very large, leading to a smoothness value s in the hundreds in both cases, which appears unrealistic.
The paradox is resolved by considering that real datasets actually live on lower-dimensional manifolds.
As far as kernel learning is concerned, our ﬁndings support that the correct deﬁnition of dimension should be based on how the nearest-neighbors distance δ ( n ) scales with n : δ ( n ) ∼ n − 1 / d eff .
Direct measurements of δ ( n ) support that MNIST and
CIFAR10 live on manifolds of lower dimensions d
eﬀMNIST ≈ 15 and d eﬀCIFAR10 ≈ 35 .
Considering the effective dimensions that we ﬁnd, the observed values for β would be obtained for Gaussian ﬁelds of smoothness s MNIST ≈ 3 and s CIFAR10 ≈ 1 , values that appear intuitively more reasonable.
More generally this analogy with Gaussian ﬁelds allows one to associate a smoothness index s to any dataset once β and d eﬀ are measured, which may turn out to be a useful characterization of data complexity in the future.
R ELATED WORKS
Our set-up of Teacher-Student learning with kernels is also referred to as kriging , or Gaussian process regression, and it was originally developed in the geostatistics community (Matheron, 1963).
In
Section 5 we present a theorem that allows one to know the rate at which the test error decreases as we increase the number of training points, assumed to lie on a high-dimensional regular lattice.
Similar results have been previously derived in the kriging literature (Stein, 1999b) when sampling occurs on the regular lattice with the exception of the origin, where the inference is made.
Here we propose an alternative derivation that some readers might ﬁnd simpler.
We also study a slightly different problem: instead of computing the test error when the inference is carried on at the origin, we compute the average error for a test point that lie at an arbitrary point, sampled uniformly at random and not necessarily on the lattice.
In what follows we show, via extensive numerical simulations, that such predictions are accurate even when the training points do not lie on a regular lattice, but are taken at random on a hypersphere.
An exact proof of our result in such a general setting is difﬁcult and cannot be found even in the kriging literature.
To our knowledge the results that get closer to the point are those discussed in (Stein, 1999a), where the author studies one-dimensional processes where the training data are not necessarily evenly spaced.
In this work the effective dimension of the data plays an import role, as it controls how the distance between nearest neighbors scales with the dataset size.
Of course, there exists a vast literature (Grassberger and Procaccia, 1983; Costa and Hero, 2004; Hein and Audibert, 2005; Levina and
Bickel, 2005; Rozza et al., 2012; Facco et al., 2017; Allegra et al., 2019) devoted to the study of effective dimensions, where other deﬁnitions are analyzed.
The effective dimensions that we ﬁnd are compatible with those obtained with more reﬁned methods.
L EARNING CURVE FOR KERNEL METHODS APPLIED TO REAL DATA
In what follows we apply kernel methods to the MNIST and CIFAR10 datasets, each consisting of a set of images ( x µ ) nµ =1 .
We simplify the problem by considering only two classes whose label
Z ( x µ ) = ± 1 correspond to odd and even numbers for MNIST, and to two groups of 5 classes in
CIFAR10.
The goal is to infer the value of the label ˆ Z S ( x ) of an image x that does not belong to the dataset.
The S subscript reminds us that inference is performed using a positive deﬁnite kernel K S .
We perform inference in both a regression and a classiﬁcation setting.
The following algorithms and associated results can be found in (Scholkopf and Smola, 2001).
Regression.
Learning corresponds to minimizing a mean-square error:
[Equation]
For algorithms seeking solutions of the form ˆ Z S ( x ) = (cid:80) µ a µ K S ( x µ , x ) ≡ a · k S ( x ) by minimizing the man-square loss over the vector a , one obtains:
[Equation]
where the vector Z contains all the labels in the training set, Z ≡ ( Z ( x µ )) nµ =1 , and K S,µν ≡
K S ( x µ , x ν ) is the Gram matrix.
The Gram matrix is always invertible if the kernel K S is positive deﬁnite.
The generalization error is then evaluated as the expected mean-square error on unseen data, estimated by averaging over a test set composed of n test unseen data points:
[Equation]
Classiﬁcation.
We perform kernel classiﬁcation via the algorithm soft-margin SVM .
The details can be found in Appendix A. After learning from the training data with a student kernel K S , performance is evaluated via the generalization error.
It is estimated as the fraction of correctly predicted labels for data points belonging to a test set with n test elements.
In Fig.
1 we present the learning curves for (binary)
MNIST and CIFAR10, for regression and classiﬁcation.
Learning is performed both with a Gaussian kernel K ( x ) ∝ exp( −|| x || 2 / (2 σ 2 )) and a Laplace one K ( x ) ∝ exp( −|| x || /σ ) .
Remarkably, the power laws in the two tasks are essentially identical (although the estimated exponent appears to be slightly larger, in absolute value, for classiﬁcation).
Moreover, the two kernels display a very similar behavior, compatible with the same exponent: about − 0 .
4 for MNIST and − 0 .
1 for CIFAR10.
The presented data are for σ = 1000 ; in
Appendix B we show that the same behaviour is observed for different values.
[Figure]
G ENERALIZATION SCALING IN KERNEL T EACHER -S TUDENT PROBLEMS
We study β in a simpliﬁed setting where the data is assumed to follow a Gaussian distribution with known covariance.
It falls into the class of teacher-Student problems, which are characterized by a machine (the Teacher) that generates the data and another machine (the Student) that tries to learn from them.
The Teacher-Student paradigm has been broadly used to study supervised learning (Saad and Solla, 1995; Monasson and Zecchina, 1995; Opper and Saad, 2001; Engel and Van den Broeck, 2001; Zdeborová and Krzakala, 2016; Barbier et al., 2019; Gabrié et al., 2018; Aubin et al., 2018;
Franz et al., 2018).
He we restrict our attention to kernel methods: we assume that a target function is distributed according to a Gaussian random ﬁeld Z ∼ N (0 , K T ) — the Teacher — characterized by a translation-invariant isotropic covariance function K T ( x, x (cid:48) )
=
K T ( || x − x (cid:48) || ) , and that the training dataset consists the ﬁnite set of n observations Z =
( Z ( x µ )) nµ =1 .
This is equivalent to saying that the vector of training points follows a centered Gaussian distribution with a covariance matrix that depends on K T and on the location of the points ( x µ ) nµ =1 :
[Equation]
Once the Teacher has generated the dataset, the rest follows as in the kernel regression described in the previous section.
We use another translation-invariant isotropic kernel K S ( x, x (cid:48) )
— the Student — to infer the value of the ﬁeld at another point, ˆ Z S ( x ) , with a regression task, i.e. minimizing the mean-square error in Eq.
The solution is therefore given again by Eq.
Fig.
2 (a-b) shows the mean-square error obtained numerically.
In the examples the Student is always taken to be a Laplace kernel, and the Teacher is either a Laplace kernel or a Gaussian kernel.
The points ( x µ ) nµ =1 are taken uniformly at random on the unit d -dimensional hypersphere for several dimensions d and for several dataset sizes n .
We take σ S =
σ T =
d as we observed that with this choice smaller datasets were enough to approach a limiting curve — in Appendix C we show the plots for the case σ S =
σ T = 10 , which appears to converge to the same limit curve with increasing n , but at a smaller pace.
The ﬁgure shows that when n is large enough, the mean-square error behaves as a power law (dashed lines) with an exponent that depends on the spatial dimension of the data, as well as on the kernels.
The ﬁtted exponents are plotted in Fig.
2 (c-d) as a function of the spatial dimension d for different dataset sizes n .
In the next section we will discuss the theoretical prediction, that in the ﬁgure is plotted a thick black line.
The ﬁgure shows that as the dataset gets bigger, the asymptotic exponent tends to our prediction.
In Appendix D we present the learning curves of Gaussian Students with both a Laplace and a Gaussian kernel.
When both kernels are Gaussian the test error decays exponentially fast, a result that matches our theoretical prediction.
In Appendix E we also provide further numerical results for the case where the Teacher kernel is a Matérn kernel (as deﬁned therein).
A NALYTIC ASYMPTOTICS FOR THE KERNEL T EACHER -S TUDENT PROBLEM ON A LATTICE
In this section we compute analytically the exponent that describe the asymptotic decay of the generalization error when the number n of training data increases.
In order to derive the result we assume that both the Teacher Gaussian random ﬁeld lives on a bounded hypercube, x ∈ V ≡ [0 , L ] d , where L is a constant and d is the spatial dimension.
The ﬁelds and the kernels can then be thought of
[Figure]
as L -periodic along each dimension. Furthermore, to make the problem tractable we assume that the points ( x µ ) nµ =1 live on a regular lattice , covering all the hypercube V .
Therefore, the linear spacing between neighboring points is δ =
Ln − 1 / d .
This is of course a different setting than the one used in the numerical simulations presented in the previous section, yet our results below support that these differences do not matter.
Generalization error is then evaluated via the typical mean-square error
[Equation]
where the expectation is taken over both the Teacher process and the point x at which we estimate the ﬁeld, assumed to be uniformly distributed in the hypercube V .
In Appendix F we prove the following:
Theorem 1.
Let ˜ K T ( w )
=
c T || w || − α T
+ o ( || w
|| − α T ) and ˜ K S ( w ) =
c S || w || − α S
+ o ( || w
|| − α S )
as || w ||
→ ∞ , where ˜ K T ( w ) and ˜ K S ( w ) are the Fourier transforms of the kernels K T ( x ) ,
K S ( x ) respectively, assumed to be positive deﬁnite.
We assume ˜ K T ( w ) and ˜ K S ( w ) has a ﬁnite limit as || w ||
→ 0 and that K (0) < ∞ .
Then,
[Equation]
Moreover, in the case of a Gaussian kernel the result holds valid if we take the corresponding exponent to be α = ∞ .
Apart from the speciﬁc value of the exponent in Eq.
(7), Theorem 1 implies that if the Student kernel decays fast enough in the frequency domain, then β depends only on the data through the behaviour of the Teacher kernel at high frequencies.
One then recovers β
=
( α T − d ) /d , also found for the
Bayes-optimal setting where the Student is identical to the Teacher.
Consider the predictions of Theorem 1 in the cases presented in Fig.
2 (a-b) of Gaussian and Laplace kernels.
If both kernels are Laplace kernels then α T = α S =
d + 1 and E MSE ∼ n − 1 / d , which scales very slowly with the dataset size in large dimensions.
If the Teacher is a Gaussian kernel ( α T = ∞ ) and the Student is a Laplace kernel then β = 2(1 + 1 /d ) , leading to β → 2 as d → ∞ .
In Fig.
2 (c-d) we compare these predictions with the exponents extracted from Fig.
2 (a-b) .
We plot log E MSE / log n ≡ − β , against the dimension d of the data, varying the dataset size n .
The exponents extracted numerically tend to our analytical predictions when n is large enough.
Notice that, although the theory and the experiments do not assume the same distribution for the sampling points ( x µ ) nµ =1 , this does not seem to yield any difference in the asymptotic behavior of the generalization error, leading to the conjecture that our predictions are exact even when the training set is random, and does not correspond to a lattice.
The conjecture can be proven in one dimension following results of the kriging literature (Stein, 1999a), but generalization to higher d is a much harder problem.
Intuitively, for kernel learning performs an expansion, whose quality is governed by the target function smoothness and the typical distance δ min between a point and its nearest neighbors in the training set.
Both for random points or on a lattice, one has δ min ∼ n − 1 / d when n is large enough, thus both situations lead to the same β .
Theorem 1 underlines that kernel methods are subjected to the curse of dimensionality.
Indeed for appropriate students, one obtains β =
( α T − d )
/d .
Let us deﬁne the smoothness index s ≡ [( α T − d ) / 2] = βd/ 2 , which must be O ( d ) to avoid β → 0 for large d .
The two Lemmas below, derived in Appendix, indicate that the target function is s time differentiable (in a mean-square sense).
Thus learning with kernels in very large dimension can only occur if the target function is O ( d ) times differentiable, a condition that appears very restrictive in large d .
Lemma 1.
Let K ( x, x (cid:48) ) be a translation-invariant isotropic kernel such that ˜ K ( w )
=
c || w ||
− α + o ( || w || − α ) as || w ||
→ ∞ and || w || d ˜ K ( w ) → 0 as || w || → 0 .
If α > d + n for some n ∈ Z + , then
K ( x ) ∈ C n , that is, it is at least n -times differentiable.
(Proof in Appendix G).
Lemma 2.
Let Z ∼ N (0 , K ) be a d -dimensional Gaussian random ﬁeld, with K ∈ C 2 n being a 2 n -times differentiable kernel.
Then Z is n -times mean-square differentiable in the sense that
• derivatives of Z ( x ) are a Gaussian random ﬁelds; • E ∂ n 1 x 1 · · · ∂ n d x d Z ( x ) = 0 ; • E ∂ n 1 x 1 · · · ∂ n d x d Z ( x ) ·
∂ n (cid:48) 1 x 1 · · · ∂ n (cid:48)
d x d Z ( x (cid:48) )
=
∂ n 1 + n (cid:48) 1 x
[Equation]
· · ∂ n
[Figure]
+ n (cid:48)
[Figure]
x
[Figure]
K ( x − x (cid:48) )
< ∞ if the derivatives of K exist.
In particular, E ∂ mx i Z ( x ) · ∂ mx i Z ( x (cid:48) )
=
∂ 2 mx i K ( x − x (cid:48) )
< ∞ ∀ m ≤ n .
(Proof in Appendix G).
E FFECTIVE DIMENSION OF DATA SETS
[Figure]
If we approximate the high-dimensional MNIST and CIFAR10 datasets with Gaussian random ﬁelds, to obtain the curves shown in Fig.
1 and to ﬁnd the values the we report for β these ﬁelds would have to be hundreds of times differentiable, which seems unrealistic.
A possible resolution of this paradox lies in the fact that the data live in a much smaller manifold than the number of pixels of these pictures would suggest.
As argued above, a key determinant of kernel performance is the typical distance δ min between a point in the training set and its nearest neighbor.
We deﬁne the effective dimension d eﬀ accordingly from the asymptotic relationship between δ min and n :
[Equation]
For random points on a d -dimensional hypersphere δ min displays ﬂuctuations and the scaling is valid only on average and only asymptotically, that is for n larger than some characteristic scale n (cid:63) ( d ) that depends on the spatial dimension.
In Fig.
3 (a) we show how the typical δ min scales with the dataset size n for random points on hyperspheres of dimension d = 15 and d = 35 .
Notice that while for d = 15 the asymptotic regime is reached when n (cid:39) 10 4 , for d = 35 a larger dataset is needed, with n > 10 5 points (that is about the maximum size of the dataset that we can use to apply kernel methods in our simulations, due to memory constraints).
One can naturally wonder whether real data are also subjected to a scaling relation like in Eq.
(8), from which an effective dimension can be deﬁned.
Consider for instance the MNIST dataset, and sample from it a subset of n pictures.
For each point we could compute the distance from its nearest neighbor and average such quantities.
As the number of data points increases, we expect that this measure characterizes the geometry of the local manifold where the data live in, since nearest neighbors are going to be closer and closer.
In Fig.
3 (b) we present how δ min scales with n for the MNIST and CIFAR10 datasets.
Both display a power-law decay, but the exponent is not compatible with 1 / d with d the spatial dimension, namely d = 784 for
MNIST and d = 3072 for CIFAR10.
MNIST actually seems to scale pretty much like random data on a hypersphere with d = 15 , and CIFAR10 scales approximately as random data on a hypersphere with d = 35 .
For this reason, the effective dimensions of these datasets are consistent with:
[Equation]
Obviously, the intrinsic dimension of the data could vary in data space, as has been reported for
MNIST (Costa and Hero, 2004; Hein and Audibert, 2005; Rozza et al., 2012; Facco et al., 2017).
In this qualitative discussion we neglect such subtle effects.
Interestingly, our effective dimensions leads to reasonable values for the effective smoothness:
[Equation]
In particular we ﬁnd s ≈ 3 for MNIST and s ≈ 1 for CIFAR10.
C ONCLUSION
In this work we have shown for CIFAR10 and MNIST respectively that kernel regression and classiﬁcation display a power-law decay in the learning curves, quite remarkably with essentially the same exponent β , found to be larger for MNIST.
These exponents are much larger than β =
/d expected for Lipschitz target functions and smaller than β = 1 / 2 expected for RKHS target functions.
This observation led us to introduce a framework in which data are modeled as Gaussian random ﬁelds of varying smoothness, in which intermediary values of β are obtained.
It is important to note the high degree of smoothness underlying the RKHS hypothesis.
Consider realizations
Z ( x ) of a Teacher Gaussian process with covariance K T and assume that they lie in the
RKHS of the Student kernel K S , namely
[Equation]
If the Teacher and Student kernels decay in the frequency domain with exponents α T and α S respectively, convergence requires α T > α S + d , and K S (0) ∝ (cid:82) d w ˜ K S ( w ) < ∞ (true for many commonly used kernels) implies α S > d .
Then using Lemma 1 and Lemma 2 we can conclude that the realizations Z ( x ) must be at least (cid:98) d / 2 (cid:99)
-times mean-square differentiable to be RKHS.
From this perspective, the RKHS assumption appears to be very strong, and thus may not provide an accurate description of various empirical learning curves.
Our assumption that data are generated by Gaussian random processes is milder, and may thus have broader applications.
Yet, we view this approximation as a ﬁrst step on which to build on, to later include other effects such as noise in the data and deviations from Gaussianity.
A CKNOWLEDGMENTS
Anonymized for double-blind review.
A S OFT - MARGIN S UPPORT V ECTOR M ACHINES
The kernel classiﬁcation task is performed via the algorithm known as soft-margin Support Vector
Machine .
We want to ﬁnd a function ˆ Z S ( x ) such that its sign correctly predicts the label of the data.
In this context we model such a function as a linear prediction after projecting the data on a feature space via x → φ ( x ) :
[Equation]
(10) where w, b are parameters to be learned from the training data.
The kernel is related to the feature space via K S ( x, x (cid:48) )
=
φ S ( x ) · φ S ( x (cid:48) ) .
We require that Z ( x µ ) ˆ Z S ( x µ ) > 1 − ξ µ for all training points.
Ideally we want to have some large margins 1 − ξ µ = 1 , but we allow some of them to be smaller by introducing the slack variables ξ µ and penalizing large values.
To achieve this the following constrained minimization is performed:
[Equation]
subjected to ∀
[Equation]
Z
[Equation]
This problem can be expressed in a dual formulation as
[Equation]
subjected to Z · a =
[Equation]
0 ≤ a µ ≤ C
[Equation]
12) where Q S,µ,ν =
Z ( x µ ) Z ( x ν ) K S ( x µ , x ν ) and Z is the vector of the labels of the training points.
Here C ( = 10 4 in our simulations) controls the trade-off between minimizing the training error and maximizing the margins 1 − ξ µ .
For the details we refer to (Scholkopf and Smola, 2001).
If a (cid:63) is the solution to the minimization problem, than
[Equation]
The predicted label for unseen data points is then
[Equation]
The generalization error is now deﬁned as the probability that an unseen image has a predicted label different from the true one, and such a probability is again estimated as an average over a test set with n test elements:
[Equation]
B D IFFERENT KERNEL VARIANCES
In Fig.
4 we show the learning curves for kernel regression on the MNIST (parity) dataset — the same setting as in Fig.
1 (a) .
Several Laplace kernels of varying variance σ are used.
The variance ranges several orders of magnitude and the learning curves all decay with the same exponent, although for σ = 10 the algorithm achieves suboptimal performance and the test errors are increased by some factor.
[Figure]
C D IFFERENT CHOICE OF KERNEL VARIANCES
In Fig.
5 we show the learning curves for the Teacher-Student kernel regression problem, with a
Student kernel that is always Laplace and a Teacher that can be either Gaussian or Laplace.
We show how the test error decays with the size of the training dataset and how the asymptotic exponent depends on the spatial dimension.
Every experiment is run with two different choices of the kernel variances: in one case σ T = σ S =
d and in the other σ T = σ S = 10 .
We observed that scaling the variances with the spatial dimension leads faster to the results that we predicted in this paper
[Figure]
but overall the choice has little effect on the exponents (both tend towards the prediction as the dataset size is increased)
[Figure]
While the former decays algebraically with the predicted exponent, the latter decays exponentially, in agreement with the prediction β = ∞ found within our framework.
In all these plots we have taken the variances of both the Teacher and Student kernels to be equal to the dimension d = 6 .
D G AUSSIAN S TUDENTS
In this appendix we present the learning curves of Gaussian Students: the Fourier transform of these kernels decays faster than any power law and one can effectively consider α S = ∞ .
If the Teacher is Laplace ( α T =
d + 1 ) then the predicted exponent is ﬁnite and takes the values β = 1 d min( α T − d, 2 α S )
= 1 d min(1 , ∞ )
= 1 d .
Such a case is displayed in Fig.
6 (left) in dimension d = 6 .
However, if we consider the Teacher to be Gaussian as well, then the predicted exponent would be β = 1 d min( ∞ , ∞ )
= ∞ .
This case corresponds to Fig.
6 (center) : the test errors decays faster than a power law.
In Fig.
6 (right) we compare the case where both kernels are Gaussian to the case where both kernels are Laplace: while the latter decays as a power law, the former decays much faster.
E M ATÉRN T EACHERS
To further test the applicability of our theory, we show here some numerical simulations for a Teacher kernel that is a Matérn covariance function and a Laplace kernel as student. We ran the simulations in 1d: the data points are sampled uniformly on a 1-dimensional circle embedded in R 2 . Matérn kernels are parametrized by a parameter ν > 0 :
[Equation]
where z = √ 2 ν || x
|| σ ( σ being the kernel variance), Γ is the gamma function and K ν is the Bessel function of the second kind with parameter ν .
Interestingly we recover the Laplace kernel for ν = 1 /and the Gaussian kernel for ν = ∞ .
As one can ﬁnd in e.g. (Williams and Rasmussen, 2006), the exponent α T that governs the decay at high frequency of this kernels is α T =
d + 2 ν . Varying ν we can change the smoothness of the target function. For d = 1 our prediction for the learning curve exponent β is
[Equation] In Fig. 7 we verify that our prediction matches the numerical results.
[Figure]
F P ROOF OF THEOREM
We prove here Theorem 1: Theorem 1 Let ˜ K T ( w ) = c T || w || − α T
+ o ( || w
|| − α T ) and ˜ K S ( w ) =
c S || w || − α S
+ o ( || w
|| − α S )
as || w ||
→ ∞ , where ˜ K T ( w ) and ˜ K S ( w ) are the Fourier transforms of the kernels K T ( x ) ,
K S ( x ) respectively, assumed to be positive deﬁnite.
We assume ˜ K T ( w ) and ˜ K S ( w ) has a ﬁnite limit as || w ||
→ 0 and that K (0) < ∞ .
Then,
[Equation]
Moreover, in the case of a Gaussian kernel the result holds valid if we take the corresponding exponent to be α = ∞ .
Proof.
Our strategy is to compute how the mean-square test error scales with distance δ between two nearest neighbors on the d -dimensional regular lattice.
At the end, we will use the fact that δ ∝ n − 1 / d , where n is the number of sampled points on the lattice.
We denote by ˜ F ( w ) the Fourier transform of a function F : V → R :
[Equation]
[Equation]
If Z ∼ N (0 , K ) is a Gaussian ﬁeld with translation-invariant covariance K then by deﬁnition
[Equation]
Properties of the Fourier transform of a Gaussian ﬁeld:
[Equation]
Eq.
(23) comes from the fact that K ( x ) is an even, real-valued function.
The real and imaginary parts of ˜ Z ( w ) are Gaussian random variables.
They are all independent except that ˜ Z ( − w )
=
˜ Z ( w ) .
Eq.
(25) follows from the fact that Z ( x ) and K ( x ) are L -periodic functions, and therefore e iw · x ˜ K ( w ) is the Fourier transform of K ( · + x ) if w ∈ 2 πL Z d .
The solution Eq.
(3) for kernel regression has two interpretations.
In Section 4 we introduced it as the quantity that minimizes a quadratic error, but it can also be seen as the maximum-a-posteriori (MAP) estimation of another formulation of the problem (Williams and Rasmussen, 2006).
The ﬁeld Z ( x ) is assumed to be drawn from a Gaussian distribution with covariance function K S ( x ) : K S therefore plays a role in the prior distribution of the data Z
=
( Z ( x µ ) n µ =1 ) .
Inference about the value of the ﬁeld ˆ Z S ( x ) at another location is then performed by maximizing its posterior distribution, [Equation]
[Equation]
Such a posterior distribution is Gaussian, and its mean — and therefore also the value that maximizes the probability — is exactly Eq.
[Equation]
27) where where Z = (cid:0) Z ( x µ ) (cid:1) n µ =1 are the training data, k S ( x )
= (cid:0) K S ( x µ [Equation] x
[Equation] n µ =1 and K S = (cid:0) K S ( x µ , x ν ) (cid:1) n µ,ν =1 is the Gram matrix, that is invertible since the kernel K S is assumed to be positive deﬁnite.
By Fourier transforming this relation we ﬁnd
[Equation]
where we have deﬁned F
[Equation]
w
[Equation]
≡
[Equation] n ∈ Z d F [Equation] w + 2 πnδ (cid:17) for a generic function F .
Another way to reach Eq.
(28) is to consider that we are observing the quantities
[Equation]
Given that we know the prior distribution of the Fourier components on the right-hand side in Eq.
(29), we can infer their posterior distribution once their sums are constrained by the value of ˜ Z (cid:63) ( w ) , and it is straightforward to see that we recover Eq.
The mean-square error can then we written using the Parseval-Plancherel identity,
[Equation]
By taking the expectation value with respect to the Teacher and using Eq.
(23)-Eq.
(25) we can write the mean-square error as
[Equation]
where B = (cid:2) − πδ , πδ (cid:3) d is the Brillouin zone.
At high frequencies, ˜ K T ( w ) =
c T || w || − α T
+ o ( || w
|| − α T ) and ˜ K S ( w ) =
c S || w || − α S
+ o ( || w
|| − α S ) .
Therefore:
[Equation]
This equation deﬁnes the function ψ T , and a similar equation holds for the Student as well.
The hypothesis K T (0) ∝ (cid:82) d w ˜ K T ( w ) < ∞ implies α T > d and therefore (cid:80) n ∈ Z d || n || − α T < ∞ (and likewise for the Student).
Then, ψ α T (0) , ψ α S (0) are ﬁnite; furthermore, the w ’s in the sum Eq.
(31) are at most of order
O (cid:0) δ − 1 (cid:1) , therefore the terms ψ α ( wδ ) are O ( δ 0 ) and do not inﬂuence how
Eq.
(31) scales with δ .
Applying Eq.
(32), expanding for δ (cid:28) 1 and keeping only the leading orders, we ﬁnd
[Equation]
We have neglected terms proportional to, for instance, δ α T + α S , since they are subleading with respect to δ α T , but we must keep both δ α T and δ α S since we do not know a priori which one is dominant.
The additional term δ − d in the subleading terms comes from the fact that | L ∩ B| = O (cid:0) δ − d (cid:1) .
The ﬁrst term in Eq.
(33) is the simplest to deal with: since ||
wδ || is smaller than some constant for all w ∈ L ∩ B and the function ψ α T ( wδ ) has a ﬁnite limit, we have
[Equation]
We then split the second term in Eq.
(33) in two contributions:
Small ||
w
||
We consider “small” all the terms w ∈ L ∩ B such that || w || <
Γ , where Γ (cid:29) 1 is O ( δ 0 ) but large.
As δ → 0 , ψ 2 α S ( wδ ) → ψ 2 α S (0) which is ﬁnite because K (0) < ∞ .
Therefore
[Equation]
The summand is real and strictly positive because the positive deﬁniteness of the kernels implies that their Fourier transforms are strictly positive.
Moreover, as δ → 0 , L ∩B ∩{||
w || < Γ } → L ∩{|| w || < Γ } , which contains a ﬁnite number of elements, independent of δ .
Therefore
[Equation]
Large || w ||
“Large” w are those with || w || >
Γ : we recall that Γ (cid:29) 1 is O ( δ 0 ) but large.
This allows us to approximate ˜ K T , ˜ K S in the sum with their asymptotic behavior:
[Equation]
Finally, putting Eq.
(34), Eq. (36) and Eq.
(37) together,
[Equation]
The proof is concluded by considering that δ = O (cid:0) n − 1 / d (cid:1) .
In the case of a Gaussian kernel K ( x ) ∝ exp( −|| x || 2 / (2 σ 2 )) — and therefore ˜ K ( w ) ∝ exp( − σ 2 || w || 2 / 2) — one has to redo the calculations starting from Eq.
(31), but the ﬁnal result can be easily recovered by taking the limit α → + ∞ (Gaussian kernels decay faster than any power law).
G P ROOFS OF LEMMAS
Lemma 1 Let K ( x, x (cid:48) ) be a translation-invariant isotropic kernel such that ˜ K ( w ) =
c || w ||
− α + o ( || w || − α ) as || w ||
→ ∞ and || w || d ˜ K ( w ) → 0 as || w || → 0 .
If α > d + n for some n ∈ Z + , then
K ( x ) ∈ C n , that is, it is at least n -times differentiable.
Proof.
The kernel is rotational invariant in real space ( K ( x ) =
K ( || x || ) ) and therefore also in the frequency domain.
Then, calling ˆ (cid:15)
= (1 , 0 , . . . )
the unitary vector along the ﬁrst dimension x 1 ,
[Equation]
It follows that
[Equation]
We want to claim that this quantity is ﬁnite if m ≤ n .
Convergence at inﬁnity requires m < α − d , that is always smaller than or equal to n because of the hypothesis of the lemma.
Convergence in zero requires that w d + m | ˜ K ( w ) | → 0 , and we want this to hold for all 0 ≤ m < α − d , the most constraining one being the condition with m = 0 .
Lemma 2 Let Z ∼ N (0 , K ) be a d -dimensional Gaussian random ﬁeld, with K ∈ C 2 n being a 2 n -times differentiable kernel.
Then Z is n -times differentiable in the sense that
• derivatives of Z ( x ) are a Gaussian random ﬁelds; • E ∂ n 1 x 1 · · · ∂ n d x d Z ( x ) = 0 ; • E ∂ n 1 x 1 · · · ∂ n d x d Z ( x ) ·
∂ n (cid:48) 1 x 1 · · · ∂ n (cid:48)
d x d Z ( x (cid:48) )
=
∂ n 1 + n (cid:48) 1 x
[Equation]
· · ∂ n d + n (cid:48)
d x d
[Equation]
[Equation]
if the derivatives of K exist
[Equation]
In particular
[Equation]
Proof.
Derivatives of Z ( x ) are deﬁned as limits of sums and differences of the ﬁeld Z evaluated at different points, therefore they are Gaussian random ﬁelds too
[Equation]
and furthermore it is straightforward to see that their expected value is always 0 if the ﬁeld itself is zero centered
[Equation]
The correlation can be computed via induction.
Assume that E ∂ n 1 x
[Equation] [Equation]
(cid:48)
[Equation]
K ( x − x
[Equation]
) holds true.
Then, if we increment n 1 :
[Equation]
Of course by symmetry the same can be said about the increase of any other exponent.
To conclude the induction proof we simply recall that by deﬁnition E Z ( x ) Z ( x (cid:48) )
=
K ( x − x (cid:48) ) .