H ARNESSING THE P OWER OF I NFINITELY W IDE D EEP N ETS ON S MALL - DATA T ASKS
A BSTRACT
Recent research shows that the following two models are equivalent: (a) inﬁnitely wide neural networks (NNs) trained under (cid:96) 2 loss by gradient descent with inﬁnitesimally small learning rate (b) kernel regression with respect to so-called
Neural Tangent Kernels (NTKs) (Jacot et al., 2018).
An efﬁcient algorithm to compute the NTK, as well as its convolutional counterparts, appears in Arora et al.
(2019a), which allowed studying performance of inﬁnitely wide nets on datasets like CIFAR-10.
However, super-quadratic running time of kernel methods makes them best suited for small-data tasks.
We report results suggesting neural tangent kernels perform strongly on low-data tasks.
On a standard testbed of classiﬁcation/regression tasks from the UCI database, NTK SVM beats the previous gold standard, Random Forests (RF)
and also the corresponding ﬁnite nets.
On CIFAR-10 with 10 – 640 training samples, Convolutional NTK consistently beats ResNet-34 by 1% - 3% .
On VOC07 testbed for few-shot image classiﬁcation tasks on ImageNet with transfer learning (Goyal et al., 2019), replacing the linear SVM currently used with a Convolutional NTK SVM consistently improves performance.
Comparing the performance of NTK with the ﬁnite-
width net it was derived from, NTK behavior starts at lower net widths than suggested by theoretical analysis(Arora et al., 2019a).
NTK’s efﬁcacy may trace to lower variance of output.
I NTRODUCTION
Modern neural networks (NNs) have way more parameters than training data points, which allow them to achieve near-zero training error while simultaneously — for some reason yet to be understood — have low generalization error (Zhang et al., 2016).
This motivated formal study of highly overparametrized networks, including networks whose width (i.e., number of nodes in layers, or number of channels in convolutional layers) goes to inﬁnity.
A recent line of theoretical results shows that with (cid:96) 2 loss and inﬁnitesimal learning rate, in the limit of inﬁnite width the trajectory of training converges to kernel regression with a particular kernel, neural tangent kernel (NTK) (Jacot et al., 2018).
For convolutional networks, the kernel is CNTK.
See Section 2 for more discussions.
Arora et al. (2019a) gave an algorithm to exactly compute the kernel corresponding to the inﬁnite limit of various realistic NN architectures with convolutions and pooling layers, allowing them to compute performance on CIFAR-10, which revealed that the inﬁnite networks have 6 to 8 % higher error than their ﬁnite counterparts.
This is still fairly good performance for a ﬁxed kernel.
Ironically, while the above-mentioned analysis, at ﬁrst sight, appears to reduce the study of a complicated model — deep networks — to an older, simpler model — kernel regression — in practice the simpler model is computationally less efﬁcient because running time of kernel regression can be quadratic in the number of data points!
Thus computing using CNTK kernel on large datasets like
ImageNet currently appears infeasible.
Even on CIFAR-10, it seems infeasible to incorporate data augmentation.
However, kernel classiﬁers are very efﬁcient on small datasets.
Here NTKs could conceivably be practical while at the same time bringing some of the power of deep networks to these settings.
We recall that recently Olson et al.
(2018) showed that multilayer neural networks can be reasonably effective on small datasets, speciﬁcally on a UCI testbed of tasks with as few as dozens of training examples.
Of course, this required some hyperparameter tuning, although they noted that such tuning is also needed for the champion method, Random Forests (RF), which multilayer neural networks could not beat.
It is thus natural to check if NTK — corresponding to inﬁnitely wide fully-connected networks — performs well in such small-data tasks 2 .
Convex objectives arising from kernels have stable solvers with minimal hyperparameter tuning.
Furthermore, random initialization in deep network training seems to lead to higher variance in the output, which can hurt performance in small-data settings.
Can NTK’s do better?
Below we will see that in the setup of Olson et al.
(2018), NTK predictors indeed outperforms corresponding ﬁnite deep networks, and also slightly beats the earlier gold standard, Random Forests.
This suggests NTK predictors should belong in any list of off-theshelf machine learning methods.
Following are low-data settings where we used NTKs and CNTKs:
• In the testbed of 90 classiﬁcation tasks from UCI database, NTK predictor achieves superior, and arguably the strongest classiﬁcation performance.
This is veriﬁed via several standard statistical tests, including Friedman Rank, Average Accuracy, Percentage of the Maximum Accuracy (PMA) and probability of achieving 90%/95% maximum accuracy (P90 and P95), performed to compare performances of different classiﬁers on 90 datasets from UCI database.
(The authors plan to release the code, to allow off-the-shelf use of this method.
It does not require GPUs.)
We ﬁnd the performance of NN is close to that of NTK.
On every dataset from UCI database, the difference between the classiﬁcation accuracy of NN and that of NTK is within 5% .
On the other hand, on some datasets, the difference between classiﬁcation accuracy of NN (or NTK) and that of other classiﬁers like RF can be as high as 20% .
This indicates in low-data settings, NTK is indeed a good description of NN.
Furthermore, we ﬁnd NTK is more stable (smaller variance), which seems to help it achieve better accuracy on small datasets (cf.
Figure 2b).
• CNTK is useful in computer vision tasks with small-data.
On CIFAR-10, we compare CNTK with ResNet using 10 - 640 training samples and ﬁnd CNTK can beat ResNet by 1% − 3% .
We further study few-shot image classiﬁcation task on VOC07 dataset.
The standard method is to ﬁrst use a pre-trained network, e.g., ResNet-50 trained on ImageNet, to extract features and then directly apply a linear classiﬁer on the extracted features (Goyal et al., 2019).
Here we replace the linear classiﬁer with CNTK and obtain better classiﬁcation accuracy in various setups.
Paper organization.
Section 2 discusses related work.
Section 3 reviews the derivation of NTK.
Section 4 presents experiments using NN and NTK on UCI datasets.
Section 5 presents experiments using CNN and CNTK on small CIFAR-10 datasets.
Section 6 presents experiments using CNTK for the few-shot learning setting.
Additional technical details are presented in appendix.
R ELATED W ORK
Our paper is inspired by Fern´andez-Delgado et al.
(2014) which conducted extensive experiments on UCI dataset.
Their conclusion is random forest performs the best, which is followed by the SVM with Gaussian kernel.
Therefore, RF may be considered as a reference (“gold-standard”) to compare with new classiﬁers.
Olson et al.
(2018) followed this testing strategy to evaluate the performance of modern neural networks concluding that modern neural networks, even though being highly overparameterized, still give reasonable performances on these small datasets, though not as strong as
RFs.
Our paper follows the same testing strategy to evaluate the performance of NTK.
The focus of this paper, neural tangent kernel is induced from a neural network architecture.
The connection between inﬁnitely wide neural networks and kernel methods is not new (Neal, 1996;
Williams, 1997; Roux & Bengio, 2007; Hazan & Jaakkola, 2015; Lee et al., 2018; Matthews et al., 2018; Novak et al., 2019; Garriga-Alonso et al., 2019; Cho & Saul, 2009; Daniely et al., 2016;
Daniely, 2017).
However, these kernels correspond to neural network where only the last layer is trained.
Neural tangent kernel, ﬁrst proposed by Jacot et al.
(2018), is fundamentally different as
NTKs correspond to inﬁnitely wide NNs with all layer being trained.
Theoretically, a line of work study the optimization and generalization behavior of ultra-wide NNs (Allen-Zhu et al., 2018b;a;
Arora et al., 2019b; Du et al., 2018b;a; Li & Liang, 2018; Zou et al., 2018; Yang, 2019).
Recently,
Arora et al. (2019a) gave non-asymptotic perturbation bound between the NN predictor trained by gradient descent and the NTK predictor.
Empirically, Lee et al.
(2019) veriﬁed on small scale data,
NTK is a good approximation to NN.
However, Arora et al. (2019a) showed on large scale dataset,
NN can outperform NTK which may due to the effect of ﬁnite-width and/or optimization procedure.
Generalization to architectures other than fully-connected NN and CNN are recently proposed (Yang, 2019; Du et al., 2019; Bietti & Mairal, 2019).
Du et al.
(2019) showed graph neural tangent kernel (GNTK) can achieve better performance than its counter part, graph neural network (GNN), on datasets with up to 5000 samples.
N EURAL N ETWORK AND N EURAL T ANGENT K ERNEL
Since NTK is induced by a NN architecture, we ﬁrst deﬁne a NN formally.
Let x ∈ R
d be the input, and denote g (0) ( x ) = x and d 0 = d for notational convenience.
We deﬁne an L -hidden-layer fully-connected neural network recursively:
[Equation]
where W ( h ) ∈ R d h × d h − 1 is the weight matrix in the h -th layer ( h ∈ [ L ] ), σ : R → R is a coordinatewise activation function, and c σ is a scaling factor.
In this paper, for NN we will consider σ being
ReLU or ELU (Clevert et al., 2015) and for NTK we will only consider kernel functions induced by
NNs with ReLU activation.
The last layer of the neural network is
[Equation]
where W ( L +1) ∈ R 1 × d L is the weights in the ﬁnal layer, and we let w = (cid:0) W (1) , . . .
, W ( L +1) (cid:1) be all parameters in the network.
All the weights are initialized to be i.i.d.
N (0 , 1) random variables.
From now on, by NTK initialization we mean a neural network with parameterization deﬁned in
Equation 2 with all weighted being initialized to be i.i.d.
N (0 , 1) .
When the hidden widths d 1 , d 2 , . . .
, d L → ∞ , certain limiting behavior emerges along the gradient trajectory.
Let x, x (cid:
) ∈
R d be two data points, the covariance kernel of the h -th layer’s outputs, Σ ( h ) ( x, x (cid:48) )
=
f ( h ) ( x ) · f ( h ) ( x (cid:48) ) , can be recursively deﬁned in an analytical form:
[Equation]
[Table]
for h ∈ [ L ] .
Crucially, this analytical form holds not only at the initialization, but also holds during the training (when gradient descent with small learning rate is used as the optimization routine).
Formally, NTK is deﬁned as the limiting gradient kernel
[Equation]
Again, one can obtain a recursive formula
[Equation]
where we let ˙Σ ( L +1) ( x, x (cid:48) )
= 1 for convenience.
It is easy to check if we ﬁx the ﬁrst L (cid:48) layers and only train the remaining ( L + 1 − L (cid:48) ) layers, then the resulting NTK is Θ( x, x (cid:48) )
= (cid:80) L +1 h
= L (cid:48) +1 (cid:16) Σ ( h − 1)
( x, x (cid:48) ) · (cid:81)
L +1 h (cid:48) =
h ˙Σ ( h (cid:48) )
( x, x (cid:48) ) (cid:17) .
Note when L (cid:48)
=
L , then the resulting NTK is Σ
( L ) ( x, x (cid:48) ) , which is the NNGP kernel (Lee et al., 2018).
L (cid:48) can be viewed as a hyperparameter of
NTK classiﬁer, and in our UCI experiment we tune L (cid:48) .
Given a kernel function, one can directly use it for downstream classiﬁcation tasks (Scholkopf & Smola, 2001).
E XPERIMENTS ON UCI D ATASETS
In this section, we present our experimental results on UCI datasets which follow the setup of
Fern´andez-Delgado et al.
(2014) with extensive comparisons of classiﬁers, including random forest, kernel SVM, multilayer neural networks, etc.
Section 4.2 discusses the performance of NTK through detailed comparisons with other classiﬁers tested by Fern´andez-Delgado et al.
Section 4.2 compares NTK classiﬁer and the corresponding NN classiﬁer and veriﬁes how similar their predictions are.
See Table 6 in Appendix A for a summary of datasets we used.
The detailed experiment setup, including the choices the datasets, training / test splitting and ranges of hyperparameters, is described in Appendix A.
We note that usual methods of obtaining conﬁdence bounds in these low-data settings are somewhat heuristic.
4. O VERALL P ERFORMANCE C OMPARISONS
Table 1 lists the performance of 6 classiﬁers under various metrics: the three top classiﬁers identiﬁed in Fern´andez-Delgado et al.
(2014), namely, RF, Gaussian kernel and polynomial kernel, along with our new methods NTK, NN with He initialization and NN with NTK initialization.
Tableshows NTK is the best classiﬁer under all metrics, followed by RF, the best classiﬁer identiﬁed in
Fern´andez-Delgado et al.
Now we interpret each metric in more details.
Friedman Ranking and Average Accuracy.
NTK is the best (Friedman Rank 28.34, Average
Accuracy 81.95%), followed by RF (Friedman Rank 33.51, Average Accuracy 81.56%) and then followed by SVM with Gaussian kernel (Friedman Rank 35.76, Average Accuracy 81.03%).
The difference between NTK and RF is signiﬁcant (-5.17 in Friedman Rank and +0.39% in Average
Accuracy), just as the superiority of RF is signiﬁcant compared to other classiﬁers as claimed in
Fern´andez-Delgado et al.
NN (with either He initialization or NTK initialization) performs signiﬁcantly better than the polynomial kernel in terms of the Average Accuracy (80.88% and 81.02% vs. 78.21%), but in terms of Friedman Rank, NN with He initialization is worse than polynomial kernel (40.97 vs. 38.44) and NN with NTK initialization is slightly better than polynomial kernel (38.06 vs. 38.44).
On many datasets where most classiﬁers have similar performances,
NN’s rank is high as well, whereas on other datasets, NN is signiﬁcantly better than most classiﬁers, including SVM with polynomial kernel.
Therefore, NN enjoys higher Average Accuracy but suffers higher Friedman Rank.
For example, on OZONE dataset, NN with NTK initialization is only 0.25% worse than polynomial kernel but their difference in terms of rank is 56.
It is also interesting to see that NN with NTK initialization performs better than NN with He initialization (38.06 vs. 40.97 in
Friedman Rank and 81.02% vs. 80.88% in Average Accuracy).
P90/P95 and PMA.
This measures, for a given classiﬁer, the fraction of datasets on which it achieves more than 90%/95% of the maximum accuracy among all classiﬁers.
NTK is one of the best classiﬁers (ties with Gaussian kernel on P95), which shows NTK can consistently achieve superior classiﬁcation performance across a broad range of datasets.
Lastly, we consider the Percentage of the Maximum Accuracy (PMA).
NTK achieves the best average PMA followed by RF whose PMA is 0.47% below that of NTK and other classiﬁers are all below 94.6%.
An interesting observation is that NTK, NN with NTK initialization and RF have small standard deviation (5.17%, 5.89% and 5.30%) whereas all other classiﬁers have much larger standard deviation.
4. P AIRWISE C OMPARISONS
NTK vs. RF.
In Figure 1a, we compare NTK with RF.
There are 42 datasets that NTK outperforms
RF and 40 datasets that RF outperforms NTK.
The mean difference is 2.51%, which is statistically signiﬁcant by a Wilcoxon signed rank test.
We see NTK and RF perform similarly when the Bayes error rate is low with NTK being slightly better.
There are some exceptions.
For example, on the
BALANCE - SCALE dataset, NTK achieves 98% accuracy whereas RF only achieves 84.1%.
When the Bayes error rate is high, either classiﬁer can be signiﬁcantly better than the other.
NTK vs. Gaussian Kernel.
The gap between NTK and Gaussian kernel is more signiﬁcant.
As shown in Figure 1b, NTK generally performs better than Gaussian kernel no matter Bayes error rate is low or high.
There are 43 datasets that NTK outperforms Gaussian kernel and there are 34 datasets that Gaussian kernel outperforms NTK.
The mean difference is 2.22%, which is also statistically signiﬁcant by a Wilcoxon signed rank test.
On BALLOONS , HEART - SWITZERLAND , PITTSBURGBRIDGES - MATERIAL , TEACHING , TRAINS datasets, NTK is better than Gaussian kernel by at least 11% in terms of accuracy.
These ﬁve datasets all have less than 200 samples, which shows that NTK can perform much better than Gaussian kernel when the number of samples is small.
NTK vs. NN
In this section we compare NTK with NN.
The goals are (i) comparing the performance and (ii) verifying NTK is a good approximation to NN.
In Figure 2a, we compare the performance between NTK and NN with He initialization.
For most datasets, these two classiﬁers perform similarly.
However, there are a few datasets that NTK performs signiﬁcantly better.
There are 50 datasets that NTK outperforms NN with He initialization and there are 27 datasets that NN with He initialization outperforms NTK.
The mean difference is 1.96%, which is also statistically signiﬁcant by a Wilcoxon signed rank test.
In Figure 2b, we compare the performance between NTK and NN with NTK initialization.
Recall that for NN with NTK initialization, when the width goes to inﬁnity, the predictor is just NTK.
[Figure] [Figure]
Therefore, we expect these two predictors give similar performance.
Figure 2b veriﬁes our conjecture.
There is no dataset the one classiﬁer is signiﬁcantly better than the other.
We do not have the same observation in Figure 1a, 1b or 2a.
Nevertheless, NTK often performs better than NN.
There are 52 datasets that NTK outperforms NN with He initialization and there are 25 datasets NN with He initialization outperforms NTK.
The mean difference is 1.54%, which is also statistically signiﬁcant by a Wilcoxon signed rank test.
E XPERIMENTS ON S MALL CIFAR-10 D ATASET
In this section, we study the performance of CNTK on subsampled CIFAR-10 dataset.
We randomly choose n samples from CIFAR-10 training set, use them to train CNTK and ResNet-34, and test both classiﬁers on the whole test set.
In our experiments, we vary n from 10 to 1280 , and the number of convolutional layers of CNTK varies from 5 to 14 .
See Appendix B for detailed experiment setup.
The results are reported in Table 2.
It can be observed that in this setting CNTK consistently outperforms ResNet-34.
The largest gap occurs at n = 320 where 14-layer CNTK achieves 36 .
57% accuracy and ResNet achieves 33 .
The smallest improvement occurs at n = 10 : 15 .
33% vs.
[Table] [Table]
in ResNet-50.
When the number of training data is large, i.e., n = 1280 , ResNet can outperform CNTK.
It is also interesting to see that 14-layer CNTK is the best performing CNTK for all values of n , suggesting depth has a signiﬁcant effect on this task.
E XPERIMENTS ON F EW - SHOT L EARNING
In this section, we test the ability of NTK as a drop-in classiﬁer to replace the linear classiﬁer in the few-shot learning setting.
Using linear SVM on top of extracted features is arguably the most widely used strategy in few-shot learning as linear SVM is easy and fast to train whereas more complicated strategies like ﬁne-tuning or training a neural network on top of the extracted feature have more randomness and may overﬁt due to the small size of the training set.
NTK has the same beneﬁts (easy and fast to train, no randomness) as the linear classiﬁer but also allows some non-linearity in the design.
Note since we consider image classiﬁcation tasks, we use CNTK in experiments in this section.
Experiment Setup.
We mostly follow the settings of Goyal et al.
Features are extracted from layer conv1, conv2, conv3, conv4, conv5 in ResNet-50 (He et al., 2016) trained on ImageNet (Deng et al., 2009), and we use these features for VOC07 classiﬁcation task.
The number of positive examples k varies from 1 to 8 .
For each value of k , for each class in the VOC dataset, we choose k positive examples and 19 k negative examples.
For each k ∈ { 1 , 2 , . . . , 8 } and for each class, we randomly choose 10 independent sets with 20 k training samples in each set.
We
[Table] [Table]
report the mean and standard deviation of mAP on the test split of VOC07 dataset.
This setting has been used in numbers of previous few-shot learning papers (Goyal et al., 2019; Zhang et al., 2017).
We take the extracted features as the input to CNTK.
We tried CNTK with 0 - 6 convolution layers, a global average pooling layer and a fully-connected layer.
We normalize the data in the feature space, and ﬁnally use SVM to train the classiﬁers.
Note without the convolution layer, it is equivalent to directly applying linear SVM after global average pooling and normalization.
We use sklearn.svm.
LinearSVC to train linear SVMs, and sklearn.svm.
SVC to train kernel
SVMs (for CNTK).
To train SVM, we choose the cost value C from 2 [ − 19 , − 4] ∪ 10 [ − 7 , 2] and set the class weight ratio to be 2 : 1 for positive/negative classes as in Goyal et al.
Since the number of given samples is usually small, Goyal et al.
(2019) chooses to report the performance of the best cost value C .
In our experiments, we use a more standard method to perform cross-validation.
We use the ﬁrst 10 classes of VOC07 to tune C , and report the performance of selected C in the other 10 classes in Table 3-5.
We also report the performance of the best C as in
Goyal et al.
(2019), in Tables 7-9 in the appendix for completeness.
Discussions.
First, we ﬁnd CNTK is a strong drop-in classiﬁer to replace the linear classiﬁer in few-shot learning.
Tables 3-9 clearly demonstrate that CNTK is consistently better than linear classiﬁer.
Note Tables 3-9 only show the prediction accuracy in an average sense.
In fact, we ﬁnd that on every randomly sampled training set, CNTK always gives a better performance.
We conjecture that CNTK gives better performance because the non-linearity in CNTK helps prediction.
This is veriﬁed by looking at the performance gain of CNTK for different feature extractors.
Note Conv is often considered to be most useful for linear classiﬁcation.
There, CNTK only outperforms the linear classiﬁer by about 1% .
On the other hand, with Conv3 and Conv4 features, CNTK can outperform linear classiﬁer by around 2% and sometimes by 3% – 4% (last two rows in Table 5).
We believe this happens because Conv3 and Conv4 correspond to middle-level features, and thus non-linearity is indeed beneﬁcial for better accuracy.
We also observe that CNTK often performs the best with a single convolutional layer.
This is expected since features extracted by ResNet-50 already produce a good representation.
Nevertheless, we ﬁnd for middle-level features Conv3 with k = 7 or 8 , CNTK with 3 convolutional layers give the best performance.
We believe this is because with more data, utilizing the non-linearity induced by CNTK can further boost the performance.
C ONCLUSION
The Neural Tangent Kernel, discovered by mathematical curiosity about deep networks in the limit of inﬁnite width, is found to yield superb performance on low-data tasks, beating extensively tuned versions of classic methods such as random forests.
The (fully-connected) NTK classiﬁers are easy to compute (no GPU required) and thus should be a good off-the-shelf classiﬁer in many settings.
We plan to release our code to allow drop-in replacement for SVMs and linear regression.
Many theoretical questions arise.
Do NTK SVMs correspond to some inﬁnite net architecture (as
NTK ridge regression does)?
What explains generalization in small-data settings?
(This understanding is imperfect even for random forests.)
Finally one can derive NTK corresponding to other architectures, e.g., recurrent neural tangent kernel (RNTK) induced by recurrent neural networks.
It would be an interesting future research direction to test their performance on benchmark tasks.
A CKNOWLEDGMENTS
S. Arora, Z. Li and D. Yu are supported by NSF, ONR, Simons Foundation, Schmidt Foundation, Amazon Research, DARPA and SRC. S. S. Du is supported by National Science Foundation (Grant
No. DMS-1638352) and the Infosys Membership.
R. Salakhutdinov and R. Wang are supported in part by NSF IIS-1763562, AFRL CogDeCON FA875018C0014, and DARPA SAGAMORE
HR00111990016.
Part of this work was done while S. S. Du was visiting Google Brain Princeton and R. Wang was visiting Princeton University.
The authors would like to thank Amazon Web
Services for providing compute time for the experiments in this paper, and NVIDIA for GPU support.
We thank Priya Goyal for providing experiment details of Goyal et al.
We thank
Xiaolong Wang for discussing the few-shot learning task
A A DDITIONAL E XPERIMENTAL D ETAILS ON UCI
In this section, we describe our experiment setup.
Dataset Selection Since we only wish to test on small datasets, we only select UCI datasets with number of samples smaller than 5000 .
Furthermore, we only use datasets without explicit training / testing splitting from the link: http://persoal.citius.usc.es/manuel.fernandez.
delgado/papers/jmlr/data.tar.gz , which are the pre-processed datasets by Fern´andezDelgado
et al.
These datasets are originally from UCI datasets (including most of the datasets before March, 2013) and 4 real-world datasets not included in the UCI repository (see Fern´andezDelgado et al.
(2014) for details).
The datasets are pre-processed to be classiﬁcation problems and all categorical features are turned into numerical features and normalized along the samples for each feature.
The reason to discard datasets with explicit splitting is that we found there is an obvious distributional shift between training and testing data.
The pre-processed data provided from the link shows that training and testing data are normalized with different mean and standard deviation.
For example, on the AUDIOLOGY - STD dataset, the support of the ﬁrst feature is {− 0 .
761526 , 1 . 30547 } in training and {− 1 . 2 , 0 . 8 } in testing; on the ANNEALING dataset, the support of the ﬁrst feature is {− 0 .
670274 , − 0 .
79822 } in training and {− 0 .
111111 , 0 , 1 } in testing.
See Table 6 for a summary of the datasets.
[Table]
Performance Comparison Details We follow the comparison setup in Fern ´ andez-Delgado et al.
(2014) that we report 4 -fold cross-validation.
For hyperparameters, we tune them with the same validation methodology in Fern ´ andez-Delgado et al.
(2014): all available training samples are randomly split into one training and one test set, while imposing that each class has the same number of training and test samples.
Then the parameter with best validation accuracy is selected.
It is possible to give conﬁdence bounds for this parameter tuning scheme, but they are worse than standard ones for separated training/validation/testing data.
For NTK and NN classiﬁers we train them on thesedatasets.
For other classiﬁers, we use the results from Fern´andez-Delgado et al.
NTK Speciﬁcation We calculate NTK induced fully-connected neural networks with L layers where L (cid:48) bottom layers are ﬁxed, and then use C -support vector classiﬁcation implemented by sklearn.svm .
We tune hyperparameters L from 1 to 5 , L (cid:48) from 0 to L − 1 , and cost value C as powers of ten from − 2 to 4 .
The number of kernels used is 15 , so the total number of parameter combinations is 105.
Note this number is much less than the number of hyperparameter of Gaussian Kernel reported in Fern´andez-Delgado et al.
(2014) where they tune hyperparameters withcombinations.
NN Speciﬁcation We use fully-connected NN with L layers, 512 number of hidden nodes per layer and use gradient descent to train the neural network.
We tune hyperparameters L from 1 to 5 , with / without batch normalization and learning rate 0 . 1 or 1 .
We run gradient descent forepochs.
We treat NN with He initialization and NTK initialization as two classiﬁers and report their results separately.
B A DDITIONAL E XPERIMENTAL D ETAILS ON S MALL CIFAR-10 D ATASETS
We randomly choose n/ 10 samples from each class of CIFAR-10 training set and test classiﬁers on the whole testing set.
n varies from 10 to 1280 .
For each n , we repeat 20 times and report the mean accuracy and its standard deviation for each classiﬁer.
The number of convolution layers of CNTK ranges from 5-14.
After convolutional layers, we apply a global pooling layer and a fully connected layer.
We refer readers to Arora et al. (2019a) for exact formulas of CNTK.
We normalize the kernel such that each sample has unit length in feature space.
We use ResNet-34 with width 64,128,256 and default hyperparameters: learning rate 0.1, momentum 0.9, weight decay 0.0005.
We decay the learning rate by 10 at the epoch of 80 and 120, with 160 training epochs in total.
The training batch size is the minimum of the size of the whole training dataset and 160.
We report the best testing accuracy among epochs.
C A DDITIONAL R ESULTS IN F EW - SHOT L EARNING
Tables 7-9 show the performance of the best C as has been done in Goyal et al.
[Table]
[Table] [Table]