Recent research shows that the following two models are equivalent: (a) infinitely wide neural networks (NNs) trained under ℓ2 loss by gradient descent with infinitesimally small learning rate (b) kernel regression with respect to so-called Neural Tangent Kernels (NTKs) (Jacot et al., 2018 ). An efficient algorithm to compute the NTK, as well as its convolutional counterparts, appears in Arora et al. (2019a) , which allowed studying performance of infinitely wide nets on datasets like CIFAR-10. However, superquadratic running time of kernel methods makes them best suited for small-data tasks. We report results suggesting neural tangent kernels perform strongly on low-data tasks. 1. On a standard testbed of classification/regression tasks from the UCI database, NTK SVM beats the previous gold standard, Random Forests (RF), and also the corresponding finite nets. 2. On CIFAR-10 with 10 -640 training samples, Convolutional NTK consistently beats ResNet-34 by 1% -3%. 3. On VOC07 testbed for few-shot image classification tasks on ImageNet with transfer learning (Goyal et al., 2019) , replacing the linear SVM currently used with a Convolutional NTK SVM consistently improves performance. 4. Comparing the performance of NTK with the finite-width net it was derived from, NTK behavior starts at lower net widths than suggested by theoretical analysis(Arora et al., 2019a). NTK's efficacy may trace to lower variance of output. Ironically, while the above-mentioned analysis, at first sight, appears to reduce the study of a complicated model -deep networks -to an older, simpler model -kernel regression -in practice the simpler model is computationally less efficient because running time of kernel regression can be quadratic in the number of data points! 1 Thus computing using CNTK kernel on large datasets like ImageNet currently appears infeasible. Even on CIFAR-10, it seems infeasible to incorporate data augmentation. However, kernel classifiers are very efficient on small datasets. Here NTKs could conceivably be practical while at the same time bringing some of the power of deep networks to these settings. We recall that recently Olson et al. (2018) showed that multilayer neural networks can be reasonably effective on small datasets, specifically on a UCI testbed of tasks with as few as dozens of training examples. Of course, this required some hyperparameter tuning, although they noted that such tuning is also needed for the champion method, Random Forests (RF), which multilayer neural networks could not beat. It is thus natural to check if NTK -corresponding to infinitely wide fully-connected networks -performs well in such small-data tasks 2 . Convex objectives arising from kernels have stable solvers with minimal hyperparameter tuning. Furthermore, random initialization in deep network training seems to lead to higher variance in the output, which can hurt performance in small-data settings. Can NTK's do better? Below we will see that in the setup of Olson et al. (2018), NTK predictors indeed outperforms corresponding finite deep networks, and also slightly beats the earlier gold standard, Random Forests. This suggests NTK predictors should belong in any list of off-the-shelf machine learning methods. Following are low-data settings where we used NTKs and CNTKs: • In the testbed of 90 classification tasks from UCI database, NTK predictor achieves superior, and arguably the strongest classification performance. This is verified via several standard statistical tests, including Friedman Rank, Average Accuracy, Percentage of the Maximum Accuracy (PMA) and probability of achieving 90%/95% maximum accuracy (P90 and P95), performed to compare performances of different classifiers on 90 datasets from UCI database. (The authors plan to release the code, to allow off-the-shelf use of this method. It does not require GPUs.) • We find the performance of NN is close to that of NTK. On every dataset from UCI database, the difference between the classification accuracy of NN and that of NTK is within 5%. On the other hand, on some datasets, the difference between classification accuracy of NN (or NTK) and that of other classifiers like RF can be as high as 20%. This indicates in low-data settings, NTK is indeed a good description of NN. Furthermore, we find NTK is more stable (smaller variance), which seems to help it achieve better accuracy on small datasets (cf. Figure 2b ). • CNTK is useful in computer vision tasks with small-data. On CIFAR-10, we compare CNTK with ResNet using 10 -640 training samples and find CNTK can beat ResNet by 1% − 3%. We further study few-shot image classification task on VOC07 dataset. The standard method is to first use a pre-trained network, e.g., ResNet-50 trained on ImageNet, to extract features and then directly apply a linear classifier on the extracted features (Goyal et al., 2019) . Here we replace the linear classifier with CNTK and obtain better classification accuracy in various setups. Paper organization. Section 2 discusses related work. Section 3 reviews the derivation of NTK. Section 4 presents experiments using NN and NTK on UCI datasets. Section 5 presents experiments using CNN and CNTK on small CIFAR-10 datasets. Section 6 presents experiments using CNTK for the few-shot learning setting. Additional technical details are presented in appendix. Our paper is inspired by Fernández-Delgado et al. (2014) which conducted extensive experiments on UCI dataset. Their conclusion is random forest performs the best, which is followed by the SVM with Gaussian 1 The bottleneck is constructing the kernel, which scales quadratically with the number of data points (Arora et al., 2019a). The regression also requires matrix inversion, which can be cubic in the number of data points. 2 Note that NTKs can also be used in kernel SVMs, which are not known to be equivalent to training infinitely wide networks. Currently, equivalence is only known for ridge regression. We tried both.
Arora et al. REF suggests the use of neural tangent kernels in low data tasks.
Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks
ArXiv
ArXiv
Computer Science Mathematics