D ATASET M ETA -L EARNING
FROM
K ERNEL
R IDGE - R EGRESSION
Timothy Nguyen
Zhourong Chen Jaehoon Lee Google Research { timothycnguyen, zrchen, jaehlee } @google.com
A BSTRACT
One of the most fundamental aspects of any machine learning algorithm is the training data used by the algorithm.
We introduce the novel concept of (cid:15) - approximation of datasets, obtaining datasets which are much smaller than or are signiﬁcant corruptions of the original training data while maintaining similar model performance.
We introduce a meta-learning algorithm called Kernel Inducing Points ( KIP ) for obtaining such remarkable datasets, inspired by the recent developments in the correspondence between inﬁnitely-wide neural networks and kernel ridge-regression (KRR).
For KRR tasks, we demonstrate that KIP can compress datasets by one or two orders of magnitude, signiﬁcantly improving previous dataset distillation and subset selection methods while obtaining state of the art results for MNIST and CIFAR-10 classiﬁcation.
Furthermore, our KIP -learned datasets are transferable to the training of ﬁnite-width neural networks even beyond the lazy-training regime, which leads to state of the art results for neural network dataset distillation with potential applications to privacy-preservation.
I NTRODUCTION
Datasets are a pivotal component in any machine learning task.
Typically, a machine learning problem regards a dataset as given and uses it to train a model according to some speciﬁc objective.
In this work, we depart from the traditional paradigm by instead optimizing a dataset with respect to a learning objective, from which the resulting dataset can be used in a range of downstream learning tasks.
Our work is directly motivated by several challenges in existing learning methods.
Kernel methods or instance-based learning (Vinyals et al., 2016; Snell et al., 2017; Kaya & Bilge, 2019) in general require a support dataset to be deployed at inference time.
Achieving good prediction accuracy typically requires having a large support set, which inevitably increases both memory footprint and latency at inference time— the scalability issue .
It can also raise privacy concerns when deploying a support set of original examples, e.g., distributing raw images to user devices.
Additional challenges to scalability include, for instance, the desire for rapid hyper-parameter search (Shleifer & Prokop, 2019) and minimizing the resources consumed when replaying data for continual learning (Borsos et al., 2020).
A valuable contribution to all these problems would be to ﬁnd surrogate datasets that can mitigate the challenges which occur for naturally occurring datasets without a signiﬁcant sacriﬁce in performance.
This suggests the following
Question: What is the space of datasets, possibly with constraints in regards to size or signal preserved, whose trained models are all (approximately) equivalent to some speciﬁc model?
In attempting to answer this question, in the setting of supervised learning on image data, we discover a rich variety of datasets, diverse in size and human interpretability while also robust to model architectures, which yield high performance or state of the art (SOTA) results when used as training data.
We obtain such datasets through the introduction of a novel meta-learning algorithm called
Kernel Inducing Points ( KIP ).
Figure 1 shows some example images from our learned datasets.
O r i g i n a l Airplane
Automobile
Bird
Cat
K I P
K I P = 0 .(a)
Frog
Bird a i r p l a n e a u t o m o b il e b i r d c a t d ee r d o g f r o g
h
o r s e s h
i p t
r u c k
Myrlte LS 500 support a i r p l a n e a u t o m o b il e b i r d c a t d ee r d o g f r o g h
o r s e s h
i p t
r u c k
RawLS a i r p l a n e a u t o m o b il e b i r d c a t d ee r d o g f r o g h
o r s e s h
i p t
r u c k 0(b)
We explore KIP in the context of compressing and corrupting datasets, validating its effectiveness in the setting of kernel-ridge regression (KRR) and neural network training on benchmark datasets
MNIST and CIFAR-10.
Our contributions can be summarized as follows:
1. S UMMARY OF CONTRIBUTIONS
• We formulate a novel concept of (cid:15) -approximation of a dataset.
This provides a theoretical framework for understanding dataset distillation and compression.
• We introduce Kernel Inducing Points ( KIP ), a meta-learning algorithm for obtaining (cid:15) - approximation of datasets.
We establish convergence in the case of a linear kernel in Theorem 1.
We also introduce a variant called Label Solve ( LS ), which gives a closed-form solution for obtaining distilled datasets differing only via labels.
• We explore the following aspects of (cid:15) -approximation of datasets: 1.
Compression (Distillation) for Kernel Ridge-Regression: For kernel ridge regression, we improve sample efﬁciency by over one or two orders of magnitude, e.g. using 10 images to outperform hundreds or thousands of images (Tables 1, 2 vs Tables A1, A2).
We obtain state of the art results for MNIST and CIFAR-10 classiﬁcation while using few enough images (10K) to allow for in-memory inference (Tables A3, A4).
Compression (Distillation) for Neural Networks:
We obtain state of the art dataset distillation results for the training of neural networks, often times even with only a single hidden layer fully-connected network (Tables 1 and 2).
Privacy: We obtain datasets with a strong trade-off between corruption and test accuracy, which suggests applications to privacy-preserving dataset creation.
In particular, we produce images with up to 90% of their pixels corrupted with limited degradation in performance as measured by test accuracy in the appropriate regimes (Figures 3, A3, and Tables A5-A10) and which simultaneously outperform natural images, in a wide variety of settings.
• We provide an open source implementation of KIP and LS , available in an interactive Colab notebook 1 .
S ETUP
In this section we deﬁne some key concepts for our methods.
Deﬁnition 1.
A dataset in R d is a set of n distinct vectors in R d for some n ≥ 1 .
We refer to each such vector as a datapoint .
A dataset is labeled if each datapoint is paired with a label vector in R C , for some ﬁxed C .
A datapoint along with its corresponding label is a labeled datapoint .
We use the notation D =
( X, y ) , where X ∈ R n × d and y ∈ R n × C , to denote the tuple of unlabeled datapoints
X with their corresponding labels y .
We henceforth assume all datasets are labeled.
Next, we introduce our notions of approximation, both of functions (representing learned algorithms) and of datasets, which are characterized in terms of performance with respect to a loss function rather than closeness with respect to a metric.
A loss function (cid:96) : R C × R C
→ R is one that is nonnegative and satisﬁes (cid:96) ( z, z ) = 0 for all z .
Deﬁnition 2.
Fix a loss function (cid:96) and let f, ˜ f
: R d → R C be two functions.
Let (cid:15) ≥ 0 .
Given a distribution P on R d × R C , we say f and ˜ f are weakly (cid:15) -close with respect to ( (cid:96), P ) if
[Equation]
Given a distribution P on R d we say f and ˜ f are strongly (cid:15) -close with respect to ( (cid:96), P ) if
[Equation]
( f (
[Equation]
[Equation]
We drop explicit reference to ( (cid:96), P ) if their values are understood or immaterial.
Given a learning algorithm A (e.g. gradient descent with respect to the loss function of a neural network), let A D denote the resulting model obtained after training A on D .
We regard A D as a mapping from datapoints to prediction labels.
Deﬁnition 3.
Fix learning algorithms A and ˜ A .
Let D and ˜ D be two labeled datasets in R d with label space R C .
Let (cid:15) ≥ 0 .
We say ˜ D is a weak (cid:15) -approximation of D with respect to ( ˜ A, A, (cid:96), P ) if ˜ A ˜ D and A D are weakly (cid:15) -close with respect to ( (cid:96), P ) , where (cid:96) is a loss function and P is a distribution on R d × R C .
We deﬁne strong (cid:15) -approximation similarly.
We drop explicit reference to (some of) the ˜ A, A, (cid:96), P if their values are understood or immaterial.
We provide some justiﬁcation for this deﬁnition in the Appendix.
In this paper, we will measure (cid:15) - approximation with respect to 0 - 1 loss for multiway classiﬁcation (i.e. accuracy).
We focus on weak (cid:15) -approximation, since in most of our experiments, we consider models in the low-data regime with large classiﬁcation error rates, in which case, sample-wise agreement of two models is not of central importance.
On the other hand, observe that if two models have population classiﬁcation error rates less than (cid:15)/ 2 , then (2) is automatically satisﬁed, in which case, the notions of weak-approximation and strong-approximation converge.
We list several examples of (cid:15) -approximation, with (cid:15) = 0 , for the case
when ˜ A =
A are given by the following:
Example 1: Support Vector Machines.
Given a dataset D of size N , train an SVM on D and obtain
M support vectors.
These M support vectors yield a dataset ˜ D that is a strong 0 -approximation to D in the linearly separable case, while for the nonseparable case, one has to also include the datapoints with positive slack.
Asymptotic lower bounds asserting M =
O ( N ) have been shown in Steinwart (2003).
Example 2: Ridge Regression.
Any two datasets D and ˜ D that determine the same ridge-regressor are 0 -approximations of each other.
In particular, in the scalar case, we can obtain arbitrarily small 0 -approximating ˜ D as follows.
Given training data D =
( X, y ) in R d , the corresponding ridgeregressor is the predictor
[Equation]
where for λ = 0 , we interpret the inverse as a pseudoinverse.
It follows that for any given w ∈ R d × 1 , we can always ﬁnd
( ˜ X, ˜ y ) of arbitrary size (i.e. ˜ X ∈ R n × d , y ∈ R n × 1 with n arbitrarily small) that satisﬁes w =
Φ λ ( ˜ X )˜
y .
Simply choose ˜ X such that w is in the range of Φ λ ( ˜ X ) .
The resulting dataset ( ˜ X, ˜ y ) is a 0 -approximation to D .
If we have a C -dimensional regression problem, the preceding analysis can be repeated component-wise in label-space to show 0 -approximation with a dataset of size at least C (since then the rank of Φ λ ( ˜ X ) can be made at least the rank of w ∈ R d × C ).
We are interested in learning algorithms given by KRR and neural networks.
These can be investigated in unison via neural tangent kernels.
Furthermore, we study two settings for the usage of (cid:15) -approximate datasets, though there are bound to be others:
(Sample efﬁciency / compression)
Fix (cid:15) .
What is the minimum size of ˜ D needed in order for ˜ D to be an (cid:15) -approximate dataset?
(Privacy guarantee)
Can an (cid:15) -approximate dataset be found such that the distribution from which it is drawn and the distribution from which the original training dataset is drawn satisfy a given upper bound in mutual information?
Motivated by these questions, we introduce the following deﬁnitions:
Deﬁnition 4.
(Heuristic) Let ˜ D
and D be two datasets such that ˜ D is a weak (cid:15) -approximation of D , with | ˜ D | ≤ | D | and (cid:15) small.
We call | D | / | ˜ D | the compression ratio .
In other words, the compression ratio is a measure of how well ˜ D compresses the information available in D , as measured by approximate agreement of their population loss.
Our deﬁnition is heuristic in that (cid:15) is not precisely quantiﬁed and so is meant as a soft measure of compression.
Deﬁnition 5.
Let Γ be an algorithm that takes a dataset D in R d and returns a (random) collection of datasets in R d .
For 0 ≤ ρ ≤ 1 , we say that Γ is ρ -corrupted if for any input dataset D , every datapoint 3 drawn from the datasets of Γ( D ) has at least ρ fraction of its coordinates independent of
D .
In other words, datasets produced by Γ have ρ fraction of its entries contain no information about the dataset D (e.g. because they have a ﬁxed value or are ﬁlled in randomly).
Corrupting information is naturally a way of enhancing privacy, as it makes it more difﬁcult for an attacker to obtain useful information about the data used to train a model.
Adding noise to the inputs to neural network or of its gradient updates can be shown to provide differentially private guarantees (Abadi et al.
K ERNEL I NDUCING P OINTS
Given a dataset D sampled from a distribution P , we want to ﬁnd a small dataset ˜ D that is an (cid:15) - approximation to D (or some large subset thereof) with respect to ( ˜ A, A, (cid:96), P ) .
Focusing on ˜
A
=
A for the moment, and making the approximation
[Equation]
(6) this suggests we should optimize the right-hand side of (6) with respect to ˜ D , using D as a validation set.
For general algorithms ˜ A , the outer optimization for ˜ D is computationally expensive and involves second-order derivatives, since one has to optimize over the inner loop encoded by the learning algorithm
˜
A .
We are thus led to consider the class of algorithms drawn from kernel ridgeregression.
The reason for this are two-fold.
First, KRR performs convex-optimization resulting in a closed-form solution, so that when optimizing for the training parameters of KRR (in particular, the support data), we only have to consider ﬁrst-order optimization.
Second, since KRR for a neural tangent kernel (NTK) approximates the training of the corresponding wide neural network (Jacot et al., 2018; Lee et al., 2019; Arora et al., 2019a; Lee et al., 2020), we expect the use of neural kernels to yield (cid:15) -approximations of D for learning algorithms given by a broad class of neural networks trainings as well.
(This be Require:
A target labeled dataset ( X t , y t ) along with a kernel or family of kernels.
This leads to our ﬁrst-order meta-learning algorithm KIP (Kernel Inducing Points), which uses kernel-ridge regression to learn (cid:15) -approximate datasets.
It can be regarded as an adaption of the inducing point method for Gaussian processes (Snelson & Ghahramani, 2006) to the case of KRR.
Given a kernel K , the KRR loss function trained on a support dataset ( X s , y s ) and evaluated on a target dataset ( X t , y t ) is given by
[Equation]
(7) where if U and V are sets, K UV is the matrix of kernel elements ( K ( u, v ))
u ∈ U,v ∈ V .
Here λ >is a ﬁxed regularization parameter.
The KIP algorithm consists of optimizing (7) with respect to the support set (either just the X s or along with the labels y s ), see Algorithm 1.
Depending on the downstream task, it can be helpful to use families of kernels (Step 3) because then KIP produces datasets that are (cid:15) -approximations for a variety of kernels instead of a single one.
This leads to a corresponding robustness for the learned datasets when used for neural network training.
We remark on best experimental practices for sampling methods and initializations for KIP in the Appendix.
Theoretical analysis for the convergence properties of KIP for the case of a linear kernel is provided by Theorem 1.
Sample KIP -learned images can be found in Section F.
KIP variations: i)
We can also randomly augment the sampled target batches in KIP .
This effectively enhances the target dataset ( X t , y t ) , and we obtain improved results in this way, with no extra computational cost with respect to the support size.
ii)
We also can choose a corruption fraction 0 ≤ ρ < 1 and do the following.
Initialize a random ρ -percent of the coordinates of each support datapoint via some corruption scheme (zero out all such pixels or initialize with noise).
Next, do not update such corrupted coordinates during the KIP training algorithm (i.e. we only perform gradient updates on the complementary set of coordinates).
Call this resulting algorithm KIP ρ .
In this way,
KIP ρ is ρ -corrupted according to Deﬁnition 5 and we use it to obtain our highly corrupted datasets.
Label solving:
In addition to KIP , where we learn the support dataset via gradient descent, we propose another inducing point method, Label Solve ( LS ), in which we directly ﬁnd the minimum of (7) with respect to the support labels while holding X s ﬁxed.
This is simple because the loss function is quadratic in y s .
We refer to the resulting labels
[Equation]
(8) as solved labels .
As Φ 0 is the pseudo-inverse operation, y ∗ s is the minimum-norm solution among minimizers of (7).
If K X t X s is injective, using the fact that Φ 0 ( AB ) =
Φ
( B )Φ 0 ( A ) for A injective and B surjective (Greville (1966)), we can rewrite (8) as
[Equation]
0 ( K X t X s ) y t .
E XPERIMENTS
We perform three sets of experiments to validate the efﬁcacy of KIP and LS for dataset learning.
The ﬁrst set of experiments investigates optimizing KIP and LS for compressing datasets and achieving state of the art performance for individual kernels.
The second set of experiments explores transferability of such learned datasets across different kernels.
The third set of experiments investigate the transferability of KIP -learned datasets to training neural networks.
The overall conclusion is that
KIP -learned datasets, even highly corrupted versions, perform well in a wide variety of settings.
Experimental details can be found in the Appendix.
We focus on MNIST (LeCun et al., 2010) and CIFAR-10 (Krizhevsky et al., 2009)
datasets for comparison to previous methods.
For LS , we also use Fashion-MNIST.
These classiﬁcation tasks are recast as regression problems by using mean-centered one-hot labels during training and by making class predictions via assigning the class index with maximal predicted value during testing.
All our kernel-based experiments use the Neural Tangents library (Novak et al., 2020), built on top of JAX (Bradbury et al., 2018).
In what follows, we use FC m and Conv m to denote a depth m fully-connected or fully-convolutional network.
Whether we mean a ﬁnite-width neural network or else the corresponding neural tangent kernel (NTK) will be understood from the context.
We will sometimes also use the neural network Gaussian process (NNGP) kernel associated to a neural network in various places.
By default, a neural kernel refers to NTK unless otherwise stated.
RBF denotes the radial-basis function kernel.
Myrtle-
N architecture follows that of Shankar et al.
(2020), where simple combination of N convolutional layers with (2 , 2) average pooling layers are interweaved to reduce internal patch-size.
We would have used deeper and more diverse architectures for KIP , but computational limits, which will be overcome in future work, placed restrictions, see the Experiment Details in Section D.
4. S INGLE K ERNEL R ESULTS
We apply KIP to learn support datasets of various sizes for MNIST and CIFAR-10.
The objective is to distill the entire training dataset down to datasets of various ﬁxed, smaller sizes to achieve high compression ratio.
We present these results against various baselines in Tables 1 and 2.
These comparisons occur cross-architecturally, but aside from Myrtle LS results, all our results involve the simplest of kernels (RBF or FC1), whereas prior art use deeper architectures (LeNet, AlexNet,
ConvNet).
We obtain state of the art results for KRR on MNIST and CIFAR-10, for the RBF and FC1 kernels, both in terms of accuracy and number of images required, see Tables 1 and 2.
In particular, our method produces datasets such that RBF and FC1 kernels ﬁt to them rival the performance of deep convolutional neural networks on MNIST (exceeding 99.2%).
By comparing Tables 2 and A2, we see that, e.g. 10 or 100 KIP images for RBF and FC1 perform on par with tens or hundreds times more natural images, resulting in a compression ratio of one or two orders of magnitude.
For neural network trainings, for CIFAR-10, the second group of rows in Table 2 shows that FC trained on KIP images outperform prior art, all of which have deeper, more expressive architectures.
On MNIST, we still outperform some prior baselines with deeper architectures.
This, along with the state of the art KRR results, suggests that KIP , when scaled up to deeper architectures, should continue to yield strong neural network performance.
For LS , we use a mix of NNGP kernels {+
 and NTK kernels+} associated to FC1, Myrtle-5, Myrtleto learn labels on various subsets of MNIST, Fashion-MNIST, and CIFAR-10.
Our results comprise the bottom third of Tables 1 and 2 and Figure 2.
As Figure 2 shows, the more targets are used, the better the performance.
When all possible targets are used, we get an optimal compression ratio of roughly one order of magnitude at intermediate support sizes.
4. K ERNEL TO K ERNEL R ESULTS
Here we investigate robustness of KIP and LS learned datasets when there is variation in the kernels used for training and testing.
We draw kernels coming from FC and Conv layers of depths 1-3, since such components form the basic building blocks of neural networks.
Figure A1 shows that KIPdatasets trained with random sampling of all six kernels do better
on
average than KIP -datasets trained using individual kernels.
[Figure] [Table]
[Figure]
[Table]
For LS , transferability between FC1 and Myrtle-10 kernels on CIFAR-10 is highly robust, see Figure A2.
Namely, one can label solve using FC1 and train Myrtle-10 using those labels and vice versa.
There is only a negligible difference in performance in nearly all instances between data with transferred learned labels and with natural labels.
4. K ERNEL TO N EURAL N ETWORKS R ESULTS
Signiﬁcantly, KIP -learned datasets, even with heavy corruption, transfer remarkably well to the training of neural networks.
Here, corruption refers to setting a random ρ fraction of the pixels of each image to uniform noise between − 1 and 1 (for KIP , this is implemented via KIP ρ ) 5 .
The deterioriation in test accuracy for KIP -images is limited as a function of the corruption fraction, especially when compared to natural images, and moreover, corrupted KIP -images typically outperform uncorrupted natural images.
We verify these conclusions along the following dimensions:
Robustness to dataset size: We perform two sets of experiments.
(i)
First, we consider small KIP datasets (10, 100, 200 images) optimized using multiple kernels (FC1-3, Conv1-2), see Tables A5, A6.
We ﬁnd that our in-distribution transfer (the downstream neural network has its neural kernel included among the kernels sampled by KIP ) performs remarkably well, with both uncorrupted and corrupted KIP images beating the uncorrupted natural [Table] images of corresponding size.
Out of distribution networks (LeNet (LeCun et al., 1998) and Wide Resnet (Zagoruyko & Komodakis, 2016)) have less transferability: the uncorrupted images still outperform natural images, and corrupted KIP images still outperform corrupted natural images, but corrupted KIP images no longer outperform uncorrupted natural images.
(ii) We consider larger KIP datasets (1K, 5K, 10K images) optimized using a single FC1 kernel for training of a corresponding FC1 neural network, where the KIP training uses augmentations (with and without label learning), see Tables A7-A10 and Figure A3.
We ﬁnd, as before, KIP images outperform natural images by an impressive margin: for instance, on CIFAR-10, 10K KIP -learned images with 90% corruption achieves 49.9% test accuracy, exceeding 10K natural images with no corruption (acc: 45.5%) and 90% corruption (acc: 33.8%).
Interestingly enough, sometimes higher corruption leads to better test performance (this occurs for CIFAR-10 with cross entropy loss for both natural and KIP -learned images), a phenomenon to be explored in future work.
We also ﬁnd that KIP with label-learning often tends to harm performance, perhaps because the labels are overﬁtting to KRR.
Robustness to hyperparameters: For CIFAR-10, we took 100 images, both clean and 90% corrupted, and trained networks on a wide variety of hyperparameters for various neural architectures.
We considered both neural networks whose corresponding neural kernels were sampled during KIPtraining those that were not.
We found that in both cases, the KIP -learned images almost always outperform 100 random natural images, with the optimal set of hyperparameters yielding a margin close to that predicted from the KRR setting, see Figure 3.
This suggests that KIP -learned images can be useful in accelerating hyperparameter search.
R ELATED W ORK
Coresets: A classical approach for compressing datasets is via subset selection, or some approximation thereof.
One notable work is Borsos et al. (2020), utilizing KRR for dataset subselection. For an overview of notions of coresets based on pointwise approximatation of datasets, see Phillips (2016).
Neural network approaches to dataset distillation: Maclaurin et al.
; Lorraine et al.
(2020) approach dataset distillation through learning the input images from large-scale gradient-based metalearning of hyper-parameters.
They did not evaluate the utility of learned images.
Properties of distilled input data was ﬁrst analyzed in Wang et al.
One of main challenges of the method is that optimization is with respect to a speciﬁc initialization of a network, or else requires sampling many initializations which is costly.
In contrast, our method naturally incorporates meta-learning [Figure] over ensembles of networks through the inﬁnite-width limit correspondence (Lee et al., 2018;
Jacot et al., 2018).
The works Sucholutsky & Schonlau (2019); Bohdal et al.
(2020) build upon Wang et al.
(2018) by distilling labels.
More recently, Zhao et al.
(2020) proposes condensing training set by gradient matching condition and shows improvement over Wang et al.
Inducing points: Our approach has as antecedant the inducing point method for Gaussian Processes (Snelson & Ghahramani, 2006; Titsias, 2009).
However, whereas the latter requires a probabilistic framework that optimizes for marginal likelihood, in our method we only need to consider minimizing mean-square loss on validation data.
Low-rank kernel approximations: Unlike common low-rank approximation methods (Williams & Seeger, 2001; Drineas & Mahoney, 2005), we obtain not only a low-rank support-support kernel matrix with KIP , but also a low-rank target-support kernel matrix. Note that the resulting matrices obtained from KIP need not approximate the original support-support or target-support matrices since KIP only optimizes for the loss function.
Moreover, because our method produces a dataset, our compression method can be transferred to other kernels or neural networks while also preserving corruption.
Neural network kernels: Our work is motivated by the exact correspondence between inﬁnitelywide neural networks and kernel methods (Neal, 1994; Lee et al., 2018; Matthews et al., 2018;
Jacot et al., 2018;
Novak et al., 2019; Garriga-Alonso et al., 2019; Arora et al., 2019a).
These correspondences allow us to view both Bayesian inference and gradient descent training of wide neural networks with squared loss as yielding a Gaussian process or kernel ridge regression with neural kernels.
Instance-Based Encryption: A related approach to corrupting datasets involves encrypting individual images via sign corruption (Huang et al.
C ONCLUSION
We introduced novel algorithms KIP and LS for the meta-learning of datasets.
We obtained a variety of compressed and corrupted datasets, achieving state of the art results for KRR and neural network dataset distillation methods.
This was achieved even using the simplest of kernels and neural networks (shallow fully-connected networks and purely-convolutional networks without pooling), which notwithstanding their limited expressiveness, outperform most baselines that use deeper architectures.
Follow-up work will involve scaling up KIP to deeper architectures with pooling (achievable with multi-device training) for which we expect to obtain even more highly performant datasets, both in terms of overall accuracy and architectural ﬂexibility.
Finally, we obtained highly corrupt datasets whose performance match or exceed natural images, which when developed at scale, could lead to practical applications for privacy-preserving machine learning. A CKNOWLEDGMENTS
We would like to thank Dumitru Erhan, Yang Li, Hossein Mobahi, Jeffrey Pennington, Si Si, Jascha
Sohl-Dickstein and Lechao Xiao for helpful discussions and references
A R EMARKS ON D EFINITION OF (cid:15) - APPROXIMATION
Here, we provide insights into the formulation of Deﬁnition 3.
One noticeable feature of our definition is that it allows for different algorithms A and ˜ A when comparing datasets D and ˜ D .
On the one hand, such ﬂexibility is required, since for instance, a mere preprocessing of the dataset (e.g. rescaling it), should be regarded as producing an equivalent ( 0 -approximate) dataset.
Yet such a rescaling may affect the hyperparameters needed to train an equivalent model (e.g. the learning rate).
Thus, one must allow the relevant hyperparameters of an algorithm to vary when the datasets are also varying.
On the other hand, it would be impossible to compare two datasets meaningfully if the learned algorithms used to train them differ too signiﬁcantly.
For instance, if D is a much larger dataset than ˜ D , but A is a much less expressive algorithm than ˜ A , then the two datasets may be (cid:15) -approximations of each other, but it would be strange to compare D and ˜ D in this way.
Thus, we treat the notion of what class of algorithms to consider informally, and leave its speciﬁcation as a practical matter for each use case.
In practice, the pair of algorithms we use to compare datasets should be drawn from a family in which some reasonable range of hyperparameters are varied, the ones typically tuned when learning on an unknown dataset.
The main case for us with differing A and ˜ A is when we compare neural network training alongside kernel ridge-regression.
Another key feature of our deﬁnition is that datapoints of an (cid:15) -approximating dataset must have the same shape as those of the original dataset.
This makes our notion of an (cid:15) -approximate dataset more restrictive than returning a specialized set of extracted features from some initial dataset.
Analogues of our (cid:15) -approximation deﬁnition have been formulated in the unsupervised setting, e.g. in the setting of clustering data (Phillips, 2016; Jubran et al., 2019).
Finally, note that the loss function (cid:96) used for comparing datasets does not have to coincide with any loss functions optimized in the learning algorithms A and ˜ A .
Indeed, for kernel ridge-regression, training mimimizes mean square loss while (cid:96) can be 0 - 1 loss.
B T UNING KIP
Sampling: When optimizing for KRR performance with support dataset size N , it is best to learn a support set ˜ D of size N and sample this entire set during KIP training.
It is our observation that subsets of size M < N of ˜ D will not perform as well as optimizing directly for a size M dataset through KIP .
Conversely, sampling subsets of size M from a support dataset of size N during
KIP will not lead to a dataset that does as well as optimizing for all N points.
This is sensible: optimizing for small support size requires a resourceful learning of coarse features at the cost of learning ﬁne-grained features from many support datapoints.
Conversely, optimizing a large support set means the learned support set has leveraged higher-order information, which will degrade when restricted to smaller subsets.
For sampling from the target set, which we always do in a class-balanced way, we found larger batch sizes typically perform better on the test set if the train and test kernels agree.
If the train and test kernels differ, then smaller batch sizes lead to less overﬁtting to the train kernel.
Initialization: We tried two sets of initializations.
The ﬁrst (“image init”) initializes ( X s , y s ) to be a subset of ( X t , y t ) .
The second (“noise init”) initializes X s with uniform noise and y s with mean-centered, one-hot labels (in a class-balanced way).
We found image initialization to perform better.
Regularization:
The regularization parameter λ in (7) can be replaced with 1 n λ · tr( K X s X s ) , where n is the number of datapoints in X s .
This makes the loss function invariant with respect to rescaling of the kernel function K and also normalizes the regularization with respect to support size.
In practice, we use this scale-invariant regularization with λ = 10 − 6 .
Number of Training Iterations: Remarkably, KIP converges very quickly in all experimental settings we tried.
After only on the order of a hundred iterations, independently of the support size, kernel, and corruption factor, the learned support set has already undergone the majority of its learning (test accuracy is within more than 90% of the ﬁnal test accuracy).
For the platforms available to us, using a single V100 GPU, one hundred training steps for the experiments we ran involving target batch sizes that were a few thousand takes on the order of about 10 minutes.
When we add augmentations to our targets, performance continues to improve slowly over time before ﬂattening out after several thousands of iterations.
C T HEORETICAL R ESULTS
Here, we analyze convergence properties of KIP in returning an (cid:15) -approximate dataset.
In what follows, we refer to gradient-descent KIP as the case when we sample from the entire support and train datasets for each update step to KIP .
We also assume that the distribution P used to evaluate (cid:15) -approximation is supported on inputs x ∈ R d with (cid:107) x (cid:107) ≤ 1 (merely to provide a convenient normalization when evaluating loss on regression algorithms).
For the case of a linear kernel, we prove the below convergence theorem:
Theorem 1.
Let D =
( X t ,
y t ) ∈ R n t × d × R n t ×
C be an arbitrary dataset.
Let w λ ∈ R d × C be the coefﬁcients obtained from training λ ridge-regression ( λ -RR) on ( X t , y t ) , as given by (4).
For generic 6 initial conditions for the support set ( X s , y s ) ⊂ R n s × d × R n s × C and sufﬁciently small λ > 0 , gradient descent KIP with target dataset D converges to a dataset ˜ D . 2.
The dataset ˜ D is a strong (cid:15) -approximation to D with respect to algorithms ( λ -RR, 0 -RR) and loss function equal to mean-square loss, where
[Equation]
and ˜ w ∈ R d × C are the coefﬁcients of the linear classiﬁer obtained from training λ -RR on ˜ D .
If the size of ˜ D is at least C , then ˜ w is also a least squares classiﬁer for D .
In particular, if D has a unique least squares classiﬁer, then (cid:15) = 0 .
Proof.
We discuss the case where X s is optimized, with the case where both ( X s , y s ) are optimized proceeding similarly.
In this case, by genericity, we can assume y s (cid:54) = 0 , else the learning dynamics is trivial.
Furthermore, to simplify notation for the time being, assume the dimensionality of the label space is C = 1 without loss of generality.
First, we establish convergence.
For a linear kernel, we can write our loss function as
[Equation]
deﬁned on the space M n s × d of n s × d matrices.
It is the pullback of the loss function
[Equation]
under the map X s (cid:55)→ Φ λ ( X s ) =
X Ts ( X s X Ts + λI ) − 1 .
The function (A3) is quadratic in Φ and all its local minima are global minima given by an afﬁne subspace M ⊂ M d × n s .
Moreover, each point of M has a stable manifold of maximal dimension equal to the codimension of M .
Thus, the functional L has global minima given by the inverse image Φ − 1 λ ( M ) (which will be nonempty for sufﬁciently small λ ).
Next, we claim that given a ﬁxed initial ( X s , y s ) , then for sufﬁciently small λ , gradient-ﬂow of (A2) starting from ( X s , y s ) cannot converge to a non-global local minima.
We proceed as follows.
If
X = U Σ V T is a singular value decomposition of X , with Σ a n s × n s diagional matrix of singular values (and any additional zeros for padding), then Φ( X ) =
V φ (Σ) U T where φ (Σ) denotes the diagonal matrix with the map
[Equation]
applied to each singular value of Σ .
The singular value decomposition depends analytically on X (Kato (1976)).
Given that φ : R ≥
→ R ≥ 0 is a local diffeomorphism away from its maximum value at µ = µ ∗ :=
λ 1 / 2 , it follows that Φ λ : M n s × d → M d × n s is locally surjective, i.e. for every X , there exists a neighborhood U of X such that Φ λ ( U ) contains a neighborhood of Φ λ ( X ) .
Thus, away from the locus of matrices in M n s × d that have a singular value equaling µ ∗ , the function (A2) cannot have any non-global local minima, since the same would have to be true for (A3).
We are left to consider those matrices with some singular values equaling µ ∗ .
Note that as λ → 0 , we have φ ( µ ∗ ) → ∞ .
On the other hand, for any initial choice of X s , the matrices Φ λ ( X s ) have uniformly bounded singular values as a function of λ .
Moreover, as X s = X s ( t ) evolves, (cid:107) Φ λ
( X s ( t )) (cid:107) never needs to be larger than some large constant times (cid:107) Φ λ
( X s (0)) (cid:107) + (cid:107) y t (cid:107) µ + (cid:107) y s (cid:107) , where µ + is the smallest positive singular value of X t .
Consequently, X s ( t ) never visits a matrix with singular value µ ∗ for sufﬁciently small λ > 0 ; in particular, we never have to worry about convergence to a non-global local minimum.
Thus, a generic gradient trajectory γ of L will be such that Φ λ ◦ γ is a gradient-like 7 trajectory for
L R d × ns that converges to M .
We have to show that γ itself converges.
It is convenient to extend φ to a map deﬁned on the one-point compactiﬁcation [0 , ∞ ] ⊃ R ≥ 0 , so as to make φ a two-to-one map away from µ ∗ .
Applying this compactiﬁcation to every singular value, we obtain a compactiﬁcation
M n s × d of M n s × d , and we can naturally extend Φ λ to such a compactiﬁcation.
We have that γ converges to an element of ˜ M :=
Φ − 1 λ ( M ) ⊂
M n s × d , where we need the compactiﬁcation to account for the fact that when Φ λ ◦ γ converges to a matrix that has a zero singular value, γ may have one of its singular values growing to inﬁnity.
Let M 0 denote the subset of M with a zero singular value.
Then γ converges to an element of M n s × d precisely when γ does not converge to an element of ˜ M ∞ :
= Φ − 1 λ ( M 0 ) ∩ ( M n s × d \ M n s × d ) .
However, M 0 ⊂ M has codimension one and hence so does ˜ M ∞ ⊂
Φ − 1 λ ( M 0 ) .
Thus, the stable set to ˜ M ∞ has codimension one in M n s × d , and hence its complement is nongeneric.
Hence, we have generic convergence of a gradient trajectory of L to a (ﬁnite) solution.
This establishes the convergence result of Part 1.
For Part 2, the ﬁrst statement is a general one: the difference of any two linear models, when evaluated on P , can be pointwise bounded by the spectral norm of the difference of the model coefﬁcient matrices.
Thus ˜ D is a strong (cid:15) -approximation to D with respect to ( λ -RR, 0 -RR) where (cid:15) is given by (A1).
For the second statement, observe that L is also the pullback of the loss function
[Equation]
under the map X s (cid:55)→ w ( X s ) =
Φ λ ( X s )
y s .
The function L R d × C ( w ) is quadratic in w and has a unique minimum value, with the space of global minima being an afﬁne subspace W ∗ of R d given by the least squares classiﬁers for the dataset ( X t , y t ) .
Thus, the global minima of L are the preimage of W ∗ under the map w ( X s ) .
For generic initial ( X s , y s ) , we have y s ∈ R n s × C is full rank.
This implies, for n s ≥ C , that the range of all possible w ( X s ) for varying X s is all of R d × C , so that the minima of (A6) and (A3) coincide.
This implies the ﬁnal parts of Part 2.
We also have the following result about (cid:15) -approximation using the label solve algorithm:
Theorem 2.
Let D =
( X t ,
y t ) ∈ R n t × d × R n t ×
C be an arbitrary dataset.
Let w λ ∈ R d × C be the coefﬁcients obtained from training λ ridge-regression ( λ -RR) on ( X t , y t ) , as given by (4).
Let
X s ∈ R n s × d be an arbitrary initial support set and let λ ≥ 0 .
Deﬁne y ∗ s =
y ∗ s ( λ ) via (8).
Then ( X s , y ∗ s ) yields a strong (cid:15) ( λ ) -approximation of ( X t , y t ) with respect to algorithms ( λ -RR, 0 -RR) and mean-square loss, where
[Equation]
and w ∗ ( λ ) is the solution to
[Equation]
Moreover, for λ = 0 , if rank( X s ) = rank( X t ) =
d , then w ∗ ( λ ) = w 0 .
This implies y ∗ s =
X s w 0 , i.e. y ∗ s coincides with the predictions of the 0 -RR classiﬁer trained on ( X t , y t ) evaluated on X s .
Proof.
By deﬁnition, y ∗ s is the minimizer of
[Equation]
with minimum norm.
This implies y ∗ s ∈ ker (cid:16) X t
Φ λ ( X s ) (cid:17) ⊥ and that w ∗ ( λ ) =
Φ λ ( X s )
y ∗ s satisﬁes (A8).
At the same time, w ∗ ( λ ) =
Φ λ ( X s )
y ∗ s are the coefﬁcients of the λ -RR classiﬁer trained on ( X s , y ∗ s ) .
If rank( X s ) = rank( X t ) =
d , then Φ 0 ( X s ) is surjective and X t is injective, in which case
[Equation]
The results follow.
For general kernels, we make the following simple observation concerning the optimal output of
KIP .
Theorem 3.
Fix a target dataset ( X t , y t ) .
Consider the family of all subspaces S of R n t given by { im K X t X s : X s ∈ R n s × d } , i.e. all possible column spaces of K X t X s .
Then the inﬁmum of the loss (7) over all possible ( X s , y s ) is equal to inf S ∈S 12 (cid:107) Π ⊥ S y t (cid:107) 2 where Π ⊥ S is orthogonal projection onto the orthogonal complement of S (acting identically on each label component).
Proof.
Since y s is trainable, ( K X s X s + λ ) − 1 y s is an arbitrary vector in R n s × C .
Thus, minimizing the training objective corresponds to maximizing the range of the linear map
K X t
X s over all possible
X s .
The result follows.
D E XPERIMENT D ETAILS
In all KIP trainings, we used the Adam optimizer.
All our labels are mean-centered 1 -hot labels.
We used learning rates 0 .
01 and 0 .
04 for the MNIST and CIFAR-10 datasets, respectively.
When sampling target batches, we always do so in a class-balanced way.
When augmenting data, we used the ImageGenerator class from Keras, which enables us to add horizontal ﬂips, height/width shift, rotatations (up to 10 degrees), and channel shift (for CIFAR-10).
All datasets are preprocessed using channel-wise standardization (i.e. mean subtraction and division by standard-deviation).
For neural (tangent) kernels, we always use weight and bias variance σ 2 w = 2 and σ 2 b = 10 − 4 , respectively.
For both neural kernels and neural networks, we always use ReLU activation.
Convolutional layers all use a (3 , 3) ﬁlter with stride 1 and same padding.
Compute Limitations: Our neural kernel computations, implemented using Neural Tangents libraray (Novak et al., 2020) are such that computation scales (i) linearly with depth; (ii) quadratically in the number of pixels for convolutional kernels; (iii) quartically in the number of pixels for pooling layers.
Such costs mean that, using a single V100 GPU with 16GB of RAM, we were (i) only able to sample shallow kernels; (ii) for convolutional kernels, limited to small support sets and small target batch sizes; (iii) unable to use pooling if learning more than just a few images.
Scaling up KIP to deeper, more expensive architectures, achievable using multi-device training, will be the subject of future exploration.
Kernel Parameterization: Neural tangent kernels, or more precisely each neural network layer of such kernels, as implemented in Novak et al.
(2020) can be parameterized in either the “NTK” parameterization or “standard” parameterization Sohl-Dickstein et al.
The latter depends on the width of a corresponding ﬁnite-width neural network while the former does not.
Our experiments mix both these parameterizations for variety.
However, because we use a scale-invariant regularization for KRR (Section B), the choice of parameterization has a limited effect compared to other more signiﬁcant hyperparameters (e.g. the support dataset size, learning rate, etc.) 8 .
Single kernel results: (Tables 1 and 2) For FC, we used kernels with NTK parametrization.
For RBF, our rbf kernel is given by
[Equation]
where d is the dimension of the inputs and γ = 1 .
We found that treating γ as a learnable parameter during KIP had mixed results 9 and so keep it ﬁxed for simplicity.
For MNIST, we found target batch size equal to 6K sufﬁcient.
For CIFAR-10, it helped to sample the entire training dataset of 50K images per step (hence, along with sampling the full support set, we are doing full gradient descent training).
When support dataset size is small or if augmentations are employed, there is no overﬁtting (i.e. the train and test loss/accuracy stay positively correlated).
If the support dataset size is large (5K or larger), sometimes there is overﬁtting when the target batch size is too large (e.g. for the RBF kernel on CIFAR10, which is why we exclude in Table 2 the entries for 5K and 10K).
We could have used a validation dataset for a stopping criterion, but that would have required reducing the target dataset from the entire training dataset.
We train KIP for 10-20k iterations and took 5 random subsets of images for initializations.
For each such training, we took 5 checkpoints with lowest train and loss and computed the test accuracy.
This gives 25 evaluations, for which we can compute the mean and standard deviation for our test accuracy numbers in Tables 1 and 2.
Kernel transfer results: For transfering of KIP images, both to other kernels and to neural networks, we found it useful to use smaller target batch sizes (either several hundred or several thousand), else the images overﬁt to their source kernel.
For random sampling of kernels used in Figure A1 and producing datasets for training of neural networks, we used FC kernels with width 1024 and Conv kernels with width 128, all with standard parametrization.
Neural network results: Neural network trainings on natural data with mean-square loss use meancentered one-hot labels for consistency with KIP trainings.
For cross entropy loss, we use one-hot labels.
For neural network trainings on KIP -learned images with label learning, we transfer over the labels directly (as with the images), whatever they may be.
For neural network transfer experiments occurring in Table 1, Table 2, Figure 3, Table A5, and Table A6, we did the following.
First, the images were learned using kernels FC1-3, Conv1-2.
Second, we trained for a few hundred iterations, after which optimal test performance was achieved.
On MNIST images, we trained the networks with constant learning rate and Adam optimizer with cross entropy loss.
Learning rate was tuned over small grid search space.
For the FC kernels and networks, we use width of 1024.
On CIFAR-10 images, we trained the networks with constant learning rate, momentum optimizer with momentum 0.9.
Learning rate, L2 regularization, parameterization (standard vs NTK) and loss type (mean square, softmax-cross-entropy) was tuned over small grid search space.
Vanilla networks use constant width at each layer: for FC we use width of 1024, for Conv2 we use 512 channels, and for Conv8 we use 128 channels.
No pooling layers are used except for the WideResNet architecture, where we follow the original architecture of Zagoruyko & Komodakis (2016) except that our batch normalization layer is stateless (i.e. no exponential moving average of batch statistics are recorded).
For neural network transfer experiments in Figure A3, Tables A7-A10, we did the following.
Our KIP -learned images were trained using only an FC1 kernel.
The neural network FC1 has an increased width 4096, which helps with the larger number of images.
We used learning rate 4 × 10 −and the Adam optimizer.
The KIP learned images with only augmentations used target batch size equal to half the training dataset size and were trained for 10k iterations, since the use of augmentations allows for continued gains after longer training. The KIP learned images with augmentations
[Table]
and label learning used target batch size equal to a tenth of the training dataset size and were trained for 2k iterations (the learned data were observed to overﬁt to the kernel and have less transferability if larger batch size were used or if trainings were carried out longer).
All neural network trainings were run with 5 random initializations to compute mean and standard deviation of test accuracies.
In Table 2, regularized ZCA preprocessing was used for a Myrtle-10 kernel (denoted with ZCA ) on CIFAR-10 dataset. Shankar et al. (2020) and Lee et al.
(2020) noticed that for neural (convolutional) kernels on image classiﬁcation tasks, regularized ZCA preprocessing can improve performance signiﬁcantly compared to standard preprocessing.
We follow the prepossessing scheme used in Shankar et al.
(2020), with regularization strength of 10 − 5 without augmentation.
E T ABLES AND F IGURES E. K ERNEL B ASELINES
We report various baselines of KRR trained on natural images.
Tables A1 and A2 shows how various kernels vary in performance with respect to random subsets of MNIST and CIFAR-10.
Linear denotes a linear kernel, RBF denotes the rbf kernel (A9) with γ = 1 , and FC1 uses standard parametrization and width 1024.
Interestingly enough, we observe non-monotonicity for the linear kernel, owing to double descent phenomenon Hastie et al.
We include additional columns for deeper kernel architectures in Table A2, taken from Shankar et al.
(2020) for reference.
Comparing Tables 1, 2 with Tables A1, A2, we see that 10 KIP -learned images, for both RBF and
FC1, has comparable performance to several thousand natural images, thereby achieving a compression ratio of over 100.
This compression ratio narrows as the support size increases towards the size of the training data.
Next, Table A3 compares FC1, RBF, and other kernels trained on all of MNIST to FC1 and RBF trained on KIP -learned images.
We see that our KIP approach, even with 10K images (which ﬁts into memory), leads to RBF and FC1 matching the performance of convolutional kernels on the original 60K images.
Table A4 shows state of the art of FC kernels on CIFAR-10.
The prior state of the art used kernel ensembling on batches of augmented data in Lee et al.
(2020) to obtain test accuracy of 61.5% (32 ensembles each of size 45K images).
By distilling augmented images using
KIP , we are able to obtain 64.7% test accuracy using only 10K images.
E. KIP AND LS T RANSFER A CROSS K ERNELS
Figure A1 plots how KIP (with only images learned) performs across kernels.
There are seven training scenarios: training individually on FC1, FC2, FC3, Conv1, Conv [Table] [Table]
[Table] are thereby trained then evaluated by averaging over all of FC1-3, Conv1- [Table] both with the NTK and
NNGP kernels for good measure.
Moreover, the FC and Conv train kernel widths (1024 and 128) were swapped at test time (FC width 128 and Conv width 1024) [Table] as an additional test of robustness.
The average performance is recorded along the y-axis.
AvgAll leads to overall boost in performance across kernels.
Another observation is that Conv kernels alone tend to do a bit better, averaged over the kernels considered, than FC kernels alone
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
In all these plots, NNGP kernels were used and
Myrtle-10 used regularized ZCA preprocessing.
In Figure A [Table] we plot how LS learned labels using Myrtle-10 kernel transfer to the FC1 kernel and vice versa.
We vary the number of targets and support size.
We ﬁnd remarkable stability across all these dimensions in the sense that while the gains from LS may be kernel-speciﬁc, LS -labels do not perform meaningfully different from natural labels when switching the train and evaluation kernels.
E. KIP TRANSFER TO N EURAL N ETWORKS AND C ORRUPTION E XPERIMENTS
[Table]
[Table] [Table] [Table] [Table]
[Table] 35.4 ± 0. 31.7 ± 0. 27.2 ± 0.
KIP (a)49.
[Table] [Table]
47.4 ± 0. 45.0 ± 0.3*
Natural43.1 ± 0.
[Table]
38.0 ± 0. 31.7 ± 0.
KIP (a)44.5 ± 1.
51.5 ± 0. 51.0 ± 0.
[Table]
Natural45.3 ± 0.
44.8 ± 0.
40.6 ± 0.
33.8 ± 0.
KIP (a)46.
[Table]
[Table]
[Table]
[Table]
[Figure]
natural images % corruption (acc: 33.
[Figure]