D ATASET C ONDENSATION WITH G RADIENT M ATCHING
A BSTRACT
As the state-of-the-art machine learning methods in many ﬁelds rely on larger datasets, storing them and training models on them becomes more expensive.
This paper proposes a training set synthesis technique for data efﬁcient learning called
Dataset Condensation , that learns to condense a large dataset into a small set of informative samples for training deep neural networks from scratch
We formulate this goal as a gradient matching problem between the gradients of a deep neural network trained on the original data and our synthetic data
We rigorously evaluate its performance in several computer vision benchmarks and demonstrate that it signiﬁcantly outperforms the state-of-the-art methods.
Finally we explore the use of our method in continual learning and neural architecture search and show that it achieves promising gains on a tight budget of memory and computations
I NTRODUCTION
Large-scale datasets, comprising millions of samples, are becoming the norm to obtain state-ofthe-art machine learning models in multiple ﬁelds including computer vision, natural language processing and speech recognition.
At such scales, even storing and preprocessing the data becomes burdensome, and training machine learning models on them demands for specialized equipment and infrastructure.
An effective way to deal with large data is data selection that improves data efﬁciency of machine learning by identifying the most representative training samples.
While classical data selection methods, also known as coreset construction (Agarwal et al., 2004; Har-Peled & Mazumdar, 2004; Feldman et al., 2013), focus on clustering problems, recent work can be found in continual learning (Rebufﬁ et al., 2017;
Toneva et al., 2019; Castro et al., 2018; Aljundi et al., 2019) and active learning (Sener & Savarese, 2018) where there is typically a ﬁxed budget in storing and labeling training samples respectively.
These methods commonly ﬁrst deﬁne a criterion for representativeness ( e.g . in terms of compactness (Rebufﬁ et al., 2017; Castro et al., 2018), diversity (Sener & Savarese, 2018;
Aljundi et al., 2019), forgetfulness (Toneva et al., 2019)), and select the representative samples by solving an optimization problem based on the criterion, ﬁnally use the selected small set to train their model for a downstream task.
Unfortunately, these methods have two shortcomings: they typically rely on i) heuristics ( e.g . picking cluster centers) that does not guarantee any optimal solution for the downstream task ( e.g . image classiﬁcation), ii) presence of representative samples, which is neither guaranteed.
A recent method,
Dataset Distillation (DD) (Wang et al., 2018) goes beyond these limitations by learning a small set of informative images from large training data.
In particular, the authors model the network parameters as a function of the synthetic training data and learn them by minimizing the training loss over the original training data w.r.t.
synthetic data.
Unlike in the coreset methods, the synthesized data are directly optimized for the downstream task and thus the success of the method does not rely on the presence of representative samples.
Inspired from DD, we focus on learning to synthesize new samples that are optimized to train neural networks for downstream tasks and not limited to individual samples in original dataset.
Like
DD (Wang et al., 2018), our goal is to obtain the highest generalization performance with a model trained on a small set of synthetic images.
Ideally, it can obtain comparable performance to that of a model trained on the original images (see Figure 1(a)).
In particular, we investigate the following questions.
Is it possible to i) compress a large image classiﬁcation dataset into a small synthetic set, ii) train an image classiﬁcation model on the synthetic set that can be further used to classify real images, iii) learn a single set of synthetic images that can be used to train different neural net-
[Figure]
work architectures? To this end, we propose a Dataset Condensation method to learn a small set of “condensed” synthetic samples such that a deep network trained on them obtains not only similar performance but also a close solution to a network trained on the large training data in the network parameter space.
We formulate this goal as a minimization problem between two sets of gradients for the network parameters that are computed for a training loss over a large ﬁxed training set and a learnable condensed set (see Figure 1(b)).
We show that our method enables a more effective learning of synthetic images and neural networks trained on them outperforms (Wang et al., 2018) and coreset methods with a wide margin in multiple computer vision benchmarks.
In addition, learning a compact set of synthetic samples also beneﬁts other learning problems when there is a ﬁxed budget on training images.
We show that our method outperforms popular data selection methods by providing more informative training samples in continual learning.
Finally, we explore a promising use case of our method in neural architecture search, and show that – once our condensed images are learned – they can be used to train numerous network architectures extremely efﬁciently.
Our method is related to knowledge distillation (KD)
(Hinton et al., 2015) techniques (Bucilu ˇ a et al., 2006; Ba & Caruana, 2014; Romero et al., 2014) that transfer the knowledge of an ensemble of models to a single one.
Unlike KD, we distill knowledge of a large training set into a small synthetic set.
Our method is also related to Generative Adversarial Networks (Goodfellow et al., 2014a; Mirza & Osindero, 2014; Radford et al., 2015) and Variational AutoEncoders (Kingma & Welling, 2013) that synthesize high ﬁdelity samples by capturing the data distribution.
In contrast, our goal is to generate informative samples for training deep neural networks rather than to produce “real-looking” samples.
Finally our method is related to the methods that recover images by projecting the feature activations back to the input pixel space (Zeiler & Fergus, 2014), reconstruct the input image by matching the feature activations (Mahendran & Vedaldi, 2015), recover private training images for given training gradients (Zhu et al., 2019; Zhao et al., 2020), synthesize features from semantic embeddings for zero-shot learning (Sariyildiz & Cinbis, 2019).
Our goal is however to synthesize a set of condensed training images not to recover the original or missing training images.
In the remainder of this paper, we ﬁrst review the problem of dataset condensation and introduce our method in section 2, present and analyze our results in several image recognition benchmarks in section 3.1, showcase applications in continual learning and network architecture search in section 3.2, and conclude the paper with future directions in section 4.
M ETHOD 2. D ATASET CONDENSATION
Suppose we are given a large dataset consisting of m pairs of a training image and its class label
T = { ( x i , y i ) }| mi =1 where x ∈ X ⊂ R d , y ∈ { 0 , . . .
, C − 1 } , X is a d-dimensional input space and
C is the number of classes.
We wish to learn a differentiable function φ ( i.e . deep neural network) with parameters θ that correctly predicts labels of previously unseen images, i.e .
y
=
φ θ ( x ) .
One can learn the parameters of this function by minimizing an empirical loss term over the training set:
[Equation]
where L T ( θ ) = (cid:80) ( x ,y ) ∈T (cid:96) ( φ θ ( x ) , y ) , (cid:96) ( · , · ) is a task speciﬁc loss ( i.e . cross-entropy) and θ T is the minimizer of L T .
The generalization performance of the obtained model φ θ T can be written as E x ∼ P D
[ (cid:96) ( φ θ T ( x ) , y )] where P D is the data distribution.
Our goal is to generate a small set of condensed synthetic samples with their labels, S =
{ ( s i , y i ) }| ni =1 where s ∈ R d and y ∈ Y , n (cid:28) m .
Similar to eq.
(1), once the condensed set is learned, one can train φ on them as follows
[Equation]
where L S ( θ ) = (cid:80) ( s ,y ) ∈S (cid:96) ( φ θ ( s ) , y ) and θ S is the minimizer of L S .
As the synthetic set S is signiﬁcantly smaller (2-3 orders of magnitude), we expect the optimization in eq.
(2) to be signiﬁcantly faster than that in eq.
We also wish the generalization performance of φ θ S as close as possible to φ θ T , i.e . E x ∼ P D [ (cid:96) ( φ θ T ( x ) , y )] (cid:39) E x ∼ P D
[ (cid:96) ( φ θ S ( x ) , y )] over the real data distribution P D .
Discussion.
The goal of obtaining comparable generalization performance by training on the condensed data can be formulated in different ways.
One approach, which is proposed in (Wang et al., 2018) and extended in (Sucholutsky & Schonlau, 2019; Bohdal et al., 2020; Such et al., 2020), is to pose the parameters θ S as a function of the synthetic data S :
[Equation]
The method aims to ﬁnd the optimum set of synthetic images S ∗ such that the model φ θ S trained on them minimizes the training loss over the original data.
Optimizing eq.
(3) involves a nested loop optimization and solving the inner loop for θ S ( S ) at each iteration to recover the gradients for S which requires a computationally expensive procedure – unrolling the recursive computation graph for S over multiple optimization steps for θ (see (Samuel & Tappen, 2009; Domke, 2012)).
Hence, it does not scale to large models and/or accurate inner-loop optimizers with many steps.
Next we propose an alternative formulation for dataset condensation.
2. D ATASET CONDENSATION WITH PARAMETER MATCHING
Here we aim to learn S such that the model φ θ S trained on them achieves not only comparable generalization performance to φ θ T but also converges to a similar solution in the parameter space ( i.e . θ S ≈ θ T ).
Let φ θ be a locally smooth function 1 , similar weights ( θ S ≈ θ T ) imply similar mappings in a local neighborhood and thus generalization performance, i.e .
E x ∼ P D
[ (cid:96) ( φ θ T ( x ) , y )] (cid:39) E x ∼ P D [ (cid:96) ( φ θ S ( x ) , y )] .
Now we can formulate this goal as
[Equation]
where θ T = arg min θ L T ( θ ) and D ( · , · ) is a distance function.
In a deep neural network, θ T typically depends on its initial values θ 0 .
However, the optimization in eq.
(4) aims to obtain an optimum set of synthetic images only for one model φ θ T with the initialization θ 0 , while our actual goal is to generate samples that can work with a distribution of random initializations P θ 0 .
Thus we modify eq.
(4) as follows:
[Equation]
where θ T = arg min θ L T ( θ ( θ 0 ))
For brevity, we use only θ S and θ T to indicate θ S ( θ 0 ) and θ T ( θ 0 ) respectively in the next sections.
The standard approach to solving eq.
(5) employs implicit differentiation (see (Domke, 2012) for details), which involves solving an inner loop optimization for θ S .
As the inner loop optimization θ S ( S ) =
arg min θ L S ( θ ) can be computationally expensive in case of large-scale models, one can adopt the back-optimization approach in (Domke, 2012) which re-deﬁnes θ S as the output of an incomplete optimization:
[Equation]
where opt-alg is a speciﬁc optimization procedure with a ﬁxed number of steps ( ς ).
In practice, θ T for different initializations can be trained ﬁrst in an ofﬂine stage and then used as the target parameter vector in eq.
However, there are two potential issues by learning to regress θ T as the target vector.
First the distance between θ T and intermediate values of θ S can be too big in the parameter space with multiple local minima traps along the path and thus it can be too challenging to reach.
Second opt-alg involves a limited number of optimization steps as a tradeoff between speed and accuracy which may not be sufﬁcient to take enough steps for reaching the optimal solution.
These problems are similar to those of (Wang et al., 2018), as they both involve parameterizing θ S with S and θ 0 .
2. D ATASET CONDENSATION WITH CURRICULUM GRADIENT MATCHING
Here we propose a curriculum based approach to address the above mentioned challenges.
The key idea is that we wish θ S to be close to not only the ﬁnal θ T but also to follow a similar path to θ T throughout the optimization.
While this can restrict the optimization dynamics for θ , we argue that it also enables a more guided optimization and effective use of the incomplete optimizer.
We can now rewrite eq.
(5) as a sum of subproblems:
[Equation]
to θ
[Equation]
) = opt-alg
[Equation]
( L S
[Equation]
S
[Equation]
[Equation] S ) and θ T t = opt-alg [Equation] (7) where T is the number of iterations, ς S and ς T are the numbers of optimization steps for θ S and θ T respectively.
In words, we wish to generate a set of condensed samples S such that the network parameters trained on them ( θ S t ) are similar to the ones trained on the original training set ( θ T t ) at each iteration t .
In our preliminary experiments we observe that θ S t , which is parameterized with S , can successfully track θ T t by updating S and minimize D ( θ S t − 1 , θ T t − 1 ) close to zero.
In the case of one step gradient descent optimization for opt-alg , the update rule is: θ S t ← θ [Equation] and θ T t ← θ T t
[Equation]
where η θ is the learning rate.
Based on our observation ( D ( θ S t − 1 , θ T t − 1 ) ≈ 0 ), we simplify the formulation in eq.
(7) by replacing θ T t − 1 with θ S t − 1 and use θ to denote θ S in the rest of the paper:
[Equation]
We now have a single deep network with parameters θ trained on the synthetic set S which is optimized such that the distance between the gradients for the loss over the training samples L T w.r.t.
θ and the gradients for the loss over the condensed samples L S w.r.t.
θ is minimized.
In words, our goal reduces to matching the gradients for the real and synthetic training loss w.r.t.
θ via updating the condensed samples.
This approximation has the key advantage over (Wang et al., 2018) and eq.
(5) that it does not require the expensive unrolling of the recursive computation graph over the previous parameters { θ 0 , . .
. , θ t − 1 } .
The important consequence is that the optimization is signiﬁcantly faster, memory efﬁcient and thus scales up to the state-of-the-art deep neural networks ( e.g . ResNet (He et al., 2016)).
Discussion.
The synthetic data contains not only samples but also their labels ( s , y ) that can be jointly learned by optimizing eq.
(9) in theory.
However, their joint optimization is challenging, as the content of the samples depend on their label and vice-versa.
Thus in our experiments we learn to synthesize images for ﬁxed labels, e.g . one synthetic image per class.
Algorithm.
We depict the optimization details in Alg. 1.
At the outer level, it contains a loop over random weight initializations, as we want to obtain condensed images that can later be used to train previously unseen models.
Once θ , which denotes θ S for brevity, is randomly initialized, we use φ θ to ﬁrst compute the loss over both the training samples ( L T ), synthetic samples ( L S ) and their gradients w.r.t.
θ , then optimize the synthetic samples S to match these gradients ∇ θ L S to ∇ θ L T by applying ς S gradient descent steps with learning rate η S .
We use the stochastic gradient descent optimization for both opt-alg θ and opt-alg S .
Next we train θ on the updated synthetic images by minimizing the loss L S with learning rate η θ for ς θ steps.
Note that we sample each real and synthetic batch pair from T and S containing samples from a single class and the synthetic data for each class are separately (or parallelly) updated at each iteration ( t ) for the following reasons: i) this reduces memory use at train time, ii) imitating the mean gradients w.r.t.
the data from single class is easier compared to those of multiple classes.
This does not bring any extra computational cost.
[Table]
Gradient matching loss.
The matching loss D ( · , · ) in eq.
(9) measures the distance between the gradients for L S and L T w.r.t.
θ .
When φ θ is a multi-layered neural network, the gradients correspond to a set of learnable 2D ( out × in ) and 4D ( out × in × h × w ) weights for each fully connected (FC) and convolutional layer resp where out , in , h , w are number of output and input channels, kernel height and width resp.
The matching loss can be decomposed into a sum of layerwise losses as D ( ∇ θ L S , ∇ θ L T ) =
(cid:80) Ll =1 d ( ∇ θ ( l ) L S , ∇ θ ( l ) L T ) where l is the layer index, L is the number of layers with weights and
[Equation]
where A i · and B i · are ﬂattened vectors of gradients corresponding to each output node i.e . in dimensional for FC weights and in × h × w dimensional for convolutional weights.
In contrast to (Lopez-Paz et al., 2017; Aljundi et al., 2019; Zhu et al., 2019) that ignore the layer-wise structure by ﬂattening tensors over all layers to one vector and then computing the distance between two vectors, we group them for each output node.
We found that this is a better distance for gradient matching (justiﬁed in the supplementary) and enables using a single learning rate across all layers.
E XPERIMENTS 3. D ATASET CONDENSATION
First we evaluate classiﬁcation performance with the condensed images on four standard benchmark datasets: digit recognition on MNIST (LeCun et al., 1998), SVHN (Netzer et al., 2011) and object classiﬁcation on FashionMNIST (Xiao et al., 2017), CIFAR10 (Krizhevsky et al., 2009).
We test our method using six standard deep network architectures: MLP, ConvNet (Gidaris & Komodakis, 2018), LeNet (LeCun et al., 1998), AlexNet (Krizhevsky et al., 2012), VGG-11 (Simonyan & Zisserman, 2014) and ResNet-18 (He et al., 2016).
MLP is a multilayer perceptron with two nonlinear hidden layers, each has 128 units.
ConvNet is a commonly used modular architecture in few-shot learning (Snell et al., 2017; Vinyals et al., 2016; Gidaris & Komodakis, 2018) with D duplicate blocks, and each block has a convolutional layer with W ( 3 × 3 ) ﬁlters, a normalization layer N , an activation layer A and a pooling layer P , denoted as [ W, N, A, P ] × D .
The default ConvNet (unless speciﬁed otherwise) includes 3 blocks, each with 128 ﬁlters, followed by InstanceNorm (Ulyanov et al., 2016), ReLU and AvgPooling modules.
The ﬁnal block is followed by a linear classiﬁer.
We use Kaiming initialization (He et al., 2015) for network weights.
More details about the datasets and networks can be found in the supplementary.
The code will be released after review period.
[Table]
[Table]
[Figure] [Figure] [Table] The pipeline for dataset condensation has two stages: learning the condensed images (denoted as
C ) and training classiﬁers from scratch on them (denoted as T ).
Note that the model architectures used in two stages might be different.
For the coreset baselines, the coreset is selected in the ﬁrst stage.
We investigate two settings: 1 and 10 image/class learning, which means that the condensed set or coreset contains 1 and 10 images per class respectively.
Each method is run for 5 times, and 5 synthetic sets are generated in the ﬁrst stage; each generated synthetic set is used to train 20 randomly initialized models in the second stage and evaluated on the test set, which amounts to evaluating 100 models in the second stage.
In all experiments, we report the mean and standard deviation of these 100 testing results.
Baselines.
We compare our method to four coreset baselines (Random, Herding, K-Center and
Forgetting) and also to DD (Wang et al., 2018).
In Random, the training samples are randomly selected as the coreset.
Herding baseline, which selects closest samples to the cluster center, is based on (Welling, 2009) and used in (Rebufﬁ et al., 2017; Castro et al., 2018; Wu et al., 2019; Belouadah & Popescu, 2020).
K-Center (Wolf, 2011; Sener & Savarese, 2018) picks multiple center points such that the largest distance between a data point and its nearest center is minimized.
For Herding and K-Center, we use models trained on the whole dataset to extract features, compute l 2 distance to centers.
Forgetting method (Toneva et al., 2019) selects the training samples which are easy to forget during training.
We do not compare to GSS-Greedy (Aljundi et al., 2019), because it is also a similarity based greedy algorithm like K-Center, but GSS-Greedy trains an online learning model to measure the similarity of samples, which is different from general image classiﬁcation problem.
More detailed comparisons can be found in the supplementary.
Comparison to coreset methods.
We ﬁrst compare our method to the coreset baselines on
MNIST, FashionMNIST, SVHN and CIFAR10 in Table 1 using the default ConvNet in classiﬁcation accuracy.
Whole dataset indicates training on the whole original set which serves as an approximate upper-bound performance.
First we observe that our method outperforms all the baselines signiﬁcantly and achieves a comparable result ( 97 . 4% ) in case of 10 images per class to the upper bound ( 99 . 6% ) in MNIST which uses two orders of magnitude more training images per class ( 6000 ).
We also obtain promising results in FashionMNIST, however, the gap between our method and upper bound is bigger in SVHN and CIFAR10 which contain more diverse images with varying foregrounds and backgrounds.
We also observe that, (i) the random selection baseline is competitive to other coreset methods in case of 10 images per class and (ii) herding method is on average the best coreset technique.
We visualize the synthetic images produced by our method under 1 image/class setting in Figure 2.
Interestingly they are interpretable and look like “prototypes” of each class.
Comparison to DD (Wang et al., 2018).
Unlike the setting in Table 1, DD (Wang et al., 2018) reports results only for 10 images per class on MNIST and CIFAR10 over LeNet and AlexCifarNet [Table] [Table] (a customized AlexNet).
We strictly follow the experimental setting in (Wang et al., 2018), use the same architectures and report our and their original results in Table 3 for a fair comparison.
Our method achieves signiﬁcantly better performance than DD on both benchmarks; obtains 5% higher accuracy with only 1 synthetic sample per class than DD with 10 samples per class.
In addition, our method obtains consistent results over multiple runs with a standard deviation of only 0 .
6% on MNIST, while DD’s performance signiﬁcantly vary over different runs ( 8 . 1% ).
Finally our method trains 2 times faster than DD and requires 50% less memory on CIFAR10 experiments.
More detailed runtime and qualitative comparison can be found in the supplementary.
Cross-architecture generalization.
Another key advantage of our method is that the condensed images learned using one architecture can be used to train another unseen one.
Here we learn to generate 1 condensed image per class for MNIST over a diverse set of networks including MLP,
ConvNet (Gidaris & Komodakis, 2018), LeNet (LeCun et al., 1998), AlexNet (Krizhevsky et al., 2012), VGG-11 (Simonyan & Zisserman, 2014) and ResNet-18 (He et al., 2016)
(see Table 2).
Once the condensed sets are synthesized, we train every network on all the sets separately from scratch and evaluate their cross architecture performance in terms of classiﬁcation accuracy on the
MNIST test set.
Table 2 shows that the condensed images, especially the ones that are trained with convolutional networks, perform well and are thus architecture generic.
MLP generated images do not work well for training convolutional architectures which is possibly due to the mismatch between translation invariance properties of MLP and convolutional networks.
Interestingly, MLP achieves better performance with convolutional network generated images than the MLP generated ones.
The best results are obtained in most cases with ResNet generated images and ConvNet or
ResNet as classiﬁers which is inline with their performances when trained on the original dataset.
Number of condensed images.
We also study the relation between the number of condensed images per class and test performance of a ConvNet trained on them for MNIST, FashionMNIST,
SVHN and CIFAR10 in Figure 3 in absolute and relative terms – normalized by its upper-bound.
While increasing the number of condensed images improves the accuracies in all benchmarks and further closes the gap with the upper-bound performance especially in MNIST and FashionMNIST.
Our method outperforms the compared Herding method by a large margin in all cases.
Activation, normalization & pooling.
We also study the effect of various activation (sigmoid,
ReLU (Nair & Hinton, 2010; Zeiler et al., 2013), leaky ReLU (Maas et al., 2013)), pooling (max, average) and normalization functions (batch (Ioffe & Szegedy, 2015), group (Wu & He, 2018), layer (Ba et al., 2016), instance norm (Ulyanov et al., 2016)) and have the following observations:
i) leaky ReLU over ReLU and average pooling over max pooling enable learning better condensed images, as they allow for denser gradient ﬂow; ii) instance normalization obtains better classiﬁcation performance than its alternatives when used in the networks that are trained on a small set of condensed images.
We refer to the supplementary for detailed results and discussion.
3. A PPLICATIONS
Continual Learning First we apply our method to a continual-learning scenario (Rebufﬁ et al., 2017;
Castro et al., 2018) where new tasks are learned incrementally and the goal is to preserve the performance on the old tasks while learning the new ones.
We build our model on E2E method in (Castro et al., 2018) that uses a limited budget rehearsal memory (we consider 10 images/class here) to keep representative samples from the old tasks and knowledge distillation (KD) to regularize the network’s output w.r.t.
to previous predictions.
We replace its sample selection mechanism (herding) with ours such that a set of condensed images are generated and stored in the memory, keep the rest of the model same and evaluate this model on the task-incremental learning problem on the digit recognition datasets, SVHN (Netzer et al., 2011), MNIST (LeCun et al., 1998) and USPS (Hull, 1994) in the same order.
MNIST and USPS images are reshaped to 32 × 32 RGB images.
[Figure] [Figure] We compare our method to E2E (Castro et al., 2018), depicted as herding in Figure 4, with and without KD regularization.
The experiment contains 3 incremental training stages (SVHN → MNIST → USPS) and testing accuracies are computed by averaging over the test sets of the previous and current tasks after each stage.
The desired outcome is to obtain high mean classiﬁcation accuracy at T3.
The results indicate that the condensed images are more data-efﬁcient than the ones sampled by herding and thus our method outperforms E2E in both settings, while by a large margin ( 2 . 3% at T3) when KD is not employed.
Neural Architecture Search.
Here we explore the use of our method in a simple neural architecture search (NAS) experiment on CIFAR 10 which typically requires expensive training of numerous architectures multiple times on the whole training set and picking the best performing ones on a test set.
Our goal is to verify that our condensed images can be used to efﬁciently train multiple networks to identify the best network.
To this end, we construct the search space of 720 ConvNets as described in Section 3.1 by varying hyper-parameters W , N , A , P , D over an uniform grid (see supplementary for more details), train them for 100 epochs on three small proxy datasets (10 images/class) that are obtained with Random sampling, Herding and our method.
Note that we train the condensed images for once only with the default ConvNet architecture and use them to train all kinds of architectures.
We also compare to early-stopping (Li & Talwalkar, 2020) in which the model is trained on whole training set but with the same number of training iterations as the one required for the small proxy datasets, in other words, with the same time cost.
Table 4 depicts i) the average test performance of the best selected model over 5 runs when trained on the whole dataset, ii) Spearman’s rank correlation coefﬁcient between the validation accuracies obtained by training the selected top 10 models on the proxy dataset and whole dataset, iii) time for training 720 architectures on a NVIDIA GTX1080-Ti GPU, and iv) memory print of the training images.
Our method achieves the highest testing performance (84.5%) and performance correlation (0.79), meanwhile signiﬁcantly decreases the the searching time (from 8604.3 to 18.8 minutes) and storage space (from 5 × 10 4 to 1 × 10 2 images) compared to whole-dataset training.
The competitive early-stopping baseline achieves on par performance for the best performing model with ours, however, the rank correlation (0.42) of top 10 models is signiﬁcantly lower than ours (0.79) which indicates unreliable correlation of performances between early-stopping and whole-dataset training.
Furthermore, early-stopping needs 100 times as many training images as ours needs.
Note that the training time for synthetic images is around 50 minutes (if K = 500 ) which is one time off and negligible cost when training thousands even millions of candidate architectures in NAS.
C ONCLUSION
In this paper, we propose a dataset condensation method that learns to synthesize a small set of informative images.
We show that these images are signiﬁcantly more data-efﬁcient than the same number of original images and the ones produced by the previous method and they are not architecture dependent, can be used to train different deep networks.
Once trained, they can be used to lower the memory print of datasets and efﬁciently train numerous networks which are crucial in continual learning and neural architecture search resp.
As future work, we plan to explore the use of con- densed images diverse and thus challenging ImageNet (Deng et al. 2009) that contain higher resolution images with larger variations in pose, appearance of objects, background.
A I MPLEMENTATION DETAILS
In this part, we explain the implementation details for the dataset condensation, continual learning and neural architecture search experiments.
Dataset condensation.
The presented experiments involve tuning of six hyperparameters – the number of outer-loop K and inner-loop steps T , learning rates η S and number of optimization steps ς S for the condensed samples, learning rates η θ and number of optimization steps ς θ for the model weights.
In all experiments, we set K = 1000 , η S = 0 .
1 , η θ = 0 .
01 , ς S = 1 and employ
Stochastic Gradient Descent (SGD) as the optimizer.
The only exception is that we set η S to 0 .
01 for synthesizing data with MLP in cross-architecture experiments (Table 2), as MLP requires a slightly different treatment.
Note that while K is the maximum number of outer-loop steps, the optimization early-stops automatically if it converges before K steps.
For the remaining hyperparameters, we use two different sets of for each 1 and 10 image/class setting.
We set T = 1 , ς θ = 1 for 1 image/class and T = 10 , ς θ = 50 for 10 images/class.
Note that when T = 1 , it is not required to update the model parameters (Step 9 in Algorithm 1), as this model is not further used.
For those experiments where more than 10 images/class are synthesized, we set T to be the same number as the synthetic images per class and ς θ = 500 /T , e.g . T = 20 , ς θ = 25 for 20 images/class learning.
The ablation study on hyper-parameters are given in Appendix B which shows that our method is not sensitive to varying hyper-parameters.
We do separate-class mini-batch sampling for Step 6 in Algorithm 1.
Speciﬁcally, we sample a mini-batch pair B T c and B S c that contain real and synthetic images from the same class c at each inner iteration.
Then, the matching loss for each class is computed with the sampled mini-batch pair and used to update corresponding synthetic images S c by back-propogation (Step 7 and 8).
This is repeated separately (or parallelly given enough computational resources) for every class.
Training as such is not slower than using mixed-class batches.
Although our method still works well when we randomly sample the real and synthetic mini-batches with mixed labels, we found that separateclass strategy is faster to train as matching gradients w.r.t.
data from single class is easier compared to those of multiple classes.
In experiments, we randomly sample 256 real images of a class as a mini-batch to calculate the mean gradient and match it with the mean gradient that is averaged over all synthetic samples with the same class label.
The performance is not sensitive to the size of real-image mini-batch if it is greater than 32.
In all experiments, we use the standard train/test splits of the datasets – the train/test statistics are shown in Table T1.
We apply data augmentation (crop, scale and rotate) only for experiments (coreset methods and ours) on MNIST.
The only exception is that we also use data augmentation when compared to DD (Wang et al., 2018) on CIFAR10 with AlexCifarNet, and data augmentation is also used in (Wang et al., 2018).
For initialization of condensed images, we tried both Gaussian noise and randomly selected real training images, and obtained comparable performances.
[Table]
In the ﬁrst stage – while training the condensed images–, we use Batch Normalization in the VGG and ResNet networks.
For reliable estimation of the running mean and variance, we sample many real training data to estimate the running mean and variance and then freeze them ahead of Step 7.
In the second stage – while training a deep network on the condensed set –, we replace Batch
Normalization layers with Instance Normalization in VGG and ResNet, due to the fact that the batch statistics are not reliable when training networks with few condensed images.
Another minor modiﬁcation that we apply to the standard network ResNet architecture in the ﬁrst stage is replacing the strided convolutions where stride = 2 with convolutional layers where stride = 1 coupled with an average pooling layer.
We observe that this change enables more detailed (per pixel) gradients w.r.t.
the condensed images and leads to better condensed images.
Continual learning.
In this experiment, we focus on a task-incremental learning on SVHN,
MNIST and USPS with the given order.
The three tasks share the same label space, however have signiﬁcantly different image statistics.
The images of the three datasets are reshaped to 32 × 32 RGB
[Figure]
a
[Figure] [Table]
[Figure]
[Table]
size for standardization.
We use the standard splits for training sets and randomly sample 2,000 test images for each datasets to obtain a balanced evaluation over three datasets.
Thus each model is tested on a growing test set with 2,000, 4,000 and 6,000 images at the three stages respectively.
We use the default ConvNet in this experiment and set the weight of distillation loss to 1.0 and the temperature to 2.
We run 5,000 and 500 iterations for training and balanced ﬁnetuning as in (Castro et al., 2018) with the learning rates 0.01 and 0.001 respectively.
We run 5 experiments and report the mean and standard variance in Figure 4.
Neural Architecture Search.
To construct the searching space of 720 ConvNets, we vary hyperparameters W ∈ { 32 , 64 , 128 , 256 } , D ∈ { 1 , 2 , 3 , 4 } , N ∈
{ None, BatchNorm, LayerNorm,
InstanceNorm, GroupNorm } , A ∈ { Sigmoid, ReLu, LeakyReLu } , P ∈
None, MaxPooling,
AvgPooling } .
We randomly sample 5,000 images from the 50,000 training images in CIFAR as the validation set.
Every candidate ConvNet is trained with the proxy dataset, and then evaluated on the validation set.
These candidate ConvNets are ranked by the validation performance.architectures with top validation accuracies are selected to calculate Spearman’s rank correlation coefﬁcient, because the best model that we want will come from the top 10 architectures.
We train each ConvNet for 5 times to get averaged validation and testing accuracies.
We visualize the performance correlation for different proxy datasets in Figure F5.
Obviously, the condensed proxy dataset produced by our method achieves the highest performance correlation (0.79) which signiﬁcantly higher than early-stopping (0.42).
It means our method can produce more reliable results for NAS.
B F URTHER ANALYSIS
Next we provide additional results on ablative studies over various deep network layers including activation, pooling and normalization functions and also over depth and width of deep network architecture.
We also study the selection of hyper-parameters and the gradient distance metric.
An additional qualitative analysis on the learned condensed images is also given.
Ablation study on activation functions.
Here we study the use of three activation functions –
Sigmoid, ReLU, LeakyReLu (negative slope is set to 0.01)
– in two stages, when training condensed images (denoted as C ) and when training a ConvNet from scratch on the learned condensed images (denoted as T ).
The experiments are conducted in MNIST dataset for 1 condensed image/class setting.
Table T2 shows that all three activation functions are good for the ﬁrst stage while generating good condensed images, however, Sigmoid performs poor in the second stage while learning a classiﬁer on the condensed images – its testing accuracies are lower than ReLu and LeakyReLu by around 5% .
This suggests that ReLU can provide sufﬁciently informative gradients for learning condensed images, though the gradient of ReLU w.r.t.
to its input is typically sparse.
[Table]
[Table] [Table]
[Table]
Ablation study on pooling functions.
Next we investigate the performance of two pooling functions – average pooling and max pooling – also no pooling for 1 image/class dataset condensation with ConvNet in MNIST in terms of classiﬁcation accuracy.
Table T3 shows that max and average pooling both perform signiﬁcantly better than no pooling (None) when they are used in the second stage.
When the condensed samples are trained and tested on models with average pooling, the best testing accuracy ( 91 .
7 ± 0 .
5% ) is obtained, possibly, because average pooling provides more informative and smooth gradients for the whole image rather than only for its discriminative parts.
Ablation study on normalization functions.
Next we study the performance of four normalization options – No normalization, Batch (Ioffe & Szegedy, 2015), Layer (Ba et al., 2016), Instance (Ulyanov et al., 2016) and Group Normalization (Wu & He, 2018) (number of groups is set to be four) – for 1 image/class dataset condensation with ConvNet architecture in MNIST classiﬁcation accuracy.
Table T4 shows that the normalization layer has little inﬂuence for learning the condensed set, while the choice of normalization layer is important for training networks on the condensed set.
LayerNorm and GroupNorm have similar performance, and InstanceNorm is the best choice for training a model on condensed images.
BatchNorm obtains lower performance which is similar to None (no normalization), as it is known to perform poorly when training models on few condensed samples as also observed in (Wu & He, 2018).
Note that Batch Normalization does not allow for a stable training in the ﬁrst stage ( C ); thus we replace its running mean and variance for each batch with those of randomly sampled real training images.
Ablation study on network depth and width.
Here we study the effect of network depth and width for 1 image/class dataset condensation with ConvNet architecture in MNIST in terms of classiﬁcation accuracy.
To this end we conduct multiple experiments by varying the depth and width of the networks that are used to train condensed synthetic images and that are trained to classify testing data in ConvNet architecture and report the results in Table T5 and Table T6.
In Table T5, we observe that deeper ConvNets with more blocks generate better condensed images that results in better classiﬁcation performance when a network is trained on them, while ConvNet with 3 blocks performs best as classiﬁer.
Interestingly, Table T6 shows that the best results are obtained with the classiﬁer that has 128 ﬁlters at each block, while network width (number of ﬁlters at each block) in generation has little overall impact on the ﬁnal classiﬁcation performance.
Ablation study on hyper-parameters.
Our performance is not sensitive to hyper-parameter selection.
The testing accuracy for various K and T is depicted in Figure F6.
The results show that the optimum K and T are around similar values across all datasets.
Thus we simply set K to 1000 and
T to 10 for all datasets.
Similarly, for the remaining ones including learning rate, weight decay, we use a single set of hyperparameters that are observed to work well for all datasets and architectures in our preliminary experiments.
Ablation study on gradient distance metric.
To prove the effectiveness and robustness of the proposed distance metric for gradients (or weights), we compare to the traditional ones (LopezPaz et al., 2017; Aljundi et al., 2019; Zhu et al., 2019) which vectorize and concatenate the whole gradient, G T , G S ∈ R D , and compute the squared Euclidean distance (cid:107) G T − G S (cid:107) 2 and the
Cosine distance 1 − cos ( G T , G S ) , where D is the number of all network parameters.
We do
[Figure]
image/class learning experiment on MNIST with different architectures.
For simplicity, the synthetic images are learned and tested on the same architecture in this experiment.
Table T7 shows that the proposed gradient distance metric remarkably outperforms others on complex architectures ( e
[Figure]
g .
LeNet, AlexNet, VGG and ResNet) and achieves the best performances in most settings, which means it is more effective and robust than the traditional ones
[Figure]
[Table]
Further qualitative analysis We ﬁrst depict the condensed images that are learned on MNIST,
FashionMNIST, SVHN and CIFAR10 datasets in one experiment by using the default ConvNet in 10 images/class setting in Figure F7.
It is interesting that the 10 images/class results in Figure F7 are diverse which cover the main variations, while the condensed images for 1 image/class setting (see
Figure 2) look like the “prototype” of each class.
For example, in Figure F7 (a), the ten images of “four” indicate ten different styles.
The ten “bag” images in Figure F7 (b) are signiﬁcantly different from each other, similarly “wallet” (1st row), “shopping bag” (3rd row), “handbag” (8th row) and “schoolbag” (10th row).
Figure F7 (c) also shows the diverse house numbers with different shapes, colors and shadows.
Besides, different poses of a “horse” have been learned in Figure F7 (d).
C C OMPARISON TO M ORE B ASELINES
Optimal random selection.
One interesting and strong baseline is Optimal Random Selection (ORS) that we implement random selection experiments for 1,000 times and pick the best ones.
Table T8 presents the performance comparison to the selected Top 1000 (all), Top 100 and Top 10 coresets.
These optimal coresets are selected by ranking their performance.
Obviously, the condensed set generated by our method surpasses the selected Top 10 of 1000 coresets with a large margin on all four datasets.
Generative model.
We also compare to the popular generative model, namely , Conditional Generative Adversarial Networks (cGAN) (Mirza & Osindero, 2014).
The generator has two blocks which consists of the Up-sampling (scale factor=2), Convolution (stride=1), BatchNorm and LeakyReLu layers.
The discriminator has three blocks which consists of Convolution (stride=2), BatchNorm and
LeakyReLu layers.
In additional to the random noise, we also input the class label as the condition.
We generate 1 and 10 images per class for each dataset with random noise.
Table T8 shows that the images produced by cGAN have similar performances to those randomly selected coresets ( i.e .
Top 1000).
It is reasonable, because the aim of cGAN is to generate real-look images.
In contrast, our method aims to generate images that can train deep neural networks efﬁciently.
Analysis of coreset performances We ﬁnd that K-Center (Wolf, 2011; Sener & Savarese, 2018) and Forgetting (Toneva et al., 2019) don’t work as well as other general coreset methods, namely
Random and Herding (Rebufﬁ et al., 2017), in this experimental setting.
After analyzing the algorithms and coresets, we ﬁnd two main reasons.
1) K-Center and Forgetting are not designed for
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table]
[Figure]
[Table] [Table]
[Table]
training deep networks from scratch, instead they are for active learning and continual learning respectively.
The two algorithms both tend to select “hard” samples which are often outliers when only a small number of images are selected.
These outliers confuse the training, which results in worse performance.
Speciﬁcally, the ﬁrst sample per class in K-Center coreset is initialized by selecting the one closest to each class center.
The later ones selected by the greedy criterion that pursues maximum coverage are often outliers which confuse the training.
D F URTHER COMPARISON TO DD (W ANG ET AL ., 2018)
Next we compare our method to DD (Wang et al., 2018)
ﬁrst quantitatively in terms of crossarchitecture generalization, then qualitatively in terms of synthetic image quality, and ﬁnally in terms of computational load for training synthetic images.
Note that we use the original source code to obtain the results for DD that is provided by the authors of DD in the experiments.
Generalization ability comparison.
Here we compare the generalization ability across different deep network architectures to DD.
To this end, we use the synthesized 10 images/class data learned with LeNet on MNIST to train MLP, ConvNet, LeNet, AlexNet, VGG11 and ResNet18 and report the results in Table T9.
We see that that the condensed set produced by our method achieves good classiﬁcation performances with all architectures, while the synthetic set produced by DD perform poorly when used to trained some architectures, e.g . AlexNet, VGG and ResNet.
Note that DD generates learning rates to be used in every training step in addition to the synthetic data.
This is in contrast to our method which does not learn learning rates for speciﬁc training steps.
Although the tied learning rates improve the performance of DD while training and testing on the same architecture, they will hinder the generalization to unseen architectures.
Qualitative comparison.
We also provide a qualitative comparison to to DD in terms of image quality in Figure F8.
Note that both of the synthetic sets are trained with LeNet on MNIST and
AlexCifarNet on CIFAR10.
Our method produces more interpretable and realistic images than DD, although it is not our goal.
The MNIST images produced by DD are noisy, and the CIFAR10 images produced by DD do not show any clear structure of the corresponding class.
In contrast, the MNIST and CIFAR10 images produced by our method are both visually meaningful and diverse.
Training memory and time.
One advantage of our method is that we decouple the model weights from its previous states in training, while DD requires to maintain the recursive computation graph which is not scalable to large models and inner-loop optimizers with many steps.
Hence, our method requires less training time and memory cost.
We compare the training time and memory cost required by DD and our method with one NVIDIA GTX1080-Ti GPU.
Table T10 shows that our method requires signiﬁcantly less memory and training time than DD and provides an approximation reduction of 17% and 55% in memory and 71% and 51% in train time to learn MNIST and
CIFAR10 datasets respectively.
Furthermore, our training time and memory cost can be signiﬁcantly decreased by use smaller hyper-parameters, e.g . K , T and the batch size of sampled real images with a slight performance decline (refer to Figure F6).
[Figure]
E E XTENDED R ELATED W ORK
Variations of Dataset Distillation.
There exists recent work that extends Dataset Distillation (Wang et al., 2018).
For example, (Sucholutsky & Schonlau, 2019; Bohdal et al., 2020) aim to improve DD by learning soft labels with/without synthetic images.
(Such et al., 2020) utilizes a generator to synthesize images instead of directly updating image pixels.
However, the reported quantitative and qualitative improvements over DD are minor compared to our improvements.
In addition, none of these methods have seriously veriﬁed the cross-architecture generalization ability of the synthetic images.
Zero-shot Knowledge Distillation.
Recent zero-shot KD methods (Lopes et al., 2017; Nayak et al., 2019) aim to perform KD from a trained model in the absence of training data by generating synthetic data as the intermediate production to further use.
Unlike them, our method does not require pretrained teacher models to provide the knowledge, i.e . to obtain the features and labels.