Dataset Distillation via Factorization
Abstract
In this paper, we study dataset distillation (DD), from a novel perspective and introduce a dataset factorization approach, termed HaBa , which is a plug-andplay strategy portable to any existing DD baseline.
Unlike conventional DD approaches that aim to produce distilled and representative samples, HaBa explores decomposing a dataset into two components: data Ha llucination networks and
Ba ses, where the latter is fed into the former to reconstruct image samples.
The flexible combinations between bases and hallucination networks, therefore, equip the distilled data with exponential informativeness gain, which largely increase the representation capability of distilled datasets.
To furthermore increase the data efficiency of compression results, we further introduce a pair of adversarial contrastive constraints on the resultant hallucination networks and bases, which increase the diversity of generated images and inject more discriminant information into the factorization.
Extensive comparisons and experiments demonstrate that our method can yield significant improvement on downstream classification tasks compared with previous state of the arts, while reducing the total number of compressed parameters by up to 65%.
Moreover, distilled datasets by our approach also achieve ~10% higher accuracy than baseline methods in cross-architecture generalization.
Our code is available here.
Introduction
The success of deep models on a variety of vision tasks, such as image classification [26, 11, 38], object detection [37, 36] and semantic segmentation [43, 56, 29], is largely attributed to the huge amount of data used for training and various pre-trained models [57].
However, the sheer amount of data introduces significant obstacles for storage, transmission, and data pre-processing.
Besides, publishing raw data inevitably brings about privacy or copyright issue in practice [44, 10].
To alleviate these problems, Wang et al .
[52] pioneer the research of dataset distillation (DD), to distill a large dataset into a synthetic one with only a limited number of samples, so that the training efforts with the distilled dataset for downstream models on the original dataset can be largely reduced, which facilitates a series of applications like continual learning [41, 40, 54, 31] and black-box optimization [7].
Due the significant practical value of DD, many endeavours have been made on this area [62, 60, 61, 51, 24, 6, 63] to design novel supervision signals to train the synthetic datasets and to further improve their performances.
Nevertheless, there is a potential drawback in conventional settings of DD: it largely treats each synthetic sample independently and ignores the inter coherence and relationship between different instances.
As such, the information embraced by each sample, despite distilled, is by nature limited.
Using the synthetic samples for training downstream models, therefore, inevitably leads to the loss of dataset information.
Moreover, the few distilled samples are incompatible with the enormous number of parameters in a deep model and may yield the risk of overfitting.
To verify these potential issues, we conduct a pre-experiment on CIFAR10 dataset with 10 synthetic images per class, using MTT [6], the current SOTA solution on DD, as the baseline.
In addition 36th Conference on Neural Processing Systems (NeurIPS 2022).
𝐻 !
𝐻 " ...
[Figure]
Synthetic Dataset
Performance
Matching ⟸
[Figure]
to the baseline setting, we also incorporate all the checkpoint synthetic datasets after each 100 DD iterations in the convergent stage to train the downstream model.
Since the synthetic images are fine-tuned during this stage, multiple checkpoints can be viewed as related but different, which may somehow increase the diversity.
As a result, it yields overall lower test loss and hence better final results in downstream training, as shown in the blue and green curves in Fig. 2, which indicates that current DD solutions can be potentially improved by leveraging some sample-wise relationships to diversify the distilled data.
Nevertheless, simply involving more data samples may also increase the memory overhead.
This fact motivates us to ask: is it possible to encode some shared relationships in a dataset implicitly, instead of storing samples directly, to avoid such additional storage costs?
[Figure]
We show in this paper that, it can indeed be made possible through reformulating the DD task as a factorization problem.
As shown in
Fig.
1, we propose a novel perspective dubbed HaBa , to factorize a dataset into two compositions: data Ha llucination networks and Ba ses.
A data hallucination network, or hallucinator, can take any basis as input and output the corresponding hallucinated image.
Supervised by the training objective of DD, a set of hallucinators can synthesize multiple samples from a common basis and are optimized to extract effective relationships among different samples in original datasets explicitly.
In this way, information of |H|×|B| images can be included for a factorization result with |H| hallucinators and |B| bases via arbitrary pair-wise combination, which improves the data efficiency of traditional DD exponentially.
As shown in the yellow curve in Fig.
2, with the same budget on the storage, our strategy achieves better test performance compared with the MTT baseline.
To further increase the informativeness of factorized results, we introduce a pair of adversarial contrastive constraints to promote sample-wise diversity.
The goal of HaBa is to minimize the correlation among images composed of different hallucinators but a common basis, while an adversary tries to maximize it.
Such an adversarial scheme, in turn, enforces the hallucinators to produce diversified images and increases the amount of useful information.
Notably, HaBa is a versatile strategy that can be built upon existing DD baselines, since it is compatible with any training objective for measuring the similarity between downstream performances as shown in Fig.
1, We conduct extensive experiments to demonstrate the advantages of the proposed method over baseline ones.
In all benchmarks and comparisons, HaBa produces significant and consistent improvement on training downstream models, while reducing the total number of compressed parameters by up to 65%.
Furthermore, it demonstrates strong cross-architecture generalization ability with accuracy improvement higher than 10%.
Our contributions are summarized as follows:
• We study dataset factorization, a novel perspective to explore dataset distillation, and propose a novel approach termed HaBa for hallucinator-basis factorization.
• We present a pair of adversarial contrastive objectives to further increase the data diversity and information capability.
• HaBa is a plug-and-play scheme compatible with all existing training objectives of DD and can yield significant and consistent improvement over the state of the arts.
Related Works
The goal of dataset distillation (DD) is to optimize a smaller synthetic dataset such that it is capable to take place of original one for training downstream tasks, which is different from coreset selection [1, 8, 15, 42, 48], another branch for dataset compression, directly selecting samples from raw datasets.
In this section, we provide a detailed review of previous methods in DD.
Motivated from knowledge distillation [18, 14, 59, 58] aiming at model compression, Wang et al .
[52] introduce the concept of dataset distillation for dataset compression.
The idea is to optimize the synthetic images so that they can minimize loss functions of downstream tasks, where a bilevel optimization algorithm [13] is involved.
Following this routine, several works further consider learnable labels beyond samples [4, 46].
Subsequently, Zhao et al .
[62] and several following approaches [60, 28] consider matching gradients of a downstream model produced by synthetic samples and real images, which improve the performance significantly.
Most recently, Cazenavette et al .
[6] argue that single-iteration gradient matching may lead to inferior performance due to error accumulation across multiple steps and thereby propose to match long-range training dynamics of an expert trained on the original dataset.
As an alternative method to profile training effects produced by different sets, Nguyen et al .
[33, 34] also introduce the kernel ridge-regression approach based on the
Neural Tangent Kernel (NTK) in infinitely wide convolutional networks [20].
Apart from matching training effects, there are also methods matching data distributions between original and synthetic datasets.
For instance, Zhao et al .
[61] propose a simple but effective Maximum
Mean Discrepancy (MMD) constraint for DD, which does not involve the training of downstream models and enjoys superior training efficiency.
Wang et al .
[51] propose CAFE, explicitly attempting to align the synthetic and real distributions in the feature space of a downstream network.
Above mentioned methods are dedicated to exploring suitable training objectives and pipelines for
DD.
However, there are few works concerning improving the data efficiency for distilled samples.
Although Zhao et al .
[60] propose differentiable siamese augmentation (DSA) to enrich the training data, the augmentation operations used, e.g. , crop, flip, scale, and rotation, cannot encode any information about the target datasets.
In this paper, we study the task in a factorization perspective, to factorize a dataset into two different compositions: data hallucination networks and bases.
Both parts carry important knowledge of the raw dataset.
For downstream training, hallucinators and bases can perform arbitrary pair-wise combination, i.e. , sending any basis to any hallucinator, to create a training sample.
The idea of factorization can improve the diversity of distilled training datasets significantly, without introducing additional costs for storage.
It is also a versatile strategy compatible with all aforementioned DD methods, which will be demonstrated in the experiment part.
Concurrent Works on Efficient Distilled Dataset Parameterization:
As a concurrent work, Kim et al .
[24] propose IDC for efficient synthetic data parameterization.
It reveals that only storing down-sample version of synthetic images and conducting bilinear upsampling in downstream training would not hurt the performance much.
Thus, given the same budget of storage, it can store 4 × number of 2 × down-sample synthetic images compared with the baseline.
Both IDC and HaBa in this paper are dedicated to improving the data efficiency of synthetic parameters.
Interestingly, according to the definition of our hallucinator-basis factorization, IDC can in fact be treated as a special case of HaBa, where the hallucinator is a parameter-free upsampling function and each basis has a smaller spatial size.
Nevertheless, the main focuses for IDC and HaBa are different and they are in fact two orthogonal techniques, which can readily join force to enhance the baseline performance, as discussed in Sec.
Methods
In this section, we elaborate our proposed method HaBa for dataset distillation (DD).
Assume that there is an original dataset T
= { ( x i , y i ) } |T
| i =1 with |T | pairs of a training sample
x i and the corresponding label
y i .
DD targets a synthetic dataset S =
{ (ˆ x i , ˆ y i ) } |S| i =1 with |S| ≪
|T | and expects that a model trained on S can have similar performance than that trained on T .
𝐻 !
𝐻 " ℋ
D o w n
S t r ea m
A dv .
𝒯 ℬ ℒ !"
#$ ℒ %% ℒ &’#.
ℒ &’).
For ℬ and ℋ
For Adv.
Sample
Sample
Sample 𝜇 𝜎
Encoder
Decoder
H
[Figure] [Figure]
We adopt batch size 2 here for clarity Architecture of hallucinator in detail.
Traditional DD methods treat each synthetic sample independently and ignore the inner relationship between different samples within a dataset, which results in poor data/information efficiency.
Focusing on such drawback, we study DD from a novel perspective and redefine it as a hallucinator-basis factorization problem:
[Equation]
where there are |H| hallucination networks and |B| bases.
The j -th hallucinator is parameterized by θ
i and we denote it by H
θ i for 1 ≤ j ≤ |H| .
For downstream training, a training data pair (˜ x ij , ˜ y ij ) is created online via sending the i -th basis, with any 1 ≤ i ≤ |B| , to the j -th hallucinator, with any 1 ≤ j ≤ |H| , i.e. , ˜ x ij = H θ j (ˆ x i ) .
In this paper, the label ˜ y ij is simply taken as ˆ y i .
An overview of our method is shown in Fig. 3(Left).
To go deeper into the technical details, we first start with the introduction of our basis and data hallucination network in Sec. 3.1.
Then, we propose an adversarial contrastive constraint to increase data diversity in Sec. 3.2.
Finally, we present the whole training pipeline of the hallucinator-basis factorization for DD in Sec. 3.3.
3.1 Basis and Hallucinator
Basis: Typically, for an image classification dataset T = { ( x i , y i ) } |T | i
=1 , x i ∈ R h × w × c
and y i ∈ { 0 , 1 , ..., C − 1 } for each 1 ≤ i
≤ |T | , where each x i is a c -channel image with a resolution of h × w , and C is the total number of classes.
In previous DD methods, the format/shape of synthetic data pairs (ˆ x, ˆ y ) has to be held the same as that of real data, so as to make sure the consistency between input and output formats in the training and test time for downstream models.
By contrast, since hallucinator networks are capable of spatial-wise and channel-wise transformation, the shape of each ˆ x i , 1 ≤ i ≤ |B| , denoted as h ′ × w ′ × c ′ , is not necessarily the same as that of original samples and thus more flexible.
And for a classification problem, we do not modify its label space in this paper for simplicity and maintain the categorical format.
Hallucinator:
Given a basis ˆ x ∈ R h ′ × w ′ × c ′ , a data hallucination network, aims to create a new image ˜ x ∈ R h × w × c based on ˆ x , which can be viewed as a conditional image generation problem.
Inspired by image style transfer [22, 19, 21, 30], a typical conditional image generation problem, we devise an encoder-transformation-decoder based architecture for hallucinators, as shown in Fig.
3(Right).
Specifically, the encoder, denoted as enc , is composed of CNN blocks, which non-linearly maps an input ˆ x to a feature space R h ′′ × w ′′ × c ′′ .
Then, an affine transformation with scale σ and shift µ is conducted on the derived feature, where σ and µ are treated as network parameters in this paper.
At last, the decoder dec under a symmetric CNN architecture with enc projects the transformed feature back to the image space.
Formally, this process can be written as:
[Equation]
where the multiplication is element-wise operation.
There are |H| hallucinators in the whole factorization pipeline and each would be trained to implicitly encode some sample-wise relations by its network parameters.
3.2 Adversarial Contrastive Constraint
Ideally, the knowledge encoded by different hallucinators should be as different/orthogonal as possible to get the most benefits for each individual.
To instantiate such regularization, let’s consider two composed images ˜ x ij and ˜ x
ik from two different hallucinators H θ j and H θ k
but a common basis ˆ x i .
The divergence between ˜ x ij and ˜ x ik is expected to be large.
To measure the divergence, a feature extractor is required to map an input image to a feature space, and how to train such a feature extractor to find an appropriate feature space is of great importance.
In this paper, we formalize the training of hallucinators and the feature extractor as a min-max game in a self-consistent manner, where the feature extractor desires to minimize the divergence between ˜ x ij and ˜ x
ik while hallucinators, as well as bases, are optimized to maximize it so that the two players can reinforce each other.
In specific, the feature extractor, denoted as F and parameterized by ψ , is typically a CNN structure for the downstream task and we adopt features at the last hidden layer before the output layer, denoted as F − 1 (˜ x ij ) and F − 1 (˜ x ik ) .
F is optimized to maximize the correlation between the two feature vectors, which can be quantified by the metric of mutual information (MI).
Inspired by the lower bound of MI [49], the objective to minimize the divergence for F is given by the following contrastive form:
[Equation]
where τ is a scalar temperature coefficient.
For the classification problem, we can alternatively adopt the supervised form of the contrastive loss L con. , where ˜ x uk with the same class label as ˜ x ij are also taken into consideration as positive samples in Eq. 3.
The supervised contrastive loss can benefit to increase the correlation of samples from the same class [23] for a more reasonable feature representation.
In addition, the feature space is expected to reflect the task-specific property for a meaningful representation.
Thus, we also incorporate the task loss L task , e.g. , cross-entropy loss in classification tasks, over the synthetic dataset as a supervision signal for F .
In this way, the overall training objective for F is defined as:
[Equation]
where λ con.
and λ task are hyper-parameters controlling the weight for each term.
F acts as an adversary to minimize the divergence between ˜ x ij and ˜ x
ik , while the synthetic dataset is expected to maximize it to increase data diversity.
To this ends, the similarity between F − 1 (˜ x ij ) and F − 1 (˜ x ik ) becomes one loss term for hallucinator-basis factorization.
In this paper, we adopt the cosine-similarity and the objective L cos.
is given by:
[Equation]
During training, the feature extractor and the factorized components are updated alternately to play this min-max game.
3.3 Factorization Training Pipeline
Following previous paradigms [62, 61, 6, 51], the synthetic dataset S is updated in an iterative algorithm.
In each iteration, we randomly sample a batch of hallucinators and bases and conduct pair-wise combinations.
The composed images are evaluated by the objective of dataset distillation
L DD and the similarity metric in Eq.
[Equation]
(6) where hyper-parameters λ DD and λ cos.
balance the loss.
Notably, the hallucinator-basis factorization is compatible with a variety of configurations of L DD by previous arts, which makes it a versatile and effective strategy for DD.
In this paper, we adopt the trajectories matching loss in Cazenavette et al .
[6] as L DD by default thanks to its superior performance.
The basic idea is to update a downstream model from a cached checkpoint ϕ ∗ t at iteration t , using the synthetic dataset S for N times, and using the real dataset T for M times respectively.
The updated parameters by the two cases, ˆ ϕ t + N and ϕ ∗ t + M are enforced to be consistent:
[Equation]
where α and β are learning rates with S and T respectively.
α is learnable in the framework while β is a hyper-parameter.
In Sec. 4.2, we also experiment with other settings of L DD .
Based on the supervised signals in Eq. 6, the gradients are backward propagated to the composed images and finally to the sampled hallucinators and bases so as to be updated using a decent algorithm such as SGD.
Since all the operations are differentiable, the training can be completed end-to-end.
Experiments 4.1 Datasets and Implementing Details
We conduct evaluations of our method on three standard image classification benchmarks: SVHN [32],
CIFAR10, and CIFAR100 [25].
There are 60,000 images for real-world digit recognition in SVHN.
For CIFAR10 and CIFAR100, there are 50,000 training images in total.
The number of classes for the three datasets are 10 , 10 , and 100 respectively.
All the images are under 32 × 32 resolution in 3-channel RGB format.
Following previous works [6], we use ZCA for image preprocessing with
Kornia implementation [39] before all the experiments.
Experiments with more datasets, including images in larger spatial scales, can be found in the supplement.
In this paper, for convenience of comparisons with prior works, we maintain the same size with images in original datasets, i.e. , h ′ =
h , w ′ = w , and c ′ = 3 for bases.
We also experiment with other sizes of bases in Sec. 4.3.
For hallucinators, the encoder and decoder contain 1 Conv-ReLU blocks.
The number of feature channel c ′′ is 3 .
We use 5 hallucinators by default.
The learning rates of hallucinators and bases, η H and η B , are the same and for the feature extractor, the learning rate η F is 0 .
Hyper-parameters λ con.
, λ task , λ DD , and λ cos.
are set as 0 . 1 , 1 , 1 , and 0 .
1 empirically.
Sensitivities of these hyper-parameters are analyzed in Sec. 4.3.
The adversary network has the same architecture as that for computing L DD .
In experiments on SVHN and CIFAR10, we incorporate all the bases in each iteration, while in experiments on CIFAR100, we adopt a batch size of 300 when the total number of bases is greater than 1,000.
We only consider random 2 hallucinators in one iteration for simplicity.
The maximal configuration of computational resources is 4 24GB 3090 GPUs.
The GPU memory consumption is dependent on that of the baseline method for L DD and is slightly higher than it due to the computation of L cos.
and L con. .
The baseline method for L DD is MTT [6] if not specified.
Other settings related to DD hold the same as the baseline.
All the quantitative results are based on the mean and standard deviation over 5 repeated experiments.
To make sure fair comparisons, the dataset size in our method is equal to the number of bases |B| and the hallucinators are treated as parameterized data augmentors working online in downstream training, just as general data augmentations, which means that the dataset size does not increase compared with the baselines.
4.2 Comparisons
Comparisons with State of the Arts: We compare HaBa with previous state of the arts for DD in standard settings, to synthesize 1, 10, and 50 images per class (IPC) respectively.
In our setting, the number of parameters in a hallucinator is is approximately equal to that for 2 synthetic images, while the size of a basis is equal to that of an image.
Taking the storage cost of 5 hallucinators into consideration, we set the number of bases per class (BPC) as IPC minus 1 in each IPC configuration when IPC is greater than 1, to make the comparisons as fair as possible.
Candidates are coreset based methods including Random [8, 35], Herding [5, 2], K-Center [12, 42], and Forgetting [47], meta learning based methods including DD [52] and LD [4], training matching based methods including DC [62], DSA [60], and MTT [6], and distribution matching based methods including DM [61] and CAFE [51].
The comparisons follow the standard protocol adopting a 3-layer
Conv-InstanceNorm-ReLU-AvgPool ConvNet with 128 channels in training and testing.
[Table]
Per Class; Ratio (%): the ratio of distilled images to whole training set.
Underline denotes results by our implementation.
[Figure]
The comparison results are shown in Tab.
and we can observe that HaBa achieves state-of-the-art performance in all datasets and settings.
Especially when the ratio of distilled images to the whole training set is less than 1%, our method can yield significant improvement over all the candidate methods, which demonstrates that the scheme of hallucinator-basis factorization improves the data efficiency for the task of dataset distillation.
Qualitative Comparisons: We visualize the factorized results by our method as well as the baseline on CIFAR10 dataset with 10 BPC in Fig.
Due to the space limitation, we only provide images generated by 2 hallucinators here.
More results can be found in the supplement.
As shown in the figure, we can find that bases mainly store some main structures and contour information.
Different hallucinators would render a basis with diverse styles and details.
Thanks to the dataset factorization scheme, the diversity of distilled images by our method is higher than that by the baseline.
Building upon Different Baselines: To reflect the versatility of the insight, we implement HaBa on multiple state-of-the-art training pipelines of DD, including DC, DM, and MTT.
We evaluate the performance of synthetic datasets on CIFAR10 and maintain the IPC of baseline methods as
BPC plus 1, which makes storage costs for synthetic datasets as close as possible for fairness.
As shown in Tab. 2, when training and testing on ConvNet, the strategy of HaBa can make a consistent improvement over all the baselines, which demonstrates that factorization is a general idea to improve the data efficiency in DD.
Cross-Architecture Performance:
For DD, a satisfactory distilled dataset should have similar training effects to the original one on downstream models with arbitrary architectures.
Thus, crossarchitecture generalization performance is an important metric for DD.
We use the synthetic datasets trained on ConvNet to train models with different structures including ResNet [16], VGG [45], and
AlexNet [26].
The results can be found in Tab. 2.
Benefiting from the increased data diversity, HaBa can improve the across-architecture accuracy significantly with a performance gain up to 17 .
The consistent and significant improvement validates the superior ability of our method to capture the informative features and thus original datasets can be replaced by the synthetic ones better.
[Table]
Comparisons under the Same Number of Final Images: In the default comparison protocol, we compare our method with the baselines using the same budget of storage, where our method can store information of exponentially more images than the baselines with the same number of parameters.
In this part, we also examine the performance of HaBa under the condition that the number of final images, i
[Table]
e. , |H| × |B| , is equal to that used by the baseline.
Intuitively, given that the objective functions of our method and the baseline are the same exactly, the performance of the baseline can be viewed as an upper bound of ours, since there are significantly less parameters in our method to carry the information of final images in this case.
Therefore, we first remove the term L cos.
from the loss function of DD in Eq.
6 to guarantee a consistent optimization objective with the baseline.
Then, we compare the performance of HaBa and the baseline using 10, 20, 30, 40, and 50 final images respectively.
Here, the number of hallucinators |H| is 2 and the number of bases is thus half of the number of final images.
As shown in the red and green curves in Fig.
7, performance of the baseline can be well approximated by ours with only half of the number of parameters, especially when the number of images is relatively large.
Remarkably, with the proposed adversary contrastive constraint, our method can even outperform the baseline consistently, as shown in the blue curve, which further demonstrates the effectiveness of the proposed solution.
Comparisons with Concurrent Works on Efficient Distilled Dataset Parameterization: As a concurrent work on efficient distilled dataset parameterization, IDC [24] is proposed to store 4 × number of 2 × down-sample synthetic images compared with the baseline.
The core is to reduce the spatial size for efficient parameterization.
For HaBa of this paper, instead, we do not modify the spatial size of bases in the default setting for better qualitative explainablity and more intuitive comparisons with the baselines.
In this sense, IDC and HaBa are in fact two orthogonal techniques and they can readily join force to enhance the baseline performance.
Here, we try using the technique of IDC and adopting 2 × down-sample synthetic images on the baseline MTT, based on which we further consider adding our HaBa and involving 5 hallucinators.
As shown in Tab. 3, with the efficient parameterization of IDC, the performance of baseline can be improved.
With HaBa in this paper, the performance can even be further improved a lot: 5.14%, 1.29%, and 4.30% in the three settings respectively, which demonstrates that IDC and HaBa work in different ways.
Applications in Continual Learning: To further demonstrate the advantage of the proposed method for improving data efficiency, following the setting of DM [61], we conduct experiments on the setting of continual learning on CIFAR-100, with 20 random classes per stage.
The average number of parameters per class is 20 × 32 × 32 × 3 .
The synthetic datasets are trained with a ConvNet with 3 blocks.
We evaluate synthetic datasets by our method and the DM baseline on the same ConvNet architecture and ResNet18.
The results in Fig.
5 demonstrate that the proposed method increases the informativeness of synthetic datasets and thus produce significantly better performance, especially in the cross-architecture setting.
4.3 Ablation Studies
Loss Terms: To validate the effectiveness of the proposed adversarial contrastive constraints, we design ablation studies on the CIFAR10 dataset over three loss terms: L cos.
in Eq.
5, L con.
in Eq. 3, and the task-specific loss L task .
Through the results in Tab. 4, we can find that deleting any one of
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
them would hurt the performance.
We also experiment with involving L con.
for downstream training, to enforce the similarity among images composed of different hallucinators and a common basis.
Observed from the last row of Tab. 4, the performance can be further improved, since L con.
helps the representation learning of related samples [23].
Note that we do not use this loss term for downstream training in other experiments for fair and standard comparison.
We examine the sensitivities of hyper-parameters λ cos.
and λ con.
used to balance the weights of loss terms L cos.
and L con.
respectively in Fig.
The results are evaluated on the CIFAR10 dataset with 10 BPC.
We can observe that the overall performance is not sensitive to the selection of these hyper-parameters and our method makes a consistent improvement over the baseline with 11 IPC.
Class-Independent Hallucinators v.s. Shared Hallucinators:
In the default setting of HaBa, each class maintains a certain number of bases independently and all the classes share the same set of hallucinators.
But what about the case that hallucinators are also made class-independent?
We study this problem experimentally in Tab. 5.
Given the same BPC, class-independent hallucinators can indeed somehow improve the performance when there are fewer synthetic samples, e.g. , 1 BPC.
However, when BPC is higher, equipping each class with an independent set of hallucinators would not benefit the performance.
There are probably two reasons: (1) shared hallucinators across all the classes extract global information of the whole dataset, which encodes more representative and universal knowledge; and (2) the class-independent case would make the number of hallucinatorstimes for the CIFAR10 dataset, which leaves a heavy burden for the optimization process.
Thus, as indicated in Tab. 5, a better solution is to make room for more bases using the memory allocated to store class-independent hallucinators initially, which would result in more satisfactory data efficiency.
Number of Channels Used by Basis: By default, the shape of a basis is the same as that of a real image, which is generally in RGB 3-channel format.
In fact, in Fig. 8(Left), we also verify that it is also possible to use single-channel basis, which can reduce the memory cost by nearly 2/3 without hurting the performance too much.
Interestingly, if the memory cost is held the same, we can choose to use 3 times BPC to store single-channel bases, rather than 3-channel ones.
This would yield impressive improvement on the test accuracy when BPC is small.
Note that for baseline results, IPC is set as the corresponding BPC plus 1.
Number of Hallucinators: We study the impact of the number of hallucinators, i.e. , |H| , in Fig.
8(Right).
We can observe that when BPC is small, including more hallucinators is helpful for the performance.
Nevertheless, when BPC is 10 or 50, the performance would not improve with more hallucinators when |H| > 10 .
One reason is that when |H| is large, the sampling of hallucinators in each iteration is sparse, which makes the joint optimization of all the hallucinators more difficult.
Data Augmentation:
The similarity between our hallucinator set and data augmentation lies that both of them can contribute to generating more samples and increasing the diversity.
However, the essential difference is that our hallucinators are optimized to encode sample-wise relationships in
[Figure]
[Figure] [Figure]
[Table]
[Figure]
[Table] [Table]
[Figure]
[Table]
a dataset, while data augmentation is based on some prior and heuristic knowledge of images.
By default, both our method and the baseline adopt the data augmentation strategy DSA [60].
To study the relationship between the two schemes experimentally, we attempt to remove DSA from baseline and our method and report the corresponding results in Tab. 6.
The evaluation is on CIFAR10 with
IPC for baseline and 10 BPC for ours.
Through the results, we can find that (1) our method without data augmentation can also outperform the baseline method with augmentation significantly, which means that the mechanism of HaBa can benefit the dataset distillation task more with the learning of global information of a dataset in hallucinators; and (2) with data augmentation, our performance can be further improved, which indicates that HaBa and DSA work in different manners.
Conclusions, Limitations, and Future Works
This paper proposes a novel hallucinator-basis factorization method dubbed HaBa for dataset distillation (DD).
It uses hallucinators to encode inner relations between different samples in original datasets, which can largely improve the data efficiency of distilled results.
To diversify the knowledge captured by different hallucinators, a pair of adversarial contrastive constraints is further introduced.
Extensive evaluations and comparisons on multiple benchmark datasets demonstrate that HaBa is capable of significantly improving the performance of downstream models trained on the synthetic dataset, using only 35% cost of memory for storage.
Moreover, it is a versatile strategy that is compatible with different configurations of DD frameworks and yields consistent improvement.
Despite the superior performance of the proposed hallucinator-basis factorization (HaBa) scheme, there are also some potential limitations.
On the one hand, compared with the baseline method HaBa built upon, the process of online pairwise combination between hallucinators and bases in training increases the cost of time and GPU memory slightly, although light-weight hallucinators are adopted.
On the other hand, it may inherited the limitations of baseline methods.
For example, when the number of images is large, further increasing the number would produce limited performance gain.
For future works, beyond the training efficiency of HaBa, introducing class-wise relationship may also be a potential research direction.
For example, it is probably optimal that one class shares hallucinators with some specific classes but does not share with others.
It is also promising to explore more advance factorization for a dataset to further improve the performance.
Acknowledgement
This research is supported by the National Research Foundation, Singapore under its Medium Sized
Centre for Advanced Robotics Technology Innovation (WBS: A-0009428-09-00).
Xinchao Wang is the corresponding author.
Appendices
In this part, we provide additional details, more results, potential limitations, and future directions of the proposed Hallucinator-Basis factorization (HaBa) for dataset disllation (DD).
First, we provide more details on the pipeline of HaBa.
Then, we conduct more experiments to demonstrate and analyze performance of our method, including results on more benchmarks with larger resolutions, as supplement to the quantitative study in the main paper.
We also provide more qualitative results by
HaBa and additional ablation studies.
Finally, we discuss some limitations and future works of our method.
; 15: end while
Appendix A Algorithm Details
To better elaborate the details of the proposed HaBa for DD, we provide an algorithmic illustration for the whole pipeline in Alg. 1, as a supplement to Sec.
3 of the main paper.
The overall algorithm takes an original dataset as well as some hyper-parameters shown in Alg. 1 as input.
The output is the distilled result including a set of hallucinators H and a set of bases B , as defined in Eq.
1 of the main paper.
The goal is to equip the distilled dataset with similar downstream performance to the original one.
Appendix B More Results
Low-Resolution Data: We provide results on the more-common benchmark datasets in DD in Tab. 7:
MNIST [27] and FashionMNIST [55].
Both datasets contain 60,000 images for training and 10, images for testing in 10 classes.
The images are under 28 × 28 resolution with 1 channel.
We build our HaBa on MTT [6] in this part.
Although the performances of DD on these two dataset seem to be saturated, our method may still yield consistent improvement over the baseline, especially when the ratio of distilled images to the whole training set is small.
ImageNet Subsets: We also evaluate the proposed scheme on the more-challenging settings of
ImageNet [9] subsets.
We follow the baseline MTT [6] for the divisions of subsets.
The 6 subsets include ImageFruit, ImageMeow, ImageNette, ImageSquawk, ImageWoof, and ImageYellow.
Each subset contains over 10,000 images, and we resize all the images to 128 × 128 resolution following the original setting.
We use ConvNet with 5 Conv-InstanceNorm-ReLU-AvgPool layers for training.
For testing, in addition to the same structure of ConvNet, we also evaluate the results under 3 other
[Table] [Table]
architectures: ResNet, VGG, and AlexNet.
To ensure the same number of parameters used for the distilled datasets, we set the number of images per class used by the baseline as the number of bases per class used by HaBa plus 1,
i
[Table]
e. , 2 IPC v.s. 1 BPC and 11 IPC v.s. 10 BPC.
Other settings follow the same configuration in the main paper.
The test performances of models trained by the distilled datasets are shown in Tab. 8.
We can observe that HaBa outperforms the baseline in almost all cases except several experiments when
IPC and BPC are small and the architectures of training and testing are the same.
Notably, in all the cross-architecture generalization settings, HaBa achieves superior performance over the baseline, which further demonstrates the improvement of data efficiency introduced by the factorization and online pair-wise combination.
More Ablations on Hallucinators:
In the default setting, the encoder and decoder of hallucinators have 1 Conv-ReLU block and the number of feature channels is 3.
In this part, we provide more results when we consider increasing the capacity of the hallucination networks.
As shown in Tab. 9, we try increasing the depth of the networks by adding more nonlinear blocks.
Although the performance
[Table]
[Table] [Table]
[Table] [Table]
[Figure]
can indeed be improved, it results in nonnegligible latency to downstream training speed, measured by the number of epochs per second.
Taking both training speed and performance into consideration, we consider using only 1 nonlinear block by default, which yields best trade-off between the two factors.
Likewise, we also try increasing the number of feature channels in halluciantors as shown in
Tab. 10.
The number of parameters is almost proportional to the number of channels.
However, the performance gain is very limited.
Thus, we simply take the number of channels in images, which isfor RGB images, as the number of feature channels in hallucinators.
More Insights on the Configurations of Hallucinators and Bases: As shown in the ablation studies in the main paper and the supplement, under the framework of hallucinator-basis factorization, there are many factors that affect the performance.
Given a fixed storage budget, how to scale the bases and hallucinators is an important topic.
Among all the factors, we empirically find that the depth of hallucinators, the number of hallucinators, the number of channels in each basis, and the number of bases are the most important ones, which will be studied in the following exploration.
Here, we consider three types of storage budget: small, medium, and large, corresponding to the cost of IPC=2, 11, and 51 for the baseline method respectively.
We consider cases of 1 and 2 convolution blocks for the depth of hallucinators, 2 and 5 for the number of hallucinators, and 1 and 3 for the number of channels in each basis.
For each setting, we adjust the number of bases to fit the given budget.
Enumerating all the configurations, there are totally 8 settings for each kind of budget.
Their results are visualized in Fig.
Based on the results we have the following observations:
• For all the three types of budget, the best performance is achieved by using deeper hallucinators.
Especially under small and medium budgets, using depth 2 can outperform using depth 1 almost consistently.
This can be explained by the more complex sample-wise relationship extracted by hallucinators.
• In our framework, bases are expected to store sample-independent information while hallucinators are used to encode shared relationship across all the samples.
When the budget is small, using 1-channel bases can achieve significantly better results.
This is because small storage budget would more rely on increasing the number of independent data samples for a better diversity.
The informativeness of each basis appears less important.
• When the budget increases, the advantage of 1-channel bases mentioned before would diminish gradually.
Especially under the large budgets, 3-channel bases outperform
[Figure]
[Figure] [Figure]
1-channel ones
consistently.
The reason is that when the number of bases is adequate, focusing on the informativeness of each basis can produce more benefit than increasing the number
• When the budget is large, using more hallucinators can yield slightly better results , which can probably be attributed to the further improvement on the diversity.
• The larger the budget is, the less insensitive the performance is, to different configurations.
Note that the above exploration is conducted without taking the downstream training speed into consideration, which is also an important metric in the task of dataset distillation.
Our opinion on the scalability is that, when downstream training overhead is not a issue, deeper hallucinators are recommended for better performance; otherwise if downstream efficiency is desired, we find thatnonlinear block is sufficient, since heavier hallucination networks can result in nonnegligible latency, especially when the total number of images is large.
Sharing Encoder and Decoder across all Hallucinators: As a variant of our default case which uses different hallucination networks, it is also feasible for the halluciantors to share a common group of encoder and decoder but use different parameters ( σ, µ ) for affine transformation, which is potential to further boost the data efficiency.
As shown in Tab. 11, the performance becomes slightly worse.
We conjecture that different convolution encoders and decoders may contribute to the diversity of the extracted patterns, which increases the representation ability of the hallucinator set.
Moreover, since we only use 1 convolution block for encoders and decoders, the number of parameters is not so significant compared with that of a basis.
Therefore, we consider making the whole network independent with each other for all hallucinators.
Results on Speech Domain: To validate the versatility of the proposed hallucinator-basis factorization solution, we further conduct experiments on the speech domain using Mini Speech Commands [53], which contains 8,000 audio clips for 8 command classes.
We adopt IDC [24] as the baseline and all the protocols for comparisons follow the official settings.
We compare our method with the coreset selection based Random and Herding, DSA [60], DM [61], and the IDC baseline [24].
The results in
Tab.
12 shows that our method can produce consistent improvement on the downstream test accuracy, which further reflects the generality of our method for different modalities.
Here, SPC denotes the number of speech spectrograms per class.
Robustness to Corruption: We further examine the generalization performance of our method and the baseline one on CIFAR10-C [17], the corrupted version of CIFAR10 dataset with 19 different types of corruption.
There are five corrupted levels from 1 (mildest) to 5 (severest) and we report the mean test accuracy across 19 domains on different levels.
Since the proposed method can increase the accuracy and alleviate the under-fitting problem on the original domain, which is one dominant component of cross-domain generalization [3], it can also demonstrate superior robustness in all corrupted data as demonstrated in Fig.
Also, the gap between performance using distilled dataset and original dataset becomes smaller with the increase of corrupted level, which suggests that our method improves the domain generalization ability potentially, thanks to the diverse training data composed of hallucinators and bases.
List of Hyper-Parameters:
In Tab. 13, we provide a clear view of the hyper-parameters used in this paper.
All the experiments follows these settings if not specified.
The performance of our method is insensitive to the values of these hyper-parameters as analyzed in the ablation studies in both the
[Figure] [Figure]
main paper and the appendix.
Other hyper-parameters not listed come from the adopted baseline methods and we follow their original settings.
TSNE Visualizations: To provide a better understanding on why the HaBa factorization can help on data efficiency in dataset distillation, we adopt TSNE [50] to visualize the features before the last linear layer of a teacher model trained on the original datasets.
In Fig.
11, we plot features of both original images and the distilled ones.
The results reveal that datasets restored from our hallucinators and bases can describe the original data distribution more finely, which means that the original datasets can be represented with the distilled ones with less information loss.
Given that the total numbers of parameters used for storing distilled datasets are the same, our method can improve the data efficiency significantly.
Visualizations of Factorized Results: We first provide the full results of HaBa factorization on
CIFAR10 dataset with 5 hallucinators and 10 BPC in Fig.
12, as a supplement to Fig.
4 in the main paper.
We also provide the distilled results on datasets with larger resolutions in Fig.
13 and 14 for the above 6 ImageNet subsets.
Here, we use 1 BPC and 2 hallucinators for visualization.
Through these results, we can find that bases in our scheme mainly define the basic contents, while different hallucinators may transform each basis to different appearances and styles.
Such difference is encouraged to be as large as possible to diversify the distilled data and thus improve data efficiency during the end-to-end training pipeline of DD.
[Figure] [Figure]