Dataset Distillation via Factorization
Anonymous Author(s)
AffiliationAddress email
Abstract
In this paper, we study dataset distillation (DD), from a novel perspective andintroduce a dataset factorization approach, termed HaBa , which is a plug-andplay strategy portable to any existing DD baseline.
Unlike conventional DDapproaches that aim to produce distilled and representative samples, HaBa exploresdecomposing a dataset into two components: data Ha llucination networks and Ba ses, where the latter is fed into the former to reconstruct image samples.
Theflexible combinations between bases and hallucination networks, therefore, equipthe distilled data with exponential informativeness gain, which largely increase therepresentation capability of distilled datasets.
To guide the compression results toextract more discriminative information, we further introduce a pair of adversarialcontrastive constraints on the resultant hallucination networks and bases, whichincrease the diversity of generated images and inject more discriminant informationinto the factorization.
Extensive comparisons and experiments demonstrate thatour method can yield significant improvement on downstream classification taskscompared with previous state of the arts, while reducing the total number ofcompressed parameters by up to 65%.
Moreover, distilled datasets by our approachalso achieve ~10% higher accuracy than baseline methods in cross-architecturegeneralization.
Introduction
The success of deep models on a variety of vision tasks, such as image classification [19, 7], objectdetection [26, 25], and semantic segmentation [29, 38] is largely attributed to the huge amountof data used for training.
However, the sheer amount of data introduces significant obstacles forstorage, transmission and data pre-processing.
Besides, publishing raw data inevitably brings aboutprivacy or copyright issue in practice [30].
To alleviate these problems, Wang et al . [37] pioneer theresearch of dataset distillation (DD), to distill a large dataset into a synthetic one with only a limitednumber of samples, with which training efforts for downstream models on the original dataset can belargely reduced.
Due the significant practical value of DD, many endeavours have been made on thisarea [41, 39, 40, 36, 17, 5] to design novel supervision signals to train the synthetic datasets and tofurther improve their performances.
Nevertheless, there is a potential drawback in conventional settings of DD: it largely treats eachsynthetic sample independently and ignores the inter coherence and relationship between differentinstances.
As such, the information embraced by each sample, despite distilled, is by nature limited.
Using the synthetic samples for training downstream models, therefore, inevitably leads to the loss ofdataset information.
Moreover, the few distilled samples are incompatible with the enormous numberof parameters in a deep model and may yield the risk of overfitting.
To verify these potential issues, we conduct a pre-experiment on CIFAR10 dataset with 10 syntheticimages per class, using MTT [5], the SOTA solution on DD, as the baseline.
In addition to thebaseline setting, we also incorporate all the checkpoint synthetic datasets after each 100 DD iterations
𝐻 !
𝐻 " ...
[Figure]
Synthetic
[Figure]
Performance
Matching ⟸
[Figure]
in the convergent stage to train the downstream model. Since the synthetic images are fine-tunedduring this stage, multiple checkpoints can be viewed as related but different, which may somehowincrease the diversity.
As a result, it yields overall lower test loss and hence better final results indownstream training, as shown in the blue and green curves in Fig. 2, which indicates that current
DD solutions can be potentially improved by leveraging some sample-wise relationships to diversifythe distilled data.
Nevertheless, simply involving more data samples may also increase the memoryoverhead.
This fact motivates us to ask: is it possible to encode some shared relationships in a datasetimplicitly, instead of storing samples directly, to avoid such additional storage costs?
[Figure]
We show in this paper that, it can indeed be made possible throughreformulating the DD task as a factorization problem.
As shown in
Fig.
1, we propose a novel perspective dubbed HaBa , to factorize adataset into two compositions: data Ha llucination networks and Ba ses.
A data hallucination network, or hallucinator, can take any basis asinput and output the corresponding hallucinated image.
Supervisedby the training objective of DD, a set of hallucinators can synthesizemultiple samples from a common basis and are optimized to extracteffective relationships among different samples in original datasetsexplicitly.
In this way, information of |H|×|B| images can be includedfor a factorization result with |H| hallucinators and |B| bases viaarbitrary pair-wise combination, which improves the data efficiencyof traditional DD exponentially. As shown in the yellow curve in Fig.2, with the same budget on the storage, our strategy achieves bettertest performance compared with the MTT baseline.
To further increase the informativeness of factorized results, we introduce a pair of adversarialcontrastive constraints to promote sample-wise diversity.
The goal of HaBa is to minimize thecorrelation among images composed of different hallucinators but a common basis, while an adversarytries to maximize it.
Such an adversarial scheme, in turn, enforces the hallucinators to producediversified images and increases the amount of useful information.
Notably, HaBa is a versatile strategy that can be built upon existing DD baselines, since it is compatiblewith any training objective for measuring the similarity between downstream performances as shownin Fig.
1, We conduct extensive experiments to demonstrate the advantages of the proposed methodover baseline ones.
In all benchmarks and comparisons, HaBa produces significant and consistentimprovement on training downstream models, while reducing the total number of compressedparameters by up to 65%.
Furthermore, it demonstrates strong cross-architecture generalizationability with accuracy improvement higher than 10%.
Our contributions are summarized as follows:
• We study dataset factorization, a novel perspective to explore dataset distillation, and proposea novel approach termed HaBa for hallucinator-basis factorization.•
We present a pair of adversarial contrastive objectives to further increase the data diversityand information capability.
• HaBa is a plug-and-play scheme compatible with all existing training objectives of DD andcan yield significant and consistent improvement over the state of the arts.
Related Works
The goal of dataset distillation (DD) is to optimize a smaller synthetic dataset such that it is capable totake place of original one for training downstream tasks, which is different from coreset selection [1,6, 11, 28, 34], another branch for dataset compression, directly selecting samples from raw datasets.
In this section, we provide a detailed review of previous methods in DD.
Motivated from knowledge distillation [13, 10] aiming at model compression, Wang et al .
[37]introduce the concept of dataset distillation for dataset compression.
The idea is to optimize thesynthetic images so that they can minimize loss functions of downstream tasks, where a bileveloptimization algorithm [9] is involved.
Following this routine, several works further considerlearnable labels beyond samples [3, 32].
Subsequently, Zhao et al .
[41] and several followingapproaches [39, 20] consider matching gradients of a downstream model produced by syntheticsamples and real images, which improve the performance significantly.
Most recently, Cazenavette etal .
[5] argue that single-iteration gradient matching may lead to inferior performance due to erroraccumulation across multiple steps and thereby propose to match long-range training dynamics of anexpert trained on the original dataset.
As an alternative method to profile training effects produced bydifferent sets, Nguyen et al .
[22, 23] also introduce the kernel ridge-regression approach based on the
Neural Tangent Kernel (NTK) in infinitely wide convolutional networks [15].
Apart from matching training effects, there are also methods matching data distributions betweenoriginal and synthetic datasets.
For instance, Zhao et al .
[40] propose a simple but effective Maximum
Mean Discrepancy (MMD) constraint for DD, which does not involve the training of downstreammodels and enjoys superior training efficiency.
Wang et al .
[36] propose CAFE, explicitly attemptingto align the synthetic and real distributions in the feature space of a downstream network.
Above mentioned methods are dedicated to exploring suitable training objectives and pipelines for DD.
However, there are few works concerning improving the data efficiency for distilled samples.
Although Zhao et al .
[39] propose differentiable siamese augmentation (DSA) to enrich the trainingdata, the augmentation operations used, e.g. , crop, flip, scale, and rotation, cannot encode anyinformation about the target datasets.
In this paper, we study the task in a factorization perspective,to factorize a dataset into two different compositions: data hallucination networks and bases.
Bothparts carry important knowledge of the raw dataset.
For downstream training, hallucinators and basescan perform arbitrary pair-wise combination, i.e. , sending any basis to any hallucinator, to createa training sample.
The idea of factorization can improve the diversity of distilled training datasetssignificantly, without introducing additional costs for storage.
It is also a versatile strategy compatiblewith all aforementioned DD methods, which will be demonstrated in the experiment part.
Methods
In this section, we elaborate our proposed method HaBa for dataset distillation (DD).
Assumethat there is an original dataset T = { ( x i , y i ) } |T | i
=1 with |T | pairs of a training sample x i and thecorresponding label
y i .
DD targets a synthetic dataset S =
{ (ˆ x i , ˆ y i ) } |S| i =1 with |S| ≪
|T | and expectsthat a model trained on S can have similar performance than that trained on T .
Traditional DD methods treat each synthetic sample independently and ignore the inner relationshipbetween different samples within a dataset, which results in poor data/information efficiency.
Focusingon such drawback, we study DD from a novel perspective and redefine it as a hallucinator-basisfactorization problem:
[Equation]
(1) where there are |H| hallucination networks and |B| bases.
The j -th hallucinator is parameterized byθ
i and we denote it by H
θ i for 1 ≤ j ≤ |H| .
For downstream training, a training data pair (˜ x ij , ˜ y ij )is created online via sending the i -th basis, with any 1 ≤ i ≤ |B| , to the j -th hallucinator, with any1 ≤ j ≤ |H| , i.e. , ˜ x ij = H θ j (ˆ x i ) .
In this paper, the label ˜ y ij is simply taken as ˆ y i .
An overview of our method is shown in Fig. 3(Left).
To go deeper into the technical details, we firststart with the introduction of our basis and data hallucination network in Sec. 3.1.
Then, we proposean adversarial contrastive constraint to increase data diversity in Sec. 3.2.
Finally, we present thewhole training pipeline of the hallucinator-basis factorization for DD in Sec. 3.3.
𝐻 !
𝐻 " ℋ
D o w n
S t r ea m
A dv .
𝒯 ℬ ℒ !"
#$ ℒ %% ℒ &’#.
ℒ &’).
For ℬ and ℋ
For Adv.
Sample
Sample
Sample 𝜇 𝜎
Encoder
Decoder
H
[Figure]
We adopt batch size 2 here for clarity
Architecture of in detail.
3.1 Basis and Hallucinator
Basis: Typically, for an image classification dataset T = { ( x i , y i ) } |T | i
=1 , x i ∈ R h × w × c
and y i ∈{ 0 , 1 , ..., C − 1 } for each 1 ≤ i
≤ |T | , where each x i is a c -channel image with a resolution of h × w ,and C is the total number of classes.
In previous DD methods, the format/shape of synthetic datapairs (ˆ x, ˆ y ) has to be held the same as that of real data, so as to make sure the consistency betweeninput and output formats in the training and test time for downstream models.
By contrast, sincehallucinator networks are capable of spatial-wise and channel-wise transformation, the shape of eachˆ x i , 1 ≤ i ≤ |B| , denoted as h ′ × w ′ × c ′ , is not necessarily the same as that of original samples andthus more flexible.
And for a classification problem, we do not modify its label space in this paperfor simplicity and maintain the categorical format.
Hallucinator:
Given a basis ˆ x ∈ R h ′ × w ′ × c ′ , a data hallucination network, aims to create a newimage ˜ x ∈ R h × w × c based on ˆ x , which can be viewed as a conditional image generation problem.
Inspired by image style transfer [14], a typical conditional image generation problem, we devisean encoder-transformation-decoder based architecture for hallucinators, as shown in Fig.
3(Right).
Specifically, the encoder, denoted as enc , is composed of CNN blocks, which non-linearly maps aninput ˆ x to a feature space R h ′′ × w ′′ × c ′′ .
Then, an affine transformation with scale σ and shift µ isconducted on the derived feature, where σ and µ are treated as network parameters in this paper.
Atlast, the decoder dec under a symmetric CNN architecture with enc projects the transformed featureback to the image space.
Formally, this process can be written as:
[Equation]
where the multiplication is element-wise operation.
There are |H| hallucinators in the whole factorization pipeline and each would be trained to implicitly encode some sample-wise relations by itsnetwork parameters.
3.2 Adversarial Contrastive Constraint
Ideally, the knowledge encoded by different hallucinators should be as different/orthogonal as possibleto get the most benefits for each individual.
To instantiate such regularization, let’s consider twocomposed images ˜ x ij and ˜ x
ik from two different hallucinators H θ j and H θ k
but a common basis ˆ x i .
The divergence between ˜ x ij and ˜ x ik is expected to be large.
To measure the divergence, a featureextractor is required to map an input image to a feature space, and how to train such a feature extractorto find an appropriate feature space is of great importance.
In this paper, we formalize the training of hallucinators and the feature extractor as a min-max gamein a self-consistent manner, where the feature extractor desires to minimize the divergence between˜ x ij and ˜ x
ik while hallucinators, as well as bases, are optimized to maximize it so that the two playerscan reinforce each other.
In specific, the feature extractor, denoted as F and parameterized by ψ , istypically a CNN structure for the downstream task and we adopt features at the last hidden layer beforethe output layer, denoted as F − 1 (˜ x ij ) and F − 1 (˜ x ik ) .
F is optimized to maximize the correlationbetween the two feature vectors, which can be quantified by the metric of mutual information (MI).
Inspired by the lower bound of MI [35], the objective to minimize the divergence for F is given by the following contrastive form:
[Equation]
where τ is a scalar temperature coefficient.
For the classification problem, we can alternativelyadopt the supervised form of the contrastive loss L con. , where ˜ x uk with the same class label as ˜ x ijare also taken into consideration as positive samples in Eq.
The supervised contrastive loss canbenefit to increase the correlation of samples from the same class [16] for a more reasonable featurerepresentation.
In addition, the feature space is expected to reflect the task-specific property for a meaningfulrepresentation.
Thus, we also incorporate the task loss L task , e.g. , cross-entropy loss in classificationtasks, over the synthetic dataset as a supervision signal for F .
In this way, the overall trainingobjective for F is defined as:
[Equation]
where λ con.
and λ task are hyper-parameters controlling the weight for each term.
F acts as an adversary to minimize the divergence between ˜ x ij and ˜ x
ik , while the synthetic datasetis expected to maximize it to increase data diversity.
To this ends, the similarity between F − 1 (˜ x ij )and F − 1 (˜ x ik ) becomes one loss term for hallucinator-basis factorization.
In this paper, we adopt thecosine-similarity and the objective L cos.
is given by:
[Equation]
During training, the feature extractor and the factorized components are updated alternately to playthis min-max game.
3.3 Factorization Training Pipeline
Following previous paradigms [41, 40, 5, 36], the synthetic dataset S is updated in an iterativealgorithm.
In each iteration, we randomly sample a batch of hallucinators and bases and conductpair-wise combinations.
The composed images are evaluated by the objective of dataset distillation
L DD and the similarity metric in Eq.
[Equation]
where hyper-parameters λ DD and λ cos.
balance the loss.
Notably, the hallucinator-basis factorization is compatible with a variety of configurations of L DDproposed by previous arts, which makes it a versatile and effective strategy for DD.
In this paper,we adopt the trajectories matching loss in Cazenavette et al .
[5] as L DD by default, thanks to itssuperior performance.
The basic idea is to update a downstream model from a cached checkpoint ϕ ∗ tat iteration t , using the synthetic dataset S for N times, and using the real dataset T for M timesrespectively.
Then, the updated parameters by the two cases, ˆ ϕ t + N and ϕ ∗ t + M are enforced to beconsistent: [Equation]
[Equation]
where α and β are learning rates with S and T respectively.
α is learnable in the framework while βis a hyper-parameter.
We also experiment with other settings of L DD and the results can be found in
Sec. 4.
Based on the supervised signals in Eq. 6, the gradients are backward propagated to the composedimages and finally to the sampled hallucinators and bases so as to be updated using a decent algorithmsuch as SGD.
Since all the operations are differentiable, the training can be completed end-to-end.
[Table]
Per Class; Ratio (%): the ratio of distilled images to whole training set.
Underline denotes results by our implementation.
Experiments
4.1 Datasets and Implementing Details
We conduct evaluations of our method on three standard image classification benchmarks: SVHN [21], CIFAR10, and CIFAR100 [18].
There are 60,000 images for real-world digit recognition in SVHN.
For CIFAR10 and CIFAR100, there are 50,000 training images in total.
The number of classes forthe three datasets are 10 , 10 , and 100 respectively.
All the images are under 32 × 32 resolution in3-channel RGB format.
Following previous works [5], we use ZCA for image preprocessing with Kornia implementation [27] before all the experiments.
Experiments with more datasets, includingimages in larger spatial scales, can be found in the supplement.
In this paper, for convenience of comparisons with prior works, we maintain the same size withimages in original datasets, i.e. , h ′ =
h , w ′ = w , and c ′ = 3 for bases.
We also experiment withother sizes of bases in Sec. 4.3.
For hallucinators, the encoder and decoder contain 1 Conv-ReLUblocks.
The number of feature channel c ′′ is 3 .
We use 5 hallucinators by default.
The learning ratesof hallucinators and bases, η H and η B , are the same and for the feature extractor, the learning rateη F is 0 .
Hyper-parameters λ con.
, λ task , λ DD , and λ cos.
are set as 0 . 1 , 1 , 1 , and 0 .
1 empirically.
Sensitivities of these hyper-parameters are analyzed in Sec. 4.3.
The adversary network has the samearchitecture as that for computing L DD .
In experiments on SVHN and CIFAR10, we incorporate allthe bases in each iteration, while in experiments on CIFAR100, we adopt a batch size of 300 whenthe total number of bases is greater than 1,000.
We only consider random 2 hallucinators in oneiteration for simplicity.
The maximal configuration of computational resources is 4 24GB 3090 GPUs.
The GPU memory consumption is dependent on that of the baseline method for L DD and is slightlyhigher than it due to the computation of L cos.
and L con. .
The baseline method for L DD is MTT [5]if not specified.
Other settings related to DD hold the same as the baseline.
All the quantitativeresults are based on the mean and standard deviation over 5 repeated experiments.
To make sure faircomparisons, the dataset size in our method is equal to the number of bases |B| and the hallucinatorsare treated as parameterized data augmentors working online in downstream training, just as generaldata augmentations, which means that the dataset size does not increase compared with the baselines.
4.2 Comparisons
Comparisons with State of the Arts: We compare HaBa with previous state of the arts for DDin standard settings, to synthesize 1, 10, and 50 images per class (IPC) respectively.
In our setting,the number of parameters in a hallucinator is is approximately equal to that for 2 synthetic images,while the size of a basis is equal to that of an image.
Taking the storage cost of 5 hallucinatorsinto consideration, we set the number of bases per class (BPC) as IPC minus 1 in each IPC con- [Figure] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] figuration when IPC is greater than 1, to make the comparisons as fair as possible. Candidates arecoreset based methods including Random [6, 24], Herding [4, 2], K-Center [8, 28], and Forgetting [33], meta learning based methods including DD [37] and LD [3], training matching basedmethods including DC [41], DSA [39]
[Table]
and MTT [5], and distribution matching based methodsincluding DM [40] and CAFE [36].
The comparisons follow the standard protocol adopting a 3-layer
Conv-InstanceNorm-ReLU-AvgPool ConvNet with 128 channels in training and testing.
The comparison results are shown in Tab.
and we can observe that HaBa achieves state-of-the-artperformance in all datasets and settings.
Especially when the ratio of distilled images to the wholetraining set is less than 1%, our method can yield significant improvement over all the candidatemethods, which demonstrates that the scheme of hallucinator-basis factorization improves the dataefficiency for the task of dataset distillation.
Qualitative Comparisons: We visualize the factorized results by our method as well as the baselineon CIFAR10 dataset with 10 BPC in Fig.
Due to the space limitation, we only provide imagesgenerated by 2 hallucinators here.
More results can be found in the supplement.
As shown in thefigure, we can find that bases mainly store some main structures and contour information.
Differenthallucinators would render a basis with diverse styles and details.
Thanks to the dataset factorizationscheme, the diversity of distilled images by our method is higher than that by the baseline.
Building upon Different Baselines: To reflect the versatility of the insight, we implement HaBaon multiple state-of-the-art training pipelines of DD, including DC, DM, and MTT.
We evaluatethe performance of synthetic datasets on CIFAR10 and maintain the IPC of baseline methods as BPC plus 1, which makes storage costs for synthetic datasets as close as possible for fairness.
Asshown in Tab. 2, when training and testing on ConvNet, the strategy of HaBa can make a consistentimprovement over all the baselines, which demonstrates that factorization is a general idea to improvethe data efficiency in DD.
Cross-Architecture Performance:
For DD, a satisfactory distilled dataset should have similartraining effects to the original one on downstream models with arbitrary architectures.
Thus, crossarchitecture generalization performance is an important metric for DD.
We use the synthetic datasetstrained on ConvNet to train models with different structures including ResNet [12], VGG [31], and
AlexNet [19].
The results can be found in Tab. 2.
Benefiting from the increased data diversity, HaBa [Table] [Table] can improve the across-architecture accuracy significantly with a performance gain up to 17 .
The consistent and significant improvement validates the superior ability of our method to capture theinformative features and thus original datasets can be replaced by the synthetic ones better.
Comparisons under the Same Number of Final Images: In the default comparison protocol, wecompare our method with the baselines using the same budget of storage, where our method can storeinformation of exponentially more images than the baselines with the same number of parameters. In this part, we also examine the performance of HaBa under the condition that the number of finalimages, i.e. , |H| × |B| , is equal to that used by the baseline. Intuitively, given that the objectivefunctions of our method and the baseline are the same exactly, the performance of the baseline can beviewed as an upper bound of ours, since there are significantly less parameters in our method to carrythe information of final images in this case. Therefore, we first remove the term L cos.
from the lossfunction of DD in Eq. 6 to guarantee a consistent optimization objective with the baseline. Then,we compare the performance of HaBa and the baseline using 10, 20, 30, 40, and 50 final imagesrespectively. Here, the number of hallucinators |H| is 2 and the number of bases is thus half of thenumber of final images.
As shown in the red and green curves in Fig. 6, performance of the baselinecan be well approximated by ours with only half of the number of parameters, especially when thenumber of images is relatively large. Remarkably, with the proposed adversary contrastive constraint,our method can even outperform the baseline consistently, as shown in the blue curve, which furtherdemonstrates the effectiveness of the proposed solution.
4.3 Ablation Studies
Loss Terms: To validate the effectiveness of the proposed adversarial contrastive constraints, wedesign ablation studies on the CIFAR10 dataset over three loss terms: L cos.
in Eq.
5, L con.
in Eq.
3,and the task-specific loss L task .
Through the results in Tab. 3, we can find that deleting any one ofthem would hurt the performance.
We also experiment with involving L con.
for downstream training,to enforce the similarity among images composed of different hallucinators and a common basis.
Observed from the last row of Tab. 3, the performance can be further improved, since L con.
helps therepresentation learning of related samples [16].
Note that we do not use this loss term for downstreamtraining in other experiments for fair and standard comparison.
We examine the sensitivities of hyper-parameters λ cos.
and λ con.
used to balance the weights ofloss terms L cos.
and L con.
respectively in Fig.
The results are evaluated on the CIFAR10 datasetwith 10 BPC.
We can observe that the overall performance is not sensitive to the selection of thesehyper-parameters and our method makes a consistent improvement over the baseline with 11 IPC.
Class-Independent Hallucinators v.s. Shared Hallucinators:
In the default setting of HaBa, eachclass maintains a certain number of bases independently and all the classes share the same set ofhallucinators.
But what about the case that hallucinators are also made class-independent?
We studythis problem experimentally in Tab. 4.
Given the same BPC, class-independent hallucinators canindeed somehow improve the performance when there are fewer synthetic samples, e.g. , 1 BPC.
However, when BPC is higher, equipping each class with an independent set of hallucinators wouldnot benefit the performance.
There are probably two reasons: (1) shared hallucinators across allthe classes extract global information of the whole dataset, which is also of great importance tocapture the discriminative information; and (2) the class-independent case would make the number ofhallucinators 10 times for the CIFAR10 dataset, which leaves a heavy burden for the optimizationprocess.
Thus, as indicated in Tab. 4, a better solution is to make room for more bases using thememory allocated to store class-independent hallucinators initially, which would result in moresatisfactory data efficiency.
[Figure] [Figure]
[Figure] [Table]
[Figure]
[Table] [Table]
[Figure]
[Table] [Table]
Basis: By default, the shape of a basis is the same as that of a realimage, which is generally in RGB 3-channel format.
In fact, in Fig. 7(Left), we also verify that it isalso possible to use single-channel basis, which can reduce the memory cost by nearly 2/3 withouthurting the performance too much.
Interestingly, if the memory cost is held the same, we can chooseto use 3 times BPC to store single-channel bases, rather than 3-channel ones.
This would yieldimpressive improvement on the test accuracy when BPC is small.
Note that for baseline results, IPCis set as the corresponding BPC plus 1.
Number of Hallucinators: We study the impact of the number of hallucinators, i.e. , |H| , in Fig.7(Right).
We can observe that when BPC is small, including more hallucinators is helpful for theperformance.
Nevertheless, when BPC is 10 or 50, the performance would not improve with morehallucinators when |H| > 10 .
One reason is that when |H| is large, the sampling of hallucinators ineach iteration is sparse, which makes the joint optimization of all the hallucinators more difficult.
Data Augmentation:
The similarity between our hallucinator set and data augmentation lies thatboth of them can contribute to generating more samples and increasing the diversity.
However, theessential difference is that our hallucinators are optimized to encode sample-wise relationships ina dataset, while data augmentation is based on some prior and heuristic knowledge of images.
Bydefault, both our method and the baseline adopt the data augmentation strategy DSA [39].
To studythe relationship between the two schemes experimentally, we attempt to remove DSA from baselineand our method and report the corresponding results in Tab. 5.
The evaluation is on CIFAR10 with
IPC for baseline and 10 BPC for ours.
Through the results, we can find that (1) our method withoutdata augmentation can also outperform the baseline method with augmentation significantly, whichmeans that the mechanism of HaBa can benefit the dataset distillation task more with the learning ofglobal information of a dataset in hallucinators; and (2) with data augmentation, our performance canbe further improved, which indicates that HaBa and DSA work in different manners.
Conclusions
This paper proposes a novel hallucinator-basis factorization method dubbed HaBa for dataset distillation (DD).
It uses hallucinators to encode inner relations between different samples in originaldatasets, which can largely improve the data efficiency of distilled results.
To diversify the knowledgecaptured by different hallucinators, a pair of adversarial contrastive constraints is further introduced.
Extensive evaluations and comparisons are conducted on multiple benchmark datasets, which demonstrate that HaBa is capable of significantly improving the performance of downstream models trainedon the synthetic dataset, using only 35% cost of memory for storage.
Moreover, it is a versatilestrategy that is compatible with different configurations of DD frameworks and yields consistentimprovement.
In our further work, we will focus on enhancing the efficiency of HaBa.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Please refer to the supplement.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Please refer to
the supplement.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] Please refer to Sec. 4.1.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We run all the experiments 5 times and report the meanand standard deviation of the performance.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please refer to Sec. 4.1.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]