Dataset Factorization for Condensation
Anonymous Author(s)
AffiliationAddress email
Abstract
In this paper, we study dataset condensation (DC) from a novel perspective andintroduce a dataset factorization approach, termed HaBa , which is a plug-and-playstrategy portable to any existing DC baseline.
Unlike conventional DC approachesthat aim to produce condensed and representative samples, HaBa explores decomposing a dataset into two components: data Ha llucination networks and Ba ses,where the latter is fed into the former to reconstruct image samples.
The flexible combinations between bases and hallucination networks, therefore, equip thecondensed data with exponential informativeness gain, which largely increase therepresentation capability of condensed datasets.
To guide the compression results tocover the original distribution thoroughly, we further introduce a pair of adversarialcontrastive constrains on the resultant hallucination networks and bases, whichincrease the diversity of generated images and inject more discriminant informationinto the factorization.
Extensive comparisons and experiments demonstrate thatour method can yield significant improvement on downstream classification taskscompared with previous state of the arts, while reducing the total number of compressed parameters by up to 65%.
Moreover, condensed datasets by our approachalso achieve ~10% higher accuracy than baseline methods in cross-architecturegeneralization.
Introduction
The success of deep models on a variety of vision tasks, such as image classification [19, 7], objectdetection [26, 25] and semantic segmentation [29, 38], is largely attributed to the huge amount ofdata used for training.
However, the sheer amount of data introduces significant obstacles for storage,transmission, and data pre-processing.
Besides, publishing raw data inevitably brings about privacyor copyright issue in practice [30].
To alleviate these problems, dataset condensation (DC) [41]has been proposed to compress a dataset into synthetic samples, with which the training effect fordownstream models on the original dataset can be largely reduced.
Due the significant practical valueof DC, many endeavours have been made [22, 40, 36, 5] to design novel supervision signals to trainthe synthetic datasets and to improve their performances on in-the-wild test sets.
Nevertheless, there is a potential drawback in conventional settings of DC: it largely treats eachsynthetic sample independently and ignores the inter coherence and relationship between differentinstances.
As such, the information embraced by each sample, despite condensed, is by nature limited.
Using the synthetic samples for training downstream models, therefore, inevitably leads to the lossof dataset information.
Moreover, the few condensed samples are incompatible with the enormousnumber of parameters in a deep model and may yield the risk of overfitting.
To verify these potential issues, we conduct a pre-experiment on CIFAR10 dataset with 10 syntheticimages per class, using MTT [5], the SOTA solution on DC, as the baseline.
In addition to thebaseline setting, we also incorporate all the checkpoint synthetic datasets after each 100 DC iterationsin the convergent stage to train the downstream model. Since the synthetic images are fine-tuned
𝐻 !
𝐻 " ...
[Figure]
Synthetic Dataset
Performance
Matching ⟸
[Figure]
during this stage, including multiple checkpoints may somehow increase the diversity.
As a result, ityields overall lower test loss and hence better final results in downstream training, as shown in theblue and green curves in Fig. 2, which indicates that there is still room for improvement for current
DC solutions.
Nevertheless, simply involving more data samples may also increase the memoryoverhead.
This fact motivates us to ask: is it possible to encode some sample-wise relationships in adataset implicitly, instead of storing samples directly, to avoid such additional storage costs?
[Figure]
We show in this paper that, it can indeed be made possible throughreformulating the DC task as a factorization problem.
As shown in
Fig.
1, we propose a novel perspective dubbed HaBa , to factorize adataset into two compositions: data Ha llucination networks and Ba ses.
A data hallucination network, or hallucinator, can take any basis asinput and output the corresponding hallucinated image.
Supervisedby the training objective of DC, a set of hallucinators can synthesizemultiple samples from a common basis and are optimized to extracteffective relationships among different samples in original datasetsexplicitly.
In this way, information of |H|×|B| images can be includedfor a factorization result with |H| hallucinators and |B| bases viaarbitrary pair-wise combination, which improves the data efficiency oftraditional DC exponentially and thus achieves better test performance,as shown in Fig.
To further increase the informativeness of factorized results, we introduce a pair of adversarial contrastive constrains to promote sample-wise diversity.
The goal of HaBa is to minimize the correlation among images composed of different hallucinators but a commonbasis, while an adversary tries to maximize it.
Such an adversarial scheme, in turn, enforces thehallucinators to produce diversified images and recover original data distribution thoroughly.
Notably, HaBa is a versatile strategy that can be built upon existing DC baselines, since it is compatiblewith any training objective for measuring the similarity between downstream performances as shownin Fig.
1, We conduct extensive experiments to demonstrate the advantages of the proposed methodover baseline ones.
In all benchmarks and comparisons, HaBa produces significant and consistentimprovement on downstream classification tasks, while reducing the total number of compressedparameters by up to 65%.
Furthermore, it demonstrates strong cross-architecture and cross-domaingeneralization ability with accuracy improvement higher than 10%.
Our contributions are summarizedas follows:
• We study dataset factorization, a novel perspective to explore dataset condensation, andpropose a novel approach termed HaBa to frame the factorization as a hallucinator-basisproblem.•
We present a pair of adversarial contrastive objectives to further increase the data diversityand information capability.•
HaBa is a plug-and-play scheme compatible with all existing training objectives of DC andcan yield significant and consistent improvement over the state of the arts.
Related Works
The goal of dataset condensation (DC) is to optimize some synthetic images such that they arecapable to take place of original datasets for training downstream tasks, which is different fromcoreset selection [1, 6, 11, 28, 34], another branch for dataset compression, directly selecting samplesfrom raw datasets.
In this section, we provide a detailed review of previous methods in DC.
Motivated from knowledge distillation [14, 10] aiming at model compression, Wang et al .
[37]introduce the concept of dataset distillation (DD) for dataset compression.
The idea is to optimizethe synthetic images so that they can minimize loss functions of downstream tasks, where a bileveloptimization algorithm [9] is involved.
Following this routine, several works further considerlearnable labels beyond samples [3, 32].
One drawback of such bilevel optimization based methodsis that supervision signals from downstream tasks, e.g. , cross-entropy loss for classification, arenot rigorous enough for synthetic samples, which results in a large solution space with a highvariance of downstream performance.
In addition, they can hardly be scaled up for larger datasets,since the quadratic back-propagation results in significant memory consumption.
To alleviate theseproblems, Zhao et al .
[41] and several following approaches [39, 20] consider matching gradients ofa downstream model produced by synthetic samples and real images.
Most recently, Cazenavette etal .
[5] argue that single-iteration gradient matching may lead to inferior performance due to erroraccumulation across multiple steps and thereby propose to match long-range training dynamics of anexpert trained on the original dataset.
As an alternative method to profile training effects produced bydifferent sets, Nguyen et al .
[22, 23] also introduce the kernel ridge-regression approach based on the
Neural Tangent Kernel (NTK) in infinitely wide convolutional networks [16].
Apart from matching training effects, there are also methods matching data distributions betweenoriginal and synthetic datasets.
For instance, Zhao et al .
[40] propose a simple but effective Maximum
Mean Discrepancy (MMD) constrain for DC, which does not involve the training of downstreammodels and enjoys superior training efficiency.
Wang et al .
[36] propose CAFE, explicitly attemptingto align the synthetic and real distributions in the feature space of a downstream network.
Above mentioned methods are dedicated to exploring suitable training objectives and pipelinesfor DC.
However, there are few works concerning improving the data efficiency for condensedsamples.
Although Zhao et al .
[39] propose differentiable siamese augmentation (DSA) to enrich thetraining data, the augmentation operations used, e.g. , crop, flip, scale, and rotation, cannot encode anyinformation about the target datasets.
In this paper, we study the task in a factorization perspective,to factorize a dataset into two different compositions: data hallucination networks and bases.
Bothparts carry important knowledge of the raw dataset.
For downstream training, hallucinators andbases can perform arbitrary pair-wise combination, i.e. , send any basis to any hallucinator, to create atraining sample.
The idea of factorization can improve the diversity of condensed training datasetssignificantly, without introducing additional costs for storage.
It is also a versatile strategy compatiblewith all aforementioned DC methods, which will be demonstrated in the experiment part.
Methods
In this section, we elaborate our proposed method HaBa for dataset condensation (DC).
Assumethat there is an original dataset T = { ( x i , y i ) } |T | i
=1 with |T | pairs of a training sample x i and thecorresponding label
y i .
DC targets a synthetic dataset S =
{ (ˆ x i , ˆ y i ) } |S| i =1 with |S| ≪
|T | and expectsthat a model trained on S can have similar performance than that trained on T .
Traditional DC methods treat each synthetic sample independently and ignore the inner relationshipbetween different samples within a dataset, which results in poor data/information efficiency.
Focusingon such drawback, we study DC from a novel perspective and redefine it as a hallucinator-basisfactorization problem:
[Equation]
, (1) where there are |H| hallucination networks and |B| bases.
The j -th hallucinator is parameterized byθ
i and we denote it by H
θ i for 1 ≤ j ≤ |H| .
For downstream training, a training data pair (˜ x ij , ˜ y ij )is created online via sending the i -th basis, with any 1 ≤ i ≤ |B| , to the j -th hallucinator, with any1 ≤ j ≤ |H| , i.e. , ˜ x ij = H θ j (ˆ x i ) .
In this paper, the label ˜ y ij is simply taken as ˆ y i .
𝐻 !
𝐻 " ℋ
D o w n
S t r ea m
A dv .
𝒯 ℬ ℒ !"
#$ ℒ %& ℒ ’(#.
ℒ ’!
For ℬ and ℋ
For Adv.
Sample
Sample
Sample 𝜇 𝜎
Encoder
Decoder
H
[Figure]
[Figure] An overview of our method is shown in Fig. 3(Left). To go deeper into the technical details, we firststart with the introduction of our basis and data hallucination network in Sec. 3.1. Then, we proposean adversarial contrastive constrain to increase data diversity in Sec. 3.2.
Finally, we present thewhole training pipeline of the hallucinator-basis factorization for DC in Sec. 3.3.
3.1 Basis and Hallucinator
Basis: Typically, for an image classification dataset T = { ( x i , y i ) } |T | i
=1 , x i ∈ R h × w × c
and y i ∈{ 0 , 1 , ..., C − 1 } for each 1 ≤ i
≤ |T | , where each x i is a c -channel image with a resolution of h × w ,and C is the total number of classes.
In previous DC methods, the format/shape of synthetic datapairs (ˆ x, ˆ y ) has to be held the same as that of real data, so as to make sure the consistency betweeninput and output formats in the training and test time for downstream models.
By contrast, sincehallucinator networks are capable of spatial-wise and channel-wise transformation, the shape of eachˆ x i , 1 ≤ i ≤ |B| , denoted as h ′ × w ′ × c ′ , is not necessarily the same as that of original samples andthus more flexible.
And for a classification problem, we do not modify its label space in this paperfor simplicity and maintain the categorical format.
Hallucinator:
Given a basis ˆ x ∈ R h ′ × w ′ × c ′ , a data hallucination network, aims to create a newimage ˜ x ∈ R h × w × c based on ˆ x , which can be viewed as a conditional image generation problem.
Inspired by image style transfer [15], a typical conditional image generation problem, we devisean encoder-transformation-decoder based architecture for hallucinators, as shown in Fig.
3(Right).
Specifically, the encoder, denoted as enc , is composed of CNN blocks, which non-linearly maps aninput ˆ x to a feature space R h ′′ × w ′′ × c ′′ .
Then, an affine transformation with scale σ and shift µ isconducted on the derived feature, where σ and µ are treated as network parameters in this paper.
Atlast, the decoder dec under a symmetric CNN architecture with enc projects the transformed featureback to the image space.
Formally, this process can be written as:
[Equation]
where the multiplication is element-wise operation.
There are |H| hallucinators in the whole factorization pipeline and each would be trained to implicitly encode some sample-wise relations by itsnetwork parameters.
3.2 Adversarial Contrastive Constrain
Ideally, the knowledge encoded by different hallucinators should be as different/orthogonal as possibleto get the most benefits for each individual.
To instantiate such regularization, let’s consider twocomposed images ˜ x ij and ˜ x
ik from two different hallucinators H θ j and H θ k
but a common basis ˆ x i .
The divergence between ˜ x ij and ˜ x ik is expected to be large.
To measure the divergence, a featureextractor is required to map an input image to a feature space, and how to train such a feature extractorto find an appropriate feature space is of great importance.
In this paper, we formalize the training of hallucinators and the feature extractor as a min-max gamein a self-consistent manner, where the feature extractor desires to minimize the divergence between˜ x ij and ˜ x
ik while hallucinators, as well as bases, are optimized to maximize it so that the two playerscan reinforce each other.
In specific, the feature extractor, denoted as F and parameterized by ψ , istypically a CNN structure for the downstream task and we adopt features at the last hidden layer before [Figure] the output layer, denoted as F − 1 (˜ x ij ) and F − 1 (˜ x ik ) .
F is optimized to maximize the correlationbetween the two feature vectors, which can be quantified by the metric of mutual information (MI).
Inspired by the lower bound of MI [35], the objective to minimize the divergence for F is given bythe following contrastive form:
[Equation]
where τ is a scalar temperature coefficient.
For the classification problem, we can alternativelyadopt the supervised form of the contrastive loss L con. , where ˜ x uk with the same class label as ˜ x ijare also taken into consideration as positive samples in Eq.
The supervised contrastive loss canbenefit to increase the correlation of samples from the same class [17] for a more reasonable featurerepresentation.
In addition, the feature space is expected to reflect the task-specific property for a meaningfulrepresentation.
Thus, we also incorporate the task loss L task , e.g. , cross-entropy loss in classificationtasks, over the synthetic dataset as a supervision signal for F .
In this way, the overall trainingobjective for F is defined as:
[Equation]
(4) where λ con.
and λ task are hyper-parameters controlling the weight for each term.
F acts as an adversary to minimize the divergence between ˜ x ij and ˜ x
ik , while the synthetic datasetis expected to maximize it to increase data diversity.
To this ends, the similarity between F − 1 (˜ x ij )and F − 1 (˜ x ik ) becomes one loss term for hallucinator-basis factorization.
In this paper, we adopt thecosine-similarity and the objective L cos.
is given by:
[Equation]
During training, the feature extractor and the factorized components are updated alternately to playthis min-max game.
3.3 Factorization Training Pipeline
Following previous paradigms [41, 40, 5, 36], the synthetic dataset S is updated in an iterativealgorithm.
In each iteration, we randomly sample a batch of hallucinators and bases and conductpair-wise combinations.
The composed images are evaluated by the objective of dataset condensation
L DC and the similarity metric in Eq.
[Equation]
[Equation]
where hyper-parameters λ DC and λ cos.
balance the loss.
Notably, the hallucinator-basis factorization is compatible with a variety of configurations of L DCproposed by previous arts, which makes it a versatile and effective strategy for DC. In this paper, [Table]
Per Class; Ratio (%): the ratio of condensed images to whole training set.
Underline denotes results by our implementation.
we adopt the trajectories matching loss in Cazenavette et al .
[5] as L DC by default, thanks to itssuperior performance.
The basic idea is to update a downstream model from a cached checkpoint ϕ ∗ tat iteration t , using the synthetic dataset S for N times, and using the real dataset T for M timesrespectively.
Then, the updated parameters by the two cases, ˆ ϕ t + N and ϕ ∗ t + M are enforced to beconsistent:
[Equation]
where α and β are learning rates with S and T respectively.
α is learnable in the framework while βis a hyper-parameter.
We also experiment with other settings of L DC and the results can be found in
Sec. 4.
Based on the supervised signals in Eq. 6, the gradients are backward propagated to the composedimages and finally to the sampled hallucinators and bases so as to be updated using a decent algorithmsuch as SGD.
Since all the operations are differentiable, the training can be completed end-to-end.
Experiments
4.1 Datasets and Implementing Details
We conduct evaluations and comparisons of our Hallucinator-Basis factorization (HaBa) on threestandard image classification benchmarks: SVHN [21], CIFAR10, and CIFAR100 [18].
There are60,000 images for real-world digit recognition in SVHN.
For CIFAR10 and CIFAR100, there are50,000 training images in total.
The number of classes for the three datasets are 10 , 100 , and 200respectively.
All the images are under 32 × 32 resolution in 3-channel RGB format.
Followingprevious works [5], we use ZCA for image preprocessing with Kornia implementation [27] before allthe experiments.
Experiments with more datasets, including images in larger spatial scales, can befound in the supplement.
In this paper, for convenience of comparisons with prior works, we maintain the same size withimages in original datasets, i.e. , h ′ =
h , w ′ = w , and c ′ = 3 for bases.
We also experiment withother sizes of bases in Sec. 4.3.
And for hallucinators, the encoder and decoder contain 1 Conv-ReLUblocks.
The number of feature channel c ′′ is 3 .
We use 5 hallucinators by default if not specified.
Thelearning rates of hallucinators and bases, η H and η B , are the same and for the feature extractor, thelearning rate η F is 0 .
Hyper-parameters λ con.
, λ task , λ DC , and λ cos.
are set as 0 . 1 , 1 , 1 , and 0 .
[Table]
empirically.
Sensitivities of these hyper-parameters are analyzed in Sec. 4.3.
The adversary networkhas the same architecture as that for computing L DC .
In experiments on SVHN and CIFAR10, weincorporate all the bases in each iteration, while in experiments on CIFAR100, we adopt a batch sizeof 300 when the total number of bases is greater than 1,000.
We only consider random 2 hallucinatorsin one iteration for simplicity.
The maximal configuration of computational resources is 4 24GB3090 GPUs.
The GPU memory consumption is dependent on that of the baseline method for L DCand is slightly higher than it due to the computation of L cos.
and L con. .
The baseline method for L DC is MTT [5] if not specified.
Other settings related to DC hold the same as the baseline.
All thequantitative results are based on the mean and standard deviation over 5 repeated experiments.
4.2 Comparisons
Comparisons with State of the Arts: We compare HaBa with previous state of the arts for DCin standard settings, to synthesize 1, 10, and 50 images per class (IPC) respectively.
In our setting,the number of parameters in a hallucinator is slightly less than that for 2 synthetic images (5,568v.s. 6,144), while the size of a basis is equal to that of an image.
Taking the storage cost of 5hallucinators into consideration, we set the number of bases per class (BPC) as IPC minus 1 ineach IPC configuration when IPC is greater than 1, to make the comparisons as fair as possible.
Candidates are coreset based methods including Random [6, 24], Herding [4, 2], K-Center [8, 28],and Forgetting [33], meta learning based methods including DD [37] and LD [3], training matchingbased methods including DC [41], DSA [39], and MTT [5], and
distribution matching based methodsincluding DM [40] and CAFE [36].
The comparisons follow the standard protocol adopting a 3-layer
Conv-InstanceNorm-ReLU-AvgPool ConvNet with 128 channels in training and testing.
The comparison results are shown in Tab.
and we can observe that HaBa achieves state-of-the-artperformance in all datasets and settings.
Especially when the ratio of condensed images to the wholetraining set is less than 1%, our method can yield significant improvement over all the candidatemethods, which demonstrates that the scheme of hallucinator-basis factorization improves the dataefficiency for the task of dataset condensation.
Qualitative Comparisons: We visualize the factorized results by our method as well as the baselineon CIFAR10 dataset with 10 BPC in Fig.
Due to the space limitation, we only provide imagesgenerated by 2 hallucinators here.
Full results and results in more settings can be found in thesupplement.
As shown in the figure, we can find that bases mainly store some main structures andcontour information of objects.
Different hallucinators would render a basis with diverse styles anddetails.
Thanks to the dataset factorization scheme, the diversity of condensed images by our methodis higher than that by the baseline.
Building upon Different Baselines: To reflect the versatility of the insight, we implement HaBaon multiple state-of-the-art training pipelines of DC, including DC, DM, and MTT.
We evaluate theperformance of synthetic datasets on CIFAR10 and maintain the IPC of baseline methods as BPC plus1, which makes storage costs for synthetic datasets as close as possible for fairness.
As shown in Tab.2, when training and testing on ConvNet, the strategy of HaBa can make a consistent improvement [Table] [Table] over all the baselines, which demonstrates that factorization is a general idea to improve the dataefficiency in DC.
Cross-Architecture Performance:
For DC, a satisfactory condensed dataset should have similartraining effects to the original one on downstream models with arbitrary architectures.
Thus, crossarchitecture generalization performance is an important metric for DC.
We use the synthetic datasetstrained on ConvNet to train models with different structures including ResNet [12], VGG [31], and
AlexNet [19].
The results can be found in Tab. 2.
Impressively, HaBa can improve the acrossarchitecture accuracy even more significantly with a performance gain up to 17 .
The consistentand significant improvement validates the superior ability of our method to recover the original datadistribution and thus original datasets can be replaced by the synthetic ones better.
Cross-Domain Performance: We further examine the domain generalization performance of ourmethod and the baseline one on CIFAR10-C [13], the corrupted version of CIFAR10 dataset with19 different types of corruption.
There are five corrupted levels from 1 (mildest) to 5 (severest) andwe report the mean test accuracy across 19 domains on different levels.
As demonstrated in Fig.7, our method outperforms the baseline consistently in all settings and corrupted levels. Also, thegap between performance using condensed dataset and original dataset becomes smaller with theincrease of corrupted level, which suggests that our method improves the domain generalizationability potentially, thanks to the diverse training data composed of hallucinators and bases.
4.3 Ablation Studies
Loss Terms: To validate the effectiveness of the proposed adversarial contrastive constrains, wedesign ablation studies on the CIFAR10 dataset over three loss terms: L cos.
in Eq.
5, L con.
in Eq.
3,and the task-specific loss L task .
Through the results in Tab. 3, we can find that deleting any one ofthem would hurt the performance.
We also experiment with involving L con.
for downstream training,to enforce the similarity among images composed of different hallucinators and a common basis.
Observed from the last row of Tab. 3, the performance can be further improved, since L con.
helps therepresentation learning of related samples [17].
Note that we do not use this loss term for downstreamtraining in other experiments for fair and standard comparison.
We examine the sensitivities of hyper-parameters λ cos.
and λ con.
used to balance the weights ofloss terms L cos.
and L con.
respectively in Fig.
The results are evaluated on the CIFAR10 datasetwith 10 BPC.
We can observe that the overall performance is not sensitive to the selection of thesehyper-parameters and our method makes a consistent improvement over the baseline with 11 IPC.
Class-Independent Hallucinators v.s. Shared Hallucinators:
In the default setting of HaBa, eachclass maintains a certain number of bases independently and all the classes share the same set ofhallucinators.
But what about the case that hallucinators are also made class-independent?
We studythis problem experimentally in Tab. 4.
Given the same BPC, class-independent hallucinators canindeed somehow improve the performance when there are fewer synthetic samples, e.g. , 1 BPC.
However, when BPC is higher, equipping each class with an independent set of hallucinators wouldyield inferior performance, even worse than the baseline when BPC is 50.
There are probablytwo reasons: (1) shared hallucinators across all the classes extract global information of the wholedataset, which is also of great importance to recover the original data distribution; and (2) the classindependent case would make the number of hallucinators 10 times for the CIFAR10 dataset, whichleaves a heavy burden for the optimization process.
Thus, as indicated in Tab. 4, a better solution isto make room for more bases using the memory allocated to store class-independent hallucinatorsinitially, which would result in data efficiency.
[Table] [Figure]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure]
Number of Channels Used by Basis: By default, the shape of a basis is the same as that of a realimage, which is generally in RGB 3-channel format.
In fact, in Fig. 6(Left), we also verify that it isalso possible to use single-channel basis, which can reduce the memory cost by nearly 2/3 withouthurting the performance too much.
Interestingly, if the memory cost is held the same, we can chooseto use 3 times BPC to store single-channel bases, rather than 3-channel ones.
This would yieldimpressive improvement on the test accuracy when BPC is small.
Note that for baseline results, IPCis set as the corresponding BPC plus 1.
Number of Hallucinators: We study the impact of the number of hallucinators, i.e. , |H| , in Fig.6(Right).
We can observe that when BPC is small, including more hallucinators is helpful for theperformance.
Nevertheless, when BPC is 10 or 50, the performance would not improve with morehallucinators when |H| > 10 .
One reason is that when |H| is large, the sampling of hallucinators ineach iteration is sparse, which makes the joint optimization of all the hallucinators more difficult.
Data Augmentation:
The similarity between our hallucinator set and data augmentation lies thatboth of them can contribute to generating more samples and increasing the diversity.
However, theessential difference is that our hallucinators are optimized to encode sample-wise relationships ina dataset, while data augmentation is based on some prior and heuristic knowledge of images.
Bydefault, both our method and the baseline adopt the data augmentation strategy DSA [39].
To studythe relationship between the two schemes experimentally, we attempt to remove DSA from baselineand our method and report the corresponding results in Tab. 5.
The evaluation is on CIFAR10 with
IPC for baseline and 10 BPC for ours.
Through the results, we can find that (1) our method withoutdata augmentation can also outperform the baseline method with augmentation significantly, whichmeans that the mechanism of HaBa can benefit the condensation task more with the learning of globalinformation of a dataset in hallucinators; and (2) with data augmentation, our performance can befurther improved, which indicates that HaBa and DSA work in different manners.
Conclusions
This paper proposes a novel hallucinator-basis factorization method dubbed HaBa for dataset condensation (DC).
It uses hallucinators to encode inner relations between different samples in originaldatasets, which can largely improve the data efficiency of condensed results.
To diversify theknowledge captured by different hallucinators, a pair of adversarial contrastive constrains is furtherintroduced.
Extensive evaluations and comparisons are conducted on multiple benchmark datasets,which demonstrate that HaBa is capable of significantly improving the performance of downstreammodels trained on the synthetic dataset, using only 35% cost of memory for storage.
Moreover, itis a versatile strategy that is compatible with different configurations of DC frameworks and yieldsconsistent improvement.
In our further work, we will focus on enhancing the efficiency of HaBa.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] The training efficiency can be further improved.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Please refer to
the supplement.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] Please refer to Sec. 4.1.(c)
Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] We run all the experiments 5 times and report the meanand standard deviation of the performance.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please refer to Sec. 4.1.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]