Active Learning is a Strong Baseline for Data Subset Selection
Abstract
Data subset selection refers to the process of finding a small subset of training data such that the predictive performance of a classifier trained on it is close to that of a classifier trained on the full training data.
A variety of sophisticated algorithms have been proposed specifically for data subset selection.
A closely related problem is the active learning problem developed for semi-supervised learning.
The key step of active learning is to identify an important subset of unlabeled data by making use of the currently available labeled data.
This paper starts with a simple observation – one can apply any off-the-shelf active learning algorithm in the context of data subset selection.
The idea is very simple – we pick a small random subset of data and pretend as if this random subset is the only labeled data, and the rest is not labeled.
By pretending so, one can simply apply any off-the-shelf active learning algorithm.
After each step of sample selection, we can “reveal” the label of the selected samples (as if we label the chosen samples in the original active learning scenario) and continue running the algorithm until one reaches the desired subset size.
We observe that surprisingly, this active learning-based algorithm outperforms all the current data subset selection algorithms on the benchmark tasks.
We also perform a simple controlled experiment to understand better why this approach works well.
As a result, we find that it is crucial to find a balance between easy-to-classify and hard-to-classify examples when selecting a subset.
Introduction
Modern deep learning have achieved unprecedented success by leveraging hyper-scale networks trained on ever-larger datasets, e.g. , GPT-3 [1], CLIP [2] and ViT [3].
However, with such a massive amount of data, practitioners often suffer from huge computational costs for model training, hyperparameter tuning and model architecture search.
For example, training GPT-3 on 45 Tera-bytes of text data takes weeks or months even with intense GPU parallelization [4].
Reducing these computational costs is of vital important because it can accelerate model development cycles, reduce the energy consumption ( e.g. , CO2 emission) [5], and even facilitate democratization of AI [6].
In this regard, many attempts have been devoted to prune the training dataset to enables efficient learning.
Data subset selection aims to reduce the size of training set by finding a core subset that generalizes on par with the full training set [7, 8].
Previous studies try to sort and select some fraction of training examples according to their difficulty for model training under the assumption that hard examples are more helpful for generalization.
To do so, various score functions have been proposed, e.g. , uncertainty-based [9], loss-based [10, 6, 11], geometry-based [12, 13, 14], and gradient-based [15, 16, 17] scores.
The prior art has proved their effectiveness by comparing the predictive performance of the model trained on the selected subsets and that of the model trained on the full data.
[Figure]
A closely related, but different, problem is active learning (AL).
AL was originally developed for the following semi-supervised learning setting.
One is given with small labeled data and large unlabeled data.
The given labeled data set is not large enough, so one would like to obtain a larger labeled dataset by labeling the unlabeled data.
However, not all unlabeled examples are equal, i.e. , some of them will be more useful than the others once labeled.
Since labeling is costly and time-consuming, it is critical to identify which examples are more useful than the others so that we can prioritize labeling important ones given a labeling budget.
AL is a principled approach to solving this problem – see [18] for a detailed survey of AL for deep learning.
While AL is developed with semi-supervised learning problems in mind, it is inherently related to the data subset selection problem in that both are concerned with finding important subsets.
In fact, it is straightforward to observe that any AL algorithm can be modified so that it can be used for the data subset problem.
The idea is strikingly simple – we start with a small random subset of data and pretend as if it is the only labeled data; train a classifier on the selected subset; augment the subset with a number of training examples selected by an AL algorithm; and repeat the training and augmenting steps until selecting the target number of subset examples.
In this paper, we show that AL can be a strong baseline for data subset selection.
As shown in Figure 1, we observe that a simple AL algorithm, such as AL (Margin) [19], outperforms every state-ofthe-art data subset selection algorithm in terms of test accuracy regardless of the selection ratios.
We demonstrate the superiority of this baseline using various architectures and datasets including
CIFAR10, CIFAR100, and a subset of ImageNet.
This is very surprising in that AL does not use the ground truth labels when selecting examples whereas the subset selection does; AL uses less information than subset selection but achieves better performance.
We also perform a simple controlled experiment to better understand why this baseline works well.
As a result, we find that it is crucial to find a balance between easy-to-classify and hard-to-classify examples when selecting a subset.
Accordingly, we contend that it now becomes an open problem to develop a new subset selection algorithm that outperforms our simple AL baseline.
Preliminary 2.1 Data Subset Selection
Problem Setup.
Let D =
( x i , y i ) } mi =1 be a given training set obtained from a joint distribution
X × Y , where m denotes the number of training examples, X and Y denote the input space and label space, respectively.
Data subset selection aims to identify the most informative subset S ⊂ D , so that the model θ S trained on S can maintain possibly close generalization performance to the model θ D trained on the entire training set D .
Related Work.
Coleman et al.
[9] shows that the uncertainty-based scores, e.g., Confidence [20],
Margin [19], and Entropy [21], can be effective metrics for subset selection in that selecting lower confident examples is more helpful for model generalization than selecting higher confident ones.
Some work used the geometric distance in the feature space to avoid selecting examples with redundant information.
Herding [12] incrementally extends the selected coreset by greedily adding a data example which can minimize the distance between the center of the coreset and that of the original training set.
kCenterGreedy [14] selects k examples that maximize the distance coverage on the entire unlabeled data.
Recently, many approaches try to directly exploit the components of deep learning with given ground-truth labels.
Forgetting [10] selects examples that are easy to be forgotten by the classifier, and it finds such samples by counting how frequently predicted label changes during several warm-up training epochs.
GraNd [6] uses the average norm of gradient vector to measure the contribution of each examples for minimizing the training loss.
GradMatch [16] and CRAIG [15] try to find an optimal coreset that its gradient can be matched with the gradient of full training set.
Glister [17] introduces a bi-level optimization framework that the outer loop is for selecting the coreset which can be solved by a greedy algorithm.
Submodular functions, such as Graph Cut, Facility Location, and Log Determinant, which measure the diversity of information, have also been shown to be useful for data subset selection [22].
2.2 Active Learning
Problem Setup.
Let U
= { x i } mi =1 be a given unlabeled set, where x i ∼ X .
Active learning aims to select the most informative examples from U and turn them into the labeled set L ⊂ D by assigning the ground-truth label y on each example x with a human oracle.
The AL process—selecting, labeling, and training—repeats for multiple rounds, say r rounds, until attaining the target number of examples.
Usually, the initial labeled set L 0 are selected randomly.
The labeled set L extends gradually by adding a newly selected set for each rounds, i.e. , L←L∪L r .
Related Work.
Numerous AL scores for measuring informativeness of examples without given the ground-truth labels have been proposed [18, 23].
One typical type is uncertainty-based score, such as soft-max Confidence [20], Margin [19], and Entropy [21], which is calculated on the final prediction probability of the model.
Some approaches measures uncertainty by using the Monte Carlo
Dropout [24] throughout multiple forward passes [25, 26].
LL [27] uses the predicted loss obtained by a small loss prediction module which is jointly learned with the target model.
Another popular type of methods is diversity-based sampling which incorporates a clustering [28, 29, 30] or greedy selection algorithm [14] to select diverse examples that well-represent the entire data distribution.
2.3 Applying Active Learning Algorithm to Data Subset Selection
Given a fully labeled dataset D , one can pretend that label information is not given and view it as
U .
Then, one can choose a small labeled random subset L 0 from D .
One can obtain the indices of chosen samples by running one step of an active learning algorithm on L 0 .
By retrieving the labels of these samples from D , one can simulate the oracle labeling process, hence obtaining L 1 .
One can repeat this process until this process reaches the desired subset size.
Experiment 3.1 Experiment Setting
Datasets.
We perform the data subset selection and active learning on three datasets; CIFAR10 [31],
CIFAR100 [31], and ImageNet-30 [32], a subset of ImageNet [33] containing 30 classes.
Algorithms.
We compare with a random selection, six subset selection algorithms, including
Margin [19], Forgetting [10], GraNd [6], kCenterGreedy [14], GraphCut [22], and Glister [17], and three AL algorithms, AL (Margin) [19], AL (Conf) [20], and AL (LL) [27].
Implementation Details.
For all datasets, we conduct the performance comparison with the selection ratios of { 10% , 20% , . . .
For data subset selection methods, following the prior work [7], a warm-up training of 10 epochs on the entire training set is preceded to calculate the difficulty scores.
For AL baselines, we randomly select 2% of training examples as the initial labeled set, and add 2% of training examples at every AL round.
We run every experiment three times and report the average of the best accuracy.
More implementation details with training configurations can be found in Appendix
A.
The code is available at https://github.com/dongmean/AL_vs_SubsetSelection .
3.2 Performance Comparison.
Table 1 shows the detailed performance of data subset selection methods and AL baselines over various selection ratios on CIFAR10, CIFAR100, and ImageNet30 with ResNet-18. See Appendix A for the result with VGG architecture.
The overall performance curves for CIFAR10 are illustrated in
Figure 1 and those for CIFAR100 and ImageNet30 are depicted in Figure 3 in Appendix B.
In general, AL baselines outperform all subset selection methods over the most selection ratios.
Specifically, AL (Margin) succeeds to maintain the full test accuracy within an error of 0.5 % until the selection ratio of 50 % for CIFAR10, 80 % for CIFAR100, and 70 % for ImageNet30.
On the other hand, the performance of subset selection methods drastically degrade when the selection ratio is very low and is sometimes even worse than Random, e.g. , for CIFAR10, every subset selection method except Forgetting is worse than Random at the selection ratios of 10 % and 20 % .
[Table]
3.3 Quick Analysis: Why a Simple AL Algorithm is Better than Data Subset Selection?
One main difference between data subset selection and AL is that AL starts from a small random initial set.
We study the effect of the random set to subset selection with a simple controlled experiment.
[Figure]
Experiment Setup.
We first train a warm-up classifier on the entire CIFAR10 for 10 epochs. Our goal is to compare the accuracy of the model trained on subset of two cases. For the first case, we randomly select 5,000 examples from CIFAR as an initial set.
Then, using the warm-up classifier, we calculate the Margin score on the remaining 45,000 examples and divide them into 9 bins, i.e. , 5,000 examples per bin, according to the score.
Next, we compose 5 subsets by only combining each odd-numbered bin with the random initial set (10,000 examples per each subset). For the second case, using the warm-up classifier, we calculate the Margin score on the entire 50,000 examples and divide them into 5 bins, i.e. , 10,000 examples per bin, according to the score.
Result.
Figure 2 shows the performance of selected subset with or without the random initial set.
With the random initial set (blue line), the performance increases according to the uncertainty level of each bin; the hardest examples are the most helpful in improving model performance.
In contrast, without the random initial set (yellow line), the model performance is not the best at the bin of highest uncertainty, which can explain the drastic performance drop of subset selection methods in low selection ratio in Section 3.2.
This indicates that the hardest examples give the greatest benefit to model generalization only when they are used together with easy examples.
Based on this finding, we further show the data subset selection can be improved by adding two main components in AL; random initial set and multi-round data selection (the result can be found in Appendix D).
Conclusion
In this work, we showed that AL can be a strong baseline for data subset selection via extensive experimental results on multiple datasets.
This is surprising as AL does not use the ground-truth label information when selecting samples.
We expect that our observation calls a new attention to developing a new data subset selection algorithm that can outperform the AL baselines. A Implementation Details
For CIFAR10 and CIFAR100 datasets, we train ResNet-18 [34] and VGG-19 [35] from scratch for 200 epochs using SGD with batch size 128, momentum 0.9, weight decay 5 × 10 − 4 , and initial learning rate 0.1 with the cosine decay scheduler.
For data augmentation, we apply random horizontal flipping and random crop with 4-pixel padding.
For ImageNet-30 dataset, we train ResNet-18 and
VGG-16 from scratch for 200 epochs using SGD with batch size 128, weight decay 5 × 10 − 4 , and initial learning rate 0.1 with the cosine decay scheduler.
For data augmentation, we resize the images to 256 × 256 , randomly crop it to 224 × 224 , and apply random horizontal flipping.
For implementation of all data subset selection algorithms, we use the code in DeepCore library 3 .
The hyperparameters for all algorithms are favorably configured following the original papers.
All algorithms are implemented with PyTorch 1.8.0 and executed on a single NVIDIA Tesla A100 GPU.
B Performance Curves on CIFAR100 and ImageNet
Figure 3 illustrates the overall performance curves of data subset selection algorithms and an AL baseline on CIFAR100 and ImageNet30 with ResNet-18.
Similar to the result in Figure 1 on CIFAR10,
AL (Margin) outperforms all the subset selection algorithms over the most selection ratios.
Among the subset selection algorithms, Margin shows the most extreme performance drop as the selection ratio becomes lower.
C Results with VGG Architecture
Table 2 shows the detailed performance of data subset selection and an AL baseline with VGGarchitecture for CIFAR10 and CIFAR100 and VGG-16 architecture for ImageNet30.
Similar to
Section 3.2, AL (Margin) wins all data subset selection methods regardless of the selection ratios.
Specifically, AL (Margin) succeeds to maintain the full test accuracy within an error of 0.5 % until the fraction ratio of 50 % for CIFAR10, 80 % for CIFAR100, and 70 % for ImageNet30.
This indicates that AL is better than data subset selection is consistent across the network architectures.
D Result of Simple Modifications of Data Subset Selection
Experiment Setup.
We make two modified versions of a data subset selection algorithm (Margin) where each of which incorporates two main components of AL; 1) random initial set, and 2) multiround selection, respectively.
For the first version, we randomly extract 2% of data examples from the entire training set and perform data subset selection with the warm-up training on the non-extracted 98% of training set.
Then, when selecting the final subset, we combine the randomly extracted examples with some fraction of hardest examples, e.g. , when the target selection ratio is 10% , we select 8% of hardest examples from the training set and combine it with already extracted 2% random examples.
For the second version, we repeatedly remove 2% of the easiest examples from the entire training set and redo the warm-up training on the remaining set whenever we remove the examples until reaching to the target selection ratio.
This version also selects the examples that are less hard than the original data subset selection with one-shot warm-up training, because the accuracy of warm-up training gradually decreases as less amount of examples is remained by the repeated example removal; the uncertainty score becomes less confident.
We train ResNet-18 [34] with the same training configuration in Appendix A
Result.
Table 3 shows the performance of two modified versions of Margin.
Overall, both versions outperforms the original Margin, which means balancing easy-to-classify and hard-to-classify is beneficial to data subset selection.
Nevertheless, each version is not yet better than the AL (Margin), which benefits both random initial set and multi-round selection.
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]