C OVERAGE - CENTRIC C ORESET S ELECTION FOR H IGH P RUNING R ATES
Anonymous authors
Paper under double-blind review
A BSTRACT
One-shot coreset selection aims to select a subset of the training data, given a pruning rate, that can achieve high accuracy for models that are subsequently trained only with that subset.
State-of-the-art coreset selection methods typically assign an importance score to each example and select the most important examples to form a coreset.
These methods perform well at low pruning rates; but at high pruning rates, they have been found to suffer a catastrophic accuracy drop , performing worse than even random coreset selection.
In this paper, we explore the reasons for this accuracy drop both theoretically and empirically.
We extend previous theoretical results on the bound for model loss in terms of coverage provided by the coreset.
Inspired by theoretical results, we propose a novel coverage-based metric and, based on the metric, find that coresets selected by importance-based coreset methods at high pruning rates can be expected to perform poorly compared to random coresets because of worse data coverage.
We then propose a new coreset selection method, Coreset-centric Coreset Selection (CCS), where we jointly consider overall data coverage based on the proposed metric as well as importance of each example.
We evaluate CCS on four datasets and show that they achieve significantly better accuracy than state-of-the-art coreset selection methods as well as random sampling under high pruning rates, and comparable performance at low pruning rates.
For example, CCS achieves 7.04% better accuracy than random sampling and at least 20.16% better than popular importance-based selection methods on CIFAR10 with a 90% pruning rate.
I NTRODUCTION
In this paper, we consider one-shot coreset selection that aims to select a small subset of the training data that can later be used to train future models while retaining high accuracy (Coleman et al., 2019;
Toneva et al., 2018)
One-shot coreset selection is important because full datasets can be massive in many applications and training on them can be computationally expensive.
Coresets also help in identifying representative examples that can summarize a large dataset for the purpose of subsequent training.
A favored way to select coresets is to assign an importance score to each example and select more important examples to form the coreset (Paul et al., 2021; Sorscher et al., 2022).
Unfortunately, current SOTA methods for one-short coreset selection suffer a catastrophic accuracy drop under high pruning rates (Guo et al., 2022; Paul et al., 2021).
For example, for the CIFAR10, a SOTA method (forgetting score (Toneva et al., 2018)) achieves 95.36% accuracy with a 30% pruning rate (outperforming random selection), but that accuracy drops to only 34.03% at a 90% pruning rate (which is significantly worse than random coreset selection).
This accuracy drop is currently unexplained and limits the extent to which coresets can be practically reduced in size.
Our main contribution is to provide both theoretical and emperical insights into reasons for the catastrophic accuracy drop and to propose a new coreset selection algorithm that overcomes this issue.
We extend classical geometrical set cover problem to a density-based distribution cover problem and provide theoretical bounds on model loss as a function of properties of a coreset.
Furthermore, based on theoretical analysis, we propose a novel metric AUC pr to quantify data coverage (Section 3.1).
With the proposed metric, we show that coresets selected by SOTA methods at high pruning rates have much worse data coverage than random pruning, suggesting a linkage between poor data coverage of SOTA methods and poor accuracy at high pruning rates.
We note that data coverage has also been studied in active learning setting, but techniques from active learning do not trivially extend to one-shot coreset selection.
We discuss the similarity and differences to active learning in Section 5.
We then propose a novel algorithm, Coverage-centric Coreset Selection (CCS), that addresses catastrophic accuracy drop by improving data coverage.
Different from SOTA methods that prune unimportant (easy) examples first, CCS applies stratified sampling across importance scores to achieve better coverage (Section 3.3).
We find that CCS overcomes catastrophic accuracy drop at high pruning rates, based on evaluation on multiple datasets (CIFAR10, CIFAR100 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011),
CINIC
(Darlow et al., 2018)), and ImageNet (Deng et al., 2009).
CCS outperformed SOTA methods by a significant margin at high pruning rates.
Furthermore, CCS outperformed random selection at high pruning rates, which SOTA methods failed to do.
For example, at a 90% pruning rate, on
CIFAR 10 (CIFAR100), CCS achieves 7 .
04% ( 10 . 83% ) better accuracy than random sampling and at least 20 .
63% ) better than other baseline methods 1 (Section 4).
At low pruning rates,
CCS achieves comparable performance to baselines, outperforming random selection.
To summarize, our contributions are as follows:
We extend the geometric set cover problem to a density-based distribution cover problem and provide a theoretical bound on model loss as a function of properties of a coreset.
Inspired by the theoretical results, we propose a novel metric AUC pr to quantify data coverage for a coreset (Section 3.1).
Based on this metric, we show that SOTA coreset selections tend to have poor data coverage at high pruning rates – worse than random selection, thus suggesting a linkage between coverage and observed catastrophic accuracy drop.
To improve coverage in coreset selection, we propose a Coverage-centric Coreset Selection (CCS) method where we we jointly consider both coverage and importance of each example based on a given importance metric.
Unlike SOTA methods, which generally pick highest importance examples, CCS uses stratified sampling across importance scores.
We evaluate CCS on five different datasets and compare with five difference baselines with multiple importance metrics, and we find that CCS significantly outperforms baselines methods as well as random coreset selection at high pruning rates.
P RELIMINARIES 2. O NE - SHOT C ORESET SELECTION
Consider a classification task with a training dataset containing N examples drawn i.i.d.
from an underlying distribution P .
We denote the training dataset as S =
{ ( x i , y i ) } Ni =1 , where x i is the data, and y i is the ground-truth label.
The goal of one-shot coreset selection is to select a training subset given a pruning rate α before training to maximize the accuracy of models trained on this subset, which can be formulated as the following optimization problem (Sener & Savarese, 2017):
[Equation]
where l is the loss function, and h S is the model trained with a labelled dataset S .
SOTA methods typically assign an importance score (also called a difficulty score or importance metric) to each example and preferably select more important (difficult) examples to form the coreset.
One proposed importance score is the Forgetting score (Toneva et al., 2018), which is defined as the number of times an example is incorrectly classified after having been correctly classified earlier during model training.
Another is area under the margin (AUM) (Pleiss et al., 2020), which is a data difficulty metric that identifies mislabeled data.
Another importance metric, EL2N score (Paul et al., 2021) estimates data difficulty by the L2 norm of error vectors.
SOTA algorithms based on these metrics all prune examples with low importance first (Sorscher et al., 2022).
As discussed in (Sorscher et al., 2022), harder examples usually contain more information and are treated as more important data and thus more desirable for inclusion in a coreset.
2. C ATASTROPHIC ACCURACY DROP WITH HIGH PRUNING RATES
[Figure]
Although SOTA coreset methods (Toneva et al., 2018;
Coleman et al., 2019; Paul et al., 2021) report encouraging performance at low pruning rates, a catastrophic accuracy drop is observed as pruning rate increases (Guo et al., 2022; Paul et al., 2021).
For instance, as shown in
Figure 1, SOTA methods based on forgetting number and
AUM outperform random sampling when the pruning rate is less than 70% but considerably underperform random sampling at 90% pruning rate, achieving only 34 .
03% and 28 .
06% accuracy, respectively, which are much worse than 79 .
04% for random sampling.
We hypothesize that this accuracy drop is a result of bad data coverage caused by the biased pruning of importance-based methods.
In this paper, we conduct an in-depth study on this catastrophic accuracy drop and propose a novel coreset selection method that overcomes the catastrophic accuracy drop at high pruning rates.
T HE COVERAGE OF CORESETS
We start by studying data coverage provided by different coresets.
We first extend the classical geometric set cover to a density-based distribution cover problem and propose a novel metric to compare the data coverage of different coresets (Section 3.1).
We then show that biased pruning on easy data leads to poor data coverage of coresets with high pruning rates (Section 3.2).
Finally, we design a novel coverage-centric coreset selection method based on stratified sampling (Section 3.3).
3. D ENSITY - BASED PARTIAL COVERAGE
To compare the data coverage of different coreset methods, we need to quantitatively measure how well a dataset S covers a distribution P .
In classical geometric set cover setting (Fowler et al., 1981;
Sener & Savarese, 2017), we say a set S ′ is a r -cover of another set S , when a set of r -radius balls centered at each element in S ′ covers the entire S .
The radius r can be used as a metric to measure coverage of S ′ on S .
To measure how well a set covers a distribution, we extend the classical geometric set cover to the density-based distribution cover .
Instead of covering a set, we study the covering on a distribution
P µ and consider probability density in different areas of the input space.
Instead of taking a complete cover, we introduce the cover percentage p to describe how a set covers different percentages of a distribution to better understand the trade-off between cover radius r and cover percentage p :
Definition 1 ( p -partial r -cover) .
Given a metric space ( X, d ) , a set S ⊂ X is a p -partial r -cover for a distribution P µ on the space X if:
[Equation]
where B d ( x , r ) = { x ′ ∈ X : d ( x, x ′ ) ≤ r } is a r -radius ball whose center is x , p measures coverage (which we will usually express as a percentage and refer to as percentage coverage ), and µ is the probability measure 2 corresponding to probability density function P µ .
The probability measure µ represents the density information in the input space.
For a model trained on a r -cover coreset, Sener & Savarese (2017) proved a bounded risk on the entire training dataset.
In Theorem 1 below, we extend their result to a p -partial r -cover.
The proof of Theorem 1 relies on the same assumptions as in (Sener & Savarese, 2017), in particular Lipschitzcontinuity loss function and zero training error on the coreset.
The zero training error assumption may not always be realistic, but Sener & Savarese (2017) found that the resulting bounds still hold and we rely on the same assumption.
The proof of theorem 1 can be found in the Appendix C. µ : P µ (
) dx
= dµ ( x ) .
Theorem 1.
Given n i.i.d.
samples drawn from P µ as S
= { x i , y i } i ∈ [ n ] where y i ∈ [ C ] is the class label for example x i , a coreset S
which is a p -partial r -cover for P µ on the input space X , and an ϵ > 1 − p , if the loss function l ( · , y, w ) is λ l -Lipschitz continuous for all y , w and bounded by
L, regression function is λ η -Lipschitz, and l ( x , y ; h S ′ )
= 0 , ∀ ( x , y ) ∈ S ′ , then with probability at least 1 − ϵ :
[Equation]
Theorem 1 shows that the model trained on a p -partial r -cover coreset S ′ has a bounded risk on the entire training dataset S .
There are several implications of Theorem 1: 1) A model trained on a p -partial r -cover coreset has a training risk bounded with the covering radius r and an additional term.
This term goes to zero as n increases to infinity.
Also, we want a small ϵ , a large p and a small radius r to get a high probability small bound on the loss.
The probability that Equation 2 holds is bounded by cover percentage p (since 1 − ϵ < p ).
This indicates that the bound in Theorembecomes meaningless (low probability) when the cover percentage p is too small.
3) Theorem 1 is a more general form of Theorem 1 in (Sener & Savarese, 2017); if we set p = 1 (i.e., complete cover), these two theorems are the same.
Definition 1 shows that, for a given set S and a distribution P µ , the cover percentage p increases with the cover radius r .
Conversely, given a cover percentage p and a set S , we can get a minimum cover radius r for the S to achieve p percent coverage on distribution P µ .
For a given set S and a distribution P µ , we can plot a p - r curve to represent this relationship.
[Figure]
The dataset without any pruning has the lowest curve (blue).
Forgetting curve (green) and random curve (orange) have a crossover around p = 80% .
In practice, the distribution P µ is unknown.
So, we estimate the cover percentage p for a given subset S of the training dataset and a radius r by measuring the fraction of the test dataset that is covered 3 .
To assess coverage for a radius r , we use the L2 distance between activations of the final convolutional layer of a model trained with the entire dataset as the distance metric d , as suggested in (Babenko & Lempitsky, 2015; Sharif Razavian et al., 2014; Hsieh et al., 2018).
Figure 2 shows the p - r curves for several subsets of CIFAR10 training dataset: (1) entire dataset (no pruning); (2) random coreset at 90% pruning rate (i.e., random 10% of the training data); and (3) coreset selected based on forgetting score as the importance metric, as proposed by Toneva et al.
As expected, the required radius r increases to get the desired cover percentage p for all coresets.
Also coreset with no pruning has the lowest curve, as expected.
In general, lower curves are better and we expect models based on the corresponding datasets to have lower loss.
But, we find that there is crossover between forgetting curve (green) and random curve (orange) around p = 80% , making a comparison less obvious.
We propose to use area under the p - r curve, AUC pr , as a proxy metric to assess the quality of a coreset selection strategy.
Lower values of AUC pr suggest better coverage by the coreset.
We note that AUC pr is the expected minimum distance between examples following the underlying distribution P µ to those in the coreset, as stated in the following proposition (proved in Appendix C):
Proposition 1.
Given a metric space ( X, d ) , a distribution P µ , cover percentage p , a set S , and f r ( p ) :
[0 , 1] → [0 , + ∞ ] representing the mapping between p and r , if f r is Riemannintegrable, the AUC of the p - r curve AUC pr ( S )
= (cid:82) 1 0 f r ( p ) dp
=
E x ∼ P µ [ d ( S, x )] , where d ( S, x ) =
min x ′ ∈ S d ( x ′ , x ) .
In practice, we can assess AUC pr with test set D test : AUC pr ( S ) =
E x ∈ D test [min x ′ ∈ S d ( x ′ , x )] .
We also note that this expected minimum distance (and thus AUC) is related to r in Theorem 1; lower values of AUC pr are preferred for lower loss and thus higher model accuracy.
3. C OVERAGE ANALYSIS ON CORESETS
[Table]
[Figure]
Different colors stands for different coreset selected methods.
A larger circle size indicates a higher pruning ratio on the
Smaller AUC pr often ends up with better accuracy.
With high pruning rates (larger balls), forgetting and AUM have a higher AUC pr and worse accuracy.
Table 1 shows the AUC pr for different CIFAR10 coresets with different pruning rates.
With a 30% or 50% pruning rate, forgetting and AUM have similar AUC pr to random sampling.
However, with a pruning rate larger than 70% , forgetting and AUM have larger AUC pr than random pruning, suggesting that forgetting and AUM can be expected to have a worse accuracy than random pruning.
The reason behind higher AUC pr for forgetting and AUM is that they prune easy examples, which tend to be located in the high-density area of distribution since more common examples often tend to be well-separated easier examples for classifiers.
Low pruning rate coresets still contain plenty of easy examples for all methods; thus coverage is not a significant factor at low pruning rates.
However, with high pruning rates, coresets can completely exclude examples from the high-density areas, reducing coverage and raising AUC pr , and thus potentially raising loss and reducing accuracy.
For the data points in Table 1, Figure 3 plots model accuracy and AUC pr .
Size of the ball at a point represents pruning rate for that point – larger ball size indicates a larger pruning rate.
The plot generally confirms our intuitions from Section 3.1 that higher AUC pr values tend to lead to lower model accuracy.
3. M ETHODOLOGY : C OVERAGE - CENTRIC CORESET SELECTION
Based on our density-based distribution setting in Section 3.1, if the sampling budget is limited, easy examples in the high-density area provide more coverage than hard examples in the low-density area. SOTA methods prune low-importance (easy) data from the high-density area, which contributes to a larger AUC pr value and thus possibly lower model accuracy at high pruning rates.
Inspired by this insight, we propose a novel coreset selection method called Coverage-centric Coreset Selection (CCS), presented in Algorithm 1.
CCS is still based on existing importance scores.
However, unlike SOTA methods that simply select important examples first, CCS combines stratified sampling on top of importance scores and pruning hard strategy to improve coverage and lower the value of AUC pr . {+
 Compared to SOTA methods, CCS still assigns the sampling budget to the high-density area containing easy examples at high-pruning rates, which provides larger coverage to the underlying distribution. Compared to random sampling, CCS assigns a larger sampling budget to the low-density area, where hard examples are informative for training. Select stratum fewest examples */ stratum straum+}
Stratified sampling in CCS first divides the dataset into different non-overlapping strata based on importance scores.
Each stratum consists of a fixed-length score range, but may have different numbers of examples.
We fix an initial budget on the number of examples to be chosen from each strata, based on the desired pruning rate.
But, if a stratum has fewer examples than the budget, the remaining budget is evenly assigned to other strata.
[Figure]
Thus stratified sampling random sam pling in that examples from “dense” stratum are less likely to be sampled than examples in “sparse”
Also, with a low pruning rate, stratified sampling tends to first discard data from lowimportance strata similar to SOTA methods.
But, at high pruning rates, stratified sampling diverges from SOTA methods in that low-importance strata still get budget to guarantee data coverage.
Before doing stratified sampling, CCS prunes β percent “hard” examples based on importance scores.
This is based on two insights: (1) Mislabeled examples often also have higher importance scores (Swayamdipta et al., 2020) but do not benefit accuracy.
As the pruning rate increases, the percentage of mislabeled data in the coreset also rises, which hurts model accuracy.
(2) When data is really scarce, we need more data from higher-density areas to get better coverage.
Pruning hard examples help allocate more budget to high-density strata.
In practice, we use grid search to find β .
We discuss how β impacts training in Section 4.2 and further details on selecting β in Appendix B.
In Figure 4, we show an example on CIFAR10 to compare the fraction of data selected between a
SOTA method (AUM) and CCS (also based on AUM) for 70% pruning rate.
The data selected by the SOTA method
[Table]
E VALUATION
We next evaluate CCS against five baseline methods (random, entropy, forgetting, EL2N
[Table]
and AUM) on five datasets (CIFAR10, CIFAR100, SVHN, CINIC10, and ImageNet). In total, we trained over 800 models for performance comparison.
We found that CCS outperforms other baselines at high pruning rates while achieving comparable accuracy at low pruning rates.
We also conduct an ablation study (Section 4.2) to better understand how hyperparameters and different components influence the accuracy.
Due to space limits, we include experimental setting in Appendix.
B. All training is repeated 5 times with different random seeds to calculate mean accuracy with standard deviation.
Baselines.
We compare CCS with five baselines, with the latter four being SOTA methods: 1)
Random : Select examples with uniform random sampling to form a coreset.
2) Entropy (Coleman et al., 2019):
Select highest entropy examples for coreset, since entropy reflects uncertainty of training examples and thus more importance to training.
Forgetting score (Toneva et al., 2018):
Select examples with highest Forgetting scores, which is the number of times an example is incorrectly classified after being correctly classified earlier during model training; 4) EL2N (Paul et al., 2021):
Select examples with highest EL2N scores, which estimate data difficulty or importance by the L2 norm of error vectors.
5) Area under the margin (AUM) (Pleiss et al., 2020): Select examples with highest AUM scores, which measures the probability gap between the target class and the next largest class across all training epochs.
A larger AUM suggests higher difficulty and importance.
SOTA baseline methods prune less important examples first when selecting coresets, based on a chosen importance metric.
We show results of using CCS with three previously-proposed importance metrics: Forgetting score, EL2N, and AUM that work well at low pruning rates.
4. C ORESET PERFORMANCE COMPARISON
We found all three CCS variants to significantly outperform SOTA schemes as well as random selection.
Table 2 shows the evaluation results on the CIFAR10 dataset for 5 pruning rates.
We observe that the four SOTA methods tend to have similar or higher accuracy than random for 50% or lower pruning rates.
At high pruning rates (e.g., 80% and 90%), however, they consistently have a worse performance than random.
In contrast, all three CCS-based schemes achieve comparable accuracy at low pruning rates, while trouncing their counterparts at high pruning rates.
For example, at 90% pruning rate, best CCS-scheme achieve 7.64% better accuracy than random sampling and 20.16% better than the best-performing SOTA method.
We also noticed that the choice of importance metric impacted CCS accuracy, but only marginally.
CCS worked well for all three importance metrics.
Findings on CIFAR100 are similar (see Table 3).
CIFAR100 is a more challenging dataset than
CIFAR10, since CIFAR100 only contains 500 images per class.
This can cause fewer redundant easy images in the dataset.
CCS achieves comparable accuracy to SOTA methods at low pruning rates (up to 50%) and much higher accuracy as pruning rate increases.
At 90% pruning rate, the accuracy of best CCS method is 10 .
83% higher than random sampling and 26 .
63% higher than best-performing
SOTA method.
We found similar findings on SVHN, CINIC10 and ImageNet (detailed results in the
Appendix D).
CCS outperformed all baselines at high pruning rates on all these datasets.
[Table]
4. A BLATION STUDY & ANALYSIS
[Figure] We first observe that, when k = 1 and β = 0 , CCS is equivalent to random sampling, i.e., random sampling is a special case of CCS.
This guarantees that CCS, with appropriate choice of hyperparameters, will not underperform random sampling.
In practice, larger values of k tend to be more useful to get the stratified sampling effect and to outperform random.
Also, as shown in Figure 5(a), for large k values, we found that the number of strata is not a sensitive hyperparameter.
We then study the hard cutoff rate β .
From Figure 5(b), we observe that the optimal hard cutoff rate β goes up as the pruning rate increases.
This is expected since a larger hard cutoff rate helps shift more budget to high-density strata (see Section 3.3), When the total data budget is limited, better coverage on high-density strata helps improve model performance.
We also observe that the optimal β is 0 when the pruning rates is 30% and 50% .
A possible reason is that the percentage of mislabeled data is low in CIFAR10 (Pleiss et al., 2020), and deep neural network is robust to a small portion of noisy data (Rolnick et al., 2017).
Grid search for β can introduce extra costs in selecting effective coresets.
However, we note that our paper shows the possibility of achieving better accuracy than random sampling at a high pruning rate.
We leave how to efficiently finetune β for future work.
[-Forgetting 95 .95 .
-]
[-Stratified only 95 .95 .
-]
[-CCS (Stratified + Prune hard) 
-]
Next, we study the impact of stratified sampling and pruning hard examples on model accuracy.
We measure the forgetting score for CIFAR10 and then apply only stratified sampling or only pruning hard examples to select the coreset, respectively, in Table 4.
By comparing stratified sampling only with the vanilla forgetting method (pruning easy first), we observe that applying stratified sampling improves accuracy compared to the vanilla forgetting method, but it is still inferior to random sampling when we prune 80% , or 90% fraction of examples.
Sorscher et al.
(2022) discussed that pruning hard data is a better strategy than pruning easy data when the original entire data is scarce.
In our evaluation, we observe that pruning hard data outperforms pruning easy data at high pruning rates, but still has worse accuracy than CCS.
[Table]
We also considered using a proportional sampling strategy to improve coverage that samples examples proportional after mapping importance score to a probability for one dataset.
Unfortunately, the strategy underperformed random sampling at high pruning rates (results in Appendix D.5).
Beating random is crucial for any strategy since random is the simplest selection strategy.
This evaluation suggested it is non-trivial to beat random at high pruning rates, even for a strategy that improves coverage.
CCS achieves that remarkable result – it consistently beats random for five different sets.
R ELATED WORK
Besides the coreset methods based on entropy Coleman et al.
(2019), forgetting score Toneva et al.
(2018), AUM Pleiss et al.
(2020), and EL2N scores Paul et al.
(2021) discussed in Section 2.1,
Sorscher et al.
(2022) use k -means to estimate importance scores, and Sener & Savarese (2017) apply greedy k -center to choose the coreset with good data coverage.
However, k -means and kcenter algorithms need to calculate the distance matrix between examples, which leads to an O ( n 2 ) time complexity and are expensive to scale to large datasets.
Data selection is also studied in active learning and some methods recognize importance of diversity or coverage in data selection (Ash et al., 2019;
Citovsky et al., 2021; Beluch et al., 2018; Hsu &
Lin, 2015; Chang et al., 2017).
While our scheme shares the idea of improving coverage with these methods, one-shot coreset selection philosophically differs from dataset selection in active learning: one-shot coreset selection aims to select a model-agnostic coreset for training new models from scratch, while active learning aims to select a subset that, given the current state of the model, is to improve the model in the next training round.
For example, BADGE (Ash et al., 2019), a diversitybased selection method in active learning, selects a subset by performing k-means++ on the gradient embedding based on the latest model and then repeats that for subsequent rounds during training.
Another significant difference is that in active learning, total number of examples queried across all rounds of training can be much larger than the size of a one-shot coreset.
For instance, in experimental results of Ash et al.
(2019), BADGE queried a different set of 10K (20%) examples on every round for the CIFAR-10 dataset.
The total number of different examples queried across all rounds is even larger.
In contrast, CCS finds a fixed small subset (e.g. 10% data) that can be used to train new models from scratch while achieving high accuracy.
Another popular coreset selection area is the coreset selection for classical clustering problems such as k-means and k-median (Feldman et al., 2020; Feldman & Langberg, 2011; Cohen-Addad et al., 2021;
Huang et al., 2019).
However, these coreset methods are usually designed based on the good mathematical property of classical clustering.
For example, importance sampling is a common practice for coreset for clustering (Huang & Vishnoi, 2020), but it does not generalize well in the deep learning classification scenario.
C ONCLUSION
We conduct an in-depth study to understand the catastrophic accuracy drop issue of coreset selection.
By extending classical geometric set cover to density-based partial cover, we propose a novel data coverage metric and show that existing importance-based coreset selections lead to poor data coverage, which in turn degrades model performance.
Based on this observation, we design a novel
Coverage-centric Coreset Selection (CCS) method based on stratified sampling to improve the data coverage.
Our evaluation results on four datasets show that our method outperforms other methods under high pruning rates and achieves comparable performance for low pruning rates.
REPRODUCIBILITY STATEMENT
In the supplemental, we include a zip file containing our code and corresponding instructions.
We also include detailed experimental setting in the appendix.
A O VERVIEW
This appendix provides details on our experimental setting, theoretical analysis, and additional evaluation results.
In Section B, we introduce the detailed setup of our experiment to improve the reproducibility of our paper.
In Section C, we provide proof for Theorem 1 and Proposition 1.
At last, in Section D, we present evaluation results on SVHN and CINIC10.
We also conduct a study on the transferability of the coreset selected by CCS.
B D ETAILED E XPERIMENTAL SETTING
CIFAR10 and CIFAR100 (Krizhevsky et al., 2009).
We use ResNet18 (He et al., 2016) as the network architecture for CIFAR10/CIFAR100.
For all coresets with different pruning rates, we train models for 40 , 000 iterations with a 256 batch size (about 200 epochs over the entire dataset).
We use the SGD optimizer ( 0 . 9 momentum and 0 . 0002 weight decay) with a 0 .
1 initial learning rate.
The learning rate scheduler is the cosine annealing learning rate scheduler (Loshchilov & Hutter, 2017) with a 0 . 0001 minimum learning rate.
We use a 4-pixel padding crop and a randomly horizontal flip as data augmentation.
SVHN (Netzer et al., 2011).
We use ResNet18 (He et al., 2016) as the network architecture.
For all coresets with different pruning rates, we train models for 30 , 000 iterations with a 256 batch size (about 100 epochs over the entire dataset).
We use the SGD optimizer ( 0 . 9 momentum and 0 .weight decay) with a 0 .
1 initial learning rate.
The learning rate scheduler is the cosine annealing learning rate scheduler (Loshchilov & Hutter, 2017) with a 0 . 0001 minimum learning rate.
CINIC
(Darlow et al., 2018).
The CINIC10 dataset contains 270 , 000 images in total and is evenly split to three subsets: training, valid, and test.
Guided by (Darlow et al., 2018), we combine the training and valid set to form a large training dataset containing 180 , 000 images and measure the test accuracy with the test set (containing 90 , 000 examples).
We use ResNet18 (He et al., 2016) as the network architecture.
For all coresets with different pruning rates, we train models for 70 , 000 iterations with a 256 batch size (about 100 epochs over the entire dataset).
We use the SGD optimizer ( 0 . 9 momentum and 0 . 0002 weight decay) with a 0 .
1 initial learning rate.
The learning rate scheduler is the cosine annealing learning rate scheduler (Loshchilov & Hutter, 2017) with a 0 . 0001 minimum learning rate.
We use a 4-pixel padding crop and a randomly horizontal flip as data augmentation.
ImageNet (Deng et al., 2009).
We use ResNet34 (He et al., 2016) as the network architecture.
For all coresets with different pruning rates, we train models for 300 , 000 iterations with a 256 batch size (about 60 epochs over the entire dataset).
We use the SGD optimizer ( 0 . 9 momentum and 0 .weight decay) with a 0 .
1 initial learning rate.
The learning rate scheduler is the cosine annealing learning rate scheduler (Loshchilov & Hutter, 2017).
Difficulty score calculation .
For CIFAR10 and CIFAR100, we train a model with the entire dataset for 200 epochs to calculate the forgetting score and AUM and use the last checkpoint to calculate entropy for each training example.
We use the first 10 epoch’s training information to calculate the
EL2N score.
For SVHN and CINIC10, we train a model with the entire dataset for 100 epochs to calculate the forgetting score and AUM and use the last checkpoint to calculate entropy for each training example.
For ImageNet, we train the entire dataset for 90 epochs to calculate the forgetting and AUM score.
We use the first 10 epoch’s training information to calculate the EL2N score.
Coverage-centric methods setting .
We set the number of strata k = 50 for all datasets and pruning rates.
We use the grid search with 0 .
1 step size to find an optimal hard cutoff rate β for different datasets and pruning rates.
For each dataset, we list the optimal β value for every α as follows in the format of tuple ( α, β ) .
For CIFAR10, the optimal setting is (30% , 0) , (50% , 0) , (70% , 10%) , (80% , 10%) , (90% , 30%) .
For CIFAR100, the optimal setting is (30% , 10%) , (50% , 20%) , (70% , 20%) , (80% , 40%) , (90% , 50%) .
For SVHN, the optimal setting is (30% , 0) , (50% , 0) , (70% , 10%) , (90% , 10%) , (95% , 20%) .
For CINIC10, the optimal setting is (30% , 0) , (50% , 0) , (70% , 10%) , (80% , 10%) , (90% , 20%) .
For ImageNet, the optimal setting is (30% , 0) , (50% , 10%) , (70% , 20%) , (80% , 20%) , (90% , 30%) .
Each model is trained on a NVIDIA 2080TI GPU.
We also include the implementation in the supplementary material.
C T HEORETICAL ANALYSIS
We include full proofs of Theorem 1 and Proposition 1 in this section.
C. P ROOF OF T HEOREM
Our proof flow is similar to the proof of Theorem in Sener & Savarese (2017).
Lemma 1.
Berlind & Urner (2015) Fix some p, p ′ ∈ [0 , 1] and y ′ ∈ { 0 , 1 } .
Then,
[Equation]
Theorem 1.
Given n i.i.d.
samples drawn from P µ as S
= { x i , y i } i ∈ [ n ] where y i ∈ [ C ] is the class label for example x i , a coreset S
which is a p -partial r -cover for P µ on the input space X , and an ϵ > 1 − p , if the loss function l ( · , y, w ) is λ l -Lipschitz continuous for all y , w and bounded by
L, regression function is λ η -Lipschitz, and l ( x , y ; h S ′ )
= 0 , ∀ ( x , y ) ∈ S ′ , then with probability at least 1 − ϵ :
[Equation]
Proof.
We first bound E
y
i ∼ η ( x i )
[ l ( x i , y i ; h S ′ )] , where η ( x ) is the output of the model, and we represent { y i = k } ∼ η k ( x i ) with y
i ∼ η k ( x i ) .
We have a condition which states that there exists an x j in δ ball around x i such that x j has 0 loss.
Since S ′ is a p -partial r -cover for the distribution, for x i ∼ µ , with the probability p , there is x j ∈ S ′ such that | x i − x j | ≤ r and x j has 0 loss.
The following inequality holds with the probability p :
[Equation]
By applying Lemma 1 and 0 training loss condition on x j
[Equation]
we have:
[Equation]
By applying the Lipschitz property of regression function and loss function
[Equation]
we have:
[Equation]
By applying the loss bound and the p -partial r -cover constraint
[Equation]
we have:
[Equation]
At last, we have the following inequality with probability p :
[Equation]
Also, by the Hoeffding’s Bound (Hoeffding, 1994), with probability at least 1 − γ
[Equation]
we have:
[Equation]
By combining Equation 4 and Equation 5, we have that with probability at least 1 − ϵ :
[Equation]
with p > 1 − ϵ .
C. P ROOF OF P ROPOSITION
Proposition 1.
Given a metric space ( X, d ) , a distribution P µ , cover percentage p , a set S , and f r ( p ) :
[0 , 1] → [0 , + ∞ ] representing the mapping between p and r , if f r is Riemannintegrable, the AUC of the p - r curve AUC pr ( S )
= (cid:82) 1 0 f r ( p ) dp
=
E x ∼ P µ [ d ( S, x )] , where d ( S, x ) =
min x ′ ∈ S d ( x ′ , x ) .
Proof.
Since f r is Riemann-integrable, we can calculate (cid:82) 1 0 f r ( p ) dp , which is AUC pr , with Riemann sum.
Specifically, by evenly dividing [0 , 1] to 0 = p 0 < p 1 < p 2 < ... <
p n = 1 , we have:
[Equation]
Since p i
= P µ ( d ( S , x ) ≤ f r ( p i )) , we have
[Equation]
We know f r is an increasing function and
f r (0) = 0 .
Let ∆ i = f r ( p i ) − f r ( p i − 1 ) , we have:
[Equation]
As n → ∞ , we have ∆ i → 0 , so we have:
[Equation]
which is the expectation of the minimum distance to data in S .
So we conclude that AUC pr ( S ) is the expectation of the minimum distance to data in S .
D A DDITIONAL EVALUATION RESULTS D. E VALUATION RESULTS ON SVHN AND CINIC
In this section, we present our comparison experiment on SVHN and CINIC10 in Table 5 and
Table 6, respectively.
Since SVHN classification is simpler than other classification tasks, we also study the performance of coreset with a 95% pruning rate.
We find similar comparison results that we discussed in Section 4.1.
For both datasets, we observe that CCS achieves comparable accuracy with low pruning rates, but higher accuracy than random and other baseline coreset selection methods.
D. C ORESET COVERAGE ANALYSIS
Figure 6 and Table 7 compare AUC pr between CCS with forgetting score and the vanilla forgetting methods (pruning easy first) on the CIFAR10 dataset.
We observe that CCS has a lower AUC pr and better accuracy with the same pruning rate, which suggests that the proposed coverage-centric method does improve the data coverage of the coresets.
[Table]
against other baseline methods.
Pruning rate is the fraction of examples that are removed from the original dataset.
We report the testing accuracy of the model trained on the pruned dataset.
The model trained with the entire dataset has 89 .
97% accuracy.
When we prune out 30% or 50% fraction of data, CCS achieves comparable accuracy.
With a 70% , 80% , or 90% pruning rate, CCS outperforms baseline methods by a large margin.
[Table]
which suggests better data coverage.
[Table] [Table]
[Figure]
D. T RANSFERABILITY OF CORESETS
Similar to Coleman et al.
(2019), we also conduct a coreset transferability study.
We train a
ResNet18 model with the entire CIFAR10 dataset and use the forgetting score to select coresets with different pruning rates.
Then we train the ResNet50 model with these coresets.
The evaluation results are shown in Table 8.
We find that coresets show good transferability across these two architectures.
D. A DDITIONAL EVALUATION ON I MAGE N ET
In this part, we present our comparison experiment on ImageNet, a larger scale dataset compared to the other four datasets evaluated, in Table 9.
Due to the large computational cost for the ImageNet training, we only apply CCS on the AUM importance score, and similar to the prior work Sorscher et al.
(2022), we train each model for one time.
From Table 9, we notice that CCS outperforms all baseline methods at high pruning rates and achieves comparable performance at low pruning rates.
We find that EL2N and AUM still experience the catastrophic accuracy drop at high pruning rates.
Specifically, compared to random sampling, CCS achieves a 5 .
02% improvements at a 90% pruning rate.
The evaluation results on
ImageNet indicate that CCS is also scaled to the large dataset.
Forgetting performs very similarly to random at high pruning rates.
We think the reason behind this is that the training dynamics of ImageNet are only collected for 90 epochs, which causes an example to be forgotten at most 45 times.
Considering the huge number of the ImageNet dataset, many example can share the same forgetting number.
Since coreset selection will randomly sample examples when the scores of these examples are the same, forgetting method for ImageNet adds more randomness to the coreset selection.
D. D ISCUSSION ON IMPORTANCE SAMPLING BASED METHOD
Another potential way to resolve the coverage issue is the importance sampling based method: sample each example in the data set with a probability based to its importance score.
Examples with less importance are assigned smaller sampling probability and examples with larger importance are assigned larger sampling probability.
We present preliminary experiments on CIFAR10, the AUM as the underlying importance metric, and using a softmax function to map importance scores to probabilities.
Unfortunately, experimental results in Table 10 show that the above scheme failed to beat random at high pruning rates.
It did beat the SOTA method at high pruning rates.
Not beating random is a big deal, since random selection is the simplest possible strategy for constructing a coreset.
The above result also suggests that it is not easy to design a coreset selection scheme that beats random selection at high pruning rates.
Our proposed scheme, CCS does that on five of the datasets
[Table] [Table]
that we evaluated it on, and it is the first coreset selection scheme to achieve that, as far as we are aware.
We note that the relationship between data importance score and good-property sampling probability is not obvious.
Unlike many statistical machine learning methods like KNN or k-means, which usually have a good math property, data importance scores in deep learning classification are usually heuristically calculated (as we discussed at the beginning of Section 4), which makes it non-trivial to figure out an effective way to calculate probabilities based on heuristic importance scores.
For the results in Table 10, we used the popular softmax function.
More specifically, since a smaller
AUM score indicates a larger importance, we calculated the sampling probability p
i for the
i th example as follows:
[Equation]
where s i is the AUM score for each example, s max is the maximum possible AUM score, and N is the number of examples.
We do not rule out that a better-designed importance sampling technique exists, say, by rethinking the relationship between importance score and probability of selection.
CCS uses stratified sampling, which we found to work really well.
CCS thus provides a good baseline for any further work in this area.