L EVERAGING I MPORTANCE W EIGHTS IN S UBSET S ELECTION
A BSTRACT
We present a subset selection algorithm designed to work with arbitrary model families in a practical batch setting.
In such a setting, an algorithm can sample examples one at a time but, in order to limit overhead costs, is only able to update its state (i.e. further train model weights) once a large enough batch of examples is selected.
Our algorithm, IWeS, selects examples by importance sampling where the sampling probability assigned to each example is based on the entropy of models trained on previously selected batches.
IWeS admits signiﬁcant performance improvement compared to other subset selection algorithms for seven publicly available datasets.
Additionally, it is competitive in an active learning setting, where the label information is not available at selection time.
We also provide an initial theoretical analysis to support our importance weighting approach, proving generalization and sampling rate bounds.
I NTRODUCTION
Deep neural networks have shown remarkable success in several domains such as computer vision and natural language processing.
In many tasks, this is achieved by heavily relying on extremely large labeled datasets.
In addition to the storage costs and potential security/privacy concerns that come along with large datasets, training modern deep neural networks on such datasets also incur high computational costs.
With the growing size of datasets in various domains, algorithm scalability is a real and imminent challenge that needs to be addressed.
One promising way to solve this problem is with data subset selection, where the learner aims to ﬁnd the most informative subset from a large number of training samples to approximate (or even improve upon) training with the entire training set.
Such ideas have been extensively studied in k-means and k-median clustering (Har-Peled & Mazumdar, 2004), subspace approximation (Feldman et al., 2010), computational geometry (Agarwal et al., 2005), density estimation (Turner et al., 2021), to name a few.
One particular approach for solving data subsampling involves the computation of coresets, which are weighted subsets of a dataset that can act as the proxy for the whole dataset to solve some optimization task.
Coreset algorithms are primarily motivated with theoretical guarantees that bound the difference between the training loss (or other such objective) over the coreset and that over the full dataset under different assumptions on the losses and hypothesis classes (Mai et al., 2021;
Munteanu et al., 2018; Curtin et al., 2019; Karnin & Liberty, 2019).
However, in practice, most competitive subset selection algorithms, that are designed for general loss functions and arbitrary function classes, focus only on selecting informative subsets of the data and typically do not assign weights to the selected examples.
These methods are, for example, based on some notion of model uncertainty (Scheffer et al., 2001), information gain (Argamon-Engelson & Dagan, 1999), loss gradients (Paul et al., 2021; Ash et al., 2019), or diversity (Sener & Savarese, 2018).
Counter to this trend, we show that weighting the selected samples can be very beneﬁcial.
In this work, we present a subset selection algorithm called IWeS that is designed for general loss functions and hypothesis classes and that selects examples by importance sampling, a theoretically motivated and unbiased sampling technique.
Importance sampling is conducted according to a specially crafted probability distribution and, importantly, each sampled example is weighted inversely proportional to its sampling probability when computing the training loss.
We develop two types of sampling probability for different practical requirements (e.g. computational constraints and label availability), but in both cases, the sampling probability is based on the example’s entropy-based score computed using a previously trained model.
We note, the IWeS algorithm is similar to the
IWAL active learning algorithm of Beygelzimer et al.
(2009) as both are based on importance sampling.
However, in contrast to IWAL, IWeS uses a different sampling probability deﬁnition with a focus on providing a practical method that is amenable to large deep networks and complex hypothesis classes.
Through extensive experiments, we ﬁnd that the IWeS algorithm is competitive for deep neural networks over several datasets.
We compare our algorithm against four types of baselines whose sampling strategies leverage: the model’s uncertainty over examples, diversity of selected examples, gradient information, and random sampling.
Finally, we analyze a closely related albeit less practical algorithm that inspires the design of IWeS, called IWeS-V, proving it admits generalization and sampling rate guarantees that hold for general loss functions and hypothesis classes.
The contributions of this work can be summarized as follows:
We present the I mportance We ighted S ubset Selection (IWeS) algorithm that selects examples by importance sampling with a sampling probability based on a model’s entropy, which is applicable to (and practical for) arbitrary model families including modern deep networks.
In addition to the subset selection framework, IWeS also works in the active learning setting where the examples are unlabeled at selection time.
We demonstrate that IWeS achieves signiﬁcant improvement over several baselines (Random, Margin, Least-Conﬁdent, Entropy, Coreset, BADGE) using VGG16 model for six common multi-class datasets (CIFAR10, CIFAR10-corrupted, CIFAR100, SVHN, Eurosat, Fashion MNIST), and using ResNet101 model for the large-scale multi -label OpenImages dataset.
We provide a theoretical analysis for a closely related algorithm, IWeS-V, in Section 4.
We prove a O (1 / √ T ) generalization bound, which depends on the full training dataset size T .
We further give a new deﬁnition of disagreement coefﬁcient and prove a sampling rate bound by leveraging label information, which is tighter compared with the label complexity bound provided by
Beygelzimer et al.
(2009) that does not use label information.
1. R ELATED W ORK
Uncertainty.
Uncertainty sampling, which selects examples that the model is least conﬁdent on, is favored by practitioners (Mussmann & Liang, 2018) and rather competitive among many recent algorithms (Yang & Loog, 2018).
Uncertainty can be measured through entropy (Argamon-Engelson & Dagan, 1999), least conﬁdence (Culotta & McCallum, 2005), and most popular is the margin between the most likely and the second most likely labels (Scheffer et al., 2001).
More recent works measure the model uncertainty indirectly, such as selecting examples based on an estimated loss (Yoo & Kweon, 2019) as well as leveraging variational autoencoders and adversarial networks to ﬁnd points not well represented by the current labeled data (Sinha et al., 2019;
Kim et al., 2021).
Beygelzimer et al.
(2009) makes use of a disagreement-based notion of uncertainty and constructs an importance weighted predictor with theoretical guarantees called IWAL, which is further enhanced by Cortes et al.
However, IWAL is not directly suitable for use with complex hypothesis spaces, such as deep networks, since it requires solving a non-trivial optimization over a subset of the hypothesis class, the so-called version space, in order to compute sampling probabilities.
We further discuss these difﬁculties in Section 4.
Diversity.
In another line of research, subsets are selected by enforcing diversity such as in the FASS (Wei et al., 2015) and Coreset (Sener & Savarese, 2018) algorithms.
Wei et al.
(2015) introduces a submodular sampling objective that trades off between uncertainty and diversity by ﬁnding a diverse set of samples from amongst those that the current trained model is most uncertain about.
It was further explored by Kaushal et al.
(2019) who designed a uniﬁed framework for data subset selection with facility location and dispersion-based diversity functions.
Sener & Savarese (2018) show that the task of identifying a coreset in an active learning setting can be mapped to solving the k-center problem.
Further recent works related to coreset idea are Mirzasoleiman et al.
Killamsetty et al.
(2021), where the algorithms select representative subsets of the training data to minimize the estimation error between the weighted gradient of selected subset and the full gradient.
Loss Gradient.
Another class of algorithms selects a subset by leveraging the loss gradients.
For example, the GRAND score (Paul et al., 2021), or closely related EL2N score, leverages the average gradient across several different independent models to measure the importance of each sample.
However, as such, it requires training several neural networks, which is computationally expensive.
BADGE (Ash et al., 2019) is a sampling strategy for deep neural networks that uses k-MEANS++ on the gradient embedding of the networks to balance between uncertainty and diversity.
ISAL (Liu et al., 2021) measures the potential impact of each example by leveraging the loss gradient and Hessian.
Finally, for the sake of completeness, we note that importance weighting type approaches have also been used for the selection of examples within an SGD minibatch (Katharopoulos & Fleuret, 2018; Johnson & Guestrin, 2018), which can be thought of a change to the training procedure itself.
In contrast, the problem setting we consider in this work requires explicitly producing a (weighted) subset of the training data and treats the training procedure itself as a black-box.
These are a sampling of data subset selection algorithms, and we refer the reader to (Guo et al., 2022) for a more detailed survey.
In this work, we choose at least one algorithm from each of the categories mentioned above, in particular, Margin (Scheffer et al., 2001), BADGE (Ash et al., 2019), and Coreset (Sener & Savarese, 2018) to compare against empirically in Section 3.
However, before that, we ﬁrst formally deﬁne the IWeS algorithm in the following section.
T HE IW E S A LGORITHM
{+Algorithm 
+}
I mportance We ighted S ubset Selection (IWeS)
Require:
Labeled pool P , seed set size k 0 , subset batch size k , number of {+iterations R , weight cap parameter u . 
 Initialize the subset S = ∅ . S 0 ← Draw k 0+} examples from P uniformly at random. 3: Set S =
{ ( x , y, 1) : ( x , y ) ∈ S 0 } and P = P\S4: for r = 1 , 2 , . . .
, R do 5:
Set S r = ∅ .
Train f r , g r on S using the weighted loss and independent random initializations.
7: while |S r | < k do 8:
Select ( x , y ) uniformly at random from P .
Set p ( x , y ) using entropy-based disagreement or entropy criteria shown in Eq (1) and (2).
Q ∼ Bernoulli ( p ( x , y )) .
11: if Q = 1 then 12:
Set S r = S
r ∪ (cid:110)(cid:16) x , y, min (cid:16) 1 p ( x ,y ) , u (cid:17)(cid:17)(cid:111) and P =
P\ { ( x , y ) } .
13: end if 14: end while 15:
S = S ∪ S r .
16: end for 17:
Train f R +1 on S using the weighted loss.
18: return S , f R +
We consider a practical batch streaming setting, where an algorithm processes one example at a time without updating its state until a batch of examples is selected.
That is, like in standard streaming settings, the algorithm receives a labeled example, and decides whether to include it in the selected subset or not.
Yet, the algorithm is only allowed to update its state after a ﬁxed batch of examples have been selected in order to limit the overhead costs (e.g. this typically can include retraining models and extracting gradients).
Unlike the pool-based setting where the algorithm receives the entire labeled pool beforehand, a batch streaming setting can be more appropriate when facing a vast training data pool since the algorithm can only process a subset of the pool without iterating over the whole pool.
Note that any batch streaming algorithm can also be used in a pool-based setting, by simply streaming through the pool in a uniformly random fashion.
At a high level, the
IWeS algorithm selects examples by importance sampling where the sampling probability is based on the entropy of models trained on previously selected data.
We deﬁne two sampling probabilities that allow us to trade-off between performance and the computational cost, as well as label-aware or an active learning setting leading to less label annotation costs.
As we will subsequently see, these sampling deﬁnitions are both easy to use and work well in practice.
To deﬁne the algorithm in more detail, we let X ∈ R d and Y = { 1 , . .
, c } denote the input space and the multi-class label space, respectively.
We assume the data ( x , y ) is drawn from an unknown joint distribution D on X ×Y .
Let H =
{ h : X → Z} be the hypothesis class consisting of functions mapping from X to some prediction space
Z ⊂ R Y and let (cid:96) :
Z × Y → R denote the loss.
The pseudocode of IWeS is shown in Algorithm 1.
Initially, a seed set S
( |S 0 | = k 0 ) is selected uniformly at random from the labeled pool P .
Then the algorithm proceeds in rounds r ∈ [1 , . . .
, R ] and it consists of two main components: training and sampling.
At the training step at round r , the model(s) are trained using the importance-weighted loss, namely f r = arg min h ∈H (cid:80) ( x ,y,w ) ∈S w · (cid:96) ( h ( x ) , y ) on the subset S , selected so far, in the previous r − 1 rounds.
Depending on the sampling strategy, we may need to randomly initialize two models f r , g r in which case they are trained independently on the same selected subset S , but with different random initializations.
At the sampling step at round r , the IWeS algorithm calculates a sampling probably for example ( x , y ) ∈ S based on one of the following deﬁnitions:
• Entropy-based Disagreement.
We deﬁne the sampling probability based on the disagreement on two functions with respect to entropy restricted to the labeled example ( x , y ) .
That is,
[Equation]
where P f r ( y | x ) is the probability of class y with model f r given example x. If the two functions, f r , g r , disagree on the labeled example ( x , y ) , then p ( x , y ) will be small and the example will be less likely to be selected.
This deﬁnition is the closest to the IWeS-V algorithm analyzed in Section 4 and achieves the best performance when the computational cost of training two models is not an issue.
In Appendix A, we show an efﬁcient version of entropy-based disagreement that utilizes only one model and achieves similar performance.
• Entropy.
We deﬁne the sampling probability by the normalized entropy of the model f r trained on past selected examples:
[Equation]
The sampling probability p ( x , · ) is high whenever the model class probability P f r ( y [Equation]
| x ) is close to 1 / |Y| , which is when the model is not conﬁdent about its prediction as it effectively randomly selects a label from Y .
This deﬁnition does not use the label y and thus it can be used in an active learning setting where the algorithm can only access the unlabeled examples.
Another advantage is that it only requires training one model, thereby saving some computational cost.
We note that entropy-based sampling has been used in algorithms such as uncertainty sampling as discussed in the related works section, but using entropy to deﬁne importance weights has not been done in past literature.
Based on one of these deﬁnitions, the IWeS algorithm then decides whether to include the example into the selected subset S by ﬂipping a coin Q with chosen sampling probability p ( x , y ) .
If the example is selected, the example’s corresponding weight w is set to [Equation] , and the example is removed from the labeled
pool P = P\ { ( x , y ) } .
This process is repeated until k examples have been se lected.
Below we use IWeS-dis as an abbreviation for IWeS algorithm with Entropy-based Disagreement sampling probability and IWeS-ent for IWeS algorithm with Entropy sampling probability.
The weighted loss used to train the model can be written as 1 |P| (cid:80) i ∈P
Q i p ( x i ,y i ) (cid:96) ( f ( x i ) , y i ) and it is an unbiased estimator of the population risk E ( x ,y ) ∼D [ (cid:96) ( f ( x ) , y )] .
Yet such estimator can have a large variance when the model is highly conﬁdent in its prediction, that is whenever P f r ( y | x ) is large, then p ( x , y ) is small.
This may lead to training instability and one pragmatic approach to addressing this issue is by “clipping” the importance sampling weights (Ionides, 2008; Swaminathan & Joachims, 2015).
Thus in our algorithm, we let u be the upper bound on the weight of the selected example.
Although this clipping strategy introduces an additional parameter, we ﬁnd it is not too sensitive and, as mentioned in the empirical section, set it to a ﬁxed constant throughout our evaluation.
The primary computational cost of the algorithm, at each sampling round r , comes from (1) training the model and (2) the inference cost associated with computing the sampling probabilities on a constant fraction of the unlabeled pool of examples.
As described in the next section, besides for random sampling, all compared sampling algorithms incur similar re-training and model scoring costs at each sampling round.
E MPIRICAL E VALUATION
We compare IWeS with state-of-the-art baselines on several image classiﬁcation benchmarks.
Speciﬁcally, we consider six multi-class datasets (CIFAR10, CIFAR100 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), EUROSAT (Helber et al., 2019), CIFAR10 Corrupted (Hendrycks & Dietterich, 2019), Fashion MNIST (Xiao et al., 2017) and one large-scale multi-label OpenImages dataset (Krasin et al., 2017).
In the multi-class setting, each image is associated with only one label.
On the other hand, the multi-label OpenImages dataset consists of 19,957 classes over 9M images, where each image contains binary labels for a small subset of the classes (on average 6 labels per image).
Further details of each dataset can be found in Table 1 and Table 2 in the appendix.
For all experiments, we consider a diverse set of standard baselines from both subset selection and active learning literature (discussed in Section 1.1).
• Uncertainty Sampling selects top k examples on which the current model admits the highest uncertainty.
There are three popular ways of deﬁning model uncertainty s ( x ) of an example x, namely margin sampling, entropy sampling, and least conﬁdent sampling, and all are based on P f
[ˆ y | x ] , the probability of class ˆ
y given example x
according to the model f .
Margin sampling deﬁnes the model uncertainty of an example x as s ( x ) = 1 − ( P f
[ˆ y 1 | x ] − P f
[ˆ y 2 | x ]) where ˆ y 1 = argmax y ∈Y P f
[ y | x ] , ˆ y 2 = argmax y ∈Y\
y 1 P f
[ y | x ] are the ﬁrst and second most probable classes for model f .
For entropy sampling, model uncertainty is deﬁned as s ( x ) =
− (cid:80)
y ∈Y P
f
(ˆ y | x ) log( P f (ˆ y | x )) while for least conﬁdence sampling, it is deﬁned as s ( x )
= 1 − max y ∈Y P
f
(ˆ y | x ) .
• BADGE of Ash et al.
(2019) selects k examples by using the k -MEANS++ seeding algorithm using the gradient vectors, computed with respect to the penultimate layer using the most likely labels given by the latest model checkpoint.
• Coreset ( k -Center) of Sener & Savarese (2018) selects a subset of examples using their embeddings derived from the penultimate layer using the latest model checkpoint.
In particular, the k examples are chosen using a greedy 2-approximation algorithm for the k -center problem.
• Random Sampling selects k examples uniformly at random.
3. M ULTI - CLASS E XPERIMENTS
Here, we compare the IWeS algorithm against the baselines on the six multi-class image datasets.
We use the VGG16 architecture with weights that were pre-trained using ImageNet as well as add two fully-connected 4096 dimensional layers and a ﬁnal prediction layer.
Xavier uniform initialization is used for the ﬁnal layers.
For each dataset, we tune the learning rate by choosing the rate from the set { 0 . 001 , 0 .
005 , 0 . 01 , 0 . 1 } that achieves best model performance on the seed set.
We use batch SGD with the selected learning rate and ﬁx SGD’s batch size to 100.
At each sampling round r , the model is trained to convergence on all past selected examples for at least 20 epochs.
For IWeS, we set the weight capping parameter to 2 for all datasets except for CIFAR10 which we decreased to 1.5 in order to reduce training instability.
The embedding layer for BADGE and Coreset is extracted from the penultimate layer having a dimension of 4096.
The effective dimension of the gradient vector in BADGE grows with the number of labels, which is problematic for CIFAR100 as it has 100 classes.
More speciﬁcally, the runtime of BADGE is given by O ( dkT ) , which can be large for CIFAR100 since the dimension of the gradient vector from the penultimate layer is d = 4096 × 100 , the size of the labeled pool is T =50K, and the number of examples selected in each round is k =
5K.
In order to solve this inefﬁciency for
CIFAR100, we split the labeled pool randomly into 100 partitions and ran separate instances of the algorithm in each partition with batch size k/ 100 .
Each algorithm is initialized with a seed set that is sampled uniformly at random from the pool.
After that, sampling then proceeds in a series of rounds r where the model is frozen until a batch k of examples is selected.
The seed set size and sampling batch size k are set to 1K for CIFAR10, SVHN,
EUROSAT, CIFAR10 Corrupted, Fashion MNIST, and to 5K for CIFAR100.
The initial seed set size allows at least 50 initial examples per class on average.
The experiment was repeated for 5 trials.
Any trial that encountered divergent training, i.e. the resulting model accuracy is more than three times below the standard error of model’s accuracy on seed set, was dropped.
We note that this happened infrequently (less than 3% of the time) and all reported averaged results have at least 3 trials.
Figure 1 shows the mean and standard error of VGG16 model’s accuracy on a held out test set comparing IWeS-dis to the baseline methods.
The IWeS-dis algorithm either outperforms or matches the performance of the baseline algorithms for all datasets.
We also ﬁnd that margin sampling consistently performs well against the remaining baseline algorithms and that BADGE either matches the performance of margin sampling or slightly underperforms on some datasets (Eurosat, Fashion
MNIST, CIFAR100).
Coreset admits a similar and at times slightly poorer performance compared to random sampling.
Next, Figure 2 compares the two variants of our algorithm: IWeS-dis and IWeS-ent.
We ﬁnd that the IWeS-dis performs slightly better than IWeS-ent on most of the datasets.
This is not surprising since the IWeS-dis sampling probability leverages label information and more computational power, i.e. trains two models.
As explained in Section 4, it also better ﬁts our theoretical motivation.
Nevertheless, it is important to note that IWeS-ent, without the label information, still consistently outperforms or matches the performance of the baselines for all the datasets.
3. M ULTI - LABEL O PEN I MAGES E XPERIMENTS
Here, we evaluate the performance of the IWeS algorithm on OpenImages v6.
We train a ResNet model implemented using tf-slim on 64 Cloud two core TPU v4 acceleration, and apply batch SGD with batchsize of 6144 and an initial learning rate of 10 − 4 with decay logarithmically every 5 × 10 examples.
We add a global pooling layer with a fully connected layer of 128 dimensions as the ﬁnal layers of the networks, which is needed by BADGE and Coreset.
The model is initialized with weights that were pre-trained on the validation split using 150K SGD steps, and at each sampling round, the model is trained on all past selected examples with an additional 15K SGD steps.
In the previous section, our results show that the IWeSdis algorithm only slightly outperforms the IWeS-ent algorithm on a few datasets.
Also, since the IWeS-dis requires training two neural networks, which is computationally expensive in this scenario, we only test the performance of IWeS-ent.
Since IWeS-ent does not use the label information, we can also measure the performance of the algorithm in an active learning setting.
Since OpenImages is a multi-label dataset, the sampling algorithms must not only select the image, but also the class.
That is, each example selected by an algorithm consists of an image-class pair with a corresponding binary label indicating whether the corresponding class is present in the image or not.
In order to adapt IWeS-ent to the multi-label setting, the entropy sampling probability for each image-class pair is deﬁned as p ( x , · ) = − P f r ( y | x ) log 2 P f r ( y | x ) − (1 − P f r ( y | x )) log 2 (1 − P f r ( y | x )) , where P f r ( y | x ) is the model class probability of a positive label at round r .
A seed set of size 300K is sampled uniformly at random from the pool, and at each sampling round r , the algorithms select 100K examples.
Similarly to the previous section, in order to run BADGE on OpenImages, we divide the pool into 100 partitions and run separate instances of the algorithm in each partition.
For IWeS, the weight capping parameter is set to 10.
Figure 3 shows the mean and standard error across 5 trials of the pooled average precision (Pooled
AP) metric for each algorithm.
As the number of selected examples increases, IWeS-ent outperforms all other baselines methods on the OpenImages dataset.
We also ﬁnd that BADGE performs similarly or even slightly worse than the uncertainty-based sampling algorithms when the number of selected examples is smaller than 800K, and then outperforms all uncertainty-based sampling as the number of selected examples increases.
Coreset initially performs better than random sampling, but at later sampling rounds, it admits a similar performance to random sampling.
T HEORETICAL M OTIVATION
In order to theoretically motivate the IWeS algorithm, we analyze a closely related algorithm which we call IWeS-V, adapted from the IWAL algorithm of Beygelzimer et al.
We prove that
IWeS-V admits generalization bounds that scale with the dataset size T and sampling rate bounds that are in terms of a new disagreement coefﬁcient tailored to the subset selection framework.
Below, we let L ( h ) =
E ( x ,y ) ∼D
[ (cid:96) ( h ( x ) , y )] denote the expected loss of hypothesis h ∈ H , and h ∗ =
argmin h ∈H L ( h ) be the best-in-class hypothesis.
Without loss of generality, we consider a bounded loss (cid:96) :
Z × Y → [0 , 1] mapping to the interval [0 , 1] .
Such a loss can be achieved by any bounded loss after normalization.
For simplicity, we assume H is a ﬁnite set, but our results can be easily extended by standard covering arguments to more general hypothesis sets such as ﬁnite VC-classes.
The IWeS-V algorithm operates on an i.i.d.
example ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
, ( x T , y T ) drawn from D sequentially.
It maintains a version space H t at any time t , with H 1 = H .
At time t , IWeS-V ﬂips a coin Q t ∈ { 0 , 1 } with bias p t deﬁned as
[Equation]
(cid:96) (
[Equation] y
[Equation]
[Equation]
where H t = (cid:110)
h ∈
H t − 1 : 1 t (cid:80) t s =1 Q s p s (cid:96) ( h ( x s ) , y s ) ≤ min h (cid:48)
∈H t − 1 1 t (cid:80) t s =1 Q s p s (cid:96) ( h (cid:48) ( x s ) , y s )
+ ∆ t − 1 (cid:111) with ∆ t − 1 = (cid:113) 8log(2 T ( T +1) |H| 2 /δ )
t −.
The example is selected if Q t = 1 and otherwise it is discarded.
The main idea behind this algorithm is thus to deﬁne a sampling probability that is in terms of the disagreement between two hypothesis f, g that are not too far from the best model trained on the past selected data, i.e. min h ∈H t − 1 1 t (cid:80) ts
=1 Q s p s (cid:96) ( h ( x s ) , y s ) .
The formal IWeS-V algorithm pseudo-code (Algorithm 2) and all the theorem proofs can be found in Appendix B.
For general, e.g. non-linear, hypothesis classes it is computationally infeasible to ﬁnd two hypotheses f, g ∈ H t that maximize the expression in equation (3).
This main impracticality of IWeS-V is reason why we developed the IWeS algorithm of the previous section.
This drawback is also shared by the IWAL algorithm of Beygelzimer et al.
(2009), which computes a sampling probability very similar to that of equation (3), but with an additional maximization over the choice of y ∈ Y in the deﬁnition of the sampling probability p t .
Before continuing we explain how our practical algorithm IWeS-dis, speciﬁcally using sampling probability in equation (1), is closely related to the IWeS-V algorithm.
Recall that the IWeS algorithm trains two models f and g each minimizing the importance-weighted loss using the data sampled so far.
Therefore, each model exhibits reasonable training loss, i.e. they are expected to be included in the version space H t of good hypothesis, while the different random initializations (in the case of non-convex neural network hypotheses) results in models that still differ in certain regions of the feature space.
Thus, the difference in equation (1) can be thought of as a less aggressive version of the difference found in the maximization of equation (3).
Another dissimilarity between the two is that the IWeS-dis algorithm is deﬁned for the batch streaming setting while the IWeS-dis algorithm and its analysis is developed for the streaming setting.
Said differently, the IWeS-V algorithm can be seen as a special case of the IWeS-dis algorithm with target subset size of 1.
To extend the theoretical guarantees of IWeS-V to the batch streaming setting, we can follow a similar analysis developed by Amin et al.
(2020) to also ﬁnd that the effects of delayed feedback in the batch streaming setting are in fact mild as compared to the streaming setting.
4. G ENERALIZATION BOUND
Next, we turn to the topic of generalization guarantees and we review an existing bound for coreset based algorithms.
The guarantees of coreset algorithms are generally focused on showing that a model’s training loss on the selected subset is close to the same model’s training loss on the whole dataset.
That is, given dataset P = { ( x i , y i ) }
Ti =1 ∼ D T , the learner seek to select a subset m < T of examples S
= { ( x (cid:48)
i , y (cid:48) i ) } mi =1 along with a corresponding set of weights w 1 , . . .
, w m such that for some small (cid:15) > 0 and for all h ∈ H , the additive error coreset guarantee holds (cid:12)(cid:12)(cid:12)(cid:80) mi
=1 w i (cid:96) ( h ( x (cid:48) i ) , y (cid:48)
i ) − (cid:80)
Ti =1 (cid:96) ( h ( x i ) , y i ) (cid:12)(cid:12)(cid:12) ≤ (cid:15)T .
The following proposition, which is a minor extension of Fact 8 of Karnin & Liberty (2019), allows us to convert a coreset guarantee into a generalization guarantee.
Proposition 4.1.
Let h (cid:48) =
argmin h ∈H (cid:80) mi
=1 w i (cid:96) ( h ( x (cid:48) i ) , y (cid:48) i ) , and let the additive error coreset guarantee hold for any (cid:15) > 0 , then for any δ > 0 , with probability at least 1 − δ , it holds that
L ( h (cid:48) )
≤ L ( h ∗ ) + 2 (cid:15) + 2 (cid:112) ln(4 /δ ) / 2 T.
As shown above, the generalization guarantee depends linearly on (cid:15) which in turn depends on the size of the subset m .
To give a few examples, Karnin & Liberty (2019) show that for hypotheses that are deﬁned as analytic functions of dot products (e.g. generalized linear models) this dependence on m is (cid:15)
=
O (1 /m ) , while for more complex Kernel Density Estimator type models the dependence is (cid:15) =
O (1 / √ m ) .
See Mai et al.
(2021), Table 1, for examples on the dependency between (cid:15) and m under different data distributions assumptions (e.g. uniform, deterministic, (cid:96) 1 Lewis) and for speciﬁc loss functions (e.g. log loss, hinge loss).
We now provide a generalization guarantee for the IWeS-V algorithm, which depends on the size of the labeled pool size T .
The proof follows from that in Beygelzimer et al.
Theorem 4.2.
Let h ∗ ∈
H be the minimizer of the expected loss function h ∗ = argmin
h ∈H L ( h ) .
For any δ > 0 , with probability at least 1 − δ , for any t ≥ 1 with t ∈ { 1 , 2 . . . , T } , we have that h ∗ ∈ H t and that L ( f ) − L ( g ) ≤ 2∆ t − 1 for any f, g ∈ H t .
In particular, if h T is the output of
IWeS-V, then L ( h T ) − L ( h ∗ ) ≤ 2∆ T − 1 =
O (cid:0)(cid:112) log( T/δ ) /T
(cid:1) .
Unlike the distribution-speciﬁc and loss-speciﬁc theoretical guarantees proposed in the coreset literature, Theorem 4.2 holds for any bounded loss function and general hypothesis classes.
If we ignore log terms and consider the more complex Kernel Density Estimator class of hypotheses, the coreset method of Karnin & Liberty (2019) requires m = O ( T ) coreset samples in order to achieve an overall O (1 / √ T ) generalization bound.
As we will see in the next section, the required IWeS sampling rate can also be as high as O ( T ) , but critically is scaled by the best-in-class loss, which in favorable cases is signiﬁcantly smaller than one.
4. S AMPLING R ATE BOUNDS
Hanneke (2007) proves that the expected number of labeled examples needed to train a model in an active learning setting can be characterized in terms of the disagreement coefﬁcient of the learning problem.
Later, Beygelzimer et al.
(2009) generalizes this notion to arbitrary loss functions, and in this work, we further generalize this for the subset selection setting.
Recall that the disagreement coefﬁcient θ AL in Beygelzimer et al.
(2009) for the active learning setting is deﬁned as
[Equation]
where B AL ( h ∗ , r ) = { h ∈ H :
ρ AL ( h, h ∗ ) ≤ r } with ρ AL ( f, g )
=
E x ∼X [sup y ∈Y | (cid:96) ( f ( x ) , y ) − (cid:96) ( g ( x ) , y ) | ] .
Informally, this coefﬁcient quantiﬁes how much disagreement there is among a set of classiﬁers that is close to the best-in-class hypothesis.
In the subset selection setting, labels are available at sample time and, thus, we are able to deﬁne the following disagreement coefﬁcient:
Deﬁnition 4.1.
Let ρ S ( f, g )
=
E ( x ,y ) ∈D [ | (cid:96) ( f ( x ) , y ) − (cid:96) ( g ( x ) , y ) | ] and B S ( h ∗ , r ) =
{ h ∈ H : ρ S ( h, h ∗ ) ≤ r } for r ≥ 0 .
The disagreement coefﬁcient in the subset selection setting is deﬁned as
[Equation]
The main difference between the above coefﬁcient and that of Beygelzimer et al.
(2009) is that there is no supremum over all label y ∈ Y both in the deﬁnition of the distance ρ and the coefﬁcient’s numerator.
Instead, the supremum is replaced with an expectation over the label space.
The following theorem leverages θ S to derive an upper bound on the expected number of selected examples for the IWeS-V algorithm.
Below, let F t = { ( x i , y i , Q i ) } ti =1 be the observations of the algorithm up to time t .
Theorem 4.3.
For any δ > 0 , with probability at least 1 − δ , the expected sampling rate of the
IWeS-V algorithm is:
[Equation]
Suppressing lower order terms, the above expected sampling rate bound is small whenever the product of the disagreement coefﬁcient and the expected loss of the best-in-class is small.
In such cases, by combining the above theorem with the generalization guarantee, it holds that IWeS-V returns a hypothesis trained on a only fraction of the points that generalizes as well as a hypothesis trained on the full dataset of size T .
Theorem 4.3 can be further improved by adapting the ideas found in
Cortes et al.
to the IWeS-V algorithm.
See Appendix B.4 for this enhanced analysis.
The form of this sampling rate bound is similar to that of Beygelzimer et al.
More concretely, under the assumption that a loss function has bounded slope asymmetry, that is K (cid:96) = sup
[Equation]
[Equation]
) − (cid:96) ( z (cid:48) ,y ) | min
y ∈Y | (cid:96) ( z,y ) − (cid:96) ( z (cid:48) ,y ) | is bounded, with probability at least 1 − δ , the expected number of examples selected by the IWAL algorithm is given by O (cid:16) θ AL K (cid:96) (cid:16) L ( h ∗ ) T + (cid:112) T log( T/δ ) (cid:17)(cid:17) .
Thus, the main difference between the sampling rate bound of the IWAL algorithm and the IWeS-V algorithm are the factors that depends on the two disagreement coefﬁcients: θ AL K (cid:96) and θ S .
Since θ S leverages the label information we may expect it to provide a tighter bound, compared to using the label-independent disagreement θ AL .
Theorem 4.4 shows that this is indeed the case.
Theorem 4.4.
If the loss function has a bounded slope asymmetry K (cid:96) , then θ S ≤ θ AL K (cid:96) .
The above theorem in conjunction with the sampling rate guarantees thus proves that the sampling rate bound of IWeS of Theorem 4.3 is tighter than the sampling rate bound of the IWAL algorithm.
C ONCLUSION
In this paper we have introduced a subset selection algorithm, IWeS that is designed for arbitrary hypothesis classes including deep networks.
We have shown that the IWeS algorithm outperforms several natural and important baselines across multiple datasets.
In addition, we have developed an initial theoretical motivation for our approach based on the importance weighted sampling mechanism.
A natural next step is enforcing a notion of diversity as it will likely provide improved performance in the large-batch sampling setting and thus, we plan to adapt the diversity-based method in
Citovsky et al.
(2021) by replacing the uncertainty sampling component with the IWeS algorithm.
Supplementary Material A E XTENDED A LGORITHMIC AND E XPERIMENTAL D ETAILS A. D ATASET D ETAILS
Table 1 and Table 2 provide further details of the six multi-class classiﬁcation datasets and the multilabel Open Images dataset, respectively
[Table]
[Table]
A. M ORE E XPERIMENT D ETAILS AND R ESULTS
When running IWeS, at each iteration, we pass over the labeled pool sequentially in a uniform random order, removing each selected example from the pool.
If we exhaust the entire pool before selecting k examples, we start again at the beginning of the sequence and iterate over the remaining examples.
In order to reduce the number of passes required for the smaller datasets, we scale the sampling probabilities of the remaining points uniformly by 1+ j 10 , where j is the number of passes so far.
Figure 4 compares IWeS-dis and IWeS-ent for additional three datasets that we omit in Section 3.
, IWeS-dis, margin sampling and random sampling.
A. E FFICIENT VERSION OF ENTROPY - BASED DISAGREEMENT
Here we develop an efﬁcient version of entropy-based disagreement which we call IWeS-loss by deﬁning the sampling probability to be proportional to the model’s entropy restricted to the labeled examples ( x , y ) as follows:
[Equation]
where P f r ( y | x ) is the probability of class y with model f r given example x. As P f r ( y | x ) increase from 0 to 1, this sampling probability ﬁrst increase then decrease.
Thus the sampling probability is high whenever the model is not conﬁdent about the model prediction.
Unlike IWeS-dis, this deﬁnition only requires training one model, thereby saving some computational cost.
Figure 5 compares the two variants IWeS-dis and IWeS-loss.
We ﬁnd that the performance are similar across all datasets, with IWeS-loss behave slightly better than IWeS-dis at the ﬁrst two sample iterations (i.e. Fashion MNIST, CIFAR10, Eurosat), and IWeS-dis slightly outperform IWeS-loss as the number of sample iteration increases (i.e. SVHN, Eurosat).
B P ROOFS OF T HEORETICAL G UARANTEE B. P ROOF OF P ROPOSITION 4.
Proposition 4.1.
Let h (cid:48) =
argmin h ∈H (cid:80) mi
=1 w i (cid:96) ( h ( x (cid:48) i ) , y (cid:48) i ) , and let the additive error coreset guarantee hold for any (cid:15) > 0 , then for any δ > 0 , with probability at least 1 − δ , it holds that
L ( h (cid:48) )
≤ L ( h ∗ ) + 2 (cid:15) + 2 (cid:112) ln(4 /δ ) / 2 T.
Proof of Proposition 4.1.
[Equation]
(Hoeffding inequality) ≤
[Equation]
T m (cid:88) i =
w i (cid:96) ( h (cid:48) ( x (cid:48) i ) , y (cid:48) i ) + (cid:15) + (cid:114) ln(4 /δ )
2 T ( 0 ≤ (cid:96) ( · ) ≤ 1 ) ≤
T m (cid:88) i =
w i (cid:96) ( h ∗ ( x (cid:48) i ) , y i ) + (cid:15) + (cid:114) ln(4 /δ )
2 T (By the deﬁnition of h (cid:48) ) ≤
T
T (cid:88) i = (cid:96) ( h ∗ ( x i ) , y i ) + 2 (cid:15) + (cid:114) ln(4 /δ )
2 T ≤ E ( x ,y ) ∼D
[ (cid:96) ( h ∗ ( x ) , y )] + 2 (cid:15) +(cid:114) ln(4 /δ )
2 T (Hoeffding inequality)
[Equation]
B. P ROOFS FOR STATEMENTS IN S ECTION 4.
Algorithm 2 contains a detailed pseudocode of the IWeS-V algorithm referred to in the main body of the paper.
Algorithm 2 IWeS-V
Require:
Labeled Pool P .
1: Initialize: S 0 = ∅ , H 0 = H .
2: for t = 1 , 2 , . . .
, T do 3:
Sample ( x t , y t ) uniformly at random from P .
Update:
H t = (cid:40)
h ∈
H t − 1 : t −t − 1 (cid:88) i =
Q i p i (cid:96) ( h ( x i ) , y i ) ≤ min h ∈H t − t −t − 1 (cid:88) i =
Q i p i (cid:96) ( h ( x i ) , y i ) + ∆ t −(cid:41) where ∆ t − 1 = (cid:113) 8log(2 T ( T +1) |H| 2 /δ )
t −.
Set p t = max f,g ∈H t (cid:96) ( f ( x t ) , y t ) − (cid:96) ( g ( x t ) , y t ) .
Q t ∼ Bernoulli ( p t ) .
7: if Q t = 1 then 8:
Set S t =
S t − 1 ∪ (cid:110)(cid:16) x t , y t , 1 p t (cid:17)(cid:111) 9: else 10:
S t =
S t − 1 .
11: end if 12: h t = arg min h ∈ H (cid:80) ( x ,y,w ) ∈S t w · (cid:96) ( h ( x ) , y ) .
13: end for 14: return h T
In order to prove Theorem 4.2, we ﬁrst present the following Lemma.
Lemma B.1.
Deﬁne the weighted empirical loss as L t − 1 ( f ) =
1 t − 1 (cid:16)(cid:80) t − 1 i =1 Q i p i (cid:96) ( f ( x i ) , y i ) (cid:17) .
For all δ > 0 with probability at least 1 − δ , for all t ∈ { 1 , . . .
, T } , and for all f, g ∈ H t − 1 , we have | L t − 1 ( f ) − L t − 1 ( g ) − L ( f ) + L ( g ) |
[Equation]
t − 1 .
Proof of Lemma B.1.
Fix any t ∈ { 1 , . .
, T } .
For any f, g ∈ H t − 1 , deﬁne
[Equation]
for i ∈ { 1 , . . .
, t − 1 } .
The sequence Z i is a martingale difference since
E [ Z i | Z 1 , . . .
Z i − 1 ] =
[Equation]
Due to the fact that
[Equation]
as p i ≥ | (cid:96) ( f ( x i ) , y i ) − (cid:96) ( g ( x i ) , y i ) | for f, g ∈ H t − 1 .
Thus, we can apply Azuma’s inequality,
[Equation]
where we used the fact that ( x t , y t ) is i.i.d.
Since H t − 1 is a random subset of H , we take the union bound over f, g ∈ H and t − 1 .
Then we take another union bound over T to ﬁnish the proof.
Next we provide the proof of Theorem 4.2.
Theorem 4.2.
Let h ∗ ∈
H be the minimizer of the expected loss function h ∗ = argmin
h ∈H L ( h ) .
For any δ > 0 , with probability at least 1 − δ , for any t ≥ 1 with t ∈ { 1 , 2 . . . , T } , we have that h ∗ ∈ H t and that L ( f ) − L ( g ) ≤ 2∆ t − 1 for any f, g ∈ H t .
In particular, if h T is the output of
IWeS-V, then L ( h T ) − L ( h ∗ ) ≤ 2∆ T − 1 =
O (cid:0)(cid:112) log( T/δ ) /T
(cid:1) .
Proof of Theorem 4.2.
We show that h ∗ ∈ H t by induction.
Base case holds as h ∗ ∈ H 1 = H .
Now assuming that h ∗ ∈ H t − 1 holds, we show that h ∗ ∈ H t .
Let h (cid:48) =
argmin f ∈H t − 1 L t − 1 ( f ) .
By Lemma B.1,
[Equation]
since L ( h ∗ ) − L ( h (cid:48) ) ≤ 0 by deﬁnition of h ∗ .
Thus, L t − 1 ( h ∗ ) ≤ L t − 1 ( h
[Equation]
) + ∆ t − 1 which means that h ∗ ∈ H t by deﬁnition of H t .
Since H t ⊆
H t − 1 , Lemma B.1 implies that for any f, g ∈ H t ,
[Equation]
Noting that h ∗ , h t ∈ H t completes the proof.
B. P ROOFS FOR STATEMENTS IN S ECTION 4.
Theorem 4.3.
For any δ > 0 , with probability at least 1 − δ , the expected sampling rate of the
IWeS-V algorithm is: (cid:80) Tt =1 E ( x t ,y t )
∼D (cid:2) p t (cid:12)(cid:12)
F t − 1 (cid:3) =
O (cid:16) θ S (cid:16) L ( h ∗ ) T + (cid:112) T log( T/δ ) (cid:17) (cid:17) .
Proof of Theorem
[Equation]
By Theorem
[Equation]
[Equation]
H t ⊂ { h ∈ H :
L ( h ) ≤ L ( h ∗ ) + 2∆ t − 1 } .
Using this fact and that h ∗ is the best in class
[Equation]
it holds that
[Equation]
Thus
[Equation]
letting r = 2 L ( h ∗ )
+ 2∆ t −
[Equation]
it holds that H t ⊂ B S ( h ∗ , r ) .
Then,
[Equation]
By summing the above over t ∈ [ T ] , the sample complexity bound of IWeS-V is then given by
O (cid:16) θ S L ( h ∗ ) T + θ S √ T (cid:17) , which completes the proof.
Theorem 4.4.
If the loss function has a bounded slope asymmetry K (cid:96) , then θ S ≤ θ AL K (cid:96) .
Proof of Theorem 4.4.
We ﬁrst prove
[Equation]
The left hand side follows directly from the deﬁnition.
For the right hand side, assume h ∈
B S ( h ∗ , r ) , then by the deﬁnition of K (cid:96) , we have
[Equation]
where the second inequality comes from
[Equation]
which follows by basic properties of expectations.
As a result
[Equation]
we have
[Equation]
deﬁnition of θ S )
[Equation]
equation
[Equation]
[Equation]
which concludes the proof
[Equation]
B. T HE E NHANCED -IW E S-V A LGORITHM
Here we analyze a modiﬁed algorithm
[Equation]
which we called the Enhanced-IWeS-V
[Equation]
that uses new slack term deﬁned as
[Equation]
to deﬁne the version space in the IWeS-V algorithm.
The theorem below proves that the EnhancedIWeS-V admits improved sampling rate bound that is smaller than the bound in Theorem 4.3 since the square root term also scales with L ( h ∗ ) .
In particular, under realizable setting where L ( h ∗ ) = 0 , the sampling rate bound is O (cid:0) log 3 ( T ) (cid:1) , which depends poly-logarithmic in T and is smaller than the O (cid:16)(cid:112) T log ( T ) (cid:17) bound from Theorem 4.3.
Theorem B.2.
For all δ > 0 , for all T ≥ 3 , with probably at least 1 − δ , the expected sampling rate of the Enhanced-IWeS-V algorithm is:
[Equation]
Proof of Theorem B.2.
The proof follows from that of Lemma 6 in Cortes et al.
From Theorem 4.3, for t ≥ 3 ,
[Equation]
Plugging in the expression of ∆ EIWeS t − 1 into ∆ t − 1 , and applying a concentration inequality to relate (cid:80)
Tt =1 p t to (cid:80) Tt =1 E ( x t ,y t ) ∼D (cid:2) p t (cid:12)(cid:12)
F t − 1 (cid:3) , we end up with a recursion on E ( x t ,y t )
∼D [ p t |F t − 1 ] :
E
[Equation]
t ,y t
[Equation]
∼D (cid:2) p t
[Equation]
where c 1 =(cid:114) log
[Equation]
(cid:17) , and c 2 is a constant.
{+Then we show for all t ≥ 3 , for constant c 
+}
= O (cid:16)(cid:112) log ( T |H| /δ )
(cid:17) and c 4 =
O (cid:0) log 2 ( T |H| /δ )
(cid:1) , we have
[Equation]
The above can be proved by induction as in Cortes et al.
By summing the above over t ∈ [ T ] gives us the result.
We absorb the dependency on |H| inside the big-O notation.