Active Learning Through a Covering Lens
Abstract
Deep active learning aims to reduce the annotation cost for the training of deep models, which is notoriously data-hungry.
Until recently, deep active learning methods were ineffectual in the low-budget regime, where only a small number of examples are annotated.
The situation has been alleviated by recent advances in representation and self-supervised learning, which impart the geometry of the data representation with rich information about the points.
Taking advantage of this progress, we study the problem of subset selection for annotation through a “covering” lens, proposing ProbCover – a new active learning algorithm for the low budget regime, which seeks to maximize Probability Coverage .
We then describe a dual way to view the proposed formulation, from which one can derive strategies suitable for the high budget regime of active learning, related to existing methods like Coreset .
We conclude with extensive experiments, evaluating ProbCover in the low-budget regime.
We show that our principled active learning strategy improves the state-of-the-art in the low-budget regime in several image recognition benchmarks.
This method is especially beneficial in the semi-supervised setting, allowing state-of-the-art semi-supervised methods to match the performance of fully supervised methods, while using much fewer labels nonetheless.
Code is available at https://github.com/avihu111/TypiClust.
Introduction
For the most part, deep learning technology critically depends on access to large amounts of annotated data.
Yet annotations are costly and remain so even in our era of Big Data .
Deep active learning (AL) aims to alleviate this problem by improving the utility of the annotated data.
Specifically, given a fixed budget b of examples that can be annotated, and some deep learner, AL algorithms aim to query those b examples that will most benefit this learner.
In order to optimally choose unlabeled examples to be annotated, most deep AL strategies follow some combination of two main principles: 1) Uncertainty sampling [e.g., 25, 47, 3, 13, 14, 35, 24], in which examples that the learner is most uncertain about are picked, to maximize the added value of the new annotations.
2) Diversity Sampling [e.g., 1, 21, 15, 37, 17, 39, 36, 16, 46, 41, 45], in which examples are chosen from diverse regions of the data distribution, to represent it wholly and reduce redundancy in the annotation.
Most AL methods fail to improve over random selection when the annotation budget is very small [34, 38, 7, 31, 53, 20, 2], a phenomenon sometimes termed "cold start" [15, 48, 22].
When the budget contains only a few examples, they struggle to improve the model’s performance, and even fail to reach the accuracy of the random baseline.
Recently, it was shown that uncertainty sampling is inherently unsuited for the low-budget regime, which may explain the cold start phenomenon [18].
The low-budget scenario is relevant in many applications, especially those requiring an expert tagger
[Figure]
whose time is expensive (e.g., a radiologist tagger for tumor detection).
If we want to expand deep learning to new domains, overcoming the cold start problem is an ever-important task.
In this work, we focus on understanding the very low budget regime of AL, where the budget of b examples cannot dependably represent the anotated data distribution.
To face up to this challenge, in
Sections 2.1-2.2 we model the problem as Max Probability Cover , defined as follows: given some data distribution, and a radius δ , select the b examples that maximize the probability of the union of balls with radius δ around each example.
We further show that under a separation assumption that is realistic in semantic embedding spaces, Max Probability Cover is befitting the nearest-neighbor classification model, in that it minimizes an upper bound on its generalization error.
In Section 2.4 we show a connection with existing deep AL methods, like Coreset [36], and explain why those methods are more suitable for the high-budget regime than the low-budget regime.
This phenomenon is visualized in Fig.
1, where we see that with only a few examples to choose, Coreset – an AL strategy that employs the principle of diversity sampling – chooses distant and often abnormal points, while ProbCover chooses representative examples.
When using the empirical data distribution, we further show that Max Probability Cover can be reduced to Max Coverage – a known classical NP-hard problem [33] (see Section 2.2).
To obtain a practical AL strategy, in Section 3 we adapt a greedy algorithm for the selection of b examples from a fixed finite training set of unlabeled examples (the training set), which guarantees 1 − 1 e approximation to the problem.
We call this new method ProbCover .
In Section 4 we empirically evaluate the performance of ProbCover on several computer vision datasets, including CIFAR-10, CIFAR-100, Tiny-ImageNet, ImageNet and its subsets.
ProbCover is thus shown to significantly outperform all alternative deep AL methods in the very low-budget regime.
Additionally, ProbCover improves the performance of state-of-the-art
semi-supervised methods, which were thought until recently to make AL redundant [6], allowing for the learning of computer vision tasks with very few annotated examples.
Relation to prior art Recent work investigated AL methods based on an approximation to a facility location problem [44, 29, 52, 36, 42], which is a variant of the covering problem.
In the minimax facility location problem [12], the entire distribution is covered with a fixed number of balls, which can vary in size, whereas in ProbCover the size of the balls is fixed, and we are allowed to cover only part of the total distribution.
While this difference may seem minor, in the low-budget regime, when the budget is not large enough to represent the data, examples chosen by the facility location problem are not representative (as illustrated in Fig. 1), which leads to poor performance (as shown in Fig. 10).
(i) Develop a theoretical framework to analyze AL strategies in embedding spaces, with "dual" low and high-budget interpretations.
(ii) Introduce ProbCover , a low-budget AL strategy motivated by our framework, which significantly outperforms other methods in the low-budget regime.
(iii) Demonstrate the outstanding competence of ProbCover in semi-supervised learning with very few labeled examples.
Theoretical Analysis
To address the challenge of active learning in low budgets, we adopt a point coverage framework.
Computationally, we analyze the generalisation of the Nearest Neighbor (NN) classification model, as this model depends exclusively on distances from a set of training examples and does not involve any additional inductive bias.
Thus in Section 2.1 we develop a bound on the generalization error in 1-NN models.
It follows from the analysis (see discussion below) that if full coverage is required, which is only practical in the high budget regime, the minimization of this bound translates to the optimal minimax facility location problem, which is known to be NP-hard and which the AL Coreset algorithm by Sener and Savarese [36] is designed to approximate.
In contrast, in the low budget regime, the aforementioned bound can best be optimized by seeking a labeled set L whose probability of covering the unlabeled set is maximal.
In Section 2.2 we show that this problem is also NP-hard.
Furthermore, when the data distribution is not known and is being approximated by the empirical distribution, we show that it is equivalent to the classical
Max Coverage problem.
ProbCover described in Section 3, is designed to solve this problem.
In
Section 2.4 we discuss a sense in which the high budget and low budget problems are dual.
2.1 Bounding the Generalization Error
We shall now derive a bound on the generalization error of the 1 Nearest Neighbor (1-NN) classifier.
We start with some necessary notations and definitions.
Most important is the assumption of δ -purity, which states that most of the time, points that are less than δ apart have the same label.
We then prove a lemma, showing that given a labeled set L and the coverage it achieves, and given the δ -purity assumption, the probability of a point being inside this cover and still being falsely labeled is small.
From this, we finally derive a bound on the generalization error, which is stated in Thm. 1.
Notations Let X denote the input domain whose underlying probability function is denoted P , and let Y =
[ k ] denote the target domain.
Assume that a true labeling function f : X → Y exists.
Let
X =
{ x i } mi =1 denote an unlabeled set of points, and b ≤ m the annotation budget.
Let L ⊆
X denote the labeled set, where | L |
=
b .
Let B δ ( x ) =
{ x ′ :
∥ x ′ − x ∥ 2 ≤ δ } denote a ball centered at x of radius δ .
Let C ≡ C
( L, δ ) =
(cid:83) x ∈
L B δ ( x ) denote the region covered by δ -balls centered at the labeled examples in L .
We call C
( L, δ ) the covered region and P ( C ) the coverage .
Definition 2.1.
We say that a ball B δ ( x ) is pure if ∀ x ′ ∈
B δ ( x ) : f ( x ′ )
=
f ( x ) .
Definition 2.2.
We define the purity of δ as π ( δ ) =
P ( { x : B δ ( x ) is pure } ) .
Notice that π ( δ ) is monotonically decreasing.
Let ˆ f denote the 1-NN classifier based on L .
We split the covered region C ( L, δ ) into two sets:
C right = { x ∈ C :
ˆ f ( x ) =
f ( x ) } ,
C wrong =
C \ C right .
Lemma 1.
C wrong ⊆ { x : B δ ( x ) is not pure } .
Proof.
Let x ∈ C wrong .
Let c ∈
L denote the nearest neighbor to x .
Then they have the same predicted label, ˆ f ( x ) =
ˆ f ( c ) , and f ( c )
=
ˆ f ( c ) because c is labeled.
Since x is wrongly labeled, ˆ f ( x ) ̸ = f ( x ) , which implies that
[Equation]
Finally, since x ∈ C wrong ⊆
C is in the coverage, d ( x, c ) <
δ , which means that c ∈ B δ ( x ) with a different label and so B δ ( x ) is not pure.
Corollary 1.
P ( C wrong
[Equation]
≤ P
[Equation]
{ x
[Equation]
B
[Equation]
x ) is not pure
[Equation]
= 1 − π
[Equation]
Theorem 1.
The generalization error of the 1-NN classifier ˆ f is bounded as follows
[Equation]
Proof.
[Equation]
Note that (1) gives us a different bound for different δ values, which also depends on the labeled set
L .
This bound introduces a trade-off: as δ increases, the coverage increases, but the purity decreases.
Ideally, we should seek a pair { δ, L } that achieves the tightest bound.
Discussion We can interpret (1) in the context of two boundary conditions of AL: high-budget and low-budget.
In the high-budget regime, achieving full coverage P ( C ) = 1 is feasible as we have many points, and the remaining challenge is to reduce 1 − π ( δ ) .
Accordingly, since π ( δ ) is monotonically decreasing, we seek to minimize δ subject to the constraint P ( C )
= 1 .
This is similar to Coreset [36].
In the low-budget regime, full coverage entails very low purity, which (if sufficiently low) makes the bound trivially 1.
Thus, instead of insisting on full coverage, we fix a δ that yields "large enough" purity π ( δ ) > 0 , and then seek a labeled set L that maximizes the coverage P ( C ) .
We call this problem Max Probability Cover .
2.2 Max Probability Cover
Definition 2.3 ( Max Probability Cover ) .
Fix δ > 0 , and obtain a subset L ⊂ X, | L | =
b , that maximizes the probability of the covered area P ( C ( L, δ ))
[Equation]
An optimal solution to (2) would minimize the bound in (1), when δ is fixed.
Unfortunately, when moving to practical settings there are two obstacles.
The first is complexity:
Theorem 2.
Max Probability Cover is NP-hard.
Proof.
(Sketch, see full proof in App.
B)
We construct a reduction from an established NP-hard problem ( Max Coverage , see Def. A.1) to Max Probability Cover .
For the collection of subsets
S =
{ S 1 , . . .
, S m } , we consider the space R m and a collection of δ -balls { B δ ( x i ) } mi =1 with the exhaustive intersection property.
This means that any subset of the balls has at least one point that is contained in all the balls in the subset, but not contained in any other ball (see example in Fig. 2).
The existence of such a collection of balls in R m , ∀ m , is proved in Lemma 3 (see App. B).
We then assign each S i to B δ ( x i ) , and each element in S i is mapped to a point in the intersection of all the balls assigned to subsets that contain it.
Each such point then defines a Dirac measure, the normalized sum of which determines a probability distribution on R m .
The selection of δ -balls that is the solution to the Max Probability Cover can be translated back to a selection of subsets, which is the solution to the original Max Coverage problem.
[Figure]
2.3 Using the Empirical Distribution
When employing Max Probability Cover , the second practical problem concerns the data distribution, which is hardly ever known apriori.
In fact, even when known, the subsequent probabilistic computations are often intractable and hard to approximate.
Instead, we may use the empirical distribution ˜
P
[Equation]
A
[Equation]
mi [Equation] x i ∈ A as an approximation, which gives us the following useful result:
Proposition.
When P is the empirical distribution ˜ P , the Max Probability Cover objective function is equivalent to the Max Coverage objective, with { B δ ( x i ) ∩ X } mi =1 as the collection of subsets.
Proof.
Given a labeled set L = { x i } bi =1 , we show equality of objectives up to constant 1 | X |
[Equation]
2.4 The "Duality" of Max Probability Cover and Coreset
The Coreset AL method by Sener and Savarese [36] minimizes the objective
[Equation]
We can rewrite the above in the language of distributions as
[Equation]
If we use the empirical distribution then δ ( L ) = δ ′ ( L ) .
In this framework we can say that Max
Probability Cover and Coreset are dual problems in the following loose sense:
Max Probability Cover minimizes the generalization error bound (1) when we fix δ and seek to maximize the coverage, which is suitable for the low budget regime.
Coreset minimizes the generalization error bound (1) when we fix the coverage to 1 and minimize δ , which is suitable for the high budget regime because only then can we fix the coverage to 1.
This duality is visualized in Fig.
Method: ProbCover
To deliver a practical method, we first note that our approach implicitly relies on the existence of a good embedding space [4, 9, 51], where distance is correlated with semantic similarity, and where similar points are likely to bunch together in high-density regions.
As is now customary [e.g., 29, 18], we use an embedding space derived by training a self-supervised task over the large unlabeled pool.
In such a space similar labels often correspond to short distances, making 1-NN classification suitable, and also providing for the existence of large enough δ balls with good purity and coverage properties.
Secondly, we note that Max Coverage is NP-hard and cannot be solved efficiently.
Instead, as its objective is submodular and monotone [26], we use the greedy approximate algorithm that achieves (cid:0) 1 − 1 e (cid:1) -approximation [26].
A better approximation is impractical, as shown in App.
D.1.
See
App.
E for additional time and space complexity analysis.
Below, we describe the greedy algorithm in Section 3.1, and the estimation of ball size δ in Section 3.2.
3.1 Greedy Algorithm
Algorithm 1 ProbCover
Input: unlabeled pool U , labeled pool L , budget b , ball-size δ ,
Output: a set of points to query
X ← Embedding of representation learning algorithm on U ∪ L
G =
( V = X,
E = { ( x, x ′ ) :
x ′ ∈
B
δ ( x ) } ) for all c ∈ L do
Remove the incoming edges to covered vertices, { ( x ′ , x ) ∈ E : ( c, x ) ∈ E } , from E end for
Queries ← ∅ for all i=1,...
,b do
Add c ∈ U with the highest out-degree in G to Queries
Remove the incoming edges to covered vertices, { ( x ′ , x ) ∈ E : ( c, x ) ∈ E } , from E end for return Queries
The algorithm (see Alg.
below for pseudo-code) goes as follows:
First, construct a directed graph
G =
( V, E ) , with V =
X the embedding of the data space, and ( x, x ′ ) ∈
E
⇐⇒ x ′ ∈
B δ ( x )
⇐⇒
d
( x, x ′ ) ≤ δ .
In G , each vertex represents a specific example, and there is an edge between two vertices ( x, x ′ ) if x ′ is covered by the δ -ball centered at x (distances are measured in the embedding space).
The algorithm then performs b iterations of the following two steps:
(i) Pick the vertex x max with the highest out-degree for annotation; (ii) Remove all incoming edges to x max and its neighbors.
As ProbCover uses a sparse representation of the adjacency graph, it is able to scale to large datasets while requiring limited space resources.
The complexity analysis of the algorithm, and specifically the complexity of constructing the adjacency graph and of the sample selection , are discussed in the
Appendix E.
3.2 Estimating δ
Our algorithm requires the specification of hyper-parameter δ , the ball radius, whose value depends on details of the embedding space (see App.
C.1 for embeddings used).
In choosing δ , we need to consider the trade-off between large coverage P ( C ) and high purity π ( δ ) .
We resolve this trade-off with the following heuristic, where we pick the largest δ possible, while maintaining purity above a certain threshold α ∈ (0 , 1) .
Specifically,
[Equation]
Importantly, α is more intuitive to tune, and is kept constant across different datasets (unlike δ ).
We still need to estimate the purity π ( δ ) , which depends on the labels, from unlabeled data.
To this end, we estimate purity using unsupervised representation learning and clustering.
First, we cluster self-supervised features using k -means with k equal to the number of classes.
For a given δ , we compute the purity π ( δ ) using the clustering labels as pseudo-labels for each example.
Searching for the best δ , we repeat the process and pick the largest δ so that at least α = 0 .
95 of the balls are pure.
In Fig.
3, we plot the percentage of pure balls across different datasets as a function of δ , where the dashed line represents the δ ∗ chosen by ProbCover .
[Figure]
Empirical Results
We report a set of empirical results, comparing ProbCover to other AL strategies in a variety of settings.
We focus on the very low budget regime, with a budget size b of a similar order of magnitude as the number of classes.
Note that since the data is picked from an unlabeled pool, chances are that the initial labeled set is not going to be balanced across classes, and in the early stages of training some classes will almost always be missing.
ProbCover ’s excellent performance nevertheless, as seen below, demonstrates its robustness in the presence of this hurdle.
4.1 Methodology
Three deep AL frameworks are evaluated:
(i) Fully supervised : train a ResNet-18 only on the annotated data, as a fully supervised task.
(ii)
Semi-supervised by transfer learning : create a representation of the data by training with a self-supervised task on the unlabeled data, then construct a 1-NN classifier using the ensuing representation in a supervised manner.
This framework is intended to capture the basic benefits of semi-supervised learning, regardless of the added benefits provided by modern semi-supervised learning methods and the more sophisticated derivation of pseudo-labels.
(iii) Fully semi-supervised : train a competitive semi-supervised model on both the annotated and unlabeled data.
In our experiments we use FlexMatch by Zhang et al.
In frameworks (i) and (ii) we adopt the evaluation kit created by Munjal et al.
[32], in which we can compare multiple deep AL strategies in a principled way.
In framework (iii), we adopt the code and hyper-parameters provided by FlexMatch.
When evaluating frameworks (i) and (ii), we compare ProbCover to 9 deep AL strategies as baselines.
(1) Random query uniformly.
(2)-(4) query examples with the lowest score, using the following basic scores: (2) Uncertainty – max softmax output, (3)
Margin – margin between the two highest softmax outputs, (4) Entropy – inverse entropy of softmax outputs.
(5) BADGE [1].
(6) DBAL [14].
(7) TypiClust [18].
(8) BALD [25].
(9) W-Dist [29], see also App.
D.4.
(10) Coreset [36].
We note that while most baseline methods are suitable for the high budget regime, TypiClust and
W-Dist are also suitable for the low budget regime.
Similarly to ProbCover , TypiClust requires a good embedding space to work properly.
When comparing ProbCover and TypiClust , and in order to avoid possible confounds, we use the same embedding space for both methods.
These AL methods are evaluated on the following classification datasets: CIFAR-10/100 [27],
TinyImageNet, [28], ImageNet [11] and its subsets (following Van Gansbeke et al.
When considering CIFAR-10/100 and TinyImageNet, we use as input the embedding of SimCLR [8] across all methods.
When considering ImageNet we use as input the embedding of DINO [5] throughout.
Results on ImageNet-50/100 are deferred to App.
D.1.
Details concerning specific networks and hyper-parameters can be found in App.
C, and in the attached code in the supplementary material.
When evaluating frameworks (i) and (ii), we perform 5 active learning rounds, querying a fixed budget of b examples in each round.
In framework (iii), as FlexMatch is computationally demanding, we only evaluate methods on their initial pool selection capabilities.
4.2 Main Results
(i) Fully supervised framework.
We evaluate different AL methods based on the performance of a deep neural network trained directly on the raw queried data.
In each round, we query b samples where b is equal to the number of classes in each dataset, and train a ResNet-18 on the accumulated queried set.
We repeat this for 5 active learning rounds, and plot the mean accuracy of 5 repetitions (3 for ImageNet) in Fig.
4 (see App.
D.1 for additional results).
[Figure]
(ii) Semi-supervised by transfer learning.
In this framework, we make use of pretrained selfsupervised features, and measure classification performance using the 1-NN classifier.
Accordingly, each point is classified by the label of its nearest neighbor (within the selected labeled set L ) in the self-supervised features space.
In low budgets, this framework outperforms the fully-supervised framework (i), though it is not as effective as the full-blown semi-supervised learning framework (iii).
This supports the generality of our findings, not limited to any specific semi-supervised method.
Similarly to Fig.
4, in Fig.
5 we plot the mean accuracy of 5 repetitions for the different tasks.
[Figure]
(iii) Semi-supervised framework.
We compare the performance of different AL strategies used prior to running FlexMatch, a state-of-the-art semi-supervised method.
In Fig.
6 we show results withrepetitions of FlexMatch, using the labeled sets provided by different AL strategies and budget b equal to the number of classes.
We see that ProbCover outperforms random sampling and other AL baselines by a large margin.
We note that in agreement with previous works [6, 18], AL strategies that are suited for high budgets do not improve the results of random sampling, while AL strategies that are suited for low budgets achieve large improvements.
4.3 Ablation Study
We report a set of ablation studies, evaluating the added value of each step of ProbCover .
[Figure]
Random initial selection When following the uncertainty sampling principle, as many AL methods do, a trained learner is needed.
Any such method requires therefore a non-empty initial pool of labeled examples to train a rudimentary learner, from which uncertainty selection can be bootstrapped.
In the set of methods evaluated here (see Section 4.1), only two - ProbCover and TypiClust - are not affected by this problem.
This can be seen in Fig. 4, noting that only these two methods do better than random in the initial step.
Is this the only reason they outperform other methods in low budgets?
To address this question, we repeat the experiments reported in Fig.
4a-4b, using an initial random set of annotated examples across the board and by all methods.
Results are reported in Fig.
When comparing Fig.
4a-4b and Fig.
7, we see that the advantage of ProbCover and TypiClust goes beyond the initial set selection, and remains in effect even if this factor is eliminated.
[Figure] [Figure]
RGB space distances As discussed in Section 3, our approach relies on the existence of a good embedding space, where distance is correlated with semantic similarity.
We now verify this claim by repeating the basic fully-supervised experiments (Fig. 4) with one difference: ProbCover can only use the original RGB space representation to compute distances.
Results are shown in Fig.
When comparing the original ProbCover with its variant using RGB space, a significant drop in performance is seen as expected, demonstrating the importance of the semantic embedding space.
The interaction between δ and budget size To understand the interaction between the hyperparameter δ and budget b , we repeat our basic experiments (Fig. 4) with different choices of δ and b using CIFAR-10.
For each pair ( δ, b ) , we select an initial pool of b examples using ProbCover with δ balls, and report the difference in accuracy from the selection of b random points.
Average results across 3 repetitions are shown in Fig.
9 as a function of b .
We see that as the budget b increases, smaller δ ’s are preferred.
Coreset vs. ProbCover .
In Section 2.4 we argue that ProbCover is suitable for low budgets, while
Coreset is suitable for high budgets.
To verify this claim, we compare their performance under the following 3 setups while using the same embedding space, and report results on CIFAR-10:
• Low budget - Select an initial pool of 100 samples using the SimCLR representation.
[Figure]
• High budget - Train a model on 5K randomly selected examples.
Then select an additional set of 5K examples using the learner’s latent representation.
This is the setup used by Sener and Savarese [36].
• Mid budget - Same as high budget, except the initial pool size and added budget are 1K.
Results are reported in Fig.
In the low budget regime, ProbCover outperforms Coreset as would be expected.
In the mid-budget regime, where the feature space of the learner is informative, only
ProbCover achieves significant improvement over random selection.
In the high budget regime,
Coreset improves over random selection, while ProbCover is least effective.
Summary and Discussion
We study the problem of AL in the low-budget regime.
We model the problem as Max Probability
Cover , showing that under certain assumptions on the data distribution, which are likely to hold in self-supervised embedding spaces, it optimizes an upper-bound on the generalization error of a 1-NN classifier.
We devise an AL strategy termed ProbCover , which approximates the optimal solution.
We empirically evaluate it in supervised and semi-supervised frameworks across different datasets, showing that ProbCover significantly outperforms other methods in the low-budget regime.
In future work we intend to investigate: (i) possible avenues for improving the choice of δ by making use of already known labels, or inferring a score for δ through the topology of the resulting covering graph; (ii) extensions of the current formulation of Max Probability Cover , by making δ - the radius of the balls - dependent on the samples rather than uniform; (iii) soft-coverage approaches, where the covering notion is not binary but some continuous measure, which may allow us to do away with δ .
Acknowledgments
This work was supported by the Israeli Ministry of Science and Technology and by the Gatsby
Charitable Foundations.
We are grateful to our dedicated NeurIPS AC, who acted upon the lengthy discussion between us and the reviewers, as seen on OpenReview.
Appendix A Some Definitions
In the proof of the main theorem below we use Max Coverage , a known NP-hard problem.
We recapitulate its definition as follows:
Definition A.1 ( Max Coverage ) .
Let b ∈
N denote an integer, U denote a set of elements, and let
S =
{ S 1 , . . .
, S m } denote a collection of subsets of U .
In the problem of Max Coverage we wish to find b subsets in S with union of maximum cardinality argmax
[Equation]
For convenience, we also repeat the definition of Max Probability Cover from Section 2.2:
Definition A.2 ( Max Probability Cover ) .
Let X denote the input probability space.
Let X
= { x i } mi =1 , x i ∈
X denote a set of points.
Let b ∈
N denote an integer, and fix δ > 0 .
In the problem of Max Probability Cover we wish to find a subset L ⊂ X, | L | =
b , that maximizes the following probability
[Equation]
B Max Probability Cover is NP-Hard
We first describe a constructive procedure to generate a collection of balls in R m with a property we call exhaustive intersection (Def. B.2).
We then use this collection in order to construct a reduction from Max Coverage to Max Probability Cover .
B.1 Exhaustive Intersection : Constructive Procedure
Let { e i } mi =1 denote the natural basis of R m .
Let B r ( p ) denote the open ball of radius r around p , and
Let B r
[ p ] similarly denote the closed ball.
Definition B.1 (Inversion mapping) .
We define the inversion mapping ι : R m \ { 0 } → R m \ { 0 } as
[Equation]
Lemma 2.
Let X i = { x ⊆ R m | x · e i > 1 } denote a halfspace.
Then ι ( X i ) ⊆
B 12 ( 12 e i ) .
Proof.
Let x ∈ X i .
Membership in the two sets is defined by satisfying the equations x · e i > 1 and d ( ι ( x ) , 12 e i ) <
We will show that they are equivalent (see visualization in Fig. 11a).
We use the polarization identity
[Equation]
[Figure]
Corollary 2.
x ̸ = 0 , x · e i < 1 ⇐⇒ ι ( x ) / ∈ B 12 [ 12 e i ] .
Definition B.2 (Exhaustive intersection) .
A collection of sets ( A 1 , . . .
, A m ) is said to have the exhaustive intersection property if for any subset of indices
I ⊂ [ m ] there exists a point
x
I with
x I ∈ (cid:84)
i ∈ I
A
i 2.
x I / ∈ (cid:83) j ∈
[ m ] \ I
A j
Put differently, x
I ∈
A
i ⇐⇒ i ∈
I (see the Venn diagram example in Fig. 11b).
Lemma 3.
The collection of balls { B 12 ( 12 e i ) } mi =1 in R m satisfies the exhaustive intersection property.
Proof.
Let I ⊆
[ m ] denote a subset of indices and define ˜ x
I = (cid:80) j ∈ I
e j , x I =
ι
(˜ x I ) .
From
Lemma 2, for any j ∈ I, ˜ x I · e j = 2 ⇒ x I ∈
B 12 ( 12 e j ) , which proves the first part of Def.
B.2.
From Cor. 2, any j / ∈ I, ˜ x
I · e j = 0 ⇒ x I / ∈ B 12 ( 12 e j ) , which proves the second part.
B.2 Reduction from Max Coverage to Max Probability Cover
To improve readability, we restate in App.
A the precise definitions of Max Probability Cover (Def. A.2) and Max Coverage (Def. A.1).
Before we start, we note that in order for Max Probability
Cover to be computable and not be trivially polynomial, we must assume that the encoding of the input distribution to Max Probability Cover is of polynomial size in the number of points.
Theorem 2.
Max Probability Cover is NP-hard.
Proof.
We construct a polynomial-time reduction from any Max Coverage problem P to another problem ˜ P , which is an instance of Max Probability Cover .
Let In = ( S 1 , . . .
, S m , b ) be the input to P , and let U = (cid:83) mi
=1 S i .
We define a mapping from the input of P to the input of ˜ P : First, we fix the input space of ˜ P to R m , where m is the number of sets in P , and fix the set of m points X in ˜ P to { 12 e i } mi =1 ⊆
R m .
We let b remain the same, and fix the radius δ = 1 2 .
We abbreviate B i =
B 1 2 ( 1 2 e i ) .
Next, we define a mapping
T : U → R m as T ( u ) =
ι ( (cid:80) mi =1 2 · 1 u ∈ S
i e i ) .
From the proof of Lemma 3 we conclude that
[Equation]
Finally, we define the probability distribution P as P ( A ) =
| U | (cid:80)
u ∈ U 1 T ( u ) ∈ A (see visualization in
Fig. 11c).
Lemma 4.
P is a valid probability distribution.
Proof.
P is a finite sum of Dirac measures 1 T ( u ) ∈ A , and as such it is a measure itself.
Hence we only need to show that it is normalized, ie P ( R m )
= 1 :
[Equation]
In summary, the input of ˜ P is the following:
• Dataset: X ( In ) =
{ 12 e i } mi =1 ⊆
R m .
• Budget:
b ( In ) =
b .
• Ball radius: δ ( In ) =
12 . • Distribution: ( P ( In ))
( A ) =
| U | (cid:80)
u ∈ U 1 T ( u ) ∈
A
Before we continue, we require another short lemma:
Lemma 5.
For all u ∈ U , I ⊆
[ m ] ,
[Equation]
Proof.
We prove the conditions are equivalent:
[Equation]
To show that a reduction is valid for an optimization problem, we need to show that the objectives are equivalent.
The objective in Max Coverage is the size of the union, whereas in Max Probability
Cover it is the probability of the δ -ball union.
[Equation]
Since | U | is constant the optimization is equivalent.
Finally, we show that the induced probability can be specified in polynomial space: as | U | is polynomial in the size of the input, it follows that the distribution as a normalized sum of indicator functions can be specified as a table of the embedding of size | U | · m , which is polynomial.
C Implementation Details
Source code used in this work is available at the following url: https://github.com/avihu111/TypiClust
C.1 Selection Method
Representation Learning: CIFAR10, CIFAR100, TinyImageNet.
To extract semantically meaningful features, we trained SimCLR using the code provided by Van Gansbeke et al.
[40] for CIFAR-10,
CIFAR-100 and TinyImageNet.
Specifically, we used ResNet-18 [19] with an MLP projection layer to a 128 -dim vector, trained for 500 epochs.
All the training hyper-parameters were identical to those used by SCAN (all details can be found in Van Gansbeke et al.
After training, we used thedimensional penultimate layer as the representation space.
Representation Learning: ImageNet.
We extracted features from the official (ViT-S/16) DINO weights pre-trained on ImageNet.
We used the L2 normalized penultimate layer for the embedding.
All the exact hyper-parameters can be found at Caron et al.
Randomness in ProbCover Selections.
In order to reduce the correlation between different repetitions using ProbCover , we added the following modification to the selection algorithm: instead of taking the node with the highest degree at each iteration, we selected randomly one of the 5 nodes with the highest degree.
We verified that both algorithms achieved similar performance, where the deterministic version has slightly better results.
C.2 Fully Supervised Evaluation
We trained a ResNet18 on the labeled set, using the AL comparison framework created by Munjal et al.
[32], and following the protocol described in [18] (see details in [18] and the shared code).
C.3 1-NN Classification with Self-Supervised Embeddings
In these experiments, we also used the framework by Munjal et al.
We extracted an embedding similar to § C.1, with which we trained a 1-NN classifier using the default parameters of scikit-learn .
C.4 Semi-Supervised Classification
When training FlexMatch [50], we used the AL framework by Zhang et al.
All experiments involved 3 repetitions.
CIFAR-10.
We used the standard hyper-parameters used by FlexMatch [50].
Specifically, we trained
WideResNet-28 for 400k iterations using the SGD optimizer, with 0 .
03 learning rate, 64 batch size, 0 .
9 momentum, 0 . 0005 weight decay, 2 widen factor, and 0 .
1 leaky slope.
The weak augmentations used are identical to those used in FlexMatch and include random crops and horizontal flips, while the strong augmentations were generated by RandAugment [10].
CIFAR-100.
Similar to CIFAR-10, but increasing the widen factor to 8 .
TinyImageNet.
We trained ResNet-50, for 1.1m iterations.
We used an SGD optimizer, with a 0 .
03 learning rate, 32 batch size, 0 .
9 momentum, 0 .
0003 weight decay, and 0 .
1 leaky slope.
The augmentations were similar to those used in FlexMatch.
D Additional Empirical Results D.1 Improving the greedy approximation
The greedy approximation used in ProbCover guarantees 1 − 1 e approximation to the maximum cover problem.
Hunt et al.
[23] showed that a polynomial time approximation scheme (PTAS) exists for this problem, suggesting the possibility of better polynomial approximations.
However, Marx [30] proved that there is no efficient PTAS to this problem, implying that such polynomial approximations may not be practical.
For example, to achieve a 1 − 1 e approximation using the PTAS suggested in Marx [30] would require O ( n 100 ) time.
Thus, a significantly better approximation than the greedy solution is left for future work.
Instead, we improved the greedy algorithm by choosing at each iteration the optimal 2 balls in a greedy way.
While this greedy solution achieves a better approximation in theory, in practice we did not see any improvement over the single-ball greedy solution.
D.2 ImageNet subsets
When evaluating ProbCover on ImageNet-50 and ImageNet-100, we report a similar qualitative behavior as seen in other datasets: ProbCover performs better than all baselines in the very low-budget regime, using 5 AL rounds with a budget equal to b = 50 examples.
More concretely, in Fig.
12 we show results corresponding to Figs.
3-5 when using ImageNet-50.
[Figure]
D.3 TypiClust vs ProbCover on SCAN feature space
Both ProbCover and TypiClust use a unsupervised self-representation embedding as part of an active learning query selection algorithm.
In Section 4.2, when comparing ProbCover to TypiClust , we used the same embedding in both of them, to avoid possible confounds relating to the choice of the specific representation algorithm.
As TypiClust reached the best performance using SimCLR representation in most budgets and frameworks on CIFAR-10 and CIFAR-100, we chose that embedding space to compare to ProbCover .
However, in the fully-supervised framework, with a budget of 10 examples, TypiClust yields better results using the embedding space of SCAN.
In Fig. 13, we plot comparison between ProbCover and TypiClust in this budget, when both are using the embedding space of SCAN.
We find a similar trend to the results reported in Section 4.2:
ProbCover achieves higher accuracy than TypiClust , and both surpass random sampling in this budget.
D.4 Comparison with W-Dist
In Section 4, we compare ProbCover to several other active learning baselines, including W-Dist [29].
As this method is computationally demanding, we are only able to evauate its performance on the
CIFAR-10 and CIFAR-100 datasets, which are the smallest datasets we consider.
We note that the results differ from the results reported in the original paper.
This stems from several things: firstly, we use 1-NN classification instead of linear classification in the self-supervised scenario.
Secondly, the implementation of the Wasserstein method is ours, based on the pseudo-code published in the original work, as no official implementation is available, though we did our best to follow the instructions of the original paper.
Thirdly, the method is unique in that it requires a long time to select samples (the original version set a 3 hours timeout for the selection of every 10samples).
Instead of the long timeouts suggested in the original work, we used 20 minutes timeouts per round, which reached similar results.
E Time and Space complexity of ProbCover
During the training of a neural network, ProbCover is executed a single time in order to select the best subset to query for human annotation for subsequent network training.
For the complexity calculation below, let n denote the number of examples in the unlabeled and labeled pool | U ∪ L | , d the dimension of the data embedding space, and b the query budget.
ProbCover can be split into two steps:
E.1 Adjacency graph
Constructing the adjacency graph requires computing pairwise distances in the embedding space.
Time Complexity : O ( n 2 d ) time.
In practice, it takes roughly 10 minutes on a single NVIDIA
A4000 GPU even on the largest dataset we consider – ImageNet with DINO embedding, where n = 1281167 , d = 384 .
Space Complexity : naively we have O ( n 2 ) , which is impractical for large datasets like ImageNet.
However, we only need to save edges whose distance is smaller than δ .
We store the edges using a sparse matrix in coordinate list (COO) format, so the space complexity is O ( | E | ) , where E is the set of edges in the graph.
Although O ( | E | ) is still O ( n 2 ) in the worst case, in practice, the average degree of each vertex in the graph using radius δ is a few orders of magnitude smaller than n , resulting in manageable space complexity.
For example, When selecting samples from ImageNet with δ = 0 . 55 , the average degree was 24 and the algorithm total memory consumption was 12 GB .
E.2 Sample selection
We iteratively select samples from the current sparse graph, removing incoming edges to newly covered samples.
We note that unlike the adjacency graph creation, the sample selection cannot be parallelized, as each selection step depends on the previous step.
Time Complexity
Breaking down the steps in the selection of a single sample:
• Calculating node degrees – O ( | E | ) time.
• Finding node with a maximal degree – O ( n ) time.
Removing covered points’ incoming edges from the graph – O ( | E | ) time.
All in all, the complexity is O ( | E | + n ) for selecting a sample, and O ( n 2 k ) overall in the worst case.
As we select more and more points, more edges are removed, making the selection of later samples faster.
In practice, thanks to the vectorization of these steps it takes roughly 15 minutes to select k = 1000 samples from ImageNet on a single CPU, and a couple of seconds in CIFAR-10/100.