M I SAL: A CTIVE L EARNING FOR E VERY B UDGET
Anonymous authors
Paper under double-blind review
A BSTRACT
In supervised Active Learning (AL), the learner can manipulate the labeled training set by choosing examples to be labeled by an oracle.
The size of the labeled set is termed budget .
Recent years have seen significant progress in this domain in the context of deep active learning.
In particular, it has been shown that in general, different families of AL strategies are suitable for high and low budgets.
Here we address for the first time the problem of deciding which family of strategies is most suitable for a given budget in a given task.
We start from the theoretical analysis of a mixture model, which motivates a computational approach to decide on the most suitable family of methods for the task and budget at hand.
We then propose a practical decision algorithm, which determines what family of strategies should be preferred.
Using this algorithm, we introduce MiSAL – a mixed strategy active learning algorithm.
MiSAL combines AL strategies from different families, resulting in a method that fits all budgets.
We support the analysis by an empirical study, showing the superiority of our method when dealing with image datasets.
I NTRODUCTION
In active learning, a learner is allowed to actively interfere with the learning process in order to improve the outcome.
Here we consider active learning in the context of the traditional supervised learning framework, in which a learner can influence and shape the construction of the labeled training set.
Specifically, we assume that the learner is given an initial set of labeled examples (which could be empty) and a set of unlabeled examples.
The learner is then asked to choose a subset of the unlabeled set (termed the active set), up to a fixed budget, and subsequently receive the labels of the selected examples from an oracle.
The challenge is to do this in an optimal manner.
An effective active learner is expected to do better than the random selection of the active set.
The optimal active learning strategy clearly depends on the learner’s inductive biases and the nature of the problem.
But even when all this is fixed, Hacohen et al.
(2022) showed that the optimal strategy still varies, depending on the size of the set of labeled examples (henceforth termed budget ).
Specifically, when the budget is large, methods based on uncertainty-sampling or related criteria are most effective, while typicality-based methods are effective when the budget is small (see Fig. 1).
In practice, given specific data and budget, it is unclear how to determine whether the budget is considered large or small, making it unclear which active learning strategy should be used.
We aim to alleviate this problem, by suggesting a practical algorithm to determine which active learning algorithm should be used.
Starting from an abstract theoretical analysis in Section 2, we describe an algorithm to decide upon this question.
We then derive an active learning strategy that is suitable for all budgets as described in Section 3.
The results of an extensive empirical study, showing that the proposed method is beneficial (and often superior) at all budgets, are described in Section 4.
R ELATION TO PRIOR WORK
By now there is a large body of fundamental work on active learning, see for example the surveys by (Settles, 2009; Schröder & Niekler, 2020).
The natural approach, which dominates deep active learning and recent work in particular, aims to identify data whose contribution to the learner achieves maximal added value with respect to what the learner already knows.
In the context of supervised learning, the learner may be trained on a fairly large set of already known labeled examples.
Afterward, it is allowed to pick unknown examples for human annotation based on the outcome of training.
These methods often employ either uncertainty sampling (Lewis & Gale, 1994; Ranganathan et al., 2017; Gissin & Shalev-Shwartz, 2019; Sinha et al., 2019) – seeking examples that the learner is least certain about, diversity sampling (Hu et al., 2010;
Elhamifar et al., 2013; Sener & Savarese, 2018;
Shui et al., 2020) – seeking examples that effectively span the data space, or some combination of both (Gal et al., 2017;
Kirsch et al., 2019; Ash et al., 2020).
But what if not much is known to the learner apriori, and effective training is not possible before the selection of queries?
Recently, leveraging advances in unsupervised representation learning (e.g.,
Van Gansbeke et al., 2020; Chen et al., 2020), another set of active learning methods has emerged (Mahmood et al., 2021;
Hacohen et al., 2022; Yehuda et al., 2022).
Such works assume a very small (sometimes even empty) set of apriori known labels, and they rely instead on a representation space as obtained by some unsupervised learner.
Methods designed to be effective in this domain tend to be qualitatively different from the more traditional methods, seeking examples that can be easily learned rather than seeking confusing examples.
In light of this dichotomy, a new exciting question has emerged when trying to put it all together: in a particular context, what family of methods should be preferred – those designed for high budgets or those designed for low budgets?
In particular, can we identify in advance, before we select any query, which family of methods to use?
Or maybe a mixture of such methods is to be preferred?
To the best of our knowledge, this is the first work that aims to answer this question, developing a practical deep active learning strategy for all budgets.
The method is versatile, in that it can incorporate state-of-the-art AL strategies suitable for either the high-budget or low-budget domains.
T HEORETICAL A NALYSIS
In the active learning scenario considered here, a learner is given an initial set of labeled examples (which can be empty) and a set of unlabeled examples.
The learner is allowed to choose a subset of the unlabeled set, up to a fixed budget, and obtain their labels from an oracle to be used for further training.
The selection of examples to be labeled can be done repeatedly, querying in each iteration only part of the total budget.
In order to focus the analysis, we consider henceforth a single iteration.
In each iteration, the learner is given two sets of examples – one labeled and one unlabeled, and the task is to optimally select a fixed-sized subset of the unlabeled set to be labeled.
In Section 2.1 we introduce some notations and a computational model, which is used in Sections 2.22.3 to derive optimal active learning strategies and reveal how they depend on the learning budget.
2. P RELIMINARIES
Notations Consider an active learning scenario with an unlabeled set of examples U , a fixed budget
B ∈ N , an active learner L , and an initial set of labeled examples L of fewer than B examples.
The goal of L is to find an optimal set of examples
A ⊆ U (referred to as the active set ), of size m ≡ | A | =
B − | L | , which would maximize the benefit to learner L when it is to be trained with the labeled set T =
A ∪ L of size | T | = B .
Active set selection Let H L ( T ) denote a hypothesis delivered by learner L when trained with T , and let Er ( H L ( T )) denote its generalization error.
Since deep learning training is stochastic, H L ( T ) is itself a random variable.
Let E L ( B ) =
E [ Er ( H L ( T ))] denote the mean generalization error of L over all possible hypotheses and all training sets T of size B .
Model definition and assumptions We adopt the mixture model and assumptions introduced in (Hacohen et al., 2022).
The input domain of the mixture model is assumed to be composed of two disjoint regions, R l and R h such that R l ∪ R h is the entire input domain.
The mixture model is composed of 2 independent general learners, each learning one of the disjoint regions of the input. We study the error function of the mixture model on the entire input domain, which depends on the error function of each of the learners in its corresponding region. While the error functions of each learner may be different, we assume for simplicity that their mean generalization error takes on a universal form.
In other words, the mean generalization error of each learner is a function that depends only on the size of the training set, up to some constant α which may differ for each region.
Formally, the error function of each learner i ∈ { l, h } can be written as E ( α i B i ) , where E : R → [0 , 1] is the universal error function, B i is the budget in region i , and α i ∈ R .
We further assume that E has two properties: (i) Efficiency: E ( αB ) is strictly monotonically decreasing, namely, on average the learner benefits from additional examples.
(ii) Realizability: lim B →∞ E ( αB )
= 0 .
Given the above assumptions, the generalization error of learner L can be formalized as:
[Equation]
Above p denotes the probability of region R l .
We assume w.l.o.g. that α l = 1 and α h ≡ α > 0 .
Without loss of generality, we assume that R l requires fewer examples to be adequately learned than
R h , which implies that α < p 1 − p .
With this assumption, the prior art shows that it is beneficial to sample from the easier-to-learn region R l when the budget is low, and sample from the harder-to-learn region R h when the budget is high.
Within this framework, the exact transition point between the low and the high budgets depends on the specific parameters of the problem at hand.
2. O PTIMAL MIXED STRATEGY FOR QUERY SELECTION
We now analyze active learning strategies using the mixture model defined in Section 2.1.
In this model, there exist two pure query selection strategies, denoted S l and S h .
S l samples points from R l , and is beneficial for low budgets.
S h samples points from R h , and is beneficial for high budgets.
Given a fixed budget B , we analyze the family of mixed strategies – each defined by q ∈ [0 , 1] , in which qB points are sampled using S l and (1 − q ) B points are sampled using S h .
We seek the optimal mixed strategy denoted ˆ q , which delivers a labeled set of size B with the lowest mean error.
First, we note that the mean generalization error of mixed strategy q is:
[Equation]
Since E is differentiable by assumption, we can find the optimum at ˆ q by differentiating (2) (see derivation in App.
A), to obtain:
[Equation]
If (3) has a solution for ˆ q and if this solution is unique and minimal, it defines an optimal mixed strategy ˆ q E ( B, p, α ) .
∀ B such that ˆ q E ( B, p, α )
= p , this optimal mixed strategy is equivalent to random selection from the input domain 1 .
We denote such budgets by B equiv , where from (3)
[Equation]
By definition, when the budget size is B equiv , random query selection is optimal.
In the analysis below, we assume for simplicity 2 that ˆ q E ( B, p, α ) is non-increasing ∀ B and strictly monotonic at B equiv , implying that B equiv is unique.
A visualization of this case can be seen in Fig.
2, where we plot the closed form solution of ˆ q a ( B, p, α ) with an exponential universal error function E ( x ) = e − ax .
[Figure]
2. O PTIMAL SELECTION STRATEGY : DEPENDENCE ON THE SIZE OF THE ACTIVE SET
In the active learning scenario as defined in Section 2.1 we have T =
L ∪ A , where set L is fixed and the optimal selection of set A is sought.
Let m denote the size of the required set, namely, | A | = m and | L | =
B − m .
Not knowing anything about the origin of L , we further assume that it has been selected randomly from the input domain, which is equivalent to the mixed strategy q = p .
In this case, the selection of set T is more naturally conceived as a mixture between 3 strategies {S r , S l , S h } , where strategy S r randomly selects data from the input domain.
This is captured by the notation S ( r r , r l , r h ) , where ( r r , r l , r h ) in the 3-simplex, for a query selection strategy that selects r r B points with S r , r l B points with S l , and r h B points with S h .
Note that this notation is unique only if either r l = 0 or r h =
(or both).
Optimal mixed strategy
The unique optimal strategy, which corresponds to mixture ˆ q ≡ ˆ q E ( B, p, α ) , is captured by this notation as:
[Equation]
Above ˆ r is obtained by noting that ˆ r + (1 − ˆ r ) p = ˆ q , which implies that
[Equation]
Achievable optimal mixed strategy Are all optimal mixed strategies feasible?
Since the size of set A is fixed at m , we can deduce from (5), and the definition of vector ( r r , r l , r h ) , that strategy ˆ r is not feasible whenever one of the following two conditions holds ˆ
[Equation]
or
[Equation]
6) ˆ
[Equation]
Above, B low is defined to be the solution of (3) when inserting ˆ q → p + m (1 − p ) B , and B high is the solution of (3) when inserting ˆ q → p − m (1 − p ) B (see derivation of B low and B high in App.
A).
We may conclude that the optimal strategy (4) that is also achievable is:
[Equation]
Comparing (7) to (4), we note that at low budgets ( B < B low ) one selects the active set with pure strategy S l since more than m points (the size of the active set) are missing from region R l to achieve optimal performance.
At high budgets ( B > B high ) one selects the active set with pure strategy S h since more than m points are missing from region R h to achieve optimal performance.
Achievable optimal mixed strategy: incremental selection Inspecting the definition of the transition points in the optimal achievable strategy, defined in (6), we observe that lim m → 0 B low =
lim m → 0 B high =
B equiv .
In other words, if the active set A is small enough, the achievable optimal mixed strategy can be effectively approximated by the following strategy
[Equation]
In (8), if the budget size is smaller than B low then the active set is sampled from R l , and if the budget size is larger than B high then the active set is sampled from R h .
Otherwise, the active set is randomly sampled from all the data.
The deviation of (8) from the optimal strategy (7) lies in a segment of budget values whose size tends to 0 as m decreases (see visualization in Fig. 2b).
If m is not sufficiently small to justify the use of strategy (8), query selection may be done incrementally: First, m is to be divided to smaller segments of length m ′ , where m ′ is small enough to justify the use of (8).
Second, strategy (8) is used repeatedly mm ′ times, to build the set A iteratively.
While taking longer to run, this iterative strategy can be implemented more easily and more robustly than strategy (7), since it does not require the evaluation of the elusive mixture coefficient ˆ r .
Further support for the use of strategy (8) can be derived from our empirical results (see Section 2. below).
In our experiments, the difference in the mean generalization error between strategies (7) and (8) is marginal even when the truly optimal ˆ r is manually handed to the algorithm.
This supports the adaptation of strategy (8) in practice with a small number of repetitions.
2. V ALIDATION AND VISUALIZATION OF THEORETICAL RESULTS
Visualization In Fig.
3, we plot the error of the strategies defined in (7) and (8), using the same exponential example as in Fig.
The orange curve represents a fairly large active set, where m (the size of active set A ) is equal to 30% of budget B , while the blue curve represents a very small active set, where m is equal to 1% of the budget B .
We can see that the smaller m is, the smaller the gap between the optimal strategy in (7) and the achievable strategy in (8) becomes.
Fig.2b shows plots of the optimal mixture coefficient ˆ q as a function of budget size B , when considering different values for the active set size m , including 1% , 30% and 100% of B .
We see that as the theory predicts, the smaller m is, the more step-like the optimal ˆ q becomes, suggesting that in most cases, the entire active set A should be sampled from the same strategy.
Validation
Our analysis uses a mixture model of idealized general learners.
We now validate that similar phenomena occur in practice when training deep networks on different computer vision tasks.
As our choice for pure strategies S l and S h , we use TypiClust and margin respectively (see details in
Section 4.1).
Fig.
4 shows results when training N = 20 networks using mixed strategies S (1 − r, 0 , r ) and S (1 − r, r, 0) on CIFAR-10 and CIFAR-100.
We compare the mean performance of each strategy to the performance of N = 20 networks trained with the random query selection strategy S (1 , 0 , 0) .
We note that other choices for S l and S h yield similar qualitative results, as can be seen in Tables 1-2.
Inspecting these results, we find similar trends to those shown in the theoretical analysis.
The most beneficial value of mixture coefficient r decreases as the budget increases until a certain transition
[Figure]
point (corresponding to B equiv ).
From this transition point onward, the bigger the budget is, the more beneficial it is to select additional examples by one of the high-budget strategies.
In fact, as in the theoretical analysis, when the budget is low it is beneficial to use a pure low-budget strategy, when the budget is high it is beneficial to use a pure high-budget strategy, and the transition area in between (corresponding to the segment B low ≤ B ≤ B high ) is typically rather short (see Figs.
4 and 5).
M I SAL: M IXED S TRATEGY FOR A CTIVE L EARNING
Following the theoretical discussion in Section 2.3, our method adopts the achievable optimal mixed strategy as prescribed in (8).
The method involves two main steps.
In the first step, a strategy beneficial for low budgets and another beneficial for high budgets is chosen.
For each strategy, we evaluate whether it is more effective than random sampling at the given budget B , and estimate its added generalization accuracy for the given query size m as explained in Section 3.1.
This step determines the budget’s actual domain: (i) low ( B ≤ B low ), (ii) transition ( B low ≤ B ≤ B high ), or (iii) high ( B ≥ B high ).
In the second step, a strategy is chosen from among the most competitive ones in the relevant domain, and is subsequently used to select m points for active set A .
3. D ECIDING ON THE SUITABLE BUDGET REGIME
Let S r denote the strategy that selects query points randomly.
Given two strategies S l and S h and given budget B , the method outlined in Alg. 1 below aims to determine (separately for each strategy) whether its expected performance is better than random query selection, or not.
In other words, we need to determine whether m additional points, as chosen by either S l or S h respectively, reduce the mean generalization error more significantly than m randomly selected points.
Unfortunately, this test cannot be decided directly without additional labels.
Instead, our method computes the result of a surrogate test, where a small set of points as chosen by each respective strategy is removed from the labeled set.
It then compares the reduction in mean generalization error to the removal of randomly chosen points.
The proposed surrogate test raises another problem: in most active learning strategies, in particular those suitable for high budgets, the outcome relies on a learner that is trained on the labeled set L and is thus exposed to the points that are to be removed.
As a result, the test is strongly biased to underestimate the cost of removing known points, as also seen in our empirical study (see Fig. 6).
Instead, in order to achieve an unbiased surrogate test, we restrict the choice of active learning strategies to methods that rely only on the unlabeled set U .
We denote these methods by S ′ l and S ′ h .
The final method can now be summarized as follows (see Alg. 1): We begin by considering the unlabeled pool U , and obtain an effective representation for all the data using a self-supervised learning task.
In the feature space thus defined, we create three subsets of L : data l , data h and data r .
Each subset is obtained by removing a small number of examples (but not less than 1) from each
Input: Unlabeled pool U , Labeled pool L , Budget B , learner L , ϵ , 2 Strategies: S ′ l , S ′ h
Output: Decision variable S features ← features of data L obtained by a self-supervised task trained on U k ← # of classes, c ← max { (cid:4) ϵk (cid:5) , 1 } data l , data h ← remove
c examples per class as chosen by S l , S h respectively data r ← remove c examples per class randomly acc l , acc h , acc r ← accuracy of L trained on data l , data h , data r respectively if acc l < acc r and acc l ≤ acc h then
S ← ‘ low ′ else if acc h < acc r and acc h ≤ acc l then
S ← ‘ high ′ else
S ← ‘ rand ′ end if return S class as chosen by S ′ l , S ′ h , and S r respectively.
We then train the learner on each of these datasets (this is repeated multiple times for S r ).
Finally, we compute the accuracy of each method using cross-validation, using 1% of the training data as a validation set in multiple repetitions.
We choose the strategy whose accuracy is the lowest – this is the strategy that is most critical for learning, hence removing examples according to it decreases the performance the most.
3. M IXED ACTIVE LEARNING STRATEGY
The active learning strategy for query selection is described below in Alg. 2.
First, it selects two active learning strategies, S ′ l , and S ′ h , which are known to be beneficial in the low and high budget regimes respectively, and which additionally rely in their computation only on the unlabeled set U .
Together with the data, they are given as input to Alg. 1, which in turn returns a decision, indicating what regime best describes the given budget B .
In its second step, Alg. 2 selects two active learning strategies, S l , and S h , which are known to be beneficial in the low and high budget regimes respectively, with no additional restrictions.
This allows the selection of strategies more competitive than S ′ l and S ′ h .
The additional flexibility is especially important in the high budget regime, where the most competitive strategies typically rely in their computation on all the data, including both L and U .
Alg.
2 then uses the preferred AL strategy ( S l , S h or S r ) to select the active set A .
Algorithm
-Strategy Active Learning (MiSAL)
Input: Unlabeled pool U , Labeled pool L , Budget B , learner L
Output: active set
A of B − | L | examples m ← B − | L |
S ′ l , S ′ h ← competitive AL methods that rely only on U
S ← output of Alg.
1 given ( U , L , B, L , m, S ′ l , S ′ h )
S l , S h ← competitive AL methods (unrestricted) if S =
= ‘ low ′ then
A ← m points selected by S l from U else if S
==
‘ high ′ then A ← m points selected by S h from U else A ← m random points from U end if return A
The use of S ′ l and S ′ h in the call to Alg. 1, rather than S l and S h , is a necessary evil as explained in Section 3.1.
The rationale for expecting our method to succeed lies in the theoretical analysis presented in Section 2, which supports the hypothesis that the transition points B low and B high may be universal, or approximately so, across pure strategies.
Our empirical results, reported in Section 4, support this supposition.
E MPIRICAL R ESULTS
We now describe the empirical results of our integrated active learning strategy, where Alg. 2 is used to generate the active set.
After labels are obtained for the active set, the training of the deep model proceeds as is customary in deep supervised learning, using all the available labels in A ∪ L .
M ETHODOLOGY
Our experimental framework is based on the code of (Munjal et al., 2020), which allows for the comparison of different active learning strategies in a robust and fair way.
While the architecture trained by this framework does not achieve state-of-the-art results on CIFAR and ImageNet, it makes it possible to compare many active learning methods in a competitive environment, while using an architecture with which these strategies have been shown to be beneficial.
In the following experiments, we trained ResNet-18 (He et al., 2015) on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009) and ImageNet-50 – a subset of ImageNet (Deng et al., 2009) containing 50 classes as done in (Van Gansbeke et al., 2020).
Hyper-parameters are the same as in Munjal et al. , see App.
B.
As our choice of competitive pure active learning strategies that use only the unlabeled set U for training (see Alg. 2), we choose TypiClust (Hacohen et al., 2022) for S ′ l and inverse TypiClust for S ′ h .
In the latter strategy, the most atypical examples are selected; we note that inverse TypiClust is an effective strategy for high budgets, see App. C.1.
As our unrestricted choice of competitive pure AL strategies, we choose ProbCover (Yehuda et al., 2022) for S l and BADGE (Ash et al., 2020) for S h .
Other choices yield similar patterns of improvement, as can be verified from Tables 1-2.
In the experiments below, we use several active learning strategies, including Min margin , Max entropy , Least confidence , DBAL (Gal et al., 2017), CoreSet (Sener & Savarese, 2018), BALD (Kirsch et al., 2019), BADGE (Ash et al., 2020), TypiClust (Hacohen et al., 2022) and ProbCover (Yehuda et al., 2022).
When available, we use for each strategy the code provided in (Munjal et al., 2020).
For low-budget strategies, which are not implemented in (Munjal et al., 2020), we used the available code provided in the git of each paper.
When using Alg. 2, the underline feature space is SimCLR.
Other choices of feature spaces yield similar results, see the comparison in App. 8.
4. E VALUATING A LG . 1 IN ISOLATION
We start by isolating the strategy selection test as describe in Alg.
1 in Section 3.1.
In order to generate the 3 subsets of labeled examples data l , data h and data r , we remove 5% of the labeled data (but not less than 1 datapoint per class).
Results are shown in Fig.
We can see that in the low-budget regime, removing examples according to S ′ l yields worse performance as compared to the removal of random examples, while better performance is seen in the high-budget regime.
The opposite behavior is seen when removing examples according to the high-budget strategy S ′ h .
[Figure]
4. O PTIMAL STRATEGY : RESULTS
Tables 1-2 show the performance of our proposed method in comparison with the performance of the other baselines.
In all these experiments, the integrated strategy is successful in its identification of a suitable budget regime. As a result, it works well both in the low and high-budget regimes, matching or surpassing both the low and high-budget strategies at all budgets.
As MiSAL chooses an active learning strategy dynamically for each budget, any state-of-the-art improvements for either low or high budgets AL strategies would also reflect an improvement of MiSAL as well.
[Table]
[Table]
Why should the choice of AL strategies in Alg. 1 be restricted?
For a given task and a given budget, Alg. 1 is designed to identify the suitable family of AL strategies, whether low or high budget, using a respective choice of S ′ l and S ′ h .
To achieve the results reported above, we use TypiClust for S ′ l and inverse TypiClust for S ′ h .
In Section 3.1 we discuss why we do not use to this end the subsequent selection of the most competitive strategies S l and S h , arguing that the selection must be restricted to AL strategies that do not rely on the labeled set L .
We now demonstrate what happens when the selection is not restricted in this manner, and in particular if S ′ h is chosen to be a competitive
AL strategy that relies on the labeled set L for its successful outcome.
Specifically, we repeat the experiments whose results are reported in Fig.
5a, but where strategy S ′ h – the one used for the removal of examples – is BADGE.
Results are shown in Fig.
Unlike Fig.
5a, there is no transition point, as it is always beneficial to remove examples selected by BADGE rather than random examples.
This is because the added value of all points used for training diminishes after training is completed.
S UMMARY AND D ISCUSSION
It has been shown in previous work that different active learning strategies are suited for different budgets.
In this paper, we present a hybrid integrated method, which combines competitive methods from the two different domains in order to achieve an active learning strategy that is suitable for all budgets.
Our main contribution is twofold: First, we provide a theoretical analysis of the transition between low and high budgets, suggesting that this phenomenon may be universal.
The main technical contributions, motivated by this analysis, involve: (i) the design of an effective test to distinguish between the domains in a given scenario, (ii) practical implementation.
Finally, the performance of our method is shown to match or surpass other active learning strategies at all budgets.
A PPENDIX A D ERIVATION OF T RANSITION P OINTS
Recall that the mean generalization error of mixed strategy q is:
[Equation]
for differentiable function E .
The mixture coefficient q , which obtains the minimal generalization error, must satisfy
[Equation]
The transition points can now be defined as follows:
• B equiv is obtained by solving (9) with q = p .
• B low is obtained by solving (9) with q = p + m (1 − p ) B .
• B high is obtained by solving (9) with q = p − m (1 − p ) B .
B H YPER - PARAMETERS B. S UPERVISED TRAINING
When training on CIFAR-10 and CIFAR-100, we used a ResNet-18 trained over 50 epochs.
We used an SGD optimizer, with 0 .
9 Nesterov momentum, 0 .
0003 weight decay, cosine learning rate scheduling with a base learning rate of 0 . 025 , and batch size of 100 examples.
We used random croppings and horizontal flips for augmentations.
An example use of these parameters can be found at (Munjal et al., 2020).
When training ImageNet-50, we used the same hyper-parameters as CIFAR-10/100, only changing the base learning rate to 0 .
01 and the batch size to 50 .
B. U NSUPERVISED REPRESENTATION LEARNING
CIFAR-10/100 We trained SimCLR using the code provided by (Van Gansbeke et al., 2020) for
CIFAR-10 and CIFAR-100.
Specifically, we used ResNet18 with an MLP projection layer to avector, trained for 500 epochs.
All the training hyper-parameters were identical to those used by
SCAN.
After training, we used the 512 dimensional penultimate layer as the representation space.
As in SCAN, we used an SGD optimizer with 0 .
9 momentum and an initial learning rate of 0 .
4 with a cosine scheduler.
The batch size was 512 and a weight decay of 0 . 0001 .
The augmentations were random resized crops, random horizontal flips, color jittering, and random grayscaling.
We refer to (Van Gansbeke et al., 2020) for additional details.
We used the L2 normalized penultimate layer as embedding.
ImageNet-50 We extracted embedding from the official (ViT-S/16) DINO weights pre-trained on
ImageNet.
We used the L2 normalized penultimate layer as embedding.
C A DDITIONAL EXPERIMENTAL RESULTS C. H IGH B UDGET S TRATEGIES
In Section.
4, we are required to use a high budget strategy S ′ h , which relies in its computation only on the unlabeled set U .
We use reverse-TypiClust , which is calculated similarly to TypiClust , only using the most atypical example at each iteration instead of the most typical example.
In Fig.
7, we plot the performance of such a strategy on CIFAR-10, as a function of budget B , similarly to the analysis done in Fig. 4.
We see that while reverse-TypiClust is not a competitive high-budget strategy, it still outperforms random sampling in the high-budget regime, making it a suitable AL strategy for this regime.
C. O THER FEATURE SPACES C.2. R EMOVING DATA ACCORDING TO OTHER FEATURE SPACES
In section 3.2, we propose an active learning method that determines the budget size by removing examples in a given feature space.
The feature space used in section 3.2 was obtained by SimCLR, as these features proved beneficial to several low-budget active learning methods.
In this section, we check the dependency of MiSAL on the specific choice of feature space.
In Fig.
8, we plot the strategy selection test as described in Alg.
1 in Section 3.1.
The plotted results are trained on CIFAR-10.
In order to generate the 3 subsets of labeled examples data l , data h and data r , we remove 5% of the labeled data (but not less than 1 datapoint per class).
This test is done usingdifferent feature spaces 1.
MoCo (He et al., 2020), a transformer based approach.
SimCLR, as done in section 3.2. 3.
SCAN (Van Gansbeke et al., 2020).
Similarly to the results reported in section 3.2, we can see that using any of the 3 feature spaces resulted in a similar result – MiSAL would behave similarly regardless of the choice of the underlying feature space.
[Figure]
versa.
C.2. O THER FEATURE SPACES IN T YPI C LUST
In Table 1 and Table 2, we plot the results of different AL strategies across different datasets and budgets.
Low-budget strategies such as TypiClust and ProbCover require the choice of feature space to work properly.
Following the original papers, we used the feature space given by SimCLR trained on the entire unlabeled pool U .
To check whether the choice of the feature space affects the results of the low-budget performance, we trained TypiClust on the TinyImageNet dataset with various choices of feature spaces.
In Fig.
9, we plot 5 active learning iterations with an active set of m = 1000 of ResNet-50 trained on
TinyImageNet.
We considered 5 different feature spaces: 1.
MoCo (He et al., 2020), a transformer based approach.
DINO (Caron et al., 2021), an SSL-based approach.
SimCLR, which was used in the original TypiClust paper.
SWAV an SSL-based approach.
A simple autoencoder on the pixel values (AE).
We found that except for the AE, all methods perform similarly, suggesting that the choice of the representation space has little effect on the training of low-budget methods such as
TypiClust.