S CALABLE
B ATCH -M ODE
D EEP
B AYESIAN
A
CTIVE
L EARNING VIA E QUIVALENCE C LASS A NNEALING
Anonymous authors
Paper under double-blind review
A BSTRACT
Active learning has demonstrated data efficiency in many fields.
Existing active learning algorithms, especially in the context of batch-mode deep Bayesian active models, rely heavily on the quality of uncertainty estimations of the model, and are often challenging to scale to large batches.
In this paper, we propose
Batch-BAL AN C E , a scalable batch-mode active learning algorithm, which combines insights from decision-theoretic active learning, combinatorial information measure, and diversity sampling.
At its core, Batch-BAL AN C E relies on a novel decision-theoretic acquisition function that facilitates differentiation among different equivalence classes .
Intuitively, each equivalence class consists of hypotheses (e.g., posterior samples of deep neural networks) with similar predictions, and
Batch-BAL AN C E adaptively adjusts the size of the equivalence classes as learning progresses.
To scale up the computation of queries to large batches, we further propose an efficient batch-mode acquisition procedure, which aims to maximize a novel combinatorial information measure defined through the acquisition function.
We show that our algorithm can effectively handle realistic multi-class classification tasks, and achieves compelling performance on several benchmark datasets for active learning under both low- and large-batch regimes.
I NTRODUCTION
Active learning (AL) (Settles, 2012) characterizes a collection of techniques that efficiently select data for training machine learning models.
In the pool-based setting, an active learner selectively queries the labels of data points from a pool of unlabeled examples and incurs a certain cost for each label obtained.
The goal is to minimize the total cost while achieving a target level of performance.
A common practice for active learning is to devise efficient surrogates, aka acquisition functions , to assess the effectiveness of unlabeled data points in the pool.
There have been a vast body of literature and empirical studies (Huang et al., 2010; Houlsby et al., 2011; Wang & Ye, 2015; Hsu & Lin, 2015; Huang et al., 2016; Sener & Savarese, 2017; Ducoffe &
Precioso, 2018; Ash et al., 2019; Liu et al., 2020; Yan et al., 2020) suggesting a variety of heuristics as potential acquisition functions for AL.
Among these methods, Bayesian Active Learning by
Disagreement (BALD) (Houlsby et al., 2011) has attained notable success in the context of deep
Bayesian AL, while maintaining the expressiveness of Bayesian models (Gal et al., 2017; Janz et al., 2017;
Shen et al., 2017).
Concretely, BALD relies on a most informative selection (MIS) strategy—a classical heuristic that dates back to Lindley (1956)—which greedily queries the data point exhibiting the maximal mutual information with the model parameters at each iteration.
Despite the overwhelming popularity of such heuristics due to the algorithmic simplicity (MacKay, 1992; Chen et al., 2015; Gal & Ghahramani, 2016), the performance of these AL algorithms unfortunately is sensitive to the quality of uncertainty estimations of the underlying model, and it remains an open problem in deep AL to accurately quantify the model uncertainty, due to limited access to training data and the challenge of posterior estimation.
In figure 1, we demonstrate the potential issues of MIS-based strategies introduced by inaccurate posterior samples from a Bayesian Neural Network (BNN) on a multi-class classification dataset.
Here, the samples (i.e. hypotheses) from the model posterior are grouped into equivalence classes (ECs) (Golovin et al., 2010) according to the Hamming distance between their predictions as shown in figure 1a.
Informally, an equivalence class contains hypotheses that are close in their predictions for a randomly selected set of examples (See section 2.2 for its formal definition).
We note from figure 1b that the probability mass of the models sampled from the BNN is centered around the mode of the approximate posterior distribution, while little coverage is seen on models of higher accuracy.
Consequently, MIS tends to select data points that reveal the maximal information w.r
[Figure]
t. the sampled distribution , rather than guiding the active learner towards learning high accuracy models.
[Figure]
In addition to the robustness concern, another challenge for deep AL is the scalability to large batches of queries.
In many real-world applications, fully sequential data acquisition algorithms are often undesirable especially for large models, as model retraining becomes the bottleneck of the learning system (Mittal et al., 2019;
Ostapuk et al., 2019).
Due to such concerns, batch-mode algorithms are designed to reduce the computational time spent on model retraining and increase labeling efficiency.
Unfortunately, for most acquisition functions, computing the optimal batch of queries function is NP-hard (Chen & Krause, 2013); when the evaluation of the acquisition function is expensive or the pool of candidate queries is large, it is even computationally challenging to construct a batch greedily (Gal et al., 2017; Kirsch et al., 2019; Ash et al., 2019).
Recently, efforts in scaling up batch-mode active learning algorithms often involve diversity sampling strategies to downsample the pool (Sener & Savarese, 2017; Ash et al., 2019; Citovsky et al., 2021;
Kirsch et al., 2021a).
Unfortunately, these diversity selection strategies either ignores the downstream learning objective (e.g., using clustering as by (Citovsky et al., 2021)), or inherit the limitations of the sequential acquisition functions (e.g., sensitivity to uncertainty estimate as elaborated in figure(Kirsch et al., 2021a)).
Motivated by these two challenges, this paper aims to simultaneously (1) mitigate the limitations of uncertainty-based deep AL heuristics due to inaccurate uncertainty estimation, and (2) enable efficient computation of batches of queries at scale.
We propose Batch-BAL AN C E —an efficient batchmode deep Bayesian AL framework—which employs a decision-theoretic acquisition function inspired by Golovin et al.
; Chen et al.
Concretely, Batch-BAL AN C E utilizes BNNs as the underlying hypotheses, and uses Monte Carlo (MC) dropout (Gal & Ghahramani, 2016;
Kingma et al., 2015) or Stochastic gradient Markov Chain Monte Carlo (SG-MCMC) (Welling & Teh, 2011;
Chen et al., 2014; Ding et al., 2014;
Li et al., 2016a) to estimate the model posterior.
It then selects points that can most effectively tell apart hypotheses from different equivalence classes (as illustrated in figure 1).
Intuitively, such disagreement structure is induced by the pool of unlabeled data points; therefore our selection criterion takes into account the informativeness of a query with respect to the target models (as done in BALD), while putting less focus on differentiating models with little disagreement on target data distribution.
As learning progresses, Batch-BAL AN C E adaptively anneals the radii of the equivalence classes, resulting in selecting more “difficult examples” that distinguish more similar hypotheses as the model accuracy improves (section 3.1).
When computing queries in small batches, Batch-BAL AN C E employs an importance sampling strategy to efficiently compute the expected gain in differentiating equivalence classes for a batch of examples and chooses samples within a batch in a greedy manner.
To scale up the computation of queries to large batches, we further propose an efficient batch-mode acquisition procedure, which aims to maximize a novel combinatorial information measure (Kothawade et al., 2021) defined through our novel acquisition function.
The resulting algorithm can efficiently scale to realistic batched learning tasks with reasonably large batch sizes (section 3.2, section 3.3, appendix B).
Finally, we demonstrate the effectiveness of variants of Batch-BAL AN C E via an extensive empirical study, and show that they achieve compelling performance—sometimes by a large margin—on several benchmark datasets (section 4, appendix D) for both small and large batch settings.
B ACKGROUND AND PROBLEM SETUP
In this section, we introduce useful notations and formally state the (deep) Bayesian AL problem.
We then describe two important classes of existing AL algorithms along with their limitations, as a warm-up discussion before introducing our algorithm in section 3.
2. P ROBLEM S ETUP
Notations We consider pool-based Bayesian AL, where we are given an unlabelled dataset D pool drawn i.i.d.
from some underlying data distribution.
Further, assume a labeled dataset D train and a set of hypotheses
H = { h 1 , . .
h n } .
We would like to distinguish a set of (unknown) target hypotheses among the ground set of hypotheses H .
Let H denote the random variable that represents the target hypotheses.
Let p( H ) be a prior distribution over the hypotheses.
In this paper, we resort to BNN with parameters ω ∼ p( ω | D train ) 1 .
Problem statement An AL algorithm will select samples from D pool and query labels from experts.
The experts will provide label y for given query x ∈ D pool .
We assume labeling each query x incurs a unit cost.
Our goal is to find an adaptive policy for selecting samples that allows us to find a hypotheses with target error rate σ ∈ [0 , 1] while minimizing the total cost of the queries.
Formally, a policy π is a mapping π from the labeled dataset D train to samples in D pool .
We use D π train to denote the set of examples chosen by π .
Given the labeled dataset D π train , we define p ERR ( π ) as the expected error probability w.r.t.
the posterior p( ω | D π train ) .
Let the cost of a policy π be cost( π ) ≜ max |D π train | , i.e., the maximum number of queries made by policy π over all possible realizations of the target hypothesis H ∈ H .
Given a tolerance parameter σ ∈ [0 , 1] , we seek a policy with the minimal cost, such that upon termination, it will get expected error probability less than σ .
Formally, we seek arg min π cost( π ) , s . t .
p ERR ( π ) ≤ σ .
2. T HE EQUIVALENCE - CLASS - BASED SELECTION CRITERION
As alluded in section 1 and figure 1, the MIS strategy can be ineffective when the samples from the model posterior are heavily biased and cluttered toward sub-optimal hypotheses.
We refer the readers to appendix A.1 for details of a stylized example where a MIS-based strategy (such as
BALD) can perform arbitrarily worse than the optimal policy.
A “smarter” strategy would instead leverage the structure of the hypothesis space induced by the underlying (unlabeled) pool of data points.
In fact, this idea connects to an important problem for approximate AL, which is often cast as learning equivalence classes (Golovin et al., 2010):
Definition 2.1 (Equivalence Class) Let ( H , d ) be a metric space where H is a hypothesis class and d is a metric.
For a given set V ⊆ H and centers S = { s 1 , ..., s k } ⊆ V of size k , let r S : V → [ k ] be a partition function over
V and D i := { h ∈ V | r S ( h ) = i } , such that ∀ i, j ∈ [ k ] , r S ( s i )
=
i and ∀
h ∈
D
i , d ( h, s i ) ≤ d ( h, s j ) .
Each D i ⊆
V is called an equivalence class induced by s i ∈ S .
Consider a pool-based AL problem with hypothesis space H , a sampled set V ⊆ H , and an unlabeled dataset ¯ D pool which is drawn i.i.d.
from the underlying data distribution.
Each hypothesis h ∈ H can be represented by a vector v h indicating the predictions of all samples in ¯ D pool .
We can construct equivalence classes with the Hamming distance, which is denoted as d H ( h, h ′ ) , and equivalence class number k on sampled hypotheses V .
Let d S H ( V ) :
= max h,h ′ ∈V : r S ( h )=
r S ( h ′ )
d H ( h, h ′ ) be the maximal diameter of equivalence classes induced by S .
Therefore, the error rates of any unordered pair of hypotheses { h, h ′ } that lie in the same equivalence class are at most d S H ( V ) away from each other.
If we construct the k equivalence-class-inducing centers (as in definition 2.1) as the solution of the max-diameter clustering problem: C = arg min |S| = k d S H ( V ) , we can obtain the minimal worstcase relative error (i.e. difference in error rate) between hypotheses pair { h, h ′ } that lie in the same equivalence class.
We denote E
= {{ h, h ′ } :
r C ( h ) ̸ = r C ( h ′ ) } as the set of all (unordered) pairs of hypotheses (i.e. undirected edges) corresponding to different equivalence classes with centers in C .
Limitation of existing EC-based algorithms Existing EC-based AL algorithms (e.g., EC(Golovin et al., 2010) as described in appendix A.2 and ECED (Chen et al., 2016) as in appendix A.3) are not directly applicable to deep Bayesian AL tasks.
This is because computing the acquisition function (equation 4) needs to integrate over the hypotheses space, which is intractable for large models (such as deep BNN).
Moreover, it is nontrivial to extend to batch-mode setting since the number of possible candidate batches and the number of label configurations for the candidate batch grows exponentially with the batch size.
Therefore, we need efficient approaches to approximate the ECED acquisition function when dealing with BNNs in both fully sequential setting and batchmode setting.
O UR A PPROACH
We first introduce our acquisition function for the sequential setting, namely BAL AN C E (as in
Bayesian Active Learning via Equivalence Class Annealing), and then present the batch-mode extension under both small and large batch-mode AL settings.
3. T HE BAL AN C E ACQUISITION FUNCTION
We resort to Monte Carlo method to estimate the acquisition function.
Given all available labeled samples D train at each iteration, hypotheses ω are sampled from the BNN posterior.
We instantiate our methods with two different BNN posterior sampling approaches: MC dropout (Gal & Ghahramani, 2016) and cSG-MCMC (Zhang et al., 2019).
MC dropout is easy to implement and scales well to large models and datasets very efficiently (Kirsch et al., 2019; Gal & Ghahramani, 2016; Gal et al., 2017). However, it is often poorly calibrated (Foong et al., 2020; Fortuin et al., 2021). cSG-MCMC is more practical and indeed has high-fidelity to the true posterior (Zhang et al., 2019;
Fortuin et al., 2021; Wenzel et al., 2020).
In order to determine if there is an edge { ˆ ω, ˆ ω ′ } that connects a pair of sampled hypotheses ˆ ω, ˆ ω ′ (i.e., if they are in different equivalence classes), we calculate the Hamming distance d H (ˆ ω, ˆ ω ′ ) between the predictions of ˆ ω, ˆ ω ′ on the unlabeled dataset ¯ D pool .
If the distance is greater than some threshold τ , we consider the edge { ˆ ω, ˆ ω ′ } ∈
ˆ E ; otherwise not.
We define the acquisition function of
BAL AN C E for a set x 1: b ≜ { x 1 , ..., x b } as:
[Equation]
where λ ω,y 1: b ≜
[Equation]
[Equation]
p
[Equation]
1: b | [Equation] 1: b ) is the likelihood ratio 2 (Chen et al., 2016), and 1 d H (ˆ ω k [Equation]
ˆ ω
[Equation]
k )
>
τ is the indicator function.
We can adaptively anneal τ by setting τ proportional to BNN’s validation error rate ε in each AL iteration.
In practice, we cannot directly compute equation 1; instead we estimate it with sampled BNN posteriors:
We first acquire K pairs of BNN posterior samples { ˆ ω, ˆ ω ′ } .
The Hamming distances d H (ˆ ω, ˆ ω ′ ) between these pairs of BNN posterior samples are computed.
Next, we calculate the weight discount factor 1 − λ ˆ ω k ,y 1: b λ ˆ ω ′ k ,y 1: b for each possible label y and each pair { ˆ ω, ˆ ω ′ } where d H (ˆ ω, ˆ ω ′ ) >
τ .
At last, we take the expectation of the discounted weight over all y 1: b configurations.
In summary, ∆ BAL AN C E ( x 1: b ) is approximated as
[Equation]
D train is omitted for simplicity of notations.
Note that in our algorithms we never explicitly construct equivalence classes on BNN posterior samples, due to the fact that (1) it is intractable to find the exact solution for the max-diameter clustering problem and (2) an explicit partitioning of the hypotheses samples tends to introduce “unnecessary” edges where the incident hypotheses are closeby (e.g., if a pair of hypotheses lie on the adjacent edge between two hypothesis partitions), and therefore may overly estimate the utility of a query.
Nevertheless, we conducted an empirical study of a variant of BAL AN C E with explicit partitioning (which underperforms BAL AN C E ).
We defer detailed discussion on this approach, as well as empirical study, to the appendix D.4.
In the fully sequential setting, we choose one sample x with top ∆ BAL AN C E ( x ) in each AL iteration.
In the batch-mode setting, we consider two strategies for selecting samples within a batch: greedy selection strategy for small batches and acquisition-function-driven clustering strategy for large batches.
We refer to our full algorithm as Batch-BAL AN C E (algorithm 1) and expand on the batch-mode extensions in the following two subsections.
3. G REEDY SELECTION STRATEGY
To avoid the combinatorial explosion of possible batch number, the greedy selection strategy selects sample x with maximum ∆ BAL AN C E ( x 1: b − 1 ∪ { x } ) in the b -th step of a batch.
However, the config- uration y 1: b of a subset x 1: b expands exponentially with subset size b .
In order to efficiently estimate ∆ BAL AN C E ( x 1: b ) , we employ an importance sampling method.
The current M configuration samples of y 1: b are drawn by concatenating previous drawn M samples of y 1: b − 1 and M samples of y b (samples drawn from proposal distribution).
The pseudocode for the greedy selection strategy is provided in algorithm 2.
We refer the readers to appendix B.2 for details of importance sampling and to appendix B.3 for details of efficient implementation. [Table] [Table]
3. S TOCHASTIC BATCH SELECTION WITH POWER SAMPLING AND BAL AN C E -C LUSTERING
A simple approach to apply our new acquisition function to large batch is stochastic batch selection (Kirsch et al., 2021a), where we randomly select a batch with power distribution p( x ) ∼ ∆ BAL AN C E ( x ) β .
We call this algorithm PowerBAL AN C E .
Next, we sought to further improve PowerBAL AN C E through a novel acquisition-function-driven clustering procedure.
Inspired by Kothawade et al.
(2021), we define a combinatorial information measure I ∆ BAL AN C E ( x, y ) for any two data samples x and y based on our acquisition function:
[Equation]
Intuitively, I ∆ BAL AN C E ( x, y ) captures the amount of overlap between x and y w.r.t.
∆ BAL AN C E .
Therefore, it is natural to use it as a similarity measure for clustering, and use the cluster centroids as candidate queries.
The BAL AN C E -Clustering algorithm is illustrated in algorithm 3.
Concretely, we first sample a subset C ⊂ D pool with p( x ) ∼ ∆ BAL AN C E ( x ) β similar to (Kirsch et al., 2021a).
The BAL AN C E -Clustering then runs an Lloyd’s algorithm (with a non-Euclidean metric) to find B cluster centroids (see Line 3-8 in algorithm 3): it takes the subset C , { ˆ ω k , ˆ ω ′ k } Kk =1 , threshold τ , coldness parameter β , and cluster number B as input.
It first samples initial centroids O with p( x ) ∼ ∆ BAL AN C E ( x ) β .
Then, it iterates the process of adjusting the clusters and centroids until convergence and outputs B cluster centroids as candidate queries.
E XPERIMENTS
In this section, we sought to show the efficacy of Batch-BAL AN C E on several diverse datasets, under both small batch setting and large batch setting.
In the main paper, we focus on accuracy as the key performance metric as is commonly used in the literature; supplemental results with different evaluation metrics, including macro-average AUC, F1, and NLL, are provided in appendix D.
4. E XPERIMENTAL SETUP
Datasets In the main paper, we consider four datasets (i.e. MNIST (LeCun et al., 1998), RepeatedMNIST (Kirsch et al., 2019), Fashion-MNIST (Xiao et al., 2017) and EMNIST (Cohen et al., 2017))
as benchmarks for the small-batch setting, and two datasets (i.e. SVHN (Netzer et al., 2011), CIFAR (Krizhevsky et al., 2009)) as benchmarks for the large-batch setting.
The reason for making the splits is that for the more challenging classification tasks on SVHN and CIFAR-10, the performance improvement for all baseline algorithms from a small batch (e.g., with batch size < 50 ) is hardly visible.
We split each dataset into unlabeled AL pool D pool , initial training dataset D train , validation dataset D val , test dataset D test and unlabeled dataset ¯ D pool .
¯ D pool is only used for calculating the Hamming distance between hypotheses and is never used for training BNNs.
For more experiment details about datasets, see appendix C.
BNN models At each AL iteration, we sample BNN posteriors given the acquired training dataset and select samples from D pool to query labels according to the acquisition function of a chosen algorithm.
To avoid overfitting, we train the BNNs with MC dropout at each iteration with early stopping.
for MNIST, Repeated-MNIST, EMNIST, and FashionMNIST, we terminate the training of BNNs with patience of 3 epochs.
For SVHN and CIFAR-10, we terminate the training of BNNs with patience of 20 epochs.
The BNN with the highest validation accuracy is picked and used to calculate the acquisition functions.
Additionally, we use weighted cross-entropy loss for training the
BNN to mitigate the bias introduced by imbalanced training data.
The BNN models are reinitialized in each AL iteration similar to Gal et al.
(2017); Kirsch et al.
It decorrelates subsequent acquisitions as the final model performance is dependent on a particular initialization.
We use Adam optimizer (Kingma & Ba, 2017) for all the models in the experiments.
For cSG-MCMC, we use ResNet-18 (He et al., 2016) and run 200 epochs in each AL iteration.
We set the number of cycles to 4 and initial step size to 0.5.
3 samples are collected in each cycle.
Acquisition criterion for Batch-BAL AN C E under different bach sizes For small AL batch with
B < 50 , Batch-BAL AN C E takes the greedy selection approach.
For large AL batch with B ≥ 50 ,
BAL AN C E takes the clustering approach described in section 3.3.
In the small batch-mode setting, if b < 4 , Batch-BAL AN C E enumerates all y 1: b configurations to compute the acquisition function ∆ (Batch − )
BAL AN C E according to equation 2; otherwise, it uses M = 10 , 000 MC samples of y 1: b and importance sampling to estimate ∆
[Table]
according to equation 5.
All our results report the median of 6 trials, with lower and upper quartiles.
Baselines For the small-batch setting, we compare Batch-BAL AN C E with Random, Variation
Ratio (Freeman & Freeman, 1965), Mean STD (Kendall et al., 2015) and BatchBALD.
To the best of the authors’ knowledge, Batch-BALD still achieves state-of-the-art performance for deep Bayesian
AL with small batches.
For large-batch setting, it is no longer feasible to run BatchBALD (Citovsky et al., 2021); we consider other baseline models both in Bayesian setting, e.g., PowerBALD, and
Non-Bayesian setting, e.g., CoreSet and BADGE.
4. C OMPUTATIONAL COMPLEXITY ANALYSIS
Table 1 shows the computational complexity of the batch-mode AL algorithms evaluated in this paper.
Here, C denotes the number of classes, B denotes the acquisition size, K is the pair number of posterior samples and M is the sample number for y 1: b configurations.
We assume the number of the hidden units is H .
T is # iterations for BAL AN C E -Clustering to converge and is usually less than 5.
In figure 2 we plot the computation time for a single batch (in seconds) by different algorithms.
As the batch size increases, variants of Batch-BAL AN C E (including Batch-BAL AN C E and
PowerBAL AN C E as its special case) both outperforms CoreSet in run time.
In later subsections, we will demonstrate that this gain in computational efficiency does not come at a cost of performance.
We refer interested readers to section B.4 for extended discussion of computational complexity.
[Table] [Figure]
n t
i
[Figure]
4. B ATCH - MODE DEEP B AYESIAN AL WITH SMALL BATCH SIZE
We compare 5 different models with acquisition sizes B = 1 , B = 3 and B = 10 on MNIST dataset.
K = 100 for all the methods.
The threshold τ for Batch-BAL AN C E is annealed by setting τ to ε/ 2 in each AL loop.
Note that when B = 3 , we can compute the acquisition function with all y 1: b configurations for b = 1 , 2 , 3 .
When b ≥ 4 , we approximate the acquisition function with importance sampling.
Figure 3 (a)-(c) show that Batch-BAL AN C E are consistently better than other baseline methods for MNIST dataset.
We then compare Batch-BAL AN C E with other baseline methods on three datasets with balanced classes—Repeated-MNIST, Fashion-MNIST and EMNIST-Balanced.
The acquisition size B for
Repeated-MNIST and Fashion-MNIST is 10 and is 5 for EMNIST-Balanced dataset.
The threshold τ of Batch-BAL AN C E is annealed by setting τ = ε/ 4 3 .
The learning curves of accuracy are shown in figure 3 (d)-(f).
For Repeated-MNIST dataset, BALD performs poorly and is worse than random selection.
BatchBALD is able to cope with the replication after certain number of AL loops, which is aligned with result shown in Kirsch et al.
Batch-BAL AN C E is able to beat all the other methods on this dataset.
An ablation study about repetition number and performance can be found in appendix D.2.
For Fashion-MNIST dataset, Batch-BAL AN C E outperforms random selection but the other methods fail.
For EMNIST dataset, Batch-BAL AN C E is slightly better than BatchBALD.
We further compare different algorithms with two unbalanced datasets: EMNIST-ByMerge and
EMNIST-ByClass.
The τ for Batch-BAL AN C E is set ε/ 4 in each AL loop.
B = 5 and K = 10 for all the methods.
As pointed out by Kirsch et al.
(2019), BatchBALD performs poorly in unbalanced dataset settings.
BAL AN C E and Batch-BAL AN C E can cope with the unbalanced data settings.
The result is shown in figure 3 (g) and (h).
Further results on other datasets and under different metrics are provided in appendix D.
4. B ATCH - MODE DEEP B AYESIAN AL WITH LARGE BATCH SIZE
Batch-BAL AN C E with MC dropout
We test different AL models on two larger datasets with larger batch size.
The acquisition batch size B is set 1,000 and τ = ε/ 8 .
We use VGG-11 as the
[Figure] [Figure] BNN and train it on all the labeled data with patience equal to 20 epochs in each AL iteration.
The
VGG-11 is trained using SGD with fixed learning rate 0.001 and momentum 0.9.
The size of C for Batch-BAL AN C E is set to 2 B .
Similar to PowerBALD (Kirsch et al., 2021a), we also find that
PowerBALanCe and BatchBALanCe are insensitive to β and β = 1 works generally well.
We thus set the coldness parameter β
= 1 for all algorithms.
The performance of different AL models on these two datasets is shown in figure 4 (a) and (b).
PowerBALD, PowerBAL AN C E , BADGE, and BatchBAL AN C E get similar performance on
SVHN dataset.
For CIFAR-10 dataset, BatchBAL AN C E shows compelling performance.
Note that
PowerBAL AN C E also performs well compared to other methods.
Batch-BAL AN C E with cSG-MCMC We test different AL models with cSG-MCMC on CIFAR10.
The acquisition batch size B is 5,000.
In order to apply CoreSet algorithm to BNN, we use the average activations of all posterior samples’ final fully-connected layers as the representations.
For
BADGE, we use the label with maximum average predictive probability as the hallucinated label and use the average loss gradient of the last layer induced by the hallucinated label as the representation.
We can see from figure 4 (c) that Batch-BAL AN C E achieve the best performance.
R ELATED WORK
Pool-based batch-mode active learning Batch-mode AL has shown promising performance for practical AL tasks.
Recent works, including both Bayesian (Houlsby et al., 2011; Gal et al., 2017;
Kirsch et al., 2019) and non-Bayesian approaches (Sener & Savarese, 2017; Ash et al., 2019;
Citovsky et al., 2021; Kothawade et al., 2021;
Hacohen et al., 2022; Karanam et al., 2022), have been enormous and we hardly do it justice here.
We mention what we believe are most relevant in the following.
Among the Bayesian algorithms, Gal et al.
(2017) choose a batch of samples with top acquisition functions.
These methods can potentially suffer from choosing similar and redundant samples inside each batch.
Kirsch et al.
(2019) extended Houlsby et al.
and proposed a batch-mode deep Bayesian AL algorithm, namely BatchBALD.
Chen & Krause (2013) formalized a class of interactive optimization problems as adaptive submodular optimization problems and prove a greedy batch-mode approach to these problems is near-optimal as compared to the optimal batch selection policy.
ELR (Roy & McCallum, 2001) focuses on a Bayesian estimate of the reduction in classification error and takes a one-step-look-ahead startegy.
Inspired by ELR, WMOCU (Zhao et al., 2021) extends MOCU (Yoon et al., 2013) with a theoretical guarantee of convergence.
However, none of these algorithms extend to the batch setting.
Among the non-Bayesian approaches, Sener & Savarese (2017) proposed a CoreSet approach to select a subset of representative points as a batch.
BADGE (Ash et al., 2019) selects samples in a batch by using the k-MEAMS++ seeding algorithm on the D pool representations, which are the gradient embeddings of DNN’s last layer induced by hallucinated labels.
Contemporary works propose AL algorithms that work for different settings including text classification (Tan et al., 2021), domain shift and outlier (Kirsch et al., 2021b), low-budget regime (Hacohen et al., 2022), very large batch (e.g., 100K or 1M) (Citovsky et al., 2021), rare classes and out-of-distribution data (Kothawade et al., 2021).
Bayesian neural networks Bayesian methods have been shown to improve the generalization performance of DNNs (Hernández-Lobato & Adams, 2015; Blundell et al., 2015; Li et al., 2016b;
Maddox et al., 2019), while providing principled representations of uncertainty.
MCMC methods provides the gold standard of performance with smaller neural networks (Neal, 2012).
SG-MCMC methods (Welling & Teh, 2011; Chen et al., 2014;
Ding et al., 2014; Li et al., 2016a) provide a promising direction for sampling-based approaches in Bayesian deep learning.
cSG-MCMC (Zhang et al., 2019) proposes a cyclical stepsize schedule, which indeed generates samples with high fidelity to the true posterior (Fortuin et al., 2021; Izmailov et al., 2021).
Another BNN posterior approximation is MC dropout (Gal & Ghahramani, 2016; Kingma et al., 2015).
We investigate both the cSG-MCMC and MC dropout methods as representative BNN models in our empirical study.
Semi-supervised learning Semi-supervised learning leverages both unlabeled and labeled examples in the training process (Kingma et al., 2014; Rasmus et al., 2015).
Some work has combined
AL and semi-supervised learning (Wang et al., 2016; Sener & Savarese, 2017; Sinha et al., 2019).
Our methods are different from these methods since our methods never leverage unlabeled data to train the models, but rather use the unlabeled pool to inform the selection of data points for AL.
C ONCLUSION AND DISCUSSION
We have proposed a scalable batch-mode deep Bayesian active learning framework, which leverages the hypothesis structure captured by equivalence classes without explicitly constructing them.
Batch-BAL AN C E selects a batch of samples at each iteration which can reduce the overhead of retraining the model and save labeling effort.
By combining insights from decision-theoretic active learning, combinatorial information measure, and diversity sampling, the proposed algorithms achieve compelling performance efficiently on active learning benchmarks both in small batch-mode and large batch-mode settings.
Given the promising empirical results on the standard benchmark datasets explored in this paper, we are further interested in understanding the theoretical properties of the equivalence annealing algorithm under controlled studies as future work.
Reproducibility Statement.
We will open-source the codes for reproducing the results in our paper.
We encourage the reader to refer to section 3 and appendix B for details about our methods and to refer to section 4, appendix C, and appendix D for details of our experimental setup and results.
C ONTENTS
Introduction
2 Background and problem setup 2.
Problem Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The equivalence-class-based selection criterion . . . . . . . . . . . . . . . . . . .
3 Our Approach 3.
The BAL AN C E acquisition function . . . . . . . . . . . . . . . . . . . . . . . . .
Greedy selection strategy .
Stochastic batch selection with power sampling and BAL AN C E -Clustering . . . .
4 Experiments 4.
Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Computational complexity analysis . .
Batch-mode deep Bayesian AL with small batch size . . . . . . . . . . . . . . . .
Batch-mode deep Bayesian AL with large batch size . . . . . . . . . . . . . . . .
5 Related work 6 Conclusion and discussion
A Preliminary works
A.
1 The most informative selection criterion
A.2 Equivalence class edge cutting
A.
The equivalence class edge discounting algorithm
B Algorithmic details
B.1 Derivation of
acquisition functions of Batch-BAL AN C E . . . . . . B configurations . . .
complexity discussion . . . . . . . . . . . . . . . . . . . .
C Experimental setup: Datasets and implementation details
C.1 Datasets used in the main paper . .
C.2 Implementation details on the empirical example in figure 1 . . . . . . . . . . . . .
D Supplemental empirical results
different
D.
Experiments on other datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . D metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
[Table] [Table]
[Table] [Table]
[Table]
A P RELIMINARY WORKS A. T HE MOST INFORMATIVE SELECTION CRITERION
BALD uses mutual information between the model prediction for each sample and parameters of the model as the acquisition function.
It captures the reduction of model uncertainty by receiving a label y of a data point x
: I ( y ; ω | x, D train )
=
H ( y | x, D train ) − E p( ω |D train )
[ H ( y | x, ω, D train )] where H denotes the Shannon entropy (Shannon, 1948).
Kirsch et al.
(2019) further proposed BatchBALD as an extension of BALD whereby the mutual information between a joint of multiple data points and the model parameters is estimated as
[Equation]
Limitation of the BALD algorithm
BALD can be ineffective when the hypothesis samples are heavily biased and cluttered towards sub-optimal hypotheses.
Below, we provide a concrete example where such selection criterion may be undesirable
[Equation]
[Table]
+vE7aV3Xvpn79cF1rNoo4ynAG53AJHtxCE+6hBT4w4PAMr/DmSOfFeXc+lq0lp5g5hT9wPn8AFHKO2A==</latexit> h n
Consider the problem shown in figure 5.
The hypothesis class H = { h 1 , . . .
, h n } is structured such that
[Equation]
where d H ( h i , h j ) denotes the fraction of labels
h
i and h j disagree upon when making predictions on i.i.d.
samples of data points.
We further assume that for any subset of hypotheses S ⊆ H , there exists a data point whose label they agree upon.
Assume each hypothesis h
i has an equal probability and the target error rate is σ .
On the one hand, note that BALD does not consider d H ( h i , h j ) , and therefore on average it requires log n examples to identify any target hypothesis.
On the other hand, to achieve a target error rate of σ , one only needs to differentiate all pairs of hypotheses
h i , h j of distance d H ( h i , h j ) >
σ (i.e., by selecting training examples to rule out at least one of h i , h j ).
Therefore, a “smarter” AL policy could query examples to sequentially check the consistency of h 1 , h 2 , . . .
, h n until all remaining hypotheses are within distance σ .
It is easy to check that this requires log(1 /σ ) examples before reaching the error rate σ .
The gap between BALD and the above policy log n log(1 /σ ) could be large as n increases.
A. E QUIVALENCE CLASS EDGE CUTTING
Consider the problem statement in section 2.1.
If σ = 0 and tests are noise-free, this problem can be solved near-optimally by the equivalence class edge cutting (EC 2 ) algorithm (Golovin et al., 2010).
EC 2 employs an edge-cutting strategy based on a weighted graph G =
( H , E ) , where vertices represent hypotheses and edges link hypotheses that we want to distinguish between.
Here
E ≜ {{ h, h ′ } : r ( h ) ̸
= r ( h ′ ) } contains all pairs of hypotheses that have different equivalence classes.
We define a weight function W : E → R ≥ 0 by W ( { h, h ′ } ) ≜ p( h ) · p( h ′ ) .
A sample x with label y is said to "cut" an edge, if at least one hypothesis is inconsistent with y .
Denote
E ( x, y ) ≜ {{ h, h ′ } ∈ E : p( y | x, h ) = 0 ∨ p( y | x, h ′ )
=
0 } as the set of edges cut by labeling x as y .
The EC 2 objective is then defined as the total weight of edges cut by the current D train : f EC [Equation] [Equation] x, y )
[Equation] EC 2 algorithm greedily maximizes this objective per iteration.
The acquisition function for EC 2 is
[Equation]
A. T HE EQUIVALENCE CLASS EDGE DISCOUNTING ALGORITHM
In the noisy setting, the acquisition function of Equivalence Class Edge Discounting algorithm (ECED) (Chen et al., 2016) takes undesired contribution by noise into account.
Given a data point and its label ( x, y ) , ECED discounts all model parameters by their likelihood ratio:
λ h,y ≜ p( y | h,x ) max y ′ p( y ′ | h,x ) .
After we get D train , the value of assigning label y to a data point x is defined as the total amount of edge weight discounted: δ ( x, y | D train ) ≜ (cid:80) { h,h ′ }∈E p( h, D train )p( h ′ , D train ) · (1 − λ h,y λ h ′ ,y ) , where E = {{ h, h ′ } : r ( h ) ̸
= r ( h ′ ) } consists of all unordered pairs of hypothesis corresponding to different equivalence classes.
Further, ECED augments the above value function δ with an offset value such that the value of a non-informative test is 0.
The offset value of labeling x as label y is defined as: ν ( x, y | D train ) ≜ (cid:80) { h,h ′ }∈E p( h, D train )p( h ′ , D train ) · (1 − max h λ 2 h,y ) .
The overall acquisition function function of ECED is:
[Equation]
B A LGORITHMIC DETAILS B. D ERIVATION OF ACQUISITION FUNCTIONS OF BAL AN C E AND B ATCH -BAL AN C E
In each AL loop, the ECED algorithm selects a sample from AL pool according to the acquisition function
[Equation]
, where E is the total edges with adjacent nodes in different equivalence classes and λ ω,y = p( y | ω ) max y ′ p( y ′ | ω ) .
W ω,ω ′ is the weight for edge { ω, ω ′ } which is maintained by ECED algorithm.
After we observe y of selected x , we update the weights of all edges with W ω,ω ′ =
W ω,ω ′ · p( y | ω )p( y | ω ′ ) .
In the deep Bayesian AL setting, the offset term 1 − max ω λ 2 ω,y can be removed when we use deep BNN.
However, we can not enumerate all { ω, ω ′ } ∈ E in this setting since there are an infinite number of hypotheses in the hypothesis space.
Moreover, we can not even estimate the acquisition function of ECED on a subset of sampled hypotheses by MC dropouts since building equivalence classes with best ϵ is NP-hard.
If we sample { ω, ω ′ } according to posterior p( ω | D train ) and check whether { ω, ω ′ } ∈
ˆ E by Hamming distance in the way we describe in section 3.1, we will get
[Equation]
Inspired by the weight discounting mechanism of ECED, we define the acquisition function of BALAN C E ∆ BAL AN C E
( x | D train ) as
[Equation]
After we get K pairs of MC dropouts
[Equation]
the acquisition function ∆ BAL AN C E ( x | D train ) can be ap- proximated as follows:
[Equation]
In batch-mode setting, the acquisition function of Batch-BAL AN C E for a batch x 1: b is
[Equation]
Similar to the fully sequential setting
[Equation]
we can approximate ∆ Batch − BAL AN C E ( x 1: b | D train ) with K pairs of MC dropouts.
The x 1: b are chosen in a greedy manner.
For iteration b inside a batch, the x 1: b − 1 are fixed and x b is selected according to
[Equation]
I MPORTANCE SAMPLING OF CONFIGURATIONS
When b becomes large
[Equation]
it is infeasible to enumerate all label configurations y 1: b .
We use M MC samples of y 1: b to estimate the acquisition function and importance sampling to further reduce the computational time 4 .
Given that p( y 1: b | ω ) can be factorized as p( y 1: b − 1 | ω ) · p( y b | ω ) , the acquisition function can be written as:
[Equation]
Suppose we have M samples of y 1: b − 1 from p( y 1: b − 1 )
[Equation]
we perform importance sampling using p( y 1: b − 1 ) to estimate the acquisition function:
[Equation]
Here we save p(ˆ y 1: b − 1 | ˆ ω k ) and p(ˆ y 1: b − 1 | ˆ ω ′ k ) for M samples in ˆ P 1: b − 1 and ˆ P ′ 1: b − 1 .
The shape of ˆ P 1: b − 1 and ˆ P ′ 1: b − 1 is K × M .
⊙ is element-wise matrix multiplication and ⊗ is the outer- product operator along first dimension.
After the outer product operation, we can reshape the matrix by flattening all the dimensions after the 1st dimension.
1 is a matrix of 1s with shape K × 1 .
ˆ P ⊤ 1: b − 1 ˆ P b and ˆ P ′⊤ 1: b − 1 ˆ P ′ b are of shape M × C and their sum is reshape to 1 × MC after divided by
[Equation]
B.
E FFICIENT IMPLEMENTATION FOR GREEDY SELECTION
In algorithm 2, we can store p(ˆ y 1: b − 1 | ˆ ω k ) in a matrix ˆ P 1: b − 1 and p(ˆ y 1: b − 1 | ˆ ω ′ k ) in matrix ˆ P 1: b −for iteration b − 1 .
The shape of ˆ P 1: b − 1 and ˆ P ′ 1: b − 1 is K × C b − 1 .
p(ˆ y b | ˆ ω k ) can be stored in ˆ P b and p(ˆ y b | ˆ ω ′ k ) in ˆ P ′ b .
The shape of ˆ P b and ˆ P ′ b is K × C .
Then, we compute probability of p(ˆ y 1: b ) as follows:
[Equation]
The ˆ P ⊤ 1: b − 1 ˆ P b and ˆ P ′⊤ 1: b − 1 ˆ P ′ b can be flattened to shape 1 × C b after matrix multiplication.
We store max ˆ y 1: b − 1 p(ˆ y 1: b − 1 | ˆ ω k ) in a matrix ˆ A 1: b − 1 and max ˆ y ′ 1: b − 1 p(ˆ y ′ 1: b − 1 | ˆ ω ′ k ) in a matrix ˆ A ′ 1: b − 1 .
The shape of ˆ A 1: b − 1 and ˆ A ′ 1: b − 1 is K × 1 .
We can compute λ ˆ ω, ˆ y 1: b inside edge weight discount expression by
[Equation]
⊙ is element-wise matrix multiplication and ⊗ is the outer-product operator along the first dimension.
After the outer product operation
[Equation]
we can reshape the matrix by flattening all the dimensions after 1st dimension to maintain consistency.
Similarly, we can compute ˆ A ′ 1: b , p(ˆ y 1: b | ˆ ω ′ k ) and λ ˆ ω ′ , ˆ y 1: b with matrix operations.
The indicator function 1 d H (ˆ ω k , ˆ ω ′ k ) >
τ can be stored in a matrix with shape K × 1 .
The acquisition function can be computed with all matrix operations as follows:
[Equation]
=
[Equation]
ˆ
[Equation]
B. D ETAILED COMPUTATIONAL COMPLEXITY DISCUSSION
As demonstrated in figure 2, figure 6, and table 1, the computational complexity of our algorithm
PowerBAL AN C E shares is comparable to PowerBALD. They all need to estimate the acquisition function value for each data point in the AL pool and then choose the top B data points after adding Gumbel-distributed noise to the log values. However, the power sampling-based methods have limited performance due to the lack of interaction between selected samples and non-selected samples during sampling.
We can further improve the performance of PowerBAL AN C E with BatchBAL AN C E .
The computation complexity of Batch-BAL AN C E for large batch setting are proportional to B 2 when downsampled with subset size |C| =
cB and c is a small constant.
Its computational complexity is similar to that of BADGE and CoreSet.
[Figure] [Figure]
C E XPERIMENTAL SETUP : D ATASETS AND IMPLEMENTATION DETAILS C. D ATASETS USED IN THE MAIN PAPER
MNIST.
We randomly split MNIST training dataset into D val with 10,000 samples, ¯ D pool with 10,000 samples and D pool with the rest.
The initial training dataset contains 20 samples withsamples in each class chosen from the AL pool.
The BNN model architecture is similar to Kirsch et al.
It consists of two blocks of [convolution, dropout, max-pooling, relu] followed by a two-layer MLP that a two-layer MLP and one dropout between the two layers.
The dropout probability is 0.5 in the dropout layers.
Repeated-MNIST.
Kirsch et al.
(2019) show that applying BALD to a dataset that contains many (near) replicated data points leads to poor performance.
We again randomly split the MNIST training dataset similar to the settings used on MNIST dataset.
We replicate all the samples in AL pool two times and add isotropic Gaussian noise with a standard deviation of 0.1 after normalizing the dataset.
The BNN architecture is the same as the one used on MNIST dataset.
EMNIST.
We further consider the EMNIST dataset under 3 different settings: EMNIST-Balanced,
EMNIST-ByClass, and EMNIST-ByMerge.
The EMNIST-Balanced contains 47 classes with balanced digits and letters.
EMNIST-ByMerge includes digits and letters for a total of 47 unbalanced classes.
EMNIST-ByClass represents the most useful organization for classification as it contains the segmented digits and characters for 62 classes comprising [0-9],[a-z], and [A-Z].
We randomly split the training set into D val with 18,800 images, ¯ D pool with 18,800 images and D pool with the rest of the samples.
Similar to Kirsch et al.
(2019), we do not use an initial dataset and instead perform the initial acquisition step with the randomly initialized model.
The model architecture contains three blocks of [convolution, dropout, max-pooling, relu], with 32, 64, and 128 3x3 convolution filters and 2x2 max pooling.
We add a two-layer MLP following the three blocks.
4 dropout layers in total are in each block and MLP with dropout probability 0.5.
Fashion-MNIST.
Fashion-MNIST is a dataset of Zalando’s article images that consists of a training set of 60,000 examples and a test set of 10,000 examples.
Each example is a 28x28 grayscale image, associated with a label from 10 classes.
We randomly split Fashion-MNIST training dataset into
D val with 10,000 samples, ¯ D pool with 10,000 samples, and D pool with the rest of samples.
We obtain the initial training dataset that contains 20 samples with 2 samples in each class randomly chosen from the AL pool.
The model architecture is similar to the one used on EMNIST dataset with 10 units in the last MLP.
SVHN.
We randomly select initial training dataset with 5,000 samples, ¯ D pool with 2,000 samples, and validation dataset D val with 5,000 samples.
Similarly for CIFAR-10 dataset,
CIFAR-10.
we random select initial training dataset with 5,000 samples, ¯ D pool with 5,000 samples, and validation dataset D val with 5,000 samples.
C. I MPLEMENTATION DETAILS ON THE EMPIRICAL EXAMPLE IN FIGURE
We show an empirical example in figure 1 to provide some intuition as to why BAL AN C E and
Batch-BAL AN C E are effective in practice.
We train a BNN with an imbalanced MNIST training subset that contains 28 images for each digit in [1-8] and 1 image for digits 0 and 9.
The crossentropy loss is reweighted to balance the training dataset during training.
We obtain 200 posterior samples of BNN and use them to get the predictions on ¯ D pool .
We compute the Hamming distances for predictions all sample pairs and use these precomputed distances to plot the predictions with t-SNE (Van der Maaten & Hinton, 2008).
The equivalence classes are approximated by farthest-first traversal algorithm (FFT) (Gonzalez, 1985).
In figure 1, the equivalence classes are highly imbalanced.
The ground truth ¯ D pool dataset labels represent the target hypotheses embedding.
This figure highlights the scenario where the equivalence class-based methods, e.g. ECED and BAL AN C E are better than BALD.
D S UPPLEMENTAL EMPIRICAL RESULTS
In this section, we provide additional experimental details and supplemental results to demonstrate the competing algorithms.
D. E FFECT OF DIFFERENT CHOICES OF HYPERPARAMETERS
We compare BALD and BAL AN C E with batch size B = 1 and different K ’s on an imbalanced
MNIST dataset which is created by removing a random portion of images for each class in the training dataset.
figure 7 (a) shows that BAL AN C E performs the best with a large margin to the curve of BALD.
Note that BAL AN C E with K = 50 is also better than BALD with K = 100 .
[Figure]
We also study the influence of τ for BAL AN C E on MNIST dataset.
Denote the validation error rate of BNN model by ε .
BAL AN C E with fixed τ = 0 .
3 and annealing τ = ε/ 2 , ε/ 4 , ε/ 8 are run on MNIST dataset and the learning curves are shown in figure 7 (b).
The BAL AN C E is robust to τ .
However, when τ is set 0.3 and the test accuracy gets around 0.88, the accuracy improvement becomes slow.
The reason for this slow improvement is that the threshold τ is too large and all the pairs of posterior samples are treated as in the same equivalence class and the acquisition functions {+120 K 0 .0 .0 .0 .
 .m e a n ± s t d o f e+}
s t
i m a t e d a c qu i s i t i o n f un c t i o n for all the samples in the AL pool are zeros. In another word, the BAL AN C E degrades to random selection when τ is too large.
We further pick an data point from this imbalanced MNIST dataset and gradually increase the posterior sample number K to estimate the acquisition function value ∆ BAL AN C E for this data point.
For each posterior sample number K , we estimate the acquisition function ∆ BAL AN C E 10 times withsets of posterior sample pairs.
The mean and std for this K are calculated and shown in figure 8.
D. E XPERIMENTS ON OTHER DATASETS
We compare different AL algorithms on tabular datasets including Human Activity Recognition
Using Smartphones Data Set (Anguita et al., 2013) (HAR), Gas Sensor Array Drift (Vergara et al., 2012) (DRIFT), and Dry Bean Dataset (Koklu & Ozkan, 2020), as well as a more difficult dataset
CINIC-10 (Darlow et al., 2018).
HAR, DRIFT and Dry Bean Dataset We run 6 AL trials for each dataset and algorithm.
In each iteration, the BNNs are trained with a learning rate of 0.01 and patience equal to 3 epochs.
The BNNs all contain three-layer MLP with ReLU activation and dropout layers in between.
The datasets are all split into starting training set, validation set, testing set, and AL pool.
The AL pool is also used as ¯ D pool .
The τ for Batch-BAL AN C E is set ε/ 4 in each AL loop.
See table 2 for more experiment details of these 3 datasets.
dataset val set size test set size hidden unit # sample # per epoch K
[Table]
The learning curves of all 5 algorithms on these 3 tabular datasets are shown in figure 9.
BatchBAL AN C E outperforms all the other algorithms for these 3 datasets.
For HAR dataset, both BatchBAL AN C E and BatchBALD work better than random selection.
In figure 9 (b) and (c), Mean STD,
Variation Ratio and BatchBALD perform worse than random selection.
We find similar effect for some other imbalanced datasets.
[Figure] CINIC-10 CINIC-10 is a large dataset with 270K images from two sources: CIFAR-
(Krizhevsky et al., 2009) and ImageNet (Rasmus et al., 2015).
The training set is split into an
AL pool with 120K samples, 40K ¯ D pool samples, 20K validation samples and 200 starting training samples with 20 samples in each class.
We use VGG-11 as the BNN.
The number of sampled MC dropout pairs is 50 and the acquisition size is 10.
We run 6 trials for this experiment.
The learning curves of 5 algorithms are shown in figure 10.
We can see from figure 10 that Batch-BAL AN C E performs better than all the other algorithms by a large margin in this setting
[Figure]
the CINIC-10 dataset.
Repeated-MNIST with different amounts of repetitions
In order to show the effect of redundant data points on BathBALD and Batch-BAL AN C E , we ran experiments on Repeated-MNIST with an increasing number of repetitions.
The learning curves of accuracy for Repeated-MNIST with different repetition numbers can be seen in figure 11.
A detailed model accuracy on the test dataset when acquired training dataset size is 130 is shown in table 3.
Even though Batch-BAL AN C E can improve data efficiency (Kirsch et al., 2019), there are still large gaps between learning curves of
Batch-BALD and Batch-BAL AN C E and the gaps become larger when the number of repetitions increases.
In order to compare our algorithms with other AL algorithms in this small batch size regime, we further run PowerBAL AN C E , PowerBALD, BADGE and CoreSet on the Repeated-MNIST with repeat number 3.
As shown in figure 12, Batch-BAL AN C E achieves the best performance.
Note that both
[Figure] and CIFAR-100 For CIFAR-100, we use 100 fine-grained labels. The dataset is split into initial training dataset with 5,000 samples, ¯ D pool with 5,000 samples, and validation dataset D val with 5, samples.
Experiment is conducted with batch size B = 5 , 000 and budget 25,000.
The cSG-MCMC is used for BNN with epoch number 200, initial step size 0.
[Figure]
and cycle number 4.
We can see in figure 13 that both PowerBAL AN C E and Batch-BAL AN C E perform well in this dataset.
D. A DDITIONAL EVALUATION METRICS
Besides accuracy, we compared macro-average AUC, macro-average F1, and NLL for 5 different methods on EMNIST-Balanced and EMNIST-ByMerge datasets in figure 14.
The acquisition size for all the AL algorithms is 5.
Batch-BAL AN C E is annealed by setting τ = ε/ 4 .
A macro-average
AUC computes the AUC independently for each class and then takes the average.
Both macroaverage AUC and macro-average F1 take class imbalance into account.
As shown in figure 14,
[Figure] [Figure] Batch-BAL AN C E attains better data efficiency compared with baseline models on both balanced and imbalanced datasets.
We also evaluated the negative log-likelihood (NLL) for different AL algorithms.
NLL is a popular metric for evaluating predictive uncertainty (Quinonero-Candela et al., 2005).
As shown in figure 14,
Batch-BAL AN C E maintains a better or comparable quality of predictive uncertainty over test data.
D. BAL AN C E VIA EXPLICIT PARTITIONING OVER THE HYPOTHESIS POSTERIOR SAMPLES
Another way of estimating the acquisition function is to construct the equivalence classes explicitly first (e.g. by partitioning the hypothesis spaces into k Voronoi cells via max-diameter clustering and calculate the weight discounts of edges that connect different equivalence classes.
Intuitively, explicitly constructing equivalence classes may introduce unnecessary edges as two closeby hypotheses can be partitioned into different equivalence classes; therefore leading to an overestimate of the edge weight discounted.
We call this algorithm BAL AN C E -Partition. E M N I S
T - B a l a n ce d
[Figure]
E M
N I S T - B y M e r g e [Figure] AN C E and Batch-BAL AN C E , we sampled K pairs of MC dropouts to estimate the acquisition function of BAL AN C E -Partition.
All the representations of 2 K MC dropouts on ¯ D pool are generated.
We run FFT (Gonzalez, 1985) with Hamming distances and threshold τ on these representations to get approximated ECs.
Each data point has at most τ Hamming distance to the corresponding cluster center.
FFT is a 2-approx algorithm and the optimal solution with the same cluster number has cluster diameter ≥ τ 2 .
After equivalence classes are returned,
BAL AN C E -Partition calculates the edges discounts of all edges that connect different equivalence classes and estimates the acquisition function values of each data sample in the AL pool.
Although a faster method that utilizes complete homogeneous symmetric polynomials (Javdani et al., 2014) can be implemented to estimate the acquisition function values for BAL AN C E -Partition, experiments in figure 15 show that BAL AN C E -Partition can not achieve better performance than
BAL AN C E and increasing the MC dropout number does not improve performance significantly.
[Table]
D. C OEFFICIENT OF VARIATION
To gain more insight into why BAL AN C E and Batch-BAL AN C E work consistently better than
BALD and BatchBALD,
we further investigate the dispersion of the estimated acquisition function values for those methods.
Since Batch-BAL AN C E and BatchBALD extend their fully sequential algorithms similarly in a greedy manner, we only compare the acquisition functions of BAL AN C E and BALD.
The coefficient of variation (CV) is chosen for the comparison of dispersion.
It is defined as the ratio of the standard deviation to the mean.
CV is a standardized measure of the dispersion of a probability distribution or frequency distribution.
The value of CV is independent of the unit in which it is taken.
[Figure] We conduct the experiment on the imbalanced MNIST dataset in the setting of appendix C.2.
We estimate the acquisition function values of BAL AN C E and BALD 5 times with 5 sets of K MC dropouts for each sample in the AL pool.
Then, the CVs are calculated for these estimations.
In figure [Figure] we show histograms of CVs for both methods.
The estimated acquisition function values of BAL AN C E are less dispersed, which shows potential for better performance.
[Figure]
D. P REDICTIVE VARIANCE
In order to directly compare the accuracy improvement of batches selected by different algorithms, instead of along the course of an AL trial [Figure] we conduct experiments with training sets of various sizes and compare the accuracy improvement of batches selected by AL algorithms with the same training set.
The initial training set has 10 sampled randomly from Repeated-MNIST.
In each step, we select 10 random samples and add them to training set.
Hypotheses are drawn from BNN posterior given the current training set.
We perform different AL algorithms and select batches with batch size 20.
After each batch is added to training set, we can estimate the accuracy improvement of the batch.
In each step, we perform each AL algorithm 20 times and estimate the mean and std of accuracy improvement.
The mean and std of BNNs’ accuracy are shown in figure 17.
We can see in figure 17 that our algorithms consistently select batches that have high accuracy improvement and low variance. [Figure]
[Figure]
cSG-MCMC can be improved by sampling with multiple chains (Zhang et al., 2019).
In order to evaluate different AL algorithms with this improved parallel cSG-MCMC method, we conduct experiment on CIFAR-10 dataset with batch size B = 2 , 000 .
Only BADGE and Batch-BAL AN C E are run due to their competitive performances in other settings.
We sample posteriors with 3 chains.
Each chain trains the model 200 epochs.
The cycle number for each chain is 4 and 3 posterior samples are collected in each cycle.
The result is shown in figure 18, Batch-BAL AN C E achieves better performance than BADGE.