Active Learning in Bayesian Neural Networks: Balanced Entropy Learning Principle
Abstract
Acquiring labeled data is challenging in many machine learning applications withlimited budgets.
Active learning gives a procedure to select the most informativedata points and improve data efficiency by reducing the cost of labeling.
The infomax learning principle maximizing mutual information such as BALD has beensuccessful and widely adapted in various active learning applications.
However,this pool-based specific objective inherently introduces a redundant selection.
Inthis paper, we design and propose a new uncertainty measure, Balanced Entropy
Acquisition (BalEntAcq), which captures the information balance between theuncertainty of underlying softmax probability and the label variable.
To do this,we approximate each marginal distribution by Beta distribution.
Beta approximation enables us to formulate BalEntAcq as a ratio between a shifted entropyand the marginalized joint entropy.
The closed-form expression of BalEntAcqfacilitates parallelization by estimating two parameters in each marginal Betadistribution.
BalEntAcq is a purely standalone measure without requiring anyrelational computations with other data points.
Nevertheless, BalEntAcq capturesa well-diversified selection near the decision boundary with a margin, unlike otherexisting uncertainty measures such as BALD, Entropy, or Mean Standard Deviation (MeanSD).
Finally, we demonstrate that our balanced entropy learningprinciple with BalEntAcq consistently outperforms well-known linearly scalableactive learning methods, including a recently proposed PowerBALD, a simplebut diversified version of BALD, by showing experimental results obtained from
MNIST, CIFAR-100, SVHN, and TinyImageNet datasets.
Introduction
Acquiring labeled data is challenging in many machine learning applications with limited budgets.
As the dataset size gets bigger and bigger for training a complex model, labeling data by humansbecomes more expensive.
Active learning gives a procedure to select the most informative data pointsand improve data efficiency by reducing the cost of labeling.
The active learning problem is well-aligned with a subset selection problem that can find the mostefficient but minimal subset from the data pool [70, 34, 18, 66, 86, 87, 85].
The difference is thatactive learning is typically an iterative process where a model is trained and a collection of datapoints is selected to be labeled from an unlabelled data pool.
Therefore, it is still a theoretically verychallenging but important problem.
It is now commonly accepted that standard deep learning models do not capture model uncertaintycorrectly.
The simple predictive probabilities are usually erroneously described as model confidence[31].
So there is a risk that a model can be misdirecting its outputs with high confidence.
However, the predictive distribution generated from Bayesian deep learning models better captures the uncertainty from the data [26, 51, 67, 17].
Therefore, we focus on developing an active learning framework inthe Bayesian deep neural network model by leveraging the Monte-Carlo (MC) dropout method as aproxy of the Gaussian process [26] which may facilitate further analysis.
1.1 Our contributions
Our proposed active learning method is well-aligned with Bayesian experimental design [89, 14, 80,62, 24] with an assumption that the forward active learning iterative process follows the Bayesianprior-posterior framework.
Furthermore, our approach is also aligned with Bayesian uncertaintyquantification methods [40, 1, 35, 41, 26, 27, 48, 67, 47] with an assumption that the working neuralnetwork model is a Bayesian network [49].
In this paper, we extend and improve recent advances in both aspects of Bayesian experimentaldesign and Bayesian uncertainty quantification.
We investigate the generalized notion of the jointentropy between model parameters and the predictive outputs by leveraging a point process entropy[64, 25, 73, 16, 5].
By approximating the marginals using Beta distributions, we then derive anexplicit formula of the marginalized joint entropy by estimating Beta parameters from Bayesiandeep learning models.
As a Bayesian experiment, we revisit the well-known entropy and mutualinformation measures given expected cross-entropy loss.
We show that well-known acquisitionmeasures are functions of marginal distributions through analytical formulas.
We propose our newuncertainty measure, Balanced Entropy Acquisition (BalEntAcq), which captures the informationbalance between the uncertainty of underlying softmax probability and the label variable.
Finally, wedemonstrate that our balanced entropy learning principle with BalEntAcq consistently outperformswell-known linearly scalable active learning methods, including a recently proposed PowerBALD[47] for mitigating the redundant selection in BALD [27], by showing experimental results obtainedfrom MNIST, CIFAR-100, SVHN, and TinyImageNet datasets.
Background
2.1 Problem formulation
We write an unlabeled dataset D pool and the labeled training set D training ⊆
D pool in each activelearning iteration.
We denote by D ( n ) training if it’s necessary to indicate the specific n -th iteration step.
Given D training , we train a Bayesian deep neural network model Φ with model parameters ω ∼ p ( ω ) .
Then for a data point x given D training , the Bayesian deep neural network Φ produces the predictionprobability: Φ ( x , ω ) :
= ( P 1 ( x , ω ) , · · · , P C ( x , ω )) ∈
∆ C where ∆ C = { ( p 1 , · · · , p C ) : p 1 + · · · +p C = 1 , p i ≥ 0 for each i } and C is the number of classes.
For the final class output Y , it is assumedto be a multinoulli distribution (or categorical distribution):
[Equation]
For the sake of brevity, we sometimes omit x or ω by writing Φ ( ω ) , P i ( ω ) , Y ( ω ) or Φ , P i , Y unlesswe need further clarifications on each data point x .
Under this formulation, the oracle (active learningalgorithm) selects a subset of data points to add to the next training set, i.e. at ( n + 1) -th iteration,the training set is determined by D ( n +1) training
=
D ( n ) training ∪ { Next training batch from Oracle } .
Once thenext training batch is selected, the selected batch will be labeled.
This means that the ground truthlabel information of the selected data is added in training set D ( n +1) training in the next round.
Then thegoal in active learning is to minimize the number of selected data points to reach a certain level ofprediction accuracy.
2.2 Examples of uncertainty based active learning methods
In this section, we list up well-known uncertainty measures suitable for Bayesian active learning.
Random : Rand [ x ] := U ( ω ′ ) where U ( · ) is a uniform distribution which is independent to ω .Random acquisition function assigns a random uniform value on [0 ,
BALD (Bayesian active learning by disagreement) [58, 35, 27]: BALD [ x ] := I ( ω, Y ( x , ω )) ,where I ( · , · ) represents a mutual information between random measures.
BALD captures themutual information between the model parameters and the predictive output of the data point.
Inpractice, we calculate the mutual information between the predictive output and the predictiveprobabilities.3.
Entropy [82]: Ent [ x ] := − (cid:80) i
( E P i ) log ( E P i ) .
Entropy is the Shannon entropy with respect tothe expected predictive probability.
Entropy can be the uncertainty of the prediction probability.
Moreover, under the cross-entropy loss, we may also interpret the entropy measure as an expectedloss gain since − log ( E P i ) is the cross-entropy loss given the ground truth label is the class i .4.
Mean standard deviation (MeanSD) [14, 40, 1]: MeanSD [ x ] := 1 C (cid:80) i (cid:113) E P 2 i − ( E P i ) 2 .
Meanstandard deviation captures the average of the standard deviations for each marginal distribution.5.
PowerBALD [21, 47]: PowerBALD [ x ] := log BALD [ x ] + Z , where Z is an independentlygenerated random value from Pareto distribution with the exponent α > 0 .
We use α = 1 as adefault choice suggested by [47].
The motivation of this randomized acquisition is to mitigate theredundant selection by diversifying selected multi-batch points.
In general, we do not know whichexponent will be the optimal choice.
In a multiple acquisition scenario, we simply add the above uncertainty values for each data point x i :
[Equation]
where AcqFunc ∈ { Rand , BALD , Ent , MeanSD , PowerBALD } .
2.3 Summary of other active learning approaches
Cohn et al.
[14] provided one of the first statistical analyses in active learning, establishing howto synthesize queries that reduce the model’s forward-looking error by minimizing its varianceleveraging MacKay’s closed-form variance approximation [60].
In this fashion, there exists a line ofworks in Bayesian experimental design [11, 58, 89, 14, 80, 90, 24, 23, 38] with an assumption thatthe forward active learning iterative process follows Bayesian prior-posterior framework.
On the other hand, in active learning, accommodating both the information uncertainty and thediversification of the acquired samples is essential to improve the performance under multi-batchacquisition scenarios.
In a theoretical perspective, the most natural way to combine the uncertaintyand the diversification seems to leverage reasonable sub-modular functions, e.g. Nearest neighborset function [92], BatchBALD [48], Determinantal Point Process [6] and SIMILAR [50] with submodular information measures, and then/or apply a fast linear-time algorithm to find a diversifiedmulti-batch with a provable performance guarantee [69, 70, 20, 95, 79, 36, 37, 57].
Although a fastlinear-time solver is available for general sub-modular functions, there still exists a gap with practicalimplementation, such as high memory requirements, which makes the computation unscalable foridentifying multi-batch acquisition points, e.g., BatchBALD [48].
Similar to the sub-modular functionoptimization, there exist many customized optimization approaches, e.g. CoreSet [81] and moreapproaches [29, 39, 19, 94, 91].
Another recent approach is to look at parameters of the neural network and to diversify points suchas BADGE [4] with gradients and BAIT [3] with Fisher information.
There also exist networkarchitectural design focused approaches such as Learning loss by designing loss prediction layers[96], UncertainGCN and CoreGCN [8] with graph neural networks , VAAL [84] and TA-VAAL
[42]by applying adversarial learning methods.
Bayesian neural network model
We adopt the Bayesian neural network framework introduced in Gal et al.
The core idea inthe Bayesian neural network is leveraging the MC dropout feature to generate a distribution of thepredictive probability as an output at inference time.
Under mild assumptions, it turns out that it isequivalent to an approximation to a Gaussian Process [77, 68, 93, 26, 56].
Each
softmax
probability marginal approximately follows
Beta distribution
We may consider a Bayesian neural network model Φ as a random measure, i.e., stochastic processparametrized by D training over the data set D pool .
Given a data point x ∈ D pool , Φ ( x , ω ) producesa random probability distribution in a simplex ∆ C .
This analogy has a close connection with theconstruction of random discrete distribution, originally introduced by Kingman [45].
Since then,random measure construction has been extensively developed in Bayesian nonparametrics, and it iswell-known that Dirichlet probability having Beta marginals plays the central role in the constructionof the random discrete distribution [46, 22, 75, 74, 7, 72, 78].
It is the main motivation of the Betadistribution approximation.
Many kinds of literature similarly assume the Dirichlet distribution afterthe softmax in the Bayesian neural network.
As illustrated by Milios et al.
[65], we may follow the construction of Dirichlet distribution.
Following the approach by Ferguson [22], a Dirichlet probability can be constructed through a collection ofindependent Gamma distributions.
On the other hand, each marginal in Gaussian Process (approximated by Bayesian neural network) in the softmax output having dependent components followsa log-normal distribution (before the normalization, but after the exponentiation in softmax).
Thenby applying the shape similarity between a log-normal distribution and Gamma distribution, theconstruction of random probability from log-normal distributions would produce an approximated
Dirichlet distribution.
Therefore we may assume that the marginal distribution would approximatelyfollow the Beta distribution.
Alternatively, as an analytical approach, we may see that Beta approximation can be justified through
Laplace approximation [61, 32, 33, 17].
There exists a mapping between multivariate Gaussiandistribution and Dirichlet distribution under a softmax basis.
Then Beta distribution follows as amarginal distribution of Dirichlet distribution.
Therefore we may assume that Beta approximationexists through Laplace approximation under the assumption that the Bayesian neural network producesthe multivariate Gaussian distribution (as a marginalized Gaussian process over finite rank covariatefunction) before the softmax layer [68, 93, 26, 56].
In practice, once we estimate the sample mean and sample variance for each marginal of Φ ( x , ω ) ,we can estimate two parameters of the Beta distribution as follows.
Assume that P i ∼ Beta ( α i , β i ) .
If E P i
= m
i and Var P
i =
σ 2 i , then
[Equation]
When P i ∼ Beta ( α i , β i ) , E P i
= α
i
α
i
+ β i = m
and Var P i = α
i β i ( α i + β i ) 2 ( α i + β i +1)
=
σ 2 i .
Solving theequation with respect to α i and β i , then the (3) follows.
3.2 Marginalized joint entropy in Bayesian neural network
In this section, we derive a marginalized joint entropy in the Bayesian neural network, which shall befurther discussed in constructing our main results.
We may formulate the Bayesian neural network Φas a well-known encoder-decoder framework.
The sender sends a message ( x , ω ) with a random keyω through the Bayesian neural network, then the receiver receives a message Y ( x , ω ) .
Under this framework, controlling ω is difficult, but we can control the family of the encodedmessages Φ ( x , ω ) in a tractable manner [27, 43, 88].
We can easily prove that the mutual informationbetween ω and Y is the same as the mutual information between the encoded Φ ( x , ω ) and thepredictive output Y since Y depends only on Φ ( x , ω ) :
[Equation]
[Equation]
x , ω )) , (5) where H ( Y ( x , ω )) represents the Shannon entropy by marginalizing out the randomness of ω in
Y ( x , ω ) and I ( · , · ) represents a mutual information between two quantities.
The formulations of the mutual information (4) - (5) look natural, but we need to note that ωor Φ ( x , ω ) is on a continuous domain, and Y ( x , ω ) is on a discrete domain.
This combineddomain implies that we cannot directly apply Shannon entropy and differential entropy notions [15].
One immediate question is what the joint entropy between Φ ( x , ω ) and Y ( x , ω ) is.
For this, wecan leverage point process entropy [64, 25, 73, 16, 5] by generalizing the notion of the entropy in this combined domain.
We consider the joint entropy of Φ ( x , ω ) and Y ( x , ω ) , denoting by
H (Φ ( x , ω ) , Y ( x , ω )) through the point process entropy.
We write a Janossy density function [16]j ( p , y = i ) of (Φ ( x , ω ) , Y ( x , ω )) on ∆ C ×
[ C ] as follows:
[Equation]
where p :=
( p 1 , · · · , p C ) and f ( · ) is a density function of Φ ( x , ω ) .
Then the joint entropy ofΦ ( x , ω ) and Y ( x , ω ) can be defined as
[Equation]
By plugging (6) into (7), we have the following identity.
[Equation]
where H ( · ) represents the usual Shannon entropy, and h ( · ) represents the usual differential entropy.
By applying Jensen’s inequality, we may derive a marginalized joint entropy as an upper bound ofthe joint entropy:
[Equation]
where we ambiguously write f ( · ) to be a density function for each P i .
Assume that each P i ∼
Beta ( α i , β i ) by applying Beta approximation.
We then define a quantity of the marginalized jointentropy from (9) and we find an equivalent formulation as follows:
[Equation]
where P + i is the conjugate Beta posterior entropy of P i which follows P + i ∼ Beta ( α i + 1 , β i ) .
Weremark that h ( P + i ) can be easily calculated by the closed form entropy formula of Beta distribution.i.e.
[Equation]
where B ( · , · ) is the Beta function, and Ψ( · ) is the Digamma function.
We call the first term in (10)to be the posterior uncertainty.
We may interpret the posterior uncertainty as an expected posteriorentropy assuming that we observed a positive sample of the class toward P i for each i withoutknowing the true class label.
The first term is always non-positive, and is maximized (equals to 0 )when each P + i is Beta (1 , 1) , i.e., Uniform on [0 , 1] .
So −∞ < MJEnt [ x ] ≤ H ( Y ) .
The secondentropy term can be decomposed into two uncertainty terms:
[Equation]
The epistemic uncertainty captures the model uncertainty (as BALD), and the aleatoric uncertaintycaptures the data uncertainty [63].
Therefore the marginalized joint entropy, MJEnt [ x ] is a decomposition of three types of uncertainty values.
3.3 Entropy is for maximizing an expected cross-entropy loss
Given a ground-truth label { Y = i } , the cross-entropy loss of the neural network can be given asloss (Φ ( x , ω ) , Y = i ) =
− log E P i .
Therefore we can calculate the expected cross-entropy losswithout knowing the truth label:
[Equation]
Based on the re-formulation, we may interpret that entropy acquisition is for maximizing an expectedcross-entropy loss in a selection of acquisition points, aligning the idea with the learning loss [96].
The natural question is, "Once we acquire a data point that maximizes entropy acquisition, can weremove/or learn this expected cross-entropy amount of loss at the future stage of the active learning?".
The answer could be "No."
The exhaustive loss acquisition could only happen if the neural networkperfectly over-fits the training data.
Therefore, there exists a gap between a realistic neural networktraining scenario and the objective of the entropy acquisition.
Our equivalent loss interpretationgives us an insight into why the entropy acquisition might not be successful in practice, even in thesingle-point acquisition scenario.
3.4 BALD is a function of marginals and is strongly aligned with maximizing an expectedcross-entropy loss difference upto the next iteration
We have the mutual information between ω and Y and it is the same as the mutual informationbetween the encoded message and the channel output since Y depends only on Φ ( x , ω )
[Equation]
[Equation]
(12) where I ( · , · ) represents mutual information between two quantities.
By assuming that Φ ( x , ω )follows a Dirichlet distribution, we can calculate the mutual information analytically [2].
Thenby investigating further into the analytical mutual information formula, we see that the marginaldistributions P i ’s in Φ ( x , ω ) are sufficient to estimate BALD.
Therefore we can state BALD through
Beta marginal distributions as follows.
See Appendix for more details.
Theorem 3.1.
Under Beta marginal distribution approximation, let P i ∼ Beta ( α i , β i ) in Φ ( x , ω ) .
Then the mutual information BALD [ x ] can be estimated as follows:
BetaMarginalBALD
[Equation] [Equation]
As a Bayesian experimental design process, we may assume that each Beta marginal distribution
P i with the ground-truth label { Y = i } of the next trained model would follow the Beta posteriordistribution P + i .
Without this assumption, existing choices of acquisition functions such as BALD or
MeanSD might not be well-justified.
For example, what is the implication of maximizing mutualinformation through the active learning process with a Bayesian neural network?
How is it differentfrom the maximization of the entropy acquisition?
To answer these questions, leveraging our Betamarginalization and considering the similar idea of expected information gain [24], we may considerthe expected cross-entropy loss difference between the current stage model and the next stage model.
ExpectedEffectiveLoss
[Equation]
ExpectedEffectiveLoss captures the effective amount of cross-entropy loss for the model to learnafter the acquisition.
By definition, we see that ExpectedEffectiveLoss aims to exclude the undesirableover-fitting scenario assumption unlike Entropy acquisition.
Since Digamma function Ψ( x ) ∼ log x − 12 x where f ( x ) ∼ g ( x ) implies lim x →∞ f ( x ) /g ( x ) = 1 ,we may expect that BetaMarginalBALD [ x ] and ExpectedEffectiveLoss [ x ] would behave similarly.
Figure 1 shows the Spearman’s rank correlations among different acquisition measures upto aclass dimension C = 10 , 000 .
We observe that BetaMarginalBALD behaves equally like theoriginal BALD and we confirm that BALD and MeanSD are strongly aligned with maximizing
ExpectedEffectiveLoss.
Therefore, acquiring points through BALD or MeanSD could be a betterstrategy than Entropy because BALD or MeanSD takes into account the effective loss acquisitioninstead of the unrealistic full amount of the loss acquisition.
Balanced entropy learning principle
The previous section shows that well-known acquisition measures have an objective toward thecross-entropy loss, and they are closely related to marginal distributions.
According to Farquharet al.
[21], to be successful in active learning, they hypothesize that it is crucial to find a goodbalance between active learning bias and over-fitting bias under over-parametrized neural networks.
In parallel to their hypothesis, we define the balanced entropy (BalEnt) to be a ratio between themarginalized joint entropy (9) and the shifted entropy:
[Equation]
Recall that we call the first term in MJEnt [ x ] to be posterior uncertainty, and it is an expected posteriorentropy of underlying marginals.
BalEnt captures the information balance between the posterioruncertainty from the model Φ and entropy of the label variable Y .
4.1 Implications of balanced entropy
To understand the implication of BalEnt [ x ] , we can prove the following Theorem 4.1.
Theorem 4.1.
Let ∆ − 1 :=
⌊ 2 e H ( Y ) ⌋ and Υ :=
{ I n } , a collection of evenly divided intervals in[0 , 1] where I n := (cid:2) ( n − 1)∆ , n ∆ (cid:1) for n = 1 , · · · , (∆ − 1 − 1) and I ∆ − 1 :=
[1 − ∆ , 1] .
Let ¯ P i be adiscretized random variable over Υ of P i from Φ ( x , ω ) .
For any estimator ˆ P i of ¯ P i given the label{ Y = i } we have
[Equation]
where ϵ 1 , ϵ 2 ≥ 0 are adjustment terms depending on ∆ such that ϵ 1 → 0 ϵ 2 → 0 as ∆ → 0 .
Theorem 4.1 tries to answer the following inverse problem.
For the unlabeled data point, x , if we knowthe information of the label { Y = i } , how much can we reliably estimate the underlying probability
P i from the model Φ ?
As we know that − log P i is the cross-entropy loss of the trained model with
Y , it equivalently answers the estimation error probability of the loss prediction under a unit precisionup to − log ∆ level.
For the precision level, we are assuming to carry − log ∆
≈ H ( Y ) + log 2 natsnatural unit of information, re-scaled amount of bits, matching the enumerator with MJEnt [ x ] term.
Itis not clear how to determine a better choice of the precision level − log ∆ .
But we may understand thedenominator H ( Y )
+ log 2 is for normalizing the term BalEnt [ x ] ≤ 1 as a probability.
Then the signof BalEnt [ x ] becomes very important.
BalEnt [ x ] ≥ 0 implies that it could be impossible to perfectlypredict the loss − log P i given currently available information.
i.e., there could exist informationimbalance between the model and the label approximately starting from BalEnt [ x ] = 0 .
Therefore,insight from Theorem 4.1 suggests us a new direction for our main active learning principle.
Wedefine our primary acquisition function, namely, balanced entropy learning acquisition (BalEntAcq),as follows:
[Equation]
Since the information imbalance exists at least from BalEnt [ x ] = 0 , we prioritize to fill the information gap from BalEnt [ x ] = 0 toward positively increasing direction.
If we try to fill theinformation imbalance gap from the highest BalEnt [ x ] , the information imbalance would still existaround BalEnt [ x ] = 0 area.
Therefore, it might not improve the active learning performance much.
See Appendix A12.2 and A12.3 for different prioritization and precision level results.
That’s themotivation why we take the reciprocal of BalEnt [ x ] when BalEnt [ x ] ≥ 0 .
[Figure]
4.2 Toy example illustration
To illustrate the behavior of BalEntAcq and its relationship with other uncertainty measures, wetrain a simple Bayesian neural network with a 3-class moon dataset in R 2 .
Then we calculate eachacquisition measure for all fixed lattice points in the square domain by assuming that the unlabeledpool is highly regularized (or uniform).
i.e., by evenly discretizing the domain, we obtain eachuncertainty value for each lattice point.
The total number of lattice points is around 0.6 million.
Then we choose top- K high uncertainty values for each method to observe the prioritized region foreach method.
We use K = 25 and K = 500 .
Figure 2 illustrates the top- K points selected by eachmethod.
The most significant phenomenon is that BalEntAcq’s selection is highly diversified nearthe decision boundary showing a bifurcated margin because we are prioritizing the surface area of{ BalEnt [ x ] ≥ 0 } .
This is well-aligned with the strategy avoiding high aleatoric points.
(See Appendix
A.13)
Then we can imagine to conduct a uniform sampling on each contour surface { BalEnt [ x ] = λ }for each λ ≥ 0 , as we move to the surface for each λ < 0 .
That’s why we observe bifurcated butdiversified and balanced selection near the decision boundary with BalEngAcq in Figure 2-(a) when
K = 25 .
On the other hand, there is a preferred area for each method from other measures except
PowerBALD.
PowerBALD shows a good diversification, but it could select non-informative points.
Experimental Results
In this section, we demonstrate the performance of BalEntAcq from MNIST [55], CIFAR-100 [52],
SVHN [71], and TinyImageNet [54] datasets under various scenarios.
We used a single NVIDIA
A100 GPU for each experiment, and details about the experiments are explained in Appendix A.12.
We test Random, BALD, Entropy, MeanSD, PowerBALD, and BalEntAcq measures.
We add
BADGE for a better baseline.
Note that all acquisition measures except BADGE in our experimentsare standalone quantities, so all can be easily parallelized.
Single acquisition active learning with MNIST.
MNIST [55] is the most popular and elementarydataset to validate the performance of image-based deep learning models initially.
We use a simple
CNN model applying dropouts to all layers with a single acquisition size.
The primary purpose ofthis single acquisition experiment is to validate our proposed balanced entropy approach by removingthe contribution of diversification unlike multi-batch acquisition scenario.
Fixed features with CIFAR-100 and 3 × CIFAR-100.
In recent years, significant efforts have beenmade on building an efficient framework of unsupervised or self-supervised feature learning such as
SimCLR [12, 13], MoCo [30], BYOL [28], SwAV [9], DINO [10], etc.
As an application in activelearning, we may leverage the feature space from the unsupervised feature learning without explicitlyknowing true labels but construct a good representation space.
In our experiments, we adopt SimCLR[12] for simplicity with ResNet-50 to build a feature space for CIFAR-100.
With 3 × CIFAR-100 dataset, we observe the effect of the redundant information treatment for eachmethod by adding three identical points.
We use the same fixed feature obtained from SimCLR with
CIFAR-100.
We may observe how each method effectively diversifies the selection under a redundantdata pool scenario by fixing the feature space.
Pre-trained backbone with SVHN and strong data augmentation with TinyImageNet.
In thisexperiment, we follow a typical image classification scenario in practice.
We use the ResNet-18backbone for SVHN and the ResNet-50 backbone for TinyImageNet with ImageNet pre-trainedmodel for model architecture, and the last linear classification layer is replaced with a simple Bayesianneural network with dropouts.
We then apply strong data augmentations for TinyImageNet, includingrandom crop, random flip, random color jitter, and random grayscale.
Under this scenario, thefeature space from the backbone is continuously evolving and keeps confused as the training andactive learning process proceeds.
Because of the strong data augmentation and batch normalizationin ResNet-18 or ResNet-50, the decision boundary keeps confused, implying that the Bayesianexperimental design assumption might not hold.
However, we still want to observe the generalbehavior of each measure and how to improve the accuracy under a more dynamic feature space.
[Figure] [Figure]
[Table]
Discussion.
In all datasets, BalEntAcq consistently outperforms other linearly scalable baselinesas shown in Table 1.
BADGE shows better performances when we fix the feature space, but it’snot a linearly scalable method.
Under dynamic feature scenarios in SVHN or TinyImageNet, weobserve that our BalEntAcq performs better.
Overall, considering the acquisition calculation time(see Appendix A.14), our BalEntAcq could be a better choice.
Figure 3 shows the full activelearning curves.
For CIFAR-100 and 3 × CIFAR-100 cases, by fixing features, we control/remove allother effects possibly affecting the model’s performance, such as data augmentation or the role ofbackbone in the classification.
As demonstrated in Figure 2, BalEntAcq is very efficient in selectingdiversified points along the decision boundary.
Instead, PowerBALD suffers from improving accuracybecause it focuses more on diversification/randomization by missing the information near the decisionboundary.
For SVHN or TinyImageNet, BalEntAcq shows better performance again.
We supposethat diversification near the decision boundary in BalEntAcq also plays the data exploration becausethe representation space keeps evolving with the backbone training.
Conclusion
In this paper, we designed and proposed a new uncertainty measure, Balanced Entropy Acquisition(BalEntAcq), which captures the information balance between the underlying probability and thelabel variable through Beta approximation with a Bayesian neural network.
BalEntAcq offers adiversified selection and is unique compared to other uncertainty measures.
Moreover, we expect thatour proposed balanced entropy measure does not have to be confined to active learning problems ingeneral.
BalEntAcq can be applied to improve the diversified selection process or accuracy estimationin a different type of Bayesian neural network framework.
Therefore, we look forward to havingfurther follow-up studies with broad applications beyond the active learning problems.
Limitations
As we specified in the introduction our focus is MC-dropout-based Bayesian neural networks;our experiments have been limited to dropout-based Bayesian neural networks.
However, ourtheoretical development does not require special architectural assumptions if we can apply Betaapproximation.
So one can apply our proposed method to any Bayesian classification network with
Beta approximation.
Moreover, considering the similarity with recent theoretically guaranteed activelearning algorithm with abstention [59, 83, 76, 97] (see Appendix A.13), we expect to replicatethe similar out-performance in other types of the Bayesian networks, e g., Gaussian process [77],ensemble network [53], variational-dropout network [44], Laplace Redux [17] and so on.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to
answer these
questions.
For each
question, change the default [TODO to
Yes
No]
or [N
/A
You arestrongly encouraged to include a justification to your answer , either by referencing the appropriate section of 602 your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data are proprietary.•
Did you include the license to the code and datasets?
[N/A]Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist 607 section does not count towards the page limit.
In your paper, please delete this instructions block and only keepthe Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?[Yes]
Yes]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]