L EAST P ROBABLE D ISAGREEMENT R EGION FOR A CTIVE L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Active learning strategy to query unlabeled samples nearer the estimated decision boundary at each step has been known to be effective when the distance from the sample data to the decision boundary can be explicitly evaluated; however, in numerous cases in machine learning, especially when it involves deep learning, conventional distance such as the (cid:96) p from sample to decision boundary is not readily measurable.
This paper deﬁnes a theoretical distance of unlabeled sample to the decision boundary as the least probable disagreement region (LPDR) containing the unlabeled sample, and it discusses how this theoretical distance can be empirically evaluated with a lower order of time complexity.
Monte Carlo sampling of the hypothesis is performed in approximating the theoretically deﬁned distance.
Experimental results on various datasets show that the proposed algorithm consistently outperforms all other high performing uncertainty based active learning algorithms and leads to state-of-the-art active learning performance on CIFAR10,
CIFAR100, Tiny ImageNet and Food101 datasets.
Only the proposed algorithm outperforms random sampling on CIFAR100 dataset using K-CNN while all other algorithms fail to do so.
I NTRODUCTION
Active learning (Cohn et al., 1996) is a subﬁeld of machine learning to attain data efﬁciency with fewer labeled training data when it is allowed to choose the training data from which to learn.
For many real-world learning problems, large collections of unlabeled samples is assumed available, and based on a certain query strategy, the label of the most informative data is iteratively queried to an oracle to be used in retraining the model (Bouneffouf et al., 2014; Roy & McCallum, 2001; Sener & Savarese, 2017b;
Settles et al., 2008; Sinha et al., 2019; Sener & Savarese, 2017a;
Pinsler et al., 2019; Shi & Yu, 2019; Gudovskiy et al., 2020).
Active learning attempts to achieve high accuracy using as few labeled samples as possible (Settles, 2009).
Of the possible query strategies, uncertainty-based sampling (Culotta & McCallum, 2005; Scheffer et al., 2001; Mussmann & Liang, 2018), which enhances the current model by labeling unlabeled samples that are difﬁcult for the model to predict, is a simple strategy commonly used in pool-based active learning (Lewis & Gale, 1994).
Nevertheless, many existing uncertainty-based algorithms have their own limitations.
Entropy (Shannon, 1948) based uncertainty sampling can query unlabeled samples near the decision boundary for binary classiﬁcation, but it does not perform well in multiclass classiﬁcation as entropy does not equate well with the distance to a complex decision boundary (Joshi et al., 2009).
Another approach based on MC-dropout sampling (Gal et al., 2017) which uses a mutual information based BALD (Houlsby et al., 2011) as an uncertainty measure identiﬁes unlabeled samples that are individually informative.
This approach, however, is not necessarily informative when it is jointly considered with other samples for label acquisition.
To address this problem, BatchBALD (Kirsch et al., 2019) is introduced.
However, BatchBALD computes, theoretically, all possible joint mutual information of batch, and is infeasible for large query size.
The ensemble method (Beluch et al., 2018), one of the query by committee (QBC) algorithm
(Seung et al., 1992), has been shown to perform well in many cases.
The fundamental premise behind the QBC is minimizing the version space (Mitchell, 1982), which is the set of hypotheses that are consistent with labeled samples.
However, the ensemble method requires high computation load because all networks that make up the ensemble must be trained.
This paper deﬁnes a theoretical distance referred to as the least probable disagreement region (LPDR) from sample to the estimated decision boundary, and in each step of active learning, labels of unlabeled samples nearest to the decision boundary in terms of LPDR are obtained to be used for retraining the classiﬁer to improve accuracy of the estimated decision boundary.
It is generally understood that labels to samples near the decision boundary are the most informative as the samples are uncertain.
Indeed in Balcan et al.
(2007), selecting unlabeled samples with the smallest margin to the linear decision boundary and thereby minimal certainty attains exponential improvement over random sampling in terms of sample complexity.
In deep learning, it is difﬁcult to identify samples nearest to the decision boundary as sample distance to decision boundary is difﬁcult to evaluate.
An adversarial approach (Ducoffe & Precioso, 2018) to approximate the sample distance to decision boundary has been studied but this method does not show preservation of the order of the sample distance and requires considerable computation in obtaining the distance.
D ISTANCE : L EAST P ROBABLE D ISAGREEMENT R EGION (LPDR)
This paper proposes an algorithm for selecting unlabeled data that are close to the decision boundary which can not be explicitly deﬁned in many of cases.
Let X , Y , H and D be the instance space, the label space, the set of hypotheses h : x → y and the joint distribution over ( x , y ) ∈ X × Y .
The distance between two hypotheses ˆ h and h is deﬁned as the probability of the disagreement region for ˆ h and h . This distance was originally deﬁned in Hanneke et al. and Hsu (2010):
[Equation]
This paper deﬁnes the sample distance d of x to the hypothesis ˆ h ∈ H based on ρ as the least probable disagreement region (LPDR) that contains x :
[Equation]
2) where H ( x , ˆ h ) = { h ∈ H : ˆ h ( x ) (cid:54) =
h ( x ) } .
Figure 1 shows an example of LPDR. Let’s deﬁne H = { h θ : h θ ( x ) =
I [ x > θ ] } on input x sampled from uniform distribution D = U [0 , 1] where I [ · ] is an indicator function. Suppose x = x 0 and ˆ h = h a ∈ H when a < x 0 . Here, H ( x 0 , h a ) consists of all hypotheses whose prediction on x 0 is in disagreement with h a ( x 0 )
= 1 , i.e., H ( x 0 , h a ) = { h b ∈ H : h b ( x 0 ) = 0 } = { h b ∈ H : b > x 0 } . Then, the LPDR between x 0 and h a , d ( x 0 , h a ) = x 0 − a
as the inﬁmum of the distance between h a and h b ∈ H ( x 0 , h a ) is ρ ( h a , h x 0 ) = x 0 − a .
Here, the sample distribution D is unknown, and H ( x , ˆ h ) may be uncountably inﬁnite. Therefore, a systematic and empirical method for evaluating the distance is required. One might the procedure below: Sample hypotheses sets H (cid:48)
= { h (cid:48) : ρ (ˆ h, h (cid:48) ) ≤ ρ (cid:48) } in terms of ρ (cid:48) , and perform grid search to determine the smallest ρ (cid:48) such that there exists h (cid:48) ∈ H (cid:48) satisfying ˆ h ( x ) (cid:54)
=
h (cid:48) ( x ) for a given x .
Sampling the hypotheses within the ball can be performed by sampling the corresponding parameters with the assumption that the expected hypothesis distance is monotonically increasing for the expected distance between the corresponding parameters (see Assumption 1).
This scheme is based on performing grid search on ρ (cid:48) and is therefore computationally inefﬁcient.
However, unlabeled samples can be ordered according to d without grid search with the assumption that there exists a H (cid:48) such that variation ratio V ( x ) = 1 − f ( x ) m / |H (cid:48) | and d ( x , ˆ h ) have strong negative correlation where f ( x ) m = max c (cid:80) h (cid:48) ∈H (cid:48) I [ h (cid:48) ( x ) =
c ] (see Assumption 2).
Assumption 1.
The expected distance between ˆ h and randomly sampled h is monotonically increasing in the expected distance between the corresponding ˆ w and w , i e., E [ (cid:107) ˆ w − w 1 (cid:107) | ˆ w ] ≤ E [ (cid:107) ˆ w − w 2 (cid:107) | ˆ w ] implies that E [ ρ (ˆ h, h 1 ) | ˆ h ] ≤ E
[ ρ (ˆ h, h 2 )
| ˆ h ] where ˆ w , w 1 and w 2 are the parameters pertaining to ˆ h, h 1 and h 2 respectively.
Assumption 2.
There exists a hypothesis set H (cid:48) sampled around ˆ h having the property that large variation ratio for a given sample data implies small sample distance to ˆ h with high probability, i.e., there exists H (cid:48) such that V ( x 1 ) ≥ V ( x 2 ) implies that d ( x 1 , ˆ h ) ≤ d ( x 2 , ˆ h ) with high probability.
E MPIRICAL S TUDIES OF LPDR 3. H YPOTHESES AND P ARAMETERS I N D EEP N ETWORKS : A SSUMPTION
The distance between two hypotheses can be approximated by vectors of the predicted labels on random samples by the hypotheses:
[Equation]
where x ( i ) is the i th sample for i ∈ [ m ] .
The h is sampled by sampling model parameter w ∼ N ( ˆ w , I σ 2 ) where ˆ w is the model parameter of ˆ h , and the expectation of distances between w and ˆ w depends on σ .
The ρ e is obtained by the average of 100 times for a ﬁxed σ .
The left-hand side of Figure 2 shows the relationship between ρ e and σ on various datasets and deep networks.
The ρ e increases almost monotonically as σ increases.
This implies that the order is preserved between the σ and ρ e .
Furthermore, the ρ e is almost linearly proportional to log( σ ) in the ascension of the graph, i.e., σ ∝ e βρ e for some β > 0 .
The right-hand side of Figure 2 shows V with respect to σ for each unlabeled sample on MNIST.
The sample distance to the decision boundary can be expressed as σ at which the variation ratio is not zero for the ﬁrst time (white arrow), where the indices of unlabeled samples in y-axis are ordered by LPDR.
The variation ratio increases as the σ increases, and it is expected that the data point with short distance has the large variation ratio compared to the data point with long distance on a certain range of σ .
3. LPDR VS V ARIATION R ATIO : A SSUMPTION
The left-hand side of Figure 3 shows Spearman’s rank correlation coefﬁcient (Spearman, 1904) between LPDR and the variation ratio with respect to σ .
The correlation is calculated using only unlabeled samples whose variation ratio is not 0 .
The strong rank correlation is veriﬁed when the σ has the appropriate value.
Too larger value of σ generates hypotheses too far away from ˆ h, which is not helpful to measure the distance.
The right-hand side of Figure 3 shows an example of σ ( log( σ ) =
− 5 . 0 ) which makes LPDR and the variation ratio have a strong negative correlation on MNIST, that is, the data point with larger variation is closer to the decision boundary.
Results for various datasets and networks are presented in Appendix C.
The time complexity is discussed to validate the efﬁciency of using variation ratio.
Let m , N and n σ be the unlabeled sample size, |H (cid:48) | and the number of grid for σ respectively.
Ordering unlabeled samples in terms of LPDR by grid search with respect to σ requires the time complexity of m × N × n σ (see the right-hand side of Figure 2).
However, using variation ratio for ordering unlabeled samples reduces the time complexity to m × N .
In the case of n σ = cN for some c > 0 , then the time complexity can be reduced from O ( mN 2 ) to O ( mN ) .
A LGORITHM FOR LPDR 4. F RAMEWORK
Let L t
and U t be the labeled and unlabeled samples at step t .
At step t , LPDR trains model parameters ˆ w t using labeled samples L t , and constructs H (cid:48) by sampling the model parameters w (cid:48) n ∼ N ( ˆ w t , I σ 2 ) for n ∈ [ N ] .
Then, LPDR queries the top q unlabeled samples having highest variation ratio from the pool data P t ⊂ U t of size m .
4. C ONSTRUCTION OF S AMPLED H YPOTHESIS S ET
It is important to set an appropriate σ when constructing H (cid:48) as variation ratios goes to 0 with decreasing σ (see the right-hand side of Figure 2) and the rank correlation goes to zero with increasing σ (see the left-hand side of Figure 3).
Theoretically, let’s consider the binary classiﬁcation with logistic regression where the predicted label is deﬁned as y = sgn
( x T w ) and sup x ∈X (cid:107) x (cid:107) ∞ < ∞ .
Then the following theorem holds and the proof is described in Appendix A.
Theorem 1.
Suppose that w (cid:48) n for n = 1 , . .
. , N are generated with the variance of σ 2 .
For all x , the followings hold: 1) As N → ∞ , 1 − f ( x ) m /N goes to 0 in probability as σ 2 goes to 0 , 2) As
N
→ ∞ , 1 − f ( x ) m /N goes to 1 / 2 in probability for binary classiﬁcation using logistic regression as σ 2 goes to ∞ .
The implication of Theorem 1 is that when σ is too small or too large, it would be difﬁcult to compare the sample distances of unlabeled samples.
In this active learning task, at least q most informative unlabeled samples must be identiﬁed.
To meet this condition, it is reasonable to set ρ (cid:48) n denoted in the
Algorithm 1 as ρ ∗ =
q/m , which is not very small and is less than 1 / 2 in general, for N hypotheses.
This can be attained by updating σ (cid:48)
n as σ (cid:48) n +1 = σ (cid:48) n e − β
( ρ (cid:48)
n − ρ ∗ ) where β > 0 (see Appendix D).
The Figure 10 in Appendix E shows the ﬁnal test accuracy with respect to target ρ (cid:48) n on MNIST dataset.
The LPDR performs best when the target ρ (cid:48)
n is roughly ρ ∗ .
In addition, the range of target ρ (cid:48) n , associated with the best performance, is wide; thus, LPDR is relatively robust against target ρ (cid:48) n .
Furthermore, LPDR is robust against hyperparameters β , N and sampling layers (see Appendix F). Input :
L 0 , U 0 : Initial labeled and unlabeled samples m , q : Size of pool data and number of queries σ 20 : Initial variance for sampling ρ ∗ : Target hypothesis distance (= q/m ) Procedure: 1: for step t = 0 , 1 , 2 , . . . , T −2: Train parameters ˆ w t with L t , then evaluate its empirical error ˆ ε t on L t 3: σ t → σ (cid:48)4: for n = 1 , 2 , . .
. , N 5:
Sample parameters w (cid:48) n ∼ N ( ˆ w t , I σ (cid:48) 2 n ) for h (cid:48) n 6:
Compute γ n = e − ( ε (cid:48) n − ˆ ε t ) + where ε (cid:48) n is empirical error of w (cid:48) n on L t 7:
Compute ρ (cid:48) n = ρ e (ˆ h t , h (cid:48) n ) 8:
Update σ (cid:48) n +1 = σ (cid:48) n e − β
( ρ (cid:48)
n − ρ ∗ ) where β >9: end for 10: σ (cid:48)
{+N +
+}
→
σ t + 11:
Compute V w ( x ( i ) )
= 1 − f
( i ) w / (cid:80) Nn
=1 γ n where f ( i ) m
= max c (cid:80) Nn =1 γ n I (cid:2) h (cid:48)
n ( x ( i ) )
=
c (cid:3) 12:
Get I ∗ = arg max I⊂I P t , |I| =
q (cid:80) i ∈I V w
x ( i ) )
where I P t = (cid:8)
j : x ( j ) ∈ P t ⊆
U t (cid:9) 13:
Update L t +1 =
L t ∪ (cid:8)(cid:0) x
( i ) , y ( i ) (cid:1)(cid:9)
i ∈I ∗ and U t +1 =
U t \ (cid:8) x
( i ) (cid:9)
i
∈I ∗ 14: end for
Meanwhile, the efﬁciency of querying samples in the disagreement region of the version space is well known both theoretically (Hanneke et al., 2014) and empirically (Beluch et al., 2018).
When the trained hypothesis ˆ h t is in the version space, the sampled hypotheses h (cid:48) n s are in the version space with high probability, but there are cases where they are outside the version space (see Appendix G).
Thus, LPDR gives weight γ n on the prediction of sampled hypothesis h (cid:48) n where γ n = e − ( ε (cid:48) n − ˆ ε t ) + is a function of ˆ ε t = err L t (ˆ h t ) and ε (cid:48) n = err L t ( h (cid:48) n ) .
Here, ( · ) + is max { 0 , ·} and err L ( h ) is the empirical error of h on L .
Then, LPDR uses weighted variation ratio V w as a function of the weighted frequency of the modal class f w as deﬁned below: [Equation]
[Equation]
where f ( i ) w = max c (cid:80) Nn =1 γ n I (cid:2) h (cid:48) n ( x ( i ) )
=
c (cid:3) and x ( i ) ∈ P t ⊆ U t .
If H (cid:48) is a subset of the version space in realizable case, the sample complexity of LPDR follows Hanneke’s theorem (Hanneke et al., 2014).
Let Λ be the sample complexity deﬁned as the smallest integer t such that for all t (cid:48) ≥ t , err( h t (cid:48) ) ≤ (cid:15)
where err( h ) :
= P D [ h ( X ) (cid:54) =
Y ] with probability at least 1 − δ .
Then, LPDR achieves a sample complexity Λ such that, for D in the realizable case, ∀ (cid:15), δ ∈ (0 , 1) ,
[Equation]
where D and ξ are the VC-dimension of H and the disagree coefﬁcient with respect to H and D .
When ξ = O (1) , in terms of (cid:15) , the number of labeled samples required by LPDR is just
O (log(1 /(cid:15) ) · log log(1 /(cid:15) )) , while the number of labeled samples by a passive learning is Ω(1 /(cid:15) ) .
Therefore, in this case, LPDR provides an exponential improvement over passive learning in sample complexity (Hsu, 2010).
E XPERIMENTS
This section discusses experimental results on 8 benchmark datasets: MNIST (LeCun et al., 1998),
CIFAR10 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2019), EMNIST (Cohen et al., 2017),
CIFAR100 (Krizhevsky et al., 2009), Tiny ImageNet (subset of the ILSVRC dataset containing [Table] 200 categories rather than the usual 1000 categories; Russakovsky et al., 2015), Food101 (Bossard et al., 2014) and HAM10000 (Tschandl et al., 2018) datasets.
For fair comparison with other active learning algorithms, simple two layered CNN, referred to as ‘S-CNN’ (Chollet et al., 2015) is used for MNIST and four layered CNN, referred to as ‘K-CNN’ (Chollet et al., 2015) is used for
CIFAR10, SVHN, EMNIST and CIFAR100.
Additionally, Wide-ResNet (WRN-16-8; Zagoruyko & Komodakis, 2016) is used for CIFAR100, Tiny ImageNet, Food101 and HAM10000.
Figure 4–7 show magniﬁed plots of test accuracy to accentuate the difference in performance among different methods: initial labeled sample sizes are not shown in the ﬁgures.
Figures that include initial labeled sample size are presented in Appendix H.
5. E XPERIMENTAL S ETTINGS
Experimental settings regarding total number of epochs, data size and acquisition size are summarized in Table 1, and other details concerning the model, optimizer, batch size, learning rate and hyperparameters are presented in Appendix B.
5. R ESULTS FOR MNIST, CIFAR10, SVHN AND EMNIST
A number of experiments are conducted to compare performance of LPDR with other high performing uncertainty based active learning algorithms on 8 datasets.
Figure 4 shows the test accuracy with respect to the number of labeled samples on MNIST, CIFAR10, SVHN and EMNIST datasets.
Each algorithm is denoted such as ‘LPDR’:
the proposed algorithm, ‘Random’: random sampling, ‘Entropy’: entropy based uncertainty sampling, ‘MC-BALD’: MC dropout sampling using BALD, ‘MC-VarR’:
MC dropout sampling using variation ratio (Ducoffe & Precioso, 2015) and ‘ENSVarR’: ensemble method.
Overall, LPDR either performs best or comparable with all other algorithms.
Its performance is consistent regardless of the benchmark datasets.
In the early step, LPDR signiﬁcantly outperforms all other algorithms on MNIST and CIFAR10 datasets.
Of all the algorithms compared, Entropy performed the worst.
MC-BALD performed well only on SVHN dataset: it seems that the performance of BALD is highly dependent on the dataset.
With the query size set to 1, LPDR outperforms BatchBALD on MNIST dataset (see Appendix I).
Although MC-VarR and
ENS-VarR are based on different sampling methods, both perform similarily-both outperforming all others on EMNIST dataset, while showing a signiﬁcant drop in performance compared to LPDR on SVHN and CIFAR10 datasets.
It is observed that the performances of other algorithms have a relatively strong data dependency compared to LPDR.
On CIFAR10 dataset, the performances of
MC-VarR and ENS-VarR are no better than that of Random, and Entropy and MC-BALD have lower performance than Random.
These results can be attributed to the low network capacity compared to the data complexity.
This issue will be discussed in next section.
5. R ESULTS FOR CIFAR100 WITH K-CNN AND W IDE -R ES N ET
In order to compare the performance of the algorithms with respect to the network capacity experiments are conducted using networks of different capacity but on the same dataset.
Figure 5 shows the results of test accuracy with respect to the number of labeled samples on CIFAR100 dataset with
K-CNN and WRN-16-8.
The left-hand ﬁgure is the results of using K-CNN, which has a relatively smaller network capacity than that of WRN-16-8.
With the exception of LPDR, the performances of all algorithms are much worse than that of Random.
The right-hand ﬁgure is the result of using
WRN-16-8, which has a relatively larger network capacity.
In contrast to the results for K-CNN, most algorithms outperform Random.
With a large network capacity, the performance gap between
LPDR and the other algorithms is reduced, but LPDR still outperforms others.
LPDR is able to perform consistently better than Random regardless of the network capacity and it seems to be particularly effective with low capacity networks. outperforms all algorithms in more
. 5. R ESULTS FOR T INY I MAGE N ET AND F OOD
Experiments on a more difﬁcult task are conducted.
Figure 6 shows test accuracy with respect to the number of labeled samples on Tiny ImageNet and Food101 datasets with WRN-16-8.
Tiny
ImageNet and Food101 are considered to be more difﬁcult than CIFAR100.
Even on more difﬁcult tasks, LPDR outperforms all other algorithms.
5. R ESULTS FOR HAM
Additional experiments are conducted to compare the performance of the algorithms on imbalanced
HAM10000 dataset with WRN-16-
Figureshows the results of the test accuracy with respect to the number of labeled samples.
The LPDR outperforms all other algorithms compared.
Figure 15 in
Appendix J shows the results of AUC with respect to the number of labeled samples.
The LPDR performs comparable with all other algorithms.
To sum up the comparing algorithms across all experimental settings and repetitions, rank and DolanMore curves are presented in Appendix K.
The
LPDR consistently achieves top rank for all steps and signiﬁcantly outperforms the other algorithms in all experimental settings.
R ELATED W ORK
Other than uncertainty-based sampling framework (Culotta & McCallum, 2005; Scheffer et al., 2001; Mussmann & Liang, 2018; Lewis & Gale, 1994; Gal et al., 2017; Kirsch et al., 2019;
Beluch et al., 2018) for active learning, decision-theoretic framework based methods such as expected model change (Settles et al., 2008) have certain relevance to the proposed LPDR as unlabeled samples nearer the decision boundary which LPDR is attempting to identify have larger gradients leading larger model change.
Recently, adversarial approaches are proposed to discriminate labeled and unlabeled samples (Gissin & Shalev-Shwartz, 2019; Sinha et al., 2019; Zhang et al., 2020), and after performing adversarial learning, any unlabeled samples that is most conﬁdently predicted as unlabeled is queried and used to retrain the network.
Here adversarial learning is used to indirectly identify sample near the decision boundary.
C ONCLUSION
This paper deﬁnes a theoretical distance of unlabeled sample to the decision boundary referred to as the least probable disagreement region (LPDR) containing the unlabeled sample for active learn- ing.
LPDR can be evaluated empirically with low computational load by making two assumptions regarding parameters of the hypothesis space, variation ratio and the LPDR.
The two assumptions are empirically veriﬁed.
Experimental results on various datasets show that LPDR consistently outperforms all other high performing uncertainty based active learning algorithms and leads to state-of-the-art active learning performance on CIFAR10, CIFAR100, Tiny ImageNet, and Food101 datasets.
In addition, LPDR is able to perform consistently better than random sampling regardless of the network capacity while all other algorithms compared fail to do so.
LPDR is simple enough to be applied to various classiﬁcation tasks with deep networks: the implementation requires only sampling a subset of parameters (parameters in the last FC layer of the deep network).
Additionally, LPDR is capable of quick and reliable performance in a variety of different settings with only a computational load that is not much higher than that of other uncertainty sampling methods.
In conclusion, LPDR is an effective uncertainty based sampling algorithm in pool-based active learning.
A PPENDIX A P ROOF OF T HEOREM
Assume that (cid:107) ˆ w t (cid:107) = 1 without the loss of generality, and (cid:107) x (cid:107) (cid:54) = 0 to avoid the null case.
The predicted label of x by w (cid:48) n disagrees with that by ˆ w t if sgn (cid:0) x T w (cid:48) n (cid:1) (cid:54) = sgn (cid:0) x T
ˆ w t (cid:1) , here, sgn(0) = 1 .
Note that [Equation] where e (cid:48) n = ( Z n 1 , . . .
, Z n | w | ) T , and Z nk s are independent random variables from N (0 , 1) .
The event of { sgn (cid:0) x T w (cid:48) n (cid:1) (cid:54) = sgn (cid:0) x T ˆ w t (cid:1) } is equal to that of E 1 ∪ E 2 where E 2 = { σ x T e (cid:48) n ≥ 0 ,
x T ˆ w t < 0 } and E 2 = { σ x T e (cid:48) n < 0 , x T ˆ w t ≥ 0 } .
Thus, the proof has two folds: the cases of 1) E 1 and 2) E 2 .
In the ﬁrst fold,
[Equation]
(cid:19) where Z ∼ N (0 , 1) , Φ is the cumulative distribution function of the normal distribution, and a ( x , ˆ w t ) =
| x T ˆ w t | / (cid:107) x (cid:107) .
Note that σ x T e (cid:48) n ∼ N (0 , σ 2 (cid:107) x (cid:107) 2 ) .
Consequently, P [ E 1 ] < 1 / 2 due to a ( x , ˆ w t ) > 0 .
Hence, the following
[Equation]
goes to value greater than 1/2 in probability as N → ∞ because Var( f ( x ) m /N )
→ 0 as N → ∞ .
Therefore, as N → ∞ , ∀ x , the variation ratio is
[Equation]
in probability. This is due to that f ( x ) m is the frequency of mode class with probability tending toas N → ∞ . By the smoothness of Φ ,
[Equation]
f [Equation] x
[Equation]
and
[Equation]
as
[Equation]
→ ∞ .
Next, in the second fold,
[Equation]
Consequently, P [ E 2 ] < 1 / 2 . Hence the following [Equation] goes to the value greater than 1/2 in probability as N → ∞ because Var( f ( x ) m /N )
→ 0 as N → ∞ . Therefore, as N → ∞ , ∀ x , the variation ratio is [Equation] in probability.
This is due to that f ( x ) m is the frequency of mode class with probability tending toas N → ∞ .
By the smoothness of Φ ,
[Equation]
and
[Equation]
This completes the proof.
[Equation]
B E XPERIMENTAL S ETTINGS B. D ATASETS
MNIST (LeCun et al., 1998) is a dataset of handwritten digits which has a training set of 60 ,samples and a test set of 10 , 000 samples in 10 classes.
Each sample is a black and white image and 28 × 28 in size.
CIFAR10 and CIFAR100 (Krizhevsky et al., 2009) are labeled subsets of the 80 million tiny images dataset which have a training set of 50 , 000 samples and a test set of 10 , 000 samples in 10 andclasses respectively.
Each sample is a color image and 32 × 32 in size.
SVHN (Netzer et al., 2019) is a real-world digit image dataset which has a training set of 73 ,samples and a test set of 26 , 032 samples in 10 classes.
Each sample is a color image and 32 × 32 in size.
EMNIST (Cohen et al., 2017) is a dataset of handwritten character digits which has a training set of 80 , 000 samples and a test set of 10 , 000 samples in 47 classes.
Each sample is a black and white image and 28 × 28 in size.
Tiny ImageNet is a subset of the ILSVRC (Russakovsky et al., 2015) dataset which has 100 ,samples in 200 classes.
Each sample is a color image and 64 × 64 in size.
In experiments, Tiny
ImageNet is split into two parts: a training set of 90 , 000 samples and a test set of 10 , 000 samples.
Food101 (Bossard et al., 2014) is a ﬁne grained dataset which has a training set of 75 , 750 samples and a test set of 25 , 250 samples in 101 classes.
Each sample is a color image and resized to 75 × 75 .
HAM10000 (Tschandl et al., 2018) is a imbalanced dataset which has 10 , 015 samples in 7 classes.
Each sample is a color image and resized to 75 × 75 .
In experiments, HAM10000 is split into two parts: a training set of 8 , 515 samples and a test set of 1 , 500 samples.
All datasets are used without any preprocessing of images.
B. S ETTINGS
S-CNN, which is the Keras MNIST CNN implementation (Chollet et al., 2015), consists of [ 3 × 3 × 32 conv - 3 × 3 × 64 conv - 2 × 2 maxpool - dropout ( 0 . 25 )
- 128 dense - dropout ( 0 . 5 ) -
#class dense - softmax] layers.
K-CNN, which is the Keras CIFAR CNN implementation (Chollet et al., 2015), consists of [two 3 × 3 × 32 conv - 2 × 2 maxpool - dropout ( 0 . 25 ) - two
3 × 3 × 64 conv - 2 ×maxpool - dropout ( 0 . 25 ) - 512 dense - dropout ( 0 . 5 ) - #class dense - softmax] layers.
WRN-16is a wide residual network that has 16 convolutional layers and a widening factor 8 (Zagoruyko &
Komodakis, 2016).
The optimizer, initial learning rate, learning rate schedule and batch size for each experimental setting are described in Table 2.
He normal initialization is used for all models.
All experiments are run for a ﬁxed number of acquisition steps until a certain amount of training data is labeled.
Results are averaged over 5 repetitions.
For all datasets, the initial labeled samples for each repetition are randomly sampled according to the distribution of the training set.
For MC dropout we use 100 forward passes, and ensemble consists of 5 networks of identical architecture but different random initialization and random batches.
For LPDR, we set σ 0 = 0 .
01 , β = 1 , N =and parameter sampling is applied to the last dense layer of each network.
[Table]
C R ANK C ORRELATION BETWEEN LPDR AND V ARIATION R ATIO
Figure 8 shows an example of negative Spearman’s rank correlation between LPDR and the variation ratio for each experimental setting. Samples with increasing LPDR or variation ratio are ranked from high to low.
The σ is selected to satisfy ρ (cid:48) n = ρ ∗ =
q/m at initial step.
D R EGULATING ρ (cid:48) n BY THE V ARIANCE OF S AMPLING
The left-hand side of Figure 9 shows the ρ (cid:48) n with respect to the active learning progress.
For all experiments, LPDR reliably guides the ρ (cid:48) n to be ρ ∗ =
q/m (MNIST: 0 .
01 , CIFAR10: 0 .
05 , SVHN: 0 .
05 , EMNIST: 0 .
075 , CIFAR100 (KCNN): 0 .
1 , CIFAR100 (WRN): 0 . 2 , Tiny ImageNet: 0 . 25 ,
Food101: 0 .
2 and HAM10000: 0 . 1 )
after the initial few steps.
The right-hand side of Figure 9 shows log( σ ) with respect to the active learning progress.
For all experiments, the variance of sampling increases as the labeling proceeds.
This is because larger variance is required to make the ρ (cid:48)
n = ρ ∗ since unlabeled samples move away from the learned decision boundary from labeled samples due to an increase in network conﬁdence as the number of labeled samples increases.
E F INAL TEST ACCURACY VS T ARGET ρ (cid:48) n
The Figure 10 shows the ﬁnal test accuracy with respect to target ρ (cid:48) n on MNIST dataset.
The results show that at around ρ ∗ (= 0 . 02) , it performs the best for q = 20 and m = 1000 .
In addition, the range of target ρ (cid:48) n , associated with the best performance, is wide ( 0 .
01 ∼ 0 . 1 ); thus, LPDR is robust against the target ρ (cid:48)
n in the wide range.
F R OBUSTNESS OF LPDR AGAINST H YPERPARAMETERS
LPDR has four hyperparameters: 1) the initial variance of sampling σ 0 ; 2) the positive hyperparameter for regulating the variance of sampling β ; 3) the number of sampled hypotheses N , and 4) the layer index of the network to which sampling is applied.
The σ 0 has no signiﬁcant effect on the performance of LPDR since σ is adaptively regulated based on the ρ (cid:48) n while sampling the sampled hypothesis.
Thus, σ 0 is not examined in detail.
Figure 11 shows the performance comparison with respect to the hyperparameters of LPDR on MNIST and CIFAR10 datasets.
The left ﬁgures show that there is no signiﬁcant difference in the performance of LPDR for various β ∈ { 0 . 1 , 1 , 10 } on both datasets.
The robustness of LPDR against β is based on the sufﬁcient buffer for regulating σ since the range of target ρ (cid:48) n associated with the best performance is wide.
The middle ﬁgures show that there is no signiﬁcant difference in the performance of LPDR for various N ∈ { 5 , 10 , 20 , 50 , 100 , 200 } on both datasets. The robustness of LPDR against N is based on the sufﬁcient discrimination in the variation ratio for identifying q most informative unlabeled samples with a small number of sampled hypotheses by setting ρ ∗ = q/m . The right ﬁgures show that there is no signiﬁcant difference in the performance of LPDR for the sampling to the parameters of last layer and to the parameters of all layers of the networks on both datasets.
G E MPIRICAL E RRORS OF L EARNED AND S AMPLED H YPOTHESES
Figure 12 shows the empirical error of the learned and the sampled hypotheses with respect to the acquisition step for all experimental settings.
In many cases, the empirical error of the learned hypothesis becomes zero, thus it is placed in the version space while the sampled hypothesis is often placed outside the version space, e.g., in SVHN dataset.
Even in the cases of EMNIST and
CIFAR100 with K-CNN datasets, as the number of labeled samples increases, the empirical error of the learned and the sampled hypothesis increases.
To address this situation, LPDR incorporates the weighted hypotheses based on the prediction error difference between the learned and the sampled hypotheses, and it works well empirically.
H P LOTS FOR T EST A CCURACY
Figure 13 shows the test accuracy with respect to the number of labeled samples from initial to ﬁnal step for all experimental settings.
I LPDR VS MC-B ATCH BALD
Figure 14 shows the performance comparison between LPDR and MC-BALD on MNIST dataset using S-CNN when the query size is 1 or 20 .
LPDR signiﬁcantly outperforms MC-BatchBALD on
MNIST dataset when q = 1 such that MC-BatchBALD is completely identical to MC-BALD.
LPDR is also expected to outperform MC-BatchBALD even when q > 1 : LPDR with q > 1 performs better than MC-BALD with q = 1 that MC-BatchBALD with q > 1 does not exceed (Kirsch et al., 2019).
J AUC OF HAM10000 DATASET
On imbalanced dataset the performance comparison is performed not only for accuracy but also for AUC.
Figure 15 shows the results of AUC with respect to the number of labeled samples on
HAM10000 dataset.
LPDR performs comparable with Entropy or ENS-VarR performing better than other algorithms.
K R ANK AND D OLAN -M ORE CURVES
[Table]
Rank curves and Dolan-More curves are used to compare the performance of the algorithms across all experimental settings and repetitions.
Figure 16 shows the rank and Dolan-More curves for all algorithms considered in the experiment.
The rank curve of each algorithm in the left-hand represents the mean of ranks on all datasets at each steps of active learning.
LPDR consistently is top-ranked for all steps.
The right-hand ﬁgure shows Dolan-More curves deﬁned as follows (Dolan & Mor´e, 2002).
Let acc pa be the ﬁnal test accuracy of the a algorithm on the p problem.
After deﬁning the performance gap as ∆ pa = max x (acc px ) − acc pa , we can deﬁne Dolan-More curve R a ( · ) as a function of the performance gap factor τ :
[Equation]
where n p is the total number of evaluations for the problem p .
Thus, R a ( τ ) is the ratio of problems with performance gap between algorithm a and the best performing competitor not more than τ .
Note that R a (0) is the ratio of problems on which algorithm a performs the best.
LPDR has the highest value R a (0)
= 43 .
3% , and LPDR maintains the highest R a ( τ ) for all τ .
Table 3 presents the mean and the standard deviation of performance gap from the best competitor for all steps of each algorithm on each dataset.
Consistent with all the results so far, LPDR signiﬁcantly outperforms the other algorithms in all experimental settings.