Original: L EAST D ISAGREE M ETRIC - BASED A CTIVE L EARNING
Revised: L EAST D ISAGREE M ETRIC - BASED A CTIVE L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The most popular class of active learners today queries for the labels of the samples for which the prediction is most uncertain and uses the labeled samples to update its prediction.
Revised: The most popular class of active learners today queries for the labels of the samples for which the prediction is most uncertain and uses the labeled samples to update its prediction.

Original: Unfortunately, quantifying uncertainty is an open question.
Revised: Unfortunately, quantifying uncertainty is an open question.

Original: This paper mathematically defines uncertainty in terms of the least disagree metric (LDM), which is the smallest perturbation required to alter the sample prediction.
Revised: This paper mathematically defines uncertainty in terms of the least disagree metric (LDM), which is the smallest perturbation required to alter the sample prediction.

Original: Based on this metric, the predictor is updated by querying the label of the most uncertain samples.
Revised: Based on this metric, the predictor is updated by querying the label of the most uncertain samples.

Original: Given a finite-sized training set, empirical LDM is incorporated into an active learning algorithm and used to approximate the theoretical LDM of each sample.
Revised: Given a finite-sized training set, empirical LDM is incorporated into an active learning algorithm and used to approximate the theoretical LDM of each sample.

Original: Theoretical convergence properties between the empirical and the mathematical definition of LDM are provided.
Revised: Theoretical convergence properties between the empirical and the mathematical definition of LDM are provided.

Original: Experimental results show that our algorithm mostly outperforms other high-performing active learning algorithms and leads to state-of-the-art performance on various datasets and deep networks.
Revised: Experimental results show that our algorithm mostly outperforms other high-performing active learning algorithms and leads to state-of-the-art performance on various datasets and deep networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Active learning (Cohn et al., 1996) is a sub-field in machine learning for attaining sample efficiency by sequentially selecting unlabeled samples for their labels.
Revised: Active learning (Cohn et al., 1996) is a sub-field in machine learning for attaining sample efficiency by sequentially selecting unlabeled samples for their labels.

Original: When selection is performed from a large collection of unlabeled samples, this type of active learning is coined pool-based active learning.
Revised: When selection is performed from a large collection of unlabeled samples, this type of active learning is coined pool-based active learning.

Original: Of the various active learning strategies, uncertainty-based sampling (Lewis & Gale, 1994; Sharma & Bilgic, 2017) for its simplicity and relatively low computational load is the most popular.
Revised: Of the various active learning strategies, uncertainty-based sampling (Lewis & Gale, 1994; Sharma & Bilgic, 2017; Nguyen et al., 2022) for its simplicity and relatively low computational load is the most popular.

Original: Here, the focus is on determining the uncertainty of each unlabeled sample for a given predictor.
Revised: Here, the focus is on determining the uncertainty of each unlabeled sample for a given predictor.

Original: However, there is no consensus on quantifying the uncertainty (Balcan et al., 2007; Settles, 2009; Houlsby et al., 2011; Yang et al., 2015; Sharma & Bilgic, 2017; Beluch et al., 2018; Ash et al., 2020).
Revised: However, there is no consensus on quantifying the uncertainty (Balcan et al., 2007; Settles, 2009; Houlsby et al., 2011; Yang et al., 2015; Sharma & Bilgic, 2017; Beluch et al., 2018; Ash et al., 2020).

Original: In this paper, we measure the uncertainty of a sample by perturbing the predictor to see how the predicted label is altered w.r.t.
Revised: In this paper, we measure the uncertainty of a sample by perturbing the predictor to see how the predicted label is altered w.r.t.

Original: that perturbation.
Revised: that perturbation.

Original: To the best of our knowledge, this approach has rarely been considered in active learning literature.
Revised: To the best of our knowledge, this approach has rarely been considered in active learning literature.

Original: In active learning, the most similar works to ours are the query-by-committee (QBC; Seung et al., 1992) and disagreement region-based approach (Hanneke, 2014), in which a committee of diverse experts is formed to identify the most disagreed samples in that the experts give differing predictions.
Revised: In active learning, the most similar works to ours are the query-by-committee (QBC; Seung et al., 1992) and disagreement region-based approach (Hanneke, 2014), in which a committee of diverse experts is formed to identify the most disagreed samples in that the experts give differing predictions.

Original: Despite its strong theoretical guarantee of achieving exponential sample complexity (Hanneke, 2014), the computational cost of creating numerous experts (e.g. by training neural networks several times) is very heavy and often in modern deep learning frameworks, infeasible.
Revised: Despite its strong theoretical guarantee of achieving exponential sample complexity (Hanneke, 2014), the computational cost of creating numerous experts (e.g. by training neural networks several times) is very heavy and often in modern deep learning frameworks, infeasible.

Original: As we will see later, our approach suffers from none of the aforementioned problems, largely because ours is based on a completely different framework.
Revised: As we will see later, our approach suffers from none of the aforementioned problems, largely because ours is based on a completely different framework.

Original: The main contributions of this paper are as follows:
Revised: The main contributions of this paper are as follows:

Original: This paper proposes the least disagree metric (LDM) as the measure of the sample’s uncertainty for the predictor, defined as the least probability of disagreement of the predicted label with a perturbed predictor.
Revised: This paper proposes the least disagree metric (LDM) as the measure of the sample’s uncertainty for the predictor, defined as the least probability of disagreement of the predicted label with a perturbed predictor.

Original: This paper then introduces a finite sample approximation to the true LDM, called empirical LDM , along with some theoretical guarantees.
Revised: This paper then introduces a finite sample approximation to the true LDM, called empirical LDM , along with some theoretical guarantees.

Original: We then provide a brute-force algorithm that evaluates the empirical LDM. 3.
Revised: We then provide a brute-force algorithm that evaluates the empirical LDM. 3.

Original: This paper proposes an active learning algorithm querying unlabeled samples based on (empirical) LDM.
Revised: This paper proposes an active learning algorithm querying unlabeled samples based on (empirical) LDM.

Original: Experiments show the algorithm leads to state-of-the-art performance on various datasets and deep networks.
Revised: Experiments show the algorithm leads to state-of-the-art performance on various datasets and deep networks.

Original:  L EAST D ISAGREE M ETRIC (LDM)
Revised:  L EAST D ISAGREE M ETRIC (LDM)

Original: This section mathematically defines the least disagree metric (LDM) of a sample for a given predictor in terms of the sample distribution.
Revised: This section mathematically defines the least disagree metric (LDM) of a sample for a given predictor in terms of the sample distribution.

Original: Henceforth, it will be assumed that the predictor belongs to a certain hypothesis space.
Revised: Henceforth, it will be assumed that the predictor belongs to a certain hypothesis space.

Original: For a given finite sample set, an empirical LDM is incorporated into a brute-force algorithm for estimating the theoretical LDM of each sample.
Revised: For a given finite sample set, an empirical LDM is incorporated into a brute-force algorithm for estimating the theoretical LDM of each sample.

Original: This paper will focus on the multi-class classification task.
Revised: This paper will focus on the multi-class classification task.

Original: 2. D EFINITION OF LDM
Revised: 2. D EFINITION OF LDM

Original: Let X
Revised: Let X

Original: and Y be the instance and label space with Y , and H be the hypothesis space of h : X → Y .
Revised: and Y be the instance and label space with Y , and H be the hypothesis space of h : X → Y .

Original: Let D be the joint distribution over X × Y , and D X be the instance distribution.
Revised: Let D be the joint distribution over X × Y , and D X be the instance distribution.

Original: Recall the disagree metric between two hypotheses (Hanneke, 2014) defined as
Revised: Recall the disagree metric between two hypotheses (Hanneke, 2014) defined as

Original: [Equation]
Revised: [Equation]

Original:  For a reference hypothesis ˆ h ∈ H and x 0 ∈ X , let H ˆ h, x 0 be the set of hypotheses disagreeing with ˆ h in their prediction of x 0 i.e.
Revised: where P X ∼D X is the probability measure on X , induced by D X . For a reference hypothesis ˆ h ∈ H and x 0 ∈ X , let H ˆ h, x 0 be the set of hypotheses disagreeing with ˆ h in their prediction of x 0 i.e.

Original: [Equation]
Revised: [Equation]

Original: Based on the above set and disagree metric, the uncertainty of a sample to a reference hypothesis is defined as follows:
Revised: Based on the above set and disagree metric, the uncertainty of a sample to a reference hypothesis is defined as follows:

Original: Definition 1.
Revised: Definition 1.

Original: For given ˆ h ∈ H and x 0 ∈ X , the least disagree metric (LDM) is defined as 
Revised: For given ˆ h ∈ H and x 0 ∈ X , the least disagree metric (LDM) is defined as [Equation]

Original: [Equation]
Revised: [Equation]

Original: Conceptually, a sample with a small LDM indicates that even a small perturbation in the predictor can alter sample prediction, and the opposite for a sample with a large LDM i.e. the sample with the smallest LDM is the most uncertain.
Revised: Conceptually, a sample with a small LDM indicates that even a small perturbation in the predictor can alter prediction and vice versa. Precisely, we claim that [Equation] where D H (ˆ h ) is a distribution over the H , induced by perturbing the ˆ h .

Original: This is the basic idea behind our active learning algorithm described in Section 3.
Revised: That is, the sample with the smallest LDM is the most uncertain. This intuition is verified in Appendix B.1.

Original: Here x is uniformly distributed on X ⊂ R 2 .
Revised: Here x is uniformly distributed on X ⊂ R 2 .

Original: The h 2 and h 3 disagree with ˆ h in prediction for x 0 , and L (ˆ h, x 0 )
Revised: The h 2 and h 3 disagree with ˆ h in prediction for x 0 , and L (ˆ h, x 0 )

Original: =
Revised: =

Original: inf h∈H ˆ h, x 0 ρ
Revised: inf h

Original: [Equation]
Revised: [Equation]

Original: h
Revised: ˆ h, x

Original: [Equation]
Revised: [Equation]

Original:  ˆ
Revised: h, ˆ

Original: [Equation] 
Revised: [Equation] [Equation]

Original: θ
Revised: |

Original: [Equation] 
Revised: [Equation] [Equation]

Original: To illustrate this concept more clearly, we provide a simple example.
Revised: To illustrate this concept more clearly, we provide a simple example.

Original: Consider a twodimensional binary classification with a set of linear classifiers,
Revised: Consider a twodimensional binary classification with a set of linear classifiers,

Original: [Equation]
Revised: [Equation]

Original: where x is uniformly distributed on X = { x : ∥ x ∥ ≤ 1 } ⊂ R 2 .
Revised: where x is uniformly distributed on X = { x : ∥ x ∥ ≤ 1 } ⊂ R 2 .

Original: The decision boundary of w is denoted as l w = { x : x T w = 0 } .
Revised: The decision boundary of w is denoted as l w = { x : x T w = 0 } .

Original: For given ˆ h and x 0 , let θ ∈ ( − π, π ) be the (radian) angle between l w and l ˆ w .
Revised: For given ˆ h and x 0 , let θ ∈ ( − π, π ) be the (radian) angle between l w and l ˆ w .

Original: Then we have that
Revised: Then we have that

Original: [Equation]
Revised: [Equation]

Original: since ρ ( h, ˆ h ) is the probability of X ∈ { x ∈
Revised: since ρ ( h, ˆ h ) is the probability of X ∈ { x ∈

Original: X | h ( x ) ̸
Revised: X | h ( x ) ̸

Original: = ˆ h ( x ) } and h ( x ) ̸
Revised: = ˆ h ( x ) } and h ( x ) ̸

Original: = ˆ h ( x ) for all x in the region between l w and l ˆ w .
Revised: = ˆ h ( x ) for all x in the region between l w and l ˆ w .

Original: This is shown in Figure 1; for instance,
Revised: This is shown in Figure 1; for instance,

Original: [Equation]
Revised: [Equation]

Original: and thus,
Revised: and thus,

Original: [Equation]
Revised: [Equation]

Original: 2. E MPIRICAL LDM
Revised: 2. E MPIRICAL LDM

Original: In most cases, LDM is not computable.
Revised: In most cases, LDM is not computable for the following two reasons: 1) ρ is generally intractable, especially when D X and the form of h are both complicated, e.g., neural networks over real-world image datasets, and 2) one needs to take an infimum over H , which is usually an infinite set.

Original: We can consider two approximations: 1) P in the definition of ρ is replaced by an empirical probability based on S samples, and 2) H is replaced by a finite hypothesis set H N of cardinality N .
Revised: We can consider two approximations: 1) P in the definition of ρ is replaced by an empirical probability based on S samples, and 2) H is replaced by a finite hypothesis set H N of cardinality N .

Original: More precisely, we consider
Revised: More precisely, we consider

Original: [Equation]
Revised: [Equation]

Original: where H ˆ h, x 0 N [Equation] | 
Revised: where H ˆ h, x 0 N := (cid:110) h ∈ H N | h ∈

Original: [Equation] ∈ H[Equation]
Revised: H

Original: [Equation] , I [ · ] is an indicator function, and X 1 , . . .
Revised: ˆ h, x 0 (cid:111) , I [ · ] is an indicator function, and X 1 , . . .

Original: , X S i.i.d.
Revised: , X S i.i.d.

Original: ∼ D X .
Revised: ∼ D X .

Original: Here, S is the number of instances (sampled i.i.d.) for approximating ρ , and N is the number of sampled hypotheses for approximating L .
Revised: Here, S is the number of instances (sampled i.i.d.) for approximating ρ , and N is the number of sampled hypotheses for approximating L .

Original: Assumption 1.
Revised: Assumption 1.

Original: Our hypothesis space H is a complete, separable metric space with metric d H ( · , · ) .
Revised: Our hypothesis space H is a complete, separable metric space with metric d H ( · , · ) .

Original: Assumption 2.
Revised: Assumption 2.

Original: ρ ( · , · ) is B -Lipschitz for some B > 

Revised: ρ ( · , · ) is B -Lipschitz for some B >

Original: i.e. ρ ( h, g ) ≤ Bd H ( h, g ) , ∀ h, g ∈ H .
Revised: i.e. ρ ( h, g ) ≤ Bd H ( h, g ) , ∀ h, g ∈ H .

Original: In the following theorem- its proof is deferred to Appendix A, we show that our proposed estimator is asymptotically consistent.
Revised: In the following theorem- its proof is deferred to Appendix A, we show that our proposed estimator is asymptotically consistent.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let ˆ h ∈ H and x 0 ∈ X be arbitrary.
Revised: Let ˆ h ∈ H and x 0 ∈ X be arbitrary.

Original: Suppose that H 1 ⊆ H 2 ⊆ · · · ⊂ H is arbitrary increasing sequence of sets satisfying the following: there exists ε > 0 s.t. 
Revised: Suppose that H 1 ⊆ H 2 ⊆ · · · ⊂ H is arbitrary increasing sequence of sets satisfying the following: there exists ε > 0 s.t. [Equation] ∗ ∈H

Original: [Equation]
Revised: [Equation]

Revised: h

Revised: [Equation]

Original:  Then, as 1 min( S, N ) → ∞ satisfying S = ω (log( CN )) , (cid:12)(cid:12)(cid:12) L N,S
Revised: [Equation] min [Equation] Then, as 1 min( S, N ) → ∞ satisfying S = ω (log( CN )) , (cid:12)(cid:12)(cid:12) L N,S

Original: (ˆ h, x 0 ) − L (ˆ h, x 0 ) (cid:12)(cid:12)(cid:12) ≤ ε in probability.
Revised: (ˆ h, x 0 ) − L (ˆ h, x 0 ) (cid:12)(cid:12)(cid:12) ≤ ε in probability.

Original: Corollary 2.
Revised: Corollary 2.

Original: If Eqn.
Revised: If Eqn.

Original: (3) holds for any ε > 0 , then under the same assumptions as Theorem 1,
Revised: (3) holds for any ε > 0 , then under the same assumptions as Theorem 1,

Original: L N,S (ˆ h, x 0 ) P
Revised: L N,S (ˆ h, x 0 ) P

Original: → L (ˆ h, x 0 ) .
Revised: → L (ˆ h, x 0 ) .

Original: One important consequence is that the ordering of the empirical LDM is preserved in probability:
Revised: One important consequence is that the ordering of the empirical LDM is preserved in probability:

Original: Corollary 3.
Revised: Corollary 3.

Original: Assume that L (ˆ h, x i ) <
Revised: Assume that L (ˆ h, x i ) <

Original: L (ˆ h, x j ) .
Revised: L (ˆ h, x j ) .

Original: Under the same assumptions as Corollary 2, we have that L N,S (ˆ h, x i ) <
Revised: Under the same assumptions as Corollary 2, we have that L N,S (ˆ h, x i ) <

Original: L N,S (ˆ h, x j ) in probability.
Revised: L N,S (ˆ h, x j ) in probability.

Original: 2. B RUTE - FORCE S EARCH FOR E MPIRICAL LDM
Revised: 2. B RUTE - FORCE S EARCH FOR E MPIRICAL LDM

Original: Assuming that the hypothesis set is parameterized i.e. h ∈ H is of the form h ( · ; w ) with w ∈ R p ,
Revised: Assuming that the hypothesis set is parameterized i.e. h ∈ H is of the form h ( · ; w ) with w ∈ R p ,

Original: Algorithm 1 describes the algorithm for evaluating empirical LDM of x for ˆ h .
Revised: Algorithm 1 describes the algorithm for evaluating empirical LDM of x for ˆ h .

Original: Prepare a set of variances { σ  k } Kk =1 such that σ k < σ k +1 and stop condition s .
Revised: Prepare a set of variances { σ 2 k } Kk =1 such that σ k < σ k +1 and stop condition s .

Original: Initialize LS (ˆ h, x) = 1 .
Revised: Initialize L x = 1 .

Original: For each k , h is sampled with w ∼ N ( ˆ w , I σ 2 k ) , and if h ( x ) ̸
Revised: For each k , h is sampled with w ∼ N ( ˆ w , I σ 2 k ) , and if h ( x ) ̸

Original: = ˆ h
Revised: = ˆ h

Original: ( x ) then update LS (ˆ h, x) as min {  ρ S ( h, ˆ h ), L S (ˆ h, x ) } .
Revised: ( x ) then update L x as min { L x , ρ S ( h, ˆ h ) } .

Original: When LS (ˆ h, x) does not change s times consecutively, move on to k +1 .
Revised: When L x does not change s times consecutively, move on to k + 1 .

Original: The final output LS (ˆ h, x) is the desired empirical LDM.
Revised: The final output L x is the desired empirical LDM.

Original: The reason for using several σ 2’s when sampling the weights is two-fold.
Revised: When sampling the weights, the reason for using several σ 2 is two-fold.

Original: We would want the sampled hypothesis h to satisfy h ( x ) ̸
Revised: We would want the sampled hypothesis h to satisfy h ( x ) ̸

Original: = ˆ h ( x ) .
Revised: = ˆ h ( x ) .

Original: Importantly, such probability (over the randomness of w and thus h ) is expected to be monotone increasing in σ 2 , as larger σ 2 indicates that the sampled hypothesis h is further away from ˆ h .
Revised: Importantly, such probability (over the randomness of w and thus h ) is expected to be monotone increasing in σ 2 , as larger σ 2 indicates that the sampled hypothesis h is further away from ˆ h .

Original: For the same reason, it is expected that the value E w [ ρ S ( h, ˆ h )] is monotone increasing in σ 2 .
Revised: For the same reason, it is expected that the value E w [ ρ S ( h, ˆ h )] is monotone increasing in σ 2 .

Original: In other words, with too large σ 2 , the minimum of ρ S ( h, ˆ h ) over the sampled h ’s may be too far away from the true LDM, especially when the true LDM is close to 0 .
Revised: In other words, with too large σ 2 , the minimum of ρ S ( h, ˆ h ) over the sampled h ’s may be too far away from the true LDM, especially when the true LDM is close to 0 .

Original: This intuition is verified in Appendix B.
Revised: This intuition is verified in Appendix B.2. [Figure]

Revised: Figure 2a shows the empirical LDM, computed by our Algorithm 1, of a sample in binary classification with the linear classifier as described in Figure 1.

Revised: The true LDM of the sample is 0 . 01 , and the empirical LDM is evaluated 100 times for S ∈ { 1000 , 10000 , 100000 } and s ∈ { 1 , 2 , 5 , 10 , 20 , 50 , 100 , 200 , 500 , 1000 } .

Revised: The empirical LDM is very close to the true LDM when s = 10 , and it reaches the true LDM when s ≥ 20 with the gap being roughly 10 − 4 .

Revised: This suggests that even with a moderate value of s , our approach can output empirical LDM that is sufficiently close to the true LDM.

Original: Input: { σ 2 k } Kk =1 : Set of Variances s : Stop condition for parameter sampling
Revised: Figure 2b shows the empirical LDMs of MNIST samples for a four-layered CNN. The empirical LDMs are evaluated 100 times at S = 60000 , which is the total number of samples of MNIST. We denote x i as the i th sample ordered by the computed empirical LDM.

Revised: Observe that the empirical

Revised: LDMs are monotonically decreasing as s increases, but even with s = 1000 , the empirical LDMs do not seem to converge.

Revised: Furthermore, when s = 1000 , as our algorithm samples 1M hypotheses, it takes roughly 20 min to evaluate the empirical LDM of a single sample, suggesting that it is computationally infeasible to obtain empirical LDM that is very close to the true LDM; this is further discussed in Appendix D.1.

Revised: Even though this is the case, one important observation is that the empirical LDM maintains the relative ranking between the samples.

Revised: Figure 2c shows the rank and Pearson’s correlations of the empirical LDMs to that at s = 1000 .

Revised: In MNIST, when s = 10 and s = 1000 , the rank and Pearson’s correlations reach 0 .

Revised: 999 and 0 .

Revised: 997 , respectively.

Revised: We set s =in Algorithm 1 in our experiments, as the runtime of our algorithm is of few seconds.

Revised: As we will discuss later, the ranking of the samples via their estimated empirical LDM is what is important.

Revised:  LDM- BASED A CTIVE L EARNING

Revised: This section introduces LDM-S, the LDM-based batch sampling algorithm for pool-based active learning.

Revised: This is the setting in which we have a set of unlabeled samples, U , and we simultaneously query q samples from randomly sampled pool data P ⊂ U of size m .

Revised: 3. LDM- BASED B ATCH S AMPLING

Revised: One LDM-based approach is to choose q samples with the lowest LDMs, which can be thought of as choosing the most uncertain samples.

Revised: However, as shown in Appendix D.2, this strategy often does not lead to good performance; upon further inspection, we observed that there’s a significant overlap of information in the selected batch.

Revised: This problem is prevalent in batch active learning, and one popular approach to mitigate this is to consider diversity to minimize redundancy among the selected samples (Kirsch et al., 2019; Ash et al., 2020; Citovsky et al., 2021;

Revised: Yang et al., 2021).

Original:  Function: LS (ˆ h, x) =for k = 1 to K do c
Revised: We incorporate diversity via a modification of the k -means++ seeding algorithm (Arthur & Vassilvitskii, 2007), which was reported to be the best algorithm to increase batch diversity without introducing additional hyperparameters (Ash et al., 2020). Intuitively, the k -means++ seeding selects centroids by iteratively sampling points proportional to their squared distance from the nearest Input: ˆ h, x : Given hypothesis and sample { σ 2 k } Kk =1 : Set of Variances s : Stop condition for parameter sampling Function: L x =for k = 1 to K do c

Original: =while c < s do w ∼ N ( ˆ w , I σ 2 k ) c
Revised: =while c < s do w ∼ N ( ˆ w , I σ 2 k ) c

Original: = c
Revised: = c

Original: +if h ( x ) ̸
Revised: +if h ( x ) ̸

Original: = ˆ h ( x ) and L  S (ˆ h, x ) >
Revised: = ˆ h ( x ) and L x > ρ S ( h, ˆ h ) then

Original: ρ S ( h, ˆ h ) then LS (ˆ h, x) ← ρ S ( h, ˆ h ) c
Revised: L x ← ρ S ( h, ˆ h ) c

Original: =end if end while end for return: LS (ˆ h, x)
Revised: =end if end while end for return: L x

Original: Input: L 0 , U 0 : Initial labeled and unlabeled samples m, q : pool and query size
Revised: Input: L 0 , U 0 : Initial labeled and unlabeled samples m, q : pool and query size

Original: Procedure:for t = 0 to T − 1 do
Revised: Procedure:for t = 0 to T − 1 do

Original: Obtain ˆ w by training on L t Randomly sample P ⊂ U t with |P| = m
Revised: Obtain ˆ w by training on L t Randomly sample P ⊂ U t with |P| = m

Original: Compute LS (ˆ h, x) for x ∈ P by Algorithm  Compute γ ( x ) using Eq.
Revised: Evaluate L x for x ∈ P by Algorithm 1 Compute γ ( x ) using Eq.

Original: Q 1 ← { x 1 } where x 1 = arg min x ∈P LS (ˆ h, x) for n = 2 to q do p ( x ) = γ ( x ) ∗ min x ′ ∈Q n − 1 d cos ( z x , z x ′ )
Revised: Q 1 ← { x 1 } where x 1 = arg min x ∈P L x for n = 2 to q do p ( x ) = γ ( x ) ∗ min x ′ ∈Q n − 1 d cos ( z x , z x ′ )

Original: Sample x n ∈ P w.p. P ( x n ) = p ( x n ) 2 (cid:80) x
Revised: Sample x n ∈ P w.p. P ( x n ) = p ( x n ) 2 (cid:80) x

Original: j ∈P p ( x j )
Revised: j ∈P p ( x j )

Original: Q n ← Q n − 1 ∪ { x n } end for
Revised: Q n ← Q n − 1 ∪ { x n } end for

Original: L t +1 ← L t ∪{ ( x i , y i ) } x i ∈Q q , U t +1 ← U t \Q q end for LDM- BASED A CTIVE L EARNING This section introduces LDM-S, the LDM-based batch sampling algorithm for pool-based active learning. This is the setting in which we have a set of unlabeled samples, U , and we simultaneously query q samples from randomly sampled pool data P ⊂ U of size m .
Revised: L t +1 ← L t ∪{ ( x i , y i ) } x i ∈Q q , U t +1 ← U t \Q q end for centroid that has already been chosen, which tends to select a diverse batch.

Original: 3. LDM- BASED B ATCH S AMPLING

Original: One LDM-based approach is to choose q samples with the lowest LDMs, which can be thought of as choosing the most uncertain samples.

Original: However, as shown in Section 4.1, this strategy often does not lead to good performance; upon further inspection, we observed that there’s a significant overlap of information in the selected batch.

Original: This problem is prevalent in batch active learning, and one popular approach to mitigate this is to consider diversity to minimize redundancy among the selected samples (Kirsch et al., 2019; Ash et al., 2020; Citovsky et al., 2021;

Original: Yang et al., 2021).

Original: We incorporate diversity via a modification of the k -means++ seeding algorithm (Arthur & Vassilvitskii, 2007), which was reported to be the best algorithm to increase batch diversity without introducing additional hyperparameters (Ash et al., 2020).

Original: Intuitively, the k -means++ seeding selects centroids by iteratively sampling points proportional to their squared distance from the nearest centroid that has already been chosen, which tends to select a diverse batch.

Original: Our proposed modification is to use the cosine distance between features of samples instead of the usual ℓ 2 -distance.
Revised: Our proposed modification is to use the cosine distance between features of samples instead of the usual ℓ 2 -distance.

Original: The reason for considering cosine distance is that for the final prediction, the scales of the features do not matter, and the reason for considering features is that the perturbation is applied to the weights of the last layer that takes the features as input.
Revised: The reason for considering cosine distance is that for the final prediction, the scales of the features do not matter, and the reason for considering features is that the perturbation is applied to the weights of the last layer that takes the features as input.

Original: Based on the above discussions, we introduce two seeding methods that take both LDM and batch diversity into account:
Revised: Based on the above discussions, we introduce two seeding methods that take both LDM and batch diversity into account:

Original: Seeding on the subset with small LDM.
Revised: Seeding on the subset with small LDM.

Original: Let P l ⊂ P be the set of top l samples with the smallest
Revised: Let P l ⊂ P be the set of top l samples with the smallest

Original: LDM, where l is about 2-5 times the query size q .
Revised: LDM, where l is about 2-5 times the query size q .

Original: The LDM-based seeding algorithm starts by selecting unlabeled samples with the smallest LDM in P l .
Revised: The LDM-based seeding algorithm starts by selecting unlabeled samples with the smallest LDM in P l .

Original: The next distinct unlabeled sample is sampled from P l by the following probability:
Revised: The next distinct unlabeled sample is sampled from P l by the following probability:

Original: [Equation]
Revised: [Equation]

Original: where Q is the set of selected unlabeled samples for querying, d cos ( · , · ) is the cosine distance, and z x , z x ′ are the feature vectors of x ∈ P l and x ′ ∈ Q , respectively.
Revised: where Q is the set of selected unlabeled samples for querying, d cos ( · , · ) is the cosine distance, and z x , z x ′ are the feature vectors of x ∈ P l and x ′ ∈ Q , respectively.

Original: Repeating until |Q| =
Revised: Repeating until |Q| =

Original: q completes the selection of unlabeled samples for querying.
Revised: q completes the selection of unlabeled samples for querying.

Original: However, this method requires setting an[Table] additional hyperparameter l , and in the chosen subset P l , the selection probability does not explicitly take LDM into account.
Revised: However, this method requires setting an additional hyperparameter l , and in the chosen subset P l , the selection probability does not explicitly take LDM into account.

Original: Thus the LDMs of the resulting query set are distributed evenly, which is undesirable as we expect for the samples with low LDMs to be chosen more often [Table]
Revised: Thus the LDMs of the resulting query set are distributed evenly, which is undesirable as we expect for the samples with low LDMs to be chosen more often.

Original: Seeding by LDM-weighted distribution.
Revised: Seeding by LDM-weighted distribution.

Original: Let P q ⊂ P be the set of top q samples with the smallest LDM and P c= P \P q .
Revised: Let P q ⊂ P be the set of top q samples with the smallest LDM and P c

Original: In the LDM-weighted seeding algorithm

Original: [Table]
Revised: = P \ P q .

Original:  we propose the following weighting scheme:  the weights of samples in P q are equal, 2) the weight of each sample in P c decreases exponentially as its LDM increases, 3) the sum of weights of samples in P q is equal to the sum of weights of samples in P c . 
Revised: In the LDM-weighted seeding algorithm, we propose the following weighting scheme: 1) the weights of samples in P q are equal, 2) the weight of each sample in P c decreases exponentially as its LDM increases, 3) the sum of weights of samples in P q is equal to the sum of weights of samples in P c . The first condition is to ensure that we choose samples with the q smallest LDM (equivalently, the most uncertain samples).

Revised: The second condition is for injecting diversity while taking the remaining LDMs into account as well.

Revised: The last condition is to balance the effects of the first and second conditions.

Original: Precisely, the weights of the samples in P are defined as follows: 
Revised: Precisely, the weights of x ∈ P are defined as follows: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ˜ γ ( x ) = exp (cid:16) − LS (ˆ h, x) − L q L q (cid:17) and L q = max x ∈P q LS (ˆ h, x) .
Revised: where ˜ γ ( x ) = exp (cid:16) − L x − L q L q (cid:17) and L q = max x ∈P q L x .

Original: Then, the Eq. 4 is replaced with the following:
Revised: Then, the Eq. 4 is replaced with the following:

Original: [Equation]
Revised: [Equation]

Original:  Note that this resolves the disadvantages of the first approach; there’s no hyperparameter, and the selection probability is explicitly impacted by LDMs while having diversity as well.
Revised: [Table] Note that this resolves the disadvantages of the first approach; there’s no hyperparameter [Table] and the selection probability is explicitly impacted by LDMs while having diversity as well.

Original: This difference is shown in Figure 3.
Revised: This difference is shown in Figure 3.

Original: 3. A LGORITHM FOR LDM- WEIGHTED S EEDING
Revised: 3. A LGORITHM FOR LDM- WEIGHTED S EEDING

Original: We now introduce LDM-S in Algorithm 2, the LDM-weighted seeding algorithm for active learning.
Revised: We now introduce LDM-S in Algorithm 2, the LDM-weighted seeding algorithm for active learning.

Original: Let L t
Revised: Let L t

Original: and U t be the set of labeled and unlabeled samples at iteration t respectively.
Revised: and U t be the set of labeled and unlabeled samples at iteration t .

Original: At each step t , the ˆ w is obtained by training on L t , and the set of pool samples P ⊂
Revised: At each step t , the ˆ w is obtained by training on L t [Table] and the set of pool samples P ⊂

Original: U t with |P| = m is drawn uniformly at random.
Revised: U t with |P| = m is drawn uniformly at random.

Original: Then for each x ∈ P ,its empirical LDM LS (ˆ h, x) andits sample weight γ ( x ) are evaluatedusing by Algorithm 1 and Eq. 5, respectively.
Revised: Then for each x ∈ P , L x and γ ( x ) are evaluated by Algorithm 1 and Eq. 5, respectively.

Original: The set of selected unlabeled samples, Q 1 , is initialized as { x 1 } where x 1 = arg min x ∈P LS (ˆ h, x) .
Revised: The set of selected unlabeled samples, Q 1 , is initialized as { x 1 } where x 1 = arg min x ∈P L x .

Original: For n = 2 , . . .
Revised: For n = 2 , . . .

Original: , q , the algorithm samples x n ∈ P with probability P ( x ) =
Revised: , q , the algorithm samples x n ∈ P with probability P ( x ) =

Original: p ( x ) 2 / (cid:80) x j ∈P p ( x j ) 2 where p ( x ) = γ ( x ) ∗ min x ′ ∈Q n − 1 d cos ( z x , z x ′ ) and appends it to Q n .
Revised: p ( x ) 2 / (cid:80) x j ∈P p ( x j ) 2 where p ( x ) = γ ( x ) ∗ min x ′ ∈Q n − 1 d cos ( z x , z x ′ ) and appends it to Q n .

Original: Lastly, the algorithm queries the label
Revised: Lastly, the algorithm queries the label

Original: y i of each x i ∈ Q q , and the algorithm continues until t =
Revised: y i of each x i ∈ Q q , and the algorithm continues until t =

Original: T − 1 .
Revised: T − 1 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section presents empirical results of the effect of diverse sampling on the LDM-based algorithm, as well as a comprehensive performance comparison with various uncertainty-based active learning algorithms.
Revised: This section presents empirical results of the effect of diverse sampling on the LDM-based algorithm, as well as a comprehensive performance comparison with various uncertainty-based active learning algorithms.

Original: Six image classification benchmark datasets are considered: MNIST (Lecun et al., 1998), CIFAR10 (Krizhevsky, 2009), SVHN (Netzer et al., 2011), CIFAR100 (Krizhevsky, 2009), Tiny ImageNet (Le & Yang, 2015), and FOOD101 (Bossard et al., 2014) datasets.
Revised: Six image classification benchmark datasets are considered: MNIST (Lecun et al., 1998), CIFAR10 (Krizhevsky, 2009), SVHN (Netzer et al., 2011), CIFAR100 (Krizhevsky, 2009), Tiny ImageNet (Le & Yang, 2015), and FOOD101 (Bossard et al., 2014) datasets.

Original: S-CNN,
Revised: S-CNN,

Original: K-CNN (Chollet et al., 2015) and Wide-ResNet (WRN-16-8; Zagoruyko & Komodakis, 2016) are used to evaluate the performance.
Revised: K-CNN (Chollet et al., 2015) and Wide-ResNet (WRN-16-8; Zagoruyko & Komodakis, 2016) are used to evaluate the performance.

Original: All results are averaged over 5 repetitions.
Revised: All results are averaged over 5 repetitions.

Original: The active learning settings regarding data, initial, and query sizes are summarized in Table 1.
Revised: The active learning settings regarding data, initial, and query sizes are summarized in Table 1.

Original: More details of datasets, networks, and training settings are presented in Appendix C.
Revised: More details of datasets, networks, and training settings are presented in Appendix C.

Original: [Figure]

Original: 4. N EED FOR D IVERSITY IN B ATCH A CTIVE L EARNING

Original: In the batch selection setting where samples are selected in the order of the smallest empirical LDM, there may be some overlap in the information of the selected samples’ features, thereby reducing the performance.

Original: In other words, querying unlabeled samples with the smallest LDMs may not lead to the best performance.

Original: To confirm this phenomenon, we query the k th batch of size q = 20 for k ∈ [50] from MNIST sorted in ascending order of LDM and compare the improvements in test accuracy.

Original: As shown in Figure 2a where 100 samples are labeled, the smallest LDM leads to the best performance, whereas in Figure 2d where 300 samples are labeled, the smallest LDM does not lead to the best performance.

Original: To see why this is the case, we’ve plotted the t-SNE plots (van der

Original: Maaten & Hinton, 2008) of the first and eighth batches for each case.

Original: In the first case, as shown in

Original: Figure 2b–2c, the samples of the first and eighth batches are all spread out, so there is no overlap of information between the samples in each batch; taking a closer look, it seems that smaller LDM leads to the samples being more spread out.

Original: However, in the second case, as shown in Figure 2e–2f, the samples of the first batch are close to one another i.e. there is a significant overlap of information between the samples in that batch.

Original: Surprisingly, this is not the case for the eighth batch, which consists of samples of larger LDMs.

Original: From this, we conclude that in the batch selection setting, even when using the LDM-based approach, diversity should be taken into consideration.

Original: 4. E FFECT OF D IVERSE S AMPLING ON LDM- BASED A LGORITHM
Revised: 4. E FFECT OF D IVERSE S AMPLING ON LDM- BASED A LGORITHM

Original: We first examine the effect of diverse sampling on our LDM-based active learning algorithm by considering various combinations of LDM and diverse sampling.
Revised: We first examine the effect of diverse sampling on our LDM-based active learning algorithm by considering various combinations of LDM and diverse sampling.

Original: Figure 3 shows the test accuracy with respect to the number of labeled samples and the histogram of samples’ indices selected from pool data sorted by their LDMs on MNIST, CIFAR10, and CIFAR100 datasets, with the sizes of the seeding subsets being 5 q, 5 q , and 2 q respectively.
Revised: Figure 3 shows the test accuracy with respect to the number of labeled samples and the histogram of samples’ indices selected from pool data sorted by their LDMs on MNIST, CIFAR10, and CIFAR100 datasets, with the sizes of the seeding subsets being 5 q, 5 q , and 2 q respectively.

Original: LDM-weighted seeding selects more samples with small LDM while pursuing batch diversity, which leads to significant performance improvement compared to when batch diversity is not considered.
Revised: LDM-weighted seeding selects more samples with small LDM while pursuing batch diversity, which leads to significant performance improvement compared to when batch diversity is not considered.

Original: However, seeding on a subset of samples with small LDMs shows no significant performance improvement when diversity is not considered.
Revised: However, seeding on a subset of samples with small LDMs shows no significant performance improvement when diversity is not considered.

Original: This is because, despite increasing diversity, only a few samples with small LDMs are chosen.
Revised: This is because, despite increasing diversity, only a few samples with small LDMs are chosen.

Original: As a sanity check, we also consider ‘seeding on pool data’, which is the modified k -means++ seeding on pool 
Revised: As a sanity check, we also consider ‘seeding on pool data’, which is the modified k -means++ seeding on pool data without considering LDM.

Revised: The result does indeed show that the success of our LDM-S cannot be solely attributed to the effect of diverse sampling.

Original: [Figure]
Revised: [Figure]

Original: data without considering LDM. The result does indeed show that the success of our LDM-S cannot be solely attributed to the effect of diverse sampling.
Revised: [Figure] Overall, LDM-S consistently performs best or is comparable with all other algorithms.

Original: 4. C OMPARING LDM-S TO O THER A LGORITHMS
Revised: 4. C OMPARING LDM-S TO O THER A LGORITHMS

Original: We now compare the performance of LDM-S with the baseline active learning algorithms, including state-of-the-art algorithms.
Revised: We now compare the performance of LDM-S with the baseline active learning algorithms, including state-of-the-art algorithms.

Original: Figure 4 shows the test accuracy with respect to the number of labeled samples on MNIST with S-CNN, CIFAR10, and SVHN with K-CNN, and CIFAR100, Tiny ImageNet, and FOOD101, with WRN-16-8 2 . 
Revised: Figure 4 shows the test accuracy with respect to the number of labeled samples on MNIST with S-CNN, CIFAR10, and SVHN with K-CNN, and CIFAR100, Tiny ImageNet, and FOOD101, with WRN-16-8 2 . [Table]

Original: Each algorithm is denoted as follows ‘Entropy’: entropy-based uncertainty sampling (Shannon, 1948), ‘Coreset’: core-set selection (Sener & Savarese, 2018), ‘MC-BALD’: MC-dropout sampling with BALD (Gal et al., 2017), ‘ENS-VarR’: ensemble method with variation ratio (Beluch et al., 2018), and ‘BADGE’: batch active learning by diverse gradient embeddings (Ash et al., 2020).
Revised: Each algorithm is denoted as follows ‘Entropy’: entropy-based uncertainty sampling (Shannon, 1948), ‘Coreset’: core-set selection (Sener & Savarese, 2018), ‘MC-BALD’: MC-dropout sampling with BALD (Gal et al., 2017), ‘ENS-VarR’: ensemble method with variation ratio (Beluch et al., 2018), and ‘BADGE’: batch active learning by diverse gradient embeddings (Ash et al., 2020).

Original: For ‘MC-BALD’, we use 100 forward passes, and for ‘ENS-VarR’, we use an ensemble consisting ofnetworks of identical architecture but different random initialization and random batches.
Revised: For ‘MC-BALD’, we use 100 forward passes, and for ‘ENS-VarR’, we use an ensemble consisting ofnetworks of identical architecture but different random initialization and random batches.

Original: Overall, LDM-S either consistently performs best or is at par with other algorithms for all datasets, while the performance of the algorithms except LDM-S varies depending on datasets.
Revised: Overall, LDM-S either consistently performs best or is at par with other algorithms for all datasets, while the performance of the algorithms except LDM-S varies depending on datasets.

Original: For instance, ‘Entropy’ and ‘Coreset’ show poor performance compared to other uncertainty-based algorithms, including ours, on MNIST, CIFAR10, and SVHN, while ‘Coreset’ performs at par with ours on
Revised: For instance, ‘Entropy’ and ‘Coreset’ show poor performance compared to other uncertainty-based algorithms, including ours, on MNIST, CIFAR10, and SVHN, while ‘Coreset’ performs at par with ours on

Original: FOOD101.
Revised: FOOD101.

Original: ‘ENS-VarR’, although comparable to other algorithms, still underperforms compared to LDM-S on all datasets.
Revised: ‘ENS-VarR’, although comparable to other algorithms, still underperforms compared to LDM-S on all datasets.

Original: A similar trend can  be observed for ‘MC-BALD’ and ‘BADGE’ as well.
Revised: A similar trend can also be observed for ‘MC-BALD’ and ‘BADGE’.

Revised: Furthermore, the runtime of LDM-S is comparable to Entropy, MC-BALD, Coreset, and BADGE; see Appendix E.1 for more details.

Revised: In Appendix D.3, the performance of LDM-S is compared with the standard uncertainty methods (Entropy, MC-BALD, ENS-VarR) to which weighted seeding is applied, with which these methods could be further benefitted.

Revised: Even in that case, LDM-S still outperforms the other methods, suggesting that the performance gains of LDM-S are attributed to our newly introduced LDM’s superiority over the other uncertainty measures, and are not mere artifacts of the seeding.

Original: All in all, Table 2 presents the mean and standard deviation of the averaged performance difference relative to Random overall steps for each repetition.
Revised: All in all, Table 2 presents the mean and standard deviation of the averaged performance difference relative to Random overall steps for each repetition.

Original: The positive or negative value indicates respectively higher or lower performance compared to Random, and the asterisk ( ∗ ) indicates the p-value is less than 0.05 in paired sample t-test for the null of no difference versus the alternative 
Revised: The positive or negative value indicates respectively higher or lower performance compared to Random, and the asterisk ( ∗ ) indicates the p-value is less than 0.05 in paired sample t-test for the null of no difference versus the alternative that the LDM-S is better than others.

Original: [Figure]
Revised: We observe that LDM-S either consistently performs best or is comparable with other algorithms for all datasets, while the performance of the algorithms except LDM-S varies depending on the datasets. In order to provide a comprehensive comparison across datasets, the performance profile, which is known as Dolan-Mor´e plot (Dolan & Mor´e, 2002), is examined. This curve has been widely considered in benchmarking active learning (Tsymbalov et al., 2018; 2019), optimization profiles (Dolan & Mor ´ e, 2002), and even general deep learning tasks (Burnaev et al., 2015a;b). To introduce the Dolan-Mor´e plot, let acc D,r,ta be the test accuracy of algorithm a at step t , for dataset D and repetition r , and ∆ D,r,ta = max a ′ (acc D,r,ta

Original: [Table]
Revised: [Equation]

Original: that the LDM-S is better than others.
Revised: ) − acc D,r,ta .

Original: We observe that LDM-S either consistently performs best or is comparable with other algorithms for all datasets, while the performance of the algorithms except LDM-S varies depending on the datasets.
Revised: Then, we define the performance profile as [Equation] where T D,r is the number of steps for dataset D at repetition r .

Revised: Intuitively, R a ( δ ) is the fraction of cases where the performance gap between algorithm a and the best competitor is less than δ .

Revised: Specifically, when δ = 0 , R a (0) is the fraction of cases on which algorithm a performs the best.

Revised: Figure 5 shows the performance profile w.r.t.

Revised: δ , for all algorithms.

Revised: Overall, it is clear that LDM-S retains the highest R a ( δ ) over all considered δ ’s. We also observe that R LDM-S (0)

Revised: =

Revised: 40% while the other algorithms have a value less than 20% .

Revised: All in all, this clearly shows that our LDM-S outperforms the other considered algorithms.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There are various active learning strategies such as uncertainty sampling (Lewis & Gale, 1994;
Revised: There are various active learning strategies such as uncertainty sampling (Lewis & Gale, 1994;

Original: Scheffer et al., 2001; Culotta & McCallum, 2005; Wang et al., 2010; Sharma & Bilgic, 2017), expected model change (Settles et al., 2007; Freytag et al., 2014; Ash et al., 2020), expected error reduction (Roy & McCallum, 2001;
Revised: Scheffer et al., 2001; Culotta & McCallum, 2005; Wang et al., 2010; Sharma & Bilgic, 2017), expected model change (Settles et al., 2007; Freytag et al., 2014; Ash et al., 2020), expected error reduction (Roy & McCallum, 2001;

Original: Yoo & Kweon, 2019; Zhao et al., 2021a), variance reduction (Schein & Ungar, 2007), uncertainty reduction (Zhao et al., 2021b), core-set approach (Sener & Savarese, 2018; Mahmood et al., 2022), clustering (Yang et al., 2021; Citovsky et al., 2021),
Revised: Yoo & Kweon, 2019; Zhao et al., 2021a), variance reduction (Schein & Ungar, 2007), uncertainty reduction (Zhao et al., 2021b), core-set approach (Sener & Savarese, 2018; Mahmood et al., 2022), clustering (Yang et al., 2021; Citovsky et al., 2021),

Original: Bayesian active learning (Pinsler et al., 2019; Shi & Yu, 2019), discriminative sampling (Sinha et al., 2019; Zhang et al., 2020;
Revised: Bayesian active learning (Pinsler et al., 2019; Shi & Yu, 2019), discriminative sampling (Sinha et al., 2019; Zhang et al., 2020;

Original: Gu et al., 2021; Caramalau et al., 2021), Fisher information (Ash et al., 2021), multi-armed bandit (Bouneffouf et al., 2014), bidirectional exploration (Zhang et al., 2015), and data augmentation (Kim et al., 2021)
Revised: Gu et al., 2021; Caramalau et al., 2021), Fisher information (Ash et al., 2021), multi-armed bandit (Bouneffouf et al., 2014), bidirectional exploration (Zhang et al., 2015), and data augmentation (Kim et al., 2021)

Original: For the uncertainty-based approach, various forms of uncertainty measures have been studied.
Revised: For the uncertainty-based approach, various forms of uncertainty measures have been studied.

Original: Entropy (Shannon, 1948) based uncertainty sampling strategies query unlabeled samples yielding the maximum entropy from the predictive distribution, but it does not perform well for multiclass classification tasks as the entropy is heavily influenced by probabilities of less important classes (Joshi et al., 2009).
Revised: Entropy (Shannon, 1948) based uncertainty sampling strategies query unlabeled samples yielding the maximum entropy from the predictive distribution, but it does not perform well for multiclass classification tasks as the entropy is heavily influenced by probabilities of less important classes (Joshi et al., 2009).

Original: Margin based strategies query unlabeled samples closest to the decision boundary, and it is generally understood that unlabeled sample closest to the decision boundary is the most uncertain (Balcan et al., 2007; Kremer et al., 2014; Ducoffe & Precioso, 2018).
Revised: Margin based strategies query unlabeled samples closest to the decision boundary, and it is generally understood that unlabeled sample closest to the decision boundary is the most uncertain (Balcan et al., 2007; Kremer et al., 2014; Ducoffe & Precioso, 2018).

Original: However, it is difficult to identify samples closest to the decision boundary for multiclass classification with the deep network as the Euclidean distance is often not readily measurable (Ducoffe & Precioso, 2018;
Revised: However, it is difficult to identify samples closest to the decision boundary for multiclass classification with the deep network as the Euclidean distance is often not readily measurable (Ducoffe & Precioso, 2018;

Original: Mickisch et al., 2020).
Revised: Mickisch et al., 2020).

Original: Mutual information based strategies, such as BALD (Houlsby et al., 2011),
Revised: Mutual information based strategies, such as BALD (Houlsby et al., 2011),

Original: DBAL (Gal et al., 2017), and BatchBALD (Kirsch et al., 2019), query unlabeled samples yielding the maximum mutual information between predictions and model parameters.
Revised: DBAL (Gal et al., 2017), and BatchBALD (Kirsch et al., 2019), query unlabeled samples yielding the maximum mutual information between predictions and model parameters.

Original: The DBAL approximates the posterior of the model parameters of the deep network by MC-dropout sampling, but each batch selection is independently conducted, and this leads to data inefficiency as correlations between data points in the batch are not taken into account (Kirsch et al., 2019).
Revised: The DBAL approximates the posterior of the model parameters of the deep network by MC-dropout sampling, but each batch selection is independently conducted, and this leads to data inefficiency as correlations between data points in the batch are not taken into account (Kirsch et al., 2019).

Original: To address this deficiency, BatchBALD was introduced, but because BatchBALD theoretically computes all possible mutual information between batch-wise predictions and model parameters.
Revised: To address this deficiency, BatchBALD was introduced, but because BatchBALD theoretically computes all possible mutual information between batch-wise predictions and model parameters.

Original: For this reason, it is not appropriate for large query sizes.
Revised: For this reason, it is not appropriate for large query sizes.

Original: Variation ratio (Freeman, 1965) with ensemble method (Beluch et al., 2018) based on query by committee (QBC) strategy (Seung et al., 1992) queries unlabeled samples yielding the maximum variation ratio in labels predicted by the multiple networks, but it requires high computational load: each network belonging to the ensemble must be individually trained.
Revised: Variation ratio (Freeman, 1965) with ensemble method (Beluch et al., 2018) based on query by committee (QBC) strategy (Seung et al., 1992) queries unlabeled samples yielding the maximum variation ratio in labels predicted by the multiple networks, but it requires high computational load: each network belonging to the ensemble must be individually trained.

Original: Gradient based strategy (Ash et al., 2020) measures uncertainty as the gradient magnitude with respect to parameters in the final layer and queries unlabeled samples where these gradients span a diverse set of directions, but it requires a high computational load when the dimension of parameters is large.
Revised: Gradient based strategy (Ash et al., 2020) measures uncertainty as the gradient magnitude with respect to parameters in the final layer and queries unlabeled samples where these gradients span a diverse set of directions, but it requires a high computational load when the dimension of parameters is large.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper defines the least disagree metric (LDM), which measures the uncertainty of samples by perturbing the predictor, and introduces a hypothesis sampling method for evaluating approximated
Revised: This paper defines the least disagree metric (LDM), which measures the uncertainty of samples by perturbing the predictor, and introduces a hypothesis sampling method for evaluating approximated

Original: LDM (empirical LDM).
Revised: LDM (empirical LDM).

Original: In addition, this paper proposes an LDM-based active learning algorithm to select unlabeled samples with small LDM while pursuing batch diversity.
Revised: In addition, this paper proposes an LDM-based active learning algorithm to select unlabeled samples with small LDM while pursuing batch diversity.

Original: The proposed algorithm either consistently performs best or is comparable with other high-performing active learning algorithms, leading to state-of-the-art active learning performance on most of the datasets considered in this paper.
Revised: The proposed algorithm either consistently performs best or is comparable with other high-performing active learning algorithms, leading to state-of-the-art active learning performance on most of the datasets considered in this paper.

Original: Our algorithm is simple and relatively light in terms of computational cost, and thus we expect it to be a meaningful addition to the field of uncertainty-based active learning.
Revised: Our algorithm is simple and relatively light in terms of computational cost, and thus we expect it to be a meaningful addition to the field of uncertainty-based active learning

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: We consider multi-class classification, which we recall here from Section 2.
Revised: We consider multi-class classification, which we recall here from Section 2.

Original: Let X
Revised: Let X

Original: and Y be the instance and label space with Y =
Revised: and Y be the instance and label space with Y =

Original: { e i } Ci =1 , where e i is the
Revised: { e i } Ci =1 , where e i is the

Original: i th standard basis vector of R C (i.e. one-hot encoding of the label i ), and H be the hypothesis space of h : X → Y .
Revised: i th standard basis vector of R C (i.e. one-hot encoding of the label i ), and H be the hypothesis space of h : X → Y .

Original: We also recall some definitions: H ˆ h, x 0 = { h ∈ H | h ( x 0 )
Revised: We also recall some definitions: H ˆ h, x 0 = { h ∈ H | h ( x 0 )

Original: ̸
Revised: ̸

Original: = ˆ h ( x 0 ) } and H ˆ h, x 0 N :=
Revised: = ˆ h ( x 0 ) } and H ˆ h, x 0 N :=

Original: (cid:110) h ∈ H N |
Revised: (cid:110) h ∈ H N |

Original: h ∈ H
Revised: h ∈ H

Original: ˆ h, x 0 (cid:111) , where for computation purpose H is replaced with H N of cardinality N .
Revised: ˆ h, x 0 (cid:111) , where for computation purpose H is replaced with H N of cardinality N .

Original: Lastly, we use S samples
Revised: Lastly, we use S samples

Original: X i ∼ D X to construct the Monte-Carlo estimate of ρ ( h, ˆ h ) .
Revised: X i ∼ D X to construct the Monte-Carlo estimate of ρ ( h, ˆ h ) .

Original: Let S ( N ) → ∞ be an arbitrary monotone increasing sequence in N such that S ( N )
Revised: Let S ( N ) → ∞ be an arbitrary monotone increasing sequence in N such that S ( N )

Original: = ω (log( CN )) .
Revised: = ω (log( CN )) .

Original: By the triangle inequality, we have that
Revised: By the triangle inequality, we have that

Original: [Equation]
Revised: [Equation]

Original: where we denote (cid:101) L N := min h ∈H ˆ h, x 0 N ρ ( h, ˆ h ) .
Revised: where we denote (cid:101) L N := min h ∈H ˆ h, x 0 N ρ ( h, ˆ h ) .

Original: ∆ 1 ( N ) P →
Revised: ∆ 1 ( N ) P →

Original: By definition,
Revised: By definition,

Original: [Equation]
Revised: [Equation]

Original: As ∆ 1 ( N ) is a difference of infimums of a sequence of functions, over a sequence of sets, we need to establish some uniform convergence-type result.
Revised: As ∆ 1 ( N ) is a difference of infimums of a sequence of functions, over a sequence of sets, we need to establish some uniform convergence-type result.

Original: This is done by invoking “general” GlivenkoCantelli Theorem, which we recall here:
Revised: This is done by invoking “general” GlivenkoCantelli Theorem, which we recall here:

Original: Theorem 4 (Theorem 4.2 of Wainwright (2019)) .
Revised: Theorem 4 (Theorem 4.2 of Wainwright (2019)) .

Original: Let Y 1 , · · · , Y n i.i.d.
Revised: Let Y 1 , · · · , Y n i.i.d.

Original: ∼ P for some distribution P over X .
Revised: ∼ P for some distribution P over X .

Original: For any b -uniformly bounded function class F , any positive integer n ≥ 1 and any scalar δ ≥ 0 , we have
Revised: For any b -uniformly bounded function class F , any positive integer n ≥ 1 and any scalar δ ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: with P -probability at least 1 − exp (cid:16) − nδ 2 8 b (cid:17) .
Revised: with P -probability at least 1 − exp (cid:16) − nδ 2 8 b (cid:17) .

Original: Here, R n ( F ) is the Rademacher complexity of F .
Revised: Here, R n ( F ) is the Rademacher complexity of F .

Original: In our case, n = S ( N ) , P = D X , b = 1 , and
Revised: In our case, n = S ( N ) , P = D X , b = 1 , and

Original: [Equation]
Revised: [Equation]

Original: Here, the usual Rademacher identities (e.g. see Section 4.2 of Wainwright (2019)) does not apply, as we are considering the composition of vector-valued functions and vector-input functions.
Revised: Here, the usual Rademacher identities (e.g. see Section 4.2 of Wainwright (2019)) does not apply, as we are considering the composition of vector-valued functions and vector-input functions.

Original: But for our setting, we provide a simple yet effective upper bound on the empirical Rademacher complexity for multi-class:
Revised: But for our setting, we provide a simple yet effective upper bound on the empirical Rademacher complexity for multi-class:

Original: Lemma 5.
Revised: Lemma 5.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For simplicity, we denote E ≜ E { X i } S ( N ) i =1 , σ , where the expectation is w.r.t.
Revised: For simplicity, we denote E ≜ E { X i } S ( N ) i =1 , σ , where the expectation is w.r.t.

Original: X i ∼ D X i.i.d., and σ is the S ( N ) -dimensional
Revised: X i ∼ D X i.i.d., and σ is the S ( N ) -dimensional

Original: Rademacher variable.
Revised: Rademacher variable.

Original: Also, let l : [ S ( N )]
Revised: Also, let l : [ S ( N )]

Original: → [ C ] be the labeling function for fixed samples { X i } S ( N ) i
Revised: → [ C ] be the labeling function for fixed samples { X i } S ( N ) i

Original: =1 i.e. l ( i ) =
Revised: =1 i.e. l ( i ) =

Original: arg max c ∈ [ C ] (cid:104)
Revised: arg max c ∈ [ C ] (cid:104)

Original: ˆ h
Revised: ˆ h

Original: ( X i ) (cid:105) c .
Revised: ( X i ) (cid:105) c .

Original: As ˆ h outputs one-hot encoding for each i , l ( i ) is unique and thus well-defined.
Revised: As ˆ h outputs one-hot encoding for each i , l ( i ) is unique and thus well-defined.

Original: Thus, we have that f ( X i ) =
Revised: Thus, we have that f ( X i ) =

Original: I [ h ( X i ) ̸
Revised: I [ h ( X i ) ̸

Original: = ˆ h
Revised: = ˆ h

Original: ( X i )]
Revised: ( X i )]

Original: = 1 − h l ( i ) ( X i ) , where we denote h
Revised: = 1 − h l ( i ) ( X i ) , where we denote h

Original: = ( h 1 , h 2 , · · · , h C ) with h j : X → { 0 , 1 } .
Revised: = ( h 1 , h 2 , · · · , h C ) with h j : X → { 0 , 1 } .

Original: By definition,
Revised: By definition,

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from Massart’s Lemma (Massart, 2000) and the fact that H ˆ h, x 0 N is a finite set of cardinality at most N .
Revised: where the last inequality follows from Massart’s Lemma (Massart, 2000) and the fact that H ˆ h, x 0 N is a finite set of cardinality at most N .

Original: Choosing δ = (cid:113) 8log( CN ) S ( N ) , we have that with probability at least 1 − 1 CN , sup
Revised: Choosing δ = (cid:113) 8log( CN ) S ( N ) , we have that with probability at least 1 − 1 CN , sup

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12) ρ S ( N ) ( h
Revised: (cid:12)(cid:12)(cid:12) ρ S ( N ) ( h

Original: [Equation]
Revised: [Equation]

Original: The following lemma completes the proof of the first part:
Revised: The following lemma completes the proof of the first part:

Original: Lemma 6.
Revised: Lemma 6.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let δ > 0 be arbitrary, and choose any N such that N > 1 Cδ and 4 (cid:113) 2log( CN ) S ( N ) < δ .
Revised: Let δ > 0 be arbitrary, and choose any N such that N > 1 Cδ and 4 (cid:113) 2log( CN ) S ( N ) < δ .

Original: From Eqn.
Revised: From Eqn.

Original: (7) we have: with probability at least 1 − δ , sup h ∈H ˆ h, x 0 N
Revised: (8) we have: with probability at least 1 − δ , sup h ∈H ˆ h, x 0 N

Original: [Equation]
Revised: [Equation]

Original: First, we choose a sequence { h j } ⊂ H ˆ h, x 0 N such that ρ ( h j , ˆ h ) → inf h ∈H N ρ ( h, ˆ h ) for j → ∞ .
Revised: First, we choose a sequence { h j } ⊂ H ˆ h, x 0 N such that ρ ( h j , ˆ h ) → inf h ∈H N ρ ( h, ˆ h ) for j → ∞ .

Original: Then, inf
Revised: Then, inf

Original: [Equation]
Revised: [Equation]

Original: Similarly, by choosing a sequence { g j } ⊂ H ˆ h, x 0 N such that ρ S ( N ) ( g j , ˆ h ) → inf h ∈H N ρ S ( N ) ( h, ˆ h ) , we have that inf
Revised: Similarly, by choosing a sequence { g j } ⊂ H ˆ h, x 0 N such that ρ S ( N ) ( g j , ˆ h ) → inf h ∈H N ρ S ( N ) ( h, ˆ h ) , we have that inf

Original: [Equation]
Revised: [Equation]

Original: N ρ
Revised: N ρ

Original: ( h, ˆ h ) − δ.
Revised: ( h, ˆ h ) − δ.

Original: Combining them, we have that for any δ > 0 , there exists some M ( δ ) such that for any N > M ( δ ) ,
Revised: Combining them, we have that for any δ > 0 , there exists some M ( δ ) such that for any N > M ( δ ) ,

Original: [Equation]
Revised: [Equation]

Original: Denoting G N ≜ inf h
Revised: Denoting G N ≜ inf h

Original: [Equation]
Revised: [Equation]

Original:  , above is equivalent to
Revised: ) , above is equivalent to

Original: [Equation]
Revised: [Equation]

Original:  where d KF ( X, Y )
Revised: G [Equation] [Equation] ≤ δ, where d KF ( X, Y )

Original: =
Revised: =

Original: inf { δ ≥ 0 | P [ | X − Y | ≥ δ ] ≤ δ } is the Ky-Fan metric , which induces a metric structure on the given probability space with the convergence in probability (see Section 9. of Dudley (2002)).
Revised: inf { δ ≥ 0 | P [ | X − Y | ≥ δ ] ≤ δ } is the Ky-Fan metric , which induces a metric structure on the given probability space with the convergence in probability (see Section 9. of Dudley (2002)).

Original: In other words, as d KF ( G N , 0) → 0 , we conclude that G N P → 0 .
Revised: In other words, as d KF ( G N , 0) → 0 , we conclude that G N P → 0 .

Original: [Equation]
Revised: [Equation]

Original: Let γ > 0 be arbitrary, and denote
Revised: Let γ > 0 be arbitrary, and denote

Original: [Equation]
Revised: [Equation]

Original: In this section, any convergence is w.r.t.
Revised: In this section, any convergence is w.r.t.

Original: the limit N → ∞ with ε fixed.
Revised: the limit N → ∞ with ε fixed.

Original: It’s easy to see that Z εN is monotone decreasing i.e. Z ε 1 ≥ Z ε 2 ≥ · · · .
Revised: It’s easy to see that Z εN is monotone decreasing i.e. Z ε 1 ≥ Z ε 2 ≥ · · · .

Original: Defining the events E N :=
Revised: Defining the events E N :=

Original: { Z εN < γ/B } , we have that
Revised: { Z εN < γ/B } , we have that

Original: E 1 ⊆
Revised: E 1 ⊆

Original: E 2 ⊆ · · · .
Revised: E 2 ⊆ · · · .

Original: From our assumption and the fact that probability measure is continuous along any monotone sequence of events, we have that
Revised: From our assumption and the fact that probability measure is continuous along any monotone sequence of events, we have that

Original: [Equation]
Revised: [Equation]

Original: This implies that w.p. 1 the following holds: for any ζ > 0 there exists N 0 s.t.
Revised: This implies that w.p. 1 the following holds: for any ζ > 0 there exists N 0 s.t.

Original: for all N ≥ N 0 , there exists h ∈ H N and h ∗
Revised: for all N ≥ N 0 , there exists h ∈ H N and h ∗

Original: ∈ H
Revised: ∈ H

Original: ˆ h, x 0 with ρ ( h ∗ , ˆ h ) − L
Revised: ˆ h, x 0 with ρ ( h ∗ , ˆ h ) − L

Original: (ˆ h, x 0 ) <
Revised: (ˆ h, x 0 ) <

Original: [Equation]

Original:  s.t.
Revised: ε s.t.

Original: [Equation]

Original:  ( h
Revised: d H ( h

Original: [Equation]
Revised: [Equation]

Original: , h ) ≤
Revised: , h ) ≤

Original: [Equation]
Revised: [Equation]

Original: + ζB .
Revised: + ζB .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: = [Equation] As γ and ζ were arbitrary, we conclude that w.p. 1 , lim N →∞ ∆ 2 ( N ) ≤ ε , and we are done.
Revised: As γ and ζ were arbitrary, we conclude that w.p. 1 , lim N →∞ ∆ 2 ( N ) ≤ ε , and we are done.

Original: B  V ERIFICATION OFTHE I NTUITION BEHIND σ 2 IN A LGORITHM
Revised: B T HEORETICAL V ERIFICATIONS OF I NTUITIONS

Original: Recall from Section 2.2 that the intuition behind using multiple σ 2 was that it controls the trade-off between the probability of obtaining a hypothesis with a different prediction than that of ˆ h and the scale of ρ S ( h, ˆ h ) .

Original: First, we theoretically prove this intuition for two-dimensional binary classification with a set of linear classifiers,H
Revised: Here, we consider two-dimensional binary classification with a set of linear classifiers,

Original:  =
Revised: H =

Original: { h  : h  ( x ) = sgn
Revised: { h w : h w ( x ) = sgn

Original: ( x T w ) , w ∈ R 2 } .
Revised: ( x T w ) , w ∈ R 2 } .

Original: For simplicity, we assume that x is uniformly distributed on X
Revised: We assume that x is uniformly distributed on X

Original: = { x : ∥ x ∥ ≤ 1 } ⊂ R 2 .
Revised: = { x : ∥ x ∥ ≤ 1 } ⊂ R 2 .

Revised: As we parametrize H using w , following our Algorithm 1, we set D H (ˆ h ) =

Revised: N ( ˆ w , I σ 2 ) , where ˆ h = h ˆ w .

Revised: For simplicity, we omit the dependency on w and ˆ w .

Original: Lemma 7.
Revised: Recall from Section 2.1 that the intuition behind a sample with a small LDM indicates that even a small perturbation in the predictor can alter sample prediction. We theoretically prove this intuition Proposition 7.

Original: Suppose that h  is sampled with w ∼ N ( ˆ w , I σ 2 ) where w , ˆw ∈ W are parameters of h, ˆ h respectively, then E [ ρ (h, ˆ h )] is continuous and strictly increasing with σ .
Revised: Suppose that h = h w is sampled with w ∼ N ( ˆ w , I σ 2 ) . Then, L ( ˆ h, x 1 ) < L ( ˆ h, x 2 )

Revised: ⇐⇒ P

Revised: [ h ( x 1 ) ̸

Revised: = ˆ h ( x 1 )] > P [ h ( x 2 ) ̸

Revised: = ˆ h ( x 2 )] .

Revised: [Figure]

Revised: Proof of Proposition 7.

Revised: One important observation is that by the duality between w and x (Tong & Chang, 2001), in R 2 , w is a point and x is represented by the hyperplane, l x = { w ∈ R 2 : sgn( x T w ) = 0 } .

Revised: Suppose that h is sampled with w ∼ N ( ˆ w , I σ 2 ) , and let ˆ θ be the angle of ˆ w , θ x be the angle between l x and positive x-axis, and W ˆ w , x be the half-plane divided by l x which does not contain ˆ w :

Revised: [Equation]

Revised: as in Figure 6a.

Revised: Then, L (ˆ h, x ) = | θ x − ˆ θ | /π

Revised: and P [ h ( x ) ̸

Revised: = ˆ h ( x )]

Revised: = P [ w ∈ W ˆ w , x ] .

Revised: Let d 1 , d 2 be the distances between ˆ w and l x 1 , l x 2 respectively, and

Revised: [Equation]

Revised: as in Figure 6b.

Revised: Suppose that d 1 < d 2 , then | θ x 1 − ˆ θ | < | θ x 2 − ˆ θ | since d i = ∥

Revised: ˆ w ∥ sin | θ x i − ˆ θ | , and

Revised: [Equation]

Revised: by the followings:

Revised: [Equation]

Revised: where W 1 , W 2 , and W ˆ w , x 1 ∩ W ˆ w , x 2 are disjoint.

Revised: Note that W 1 and W 2 are one-to-one mapped by the symmetry at origin, but the probabilities are different by the biased location of ˆ w, i.e, ϕ ( w 1 | ˆ w , σ 2 ) >

Revised: ϕ ( w 2 | ˆ w , σ 2 ) for all pairs of ( w 1 , w 2 ) ∈

Revised: W 1 × W 2 that are symmetric at the origin.

Revised: Here ϕ ( ·| ˆ w , σ 2 ) is the probability density function of the bivariate normal distribution with mean ˆ w and covariance I σ 2 .

Revised: Thus,

Revised: [Equation]

Revised: B. V ARYING σ 2 IN A LGORITHM

Revised: Recall from Section 2.2 that the intuition behind using multiple σ 2 was that it controls the trade-off between the probability of obtaining a hypothesis with a different prediction than that of ˆ h and the scale of ρ S ( h, ˆ h ) .

Revised: Theoretically, we show the following for two-dimensional binary classification with linear classifiers:

Revised: Proposition 8.

Revised: Suppose that h is sampled with w ∼ N ( ˆ w , I σ 2 ) where w , ˆ w ∈ W are parameters of h, ˆ h respectively, then E [ ρ ( h, ˆ h )] is continuous and strictly increasing with σ .

Original: We now empirically show that this intuition holds for general deep learning architectures.
Revised: We now empirically show that this intuition holds for general deep learning architectures.

Original: Figureshows the relationship between E [ ρ S ( h, ˆ h )] and log σ for MNIST, CIFAR10, SVNH, CIFAR100,
Revised: Figureshows the relationship between E [ ρ S ( h, ˆ h )] and log σ for MNIST, CIFAR10, SVNH, CIFAR100,

Original: Tiny ImageNet, and FOOD101 datasets where h is sampled with w ∼ N ( ˆ w , I σ 2 ) .
Revised: Tiny ImageNet, and FOOD101 datasets where h is sampled with w ∼ N ( ˆ w , I σ 2 ) .

Original: The E [ ρ S ( h, ˆ h )] is monotonically increasing with log σ in all experimental settings.
Revised: The E [ ρ S ( h, ˆ h )] is monotonically increasing with log σ in all experimental settings.

Original: [Figure]
Revised: [Figure]

Original: with perturbation strength in B. P ROOF OF L EMMA
Revised: Proof of Proposition 8.

Original: One important observation is that by the duality between w and x (Tong & Chang, 2001), in W , w is a point and x is represented by the hyperplane, l x = { w ∈ W : sgn( x T w ) = 0 } .
Revised: By the duality between w and x , in W , w is a point and x is represented by the hyperplane, l x = { w ∈ W : sgn( x T w ) = 0 } .

Original: Let h be a sampled hypothesis with w ∼ N ( ˆ w , I σ 2 ) , ˆ θ be the angle of ˆ w =
Revised: Let h be a sampled hypothesis with w ∼ N ( ˆ w , I σ 2 ) , ˆ θ be the angle of ˆ w =

Original: ( ˆ w 1 , ˆ w 2 ) T , i.e., tan ˆ θ = ˆ w 2 / ˆ w 1 , θ be the angle of w = ( w 1 , w 2 ) T , i.e., tan θ =
Revised: ( ˆ w 1 , ˆ w 2 ) T , i.e., tan ˆ θ = ˆ w 2 / ˆ w 1 , θ be the angle of w = ( w 1 , w 2 ) T , i.e., tan θ =

Original: w 2 /w 1 , and θ x be the angle between l x and positive x-axis.
Revised: w 2 /w 1 , and θ x be the angle between l x and positive x-axis.

Original: Here, θ, θ x ∈ [ − π
Revised: Here, θ, θ x ∈ [ − π

Original: + ˆ θ, π + ˆ θ ] in convenience.
Revised: + ˆ θ, π + ˆ θ ] in convenience.

Original: When θ x or π + θ x is between θ and ˆ θ , h ( x ) ̸
Revised: When θ x or π + θ x is between θ and ˆ θ , h ( x ) ̸

Original: = ˆ h ( x ) , otherwise h ( x ) =
Revised: = ˆ h ( x ) , otherwise h ( x ) =

Original: ˆ h ( x ) .
Revised: ˆ h ( x ) .

Original: Thus, ρ ( h, ˆ h ) =
Revised: Thus, ρ ( h, ˆ h ) =

Original: | θ − ˆ θ | /π .
Revised: | θ − ˆ θ | /π .

Original: Using Box-Muller transform (Box & Muller, 1958), w can be generated by
Revised: Using Box-Muller transform (Box & Muller, 1958), w can be generated by

Original: [Equation]
Revised: [Equation]

Original: where u and v are independent uniform random variables on [0 , 1] .
Revised: where u and v are independent uniform random variables on [0 , 1] .

Original: Then, ∥ w − ˆ w ∥ = σ √− 2 log u and ( w 2 − ˆ w 2 ) / ( w 1 − ˆ w 1 ) = tan(2 πv ) , i.e., the angle of w − ˆ w is 2 πv .
Revised: Then, ∥ w − ˆ w ∥ = σ √− 2 log u and ( w 2 − ˆ w 2 ) / ( w 1 − ˆ w 1 ) = tan(2 πv ) , i.e., the angle of w − ˆ w is 2 πv .

Original: Here, 
Revised: Here, ∥ ˆ w

Original: [Equation]
Revised: [Equation]

Original: by using the perpendicular line from ˆ w to the line passing through the origin and w (see the Figurefor its geometry), and Eq. 8 is satisfied for all θ .
Revised: by using the perpendicular line from ˆ w to the line passing through the origin and w (see the Figurefor its geometry), and Eq. 9 is satisfied for all θ .

Original: For given u and v , θ is continuous and the derivative of θ with respect to σ is
Revised: For given u and v , θ is continuous and the derivative of θ with respect to σ is

Original: [Equation]
Revised: [Equation]

Original: thus
Revised: thus

Original: [Equation]
Revised: [Equation]

Original: Then,dρ
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where v / ∈

Original: [Equation]

Original: ˆ

Original: [Equation] ˆ θ 2 π (cid:111) . [Figure] Thus, ρ ( h, ˆ h ) is continuous and strictly increasing with σ when v ̸ = ˆ θ/ 2 π and v ̸ =
Revised: Thus, ρ ( h, ˆ h ) is continuous and strictly increasing with σ when v ̸ = ˆ θ/ 2 π and v ̸ =

Original: ( π + ˆ θ ) / 2 π .
Revised: ( π + ˆ θ ) / 2 π .

Original: Let ρ ( h, ˆ h ) =
Revised: Let ρ ( h, ˆ h ) =

Original: g ( σ, u, v ) , then 
Revised: g ( σ, u, v ) , then E [ ρ ( h, ˆ h )]

Original: [Equation]
Revised: = (cid:82) g ( σ, u, v ) f ( u ) f

Original:  where h ( u ) = I [0 < u < 1] and h ( v ) =
Revised: ( v ) dudv where h ( u ) = I [0 < u < 1] and h ( v ) =

Original: I [0 < v < 1] .
Revised: I [0 < v < 1] .

Original: For 0 < σ 1 < σ 2 ,
Revised: For 0 < σ 1 < σ 2 ,

Original: E 
Revised: E [ g ( σ 2 , u, v )] − E [ g ( σ 1 , u, v )] = (cid:90) ( g ( σ 2 , u, v ) − g ( σ 1 , u, v )) f ( u ) f ( v ) dudv >

Original: [Equation]
Revised: [Figure]

Original: C D ATASETS , N ETWORKS AND E XPERIMENTAL S ETTINGS C. B ENCHMARK D ATASETS
Revised: C D ATASETS , N ETWORKS AND E XPERIMENTAL S ETTINGS C. B ENCHMARK D ATASETS

Original: MNIST (Lecun et al., 1998) is a handwritten digit dataset which has 60 , 000 training samples and 10 , 000 test samples in 10 classes.
Revised: MNIST (Lecun et al., 1998) is a handwritten digit dataset which has 60 , 000 training samples and 10 , 000 test samples in 10 classes.

Original: Each sample is a black and white image and 28 × 28 in size.
Revised: Each sample is a black and white image and 28 × 28 in size.

Original: CIFAR10 and CIFAR100 (Krizhevsky, 2009) are tiny image datasets which has 50 , 000 training samples and 10 , 000 test samples in 10 and 100 classes respectively.
Revised: CIFAR10 and CIFAR100 (Krizhevsky, 2009) are tiny image datasets which has 50 , 000 training samples and 10 , 000 test samples in 10 and 100 classes respectively.

Original: Each sample is a color image and 32 × 32 in size.
Revised: Each sample is a color image and 32 × 32 in size.

Original: SVHN (Netzer et al., 2011) is a real-world digit dataset which has 73 , 257 training samples and 26 , 032 test samples in 10 classes.
Revised: SVHN (Netzer et al., 2011) is a real-world digit dataset which has 73 , 257 training samples and 26 , 032 test samples in 10 classes.

Original: Each sample is a color image and 32 × 32 in size.
Revised: Each sample is a color image and 32 × 32 in size.

Original: Tiny ImageNet  is a subset of the ILSVRC (Russakovsky et al., 2015) dataset which has 100 ,samples in 200 classes.
Revised: Tiny ImageNet (Le & Yang, 2015) is a subset of the ILSVRC (Russakovsky et al., 2015) dataset which has 100 , 000 samples in 200 classes.

Original: Each sample is a color image and 64 × 64 in size.
Revised: Each sample is a color image and 64 × 64 in size.

Original: In experiments, Tiny ImageNet is split into two parts: 90 , 000 samples for training and 10 , 000 samples for test.
Revised: In experiments, Tiny ImageNet is split into two parts: 90 , 000 samples for training and 10 , 000 samples for test.

Original: Food101 (Bossard et al., 2014) is a fine-grained food image dataset which has 75 , 750 training samples and 25 , 250 test samples in 101 classes.
Revised: Food101 (Bossard et al., 2014) is a fine-grained food image dataset which has 75 , 750 training samples and 25 , 250 test samples in 101 classes.

Original: Each sample is a color image and resized to 75 × 75 .
Revised: Each sample is a color image and resized to 75 × 75 .

Original: All datasets are used without any preprocessing of images.
Revised: All datasets are used without any preprocessing of images.

Original: C. D EEP N ETWORKS
Revised: C. D EEP N ETWORKS

Original: S-CNN (Chollet et al., 2015) consists of [3 × 3 × 32 conv − 3 × 3 × 64 conv − 2 × 2 maxpool − dropout ( 0 . 25 )
Revised: S-CNN (Chollet et al., 2015) consists of [3 × 3 × 32 conv − 3 × 3 × 64 conv − 2 × 2 maxpool − dropout ( 0 . 25 )

Original: − 128 dense − dropout ( 0 . 5 )
Revised: − 128 dense − dropout ( 0 . 5 )

Original: − # class dense − softmax] layers, and it is used for
Revised: − # class dense − softmax] layers, and it is used for

Original: MNIST.
Revised: MNIST.

Original: K-CNN (Chollet et al., 2015) consists of [two 3 × 3 × 32 conv − 2 ×
Revised: K-CNN (Chollet et al., 2015) consists of [two 3 × 3 × 32 conv − 2 ×

Original: 2 maxpool - dropout ( 0 . 25 ) − two 3 × 3 × 64 conv − 2 ×2 maxpool - dropout ( 0 . 25 ) − 512 dense − dropout ( 0 . 5 ) − # class densesoftmax] layers, and it is used for CIFAR10, SVHN, and CIFAR100.
Revised: 2 maxpool - dropout ( 0 . 25 ) − two 3 × 3 × 64 conv [Table] ×

Original: [Table]
Revised: [Table]

Original: WRN-16-8 (Zagoruyko & Komodakis, 2016) is a wide residual network that has 16 convolutional layers and a widening factor 8, and it is used for CIFAR100 and Tiny ImageNet.
Revised: - dropout [Table] [Table] 512 dense

Revised: [Table]

Revised: dropout

Revised: [Table]

Revised: 0 . 5 ) − # class densesoftmax] layers, and it is used for CIFAR10, SVHN, and CIFAR100.

Revised: WRN-16-8 (Zagoruyko & Komodakis, 2016) is a wide residual network that has 16 convolutional layers and a widening factor 8, and it is used for CIFAR100 and Tiny ImageNet.

Original: C. E XPERIMENTAL S ETTINGS
Revised: C. E XPERIMENTAL S ETTINGS

Original: Training settings regarding a number of epochs, batch size, optimizer, learning rate, and learning rate schedule are summarized in Table 3.
Revised: Training settings regarding a number of epochs, batch size, optimizer, learning rate, and learning rate schedule are summarized in Table 3.

Original: The model parameters are initialized with He normal initialization (He et al., 2015) for all experimental settings.
Revised: The model parameters are initialized with He normal initialization (He et al., 2015) for all experimental settings.

Original: For all experiments, the initial labeled samples for each repetition are randomly sampled according to the distribution of the training set. 
Revised: For all experiments, the initial labeled samples for each repetition are randomly sampled according to the distribution of the training set. [Table] [Figure]

Original: DR ESULTS FOR T EST A CCURACY
Revised: D A BLATION S TUDY D. S TOP C ONDITION

Revised: Figure 9 shows the number of sampled hypotheses and the runtime for evaluating empirical LDM of pool data in MNIST, as the stop condition varies.

Revised: Both are monotonically increasing with the stop condition, which implies that large stop condition, although it may result in empirical LDM closer to the true LDM, potentially requires huge computing power, which is very inefficient

Revised: . D. N EED FOR D IVERSITY IN B ATCH A CTIVE L EARNING

Revised: In the batch selection setting where samples are selected in the order of the smallest empirical LDM, there may be some overlap in the information of the selected samples’ features, thereby reducing the performance.

Revised: In other words, querying unlabeled samples with the smallest LDMs may not lead to the best performance.

Revised: To confirm this phenomenon, we query the k th batch of size q = 20 for k ∈ [50] from MNIST sorted in ascending order of LDM and compare the improvements in test accuracy.

Revised: As shown in Figure 10a where 100 samples are labeled, the smallest LDM leads to the best performance, whereas in Figure 10d where 300 samples are labeled, the smallest LDM does not lead to the best performance.

Revised: To see why this is the case, we’ve plotted the t-SNE plots (van der

Revised: Maaten & Hinton, 2008) of the first and eighth batches for each case.

Revised: In the first case, as shown in

Revised: Figure 10b–10c, the samples of the first and eighth batches are all spread out, so there is no overlap of information between the samples in each batch; taking a closer look, it seems that smaller LDM leads to the samples being more spread out.

Revised: However, in the second case, as shown in Figure 10e– 10f, the samples of the first batch are close to one another i.e. there is a significant overlap of information between the samples in that batch.

Revised: Surprisingly, this is not the case for the eighth batch, which consists of samples of larger LDMs.

Revised: From this, we conclude that in the batch selection setting, even when using the LDM-based approach, diversity should be taken into consideration.

Revised: D. C OMPARING WITH O THER U NCERTAINTY M ETHODS WITH S EEDING

Revised: To clarify whether the gains of LDM-S over the standard uncertainty methods are due to weighted seeding or due to the superiority of LDM, the performance of LDM-S is compared with those meth-

Revised: [Figure]

Revised: [Figure]

Revised: ods to which weighted seeding is applied.

Revised: Figure 11 shows the test accuracy with respect to the number of labeled samples on MNIST, CIFAR10, SVHN and CIFAR100 datasets.

Revised: Overall, even when weighted seeding is applied to the standard uncertainty methods, LDM-S still performs better on all datasets.

Revised: Therefore, the performance gains of LDM-S can be attributed to LDM’s superiority over the standard uncertainty measures.

Revised: [Table]

Original: Figure 7 shows the test accuracy with respect to  the number of labeled samples from initial to final step for all experimental settings.
Revised: [Figure] E A DDITIONAL R ESULTS E. R UNTIME Table 4 present the mean of runtime (min) to perform active learning for each algorithm and each dataset. The value in parentheses is the ratio of the runtime for each algorithm to that of Entropy.

Original: [Figure]
Revised: Overall, the runtime of LDM-S increased by only 3 ∼ 6% compared to Entropy, and it is a little faster than MC-BALD, Coreset, and BADGE. ENS-VarR requires about 5 times more computational load than Entropy, as all networks in the ensemble are individually trained in that method. The only exception is MNIST in which the runtime of LDM-S increases by 70% compared to Entropy; this can be attributed to the relatively small training time compared to acquisition time, which is due to the simplicity of MNIST. E. R ESULTS FOR T EST A CCURACY

Revised: Figure 12 shows the test accuracy with respect to the number of labeled samples from initial to final step for all experimental settings.