Original: U NCERTAINTY - AWARE A CTIVE L EARNING FOR O PTIMAL B AYESIAN C LASSIFIER
Revised: U NCERTAINTY - AWARE A CTIVE L EARNING FOR O PTIMAL B AYESIAN C LASSIFIER

Original: A BSTRACT
Revised: A BSTRACT

Original: For pool-based active learning, in each iteration a candidate training sample is chosen for labeling by optimizing an acquisition function.
Revised: For pool-based active learning, in each iteration a candidate training sample is chosen for labeling by optimizing an acquisition function.

Original: In Bayesian classiﬁcation, expected Loss Reduction (ELR) methods maximize the expected reduction in the classiﬁcation error given a new labeled candidate based on a one-step-lookahead strategy.
Revised: In Bayesian classiﬁcation, expected Loss Reduction (ELR) methods maximize the expected reduction in the classiﬁcation error given a new labeled candidate based on a one-step-lookahead strategy.

Original: ELR is the optimal strategy with a single query; however, since such myopic strategies cannot identify the long-term effect of a query on the classiﬁcation error, ELR may get stuck before reaching the optimal classiﬁer.
Revised: ELR is the optimal strategy with a single query; however, since such myopic strategies cannot identify the long-term effect of a query on the classiﬁcation error, ELR may get stuck before reaching the optimal classiﬁer.

Original: In this paper, inspired by the mean objective cost of uncertainty (MOCU), a metric quantifying the uncertainty directly affecting the classiﬁcation error, we propose an acquisition function based on a weighted form of MOCU.
Revised: In this paper, inspired by the mean objective cost of uncertainty (MOCU), a metric quantifying the uncertainty directly affecting the classiﬁcation error, we propose an acquisition function based on a weighted form of MOCU.

Original: Similar to ELR, the proposed method focuses on the reduction of the uncertainty that pertains to the classiﬁcation error.
Revised: Similar to ELR, the proposed method focuses on the reduction of the uncertainty that pertains to the classiﬁcation error.

Original: But unlike any other existing scheme, it provides the critical advantage that the resulting Bayesian active learning algorithm guarantees convergence to the optimal classiﬁer of the true model.
Revised: But unlike any other existing scheme, it provides the critical advantage that the resulting Bayesian active learning algorithm guarantees convergence to the optimal classiﬁer of the true model.

Original: We demonstrate its performance with both synthetic and real-world datasets.
Revised: We demonstrate its performance with both synthetic and real-world datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In supervised learning, labeling data is often expensive and highly time consuming.
Revised: In supervised learning, labeling data is often expensive and highly time consuming.

Original: Active learning is one ﬁeld of research that aims to address this problem and has been demonstrated for sampleefﬁcient learning with less required labeled data (Gal et al., 2017; Tran et al., 2019; Sinha et al., 2019).
Revised: Active learning is one ﬁeld of research that aims to address this problem and has been demonstrated for sampleefﬁcient learning with less required labeled data (Gal et al., 2017; Tran et al., 2019; Sinha et al., 2019).

Original: In this paper, we focus on pool-based Bayesian active learning for classiﬁcation with 01 loss function.
Revised: In this paper, we focus on pool-based Bayesian active learning for classiﬁcation with 01 loss function.

Original: Bayesian active learning starts from the prior knowledge of uncertain models.
Revised: Bayesian active learning starts from the prior knowledge of uncertain models.

Original: By optimizing an acquisition function, it chooses the next candidate training sample to query for labeling, and then based on the acquired data, updates the belief of uncertain models through Bayes’ rule to approach the optimal classiﬁer of the true model, which minimizes the classiﬁcation error.
Revised: By optimizing an acquisition function, it chooses the next candidate training sample to query for labeling, and then based on the acquired data, updates the belief of uncertain models through Bayes’ rule to approach the optimal classiﬁer of the true model, which minimizes the classiﬁcation error.

Original: In active learning, maximizing the performance of the model trained on queried candidates is the ultimate objective.
Revised: In active learning, maximizing the performance of the model trained on queried candidates is the ultimate objective.

Original: However, most of the existing methods do not directly target the learning objective.
Revised: However, most of the existing methods do not directly target the learning objective.

Original: For example, Maximum Entropy Sampling (MES) or Uncertainty Sampling, simply queries the candidate with the maximum predictive entropy (Lewis & Gale, 1994; Sebastiani & Wynn, 2000; Mussmann & Liang, 2018); but the method fails to differentiate between the model uncertainty and the observation uncertainty.
Revised: For example, Maximum Entropy Sampling (MES) or Uncertainty Sampling, simply queries the candidate with the maximum predictive entropy (Lewis & Gale, 1994; Sebastiani & Wynn, 2000; Mussmann & Liang, 2018); but the method fails to differentiate between the model uncertainty and the observation uncertainty.

Original: Bayesian Active Learning by Disagreement (BALD) seeks the data point that maximizes the mutual information between the observation and the model parameters (Houlsby et al., 2011; Kirsch et al., 2019).
Revised: Bayesian Active Learning by Disagreement (BALD) seeks the data point that maximizes the mutual information between the observation and the model parameters (Houlsby et al., 2011; Kirsch et al., 2019).

Original: Besides BALD, there are also other methods reducing the model uncertainty in different forms (Golovin et al., 2010; Cuong et al., 2013).
Revised: Besides BALD, there are also other methods reducing the model uncertainty in different forms (Golovin et al., 2010; Cuong et al., 2013).

Original: However, not all the model uncertainty will affect the performance of the learning task of interest.
Revised: However, not all the model uncertainty will affect the performance of the learning task of interest.

Original: Without identifying whether the uncertainty is related to the classiﬁcation error or not, these methods can be inefﬁcient in the sense that it may query candidates that do not directly help improve prediction performance.
Revised: Without identifying whether the uncertainty is related to the classiﬁcation error or not, these methods can be inefﬁcient in the sense that it may query candidates that do not directly help improve prediction performance.

Original: [Figure]
Revised: [Figure]

Original: In this paper we focus on the active learning methods directly maximizing the learning model performance.
Revised: In this paper we focus on the active learning methods directly maximizing the learning model performance.

Original: There exist such active learning methods by Expected Loss Reduction (ELR) that aim to maximize the expected reduction in loss based on a one-step-look-ahead manner (Roy & McCallum, 2001; Zhu et al., 2003; Kapoor et al., 2007).
Revised: There exist such active learning methods by Expected Loss Reduction (ELR) that aim to maximize the expected reduction in loss based on a one-step-look-ahead manner (Roy & McCallum, 2001; Zhu et al., 2003; Kapoor et al., 2007).

Original: The ELR methods can focus on only the uncertainty related to the loss function to achieve sample-efﬁcient learning.
Revised: The ELR methods can focus on only the uncertainty related to the loss function to achieve sample-efﬁcient learning.

Original: In fact, ELR is the optimal strategy for active learning with a single query (Roy & McCallum, 2001).
Revised: In fact, ELR is the optimal strategy for active learning with a single query (Roy & McCallum, 2001).

Original: However, a critical shortcoming of previous ELR schemes is that none of them provide any theoretical guarantee regarding their longterm performance.
Revised: However, a critical shortcoming of previous ELR schemes is that none of them provide any theoretical guarantee regarding their longterm performance.

Original: In fact, since these methods are myopic and cannot identify the long-term effect of a query on the loss functions, without special design on the loss function, they may get stuck before reaching the optimal classiﬁer.
Revised: In fact, since these methods are myopic and cannot identify the long-term effect of a query on the loss functions, without special design on the loss function, they may get stuck before reaching the optimal classiﬁer.

Original: To the best of our knowledge, there is currently no method that directly maximizes the model performance while simultaneously guaranteeing the convergence to the optimal classiﬁer.
Revised: To the best of our knowledge, there is currently no method that directly maximizes the model performance while simultaneously guaranteeing the convergence to the optimal classiﬁer.

Original: Fig.
Revised: Fig.

Original: 1a provides an example of binary classiﬁcation with one feature where both BALD and ELR methods fail.
Revised: 1a provides an example of binary classiﬁcation with one feature where both BALD and ELR methods fail.

Original: In the ﬁgure, the red lines indicate the upper and lower bounds of the prediction probability of class 1, illustrating the model with higher probability uncertainty on the sides ( x → ± 4 ) than that in the middle ( x = 0 ).
Revised: In the ﬁgure, the red lines indicate the upper and lower bounds of the prediction probability of class 1, illustrating the model with higher probability uncertainty on the sides ( x → ± 4 ) than that in the middle ( x = 0 ).

Original: Querying candidates on the sides will provide more information of the model parameters, and therefore is preferred in BALD.
Revised: Querying candidates on the sides will provide more information of the model parameters, and therefore is preferred in BALD.

Original: However, since the possible probabilities on the sides are always larger than or less than 0.5, querying candidates on the sides will not help reduce the classiﬁcation error.
Revised: However, since the possible probabilities on the sides are always larger than or less than 0.5, querying candidates on the sides will not help reduce the classiﬁcation error.

Original: On the other hand, ELR queries candidates that help reduce the classiﬁcation error the most, so it prefers data in the middle whose optimal labels are uncertain given the prior knowledge.
Revised: On the other hand, ELR queries candidates that help reduce the classiﬁcation error the most, so it prefers data in the middle whose optimal labels are uncertain given the prior knowledge.

Original: The performance shown in Fig.
Revised: The performance shown in Fig.

Original: 1b agrees with our analysis.
Revised: 1b agrees with our analysis.

Original: Fig.
Revised: Fig.

Original: 1b shows the performance averaged over 1000 runs, with more details and discussions of the example included in
Revised: 1b shows the performance averaged over 1000 runs, with more details and discussions of the example included in

Original: Appendix C. BALD performs inefﬁciently at the beginning by querying points on both sides.
Revised: Appendix C. BALD performs inefﬁciently at the beginning by querying points on both sides.

Original: On the other hand, the ELR method performs the best at the beginning, but becomes inefﬁcient after some iterations ( ∼ 100), indicating some of its runs get stuck before reaching the optimal classiﬁer.
Revised: On the other hand, the ELR method performs the best at the beginning, but becomes inefﬁcient after some iterations ( ∼ 100), indicating some of its runs get stuck before reaching the optimal classiﬁer.

Original: In this paper, we consider the algorithm to “get stuck” when the acquisition function value is 0 for all the candidates in the pool and the algorithm degenerates to uniform random sampling.
Revised: In this paper, we consider the algorithm to “get stuck” when the acquisition function value is 0 for all the candidates in the pool and the algorithm degenerates to uniform random sampling.

Original: In this paper, we analyze the reason why ELR methods may get stuck before reaching the optimal classiﬁer, and propose a new strategy to solve this problem.
Revised: In this paper, we analyze the reason why ELR methods may get stuck before reaching the optimal classiﬁer, and propose a new strategy to solve this problem.

Original: Our contributions are in four parts: 1.
Revised: Our contributions are in four parts: 1.

Original: We show that ELR methods may get stuck, preventing active learning from reaching the optimal classiﬁer efﬁciently.
Revised: We show that ELR methods may get stuck, preventing active learning from reaching the optimal classiﬁer efﬁciently.

Original: We propose a novel weighted-MOCU active learning method that can focus only on the uncertainty related to the loss for efﬁcient active learning and is guaranteed to converge to the optimal classiﬁer of the true model.
Revised: We propose a novel weighted-MOCU active learning method that can focus only on the uncertainty related to the loss for efﬁcient active learning and is guaranteed to converge to the optimal classiﬁer of the true model.

Original: We provide the convergence proof of the weightedMOCU method.
Revised: We provide the convergence proof of the weightedMOCU method.

Original: We demonstrate the sample-efﬁciency of our weighted-MOCU method with both synthetic and real-world datasets.
Revised: We demonstrate the sample-efﬁciency of our weighted-MOCU method with both synthetic and real-world datasets.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Optimal Bayesian classiﬁer.
Revised: Optimal Bayesian classiﬁer.

Original: Consider a classiﬁcation problem with candidates x ∈ X and class labels y ∈ Y
Revised: Consider a classiﬁcation problem with candidates x ∈ X and class labels y ∈ Y

Original: = { 0 , 1 , . .
Revised: = { 0 , 1 , . .

Original: , M − 1 } .
Revised: , M − 1 } .

Original: The predictive probability p ( y | x, θ ) is modeled with parameters θ .
Revised: The predictive probability p ( y | x, θ ) is modeled with parameters θ .

Original: Assume θ is uncertain with a distribution π ( θ ) within the uncertainty class Θ .
Revised: Assume θ is uncertain with a distribution π ( θ ) within the uncertainty class Θ .

Original: The classiﬁcation problem is to ﬁnd a classiﬁer ψ : X → Y , which assigns a predicted class label to a given candidate.
Revised: The classiﬁcation problem is to ﬁnd a classiﬁer ψ : X → Y , which assigns a predicted class label to a given candidate.

Original: The expected 0-1 loss of the classiﬁer ψ for a candidate x , dependent on θ , is deﬁned as C θ ( ψ, x ) , which can be derived to be the classiﬁcation error:
Revised: The expected 0-1 loss of the classiﬁer ψ for a candidate x , dependent on θ , is deﬁned as C θ ( ψ, x ) , which can be derived to be the classiﬁcation error:

Original: C θ ( ψ, x ) = 1 − p ( y = ψ ( x ) | x, θ ) .
Revised: C θ ( ψ, x ) = 1 − p ( y = ψ ( x ) | x, θ ) .

Original: The optimal classiﬁer with θ , ψ θ is deﬁned as the classiﬁer minimizing the classiﬁcation error:
Revised: The optimal classiﬁer with θ , ψ θ is deﬁned as the classiﬁer minimizing the classiﬁcation error:

Original: ψ θ ( x ) = arg max y p ( y | x, θ ) .
Revised: ψ θ ( x ) = arg max y p ( y | x, θ ) .

Original: So we have: C θ ( ψ θ , x ) =
Revised: So we have: C θ ( ψ θ , x ) =

Original: min ψ C θ ( ψ, x ) =
Revised: min ψ C θ ( ψ, x ) =

Original: min y { 1 − p ( y | x, θ ) } .
Revised: min y { 1 − p ( y | x, θ ) } .

Original: When there is model uncertainty with π ( θ ) , an Optimal Bayesian Classiﬁer (OBC) ψ π ( θ ) is the classiﬁer that has the minimum expected loss over π ( θ ) (Dalton & Dougherty, 2013):
Revised: When there is model uncertainty with π ( θ ) , an Optimal Bayesian Classiﬁer (OBC) ψ π ( θ ) is the classiﬁer that has the minimum expected loss over π ( θ ) (Dalton & Dougherty, 2013):

Original: [Equation]
Revised: [Equation]

Original: where p ( y | x ) =
Revised: where p ( y | x ) =

Original: E π ( θ ) [ p ( y | x, θ )] is the predictive distribution.
Revised: E π ( θ ) [ p ( y | x, θ )] is the predictive distribution.

Original: It’s easily to see ψ π ( θ ) ( x ) =
Revised: It’s easily to see ψ π ( θ ) ( x ) =

Original: arg max y p ( y | x ) .
Revised: arg max y p ( y | x ) .

Original: Active learning.
Revised: Active learning.

Original: Active learning collects the training dataset D in a sequential way.
Revised: Active learning collects the training dataset D in a sequential way.

Original: For poolbased active learning, in each iteration, we choose a candidate x from the set of potential training samples X to query for the class label by optimizing an acquisition function U ( x ) .
Revised: For poolbased active learning, in each iteration, we choose a candidate x from the set of potential training samples X to query for the class label by optimizing an acquisition function U ( x ) .

Original: Then, in the
Revised: Then, in the

Original: Bayesian setting, by including the observed data pair ( x, y ) to D , we update the posterior distribution based on Bayes’ rule.
Revised: Bayesian setting, by including the observed data pair ( x, y ) to D , we update the posterior distribution based on Bayes’ rule.

Original: In each iteration, the acquisition function depends on the posterior distribution of model parameters
Revised: In each iteration, the acquisition function depends on the posterior distribution of model parameters

Original: π ( θ | D ) .
Revised: π ( θ | D ) .

Original: In the following discussion, to simplify notations, we omit D from the notations and use π ( θ ) and p ( y | x ) to respectively denote the posterior and predictive distributions conditioned on D .
Revised: In the following discussion, to simplify notations, we omit D from the notations and use π ( θ ) and p ( y | x ) to respectively denote the posterior and predictive distributions conditioned on D .

Original: When a new observed data point is included, the distributions are updated by Bayes’ rule and the total probability rule as: π ( θ | x, y )
Revised: When a new observed data point is included, the distributions are updated by Bayes’ rule and the total probability rule as: π ( θ | x, y )

Original: =
Revised: =

Original: π ( θ ) p ( y | x,θ ) p ( y | x ) and p ( y (cid:48)
Revised: π ( θ ) p ( y | x,θ ) p ( y | x ) and p ( y (cid:48)

Original: | x (cid:48) , x, y ) =
Revised: | x (cid:48) , x, y ) =

Original: E π ( θ | x,y )
Revised: E π ( θ | x,y )

Original: [ p ( y (cid:48)
Revised: [ p ( y (cid:48)

Original: | x (cid:48) , θ )] .
Revised: | x (cid:48) , θ )] .

Original: The acquisition function of ELR methods in the Bayesian setting can be deﬁned by the expected
Revised: The acquisition function of ELR methods in the Bayesian setting can be deﬁned by the expected

Original: OBC prediction error reduction after observing the new pair ( x, y ) (Roy & McCallum, 2001):
Revised: OBC prediction error reduction after observing the new pair ( x, y ) (Roy & McCallum, 2001):

Original: [Equation]
Revised: [Equation]

Original: where p ( x (cid:48) ) is the distribution over X , independent of θ and D .
Revised: where p ( x (cid:48) ) is the distribution over X , independent of θ and D .

Original: ELR methods assume that we use
Revised: ELR methods assume that we use

Original: OBC as the classiﬁer, and in each iteration we should choose the query that maximize the decrease in OBC prediction error.
Revised: OBC as the classiﬁer, and in each iteration we should choose the query that maximize the decrease in OBC prediction error.

Original: The ﬁrst term in (2) is the OBC prediction error of ψ π ( θ ) , and the second term is the expected prediction error of ψ π ( θ | x,y ) , the one-step-look-ahead OBC, with respect to p ( y | x ) .
Revised: The ﬁrst term in (2) is the OBC prediction error of ψ π ( θ ) , and the second term is the expected prediction error of ψ π ( θ | x,y ) , the one-step-look-ahead OBC, with respect to p ( y | x ) .

Original: In the following section, we analyze why this acquisition function is sample-efﬁcient as it directly targets at classiﬁcation error reduction while ignoring irrelevant uncertainty with respect to the learning task; but it may get stuck before converging to the true optimal classiﬁer (optimal classiﬁer of the true model).
Revised: In the following section, we analyze why this acquisition function is sample-efﬁcient as it directly targets at classiﬁcation error reduction while ignoring irrelevant uncertainty with respect to the learning task; but it may get stuck before converging to the true optimal classiﬁer (optimal classiﬁer of the true model).

Original:  MOCU- BASED ACTIVE LEARNING 3. M EAN OBJECTIVE COST OF UNCERTAINTY
Revised:  MOCU- BASED ACTIVE LEARNING 3. M EAN OBJECTIVE COST OF UNCERTAINTY

Original: To analyze ELR methods, we borrow the idea of the Mean Objective Cost of Uncertainty (MOCU) for active learning with respect to the corresponding posterior π ( θ ) .
Revised: To analyze ELR methods, we borrow the idea of the Mean Objective Cost of Uncertainty (MOCU) for active learning with respect to the corresponding posterior π ( θ ) .

Original: MOCU is a general objectiveoriented uncertainty quantiﬁcation framework (Yoon et al., 2013).
Revised: MOCU is a general objectiveoriented uncertainty quantiﬁcation framework (Yoon et al., 2013).

Original: For active learning, MOCU can be deﬁned as the expected loss difference between the OBC and the optimal classiﬁer:
Revised: For active learning, MOCU can be deﬁned as the expected loss difference between the OBC and the optimal classiﬁer:

Original: [Equation]
Revised: [Equation]

Original: The second line is derived by the deﬁnition of ψ θ and (1).
Revised: The second line is derived by the deﬁnition of ψ θ and (1).

Original: The ﬁrst term in (3) is the OBC error as the loss.
Revised: The ﬁrst term in (3) is the OBC error as the loss.

Original: In the second term, ψ θ is the optimal classiﬁer with a speciﬁc θ .
Revised: In the second term, ψ θ is the optimal classiﬁer with a speciﬁc θ .

Original: For the terms inside the expectation operator, we have C θ ( ψ π ( θ ) , x (cid:48) )
Revised: For the terms inside the expectation operator, we have C θ ( ψ π ( θ ) , x (cid:48) )

Original: − C θ ( ψ θ , x (cid:48) ) ≥ 0 .
Revised: − C θ ( ψ θ , x (cid:48) ) ≥ 0 .

Original: Therefore, the second term in (3) is a lower bound of the OBC prediction error.
Revised: Therefore, the second term in (3) is a lower bound of the OBC prediction error.

Original: MOCU captures the difference between the OBC error and its lower bound.
Revised: MOCU captures the difference between the OBC error and its lower bound.

Original: When MOCU is 0, the OBC converges to the true optimal classiﬁer and we cannot reduce the OBC prediction error further.
Revised: When MOCU is 0, the OBC converges to the true optimal classiﬁer and we cannot reduce the OBC prediction error further.

Original: In that case, we say that OBC has reached the true optimal classiﬁer.
Revised: In that case, we say that OBC has reached the true optimal classiﬁer.

Original: [Equation]
Revised: [Equation]

Original: As in ELR methods, we can deﬁne an acquisition function by the reduction of MOCU in a one-steplook-ahead manner:
Revised: As in ELR methods, we can deﬁne an acquisition function by the reduction of MOCU in a one-steplook-ahead manner:

Original: [Equation]
Revised: [Equation]

Original: We can show that the second term in (3), the lower bound of the OBC error, is cancelled in (5).
Revised: We can show that the second term in (3), the lower bound of the OBC error, is cancelled in (5).

Original: The acquisition function (5) hence captures the expected reduction of the OBC error given new data and is equivalent to the ELR acquisition function (2).
Revised: The acquisition function (5) hence captures the expected reduction of the OBC error given new data and is equivalent to the ELR acquisition function (2).

Original: Expanding the second term in (5), we have:
Revised: Expanding the second term in (5), we have:

Original: [Equation]
Revised: [Equation]

Original: Since (cid:80) y p
Revised: Since (cid:80) y p

Original: ( y | x ) π ( θ | x, y )
Revised: ( y | x ) π ( θ | x, y )

Original: =
Revised: =

Original: π ( θ ) , as x is assumed to be independent of θ so that we have π ( θ | x ) =
Revised: π ( θ ) , as x is assumed to be independent of θ so that we have π ( θ | x ) =

Original: π ( θ ) , we can rewrite the ﬁrst term in (5) as:
Revised: π ( θ ) , we can rewrite the ﬁrst term in (5) as:

Original: [Equation]
Revised: [Equation]

Original: Combining (6) and (7) and canceling the C θ ( ψ θ , x (cid:48) )
Revised: Combining (6) and (7) and canceling the C θ ( ψ θ , x (cid:48) )

Original: terms (the lower bound of the OBC error), (6) can be derived as:
Revised: terms (the lower bound of the OBC error), (6) can be derived as:

Original: [Equation]
Revised: [Equation]

Original: π ( θ | x,y ) , x (cid:48) )]] } , (8) which is just the ELR acquisition function in (2).
Revised: π ( θ | x,y ) , x (cid:48) )]] } , (8) which is just the ELR acquisition function in (2).

Original: Therefore, we can conclude that MOCU-based methods are equivalent to ELR methods.
Revised: Therefore, we can conclude that MOCU-based methods are equivalent to ELR methods.

Original: Another property we can observe from (8) is that U MOCU ( x ; π ( θ )) ≥ 0 .
Revised: Another property we can observe from (8) is that U MOCU ( x ; π ( θ )) ≥ 0 .

Original: By deﬁnition, ψ π ( θ | x,y ) is the OBC with the minimum expected classiﬁcation error over π ( θ | x, y ) .
Revised: By deﬁnition, ψ π ( θ | x,y ) is the OBC with the minimum expected classiﬁcation error over π ( θ | x, y ) .

Original: Therefore,
Revised: Therefore,

Original: E π
Revised: E π

Original: ( θ | x,y )
Revised: ( θ | x,y )

Original: [ C θ ( ψ π ( θ | x,y ) , x (cid:48) )] ≤ E π ( θ | x,y )
Revised: [ C θ ( ψ π ( θ | x,y ) , x (cid:48) )] ≤ E π ( θ | x,y )

Original: [ C θ ( ψ π ( θ ) , x (cid:48) )]
Revised: [ C θ ( ψ π ( θ ) , x (cid:48) )]

Original: and we have U MOCU ( x ; π ( θ )) ≥ 0 , indicating collecting new data will reduce MOCU.
Revised: and we have U MOCU ( x ; π ( θ )) ≥ 0 , indicating collecting new data will reduce MOCU.

Original: 3. A NALYSIS OF ELR METHODS
Revised: 3. A NALYSIS OF ELR METHODS

Original: In the following analysis, we assume that Θ contains the true model θ r and π ( θ r ) > 0 .
Revised: In the following analysis, we assume that Θ contains the true model θ r and π ( θ r ) > 0 .

Original: We ﬁrst analyze ELR methods by the MOCU reduction to show that ELR and MOCU-based active learning ignores the uncertainty irrelevant to the OBC prediction.
Revised: We ﬁrst analyze ELR methods by the MOCU reduction to show that ELR and MOCU-based active learning ignores the uncertainty irrelevant to the OBC prediction.

Original: By that, we indicate that not all the model uncertainties directly affect the OBC prediction.
Revised: By that, we indicate that not all the model uncertainties directly affect the OBC prediction.

Original: Denote the contribution to the MOCU at point x as K ( x, π ( θ ))
Revised: Denote the contribution to the MOCU at point x as K ( x, π ( θ ))

Original: =
Revised: =

Original: E π ( θ )
Revised: E π ( θ )

Original: [ C θ ( ψ π ( θ ) , x ) − C θ ( ψ θ , x )] , so that M ( π ( θ )) =
Revised: [ C θ ( ψ π ( θ ) , x ) − C θ ( ψ θ , x )] , so that M ( π ( θ )) =

Original: E p ( x ) [ K ( x, π ( θ ))] .
Revised: E p ( x ) [ K ( x, π ( θ ))] .

Original: If
Revised: If

Original: K ( x, π ( θ ))
Revised: K ( x, π ( θ ))

Original: = 0 , then we have ∀ θ ∈ supp ( π ) , ψ θ ( x ) = ψ π ( θ ) ( x ) , i.e. arg max y p
Revised: = 0 , then we have ∀ θ ∈ supp ( π ) , ψ θ ( x ) = ψ π ( θ ) ( x ) , i.e. arg max y p

Original: ( y | x, θ ) =
Revised: ( y | x, θ ) =

Original: arg max y p ( y | x ) .
Revised: arg max y p ( y | x ) .

Original: This means that for all the possible models, the optimal predictions are the same, and the OBC prediction on x will not be affected by the remaining uncertainty of p ( y | x, θ ) , if any.
Revised: This means that for all the possible models, the optimal predictions are the same, and the OBC prediction on x will not be affected by the remaining uncertainty of p ( y | x, θ ) , if any.

Original: In fact, K ( x, π ( θ )) = 0 does not necessarily mean that there is no uncertainty associated with p ( y | x, θ ) , for which it requires that the value of p ( y | x, θ ) is the same ∀ θ ∈ supp ( π ) , apparently a stronger statement than K ( x, π ( θ )) being 0.
Revised: In fact, K ( x, π ( θ )) = 0 does not necessarily mean that there is no uncertainty associated with p ( y | x, θ ) , for which it requires that the value of p ( y | x, θ ) is the same ∀ θ ∈ supp ( π ) , apparently a stronger statement than K ( x, π ( θ )) being 0.

Original: Therefore, not all the uncertainties of p ( y | x, θ ) are captured in MOCU when K ( x, π ( θ ))
Revised: Therefore, not all the uncertainties of p ( y | x, θ ) are captured in MOCU when K ( x, π ( θ ))

Original: = 0 .
Revised: = 0 .

Original: We consider the uncertainty in p ( y | x, θ ) to be “objectiveirrelevant” to the OBC prediction
Revised: We consider the uncertainty in p ( y | x, θ ) to be “objectiveirrelevant” to the OBC prediction

Original: if K ( x, π ( θ )) = 0 .
Revised: if K ( x, π ( θ )) = 0 .

Original: In the active learning procedure, when a new observation is obtained, it reduces the uncertainty of the parameter θ ; and as a result, it reduces the uncertainty of p ( y | x, θ ) for each x ∈ X .
Revised: In the active learning procedure, when a new observation is obtained, it reduces the uncertainty of the parameter θ ; and as a result, it reduces the uncertainty of p ( y | x, θ ) for each x ∈ X .

Original: If an observation only reduces objective-irrelevant uncertainty, the value of MOCU will not change.
Revised: If an observation only reduces objective-irrelevant uncertainty, the value of MOCU will not change.

Original: For example, in Fig.
Revised: For example, in Fig.

Original: 1a, the uncertainty of p ( y | x, θ ) in the region close to x = ± 4 is objective-irrelevant.
Revised: 1a, the uncertainty of p ( y | x, θ ) in the region close to x = ± 4 is objective-irrelevant.

Original: Evaluating points at x → ± 4 will only reduce irrelevant uncertainty and it will not be considered in either MOCU- or ELR-based methods.
Revised: Evaluating points at x → ± 4 will only reduce irrelevant uncertainty and it will not be considered in either MOCU- or ELR-based methods.

Original: That explains why in the ﬁrst several active learning iterations, the ELR or MOCU-based active learning can be more efﬁcient than the methods guided by total uncertainty reduction, such as BALD.
Revised: That explains why in the ﬁrst several active learning iterations, the ELR or MOCU-based active learning can be more efﬁcient than the methods guided by total uncertainty reduction, such as BALD.

Original: Now we explain why ELR methods may get stuck before the OBC converges to the true optimal classiﬁer.
Revised: Now we explain why ELR methods may get stuck before the OBC converges to the true optimal classiﬁer.

Original: When we have ∀ x ∈ X , U MOCU ( x ; π ( θ ))
Revised: When we have ∀ x ∈ X , U MOCU ( x ; π ( θ ))

Original: = 

Revised: =

Original: and assume the tie is broken randomly, the acquisition function will suggest any random candidate in the pool.
Revised: and assume the tie is broken randomly, the acquisition function will suggest any random candidate in the pool.

Original: When that happens, we say that
Revised: When that happens, we say that

Original: ELR methods get stuck if the OBC has not reached the true optimal classiﬁer; i.e., M ( π ( θ )) is still larger than 0.
Revised: ELR methods get stuck if the OBC has not reached the true optimal classiﬁer; i.e., M ( π ( θ )) is still larger than 0.

Original: Since p ( y (cid:48) | x (cid:48) )
Revised: Since p ( y (cid:48) | x (cid:48) )

Original: =
Revised: =

Original: E π ( θ ) [ p ( y (cid:48)
Revised: E π ( θ ) [ p ( y (cid:48)

Original: | x (cid:48) , θ )] is a linear function of π ( θ ) , the term min y (cid:48) { 1 − p ( y (cid:48)
Revised: | x (cid:48) , θ )] is a linear function of π ( θ ) , the term min y (cid:48) { 1 − p ( y (cid:48)

Original: | x (cid:48) ) } in (4) is the minimum among M linear functions, and thus a concave piece-wise linear function.
Revised: | x (cid:48) ) } in (4) is the minimum among M linear functions, and thus a concave piece-wise linear function.

Original: Within each linear piece, ψ π ( θ ) ( x (cid:48) )
Revised: Within each linear piece, ψ π ( θ ) ( x (cid:48) )

Original: =
Revised: =

Original: arg max y (cid:48) p ( y (cid:48)
Revised: arg max y (cid:48) p ( y (cid:48)

Original: | x (cid:48) ) are the same for different π ( θ ) .
Revised: | x (cid:48) ) are the same for different π ( θ ) .

Original: The second term
Revised: The second term

Original: E π ( θ ) [min y (cid:48) { 1 − p ( y (cid:48) | x (cid:48) , θ ) } ] in (4) is a linear function of π ( θ ) .
Revised: E π ( θ ) [min y (cid:48) { 1 − p ( y (cid:48) | x (cid:48) , θ ) } ] in (4) is a linear function of π ( θ ) .

Original: Subtracting it from the ﬁrst term and averaging the resulting difference over p ( x (cid:48) ) maintain the concavity and the piece-wise linearity.
Revised: Subtracting it from the ﬁrst term and averaging the resulting difference over p ( x (cid:48) ) maintain the concavity and the piece-wise linearity.

Original: [Figure]
Revised: [Figure]

Original: Therefore, MOCU deﬁned in (4) is a concave piece-wise linear function of π ( θ ) .
Revised: Therefore, MOCU deﬁned in (4) is a concave piece-wise linear function of π ( θ ) .

Original: Moreover, within a linear piece of MOCU, ψ π ( θ ) ( x (cid:48) ) =
Revised: Moreover, within a linear piece of MOCU, ψ π ( θ ) ( x (cid:48) ) =

Original: arg max y (cid:48) p ( y (cid:48)
Revised: arg max y (cid:48) p ( y (cid:48)

Original: | x (cid:48) ) , i.e. the OBC prediction, is the same for each of x (cid:48)
Revised: | x (cid:48) ) , i.e. the OBC prediction, is the same for each of x (cid:48)

Original: ∈ X .
Revised: ∈ X .

Original: To gain some intuition, we study a binary classiﬁcation problem with the uncertainty class of two possible models Θ
Revised: To gain some intuition, we study a binary classiﬁcation problem with the uncertainty class of two possible models Θ

Original: = { θ 1 , θ 2 } and a candidate pool of two training samples to query:
Revised: = { θ 1 , θ 2 } and a candidate pool of two training samples to query:

Original: X = { x 1 , x 2 } .
Revised: X = { x 1 , x 2 } .

Original: Further details of the model setup can be found in Appendix D. Since π ( θ 1 )
Revised: Further details of the model setup can be found in Appendix D. Since π ( θ 1 )

Original: = 1 − π ( θ 2 ) , we can express the MOCU function as a univariate function of π ( θ 1 ) as shown in Fig.
Revised: = 1 − π ( θ 2 ) , we can express the MOCU function as a univariate function of π ( θ 1 ) as shown in Fig.

Original: It is clear that the MOCU function is a concave piece-wise linear function.
Revised: It is clear that the MOCU function is a concave piece-wise linear function.

Original: Since π ( θ ) =
Revised: Since π ( θ ) =

Original: E p ( y | x ) [ π ( θ | x, y )] , from (5), the acquisition function is deﬁned as
Revised: E p ( y | x ) [ π ( θ | x, y )] , from (5), the acquisition function is deﬁned as

Original: M [ E p ( y | x ) [ π ( θ | x, y )]] − E p ( y | x ) [ M ( π ( θ | x, y ))] .
Revised: M [ E p ( y | x ) [ π ( θ | x, y )]] − E p ( y | x ) [ M ( π ( θ | x, y ))] .

Original: Based on the concavity of MOCU M ( · ) and
Revised: Based on the concavity of MOCU M ( · ) and

Original: Jensen’s inequality, we have the acquisition function U MOCU ( x ; π ( θ )) ≥ 0 .
Revised: Jensen’s inequality, we have the acquisition function U MOCU ( x ; π ( θ )) ≥ 0 .

Original: The equality holds if for all y ∈ Y , π ( θ | x, y ) falls into the same linear piece of MOCU.
Revised: The equality holds if for all y ∈ Y , π ( θ | x, y ) falls into the same linear piece of MOCU.

Original: In this case, the single query ( x, y ) cannot provide enough evidence to shift the OBC prediction, so that arg max y (cid:48) p ( y (cid:48) | x (cid:48) , x, y ) =
Revised: In this case, the single query ( x, y ) cannot provide enough evidence to shift the OBC prediction, so that arg max y (cid:48) p ( y (cid:48) | x (cid:48) , x, y ) =

Original: arg max y (cid:48) p ( y (cid:48)
Revised: arg max y (cid:48) p ( y (cid:48)

Original: | x (cid:48) ) for each of x (cid:48)
Revised: | x (cid:48) ) for each of x (cid:48)

Original: ∈ X , even though p ( y (cid:48) | x (cid:48) , x, y ) (cid:54) =
Revised: ∈ X , even though p ( y (cid:48) | x (cid:48) , x, y ) (cid:54) =

Original: p ( y (cid:48)
Revised: p ( y (cid:48)

Original: | x (cid:48) ) .
Revised: | x (cid:48) ) .

Original: In Fig.
Revised: In Fig.

Original: 2, we have shown in the binary classiﬁcation problem there exists such a case that ˜ π
Revised: 2, we have shown in the binary classiﬁcation problem there exists such a case that ˜ π

Original: ( θ 1 | x 1 , y 1 = 0) and ˜ π
Revised: ( θ 1 | x 1 , y 1 = 0) and ˜ π

Original: ( θ 1 | x 1 , y 1 = 1) are within the interval of the same linear piece of the corresponding MOCU function.
Revised: ( θ 1 | x 1 , y 1 = 1) are within the interval of the same linear piece of the corresponding MOCU function.

Original: When MOCU is larger than 0, if all candidates cannot provide enough evidence to change the OBC predictions, ELR and MOCU-based methods will get stuck before converging to the true optimal classiﬁer.
Revised: When MOCU is larger than 0, if all candidates cannot provide enough evidence to change the OBC predictions, ELR and MOCU-based methods will get stuck before converging to the true optimal classiﬁer.

Original: This is due to the myopic nature of the acquisition function ignoring the long-term effect of querying each candidate.
Revised: This is due to the myopic nature of the acquisition function ignoring the long-term effect of querying each candidate.

Original: In summary, ELR methods are efﬁcient by ignoring objective-irrelevant uncertainty.
Revised: In summary, ELR methods are efﬁcient by ignoring objective-irrelevant uncertainty.

Original: But if sampling one data point can only provide little information and cannot help improve OBC prediction in the current iteration, they will ignore its long-term effect on prediction performance.
Revised: But if sampling one data point can only provide little information and cannot help improve OBC prediction in the current iteration, they will ignore its long-term effect on prediction performance.

Original: As a result, they may get stuck before converging to the optimal classiﬁer.
Revised: As a result, they may get stuck before converging to the optimal classiﬁer.

Original: To keep the efﬁcient sampling property by ignoring objective-irrelevant uncertainty but avoid getting stuck, we propose a one-step-look-ahead acquisition function based on a weighted version of MOCU, which can capture the change of the predictive probability from one-step query candidates.
Revised: To keep the efﬁcient sampling property by ignoring objective-irrelevant uncertainty but avoid getting stuck, we propose a one-step-look-ahead acquisition function based on a weighted version of MOCU, which can capture the change of the predictive probability from one-step query candidates.

Original: If that change can potentially shift the OBC predictions in the long run, our acquisition function will have a positive value, and thereby avoid the issues that ELR methods suffer from.
Revised: If that change can potentially shift the OBC predictions in the long run, our acquisition function will have a positive value, and thereby avoid the issues that ELR methods suffer from.

Original: 3. W EIGHTED MOCU- BASED ACTIVE LEARNING
Revised: 3. W EIGHTED MOCU- BASED ACTIVE LEARNING

Original: In this section, we propose a modiﬁed MOCU-based acquisition function that has the theoretical guarantee to converge to the optimal classiﬁer.
Revised: In this section, we propose a modiﬁed MOCU-based acquisition function that has the theoretical guarantee to converge to the optimal classiﬁer.

Original: Speciﬁcally, we propose a modiﬁed MOCU function that multiplies a weight with each loss difference between the OBC ψ π ( θ ) and the optimal classiﬁer ψ θ in the original MOCU deﬁnition:
Revised: Speciﬁcally, we propose a modiﬁed MOCU function that multiplies a weight with each loss difference between the OBC ψ π ( θ ) and the optimal classiﬁer ψ θ in the original MOCU deﬁnition:

Original: [Equation]
Revised: [Equation]

Original: where w ( π ( θ ) , x (cid:48) , θ ) > 0 is the weighting function.
Revised: where w ( π ( θ ) , x (cid:48) , θ ) > 0 is the weighting function.

Original: The corresponding acquisition function is:
Revised: The corresponding acquisition function is:

Original: [Equation]
Revised: [Equation]

Original: In (9), as more data are collected and the model parameter distribution π ( θ ) changes, w ( π ( θ ) , x (cid:48) , θ ) will change accordingly.
Revised: In (9), as more data are collected and the model parameter distribution π ( θ ) changes, w ( π ( θ ) , x (cid:48) , θ ) will change accordingly.

Original: The change of w ( π ( θ ) , x (cid:48) , θ ) cannot affect the value of the weighted
Revised: The change of w ( π ( θ ) , x (cid:48) , θ ) cannot affect the value of the weighted

Original: MOCU if C θ ( ψ π ( θ ) , x (cid:48) )
Revised: MOCU if C θ ( ψ π ( θ ) , x (cid:48) )

Original: − C θ ( ψ θ , x (cid:48) )
Revised: − C θ ( ψ θ , x (cid:48) )

Original: = 0 , ∀ θ ∈ supp ( π ( θ )) , indicating the uncertainty at x (cid:48) is objective-irrelevant.
Revised: = 0 , ∀ θ ∈ supp ( π ( θ )) , indicating the uncertainty at x (cid:48) is objective-irrelevant.

Original: This makes sure that the acquisition function based on the weighted MOCU will inherit the property of MOCU-based active learning to directly target at classiﬁcation error reduction while ignoring irrelevant uncertainty.
Revised: This makes sure that the acquisition function based on the weighted MOCU will inherit the property of MOCU-based active learning to directly target at classiﬁcation error reduction while ignoring irrelevant uncertainty.

Original: On the other hand, by introducing the predictive probability into the weighting functions, the probability change from one-step samples can be captured by the weighted-MOCU based acquisition function such that it can have theoretical convergence guaranteed to the optimal classiﬁer as shown below.
Revised: On the other hand, by introducing the predictive probability into the weighting functions, the probability change from one-step samples can be captured by the weighted-MOCU based acquisition function such that it can have theoretical convergence guaranteed to the optimal classiﬁer as shown below.

Original: We would like to emphasize that there are also active learning algorithms, such as the ones based on the cyclic sampling and (cid:15) -greedy policies (Hoang et al., 2014), that can almost surely converge to the true model, and as a result, the OBC converges to the true optimal classiﬁer.
Revised: We would like to emphasize that there are also active learning algorithms, such as the ones based on the cyclic sampling and (cid:15) -greedy policies (Hoang et al., 2014), that can almost surely converge to the true model, and as a result, the OBC converges to the true optimal classiﬁer.

Original: However, these policies focus on the total uncertainty reduction to derive the full knowledge of the true model, which is unnecessary and therefore inefﬁcient, since we only need the knowledge of the true optimal classiﬁer if the classiﬁcation performance is the primary concern.
Revised: However, these policies focus on the total uncertainty reduction to derive the full knowledge of the true model, which is unnecessary and therefore inefﬁcient, since we only need the knowledge of the true optimal classiﬁer if the classiﬁcation performance is the primary concern.

Original: Unlike such policies, our weighted-MOCU based policy directly reduces the objective uncertainty affecting classiﬁcation, and as a result, it is much more efﬁcient by focusing only on those queries that are helpful for improving the prediction.
Revised: Unlike such policies, our weighted-MOCU based policy directly reduces the objective uncertainty affecting classiﬁcation, and as a result, it is much more efﬁcient by focusing only on those queries that are helpful for improving the prediction.

Original: As a result, our proposed algorithm guarantees efﬁciency both in the short term as well as in the longer term.
Revised: As a result, our proposed algorithm guarantees efﬁciency both in the short term as well as in the longer term.

Original: In the following, we design a weighting function to make M w ( π ( θ ))
Revised: In the following, we design a weighting function to make M w ( π ( θ ))

Original: = 

Revised: =

Original: if and only if ∀ x ∈
Revised: if and only if ∀ x ∈

Original: X , U w ( x ; π ( θ ))
Revised: X , U w ( x ; π ( θ ))

Original: = 

Revised: =

Original: and show that active learning based on this weighted MOCU converges to the optimal classiﬁer.
Revised: and show that active learning based on this weighted MOCU converges to the optimal classiﬁer.

Original: Speciﬁcally, we propose the following weighting function:
Revised: Speciﬁcally, we propose the following weighting function:

Original: [Equation]
Revised: [Equation]

Original: where 0 < c ≤ 1 is a parameter controlling the approximation of the weighted MOCU to the original MOCU, with smaller c giving a better approximation.
Revised: where 0 < c ≤ 1 is a parameter controlling the approximation of the weighted MOCU to the original MOCU, with smaller c giving a better approximation.

Original: The choice of c depends on the speciﬁc classiﬁcation problem and the total query budget.
Revised: The choice of c depends on the speciﬁc classiﬁcation problem and the total query budget.

Original: Methods using a smaller c approximate the ELR methods better, hence they will perform well in the ﬁrst several iterations but may converge slowly in the long run.
Revised: Methods using a smaller c approximate the ELR methods better, hence they will perform well in the ﬁrst several iterations but may converge slowly in the long run.

Original: On the other hand, when c is closer to 1, the acquisition function weighs more heavily on long-term beneﬁts.
Revised: On the other hand, when c is closer to 1, the acquisition function weighs more heavily on long-term beneﬁts.

Original: It is clear that K ( x (cid:48) , π ( θ )) ≥ 0 by (13).
Revised: It is clear that K ( x (cid:48) , π ( θ )) ≥ 0 by (13).

Original: For binary classiﬁcation, max y (cid:48) p ( y (cid:48) | x (cid:48) ) ≥ 0 .
Revised: For binary classiﬁcation, max y (cid:48) p ( y (cid:48) | x (cid:48) ) ≥ 0 .

Original: As E π ( θ ) [max y (cid:48) p ( y (cid:48) | x (cid:48) , θ )] ≤ 1 , from (14), we have K ( x (cid:48) , π ( θ )) ≤ 0 .
Revised: As E π ( θ ) [max y (cid:48) p ( y (cid:48) | x (cid:48) , θ )] ≤ 1 , from (14), we have K ( x (cid:48) , π ( θ )) ≤ 0 .

Original: 5 , demonstrating that the weighting function in (11) satisﬁes the requirement w ( π ( θ ) , x (cid:48) , θ ) ≥ 0 .
Revised: 5 , demonstrating that the weighting function in (11) satisﬁes the requirement w ( π ( θ ) , x (cid:48) , θ ) ≥ 0 .

Original: 5 > 0 .
Revised: 5 > 0 .

Original: Note that this simple weighting function does not change with respect to the model parameter values.
Revised: Note that this simple weighting function does not change with respect to the model parameter values.

Original: Substituting it into the weighted MOCU expression, we have:
Revised: Substituting it into the weighted MOCU expression, we have:

Original: [Equation]
Revised: [Equation]

Original: (15) which is a strictly concave function of K .
Revised: (15) which is a strictly concave function of K .

Original: We also illustrate the weighted-MOCU function in Fig.for the same example in Section 3.2.
Revised: We also illustrate the weighted-MOCU function in Fig.for the same example in Section 3.2.

Original: As shown in the ﬁgure, a smaller c provides better approximation to the MOCU function, and all the weighted MOCU functions are strictly concave functions of π ( θ 1 ) instead of being piece-wise linear, which guarantees that the acquisition function
Revised: As shown in the ﬁgure, a smaller c provides better approximation to the MOCU function, and all the weighted MOCU functions are strictly concave functions of π ( θ 1 ) instead of being piece-wise linear, which guarantees that the acquisition function

Original: U w ( x 1 ; ˜ π ( θ )) is positive.
Revised: U w ( x 1 ; ˜ π ( θ )) is positive.

Original: In general, weighted MOCU is strictly concave along most of the directions and only changes linearly along the directions that K ( x, π ( θ )) is constant for x ∈ X , which correspond to the queries that only reduce irrelevant uncertainties.
Revised: In general, weighted MOCU is strictly concave along most of the directions and only changes linearly along the directions that K ( x, π ( θ )) is constant for x ∈ X , which correspond to the queries that only reduce irrelevant uncertainties.

Original: Such a property can guarantee the convergence to the true optimal classiﬁer.
Revised: Such a property can guarantee the convergence to the true optimal classiﬁer.

Original: Before presenting the theoretical convergence guarantee of the weighted-MOCU based active learning, we summarize the computation of our weighted-MOCU based acquisition function in Algorithm 1, which can replace ELR and MOCU-based acquisition functions in Bayesian active learning algorithms with the pseudo-code given in Appendix B.
Revised: Before presenting the theoretical convergence guarantee of the weighted-MOCU based active learning, we summarize the computation of our weighted-MOCU based acquisition function in Algorithm 1, which can replace ELR and MOCU-based acquisition functions in Bayesian active learning algorithms with the pseudo-code given in Appendix B.

Original: We estimate the computational complexity of Algorithm 1 for the discrete feature and parameter spaces.
Revised: We estimate the computational complexity of Algorithm 1 for the discrete feature and parameter spaces.

Original: Assume that the size of the discrete feature space is N x = |X| and the size of the uncertainty set of classiﬁers is N θ =
Revised: Assume that the size of the discrete feature space is N x = |X| and the size of the uncertainty set of classiﬁers is N θ =

Original: | Θ | .
Revised: | Θ | .

Original: We study the complexity of calculating the weighted MOCU.
Revised: We study the complexity of calculating the weighted MOCU.

Original: In the W MOCU function, the OBC error evaluation in line 19 is called for O ( N x N θ ) times.
Revised: In the W MOCU function, the OBC error evaluation in line 19 is called for O ( N x N θ ) times.

Original: In A CQUISITION F UN , W MOCU is called for constant times.
Revised: In A CQUISITION F UN , W MOCU is called for constant times.

Original: Hence, the total complexity of calculating the acquisition function in weighted-MOCU based active learning is O ( N x N θ ) .
Revised: Hence, the total complexity of calculating the acquisition function in weighted-MOCU based active learning is O ( N x N θ ) .

Original: Compared with the ELR method, there is O ( N x ) additional computation associated with computing the weight (1 − cK ) in line 26.
Revised: Compared with the ELR method, there is O ( N x ) additional computation associated with computing the weight (1 − cK ) in line 26.

Original: Hence, the incurred computational complexity is of the same order as the original ELR and MOCU-based methods.
Revised: Hence, the incurred computational complexity is of the same order as the original ELR and MOCU-based methods.

Original: 1: function A CQUISITION F UN ( x, π θ | D , c ) 2: wmocu current = W MOCU ( π θ | D ) 3: wmocu next =4: for y in { 0 , 1 } do 5: for θ in Θ do 6:
Revised: 1: function A CQUISITION F UN ( x, π θ | D , c ) 2: wmocu current = W MOCU ( π θ | D ) 3: wmocu next =4: for y in { 0 , 1 } do 5: for θ in Θ do 6:

Original: Generate array p ( θ, y | D, x ) =
Revised: Generate array p ( θ, y | D, x ) =

Original: π θ | D · p ( y | x, θ ) 7: end for 8: p ( y | D, x )
Revised: π θ | D · p ( y | x, θ ) 7: end for 8: p ( y | D, x )

Original: = (cid:80) θ p ( θ, y | D, x ) 9: π θ | D,x,y =
Revised: = (cid:80) θ p ( θ, y | D, x ) 9: π θ | D,x,y =

Original: p
Revised: p

Original: ( θ, y | D, x ) /p
Revised: ( θ, y | D, x ) /p

Original: ( y | D, x ) 10: wmocu next = wmocu next + p
Revised: ( y | D, x ) 10: wmocu next = wmocu next + p

Original: ( y | D, x ) · W MOCU ( π θ | D,x,y , c ) 11: end for 12: return wmocu current − wmocu next 13: end function 14: function W MOCU ( π θ | D , c ) 15: wmocu =16: for x (cid:48) in X do 17: bayesian error =18: for θ in Θ do 19: bayesian error =
Revised: ( y | D, x ) · W MOCU ( π θ | D,x,y , c ) 11: end for 12: return wmocu current − wmocu next 13: end function 14: function W MOCU ( π θ | D , c ) 15: wmocu =16: for x (cid:48) in X do 17: bayesian error =18: for θ in Θ do 19: bayesian error =

Original: bayesian error
Revised: bayesian error

Original: + π θ | D · (1 − max y (cid:48) p ( y (cid:48)
Revised: + π θ | D · (1 − max y (cid:48) p ( y (cid:48)

Original: | x (cid:48) , θ )) 20: end for 21: for y (cid:48) in { 0 , 1 } do 22: p ( y (cid:48) | D, x (cid:48) )
Revised: | x (cid:48) , θ )) 20: end for 21: for y (cid:48) in { 0 , 1 } do 22: p ( y (cid:48) | D, x (cid:48) )

Original: = (cid:80)
Revised: = (cid:80)

Original: θ π θ | D · p ( y (cid:48) | x (cid:48) , θ ) 23: end for 24: obc error = 1 − max y (cid:48) p ( y (cid:48) | D, x (cid:48) ) 25:
Revised: θ π θ | D · p ( y (cid:48) | x (cid:48) , θ ) 23: end for 24: obc error = 1 − max y (cid:48) p ( y (cid:48) | D, x (cid:48) ) 25:

Original: K = obc error − bayesian error 26: wmocu = wmocu + p ( x (cid:48) ) · [(1 − cK ) K ] 27: end for 28: return wmocu 29: end function
Revised: K = obc error − bayesian error 26: wmocu = wmocu + p ( x (cid:48) ) · [(1 − cK ) K ] 27: end for 28: return wmocu 29: end function

Original: Theoretical convergence guarantee.
Revised: Theoretical convergence guarantee.

Original: Now we show that if active learning for a binary classiﬁcation problem is guided by the acquisition function deﬁned by (10) and (11), MOCU will converge to 0 almost surely and hence the procedure will converge to learning the optimal classiﬁer of the true model.
Revised: Now we show that if active learning for a binary classiﬁcation problem is guided by the acquisition function deﬁned by (10) and (11), MOCU will converge to 0 almost surely and hence the procedure will converge to learning the optimal classiﬁer of the true model.

Original: We assume that both X and Θ are discrete with ﬁnite elements; the true model parameter θ r ∈ Θ and the prior distribution π 0 ( θ ) over Θ satisﬁes π 0 ( θ r ) > 0 .
Revised: We assume that both X and Θ are discrete with ﬁnite elements; the true model parameter θ r ∈ Θ and the prior distribution π 0 ( θ ) over Θ satisﬁes π 0 ( θ r ) > 0 .

Original: We denote the posterior by π n ( θ ) and predictive probability p n ( y | x ) in the n -th weighted MOCU based active learning iteration, respectively.
Revised: We denote the posterior by π n ( θ ) and predictive probability p n ( y | x ) in the n -th weighted MOCU based active learning iteration, respectively.

Original: In the following, we give important lemmas ﬁrst.
Revised: In the following, we give important lemmas ﬁrst.

Original: All the proofs of the presented lemmas can be found in Appendix A.
Revised: All the proofs of the presented lemmas can be found in Appendix A.

Original: Lemma 

Revised: Lemma

Original: Given π ( θ ) , M ( π ( θ ))
Revised: Given π ( θ ) , M ( π ( θ ))

Original: = 

Revised: =

Original: if and only if M w ( π ( θ )) = 0 .
Revised: if and only if M w ( π ( θ )) = 0 .

Original: Lemma 1 indicates that if M w ( π ( θ )) = 0 , the OBC ψ π ( θ ) converges to the optimal classiﬁer ψ θ r as explained in the ﬁrst paragraph in Section 3.1.
Revised: Lemma 1 indicates that if M w ( π ( θ )) = 0 , the OBC ψ π ( θ ) converges to the optimal classiﬁer ψ θ r as explained in the ﬁrst paragraph in Section 3.1.

Original: Lemma 2 Deﬁne G ( x (cid:48) , π ( θ ))
Revised: Lemma 2 Deﬁne G ( x (cid:48) , π ( θ ))

Original: = (1 − cK ( x (cid:48) , π ( θ )))
Revised: = (1 − cK ( x (cid:48) , π ( θ )))

Original: K ( x (cid:48) , π ( θ )) , 0 < c ≤ 1 .
Revised: K ( x (cid:48) , π ( θ )) , 0 < c ≤ 1 .

Original: G ( x (cid:48) , π ( θ )) is a concave function of π ( θ ) .
Revised: G ( x (cid:48) , π ( θ )) is a concave function of π ( θ ) .

Original: It is important to choose a weighting scheme that renders a concave function G as it guarantees the acquisition function to be larger than or equal to 0, so that adding a new observation helps to reduce weighted MOCU to effectively guide active learning.
Revised: It is important to choose a weighting scheme that renders a concave function G as it guarantees the acquisition function to be larger than or equal to 0, so that adding a new observation helps to reduce weighted MOCU to effectively guide active learning.

Original: [Equation]
Revised: [Equation]

Original: Lemma 

Revised: Lemma

Original: At the n -th active learning iteration,if U
Revised: At the n -th active learning iteration,

Original: [Equation]

Original: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: This lemma states that if the acquisition function values of all candidates with respect to π ( θ ) are 0, the weighted MOCU is 0.
Revised: This lemma states that if the acquisition function values of all candidates with respect to π ( θ ) are 0, the weighted MOCU is 0.

Original: By Lemma 1, so is MOCU.
Revised: By Lemma 1, so is MOCU.

Original: With these, we can conclude that the OBC with respect to π ( θ ) has converged to the optimal classiﬁer.
Revised: With these, we can conclude that the OBC with respect to π ( θ ) has converged to the optimal classiﬁer.

Original: This is signiﬁcant when comparing with original ELR and MOCU-based methods as we have shown that this is not the case for them, which may get stuck earlier and therefore lose the long-term efﬁciency.
Revised: This is signiﬁcant when comparing with original ELR and MOCU-based methods as we have shown that this is not the case for them, which may get stuck earlier and therefore lose the long-term efﬁciency.

Original: Lemma 

Revised: Lemma

Original: If following some policy a candidate x is measured inﬁnitely often almost surely, then lim n →∞ U w ( x ; π n ( θ ))
Revised: If following some policy a candidate x is measured inﬁnitely often almost surely, then lim n →∞ U w ( x ; π n ( θ ))

Original: = 0 almost surely.
Revised: = 0 almost surely.

Original: Intuitively, if a candidate has been measured many times, there is no beneﬁt to measure it again.
Revised: Intuitively, if a candidate has been measured many times, there is no beneﬁt to measure it again.

Original: With these lemmas, we can prove the convergence of weighted-MOCU based active learning:
Revised: With these lemmas, we can prove the convergence of weighted-MOCU based active learning:

Original: Theorem 1 Assume that both X and Θ are discrete with ﬁnite elements, the true model parameter θ r ∈ Θ and the prior distribution π 0 ( θ ) over Θ satisﬁes
Revised: Theorem 1 Assume that both X and Θ are discrete with ﬁnite elements, the true model parameter θ r ∈ Θ and the prior distribution π 0 ( θ ) over Θ satisﬁes

Original: π 0 ( θ r ) > 0 ; then for the active learning algorithm deﬁned by the acquisition function (10), we have lim n →∞ M ( π n ( θ ))
Revised: π 0 ( θ r ) > 0 ; then for the active learning algorithm deﬁned by the acquisition function (10), we have lim n →∞ M ( π n ( θ ))

Original: = 0 almost surely.
Revised: = 0 almost surely.

Original: Proof.
Revised: Proof.

Original: As the number of active learning iterations n → ∞ , following the acquisition function (10), some of the candidates can be measured inﬁnite times.
Revised: As the number of active learning iterations n → ∞ , following the acquisition function (10), some of the candidates can be measured inﬁnite times.

Original: Deﬁne X A ⊂ X as the set whose candidates have been measured inﬁnite times.
Revised: Deﬁne X A ⊂ X as the set whose candidates have been measured inﬁnite times.

Original: Denote the measuring sequence of the candidates following (10) as { x n } , we have: ∃ N, s.t. ∀ n > N, x n ∈ X A .
Revised: Denote the measuring sequence of the candidates following (10) as { x n } , we have: ∃ N, s.t. ∀ n > N, x n ∈ X A .

Original: Based on Lemma 5,
Revised: Based on Lemma 5,

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: On the other hand, since with the weighted MOCU
Revised: On the other hand, since with the weighted MOCU

Original: U w ( x n ; π n ( θ ))
Revised: U w ( x n ; π n ( θ ))

Original: =
Revised: =

Original: max x ∈X U w ( x ; π n ( θ )) , then lim n →∞ U w ( x n ; π n ( θ ))
Revised: max x ∈X U w ( x ; π n ( θ )) , then lim n →∞ U w ( x n ; π n ( θ ))

Original: = 0 indicates that ∀ x ∈ X ,
Revised: = 0 indicates that ∀ x ∈ X ,

Original: U w ( x ; π n ( θ )) uniformly converges to 0.
Revised: U w ( x ; π n ( θ )) uniformly converges to 0.

Original: Based on
Revised: Based on

Original: Lemma 4, lim n →∞ M wn
Revised: Lemma 4, lim n →∞ M wn

Original: = 

Revised: =

Original: and we can conclude the proof with Lemma 1.
Revised: and we can conclude the proof with Lemma 1.

Original:  E MPIRICAL RESULTS
Revised:  E MPIRICAL RESULTS

Original: We benchmark our weighted-MOCU method with other active learning algorithms, including random sampling, MES (Sebastiani & Wynn, 2000), BALD (Houlsby et al., 2011) and ELR (Roy & McCallum, 2001), on both simulated and real-world classiﬁcation datasets.
Revised: We benchmark our weighted-MOCU method with other active learning algorithms, including random sampling, MES (Sebastiani & Wynn, 2000), BALD (Houlsby et al., 2011) and ELR (Roy & McCallum, 2001), on both simulated and real-world classiﬁcation datasets.

Original: In the following experiments, we set c = 1 for the weighted MOCU function.
Revised: In the following experiments, we set c = 1 for the weighted MOCU function.

Revised: The code for our experiments is made available at https://github.com/QianLab/WMOCU_AL .

Original: Simulated experiments.
Revised: Simulated experiments.

Original: In addition to the one-dimensional simulated example introduced in Section 1, we test our model on a similar simulation setting as the block in the middle dataset in (Houlsby et al., 2011), where noisy observations with ﬂip error are simulated in a block region on the decision boundary.
Revised: In addition to the one-dimensional simulated example introduced in Section 1, we test our model on a similar simulation setting as the block in the middle dataset in (Houlsby et al., 2011), where noisy observations with ﬂip error are simulated in a block region on the decision boundary.

Original: We generate data based on a two-dimensional Bayesian logistic regression model: p ( y = 1 | x , w , b ) = 1 1+exp( − w T x − b ) with x ∈ [ − 4 , 4] 2 .
Revised: We generate data based on a two-dimensional Bayesian logistic regression model: p ( y = 1 | x , w , b ) = 1 1+exp( − w T x − b ) with x ∈ [ − 4 , 4] 2 .

Original: The block region is within [ − 0 .
Revised: The block region is within [ − 0 .

Original: 5]with the ﬂip error rate equal to 0.3.
Revised: 5]with the ﬂip error rate equal to 0.3.

Original: For the model parameter prior, w 1 ∼ U (0 . 3 , 0 .
Revised: For the model parameter prior, w 1 ∼ U (0 . 3 , 0 .

Original: 8) is uniformly distributed and w 2 ∼ U ( − 0 .
Revised: 8) is uniformly distributed and w 2 ∼ U ( − 0 .

Original: 25 , 0 . 25) and b ∼ U ( − 0 .
Revised: 25 , 0 . 25) and b ∼ U ( − 0 .

Original: 25 , 0 . 25) ; w 1 , w 2 and b are independent.
Revised: 25 , 0 . 25) ; w 1 , w 2 and b are independent.

Original: We randomly sample 100 particles from the parameter prior with one of the particles as the true model parameter.
Revised: We randomly sample 100 particles from the parameter prior with one of the particles as the true model parameter.

Original: The ﬁve active learning algorithms are compared for 500 iterations by the OBC error with respect to the testing data generated from the true model.
Revised: The ﬁve active learning algorithms are compared for 500 iterations by the OBC error with respect to the testing data generated from the true model.

Original: We repeat the simulations for 500 runs and plot the average performance with standard deviation bars in Fig.
Revised: We repeat the simulations for 500 runs and plot the average performance with standard deviation bars in Fig.

Original: The error regret is deﬁned as the error difference between the OBC and the true optimal classiﬁer.
Revised: The error regret is deﬁned as the error difference between the OBC and the true optimal classiﬁer.

Original: From the ﬁgure,
Revised: From the ﬁgure,

Original: [Figure]
Revised: [Figure]

Original: r r
Revised: r r

Original: [Figure]
Revised: [Figure]

Original: g
Revised: g

Original: [Figure]
Revised: [Figure]

Original: e t randomMESBALDELRWeighted
Revised: e t randomMESBALDELRWeighted

Original: [Figure]
Revised: [Figure]

Original: MES simply chooses the candidates with the predictive probability closest to 0.5, it can sample many noisy observations from the block region.
Revised: MES simply chooses the candidates with the predictive probability closest to 0.5, it can sample many noisy observations from the block region.

Original: ELR performs well in the ﬁrst several iterations but poorly after 200 samples.
Revised: ELR performs well in the ﬁrst several iterations but poorly after 200 samples.

Original: Our weighted MOCU performs the best.
Revised: Our weighted MOCU performs the best.

Original: We have also benchmarked our weighted-MOCU based method with other active learning methods for a synthetic multi-class classiﬁcation problem.
Revised: We have also benchmarked our weighted-MOCU based method with other active learning methods for a synthetic multi-class classiﬁcation problem.

Original: We assume that the probabilistic model p ( y | x , σ 2 y ) =
Revised: We assume that the probabilistic model p ( y | x , σ 2 y ) =

Original: f y ( x , σ 2 y ) / (cid:80) y (cid:48)
Revised: f y ( x , σ 2 y ) / (cid:80) y (cid:48)

Original: f ( x , σ 2 y (cid:48) ) with x ∈ [ − 2 , 2] 2 , y ∈ { 0 , 1 , 2 } and f y = exp( − ( x − m y ) 2 / 2 σ 2 y ) .
Revised: f ( x , σ 2 y (cid:48) ) with x ∈ [ − 2 , 2] 2 , y ∈ { 0 , 1 , 2 } and f y = exp( − ( x − m y ) 2 / 2 σ 2 y ) .

Original: We set m y to be (0 , 0) , (1 , 0) , (0 , 1) for y = 0 , 1 , 2 respectively; and σ 2 y ∼ U (1 , 5) being the uncertain parameters.
Revised: We set m y to be (0 , 0) , (1 , 0) , (0 , 1) for y = 0 , 1 , 2 respectively; and σ 2 y ∼ U (1 , 5) being the uncertain parameters.

Original: the previous binary classiﬁcation experiment, we test for 300 runs and plot the average performance with standard deviations in Fig.
Revised: the previous binary classiﬁcation experiment, we test for 300 runs and plot the average performance with standard deviations in Fig.

Original: We can observe that ELR performs poorly in the long run while our Weighted MOCU has better empirical performance on par with BALD
Revised: We can observe that ELR performs poorly in the long run while our Weighted MOCU has better empirical performance on par with BALD

Original: More results and discussion are in Appendix D & E.
Revised: More results and discussion are in Appendix D & E.

Original: [Figure]
Revised: [Figure]

Original: e rr o r randomMESBALDELRWeighted _MOCU
Revised: e rr o r randomMESBALDELRWeighted _MOCU

Original: Real-world benchmark experiments.
Revised: Real-world benchmark experiments.

Original: We also present the results on the UCI User Knowledge dataset (Kahraman et al., 2013).
Revised: We also present the results on the UCI User Knowledge dataset (Kahraman et al., 2013).

Original: The dataset includes 403 samples assigned to 4 classes (High,
Revised: The dataset includes 403 samples assigned to 4 classes (High,

Original: Medium, Low, Very Low) with each sample having ﬁve features in [0 , 1] 5 .
Revised: Medium, Low, Very Low) with each sample having ﬁve features in [0 , 1] 5 .

Original: We have grouped the samples into two classes with 224 samples in High or
Revised: We have grouped the samples into two classes with 224 samples in High or

Original: Medium, 179 in Low or Very Low.
Revised: Medium, 179 in Low or Very Low.

Original: We consider the ﬁrst and ﬁfth features for classiﬁcation and equally divide the feature space into 4 × 4 bins.
Revised: We consider the ﬁrst and ﬁfth features for classiﬁcation and equally divide the feature space into 4 × 4 bins.

Original: For the i -th bin, the probability of candidates belonging to High or Medium is denoted by θ i , 1 ≤ i ≤ 16 and θ i ’s are independent and θ i ∼ Beta ( α i , β i ) , with hyperparameters
Revised: For the i -th bin, the probability of candidates belonging to High or Medium is denoted by θ i , 1 ≤ i ≤ 16 and θ i ’s are independent and θ i ∼ Beta ( α i , β i ) , with hyperparameters

Original: α i and β i .
Revised: α i and β i .

Original: We present the results with the uncertainty class by setting α i =
Revised: We present the results with the uncertainty class by setting α i =

Original: β i = 10 in eight randomly chosen bins and for the other bins, α i = 5 , β i = 

Revised: β i = 10 in eight randomly chosen bins and for the other bins, α i = 5 , β i =

Original: if the true frequency of High or
Revised: if the true frequency of High or

Original: Medium in the i -th bin is lower than 0.5 and α i = 2 , β i = 5 otherwise.
Revised: Medium in the i -th bin is lower than 0.5 and α i = 2 , β i = 5 otherwise.

Original: We have randomly drawn 150 samples from each class as the candidate pool and perform the ﬁve different active learning algorithms.
Revised: We have randomly drawn 150 samples from each class as the candidate pool and perform the ﬁve different active learning algorithms.

Original: We repeat the whole procedure 150 times and the average error rates are shown in Fig.
Revised: We repeat the whole procedure 150 times and the average error rates are shown in Fig.

Original: While ELR clearly gets stuck in this setup, our Weighted MOCU method can converge to the optimal classiﬁer with less samples than all the competing methods.
Revised: While ELR clearly gets stuck in this setup, our Weighted MOCU method can converge to the optimal classiﬁer with less samples than all the competing methods.

Original: BALD performs poorly as the bins with α = β
Revised: BALD performs poorly as the bins with α = β

Original: = 10 have less uncertainty but have more impact on OBC prediction and BALD fails to identify that.
Revised: = 10 have less uncertainty but have more impact on OBC prediction and BALD fails to identify that.

Original: More comprehensive results and discussion, including results on the UCI Letter
Revised: More comprehensive results and discussion, including results on the UCI Letter

Original: Recognition dataset (Dua & Graff, 2017), can be found in Appendix F.
Revised: Recognition dataset (Dua & Graff, 2017), can be found in Appendix F.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have identiﬁed potential convergence problems of existing ELR methods and proposed a novel active learning strategy for classiﬁcation based on weighted MOCU.
Revised: We have identiﬁed potential convergence problems of existing ELR methods and proposed a novel active learning strategy for classiﬁcation based on weighted MOCU.

Original: Our weighted MOCU directly targets at decreasing the classiﬁcation error and ignores uncertainty irrelevant to the classiﬁcation performance.
Revised: Our weighted MOCU directly targets at decreasing the classiﬁcation error and ignores uncertainty irrelevant to the classiﬁcation performance.

Original: More critically, it can capture continuous change in objective-relevant uncertainty.
Revised: More critically, it can capture continuous change in objective-relevant uncertainty.

Original: Hence, our new active learning can be efﬁcient both at the beginning and in the long run with the guarantee of converging to the optimal classiﬁer.
Revised: Hence, our new active learning can be efﬁcient both at the beginning and in the long run with the guarantee of converging to the optimal classiﬁer.

Original: Empirical results have demonstrated active learning guided by weighted MOCU leads to sample-efﬁcient learning.
Revised: Empirical results have demonstrated active learning guided by weighted MOCU leads to sample-efﬁcient learning.

Original: Future work includes theoretical analysis of MOCU-guided active learning for multi-class classiﬁcation, as well as developing optimization methods for active learning in continuous space.
Revised: Future work includes theoretical analysis of MOCU-guided active learning for multi-class classiﬁcation, as well as developing optimization methods for active learning in continuous space.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: X. Qian was supported in part by the National Science Foundation (NSF) Awards 1553281, 1812641, and 1934904.
Revised: X. Qian was supported in part by the National Science Foundation (NSF) Awards 1553281, 1812641, and 1934904.

Original: B.-J. Yoon was supported in part by the NSF Award 1835690.
Revised: B.-J. Yoon was supported in part by the NSF Award 1835690.

Original: The work of E. R. Dougherty and F. J. Alexander was supported by the U.S. Department of Energy, Ofﬁce of
Revised: The work of E. R. Dougherty and F. J. Alexander was supported by the U.S. Department of Energy, Ofﬁce of

Original: Science, Ofﬁce of Advanced Scientiﬁc Computing Research, Mathematical Multifaceted Integrated
Revised: Science, Ofﬁce of Advanced Scientiﬁc Computing Research, Mathematical Multifaceted Integrated

Original: Capability Centers program under Award DE-SC
Revised: Capability Centers program under Award DE-SC

Original: A. P ROOFS OF LEMMAS
Revised: A. P ROOFS OF LEMMAS

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: Based on (3), since C θ ( ψ π ( θ ) , x (cid:48) ) − C θ ( ψ θ , x (cid:48) ) ≥ 0 , so M ( π ( θ ))
Revised: Based on (3), since C θ ( ψ π ( θ ) , x (cid:48) ) − C θ ( ψ θ , x (cid:48) ) ≥ 0 , so M ( π ( θ ))

Original: = 

Revised: =

Original: iff
Revised: iff

Original: C θ ( ψ π ( θ ) , x (cid:48) )
Revised: C θ ( ψ π ( θ ) , x (cid:48) )

Original: − C θ ( ψ θ , x (cid:48) )
Revised: − C θ ( ψ θ , x (cid:48) )

Original: =
Revised: =

Original: 0 ∀ x (cid:48)
Revised: 0 ∀ x (cid:48)

Original: ∈ X , ∀ θ ∈ supp ( π ) .
Revised: ∈ X , ∀ θ ∈ supp ( π ) .

Original: In addition, in (9), w ( π ( θ ) , x (cid:48) , θ ) > 0 , so
Revised: In addition, in (9), w ( π ( θ ) , x (cid:48) , θ ) > 0 , so

Original: C θ ( ψ π ( θ ) , x (cid:48) )
Revised: C θ ( ψ π ( θ ) , x (cid:48) )

Original: − C θ ( ψ θ , x (cid:48) )
Revised: − C θ ( ψ θ , x (cid:48) )

Original: =
Revised: =

Original: 0 ∀ x (cid:48)
Revised: 0 ∀ x (cid:48)

Original: ∈ X , ∀ θ ∈ supp ( π ) iff M w ( π ( θ ))
Revised: ∈ X , ∀ θ ∈ supp ( π ) iff M w ( π ( θ ))

Original: = 0 , which concludes the proof.
Revised: = 0 , which concludes the proof.

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: In the following proof, we omit the argument x (cid:48) in G and K for simplicity.
Revised: In the following proof, we omit the argument x (cid:48) in G and K for simplicity.

Original: Owing to the concavity of the min operator, min y (cid:48)
Revised: Owing to the concavity of the min operator, min y (cid:48)

Original: E π ( θ ) [1 − p ( y (cid:48)
Revised: E π ( θ ) [1 − p ( y (cid:48)

Original: | x (cid:48) , θ )] is a concave function of π ( θ ) .
Revised: | x (cid:48) , θ )] is a concave function of π ( θ ) .

Original: With E π ( θ ) [min y (cid:48) (1 − p ( y (cid:48) | x (cid:48) , θ )] being a linear function of π ( θ ) , based on (13), K ( π ( θ )) equals to a concave function subtracting a linear function and thus is also a concave function.
Revised: With E π ( θ ) [min y (cid:48) (1 − p ( y (cid:48) | x (cid:48) , θ )] being a linear function of π ( θ ) , based on (13), K ( π ( θ )) equals to a concave function subtracting a linear function and thus is also a concave function.

Original: As analyzed in Section 3.2, 0 ≤ K ( π ( θ )) ≤ 0 .
Revised: As analyzed in Section 3.2, 0 ≤ K ( π ( θ )) ≤ 0 .

Original: We deﬁne T ( κ ) =
Revised: We deﬁne T ( κ ) =

Original: (1 − cκ ) κ, κ ∈ [0 , 0 .
Revised: (1 − cκ ) κ, κ ∈ [0 , 0 .

Original: 5] , a strictly increasing and strictly concave function with 0 < c ≤ 1 .
Revised: 5] , a strictly increasing and strictly concave function with 0 < c ≤ 1 .

Original: G ( π ( θ ))
Revised: G ( π ( θ ))

Original: =
Revised: =

Original: T [ K ( π ( θ ))] is a composite function of T and K .
Revised: T [ K ( π ( θ ))] is a composite function of T and K .

Original: So we conclude the proof with the property of the concavity for the composite functions:
Revised: So we conclude the proof with the property of the concavity for the composite functions:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality is because T is increasing and K is concave; and the second inequality holds as
Revised: The ﬁrst inequality is because T is increasing and K is concave; and the second inequality holds as

Original: T is a concave function.
Revised: T is a concave function.

Original: Proof of Lemma 3.
Revised: Proof of Lemma 3.

Original: Since π ( θ ) = (cid:80) y p ( y | x ) π ( θ | x, y ) , by Jensen’s inequality, we have
Revised: Since π ( θ ) = (cid:80) y p ( y | x ) π ( θ | x, y ) , by Jensen’s inequality, we have

Original: G ( x (cid:48) , π ( θ )) ≥ E y | x [ G ( x (cid:48) , π ( θ | x, y ))] as G is a concave function.
Revised: G ( x (cid:48) , π ( θ )) ≥ E y | x [ G ( x (cid:48) , π ( θ | x, y ))] as G is a concave function.

Original: So the weighted MOCU acquisition function:
Revised: So the weighted MOCU acquisition function:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 4.
Revised: Proof of Lemma 4.

Original: We will prove the contrapositive of the lemma: assuming M w ( π n ( θ )) > 0 , ∃ x ∈ X s.t.
Revised: We will prove the contrapositive of the lemma: assuming M w ( π n ( θ )) > 0 , ∃ x ∈ X s.t.

Original: U w ( x ; π n ( θ ))
Revised: U w ( x ; π n ( θ ))

Original: > 0 .
Revised: > 0 .

Original: Based on (15), M w ( π n ( θ )) > 0 indicating
Revised: Based on (15), M w ( π n ( θ )) > 0 indicating

Original: ∃ x ∈ X s.t. K ( x, π n ( θ ))
Revised: ∃ x ∈ X s.t. K ( x, π n ( θ ))

Original: > 0 .
Revised: > 0 .

Original: It is sufﬁcient to show that if K ( x, π n ( θ )) > 0 , then U w ( x ; π n ( θ )) > 0 .
Revised: It is sufﬁcient to show that if K ( x, π n ( θ )) > 0 , then U w ( x ; π n ( θ )) > 0 .

Original: To prove that, we only need to prove
Revised: To prove that, we only need to prove

Original: G ( x, π n ( θ )) > E p n ( y | x ) [ G ( x, π n ( θ | x, y ))] ; then by (17), U w ( x ; π n ( θ )) > 0 .
Revised: G ( x, π n ( θ )) > E p n ( y | x ) [ G ( x, π n ( θ | x, y ))] ; then by (17), U w ( x ; π n ( θ )) > 0 .

Original: Since G is a concave function, we know G ( x, π n ( θ )) ≥ E p n ( y | x ) [ G ( x, π n ( θ | x, y ))] .
Revised: Since G is a concave function, we know G ( x, π n ( θ )) ≥ E p n ( y | x ) [ G ( x, π n ( θ | x, y ))] .

Original: With π n ( θ ) = (cid:80) y
Revised: With π n ( θ ) = (cid:80) y

Original: p
Revised: p

Original: n ( y | x ) π n ( θ | x, y ) , we can rewrite (16) as:
Revised: n ( y | x ) π n ( θ | x, y ) , we can rewrite (16) as:

Original: [Equation]
Revised: [Equation]

Original: The second equality holds only if ∀ y ∈ { 0 , 1 } , K ( x, π n ( θ | x, y ))
Revised: The second equality holds only if ∀ y ∈ { 0 , 1 } , K ( x, π n ( θ | x, y ))

Original: =
Revised: =

Original: K ( x, π n ( θ )) , which means that to prove G ( x, π n ( θ ))
Revised: K ( x, π n ( θ )) , which means that to prove G ( x, π n ( θ ))

Original: > E y | x [ G ( x, π n ( θ | x, y ))] , we just need to show ∃ y ∈ { 0 , 1 } , K ( x, π n ( θ | x, y ))
Revised: > E y | x [ G ( x, π n ( θ | x, y ))] , we just need to show ∃ y ∈ { 0 , 1 } , K ( x, π n ( θ | x, y ))

Original: (cid:54) =
Revised: (cid:54) =

Original: K ( x, π n ( θ )) .
Revised: K ( x, π n ( θ )) .

Original: In the following proof, we will show if K ( x, π n ( θ )) > 0 , then ∃ y ∈ { 0 , 1 } ,
Revised: In the following proof, we will show if K ( x, π n ( θ )) > 0 , then ∃ y ∈ { 0 , 1 } ,

Original: s.t. K
Revised: s.t. K

Original: ( x, π n ( θ | x, y )) (cid:54) =
Revised: ( x, π n ( θ | x, y )) (cid:54) =

Original: K ( x, π n ( θ )) .
Revised: K ( x, π n ( θ )) .

Original: Denote ˆ y = arg max y p n ( y | x ) .
Revised: Denote ˆ y = arg max y p n ( y | x ) .

Original: By (14) we have:
Revised: By (14) we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since K ( x, π n ( θ )) > 0 , the parameter set Θ no = { θ ∈ supp ( π n ) : arg max y p ( y | x, θ ) (cid:54)
Revised: Since K ( x, π n ( θ )) > 0 , the parameter set Θ no = { θ ∈ supp ( π n ) : arg max y p ( y | x, θ ) (cid:54)

Original: =
Revised: =

Original: ˆ y } is not empty.
Revised: ˆ y } is not empty.

Original: We only keep the nonzero terms in K :
Revised: We only keep the nonzero terms in K :

Original: [Equation]
Revised: [Equation]

Original: For binary classiﬁcation, ˆ y = arg max y p n ( y | x ) , indicating that the predictive probability p n (ˆ y | x ) ≥ 0 .
Revised: For binary classiﬁcation, ˆ y = arg max y p n ( y | x ) , indicating that the predictive probability p n (ˆ y | x ) ≥ 0 .

Original: For θ ∈ Θ o , p (ˆ y | x, θ ) < 0 .
Revised: For θ ∈ Θ o , p (ˆ y | x, θ ) < 0 .

Original: 5 , we have: if θ ∈ Θ o , π n ( θ | x, ˆ y ) =
Revised: 5 , we have: if θ ∈ Θ o , π n ( θ | x, ˆ y ) =

Original: π n ( θ ) p (ˆ y | x,θ ) p n (ˆ y | x ) <
Revised: π n ( θ ) p (ˆ y | x,θ ) p n (ˆ y | x ) <

Original: π n ( θ ) .
Revised: π n ( θ ) .

Original: If we observe ( x, ˆ y ) in ( n + 1) -th iteration, the updated posterior predictive probability p n (ˆ y | x, { x, ˆ y } ) ≥ p n (ˆ y | x ) ≥ 0 . 5 and therefore max y p n ( y | x, { x, ˆ y } )
Revised: If we observe ( x, ˆ y ) in ( n + 1) -th iteration, the updated posterior predictive probability p n (ˆ y | x, { x, ˆ y } ) ≥ p n (ˆ y | x ) ≥ 0 . 5 and therefore max y p n ( y | x, { x, ˆ y } )

Original: =
Revised: =

Original: ˆ y .
Revised: ˆ y .

Original: Hence,
Revised: Hence,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since K ( π n ( θ | x, ˆ y ) , x ) (cid:54) =
Revised: Since K ( π n ( θ | x, ˆ y ) , x ) (cid:54) =

Original: K ( π n ( θ ) , x ) , we have G ( x, π n ( θ ))
Revised: K ( π n ( θ ) , x ) , we have G ( x, π n ( θ ))

Original: >
Revised: >

Original: E p n ( y | x ) [ G ( x, π n ( θ | x, y ))] and
Revised: E p n ( y | x ) [ G ( x, π n ( θ | x, y ))] and

Original: [Equation]
Revised: [Equation]

Original: This concludes our proof.
Revised: This concludes our proof.

Original: Proof of Lemma 5.
Revised: Proof of Lemma 5.

Original: Adding a new data point ( x, y ) to D , the posterior change is: π n ( θ | x, y ) =
Revised: Adding a new data point ( x, y ) to D , the posterior change is: π n ( θ | x, y ) =

Original: π n ( θ ) p ( y | x,θ ) p n ( y | x ) .
Revised: π n ( θ ) p ( y | x,θ ) p n ( y | x ) .

Original: Deﬁne Θ x = { θ ∈ Θ : p ( y | x, θ ) = p ( y | x, θ r ) } .
Revised: Deﬁne Θ x = { θ ∈ Θ : p ( y | x, θ ) = p ( y | x, θ r ) } .

Original: Denote N x ( n ) as the times of the candidate x being queried at the n -th iteration.
Revised: Denote N x ( n ) as the times of the candidate x being queried at the n -th iteration.

Original: Based on the posterior consistency theory we have (cid:80) θ ∈ Θ x π n ( θ ) a.s. −−→ 1 as N x ( n ) → ∞ (Gelman et al., 2013).
Revised: Based on the posterior consistency theory we have (cid:80) θ ∈ Θ x π n ( θ ) a.s. −−→ 1 as N x ( n ) → ∞ (Gelman et al., 2013).

Original: Since p n ( y | x ) = (cid:80) θ ∈ Θ π n ( θ ) p ( y | x, θ ) , we have lim n →∞ p n ( y | x ) a.s. −−→ p ( y | x, θ r ) .
Revised: Since p n ( y | x ) = (cid:80) θ ∈ Θ π n ( θ ) p ( y | x, θ ) , we have lim n →∞ p n ( y | x ) a.s. −−→ p ( y | x, θ r ) .

Original: Hence lim n →∞ π n ( θ | x, y ) − π n ( θ ) = 0 almost surely, which indicates lim n →∞ U w ( x ; π n ( θ ))
Revised: Hence lim n →∞ π n ( θ | x, y ) − π n ( θ ) = 0 almost surely, which indicates lim n →∞ U w ( x ; π n ( θ ))

Original: = 0 almost surely.
Revised: = 0 almost surely.

Original: B. W EIGHTED -MOCU BASED ACTIVE LEARNING & COMPUTATIONAL COMPLEXITY
Revised: B. W EIGHTED -MOCU BASED ACTIVE LEARNING & COMPUTATIONAL COMPLEXITY

Original: The pseudo-code of the general active learning procedure is provided in Algorithm 2.
Revised: The pseudo-code of the general active learning procedure is provided in Algorithm 2.

Original: The function
Revised: The function

Original: A CQUISITION F UN can be acquisition functions of various methods, including weighted-MOCU,
Revised: A CQUISITION F UN can be acquisition functions of various methods, including weighted-MOCU,

Original: ELR, BALD, etc.
Revised: ELR, BALD, etc.

Original: Computational complexity We study the complexity of the complete active learning procedure.
Revised: Computational complexity We study the complexity of the complete active learning procedure.

Original: As we analyzed in the main text for the computation of the weighted MOCU acquisition function, the W MOCU function is called for O ( N x N θ ) times.
Revised: As we analyzed in the main text for the computation of the weighted MOCU acquisition function, the W MOCU function is called for O ( N x N θ ) times.

Original: In A CQUISITION F UN , W MOCU is called for constant times.
Revised: In A CQUISITION F UN , W MOCU is called for constant times.

Original: Finally, in the main procedure, in each iteration, A CQUISITION F UN is called for each x .
Revised: Finally, in the main procedure, in each iteration, A CQUISITION F UN is called for each x .

Original: Hence, the total complexity of Weighted MOCU-based active learning is O ( TN 2 x N θ ) .
Revised: Hence, the total complexity of Weighted MOCU-based active learning is O ( TN 2 x N θ ) .

Original: [Figure]
Revised: [Figure]

Original: m u t
Revised: m u t

Original: u a l i n f o r m a t
Revised: u a l i n f o r m a t

Original: i o n (a) Acquisition function of BALD
Revised: i o n (a) Acquisition function of BALD

Original: [Figure]
Revised: [Figure]

Original: d e s i g n v a l u e (
Revised: d e s i g n v a l u e (

Original: [Figure]
Revised: [Figure]

Original: Acquisition function of ELR
Revised: Acquisition function of ELR

Original: C. D ETAILS OF THE ONE - DIMENSIONAL ACTIVE LEARNING EXAMPLE IN I NTRODUCTION
Revised: C. D ETAILS OF THE ONE - DIMENSIONAL ACTIVE LEARNING EXAMPLE IN I NTRODUCTION

Original: The example in the Introduction of the main text is a binary classiﬁcation problem with y ∈ { 0 , 1 } based on only one feature x ∈ [ − 4 , 4] .
Revised: The example in the Introduction of the main text is a binary classiﬁcation problem with y ∈ { 0 , 1 } based on only one feature x ∈ [ − 4 , 4] .

Original: The underlying discriminative model is based on:
Revised: The underlying discriminative model is based on:

Original: [Equation]
Revised: [Equation]

Original: 1: function M AIN P ROCEDURE ( ) 2:
Revised: 1: function M AIN P ROCEDURE ( ) 2:

Original: Set a discrete candidate set X , the probability array p x , and iteration number T 3:
Revised: Set a discrete candidate set X , the probability array p x , and iteration number T 3:

Original: Set the discrete parameter set Θ and the corresponding probability array π θ 4:
Revised: Set the discrete parameter set Θ and the corresponding probability array π θ 4:

Original: Initialize the data set D = ∅ 5: π θ | D = π θ 6: for t = 1 to T do 7: for x in X do 8:
Revised: Initialize the data set D = ∅ 5: π θ | D = π θ 6: for t = 1 to T do 7: for x in X do 8:

Original: Store A CQUISITION F UN ( x, π θ | D ) to the array U X 9: end for 10:
Revised: Store A CQUISITION F UN ( x, π θ | D ) to the array U X 9: end for 10:

Original: Optimize U X
Revised: Optimize U X

Original: and ﬁnd the maximum point x ∗ 11:
Revised: and ﬁnd the maximum point x ∗ 11:

Original: Obtain the label y ∗ corresponds to x ∗ and update D = D ∪ { x ∗ , y ∗ } 12: for θ in Θ do 13:
Revised: Obtain the label y ∗ corresponds to x ∗ and update D = D ∪ { x ∗ , y ∗ } 12: for θ in Θ do 13:

Original: Update π θ | D ∝ π θ | D · p ( y ∗ | x ∗ , θ ) 14: end for 15:
Revised: Update π θ | D ∝ π θ | D · p ( y ∗ | x ∗ , θ ) 14: end for 15:

Original: X =
Revised: X =

Original: X / { x ∗ } 16: end for 17: end function where θ =
Revised: X / { x ∗ } 16: end for 17: end function where θ =

Original: ( a, b ) T is the uncertain parameter vector, with a and b independently uniformly distributed on the intervals [ − 0 . 1 , 0 . 1] and [ − 0 .
Revised: ( a, b ) T is the uncertain parameter vector, with a and b independently uniformly distributed on the intervals [ − 0 . 1 , 0 . 1] and [ − 0 .

Original: 2 , 0 . 2] respectively.
Revised: 2 , 0 . 2] respectively.

Original: The discriminative model equals to a sigmoid function S ( x ) plus the perturbation (cid:15) ( x, a, b ) , a mixed Gaussian function that changes with a and b , the uncertainty class of classiﬁers can be constructed by such deviations on S ( x ) .
Revised: The discriminative model equals to a sigmoid function S ( x ) plus the perturbation (cid:15) ( x, a, b ) , a mixed Gaussian function that changes with a and b , the uncertainty class of classiﬁers can be constructed by such deviations on S ( x ) .

Original: The discriminative model has higher uncertainty near x ± 4 , which depends on the value of b , than the uncertainty near x = 0 , which depends on a .
Revised: The discriminative model has higher uncertainty near x ± 4 , which depends on the value of b , than the uncertainty near x = 0 , which depends on a .

Original: Value of a and b has negligible inﬂuence on the p ( y = 1 | x ) near x ± 4 and x = 0 , respectively.
Revised: Value of a and b has negligible inﬂuence on the p ( y = 1 | x ) near x ± 4 and x = 0 , respectively.

Original: So by observing data at x = ± 4 , the uncertainty on p c ( y = 1 | x = 0) will not be reduced signiﬁcantly.
Revised: So by observing data at x = ± 4 , the uncertainty on p c ( y = 1 | x = 0) will not be reduced signiﬁcantly.

Original: In Figs.
Revised: In Figs.

Original: S1(a) and (b), we show how the acquisition functions of BALD and ELR change with respect to x .
Revised: S1(a) and (b), we show how the acquisition functions of BALD and ELR change with respect to x .

Original: It is clear that the acquisition function of BALD at x = ± 4 should have the largest value since p ( y | x ) has the highest uncertainty.
Revised: It is clear that the acquisition function of BALD at x = ± 4 should have the largest value since p ( y | x ) has the highest uncertainty.

Original: On the other hand, since p ( y | x = ± 4) is always above or below 0.5, the speciﬁc value of p ( y | x, a, b ) will not affect the corresponding optimal Bayesian classiﬁer (OBC) and therefore the loss reduction in Fig.
Revised: On the other hand, since p ( y | x = ± 4) is always above or below 0.5, the speciﬁc value of p ( y | x, a, b ) will not affect the corresponding optimal Bayesian classiﬁer (OBC) and therefore the loss reduction in Fig.

Original: S1(b) at x = ± 4 is always 0; the acquisition function of ELR around x = 0 can be larger than 0, since knowing the speciﬁc value of p ( y | x = 0 , a, b ) can reduce the classiﬁcation error.
Revised: S1(b) at x = ± 4 is always 0; the acquisition function of ELR around x = 0 can be larger than 0, since knowing the speciﬁc value of p ( y | x = 0 , a, b ) can reduce the classiﬁcation error.

Original: D. D ETAILS OF THE BINARY CLASSIFICATION EXAMPLE IN S ECTION 3. 
Revised: D. D ETAILS OF THE BINARY CLASSIFICATION EXAMPLE IN S ECTION 3.

Original: In the binary classiﬁcation problem, Θ = { θ 1 , θ 2 } , X = { x 1 , x 2 } .
Revised: In the binary classiﬁcation problem, Θ = { θ 1 , θ 2 } , X = { x 1 , x 2 } .

Original: The probabilistic model setting for the two candidates is symmetric:
Revised: The probabilistic model setting for the two candidates is symmetric:

Original: [Equation]
Revised: [Equation]

Original: There are three intervals corresponding to the linear function pieces of MOCU in Fig. 2: [0 , 0 .
Revised: There are three intervals corresponding to the linear function pieces of MOCU in Fig. 2: [0 , 0 .

Original: 67] and (0 .
Revised: 67] and (0 .

Original: In the three intervals, ψ π ( θ ) ( x 1 ) , the OBC predictions of x 1 are 1, 1 and 0, respectively; ψ π ( θ ) ( x 2 ) , the OBC predictions of x 2 are 1, 0, and 0, respectively.
Revised: In the three intervals, ψ π ( θ ) ( x 1 ) , the OBC predictions of x 1 are 1, 1 and 0, respectively; ψ π ( θ ) ( x 2 ) , the OBC predictions of x 2 are 1, 0, and 0, respectively.

Original: In Fig.
Revised: In Fig.

Original: 2 we set the prior ˜ π
Revised: 2 we set the prior ˜ π

Original: ( θ 1 ) = 0 .
Revised: ( θ 1 ) = 0 .

Original: 15 , then based on the Bayes’s rule we can obtain the posterior with the observations of ( x 1 , y 1 ) .
Revised: 15 , then based on the Bayes’s rule we can obtain the posterior with the observations of ( x 1 , y 1 ) .

Original: Based on the observation result of y 1 , the posteriors are ˜ π ( θ 1 | x 1 , y 1 = 0) = 0 .
Revised: Based on the observation result of y 1 , the posteriors are ˜ π ( θ 1 | x 1 , y 1 = 0) = 0 .

Original: 2609 and ˜ π ( θ 1 | x 1 , y 1 = 1) = 0 .
Revised: 2609 and ˜ π ( θ 1 | x 1 , y 1 = 1) = 0 .

Original: 0916 , both of which fall into the ﬁrst linear piece of MOCU.
Revised: 0916 , both of which fall into the ﬁrst linear piece of MOCU.

Original: E. M ULTI - CLASS CLASSIFICATION
Revised: E. M ULTI - CLASS CLASSIFICATION

Original: Although we have shown in the main text that our weighted-MOCU can achieve good empirical performance of converging to OBC with the simulated multi-class classiﬁcation experiment, active learning for multi-class classiﬁcation problems can be complicated.
Revised: Although we have shown in the main text that our weighted-MOCU can achieve good empirical performance of converging to OBC with the simulated multi-class classiﬁcation experiment, active learning for multi-class classiﬁcation problems can be complicated.

Original: The weighting function (10) adopted in the main text may not have the same theoretical convergence guarantee to the optimal classiﬁer if applied to multi-class classiﬁcation problems.
Revised: The weighting function (10) adopted in the main text may not have the same theoretical convergence guarantee to the optimal classiﬁer if applied to multi-class classiﬁcation problems.

Original: Here we just show a counter example, for which Lemma 4 does not hold if using the same weighting function.
Revised: Here we just show a counter example, for which Lemma 4 does not hold if using the same weighting function.

Original: Assume a three-class classiﬁcation problem y ∈ { 0 , 1 , 2 } .
Revised: Assume a three-class classiﬁcation problem y ∈ { 0 , 1 , 2 } .

Original: The candidate pool only has one candidate X = { x } and the parameter set Θ
Revised: The candidate pool only has one candidate X = { x } and the parameter set Θ

Original: = { θ 1 , θ 2 , θ 3 } .
Revised: = { θ 1 , θ 2 , θ 3 } .

Original: In addition we set the probabilistic model p ( y | x, θ ) and the prior π ( θ ) as shown in Tables S1 and S2, and calculate the posterior and posterior predictive probabilities.
Revised: In addition we set the probabilistic model p ( y | x, θ ) and the prior π ( θ ) as shown in Tables S1 and S2, and calculate the posterior and posterior predictive probabilities.

Original: In the tables, y o denotes the one-step-look-ahead observation corresponding to x , and x is omitted for simplicity.
Revised: In the tables, y o denotes the one-step-look-ahead observation corresponding to x , and x is omitted for simplicity.

Original: Without loss generality, we just set the weighted MOCU parameter c = 1 .
Revised: Without loss generality, we just set the weighted MOCU parameter c = 1 .

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: y |
Revised: y |

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Here two properties in the setting are worth mentioning:
Revised: Here two properties in the setting are worth mentioning:

Original: π ( θ 1 ) is close to 1 and as a result ∀ y o ∈ { 0 , 1 , 2 } , we have max y p
Revised: π ( θ 1 ) is close to 1 and as a result ∀ y o ∈ { 0 , 1 , 2 } , we have max y p

Original: ( y )
Revised: ( y )

Original: = max y p
Revised: = max y p

Original: ( y | y o )
Revised: ( y | y o )

Original: = max y p
Revised: = max y p

Original: ( y | θ 1 )
Revised: ( y | θ 1 )

Original: = 0 . 4 ; 2.
Revised: = 0 . 4 ; 2.

Original: p ( y | θ 2 ) and p ( y | θ 3 ) are symmetric and π ( θ 2 )
Revised: p ( y | θ 2 ) and p ( y | θ 3 ) are symmetric and π ( θ 2 )

Original: =
Revised: =

Original: π ( θ 3 ) , as a result ∀ y o ∈ { 0 , 1 , 2 } , π ( θ 1 ) =
Revised: π ( θ 3 ) , as a result ∀ y o ∈ { 0 , 1 , 2 } , π ( θ 1 ) =

Original: π ( θ 1 | y o )
Revised: π ( θ 1 | y o )

Original: = 0 .
Revised: = 0 .

Original: 8 and therefore E π ( θ ) [max y (cid:48) p ( y | θ )]
Revised: 8 and therefore E π ( θ ) [max y (cid:48) p ( y | θ )]

Original: =
Revised: =

Original: E π ( θ | y o )
Revised: E π ( θ | y o )

Original: [max y (cid:48) p ( y | θ )]
Revised: [max y (cid:48) p ( y | θ )]

Original: = 0 .
Revised: = 0 .

Original: 8 × 0 .
Revised: 8 × 0 .

Original: 4 + 0 .
Revised: 4 + 0 .

Original: 2 × 0 .
Revised: 2 × 0 .

Original: 5 = 0 .
Revised: 5 = 0 .

Original: Recall that the K function and weighted MOCU are:
Revised: Recall that the K function and weighted MOCU are:

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have ∀ y o ∈ { 0 , 1 , 2 } , K ( π ( θ ))
Revised: Therefore, we have ∀ y o ∈ { 0 , 1 , 2 } , K ( π ( θ ))

Original: =
Revised: =

Original: K ( π ( θ | y o ))
Revised: K ( π ( θ | y o ))

Original: = 0 .
Revised: = 0 .

Original: 02 and M w ( π ( θ ))
Revised: 02 and M w ( π ( θ ))

Original: =
Revised: =

Original: M w ( π ( θ | y o )) > 0 .
Revised: M w ( π ( θ | y o )) > 0 .

Original: On the other hand,
Revised: On the other hand,

Original: [Equation]
Revised: [Equation]

Original: which means that the algorithm may get stuck.
Revised: which means that the algorithm may get stuck.

Original: Here we just give an extreme case where only one candidate is in the search pool, but it is straightforward to build a more practical example based on what we have shown here.
Revised: Here we just give an extreme case where only one candidate is in the search pool, but it is straightforward to build a more practical example based on what we have shown here.

Original: We can see from the example that, unlike in the cases of binary classiﬁcation problems, the weighting function 1 − cK may remain unchanged for a single observation in multi-class problems.
Revised: We can see from the example that, unlike in the cases of binary classiﬁcation problems, the weighting function 1 − cK may remain unchanged for a single observation in multi-class problems.

Original: Because of this, the weighted-MOCU algorithm may get stuck.
Revised: Because of this, the weighted-MOCU algorithm may get stuck.

Original: Since OBC prediction is the maximum of the predictive distribution p ( y | x ) , the weight function is introduced to capture the changes of p ( y | x ) , as that indicates the potential shift of OBC prediction in the long run.
Revised: Since OBC prediction is the maximum of the predictive distribution p ( y | x ) , the weight function is introduced to capture the changes of p ( y | x ) , as that indicates the potential shift of OBC prediction in the long run.

Original: K is a function of max y p ( y | x ) , in binary case, max y p ( y | x ) must change as p ( y | x ) changes.
Revised: K is a function of max y p ( y | x ) , in binary case, max y p ( y | x ) must change as p ( y | x ) changes.

Original: However, in multi-class problems, the probability of the optimal label max y p ( y | x ) may remain unchanged, when the probability of other labels change, just like in the example above where max y p ( y )
Revised: However, in multi-class problems, the probability of the optimal label max y p ( y | x ) may remain unchanged, when the probability of other labels change, just like in the example above where max y p ( y )

Original: = max y p
Revised: = max y p

Original: ( y |
Revised: ( y |

Original: y o = 1) .
Revised: y o = 1) .

Original: In the next section, we propose a weighting function that can capture the change of any element in p ( y | x ) .
Revised: In the next section, we propose a weighting function that can capture the change of any element in p ( y | x ) .

Original: F. A NOTHER W EIGHTED MOCU S CHEME FOR M ULTI - CLASS C LASSIFICATION
Revised: F. A NOTHER W EIGHTED MOCU S CHEME FOR M ULTI - CLASS C LASSIFICATION

Original: To extend the weighted MOCU scheme suit for the multi-class problem, we propose a weight function that can capture the change of p ( y | x ) .
Revised: To extend the weighted MOCU scheme suit for the multi-class problem, we propose a weight function that can capture the change of p ( y | x ) .

Original: The weighting function is deﬁned as the softmax of
Revised: The weighting function is deﬁned as the softmax of

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where p ( y | x ) is the posterior predictive distribution at the current active learning iteration.
Revised: where p ( y | x ) is the posterior predictive distribution at the current active learning iteration.

Original: We compare this Weighted MOCU with other active learning algorithms empirically on the synthetic three-class classiﬁcation problem and the performance comparison is shown in Fig.
Revised: We compare this Weighted MOCU with other active learning algorithms empirically on the synthetic three-class classiﬁcation problem and the performance comparison is shown in Fig.

Original: S2.
Revised: S2.

Original: This new
Revised: This new

Original: Weighted MOCU (Weighted MOCU2) performs slightly better than other algorithms on this multiclass classiﬁcation problem.
Revised: Weighted MOCU (Weighted MOCU2) performs slightly better than other algorithms on this multiclass classiﬁcation problem.

Original: [Figure]
Revised: [Figure]

Original: G. A DDITIONAL SYNTHETIC EXPERIMENTS
Revised: G. A DDITIONAL SYNTHETIC EXPERIMENTS

Original: We run the same synthetic experiment of Fig.
Revised: We run the same synthetic experiment of Fig.

Original: 3 with a different prior setting: w 1 ∼ U (0 . 3 , 0 .
Revised: 3 with a different prior setting: w 1 ∼ U (0 . 3 , 0 .

Original: 8) , w 2 ∼ U ( − 0 .
Revised: 8) , w 2 ∼ U ( − 0 .

Original: and b ∼ U ( − 0 .
Revised: and b ∼ U ( − 0 .

Original: , and the results is shown in Figure S3.
Revised: , and the results is shown in Figure S3.

Original: The performance shows that only our Weighted MOCU method performs better than the random benchmark.
Revised: The performance shows that only our Weighted MOCU method performs better than the random benchmark.

Original: Here we benchmark different active learning strategies for OBC with another synthetic example.
Revised: Here we benchmark different active learning strategies for OBC with another synthetic example.

Original: Assume the classiﬁcation problem with two dimensional input features x = ( x 1 , x 2 ) ∈ R 2 and binary class labels y ∈ { 0 , 1 } .
Revised: Assume the classiﬁcation problem with two dimensional input features x = ( x 1 , x 2 ) ∈ R 2 and binary class labels y ∈ { 0 , 1 } .

Original: The computational model is derived by a decision boundary in a quadratic form: x 2 = ax 21 + bx 1 + c , i.e. p ( y = 1 | x , a, b, c ) = 1 ( x 2 > ax 21 + bx 1 + c ) .
Revised: The computational model is derived by a decision boundary in a quadratic form: x 2 = ax 21 + bx 1 + c , i.e. p ( y = 1 | x , a, b, c ) = 1 ( x 2 > ax 21 + bx 1 + c ) .

Original: The parameter vector θ =
Revised: The parameter vector θ =

Original: ( a, b, c ) ∈ R 3 is uncertain and the true model is characterized by a true parameter θ ∗ .
Revised: ( a, b, c ) ∈ R 3 is uncertain and the true model is characterized by a true parameter θ ∗ .

Original: Unlike Monte Carlo sampling in the main text, here we consider a discrete grid setting for both input space and parameter space with discretization for each variable as follows:
Revised: Unlike Monte Carlo sampling in the main text, here we consider a discrete grid setting for both input space and parameter space with discretization for each variable as follows:

Original: x 1 ranges in [ − 0 .
Revised: x 1 ranges in [ − 0 .

Original: 5 , 0 . 5] with increment 0.
Revised: 5 , 0 . 5] with increment 0.

Original: x 2 ranges in [0 , 2] with increment 0.1.
Revised: x 2 ranges in [0 , 2] with increment 0.1.

Original: a ranges in [-4.3, -3.8] with increment 0.05, 4.
Revised: a ranges in [-4.3, -3.8] with increment 0.05, 4.

Original: b ranges in [-0.25, 0.25] with increment 0.05, 5.
Revised: b ranges in [-0.25, 0.25] with increment 0.05, 5.

Original: c ranges in [1, 2] with increment 0.05.
Revised: c ranges in [1, 2] with increment 0.05.

Original: For now, we simply assume that the distributions over the feature space and parameter space are all uniform to illustrate the effectiveness of MOCU-based active learning.
Revised: For now, we simply assume that the distributions over the feature space and parameter space are all uniform to illustrate the effectiveness of MOCU-based active learning.

Original: With prior knowledge of the system of interest, knowledge-driven prior should be incorporated.
Revised: With prior knowledge of the system of interest, knowledge-driven prior should be incorporated.

Original: Following the weightedMOCU based active learning algorithm in Algorithm 1, we can sequentially query the true system and reduce the model uncertainty in a way that maximally reduces the classiﬁcation error of the corresponding OBC.
Revised: Following the weightedMOCU based active learning algorithm in Algorithm 1, we can sequentially query the true system and reduce the model uncertainty in a way that maximally reduces the classiﬁcation error of the corresponding OBC.

Original: [Figure]
Revised: [Figure]

Original: Now we assume that when querying the system, the class label is given with a heterogeneous random ﬂipping error with the error probability being a function of x 1 : p ( y = 1 | z = 0)
Revised: Now we assume that when querying the system, the class label is given with a heterogeneous random ﬂipping error with the error probability being a function of x 1 : p ( y = 1 | z = 0)

Original: =
Revised: =

Original: p ( y = 

Revised: p ( y =

Original: | z = 1) = 0 .
Revised: | z = 1) = 0 .

Original: 3 × (1 − 4 x 21 ) + 0 . 1 .
Revised: 3 × (1 − 4 x 21 ) + 0 . 1 .

Original: Therefore, when x 1 = 0 , the ﬂipping error is 0.4; and when x 1 = ± 0 .
Revised: Therefore, when x 1 = 0 , the ﬂipping error is 0.4; and when x 1 = ± 0 .

Original: 5 , the ﬂipping error is 0.1.
Revised: 5 , the ﬂipping error is 0.1.

Original: We have implemented the same methods as in the main text with 50 iterations and 100 runs, The active learning results are illustrated in Fig.
Revised: We have implemented the same methods as in the main text with 50 iterations and 100 runs, The active learning results are illustrated in Fig.

Original: S4.
Revised: S4.

Original: As we can see, in this ﬁgure,
Revised: As we can see, in this ﬁgure,

Original: MES does not perform well as it cannot differentiate between model uncertainty and observation error.
Revised: MES does not perform well as it cannot differentiate between model uncertainty and observation error.

Original: ELR performs similarly to BALD and our weighted-MOCU based method at the beginning, but then it gets stuck before ﬁnding the true boundary.
Revised: ELR performs similarly to BALD and our weighted-MOCU based method at the beginning, but then it gets stuck before ﬁnding the true boundary.

Original: BALD and our weighted MOCU perform similarly.
Revised: BALD and our weighted MOCU perform similarly.

Original: This is because in this setting p ( y | x, θ ) is either 1 or 0, so there is no irrelevant uncertainty with which p ( y | x, θ ) is always larger or smaller than 0.5 but the value is uncertain.
Revised: This is because in this setting p ( y | x, θ ) is either 1 or 0, so there is no irrelevant uncertainty with which p ( y | x, θ ) is always larger or smaller than 0.5 but the value is uncertain.

Original: In addition to the average performance comparison, we deliberately choose one of the runs in which the ELR method gets stuck to better illustrate the difference between the existing ELR methods and the proposed weighted-MOCU based method.
Revised: In addition to the average performance comparison, we deliberately choose one of the runs in which the ELR method gets stuck to better illustrate the difference between the existing ELR methods and the proposed weighted-MOCU based method.

Original: In this run, the randomly chosen parameters are ( a = − 3 , b = 0 , c = 1 . 9) .
Revised: In this run, the randomly chosen parameters are ( a = − 3 , b = 0 , c = 1 . 9) .

Original: Fig.
Revised: Fig.

Original: S5(a) shows the error regret (the OBC error minus the true optimal classiﬁer error) comparison, in which ELR gets stuck and the weighted-MOCU based method reaches 0.
Revised: S5(a) shows the error regret (the OBC error minus the true optimal classiﬁer error) comparison, in which ELR gets stuck and the weighted-MOCU based method reaches 0.

Original: Notice that the y-axis is in the logarithm scale, so the vertical line in the WMOCU plot implies that the value turns to 0.
Revised: Notice that the y-axis is in the logarithm scale, so the vertical line in the WMOCU plot implies that the value turns to 0.

Original: Error regret equals to 0 indicates that the OBC classiﬁer equals to the true optimal classiﬁer, but in practice we don’t know the true optimal classiﬁer, so we need the value of MOCU to quantify the expected error difference between OBC and the optimal classiﬁer of each θ =
Revised: Error regret equals to 0 indicates that the OBC classiﬁer equals to the true optimal classiﬁer, but in practice we don’t know the true optimal classiﬁer, so we need the value of MOCU to quantify the expected error difference between OBC and the optimal classiﬁer of each θ =

Original: ( a, b, c ) .
Revised: ( a, b, c ) .

Original: Fig.
Revised: Fig.

Original: S5(b) shows the changes of MOCU value during the two active learning procedures.
Revised: S5(b) shows the changes of MOCU value during the two active learning procedures.

Original: Not surprisingly, the MOCU value during the iterations of the ELR method also gets stuck, while the MOCU value in the iterations of the weighted-MOCU method continues to decrease.
Revised: Not surprisingly, the MOCU value during the iterations of the ELR method also gets stuck, while the MOCU value in the iterations of the weighted-MOCU method continues to decrease.

Original: Fig.
Revised: Fig.

Original: S5(c) shows the changes of the maximum value of acquisition function in each iteration.
Revised: S5(c) shows the changes of the maximum value of acquisition function in each iteration.

Original: The acquisition function of ELR decrease to 0 after 22 iterations, and that explains why ELR gets stuck.
Revised: The acquisition function of ELR decrease to 0 after 22 iterations, and that explains why ELR gets stuck.

Original: On the other hand, the maximum acquisition function of WMOCU is always positive as the corresponding MOCU is positive, until it gets close to 10 − 16 , which is the rounding error in ﬂoating point arithmetic.
Revised: On the other hand, the maximum acquisition function of WMOCU is always positive as the corresponding MOCU is positive, until it gets close to 10 − 16 , which is the rounding error in ﬂoating point arithmetic.

Original: In theory, as the observation is noisy, we can not be sure of the optimal prediction.
Revised: In theory, as the observation is noisy, we can not be sure of the optimal prediction.

Original: Therefore, the MOCU and the acquisition function of weighted-MOCU should always be positive, which is demonstrated in the ﬁgures.
Revised: Therefore, the MOCU and the acquisition function of weighted-MOCU should always be positive, which is demonstrated in the ﬁgures.

Original: We have also performed an experiment to show the algorithm performance change under different noise levels.
Revised: We have also performed an experiment to show the algorithm performance change under different noise levels.

Original: We set the ﬂipping error rate as p ( y (cid:54) =
Revised: We set the ﬂipping error rate as p ( y (cid:54) =

Original: z | x ) =
Revised: z | x ) =

Original: (cid:15) × (1 − 4 x 21 ) + (cid:15), 0 ≤ (cid:15) ≤ 0 .
Revised: (cid:15) × (1 − 4 x 21 ) + (cid:15), 0 ≤ (cid:15) ≤ 0 .

Original: Therefore, when x 1 = 0 , the ﬂipping error is 2 (cid:15) ; and when x 1 = ± 0 .
Revised: Therefore, when x 1 = 0 , the ﬂipping error is 2 (cid:15) ; and when x 1 = ± 0 .

Original: 5 , the ﬂipping error is (cid:15) .
Revised: 5 , the ﬂipping error is (cid:15) .

Original: We perform the same methods with 100 iterations and 100 runs on the noise level (cid:15) = 0 .
Revised: We perform the same methods with 100 iterations and 100 runs on the noise level (cid:15) = 0 .

Original: 05 and (cid:15) = 0 .
Revised: 05 and (cid:15) = 0 .

Original: The resulting active learning performance curves are illustrated in Fig.
Revised: The resulting active learning performance curves are illustrated in Fig.

Original: S6.
Revised: S6.

Original: We can
Revised: We can

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: r r
Revised: r r

Original: [Figure]
Revised: [Figure]

Original: g r
Revised: g r

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: see from the ﬁgure that the performance of MES degrades signiﬁcantly with high noise while the performance of other methods does not appear to be very sensitive to the increasing noise
Revised: see from the ﬁgure that the performance of MES degrades signiﬁcantly with high noise while the performance of other methods does not appear to be very sensitive to the increasing noise

Original: We here present the complete results on the UCI User Knowledge dataset (Kahraman et al., 2013).
Revised: We here present the complete results on the UCI User Knowledge dataset (Kahraman et al., 2013).

Original: In addition to the uncertainty class setup in the main text, we have tested two other setups of hyperparameter values: 1) ‘uniform prior’ with α i
Revised: In addition to the uncertainty class setup in the main text, we have tested two other setups of hyperparameter values: 1) ‘uniform prior’ with α i

Original: = β
Revised: = β

Original: i = 1 , and 2) ’good prior’ with α i =
Revised: i = 1 , and 2) ’good prior’ with α i =

Original: β i = 10 in eight bins chosen randomly, for other bins
Revised: β i = 10 in eight bins chosen randomly, for other bins

Original: α i = 5 , β i = 

Revised: α i = 5 , β i =

Original: if the true frequency of High or Medium
Revised: if the true frequency of High or Medium

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: in the i -th bin is higher than 0.5 and α
Revised: in the i -th bin is higher than 0.5 and α

Original: i = 2 , β i = 5 if the frequency is lower than 0.5.
Revised: i = 2 , β i = 5 if the frequency is lower than 0.5.

Original: We also randomly draw 150 samples from each class as the candidate pool and perform the ﬁve different active learning algorithms.
Revised: We also randomly draw 150 samples from each class as the candidate pool and perform the ﬁve different active learning algorithms.

Original: We repeat the whole procedure 150 times and the average error rates are shown in Fig.
Revised: We repeat the whole procedure 150 times and the average error rates are shown in Fig.

Original: S7.
Revised: S7.

Original: In both Fig.
Revised: In both Fig.

Original: S7a and Fig.
Revised: S7a and Fig.

Original: S7b, ELR performs the best in these two setups while our Weighted MOCU performs similarly.
Revised: S7b, ELR performs the best in these two setups while our Weighted MOCU performs similarly.

Original: BALD performs reasonably in Fig.
Revised: BALD performs reasonably in Fig.

Original: S7a
Revised: S7a

Original: but it again performs poorly in Fig.
Revised: but it again performs poorly in Fig.

Original: S7b.
Revised: S7b.

Original: This is because the bins with α = β = 10 have less uncertainty but have more impact on OBC prediction and BALD fails to identify that in this setup again.
Revised: This is because the bins with α = β = 10 have less uncertainty but have more impact on OBC prediction and BALD fails to identify that in this setup again.

Original: We also present the results on the UCI Letter Recognition dataset (Dua & Graff, 2017).
Revised: We also present the results on the UCI Letter Recognition dataset (Dua & Graff, 2017).

Original: Letter
Revised: Letter

Original: Recognition is a multi-class classiﬁcation dataset with each sample having 16 numerical features generated from typed images of the capital letters in the English alphabet.
Revised: Recognition is a multi-class classiﬁcation dataset with each sample having 16 numerical features generated from typed images of the capital letters in the English alphabet.

Original: We select two pairs of hard-to-distinguish letters: E vs. F and D vs. P.
Revised: We select two pairs of hard-to-distinguish letters: E vs. F and D vs. P.

Original: The total number of training samples is 1543 and 1608 for E vs. F and D vs. P, respectively.
Revised: The total number of training samples is 1543 and 1608 for E vs. F and D vs. P, respectively.

Original: Active learning algorithms are applied with Bayesian logistic regression models.
Revised: Active learning algorithms are applied with Bayesian logistic regression models.

Original: We randomly take 100 data points ﬁrst to construct the prior, and use the rest of the data as the pool to test the ﬁve active learning algorithms.
Revised: We randomly take 100 data points ﬁrst to construct the prior, and use the rest of the data as the pool to test the ﬁve active learning algorithms.

Original: For prior construction, we train a logistic regression model on the 100 data points and take the trained parameters as the mean of a normal distributed prior with the variance equal to 1.
Revised: For prior construction, we train a logistic regression model on the 100 data points and take the trained parameters as the mean of a normal distributed prior with the variance equal to 1.

Original: Then we sample 1000 particles from the prior as the uncertain parameter set.
Revised: Then we sample 1000 particles from the prior as the uncertain parameter set.

Original: We repeat the whole procedure 100 times and the average error rates are shown in Fig. S8.
Revised: We repeat the whole procedure 100 times and the average error rates are shown in Fig. S8.

Original: Unlike the synthetic datasets, the real-world datasets have no corresponding true models.
Revised: Unlike the synthetic datasets, the real-world datasets have no corresponding true models.

Original: We can only ﬁnd the optimal models that approximate the data best.
Revised: We can only ﬁnd the optimal models that approximate the data best.

Original: However, we can still see the trends of different algorithms.
Revised: However, we can still see the trends of different algorithms.

Original: Compared with random sampling, all the algorithms quickly converge to the optimal models.
Revised: Compared with random sampling, all the algorithms quickly converge to the optimal models.

Original: ELR performs the best in the ﬁrst several iterations, while converges slowly in the latter iterations.
Revised: ELR performs the best in the ﬁrst several iterations, while converges slowly in the latter iterations.

Original: Our weighted MOCU based method is again demonstrated to converge faster than other competing methods.
Revised: Our weighted MOCU based method is again demonstrated to converge faster than other competing methods.

Original: It is clear from all our experiments for both simulated and real-world data that, in addition to its theoretical guarantee for active learning with OBC, our weighted MOCU method has achieved consistently better or similar empirical performance compared to the best performing ones among the existing pool-based active learning methods, approaching the corresponding OBCs faster with fewer labeled samples.
Revised: It is clear from all our experiments for both simulated and real-world data that, in addition to its theoretical guarantee for active learning with OBC, our weighted MOCU method has achieved consistently better or similar empirical performance compared to the best performing ones among the existing pool-based active learning methods, approaching the corresponding OBCs faster with fewer labeled samples.

Original: [Figure]
Revised: [Figure]