Original: D EEP A CTIVE L EARNING WITH N OISE S TABILITY
Revised: D EEP A CTIVE L EARNING WITH N OISE S TABILITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Uncertainty estimation for unlabeled data is crucial to active learning.
Revised: Uncertainty estimation for unlabeled data is crucial to active learning.

Original: With a deep neural network employed as the backbone model, the data selection process is highly challenged due to the potential over-conﬁdence of the model inference.
Revised: With a deep neural network employed as the backbone model, the data selection process is highly challenged due to the potential over-conﬁdence of the model inference.

Original: Existing methods usually resort to multi-pass model training or adversarial training to handle this challenge.
Revised: Existing methods usually resort to multi-pass model training or adversarial training to handle this challenge.

Original: This tends to result in complex and inefﬁcient pipelines, which would render the methods impractical.
Revised: This tends to result in complex and inefﬁcient pipelines, which would render the methods impractical.

Original: In this work, we propose a novel Single-Training Multi-Inference algorithm that leverages noise stability to estimate data uncertainty.
Revised: In this work, we propose a novel Single-Training Multi-Inference algorithm that leverages noise stability to estimate data uncertainty.

Original: A key idea behind is a magnitude measurement of output derivation from the original observation when the model parameters are randomly perturbed by noise.
Revised: A key idea behind is a magnitude measurement of output derivation from the original observation when the model parameters are randomly perturbed by noise.

Original: We provide analysis by using the small Gaussian noise theory and show that our method has a strong connection with the classical theory of variance reduction.
Revised: We provide analysis by using the small Gaussian noise theory and show that our method has a strong connection with the classical theory of variance reduction.

Original: That is, by labelling a data sample of higher uncertainty, as indicated by the inverse noise stability, our method contributes more to reducing the variance of existing training samples.
Revised: That is, by labelling a data sample of higher uncertainty, as indicated by the inverse noise stability, our method contributes more to reducing the variance of existing training samples.

Original: Despite its simplicity, our method outperforms the state-of-the-art active learning baselines in image classiﬁcation and semantic segmentation tasks.
Revised: Despite its simplicity, our method outperforms the state-of-the-art active learning baselines in image classiﬁcation and semantic segmentation tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The success of training a deep neural network highly depends on a huge amount of labeled data.
Revised: The success of training a deep neural network highly depends on a huge amount of labeled data.

Original: Nevertheless, in the era of big data, much data is unlabeled.
Revised: Nevertheless, in the era of big data, much data is unlabeled.

Original: This imposes a challenge on exploiting appropriate learning fashions.
Revised: This imposes a challenge on exploiting appropriate learning fashions.

Original: For instance, given a pool of unlabeled data, one can employ unsupervised manner to learn a model.
Revised: For instance, given a pool of unlabeled data, one can employ unsupervised manner to learn a model.

Original: However, the performance of unsupervised or semi-supervised learning is still bound to that of supervised learning (Yoo & Kweon, 2019).
Revised: However, the performance of unsupervised or semi-supervised learning is still bound to that of supervised learning (Yoo & Kweon, 2019).

Original: An ideal option could be annotating all the given data, so as to use the supervised way to learn a model.
Revised: An ideal option could be annotating all the given data, so as to use the supervised way to learn a model.

Original: Nevertheless, the cost for data annotation could be extremely high, such as in medical area (Gorriz et al., 2017;
Revised: Nevertheless, the cost for data annotation could be extremely high, such as in medical area (Gorriz et al., 2017;

Original: Konyushkova et al., 2017).
Revised: Konyushkova et al., 2017).

Original: Due to the limited labeling budget, it may be more feasible to annotate a portion of the given data.
Revised: Due to the limited labeling budget, it may be more feasible to annotate a portion of the given data.

Original: This is also aligned with the prerequisite of semi-supervised learning.
Revised: This is also aligned with the prerequisite of semi-supervised learning.

Original: Then one can ask: which data is deserved for being annotated?
Revised: Then one can ask: which data is deserved for being annotated?

Original: Active learning was proposed to solve this question.
Revised: Active learning was proposed to solve this question.

Original: One of the main ideas, namely uncertaintybased method , aims to select a portion of the most uncertain or representative data from an unlabeled pool for annotation.
Revised: One of the main ideas, namely uncertaintybased method , aims to select a portion of the most uncertain or informative data from an unlabeled pool for annotation.

Original: The newly annotated data are then used to train a task model.
Revised: The newly annotated data are then used to train a task model.

Original: Note that uncertainty estimation with deep neural networks (DNNs) remains challenging, due to the potential over-conﬁdence of deep models.
Revised: Note that uncertainty estimation with deep neural networks (DNNs) remains challenging, due to the potential over-conﬁdence of deep models.

Original: That is, the primitive softmax output 1 as a score does not necessarily reﬂect a reliable uncertainty or conﬁdence for the prediction.
Revised: That is, the primitive softmax output 1 as a score does not necessarily reﬂect a reliable uncertainty or conﬁdence for the prediction.

Original: Several methods have been proposed to address this challenge.
Revised: Several methods have been proposed to address this challenge.

Original: For example, MC-Dropout (Gal & Ghahramani, 2016) implements a Single-Training Multi-Inference algorithm by aggregating multiple conﬁdences sampled by
Revised: For example, MC-Dropout (Gal & Ghahramani, 2016) implements a Single-Training Multi-Inference algorithm by aggregating multiple conﬁdences sampled by

Original: Monte Carlo dropout.
Revised: Monte Carlo dropout.

Original: The algorithm has been shown to be equivalent to the approximate Bayesian inference.
Revised: The algorithm has been shown to be equivalent to the approximate Bayesian inference.

Original: Another typical idea is query-by-committee (Freund et al., 1997; Gorriz et al., 2017), or QBC in short, which requires training multiple auxiliary models as a committee.
Revised: Another typical idea is query-by-committee (Freund et al., 1997; Gorriz et al., 2017), or QBC in short, which requires training multiple auxiliary models as a committee.

Original: Besides, many other efforts suggest to use specially designed auxiliary modules or training fashions, such as variational auto-encoder (Sinha et al., 2019), adversarial learning (Ducoffe & Precioso, 2018; Mayer &
Revised: Besides, many other efforts suggest to use specially designed auxiliary modules or training fashions, such as variational auto-encoder (Sinha et al., 2019), adversarial learning (Ducoffe & Precioso, 2018; Mayer &

Original: Timofte, 2020) and Graph Convolutional Network (Caramalau et al., 2021).
Revised: Timofte, 2020) and Graph Convolutional Network (Caramalau et al., 2021).

Original: In this work, we study the problem of deep active learning from a different viewpoint, inspired by a relevant topic of noise stability .
Revised: In this work, we study the problem of deep active learning from a different viewpoint, inspired by a relevant topic of noise stability .

Original: Noise stability, in terms of the output stability w.r.t.
Revised: Noise stability, in terms of the output stability w.r.t.

Original: the input 1 Considering  noise, is well studied in literature.
Revised: the input 1 Considering classiﬁcation problems noise, is well studied in literature.

Original: For example, the authors in (Bishop, 1995) propose to train a model with perturbed input samples as a regularizer to improve the model performance.
Revised: For example, the authors in (Bishop, 1995) propose to train a model with perturbed input samples as a regularizer to improve the model performance.

Original: A recent theoretical work (Arora et al., 2018) shows that the interlayer cushion, evaluated by noise stability, acts as a good indicator of the generalization bounds for DNNs.
Revised: A recent theoretical work (Arora et al., 2018) shows that, the stability of each layer’s computation to noise injected at lower layers acts as a good indicator of the generalization bounds for DNNs.

Original: Behind these achievements is an intuitive implication that, a model (or more exactly, its parameters) robust to perturbed input examples tends to be easy to recognize future examples.
Revised: Behind these achievements is an intuitive implication that, a model (or more exactly, its parameters) robust to perturbed input examples tends to be easy to recognize unseen examples.

Original: For the active learning problem, we give an analogous intuition by exchanging the role of the input and parameter that, an example robust to the perturbed current parameters tends to be easy to be recognized by a future model.
Revised: For the active learning problem, we give an analogous intuition by exchanging the role of the input and parameter that, an example robust to the perturbed current parameters tends to be easy to be recognized by a future model.

Original: In other words, examples less robust to the parameter perturbation would be regarded as having higher uncertainty .
Revised: In other words, examples less robust to the parameter perturbation would be regarded as having higher uncertainty .

Original: Speciﬁcally, we introduce a simple algorithm of uncertainty estimation by computing the distance between outputs of the clean and perturbed input, when imposing a small noise on the parameters.
Revised: Speciﬁcally, we introduce a simple algorithm of uncertainty estimation by measuring how far does the output deviate from the original value, when imposing a small noise on the parameters.

Original: The induced distance is used as a measurement of uncertainty used to select unlabeled examples in active learning.
Revised: The induced distance is used as a measurement of uncertainty used to select unlabeled examples in active learning.

Original: We provide a theoretical analysis under the condition that the noise magnitude is small.
Revised: We provide a theoretical analysis under the condition that the noise magnitude is small.

Original: By imposing standard multivariate Gaussian noise to the parameter, we prove that the proposed noise stability is equivalent to the parameter-output Jacobian norm under the ﬁrst order
Revised: By imposing standard multivariate Gaussian noise to the parameter, we prove that the proposed noise stability is equivalent to the parameter-output Jacobian norm under the ﬁrst order

Original: Taylor assumption.
Revised: Taylor assumption.

Original: Furthermore, we show that, our algorithm has a close connection with variance reduction (Cohn et al., 1996).
Revised: Furthermore, we show that, our algorithm has a close connection with variance reduction (Cohn et al., 1996).

Original: That is, selecting unlabeled samples with low noise stability would yield the same effect of reducing the prediction variance of existing training samples.
Revised: That is, selecting unlabeled samples with low noise stability would yield the same effect of reducing the prediction variance of existing training samples.

Original: Our method is easy to implement and free of customized auxiliary models.
Revised: Our method is easy to implement and free of customized auxiliary models.

Original: Therefore, it can be exploited in various tasks, such as image classiﬁcation and semantic segmentation, leading to its task-agnostic nature.
Revised: Therefore, it can be exploited in various tasks, such as image classiﬁcation and semantic segmentation, leading to its task-agnostic nature.

Original: We conduct extensive experiments to evaluate our method on various datasets including Cifar10 (Krizhevsky et al., 2009), Cifar100 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), Caltech101 (Fei-Fei et al., 2006), Cityscapes (Cordts et al., 2016), and cryo-ET (Chen et al., 2017).
Revised: We conduct extensive experiments to evaluate our method on various datasets including Cifar10 (Krizhevsky et al., 2009), Cifar100 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), Caltech101 (Fei-Fei et al., 2006), Cityscapes (Cordts et al., 2016), and cryo-ET (Chen et al., 2017).

Original: The performance of our method signiﬁcantly exceeds that of the state-of-the-art.
Revised: The performance of our method signiﬁcantly exceeds that of the state-of-the-art.

Original: The contributions of this work are summarized as follows.
Revised: The contributions of this work are summarized as follows.

Original: We propose a novel effective method of noise stability to select unlabeled data for active learning.
Revised: We propose a novel effective method of noise stability to select unlabeled data for active learning.

Original: The proposed method is free of any auxiliary models or special training fashions.
Revised: The proposed method is free of any auxiliary models or special training fashions.

Original: We provide a theoretical analysis to show that, selecting unlabeled data with higher noise stability is equivalent to selecting that with higher Jacobian norm w.r.t.
Revised: We provide a theoretical analysis to show that, selecting unlabeled data with higher noise stability is equivalent to selecting that with higher Jacobian norm w.r.t.

Original: the parameter.
Revised: the parameter.

Original: We also establish a connection between noise stability and variance reduction for the existing training set.
Revised: We also establish a connection between noise stability and variance reduction for the existing training set.

Original: We conduct extensive experiments on benchmark datasets of image classiﬁcation, semantic segmentation, and 3D cryo-ET subtomogram classiﬁcation.
Revised: We conduct extensive experiments on benchmark datasets of image classiﬁcation, semantic segmentation, and 3D cryo-ET subtomogram classiﬁcation.

Original: The results demonstrate the effectiveness of the proposed method.
Revised: The results demonstrate the effectiveness of the proposed method.

Original:  A CTIVE L EARNING WITH N OISE S TABILITY 2. P ROBLEM D EFINITION
Revised:  A CTIVE L EARNING WITH N OISE S TABILITY 2. P ROBLEM D EFINITION

Original: Here we formulate the active learning problem.
Revised: Here we formulate the active learning problem.

Original: Given a pool of unlabeled data { X U } , and a labeled pool { X L , Y L } which is initially empty.
Revised: Given a pool of unlabeled data { X U } , and a labeled pool { X L , Y L } which is initially empty.

Original: Active learning aims to select a portion of data from { X U } depending on labeling budget (e.g. select 2500 samples out of 50000 at a time).
Revised: Active learning aims to select a portion of data from { X U } depending on labeling budget (e.g. select 2500 samples out of 50000 at a time).

Original: The selected data
Revised: The selected data

Original: X N is then annotated by human oracles (or equivalent), and added to the labeled pool.
Revised: X N is then annotated by human oracles (or equivalent), and added to the labeled pool.

Original: That is, { X L ,
Revised: That is, { X L ,

Original: Y L } ← { X L , Y L } + { X N , Y N } , where Y N is the new annotation for X N .
Revised: Y L } ← { X L , Y L } + { X N , Y N } , where Y N is the new annotation for X N .

Original: The unlabeled pool is then updated by removing the selected data: { X U } ← { X U } − { X N } .
Revised: The unlabeled pool is then updated by removing the selected data: { X U } ← { X U } − { X N } .

Original: { X L , Y L } is then used to train a task model f ( . ; θ ) in supervised fashion by minimizing an empirical loss L (e.g. cross-entropy loss for classiﬁcation), where f ( . ; θ ) denotes a neural network f ( . ) parameterized with θ 
Revised: { X L , Y L } is then used to train a task model f ( . ; θ ) in supervised fashion by minimizing an empirical loss L (e.g. crossentropy loss for classiﬁcation), where f ( . ; θ ) denotes a neural network f ( . ) parameterized with θ . Note that in this paper,

Original: . 2. U NCERTAINTY E STIMATION WITH N OISE S TABILITY
Revised: 2. U NCERTAINTY E STIMATION WITH N OISE S TABILITY

Original: We use the noise stability to estimate uncertainty for unlabeled data.
Revised: We use the noise stability to estimate uncertainty for unlabeled data.

Original: Speciﬁcally, for an input sample x , we explore to what degree will the model’s output deviate from the original observation f ( x ; θ ) , when adding random noise to the model parameter θ .
Revised: Speciﬁcally, for an input sample x , we explore to what degree will the model’s output deviate from the original observation f ( x ; θ ) , when adding random noise to the model parameter θ .

Original: Let ξ ∆ θ denote the added noise, where ξ controls the magnitude of the noise.
Revised: Let ξ ∆ θ denote the added noise, where ξ controls the magnitude of the noise.

Original: If ∆ θ conforms to the standard multivariate Gaussian distribution 2 N ( 0 , σ 2 I ) , the criterion to quantify the degree of noise instability can be formulated as
Revised: If ∆ θ conforms to the standard multivariate Gaussian distribution 2 N ( 0 , σ 2 I ) , the criterion to quantify the degree of noise instability can be formulated as

Original: [Equation]
Revised: [Equation]

Original:  where (cid:107) .
Revised: (1) where (cid:107) .

Original: (cid:107) 2 is the L 2 norm of a vector.
Revised: (cid:107) 2 is the L 2 norm of a vector.

Original: In active learning, we select samples with largest R ( x ) for annotation.
Revised: In active learning, we select samples with largest R ( x ) for annotation.

Original: To approximate the expectation, we perform a Monto-carlo sampling from the multivariate
Revised: To approximate the expectation, we perform a Monto-carlo sampling from the multivariate

Original: Guassian distribution to generate multiple ∆ θ ( i ) .
Revised: Guassian distribution to generate multiple ∆ θ ( i ) .

Original: Then we get the approximated noise stability by
Revised: Then we get the approximated noise stability by

Original: [Equation]
Revised: [Equation]

Original: The noise magnitude should be small relative to that of the original parameter (e.g. ξ ( i ) =
Revised: The noise magnitude should be small relative to that of the original parameter (e.g. ξ ( i ) =

Original: 10 − 3 (cid:107) θ (cid:107) (cid:107) ∆ θ ( i ) (cid:107) )
Revised: 10 − 3 (cid:107) θ (cid:107) (cid:107) ∆ θ ( i ) (cid:107) )

Original: in order to avoid catastrophic perturbation to the clean model.
Revised: in order to avoid catastrophic perturbation to the clean model.

Revised: Note that, we use a single ξ ( i ) for the entire ∆ θ ( i ) , i.e. each element in ∆ θ ( i ) is equally re-scaled by ξ ( i ) .

Original: As for thethe sampling number, we ﬁnd the setting K = 5 or 10 works well in practice.
Revised: As for the sampling number, we ﬁnd the setting K = 5 or 10 works well in practice.

Original: A complete procedure of our method is presented in Algorithm 1.
Revised: A complete procedure of our method is presented in Algorithm 1.

Original:  T HEORETICAL U NDERSTANDINGS
Revised:  T HEORETICAL U NDERSTANDINGS

Original: In this section, we provide theoretical understandings about our simple method for uncertainty estimation.
Revised: In this section, we provide theoretical understandings about our simple method for uncertainty estimation.

Original: Our main conclusions are summarized as follows.
Revised: Our main conclusions are summarized as follows.

Original: • When the noise magnitude ξ is sufﬁciently small, selecting data according to R ( x ) in Eq (1) is equivalent to selecting data by the Frobenius norm of parameter-output Jacobian w.r.t.
Revised: • When the noise magnitude ξ is sufﬁciently small, selecting data according to R ( x ) in Eq (1) is equivalent to selecting data by the Frobenius norm of parameter-output Jacobian w.r.t.

Original: f . • Based on the above conclusion, selecting new samples with higher R ( x ) are expected to contribute more in reducing the prediction variance of existing training samples.
Revised: f . • Based on the above conclusion, selecting new samples with higher R ( x ) are expected to contribute more in reducing the prediction variance of existing training samples.

Original: 3. N OISE S TABILITY AS J ACOBIAN N ORM
Revised: 3. N OISE S TABILITY AS J ACOBIAN N ORM

Original: Let f ( x ; θ ) be differentiable (w.r.t.
Revised: Let f ( x ; θ ) be differentiable (w.r.t.

Original: θ ) at the point θ given x as the input.
Revised: θ ) at the point θ given x as the input.

Original: When the imposed noise ξ ∆ θ in Eq (1) has a sufﬁciently small magnitude, we use the ﬁrst-order Taylor expansion to estimate f ( x ; θ + ξ ∆ θ ) as
Revised: When the imposed noise ξ ∆ θ in Eq (1) has a sufﬁciently small magnitude, we use the ﬁrst-order Taylor expansion to estimate f ( x ; θ + ξ ∆ θ ) as

Original: [Equation]
Revised: [Equation]

Original: where J θ ( x ; θ ) is the Jacobian matrix of f with respect to the parameter θ as J θ ( x ; θ ) ( i,j )
Revised: where J θ ( x ; θ ) is the Jacobian matrix of f with respect to the parameter θ as J θ ( x ; θ ) ( i,j )

Original: =
Revised: =

Original: ∂f ( x ; θ ) ( i ) /∂
Revised: ∂f ( x ; θ ) ( i ) /∂

Original: θ ( j ) .
Revised: θ ( j ) .

Original: By substituting the Taylor approximation into Eq (1), the uncertainty estimation can be reformulated as
Revised: By substituting the Taylor approximation into Eq (1), the uncertainty estimation can be reformulated as

Original: [Equation]
Revised: [Equation]

Original: Since the elements of ∆ θ are independent with each other, it’s easy to derive that the expected uncertainty in Eq (4) is in proportion with the Frobenius norm of Jacobian as
Revised: Since the elements of ∆ θ are independent with each other, it’s easy to derive that the expected uncertainty in Eq (4) is in proportion with the Frobenius norm of Jacobian as

Original: [Equation]
Revised: [Equation]

Original: In fact, Eq (5) stands for any appropriate noise distribution with the above mentioned independence prerequisite and zero mean.
Revised: In fact, Eq (5) stands for any appropriate noise distribution with the above mentioned independence prerequisite and zero mean.

Original: We put a detailed proof of Eq (5) in Appendix A.2.
Revised: We put a detailed proof of Eq (5) in Appendix A.2.

Original: 3. C ONNECTIONS WITH P REDICTION V ARIANCE R EDUCTION
Revised: 3. C ONNECTIONS WITH P REDICTION V ARIANCE R EDUCTION

Original: We next show an interesting connection between uncertainty estimation with noise stability and existing theoretical literature of variance reduction.
Revised: We next show an interesting connection between uncertainty estimation with noise stability and existing theoretical literature of variance reduction.

Original: For convenience, we assume that ˆ y = f
Revised: For convenience, we assume that ˆ y = f

Original: ( . ; θ ) is a real-valued function parameterized with θ that results in ˆ y : R d → R , and the results are easy to generalize to vector-valued functions.
Revised: ( . ; θ ) is a real-valued function parameterized with θ that results in ˆ y : R d → R , and the results are easy to generalize to vector-valued functions.

Original: Given a labeled training set L = { x i , y i } m i =1 , we aim at minimizing the Mean Squared Error as
Revised: Given a labeled training set L = { x i , y i } m i =1 , we aim at minimizing the Mean Squared Error as

Original: [Equation]
Revised: [Equation]

Original: Then, we deﬁne by J ˆ y ( x ) the Jacobian w.r.t.
Revised: Then, we deﬁne by J ˆ y ( x ) the Jacobian w.r.t.

Original: the parameter θ at x , i.e. ∂ ˆ y ( x ) /∂
Revised: the parameter θ at x , i.e. ∂ ˆ y ( x ) /∂

Original: θ .
Revised: θ .

Original: Let A S 2 denote the Fisher Information Matrix that A S 

Revised: Let A S 2 denote the Fisher Information Matrix that A S

Original: = 1 S 2 ∂ 2 S 2 /∂
Revised: = 1 S 2 ∂ 2 S 2 /∂

Original: θ 2 .
Revised: θ 2 .

Original: Note that we omit the notation of θ in J and A to avoid ambiguity.
Revised: Note that we omit the notation of θ in J and A to without ambiguity.

Original: According to the well-established theory of variance estimation (Thisted, 1988), the output variance at the input point x can be represented as
Revised: According to the well-established theory of variance estimation (Thisted, 1988), the output variance at the input point x can be represented as

Original: [Equation]
Revised: [Equation]

Original: With the normality and local linearity assumptions, the authors in (MacKay, 1992; Cohn, 1996) further quantify the change of output variance.
Revised: With the normality and local linearity assumptions [Equation] the authors in (MacKay, 1992; Cohn, 1996) further quantify the change of output variance.

Original: Speciﬁcally, when a new sample ˜ x is labeled and added to the training set, the expected new output variance at x is
Revised: Speciﬁcally, when a new sample ˜ x is labeled and added to the training set [Equation] the expected new output variance at x is

Original: [Equation]
Revised: [Equation]

Original: where σ 2ˆ y ( x , ˜ x ) is deﬁned as
Revised: where σ 2ˆ y ( x , ˜ x ) is deﬁned as

Original: [Equation] 
Revised: [Equation] [Equation]

Original: Therefore, by adding a new sample ˜ x , the reduction of output variance σ 2ˆ y ( x ) is: 
Revised: Therefore, by adding a new sample ˜ x [Equation] the reduction of output variance σ 2ˆ y ( x ) is: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where we use u (˜ x ) with the unit length to denote the direction of J ˆ y (˜ x ) as u (˜ x )
Revised: where we use u (˜ x ) with the unit length to denote the direction of J ˆ y (˜ x ) as u (˜ x )

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Eq (10) implies that, for any direction u (˜ x ) the optimal choice for maximizing ∆ ˜ x σ 2ˆ y ( x ) is the ones with the largest (cid:107)J ˆ y
Revised: Eq (10) implies that, for any direction u (˜ x ) the optimal choice for maximizing ∆ ˜ x σ 2ˆ y ( x ) is the ones with the largest (cid:107)

Original:  (˜ x  ) (cid:107) 2 .
Revised: J θ (˜ x ; θ ) (cid:107) 2 .

Original: Though the objective is also affected by the direction u (˜ x ) , searching for an optimal direction is usually inefﬁcient due to the characteristic of almost orthogonality of high-dimensional independent vectors (Vershynin, 2018).
Revised: Though the objective is also affected by the direction u (˜ x ) , searching for an optimal direction is usually inefﬁcient due to the characteristic of almost orthogonality of high-dimensional independent vectors (Vershynin, 2018).

Original: Active learning with the objective of maximizing variance reduction has been successfully validated for shallow models such as mixture of Gaussians and kernel regression (Cohn et al., 1996).
Revised: Active learning with the objective of maximizing variance reduction has been successfully validated for shallow models such as mixture of Gaussians and kernel regression (Cohn et al., 1996).

Original: However, computing the inversion of A for each new sample is too cumbersome for deep neural networks.
Revised: However, computing the inversion of A for each new sample is too cumbersome for deep neural networks.

Original: Our work establishes a strong connection between variance reduction and noise stability, and renders the algorithm implementable easily.
Revised: Our work establishes a strong connection between variance reduction and noise stability, and renders the algorithm implementable easily.

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We compare our method with the state-of-the-art active learning methods.
Revised: We compare our method with the state-of-the-art active learning methods.

Original: Furthermore, since random selection is the most straightforward strategy for data selection, we also include it as a baseline in this paper.
Revised: Furthermore, since random selection is the most straightforward strategy for data selection, we also include it as a baseline in this paper.

Original: We ﬁrst validate our method on classical image classiﬁcation and semantic segmentation tasks.
Revised: We ﬁrst validate our method on classical image classiﬁcation and semantic segmentation tasks.

Original: Then we investigate its performance on simulated cryo-ET data (see Appendix A.3), demonstrating its competency for domain tasks.
Revised: Then we investigate its performance on simulated cryo-ET data (see Appendix A.4.1), demonstrating its competency for domain tasks.

Original: For all the tasks, the labeling budget varies from 10% to 40% corresponding to 7 active learning cycles.
Revised: For all the tasks, the labeling budget varies from 10% to 40% corresponding to 7 active learning cycles.

Original: In each cycle, 5% of all the training data will be selected for annotation.
Revised: In each cycle, 5% of all the training data will be selected for annotation.

Original: For the ﬁrst cycle, we randomly select and annotate 10% of the data from the unlabeled pool and use the selected data as initial training data for all the compared methods.
Revised: For the ﬁrst cycle, we randomly select and annotate 10% of the data from the unlabeled pool and use the selected data as initial training data for all the compared methods.

Original: Each reported result is based on three trials to obtain a reliable evaluation.
Revised: Each reported result is based on three trials to obtain a reliable evaluation.

Original: We conduct all the experiments using Pytorch (Paszke et al., 2017) and the source code will be publicly available.
Revised: We conduct all the experiments using Pytorch (Paszke et al., 2017) and the source code will be publicly available.

Original: 4. I MAGE C LASSIFICATION
Revised: 4. I MAGE C LASSIFICATION

Original: Datasets.
Revised: Datasets.

Original: We adopt four benchmark classiﬁcation datasets for evaluations, including Cifar (Krizhevsky et al., 2009), Cifar100 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), and Caltech101 (Fei-Fei et al., 2006).
Revised: We adopt four benchmark classiﬁcation datasets for evaluations, including Cifar (Krizhevsky et al., 2009), Cifar100 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), and Caltech101 (Fei-Fei et al., 2006).

Original: Each Cifar dataset contains 50000 training and 10000 testing samples, uniformly distributed across 10 and 100 classes, respectively.
Revised: Each Cifar dataset contains 50000 training and 10000 testing samples, uniformly distributed across 10 and 100 classes, respectively.

Original: SVHN includes 73257 training and 26032 testing samples, and we do not use the additional training data in SVHN following the standard settings.
Revised: SVHN includes 73257 training and 26032 testing samples, and we do not use the additional training data in SVHN following the standard settings.

Original: There are 8677 images of higher resolution (e.g. 300 × 200 pixels) in the Caltech dataset.
Revised: There are 8677 images of higher resolution (e.g. 300 × 200 pixels) in the Caltech dataset.

Original: We follow the common practice to resize the images to 256 × 256 , and center-crop the images within a 224 × 224 region.
Revised: We follow the common practice to resize the images to 256 × 256 , and center-crop the images within a 224 × 224 region.

Original: Both SVHN and Caltech101 are typical imbalanced datasets.
Revised: Both SVHN and Caltech101 are typical imbalanced datasets.

Original: In the latter, for example, the number of samples in a class varies from 40 to 800.
Revised: In the latter, for example, the number of samples in a class varies from 40 to 800.

Original: Model and Training We use ResNet-18 He et al.
Revised: Model and Training We use ResNet-18 He et al.

Original: as the task model all the classiﬁcation experiments.
Revised: as the task model all the classiﬁcation experiments.

Original: Speciﬁcally, we follow the practice in Caramalau et al.
Revised: Speciﬁcally, we follow the practice in Caramalau et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2020) to adopt a special version 3 of ResNet-18 He et al.
Revised: (2020) to adopt a special version 3 of ResNet-18 He et al.

Original: as the task model to make the model compatible with the smaller image resolution (i.e. 32 × 32 ) in Cifar10/100 and SVHN.
Revised: as the task model to make the model compatible with the smaller image resolution (i.e. 32 × 32 ) in Cifar10/100 and SVHN.

Original: For Caltech101, we use the original ResNet-18.
Revised: For Caltech101, we use the original ResNet-18.

Original: Since an active learning algorithm is generally independent of network architectures, we use the same task model for all the compared methods in each experiment.
Revised: Since an active learning algorithm is generally independent of network architectures, we use the same task model for all the compared methods in each experiment.

Original: For instance, on Cifar10, the original CoreSet Sener & Savarese (2018) and VAAL Sinha et al.
Revised: For instance, on Cifar10, the original CoreSet Sener & Savarese (2018) and VAAL Sinha et al.

Original: (2019) employ VGG-16 as the task model which will be replaced by ResNet-18 in our reproduction.
Revised: (2019) employ VGG-16 as the task model which will be replaced by ResNet-18 in our reproduction.

Original: In each cycle of the Cifar and SVHN experiments, we use a batch size of 128 samples to train the task model with an initial learning rate of 0.1.
Revised: In each cycle of the Cifar and SVHN experiments, we use a batch size of 128 samples to train the task model with an initial learning rate of 0.1.

Original: The training continues for 200 epochs and we decay the learning rate with 0.1 at the 160 th epoch.
Revised: The training continues for 200 epochs and we decay the learning rate with 0.1 at the 160 th epoch.

Original: In Caltech101, the training will continue for 50 epochs and we choose 0.01 as the initial learning rate, which will be decayed by 0.1 at the 40 th epoch.
Revised: In Caltech101, the training will continue for 50 epochs and we choose 0.01 as the initial learning rate, which will be decayed by 0.1 at the 40 th epoch.

Original: Due to the higher image resolution, a smaller batch size of 64 is used to avoid GPU memory issues.
Revised: Due to the higher image resolution, a smaller batch size of 64 is used to avoid GPU memory issues.

Original: In order to make the evaluations not impacted by different optimization routines, we employ the same optimizer (i.e. SGD (Bottou, 2010)) for all the experiments of image classiﬁcation.
Revised: In order to make the evaluations not impacted by different optimization routines, we employ the same optimizer (i.e. SGD (Bottou, 2010)) for all the experiments of image classiﬁcation.

Original: A momentum of 0.9 and a weight decay rate of 5 × 10 − 4 are used with the optimizer.
Revised: A momentum of 0.9 and a weight decay rate of 5 × 10 − 4 are used with the optimizer.

Original: Observations.
Revised: Observations.

Original: We compare our method with the state-of-the-art baselines and illustrate the results in Figure 1.
Revised: We compare our method with the state-of-the-art baselines and illustrate the results in Figure 1.

Original: The baseline methods include random selection, MC-Dropout (Gal & Ghahramani, 2016), QBC (Kuo et al., 2018), CoreSet (Sener & Savarese, 2018), VAAL (Sinha et al., 2019),
Revised: The baseline methods include random selection, MC-Dropout (Gal & Ghahramani, 2016), QBC (Kuo et al., 2018), CoreSet (Sener & Savarese, 2018), VAAL (Sinha et al., 2019),

Original: LL4AL (Yoo & Kweon, 2019), SRAAL (Zhang et al., 2020) and GCN (Caramalau et al., 2021).
Revised: LL4AL (Yoo & Kweon, 2019), SRAAL (Zhang et al., 2020) and GCN (Caramalau et al., 2021).

Original: For all these algorithms, we use the recommended hyper-parameters as suggested in their original papers.
Revised: For all these algorithms, we use the recommended hyper-parameters as suggested in their original papers.

Original: For the proposed noise stability, we use default hyper-parameters in all experiments: σ 2 = 1 ,
Revised: For the proposed noise stability, we use default hyper-parameters in all experiments: σ 2 = 1 ,

Original: K = 10 and ξ = 10 − 3 (cid:107) θ (cid:107) (cid:107) ∆ θ (cid:107) to ensure the relative noise magnitude constrained.
Revised: K = 10 and ξ = 10 − 3 (cid:107) θ (cid:107) (cid:107) ∆ θ (cid:107) to ensure the relative noise magnitude constrained.

Original: As can be seen in Figure 1, the proposed method ( NoiseStability ) consistently outperforms the baselines (in terms of accuracy) in each active learning cycle.
Revised: As can be seen in Figure 1, the proposed method ( NoiseStability ) consistently outperforms the baselines (in terms of accuracy) in each active learning cycle.

Original: In particular, for harder datasets such as
Revised: In particular, for harder datasets such as

Original: Cifar10, Cifar100 and Caltech, our method achieves remarkably higher ﬁnal accuracy against the others.
Revised: Cifar10, Cifar100 and Caltech, our method achieves remarkably higher ﬁnal accuracy against the others.

Original: For the easier task SVHN, our method yields a more rapid and stable increase of accuracy at the ﬁrst several cycles, more competitive than the baselines.
Revised: For the easier task SVHN, our method yields a more rapid and stable increase of accuracy at the ﬁrst several cycles, more competitive than the baselines.

Original: In addition to the general comparison, we present a detailed analysis below to demonstrate more merits of our method.
Revised: In addition to the general comparison, we present a detailed analysis below to demonstrate more merits of our method.

Original: • In the later cycles (e.g. 35% and 40%), our method usually exceeds the others by a larger margin (e.g. in Cifar10 and Caltech101).
Revised: • In the later cycles (e.g. 35% and 40%), our method usually exceeds the others by a larger margin (e.g. in Cifar10 and Caltech101).

Original: This indicates that when the task model remembers more samples (i
Revised: This indicates that when the task model remembers more samples (i

Original: e. the trained model is more likely to behave over-conﬁdent), estimating uncertainty by our noise stability is much more accurate than other competitors.
Revised: e. the trained model is more likely to behave over-conﬁdent), estimating uncertainty by our noise stability is much more accurate than other competitors.

Original: • The superior performance in the Caltech101 experiment suggests that our method is more suitable for imbalanced datasets than the existing baselines.
Revised: • The superior performance in the Caltech101 experiment suggests that our method is more suitable for imbalanced datasets than the existing baselines.

Original: We conjecture this is due to the simplicity of our method, since training helper models (Kuo et al., 2018; Sinha et al., 2019; Zhang et al., 2020; Caramalau et al., 2021) with imbalanced data could be adverse to these helpers themselves, thus leading to an inferior performance of the task model.
Revised: We conjecture this is due to the simplicity of our method, since training helper models (Kuo et al., 2018; Sinha et al., 2019; Zhang et al., 2020; Caramalau et al., 2021) with imbalanced data could be adverse to these helpers themselves, thus leading to an inferior performance of the task model.

Original: Lastly, we observe that in the Cifar10 experiment, the performance of our method (94.25%) exceeds that (93.6%) of the same model which is trained on the entire dataset without considering active learning.
Revised: Lastly, we observe that in the Cifar10 experiment, the performance of our method (94.25%) exceeds that (93.6%) of the same model which is trained on the entire dataset without considering active learning.

Original: This counter-intuitive observation indicates that training only with a subset of hard samples may make DNNs generalize better than training with an entire set.
Revised: This counter-intuitive observation indicates that training only with a subset of hard samples may make DNNs generalize better than training with an entire set.

Original: 4. S EMANTIC S EGMENTATION
Revised: 4. S EMANTIC S EGMENTATION

Original: Semantic segmentation is more complex than image classiﬁcation due to the requirement of high classiﬁcation accuracy at pixel level.
Revised: Semantic segmentation is more complex than image classiﬁcation due to the requirement of high classiﬁcation accuracy at pixel level.

Original: As a consequence, in several baseline methods, such as VAAL
Revised: As a consequence, in several baseline methods, such as VAAL

Original: Sinha et al.
Revised: Sinha et al.

Original: , SRAAL Zhang et al.
Revised: , SRAAL Zhang et al.

Original: (2020) and GCN Caramalau et al.
Revised: (2020) and GCN Caramalau et al.

Original: (2021), the auxiliary models need careful design in order to achieve an satisfactory performance.
Revised: (2021), the auxiliary models need careful design in order to achieve an satisfactory performance.

Original: This increases the burden of deploying Al algorithms in practice.
Revised: This increases the burden of deploying Al algorithms in practice.

Original: In contrast, our method is free of extra design and can be easily adopted for this problem, demonstrating its task-agnostic advantage.
Revised: In contrast, our method is free of extra design and can be easily adopted for this problem, demonstrating its task-agnostic advantage.

Original: Dataset.
Revised: Dataset.

Original: We use the Cityscapes Cordts et al.
Revised: We use the Cityscapes Cordts et al.

Original: (2016), a large-scale street scene dataset.
Revised: (2016), a large-scale street scene dataset.

Original: It includes images taken under various weather conditions in different seasons from 50 cities, making the task on it more challenging.
Revised: It includes images taken under various weather conditions in different seasons from 50 cities, making the task on it more challenging.

Original: We consider the standard training and validation data of this set.
Revised: We consider the standard training and validation data of this set.

Original: We crop
Revised: We crop

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: the images to a dimension of 688 × 688 , and there are 19 categories (classes) in total for each pixel.
Revised: the images to a dimension of 688 × 688 , and there are 19 categories (classes) in total for each pixel.

Original: It is worth noting that the cropping size also depends on the task model, discussed in next section.
Revised: It is worth noting that the cropping size also depends on the task model, discussed in next section.

Original: Model and Training.
Revised: Model and Training.

Original: As a common practice Zhang et al.
Revised: As a common practice Zhang et al.

Original: (2020), we choose a dilated residual network (DRN-D-22) (Yu et al., 2017) as the task model.
Revised: (2020), we choose a dilated residual network (DRN-D-22) (Yu et al., 2017) as the task model.

Original: It includes 8 layer modules and each module is composed of 1 or 2 BasicBlock of layers.
Revised: It includes 8 layer modules and each module is composed of 1 or 2 BasicBlock of layers.

Original: One can refer to the original paper (Yu et al., 2017) for the network details .
Revised: One can refer to the original paper (Yu et al., 2017) for the network details .

Original: Similar to the classiﬁcation evaluations, we train the model for 7 cycles and 50 epochs for each cycle.
Revised: Similar to the classiﬁcation evaluations, we train the model for 7 cycles and 50 epochs for each cycle.

Original: Following the suggestion in (Yu et al., 2017), we use the Adam optimizer (Kingma & Ba, 2014).
Revised: Following the suggestion in (Yu et al., 2017), we use the Adam optimizer (Kingma & Ba, 2014).

Original: We choose a learning rate of 1 × 10 − 4 , remained unchanged in the entire training.
Revised: We choose a learning rate of 1 × 10 − 4 , remained unchanged in the entire training.

Original: For the proposed noise stability, we set K = 3 and keep the remaining hyper-parameters the same as speciﬁed in the classiﬁcation experiments.
Revised: For the proposed noise stability, we set K = 3 and keep the remaining hyper-parameters the same as speciﬁed in the classiﬁcation experiments.

Original: Observations.
Revised: Observations.

Original: We compare our method with MC-Dropout (Gal & Ghahramani, 2016), QBC (Kuo et al., 2018), CoreSet (Sener & Savarese, 2018), VAAL (Sinha et al., 2019), SRAAL (Zhang et al., 2020) in this experiment.
Revised: We compare our method with MC-Dropout (Gal & Ghahramani, 2016), QBC (Kuo et al., 2018), CoreSet (Sener & Savarese, 2018), VAAL (Sinha et al., 2019), SRAAL (Zhang et al., 2020) in this experiment.

Original: We illustrate the results in Figure 2.
Revised: We illustrate the results in Figure 2.

Original: We observe that the improvement of the proposed method is solid.
Revised: We observe that the improvement of the proposed method is solid.

Original: Given Cityscapes is highy imbalanced, this experiment once again demonstrates the superiority of our method on imbalanced datasets.
Revised: Given Cityscapes is highy imbalanced, this experiment once again demonstrates the superiority of our method on imbalanced datasets.

Original: More importantly, from classiﬁcation to segmentation, our method can easily adapt to the new task by simply sampling and adding noise to the output (without being affected by the change of the output format).
Revised: More importantly, from classiﬁcation to segmentation, our method can easily adapt to the new task by simply sampling and adding noise to the output (without being affected by the change of the output format).

Original:  D ISCUSSIONS
Revised:  D ISCUSSIONS

Original: In this section, we provide further analysis on the effect of the proposed method.
Revised: In this section, we provide further analysis on the effect of the proposed method.

Original: First in Section 5.1, we discuss the choice of hyper-parameters used in noise stability.
Revised: First in Section 5.1, we discuss the choice of hyper-parameters used in noise stability.

Original: Then we evaluate our algorithm with fewer cycles and larger selection size in Section 5.2.
Revised: Then we evaluate our algorithm with fewer cycles and larger selection size in Section 5.2.

Original: Moreover, in Appendix A.4, we provide an empirical validation for the theoretical hypothesis that selecting samples by noise stability is conducive to reducing the output variance for the training set.
Revised: Moreover, in Appendix A.7, we provide an empirical validation for the theoretical hypothesis that selecting samples by noise stability is conducive to reducing the output variance for the training set.

Original: 5. S ENSITIVITY TO H YPER - PARAMETERS
Revised: 5. S ENSITIVITY TO H YPER - PARAMETERS

Original: We ﬁrst investigate how our algorithm is sensitive to the hyper-parameters (i.e. the noise magnitude, the noise distribution and the sampling times).
Revised: We ﬁrst investigate how our algorithm is sensitive to the hyper-parameters (i.e. the noise magnitude, the noise distribution and the sampling times).

Original: We conduct experiments on Cifar10 and Cifar100 for this analysis.
Revised: We conduct experiments on Cifar10 and Cifar100 for this analysis.

Original: Noise magnitude.
Revised: Noise magnitude.

Original: The criterion of noise stability in Eq (1) is sensitive to the noise magnitude.
Revised: The criterion of noise stability in Eq (1) is sensitive to the noise magnitude.

Original: Speciﬁcally, noise with too large magnitudes tends to destroy the learned knowledge in DNNs.
Revised: Speciﬁcally, noise with too large magnitudes tends to destroy the learned knowledge in DNNs.

Original: For simplicity, we denote by λ the relative (to the clean parameter) magnitude, i.e. ξ = λ (cid:107) θ (cid:107) (cid:107) ∆ θ ( i ) (cid:107) .
Revised: For simplicity, we denote by λ the relative (to the clean parameter) magnitude, i.e. ξ = λ (cid:107) θ (cid:107) (cid:107) ∆ θ ( i ) (cid:107) .

Original: We evaluate the choices of { 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 } for λ .
Revised: We evaluate the choices of { 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 } for λ .

Original: Results in Figure 3 (left) show that
Revised: Results in Figure 3 (left) show that

Original: [Figure]
Revised: [Figure]

Original: using λ between 10 − 4 and 10 − 2 is likely to yield good performance.
Revised: using λ between 10 − 4 and 10 − 2 is likely to yield good performance.

Original: From the various experiments of this work, we observe that a default choice of 10 − 3 would be suitable in general.
Revised: From the various experiments of this work, we observe that a default choice of 10 − 3 would be suitable in general.

Original: Sampling times.
Revised: Sampling times.

Original: Accurate approximations such as that for Eq (1) usually require a considerable sampling times.
Revised: Accurate approximations such as that for Eq (1) usually require a considerable sampling times.

Original: However, we surprisingly ﬁnd that the noise stability scheme used in our algorithm is very insensitive to the sampling times K .
Revised: However, we surprisingly ﬁnd that the noise stability scheme used in our algorithm is very insensitive to the sampling times K .

Original: We evaluate K with choices of { 1 , 3 , 10 , 30 , 50 } and observe very similar results, as shown in Figure 3 (middle).
Revised: We evaluate K with choices of { 1 , 3 , 10 , 30 , 50 } and observe very similar results, as shown in Figure 3 (middle).

Original: We conjecture this might be because the uncertainty estimation in active learning does not require an accurate approximation of the exact value, even nor the exact order.
Revised: We conjecture this might be because the uncertainty estimation in active learning does not require an accurate approximation of the exact value, even nor the exact order.

Original: The main factor that impacts the ﬁnal results is only the selected subset of data.
Revised: The main factor that impacts the ﬁnal results is only the selected subset of data.

Original: Noise distribution.
Revised: Noise distribution.

Original: Note that our method is not limited to the multivariate Gaussian distribution.
Revised: Note that our method is not limited to the multivariate Gaussian distribution.

Original: Any distribution with all the elements independent of each other satisﬁes the theoretical properties.
Revised: Any distribution with all the elements independent of each other satisﬁes the theoretical properties.

Original: Here we evaluate two alternative noise distributions, which are the Uniform distribution ∆ θ i ∼
Revised: Here we evaluate two alternative noise distributions, which are the Uniform distribution ∆ θ i ∼

Original: U( − 1 , 1) and the zero-mean Laplace distribution ∆ θ i ∼ Laplace(0 , b ) with b = 1 .
Revised: U( − 1 , 1) and the zero-mean Laplace distribution ∆ θ i ∼ Laplace(0 , b ) with b = 1 .

Original: Results in
Revised: Results in

Original: Figure 3 (right) show that both of them achieve similar good performance as the normal distribution.
Revised: Figure 3 (right) show that both of them achieve similar good performance as the normal distribution.

Original: 5. P ERFORMANCE ON L ARGER S ELECTION S IZE
Revised: 5. P ERFORMANCE ON L ARGER S ELECTION S IZE

Original: It remains a long-term open problem that whether data uncertainty and diversity are empirically consistent or in a trade-off relationship.
Revised: It remains a long-term open problem that whether data uncertainty and diversity are empirically consistent or in a trade-off relationship.

Original: Instead of directly addressing this dilemma, we design an experiment to investigate how our algorithm performs with an increased selection size 4 .
Revised: Instead of directly addressing this dilemma, we design an experiment to investigate how our algorithm performs with an increased selection size 4 .

Original: As the training pool is enlarged by adding a large amount of samples, both uncertainty and diversity tend to be crucial to further training, e.g. highly imbalanced (lack of diversity) data is deﬁnitely harmful to learning.
Revised: As the training pool is enlarged by adding a large amount of samples, both uncertainty and diversity tend to be crucial to further training, e.g. highly imbalanced (lack of diversity) data is deﬁnitely harmful to learning.

Original: For efﬁciency, we perform a single cycle of sample selection after cycle 1.
Revised: For efﬁciency, we perform a single cycle of sample selection after cycle 1.

Original: We use different algorithms to select unlabeled samples with the labeling budgets of 10%, 15%, 20%, 25%, 35% and 40% respectively.
Revised: We use different algorithms to select unlabeled samples with the labeling budgets of 10%, 15%, 20%, 25%, 35% and 40% respectively.

Original: Note that it is different from the standard experiments presented in Figure 1, since the sample selection here is one-off rather than iterative.
Revised: Note that it is different from the standard experiments presented in Figure 1, since the sample selection here is one-off rather than iterative.

Original: Results in Figure 4 demonstrate that our method still consistently outperforms the SOTA baseline GCN Caramalau et al.
Revised: Results in Figure 4 demonstrate that our method still consistently outperforms the SOTA baseline GCN Caramalau et al.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Active learning has been a long-term open research topic in the community.
Revised: Active learning has been a long-term open research topic in the community.

Original: Most of the methods can be categorized into three types including uncertainty-based, diversity-based and synthesis-based methods.
Revised: Most of the methods can be categorized into three types including uncertainty-based, diversity-based and synthesis-based methods.

Original: In this section we provide a brief review of these typical studies.
Revised: In this section we provide a brief review of these typical studies.

Original: Uncertainty-based.
Revised: Uncertainty-based.

Original: Traditional methods solve the problem mainly via estimating uncertainty for unlabeled data.
Revised: Traditional methods solve the problem mainly via estimating uncertainty for unlabeled data.

Original: Some typical works (Joshi et al., 2009; Lewis & Catlett, 1994; Lewis & Gale, 1994;
Revised: Some typical works (Joshi et al., 2009; Lewis & Catlett, 1994; Lewis & Gale, 1994;

Original: Roth & Small, 2006; Tong & Koller, 2001; Vijayanarasimhan & Grauman, 2014; Li & Guo, 2014) have been proposed for shallow or statistical models.
Revised: Roth & Small, 2006; Tong & Koller, 2001; Vijayanarasimhan & Grauman, 2014; Li & Guo, 2014) have been proposed for shallow or statistical models.

Original: Nevertheless, as the rapid development of deep learning, a new challenge has emerged.
Revised: Nevertheless, as the rapid development of deep learning, a new challenge has emerged.

Original: That is, how to estimate data uncertainty with deep neural
Revised: That is, how to estimate data uncertainty with deep neural

Original: [Figure]
Revised: [Figure]

Original: networks (DNNs).
Revised: networks (DNNs).

Original: Several methods (Sinha et al., 2019; Yoo & Kweon, 2019; Zhang et al., 2020;
Revised: Several methods (Sinha et al., 2019; Yoo & Kweon, 2019; Zhang et al., 2020;

Original: Mackowiak et al., 2018) have been proposed to address this challenge with the aid of specialized
Revised: Mackowiak et al., 2018) have been proposed to address this challenge with the aid of specialized

Original: DNNs and training fashions, such as VAE (variational auto-encoder) (Kingma & Welling, 2013) and adversarial learning.
Revised: DNNs and training fashions, such as VAE (variational auto-encoder) (Kingma & Welling, 2013) and adversarial learning.

Original: Sener & Savarese (2018); Kuo et al.
Revised: Sener & Savarese (2018); Kuo et al.

Original: (2018) attempt to combine a classical algorithm (e.g. K-center or 0-1 Knapsack problem) with DNNs to select the most uncertain data for annotation.
Revised: (2018) attempt to combine a classical algorithm (e.g. K-center or 0-1 Knapsack problem) with DNNs to select the most uncertain data for annotation.

Original: Note that our method belongs to this category.
Revised: Note that our method belongs to this category.

Original: In terms of methodology, the most similar existing work to ours is MC-Dropout (Gal & Ghahramani, 2016).
Revised: In terms of methodology, the most similar existing work to ours is MC-Dropout (Gal & Ghahramani, 2016).

Original: However, the motivation is quite different in that we adopt noise stability to estimate uncertainty, whereas MC-Dropout resorts to Bayesian inference to approximate the uncertainty (measured in the output entropy).
Revised: However, the motivation is quite different in that we adopt noise stability to estimate uncertainty, whereas MC-Dropout resorts to Bayesian inference to approximate the uncertainty (measured in the output entropy).

Original: Another essential difference lies in that, noise stability only needs a local sampling that generates parameter noise around a speciﬁc solution, whereas Bayesian inference requires global sampling over the entire hypothesis space.
Revised: Another essential difference lies in that, noise stability only needs a local sampling that generates parameter noise around a speciﬁc solution, whereas Bayesian inference requires global sampling over the entire hypothesis space.

Original: As a consequence, the necessary sampling times of MC-Dropout (e.g. ∼ 100 or ∼ 1000 ) is orders of magnitude larger than noise stability.
Revised: As a consequence, the necessary sampling times of MC-Dropout (e.g. ∼ 100 or ∼ 1000 ) is orders of magnitude larger than noise stability.

Revised: Our work is also relevant to recent studies on linearised Laplace approximation of the BNN posterior (Immer et al., 2021; Daxberger et al., 2021).

Revised: We put a more detailed discussion in Appendix A.8.

Original: Diversity-based.
Revised: Diversity-based.

Original: In addition to the uncertainty-based methods, several works attempt to select unlabeled data that can diversify the labeled pool.
Revised: In addition to the uncertainty-based methods, several works attempt to select unlabeled data that can diversify the labeled pool.

Original: This category of methods are also known as distribution-based (Bilgic & Getoor, 2009; Elhamifar et al., 2013; Freytag et al., 2014; Hasan & RoyChowdhury, 2015; Mac Aodha et al., 2014; Nguyen & Smeulders, 2004; Roy & McCallum, 2001;
Revised: This category of methods are also known as distribution-based (Bilgic & Getoor, 2009; Elhamifar et al., 2013; Freytag et al., 2014; Hasan & RoyChowdhury, 2015; Mac Aodha et al., 2014; Nguyen & Smeulders, 2004; Roy & McCallum, 2001;

Original: Yang et al., 2015; Settles et al., 2008).
Revised: Yang et al., 2015; Settles et al., 2008).

Original: Although the diversity-based methods were traditionally treated as essentially different from the uncertainty-based methods, it is worth noting that a recent study (Loquercio et al., 2020) argues that data diversity is correlated to data uncertainty.
Revised: Although the diversity-based methods were traditionally treated as essentially different from the uncertainty-based methods, it is worth noting that a recent study (Loquercio et al., 2020) argues that data diversity is correlated to data uncertainty.

Original: Synthesis-based.
Revised: Synthesis-based.

Original: Another typical category of methods aim to diversify the labeled pool by synthesizing new data that can beneﬁt the task model training.
Revised: Another typical category of methods aim to diversify the labeled pool by synthesizing new data that can beneﬁt the task model training.

Original: Typical methods (Zhu & Bento, 2017;
Revised: Typical methods (Zhu & Bento, 2017;

Original: Mahapatra et al., 2018; Mayer & Timofte, 2020) resort to generative models, such as GAN (Goodfellow et al., 2014) or VAE (Kingma & Welling, 2013) to synthesize diverse data samples to train the task model.
Revised: Mahapatra et al., 2018; Mayer & Timofte, 2020) resort to generative models, such as GAN (Goodfellow et al., 2014) or VAE (Kingma & Welling, 2013) to synthesize diverse data samples to train the task model.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposes a simple method that utilizes noise stability for uncertainty estimation in active learning.
Revised: This paper proposes a simple method that utilizes noise stability for uncertainty estimation in active learning.

Original: For an input sample, noise stability measures how stable can the output be when a random perturbation is added on the model parameters.
Revised: For an input sample, noise stability measures how stable can the output be when a random perturbation is added on the model parameters.

Original: Our theoretical analysis prove that the noise stability approximates Jacobian norm w.r.t.
Revised: Our theoretical analysis prove that the noise stability approximates Jacobian norm w.r.t.

Original: the parameters assuming the noise magnitude is small.
Revised: the parameters assuming the noise magnitude is small.

Original: Theoretical and empirical analysis also demonstrate that, selecting those samples mostly unstable to parameter noise has the effect of reducing the prediction variance of the training samples.
Revised: Theoretical and empirical analysis also demonstrate that, selecting those samples mostly unstable to parameter noise has the effect of reducing the prediction variance of the training samples.

Original: Experiments on image classiﬁcation, semantic segmentation, and3D cryo-ET subtomogram classiﬁcation demonstrate the efﬁcacy of the proposed method.
Revised: Experiments on image classiﬁcation, semantic , and demonstrate the efﬁcacy of the proposed method.

Original: A A PPENDIX A. S TANDARD M ULTIVARIATE G UASSIAN D ISTRIBUTION
Revised: A A PPENDIX A. S TANDARD M ULTIVARIATE G UASSIAN D ISTRIBUTION

Original: The univariate Gaussian distribution is often referred to as N ( µ, σ 2 )  where µ denotes the expectation of the distribution and σ is the standard deviation.
Revised: The univariate Gaussian distribution is often referred to as N ( µ, σ 2 ) , where µ denotes the expectation of the distribution and σ is the standard deviation.

Original: N ( µ, σ 2 ) can be described by its probability density function as p ( x ) =
Revised: N ( µ, σ 2 ) can be described by its probability density function as p ( x ) =

Original: [Equation]
Revised: [Equation]

Original: Multivariate Gaussian distribution x ∼ N ( µ , Σ ) , is a generalized form for the vector variable, where µ is the mean vector and Σ is the covariance matrix.
Revised: Multivariate Gaussian distribution x ∼ N ( µ , Σ ) , is a generalized form for the vector variable, where µ is the mean vector and Σ is the covariance matrix.

Original: In this work, we shall focus on the standard case in which µ is the zero vector 0 and Σ is a diagonal matrix with all diagonal elements being σ 2 , i.e. Σ = σ 2 I .
Revised: In this work, we shall focus on the standard case, in which µ is the zero vector 0 and Σ is a diagonal matrix with all diagonal elements being σ 2 , i.e. Σ = σ 2 I .

Original: Note that if x ∼ N ( 0 , σ 2 I ) , all the samplings { x i } are independent of each other and each conforms to the standard univariate
Revised: Note that if x ∼ N ( 0 , σ 2 I ) , all the samplings { x i } are independent of each other, and each conforms to the standard univariate

Original: Gaussian distribution as x i ∼ N (0 , σ 2 ) .
Revised: Gaussian distribution as x i ∼ N (0 , σ 2 ) .

Original: A. P ROOF OF E Q (5)
Revised: A. P ROOF OF E Q (5)

Original: We ﬁrst assume f ( . ; θ ) is a real-valued function and the Jacobian is a vector.
Revised: We ﬁrst assume f ( . ; θ ) is a real-valued function and the Jacobian is a vector.

Original: If ∆ θ ∈ R d conforms to the standard multivariate Gaussian distribution that ∆ θ ∼ N ( 0 , σ 2 I ) , each of its elements are independent with each other.
Revised: If ∆ θ ∈ R d conforms to the standard multivariate Gaussian distribution that ∆ θ ∼ N ( 0 , σ 2 I ) , each of its elements are independent with each other.

Original: That is, for any 1 ≤ i, j ≤ d , the following is true:
Revised: That is, for any 1 ≤ i, j ≤ d , the following is true:

Original: [Equation]
Revised: [Equation]

Original:  where δ ij = 1{+
+} i = j .
Revised: (11) where δ ij = {+
+} i = j .

Original: Without ambiguity, here we use J as a shorthand for J θ ( x ; θ ) .
Revised: Without ambiguity, here we use J as a shorthand for J θ ( x ; θ ) .

Original: The notation of
Revised: The notation of

Original: Gaussian distribution under the expectation is also omitted for simplicity.
Revised: Gaussian distribution under the expectation is also omitted for simplicity.

Original: Proof.
Revised: Proof.

Original: Based on Eq (11) [Equation] we can expand the inner product between J and ∆ θ in Eq (4) as
Revised: Based on Eq (11), we can expand the inner product between J and ∆ θ in Eq (4) as

Original: [Equation]
Revised: [Equation]

Original: When f ( . ; θ ) is vector-valued, we can straightforwardly put each row of the Jacobian matrix into Eq (12) and ﬁnally get
Revised: When f ( . ; θ ) is vector-valued, we can straightforwardly put each row of the Jacobian matrix into Eq (12) and ﬁnally get

Original: [Equation]
Revised: [Equation]

Original: Note that, the conclusion is true for any other zero-mean distribution as long as it satisﬁes the independence prerequisite and has a deﬁnite variance.
Revised: Note that, the conclusion is true for any other zero-mean distribution as long as it satisﬁes the independence prerequisite and has a deﬁnite variance.

Original: The multivariate Gaussian distribution is used only as a typical example.
Revised: The multivariate Gaussian distribution is used only as a typical example.

Original: A.  C RYO -ET C HALLENGES
Revised: A. M ORE C OMPARISONS WITH D EEP B AYESIAN A CTIVE L EARNING

Revised: In this section, we compare our method with more deep Bayesian active learning methods on MNIST and Cifar10. The deep Bayesian method is one of the most important schemes for uncertainty estimation. We evaluate MC-Dropout Gal & Ghahramani (2016), BALD Houlsby et al.

Revised: BatchBALD Kirsch et al.

Original: Cryo-Electron Tomography (cryo-ET) is an important imaging technique to study biological macromolecules and cells, used to produce their high-resolution 3D tomograms of cellular landscapes and structures.
Revised: and Checkpoint Ensemble Chitta et al. For all the dropout based methods (MC-Dropout, BALD and BatchBALD), which can be regarded as a kind of implicit ensemble, we set the number of Monte-Carlo sampling to 50. For Checkpoint Ensemble, we save the checkpoints for the last 10 epochs as the ensemble models. BALD (mutual information) is used as the acquisition function for the Ensemble method.

Revised: For our method, we still sample noise fortimes.

Revised: Model and Training.

Revised: For Cifar10, we use ResNet-18 with exactly the same settings as described in

Revised: Section 4.1.

Revised: For MNIST, we employ a small convolutional network composed of two 5 × 5 convolutional layers, followed by two fully connected layers.

Revised: We use the Adam optimizer with a learning rate of 0.001.

Revised: For each AL cycle, we train the model for 50 epochs on the labeled pool.

Revised: The initial labeled pool consists of 50 samples, and another 50 unlabeled samples are selected for annotation at each AL cycle.

Revised: Observations.

Revised: We present the results in Figure 5.

Revised: We observe that the basic deep Bayesian methods, such as MC-Dropout and BALD, perform better than Random selection on both datasets.

Revised: The advanced Bayesian method BatchBALD does not show superiority, compared with BALD on MNIST.

Revised: On Cifar10 with a selection size of 2500, BatchBALD almost degenerates to Random selection, and we conjecture this is due to the inaccurate joint entropy estimation.

Revised: Though using a smaller sampling number, the Ensemble and our NoiseStability methods perform the best.

Revised: Similar to our method, for MC-Dropout, BALD and BatchBALD, further increasing the number of Monte-Carlo samplings (e.g. an increment to 100) cannot bring any performance gain on the two datasets.

Revised: [Figure]

Revised: A. E XPERIMENTS ON A DDITIONAL S ETTINGS

Revised: Here we present more experimental results to evaluate our algorithm

Revised: . A.4. E XPERIMENTS ON M ORE D ATA D OMAINS

Revised: In this part, we evaluate AL methods on the Cryo-ET Challenges.

Revised: Cryo-Electron Tomography (cryoET) is an important imaging technique to study biological macromolecules and cells, used to produce their high-resolution 3D tomograms of cellular landscapes and structures.

Original: Active learning is of great interest for such kind of tasks since the annotation cost is extremely high.
Revised: Active learning is of great interest for such kind of tasks since the annotation cost is extremely high.

Original: We evaluate our active learning algorithm on cryo-ET subtomogram classiﬁcation (Chen et al., 2017), demonstrating the capability of our method in various domains.
Revised: We evaluate our active learning algorithm on cryo-ET subtomogram classiﬁcation (Chen et al., 2017), demonstrating the capability of our method in various domains.

Original: Dataset.
Revised: Dataset.

Original: We compare our noise stability against the baseline methods on one simulated cryo-ET dataset consisting of subtomogram images contaminated by noise with a SNR (signal-noise-ratio) of 0.03.
Revised: We compare our noise stability against the baseline methods on one simulated cryo-ET dataset consisting of subtomogram images contaminated by noise with a SNR (signal-noise-ratio) of 0.03.

Original: Given different conformational states, all the samples in this dataset are categorized intoclasses, namely 50c-snr003 .
Revised: Given different conformational states, all the samples in this dataset are categorized intoclasses, namely 50c-snr003 .

Original: There are 24000 training and 1000 test samples, uniformly distributed across all the classes.
Revised: There are 24000 training and 1000 test samples, uniformly distributed across all the classes.

Original: One can refer to Gubins et al.
Revised: One can refer to Gubins et al.

Original: for how to generate the data.
Revised: for how to generate the data.

Original: Model and Training.
Revised: Model and Training.

Original: The dimension of the subtomogram data (i.e. 32 × 32 × 32 ) is different with that of RGB images, making the classiﬁcation task more challenging.
Revised: The dimension of the subtomogram data (i.e. 32 × 32 × 32 ) is different with that of RGB images, making the classiﬁcation task more challenging.

Original: To adapt to this particular data dimension, we employ a customized ResNet-18 with 3D convolutional layers as the task model.
Revised: To adapt to this particular data dimension, we employ a customized ResNet-18 with 3D convolutional layers as the task model.

Original: In addition to the task model, in the baseline methods one also needs to customize the auxiliary models (e.g. VAE in VAAL (Sinha et al., 2019) and SRAAL (Zhang et al., 2020), and the loss prediction module in LL4AL (Yoo & Kweon, 2019)), whereas our method is free of this extra implementation.
Revised: In addition to the task model, in the baseline methods one also needs to customize the auxiliary models (e.g. VAE in VAAL (Sinha et al., 2019) and SRAAL (Zhang et al., 2020), and the loss prediction module in LL4AL (Yoo & Kweon, 2019)), whereas our method is free of this extra implementation.

Original: The SGD optimizer (Bottou, 2010) is employed to train the task model.
Revised: The SGD optimizer (Bottou, 2010) is employed to train the task model.

Original: The training will continue for 100 epochs and the initial learning rate is set as 0.1, and we decay it by 0.1 at the 80 th epoch.
Revised: The training will continue for 100 epochs and the initial learning rate is set as 0.1, and we decay it by 0.1 at the 80 th epoch.

Original: We choose a momentum of 0.9 and a weight decay rate of 5 × 10 − 4 , respectively.
Revised: We choose a momentum of 0.9 and a weight decay rate of 5 × 10 − 4 , respectively.

Original: For the proposed noise stability, we set K = 5 and remain the hyper-parameters the same as speciﬁed in the classiﬁcation experiments.
Revised: For the proposed noise stability, we set K = 5 and remain the hyper-parameters the same as speciﬁed in the classiﬁcation experiments.

Original: [Figure]
Revised: [Figure]

Original: Observations.
Revised: Observations.

Original: We present the comparisons of the different methods in Figure 5.
Revised: We present the comparisons of the different methods in Figure 6.

Original: At each cycle, our method demonstrates solid improvement over the others.
Revised: At each cycle, our method demonstrates solid improvement over the others.

Original: Moreover, in the later cycles, our method outperforms the others by a signiﬁcant margin, demonstrating its consistent superiority.
Revised: Moreover, in the later cycles, our method outperforms the others by a signiﬁcant margin, demonstrating its consistent superiority.

Original: Moreover, due to the noise in the data, some other methods show reduced accuracy as cycle increases (e.g.
Revised: Moreover, due to the noise in the data, some other methods show reduced accuracy as cycle increases (e.g.

Original: CoreSet (Sener & Savarese, 2018), whereas our method keeps climbing towards a higher accuracy.
Revised: CoreSet (Sener & Savarese, 2018), whereas our method keeps climbing towards a higher accuracy.

Original: A. E MPIRICAL V ALIDATION OF V ARIANCE R EDUCTION
Revised: A.4. E XPERIMENTS ON R EGRESSION P ROBLEMS

Revised: Here we show that our method can also be applied to regression problems.

Revised: We evaluate several active learning algorithms on Ames Housing dataset De Cock (2011) to predict house prices.

Revised: Dataset.

Revised: The Ames Housing dataset De Cock (2011) describes the sales of individual residential properties in Ames, Iowa from 2006 to 2010.

Revised: The data set includes 1461 labeled training examples.

Revised: There are 60 explanatory variables involved in assessing home values.

Revised: The Ames Housing dataset is considered an expanded version of the classical Boston Housing dataset, which has only 506 labeled samples and 14 attributes.

Revised: Model and Training.

Revised: We use 50% of the data for training set and the remaining for testing.

Revised: We follow common practices to transform the categorical features into numerical ones with one-hot encoding.

Revised: A four-layer MLP is employed to extract features of the input vectors, followed by another fully connected layer as the predictor.

Revised: To adapt the regression task, we re-implement MC-Dropout to compute the output variance among multiple predictions.

Revised: Methods such as BALD Houlsby et al.

Revised: (2011) and BatchBALD Kirsch et al.

Revised: (2019), which rely on the output entropy are not suitable for regression tasks.

Revised: The Adam optimizer is employed to train the task model.

Revised: The training will continue for 300 epochs and the learning rate is set as 0.001.

Revised: For the proposed noise stability, we set K = 10 and remain the hyper-parameters the same as speciﬁed in the classiﬁcation experiments.

Revised: For MC-Dropout, we set the number of Monte-Carlo sampling to 50 as usually recommended.

Revised: Observations.

Revised: We start the training with 20 labeled samples and iteratively acquire 20 unlabeled samples for annotation.

Revised: As illustrated in Figure 7, our NoiseStability yields the lowest mean absolute error in most places.

Revised: Moreover, our selected data in earlier stages reduces the error more quickly than the others, demonstrating the advantage of our method when there is less labeled data in training.

Revised: As the training gets into later stages, our method also shows a superiority than most of the others, indicating its stability when more labeled data is involved in training.

Revised: [Figure]

Revised: A.4. E XPERIMENTS WITH S MALL S ELECTION S IZE

Revised: To further validate the effectiveness of our method, we consider smaller selection size 10 andrespectively on MNIST.

Revised: Though such very small selection sizes are not frequently used in practice, they can be treated as a popular indicator to evaluate the performance of an acquisition function.

Revised: Moreover, under the extreme case in which the selection size is 1, results can reveal whether an AL algorithm will successfully select the most informative or representative samples at each AL cycle, without concerning the problem of batch acquisition.

Revised: We use the same settings as in the MNIST experiments described in Appendix A.3.

Revised: As demonstrated in Figure 8, all the algorithms perform consistently when different selection sizes are used, i.e. algorithms perform well with larger selection sizes also have good performance with smaller selection sizes.

Revised: Under smaller selection sizes, all the state-of-the-art methods outperform Random selection by a large margin.

Revised: Moreover, our NoiseStability method yields the highest accuracy.

Revised: A

Revised: E XPERIMENTS

Revised: ON L ARGE S CALE D ATASETS

Revised: Our method can be easily applied to large scale datasets with a large selection size.

Revised: Here, we compare our method with Random selection and MC-Dropout on such a large dataset, namely ImageNet Deng et al.

Revised: Dataset.

Revised: The ImageNet dataset Deng et al.

Revised: (2009) consists of a total of 14 million images with more than 20K categories.

Revised: We use the popular subset ILSVRC2012 which includes around 1.3M images distributed across 1K categories.

Revised: We randomly select 15% of the total training samples to build the initial labeled pool and iteratively select 5% of the total training samples for annotation.

Revised: Note that the data selection is conducted without replacement, as we do in all the other experiments.

Revised: Model and Training.

Revised: We employ the standard ResNet-18 model for this task.

Revised: SGD is used as the optimizer with an initial learning rate of 0.1, momentum of 0.9, weight decay of 0.0005 and batch size of 256.

Revised: We train the model for 90 epochs following the practice in Pytorch ofﬁcial code.

Revised: The learning rate decays by a factor of 0.1 every 30 epochs.

Revised: [Figure]

Revised: Observations.

Revised: As illustrated in Figure

Revised: [Figure]

Revised: MC-Dropout and Noise Stability are consistently superior to the naive Random selection.

Revised: Our method signiﬁcantly outperforms the two baselines at the later

Revised: AL cycles

Revised: [Figure]

Revised: A.4. E XPERIMENTS ON M ORE A RCHITECTURES

Revised: Since the proposed method directly manipulates the task model, we conduct additional experiments on another architecture in order to validate the architecture-agnostic nature of the proposed method.

Revised: Speciﬁcally, we choose MobileNet-v2, a compact yet effective DNN architecture with fewer parameters.

Revised: We evaluate the AL algorithms on Cifar10 and Cifar100, using exactly the same settings as in the experiments in which ResNet-18 is used.

Revised: As shown in Figure 10, our method still yields very competitive performance.

Revised: In Cifar100, our method is on par with CoreSet

Revised: [Figure]

Revised: but it outperforms the others by a decent margin.

Revised: Note that in the regression experiment in Appendix A.4.2, our method is also shown effective on MLP architectures.

Revised: These results demonstrate that our method is independent of model architectures.

Revised: A. A DDITIONAL A BLATION S TUDY

Revised: Here we conduct an ablation study to analyze the two important designs in our method.

Revised: One is the adaptive noise magnitude.

Revised: The other is the position to inject noise.

Revised: [Figure]

Revised: [Figure]

Revised: A.5. A DAPTIVE N OISE M AGNITUDE

Revised: The noise magnitude should be proper, i.e. a very large magnitude will destroy the network and a very small one makes the model less capable to distinguish uncertain samples.

Revised: We adopt an adaptive strategy to control the noise magnitude.

Revised: In order to validate its effectiveness, we evaluate our method with several constant noise magnitudes including 1e-1, 1e-2, 1e-3 and 1e-4.

Revised: As shown in Figure( left ), our method performs much worse if the constant magnitude is too small or too large.

Revised: When a proper constant is used, i.e. 1e-2 or 1e-3, the performance is comparable to the adaptive strategy.

Revised: However, selecting a reasonable constant noise magnitude for each speciﬁc model sounds ad-hoc, since model parameters w.r.t.

Revised: different architectures usually have different value distributions.

Revised: More importantly, our experimental results on ResNet-18, MobileNet-v2 and MLP have demonstrated that the adaptive noise magnitude works well across various architectures, demonstrating the rationale of using the adaptive strategy.

Revised: A.5. P OSITION OF A PPLYING N OISE

Revised: Here we investigate how the position of applying noise affects the performance.

Revised: We take ResNetas the task model that includes four layer modules , each of which is composed of a sequence of

Revised: BasicBlocks .

Revised: In this experiment, we inject noise to each layer module respectively.

Revised: As shown in

Revised: Figure 11 ( right ), changing the position of noise injection does not affect the performance much.

Revised: A. R UNNING T IME C OMPARISON

Revised: To demonstrate the time efﬁciency, we report the running time of the acquisition function used in each AL baseline in Table 1.

Revised: We conduct the experiments on Cifar10, with a selection size of 2500 and 250, respectively.

Revised: The size of the corresponding unlabelled pool is 10000 and 20000, respectively.

Revised: The running time is measured on a single Nvidia Tesla V100 GPU.

Revised: For fair comparison, we set the number of dropout sampling to 10 for all the Bayesian methods.

Revised: As can be seen, our

Revised: NoiseStability is relatively efﬁcient among these AL baselines.

Revised: Moreover, the time complexity of our method increases approximately linearly as the size of the unlabelled pool is increasing, but is almost independent of the selection size.

Revised: [Table]

Revised: A. E MPIRICAL V ALIDATION OF V ARIANCE R EDUCTION

Original: Theoretical analysis in Section 3.2 reveals the connection between noise stability and variance reduction.
Revised: The theoretical analysis in Section 3.2 reveals the connection between our NoiseStability and variance reduction.

Original: Here we validate this connection, by quantifying the variance reduction for the training samples used in cycle 1, using the model trained in the subsequent cycle.
Revised: Here we validate this connection, by quantifying the variance reduction for the training samples used in cycle 1, using the model trained in the subsequent cycle.

Original: As the original form of output variance is intractable for DNNs (Thisted, 1988), we choose MC-Dropout (Gal & Ghahramani, 2016) as the criterion of variance estimation to evaluate our method and other competitors.
Revised: As the original form of output variance is intractable for DNNs (Thisted, 1988), we choose MC-Dropout (Gal & Ghahramani, 2016) as the criterion for variance estimation to evaluate our method and the others.

Original: Speciﬁcally, we approximate the uncertainty with 1) the conﬁdence represented by Bayesian posterior probability for the true label and 2) the prediction entropy.
Revised: Speciﬁcally, we approximate the uncertainty with 1) the conﬁdence represented by Bayesian posterior probability for the true label and 2) the prediction entropy.

Original: Results are shown in Table 1. While training with more samples always reduce the output variance for training samples, selecting samples with noise stability exhibits the most signiﬁcant effect. 
Revised: As illustrated in Table 2, while training with more samples always reduce the output variance, selecting samples with our NoiseStability exhibits the best effect. [Table] A. D ISCUSSION ABOUT MORE RELATED WORKS

Original:  of the true label (higher is better) and Entropy (lower is better) are used toapproximate the variance.
Revised: Studies in active learning with a linearised Laplace approximation of the BNN posterior (Immer et al., 2021; Daxberger et al., 2021) are very relevant to our method. Here we brieﬂy discussing their connections and differences. From a high level, our criterion is the minimum mean-squared error estimation (MMSE) while theirs are Bayesian estimation, corresponding to the frequentists’ and Bayesian perspective in statistics.

Original: We use the same random seed in  theall experiments.
Revised: They are not incompatible, yet they are inextricably linked. For instance, under the jointly Gaussian conditions (such as the Gaussian priors assumed in those papers) the MAP and MMSE are known to be equivalent; They face similar computational challenges as MMSE is the conditional mean that are as difﬁcult to calculate as the MAP. Our approach is to approximate full conditional mean by the linear MMSE, which only pertains to the mean and covariances. This is a principled approach that is also used in practice in other areas such as the

Original: The Conﬁdence and 
Revised: Kalman ﬁlter. In contrast, the Laplace approximation (e.g. (3) in Immer et al. (2021)) is accurate only when θ is close to the optimal value θ ∗ (where the ﬁrst order term is negligible) and thus the theory only provides local guarantees. It is our perspective that the condition is restrictive and does not apply to practical deep learning methods.

Original: Entropy at cycle 1 for training samples are 0.64 and 0.69, respectively.
Revised: The resulting algorithms are also different: Immer et al.

Original: [Table]
Revised: (2021) is based on the determinant of the Hessian matrix; the method in Daxberger et al. (2021) involves the diagonal of inverse Hessian. According to our ﬁndings, the predictive variance reduction conditioning on any direction is always monotonic decreasing with the gradient norm and servers as the cornerstone of our approach.