Original: S IMPLE AND E FFECTIVE S TOCHASTIC N EURAL N ETWORKS
Revised: S IMPLE AND E FFECTIVE S TOCHASTIC N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Stochastic neural networks (SNNs) are currently topical, with several paradigms being actively investigated including dropout, Bayesian neural networks, variational information bottleneck (VIB) and noise regularized learning.
Revised: Stochastic neural networks (SNNs) are currently topical, with several paradigms being actively investigated including dropout, Bayesian neural networks, variational information bottleneck (VIB) and noise regularized learning.

Original: These neural network variants impact several major considerations, including generalization, network compression, and robustness against adversarial attack and label noise.
Revised: These neural network variants impact several major considerations, including generalization, network compression, and robustness against adversarial attack and label noise.

Original: However, many existing networks are complicated and expensive to train, and/or only address one or two of these practical considerations.
Revised: However, many existing networks are complicated and expensive to train, and/or only address one or two of these practical considerations.

Original: In this paper we propose a simple and effective stochastic neural network (SE-SNN) architecture for discriminative learning by directly modeling activation uncertainty and encouraging high activation variability.
Revised: In this paper we propose a simple and effective stochastic neural network (SE-SNN) architecture for discriminative learning by directly modeling activation uncertainty and encouraging high activation variability.

Original: Compared to existing SNNs, our SE-SNN is simpler to implement and faster to train, and produces state of the art results on network compression by pruning, adversarial defense and learning with label noise.
Revised: Compared to existing SNNs, our SE-SNN is simpler to implement and faster to train, and produces state of the art results on network compression by pruning, adversarial defense and learning with label noise.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Stochastic neural networks (SNNs) have a long history.
Revised: Stochastic neural networks (SNNs) have a long history.

Original: Recently various stochastic neural network instantiations have been topical in their applications to reducing overﬁtting (Gal & Ghahramani, 2016;
Revised: Recently various stochastic neural network instantiations have been topical in their applications to reducing overﬁtting (Gal & Ghahramani, 2016;

Original: Neelakantan et al., 2015) and training data requirements (Garnelo et al., 2018), providing conﬁdence estimates on predictions (Gal & Ghahramani, 2016), enabling network compression (Dai et al., 2018), improving robustness to adversarial attack (Alemi et al., 2017), improving optimization (Neelakantan et al., 2015), generative modeling (Kingma & Welling, 2014), and inputting or producing probability distributions (de Bie et al., 2019; Frogner et al., 2019).
Revised: Neelakantan et al., 2015) and training data requirements (Garnelo et al., 2018), providing conﬁdence estimates on predictions (Gal & Ghahramani, 2016), enabling network compression (Dai et al., 2018), improving robustness to adversarial attack (Alemi et al., 2017), improving optimization (Neelakantan et al., 2015), generative modeling (Kingma & Welling, 2014), and inputting or producing probability distributions (de Bie et al., 2019; Frogner et al., 2019).

Original: One of the most theoretically appealing stochastic neural network formulations is Bayesian neural networks, which place a prior distribution on the weights of the network (Graves, 2011; Blundell et al., 2015; Ritter et al., 2018).
Revised: One of the most theoretically appealing stochastic neural network formulations is Bayesian neural networks, which place a prior distribution on the weights of the network (Graves, 2011; Blundell et al., 2015; Ritter et al., 2018).

Original: However this usually necessitates more complex learning and inference procedures that rely on variational approximations or sampling.
Revised: However this usually necessitates more complex learning and inference procedures that rely on variational approximations or sampling.

Original: Another group of works instead focus on modeling the uncertainty in neural network activations.
Revised: Another group of works instead focus on modeling the uncertainty in neural network activations.

Original: Notably the variational information bottleneck (VIB) approach (Alemi et al., 2017) is motivated by information theoretic considerations (Tishby et al., 1999) to learn a hidden representation that carries maximum information about the output and minimum information about the input.
Revised: Notably the variational information bottleneck (VIB) approach (Alemi et al., 2017) is motivated by information theoretic considerations (Tishby et al., 1999) to learn a hidden representation that carries maximum information about the output and minimum information about the input.

Original: Evaluating the required mutual information terms requires modeling probability distributions over activations rather than weights.
Revised: Evaluating the required mutual information terms requires modeling probability distributions over activations rather than weights.

Original: Deep VIB (Alemi et al., 2017) leads to improved generalization, adversarial robustness and model compression algorithms (Dai et al., 2018).
Revised: Deep VIB (Alemi et al., 2017) leads to improved generalization, adversarial robustness and model compression algorithms (Dai et al., 2018).

Original: Furthermore, modeling stochastic activations through noise is often practically useful for improving exploration and local minima escape during optimization, generalization and adversarial robustness (You et al., 2018; Noh et al., 2017; Bishop, 1995; Gulcehre et al., 2016), and in some cases can be linked back to Bayesian models of weights (Noh et al., 2017; Gal & Ghahramani, 2016) when the noise added at each activation can be considered as a result of different samples from the weight posterior.
Revised: Furthermore, modeling stochastic activations through noise is often practically useful for improving exploration and local minima escape during optimization, generalization and adversarial robustness (You et al., 2018; Noh et al., 2017; Bishop, 1995; Gulcehre et al., 2016), and in some cases can be linked back to Bayesian models of weights (Noh et al., 2017; Gal & Ghahramani, 2016) when the noise added at each activation can be considered as a result of different samples from the weight posterior.

Original: In this paper we propose a simple and effective stochastic neural network (SE-SNN) that models activation uncertainty through predicting a Gaussian mean and variance at each layer, which is then sampled during the forward pass.
Revised: In this paper we propose a simple and effective stochastic neural network (SE-SNN) that models activation uncertainty through predicting a Gaussian mean and variance at each layer, which is then sampled during the forward pass.

Original: This is similar to the strategy used to model activation distributions in VAE (Kingma & Welling, 2014) and VIB (Alemi et al., 2017).
Revised: This is similar to the strategy used to model activation distributions in VAE (Kingma & Welling, 2014) and VIB (Alemi et al., 2017).

Original: Differently, we then place a non-informative prior on activation distribution and derive an activation regularizer that directly encourages high activation variability via preferring high-entropy activations.
Revised: Differently, we then place a non-informative prior on activation distribution and derive an activation regularizer that directly encourages high activation variability via preferring high-entropy activations.

Original: In conjunction with a discriminative learning loss, this means that the network is optimized for activation patterns that have
Revised: In conjunction with a discriminative learning loss, this means that the network is optimized for activation patterns that have

Original: [Figure]
Revised: [Figure]

Original: #$%)
Revised: #$%)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (#$%) )
Revised: (#$%) )

Original: [Figure]
Revised: [Figure]

Original: Layer 𝑙 ℎ (#) 𝑓 , (ℎ (#) )
Revised: Layer 𝑙 ℎ (#) 𝑓 , (ℎ (#) )

Original: 𝑓 ( (ℎ (#) )
Revised: 𝑓 ( (ℎ (#) )

Original: 𝑓 , (ℎ (#$%) ) high uncertainty while simultaneously being predictive of the target variable.
Revised: 𝑓 , (ℎ (#$%) ) high uncertainty while simultaneously being predictive of the target variable.

Original: The interplay between these two objectives leads to several appealing capabilities in pruning, adversarial defense and learning with label noise.
Revised: The interplay between these two objectives leads to several appealing capabilities in pruning, adversarial defense and learning with label noise.

Original: Pruning :
Revised: Pruning :

Original: Optimizing for high per-activation variability/uncertainty and predictive accuracy simultaneously lead to the network packing more entropy into the least signiﬁcant neurons – so that the most crucial neurons are free to operate unperturbed.
Revised: Optimizing for high per-activation variability/uncertainty and predictive accuracy simultaneously lead to the network packing more entropy into the least signiﬁcant neurons – so that the most crucial neurons are free to operate unperturbed.

Original: This leads to a simple pruning criterion based on each neuron’s entropy value.
Revised: This leads to a simple pruning criterion based on each neuron’s entropy value.

Original: Adversarial defense : By optimizing for both peractivation uncertainty and the network’s predictive accuracy, a representation-level data augmentation policy is trained that perturbs the internal features during training for increased robustness (Alemi et al., 2017; You et al., 2018).
Revised: Adversarial defense : By optimizing for both peractivation uncertainty and the network’s predictive accuracy, a representation-level data augmentation policy is trained that perturbs the internal features during training for increased robustness (Alemi et al., 2017; You et al., 2018).

Original: Label noise : With SE-SNN, per-activation uncertainty can be easily aggregated to produce per-instance uncertainty.
Revised: Label noise : With SE-SNN, per-activation uncertainty can be easily aggregated to produce per-instance uncertainty.

Original: By optimizing for per-instance uncertainty and predictive accuracy, the network allocates the uncertainty to spread the representation and prediction of the hard-to-classify instances so as to downweight their inﬂuence on parameter learning.
Revised: By optimizing for per-instance uncertainty and predictive accuracy, the network allocates the uncertainty to spread the representation and prediction of the hard-to-classify instances so as to downweight their inﬂuence on parameter learning.

Original: The result is a model robust to label noise as well as outlying training samples.
Revised: The result is a model robust to label noise as well as outlying training samples.

Original: To summarize, our contributions are: (1) A new simple yet effective stochastic neural network formulation.
Revised: To summarize, our contributions are: (1) A new simple yet effective stochastic neural network formulation.

Original: (2) We show that our SE-SNN has connections to VIB (Alemi et al., 2017), Dropout (Srivastava et al., 2014) and non-informative activation priors while being simpler to implement and faster to train, as well as impactful on a variety of practical problems.
Revised: (2) We show that our SE-SNN has connections to VIB (Alemi et al., 2017), Dropout (Srivastava et al., 2014) and non-informative activation priors while being simpler to implement and faster to train, as well as impactful on a variety of practical problems.

Original: (3) Comprehensive evaluations show excellent performance on pruning-based model compression, adversarial defense, and label noise robust learning.
Revised: (3) Comprehensive evaluations show excellent performance on pruning-based model compression, adversarial defense, and label noise robust learning.

Original:  M ETHODOLOGY Stochastic Layers
Revised:  M ETHODOLOGY Stochastic Layers

Original: We consider a neural network discriminatively trained for a predictive task such as object recognition.
Revised: We consider a neural network discriminatively trained for a predictive task such as object recognition.

Original: Instead computing ﬁxed point estimates of feature vectors, we propose to use stochastic neurons.
Revised: Instead computing ﬁxed point estimates of feature vectors, we propose to use stochastic neurons.

Original: More speciﬁcally, for an input h , a layer will output a series of univariate distributions.
Revised: More speciﬁcally, for an input h , a layer will output a series of univariate distributions.

Original: By sampling from those distributions independently, we get a random output z .
Revised: By sampling from those distributions independently, we get a random output z .

Original: Finally we apply the non-linear activation function ψ ( · ) to z and get the input for the next layer.
Revised: Finally we apply the non-linear activation function ψ ( · ) to z and get the input for the next layer.

Original: In this study, we choose to use Gaussian distribution with parameterized mean and variance, which has been popularized by VAE (Kingma & Welling, 2014) and VIB (Alemi et al., 2017) due to the ease of reparameterization.
Revised: In this study, we choose to use Gaussian distribution with parameterized mean and variance, which has been popularized by VAE (Kingma & Welling, 2014) and VIB (Alemi et al., 2017) due to the ease of reparameterization.

Original: Formally, for the l -th layer, this forward-pass process can be written as (omitting neuron index for notation simplicity),
Revised: Formally, for the l -th layer, this forward-pass process can be written as (omitting neuron index for notation simplicity),

Original: [Equation]
Revised: [Equation]

Original: This process is illustrated in Figure 1, where each standard deviation predictor f σ comes with a softplus activation f ( x ) =
Revised: This process is illustrated in Figure 1, where each standard deviation predictor f σ comes with a softplus activation f ( x ) =

Original: log(1 + exp( x ))
Revised: log(1 + exp( x ))

Original: to ensure non-negativity.
Revised: to ensure non-negativity.

Original: Supervised Learning Loss
Revised: Supervised Learning Loss

Original: We can choose to replace some or all intermediate layers of a vanilla neural network with such stochastic layers.
Revised: We can choose to replace some or all intermediate layers of a vanilla neural network with such stochastic layers.

Original: For the ﬁnal layer (i.e., the classiﬁer) we opt for a standard linear layer, and the classiﬁcation loss is the same as that of a vanilla neural network, e.g., cross-entropy.
Revised: For the ﬁnal layer (i.e., the classiﬁer) we opt for a standard linear layer, and the classiﬁcation loss is the same as that of a vanilla neural network, e.g., cross-entropy.

Original: Since a Gaussian distribution is fully reparameterizable, the network can be trained end-to-end as long as the sampling process z ∼ N ( µ, σ ) is realized by z = µ + (cid:15) · σ where (cid:15) ∼ N (0 , 1) .
Revised: Since a Gaussian distribution is fully reparameterizable, the network can be trained end-to-end as long as the sampling process z ∼ N ( µ, σ ) is realized by z = µ + (cid:15) · σ where (cid:15) ∼ N (0 , 1) .

Original: Max-entropy Regularization
Revised: Max-entropy Regularization

Original: We place a non-informative prior on the produced Gaussian (denoted as N ( µ 1 , σ 1 ) ).
Revised: We place a non-informative prior on the produced Gaussian (denoted as N ( µ 1 , σ 1 ) ).

Original: The non-informative prior is a Gaussian with arbitrary mean ( µ 1 ) and inﬁnite variance ( σ 1 ).
Revised: The non-informative prior is a Gaussian with arbitrary mean ( µ 1 ) and inﬁnite variance ( σ 21 ).

Original: This reﬂects the prior that none of neurons is meaningful for predictive purposes.
Revised: This reﬂects the prior that none of neurons is meaningful for predictive purposes.

Original: The non-informative prior leads to a regularization term that minimizes the KL divergence of the produced Gaussian ( N ( µ 2 , σ 2 ) ) and the inﬁnite-variance Gaussian
Revised: The non-informative prior leads to a regularization term that minimizes the KL divergence of the produced Gaussian ( N ( µ 2 , σ 2 ) ) and the inﬁnite-variance Gaussian

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 2 suggests that we simply need to maximize the predicted standard deviation, or equivalently the entropy of the predicted Gaussian.
Revised: 2 suggests that we simply need to maximize the predicted standard deviation, or equivalently the entropy of the predicted Gaussian.

Original: Thus we call it a max-entropy regularizer Ω .
Revised: Thus we call it a max-entropy regularizer Ω .

Original: It can be easily used in any existing neural network architecture:
Revised: It can be easily used in any existing neural network architecture:

Original: [Equation]
Revised: [Equation]

Original: where σ ( h | θ ) denotes the predicted standard deviation of hidden unit h given the neuron uncertainty prediction parameter θ .
Revised: where σ ( h | θ ) denotes the predicted standard deviation of hidden unit h given the neuron uncertainty prediction parameter θ .

Original: For numerical safety, we introduce a margin b in the loss,
Revised: For numerical safety, we introduce a margin b in the loss,

Original: [Equation]
Revised: [Equation]

Original: This means that the regularization does not punish the model as long as the entropy is larger than a threshold b .
Revised: This means that the regularization does not punish the model as long as the entropy is larger than a threshold b .

Original: Note that, this loss design is not absolutely necessary as the increment for σ shrinks (since the gradient is − 1 σ ) during training, and thus never reaches inﬁnity with a ﬁnite number of updates.
Revised: Note that, this loss design is not absolutely necessary as the increment for σ shrinks (since the gradient is − 1 σ ) during training, and thus never reaches inﬁnity with a ﬁnite number of updates.

Original: However, one can think of it as an early-stopping mechanism for this regularization term.
Revised: However, one can think of it as an early-stopping mechanism for this regularization term.

Original: So far we have introduced the stochasticity to the smallest unit of a network, i.e., a single neuron.
Revised: So far we have introduced the stochasticity to the smallest unit of a network, i.e., a single neuron.

Original: To compute the value of the regularizer over a mini-batch consisting of N training samples, we need to aggregate the entropy of multiple neurons and set the margin b on the aggregation.
Revised: To compute the value of the regularizer over a mini-batch consisting of N training samples, we need to aggregate the entropy of multiple neurons and set the margin b on the aggregation.

Original: How to aggregate exactly is task-dependent, and we next provide some suggestions for three tasks including network pruning, adversarial defense, and label noise defense.
Revised: How to aggregate exactly is task-dependent, and we next provide some suggestions for three tasks including network pruning, adversarial defense, and label noise defense.

Original: Pruning
Revised: Pruning

Original: For network pruning, we aggregate entropy over samples for each neuron, and then penalize if that neuron’s aggregated entropy is low.
Revised: For network pruning, we aggregate entropy over samples for each neuron, and then penalize if that neuron’s aggregated entropy is low.

Original: To this end, the regularizer is formulated as:
Revised: To this end, the regularizer is formulated as:

Original: [Equation]
Revised: [Equation]

Original: where i = 1 . .
Revised: where i = 1 . .

Original: N indexes the training samples, and j = 1 . . .
Revised: N indexes the training samples, and j = 1 . . .

Original: K neurons in a layer (i.e., feature channels).
Revised: K neurons in a layer (i.e., feature channels).

Original: This regularizer aims to make neurons very stochastic, to the point of compromising their reliability for computing a supervised learning task.
Revised: This regularizer aims to make neurons very stochastic, to the point of compromising their reliability for computing a supervised learning task.

Original: Thus only those neurons that are most useful for the task get their entropy lowered and thus pay the regularization cost.
Revised: Thus only those neurons that are most useful for the task get their entropy lowered and thus pay the regularization cost.

Original: Less useful neurons get their entropy maximized, allowing them to be detected and pruned after training.
Revised: Less useful neurons get their entropy maximized, allowing them to be detected and pruned after training.

Original: Label Noise
Revised: Label Noise

Original: Different from pruning, we aim to identify uncertain samples.
Revised: Different from pruning, we aim to identify uncertain samples.

Original: Therefore, we aggregate entropy over neurons for each sample, and prefer high sample-wise entropy, leading to
Revised: Therefore, we aggregate entropy over neurons for each sample, and prefer high sample-wise entropy, leading to

Original: [Equation]
Revised: [Equation]

Original: With this regularizer, inlier samples pay the entropy cost in order to produce a clean feature for classiﬁcation to satisfy the supervised learning loss.
Revised: With this regularizer, inlier samples pay the entropy cost in order to produce a clean feature for classiﬁcation to satisfy the supervised learning loss.

Original: Outlier samples, caused by either label noise or being out-of-distribution, are anyway hard to classify; the regularizer naturally inﬂates their entropy since doing so does not impact the supervised learning loss.
Revised: Outlier samples, caused by either label noise or being out-of-distribution, are anyway hard to classify; the regularizer naturally inﬂates their entropy since doing so does not impact the supervised learning loss.

Original: This high-variance representation in turn reduces their (negative) impact on network ﬁtting.
Revised: This high-variance representation in turn reduces their (negative) impact on network ﬁtting.

Original: Adversarial Defense
Revised: Adversarial Defense

Original: To defend against adversarial samples, we aim to inﬂate entropy over both the neuron- and sample-axes to produce a highly stochastic model:
Revised: To defend against adversarial samples, we aim to inﬂate entropy over both the neuron- and sample-axes to produce a highly stochastic model:

Original: [Equation]
Revised: [Equation]

Original: This looks similar to VIB’s adversarial defense (Alemi et al., 2017), but with the vital difference that our entropy regularizer is not restricted to the N (0 , 1) prior used in VIB.
Revised: This looks similar to VIB’s adversarial defense (Alemi et al., 2017), but with the vital difference that our entropy regularizer is not restricted to the N (0 , 1) prior used in VIB.

Original: It can be seen as learning a layer-wise data augmentation policy, which turns out to be very useful for adversarial defense in practice (see Sec. 4.2).
Revised: It can be seen as learning a layer-wise data augmentation policy, which turns out to be very useful for adversarial defense in practice (see Sec. 4.2).

Original:  R ELATED W ORK Connection to VIB and Sparse VD
Revised:  R ELATED W ORK Connection to VIB and Sparse VD

Original: Though we derive the max-entropy regularizer from the perspective of a non-informative activation prior, our work is closely related to VIB (Alemi et al., 2017) and sparse variational dropout (VD) (Molchanov et al., 2017a), despite their different perspectives.
Revised: Though we derive the max-entropy regularizer from the perspective of a non-informative activation prior, our work is closely related to VIB (Alemi et al., 2017) and sparse variational dropout (VD) (Molchanov et al., 2017a), despite their different perspectives.

Original: Speciﬁcally, if we replace our inﬁnite-variance Gaussian with a standard Gaussian, it becomes VIB (see Eq.
Revised: Speciﬁcally, if we replace our inﬁnite-variance Gaussian with a standard Gaussian, it becomes VIB (see Eq.

Original: 17 in (Alemi et al., 2017)).
Revised: 17 in (Alemi et al., 2017)).

Original: The max-entropy regularizer is also linked to Eq.
Revised: The max-entropy regularizer is also linked to Eq.

Original: 14 in Sparse
Revised: 14 in Sparse

Original: VD (Molchanov et al., 2017a), which also encourages large variance/entropy (at a different rate).
Revised: VD (Molchanov et al., 2017a), which also encourages large variance/entropy (at a different rate).

Original: But again, Sparse VD (Molchanov et al., 2017a) is derived with a completely different motivation: It has an intuitive explanation that the regularizer corresponds to a sparsity prior on the weights.
Revised: But again, Sparse VD (Molchanov et al., 2017a) is derived with a completely different motivation: It has an intuitive explanation that the regularizer corresponds to a sparsity prior on the weights.

Original: We note that enforcing uncertainty on activations rather than weights has a number of advantages: (i)
Revised: We note that enforcing uncertainty on activations rather than weights has a number of advantages: (i)

Original: The weight prior is intractable analytically, which leads to the fact that Sparse VD regularization is itself an approximation.
Revised: The weight prior is intractable analytically, which leads to the fact that Sparse VD regularization is itself an approximation.

Original: (ii) Deriving the regularizer from a weight prior is unnecessarily complicated for the purpose of sparsifying the model compared to ours.
Revised: (ii) Deriving the regularizer from a weight prior is unnecessarily complicated for the purpose of sparsifying the model compared to ours.

Original: In contrast, our approach sidesteps the need to sample weights and avoids keeping multiple copies of the network, which can potentially improve efﬁciency (e.g., in memory usage).
Revised: In contrast, our approach sidesteps the need to sample weights and avoids keeping multiple copies of the network, which can potentially improve efﬁciency (e.g., in memory usage).

Original: Network Compression
Revised: Network Compression

Original: Network compression based on pruning typically uses heuristics based on pruning low-importance weights or low-activation neurons (Molchanov et al., 2017b; Wen et al., 2016), often assisted by sparsity-enhancing priors such as lasso (Wen et al., 2016).
Revised: Network compression based on pruning typically uses heuristics based on pruning low-importance weights or low-activation neurons (Molchanov et al., 2017b; Wen et al., 2016), often assisted by sparsity-enhancing priors such as lasso (Wen et al., 2016).

Original: We avoid the complication of Bayesian learning of weights by proposing a simpler and direct activation prior that predisposes neurons towards deactivation unless necessary to solve the supervised task.
Revised: We avoid the complication of Bayesian learning of weights by proposing a simpler and direct activation prior that predisposes neurons towards deactivation unless necessary to solve the supervised task.

Original: Adversarial Defense
Revised: Adversarial Defense

Original: Our method is related to existing randomization-based methods for adversarial defense (Liu et al., 2018; Xie et al., 2018;
Revised: Our method is related to existing randomization-based methods for adversarial defense (Liu et al., 2018; Xie et al., 2018;

Original: Alemi et al., 2017; Ye & Zhu, 2018;
Revised: Alemi et al., 2017; Ye & Zhu, 2018;

Original: Liu et al., 2019).
Revised: Liu et al., 2019).

Original: However unlike these studies which use a ﬁxed distribution for noise (Liu et al., 2018; Alemi et al., 2017), a learned model distribution for effective randomization (Liu et al., 2019), image perturbations (Xie et al., 2018) or a learned adversarial data-generating distribution (Ye & Zhu, 2018), our randomization-based defense is both learned , and data-dependent since the variance at each layer is generated based on the output of the previous layer.
Revised: However unlike these studies which use a ﬁxed distribution for noise (Liu et al., 2018; Alemi et al., 2017), a learned model distribution for effective randomization (Liu et al., 2019), image perturbations (Xie et al., 2018) or a learned adversarial data-generating distribution (Ye & Zhu, 2018), our randomization-based defense is both learned , and data-dependent since the variance at each layer is generated based on the output of the previous layer.

Original: Label Noise Robustness
Revised: Label Noise Robustness

Original: A number of existing label noise-robust deep learning approaches require a subset of noisy data to be reliably re-annotated (cleaned) to verify which samples contain noise (Lee et al., 2017; Jiang et al., 2018).
Revised: A number of existing label noise-robust deep learning approaches require a subset of noisy data to be reliably re-annotated (cleaned) to verify which samples contain noise (Lee et al., 2017; Jiang et al., 2018).

Original: In contrast, some others including SE-SNN do not rely on additional human noise annotation.
Revised: In contrast, some others including SE-SNN do not rely on additional human noise annotation.

Original: These methods address label noise by either iterative label correction via bootstrapping (Reed et al., 2015), adding additional layers on top of a softmax classiﬁcation layer to estimate the noise pattern (Sukhbaatar et al., 2015; Goldberger & Ben-Reuven, 2017), or loss correlation (Patrini et al., 2017).
Revised: These methods address label noise by either iterative label correction via bootstrapping (Reed et al., 2015), adding additional layers on top of a softmax classiﬁcation layer to estimate the noise pattern (Sukhbaatar et al., 2015; Goldberger & Ben-Reuven, 2017), or loss correlation (Patrini et al., 2017).

Original: By allocating large uncertainty to outlying samples, SE-SNN can handle both label noise and out-of-distribution samples with correct labels.
Revised: By allocating large uncertainty to outlying samples, SE-SNN can handle both label noise and out-of-distribution samples with correct labels.

Original: This approach to label noise robustness is appealingly simple in that it requires neither explicit detection of noisy samples, nor additional annotation.
Revised: This approach to label noise robustness is appealingly simple in that it requires neither explicit detection of noisy samples, nor additional annotation.

Original: However label noise robustness has been largely ignored by existing SNNs.
Revised: However label noise robustness has been largely ignored by existing SNNs.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Experiments are carried out to evaluate the efﬁcacy of the proposed framework in three applications: neural network pruning, adversarial attack defense and learning with label noise.
Revised: Experiments are carried out to evaluate the efﬁcacy of the proposed framework in three applications: neural network pruning, adversarial attack defense and learning with label noise.

Original: 4. N EURAL N ETWORK P RUNING Competitors
Revised: 4. N EURAL N ETWORK P RUNING Competitors

Original: We follow the architecture/dataset combinations used in most recent neural network pruning studies, including LeNet-5-Caffe 1 network on MNIST (LeCun, 1998), VGG-16 (Simonyan
Revised: We follow the architecture/dataset combinations used in most recent neural network pruning studies, including LeNet-5-Caffe 1 network on MNIST (LeCun, 1998), VGG-16 (Simonyan

Original: [Table]
Revised: [Table]

Original: Zisserman, 2015)
Revised: Zisserman, 2015)

Original: on CIFAR10 (Krizhevsky & Hinton, 2009) and a variant of VGG–16 on
Revised: on CIFAR10 (Krizhevsky & Hinton, 2009) and a variant of VGG–16 on

Original: CIFAR100.
Revised: CIFAR100.

Original: Under these settings, the proposed method is compared with the following contemporary state-of-the-art methods including Generalized Dropout (GD) (Srinivas & Babu, 2016), Group
Revised: Under these settings, the proposed method is compared with the following contemporary state-of-the-art methods including Generalized Dropout (GD) (Srinivas & Babu, 2016), Group

Original: Lasso (GL) (Wen et al., 2016), Sparse Variational Dropout (VD) (Molchanov et al., 2017a), Structured
Revised: Lasso (GL) (Wen et al., 2016), Sparse Variational Dropout (VD) (Molchanov et al., 2017a), Structured

Original: Bayesian Pruning (SBP) (Neklyudov et al., 2017), Bayesian Compression with Group Normal
Revised: Bayesian Pruning (SBP) (Neklyudov et al., 2017), Bayesian Compression with Group Normal

Original: Jeffreys Prior (BC-GNJ) and Group Horseshoe Prior (BC-GHS)
Revised: Jeffreys Prior (BC-GNJ) and Group Horseshoe Prior (BC-GHS)

Original: (Louizos et al., 2017), Sparse l
Revised: (Louizos et al., 2017), Sparse l

Original: Regularization (L0) and L0 with separate λ for each layer (L0-sep)
Revised: Regularization (L0) and L0 with separate λ for each layer (L0-sep)

Original: (Louizos et al., 2018), Variational
Revised: (Louizos et al., 2018), Variational

Original: Information Bottleneck (VIBNet) (Dai et al., 2018), and Network Slimming (NS) (Liu et al., 2017).
Revised: Information Bottleneck (VIBNet) (Dai et al., 2018), and Network Slimming (NS) (Liu et al., 2017).

Original: Evaluation Metrics
Revised: Evaluation Metrics

Original: Following the majority of the existing evaluations, we monitor the test error while focusing on the following three compression/complexity metrics: (a) Model size: The ratio of nonzero weights in the compressed network versus the original model.
Revised: Following the majority of the existing evaluations, we monitor the test error while focusing on the following three compression/complexity metrics: (a) Model size: The ratio of nonzero weights in the compressed network versus the original model.

Original: (b) FLOPs:
Revised: (b) FLOPs:

Original: The number of ﬂoating point operations required to predict a label from an input image during test 2 .
Revised: The number of ﬂoating point operations required to predict a label from an input image during test 2 .

Original: (c) Run-time memory footprint:
Revised: (c) Run-time memory footprint:

Original: The ratio of the space for storing hidden feature maps during run-time in the pruned network versus the original network.
Revised: The ratio of the space for storing hidden feature maps during run-time in the pruned network versus the original network.

Original: Training
Revised: Training

Original: While many existing studies remove redundant weights , we remove redundant neurons during compression.
Revised: While many existing studies remove redundant weights , we remove redundant neurons during compression.

Original: Therefore, we can use the pipeline proposed in Molchanov et al.
Revised: Therefore, we can use the pipeline proposed in Molchanov et al.

Original: (2017b).
Revised: (2017b).

Original: Speciﬁcally, after training the neural network, an initial batch pruning stage is followed by a loop of removing the least important neuron and ﬁne-tuning.
Revised: Speciﬁcally, after training the neural network, an initial batch pruning stage is followed by a loop of removing the least important neuron and ﬁne-tuning.

Original: Since SE-SNN is designed to discount unimportant neurons through inﬂating their pre-activation variance, we ﬁnd that the network achieves this by simultaneously assigning high-variance and negative mean.
Revised: Since SE-SNN is designed to discount unimportant neurons through inﬂating their pre-activation variance, we ﬁnd that the network achieves this by simultaneously assigning high-variance and negative mean.

Original: As a result, a large portion of redundant neurons never activate their RELU non-linearity.
Revised: As a result, a large portion of redundant neurons never activate their RELU non-linearity.

Original: In the initial batch pruning stage, neuron inactivity thus provides a single-step pruning criterion before the iterative pruning begins (and one that is guaranteed not to affect the test accuracy since these neurons propagate no information).
Revised: In the initial batch pruning stage, neuron inactivity thus provides a single-step pruning criterion before the iterative pruning begins (and one that is guaranteed not to affect the test accuracy since these neurons propagate no information).

Original: The pruning then enters the second stage where the least important neuron removal + ﬁne-tuning loop continues until reaching the target trade-off between accuracy and model compression objectives.
Revised: The pruning then enters the second stage where the least important neuron removal + ﬁne-tuning loop continues until reaching the target trade-off between accuracy and model compression objectives.

Original: Since one neuron/channel is removed at each iteration, the accuracy never drops sharply.
Revised: Since one neuron/channel is removed at each iteration, the accuracy never drops sharply.

Original: We set the uncertainty loss/regularizer weighting factor and margin b (Eq. 5) as 0.0001 and 4, respectively.
Revised: We set the uncertainty loss/regularizer weighting factor and margin b (Eq. 5) as 0.0001 and 4, respectively.

Original: Testing
Revised: Testing

Original: During testing, only the mean of the learned distribution is passed between layers.
Revised: During testing, only the mean of the learned distribution is passed between layers.

Original: So there are no additional parameters and inference cost compared to the network’s deterministic counterpart.
Revised: So there are no additional parameters and inference cost compared to the network’s deterministic counterpart.

Original: The distribution generation branches are only used during training to identify redundant neurons
Revised: The distribution generation branches are only used during training to identify redundant neurons

Original: . Results on MNIST
Revised: . Results on MNIST

Original: The most commonly used benchmark and architecture is MNIST with LeNet5-Caffe.
Revised: The most commonly used benchmark and architecture is MNIST with LeNet5-Caffe.

Original: We follow the standard training and testing protocols.
Revised: We follow the standard training and testing protocols.

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: It is clear that SE-SNN achieves the best performance on FLOPs, run-time memory footprint, and test error.
Revised: It is clear that SE-SNN achieves the best performance on FLOPs, run-time memory footprint, and test error.

Original: In terms of model size, our model is only comparable with the state of art.
Revised: In terms of model size, our model is only comparable with the state of art.

Original: This is because it does not prune the linear layer as much as other methods.
Revised: This is because it does not prune the linear layer as much as other methods.

Original: Instead, it focuses on pruning the convolutional ﬁlters, hence the excellent performance on FLOPS and memory footprint.
Revised: Instead, it focuses on pruning the convolutional ﬁlters, hence the excellent performance on FLOPS and memory footprint.

Original: Results on CIFAR 
Revised: Results on CIFAR

Original: For CIFAR10, several VGG16 variants and training protocols have been proposed in different works.
Revised: For CIFAR10, several VGG16 variants and training protocols have been proposed in different works.

Original: We use the standard VGG16 architecture but change the dimension of
Revised: We use the standard VGG16 architecture but change the dimension of

Original: [Table] [Table]
Revised: [Table] [Table]

Original: linear layers from 4096 to 512, as in (Louizos et al., 2017; Dai et al., 2018).
Revised: linear layers from 4096 to 512, as in (Louizos et al., 2017; Dai et al., 2018).

Original: The results in Tablecompare our method with (Louizos et al., 2017; Dai et al., 2018).
Revised: The results in Tablecompare our method with (Louizos et al., 2017; Dai et al., 2018).

Original: The error rate for VIBNet in parentheses was obtained by further ﬁne-tuning the pruned architecture.
Revised: The error rate for VIBNet in parentheses was obtained by further ﬁne-tuning the pruned architecture.

Original: Our model achieves the best performance across all the evaluation metrics and error rates.
Revised: Our model achieves the best performance across all the evaluation metrics and error rates.

Original: Results on CIFAR 
Revised: Results on CIFAR

Original: We compare with the study in (Liu et al., 2017), which uses the same
Revised: We compare with the study in (Liu et al., 2017), which uses the same

Original: VGG16 variant that replaces two fully connected layers with three convolutional layers.
Revised: VGG16 variant that replaces two fully connected layers with three convolutional layers.

Original: This architecture improves accuracy at the expense of FLOPs and memory.
Revised: This architecture improves accuracy at the expense of FLOPs and memory.

Original: The results in Table 3 show that our model produces the best compression result while maintaining comparable accuracy.
Revised: The results in Table 3 show that our model produces the best compression result while maintaining comparable accuracy.

Original: Note that the 26 . 2% error rate achieved by our model is identical to that of the original network.
Revised: Note that the 26 . 2% error rate achieved by our model is identical to that of the original network.

Original: So if the accuracy drop is used as a compression metric, our model is as good as any competitor.
Revised: So if the accuracy drop is used as a compression metric, our model is as good as any competitor.

Original: 4. A DVERSARIAL D EFENSE Experimental Setting
Revised: 4. A DVERSARIAL D EFENSE Experimental Setting

Original: Following Alemi et al.
Revised: Following Alemi et al.

Original: (2017), we focus on two types of adversarial attacks: Fast Gradient Sign (FGS) (Goodfellow et al., 2015) and an optimization-based attack –
Revised: (2017), we focus on two types of adversarial attacks: Fast Gradient Sign (FGS) (Goodfellow et al., 2015) and an optimization-based attack –

Original: CW-L2 (Carlini & Wagner, 2017).
Revised: CW-L2 (Carlini & Wagner, 2017).

Original: We evaluate untargeted FGS attacks with attack magnitude (cid:15) ranging from 0.0 to 0.5 and untargeted CW-L2 attack on models trained on MNIST.
Revised: We evaluate untargeted FGS attacks with attack magnitude (cid:15) ranging from 0.0 to 0.5 and untargeted CW-L2 attack on models trained on MNIST.

Original: We use a popular architecture including three FC layers with 1024, 1024 and 256 output neurons respectively.
Revised: We use a popular architecture including three FC layers with 1024, 1024 and 256 output neurons respectively.

Original: The third
Revised: The third

Original: FC layer is implemented as a stochastic layer.
Revised: FC layer is implemented as a stochastic layer.

Original: We set the uncertainty loss/regularizer weighting factor and margin as 0.1 and 16, respectively.
Revised: We set the uncertainty loss/regularizer weighting factor and margin as 0.1 and 16, respectively.

Original: Results averaged over 20 runs are reported.
Revised: Results averaged over 20 runs are reported.

Original: We compare against the original (undefended) network, termed as ‘Baseline’, Deep VIB (Alemi et al., 2017) using the variational information bottleneck, Bayesian Adversarial Learning (BAL) (Ye & Zhu, 2018) putting a distribution on the adversarial data-generating process and Adv-BNN (Liu et al., 2019) learning a BNN to incorporate the effective randomness and using adversarial training to seek the best model distribution.
Revised: We compare against the original (undefended) network, termed as ‘Baseline’, Deep VIB (Alemi et al., 2017) using the variational information bottleneck, Bayesian Adversarial Learning (BAL) (Ye & Zhu, 2018) putting a distribution on the adversarial data-generating process and Adv-BNN (Liu et al., 2019) learning a BNN to incorporate the effective randomness and using adversarial training to seek the best model distribution.

Original: Since our model is stochastic, we follow the best practice recommended in (Athalye et al., 2018) for attacking stochastic models: We compute the expected gradient over multiple stochastic samples for each input when constructing attacks.
Revised: Since our model is stochastic, we follow the best practice recommended in (Athalye et al., 2018) for attacking stochastic models: We compute the expected gradient over multiple stochastic samples for each input when constructing attacks.

Original: This is because using the expected gradient over multiple posterior samples produces a better gradient estimator for the attacker, allowing it to generate samples that are much harder to defend against (Athalye et al., 2018).
Revised: This is because using the expected gradient over multiple posterior samples produces a better gradient estimator for the attacker, allowing it to generate samples that are much harder to defend against (Athalye et al., 2018).

Original: For the
Revised: For the

Original: FGS attack, we evaluate two settings, namely normal training and adversarial training, the latter of which generates and uses FGS attack samples during training to increase adversarial robustness.
Revised: FGS attack, we evaluate two settings, namely normal training and adversarial training, the latter of which generates and uses FGS attack samples during training to increase adversarial robustness.

Original: Results
Revised: Results

Original: From the comparative results shown in Figure 2a, we can see that our SE-SNN outperforms the competing defense methods over a range of FGS attack strengths when trained with adversarial attack samples (Figure 2a(R)) or with only normal samples (Figure 2a(L)).
Revised: From the comparative results shown in Figure 2a, we can see that our SE-SNN outperforms the competing defense methods over a range of FGS attack strengths when trained with adversarial attack samples (Figure 2a(R)) or with only normal samples (Figure 2a(L)).

Original: The advantage of SE-SNN is particularly pronounced when the attack magnitude is large.
Revised: The advantage of SE-SNN is particularly pronounced when the attack magnitude is large.

Original: We can also see from Figure 2b that under the stronger CW attacks the Baseline now fails completely.
Revised: We can also see from Figure 2b that under the stronger CW attacks the Baseline now fails completely.

Original: In this case SE-SNN provides the most effective defense.
Revised: In this case SE-SNN provides the most effective defense.

Original: It is worth pointing out that, unlike BAL (Ye & Zhu, 2018), which learns their models with explicit adversarial sampling, our SE-SNN can also work with training with only normal samples - Figure 2a suggests that our SE-SNN beats BAL even without being trained with adversarial attack samples (comparing SE-SNN in Figure 2a(L) to BAL in Figure 2a(R)).
Revised: It is worth pointing out that, unlike BAL (Ye & Zhu, 2018), which learns their models with explicit adversarial sampling, our SE-SNN can also work with training with only normal samples - Figure 2a suggests that our SE-SNN beats BAL even without being trained with adversarial attack samples (comparing SE-SNN in Figure 2a(L) to BAL in Figure 2a(R)).

Original: [Figure]
Revised: [Figure]

Original: 4. R OBUSTNESS AGAINST L ABEL N OISE
Revised: 4. R OBUSTNESS AGAINST L ABEL N OISE

Original: In this section, tasks including digit recognition and person re-identiﬁcation (ReID) are considered.
Revised: In this section, tasks including digit recognition and person re-identiﬁcation (ReID) are considered.

Original: Compared to Digit recognition, ReID is much more challenging.
Revised: Compared to Digit recognition, ReID is much more challenging.

Original: Particularly, ReID is an instance recognition task, which aims to match people under disjoint camera views.
Revised: Particularly, ReID is an instance recognition task, which aims to match people under disjoint camera views.

Original: Although the performance of state-of-the-art ReID models on public benchmarks approaches saturation, ReID with label noise remains an unsolved and under-studied problem.
Revised: Although the performance of state-of-the-art ReID models on public benchmarks approaches saturation, ReID with label noise remains an unsolved and under-studied problem.

Original: Datasets
Revised: Datasets

Original: The label noise robustness of our SE-SNN is evaluated for digit recognition using MNIST and ReID using Market-1501 (Zheng et al., 2015) benchmark.
Revised: The label noise robustness of our SE-SNN is evaluated for digit recognition using MNIST and ReID using Market-1501 (Zheng et al., 2015) benchmark.

Original: Market1501 is collected by 6 cameras and contains 751 training identities (12,936 images) and 750 test identities (19,281 images).
Revised: Market1501 is collected by 6 cameras and contains 751 training identities (12,936 images) and 750 test identities (19,281 images).

Original: The test set is organized into a query set and a gallery set.
Revised: The test set is organized into a query set and a gallery set.

Original: Cumulative Matching Characteristics (CMC) ranks and mean Average Precision are used as the performance measure.
Revised: Cumulative Matching Characteristics (CMC) ranks and mean Average Precision are used as the performance measure.

Original: Noise Generation
Revised: Noise Generation

Original: We consider patterned label noise which is more common in practice.
Revised: We consider patterned label noise which is more common in practice.

Original: Specifically, for MNIST, one class label will be ﬂipped to a different one at a strength p .
Revised: Specifically, for MNIST, one class label will be ﬂipped to a different one at a strength p .

Original: The ﬂipping pattern is ﬁxed for each run but varies across different runs following (Hendrycks et al., 2018).
Revised: The ﬂipping pattern is ﬁxed for each run but varies across different runs following (Hendrycks et al., 2018).

Original: For
Revised: For

Original: Market-1501, we use the commonly used ResNet-50 (He et al., 2016) trained on the clean data to obtain the feature of each training sample and ﬁnd the most visually similar samples using feature
Revised: Market-1501, we use the commonly used ResNet-50 (He et al., 2016) trained on the clean data to obtain the feature of each training sample and ﬁnd the most visually similar samples using feature

Original: Euclidean distance.
Revised: Euclidean distance.

Original: Then for randomly selected training samples, their identity labels are assigned to that of the most similar sample that has a different identity, imitating human annotation errors.
Revised: Then for randomly selected training samples, their identity labels are assigned to that of the most similar sample that has a different identity, imitating human annotation errors.

Original: Competitors
Revised: Competitors

Original: Baseline: The main baseline is the original network without adding our stochastic layers.
Revised: Baseline: The main baseline is the original network without adding our stochastic layers.

Original: For a fair comparison, we add a parallel layer to the penultimate (feature) layer in the baseline to guarantee the two networks have the same number of parameters.
Revised: For a fair comparison, we add a parallel layer to the penultimate (feature) layer in the baseline to guarantee the two networks have the same number of parameters.

Original: The input of the added layer is the same as the feature layer, and its output is element-wise added along with the output of the feature layer to get the ﬁnal feature, which is then fed to a fully connected layer for computing the classiﬁcation loss. Bootstrap_hard
Revised: The input of the added layer is the same as the feature layer, and its output is element-wise added along with the output of the feature layer to get the ﬁnal feature, which is then fed to a fully connected layer for computing the classiﬁcation loss. Bootstrap_hard

Original: and
Revised: and

Original: Bootstrap_soft (Reed et al., 2015): Both iteratively use the model-predicted labels to reﬁne the original labels that are potentially corrupted by noise.
Revised: Bootstrap_soft (Reed et al., 2015): Both iteratively use the model-predicted labels to reﬁne the original labels that are potentially corrupted by noise.

Original: They differ in whether the updated label is binary or continuous.
Revised: They differ in whether the updated label is binary or continuous.

Original: Forward Correction (Patrini et al., 2017):
Revised: Forward Correction (Patrini et al., 2017):

Original: This model predicts the label corruption matrix (capturing the label ﬂipping pattern) by ﬁrst training a classiﬁer on the noisy labels followed by corruption matrix estimation using the resulting softmax probabilities.
Revised: This model predicts the label corruption matrix (capturing the label ﬂipping pattern) by ﬁrst training a classiﬁer on the noisy labels followed by corruption matrix estimation using the resulting softmax probabilities.

Original: The estimated matrix is then employed to regularize the retraining of the model.
Revised: The estimated matrix is then employed to regularize the retraining of the model.

Original: For MNIST, we stick to the original setting, which uses the argmax at the 97th percentile of softmax probabilities for label noise detection.
Revised: For MNIST, we stick to the original setting, which uses the argmax at the 97th percentile of softmax probabilities for label noise detection.

Original: For the ReID experiment, we replace this with the argmax over all softmax probabilities for a given class following what Patrini et al.
Revised: For the ReID experiment, we replace this with the argmax over all softmax probabilities for a given class following what Patrini et al.

Original: (2017) did on datasets of more classes such as CIFAR100.
Revised: (2017) did on datasets of more classes such as CIFAR100.

Original: CleanNet (Lee et al., 2017): different from other models, this one requires a subset of noisy training samples to be re-annotated by a more reliable source (i.e., cleaned).
Revised: CleanNet (Lee et al., 2017): different from other models, this one requires a subset of noisy training samples to be re-annotated by a more reliable source (i.e., cleaned).

Original: It then learns the similarity between class and query-embedding vectors, which is further used to detect noisy samples for sample pruning.
Revised: It then learns the similarity between class and query-embedding vectors, which is further used to detect noisy samples for sample pruning.

Original: Having a cleaned subset gives CleanNet an unfair advantage over other compared models.
Revised: Having a cleaned subset gives CleanNet an unfair advantage over other compared models.

Original: The model requires at least 5 FC layers so cannot be implemented for the MNIST backbone.
Revised: The model requires at least 5 FC layers so cannot be implemented for the MNIST backbone.

Original: For person ReID, 10% of the training set without noise is used as a clean reference set to train CleanNet using the author-provided code.
Revised: For person ReID, 10% of the training set without noise is used as a clean reference set to train CleanNet using the author-provided code.

Original: After training, 20% of the whole training set deemed most likely to be noisy are removed before the ﬁnal ReID model is trained on the remainder.
Revised: After training, 20% of the whole training set deemed most likely to be noisy are removed before the ﬁnal ReID model is trained on the remainder.

Original: Note that unlike the four competitors, our SE-SNN does not have any additional steps to reﬁne the label or prune the training samples.
Revised: Note that unlike the four competitors, our SE-SNN does not have any additional steps to reﬁne the label or prune the training samples.

Original: As explained earlier, it works by discounting/neutralizing the negative inﬂuence of noisy samples (hence also works on out-of-distribution samples with correct labels).
Revised: As explained earlier, it works by discounting/neutralizing the negative inﬂuence of noisy samples (hence also works on out-of-distribution samples with correct labels).

Original: Having said that, it can be easily combined with a
Revised: Having said that, it can be easily combined with a

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: label reﬁnement procedure such as the corruption matrix based one in Forward Correction ( et
Revised: label reﬁnement procedure such as the corruption matrix based one in Forward Correction ( et

Original: Results
Revised: Results

Original: The results on MNIST are shown in Figure 3a.
Revised: The results on MNIST are shown in Figure 3a.

Original: We can see that even without any label correction/reﬁnement procedure, our SE-SNN is already very competitive – it is only beaten by
Revised: We can see that even without any label correction/reﬁnement procedure, our SE-SNN is already very competitive – it is only beaten by

Original: Forward Correction when the corruption strength (percentage of samples with label noise) is large (>0.2).
Revised: Forward Correction when the corruption strength (percentage of samples with label noise) is large (>0.2).

Original: Once our SE-SNN is combined with the same procedure adopted by Forward Correction (SE-SNN+Forward), we achieve the best result.
Revised: Once our SE-SNN is combined with the same procedure adopted by Forward Correction (SE-SNN+Forward), we achieve the best result.

Original: On the more challenging ReID task in Market-
Revised: On the more challenging ReID task in Market-

Original: (Table 4), our SE-SNN also achieves the best overall performance - even beating CleanNet which needs an additional cleaned training subset.
Revised: (Table 4), our SE-SNN also achieves the best overall performance - even beating CleanNet which needs an additional cleaned training subset.

Original: Note that the strongest competitor on MNIST, Forward
Revised: Note that the strongest competitor on MNIST, Forward

Original: Correction now is the weakest.
Revised: Correction now is the weakest.

Original: This is because, as admitted in (Patrini et al., 2017), it struggles with estimating an accurate correction matrix given a small number of training samples per class (e.g., 500 per class in CIFAR100).
Revised: This is because, as admitted in (Patrini et al., 2017), it struggles with estimating an accurate correction matrix given a small number of training samples per class (e.g., 500 per class in CIFAR100).

Original: The < 20 class size in Market-1501 means that the label correction procedure only has a detrimental effect.
Revised: The < 20 class size in Market-1501 means that the label correction procedure only has a detrimental effect.

Original: Figure 3b compares the average variance/uncertainty inferred for clean and noisy data in Market-1501.
Revised: Figure 3b compares the average variance/uncertainty inferred for clean and noisy data in Market-1501.

Original: We can see that, indeed the variance of noisy data is larger than that of clean data on average.
Revised: We can see that, indeed the variance of noisy data is larger than that of clean data on average.

Original: We also notice that even when there is no label noise, SE-SNN beats the baseline by a clear margin ( 70.20% mAP vs. 67.66%).
Revised: We also notice that even when there is no label noise, SE-SNN beats the baseline by a clear margin ( 70.20% mAP vs. 67.66%).

Original: This suggests that the mechanism of neutralizing outlying samples is still in play when the labels are clean.
Revised: This suggests that the mechanism of neutralizing outlying samples is still in play when the labels are clean.

Original: To validate this, we show some examples of outliers (those with the largest variance) in Figure 3c when there is no label noise.
Revised: To validate this, we show some examples of outliers (those with the largest variance) in Figure 3c when there is no label noise.

Original: It is clear that these are mostly caused by either poor person detection or occlusion (i.e., outliers).
Revised: It is clear that these are mostly caused by either poor person detection or occlusion (i.e., outliers).

Original: In contrast, images with the smallest variance mostly contain people detected perfectly and without occlusion – the model is thus most conﬁdent about the feature representations for these.
Revised: In contrast, images with the smallest variance mostly contain people detected perfectly and without occlusion – the model is thus most conﬁdent about the feature representations for these.

Original: More SE-SNN implementation details for the label noise robust learning and more results can be found in the Appendix A. Further analyses across all tasks including hyperparameter analysis, solving multiple tasks in one model, effects of the number of stochastic layers are given in Appendix B.
Revised: More SE-SNN implementation details for the label noise robust learning and more results can be found in the Appendix A. Further analyses across all tasks including hyperparameter analysis, solving multiple tasks in one model, effects of the number of stochastic layers are given in Appendix B.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a simple and effective stochastic neural network framework.
Revised: We proposed a simple and effective stochastic neural network framework.

Original: Our model is related to VIB and variational dropout, but provides a simpler and more direct realization via neuron regularization by a non-informative activation prior.
Revised: Our model is related to VIB and variational dropout, but provides a simpler and more direct realization via neuron regularization by a non-informative activation prior.

Original: Our extensive experiments show that this simple framework has diverse beneﬁts for network pruning, adversarial defense and label noise robust learning.
Revised: Our extensive experiments show that this simple framework has diverse beneﬁts for network pruning, adversarial defense and label noise robust learning.

Original: A E XPERIMENTAL D ETAILS OF L ABEL N OISE
Revised: A E XPERIMENTAL D ETAILS OF L ABEL N OISE

Original: For the experiments on MNIST, we follow (Patrini et al., 2017) to train a neural network with two FC layers of dimension 128, where both FC layers are implemented as stochastic layers.
Revised: For the experiments on MNIST, we follow (Patrini et al., 2017) to train a neural network with two FC layers of dimension 128, where both FC layers are implemented as stochastic layers.

Original: For
Revised: For

Original: Market-1501, following most recent state-of-the-art ReID models we use ResNet-50 (He et al., 2016) as the backbone.
Revised: Market-1501, following most recent state-of-the-art ReID models we use ResNet-50 (He et al., 2016) as the backbone.

Original: In this model, only the last unit of the ﬁnal residual block is implemented as a stochastic layer.
Revised: In this model, only the last unit of the ﬁnal residual block is implemented as a stochastic layer.

Original: For Market-1501, we train our model for two stages: (1) We ﬁne-tune ImageNet pretrained ResNet- (for identity classiﬁcation) on Market-1501 training set for 60 , 000 iterations.
Revised: For Market-1501, we train our model for two stages: (1) We ﬁne-tune ImageNet pretrained ResNet- (for identity classiﬁcation) on Market-1501 training set for 60 , 000 iterations.

Original: We use batch sizeand Adam optimizer (Kingma & Ba, 2015) with learning rate 3 .
Revised: We use batch sizeand Adam optimizer (Kingma & Ba, 2015) with learning rate 3 .

Original: 5 × 10 − 4 .
Revised: 5 × 10 − 4 .

Original: (2) We retain the model parameters trained in stage (1) and extend the last unit to a stochastic layer with randomly initialized σ layer.
Revised: (2) We retain the model parameters trained in stage (1) and extend the last unit to a stochastic layer with randomly initialized σ layer.

Original: Then, we train the stochastic layer and the classiﬁer for another 20 , 000 iterations with a lower learning rate 5 × 10 − 4 .
Revised: Then, we train the stochastic layer and the classiﬁer for another 20 , 000 iterations with a lower learning rate 5 × 10 − 4 .

Original: The margin b is set as 4 in the max-entropy regularization loss (Eq. 6) and a loss weight 10 − 2 is assigned.
Revised: The margin b is set as 4 in the max-entropy regularization loss (Eq. 6) and a loss weight 10 − 2 is assigned.

Original: For both benchmarks, due to the randomness of noisy samples in selection and label reassignment, multiple runs of experiments are conducted.
Revised: For both benchmarks, due to the randomness of noisy samples in selection and label reassignment, multiple runs of experiments are conducted.

Original: The standard deviation values of Re-ID results on
Revised: The standard deviation values of Re-ID results on

Original: Market-1501 with label-noise are shown in Table A.1.
Revised: Market-1501 with label-noise are shown in Table A.1.

Original: B F URTHER A NALYSIS B. H YPER - PARAMETERS A NALYSIS
Revised: B F URTHER A NALYSIS B. H YPER - PARAMETERS A NALYSIS

Original: In this section, we analyze the impact of loss weight, which is denoted as ω , and the margin b for regularizers of all tasks on MNIST.
Revised: In this section, we analyze the impact of loss weight, which is denoted as ω , and the margin b for regularizers of all tasks on MNIST.

Original: We report the FLOPs of the compressed model (Network pruning), adversarial defense accuracy under the FGSM attack (Adversarial defense) and test accuracy of a model trained with 40% label noise (Label noise) in Figure B.1.
Revised: We report the FLOPs of the compressed model (Network pruning), adversarial defense accuracy under the FGSM attack (Adversarial defense) and test accuracy of a model trained with 40% label noise (Label noise) in Figure B.1.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: In Figure B.1(top), we evaluate the effects of having different values of ω from [10 − 1 , 10 − 2 , 10 − 3 ] .
Revised: In Figure B.1(top), we evaluate the effects of having different values of ω from [10 − 1 , 10 − 2 , 10 − 3 ] .

Original: We can see that, for all three values and across the three tasks, the performance of our method is only marginally impacted by ω and the overall best performance is achieved when ω = 10 − 2 .
Revised: We can see that, for all three values and across the three tasks, the performance of our method is only marginally impacted by ω and the overall best performance is achieved when ω = 10 − 2 .

Original: In Figure B.1(bottom), we investigate the sensitivity of our model to the values of margin b .
Revised: In Figure B.1(bottom), we investigate the sensitivity of our model to the values of margin b .

Original: Three values, [log(1) , log(4) , log(16)] are considered.
Revised: Three values, [log(1) , log(4) , log(16)] are considered.

Original: Recall that our SE-SNN uses a non-informative prior to regularize the uncertainty of neuron activations, and a margin b is employed to upper-bound the standard deviation of the stochastic layers in SE-SNN.
Revised: Recall that our SE-SNN uses a non-informative prior to regularize the uncertainty of neuron activations, and a margin b is employed to upper-bound the standard deviation of the stochastic layers in SE-SNN.

Original: Therefore, larger values of b will lead to a higher degree of uncertainty of stochastic layers.
Revised: Therefore, larger values of b will lead to a higher degree of uncertainty of stochastic layers.

Original: From Figure B.1(bottom), we can see that increasing the value of b improves the performance of our method on adversarial defense and label noise.
Revised: From Figure B.1(bottom), we can see that increasing the value of b improves the performance of our method on adversarial defense and label noise.

Original: However, for network pruning, smaller b is preferred.
Revised: However, for network pruning, smaller b is preferred.

Original: Overall, our model is insensitive to the value of b .
Revised: Overall, our model is insensitive to the value of b .

Original: B. C OMBINATION OF NETWORK PRUNING AND LABEL NOISE
Revised: B. C OMBINATION OF NETWORK PRUNING AND LABEL NOISE

Original: The effectiveness of our SE-SNN has been demonstrated on the three tasks separately, but there is no reason why it cannot be deployed to deal with multiple tasks simultaneously.
Revised: The effectiveness of our SE-SNN has been demonstrated on the three tasks separately, but there is no reason why it cannot be deployed to deal with multiple tasks simultaneously.

Original: To validate this, in this experiment, we train SE-SNN on MNIST with 40% label noise while conducting model pruning.
Revised: To validate this, in this experiment, we train SE-SNN on MNIST with 40% label noise while conducting model pruning.

Original: We use the same architecture with three FC layers (784-128-128-10) as the previous label noise experiments, and make the ﬁrst two layers stochastic layers.
Revised: We use the same architecture with three FC layers (784-128-128-10) as the previous label noise experiments, and make the ﬁrst two layers stochastic layers.

Original: We compare the trained model with two competitors, including the baseline model (without stochastic layers) and our normal SE-SNN (without pruning) both trained with label noise.
Revised: We compare the trained model with two competitors, including the baseline model (without stochastic layers) and our normal SE-SNN (without pruning) both trained with label noise.

Original: From the results in Figure B.2, we can see that we get a pruned SE-SNN (145-26-15-10) trained with label noise achieving 95.20% test accuracy, which is even slightly better than our normal SE-SNN (95.17%) and much better than the baseline (81.04%).
Revised: From the results in Figure B.2, we can see that we get a pruned SE-SNN (145-26-15-10) trained with label noise achieving 95.20% test accuracy, which is even slightly better than our normal SE-SNN (95.17%) and much better than the baseline (81.04%).

Original: B. E FFECT OF THE NUMBER OF STOCHASTIC LAYERS
Revised: B. E FFECT OF THE NUMBER OF STOCHASTIC LAYERS

Original: In this section, we study the effect of the number of used stochastic layers (SLs).
Revised: In this section, we study the effect of the number of used stochastic layers (SLs).

Original: The experiments are run on the label noise robust learning task with network comprising two FC layers.
Revised: The experiments are run on the label noise robust learning task with network comprising two FC layers.

Original: For the network with one SL, we make the penultimate layer stochastic.
Revised: For the network with one SL, we make the penultimate layer stochastic.

Original: Note that the network with zero SL is the baseline.
Revised: Note that the network with zero SL is the baseline.

Original: Figure B.3 shows the test accuracy of models with different numbers of SLs used.
Revised: Figure B.3 shows the test accuracy of models with different numbers of SLs used.

Original: We can see that increasing the number of SLs helps improve the model’s robustness to label noise.
Revised: We can see that increasing the number of SLs helps improve the model’s robustness to label noise.

Revised: B. C ONVERGENCE CURVE

Revised: The training efﬁciency of the proposed method is studied here by examining how fast the algorithm converges.

Revised: The experiments are run on the label noise robust learning task.

Revised: Figure B.4 shows the losses of different methods during 10 training epochs.

Revised: It is observed that Bootstrap and Forward have larger loss values than the rest.

Revised: This is because they use losses in addition to the classiﬁcation loss.

Revised: Overall our method is shown to be as efﬁcient as other methods, despite the fact that we have introduced stochastic layers and used a reparameterization trick during training.