Paper:
Stochastic neural networks (SNNs) are currently topical, with several paradigms being actively investigated including dropout, Bayesian neural networks, variational information bottleneck (VIB) and noise regularized learning. These neural network variants impact several major considerations, including generalization, network compression, and robustness against adversarial attack and label noise. However, many existing networks are complicated and expensive to train, and/or only address one or two of these practical considerations. In this paper we propose a simple and effective stochastic neural network (SE-SNN) architecture for discriminative learning by directly modeling activation uncertainty and encouraging high activation variability. Compared to existing SNNs, our SE-SNN is simpler to implement and faster to train, and produces state of the art results on network compression by pruning, adversarial defense and learning with label noise.

Review:
Summary: 
This paper presents an efficient stochastic neural network architecture by directly modeling activation uncertainty and adding a regularization term to encourage high activation variability by maximizing the entropy of stochastic neurons. Compared with other existing approaches, such as Bayesian neural networks and variational information bottleneck, the proposed architecture is simpler to implement and faster to train. The authors also achieve state of the art results in various fields, including network compression by pruning, adversarial defense and learning with label noise.
Major comments: 
- Overall, I find the paper is easy to follow and the experimental evaluation shows promising results, but my major concern is about the novelty of this work, given the fact that the structure of the proposed stochastic layers is quite similar to VIBNet.
- The derivation of the max-entropy term is somewhat unclear and I think the paper needs a major revision on this part. The authors suggest using a Gaussian with a finite mean and an infinite variance as the non-informative prior for the produced Gaussian random variable z (in Eq. 1), and then minimize the KL divergence between the produced Gaussian and the infinite-variance Gaussian. However, this may be questionable from a Bayesian viewpoint, in the sense that the infinite variance leads to an improper prior as the variance increases without bound and thus may produce an improper posterior distribution. This is not discussed and needs to be clarified in Sect. 2 (Max-entropy Regularization).
- There are things unclear in the derivation (last line of Eq. 2), since log(\sigma_2) trends to infinity.
- In addition, the penalty terms for different types of tasks are directly given only based on some of the assumptions that the authors have made, there does not seem to be any theoretical justification for such choices.
Minor comments:
- Some of the notations used in this paper seem a bit confusing, which may hinder readability. For example, on page 3, in “The non-informative prior is a Gaussian with arbitrary mean (\mu_1) and infinite variance (\sigma_1)”, I guess \sigma means the standard deviation? I would like to recommend using N(\mu, \sigma^2) to denote a Gaussian distribution, where \sigma means the standard deviation and \sigma^2 the variance.
- On page 3, in “where \sigma(h|\theta) denotes the predicted standard deviation of hidden unit h given the neuron uncertainty prediction parameter \theta”, there is no discussion on the neuron uncertainty prediction parameter. Does it means the predicted standard deviation is again parameterized by \theta?