  Adding noises to artificial neural network(ANN) has been shown to be able to
improve robustness in previous work. In this work, we propose a new technique
to compute the pathwise stochastic gradient estimate with respect to the
standard deviation of the Gaussian noise added to each neuron of the ANN. By
our proposed technique, the gradient estimate with respect to noise levels is a
byproduct of the backpropagation algorithm for estimating gradient with respect
to synaptic weights in ANN. Thus, the noise level for each neuron can be
optimized simultaneously in the processing of training the synaptic weights at
nearly no extra computational cost. In numerical experiments, our proposed
method can achieve significant performance improvement on robustness of several
popular ANN structures under both black box and white box attacks tested in
various computer vision datasets.
