Manifold Regularization for Locally Stable Deep Neural Networks
Authors: Charles Jin, Martin Rinard
Keywords: regularization, deep learning, adversarial robustness
Abstract: We apply concepts from manifold regularization to develop new regularization techniques for training locally stable deep neural networks. Our regularizers encourage functions which are smooth not only in their predictions but also their decision boundaries. Empirically, our networks exhibit stability in a diverse set of perturbation models, including $\ell_2$, $\ell_\infty$, and Wasserstein-based perturbations; in particular, against a state-of-the-art PGD adversary, a single model achieves both $\ell_\infty$ robustness of 40% at $\epsilon = 8/255$ and $\ell_2$ robustness of 48% at $\epsilon = 1.0$ on CIFAR-10. We also obtain state-of-the-art verified accuracy of 21% in the same $\ell_\infty$ setting. Furthermore, our techniques are efficient, incurring overhead on par with two additional parallel forward passes through the network; in the case of CIFAR-10, we achieve our results after training for only 3 hours, compared to more than 70 hours for standard adversarial training.