Towards an Adversarially Robust Normalization Approach
Authors: Muhammad Awais, Fahad Shamshad, Sung-Ho Bae
Keywords: robustness, BatchNorm, adversarial
TL;DR: Investigation of how BatchNorm causes adversarial vulnerability and how to avoid it. 
Abstract: Batch Normalization (BatchNorm) has shown to be effective for improving and accelerating the training of deep neural networks. However, recently it has been shown that it is also vulnerable to adversarial perturbations. In this work, we aim to investigate the cause of adversarial vulnerability of the BatchNorm. We hypothesize that the use of different normalization statistics during training and inference (mini-batch statistics for training and moving average of these values at inference) is the main cause of this adversarial vulnerability in the BatchNorm layer. We empirically proved this by experiments on various neural network architectures and datasets. Furthermore, we introduce Robust Normalization (RobustNorm) and experimentally show that it is not only resilient to adversarial perturbation but also inherit the benefits of BatchNorm.