  Adversarial training (AT) is currently one of the most successful methods to
obtain the adversarial robustness of deep neural networks. However, a
significant generalization gap in the robustness obtained from AT has been
problematic, making practitioners to consider a bag of tricks for a successful
training, e.g., early stopping. In this paper, we investigate data augmentation
(DA) techniques to address the issue. In contrast to the previous reports in
the literature that DA is not effective for regularizing AT, we discover that
DA can mitigate overfitting in AT surprisingly well, but they should be chosen
deliberately. To utilize the effect of DA further, we propose a simple yet
effective auxiliary 'consistency' regularization loss to optimize, which forces
predictive distributions after attacking from two different augmentations to be
similar to each other. Our experimental results demonstrate that our simple
regularization scheme is applicable for a wide range of AT methods, showing
consistent yet significant improvements in the test robust accuracy. More
remarkably, we also show that our method could significantly help the model to
generalize its robustness against unseen adversaries, e.g., other types or
larger perturbations compared to those used during training. Code is available
at https://github.com/alinlab/consistency-adversarial.
