  Normalization techniques have become a basic component in modern
convolutional neural networks (ConvNets). In particular, many recent works
demonstrate that promoting the orthogonality of the weights helps train deep
models and improve robustness. For ConvNets, most existing methods are based on
penalizing or normalizing weight matrices derived from concatenating or
flattening the convolutional kernels. These methods often destroy or ignore the
benign convolutional structure of the kernels; therefore, they are often
expensive or impractical for deep ConvNets. In contrast, we introduce a simple
and efficient ``convolutional normalization'' method that can fully exploit the
convolutional structure in the Fourier domain and serve as a simple
plug-and-play module to be conveniently incorporated into any ConvNets. Our
method is inspired by recent work on preconditioning methods for convolutional
sparse coding and can effectively promote each layer's channel-wise isometry.
Furthermore, we show that convolutional normalization can reduce the layerwise
spectral norm of the weight matrices and hence improve the Lipschitzness of the
network, leading to easier training and improved robustness for deep ConvNets.
Applied to classification under noise corruptions and generative adversarial
network (GAN), we show that convolutional normalization improves the robustness
of common ConvNets such as ResNet and the performance of GAN. We verify our
findings via extensive numerical experiments on CIFAR-10, CIFAR-100, and
ImageNet.
