Paper:
Convolution is a central operation in Convolutional Neural Networks (CNNs), which applies a kernel to overlapping regions shifted across the image. However, because of the immense amount of correlations in real-world image data, convolutional kernels are in effect re-learning redundant data. In this work, we show that this redundancy has made neural network training challenging, and propose network deconvolution, a procedure which optimally removes pixel-wise and channel-wise correlations before the data is fed into each layer. Network deconvolution can be efficiently calculated at a fraction of the computation cost of a convolution layer. We also show that the deconvolution filters in the first layer of the network resemble the center-surround structure found in biological neurons in the visual regions of the brain. Filtering with such kernels results in a sparse representation, a desired property that has been missing in the training of neural networks. Learning from the sparse representation promotes faster convergence and superior results without the use of batch normalization. We apply our network deconvolution operation to 10 modern neural network models by replacing batch normailization within each. Our extensive experiments show the network deconvolution operation is able to deliver performance improvement in all cases on CIFAR-10, CIFAR-100, MNIST, Fashion-MNIST and ImageNet datasets.

Review:
This paper addresses the correlation present in the input data, which may affect learning kernels with redundancy. Therefore, they introduce  a deconvolution mechanism on the input features to remove spatial and channel-wise correlation,  before these features are fed to network layers. They also show the deconvolution effect in the first layer resembles the center-surround effects found in biological neuron and induce sparsity in representation. They draw analogy to batch normalization, and show superiority in terms of  convergence and accuracy. 
Overall, the concept of the paper is pretty simple and straightforward - basically it removes the correlation present in the input data, specifically in the case of convolution.The experimental results are promising and show fast convergence over using batch normalization, slightly better accuracy.
However, it seems that the deconvolution to avoid correlation is helping for classification tasks, but may not be applicable for other related tasks such as semantic segmentation, where simply using BN may work.
What is G in the overall complexity? No. of groups? Please define. 
 
One of the questions that comes to mind is considering PCA normalization for the same purpose. Essentially, the step of computing the covariance is the same and decorrelating the data by  using the orthonormal basis vectors, pretty much with a similar motivation. Also, PCA is a linear transformation, which points to a way of learning in the network training. So, my question is what will be the problem of doing PCA transformation and then performing convolutions on that transformed decorrelated data? Maybe layerwise PCA transformation may also help in performance and reduce complexity? I think this discussion should be added to the normalization and whitening section of related works.
Minor note: The recommended page limit is 8 pages, the paper has now 10 pages. For me, Figure 1 is not conveying much information, and can be easily replaced with text. Also, I do not see the need of an extra section for the neurological basis for deconvolution; the brief discussion in the intro is enough.