Paper:
EQUI-NORMALIZATION OF NEURAL NETWORKS
Modern neural networks are over-parametrized. In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and output weights, without changing the rest of the network. Inspired by the SinkhornKnopp algorithm, we introduce a fast iterative method for minimizing the 2 norm of the weights, equivalently the weight decay regularizer. It provably converges to a unique solution. Interleaving our algorithm with SGD during training improves the test accuracy. For small batches, our approach offers an alternative to batchand group-normalization on CIFAR-10 and ImageNet with a ResNet-18.

Review:
Summary:
This paper introduces equi-normalization (ENorm): a normalization technique that relies on the scaling invariance properties of the ReLU, similarly to Path-SGD. Their method explicitly use this property to balance the weights of the network, without changing the function computed by the network. The main difference with Path-SGD is that the network is explicitly balanced, while Path-SGD uses a regularizer to implicitly balance the network. Since it doesn’t rely on mini-batch to normalize the network, Equi-normalization could be a good alternative to BN in small mini-batch regime. The method is validated on 3 tasks (MLP on CIFAR10, CNN on CIFAR10, Reidual Network on ImageNet).
Clarity:
The paper is quite clear, although a bit long (10 pages). The related work section is particularly nice. I really appreciated the “positioning” paragraph, which really explains how the method differs from others.
Novelty:
The paper is quite incremental, due to its similarities with Path-SGD. It seems quite close to what Weight Normalization is doing as well (see detailed comments).
Pros and Cons:
+ Clearly written
+ Clearly motivated
+ Nice review of literature
- Quite incremental (close to Path-SGD / Weight Normalization), and missing actual comparison with Weight Normalization, which seems to be the direct competitor of ENorm (see detailed comments)
- Some flaws in the experimental setup (see detailed comments), particularly in the fully-connected experiment.
- Doesn’t scale (yet) to deeper architectures, which is precisely where small batch sizes become a problem for BN and thus where ENorm would be needed.
Detailed Comments:
1. Differences with Weight Normalization:
I have trouble seeing the difference between the proposed method and Weight Normalization (WN), or the more advanced Normalization Propagation (NP). It seems that both methods are performing quite similar normalization: WN reparameterize the network such that ||W[:,j]||_2 = 1, so it seems that you wouldn’t need to re-balance the network when using WN. Could the authors elaborate on this? Moreover WN is simple to implement, fast, and also works in the small batches settings as well. Finally, since those methods are quite similar, the authors should compare their method against WN in their experimental setup.
2. Initialization of the Weights:
The Xavier initialization you are using in the CIFAR10 experiments is designed to work with Tanh activations functions. For ReLUs, one should use the Kaiming initialization, which has been proven to work way better for ReLUs (He et al., 2015a).  This certainly explains the poor performances of your baseline when you increase the number of layers in Figure 4, and probably explains why you need to add a BN layer at the end of your network to help with the training of the baselines. I suggest the authors to re-run the baselines using proper initialization for ReLUs.
3. Fully-Connected Layers Benchmark:
I think the fully-connected benchmark you used is quite poor. A baseline with 1 layer only reaches ~54 % test accuracy and your method needs a 11 layer model to increase this baseline performance of about ~0.5 % only. The deep autoencoder on MNIST (see e.g. Desjardins et al., Natural neural networks, NIPS 2015), would probably be a better benchmark for fully-connected layers (of course using ReLUs in place of Sigmoids).  It would also reinforce your empirical results by adding a 3rd dataset.
4. ImageNet Experiment:
Do you use Ghost Batch Normalization in this experiments (i.e. calculating the BN statistics on each GPU separately)? It would certainly explain the poor performances of BN with tiny batch size (32 examples on 8 GPUs means only 8 examples per GPU).
5. Computation Time performances:
It is stated in the conclusion that “using ENorm during training adds a much smaller computational overhead than BN or GN …”. I can see that Table 1 gives an overview of the number of elements accessed during normalization, but I do think that a proper plot showing the accuracy versus the wall-clock time would be a better way of showing how your method compares in practice with BN or GN. Moreover, as stated previously, comparison with WN (and / or NP) need to be performed as well, especially because WN and NP are also way faster to compute than BN and GN.
6. Shortening the paper:
The recommended paper size for ICLR is 8 pages. Here would be a few pointers that could help you reduce a bit the length of the paper: Introduction and Related Work takes up 3 pages already and I think there is quite some overlap between the 2 (about BN in particular), so there is probably quite a lot of space to gain here if the authors were to reduce a bit the introduction or make the related work a sub-section of the introduction. The ENorm presentation is 4 pages long (which is quite a lot). Section 3.6 might be totally discarded since it is vanilla application of the chain rule. The extension to convolution layers and max pooling could be transferred to the appendix.
Minor Comments:
You should add “Optimizing neural networks with kronecker-factored approximate curvature” in the literature review about optimization landscape, as it is an important research direction on natural gradient.
Conclusion:
All in all, I find that this work is a bit too incremental, missing some important comparisons with other techniques and its experimental setup could definitely be improved. Also, the speedup claims should also be supported with empirical experimentation.
Revision:
I thank the authors for the all the extra experiments they performed. The paper looks good to me, and increased my evaluation accordingly.