Paper:
Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
This paper seeks to answer the question: as the (near-) orthogonality of weights is found to be a favorable property for training deep convolutional neural networks, how can we enforce it in more effective and easy-to-use ways? We develop novel orthogonality regularizations on training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and restricted isometry property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on state-of-the-art models: ResNet, WideResNet, and ResNeXt, on several most popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN and ImageNet. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and faster and more stable convergences. We have made our codes and pre-trained models publicly available.
1 .

Review:
SUMMARY This work investigates various possible soft orthogonality constraints, promoting feature diversity in convolution layers, and yielding some improvement in image classification on CIFAR-10 and CIFAR-100.  STRENGTHS - Tested on popular classification models. - Pretty good overview of prior work. - SRIP is framed in an interesting way in equation 6, for use as a regularizer. - SRIP may be fairly cheap, computationally, when roughly approximated as proposed.  CRITICISMS - Because the proposed constraints enforce filters in convolution layers to be orthogonal to each other, they do not enforce the convolutional operators to be orthogonal; rather, they promote feature diversity in convolution layers. This is indeed proposed in the referenced "all you need is a good init" (10) paper as a regularizer. Such a constraint does not target the condition number of layer or address vanishing or exploding signals through a layer. This must be clearly stated and discussed. - Regarding 'scheme change': is it useful to even have a weight decay term for non-convolutional layers? Weight decay in DNNs is in practice enforced as an L2 norm on the weights which is essentially a penalty on the spectral radius (it bounds the largest singular value). Results without scheme change are not shown. The utility of 'scheme change' is stated but not proven. - In section 3.1: if W is m*n, then W'W is n*n, not m*m. Furthermore, it is stated that m ≤ n is undercomplete and necessary for the |W'W-I| constraint to work but this is inaccurate. This is actually correct elsewhere in the text but in this section, some symbols appear incorrectly arranged. If m < n, then W may be overcomplete, restricting W'W is problematic, and WW' might work. If m > n, then it may be undercomplete and restricting W'W works. Finally, if m = n, then W may be complete (and restricting W'W works).     -- "if and only if W is undercomplete (m ≤ n)" -> '(n ≤ m)'     -- "For overcomplete W (m > n), its gram matrix ∈ m × m cannot be even close to identity, because its rank is at most n" -> '(n > m)' and 'rank is at most m' - Final performace values tend to be very close together. What is the variance when rerunning experiments? Rerunning an exact reproduction of wide resnet, variance contains the results of multiple papers that claimed SOTA over wide resnets.     -- The authors state that MC outperforms the baseline with a 6.97% error vs 7.04% error on CIFAR 10 and 25.42% for both on CIFAR-100. The performance appears identical for both. - There is no description of the way validation data was split from training data. Was there always a validation set separate from the test set? This should be detailed and made clear. If testing was done on the test set (as in wide resnet), the evaluation is incorrect. - It would be useful to compare against a norm constraint on neighbouring activations ("Regularizing RNNs by Stabilizing Activations" - 2015) or a norm constraint on the gradients between neighbouring layers ("On the difficulty of training recurrent neural networks." - 2013).  MINOR COMMENTS - Orthogonality constraints may be more useful for RNNs and GAN discriminators than for ResNets, which already have a stable gradient guarantee, or for encouraging feature diversity in convolution lahyers. - "which provides a potential link between regularizing orthogonality and spectrum." -- it's the same thing - "we gradually reduce λ (initially 0.1) by factors of 1e-3, 1e-4 and 1e-6" - not 'by factors of', rather 'to'     - learning rate "decreased by a factor of 0.2" - 'by a factor of' means divided by, which would in this case mean the lr is increasing     - similarly, 'by a factor of 0.1' - It's surprising that "DSO always outperforms selective regularization". - SRIP is interesting. There should be a discussion on the time cost of different regularization approaches. With such a rough approximation as proposed (2 power method iterations), SRIP reduces computational cost from O(n**3) for soft orthogonality to O(n**2); from matrix-matrix multiplication to matrix-vector multiplication. - Perhaps there is a better lambda schedule for soft orthogonality that would make it outperform SRIP. - The MC trick may need a higher weight at first than soft orthogonality and SRIP due to a sparser gradient signal. - If (4) is an approximation, it's not "equivalent".  TYPOS     - in Assumption 1: z in [R]^n - "but unnecessarily appropriate" - not necessarily - "phase(direction)" - space - "without bothering any assistance components." - strange wording - "could possibly to mutually" - 'to' -> 'be' - "structure is most benefit at the initial stage" - 'beneficial' - eq 5 missing vertical bar  OVERALL OPINION The evaluation of which orthogonality regularizers are possible and useful, and by how much, is a useful one for the community. The SRIP formulation is interesting. However, the scope of the analysis should be expanded and the robustness of comparisons should be improved with statistical analysis. The effect of the proposed regularizations on learned representations and on network dynamics should be clearly identified and discussed. Some errors should be corrected. I would encourage submitting this work to a workshop.  UPDATE Significant work was done in the rebuttal to address the criticisms.  Although the authors did not address the significance of results in Table 1, they (a) confirmed the reliability of results in Table 2 and (b) performed additional experiments on both SVHN and ImageNet that go a long way toward convincing me that SRIP could be a reliably useful regularizer. The authors also included additional experiments with constraints on gradient norm change, further demonstrating superior results with SRIP.  The authors demonstrated a will to correct errors and clarify misleading parts of the text. I would especially encourage that they follow through with clarifying in the beginning that the proposed work enforces orthogonality across filters, encouraging filter diversity, and does not enforce orthogonality "on linear transformations between hidden layers" (second paragraph) in the way that some of the references do. The authors clarified some concerns such as whether hyperparameters were tuned on the validation data.  Authors responded qualitatively about time complexity -- this should be in big O notation and would work in their favor to advertise in the text, along with the qualitative note about perceived wall clock time, as the low cost of the method is attractive.  In light of the additional results and the clarification going into the revision, I will raise my rating from a 4 to a 7.