Paper:
Natural Neural Networks
We introduce Natural Neural Networks, a novel family of algorithms that speed up convergence by adapting their internal representation during training to improve conditioning of the Fisher matrix. In particular, we show a specific example that employs a simple and efficient reparametrization of the neural network weights by implicitly whitening the representation obtained at each layer, while preserving the feed-forward computation of the network. Such networks can be trained efficiently via the proposed Projected Natural Gradient Descent algorithm (PRONG), which amortizes the cost of these reparametrizations over many parameter updates and is closely related to the Mirror Descent online learning algorithm. We highlight the benefits of our method on both unsupervised and supervised learning tasks, and showcase its scalability by training on the large-scale ImageNet Challenge dataset.

Review:
This paper develops a neural networks optimization method based on the idea of optimizing a version of the network where the values of the units at each layer are whitened and centered.This is reminiscent of centering approaches, which only do the centering.The method is presented as a type of approximate natural gradient descent, and competitive/superior performance compared to methods like SGD and Batch Normalization is demonstrated on various deep neural network training problems.The paper is reasonably well written and the approximation approach seems interesting, although quite similar to several methods that have appeared recently (which are cited).The empirical comparisons seems mostly solid, although I'm not convinced that they are as fair as they could be.The approximation of the natural gradient used in the approach is not given any theoretical justification, although judging by the experiments it seems to be a reasonable one in practice.   L 35: what do you mean here by "probabilistic manifold" and "constant progress"?Surely the amount of progress per iteration of natural gradient descent will change as optimization proceeds.L 44: "do not take into account the smooth change".Actually I think some of these methods initialize their inner iterations from previous solutions so they would benefit from a slowly changing FisherL 120:Assuming this kind of independence seems like a very severe approximation to make, as these quantities are obviously closely related to each other.Do you have any justification for making this approximation?L 136:PNGD seems like a poor choice for a name since it sounds like it could describe any method combining ideas of natural gradient descent with projected gradient descent, instead of the very specific approximations that you are using here (which are also neural-net specific).L 152:I don't see why this has anything obviously to do with a trust-region.It should be spelled out in more detail.L 191:I found this paragraph very hard to follow.I think you should be more explicit about what you mean here by this diagonal rescaling of U, providing an equation etc.Are the rows of U being rescaled?In this case wouldn't you then have to rescaling the columns of V to preserve the feed-forward computation?Are the columns being rescaled (which is the interpretation I'm going with)?Then I don't see how any simple manipulation of V would preserve the computation.One could recompute V as W D^-1 U^-1, whenever U is replaced with U D, but this certainly doesn't correspond to a row scaling of V.L 212: is the * supposed to be after \nabla like that?L 215: Is it supposed to be \Omega^(t+1) in brackets here?L 235: I don't understand what it means to use a "whitened mirror map".Won't defining \psi completely change the kind of update you compute?L 238: Won't the updates made to \Omega involve multiplying by inverses of matrices too?How will this make things more efficient than the naive approach?L 256: I think the square root connection is entirely spurious.If what you do is equivalent to a form of natural gradient descent, then the updates, as considered in the original theta space, won't depend on square roots in any way.This is much unlike the case with RMSprop.L 260: I don't believe that the K-FAC approach uses any low-rank approximations.L 310: An improved condition number doesn't imply that the matrix is close to diagonal.Figure 2b) doesn't look particularly diagonal to me.L 332: Why do plots c) and d) both have validation error on the y axis?L 350: Sutskever et al (2013) showed that momentum is very important when optimizing these kinds of deep autoencoders and that the coefficient matters a lot.Choosing a fixed value from {0,0.9} is probably causing SGD with momentum to underperform here.L 384: What do you mean by extra perturbations?Where is this coming from?L 416: As far as I can tell, K-FAC does something like this but without the need for skip-connections, so it's not obvious that they should be needed.