Paper:
First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks. Second-order methods, despite their better convergence rate, are rarely used in practice due to the prohibitive computational cost in calculating the second-order information. In this paper, we propose a novel Gram-Gauss-Newton (GGN) algorithm to train deep neural networks for regression problems with square loss. Our method draws inspiration from the connection between neural network optimization and kernel regression of neural tangent kernel (NTK). Different from typical second-order methods that have heavy computational cost in each iteration, GGN only has minor overhead compared to first-order methods such as SGD. We also give theoretical results to show that for sufficiently wide neural networks, the convergence rate of GGN is quadratic. Furthermore, we provide convergence guarantee for mini-batch GGN algorithm, which is, to our knowledge, the first convergence result for the mini-batch version of a second-order method on overparameterized neural networks. Preliminary experiments on regression tasks demonstrate that for training standard networks, our GGN algorithm converges much faster and achieves better performance than SGD.

Review:
Authors propose minimizing neural network using kernel ridge regression. (Formula 9 and Algorithm 1). Main difference of this method is compared to Gauss-Newton, is that it uses JJ' as curvature, which has dimensions b-by-by (batch size b), instead of J'J as curvature, which has dimensions m-by-m (number of parameters m).
When b is much smaller than m, this matrix is tractable to represent exactly. Related approach is taken by KKT (see Figure 1 of https://arxiv.org/pdf/1806.02958.pdf) which also replaces J'J with more tractable JJ'.
There is a long history of authors trying to extend second order methods to deep learning and and finding that curvature estimated on a small batch is extremely noisy, requiring large batches (see papers by Nocedal's group). Authors propose a method that estimates curvature from small batches. Given the history of failures in small-batch curvature estimation, the bar is high to show that small-batch curvature estimation works.
Bulk of the paper is dedicated to theoretical convergence and connections between concepts. Since the focus of the paper is on a new optimization method for deep learnning, I feel like convergence proofs can be moved to Appendix, and more of the paper should focus on practical aspects of the method. Also the connections to other concepts (ie, tangent kernel) are not essential to the paper and could be better left over for a tutorial paper.
I'm not convinced that their method works well enough to have practical impact.
- Their method seems to be limited to neural network with one output (ie, univariate regression task). This is a serious limitation and paper should highlight this more on this, given that vast majority of applications and benchmarks involve more than output variable.
- Practical implementation details are skimmed over. Section 3.3 brings up that to compute Jacobian, one needs to keep track of the output derivative on per-example basis. How is this accomplished? Modern frameworks like PyTorch and TensorFlow don't give an easy way to compute per-example derivatives efficiently.
- Experiments are performed on two tasks that are not well known in the literature. The choice is somewhat understandable given that their method performs for univariate regression, but also this makes it hard to evaluate whether the method works. SGD vs Gram-Gauss evaluation use parameter settings which are not comparable, so it's impossible to tell whether the improvement are due to better choice of hyper-parameters.
The changes needed to make this paper acceptable are extensive, and I would recommed a reject.
I would recommend authors attempt the following changes for future submission:
1. Make it work for multivariate regression. There's a conversion technique to represent multivariate regression in the same form as univariate regression (see Section 2.4 of "Rao, Toutenberg" Linear Models). Essentially it comes down concatenating o output Jacobians (o output classes) along the batch dimension.
2. Use this to evaluate the method on standard benchmarks like MNIST and CIFAR and show that it doesn't cause a significant worsening in quality. Given that similar approach (KKT paper) found bigger improvement on RNN task, an RNN task may be useful.
3. Give more details on implementation. How was Jacobian calculation implemented? Which framework? How was the per-example computation made tractable? Making small-scale experiments reproducible through anonymous github submission would also help