Paper:
Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space. These difficulties can be addressed by second-order approaches that apply a preconditioning matrix to the gradient to improve convergence. Unfortunately, such algorithms typically struggle to scale to high-dimensional problems, in part because the calculation of specific preconditioners such as the inverse Hessian or Fisher information matrix is highly expensive. We introduce first-order preconditioning (FOP), a fast, scalable approach that generalizes previous work on hypergradient descent (Almeida et al., 1998;Maclaurin et al., 2015;Baydin et al., 2017) to learn a preconditioning matrix that only makes use of first-order information. Experiments show that FOP is able to improve the performance of standard deep learning optimizers on visual classification and reinforcement learning tasks with minimal computational overhead. We also investigate the properties of the learned preconditioning matrices and perform a preliminary theoretical analysis of the algorithm.

Review:
This paper studies hypergradient descent for precondition matrices. The goal is to learn an adaptable preconditioning for the task while training. Specifically, they take the gradient of the loss wrt the precondition matrix and update the precondition matrix to decrease the loss. They reparametrize the precondition matrix to ensure it is positive-definite and provide low-rank approximations and they provide cheap approximations for CNNs.
Pros:
- Figure 3 and 4 show promising results on cifar10 with a 9-layer cnn.
- Figure 4 shows FOP can improve the accuracy for particular hyper-parameters. In cases improving by 2%.
Cons:
- Results on imagnet are not particularly good. The improvement is not significant.
- Why positive-definite precondition matrix rather than positive-semi-definite?
- Section 5: why is a degenerate precondition matrix bad? Fisher and Hessian for deep networks can be highly ill-conditioned.
- Theo 1 seems to have errors. The term M_t in the update rule should show up in the bound on P as an exponential term in the first upper bound.
- Figure 2: On mnist after 20 epochs the model has not reached 1% test error. Not clear if we can make any conclusions from this figure.
After rebuttal:
I keep my rating as weak reject. I reiterate that results look promising. However, the quality and accuracy of the writing are not acceptable for a paper on optimization. In my original review I only named a few problematic statements. I have to clarify that I do not think fixing only those few is enough.
I am also not convinced about the proof of Theorem 1. Basically, section 6 looks very much like section 5 from Baydin et al. 2018. Even the wording is mostly the same. Theorem 5.1 in Baydin et al. 2018 is based on their update rule in Eq 6 in the form of alpha_t = alpha_{t-1} - beta nabla^T nabla, where alpha does not appear in the second term. However, in this paper, the update rule on line 7 in Algorithm 1 is M_t = M_{t-1} + rho * eps *(.) M_{t-1}, where M_t appears in the second term. Hence, the first bound in Theorem 1 in this paper cannot simply be the same as in Baydin et al. 2018.