Paper:
Equilibrated adaptive learning rates for non-convex optimization Harm de Vries
Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us design better suited adaptive learning rate schemes. We show that the popular Jacobi preconditioner has undesirable behavior in the presence of both positive and negative curvature, and present theoretical and empirical evidence that the socalled equilibration preconditioner is comparatively better suited to non-convex problems. We introduce a novel adaptive learning rate scheme, called ESGD, based on the equilibration preconditioner. Our experiments show that ESGD performs as well or better than RMSProp in terms of convergence speed, always clearly improving over plain stochastic gradient descent.

Review:
This paper proposes a new adaptive learning rate scheme for optimizing nonlinear objective functions that arise during the training of deep neural networks. The main argument is based on recent results that indicate that the difficulty of the optimization stems from the presence of saddle points rather than local minima in the optimization path.The saddle points slow down training since the objective function tends to be flat in many directions and ill-conditioned in the neighbourhood of the saddle points. The authors propose a new method for reducing the ill-conditioning (the problem of pathological curvature) by "preconditioning" the objective function through a linear change of variables, which reduces to left-multiplying the gradient descent update step with a learned preconditioning matrix D. They focus specifically on the case where D is diagonal, and they show how a diagonal D reduces to methods for learning parameter-specific learning rates, such as the well-known Jacobi preconditioner or RMSProp.  This is a nice framework within which to consider different schemes for adaptive learning rates. As a side note, it was unclear to me whether this link has been established before, or whether this was a contribution of this paper?Their main *theoretical* contribution is to show that the Jacobi preconditioner has undesirable behaviour in the presence of both positive and negative curvature, and to derive the optimal preconditioner which they call the "equilibration preconditioner" D^E as |H|^-1. However, since D^E is a function of the Hession which is O(n^2) in the number of weights of the network, both computing it and storing it becomes computationally intractable. Their main *practical* contribution is an efficient approximate algorithm "Equilibrated Stochastic Gradient Descent" (ESGD) which estimates the optimal D^E by using the R-operator and requiring roughly two additional gradient calculations (further amortized by only performing this step every ~20 iterations) and sampling a vector from a Gaussian.The theoretical and practical contributions are verified by their empirical results which show that ESGD outperforms both standard SGD, Jacobi preconditioning and RMSProp for training deep auto-encoder networks.Overall, the paper is fairly clearly written, although I had some issues with ambiguous notation, primarily between distinguishing between element-wise operators and matrix-operators and at some points where it appears that matrices are defined element-wise but equated with full matrices, or vice versa, e.g.:* Eqn 7 and the definition for D^-1 = 1 / ||A_{i, .}||_2 just above it. Do you mean here the i'th element of D^-1 is defined like that? Otherwise, what does the subscript i refer to in A_{i,.} ? * Eqn 10 (which I'm taking to mean that each element i on the diagonal of D^E is defined as the norm of the i'th row of the Hessian, ||H_{i,.}||, is that correct?). Overall it would be helpful to add a section at the beginning on your notation, and specifying which operators act element-wise (e.g. |A|) and how you index into matrices, e.g. what do you mean by * A_{i,.} vs A_i (I'm presuming one indexes a row from a full matrix and the other an element from a diagonal matrix?), or* q_i q_j (e.g. below Eqn 11) vs q_{j,i} (in the supplementary material). Are these both the same?In the supplementary material, it was not clear to me in Proposition 3 how the q on the RHS of the inequality is still squared after applying (q_*^2)^-{1/2} which seems to be q_*^{-1}?