Paper:
Deep learning has shown that learned functions can dramatically outperform handdesigned functions on perceptual tasks. Analogously, this suggests that learned optimizers may similarly outperform current hand-designed optimizers, especially for specific problems. However, learned optimizers are notoriously difficult to train and have yet to demonstrate wall-clock speedups over hand-designed optimizers, and thus are rarely used in practice. Typically, learned optimizers are trained by truncated backpropagation through an unrolled optimization process. The resulting gradients are either strongly biased (for short truncations) or have exploding norm (for long truncations). In this work we propose a training scheme which overcomes both of these difficulties, by dynamically weighting two unbiased gradient estimators for a variational loss on optimizer performance. This allows us to train neural networks to perform optimization faster than well tuned first-order methods. Moreover, by training the optimizer against validation loss (as opposed to training loss), we are able to learn optimizers that train networks to better generalization than first order methods. We demonstrate these results on problems where our learned optimizer trains convolutional networks in a fifth of the wall-clock time compared to tuned first-order methods, and with an improvement in validation loss.

Review:
Summary:
The paper presents a method for "learning an optimizer"(also in the literature Learning to Learn and a form of Meta-Learning) by using a Variational Optimization for the "outer" optimizer loss. The mean idea of the paper is to combine both the reparametrized gradient and the score-function estimator for the Variational Objective and weight them using a product of Gaussians formula for the mean. The method is simple and clearly presented. The paper also presents issues with the standard "learning to learn" optimizers, one being the short-horizon bias and as credited by the authors has been observed before in the literature, and the second one is what is termed the "exponential explosion of gradients" which I think lacks enough justification as currently presented (see below for details). The ideas are clearly stated, although the work is not groundbreaking, but more on combining several ideas into a single one. 
Experiments: 
The authors evaluate their method on a single task which consists of optimizing a 3-layer convolutional neural network on downsampled images from ImageNet. A key idea, not new to this work, is to optimize the meta-optimizer with respect to the validation dataset rather than the training, which seems to be crucial for any meaningful training to happen. Although the experiments do show so promising results, they seem to be somewhat limited (see below for details). There is also a small ablation study on how do different features presented to the optimizer affect its performance. Given the still small-scale experiments, I'm not sure this is a significant result for the community. 
Conclusion:
As a whole, I think the idea in the paper is a good one and worth investigating further. However, the objections I have on section 2.3 and the experiments seem to indicate that there needs to be more work into this paper to make it ready for publication. 
On section 2.3 and the explosion of gradients:
There is a mistake in the equation on page 4 regarding the "gradient with respect to the learning rate". Although the derivation in Appendix A is correct, the inner product in the equation starts wrongly from j=0, where it should in fact start at j = i + 1. To be more clear the actual enrolled equation for dw^T/dt for 3 steps back is:
dw^T/dt = (I - tH^{T-1})(I - tH^{T-2})(I - tH^{T-3}) dw^{T-3} - (I - tH^{T-1})(I - tH^{T-2}) g^{T-3} - (I - tH^{T-1}) g^{T-2} - g^{T-1} 
Hence the product must start at j = i + 1. 
It is correct that in this setting the equation is a polynomial of degree T of the Hessian, however, there are several important factors that the authors have not discussed. Namely, if the learning rate is chosen accordingly such that the spectral radius of the Hessian is less than 1/t then rather than the gradient exploding the higher order term will vanish. However, even if they do vanish for large T since the Hessian plays with smaller and smaller power to more recent gradients (after correcting the mistake in the equation) than the actual T-step gradient will never vanish (in fact even if tH = I then dw^T/dt = g^{T-1}). Hence the claims of exploding gradients made in this section coupled with the very limited theoretical analysis seem to unconvincing that this is nessacarily an issue and under what circumstances they are. 
The toy example with l(w) = (w - 4)(w - 3) w^2 is indeed interesting for visualizing a case where the gradient explosion does happen. However, surprisingly here the authors rather than optimizing the learning rate, which they analyzed in the previous part of the section, they are now optimizing the momentum. The observation that at high momentum the training is unstable are not really surprising as there are fundamental reasons why too high momentum leads to instabilities and these have been analyzed in the literature. Additionally, it is not mentioned what learning rate is used, which can also play a major role in the effects observed. 
As a whole, although the example in this section is interesting, the claims made by the authors and some of the conclusions seem to lack any significant justifications, in addition to the fact that usually large over-parameterized models behave differently than small models. 
Experiments:
I have a few key issues with the experimental setup, which I think need to be addressed:
1. The CNN being optimized is quite small - only 3 layers. This allows the authors to train everything on a CPU. The key issue here, as well with previous work on Learning to Learn, is that it is not clear how scalable is this method to very Deep Networks. 
2. Figure 1 - The setup is to optimize the problem for 10000 iterations, however, I think it is pretty clear even to the naked eye that the standard first-order optimizers (Adam/RMS/Mom) have not fully converged on this problem. Hence I think its slightly unfair to compare their "final performance" after this fixed period. Additionally using the curriculum the "meta"-optimizer is trained explicitly for 10000 iterations. Hence, it is also unclear if it retains its stability after letting it run for longer. From the text it is also unclear whether the authors have optimized the parameters of the first-order methods with respect to their training or validation performance - I hope this is the latter as that is the only way to fairly compare the two approaches. 
3. Figure 6 - the results here seem to indicate that the learned optimizer transfers reasonably well, achieving similar performance to first-order methods (slightly faster validation reduction). Given however that these are plots for only 10000 iterations it is still unclear if this is scalable to larger problems.