Paper:
We consider the learning to learn problem, where the goal is to leverage deep learning models to automatically learn (iterative) optimization algorithms for training machine learning models. A natural way to tackle this problem is to replace the human-designed optimizer by an LSTM network and train the parameters on some simple optimization problems (Andrychowicz et al., 2016). Despite their success compared to traditional optimizers such as SGD on a short horizon, these learnt (meta-) optimizers suffer from two key deficiencies: they fail to converge (or can even diverge) on a longer horizon (e.g., 10000 steps). They also often fail to generalize to new tasks. To address the convergence problem, we rethink the architecture design of the meta-optimizer and develop an embarrassingly simple, yet powerful form of meta-optimizers-a coordinate-wise RNN model. We provide insights into the problems with the previous designs of each component and redesign our SimpleOptimizer to resolve those issues. Furthermore, we propose a new mechanism to allow information sharing between coordinates which enables the meta-optimizer to exploit second-order information with negligible overhead. With these designs, our proposed SimpleOptimizer outperforms previous metaoptimizers and can successfully converge to optimal solutions in the long run. Furthermore, our empirical results show that these benefits can be obtained with much smaller models compared to the previous ones.

Review:
In this paper, the authors build on the 'learning to learn' work, that aims to leverage deep learning models with optimization algorithms, most commonly with recurrent networks.  The goal is to utilize meta-learners that can adapt the optimization update strategy based on data/experience.  The authors aim to tackle problems that often arise in such meta-optimization schemes, such as covergence and generalization problems.  The paper is overall well-written, and several experiments are presented
Building on previous work, the authors propose some variations to the meta-learning schemes and architecture.  I comment on these below.
- The authors remove the bias term based on the intuition that for the meta-learners, removing the bias can lead to better convergence.  This is supported by a set of experiments in Fig 3a (but with only one learning rate?).  This experiment shows an extraordinary difference in accuracy between including and not including the RNN bias.  I think, because this difference is substantial (along with papers that do use the bias in meta-learners showing good results overall), more evidence/ablations are required.  
Can we can be sure that if the bias is non-zero (as in other reported works), we should expect a worse performance?.  Clearly this change seems to speed up the process, and in the example explained in the paper it makes sense, but could there be examples where lack of bias might lead to worse results?
- The authors claim that one of the problems of learning-to-learn algorithms is the late-stage optimization strategy.  While previous works introduce a weighting for the meta-optimizer's loss function (usually binary), the authors extend this to continuous monotonically increasing functions in order to weight the late stage steps of the algorithm more.  The authors propose compensating small loss difference happening during late optimization with a larger weight.   This naturally brings the question of which function should be used, as showing that any function will do seems difficult in practice. Have the authors tried any functions that failed or led to overfitting in comparison to other works?  Could one argue that if the loss difference is not large then perhaps the network can be more prone to overfitting at least in some cases?
Finally, the authors introduce embedding sharing between coordinates (aggregating hidden states), and also propose fine-tuning (sampling 5% of training data 5 times for warm-up).  Sharing hidden state information is expected to be useful (and has been often employed in literature), similarly to fine-tuning.  Also, 5% of dataset and 5 times seems to be an arbitrary choice - probably more related to the experimental section rather than the method itself.  
a question on fine-tuning:  if fine-tuning was used for the proposed method, have the compared methods also been pre-trained to provide fair comparisons?  This is particularly relevant to fig 4B where only the fine-tuned method slightly overperforms SGD.