Paper:
In this paper, we propose a generic and simple strategy for utilizing stochastic gradient information in optimization. The technique essentially contains two consecutive steps in each iteration: 1) computing and normalizing each block (layer) of the mini-batch stochastic gradient; 2) selecting appropriate step size to update the decision variable (parameter) towards the negative of the block-normalized gradient. We conduct extensive empirical studies on various non-convex neural network optimization problems, including multi layer perceptron, convolution neural networks and recurrent neural networks. The results indicate the blocknormalized gradient can help accelerate the training of neural networks. In particular, we observe that the normalized gradient methods having constant step size with occasionally decay, such as SGD with momentum, have better performance in the deep convolution neural networks, while those with adaptive step sizes, such as Adam, perform better in recurrent neural networks. Besides, we also observe this line of methods can lead to solutions with better generalization properties, which is confirmed by the performance improvement over strong baselines.

Review:
This paper proposes a family of first-order stochastic optimization schemes based on (1)  normalizing (batches of) stochastic gradient descents and (2) choosing from a step size updating scheme. The authors argue that iterative first-order optimization algorithms can be interpreted as a choice of an update direction and a step size, so they suggest that one should always normalize the gradient when computing the direction and then choose a step size using the normalized gradient. 
The presentation in the paper is clear, and the exposition is easy to follow. The authors also do a good job of presenting related work and putting their ideas in the proper context. The authors also test their proposed method on many datasets, which is appreciated.
However, I didn't find the main idea of the paper to be particularly compelling. The proposed technique is reasonable on its own, but the empirical results do not come with any measure of statistical significance. The authors also do not analyze the sensitivity of the different optimization algorithms to hyperparameter choice, opting to only use the default. Moreover, some algorithms were used as benchmarks on some datasets but not others. For a primarily empirical paper, every state-of-the-art algorithm should be used as a point of comparison on every dataset considered. These factors altogether render the experiments uninformative in comparing the proposed suite of algorithms to state-of-the-art methods. The theoretical result in the convex setting is also not data-dependent, despite the fact that it is the normalized gradient version of AdaGrad, which does come with a data-dependent convergence guarantee.
Given the suite of optimization algorithms in the literature and in use today, any new optimization framework should either demonstrate improved (or at least matching) guarantees in some common (e.g. convex) settings or definitively outperform state-of-the-art methods on problems that are of widespread interest. Unfortunately, this paper does neither. 
Because of these points, I do not feel the quality, originality, and significance of the work to be high enough to merit acceptance. 
Some specific comments:
p. 2: "adaptive feature-dependent step size has attracted lots of attention". When you apply feature-dependent step sizes, you are effectively changing the direction of the gradient, so your meta learning formulation, as posed, doesn't make as much sense.
p. 2: "we hope the resulting methods can benefit from both techniques". What reason do you have to hope for this? Why should they be complimentary? Existing optimization techniques are based on careful design and coupling of gradients or surrogate gradients, with specific learning rate schedules. Arbitrarily mixing the two doesn't seem to be theoretically well-motivated.
p. 2: "numerical results shows that normalized gradient always helps to improve the performance of the original methods when the network structure is deep". It would be great to provide some intuition for this.  
p. 2: "we also provide a convergence proof under this framework when the problem is convex and the stepsize is adaptive". The result that you prove guarantees a \theta(\sqrt{T}) convergence rate. On the other hand, the AdaGrad algorithm guarantees a data-dependent bound that is O(\sqrt{T}) but can also be much smaller. This suggests that there is no theoretical motivation to use NGD with an adaptive step size over AdaGrad.
p. 2-3: "NGD can find a \eps-optimal solution....when the objective function is quasi-convex. ....extended NGD for upper semi-continuous quasi-convex objective functions...". This seems like a typo. How are results that go from quasi-convex to upper semi-continuous quasi-convex an extension?
p. 3: There should be a reference for RMSProp.
p. 3: "where each block of parameters x^i can be viewed as parameters associated to the ith layer in the network". Why is layer parametrization (and later on normalization) a good way idea? There should be either a reference or an explanation.
p. 4: "x=(x_1, x_2, \ldots, x_B)". Should these subscripts be superscripts?
p. 4: "For all the algorithms, we use their default settings." This seems insufficient for an empirical paper, since most problems often involve some amount of hyperparameter tuning. How sensitive is each method to the choice of hyperparameters? What about the impact of initialization?
p. 4-8: None of the experimental results have error bars or any measure of statistical significance.
p. 5: "NG... is a variant of the NG_{UNIT} method". This method is never motivated.
p. 5-6: Why are SGD and Adam used for MNIST but not on CIFAR? 
p. 5: "we chose the best heyper-paerameter from the 56 layer residual network." Apart from the typos, are these parameters chosen from the training set or the test set? 
p. 6: Why isn't Adam tested on ImageNet?
  
POST AUTHOR RESPONSE: After reading the author response and taking into account the fact that the authors have spent the time to add more experiments and clarify their theoretical result, I have decided to upgrade my score from a 3 to a 4. However, I still do not feel that the paper is up to the standards of the conference.