Paper:
We propose NovoGrad, an adaptive stochastic gradient descent method with layerwise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well-tuned SGD with momentum, Adam, and AdamW. Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in a large batch setting, and (3) has half the memory footprint of Adam.

Review:
I am not sure about my assessment of this paper. 
The authors propose their approach as a mixture of two other approaches. Then, they proceed with an illustrative 2-D example where they apply the same hyperparameters for all tested methods. This does not seem appropriate especially given that later in the paper they show that SGD with momentum and Adam use very different hyperparameter settings such as learning rate and weight decay. Basically, SGD and NovoGrad perform very similar in Figure 6 and the difference might be due to a too large learning rate for SGD (and other approaches). 
Increasing it further does not help as Figure 7 suggests. It is not the case for NovoGrad because of its gradient normalization. However, what the authors don't show us is a different experimental setup with some trivial objective function where it would take NovoGrad an enormous amount of steps to converge to the optimum IF the algorithm is initialized far enough from the optimum. Since NovoGrad is based on normalized gradients, its steps are pretty much just signs of the original gradient multiplied by learning rate, especially in 2D. Thus, with some initial learning rate of 0.1 and when initialized in say (1, 10^10), it would take NovoGrad about 10^9 steps to converge independently on the scale of the objective function. This is not the case for SGD whose gradients are not normalized. This is actually a strong counterargument to the main claim of the paper that "NovoGrad is robust to the choice of learning rate and weight initialization". This argument in the paper is based solely on the results of the toy problem where other methods used large learning rates. 
The paper includes a set of experiments containing different methods and their hyperparameters. 
When the authors use some cosine function to schedule learning rate in Table 1, they do it for AdamW and NovoGrad but they use polynomial schedule for SGD. Why? Weight decay and learning rate values for SGD and NovoGrad are very different in the same table, why? Do the authors explain why this difference may happen, they don't. I believe that there is a better answer than just "this is the result of hyperparameter tuning". 
The authors note that when beta=0, NovoGrad becomes layer-wise NGD with decoupled weight decay. They suggest beta to be in [0.2, 0.5] and I am wondering which beta were used in different experiment (the default suggested value is not given). 
The experimental results suggest that NovoGrad performs slightly better than SGD with momentum but it has more hyperparameters and we don't know whether the results are due to different computation efforts in hyperparameter tuning. Figure 5 compares Adam and Novograd on WikiTex-103 and shows that Adam converges faster in terms training perplexity. However, Adam's test perplexity is worse than the one of NovoGrad. Interestingly, NovoGrad's test perplexity is better than its training perplexity especially in the beginning. It seems that NovoGrad is good here because it does not converge well, i.e., the results are likely due to regularization which is problematic in the original Adam.