Paper:
Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations
We investigate the parameter-space geometry of recurrent neural networks (RNNs), and develop an adaptation of path-SGD optimization method, attuned to this geometry, that can learn plain RNNs with ReLU activations. On several datasets that require capturing long-term dependency structure, we show that path-SGD can significantly improve trainability of ReLU RNNs compared to RNNs trained with SGD, even with various recently suggested initialization schemes.

Review:
In this work the authors present an adaptation of the Path-SGD algorithm for RNNs. They characterize all node-wise invariances in RNNs and show that the update rule for their modified algorithm is invariant to all feasible node-wise rescalings. They provide experiments which indicate that their modified optimization method yields improvements for plain RNNs trained with existing methods, although they still lag behind LSTMs.
The authors frame their work as a step towards bridging the gap between plain RNNs and more complex LSTMs/GRUs by using an optimization method which accounts for the geometry of the loss surface. This seems to be a worthwhile goal (since plain RNNs are computationally cheaper and easier to analyze theoretically) and their experiments show some promising results in improving performance over plain RNNs trained with existing optimization methods.   However, it is not clear to me how the method that the authors use in practice differs significantly from regular Path-SGD introduced in previous work. The authors do present an adaptation of Path-SGD to networks with shared weights, and show that the new rescaling term applied to the gradients can be divided into two terms k1 and k2. But then, they note that the second term, which accounts for interactions between shared weights along the same path, is expensive to calculate for RNNs and show some empirical evidence that including it does not help performance. In the rest of the experiments, they ignore the second term, which to my understanding is essentially what makes the method introduced here different from regular Path-SGD. Isn't ignoring the second term the same as simply applying Path-SGD as if the weights were not shared, and combining the rescaling terms corresponding to shared weights? Also, part of the justification for this work in Section 3 is that RNNs with ReLUs can encounter the "exploding activation" problem due to the fact that ReLUs do not saturate, and that taking into account the interactions between shared weights along the same path can avoid this problem. I would have thought the second term would be necessary to avoid this. I would like the authors to clarify the major differences, if any, between the approximation that they use in the experiments (which ignores the second term) and the Path-SGD method introduced in other work, and its relation to the exploding activation problem in ReLU-RNNs. I would be willing to raise my score if the authors can explain how the method that they use in practice is significantly different than the previous Path-SGD algorithm.  The experiments do indicate that using path normalization offers a consistent improvement for plain RNNs on language modeling and sequential MNIST, which is encouraging, although they have yet to catch up to the LSTM.   A few remarks: -Section 5.1: it would be better to report experiments on the same tasks that are used in evaluation, i.e. character level rather than word-level experiments.   - for addition problem experiments, it seems like the authors initialize the RNN with identity transition matrix (this could be made more clear). Note that this is close to the exact solution presented in "Recurrent Orthogonal Networks and Long Memory Tasks", Henaff et al. 2016, and gives the RNN an advantage. Also, the experiments in that paper show that LSTMs can solve the addition task for T=750. For fair comparison with LSTMs, the RNN should be initialized randomly. -Line 217: both version -> both versions