Paper:
Plain recurrent networks greatly suffer from the vanishing gradient problem while Gated Neural Networks (GNNs) such as Long-short Term Memory (LSTM) and Gated Recurrent Unit (GRU) deliver promising results in many sequence learning tasks through sophisticated network designs. This paper shows how we can address this problem in a plain recurrent network by analyzing the gating mechanisms in GNNs. We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates. We compare this model with IRNNs and LSTMs on multiple sequence modeling benchmarks. The RINs demonstrate competitive performance and converge faster in all tasks. Notably, small RIN models produce 12%-67% higher accuracy on the Sequential and Permuted MNIST datasets and reach state-of-the-art performance on the bAbI question answering dataset.

Review:
Summary: 
The authors present a simple variation of vanilla recurrent neural networks, which use ReLU hiddens and a fixed identity matrix that is added to the hidden-to-hidden weight matrix. This identity connection acts as a “surrogate memory” component, preserving hidden activations over time steps. 
The experiments demonstrate that this architecture reliably solves the addition task for up to 400 input frames. It also achieves a very good performance on sequential and permuted MNIST and achieves SOTA performance on bAbI.
The authors observe that the proposed recurrent identity network (RIN) is relatively robust to hyperparameter choices. After Le et al. (2015), the paper presents another convincing case for the application of ReLUs in RNNs.
Review: 
I very much like the paper. The motivation and architecture is presented very clearly and I am happy to also see explorations of simpler recurrent architectures in parallel to research of gated architectures!
I have a few comments and questions:
1) Clarification: In Section 2.2, do you really mean bit-wise multiplication or element-wise? If bit-wise, can you elaborate why? I might have missed something.
2) Why does the learning curve of the IRNN stop around epoch 270 in Figure 2c? Also some curves in the appendix stop abruptly without visible explosions. Were these experiments run until completion? If so, would it be possible to plot the complete curves?
3) I think for a fair comparison with LSTMs and IRNNs a limited hyperparameter search should be performed separately on all three architectures at least for the addition task. Optimal hyperparameters are usually model-specific. Admittedly, the authors mention that they do not intend to make claims about superior performance to LSTMs, however the competitive performance of small RINs is mentioned a couple of times in the manuscript.
Le et al. (2015) for instance perform a coarse grid search for each model.
4) I wouldn't say that ResNets are Gated Neural Networks, as the branches are just summed up. There is no (multiplicative) gating as in Highway Networks.
5) I think what enables the training of very deep networks or LSTMs on long sequences is the presence of a (close-to-)identity component in forward/backward propagation, not the gating. The use of ReLU activations in IRNNs (with identity initialization of the hidden-to-hidden weights) and RINs (effectively initialized with identity plus some noise) makes the recurrence more linear than with squashing activation functions.
6) Regarding the absence of gating in RINs: What is your intuition on how the model would perform in tasks for which conditional forgetting is useful. Consider for example a task with long sequences, outputs at every time step and hidden activations not necessarily being encouraged to estimate last step hidden activations. Would RINs readily learn to reset parts of the hidden state?
7) Henaff et al. (2016) might be related, as they are also looking into the addition task with long sequences.
Overall, the presented idea is novel to the best of my knowledge and the manuscript is well-written. I would recommend it for acceptance, but would like to see the above points addressed (especially 1-3 and some comments on 4-6). After a revision I would consider to increase the score.
References:
Henaff, Mikael, Arthur Szlam, and Yann LeCun. "Recurrent orthogonal networks and long-memory tasks." In International Conference on Machine Learning, pp. 2034-2042. 2016.
Le, Quoc V., Navdeep Jaitly, and Geoffrey E. Hinton. "A simple way to initialize recurrent networks of rectified linear units." arXiv preprint arXiv:1504.00941 (2015).