Paper:
Recurrent neural networks (RNNs) are particularly well-suited for modeling longterm dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate. While a number of works attempt to mitigate this effect through gated recurrent units, skip-connections, parametric constraints and design choices, we propose a novel incremental RNN (iRNN), where hidden state vectors keep track of incremental changes, and as such approximate state-vector increments of Rosenblatt's (1962) continuous-time RNNs. iRNN exhibits identity gradients and is able to account for long-term dependencies (LTD). We show that our method is computationally efficient overcoming overheads of many existing methods that attempt to improve RNN training, while suffering no performance degradation. We demonstrate the utility of our approach with extensive experiments and show competitive performance against standard LSTMs on LTD and other non-LTD tasks.

Review:
Update: the authors have addressed my issues below and I have raised my score to 8.
***
This paper on recurrent neural networks goes back to Rosenblatt's continuous time dynamics model and uses a discretised version of that equation (equation (1) in the paper) to build an incremental version of the RNN, called incremental RNN, where the transition from hidden state h_{k-1} at step k-1 to next step's h_k is done using small incremental updates until the system achieves equilibrium. It claims that it manages to solve the vanishing gradient problem by keeping all gradients \frac{\partial h_k}{\partial h_{k-1}} equal to minus identity matrix. The algorithm is then extensively evaluated on a large number of tasks and compared with plain RNNs, LSTMs, and two recently published papers on antisymmetric RNN and FastRNN.
I need to admit that after reading the paper twice, I am not sure I understand how the method works exactly (how does inserting intermediary steps and variables g_0, g_1, ... g_T enable the system to reach equilibrium: is there an iterative evaluation until convergence?) and more worringly, how the single-step SiRNN differs from a normal RNN with an extra residual connection?
According to equation (5), for T=1 and g_0=0, we have:
g_T = g_1 = \eta_k^1 ( \phi (U (h_{k-1}) + W x_k + b) - \alpha h_{k-1} ).
If the gradients are vanishing in the normal RNN, why would they not vanish here for T=1?
Propositions 1 and 2 are for the case K=\infinity, and I could not understand the proof of theorem 1 that shows why \frac{\partial h_k}{\partial h_{k-1}} = - I. This seems to be the major contribution of the paper and should be given prominence.
What is missing is clear explanation, like (Bengio et al, 1994), of the identity gradient and of how the algorithm works. These questions could be solved by including code or pseudo-code explaining how to actually implement incremental RNNs.
There are also several important papers recently published that have approached the problems of continous-time dynamics and relaxation of hidden state to equilibria.
* The paper does not mention at all Neural ODEs [2] [3] where the state flows in a continuously differentiable way thanks to the continuous-time residual network ODE formulation. Moreover, isn't the idea of inserting a relaxation to equilibrium using ODEs already implemented in the ODE-RNNs [3]?
* How do incremental updates related to Adaptive Computation Time [4]?
For this reason, I am currently tending to reject the paper, but am open to change my score upon clarifications and links to other similar work.
Additional remarks:
The first paragraph of the paper explains the Elman RNN, not RNNs in general.
Please cite [1] alongside Bengio et al (1994) for the problem of the vanishing gradient.
Define alpha in equation (1)
Notation k and K is very confusing
Blue vs. green on Figure 2 is hard to read, and where is the new initialisation?
Why do you add h_k^K to h_{k-1} in equation (8)? I thought it was g_k^K?
Keep the same colours for all experiments in figure 4.
[1] Hochreiter (1991) "Untersuchungen zu dynamischen neuronalen Netzen"
[2] Chen, Rubanova, Bettencourt & Duvenaud (2018) "Neural Ordinary Differential Equations"
[3] Rubanova, Chen & Duvenaud (2019) "Latent odes for irregularly-sampled time series"
[4] Graves (2016) "Adaptive computation time for recurrent neural networks"