Original:  I NCREMENTAL RNN: A D YNAMICAL V IEW . Anonymous authors Paper under double-blind review
Revised: RNN S I NCREMENTALLY E VOLVING ON AN E QUILIBRIUM M ANIFOLD : A P ANACEA FOR V ANISHING AND E XPLODING G RADIENTS ?

Original: A BSTRACT
Revised: A BSTRACT

Original: Recurrent neural networks (RNNs) are particularly well-suited for modeling longterm dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate.
Revised: Recurrent neural networks (RNNs) are particularly well-suited for modeling longterm dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate.

Original: While a number of works attempt to mitigate this effect through gated recurrent units, skip-connections, parametric constraints and design choices, we propose a novel incremental RNN (iRNN), where hidden state vectors keep track of incremental changes, and as such approximate state-vector increments of Rosenblatt’s (1962) continuous-time RNNs.
Revised: While a number of works attempt to mitigate this effect through gated recurrent units, skip-connections, parametric constraints and design choices, we propose a novel incremental RNN (iRNN), where hidden state vectors keep track of incremental changes, and as such approximate state-vector increments of Rosenblatt’s (1962) continuous-time RNNs.

Original: iRNN exhibits identity gradients and is able to account for long-term dependencies (LTD).
Revised: iRNN exhibits identity gradients and is able to account for long-term dependencies (LTD).

Original: We show that our method is computationally efﬁcient overcoming overheads of many existing methods that attempt to improve RNN training, while suffering no performance degradation.
Revised: We show that our method is computationally efﬁcient overcoming overheads of many existing methods that attempt to improve RNN training, while suffering no performance degradation.

Original: We demonstrate the utility of our approach with extensive experiments and show competitive performance against standard LSTMs on LTD and other non-LTD tasks.
Revised: We demonstrate the utility of our approach with extensive experiments and show competitive performance against standard LSTMs on LTD and other non-LTD tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recurrent neural networks (RNNs) in each round store a hidden state vector, h m ∈ R D , and upon receiving the input vector, x m +1 ∈ R d , linearly transform the tuple ( h m , x m +1 ) and pass it through a memoryless non-linearity to update the state over T rounds.
Revised: Recurrent neural networks (RNNs) in each round store a hidden state vector, h m ∈ R D , and upon receiving the input vector, x m +1 ∈ R d , linearly transform the tuple ( h m , x m +1 ) and pass it through a memoryless non-linearity to update the state over T rounds.

Original: Subsequently, RNNs output an afﬁne function of the hidden states as its prediction.
Revised: Subsequently, RNNs output an afﬁne function of the hidden states as its prediction.

Original: The model parameters (state/input/prediction parameters) are learnt by minimizing an empirical loss.
Revised: The model parameters (state/input/prediction parameters) are learnt by minimizing an empirical loss.

Original: This seemingly simple update rule has had signiﬁcant success in learning complex patterns for sequential input data.
Revised: This seemingly simple update rule has had signiﬁcant success in learning complex patterns for sequential input data.

Original: Nevertheless, that training RNNs can be challenging, and that performance can be uneven on tasks that require long-term-dependency (LTD), was ﬁrst noted by Hochreiter (1991), Bengio et al.
Revised: Nevertheless, that training RNNs can be challenging, and that performance can be uneven on tasks that require long-term-dependency (LTD), was ﬁrst noted by Hochreiter (1991), Bengio et al.

Original: (1994) and later by other researchers.
Revised: (1994) and later by other researchers.

Original: Pascanu et al.
Revised: Pascanu et al.

Original: (2013b) attributed this to the fact that the error gradient back-propagated in time (BPTT), for the time-step m , is dominated by product of partials of hiddenstate vectors, (cid:81) T − 1 j
Revised: (2013b) attributed this to the fact that the error gradient back-propagated in time (BPTT), for the time-step m , is dominated by product of partials of hiddenstate vectors, (cid:81) T − 1 j

Original: = m ∂h j +1 ∂h j , and these products typically exhibit exponentially vanishing decay or explosion, resulting in incorrect credit assignment during training and test-time.
Revised: = m ∂h j +1 ∂h j , and these products typically exhibit exponentially vanishing decay or explosion, resulting in incorrect credit assignment during training and test-time.

Original: Rosenblatt (1962) , on whose work we draw inspiration from, introduced continuous-time RNN (CTRNN) to mimic activation propagation in neural circuitry.
Revised: Rosenblatt (1962) , on whose work we draw inspiration from, introduced continuous-time RNN (CTRNN) to mimic activation propagation in neural circuitry.

Original: CTRNN dynamics evolves as follows:
Revised: CTRNN dynamics evolves as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, x ( t ) ∈ R
Revised: Here, x ( t ) ∈ R

Original: d is the input signal, g ( t ) ∈
Revised: d is the input signal, g ( t ) ∈

Original: R D is the hidden state vector of D neurons, ˙
Revised: R D is the hidden state vector of D neurons, ˙

Original: g i ( t ) is the rate of change of the i -th state component; τ, α ∈ R + , referred to as the post-synaptic time-constant, impacts the rate of a neuron’s response to the instantaneous activation φ ( Ug ( t )
Revised: g i ( t ) is the rate of change of the i -th state component; τ, α ∈ R + , referred to as the post-synaptic time-constant, impacts the rate of a neuron’s response to the instantaneous activation φ ( Ug ( t )

Original: + Wx ( t ) + b ) ; and
Revised: + Wx ( t ) + b ) ; and

Original: U ∈ R D × D , W ∈ R D × d , b ∈ R D are model parameters.
Revised: U ∈ R D × D , W ∈ R D × d , b ∈ R D are model parameters.

Original: In passing, note that recent RNN works that draw inspiration from ODE’s
Revised: In passing, note that recent RNN works that draw inspiration from ODE’s

Original: (Chang et al., 2019) are special cases of CTRNN ( τ = 1 , α = 0 ).
Revised: (Chang et al., 2019) are special cases of CTRNN ( τ = 1 , α = 0 ).

Original: Vanishing Gradients.
Revised: Vanishing Gradients.

Original: The qualitative aspects of the CTRNN dynamics is transparent in its integral form:
Revised: The qualitative aspects of the CTRNN dynamics is transparent in its integral form:

Original: [Equation]
Revised: [Equation]

Original: This integral form reveals that the partials of hidden-state vector with respect to the initial condition, ∂g ( t ) ∂g ( t 0 ) , gets attenuated rapidly (ﬁrst term in RHS), and so we face a vanishing gradient problem.
Revised: This integral form reveals that the partials of hidden-state vector with respect to the initial condition, ∂g ( t ) ∂g ( t 0 ) , gets attenuated rapidly (ﬁrst term in RHS), and so we face a vanishing gradient problem.

Original: We will address this issue later but we note that this is not an artifact of CTRNN but is exhibited by ODEs that have motivated other RNNs (see Sec. 2).
Revised: We will address this issue later but we note that this is not an artifact of CTRNN but is exhibited by ODEs that have motivated other RNNs (see Sec. 2).

Original: Shannon-Nyquist Sampling.
Revised: Shannon-Nyquist Sampling.

Original: A key property of CTRNN is that the time-constant τ together with the ﬁrst term − g ( t ) , is in effect a low-pass ﬁlter with bandwidth ατ − 1 suppressing high frequency components of the activation signal, φ (( Ug ( s ))
Revised: A key property of CTRNN is that the time-constant τ together with the ﬁrst term − g ( t ) , is in effect a low-pass ﬁlter with bandwidth ατ − 1 suppressing high frequency components of the activation signal, φ (( Ug ( s ))

Original: + ( Wx ( s )) + b ) .
Revised: + ( Wx ( s )) + b ) .

Original: This is good, because, by virtue of the Shannon-Nyquist sampling theorem, we can now maintain ﬁdelity of discrete samples with respect to continuous time dynamics, in contrast to conventional ODEs ( α = 0 ).
Revised: This is good, because, by virtue of the Shannon-Nyquist sampling theorem, we can now maintain ﬁdelity of discrete samples with respect to continuous time dynamics, in contrast to conventional ODEs ( α = 0 ).

Original: Additionally, since high-frequencies are already suppressed, in effect we may assume that the input signal x ( t ) is slowly varying relative to the post-synaptic time constant τ .
Revised: Additionally, since high-frequencies are already suppressed, in effect we may assume that the input signal x ( t ) is slowly varying relative to the post-synaptic time constant τ .

Original: Equilibrium.
Revised: Equilibrium.

Original: The combination of low pass ﬁltering and slowly time varying input has a signiﬁcant bearing.
Revised: The combination of low pass ﬁltering and slowly time varying input has a signiﬁcant bearing.

Original: The state vector as well as the discrete samples evolve close to the equilibrium state, i.e., g ( t )
Revised: The state vector as well as the discrete samples evolve close to the equilibrium state, i.e., g ( t )

Original: ≈
Revised: ≈

Original: φ ( Ug ( t )
Revised: φ ( Ug ( t )

Original: + Wx ( t ) + b ) under general conditions (Sec. 3).
Revised: + Wx ( t ) + b ) under general conditions (Sec. 3).

Original: Incremental Updates.
Revised: Incremental Updates.

Original: Whether or not system is in equilibrium, the integral form in Eq.
Revised: Whether or not system is in equilibrium, the integral form in Eq.

Original: 2 points to gradient attenuation as a fundamental issue.
Revised: 2 points to gradient attenuation as a fundamental issue.

Original: To overcome this situation, we store and process increments rather than the cumulative values g ( t ) and propose dynamic evolution in terms of increments.
Revised: To overcome this situation, we store and process increments rather than the cumulative values g ( t ) and propose dynamic evolution in terms of increments.

Original: Let us denote hidden state sequence as h m ∈ R D and input sequence x m ∈ R d .
Revised: Let us denote hidden state sequence as h m ∈ R D and input sequence x m ∈ R d .

Original: For m = 1 , 2 , . . .
Revised: For m = 1 , 2 , . . .

Original: , T , and a suitable β >
Revised: , T , and a suitable β >

Original: [Equation]
Revised: [Equation]

Original: Intuitively, say system is in equilibrium and − α ( µ ( x m , h m − 1 ))
Revised: Intuitively, say system is in equilibrium and − α ( µ ( x m , h m − 1 ))

Original: + φ ( Uµ ( x m , h m − 1 )
Revised: + φ ( Uµ ( x m , h m − 1 )

Original: + Wx m + b ) = 0 .
Revised: + Wx m + b ) = 0 .

Original: We note state transitions are marginal changes from previous states, namely, h m = µ ( x m , h m − 1 ) − h m − 1 .
Revised: We note state transitions are marginal changes from previous states, namely, h m = µ ( x m , h m − 1 ) − h m − 1 .

Original: Now for a ﬁxed input x m , as to which equilibrium is reached depends on h m − 1 , but are nevertheless ﬁnitely many.
Revised: Now for a ﬁxed input x m , as to which equilibrium is reached depends on h m − 1 , but are nevertheless ﬁnitely many.

Original: So encoding marginal changes as states leads to “identity” gradient.
Revised: So encoding marginal changes as states leads to “identity” gradient.

Original: Incremental RNN (iRNN) achieves Identity Gradient.
Revised: Incremental RNN (iRNN) achieves Identity Gradient.

Original: We propose to discretize Eq.
Revised: We propose to discretize Eq.

Original: 3 to realize iRNN (see Sec. 3).
Revised: 3 to realize iRNN (see Sec. 3).

Original: At time m , it takes the previous state h m − 1 ∈ R D and input x m ∈ R d and outputs h m ∈ R D after simulating the CTRNN evolution in discrete-time, for a suitable number of discrete steps.
Revised: At time m , it takes the previous state h m − 1 ∈ R D and input x m ∈ R d and outputs h m ∈ R D after simulating the CTRNN evolution in discrete-time, for a suitable number of discrete steps.

Original: We show that the proposed RNN approximates the continuous dynamics and solves the vanishing/exploding gradient issue by ensuring identity gradientIn general, we consider two options,
Revised: We show that the proposed RNN approximates the continuous dynamics and solves the vanishing/exploding gradient issue by ensuring identity gradientIn general, we consider two options,

Original: SiRNN, whose state is updated with a single CTRNN sample, similar to vanilla RNNs, and, iRNN, with many intermediate samples.
Revised: SiRNN, whose state is updated with a single CTRNN sample, similar to vanilla RNNs, and, iRNN, with many intermediate samples.

Original: SiRNN is well-suited for slowly varying inputs.
Revised: SiRNN is well-suited for slowly varying inputs.

Original: Contributions.
Revised: Contributions.

Original: To summarize, we list our main contributions:
Revised: To summarize, we list our main contributions:

Original: (A) iRNN converges to equilibrium for typical activation functions.
Revised: (A) iRNN converges to equilibrium for typical activation functions.

Original: The partial gradients of hiddenstate vectors for iRNNs converge to identity, thus solving vanishing/exploding gradient problem!
Revised: The partial gradients of hiddenstate vectors for iRNNs converge to identity, thus solving vanishing/exploding gradient problem!

Original: (B) iRNN converges rapidly, at an exponential rate in the number of discrete samplings of Eq.
Revised: (B) iRNN converges rapidly, at an exponential rate in the number of discrete samplings of Eq.

Original: SiRNN, the single-step iRNN, is efﬁcient and can be leveraged for slowly varying input sequences.
Revised: SiRNN, the single-step iRNN, is efﬁcient and can be leveraged for slowly varying input sequences.

Original: It exhibits fast training time, has fewer parameters and better accuracy relative to standard LSTMs.
Revised: It exhibits fast training time, has fewer parameters and better accuracy relative to standard LSTMs.

Original: (C) Extensive experiments on LTD datasets show that we improve upon standard LSTM accuracy as well as other recent proposals that are based on designing transition matrices and/or skip connections.
Revised: (C) Extensive experiments on LTD datasets show that we improve upon standard LSTM accuracy as well as other recent proposals that are based on designing transition matrices and/or skip connections.

Original: iRNNs/SiRNNs are robust to time-series distortions such as noise paddings (D) While our method extends directly (see Appendix A.1) to Deep RNNs, we deem these extensions complementary, and focus on single-layer to highlight our incremental perspective.
Revised: iRNNs/SiRNNs are robust to time-series distortions such as noise paddings (D) While our method extends directly (see Appendix A.1) to Deep RNNs, we deem these extensions complementary, and focus on single-layer to highlight our incremental perspective.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Gated Architectures.
Revised: Gated Architectures.

Original: Long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) is widely used in RNNs to model long-term dependency in sequential data.
Revised: Long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) is widely used in RNNs to model long-term dependency in sequential data.

Original: Gated recurrent unit (GRU) (Cho et al., 2014) is another gating mechanism that has been demonstrated to achieve similar performance of LSTM with fewer parameters.
Revised: Gated recurrent unit (GRU) (Cho et al., 2014) is another gating mechanism that has been demonstrated to achieve similar performance of LSTM with fewer parameters.

Original: Some recent gated RNNs include UGRNN (Collins et al., 2016), and FastGRNN (Kusupati et al., 2018).
Revised: Some recent gated RNNs include UGRNN (Collins et al., 2016), and FastGRNN (Kusupati et al., 2018).

Original: While mitigating vanishing/exploding gradients, they do not eliminate it.
Revised: While mitigating vanishing/exploding gradients, they do not eliminate it.

Original: Often, these models incur increased inference, training costs, and model size.
Revised: Often, these models incur increased inference, training costs, and model size.

Original: Unitary RNNs.
Revised: Unitary RNNs.

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: ; Jing et al.
Revised: ; Jing et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: ; Mhammedi et al.
Revised: ; Mhammedi et al.

Original: (2016) focus on designing well-conditioned state transition matrices, attempting to enforce unitary-property, during training.
Revised: (2016) focus on designing well-conditioned state transition matrices, attempting to enforce unitary-property, during training.

Original: Unitary property does not generally circumvent vanishing gradient (Pennington et al.
Revised: Unitary property does not generally circumvent vanishing gradient (Pennington et al.

Original: Also, it limits expressive power and prediction accuracy while also increasing training time.
Revised: Also, it limits expressive power and prediction accuracy while also increasing training time.

Original: Deep RNNs.
Revised: Deep RNNs.

Original: These are nonlinear transition functions incorporated into RNNs for performance improvement.
Revised: These are nonlinear transition functions incorporated into RNNs for performance improvement.

Original: For instance, Pascanu et al.
Revised: For instance, Pascanu et al.

Original: (2013a) empirically analyzed the problem of how to construct deep RNNs.
Revised: (2013a) empirically analyzed the problem of how to construct deep RNNs.

Original: Zilly et al.
Revised: Zilly et al.

Original: (2017) proposed extending the LSTM architecture to allow stepto-step transition depths larger than one.
Revised: (2017) proposed extending the LSTM architecture to allow step-tostep transition depths larger than one.

Original: Mujika et al.
Revised: Mujika et al.

Original: (2017) proposed incorporating the strengths of both multiscale RNNs and deep transition RNNs to learn complex transition functions from one timestep to the next.
Revised: (2017) proposed incorporating the strengths of both multiscale RNNs and deep transition RNNs to learn complex transition functions.

Original: While Deep RNNs offer richer representations relative to single-layers, it is complementary to iRNNs.
Revised: While Deep RNNs offer richer representations relative to single-layers, it is complementary to iRNNs.

Original: Residual/Skip Connections.
Revised: Residual/Skip Connections.

Original: Jaeger et al.
Revised: Jaeger et al.

Original: ; Bengio et al.
Revised: ; Bengio et al.

Original: ; Chang et al.
Revised: ; Chang et al.

Original: ; Campos et al.
Revised: ; Campos et al.

Original: ; Kusupati et al.
Revised: ; Kusupati et al.

Original: (2018) feed-forward state vectors to induce skip or residual connections, to serve as a middle ground between feed-forward and recurrent models, and to mitigate gradient decay.
Revised: (2018) feed-forward state vectors to induce skip or residual connections, to serve as a middle ground between feed-forward and recurrent models, and to mitigate gradient decay.

Original: Nevertheless, these connections, cannot entirely eliminate gradient explosion/decay.
Revised: Nevertheless, these connections, cannot entirely eliminate gradient explosion/decay.

Original: For instance, Kusupati et al.
Revised: For instance, Kusupati et al.

Original: (2018) suggest h m =
Revised: (2018) suggest h m =

Original: α m h m − 1 + β m φ ( Uh m − 1 + Wx m + b ) , and learn parameters so that α m ≈ 

Revised: α m h m − 1 + β m φ ( Uh m − 1 + Wx m + b ) , and learn parameters so that α m ≈

Original: and β m ≈ 0 .
Revised: and β m ≈ 0 .

Original: Evidently, this setting can lead to identity gradient, observe that setting β m ≈ 0 , implies little contribution from the inputs and can conﬂict with good accuracy, as also observed in our experiments.
Revised: Evidently, this setting can lead to identity gradient, observe that setting β m ≈ 0 , implies little contribution from the inputs and can conﬂict with good accuracy, as also observed in our experiments.

Original: Linear RNNs.
Revised: Linear RNNs.

Original: (Bradbury et al., 2016; Lei et al., 2018; Balduzzi & Ghifary, 2016) have focused on speeding up recurrent neural networks by replacing recurrent connections, such as hidden-to-hidden interactions, with light weight linear components.
Revised: (Bradbury et al., 2016; Lei et al., 2018; Balduzzi & Ghifary, 2016) focus on speeding up RNNs by replacing recurrent connections, such as hidden-to-hidden interactions, with light weight linear components.

Original: While this has led to reduced training time, it has resulted in signiﬁcantly increasing model size.
Revised: This reduces training time, but results in signiﬁcantly increased model size.

Original: For example, Lei et al.
Revised: For example, Lei et al.

Original: (2018)typically requires twice the number of cells for LSTM level performance.
Revised: (2018) requires twice the number of cells for LSTM level performance.

Original: ODE/Dynamical Perspective.
Revised: ODE/Dynamical Perspective.

Original: There are a few works that are inspired by ODEs, and attempt to address stability, but do not end up eliminating vanishing/exploding gradients.
Revised: Few ODE inspired architectures attempt to address stability, but do not end up eliminating vanishing/exploding gradients.

Original: Talathi & Vartak (2015) proposed a modiﬁed weight initialization strategy based on a dynamical system perspective on weight initialization processthat leads to successfully training RNNs composed of ReLUs.
Revised: Talathi & Vartak (2015) proposed a modiﬁed weight initialization strategy based on a dynamical system perspective on weight initialization process to successfully train RNNs composed of ReLUs.

Original: Niu et al.
Revised: Niu et al.

Original: (2019) analyzed RNN architectures using numerical methods of ODE and propose a family of ODE-RNNs.
Revised: (2019) analyzed RNN architectures using numerical methods of ODE and propose a family of ODE-RNNs.

Original: Chang et al.
Revised: Chang et al.

Original: (2019), propose Antisymmetric-RNN.
Revised: (2019), propose Antisymmetric-RNN.

Original: Their key idea is to express the transition matrix in Eq.
Revised: Their key idea is to express the transition matrix in Eq.

Original: 1, for the special case α = 0 , τ = 1 , as a difference: U =
Revised: 1, for the special case α = 0 , τ = 1 , as a difference: U =

Original: V − V T and note that the eigenspectrum is imaginary.
Revised: V − V T and note that the eigenspectrum is imaginary.

Original: Nevertheless, Euler discretization, in this context leads to instability, necessitating damping of the system.
Revised: Nevertheless, Euler discretization, in this context leads to instability, necessitating damping of the system.

Original: As such vanishing gradient cannot be completely eliminated.
Revised: As such vanishing gradient cannot be completely eliminated.

Original: Its behavior is analogous to FastRNN Kusupati et al.
Revised: Its behavior is analogous to FastRNN Kusupati et al.

Original: (2018), in that, identity gradient conﬂicts with high accuracy.
Revised: (2018), in that, identity gradient conﬂicts with high accuracy.

Original: In summary, we are the ﬁrst to propose evolution over the equilibrium manifold, and demonstrating identity gradients.
Revised: In summary, we are the ﬁrst to propose evolution over the equilibrium manifold, and demonstrating identity gradients.

Original: Neural ODEs (Chen et al., 2018; Rubanova et al., 2019) have also been proposed for time-series prediction to deal with irregularly sampled inputs.
Revised: Neural ODEs (Chen et al., 2018; Rubanova et al., 2019) have also been proposed for time-series prediction to deal with irregularly sampled inputs.

Original: To do this they parameterize the derivative of the hidden-state in terms of an autonomous differential equation and let the ODE evolve in continuous time until the next input arrives.
Revised: They parameterize the derivative of the hidden-state in terms of an autonomous differential equation and let the ODE evolve in continuous time until the next input arrives.

Original: As such, this is not our goal, our ODE explicitly depends on the input, and evolves until equilibrium for that input is reached.
Revised: As such, this is not our goal, our ODE explicitly depends on the input, and evolves until equilibrium for that input is reached.

Original: We introduce incremental updates to bypass vanishing/exploding gradient issues, which is not of speciﬁc concern for these works.
Revised: We introduce incremental updates to bypass vanishing/exploding gradient issues, which is not of speciﬁc concern for these works.

Original:  M ETHOD
Revised:  M ETHOD

Original: We use Euler’s method to discretize Eq.
Revised: We use Euler’s method to discretize Eq.

Original: 3 in steps δ = ητ .
Revised: 3 in steps δ = ητ .

Original: Denoting the kth step as g k = g ( kδ )
Revised: Denoting the kth step as g k = g ( kδ )

Original: [Equation]
Revised: [Equation]

Original: Rearranging terms we get a compact form for iRNN (see Fig. 1).
Revised: Rearranging terms we get a compact form for iRNN (see Fig. 1).

Original: In addition we introduce a learnable parameter η km and let it be a function of time m and the recursion-step k .
Revised: In addition we introduce a learnable parameter η km and let it be a function of time m and the recursion-step k .

Original: [Equation]
Revised: [Equation]

Original: We run the recursion for k ∈ [ K ] with some suitable initial condition.
Revised: We run the recursion for k ∈ [ K ] with some suitable initial condition.

Original: This could be g 

Revised: This could be g

Original: = 0 or initialized to the previous state, i.e., g 

Revised: = 0 or initialized to the previous state, i.e., g

Original: = h m − 1 at time m .
Revised: = h m − 1 at time m .

Original:  In many of our examples, we ﬁnd the input sequence is slowly varying, and K = 1 can also realize good empirical performance.
Revised: [Equation] g [Equation] In many of our examples, we ﬁnd the input sequence is slowly varying, and K = 1 can also realize good empirical performance.

Original: We refer to this as single-step-incremental-RNN (SiRNN).
Revised: We refer to this as single-step-incremental-RNN (SiRNN).

Original: [Equation]
Revised: [Equation]

Original: For both iRNN and SiRNN we drop the superscript whenever it is clear from the context.[Equation] unfold [Equation]
Revised: For both iRNN and SiRNN we drop the superscript whenever it is clear from the context.

Original: g K

Original: Root Finding and Transitions.
Revised: Root Finding and Transitions.

Original: The two indices k and m should not be confused.
Revised: The two indices k and m should not be confused.

Original: The index m ∈ [ T ] refers to the time index, and indexes input, x m and hidden state h m over time horizon T .
Revised: The index m ∈ [ T ] refers to the time index, and indexes input, x m and hidden state h m over time horizon T .

Original: The index k ∈
Revised: The index k ∈

Original: [ K ] is a ﬁxed-point recursion for converging to the equilibrium solution at each time m , given input x m and the hidden state h m − 1 .
Revised: [ K ] is a ﬁxed-point recursion for converging to the equilibrium solution at each time m , given input x m and the hidden state h m − 1 .

Original: We iterate over k so that at k = K , g K satisﬁes,
Revised: We iterate over k so that at k = K , g K satisﬁes,

Original: [Equation]
Revised: [Equation]

Original: The recursion (Eq. 5) at time m runs for K rounds, terminates, and recursion is reset for the new input, x m +1 .
Revised: The recursion (Eq. 5) at time m runs for K rounds, terminates, and recursion is reset for the new input, x m +1 .

Original: Indeed, Eq. 5 is a standard root-ﬁnding recursion, with g k − 1 serving as the previous solution, plus a correction term, which is the error, φ ( U ( g k − 

Revised: Indeed, Eq. 5 is a standard root-ﬁnding recursion, with g k − 1 serving as the previous solution, plus a correction term, which is the error, φ ( U ( g k −

Original: + h m − 1 ) + Wx m + b ) − α
Revised: + h m − 1 ) + Wx m + b ) − α

Original: ( g k − 1 + h m − 1 ) .
Revised: ( g k − 1 + h m − 1 ) .

Original: If the sequence converges, the resulting solution is the equilibrium point.
Revised: If the sequence converges, the resulting solution is the equilibrium point.

Original: Proposition 2 guarantees a geometric rate of convergence.
Revised: Proposition 2 guarantees a geometric rate of convergence.

Original: Identity Gradient.
Revised: Identity Gradient.

Original: We will informally (see Theorem 1) show here that partial gradients are identity.
Revised: We will informally (see Theorem 1) show here that partial gradients are identity.

Original: Say we have for sufﬁciently large K , h m =
Revised: Say we have for sufﬁciently large K , h m =

Original: g K is the equilibrium solution.
Revised: g K is the equilibrium solution.

Original: It follows that,
Revised: It follows that,

Original: [Equation]
Revised: [Equation]

Original: Taking derivatives, we have,
Revised: Taking derivatives, we have,

Original: [Equation]
Revised: [Equation]

Original: Thus if the matrix ( ∇ φ ( · ) U − αI ) is not singular, it follows that ( ∂h m ∂h m − 1 + I ) = 0 .
Revised: Thus if the matrix ( ∇ φ ( · ) U − αI ) is not singular, it follows that ( ∂h m ∂h m − 1 + I ) = 0 .

Original: SiRNN vs. iRNN.
Revised: SiRNN vs. iRNN.

Original: SiRNN approximates iRNN.
Revised: SiRNN approximates iRNN.

Original: In particular, say x m is a constant in the segment, m ∈ [ m 0 , m 0 + K ] , then SiRNN trajectory of hidden states, denoted as h 1 m 

Revised: In particular, say x m is a constant in the segment, m ∈ [ m 0 , m 0 + K ] , then SiRNN trajectory of hidden states, denoted as h 1 m

Original: + K is equal to the iRNN hidden state h Km 0 , when both SiRNN and iRNN are initialized with g 

Revised: + K is equal to the iRNN hidden state h Km 0 , when both SiRNN and iRNN are initialized with g

Original: = h m − 1 .
Revised: = h m − 1 .

Original: Thus, for slowly time-varying inputs we can expect SiRNN to closely approximate iRNN.
Revised: Thus, for slowly time-varying inputs we can expect SiRNN to closely approximate iRNN.

Original: Residual Connections vs. iRNN/SiRNN.
Revised: Residual Connections vs. iRNN/SiRNN.

Original: As such, our architecture is a special case of skip/residual connections.
Revised: As such, our architecture is a special case of skip/residual connections.

Original: Nevertheless, unlike skip connections, our connections are structured , and the dynamics driven by the error term ensures that the hidden state is associated with equilibrium and leads to identity gradient.
Revised: Nevertheless, unlike skip connections, our connections are structured , and the dynamics driven by the error term ensures that the hidden state is associated with equilibrium and leads to identity gradient.

Original: No such guarantees are possible with unstructured skip connections.
Revised: No such guarantees are possible with unstructured skip connections.

Original: Note that for slowly varying inputs, after a certain transition-time period, we should expect SiRNN to be close to equilibrium as well.
Revised: Note that for slowly varying inputs, after a certain transition-time period, we should expect SiRNN to be close to equilibrium as well.

Original: Without this imposed structure, general residual architectures can learn patterns that can be dramatically different (see Fig. 2).
Revised: Without this imposed structure, general residual architectures can learn patterns that can be dramatically different (see Fig. 2).

Original: 3. I DENTITY G RADIENT P ROPERTY AND C ONVERGENCE G UARANTEES .
Revised: 3. I DENTITY G RADIENT P ROPERTY AND C ONVERGENCE G UARANTEES .

Original: Let us now collect a few properties of Eq.
Revised: Let us now collect a few properties of Eq.

Original: 3 and Eq.
Revised: 3 and Eq.

Original: First, denote the equilibrium solutions for an arbirary input x ∈ R d , arbitrary state-vector ν ∈ R D , in an arbitrary round:
Revised: First, denote the equilibrium solutions for an arbirary input x ∈ R d , arbitrary state-vector ν ∈ R D , in an arbitrary round:

Original: [Equation]
Revised: [Equation]

Original: Whenever the equilibrium set is a singleton, we denote it as a function h eq ( x, ν ) .
Revised: Whenever the equilibrium set is a singleton, we denote it as a function h eq ( x, ν ) .

Original: For simplicity, we assume below that η
Revised: For simplicity, we assume below that η

Original: ik is a positive constant independent of k and i .
Revised: ik is a positive constant independent of k and i .

Original: Proposition 1.
Revised: Proposition 1.

Original: Suppose, φ ( · ) is a 1-Lipshitz function in the norm induced by (cid:107) · (cid:107) , and (cid:107) U (cid:107) <
Revised: Suppose, φ ( · ) is a 1-Lipshitz function in the norm induced by (cid:107) · (cid:107) , and (cid:107) U (cid:107) <

Original: α , then for any x m ∈ R d and h m − 1 ∈ R D , it follows that M eq ( x, ν ) is a singleton and as K → ∞ , the iRNN recursions converge to this solution, namely, [Equation]
Revised: α , then for any x m ∈ R d and h m − 1 ∈ R D , it follows that M eq ( x, ν ) is a singleton and as K → ∞ , the iRNN recursions converge to this solution, namely, h m = lim K →∞ g K = h eq ( x m , h m − 1 )

Original: Proof.
Revised: Proof.

Original: Deﬁne T : R D
Revised: Deﬁne T : R D

Original: → R d , with T ( g ) =
Revised: → R d , with T ( g ) =

Original: (1 − ηα ) g + η ( φ ( U ( g + h
Revised: (1 − ηα ) g + η ( φ ( U ( g + h

Original: m − 1 )
Revised: m − 1 )

Original: + Wx m + b ) − h m − 1 ) .
Revised: + Wx m + b ) − h m − 1 ) .

Original: It follows that T ( · ) is a contraction:
Revised: It follows that T ( · ) is a contraction:

Original: [Equation]
Revised: [Equation]

Original: We now invoke the Banach ﬁxed point theorem, which asserts that a contractive operator on a complete metric space converges to a unique ﬁxed point, namely, T K ( g ) → g ∗ .
Revised: We now invoke the Banach ﬁxed point theorem, which asserts that a contractive operator on a complete metric space converges to a unique ﬁxed point, namely, T K ( g ) → g ∗ .

Original: Upon substitution, we see that this point g ∗ must be such that, φ ( U ( g ∗ + h m − 1 ) + Wx m + b ) − ( g ∗ + h m − 1 ) = 0 .
Revised: Upon substitution, we see that this point g ∗ must be such that, φ ( U ( g ∗ + h m − 1 ) + Wx m + b ) − ( g ∗ + h m − 1 ) = 0 .

Original: Thus equilibrium point exists and is unique.
Revised: Thus equilibrium point exists and is unique.

Original: Result follows by setting h m (cid:44) h eq ( x m , h m − 1 ) .
Revised: Result follows by setting h m (cid:44) h eq ( x m , h m − 1 ) .

Original: Handling (cid:107) U (cid:107) ≤ α .
Revised: Handling (cid:107) U (cid:107) ≤ α .

Original: In experiments, we set α = 1 , and do not enforce (cid:107) U (cid:107) ≤ α constraint.
Revised: In experiments, we set α = 1 , and do not enforce (cid:107) U (cid:107) ≤ α constraint.

Original: Instead, we initialize U as a Gaussian matrix with IID mean zero, small variance components.
Revised: Instead, we initialize U as a Gaussian matrix with IID mean zero, small variance components.

Original: As such, the matrix norm is smaller than 1.
Revised: As such, the matrix norm is smaller than 1.

Original: Evidently, the resulting learnt U matrix does not violate this condition.
Revised: Evidently, the resulting learnt U matrix does not violate this condition.

Original: Next we show for η > 0 , iRNN converges at a linear rate, which follows directly from Proposition 1.
Revised: Next we show for η > 0 , iRNN converges at a linear rate, which follows directly from Proposition 1.

Original: Proposition 2.
Revised: Proposition 2.

Original: Under the setup in Proposition 1, it follows that,
Revised: Under the setup in Proposition 1, it follows that,

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: Remark.
Revised: Remark.

Original: Proposition 1 accounts for typical activation functions ReLU, tanh, sigmoids as well as deep RNNs (appendix A.1).
Revised: Proposition 1 accounts for typical activation functions ReLU, tanh, sigmoids as well as deep RNNs (appendix A.1).

Original: In passing we point out that, in our experiments, we learn parameters η km , and a result that accounts for this case is desirable.
Revised: In passing we point out that, in our experiments, we learn parameters η km , and a result that accounts for this case is desirable.

Original: We describe this case in Appendix A.3.
Revised: We describe this case in Appendix A.3.

Original: A fundamental result we describe below is that partials of hidden-state vectors, on the equilibrium surface is unity.
Revised: A fundamental result we describe below is that partials of hidden-state vectors, on the equilibrium surface is unity.

Original: For technical simplicity, we assume a continuously differentiable activation, which appears to exclude ReLU activations.
Revised: For technical simplicity, we assume a continuously differentiable activation, which appears to exclude ReLU activations.

Original: Nevertheless, we can overcome this issue, but requires more technical arguments.
Revised: Nevertheless, we can overcome this issue, but requires more technical arguments.

Original: The main difﬁculty stems from ensuring that derivatives along the equilibrium surface exist, and this can be realized by invoking the implicit function theorem (IFT).
Revised: The main difﬁculty stems from ensuring that derivatives along the equilibrium surface exist, and this can be realized by invoking the implicit function theorem (IFT).

Original: IFT requires continuous differentiability, which ReLUs violate.
Revised: IFT requires continuous differentiability, which ReLUs violate.

Original: Nevertheless, recent results 1 suggests that one can state implicit function theorem for everywhere differentiable functions, which includes ReLUs.
Revised: Nevertheless, recent results 1 suggests that one can state implicit function theorem for everywhere differentiable functions, which includes ReLUs.

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose φ ( · ) is a continuously differentiable, 1-Lipshitz function, with (cid:107) U (cid:107) < α .
Revised: Suppose φ ( · ) is a continuously differentiable, 1-Lipshitz function, with (cid:107) U (cid:107) < α .

Original: Then as K
Revised: Then as K

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: = − I .

Original: Furthermore, as K → ∞ the partial gradients over arbitrary number of rounds for iRNN is identity.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Deﬁne, ψ ( g, h m − 1 ) =
Revised: Deﬁne, ψ ( g, h m − 1 ) =

Original: φ ( U ( g + h m − 1 ) + Wx m + b ) − α
Revised: φ ( U ( g + h m − 1 ) + Wx m + b ) − α

Original: ( g + h m − 1 ) .
Revised: ( g + h m − 1 ) .

Original: We overload notation and view the equilibrium point as a function of h m − 1 , i.e., g ∗ ( h m − 1 )
Revised: We overload notation and view the equilibrium point as a function of h m − 1 , i.e., g ∗ ( h m − 1 )

Original: =
Revised: =

Original: h eq ( x m , h m − 1 ) .
Revised: h eq ( x m , h m − 1 ) .

Original: Invoking standard results 2 in ODE’s, it follows that g ∗ ( h m − 1 ) is a smooth function, so long as the Jacobian, ∇ g ψ ( g ∗ , h m − 1 ) with respect to the ﬁrst coordinate, g ∗ , is non-singular.
Revised: Invoking standard results 2 in ODE’s, it follows that g ∗ ( h m − 1 ) is a smooth function, so long as the Jacobian, ∇ g ψ ( g ∗ , h m − 1 ) with respect to the ﬁrst coordinate, g ∗ , is non-singular.

Original: Upon computation, we see that, ∇ g ψ ( g ∗ , h m − 1 ) =
Revised: Upon computation, we see that, ∇ g ψ ( g ∗ , h m − 1 ) =

Original: ∇ φ ( g ∗ , h m − 1 ) U − αI , is non-singular, since (cid:107)∇ φ ( g ∗ , h m − 1 ) U (cid:107) ≤ (cid:107) U (cid:107) .
Revised: ∇ φ ( g ∗ , h m − 1 ) U − αI , is non-singular, since (cid:107)∇ φ ( g ∗ , h m − 1 ) U (cid:107) ≤ (cid:107) U (cid:107) .

Original: It follows that we can take partials of the state-vectors.
Revised: It follows that we can take partials of the state-vectors.

Original: By taking the partial derivatives w.r.t.
Revised: By taking the partial derivatives w.r.t.

Original: h m −in Eq.
Revised: h m −in Eq.

Original: 5, at the equilibrium points we have [ ∇ φ ( g ∗ , h m − 1 )
Revised: 5, at the equilibrium points we have [ ∇ φ ( g ∗ , h m − 1 )

Original: U − α I ][ ∂g ∗ ∂h m − 

Revised: U − α I ][ ∂g ∗ ∂h m −

Original: + I ] = 

Revised: + I ] =

Original: (see Eq. 7).
Revised: (see Eq. 7).

Original: The rest of the proof follows by observing that the ﬁrst term is non-singular.
Revised: The rest of the proof follows by observing that the ﬁrst term is non-singular.

Revised: [Figure]

Original: Remark.
Revised: Remark.

Original: We notice that replacing h m − 1 with − h m − 1 in Eq.
Revised: We notice that replacing h m − 1 with − h m − 1 in Eq.

Original: 12 will lead to ∂heq
Revised: 12 will lead to ∂h

Revised: [Figure]

Original: ∂h m − 1 =
Revised: ∂h m − 1 =

Original: I , which also has no impact on magnitudes of gradients.
Revised: I , which also has no impact on magnitudes of gradients.

Original: As a result, both choices are suitable for circumventing vanishing or exploding gradients during training, but still may converge to different local minima and thus result in different test-time performance.
Revised: As a result, both choices are suitable for circumventing vanishing or exploding gradients during training, but still may converge to different local minima and thus result in different test-time performance.

Original: Furthermore, notice that the norm preserving property is somewhat insensitive to choices of α , so long as the non-singular condition is satisﬁed.[Figure]
Revised: Furthermore, notice that the norm preserving property is somewhat insensitive to choices of α , so long as the non-singular condition is satisﬁed.

Original: Approximate Identity. Theorem 1 guarantees identity gradients on the equilibrium surface. However, Proposition 2 only guarantees we are close to equilibrium surface for ﬁnite K . Consequently we may still have a vanishing gradient problem for sufﬁciently large T .
Revised: 3. I RNN D ESIGN I MPLICATIONS

Original: Nevertheless, this can be readily handled by suitable choices of K .

Original: This is because, while larger T degrades gradients, it is compensated by larger values of K .

Original: To ensure that end-to-end partials are no worse than 1 ± (cid:15) , we need K = O (log( T/(cid:15) )) , thanks to the geometric rate of convergence of partials, which can be established under additional conditions on transition function φ . As such log( T/(cid:15) ) is small for most problems we encountered.
Revised: [Figure]

Original: 3. I RNN D ESIGN I MPLICATIONS : L OW -R ANK M ODEL P ARAMETRIZATION

Original: Fig.
Revised: Fig.

Original: 2 depicts phase portrait and illustrates salient differences between RNN, FastRNN (RNN with skip connection), and iRNN (K=5).
Revised: 2 depicts phase portrait and illustrates salient differences between RNN, FastRNN (RNN with skip connection), and iRNN (K=5).

Original: RNN and FastRNN exhibit complex trajectories, while iRNN trajectory is smooth, projecting initial point (black circle) onto the equilibrium surface (blue) and moving within it (green).
Revised: RNN and FastRNN exhibit complex trajectories, while iRNN trajectory is smooth, projecting initial point (black circle) onto the equilibrium surface (blue) and moving within it (green).

Original: This suggests that iRNN trajectory belongs to a low-dimensional manifold.
Revised: This suggests that iRNN trajectory belongs to a low-dimensional manifold.

Original: Variation of Equilibrium w.r.t.
Revised: Variation of Equilibrium w.r.t.

Original: Input.
Revised: Input.

Original: As before, h eq be an equilibrium solution for some tuple ( h m − 1 , x m ) .
Revised: As before, h eq be an equilibrium solution for some tuple ( h m − 1 , x m ) .

Original: It follows that,
Revised: It follows that,

Original: [Equation]
Revised: [Equation]

Original: This suggests that, whenever the input undergoes a slow variation, we expect that the equilibrium point moves in such a way that U∂h eq must lie in a transformed span of W .
Revised: This suggests that, whenever the input undergoes a slow variation, we expect that the equilibrium point moves in such a way that U∂h eq must lie in a transformed span of W .

Original: Now W ∈ R D ×
Revised: Now W ∈ R D ×

Original: d with d (cid:28) D , which implies that ( α I − ∇ φ ( U ( h eq + h m − 1 ) + Wx m + b ) U is rank-deﬁcient.
Revised: d with d (cid:28) D , which implies that ( α I − ∇ φ ( U ( h eq + h m − 1 ) + Wx m + b ) U is rank-deﬁcient.

Original: Low Rank Matrix Parameterization.
Revised: Low Rank Matrix Parameterization.

Original: For typical activation functions, note that whenever the argument is in the unsaturated regime, ∇ φ ( · )
Revised: For typical activation functions, note that whenever the argument is in the unsaturated regime, ∇ φ ( · )

Original: ≈ I .
Revised: ≈ I .

Original: We then approximately get span( α I − U )
Revised: We then approximately get span( α I − U )

Original: ≈ span( W ) .
Revised: ≈ span( W ) .

Original: We can express these constraints as U =
Revised: We can express these constraints as U =

Original: α I + V H with low-rank matrices V ∈ R D ×
Revised: α I + V H with low-rank matrices V ∈ R D ×

Original: d 1 , H ∈ R d 1 × D , and further map both Uh m and Wx m onto a shared space.
Revised: d 1 , H ∈ R d 1 × D , and further map both Uh m and Wx m onto a shared space.

Original: Since in our experiments the signal vectors we encounter are low-dimensional, and sequential inputs vary slowly over time, we enforce this restriction in all our experiments.
Revised: Since in our experiments the signal vectors we encounter are low-dimensional, and sequential inputs vary slowly over time, we enforce this restriction in all our experiments.

Original: In particular, we consider,
Revised: In particular, we consider,

Original: [Equation]
Revised: [Equation]

Original: The parameter matrix P ∈ R D
Revised: The parameter matrix P ∈ R D

Original: × D maps the contributions from input and hidden states onto the same space.
Revised: × D maps the contributions from input and hidden states onto the same space.

Original: To decrease model-size we let P =
Revised: To decrease model-size we let P =

Original: U =
Revised: U =

Original: ( I + V H ) learn these parameters.
Revised: ( I + V H ) learn these parameters.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We organize this section as follows.
Revised: We organize this section as follows.

Original: First, the experimental setup, competing algorithms will be described.
Revised: First, the experimental setup, competing algorithms will be described.

Original: Then we present an ablative analysis to highlight salient aspects of iRNN and justify some of our experimental choices.
Revised: Then we present an ablative analysis to highlight salient aspects of iRNN and justify some of our experimental choices.

Original: We then plot and tabulate experimental results on benchmark datasets.
Revised: We then plot and tabulate experimental results on benchmark datasets.

Original: 4. E XPERIMENTAL S ETUP AND B ASELINES
Revised: 4. E XPERIMENTAL S ETUP AND B ASELINES

Original: Choice of Competing Methods: We choose competing methods based on the following criteria: (a) methods that are devoid of additional application or dataset-speciﬁc heuristics, (b) methods that leverage only single cell/block/layer, and (c) methods without the beneﬁt of complementary add-ons (such as gating, advanced regularization, model compression, etc.).
Revised: Choice of Competing Methods: We choose competing methods based on the following criteria: (a) methods that are devoid of additional application or dataset-speciﬁc heuristics, (b) methods that leverage only single cell/block/layer, and (c) methods without the beneﬁt of complementary add-ons (such as gating, advanced regularization, model compression, etc.).

Original: Requiring (a) is not controversial since our goal is methodological.
Revised: Requiring (a) is not controversial since our goal is methodological.

Original: Conditions (b),(c) are justiﬁable since we could also leverage these add-ons and are not germane to any particular method 3 .
Revised: Conditions (b),(c) are justiﬁable since we could also leverage these add-ons and are not germane to any particular method 3 .

Original: We benchmark iRNN against standard
Revised: We benchmark iRNN against standard

Original: [Figure] RNN, LSTM (Hochreiter & Schmidhuber, 1997), (ungated) AntisymmetricRNN (Chang et al., 2019), (ungated) FastRNN (Kusupati et al., 2018). 
Revised: RNN, LSTM (Hochreiter & Schmidhuber, 1997), (ungated) AntisymmetricRNN (Chang et al., 2019), (ungated) FastRNN (Kusupati et al., 2018). [Figure]

Original: Unitary RNN Variants.
Revised: Unitary RNN Variants.

Original: Results for methods based on unitary transitions (such as Arjovsky et al.
Revised: Results for methods based on unitary transitions (such as Arjovsky et al.

Original: ; Wisdom et al.
Revised: ; Wisdom et al.

Original: ; Vorontsov et al.
Revised: ; Vorontsov et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2018)) are not reported in the main paper (when available reported in appendix) for the following reasons: (a) They are substantially more expensive, and requiring large model sizes; (b) Apart from the benchmark copy and add tasks, results tabulated by FastRNN and Antisymmetric authors (see Zhang et al.
Revised: (2018)) are not reported in the main paper (when available reported in appendix) for the following reasons: (a) They are substantially more expensive, and requiring large model sizes; (b) Apart from the benchmark copy and add tasks, results tabulated by FastRNN and Antisymmetric authors (see Zhang et al.

Original: ; Chang et al.
Revised: ; Chang et al.

Original: (2019)) show that they are well below SOTA; (c) iRNN dominates unitary-RNN variants on add-task (see Sec. 4.3.1); (d) On copy task, while unitary invariants are superior, Vorontsov et al.
Revised: (2019)) show that they are well below SOTA; (c) iRNN dominates unitary-RNN variants on add-task (see Sec. 4.3.1); (d) On copy task, while unitary invariants are superior, Vorontsov et al.

Original: (2017) attributes it to modReLU or leaky ReLU activations.
Revised: (2017) attributes it to modReLU or leaky ReLU activations.

Original: Leaky ReLUs allow for linear transitions, and copy task being a memory task beneﬁts from it.
Revised: Leaky ReLUs allow for linear transitions, and copy task being a memory task beneﬁts from it.

Original: With hard non-linear activation, unitary RNN variants can take up to 1000’s of epochs for even 100-length sequences (Vorontsov et al.
Revised: With hard non-linear activation, unitary RNN variants can take up to 1000’s of epochs for even 100-length sequences (Vorontsov et al.

Original: Implementation.
Revised: Implementation.

Original: For all our experiments, we used the parametrized update formulation in Eq. 9 for iRNN .
Revised: For all our experiments, we used the parametrized update formulation in Eq. 9 for iRNN .

Original: We used tensorﬂow framework for our experiments.
Revised: We used tensorﬂow framework for our experiments.

Original: For most competing methods apart from
Revised: For most competing methods apart from

Original: AntisymmetricRNN, which we implemented, code is publicly available.
Revised: AntisymmetricRNN, which we implemented, code is publicly available.

Original: All the experiments were run on an Nvidia GTX 

Revised: All the experiments were run on an Nvidia GTX

Original: GPU with CUDA 9 and cuDNN 7.0 on a machine with Intel Xeon 2.
Revised: GPU with CUDA 9 and cuDNN 7.0 on a machine with Intel Xeon 2.

Original: GHz CPU with 20 cores.
Revised: GHz CPU with 20 cores.

Original: Datasets.
Revised: Datasets.

Original: Pre-processing and feature extraction details for all publicly available datasets are in the appendix A.4.
Revised: Pre-processing and feature extraction details for all publicly available datasets are in the appendix A.4.

Original: We replicate benchmark test/train split with 20% of training data for validation to tune hyperparameters.
Revised: We replicate benchmark test/train split with 20% of training data for validation to tune hyperparameters.

Original: Reported results are based on the full training set, and performance achieved on the publicly available test set.
Revised: Reported results are based on the full training set, and performance achieved on the publicly available test set.

Original: Table 4 (Appendix) and A.4 describes details for all the data sets.
Revised: Table 4 (Appendix) and A.4 describes details for all the data sets.

Original: Hyper Parameters We used grid search and ﬁne-grained validation wherever possible to set the hyper-parameters of each algorithm, or according to the settings published in (Kusupati et al., 2018;
Revised: Hyper Parameters We used grid search and ﬁne-grained validation wherever possible to set the hyper-parameters of each algorithm, or according to the settings published in (Kusupati et al., 2018;

Original: Arjovsky et al., 2016)
Revised: Arjovsky et al., 2016)

Original: ( e.g. number of hidden states).
Revised: ( e.g. number of hidden states).

Original: Both the learning rate and η ’s were initialized to 10 − 2 .
Revised: Both the learning rate and η ’s were initialized to 10 − 2 .

Original: The batch size of 128 seems to work well across all the data sets.
Revised: The batch size of 128 seems to work well across all the data sets.

Original: We used ReLU as the non-linearity and Adam (Kingma & Ba (2015)) as the optimizer for all the experiments.
Revised: We used ReLU as the non-linearity and Adam (Kingma & Ba (2015)) as the optimizer for all the experiments.

Original: 4. A BLATIVE A NALYSIS
Revised: 4. A BLATIVE A NALYSIS

Original: We perform ablative analysis on the benchmark add-task (Sec 4.3.1) for sequence length 200 for 1000 iterations and explore mean-squared error as a metric.
Revised: We perform ablative analysis on the benchmark add-task (Sec 4.3.1) for sequence length 200 for 1000 iterations and explore mean-squared error as a metric.

Original: Fig.
Revised: Fig.

Original: 3 depicts salient results.
Revised: 3 depicts salient results.

Original: (a) Identity Gradients & Accuracy: iRNN accuracy is correlated with identity gradients.
Revised: (a) Identity Gradients & Accuracy: iRNN accuracy is correlated with identity gradients.

Original: Increasing K improves gradients, and correlates with increased accuracy (Fig. 3).
Revised: Increasing K improves gradients, and correlates with increased accuracy (Fig. 3).

Original: While other models h t =
Revised: While other models h t =

Original: αh t − 1 + βφ (( U − γI ) h t − 1 + Wx t ) , can realize identity gradients for suitable choices; linear ( α = 1 , β = 1 , γ = 0 , U = 0 ), FastRNN ( α ≈ 1 , β ≈ 0 , γ = 0 ) and Antisymmetric ( α = 1 , β = 1 , U
Revised: αh t − 1 + βφ (( U − γI ) h t − 1 + Wx t ) , can realize identity gradients for suitable choices; linear ( α = 1 , β = 1 , γ = 0 , U = 0 ), FastRNN ( α ≈ 1 , β ≈ 0 , γ = 0 ) and Antisymmetric ( α = 1 , β = 1 , U

Original: =
Revised: =

Original: V − V T , (cid:107) U (cid:107) ≤ γ ), this goal may not be correlated with improved test accuracy.
Revised: V − V T , (cid:107) U (cid:107) ≤ γ ), this goal may not be correlated with improved test accuracy.

Original: FastRNN( η = 0 . 001 ), Antisymmetric ( γ = 0 .
Revised: FastRNN( η = 0 . 001 ), Antisymmetric ( γ = 0 .

Original: 01 , (cid:15) = 0 .
Revised: 01 , (cid:15) = 0 .

Original: 001 ) have good gradients but poorer test accuracy relative to FastRNN( η = 0 . 01 ), Antisymmetric( γ = 0 .
Revised: 001 ) have good gradients but poorer test accuracy relative to FastRNN( η = 0 . 01 ), Antisymmetric( γ = 0 .

Original: 01 , (cid:15) = 0 . 1 ), with poorer gradients.
Revised: 01 , (cid:15) = 0 . 1 ), with poorer gradients.

Original: (b) Identity gradient implies faster convergence: Identity gradient, whenever effective, must be capable of assigning credit to the informative parts, which in turn results in larger loss gradients, and signiﬁcantly faster convergence with number of iterations.
Revised: (b) Identity gradient implies faster convergence: Identity gradient, whenever effective, must be capable of assigning credit to the informative parts, which in turn results in larger loss gradients, and signiﬁcantly faster convergence with number of iterations.

Original: This is borne out in ﬁgure 3(a).
Revised: This is borne out in ﬁgure 3(a).

Original: iRNN for larger K is closer to identity gradient with fewer (unstable) spikes ( K = 1 , 5 , 10 ).
Revised: iRNN for larger K is closer to identity gradient with fewer (unstable) spikes ( K = 1 , 5 , 10 ).

Original: With K = 10 , 
Revised: With K = 10 , iRNN converges within 300 iterations while competing methods take about twice this time (other baselines not included here exhibited poorer performance than the once plotted).

Original: [Figure]
Revised: [Figure]

Original: iRNN converges within 300 iterations while competing methods take about twice this time (other baselines not included here exhibited poorer performance than the once plotted).

Original: (c) SiRNN (iRNN with K = 1 delivers good performane in some cases.
Revised: (c) SiRNN (iRNN with K = 1 delivers good performane in some cases.

Original: Fig. 3(a) illustrates that iRNN K
Revised: Fig. 3(a) illustrates that iRNN K

Original: = { 5 , 10 } achieves faster convergence than SiRNN, but the computational overhead per iteration roughly doubles or triples in comparison.
Revised: = { 5 , 10 } achieves faster convergence than SiRNN, but the computational overhead per iteration roughly doubles or triples in comparison.

Original: SiRNN is faster relative to competitors.
Revised: SiRNN is faster relative to competitors.

Original: For this reason, we sometimes tabulate only SiRNN, whenever it is SOTA in benchmark experiments, since accuracy improves with K but requires higher overhead.
Revised: For this reason, we sometimes tabulate only SiRNN, whenever it is SOTA in benchmark experiments, since accuracy improves with K but requires higher overhead.

Original: 4. L ONG - TERM D EPENDENCY AND O THER T ASKS
Revised: 4. L ONG - TERM D EPENDENCY AND O THER T ASKS

Original: We list ﬁve types of datasets, all of which in some way require effective gradient propagation: (1)
Revised: We list ﬁve types of datasets, all of which in some way require effective gradient propagation: (1)

Original: Conventional Benchmark LTD tasks (Add & Copy tasks) that illustrate that iRNN can rapidly learn long-term dependence; (2) Benchmark vision tasks (pixel MNIST, perm-MNIST) that may not require long-term, but nevertheless, demonstrates that iRNN achieves SOTA for short term dependencies but with less resources.
Revised: Conventional Benchmark LTD tasks (Add & Copy tasks) that illustrate that iRNN can rapidly learn long-term dependence; (2) Benchmark vision tasks (pixel MNIST, perm-MNIST) that may not require long-term, but nevertheless, demonstrates that iRNN achieves SOTA for short term dependencies but with less resources.

Original: (3) Noise Padded (LTD) Vision tasks (Noisy MNIST, Noisy CIFAR), where a large noise time segment separates information segments and the terminal state, and so the learner must extract information parts while rejecting the noisy parts; (4) short duration activity embedded in a larger time-window (HAR-2, Google-30 in Appendix Table 4 and many others A.7), that usually arise in the context of smart IoT applications and require a small model-size footprint.
Revised: (3) Noise Padded (LTD) Vision tasks (Noisy MNIST, Noisy CIFAR), where a large noise time segment separates information segments and the terminal state, and so the learner must extract information parts while rejecting the noisy parts; (4) short duration activity embedded in a larger time-window (HAR-2, Google-30 in Appendix Table 4 and many others A.7), that usually arise in the context of smart IoT applications and require a small model-size footprint.

Original: Chang et al.
Revised: Chang et al.

Original: (2019) further justify (3) and (4) as LTD, because for these datasets where only a smaller unknown segment(s) of a longer sequence is informative.
Revised: (2019) further justify (3) and (4) as LTD, because for these datasets where only a smaller unknown segment(s) of a longer sequence is informative.

Original: (5) Sequence-sequence prediction tasks (PTB language modeling) that are different from terminal prediction (reported in appendix A.7).
Revised: (5) Sequence-sequence prediction tasks (PTB language modeling) that are different from terminal prediction (reported in appendix A.7).

Original: 4.3. S TANDARD B ENCHMARK LTD T ASKS : A DDITION & C OPY M EMORY
Revised: 4.3. S TANDARD B ENCHMARK LTD T ASKS : A DDITION & C OPY M EMORY

Original: Addition and Copy tasks (Hochreiter & Schmidhuber, 1997) have long been used as benchmarks in the literature to evaluate LTD (Hori et al., 2017; Zhang et al., 2018;
Revised: Addition and Copy tasks (Hochreiter & Schmidhuber, 1997) have long been used as benchmarks in the literature to evaluate LTD (Hori et al., 2017; Zhang et al., 2018;

Original: Arjovsky et al., 2016; Martens &
Revised: Arjovsky et al., 2016; Martens &

Original: Sutskever, 2011).
Revised: Sutskever, 2011).

Original: We follow the setup described in Arjovsky et al.
Revised: We follow the setup described in Arjovsky et al.

Original: (2016) to create the adding and copying tasks.
Revised: (2016) to create the adding and copying tasks.

Original: See appendix A.4 for detailed description.
Revised: See appendix A.4 for detailed description.

Original: For both tasks we run iRNN with K = 5 .
Revised: For both tasks we run iRNN with K = 5 .

Original: Figure 4 show the average performance of various methods on these tasks.
Revised: Figure 4 show the average performance of various methods on these tasks.

Original: For the copying task we observe that iRNN converges rapidly to the naive baseline and is the only method to achieve zero average cross entropy.
Revised: For the copying task we observe that iRNN converges rapidly to the naive baseline and is the only method to achieve zero average cross entropy.

Original: For the addition task, both FastRNN and iRNN solves the addition task but FastRNN takes twice the number of iterations to reach desired 0 MSE.
Revised: For the addition task, both FastRNN and iRNN solves the addition task but FastRNN takes twice the number of iterations to reach desired 0 MSE.

Original: In both the tasks, iRNN performance is much more stable across number of online training samples.
Revised: In both the tasks, iRNN performance is much more stable across number of online training samples.

Original: In contrast, other methods either takes a lot of samples to match iRNN ’s performance or depict high variance in the evaluation metric.
Revised: In contrast, other methods either takes a lot of samples to match iRNN ’s performance or depict high variance in the evaluation metric.

Original: This shows that iRNN converges faster than the baselines (to the desired error).
Revised: This shows that iRNN converges faster than the baselines (to the desired error).

Original: These results demonstrate that iRNN easily and quickly learns the long term dependencies .
Revised: These results demonstrate that iRNN easily and quickly learns the long term dependencies .

Original: We omitted reporting unitary RNN variants for Add and Copy task.
Revised: We omitted reporting unitary RNN variants for Add and Copy task.

Original: See Sec.
Revised: See Sec.

Original: 4.1 for copy task.
Revised: 4.1 for copy task.

Original: On
Revised: On

Original: Add-task we point out that our performance is superior.
Revised: Add-task we point out that our performance is superior.

Original: In particular, for the longer T = 750 length,
Revised: In particular, for the longer T = 750 length,

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: (2016), points out that MSE does not reach zero, and uRNN is noisy.
Revised: (2016), points out that MSE does not reach zero, and uRNN is noisy.

Original: Others either (Wisdom et al., 2016) do not report add-task or report only for shorter lengths (Zhang et al., 2018).[Table] =
Revised: Others either (Wisdom et al., 2016) do not report add-task or report only for shorter lengths (Zhang et al., 2018).

Original: [Table]
Revised: [Table]

Original: 4.3. N ON LTD V ISION T ASKS : P IXEL 
Revised: 4.3. N ON LTD V ISION T ASKS : P IXEL MNIST, P ERMUTE MNIST

Original: MNIST , P ERMUTE MNIST

Original: Next, we perform experiments on the sequential vision tasks: (a) classiﬁcation of MNIST images on a pixel-by-pixel sequence; (b) a ﬁxed random permuted MNIST sequence (Lecun et al., 1998).
Revised: Next, we perform experiments on the sequential vision tasks: (a) classiﬁcation of MNIST images on a pixel-by-pixel sequence; (b) a ﬁxed random permuted MNIST sequence (Lecun et al., 1998).

Original: These tasks typically do not fall in the LTD categories (Chang et al., 2019), but are useful to demonstrate faster training, which can be attributed to better gradients.
Revised: These tasks typically do not fall in the LTD categories (Chang et al., 2019), but are useful to demonstrate faster training, which can be attributed to better gradients.

Original: For the pixel-MNIST task, Kusupati et al.
Revised: For the pixel-MNIST task, Kusupati et al.

Original: (2018) reports that it takes signiﬁcantly longer time for existing (LSTMs, Unitary, Gated, Spectral) RNNs to converge to reasonable performance.
Revised: (2018) reports that it takes signiﬁcantly longer time for existing (LSTMs, Unitary, Gated, Spectral) RNNs to converge to reasonable performance.

Original: In contrast,
Revised: In contrast,

Original: FastRNN trains at least 2 x faster than LSTMs.
Revised: FastRNN trains at least 2 x faster than LSTMs.

Original: Our results (table 1) for iRNN shows a 9 x speedup relative LSTMs, and 2 x speedup in comparison to Antisymmetric.
Revised: Our results (table 1) for iRNN shows a 9 x speedup relative LSTMs, and 2 x speedup in comparison to Antisymmetric.

Original: In terms of test accuracy, iRNN matches the performance of Antisymmetric, but with at least 3 x fewer parameters.
Revised: In terms of test accuracy, iRNN matches the performance of Antisymmetric, but with at least 3 x fewer parameters.

Original: We did not gain much with increased K values 5 .
Revised: We did not gain much with increased K values 5 .

Original: For the permuted version of this task, we seem to outperform the existing baselines 6 .
Revised: For the permuted version of this task, we seem to outperform the existing baselines 6 .

Original: In both tasks, iRNN trained at least 2 x faster than the strongest baselines.
Revised: In both tasks, iRNN trained at least 2 x faster than the strongest baselines.

Original: These results demonstrate that iRNN converges much faster than the baselines with fewer parameters.
Revised: These results demonstrate that iRNN converges much faster than the baselines with fewer parameters.

Original: 4.3. N OISE PADDING T ASKS : N OISY -MNIST, N OISY -CIFAR
Revised: 4.3. N OISE PADDING T ASKS : N OISY -MNIST, N OISY -CIFAR

Original: Additionally, as in Chang et al.
Revised: Additionally, as in Chang et al.

Original: (2019), we induce LTD by padding CIFAR- 10 with noise exactly replicating their setup, resulting in Noisy-CIFAR.
Revised: (2019), we induce LTD by padding CIFAR- 10 with noise exactly replicating their setup, resulting in Noisy-CIFAR.

Original: We extend this setting to MNIST dataset resulting in Noisy-MNIST.
Revised: We extend this setting to MNIST dataset resulting in Noisy-MNIST.

Original: Intuitively we expect our model to be resilient to such perturbations.
Revised: Intuitively we expect our model to be resilient to such perturbations.

Original: We attribute iRNN’s superior performance to the fact that it is capable of suppressing noise.
Revised: We attribute iRNN’s superior performance to the fact that it is capable of suppressing noise.

Original: For example, say noise is padded at t > τ and this results in Wx t being zero on average.
Revised: For example, say noise is padded at t > τ and this results in Wx t being zero on average.

Original: For iRNN the resulting states ceases to be updated.
Revised: For iRNN the resulting states ceases to be updated.

Original: So iRNN recalls last informative state
Revised: So iRNN recalls last informative state

Original: h τ (modulo const) unlike RNNs/variants!
Revised: h τ (modulo const) unlike RNNs/variants!

Original: Thus information from signal component is possibly better preserved.
Revised: Thus information from signal component is possibly better preserved.

Original: iterations in our experiments to demonstrate the effectiveness of our method.

Original: For some existing comparisons LSTM have achieved roughly 98 .

Original: 9 with dataset speciﬁc heuristics (Cooij- mans et al., 2016), but we could not achieve this performance in our comparison (and so have many others like (Kusupati et al., 2018; Zhang et al., 2018; Arjovsky et al., 2016)).

Original: Note that there’s no the literature.

Original: This may be main reason we could not replicate et

Original: [Table] Results for Noisy-MNIST and Noisy-CIFAR are shown in Table 2.
Revised: Results for Noisy-MNIST and Noisy-CIFAR are shown in Table 2.

Original: Note that almost all timesteps contain noise in these datasets.
Revised: Note that almost all timesteps contain noise in these datasets.

Original: LSTMs perform poorly on these tasks due to vanishing gradients.
Revised: LSTMs perform poorly on these tasks due to vanishing gradients.

Original: This  is consistent with the earlier observations (Chang et al., 2019).
Revised: This [Table] is consistent with the earlier observations (Chang et al., 2019).

Original: iRNN outperforms the baselines very comprehensively on CIFAR-10, while on MNIST the gains are smaller, as it’s a relatively easier task.
Revised: iRNN outperforms the baselines very comprehensively on CIFAR- [Table] while on MNIST the gains are smaller, as it’s a relatively easier task.

Original: These results show that iRNN is more resilient to noise and can account for longer dependencies.
Revised: These results show that iRNN is more resilient to noise and can account for longer dependencies.

Original: [Table]
Revised: [Table]

Original: 4.3. S HORT D URATION E MBEDDED A CTIVITY R ECOGNITION T ASKS : HAR-2, G OOGLE - 
Revised: 4.3. S HORT D URATION E MBEDDED A CTIVITY R ECOGNITION T ASKS : HAR-2, G OOGLE -

Original: We are interested in detecting activity embedded in a longer sequence with small footprint RNNs (Kusupati et al.
Revised: We are interested in detecting activity embedded in a longer sequence with small footprint RNNs (Kusupati et al.

Original: ): (a) Google-30 (Warden, 2018), i.e. detection of utterances of 30 commands plus background noise and silence, and (b) HAR-2 (Anguita et al., 2012), i.e. Human Activity
Revised: ): (a) Google-30 (Warden, 2018), i.e. detection of utterances of 30 commands plus background noise and silence, and (b) HAR-2 (Anguita et al., 2012), i.e. Human Activity

Original: Recognition from an accelerometer and gyroscope on a Samsung Galaxy S3 smartphone.
Revised: Recognition from an accelerometer and gyroscope on a Samsung Galaxy S3 smartphone.

Original: Table 3 shows accuracy, training time, number of parameters and prediction time.
Revised: Table 3 shows accuracy, training time, number of parameters and prediction time.

Original: Even with
Revised: Even with

Original: K = 1 , we compare well against competing methods, and iRNN accuracy improves with larger K .
Revised: K = 1 , we compare well against competing methods, and iRNN accuracy improves with larger K .

Original: Interestingly, higher K yields faster training as well as moderate prediction time, despite the overhead of additional recursions.
Revised: Interestingly, higher K yields faster training as well as moderate prediction time, despite the overhead of additional recursions.

Original: These results show that iRNN outperforms baselines on activity recognition tasks, and ﬁts within IoT/edge-device budgets.
Revised: These results show that iRNN outperforms baselines on activity recognition tasks, and ﬁts within IoT/edge-device budgets.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Drawing inspiration from Rosenblatts Continuous RNNs, we developed discrete time incremental
Revised: Drawing inspiration from Rosenblatts Continuous RNNs, we developed discrete time incremental

Original: RNN (iRNN).
Revised: RNN (iRNN).

Original: Leveraging equilibrium properties of CTRNN, iRNN solves exploding/vanishing gradient problem.
Revised: Leveraging equilibrium properties of CTRNN, iRNN solves exploding/vanishing gradient problem.

Original: We show that iRNN improved gradients are directly correlated with improved test accuracy.
Revised: We show that iRNN improved gradients are directly correlated with improved test accuracy.

Original: A number of experiments demonstrate iRNNs responsiveness to long-term dependency tasks.
Revised: A number of experiments demonstrate iRNNs responsiveness to long-term dependency tasks.

Original: In addition, due to its smooth low-dimensional trajectories, it has a lightweight footprint that can be leveraged for IoT applications. 
Revised: In addition, due to its smooth low-dimensional trajectories, it has a lightweight footprint that can be leveraged for IoT applications. A CKNOWLEDGMENTS

Revised: The authors would like to thank the Area Chair and the reviewers for their constructive comments.

Revised: This work was supported partly by the National Science Foundation Grant 1527618, the Ofﬁce of

Revised: Naval Research Grant N0014-18-1-2257 and by a gift from ARM corporation

Original: A A PPENDIX A. M ULTI -L AYER D EEP RNN N ETWORKS .
Revised: A A PPENDIX A. M ULTI -L AYER D EEP RNN N ETWORKS .

Original: We point out in passing that our framework readily admits deep multi-layered networks within a single time-step.
Revised: We point out in passing that our framework readily admits deep multi-layered networks within a single time-step.

Original: Indeed our setup is general; it applies to shallow and deep nets; small and large time steps.
Revised: Indeed our setup is general; it applies to shallow and deep nets; small and large time steps.

Original: As a case in point, the Deep Transition RNN Pascanu et al.
Revised: As a case in point the Deep Transition RNN Pascanu et al.

Original: (2013c):
Revised: (2013c):

Original: [Equation]
Revised: [Equation]

Original: is readily accounted by Theorem 1 in an implicit form:
Revised: is readily accounted by Theorem 1 in an implicit form:

Original: [Equation]
Revised: [Equation]

Original: So is Deep-RNN Hermans & Schrauwen (2013).
Revised: So is Deep-RNN Hermans & Schrauwen (2013).

Original: The trick is to transform h m → h m +
Revised: The trick is to transform h m → h m +

Original: h
Revised: h

Original: m +

Revised: m +

Original: and h m +

Revised: and h m +

Original: → h m
Revised: → h m

Original: + h
Revised: + h

Original: m +1 .
Revised: m +1 .

Original: As such, all we need is smoothness of f h , which has no restriction on # layers.
Revised: As such, all we need is smoothness of f h , which has no restriction on # layers.

Original: On the other hand, that we do not have to limit the number of time steps is the point of Theorem 1, which asserts that the partial differential of hidden states (which is primarily why vanishing/exploding gradient arises Pascanu et al.
Revised: On the other hand that we do not have to limit the number of time steps is the point of Theorem 1, which asserts that the partial differential of hidden states (which is primarily why vanishing/exploding gradient arises Pascanu et al.

Original: (2013b) in the ﬁrst place) is identity!!
Revised: (2013b) in the ﬁrst place) is identity!!

Original: A. P SEUDO C ODE AND I MPLEMENTATION
Revised: A. P SEUDO C ODE AND I MPLEMENTATION

Original: Given an input sequence and iRNN model parameters, the hidden states can be generated with the help of subroutine 1.
Revised: Given an input sequence and iRNN model parameters, the hidden states can be generated with the help of subroutine 1.

Original: This routine can be plugged into standard deep learning frameworks such as
Revised: This routine can be plugged into standard deep learning frameworks such as

Original: Tensorﬂow/PyTorch to learn the model parameters via back-propagation.
Revised: Tensorﬂow/PyTorch to learn the model parameters via back-propagation.

Original: Data: Input sequence { x m } Tm =
Revised: Data: Input sequence { x m } Tm =

Original: Require:
Revised: Require:

Original: Number of recursion steps K , Model parameters (cid:0) U, W, b, α, { η km } (cid:1) 1 Initial hidden state h 

Revised: Number of recursion steps K , Model parameters (cid:0) U, W, b, α, { η km } (cid:1) 1 Initial hidden state h

Original: =2 for m = 1 to T do
Revised: =2 for m = 1 to T do

Original: Initialize g 0 to zero or h m − for k = 1 to K do g k = g k − 1 + η km (cid:0) φ
Revised: Initialize g 0 to zero or h m − for k = 1 to K do g k = g k − 1 + η km (cid:0) φ

Original: ( U ( g k − 1 + h m − 1 )
Revised: ( U ( g k − 1 + h m − 1 )

Original: + Wx m + b ) − α
Revised: + Wx m + b ) − α

Original: ( g k − 1 + h m − 1 ) (cid:1) h m =
Revised: ( g k − 1 + h m − 1 ) (cid:1) h m =

Original: g K
Revised: g K

Original: Output: hidden states { h m } Tm =
Revised: Output: hidden states { h m } Tm =

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: A. C ONVERGENCE G UARANTEES FOR G ENERAL L EARNING R ATES .
Revised: C ONVERGENCE G UARANTEES FOR G ENERAL L EARNING R ATES .

Original: Theorem 2 (Local Convergence with Linear Rate) .
Revised: Theorem 2 (Local Convergence with Linear Rate) .

Original: Assume that the function F ( g i ) (cid:44) φ ( U ( g i + h k − 1 )
Revised: Assume that the function F ( g i ) (cid:44) φ ( U ( g i + h k − 1 )

Original: + Wx k + b ) − ( g i + h k − 1 ) and the parameter η ( i ) k in Eq.
Revised: + Wx k + b ) − ( g i + h k − 1 ) and the parameter η ( i ) k in Eq.

Original: 5 satisﬁes [ η ( i ) k ] 2 (cid:107)∇ F ( g i ) F ( g i ) (cid:107) 

Revised: 5 satisﬁes [ η ( i ) k ] 2 (cid:107)∇ F ( g i ) F ( g i ) (cid:107)

Original: + 2 η ( i 
Revised: + 2 η ( i )

Original: [Equation]
Revised: [Equation]

Original: Then there exists (cid:15) > 

Revised: Then there exists (cid:15) >

Original: such that if (cid:107) g 0 − h eq (cid:107) ≤ (cid:15) where h eq denotes the ﬁxed point, the sequence g i generated by the Euler method converges to the equilibrium solution in M eq ( h k − 1 , x k ) locally with linear rate.
Revised: such that if (cid:107) g 0 − h eq (cid:107) ≤ (cid:15) where h eq denotes the ﬁxed point, the sequence g i generated by the Euler method converges to the equilibrium solution in M eq ( h k − 1 , x k ) locally with linear rate.

Original: The proof is based on drawing a connection between the Euler method and inexact Newton methods, and leverages Thm. 2.3 in Dembo et al.
Revised: The proof is based on drawing a connection between the Euler method and inexact Newton methods, and leverages Thm. 2.3 in Dembo et al.

Original: See appendix Sec.
Revised: See appendix Sec.

Original: A.8.

Revised: A.8.

Original: Thm. 3 and Sec.
Revised: Thm. 3 and Sec.

Original: A.7.5 (for proof, empirical veriﬁcation).
Revised: A.7.5 (for proof, empirical veriﬁcation).

Original: Corollary 1.
Revised: Corollary 1.

Original: If (cid:107) I + η ( i ) k ∇ F ( g i ) (cid:107) < 1 , ∀ k, ∀ i , the forward propagation (Eq. 13) is stable and the sequence { g i } converges locally at a linear rate.
Revised: If (cid:107) I + η ( i ) k ∇ F ( g i ) (cid:107) < 1 , ∀ k, ∀ i , the forward propagation (Eq. 13) is stable and the sequence { g i } converges locally at a linear rate.

Original: The proof is based on Thm.
Revised: The proof is based on Thm.

Original: 2.3 in Dembo et al.
Revised: 2.3 in Dembo et al.

Original: (1982), Thm. 2 and Prop. 2 in Chang et al.
Revised: (1982), Thm. 2 and Prop. 2 in Chang et al.

Original: See appendix A.8.1 Corollary.
Revised: See appendix A.8.1 Corollary.

Original: A. D ATASET D ETAILS
Revised: A. D ATASET D ETAILS

Original: Table 4 and table 6 lists the statistics of all the datasets described below.
Revised: Table 4 and table 6 lists the statistics of all the datasets described below.

Original: Google-12 & Google-30 : Google Speech Commands dataset contains 1 second long utterances of 30 short words (30 classes) sampled at 16KHz.
Revised: Google-12 & Google-30 : Google Speech Commands dataset contains 1 second long utterances of 30 short words (30 classes) sampled at 16KHz.

Original: Standard log Mel-ﬁlter-bank featurization withﬁlters over a window size of 25ms and stride of 10ms gave 99 timesteps of 32 ﬁlter responses for a 1-second audio clip.
Revised: Standard log Mel-ﬁlter-bank featurization withﬁlters over a window size of 25ms and stride of 10ms gave 99 timesteps of 32 ﬁlter responses for a 1-second audio clip.

Original: For the 12 class version, 10 classes used in Kaggle’s Tensorﬂow Speech
Revised: For the 12 class version, 10 classes used in Kaggle’s Tensorﬂow Speech

Original: Recognition challenge 7 were used and remaining two classes were noise and background sounds (taken randomly from remaining 20 short word utterances).
Revised: Recognition challenge 7 were used and remaining two classes were noise and background sounds (taken randomly from remaining 20 short word utterances).

Original: Both the datasets were zero mean - unit variance normalized during training and prediction.
Revised: Both the datasets were zero mean - unit variance normalized during training and prediction.

Original: challenge HAR-2 8 : Human Activity Recognition (HAR) dataset was collected from an accelerometer and gyroscope on a Samsung Galaxy S3 smartphone.
Revised: HAR-2 8 : Human Activity Recognition (HAR) dataset was collected from an accelerometer and gyroscope on a Samsung Galaxy S3 smartphone.

Original: The features available on the repository were directly used for experiments.
Revised: The features available on the repository were directly used for experiments.

Original: The 6 activities were merged to get the binarized version.
Revised: The 6 activities were merged to get the binarized version.

Original: The classes
Revised: The classes

Original: Sitting, Laying, Walking_Upstairs and Standing, Walking, Walking_Downstairs were merged to obtain the two classes.
Revised: Sitting, Laying, Walking_Upstairs and Standing, Walking, Walking_Downstairs were merged to obtain the two classes.

Original: The dataset was zero mean - unit variance normalized during training and prediction.
Revised: The dataset was zero mean - unit variance normalized during training and prediction.

Original: Penn Treebank : 300 length word sequences were used for word level language modeling task using
Revised: Penn Treebank : 300 length word sequences were used for word level language modeling task using

Original: Penn Treebank (PTB) corpus.
Revised: Penn Treebank (PTB) corpus.

Original: The vocabulary consisted of 10,000 words and the size of trainable word embeddings was kept the same as the number of hidden units of architecture.
Revised: The vocabulary consisted of 10,000 words and the size of trainable word embeddings was kept the same as the number of hidden units of architecture.

Original: This is the setup used in (Kusupati et al., 2018; Zhang et al., 2018).
Revised: This is the setup used in (Kusupati et al., 2018; Zhang et al., 2018).

Original: Pixel-MNIST : Pixel-by-pixel version of the standard MNIST-10 dataset 9 .
Revised: Pixel-MNIST : Pixel-by-pixel version of the standard MNIST-10 dataset 9 .

Original: The dataset was zero mean - unit variance normalized during training and prediction.
Revised: The dataset was zero mean - unit variance normalized during training and prediction.

Original: Permuted-MNIST :
Revised: Permuted-MNIST :

Original: This is similar to Pixel-MNIST, except its made harder by shufﬂing the pixels with a ﬁxed permutation.
Revised: This is similar to Pixel-MNIST, except its made harder by shufﬂing the pixels with a ﬁxed permutation.

Original: We keep the random seed as 42 to generate the permutation of 784 pixels.
Revised: We keep the random seed as 42 to generate the permutation of 784 pixels.

Original: Noisy-MNIST :
Revised: Noisy-MNIST :

Original: To introduce more long-range dependencies to the Pixel-MNIST task, we deﬁne a more challenging task called the Noisy-MNIST, inspired by the noise padded experiments in Chang et al.
Revised: To introduce more long-range dependencies to the Pixel-MNIST task, we deﬁne a more challenging task called the Noisy-MNIST, inspired by the noise padded experiments in Chang et al.

Original: Instead of feeding in one pixel at one time, we input each row of a MNIST image at every time step.
Revised: Instead of feeding in one pixel at one time, we input each row of a MNIST image at every time step.

Original: After the ﬁrst 28 time steps, we input independent standard Gaussian noise for the remaining time steps.
Revised: After the ﬁrst 28 time steps, we input independent standard Gaussian noise for the remaining time steps.

Original: Since a MNIST image is of size 28 with 1 RGB channels, the input dimension is m = 28.
Revised: Since a MNIST image is of size 28 with 1 RGB channels, the input dimension is m = 28.

Original: The total number of time steps is set to T = 1000.
Revised: The total number of time steps is set to T = 1000.

Original: In other words, only the ﬁrst 28 time steps of input contain salient information, all remaining 972 time steps are merely random noise.
Revised: In other words, only the ﬁrst 28 time steps of input contain salient information, all remaining 972 time steps are merely random noise.

Original: For a model to correctly classify an input image, it has to remember the information from a long time ago.
Revised: For a model to correctly classify an input image, it has to remember the information from a long time ago.

Original: This task is conceptually more difﬁcult than the pixel-by-pixel MNIST, although the total amount of signal in the input sequence is the same.
Revised: This task is conceptually more difﬁcult than the pixel-by-pixel MNIST, although the total amount of signal in the input sequence is the same.

Original: Noisy-CIFAR :
Revised: Noisy-CIFAR :

Original: This is exactly replica of the noise paded CIFAR task mentioned in Chang et al.
Revised: This is exactly replica of the noise paded CIFAR task mentioned in Chang et al.

Original: Instead of feeding in one pixel at one time, we input each row of a CIFAR-10 image at every time step.
Revised: Instead of feeding in one pixel at one time, we input each row of a CIFAR-10 image at every time step.

Original: After the ﬁrst 32 time steps, we input independent standard Gaussian noise for the remaining time steps.
Revised: After the ﬁrst 32 time steps, we input independent standard Gaussian noise for the remaining time steps.

Original: Since a CIFAR-10 image is of size 32 with three RGB channels, the input dimension is m = 96.
Revised: Since a CIFAR-10 image is of size 32 with three RGB channels, the input dimension is m = 96.

Original: The total number of time steps is set to T = 1000.
Revised: The total number of time steps is set to T = 1000.

Original: In other words, only the ﬁrst 32 time steps of input contain salient information, all remaining 968 time steps are merely random noise.
Revised: In other words, only the ﬁrst 32 time steps of input contain salient information, all remaining 968 time steps are merely random noise.

Original: For a model to correctly classify an input image, it has to remember the information from a long time ago.
Revised: For a model to correctly classify an input image, it has to remember the information from a long time ago.

Original: This task is conceptually more difﬁcult than the pixel-by-pixel CIFAR-10, although the total amount of signal in the input sequence is the same.
Revised: This task is conceptually more difﬁcult than the pixel-by-pixel CIFAR-10, although the total amount of signal in the input sequence is the same.

Original: Addition Task : We closely follow the adding problem deﬁned in (Arjovsky et al., 2016; Hochreiter & Schmidhuber, 1997) to explain the task at hand.
Revised: Addition Task : We closely follow the adding problem deﬁned in (Arjovsky et al., 2016; Hochreiter & Schmidhuber, 1997) to explain the task at hand.

Original: Each input consists of two sequences of length T.
Revised: Each input consists of two sequences of length T.

Original: The ﬁrst sequence, which we denote x , consists of numbers sampled uniformly at random U [0 , 1] .
Revised: The ﬁrst sequence, which we denote x , consists of numbers sampled uniformly at random U [0 , 1] .

Original: The second sequence is an indicator sequence consisting of exactly two entries of 1 and remaining entries 0 .
Revised: The second sequence is an indicator sequence consisting of exactly two entries of 1 and remaining entries 0 .

Original: The ﬁrst 1 entry is located uniformly at random in the ﬁrst half of the sequence, whilst the second 1 entry is located uniformly at random in the second half.
Revised: The ﬁrst 1 entry is located uniformly at random in the ﬁrst half of the sequence, whilst the second 1 entry is located uniformly at random in the second half.

Original: The output is the sum of the two entries of the ﬁrst sequence, corresponding to where the 1 entries are located in the second sequence.
Revised: The output is the sum of the two entries of the ﬁrst sequence, corresponding to where the 1 entries are located in the second sequence.

Original: A naive strategy of predicting 1 as the output regardless of the input sequence gives an expected mean squared error of 0.167, the variance of the sum of two independent uniform distributions.
Revised: A naive strategy of predicting 1 as the output regardless of the input sequence gives an expected mean squared error of 0.167, the variance of the sum of two independent uniform distributions.

Original: Copying Task :
Revised: Copying Task :

Original: Following a similar setup to (Arjovsky et al., 2016; Hochreiter & Schmidhuber, 1997), we outline the copy memory task.
Revised: Following a similar setup to (Arjovsky et al., 2016; Hochreiter & Schmidhuber, 1997), we outline the copy memory task.

Original: Consider 10 categories, { a i } 9 i =0 .
Revised: Consider 10 categories, { a i } 9 i =0 .

Original: The input takes the form of a T + 20 length vector of categories, where we test over a range of values of T. The ﬁrstentries are sampled uniformly, independently and with replacement from { a i } 7 i =0 , and represent the sequence which will need to be remembered.
Revised: The input takes the form of a T + 20 length vector of categories, where we test over a range of values of T. The ﬁrstentries are sampled uniformly, independently and with replacement from { a i } 7 i =0 , and represent the sequence which will need to be remembered.

Original: The next T − 1 entries are set to a 8 , which can be thought of as the ’blank’ category.
Revised: The next T − 1 entries are set to a 8 , which can be thought of as the ’blank’ category.

Original: The next single entry is a 9 , which represents a delimiter, which should indicate to the algorithm that it is now required to reproduce the initial 10 categories in the output.
Revised: The next single entry is a 9 , which represents a delimiter, which should indicate to the algorithm that it is now required to reproduce the initial 10 categories in the output.

Original: The remaining 10 entries are set to a 8 .
Revised: The remaining 10 entries are set to a 8 .

Original: The required output sequence consists of T +repeated entries of a 8 , followed by the ﬁrst 10 categories of the input sequence in exactly the same order.
Revised: The required output sequence consists of T +repeated entries of a 8 , followed by the ﬁrst 10 categories of the input sequence in exactly the same order.

Original: The goal is to minimize the average cross entropy of category predictions at each time step of the sequence.
Revised: The goal is to minimize the average cross entropy of category predictions at each time step of the sequence.

Original: The task amounts to having to remember a categorical sequence of length 10, for T time steps.
Revised: The task amounts to having to remember a categorical sequence of length 10, for T time steps.

Original: A simple baseline can be established by considering an optimal strategy when no memory is available, which we deem the memoryless strategy.
Revised: A simple baseline can be established by considering an optimal strategy when no memory is available, which we deem the memoryless strategy.

Original: The memoryless strategy would be to predict a 8 for T +entries and then predict each of the ﬁnal 10 categories from the set { a i } 7 i =

Revised: The memoryless strategy would be to predict a 8 for T +entries and then predict each of the ﬁnal 10 categories from the set { a i } 7 i =

Original: i=0 independently and uniformly at random.
Revised: i=0 independently and uniformly at random.

Original: The categorical cross entropy of this strategy is 10log(8) T +
Revised: The categorical cross entropy of this strategy is 10log(8) T +

Original: DSA-19 10 : This dataset is based on Daily and Sports Activity (DSA) detection from a resourceconstrained IoT wearable device with 5 Xsens MTx sensors having accelerometers, gyroscopes and magnetometers on the torso and four limbs.
Revised: DSA-19 10 : This dataset is based on Daily and Sports Activity (DSA) detection from a resourceconstrained IoT wearable device with 5 Xsens MTx sensors having accelerometers, gyroscopes and magnetometers on the torso and four limbs.

Original: The features available on the repository were used for experiments.
Revised: The features available on the repository were used for experiments.

Original: The dataset was zero mean - unit variance normalized during training and prediction.
Revised: The dataset was zero mean - unit variance normalized during training and prediction.

Original: Yelp-5 : Sentiment Classiﬁcation dataset based on the text reviews 11 .
Revised: Yelp-5 : Sentiment Classiﬁcation dataset based on the text reviews 11 .

Original: The data consists of 500, train points and 500,000 test points from the ﬁrst 1 million reviews.
Revised: The data consists of 500, train points and 500,000 test points from the ﬁrst 1 million reviews.

Original: Each review was clipped or padded to be 300 words long.
Revised: Each review was clipped or padded to be 300 words long.

Original: The vocabulary consisted of 20000 words and 128 dimensional word embeddings were jointly trained with the network.
Revised: The vocabulary consisted of 20000 words and 128 dimensional word embeddings were jointly trained with the network.

Original: A. B ASELINE J USTIFICATION
Revised: A. B ASELINE J USTIFICATION

Original: In our experiments section, we stated that some of the potential baselines were removed due to experimental conditions enforced in the setup.
Revised: In our experiments section, we stated that some of the potential baselines were removed due to experimental conditions enforced in the setup.

Original: Here we clearly justify our choice.
Revised: Here we clearly justify our choice.

Original: Mostly the reasoning is to avoid comparing complementary add-ons and compare the bare-bone cells.
Revised: Mostly the reasoning is to avoid comparing complementary add-ons and compare the bare-bone cells.

Original: • Cooijmans et al.
Revised: • Cooijmans et al.

Original: (2016) is removed since its an add-on and can be applied to any method.
Revised: (2016) is removed since its an add-on and can be applied to any method.

Original: Besides its pixel-mnist results involve dataset speciﬁc heuristics.
Revised: Besides its pixel-mnist results involve dataset speciﬁc heuristics.

Original: • Gong et al.
Revised: • Gong et al.

Original: (2018) is also an add-on and hence can be applied to any method.
Revised: (2018) is also an add-on and hence can be applied to any method.

Original: • Zilly et al.
Revised: • Zilly et al.

Original: ; Pascanu et al.
Revised: ; Pascanu et al.

Original: (2013a)
Revised: (2013a)

Original: ; Mujika et al.
Revised: ; Mujika et al.

Original: (2017) denote deep transitioning methods.
Revised: (2017) denote deep transitioning methods.

Original: They are add-ons for any single recurrent block and hence can be applied to any recurrent cell.
Revised: They are add-ons for any single recurrent block and hence can be applied to any recurrent cell.

Original: • Gating variants of single recurrent
Revised: • Gating variants of single recurrent

Original: cells (Chang et al., 2019; Kusupati et al., 2018) have also been removed.
Revised: cells (Chang et al., 2019; Kusupati et al., 2018) have also been removed.

Original: Since iRNN can be extended to a gating variant and hence its just an add-on.
Revised: Since iRNN can be extended to a gating variant and hence its just an add-on.

Original: [Table]
Revised: [Table]

Original: A. H YPER - PARAMETERS FOR REPRODUCIBILITY
Revised: A. H YPER - PARAMETERS FOR REPRODUCIBILITY

Original: We report various hyper-parameters we use in our experiments for reproduciblity.
Revised: We report various hyper-parameters we use in our experiments for reproduciblity.

Original: As mentioned earlier we mainly use ’ReLU’ as the non-linearity and Adam as the optimizer.
Revised: As mentioned earlier we mainly use ’ReLU’ as the non-linearity and Adam as the optimizer.

Original: Apart from this, other hyper-parameters are mentioned in table 5.
Revised: Apart from this, other hyper-parameters are mentioned in table 5.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: A. A DDITIONAL E XPERIMENTS A.7. C OPYING AND A DDITION T ASKS
Revised: A. A DDITIONAL E XPERIMENTS A.7. C OPYING AND A DDITION T ASKS

Original: Figure 5 shows the results for remaining experiments for the addition task for length 100 , 400 .
Revised: Figure 5 shows the results for remaining experiments for the addition task for length 100 , 400 .

Original: [Table]
Revised: [Table]

Revised: 53k

Original: https:
Revised: URNN [Table] MNIST [Table]

Original: A.7. T RADITIONAL D ATASETS
Revised: A.7. T RADITIONAL D ATASETS

Original: Table 7 shows the results including left out baselines for Pixel-MNIST and permute-MNIST task.
Revised: Table 7 shows the results including left out baselines for Pixel-MNIST and permute-MNIST task.

Original: Here we also include star rating prediction on a scale of 1 to 5 of Yelp reviews Yelp (2017).
Revised: Here we also include star rating prediction on a scale of 1 to 5 of Yelp reviews Yelp (2017).

Original: Tableshows the results for this dataset.
Revised: Tableshows the results for this dataset.

Original: [Table]
Revised: [Table]

Original: A.7. A CTIVITY R ECOGNITION D ATASETS
Revised: A.7. A CTIVITY R ECOGNITION D ATASETS

Original: We also include activity recognition tasks: (a)Google-12 Warden (2018) , i.e. detection of utterances of 10 commands plus background noise and silence and (b) DSA-19 Altun et al.
Revised: We also include activity recognition tasks: (a)Google-12 Warden (2018) , i.e. detection of utterances of 10 commands plus background noise and silence and (b) DSA-19 Altun et al.

Original: (2010), Daily and
Revised: (2010), Daily and

Original: Sports Activity (DSA) detection from a resource-constrained IoT wearable device with 5 Xsens MTx sensors having accelerometers, gyroscopes and magnetometers on the torso and four limbs.
Revised: Sports Activity (DSA) detection from a resource-constrained IoT wearable device with 5 Xsens MTx sensors having accelerometers, gyroscopes and magnetometers on the torso and four limbs.

Original: Table 9 shows results for these activities along with some other baselines for activity recognition tasks mentioned in Sec.
Revised: Table 9 shows results for these activities along with some other baselines for activity recognition tasks mentioned in Sec.

Original: 4.3.4 and described in Sec.
Revised: 4.3.4 and described in Sec.

Original: A.4.
Revised: A.4.

Original: A.7. PTB L ANGUAGE M ODELLING
Revised: A.7. PTB L ANGUAGE M ODELLING

Original: We follow (Kusupati et al., 2018; Zhang et al., 2018) to setup our PTB experiments.
Revised: We follow (Kusupati et al., 2018; Zhang et al., 2018) to setup our PTB experiments.

Original: We only pursue one layer language modelling, but with more difﬁcult sequence length ( 300 ).
Revised: We only pursue one layer language modelling, but with more difﬁcult sequence length ( 300 ).

Original: Table 10 reports all the evaluation metrics for the PTB Language modelling task with 1 layer as setup by Kusupati et al.
Revised: Table 10 reports all the evaluation metrics for the PTB Language modelling task with 1 layer as setup by Kusupati et al.

Original: (2018), including test time and number of parameters (which we omitted from the main paper due to lack of space).
Revised: (2018), including test time and number of parameters (which we omitted from the main paper due to lack of space).

Original: A.7. L INEAR R ATE OF C ONVERGENCE TO F IXED P OINT
Revised: A.7. L INEAR R ATE OF C ONVERGENCE TO F IXED P OINT

Original: [Figure]
Revised: [Figure]

Original: Empirically we verify the local convergence to a ﬁxed point with linear rate by comparing the Euclidean distance between the approximate solutions, h ( k ) t , using Eq.with g 0 = 0 and the ﬁxed points, h t , computed using
Revised: Empirically we verify the local convergence to a ﬁxed point with linear rate by comparing the Euclidean distance between the approximate solutions, h ( k ) t , using Eq.with g 0 = 0 and the ﬁxed points, h t , computed using

Original: FSOLVE from SCIPY .
Revised: FSOLVE from SCIPY .

Original: The learnable parameters are initialized suitably and then ﬁxed.
Revised: The learnable parameters are initialized suitably and then ﬁxed.

Original: We illustrate our results in
Revised: We illustrate our results in

Original: Fig.
Revised: Fig.

Original: 6, which clearly demonstrates that the approximate solutions tend to converge with linear rate.
Revised: 6, which clearly demonstrates that the approximate solutions tend to converge with linear rate.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: A.7. T HEORETICAL V ERIFICATION
Revised: A.7. T HEORETICAL V ERIFICATION

Original: Here we include some experiments to show that our theoretical assumptions hold true.
Revised: Here we include some experiments to show that our theoretical assumptions hold true.

Original: [Figure]
Revised: [Figure]

Original: Non-Singularity of the matrix D
Revised: Non-Singularity of the matrix D

Original: For our iRNN parametrization to satisfy the conditions of having equillibrium points to be locally asymptotically stable, the eigen values of the matrix D
Revised: For our iRNN parametrization to satisfy the conditions of having equillibrium points to be locally asymptotically stable, the eigen values of the matrix D

Original: = ( ∇ φ ( · ) U − γI ) should be negative.
Revised: = ( ∇ φ ( · ) U − γI ) should be negative.

Original: We plot a histogram of the eigenvalues of D for all the points in the HARdataset.
Revised: We plot a histogram of the eigenvalues of D for all the points in the HARdataset.

Original: As illustrated in the ﬁgure 7, all the eigenvalues are negative.
Revised: As illustrated in the ﬁgure 7, all the eigenvalues are negative.

Original: A.7. I DENTITY G RADIENT COMPARISON iRNN VS RNN
Revised: A.7. I DENTITY G RADIENT COMPARISON iRNN VS RNN

Original: To verify Theorem.
Revised: To verify Theorem.

Original: 1 empirically, we train RNN and iRNN on the HAR-2 data set (see more details in Sec. 4), respectively, and plot in Fig.
Revised: 1 empirically, we train RNN and iRNN on the HAR-2 data set (see more details in Sec. 4), respectively, and plot in Fig.

Original: 8 the magnitude of gradient of the last layer h T w.r.t.
Revised: 8 the magnitude of gradient of the last layer h T w.r.t.

Original: the ﬁrst
Revised: the ﬁrst

Original: [Figure]
Revised: [Figure]

Original: layer h 1 in log scale to conﬁrm that our approach leads to no vanishing or exploding gradients when the error is back-propagated through time.
Revised: layer h 1 in log scale to conﬁrm that our approach leads to no vanishing or exploding gradients when the error is back-propagated through time.

Original: We also conducted experiments to verify that the gradient of iRNN is norm preserving (see Sec.
Revised: We also conducted experiments to verify that the gradient of iRNN is norm preserving (see Sec.

Original: A.7.8 and Figure . 3).
Revised: A.7.8 and Figure . 3).

Original: As we see clearly, RNN suffers from serious vanishing gradient issue in training, while iRNN’s backpropagated gradients is close to 1, and the variance arises mainly our approximation of ﬁxed points and stochastic behavior in training networks, demonstrating much better training stability of iRNN.
Revised: As we see clearly, RNN suffers from serious vanishing gradient issue in training, while iRNN’s backpropagated gradients is close to 1, and the variance arises mainly our approximation of ﬁxed points and stochastic behavior in training networks, demonstrating much better training stability of iRNN.

Original: A.7. G RADIENT NORM W . R . T . LOSS (cid:107) ∂L∂h 1 (cid:107)
Revised: A.7. G RADIENT NORM W . R . T . LOSS (cid:107) ∂L∂h 1 (cid:107)

Original: In addition to the gradient ratio we plot in Sec.4.2, we also show in ﬁgure 9, the more popular quantity captured in earlier works (Arjovsky et al., 2016; Zhang et al., 2018), i.e. the gradient norm w.r.t.
Revised: In addition to the gradient ratio we plot in Sec.4.2, we also show in ﬁgure 9, the more popular quantity captured in earlier works (Arjovsky et al., 2016; Zhang et al., 2018), i.e. the gradient norm w.r.t.

Original: loss (cid:107) ∂L∂h 1 (cid:107) .
Revised: loss (cid:107) ∂L∂h 1 (cid:107) .

Original: We emphasize that this quantity alone is misleading in the context of resolving the issue of vanishing/exploding gradients.
Revised: We emphasize that this quantity alone is misleading in the context of resolving the issue of vanishing/exploding gradients.

Original: Since (cid:107) ∂L∂h 1 (cid:107) =
Revised: Since (cid:107) ∂L∂h 1 (cid:107) =

Original: (cid:107) ∂L∂h T (cid:107) ∗ (cid:107) ∂h T
Revised: (cid:107) ∂L∂h T (cid:107) ∗ (cid:107) ∂h T

Original: ∂h 1 (cid:107) .
Revised: ∂h 1 (cid:107) .

Original: The long term component controlling the gradients is (cid:107) ∂h T
Revised: The long term component controlling the gradients is (cid:107) ∂h T

Original: ∂h 1 (cid:107) , but the other component, (cid:107) ∂L∂h T (cid:107) could become zero by the virtue that the loss is nearly zero.
Revised: ∂h 1 (cid:107) , but the other component, (cid:107) ∂L∂h T (cid:107) could become zero by the virtue that the loss is nearly zero.

Original: This happens in our addition task experiment, because MSE is close to zero, we experience nearly 0 value for this quantity.
Revised: This happens in our addition task experiment, because MSE is close to zero, we experience nearly 0 value for this quantity.

Original: But this is clearly because the MSE is 0 .
Revised: But this is clearly because the MSE is 0 .

Original: Also note that none of our graphs have log scale, which is not the case in earlier works.
Revised: Also note that none of our graphs have log scale, which is not the case in earlier works.

Original: The conclusion that can be drawn from the loss-gradient is that it is somewhat stable, and can inform us about quality of convergence.
Revised: The conclusion that can be drawn from the loss-gradient is that it is somewhat stable, and can inform us about quality of convergence.

Original: We also plot (cid:107) ∂h T ∂h T − 1 (cid:107) in ﬁgure 9 in order to show that indeed iRNN achieves identity gradients everywhere in the time horizon, since ﬁg. 3 had shown that the ratio of (cid:107) ∂h T ∂h 1 (cid:107) and (cid:107) ∂h T
Revised: We also plot (cid:107) ∂h T ∂h T − 1 (cid:107) in ﬁgure 9 in order to show that indeed iRNN achieves identity gradients everywhere in the time horizon, since ﬁg. 3 had shown that the ratio of (cid:107) ∂h T ∂h 1 (cid:107) and (cid:107) ∂h T

Original: ∂h T − 1 (cid:107) equalsfor iRNN .
Revised: ∂h T − 1 (cid:107) equalsfor iRNN .

Original: A.7. D IFFERENT A CTIVATION F UNCTION
Revised: A.7. D IFFERENT A CTIVATION F UNCTION

Original: We also performed some experiments for sigmoid activation on HAR-2 dataset.
Revised: We also performed some experiments for sigmoid activation on HAR-2 dataset.

Original: The results for this variant also follow similar pattern as we saw in ReLU variant.
Revised: The results for this variant also follow similar pattern as we saw in ReLU variant.

Original: A. P ROOFS A.8. L OCAL C ONVERGENCE WITH L INEAR R ATE
Revised: A. P ROOFS A.8. L OCAL C ONVERGENCE WITH L INEAR R ATE

Original: Recall that we rewrite the ﬁxed-point constraints in our iRNN as the following ODE:
Revised: Recall that we rewrite the ﬁxed-point constraints in our iRNN as the following ODE:

Original: [Equation]
Revised: [Equation]

Original: Then based on the Euler method, we have the following update rule for solving ﬁxed-points:
Revised: Then based on the Euler method, we have the following update rule for solving ﬁxed-points:

Original: [Equation]
Revised: [Equation]

Original: Inexact Newton methods Dembo et al.
Revised: Inexact Newton methods Dembo et al.

Original: (1982) refer to a family of algorithms that aim to solve the equation system F ( z ) = 0 approximately at each iteration using the following rule:
Revised: (1982) refer to a family of algorithms that aim to solve the equation system F ( z ) = 0 approximately at each iteration using the following rule:

Original: [Equation]
Revised: [Equation]

Original: where ∇ F denotes the (sub)gradient of function F , and r i denotes the error at the i -th iteration between F ( z i ) and 0 .
Revised: where ∇ F denotes the (sub)gradient of function F , and r i denotes the error at the i -th iteration between F ( z i ) and 0 .

Original: By drawing the connection between Eq. 13 and Eq. 15, we can set z i ≡ g i and s i ≡ η ( i )
Revised: By drawing the connection between Eq. 13 and Eq. 15, we can set z i ≡ g i and s i ≡ η ( i )

Original: k F
Revised: k F

Original: ( g i ) .
Revised: ( g i ) .

Original: Then based on Eq.
Revised: Then based on Eq.

Original: 15 we have
Revised: 15 we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 1 (Thm. 2.3 in Dembo et al.
Revised: Lemma 1 (Thm. 2.3 in Dembo et al.

Original: Assume that
Revised: Assume that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (cid:107) · (cid:107) denotes an arbitrary norm and the induced operator norm.
Revised: where (cid:107) · (cid:107) denotes an arbitrary norm and the induced operator norm.

Original: There exists ε > 0 such that, if (cid:107) z 0 − z ∗ (cid:107) ≤ ε , then the sequence of inexact Newton iterates { z i } converges to z ∗ .
Revised: There exists ε > 0 such that, if (cid:107) z 0 − z ∗ (cid:107) ≤ ε , then the sequence of inexact Newton iterates { z i } converges to z ∗ .

Original: Moreover, the convergence is linear in the sense that (cid:107) z i +1 − z ∗ (cid:107) ∗ ≤ τ (cid:107) z i − z ∗ (cid:107) ∗ , where (cid:107) y
Revised: Moreover, the convergence is linear in the sense that (cid:107) z i +1 − z ∗ (cid:107) ∗ ≤ τ (cid:107) z i − z ∗ (cid:107) ∗ , where (cid:107) y

Original: (cid:107) ∗
Revised: (cid:107) ∗

Original: =
Revised: =

Original: (cid:107)∇ F ( z ∗ ) y
Revised: (cid:107)∇ F ( z ∗ ) y

Original: (cid:107) .
Revised: (cid:107) .

Original: Training Steps
Revised: Training Steps

Original: G r a d i e n t n o r m |
Revised: G r a d i e n t n o r m |

Original: L h 1 |
Revised: L h 1 |

Original: Sequence length =
Revised: Sequence length =

Original: LSTM
Revised: LSTM

Original: FastRNN(eta=0.01)
Revised: FastRNN(eta=0.01)

Original: FastRNN(eta=0.001)
Revised: FastRNN(eta=0.001)

Original: Antisym(g=0.01,e=0.1)
Revised: Antisym(g=0.01,e=0.1)

Original: Antisym(g=0.01,e=0.001) iRNN(K=1) iRNN(K=5) iRNN(K=10)
Revised: Antisym(g=0.01,e=0.001) iRNN(K=1) iRNN(K=5) iRNN(K=10)

Original: (a)
Revised: (a)

Original: Training Steps
Revised: Training Steps

Original: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: G r a d i e n t n o r m | h T h
Revised: G r a d i e n t n o r m | h T h

Original: T|
Revised: T|

Original: Sequence length =
Revised: Sequence length =

Original: LSTM
Revised: LSTM

Original: FastRNN(eta=0.01)
Revised: FastRNN(eta=0.01)

Original: FastRNN(eta=0.001)
Revised: FastRNN(eta=0.001)

Original: Antisym(g=0.01,e=0.1)
Revised: Antisym(g=0.01,e=0.1)

Original: Antisym(g=0.01,e=0.001) iRNN(K=1) iRNN(K=5) iRNN(K=10)
Revised: Antisym(g=0.01,e=0.001) iRNN(K=1) iRNN(K=5) iRNN(K=10)

Original: (b)
Revised: (b)

Original: Theorem 3 (Local Convergence with Linear Rate) .
Revised: Theorem 3 (Local Convergence with Linear Rate) .

Original: Assume that the function F in Eq.
Revised: Assume that the function F in Eq.

Original: 12 and the parameter η
Revised: 12 and the parameter η

Original: ( i ) k in Eq.
Revised: ( i ) k in Eq.

Original: 13 satisfy
Revised: 13 satisfy

Original: [Equation]
Revised: [Equation]

Original: Then there exists (cid:15) > 0 such that if (cid:107) g 0 − h eq (cid:107) ≤ (cid:15) where h eq denotes the ﬁxed point, the sequence { g i } generated by the Euler method converges to the equilibrium solution in M eq ( h k − 1 , x k ) locally with linear rate.
Revised: Then there exists (cid:15) > 0 such that if (cid:107) g 0 − h eq (cid:107) ≤ (cid:15) where h eq denotes the ﬁxed point, the sequence { g i } generated by the Euler method converges to the equilibrium solution in M eq ( h k − 1 , x k ) locally with linear rate.

Original: Proof.
Revised: Proof.

Original: By substituting Eq. 16 into Eq.
Revised: By substituting Eq. 16 into Eq.

Original: 17, to prove local convergence we need to guarantee
Revised: 17, to prove local convergence we need to guarantee

Original: [Equation]
Revised: [Equation]

Original: By taking the square of both sides in Eq.
Revised: By taking the square of both sides in Eq.

Original: 19, we can show that Eq.
Revised: 19, we can show that Eq.

Original: 19 is equivalent to Eq.
Revised: 19 is equivalent to Eq.

Original: We then complete our proof.
Revised: We then complete our proof.

Original: Corollary 2.
Revised: Corollary 2.

Original: Assume that (cid:107) I + η ( i )
Revised: Assume that (cid:107) I + η ( i )

Original: k ∇ F ( g i ) (cid:107) < 1 , ∀ i, ∀ k holds.
Revised: k ∇ F ( g i ) (cid:107) < 1 , ∀ i, ∀ k holds.

Original: Then the forward propagation using
Revised: Then the forward propagation using

Original: Eq. 13 is stable and our sequence { g i } converges locally with linear rate.
Revised: Eq. 13 is stable and our sequence { g i } converges locally with linear rate.

Original: Proof.
Revised: Proof.

Original: By substituting Eq. 16 into Eq.
Revised: By substituting Eq. 16 into Eq.

Original: 17 and based on the assumption in the corollary, we have
Revised: 17 and based on the assumption in the corollary, we have

Original: [Equation]
Revised: [Equation]

Original: Further based on Prop. 2 in Chang et al.
Revised: Further based on Prop. 2 in Chang et al.

Original: and Thm. 2, we then complete our proof.
Revised: and Thm. 2, we then complete our proof.

Original: [Table]
Revised: [Table]

Original: Antisymmetric 85.
Revised: Antisymmetric 85.

Original: 8.3k
Revised: 8.3k

Original: UGRNN 84.
Revised: UGRNN 84.

Original: iRNN (K=1) 88.
Revised: iRNN (K=1) 88.

Original: 3.5k iRNN (K=3) 85.
Revised: 3.5k iRNN (K=3) 85.

Original: 3.5k iRNN (K=5) 87.
Revised: 3.5k iRNN (K=5) 87.

Original: 3.5k
Revised: 3.5k

Original: GoogleFastRNN 92.
Revised: GoogleFastRNN 92.

Original: 12k
Revised: 12k

Original: FastGRNN-LSQ 93.
Revised: FastGRNN-LSQ 93.

Original: 12k
Revised: 12k

Original: FastGRNN 92.
Revised: FastGRNN 92.

Original: RNN 73.
Revised: RNN 73.

Original: 12k
Revised: 12k

Original: SpectralRNN 91.
Revised: SpectralRNN 91.

Original: 49k
Revised: 49k

Original: EURNN 76.
Revised: EURNN 76.

Original: LSTM 92.
Revised: LSTM 92.

Original: 45k
Revised: 45k

Original: GRU 93.
Revised: GRU 93.

Original: 53k
Revised: 53k

Original: Antisymmetric 89.
Revised: Antisymmetric 89.

Original: 12k
Revised: 12k

Original: UGRNN 92.
Revised: UGRNN 92.

Original: iRNN (K=1) 93.
Revised: iRNN (K=1) 93.

Original: 8.1k iRNN (K=3) 94. 8.1k iRNN (K=5) 94.
Revised: 8.1k iRNN (K=3) 94. 8.1k iRNN (K=5) 94.

Original: 8.1k
Revised: 8.1k

Original: GoogleFastRNN 91.
Revised: GoogleFastRNN 91.

Original: 18k
Revised: 18k

Original: FastGRNN-LSQ 92.
Revised: FastGRNN-LSQ 92.

Original: 8.5k
Revised: 8.5k

Original: FastGRNN 90.
Revised: FastGRNN 90.

Original: RNN 80.
Revised: RNN 80.

Original: 12k
Revised: 12k

Original: SpectralRNN 88.
Revised: SpectralRNN 88.

Original: 24k
Revised: 24k

Original: EURNN 56.
Revised: EURNN 56.

Original: LSTM 90.
Revised: LSTM 90.

Original: 41k
Revised: 41k

Original: GRU 91.
Revised: GRU 91.

Original: 48.5k
Revised: 48.5k

Original: Antisymmetric 90.
Revised: Antisymmetric 90.

Original: 12k
Revised: 12k

Original: UGRNN 90.
Revised: UGRNN 90.

Original: iRNN (K=1) 93.
Revised: iRNN (K=1) 93.

Original: 8.5k iRNN (K=3) 91.
Revised: 8.5k iRNN (K=3) 91.

Original: 8.5k iRNN (K=5) 94.
Revised: 8.5k iRNN (K=5) 94.

Original: 8.5k
Revised: 8.5k

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  iRNN
Revised: 4k iRNN

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table]
Revised: [Table]

Original: Sigmoid 4k iRNN (K=
Revised: ReLU 4k iRNN (K=1)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Sigmoid 4k 
Revised: Sigmoid 4k iRNN (K=3) 93.

Original: [Table]
Revised: Sigmoid 4k iRNN (K=5) 95.

Original: [Table]
Revised: [Table]

Original: Sigmoid 4k pre-deﬁned recursions embedded in graph to reach equillibrium.
Revised: Sigmoid 4k -deﬁned recursions embedded in graph to reach equillibrium.