Original: I MPROVED MEMORY IN RECURRENT NEURAL NETWORKS WITH SEQUENTIAL NON - NORMAL DYNAMICS
Revised: I MPROVED MEMORY IN RECURRENT NEURAL NETWORKS WITH SEQUENTIAL NON - NORMAL DYNAMICS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training recurrent neural networks (RNNs) is a hard problem due to degeneracies in the optimization landscape, a problem also known as vanishing/exploding gradients.
Revised: Training recurrent neural networks (RNNs) is a hard problem due to degeneracies in the optimization landscape, a problem also known as vanishing/exploding gradients.

Original: Short of designing new RNN architectures, previous methods for dealing with this problem usually boil down to orthogonalization of the recurrent dynamics, either at initialization or during the entire training period.
Revised: Short of designing new RNN architectures, previous methods for dealing with this problem usually boil down to orthogonalization of the recurrent dynamics, either at initialization or during the entire training period.

Original: The basic motivation behind these methods is that orthogonal transformations are isometries of the Euclidean space, hence they preserve (Euclidean) norms and effectively deal with vanishing/exploding gradients.
Revised: The basic motivation behind these methods is that orthogonal transformations are isometries of the Euclidean space, hence they preserve (Euclidean) norms and effectively deal with vanishing/exploding gradients.

Original: However, this ignores the crucial effects of non-linearity and noise .
Revised: However, this ignores the crucial effects of non-linearity and noise .

Original: In the presence of a non-linearity, orthogonal transformations no longer preserve norms, suggesting that alternative transformations might be better suited to non-linear networks.
Revised: In the presence of a non-linearity, orthogonal transformations no longer preserve norms, suggesting that alternative transformations might be better suited to non-linear networks.

Original: Moreover, in the presence of noise, norm preservation itself ceases to be the ideal objective.
Revised: Moreover, in the presence of noise, norm preservation itself ceases to be the ideal objective.

Original: A more sensible objective is maximizing the signalto-noise ratio (SNR) of the propagated signal instead.
Revised: A more sensible objective is maximizing the signalto-noise ratio (SNR) of the propagated signal instead.

Original: Previous work has shown that in the linear case, recurrent networks that maximize the SNR display strongly non-normal, sequential dynamics and orthogonal networks are highly suboptimal by this measure.
Revised: Previous work has shown that in the linear case, recurrent networks that maximize the SNR display strongly non-normal, sequential dynamics and orthogonal networks are highly suboptimal by this measure.

Original: Motivated by this ﬁnding, here we investigate the potential of non-normal RNNs, i.e. RNNs with a non-normal recurrent connectivity matrix, in sequential processing tasks.
Revised: Motivated by this ﬁnding, here we investigate the potential of non-normal RNNs, i.e. RNNs with a non-normal recurrent connectivity matrix, in sequential processing tasks.

Original: Our experimental results show that non-normal RNNs outperform their orthogonal counterparts in a diverse range of benchmarks.
Revised: Our experimental results show that non-normal RNNs outperform their orthogonal counterparts in a diverse range of benchmarks.

Original: We also ﬁnd evidence for increased non-normality and hidden chain-like feedforward motifs in trained RNNs initialized with orthogonal recurrent connectivity matrices.
Revised: We also ﬁnd evidence for increased non-normality and hidden chain-like feedforward motifs in trained RNNs initialized with orthogonal recurrent connectivity matrices.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modeling long-term dependencies with recurrent neural networks (RNNs) is a hard problem due to degeneracies inherent in the optimization landscapes of these models, a problem also known as the vanishing/exploding gradients problem (Hochreiter, 1991; Bengio et al., 1994).
Revised: Modeling long-term dependencies with recurrent neural networks (RNNs) is a hard problem due to degeneracies inherent in the optimization landscapes of these models, a problem also known as the vanishing/exploding gradients problem (Hochreiter, 1991; Bengio et al., 1994).

Original: One approach to addressing this problem has been designing new RNN architectures that are less prone to such difﬁculties, hence are better able to capture long-term dependencies in sequential data (Hochreiter &
Revised: One approach to addressing this problem has been designing new RNN architectures that are less prone to such difﬁculties, hence are better able to capture long-term dependencies in sequential data (Hochreiter &

Original: Schmidhuber, 1997; Cho et al., 2014; Chang et al., 2017; Bai et al., 2018).
Revised: Schmidhuber, 1997; Cho et al., 2014; Chang et al., 2017; Bai et al., 2018).

Original: An alternative approach is to stick with the basic vanilla RNN architecture instead, but to constrain its dynamics in some way so as to eliminate or reduce the degeneracies that otherwise afﬂict the optimization landscape.
Revised: An alternative approach is to stick with the basic vanilla RNN architecture instead, but to constrain its dynamics in some way so as to eliminate or reduce the degeneracies that otherwise afﬂict the optimization landscape.

Original: Previous proposals belonging to this second category generally boil down to orthogonalization of the recurrent dynamics, either at initialization or during the entire training period (Le et al., 2015; Arjovsky et al., 2016;
Revised: Previous proposals belonging to this second category generally boil down to orthogonalization of the recurrent dynamics, either at initialization or during the entire training period (Le et al., 2015; Arjovsky et al., 2016;

Original: Wisdom et al., 2016).
Revised: Wisdom et al., 2016).

Original: The basic idea behind these methods is that orthogonal transformations are isometries of the Euclidean space, hence they preserve distances and norms, which enables them to deal effectively with the vanishing/exploding gradients problem.
Revised: The basic idea behind these methods is that orthogonal transformations are isometries of the Euclidean space, hence they preserve distances and norms, which enables them to deal effectively with the vanishing/exploding gradients problem.

Original: However, this idea ignores the crucial effects of non-linearity and noise .
Revised: However, this idea ignores the crucial effects of non-linearity and noise .

Original: Orthogonal transformations no longer preserve distances and norms in the presence of a non-linearity, suggesting that alternative transformations might be better suited to non-linear networks (this point was noted by Pennington et al.
Revised: Orthogonal transformations no longer preserve distances and norms in the presence of a non-linearity, suggesting that alternative transformations might be better suited to non-linear networks (this point was noted by Pennington et al.

Original: (2017) and Chen et al.
Revised: (2017) and Chen et al.

Original: (2018) before, where isometric initializations that take the non-linearity into account were proposed).
Revised: (2018) before, where isometric initializations that take the non-linearity into account were proposed).

Original: Similarly, in the presence of noise, norm preservation itself ceases to be the ideal objective.
Revised: Similarly, in the presence of noise, norm preservation itself ceases to be the ideal objective.

Original: One must instead maximize the signal-to-noise ratio (SNR) of the propagated signal.
Revised: One must instead maximize the signal-to-noise ratio (SNR) of the propagated signal.

Original: In neural networks, noise comes in both through the stochasticity of the stochastic gradient descent (SGD) algorithm and sometimes also through direct noise injection for regularization purposes, as in dropout (Srivastava et al., 2014).
Revised: In neural networks, noise comes in both through the stochasticity of the stochastic gradient descent (SGD) algorithm and sometimes also through direct noise injection for regularization purposes, as in dropout (Srivastava et al., 2014).

Original: Previous work has shown that even in a simple linear setting, recurrent networks that maximize the SNR display strongly non-normal, sequential dynamics and orthogonal networks are highly suboptimal by this measure (Ganguli et al., 2008).
Revised: Previous work has shown that even in a simple linear setting, recurrent networks that maximize the SNR display strongly non-normal, sequential dynamics and orthogonal networks are highly suboptimal by this measure (Ganguli et al., 2008).

Original: Motivated by these observations, in this paper, we investigate the potential of non-normal RNNs, i.e. RNNs with a non-normal recurrent connectivity matrix, in sequential processing tasks.
Revised: Motivated by these observations, in this paper, we investigate the potential of non-normal RNNs, i.e. RNNs with a non-normal recurrent connectivity matrix, in sequential processing tasks.

Original: Recall that a normal matrix is a matrix with an orthonormal set of eigenvectors, whereas a non-normal matrix does not have an orthonormal set of eigenvectors.
Revised: Recall that a normal matrix is a matrix with an orthonormal set of eigenvectors, whereas a non-normal matrix does not have an orthonormal set of eigenvectors.

Original: This property allows non-normal systems to display interesting transient behaviors that are not available in normal systems.
Revised: This property allows non-normal systems to display interesting transient behaviors that are not available in normal systems.

Original: This kind of transient behavior, speciﬁcally a particular kind of transient ampliﬁcation of the signal in certain non-normal systems, underlies their superior memory properties (Ganguli et al., 2008), as will be discussed further below.
Revised: This kind of transient behavior, speciﬁcally a particular kind of transient ampliﬁcation of the signal in certain non-normal systems, underlies their superior memory properties (Ganguli et al., 2008), as will be discussed further below.

Original: Our empirical results show that non-normal vanilla RNNs signiﬁcantly outperform their orthogonal counterparts in a diverse range of benchmarks.
Revised: Our empirical results show that non-normal vanilla RNNs signiﬁcantly outperform their orthogonal counterparts in a diverse range of benchmarks.

Original:  B ACKGROUND 2. M EMORY IN LINEAR RECURRENT NETWORKS WITH NOISE
Revised:  B ACKGROUND 2. M EMORY IN LINEAR RECURRENT NETWORKS WITH NOISE

Original: Ganguli et al.
Revised: Ganguli et al.

Original: (2008) studied memory properties of linear recurrent networks injected with a scalar temporal signal s t , and noise z t :
Revised: (2008) studied memory properties of linear recurrent networks injected with a scalar temporal signal s t , and noise z t :

Original: [Equation]
Revised: [Equation]

Original: The noise is assumed to be i.i.d.
Revised: The noise is assumed to be i.i.d.

Original: with z t ∼ N (0 , I ) .
Revised: with z t ∼ N (0 , I ) .

Original: Ganguli et al.
Revised: Ganguli et al.

Original: (2008) then analyzed the Fisher memory matrix (FMM) of this system, deﬁned as:
Revised: (2008) then analyzed the Fisher memory matrix (FMM) of this system, deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: For linear networks with Gaussian noise, it is easy to show that J kl ( s ≤ t ) is, in fact, independent of the past signal history s ≤ t .
Revised: For linear networks with Gaussian noise, it is easy to show that J kl ( s ≤ t ) is, in fact, independent of the past signal history s ≤ t .

Original: Ganguli et al.
Revised: Ganguli et al.

Original: (2008) speciﬁcally analyzed the diagonal of the FMM:
Revised: (2008) speciﬁcally analyzed the diagonal of the FMM:

Original: J ( k ) ≡
Revised: J ( k ) ≡

Original: J kk , which can be written explicitly as:
Revised: J kk , which can be written explicitly as:

Original: [Equation]
Revised: [Equation]

Original: where C = (cid:80) ∞ k =0 W k W k (cid:62) is the noise covariance matrix, and the norm of W k v can be roughly thought of as representing the signal strength.
Revised: where C = (cid:80) ∞ k =0 W k W k (cid:62) is the noise covariance matrix, and the norm of W k v can be roughly thought of as representing the signal strength.

Original: The total Fisher memory is the sum of J ( k ) over all past time steps k :
Revised: The total Fisher memory is the sum of J ( k ) over all past time steps k :

Original: [Equation]
Revised: [Equation]

Original: Intuitively, J ( k ) measures the information contained in the current state of the system, h t , about a signal that entered the system k time steps ago, s t − k .
Revised: Intuitively, J ( k ) measures the information contained in the current state of the system, h t , about a signal that entered the system k time steps ago, s t − k .

Original: J tot is then a measure of the total information contained in the current state of the system about the entire past signal history, s ≤ t .
Revised: J tot is then a measure of the total information contained in the current state of the system about the entire past signal history, s ≤ t .

Original: The main result in Ganguli et al.
Revised: The main result in Ganguli et al.

Original: (2008) shows that J tot = 1 for all normal matrices W (including all orthogonal matrices), whereas in general J tot ≤ N , where N is the network size.
Revised: (2008) shows that J tot = 1 for all normal matrices W (including all orthogonal matrices), whereas in general J tot ≤ N , where N is the network size.

Original: Remarkably, the memory upper bound can be achieved by certain highly non-normal systems and several examples are explicitly given in Ganguli et al.
Revised: Remarkably, the memory upper bound can be achieved by certain highly non-normal systems and several examples are explicitly given in Ganguli et al.

Original: Two of those examples are illustrated in Figure 1a (right): a uni-directional “chain” network and a chain network with feedback.
Revised: Two of those examples are illustrated in Figure 1a (right): a uni-directional “chain” network and a chain network with feedback.

Original: In the chain network, the recurrent connectivity is given by W ij = αδ j,i − 1 and in the chain with feedback network, it is given by W ij = αδ j,i − 1 + βδ j,i +1 , where α and β are the feedforward and feedback connection weights, respectively (here δ denotes the Kronecker delta function).
Revised: In the chain network, the recurrent connectivity is given by W ij = αδ j,i − 1 and in the chain with feedback network, it is given by W ij = αδ j,i − 1 + βδ j,i +1 , where α and β are the feedforward and feedback connection weights, respectively (here δ denotes the Kronecker delta function).

Original: In addition, in order to achieve optimal memory, the signal must be fed at the source neuron in these networks, i.e. v = [1 , 0 , 0 , . . .
Revised: In addition, in order to achieve optimal memory, the signal must be fed at the source neuron in these networks, i.e. v = [1 , 0 , 0 , . . .

Original: , 0] (cid:62) .
Revised: , 0] (cid:62) .

Original: Figure 1b compares the Fisher memory curves, J ( k ) , of these non-normal networks with the Fisher memory curves of two example normal networks, namely recurrent networks with identity or random orthogonal connectivity matrices.
Revised: Figure 1b compares the Fisher memory curves, J ( k ) , of these non-normal networks with the Fisher memory curves of two example normal networks, namely recurrent networks with identity or random orthogonal connectivity matrices.

Original: The two non-normal networks have extensive memory capacity, i.e.
Revised: The two non-normal networks have extensive memory capacity, i.e.

Original: J tot ∼ O ( N ) , whereas for the normal examples, J tot = 1 .
Revised: J tot ∼ O ( N ) , whereas for the normal examples, J tot = 1 .

Original: The crucial property that enables extensive memory in non-normal networks is transient ampliﬁcation : after the signal enters the network, it is ampliﬁed supralinearly for a time of length
Revised: The crucial property that enables extensive memory in non-normal networks is transient ampliﬁcation : after the signal enters the network, it is ampliﬁed supralinearly for a time of length

Original: O ( N ) before it eventually dies out (Figure 1c).
Revised: O ( N ) before it eventually dies out (Figure 1c).

Original: This kind of transient ampliﬁcation is not possible in normal networks.
Revised: This kind of transient ampliﬁcation is not possible in normal networks.

Original: [Figure]
Revised: [Figure]

Original: 2. A TOY NON - LINEAR EXAMPLE : N ON - LINEARITY AND NOISE INDUCE SIMILAR EFFECTS
Revised: 2. A TOY NON - LINEAR EXAMPLE : N ON - LINEARITY AND NOISE INDUCE SIMILAR EFFECTS

Original: The preceding analysis by Ganguli et al.
Revised: The preceding analysis by Ganguli et al.

Original: (2008) is exact in linear networks.
Revised: (2008) is exact in linear networks.

Original: Analysis becomes more difﬁcult in the presence of a non-linearity.
Revised: Analysis becomes more difﬁcult in the presence of a non-linearity.

Original: However, we now demonstrate that the non-normal networks shown in Figure 1a have advantages that extend beyond the linear case.
Revised: However, we now demonstrate that the non-normal networks shown in Figure 1a have advantages that extend beyond the linear case.

Original: The advantages in the non-linear case are due to reduced interference in these non-normal networks between signals entering the network at different time points in the past.
Revised: The advantages in the non-linear case are due to reduced interference in these non-normal networks between signals entering the network at different time points in the past.

Original: To demonstrate this with a simple example, we will ignore the effect of noise for now and consider the effect of non-linearity on the linear decodability of past signals from the current network activity.
Revised: To demonstrate this with a simple example, we will ignore the effect of noise for now and consider the effect of non-linearity on the linear decodability of past signals from the current network activity.

Original: We thus consider deterministic non-linear networks of the form (see Appendix A for additional details):
Revised: We thus consider deterministic non-linear networks of the form (see Appendix A for additional details):

Original: [Equation]
Revised: [Equation]

Original: (5) and ask how well we can linearly decode a signal that entered the network k time steps ago, s t − k , from the current activity of the network, h t .
Revised: (5) and ask how well we can linearly decode a signal that entered the network k time steps ago, s t − k , from the current activity of the network, h t .

Original: Figure 2c compares the decoding performance in a non-linear orthogonal network with the decoding performance in the non-linear chain network.
Revised: Figure 2c compares the decoding performance in a non-linear orthogonal network with the decoding performance in the non-linear chain network.

Original: Just as in the linear case with noise (Figure 2b), the chain network outperforms the orthogonal network.
Revised: Just as in the linear case with noise (Figure 2b), the chain network outperforms the orthogonal network.

Original: To understand intuitively why this is the case, consider a chain network with W ij = δ j,i − 1 and v =
Revised: To understand intuitively why this is the case, consider a chain network with W ij = δ j,i − 1 and v =

Original: , 0] (cid:62) .
Revised: , 0] (cid:62) .

Original: In this model, the responses of the N neurons after N time steps (at t = N ) are given by f ( s N ) , f
Revised: In this model, the responses of the N neurons after N time steps (at t = N ) are given by f ( s N ) , f

Original: ( f ( s N − 1 )) , ..., f ( f ( . . .
Revised: ( f ( s N − 1 )) , ..., f ( f ( . . .

Original: f ( s 1 ) . . . )) , respectively, starting from the source neuron.
Revised: f ( s 1 ) . . . )) , respectively, starting from the source neuron.

Original: Although the non-linearity f ( · ) makes perfect linear decoding of the past signal s t − k impossible, one may still imagine being able to decode the past signal with reasonable accuracy as long as f ( · ) is not “too non-linear”.
Revised: Although the non-linearity f ( · ) makes perfect linear decoding of the past signal s t − k impossible, one may still imagine being able to decode the past signal with reasonable accuracy as long as f ( · ) is not “too non-linear”.

Original: A similar intuition holds for the chain network with feedback as well, as long as the feedforward connection weight, α , is sufﬁciently stronger than the feedback connection strength, β .
Revised: A similar intuition holds for the chain network with feedback as well, as long as the feedforward connection weight, α , is sufﬁciently stronger than the feedback connection strength, β .

Original: A condition like this must already be satisﬁed if the network is to maintain its optimal memory properties and also be dynamically stable at the same time (Ganguli et al., 2008).
Revised: A condition like this must already be satisﬁed if the network is to maintain its optimal memory properties and also be dynamically stable at the same time (Ganguli et al., 2008).

Original: In normal networks, however, linear decoding is further degraded by interference from signals entering the network at different time points, in addition to the degradation caused by the nonlinearity.
Revised: In normal networks, however, linear decoding is further degraded by interference from signals entering the network at different time points, in addition to the degradation caused by the nonlinearity.

Original: This is easiest to see in the identity network (a similar argument holds for the random orthogonal example too), where the responses of the neurons after N time steps are identically given
Revised: This is easiest to see in the identity network (a similar argument holds for the random orthogonal example too), where the responses of the neurons after N time steps are identically given

Original: O r t
Revised: O r t

Original: h
Revised: h

Original: o g o n a l
Revised: o g o n a l

Original: C h a i n a b c by f ( f ( . . .
Revised: C h a i n a b c by f ( f ( . . .

Original: f ( f ( s 1 )
Revised: f ( f ( s 1 )

Original: + s 2 ) . . . )
Revised: + s 2 ) . . . )

Original: + s N ), if one assumes v = [
Revised: + s N ) assumes v = [

Original: 1] (cid:62) .
Revised: 1] (cid:62) .

Original: is

Original:  R ESULTS 3. E XPERIMENTS
Revised:  R ESULTS 3. E XPERIMENTS

Original: Because assuming an a priori ﬁxed non-normal structure for an RNN runs the risk of being too restrictive, in this paper, we instead explore the promise of non-normal networks as initializers for RNNs.
Revised: Because assuming an a priori ﬁxed non-normal structure for an RNN runs the risk of being too restrictive, in this paper, we instead explore the promise of non-normal networks as initializers for RNNs.

Original: Throughout the paper, we will be primarily comparing the four RNN architectures schematically depicted in Figure 1a as initializers: two of them normal networks (identity and random orthogonal) and the other two non-normal networks (chain and chain with feedback), the last two being motivated by their optimal memory properties in the linear case, as reviewed above.
Revised: Throughout the paper, we will be primarily comparing the four RNN architectures schematically depicted in Figure 1a as initializers: two of them normal networks (identity and random orthogonal) and the other two non-normal networks (chain and chain with feedback), the last two being motivated by their optimal memory properties in the linear case, as reviewed above.

Original: 3.1. C OPY , ADDITION , PERMUTED SEQUENTIAL MNIST
Revised: 3.1. C OPY , ADDITION , PERMUTED SEQUENTIAL MNIST

Original: Copy, addition,
Revised: Copy, addition,

Original: and
Revised: and

Original: permuted sequential MNIST tasks were commonly used as benchmarks in previous RNN studies (Arjovsky et al., 2016;
Revised: permuted sequential MNIST tasks were commonly used as benchmarks in previous RNN studies (Arjovsky et al., 2016;

Original: Bai et al., 2018; Chang et al., 2017; Hochreiter &
Revised: Bai et al., 2018; Chang et al., 2017; Hochreiter &

Original: Schmidhuber, 1997; Le et al., 2015; Wisdom et al., 2016).
Revised: Schmidhuber, 1997; Le et al., 2015; Wisdom et al., 2016).

Original: We now brieﬂy describe each of these tasks.
Revised: We now brieﬂy describe each of these tasks.

Original: Copy task: The input is a sequence of integers of length T .
Revised: Copy task: The input is a sequence of integers of length T .

Original: The ﬁrst 10 integers in the sequence deﬁne the target subsequence that is to be copied and consist of integers between 1 and 8 (inclusive).
Revised: The ﬁrst 10 integers in the sequence deﬁne the target subsequence that is to be copied and consist of integers between 1 and 8 (inclusive).

Original: The next T − 21 integers are set to 0 .
Revised: The next T − 21 integers are set to 0 .

Original: The integer after that is set to 9 , which acts as the cue indicating that the model should start copying the target subsequence.
Revised: The integer after that is set to 9 , which acts as the cue indicating that the model should start copying the target subsequence.

Original: The ﬁnal 10 integers are set to 0 .
Revised: The ﬁnal 10 integers are set to 0 .

Original: The output sequence that the model is trained to reproduce consists of T − 10 0 s followed by the target subsequence from the input that is to be copied.
Revised: The output sequence that the model is trained to reproduce consists of T − 10 0 s followed by the target subsequence from the input that is to be copied.

Original: To make sure that the task requires a sufﬁciently long memory capacity, we used a large sequence length, T = 500 , comparable to the largest sequence length considered in Arjovsky et al.
Revised: To make sure that the task requires a sufﬁciently long memory capacity, we used a large sequence length, T = 500 , comparable to the largest sequence length considered in Arjovsky et al.

Original: for the same task.
Revised: for the same task.

Original: Addition task: The input consists of two sequences of length T .
Revised: Addition task: The input consists of two sequences of length T .

Original: The ﬁrst one is a sequence of random numbers drawn uniformly from the interval [0 , 1] .
Revised: The ﬁrst one is a sequence of random numbers drawn uniformly from the interval [0 , 1] .

Original: The second sequence is an indicator sequence with 1 s at exactly two positions and 0 s everywhere else.
Revised: The second sequence is an indicator sequence with 1 s at exactly two positions and 0 s everywhere else.

Original: The positions of the two 1 s indicate the positions of the numbers to be added in the ﬁrst sequence.
Revised: The positions of the two 1 s indicate the positions of the numbers to be added in the ﬁrst sequence.

Original: The target output is the sum of the two corresponding numbers.
Revised: The target output is the sum of the two corresponding numbers.

Original: The position of the ﬁrst 1 is drawn uniformly from the ﬁrst half of the sequence and the position of the second 1 is drawn uniformly from the second half of the sequence.
Revised: The position of the ﬁrst 1 is drawn uniformly from the ﬁrst half of the sequence and the position of the second 1 is drawn uniformly from the second half of the sequence.

Original: Again, to ensure that the task requires a sufﬁciently long memory capacity, we chose T = 750 , which is the same as the largest sequence length considered in Arjovsky et al.
Revised: Again, to ensure that the task requires a sufﬁciently long memory capacity, we chose T = 750 , which is the same as the largest sequence length considered in Arjovsky et al.

Original: for the same task.
Revised: for the same task.

Original: Permuted sequential MNIST (psMNIST):
Revised: Permuted sequential MNIST (psMNIST):

Original: This is a sequential version of the standard MNIST benchmark where the pixels are fed to the model one pixel at a time.
Revised: This is a sequential version of the standard MNIST benchmark where the pixels are fed to the model one pixel at a time.

Original: To make the task hard enough, we used the permuted version of the sequential MNIST task where a ﬁxed random permutation is applied to the pixels to eliminate any spatial structure before they are fed into the model.
Revised: To make the task hard enough, we used the permuted version of the sequential MNIST task where a ﬁxed random permutation is applied to the pixels to eliminate any spatial structure before they are fed into the model.

Original: We used vanilla RNNs with N = 25 recurrent units in the psMNIST task and N = 100 recurrent units in the copy and addition tasks.
Revised: We used vanilla RNNs with N = 25 recurrent units in the psMNIST task and N = 100 recurrent units in the copy and addition tasks.

Original: We used the elu nonlinearity for the copy and the psMNIST tasks (Clevert et al., 2016), and the relu nonlinearity for the addition problem (because relu proved to be more natural for remembering positive numbers).
Revised: We used the elu nonlinearity for the copy and the psMNIST tasks (Clevert et al., 2016), and the relu nonlinearity for the addition problem (because relu proved to be more natural for remembering positive numbers).

Original: Batch size was 16 in all tasks.
Revised: Batch size was 16 in all tasks.

Original: As mentioned above, the scaled identity and the scaled random orthogonal networks constituted the normal initializers.
Revised: As mentioned above, the scaled identity and the scaled random orthogonal networks constituted the normal initializers.

Original: In the scaled identity initializer, the recurrent connectivity matrix was initialized as
Revised: In the scaled identity initializer, the recurrent connectivity matrix was initialized as

Original: W = λ
Revised: W = λ

Original: I and the input matrix V was initialized as V ij ∼ N (0 , 0 . 9 / √ N ) .
Revised: I and the input matrix V was initialized as V ij ∼ N (0 , 0 . 9 / √ N ) .

Original: In the random orthogonal initializer, the recurrent connectivity matrix was initialized as W = λ Q , where Q is a random dense orthogonal matrix, and the input matrix V was initialized in the same way as in the identity initializer.
Revised: In the random orthogonal initializer, the recurrent connectivity matrix was initialized as W = λ Q , where Q is a random dense orthogonal matrix, and the input matrix V was initialized in the same way as in the identity initializer.

Original: The feedforward chain and the chain with feedback networks constituted our non-normal initializers.
Revised: The feedforward chain and the chain with feedback networks constituted our non-normal initializers.

Original: In the chain initializer, the recurrent connectivity matrix was initialized as W ij = αδ j,i − 1 and the input matrix V was initialized as V ∼ 0 .
Revised: In the chain initializer, the recurrent connectivity matrix was initialized as W ij = αδ j,i − 1 and the input matrix V was initialized as V ∼ 0 .

Original: 9 I N × d , where I N × d denotes the N × d -dimensional identity matrix.
Revised: 9 I N × d , where I N × d denotes the N × d -dimensional identity matrix.

Original: Note that this choice of V is a natural generalization of the the source injecting input vector that was found to be optimal in the linear case with scalar signals to multi-dimensional inputs (as long as N (cid:29) d ).
Revised: Note that this choice of V is a natural generalization of the the source injecting input vector that was found to be optimal in the linear case with scalar signals to multi-dimensional inputs (as long as N (cid:29) d ).

Original: In the chain with feedback initializer, the recurrent connectivity matrix was initialized as
Revised: In the chain with feedback initializer, the recurrent connectivity matrix was initialized as

Original: W ij = 0 .
Revised: W ij = 0 .

Original: 99 δ j,i − 1 + βδ j,i +

Revised: 99 δ j,i − 1 + βδ j,i +

Original: and the input matrix V was initialized in the same way as in the chain initializer.
Revised: and the input matrix V was initialized in the same way as in the chain initializer.

Original: We used the rmsprop optimizer for all models, which we found to be the best method for this set of tasks.
Revised: We used the rmsprop optimizer for all models, which we found to be the best method for this set of tasks.

Original: The learning rate of the optimizer was a hyperparameter which we tuned separately for each model and each task.
Revised: The learning rate of the optimizer was a hyperparameter which we tuned separately for each model and each task.

Original: The following learning rates were considered in the hyper-parameter search: 8 × 10 − 4 , 5 × 10 − 4 , 3 × 10 − 4 , 10 − 4 , 8 × 10 − 5 , 5 × 10 − 5 , 3 × 10 − 5 , 10 − 5 , 8 × 10 − 6 , 5 × 10 − 6 , 3 × 10 − 6 .
Revised: The following learning rates were considered in the hyper-parameter search: 8 × 10 − 4 , 5 × 10 − 4 , 3 × 10 − 4 , 10 − 4 , 8 × 10 − 5 , 5 × 10 − 5 , 3 × 10 − 5 , 10 − 5 , 8 × 10 − 6 , 5 × 10 − 6 , 3 × 10 − 6 .

Original: We ran each model on each task 6 times using the integers from 1 to 6 as random seeds.
Revised: We ran each model on each task 6 times using the integers from 1 to 6 as random seeds.

Original: In addition, the following model-speciﬁc hyperparameters were searched over for each task:
Revised: In addition, the following model-speciﬁc hyperparameters were searched over for each task:

Original: Chain: feedforward connection weight, α ∈ { 0 . 99 , 1 . 00 , 1 . 01 , 1 . 02 , 1 . 03 , 1 . 04 , 1 . 05 } .
Revised: Chain: feedforward connection weight, α ∈ { 0 . 99 , 1 . 00 , 1 . 01 , 1 . 02 , 1 . 03 , 1 . 04 , 1 . 05 } .

Original: Chain with feedback: feedback connection weight, β ∈ { 0 . 01 , 0 . 02 , 0 .
Revised: Chain with feedback: feedback connection weight, β ∈ { 0 . 01 , 0 . 02 , 0 .

Original: [Table]
Revised: [Table]

Original: This yields a total of 7 × 11 × 6 = 462 different runs for each experiment in the non-normal models and a total of 9 × 11 × 6 = 594 different runs in the normal models.
Revised: This yields a total of 7 × 11 × 6 = 462 different runs for each experiment in the non-normal models and a total of 9 × 11 × 6 = 594 different runs in the normal models.

Original: Note that we ran more extensive hyper-parameter searches for the normal models than for the non-normal models in this set of tasks.
Revised: Note that we ran more extensive hyper-parameter searches for the normal models than for the non-normal models in this set of tasks.

Original: Figure 3a-c shows the validation losses for each model with the best hyper-parameter settings.
Revised: Figure 3a-c shows the validation losses for each model with the best hyper-parameter settings.

Original: The non-normal initializers generally outperform the normal initializers.
Revised: The non-normal initializers generally outperform the normal initializers.

Original: Figure 3d-f shows for each model the number of “successful” runs that converged to a validation loss below a criterion level (which we set to be 50% of the loss for a baseline random model).
Revised: Figure 3d-f shows for each model the number of “successful” runs that converged to a validation loss below a criterion level (which we set to be 50% of the loss for a baseline random model).

Original: The chain model outperformed all other models by this measure (despite having a smaller total number of runs than the normal models).
Revised: The chain model outperformed all other models by this measure (despite having a smaller total number of runs than the normal models).

Original: In the copy task, for example, none of the runs for the normal models was able to achieve the criterion level, whereas 46 out of 462 runs for the chain model and 11 out of 462 runs for the feedback chain model reached the criterion loss (see Appendices B & C for further results and discussion).
Revised: In the copy task, for example, none of the runs for the normal models was able to achieve the criterion level, whereas 46 out of 462 runs for the chain model and 11 out of 462 runs for the feedback chain model reached the criterion loss (see Appendices B & C for further results and discussion).

Original: 3.1. L ANGUAGE MODELING EXPERIMENTS
Revised: 3.1. L ANGUAGE MODELING EXPERIMENTS

Original: [Figure]
Revised: [Figure]

Original: To investigate if the beneﬁts of nonnormal initializers extend to more realistic problems, we conducted experiments with three standard language modeling tasks: word level Penn Treebank (PTB), character level PTB, and character level benchmarks
Revised: To investigate if the beneﬁts of nonnormal initializers extend to more realistic problems, we conducted experiments with three standard language modeling tasks: word-level Penn Treebank (PTB), character-level PTB, and character-level enwik8 benchmarks.

Original: For the language modeling experiments in this subsection, we used the code base provided by Salesforce Research (Merity et al., 2018a;b): https://github.com/ salesforce/awd-lstm-lm .
Revised: For the language modeling experiments in this subsection, we used the code base provided by Salesforce Research (Merity et al., 2018a;b): https://github.com/ salesforce/awd-lstm-lm .

Original: We refer the reader to Merity et al.
Revised: We refer the reader to Merity et al.

Original: (2018a;b) for a more detailed description of the benchmarks.
Revised: (2018a;b) for a more detailed description of the benchmarks.

Original: For the experiments in this subsection, we generally preserved the model setup used in Merity et al.
Revised: For the experiments in this subsection, we generally preserved the model setup used in Merity et al.

Original: (2018a;b), except for the following differences: 1)
Revised: (2018a;b), except for the following differences: 1)

Original: We replaced the gated RNN architectures (LSTMs and QRNNs) used in Merity et al.
Revised: We replaced the gated RNN architectures (LSTMs and QRNNs) used in Merity et al.

Original: (2018a;b) with vanilla RNNs; 2) We observed that vanilla RNNs require weaker regularization than gated RNN architectures.
Revised: (2018a;b) with vanilla RNNs; 2) We observed that vanilla RNNs require weaker regularization than gated RNN architectures.

Original: Therefore, in the word-level PTB task, we set all dropout rates to 0 .
Revised: Therefore, in the word-level PTB task, we set all dropout rates to 0 .

Original: In the character-level PTB task, all dropout rates except dropoute were set to 0 .
Revised: In the character-level PTB task, all dropout rates except dropoute were set to 0 .

Original: 1 , which was set to 0 .
Revised: 1 , which was set to 0 .

Original: In the enwik8 benchmark, all dropout rates were set to 0 ; 3) We trained the word-level PTB models for 60 epochs, the character-level PTB models for 500 epochs and the enwik models for 
Revised: In the enwik8 benchmark, all dropout rates were set to 0 ; 3) We trained the word-level PTB models for 60 epochs, the character-level PTB models for 500 epochs and the enwik8 models for 35 epochs.

Original: 35 epochs.

Original: We compared the same four models described in the previous subsection.
Revised: We compared the same four models described in the previous subsection.

Original: As in Merity et al.
Revised: As in Merity et al.

Original: (2018a)
Revised: (2018a)

Original: , we used the Adam optimizer and thus only optimized the α , β , λ hyper-parameters for the experiments in this subsection.
Revised: , we used the Adam optimizer and thus only optimized the α , β , λ hyper-parameters for the experiments in this subsection.

Original: For the hyper-parameter α in the chain model and the hyper-parameter λ in the scaled identity and random orthogonal models, we searched over 21 values uniformly spaced between 0 . 05 and 1 .
Revised: For the hyper-parameter α in the chain model and the hyper-parameter λ in the scaled identity and random orthogonal models, we searched over 21 values uniformly spaced between 0 . 05 and 1 .

Original: 05 (inclusive); whereas for the chain with feedback model, we set the feedforward connection weight, α , to the optimal value it had in the chain model and searched over 21 β values uniformly spaced between 0 . 01 and 0 .
Revised: 05 (inclusive); whereas for the chain with feedback model, we set the feedforward connection weight, α , to the optimal value it had in the chain model and searched over 21 β values uniformly spaced between 0 . 01 and 0 .

Original: 21 (inclusive).
Revised: 21 (inclusive).

Original: In addition, we repeated each experimenttimes using different random seeds, yielding a total of 63 runs for each model and each benchmark.
Revised: In addition, we repeated each experimenttimes using different random seeds, yielding a total of 63 runs for each model and each benchmark.

Original: The results are shown in Figure 4 and in Table 1.
Revised: The results are shown in Figure 4 and in Table 1.

Original: Figure 4 shows the validation loss over the course of training in units of bits per character (bpc).
Revised: Figure 4 shows the validation loss over the course of training in units of bits per character (bpc).

Original: Table 1 reports the test losses at the end of training.
Revised: Table 1 reports the test losses at the end of training.

Original: The non-normal models outperform the normal models on the word-level and character-level PTB benchmarks.
Revised: The non-normal models outperform the normal models on the word-level and character-level PTB benchmarks.

Original: The differences between the models are less clear on the enwik8 benchmark.
Revised: The differences between the models are less clear on the enwik8 benchmark.

Original: However, in terms of the test loss, the non-normal feedback chain model outperforms the other models on all three benchmarks (Table 1).
Revised: However, in terms of the test loss, the non-normal feedback chain model outperforms the other models on all three benchmarks (Table 1).

Original: Epoch 6.
Revised: Epoch 6.

Original: V a l i d a t i o n l o ss ( bp c )
Revised: V a l i d a t i o n l o ss ( bp c )

Original: Identity Orthogonal
Revised: Identity Orthogonal

Original: Chain Fb.
Revised: Chain Fb.

Original: chain a
Revised: chain a

Original: PTB word
Revised: PTB word

Original: Epoch 1.
Revised: Epoch 1.

Original: V a l i d a t i o n l o ss ( bp c ) b
Revised: V a l i d a t i o n l o ss ( bp c ) b

Original: PTB char
Revised: PTB char

Original: Epoch 1.
Revised: Epoch 1.

Original: V a l i d a t i o n l o ss ( bp c ) c enwik8 char
Revised: V a l i d a t i o n l o ss ( bp c ) c enwik8 char

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: We note that the vanilla RNN models perform signiﬁcantly worse than the gated RNN architectures considered in Merity et al.
Revised: We note that the vanilla RNN models perform signiﬁcantly worse than the gated RNN architectures considered in Merity et al.

Original: (2018a;b).
Revised: (2018a;b).

Original: We conjecture that this is because gated architectures are generally better at modeling contextual dependencies, hence they have inductive biases better suited to language modeling tasks.
Revised: We conjecture that this is because gated architectures are generally better at modeling contextual dependencies, hence they have inductive biases better suited to language modeling tasks.

Original: The primary beneﬁt of non-normal dynamics, on the other hand, is enabling a longer memory capacity.
Revised: The primary beneﬁt of non-normal dynamics, on the other hand, is enabling a longer memory capacity.

Original: Below, we will discuss whether non-normal dynamics can be used in gated RNN architectures to improve performance as well.
Revised: Below, we will discuss whether non-normal dynamics can be used in gated RNN architectures to improve performance as well.

Original: 3. H IDDEN FEEDFORWARD STRUCTURES IN TRAINED RNN S
Revised: 3. H IDDEN FEEDFORWARD STRUCTURES IN TRAINED RNN S

Original: We observed that training made vanilla RNNs initialized with orthogonal recurrent connectivity matrices non-normal.
Revised: We observed that training made vanilla RNNs initialized with orthogonal recurrent connectivity matrices non-normal.

Original: We quantiﬁed the non-normality of the trained recurrent connectivity matrices using a measure introduced by Henrici (1962): d ( W ) ≡ (cid:112) (cid:107) W (cid:107) 2F − (cid:80) i | λ i | 2 , where (cid:107) · (cid:107) F denotes the Frobenius norm and λ i is the i -th eigenvalue of W .
Revised: We quantiﬁed the non-normality of the trained recurrent connectivity matrices using a measure introduced by Henrici (1962): d ( W ) ≡ (cid:112) (cid:107) W (cid:107) 2F − (cid:80) i | λ i | 2 , where (cid:107) · (cid:107) F denotes the Frobenius norm and λ i is the i -th eigenvalue of W .

Original: This measure equals 0 for all normal matrices and is positive for non-normal matrices.
Revised: This measure equals 0 for all normal matrices and is positive for non-normal matrices.

Original: We found that d ( W ) became positive for all successfully trained RNNs initialized with orthogonal recurrent connectivity matrices.
Revised: We found that d ( W ) became positive for all successfully trained RNNs initialized with orthogonal recurrent connectivity matrices.

Original: Table 2 reports the aggregate statistics of d ( W ) for orthogonally initialized RNNs trained on the toy benchmarks.
Revised: Table 2 reports the aggregate statistics of d ( W ) for orthogonally initialized RNNs trained on the toy benchmarks.

Original: Although increased non-normality in trained RNNs is an interesting observation, the Henrici index, by itself, does not tell us what structural features in trained RNNs contribute to this increased non-normality.
Revised: Although increased non-normality in trained RNNs is an interesting observation, the Henrici index, by itself, does not tell us what structural features in trained RNNs contribute to this increased non-normality.

Original: Given the beneﬁts of chain-like feedforward non-normal structures in RNNs for improved memory, we hypothesized that training might have installed hidden chain-like feedforward structures in trained RNNs and that these feedforward structures were responsible for their increased non-normality.
Revised: Given the beneﬁts of chain-like feedforward non-normal structures in RNNs for improved memory, we hypothesized that training might have installed hidden chain-like feedforward structures in trained RNNs and that these feedforward structures were responsible for their increased non-normality.

Original: To uncover these hidden feedforward structures, we performed an analysis suggested by Rajan et al.
Revised: To uncover these hidden feedforward structures, we performed an analysis suggested by Rajan et al.

Original: In this analysis, we ﬁrst injected a unit pulse of input to the network at the beginning of the trial and let the network evolve for 100 time steps afterwards according to its recurrent dynamics with no direct input.
Revised: In this analysis, we ﬁrst injected a unit pulse of input to the network at the beginning of the trial and let the network evolve for 100 time steps afterwards according to its recurrent dynamics with no direct input.

Original: We then ordered the recurrent units by the time of their peak activity (using a small amount of jitter to break potential ties between units) and plotted the mean recurrent connection(Figure
Revised: We then ordered the recurrent units by the time of their peak activity (using a small amount of jitter to break potential ties between units) and plotted the mean recurrent connection

Original: [Table]
Revised: [Table]

Original: weights, W ij , as a function of the order difference between two units, i − j .
Revised: weights, W ij , as a function of the order difference between two units, i − j .

Original: Positive i − j values correspond to connections from earlier peaking units to later peaking units and vice versa for negative
Revised: Positive i − j values correspond to connections from earlier peaking units to later peaking units and vice versa for negative

Original: i − j values.
Revised: i − j values.

Original: In trained RNNs the mean recurrent weight proﬁle as a function of i − j had an asymmetric peak, with connections in the “forward” direction being, on average, stronger than those inthe oppositedirection.
Revised: In trained RNNs the mean recurrent weight proﬁle as a function of i − j had an asymmetric peak, with connections in the “forward” direction being, on average, stronger than those in opposite

Original: Figure 5 shows examples with orthogonally initializedRNNs trained on theaddition and the permutedsequential MNIST tasks.
Revised: Figure 5 shows examples with orthogonally initialized trained on the and the permuted MNIST tasks.

Original: Note that for a purely feedforward chain, the weight proﬁle would have a single peak at i − j = 1 and would be zero elsewhere.
Revised: Note that for a purely feedforward chain the weight proﬁle would have a single peak at i − j = 1 and would be zero elsewhere.

Original: Although the weight proﬁles for trained RNNs are not this extreme, the prominent asymmetric bump with a peak at a positive i − j value indicates a hidden chain-like feedforward structure in these networks
Revised: Although the weight proﬁles for trained RNNs are not this extreme, the prominent asymmetric bump with a peak at a positive i − j value indicates a hidden chain-like feedforward structure in these networks

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: D O BENEFITS OF NON - NORMAL DYNAMICS EXTEND TO GATED RNN ARCHITECTURES ?
Revised: D O BENEFITS OF NON - NORMAL DYNAMICS EXTEND TO GATED RNN ARCHITECTURES ?

Original: So far
Revised: So far

Original: [Table]
Revised: [Table]

Original: we have only considered vanilla RNNs.
Revised: we have only considered vanilla RNNs.

Original: An important question is whether the beneﬁts of non-normal dynamics demonstrated above for vanilla RNNs also extend to gated RNN architectures like LSTMs or GRUs (Hochreiter & Schmidhuber, 1997; Cho et al., 2014).
Revised: An important question is whether the beneﬁts of non-normal dynamics demonstrated above for vanilla RNNs also extend to gated RNN architectures like LSTMs or GRUs (Hochreiter & Schmidhuber, 1997; Cho et al., 2014).

Original: Gated RNN architectures have better inductive biases than vanilla RNNs in many practical tasks of interest such as language modeling (e.g. see Table 1 for a comparison of vanilla RNN architectures with an LSTM architecture of similar size in the language modeling benchmarks) [Table] thus it would be practically very useful if their performance could be improved through an inductive bias for non-normal dynamics.
Revised: Gated RNN architectures have better inductive biases than vanilla RNNs in many practical tasks of interest such as language modeling (e.g. see Table 1 for a comparison of vanilla RNN architectures with an LSTM architecture of similar size in the language modeling benchmarks), thus it would be practically very useful if their performance could be improved through an inductive bias for non-normal dynamics.

Original: To address this question
Revised: To address this question

Original: [Table]
Revised: [Table]

Original: we treated the input, forget, output
Revised: we treated the input [Table] forget, output

Original: [Table]
Revised: [Table]

Original: and update gates of the LSTM architecture as analogous to vanilla RNNs and initialized the recurrent and input matrices inside these gates in the same way as in the chain or the orthogonal initialization of vanilla RNNs above.
Revised: and update gates of the LSTM architecture as analogous to vanilla RNNs and initialized the recurrent and input matrices inside these gates in the same way as in the chain or the orthogonal initialization of vanilla RNNs above.

Original: We also compared these with a more standard initialization scheme where all the weights were drawn from a uniform distribution U ( −√ k, √ k ) where k is the reciprocal of the hidden layer size (labeled plain in Table 3).
Revised: We also compared these with a more standard initialization scheme where all the weights were drawn from a uniform distribution U ( −√ k, √ k ) where k is the reciprocal of the hidden layer size (labeled plain in Table 3).

Original: This is the default initializer for the LSTM weight matrices in PyTorch: https://pytorch.org/docs/stable/nn.html#lstm .
Revised: This is the default initializer for the LSTM weight matrices in PyTorch: https://pytorch.org/docs/stable/nn.html#lstm .

Original: We compared these initializers in the language modeling benchmarks.
Revised: We compared these initializers in the language modeling benchmarks.

Original: The chain initializer did not perform better than the orthogonal initializer (Table 3), suggesting that non-normal dynamics in gated RNN architectures may not be as helpful as it is in vanilla RNNs.
Revised: The chain initializer did not perform better than the orthogonal initializer (Table 3), suggesting that non-normal dynamics in gated RNN architectures may not be as helpful as it is in vanilla RNNs.

Original: In hindsight, this is not too surprising, because our initial motivation for introducing non-normal dynamics heavily relied on the vanilla RNN architecture and gated RNNs can be dynamically very different from vanilla RNNs.
Revised: In hindsight, this is not too surprising, because our initial motivation for introducing non-normal dynamics heavily relied on the vanilla RNN architecture and gated RNNs can be dynamically very different from vanilla RNNs.

Original: When we looked at the trained LSTM weight matrices more closely, we found that, although still non-normal, the recurrent weight matrices inside the input, forget, and output gates (i.e. the sigmoid gates) did not have the same signatures of hidden chain-like feedforward structures observed in vanilla RNNs.
Revised: When we looked at the trained LSTM weight matrices more closely, we found that, although still non-normal, the recurrent weight matrices inside the input, forget, and output gates (i.e. the sigmoid gates) did not have the same signatures of hidden chain-like feedforward structures observed in vanilla RNNs.

Original: Speciﬁcally, the weight proﬁles in the LSTM recurrent weight matrices inside these three gates did not display the asymmetric bump characteristic of a prominent chain-like feedforward structure, but were instead approximately monotonic functions of i − j
Revised: Speciﬁcally, the weight proﬁles in the LSTM recurrent weight matrices inside these three gates did not display the asymmetric bump characteristic of a prominent chain-like feedforward structure, but were instead approximately monotonic functions of i − j

Original: (Figure 6a-c), suggesting a qualitatively different kind of dynamics where the individual units are more persistent over time.
Revised: (Figure 6a-c), suggesting a qualitatively different kind of dynamics where the individual units are more persistent over time.

Original: The recurrent weight matrix inside the update gate (the tanh gate), on the other hand, did display the signature of a hidden chain-like feedforward structure (Figure 6d).
Revised: The recurrent weight matrix inside the update gate (the tanh gate), on the other hand, did display the signature of a hidden chain-like feedforward structure (Figure 6d).

Original: When we incorporated these two structures in different gates of the LSTMs, by using a chain initializer for the update gate and a monotonically increasing recurrent weight proﬁle for the other gates (labeled mixed in Table 3), the resulting initializer outperformed the other initializers on character-level PTB and enwik8 tasks.
Revised: When we incorporated these two structures in different gates of the LSTMs, by using a chain initializer for the update gate and a monotonically increasing recurrent weight proﬁle for the other gates (labeled mixed in Table 3), the resulting initializer outperformed the other initializers on character-level PTB and enwik8 tasks.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Motivated by their optimal memory properties in a simpliﬁed linear setting (Ganguli et al., 2008), in this paper, we investigated the potential beneﬁts of certain highly non-normal chain-like RNN architectures in capturing long-term dependencies in sequential tasks.
Revised: Motivated by their optimal memory properties in a simpliﬁed linear setting (Ganguli et al., 2008), in this paper, we investigated the potential beneﬁts of certain highly non-normal chain-like RNN architectures in capturing long-term dependencies in sequential tasks.

Original: Our results demonstrate an advantage for such non-normal architectures as initializers for vanilla RNNs, compared to the commonly used orthogonal initializers.
Revised: Our results demonstrate an advantage for such non-normal architectures as initializers for vanilla RNNs, compared to the commonly used orthogonal initializers.

Original: We further found evidence for the induction of such chainlike feedforward structures in trained vanilla RNNs even when these RNNs were initialized with orthogonal recurrent connectivity matrices.
Revised: We further found evidence for the induction of such chainlike feedforward structures in trained vanilla RNNs even when these RNNs were initialized with orthogonal recurrent connectivity matrices.

Original: -  1149 i − j -0.
Revised: -  1149 i − j -0.

Original: M e a n r e c .
Revised: M e a n r e c .

Original: w e i g h t
Revised: w e i g h t

Original: ( W I i j )
Revised: ( W I i j )

Original: TrainedUntrained
Revised: TrainedUntrained

Original: [Figure]
Revised: [Figure]

Original: gate1149 i − j -0.
Revised: gate1149 i − j -0.

Original: M e a n r e c .
Revised: M e a n r e c .

Original: w e i g h t ( W F i j ) b
Revised: w e i g h t ( W F i j ) b

Original: Forget gate1149 i − j -0.
Revised: Forget gate1149 i − j -0.

Original: M e a n r e c .
Revised: M e a n r e c .

Original: w e i g h t
Revised: w e i g h t

Original: ( W U i j ) d
Revised: ( W U i j ) d

Original: [Figure]
Revised: [Figure]

Original: gate1149 i

Original: [Figure]
Revised: [Figure]

Original: j -0.

Original: [Figure][Figure]
Revised: [Figure]

Original: r

Original: [Figure]
Revised: [Figure]

Original:  layers and  trained on  display 
Revised: Other layers and models trained on other tasks display qualitatively similar properties.

Original: The beneﬁts of these chain-like non-normal initializers do not directly carry over to more complex, gated RNN architectures such as LSTMs and GRUs.
Revised: The beneﬁts of these chain-like non-normal initializers do not directly carry over to more complex, gated RNN architectures such as LSTMs and GRUs.

Original: In some important practical problems such as language modeling, the gains from using these kinds of gated architectures seem to far outweigh the gains obtained from the non-normal initializers in vanilla RNNs (see Table 1).
Revised: In some important practical problems such as language modeling, the gains from using these kinds of gated architectures seem to far outweigh the gains obtained from the non-normal initializers in vanilla RNNs (see Table 1).

Original: However, we also uncovered important regularities in trained LSTM weight matrices, namely that the recurrent weight proﬁles of the input, forget, and output gates (the sigmoid gates) in trained LSTMs display a monotonically increasing pattern, whereas the recurrent matrix inside the update gate (the tanh gate) displays a chain-like feedforward structure similar to that observed in vanilla RNNs (Figure 6).
Revised: However, we also uncovered important regularities in trained LSTM weight matrices, namely that the recurrent weight proﬁles of the input, forget, and output gates (the sigmoid gates) in trained LSTMs display a monotonically increasing pattern, whereas the recurrent matrix inside the update gate (the tanh gate) displays a chain-like feedforward structure similar to that observed in vanilla RNNs (Figure 6).

Original: We showed that these regularities can be exploited to improve the training and/or generalization performance of gated RNN architectures by introducing them as useful inductive biases.
Revised: We showed that these regularities can be exploited to improve the training and/or generalization performance of gated RNN architectures by introducing them as useful inductive biases.

Original: A concurrent work to ours also emphasized the importance of non-normal dynamics in RNNs (Kerg et al., 2019).
Revised: A concurrent work to ours also emphasized the importance of non-normal dynamics in RNNs (Kerg et al., 2019).

Original: The main difference between Kerg et al.
Revised: The main difference between Kerg et al.

Original: and our work is that we explicitly introduce sequential motifs in RNNs at initialization as a useful inductive bias for improved long-term memory (motivated by the optimal memory properties of these motifs in simpler cases), whereas their approach does not constrain the shape of the non-normal part of the recurrent connectivity matrix, hence does not utilize sequential non-normal dynamics as an inductive bias.
Revised: and our work is that we explicitly introduce sequential motifs in RNNs at initialization as a useful inductive bias for improved long-term memory (motivated by the optimal memory properties of these motifs in simpler cases), whereas their approach does not constrain the shape of the non-normal part of the recurrent connectivity matrix, hence does not utilize sequential non-normal dynamics as an inductive bias.

Original: In some of their tasks,
Revised: In some of their tasks,

Original: Kerg et al.
Revised: Kerg et al.

Original: (2019) also uncovered a feedforward, chain-like motif in trained vanilla RNNs similar to the one reported in this paper (Figure 5).
Revised: (2019) also uncovered a feedforward, chain-like motif in trained vanilla RNNs similar to the one reported in this paper (Figure 5).

Original: There is a close connection between the identity initialization of RNNs (Le et al., 2015) and the widely used identity skip connections (or residual connections) in deep feedforward networks (He et al., 2016).
Revised: There is a close connection between the identity initialization of RNNs (Le et al., 2015) and the widely used identity skip connections (or residual connections) in deep feedforward networks (He et al., 2016).

Original: Given the superior performance of chain-like non-normal initializers over the identity initialization demonstrated in the context of vanilla RNNs in this paper, it could be interesting to look for similar chain-like non-normal architectural motifs that could be used in deep feedforward networks in place of the identity skip connections.
Revised: Given the superior performance of chain-like non-normal initializers over the identity initialization demonstrated in the context of vanilla RNNs in this paper, it could be interesting to look for similar chain-like non-normal architectural motifs that could be used in deep feedforward networks in place of the identity skip connections.

Original: A D ETAILS AND EXTENSIONS OF THE LINEAR DECODING EXPERIMENTS
Revised: A D ETAILS AND EXTENSIONS OF THE LINEAR DECODING EXPERIMENTS

Original: This appendix contains the details of the linear decoding experiments in section 2.2 and reports the results of additional linear decoding experiments.
Revised: This appendix contains the details of the linear decoding experiments in section 2.2 and reports the results of additional linear decoding experiments.

Original: The experiments in section 2.2 compare the signal propagation properties of vanilla RNNs with either random orthogonal or chain connectivity matrices.
Revised: The experiments in section 2.2 compare the signal propagation properties of vanilla RNNs with either random orthogonal or chain connectivity matrices.

Original: In both cases, the overall scale of the recurrent connectivity matrices is set to 1 .
Revised: In both cases, the overall scale of the recurrent connectivity matrices is set to 1 .

Original: The input weight vector is v = [1 , 0 , 0 , . .
Revised: The input weight vector is v = [1 , 0 , 0 , . .

Original: , 0] (cid:62) for the chain model and v i ∼ N (0 , 1 / √ n ) for the random orthogonal model (thus the overall scales of both the feedforward and the recurrent inputs are identical in the two models).
Revised: , 0] (cid:62) for the chain model and v i ∼ N (0 , 1 / √ n ) for the random orthogonal model (thus the overall scales of both the feedforward and the recurrent inputs are identical in the two models).

Original: The RNNs themselves are not trained in these experiments.
Revised: The RNNs themselves are not trained in these experiments.

Original: At each time point, an i.i.d.
Revised: At each time point, an i.i.d.

Original: random scalar signal s t ∼ N (0 , 1) is fed into the network as input (Equation 5).
Revised: random scalar signal s t ∼ N (0 , 1) is fed into the network as input (Equation 5).

Original: We simulate 250 trials for each model and ask how well we can linearly decode the signal at the ﬁrst time step, s 1 , from the recurrent activities at time step 100, h 100 .
Revised: We simulate 250 trials for each model and ask how well we can linearly decode the signal at the ﬁrst time step, s 1 , from the recurrent activities at time step 100, h 100 .

Original: We do this by linearly regressing s 1 on h(using the 250 simulated samples) and report the R 2 value for the linear regression in Figure 2.
Revised: We do this by linearly regressing s 1 on h(using the 250 simulated samples) and report the R 2 value for the linear regression in Figure 2.

Original: In simulations with noise (Figure 2b), an additional i.i.d.
Revised: In simulations with noise (Figure 2b), an additional i.i.d.

Original: random noise term, z it ∼ N (0 , σ ) , is added to each recurrent neuron i at each time step t .
Revised: random noise term, z it ∼ N (0 , σ ) , is added to each recurrent neuron i at each time step t .

Original: The standard deviation of the noise, σ , is set to 0 .in
Revised: The standard deviation of the noise, σ , is set to 0 .in

Original: the experiments shown in Figure 2b.
Revised: the experiments shown in Figure 2b.

Original: To show that the results are not sensitive to the noise scale, we ran additional experiments with lower ( σ = 0 . 01 ) and higher ( σ = 1 ) levels of noise (Figure 7).
Revised: To show that the results are not sensitive to the noise scale, we ran additional experiments with lower ( σ = 0 . 01 ) and higher ( σ = 1 ) levels of noise (Figure 7).

Original: In both cases, the chain network still outperforms the orthogonal network.
Revised: In both cases, the chain network still outperforms the orthogonal network.

Original: Note that these “linear + noise” experiments satisfy the conditions of the analytical theory in Ganguli et al.
Revised: Note that these “linear + noise” experiments satisfy the conditions of the analytical theory in Ganguli et al.

Original: (2008), so these results are as expected from the theory.
Revised: (2008), so these results are as expected from the theory.

Original: As mentioned in the main text, the “non-linear + no noise” experiments reported in Figure 2c used the elu non-linearity.
Revised: As mentioned in the main text, the “non-linear + no noise” experiments reported in Figure 2c used the elu non-linearity.

Original: To show that the results are not sensitive to the choice of the non-linearity, we also ran additional experiments with tanh and relu non-linearities (Figure 8).
Revised: To show that the results are not sensitive to the choice of the non-linearity, we also ran additional experiments with tanh and relu non-linearities (Figure 8).

Original: As with the elu non-linearity, the chain network outperforms the orthogonal network with the tanh and relu non-linearities as well, suggesting that the results are not sensitive to the choice of the non-linearity.
Revised: As with the elu non-linearity, the chain network outperforms the orthogonal network with the tanh and relu non-linearities as well, suggesting that the results are not sensitive to the choice of the non-linearity.

Original: B T HE EFFECT OF THE FEEDBACK STRENGTH PARAMETER ( β ) IN THE CHAIN WITH FEEDBACK MODEL
Revised: B T HE EFFECT OF THE FEEDBACK STRENGTH PARAMETER ( β ) IN THE CHAIN WITH FEEDBACK MODEL

Original: In this appendix, we consider the effect of the feedback strength parameter, β , for the chain with feedback model in the context of the experiments reported in section 3.1.1.
Revised: In this appendix, we consider the effect of the feedback strength parameter, β , for the chain with feedback model in the context of the experiments reported in section 3.1.1.

Original: We focus on the psMNIST task speciﬁcally, because this is the only task where the feedback chain model converges to a low loss solution for a sufﬁciently large number of hyper-parameter conﬁgurations.
Revised: We focus on the psMNIST task speciﬁcally, because this is the only task where the feedback chain model converges to a low loss solution for a sufﬁciently large number of hyper-parameter conﬁgurations.

Original: For the addition and copy tasks, there are not enough successful hyper-parameter conﬁgurations to draw reliable inferences about the effect of β (see Figure 3d-f).
Revised: For the addition and copy tasks, there are not enough successful hyper-parameter conﬁgurations to draw reliable inferences about the effect of β (see Figure 3d-f).

Original: Figure 9 shows the validation loss at the end of training as a function of β in the psMNIST task.
Revised: Figure 9 shows the validation loss at the end of training as a function of β in the psMNIST task.

Original: In this ﬁgure, we considered all networks that achieved a validation loss lower than the random baseline model (i.e. < log(10) ≈ 2 . 3 ) at the end of training (an overwhelming majority of the networks satisﬁed this criterion).
Revised: In this ﬁgure, we considered all networks that achieved a validation loss lower than the random baseline model (i.e. < log(10) ≈ 2 . 3 ) at the end of training (an overwhelming majority of the networks satisﬁed this criterion).

Original: Figure 9 shows that the ﬁnal validation loss is a monotonically increasing function of β in this task, suggesting that large feedback strengths are harmful for the model performance.
Revised: Figure 9 shows that the ﬁnal validation loss is a monotonically increasing function of β in this task, suggesting that large feedback strengths are harmful for the model performance.

Original: C C OMPARISON WITH PREVIOUS MODELS
Revised: C C OMPARISON WITH PREVIOUS MODELS

Original: In this appendix, we compare our results with those obtained by previous models, focusing speciﬁcally on the experiments in section 3.1.1 (because the tasks in this section are commonly used as RNN benchmarks).
Revised: In this appendix, we compare our results with those obtained by previous models, focusing speciﬁcally on the experiments in section 3.1.1 (because the tasks in this section are commonly used as RNN benchmarks).

Original: a b
Revised: a b

Original: O r t
Revised: O r t

Original: h
Revised: h

Original: o g o n a l
Revised: o g o n a l

Original: C h a i n uRNN: We ﬁrst note that our copy and addition tasks use the largest sequence lengths considered in
Revised: C h a i n uRNN: We ﬁrst note that our copy and addition tasks use the largest sequence lengths considered in

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: for the same tasks ( T = 500 for the copy task and T
Revised: for the same tasks ( T = 500 for the copy task and T

Original: = 750 for the addition task).
Revised: = 750 for the addition task).

Original: Hence our results are directly comparable to those reported in Arjovsky et al.
Revised: Hence our results are directly comparable to those reported in Arjovsky et al.

Original: (the random baselines shown by the dashed lines in Figure 3a-b are identical to those in Arjovsky et al.
Revised: (the random baselines shown by the dashed lines in Figure 3a-b are identical to those in Arjovsky et al.

Original: (2016) for the same conditions).
Revised: (2016) for the same conditions).

Original: The unitary evolution RNN (uRNN) model proposed in Arjovsky et al.
Revised: The unitary evolution RNN (uRNN) model proposed in Arjovsky et al.

Original: (2016) comfortably learns the copy-500 task (with 128 recurrent units), quickly reaching a near-zero loss (see their Figure 1, bottom right); however, it struggles with the addition task, barely reaching the half-baseline criterion even with 512 recurrent units (see their Figure 2, bottom right).
Revised: (2016) comfortably learns the copy-500 task (with 128 recurrent units), quickly reaching a near-zero loss (see their Figure 1, bottom right); however, it struggles with the addition task, barely reaching the half-baseline criterion even with 512 recurrent units (see their Figure 2, bottom right).

Original: This difference in the behavior of the uRNN model in the copy and addition tasks is predicted by
Revised: This difference in the behavior of the uRNN model in the copy and addition tasks is predicted by

Original: Henaff et al.
Revised: Henaff et al.

Original: (2016), where it is shown that random orthogonal and near-identity recurrent connectivity matrices have much better inductive biases in the copy and addition tasks, respectively.
Revised: (2016), where it is shown that random orthogonal and near-identity recurrent connectivity matrices have much better inductive biases in the copy and addition tasks, respectively.

Original: Because of its parametrization, uRNN behaves more similarly to a random orthogonal RNN than a near-identity
Revised: Because of its parametrization, uRNN behaves more similarly to a random orthogonal RNN than a near-identity

Original: RNN.
Revised: RNN.

Original: In contrast, our non-normal RNNs, especially the chain model, comfortably clear the half-baseline criterion both in copy-500 and addition-750 tasks (with 100 recurrent units), quickly achieving very small loss values in both tasks with the optimal hyper-parameter conﬁgurations (Figure 3a-b).
Revised: In contrast, our non-normal RNNs, especially the chain model, comfortably clear the half-baseline criterion both in copy-500 and addition-750 tasks (with 100 recurrent units), quickly achieving very small loss values in both tasks with the optimal hyper-parameter conﬁgurations (Figure 3a-b).

Original: Note that this is despite the fact that our models use fewer recurrent units than the uRNN model in Arjovsky et al.
Revised: Note that this is despite the fact that our models use fewer recurrent units than the uRNN model in Arjovsky et al.

Original: (2016) (100 vs. 128 or 512 recurrent units).
Revised: (2016) (100 vs. 128 or 512 recurrent units).

Original: nnRNN:
Revised: nnRNN:

Original: Kerg et al.
Revised: Kerg et al.

Original: (2019) report results for the copy ( T = 200 ) and psMNIST tasks only.
Revised: (2019) report results for the copy ( T = 200 ) and psMNIST tasks only.

Original: They have not reported training success for longer variants of the copy task (speciﬁcally for T = 500 ).
Revised: They have not reported training success for longer variants of the copy task (speciﬁcally for T = 500 ).

Original: Kerg et al.
Revised: Kerg et al.

Original: (2019) also have not reported successful training in the addition task, whereas our non-normal
Revised: (2019) also have not reported successful training in the addition task, whereas our non-normal

Original: RNNs showed training success both in copy-500 and addition-750 tasks (Figure 3a-b).
Revised: RNNs showed training success both in copy-500 and addition-750 tasks (Figure 3a-b).

Original: We conclude that our non-normal initializers for vanilla RNNs perform comparably to, or better than, the uRNN and nnRNN models in standard long-term memory benchmarks.
Revised: We conclude that our non-normal initializers for vanilla RNNs perform comparably to, or better than, the uRNN and nnRNN models in standard long-term memory benchmarks.

Original: One of the biggest strengths of our proposal compared to these previous models is its much greater simplicity.
Revised: One of the biggest strengths of our proposal compared to these previous models is its much greater simplicity.

Original: Both uRNN and nnRNN require a complete re-parametrization of the vanilla RNN model (nnRNN even
Revised: Both uRNN and nnRNN require a complete re-parametrization of the vanilla RNN model (nnRNN even

Original: O r t
Revised: O r t

Original: h
Revised: h

Original: o g o n a l
Revised: o g o n a l

Original: C h a i n a b
Revised: C h a i n a b

Original: [Figure]
Revised: [Figure]

Original: . chain psMNIST
Revised: ss Fb. chain psMNIST

Original: requires a novel optimization method).
Revised: requires a novel optimization method).

Original: Our method, on the other hand, proposes much simpler, easy-to that keep the standard of RNNs intact.
Revised: Our method, on the other hand, proposes much simpler, easy-to- type initializers that keep the standard of RNNs intact.

Original: critical RNN: Chen et al.
Revised: critical RNN: Chen et al.

Original: (2018) note that the conditions for dynamical isometry in vanilla RNNs are identical to those in fully-connected feed-forward networks studied in Pennington et al.
Revised: (2018) note that the conditions for dynamical isometry in vanilla RNNs are identical to those in fully-connected feed-forward networks studied in Pennington et al.

Original: Pennington et al.
Revised: Pennington et al.

Original: (2017), in turn, note that dynamical isometry is not achievable exactly in networks with relu activation, but it is achievable in networks with tanh activation, where it essentially boils down to initializing the weights to small values.
Revised: (2017), in turn, note that dynamical isometry is not achievable exactly in networks with relu activation, but it is achievable in networks with tanh activation, where it essentially boils down to initializing the weights to small values.

Original: Pennington et al.
Revised: Pennington et al.

Original: (2017) give a speciﬁc example of a dynamically isometric tanh network (with n = 400 , σ w = 1 . 05 , and σ b = 2 .
Revised: (2017) give a speciﬁc example of a dynamically isometric tanh network (with n = 400 , σ w = 1 . 05 , and σ b = 2 .

Original: 01 × 10 − 5 ).
Revised: 01 × 10 − 5 ).

Original: We set up a similar tanh RNN model, but were not able to train it successfully in the copy or addition tasks.
Revised: We set up a similar tanh RNN model, but were not able to train it successfully in the copy or addition tasks.

Original: Again, as with the nnRNN results, this shows the challenging nature of these two tasks and suggests that dynamical isometry may not be enough for successful training in these tasks.
Revised: Again, as with the nnRNN results, this shows the challenging nature of these two tasks and suggests that dynamical isometry may not be enough for successful training in these tasks.

Original: A possible reason for this is that although critical initialization takes the non-linearity into account, it still does not take the noise into account (i.e. it is not guaranteed to maximize the SNR).
Revised: A possible reason for this is that although critical initialization takes the non-linearity into account, it still does not take the noise into account (i.e. it is not guaranteed to maximize the SNR).

Original: LSTM, tanh RNN:
Revised: LSTM, tanh RNN:

Original: Consistent with the results in Arjovsky et al.
Revised: Consistent with the results in Arjovsky et al.

Original: , we were not able to successfully train LSTMs or vanilla RNNs with tanh non-linearity in the challenging copy-500 and addition-750 tasks.
Revised: , we were not able to successfully train LSTMs or vanilla RNNs with tanh non-linearity in the challenging copy-500 and addition-750 tasks.

Original: Therefore, these models were not included as baselines in section 3.1.1.
Revised: Therefore, these models were not included as baselines in section 3.1.1.

Original: [Figure]
Revised: [Figure]

Original: Epoch

Original: [Figure]
Revised: [Figure]

Original: IdentityOrthogonalChainFb .
Revised: IdentityOrthogonalChainFb .

Original: chain psMNIST
Revised: chain psMNIST