Original: A M AX -A FFINE S PLINE P ERSPECTIVE OF R ECURRENT N EURAL N ETWORKS
Revised: A M AX -A FFINE S PLINE P ERSPECTIVE OF R ECURRENT N EURAL N ETWORKS

Original: A BSTRACT
Revised: A BSTRACT

Original: We develop a framework for understanding and improving recurrent neural networks (RNNs) using max-afﬁne spline operators (MASOs).
Revised: We develop a framework for understanding and improving recurrent neural networks (RNNs) using max-afﬁne spline operators (MASOs).

Original: We prove that RNNs using piecewise afﬁne and convex nonlinearities can be written as a simple piecewise afﬁne spline operator.
Revised: We prove that RNNs using piecewise afﬁne and convex nonlinearities can be written as a simple piecewise afﬁne spline operator.

Original: The resulting representation provides several new perspectives for analyzing RNNs, three of which we study in this paper.
Revised: The resulting representation provides several new perspectives for analyzing RNNs, three of which we study in this paper.

Original: First, we show that an RNN internally partitions the input space during training and that it builds up the partition through time.
Revised: First, we show that an RNN internally partitions the input space during training and that it builds up the partition through time.

Original: Second, we show that the afﬁne slope parameter of an RNN corresponds to an input-speciﬁc template, from which we can interpret an RNN as performing a simple template matching (matched ﬁltering) given the input.
Revised: Second, we show that the afﬁne slope parameter of an RNN corresponds to an input-speciﬁc template, from which we can interpret an RNN as performing a simple template matching (matched ﬁltering) given the input.

Original: Third, by carefully examining the MASO RNN afﬁne mapping, we prove that using a random initial hidden state corresponds to an explicit (cid:96) 2 regularization of the afﬁne parameters, which can mollify exploding gradients and improve generalization.
Revised: Third, by carefully examining the MASO RNN afﬁne mapping, we prove that using a random initial hidden state corresponds to an explicit (cid:96) 2 regularization of the afﬁne parameters, which can mollify exploding gradients and improve generalization.

Original: Extensive experiments on several datasets of various modalities demonstrate and validate each of the above conclusions.
Revised: Extensive experiments on several datasets of various modalities demonstrate and validate each of the above conclusions.

Original: In particular, using a random initial hidden states elevates simple RNNs to near state-of-the-art performers on these datasets.
Revised: In particular, using a random initial hidden states elevates simple RNNs to near state-of-the-art performers on these datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recurrent neural networks (RNNs) are a powerful class of models for processing sequential inputs and a basic building block for more advanced models that have found success in challenging problems involving sequential data, including sequence classiﬁcation (e.g., sentiment analysis (Socher et al., 2013) , sequence generation (e.g., machine translation (Bahdanau et al., 2014)),
Revised: Recurrent neural networks (RNNs) are a powerful class of models for processing sequential inputs and a basic building block for more advanced models that have found success in challenging problems involving sequential data, including sequence classiﬁcation (e.g., sentiment analysis (Socher et al., 2013) , sequence generation (e.g., machine translation (Bahdanau et al., 2014)),

Original: speech recognition (Graves et al., 2013) and image captioning (Mao et al., 2015).
Revised: speech recognition (Graves et al., 2013) and image captioning (Mao et al., 2015).

Original: Despite their success, however, our understanding of how RNNs work remains limited.
Revised: Despite their success, however, our understanding of how RNNs work remains limited.

Original: For instance, an attractive theoretical result is the universal approximation property that states that an RNN can approximate an arbitrary function (Sch ¨ afer & Zimmermann, 2006; Siegelmann & Sontag, 1995; Hammer, 2000).
Revised: For instance, an attractive theoretical result is the universal approximation property that states that an RNN can approximate an arbitrary function (Sch ¨ afer & Zimmermann, 2006; Siegelmann & Sontag, 1995; Hammer, 2000).

Original: These classical theoretical results have been obtained primarily from the dynamical system (Siegelmann & Sontag, 1995; Sch¨afer & Zimmermann, 2006) and measure theory (Hammer, 2000) perspectives.
Revised: These classical theoretical results have been obtained primarily from the dynamical system (Siegelmann & Sontag, 1995; Sch¨afer & Zimmermann, 2006) and measure theory (Hammer, 2000) perspectives.

Original: These theories provide approximation error bounds but unfortunately limited guidance on applying RNNs and understanding their performance and behavior in practice.
Revised: These theories provide approximation error bounds but unfortunately limited guidance on applying RNNs and understanding their performance and behavior in practice.

Original: In this paper, we provide a new angle for understanding RNNs using max-afﬁne spline operators (MASOs) (Magnani & Boyd, 2009; Hannah & Dunson, 2013) from approximation theory.
Revised: In this paper, we provide a new angle for understanding RNNs using max-afﬁne spline operators (MASOs) (Magnani & Boyd, 2009; Hannah & Dunson, 2013) from approximation theory.

Original: The piecewise afﬁne approximations made by compositions of MASOs provide a new and useful framework to study neural networks.
Revised: The piecewise afﬁne approximations made by compositions of MASOs provide a new and useful framework to study neural networks.

Original: For example, Balestriero & Baraniuk (2018); Balestriero & Baraniuk (2018a) have provided a detailed analysis in the context of feedforward networks.
Revised: For example, Balestriero & Baraniuk (2018); Balestriero & Baraniuk (2018a) have provided a detailed analysis in the context of feedforward networks.

Original: Here, we go one step further and ﬁnd new insights and interpretations from the MASO perspective for RNNs.
Revised: Here, we go one step further and ﬁnd new insights and interpretations from the MASO perspective for RNNs.

Original: We will see that the input space partitioning and matched ﬁltering links developed in Balestriero &
Revised: We will see that the input space partitioning and matched ﬁltering links developed in Balestriero &

Original: Baraniuk (2018); Balestriero & Baraniuk (2018a) extend to RNNs and yield interesting insights into their inner workings.
Revised: Baraniuk (2018); Balestriero & Baraniuk (2018a) extend to RNNs and yield interesting insights into their inner workings.

Original: Moreover, the MASO formulation of RNNs enables us to theoretically justify the use of a random initial hidden state to improve RNN performance.
Revised: Moreover, the MASO formulation of RNNs enables us to theoretically justify the use of a random initial hidden state to improve RNN performance.

Original: For concreteness, we focus our analysis on a speciﬁc class of simple RNNs (Elman, 1990) with piecewise afﬁne and convex nonlinearities such as the ReLU (Glorot et al., 2011).
Revised: For concreteness, we focus our analysis on a speciﬁc class of simple RNNs (Elman, 1990) with piecewise afﬁne and convex nonlinearities such as the ReLU (Glorot et al., 2011).

Original: RNNs with h (2 , 2) h (2 , 1) h (2 , 3) h (1 , 2) h (1 , 1) h (1 , 3)
Revised: RNNs with h (2 , 2) h (2 , 1) h (2 , 3) h (1 , 2) h (1 , 1) h (1 , 3)

Original: W (2) r W (2) r
Revised: W (2) r W (2) r

Original: W (1) r
Revised: W (1) r

Original: W (1) r W (1) r W (2)
Revised: W (1) r W (1) r W (2)

Original: W (2)
Revised: W (2)

Original: W (2) h (2 , 0) h (1 , 0)
Revised: W (2) h (2 , 0) h (1 , 0)

Original: W (2) r
Revised: W (2) r

Original: W (1) r x (1) x (2) x (3)
Revised: W (1) r x (1) x (2) x (3)

Original: W (1)
Revised: W (1)

Original: W (1)
Revised: W (1)

Original: W (1) such nonlinearities have recently gained considerable attention due to their ability to combat the exploding gradient problem ; with proper initialization (Le et al., 2015; Talathi & Vartak, 2016) and clever parametrization of the recurrent weight (Arjovsky et al., 2016; Wisdom et al., 2016;
Revised: W (1) such nonlinearities have recently gained considerable attention due to their ability to combat the exploding gradient problem ; with proper initialization (Le et al., 2015; Talathi & Vartak, 2016) and clever parametrization of the recurrent weight (Arjovsky et al., 2016; Wisdom et al., 2016;

Original: Jing et al., 2017; Hyland & R¨atsch, 2017; Mhammedi et al., 2017; Helfrich et al., 2018), these RNNs achieve performance on par with more complex ones such as LSTMs.
Revised: Jing et al., 2017; Hyland & R¨atsch, 2017; Mhammedi et al., 2017; Helfrich et al., 2018), these RNNs achieve performance on par with more complex ones such as LSTMs.

Original: Below is a summary of our key contributions.
Revised: Below is a summary of our key contributions.

Original: Contribution 1.
Revised: Contribution 1.

Original: We prove that an RNN with piecewise afﬁne and convex nonlinearities can be rewritten as a composition of MASOs, making it a piecewise afﬁne spline operator with an elegant analytical form (Section 3).
Revised: We prove that an RNN with piecewise afﬁne and convex nonlinearities can be rewritten as a composition of MASOs, making it a piecewise afﬁne spline operator with an elegant analytical form (Section 3).

Original: Contribution 2.
Revised: Contribution 2.

Original: We leverage the partitioning of piecewise afﬁne spline operators to analyze the input space partitioning that an RNN implicitly performs.
Revised: We leverage the partitioning of piecewise afﬁne spline operators to analyze the input space partitioning that an RNN implicitly performs.

Original: We show that an RNN calculates a new, high-dimensional representation (the partition code) of the input sequence that captures informative underlying characteristics of the input.
Revised: We show that an RNN calculates a new, high-dimensional representation (the partition code) of the input sequence that captures informative underlying characteristics of the input.

Original: We also provide a new perspective on RNN dynamics by visualizing the evolution of the RNN input space partitioning through time (Section 4).
Revised: We also provide a new perspective on RNN dynamics by visualizing the evolution of the RNN input space partitioning through time (Section 4).

Original: Contribution 3.
Revised: Contribution 3.

Original: We show the piecewise afﬁne mapping in an RNN associated with a given input sequence corresponds to an input-dependent template, from which we can interpret the RNN as performing greedy template matching (matched ﬁltering) at every RNN cell (Section 5).
Revised: We show the piecewise afﬁne mapping in an RNN associated with a given input sequence corresponds to an input-dependent template, from which we can interpret the RNN as performing greedy template matching (matched ﬁltering) at every RNN cell (Section 5).

Original: Contribution 4.
Revised: Contribution 4.

Original: We rigorously prove that using a random (rather than zero) initial hidden state in an RNN corresponds to an explicit regularizer that can mollify exploding gradients.
Revised: We rigorously prove that using a random (rather than zero) initial hidden state in an RNN corresponds to an explicit regularizer that can mollify exploding gradients.

Original: We show empirically that such a regularization improves RNN performance (to state-of-the-art) on four datasets of different modalities (Section 6).
Revised: We show empirically that such a regularization improves RNN performance (to state-of-the-art) on four datasets of different modalities (Section 6).

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Recurrent Neural Networks (RNNs).
Revised: Recurrent Neural Networks (RNNs).

Original: A simple RNN unit (Elman, 1990) per layer (cid:96) and time step t , referred to as a “cell,” performs the following recursive computation
Revised: A simple RNN unit (Elman, 1990) per layer (cid:96) and time step t , referred to as a “cell,” performs the following recursive computation

Original: [Equation]
Revised: [Equation]

Original: where h ( (cid:96),t ) is the hidden state at layer (cid:96) and time step t , h (0 ,t ) :
Revised: where h ( (cid:96),t ) is the hidden state at layer (cid:96) and time step t , h (0 ,t ) :

Original: = x ( t ) which is the input sequence, σ is an activation function and W ( (cid:96) ) , W ( (cid:96) ) r , and b ( (cid:96) ) are time-invariant parameters at layer (cid:96) .
Revised: = x ( t ) which is the input sequence, σ is an activation function and W ( (cid:96) ) , W ( (cid:96) ) r , and b ( (cid:96) ) are time-invariant parameters at layer (cid:96) .

Original: h ( (cid:96), 0) is the initial hidden state at layer (cid:96) which needs to be set to some value beforehand to start the
Revised: h ( (cid:96), 0) is the initial hidden state at layer (cid:96) which needs to be set to some value beforehand to start the

Original: RNN recursive computation.
Revised: RNN recursive computation.

Original: Unrolling the RNN through time gives an intuitive view of the RNN dynamics, which we visualize in Figure 1.
Revised: Unrolling the RNN through time gives an intuitive view of the RNN dynamics, which we visualize in Figure 1.

Original: The output of the overall RNN is typically an afﬁne transformation of the hidden state of the last layer L at time step t
Revised: The output of the overall RNN is typically an afﬁne transformation of the hidden state of the last layer L at time step t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In the special case where the RNN has only one output at the end of processing the entire input sequence, the RNN output is an afﬁne transformation of the hidden state at the last time step, i.e., z ( T ) =
Revised: In the special case where the RNN has only one output at the end of processing the entire input sequence, the RNN output is an afﬁne transformation of the hidden state at the last time step, i.e., z ( T ) =

Original: W h ( L,T ) + b .
Revised: W h ( L,T ) + b .

Original: Max-Afﬁne Spline Operators (MASOs).
Revised: Max-Afﬁne Spline Operators (MASOs).

Original: A max-afﬁne spline operator (MASO) is piecewise afﬁne and convex with respect to each output dimension
Revised: A max-afﬁne spline operator (MASO) is piecewise afﬁne and convex with respect to each output dimension

Original: k
Revised: k

Original: = 1 , . .
Revised: = 1 , . .

Original: , K .
Revised: , K .

Original: It is deﬁned as a parametric function S : R D
Revised: It is deﬁned as a parametric function S : R D

Original: → R K with parameters A ∈ R K × R × D and B ∈
Revised: → R K with parameters A ∈ R K × R × D and B ∈

Original: R K × R .
Revised: R K × R .

Original: A MASO leverages K independent max-afﬁne splines (Magnani & Boyd, 2009), each with R partition regions.
Revised: A MASO leverages K independent max-afﬁne splines (Magnani & Boyd, 2009), each with R partition regions.

Original: Its output for output dimension k is produced via
Revised: Its output for output dimension k is produced via

Original: [Equation]
Revised: [Equation]

Original: where x ∈ R D and y ∈ R K are dummy variables that respectively denote the input and output of the MASO S and (cid:104)· , ·(cid:105) denotes inner product.
Revised: where x ∈ R D and y ∈ R K are dummy variables that respectively denote the input and output of the MASO S and (cid:104)· , ·(cid:105) denotes inner product.

Original: The three subscripts of the “slope” parameter [ A ] k,r,d correspond to output k , partition region r , and input signal index d .
Revised: The three subscripts of the “slope” parameter [ A ] k,r,d correspond to output k , partition region r , and input signal index d .

Original: The two subscripts of the “bias” parameter [ B ] k,r correspond to output k and partition region r .
Revised: The two subscripts of the “bias” parameter [ B ] k,r correspond to output k and partition region r .

Original: We highlight two important and interrelated MASO properties relevant to the discussions throughout the paper.
Revised: We highlight two important and interrelated MASO properties relevant to the discussions throughout the paper.

Original: First, a MASO performs implicit input space partitioning, which is made explicit by rewriting (3) as
Revised: First, a MASO performs implicit input space partitioning, which is made explicit by rewriting (3) as

Original: [Equation]
Revised: [Equation]

Original: where Q ∈ R K × R is a partition selection matrix 1 calculated as [ Q ] k,r = 1 ( r = [ r ∗ ] k ) , where [ r ∗ ]
Revised: where Q ∈ R K × R is a partition selection matrix 1 calculated as [ Q ] k,r = 1 ( r = [ r ∗ ] k ) , where [ r ∗ ]

Original: [Equation]
Revised: [Equation]

Original: = arg
Revised: = arg

Original: [Equation]
Revised: [Equation]

Original: Namely, Q contains K stacked one-hot row vectors, each of which selects the [ r ∗ ] th k partition of the input space that maximizes (4) for output dimension k .
Revised: Namely, Q contains K stacked one-hot row vectors, each of which selects the [ r ∗ ] th k partition of the input space that maximizes (4) for output dimension k .

Original: As a consequence, knowing Q is equivalent to knowing the partition of an input x that the MASO implicitly computes.
Revised: As a consequence, knowing Q is equivalent to knowing the partition of an input x that the MASO implicitly computes.

Original: We will use this property in Section 4 to provide new insights into RNN dynamics.
Revised: We will use this property in Section 4 to provide new insights into RNN dynamics.

Original: Second, given the partition r ∗ that an input belongs to, as determined by (5), the output of the MASO of dimension k from (3) reduces to a simple afﬁne transformation of the input [ y ] k
Revised: Second, given the partition r ∗ that an input belongs to, as determined by (5), the output of the MASO of dimension k from (3) reduces to a simple afﬁne transformation of the input [ y ] k

Original: =
Revised: =

Original: [ A ] k, · x + [ B ] k , where [ A ] k, · =
Revised: [ A ] k, · x + [ B ] k , where [ A ] k, · =

Original: [ A ] k, [ r ∗ ]
Revised: [ A ] k, [ r ∗ ]

Original: k and [ B ] k, · = [ B ] k, [ r ∗ ] k .
Revised: k and [ B ] k, · = [ B ] k, [ r ∗ ] k .

Original: Here, the selected afﬁne parameters A ∈ R K × D and B ∈ R K are speciﬁc to the input’s partition region [ r ∗ ] k , which are simply the [ r ∗ ] th k slice and [ r ∗ ] th k column of A and B , respectively, for output dimension k .
Revised: Here, the selected afﬁne parameters A ∈ R K × D and B ∈ R K are speciﬁc to the input’s partition region [ r ∗ ] k , which are simply the [ r ∗ ] th k slice and [ r ∗ ] th k column of A and B , respectively, for output dimension k .

Original: We emphasize that A and B are input-dependent; different inputs x induce different A and B .
Revised: We emphasize that A and B are input-dependent; different inputs x induce different A and B .

Original: We will use this property in Section 5 to link RNNs to matched ﬁlterbanks.
Revised: We will use this property in Section 5 to link RNNs to matched ﬁlterbanks.

Original:  RNN S AS P IECEWISE A FFINE S PLINE O PERATORS
Revised:  RNN S AS P IECEWISE A FFINE S PLINE O PERATORS

Original: We now leverage the MASO framework to rewrite, interpret, and analyze RNNs.
Revised: We now leverage the MASO framework to rewrite, interpret, and analyze RNNs.

Original: We focus on RNNs with piecewise afﬁne and convex nonlinearities in order to derive rigorous analytical results.
Revised: We focus on RNNs with piecewise afﬁne and convex nonlinearities in order to derive rigorous analytical results.

Original: The analysis of RNNs with other nonlinearities is left for future work.
Revised: The analysis of RNNs with other nonlinearities is left for future work.

Original: We ﬁrst derive the MASO formula for an RNN cell (1) and then extend to one layer of a timeunrolled RNN and ﬁnally to a multi-layer, time-unrolled RNN.
Revised: We ﬁrst derive the MASO formula for an RNN cell (1) and then extend to one layer of a timeunrolled RNN and ﬁnally to a multi-layer, time-unrolled RNN.

Original: Let z ( (cid:96),t ) =
Revised: Let z ( (cid:96),t ) =

Original: (cid:2) h ( (cid:96) − 1 ,t ) (cid:62) , h ( (cid:96),t
Revised: (cid:2) h ( (cid:96) − 1 ,t ) (cid:62) , h ( (cid:96),t

Original: [Equation]
Revised: [Equation]

Original: 1) (cid:62) (cid:3) (cid:62) be the input to an RNN cell that is the concatenation of the current input
Revised: 1) (cid:62) (cid:3) (cid:62) be the input to an RNN cell that is the concatenation of the current input

Original: h
Revised: h

Original: ( (cid:96) − 1 ,t ) and the previous hidden state h ( (cid:96),t − 1) .
Revised: ( (cid:96) − 1 ,t ) and the previous hidden state h ( (cid:96),t − 1) .

Original: Then we have the following result, which is a straightforward extension of
Revised: Then we have the following result, which is a straightforward extension of

Original: Proposition 4 in Balestriero & Baraniuk (2018).
Revised: Proposition 4 in Balestriero & Baraniuk (2018).

Original: Proposition 1.
Revised: Proposition 1.

Original: An RNN cell of the form (1) is a MASO with
Revised: An RNN cell of the form (1) is a MASO with

Original: [Equation]
Revised: [Equation]

Original: (7) where A ( (cid:96),t ) =
Revised: (7) where A ( (cid:96),t ) =

Original: A ( (cid:96),t ) σ [ W ( (cid:96) ) , W ( (cid:96) )
Revised: A ( (cid:96),t ) σ [ W ( (cid:96) ) , W ( (cid:96) )

Original: r ] and B ( t ) =
Revised: r ] and B ( t ) =

Original: A ( (cid:96),t ) σ b ( (cid:96) ) are the afﬁne parameters and A ( (cid:96),t ) σ is the afﬁne parameter corresponding to the piecewise afﬁne and convex nonlinearity σ ( · ) that depends on the cell input z
Revised: A ( (cid:96),t ) σ b ( (cid:96) ) are the afﬁne parameters and A ( (cid:96),t ) σ is the afﬁne parameter corresponding to the piecewise afﬁne and convex nonlinearity σ ( · ) that depends on the cell input z

Original: ( (cid:96),t ) .
Revised: ( (cid:96),t ) .

Original: We now derive an explicit afﬁne formula for a time-unrolled RNN at layer (cid:96) .
Revised: We now derive an explicit afﬁne formula for a time-unrolled RNN at layer (cid:96) .

Original: Let h ( (cid:96) − 1) =
Revised: Let h ( (cid:96) − 1) =

Original: (cid:2) h ( (cid:96) − 1 , 1) (cid:62) , · · · , h ( (cid:96) − 1 ,T ) (cid:62) (cid:3) (cid:62) be the entire input sequence to the RNN at layer (cid:96) , and let h ( (cid:96) )
Revised: (cid:2) h ( (cid:96) − 1 , 1) (cid:62) , · · · , h ( (cid:96) − 1 ,T ) (cid:62) (cid:3) (cid:62) be the entire input sequence to the RNN at layer (cid:96) , and let h ( (cid:96) )

Original: = (cid:2) h
Revised: = (cid:2) h

Original: ( (cid:96), 1) (cid:62) , · · · , h ( (cid:96),T ) (cid:62) (cid:3) (cid:62) be all the hidden states that are output at layer (cid:96) .
Revised: ( (cid:96), 1) (cid:62) , · · · , h ( (cid:96),T ) (cid:62) (cid:3) (cid:62) be all the hidden states that are output at layer (cid:96) .

Original: After some algebra and simpliﬁcation, we arrive at the following result.
Revised: After some algebra and simpliﬁcation, we arrive at the following result.

Original: Theorem 1.
Revised: Theorem 1.

Original: The (cid:96) th layer of an RNN is a piecewise afﬁne spline operator deﬁned as
Revised: The (cid:96) th layer of an RNN is a piecewise afﬁne spline operator deﬁned as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:125)
Revised: (cid:125)

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: (cid:96)
Revised: (cid:96)

Original: [Equation]
Revised: [Equation]

Original: T )
Revised: T )

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: (cid:96)
Revised: (cid:96)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1) 
Revised: 1) 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: (8) where A ( (cid:96) )
Revised: (8) where A ( (cid:96) )

Original: t : T (cid:48) =
Revised: t : T (cid:48) =

Original: (cid:16)(cid:81)
Revised: (cid:16)(cid:81)

Original: t +1 s
Revised: t +1 s

Original: = T (cid:48)
Revised: = T (cid:48)

Original: A ( (cid:96),s ) σ
Revised: A ( (cid:96),s ) σ

Original: W ( (cid:96) ) r (cid:17) for t < T (cid:48) and identity otherwise, 3 h ( (cid:96), 0) is the initial hidden state of the RNN at layer (cid:96) , and
Revised: W ( (cid:96) ) r (cid:17) for t < T (cid:48) and identity otherwise, 3 h ( (cid:96), 0) is the initial hidden state of the RNN at layer (cid:96) , and

Original: A ( (cid:96) )RNN and B ( (cid:96) )
Revised: A ( (cid:96) )RNN and B ( (cid:96) )

Original: RNN are afﬁne parameters that depend on the layer input h ( (cid:96) − 1) and the initial hidden state h ( (cid:96), 0) .
Revised: RNN are afﬁne parameters that depend on the layer input h ( (cid:96) − 1) and the initial hidden state h ( (cid:96), 0) .

Original: We present the proof for Theorem 1 in Appendix G.
Revised: We present the proof for Theorem 1 in Appendix G.

Original: The key point here is that, by leveraging
Revised: The key point here is that, by leveraging

Original: MASOs, we can represent the time-unrolled RNN as a simple afﬁne transformation of the entire input sequence (8).
Revised: MASOs, we can represent the time-unrolled RNN as a simple afﬁne transformation of the entire input sequence (8).

Original: Note that this afﬁne transformation changes depending on the partition region in which the input belongs (recall (4) and (5)).
Revised: Note that this afﬁne transformation changes depending on the partition region in which the input belongs (recall (4) and (5)).

Original: Note also that the initial hidden state affects the layer output by inﬂuencing the afﬁne parameters and contributing a bias term A ( (cid:96) ) 0:
Revised: Note also that the initial hidden state affects the layer output by inﬂuencing the afﬁne parameters and contributing a bias term A ( (cid:96) ) 0:

Original: t h ( (cid:96), 0) to the bias parameter B ( (cid:96) )RNN .
Revised: t h ( (cid:96), 0) to the bias parameter B ( (cid:96) )RNN .

Original: We study the impact of the initial hidden state in more detail in Section 6.
Revised: We study the impact of the initial hidden state in more detail in Section 6.

Original: We are now ready to generalize the above result to multi-layer RNNs.
Revised: We are now ready to generalize the above result to multi-layer RNNs.

Original: Let x = (cid:2) x (1) (cid:62) , · · · , x ( T ) (cid:62) (cid:3) (cid:62) be the input sequence to a multi-layer RNN, and let z = (cid:2)
Revised: Let x = (cid:2) x (1) (cid:62) , · · · , x ( T ) (cid:62) (cid:3) (cid:62) be the input sequence to a multi-layer RNN, and let z = (cid:2)

Original: z (1) (cid:62) , · · · , z ( T ) (cid:62) (cid:3) (cid:62) be the output sequence.
Revised: z (1) (cid:62) , · · · , z ( T ) (cid:62) (cid:3) (cid:62) be the output sequence.

Original: We state the following result for the overall mapping of a multi-layer RNN.
Revised: We state the following result for the overall mapping of a multi-layer RNN.

Original: Theorem 2.
Revised: Theorem 2.

Original: The output of an L -layer RNN is a piecewise afﬁne spline operator deﬁned as
Revised: The output of an L -layer RNN is a piecewise afﬁne spline operator deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where A RNN =
Revised: where A RNN =

Original: (cid:81) 1 (cid:96) =
Revised: (cid:81) 1 (cid:96) =

Original: L A ( (cid:96) )RNN and B RNN = (cid:80) L(cid:96)
Revised: L A ( (cid:96) )RNN and B RNN = (cid:80) L(cid:96)

Original: =1 (cid:16)(cid:81)
Revised: =1 (cid:16)(cid:81)

Original: L − 1 (cid:96) (cid:48) =
Revised: L − 1 (cid:96) (cid:48) =

Original: (cid:96) A ( (cid:96) (cid:48) )
Revised: (cid:96) A ( (cid:96) (cid:48) )

Original: RNN (cid:17) B ( (cid:96) )
Revised: RNN (cid:17) B ( (cid:96) )

Original: RNN are the afﬁne parameters of the L -layer RNN.
Revised: RNN are the afﬁne parameters of the L -layer RNN.

Original: W and b are parameters of the fully connected output layer, where W =
Revised: W and b are parameters of the fully connected output layer, where W =

Original: [ W , W , . . .
Revised: [ W , W , . . .

Original: , W ] when the RNN outputs at every time step and W =
Revised: , W ] when the RNN outputs at every time step and W =

Original: [ W , 0 , . .
Revised: [ W , 0 , . .

Original: , 0] when the RNN outputs only at the last time step.
Revised: , 0] when the RNN outputs only at the last time step.

Original: Theorem 2 shows that, using MASOs, we have a simple, elegant, and closed-form formula showing that the output of an RNN is computed locally via very simple functions.
Revised: Theorem 2 shows that, using MASOs, we have a simple, elegant, and closed-form formula showing that the output of an RNN is computed locally via very simple functions.

Original: This result is proved by recursively applying the proof for Theorem 1.
Revised: This result is proved by recursively applying the proof for Theorem 1.

Original: The afﬁne mapping formula (9) opens many doors for RNN analyses, because we can shed light on
Revised: The afﬁne mapping formula (9) opens many doors for RNN analyses, because we can shed light on

Original: RNNs by applying established matrix results.
Revised: RNNs by applying established matrix results.

Original: In the next sections, we provide three analyses that follow this programme.
Revised: In the next sections, we provide three analyses that follow this programme.

Original: First, we show that RNNs partition the input space and that they develop the partitions through time.
Revised: First, we show that RNNs partition the input space and that they develop the partitions through time.

Original: Second, we analyze the forms of the afﬁne slope parameter and link RNNs to matched ﬁlterbanks.
Revised: Second, we analyze the forms of the afﬁne slope parameter and link RNNs to matched ﬁlterbanks.

Original: Third, we study the impact of the initial hidden state to rigorously justify the use of randomness in initial hidden state.
Revised: Third, we study the impact of the initial hidden state to rigorously justify the use of randomness in initial hidden state.

Original: From this point, for simplicity, we will assume a zero initial hidden state unless otherwise stated.
Revised: From this point, for simplicity, we will assume a zero initial hidden state unless otherwise stated.

Original:  I NTERNAL I NPUT S PACE P ARTITIONING IN RNN S
Revised:  I NTERNAL I NPUT S PACE P ARTITIONING IN RNN S

Original: The MASO viewpoint enables us to see how an RNN implicitly partitions its input sequence through time, which provides a new perspective of its dynamics.
Revised: The MASO viewpoint enables us to see how an RNN implicitly partitions its input sequence through time, which provides a new perspective of its dynamics.

Original: To see this, ﬁrst recall that, for an RNN
Revised: To see this, ﬁrst recall that, for an RNN

Original: [Figure]
Revised: [Figure]

Original: cell, the piecewise afﬁne and convex activation nonlinearity partitions each dimension of the cell input z ( (cid:96),t ) into R regions (for ReLU, R = 2 ).
Revised: cell, the piecewise afﬁne and convex activation nonlinearity partitions each dimension of the cell input z ( (cid:96),t ) into R regions (for ReLU, R = 2 ).

Original: Knowing the state of the nonlinearity (which region r is activated) is thus equivalent to knowing the partition of the cell input.
Revised: Knowing the state of the nonlinearity (which region r is activated) is thus equivalent to knowing the partition of the cell input.

Original: For a multi-layer
Revised: For a multi-layer

Original: RNN composed of many RNN cells (recall Figure 1), the input sequence partition can be retrieved by accessing the collection of the states of all of the nonlinearities; each input sequence can be represented by a partition “code” that determines the partition to which it belongs.
Revised: RNN composed of many RNN cells (recall Figure 1), the input sequence partition can be retrieved by accessing the collection of the states of all of the nonlinearities; each input sequence can be represented by a partition “code” that determines the partition to which it belongs.

Original: Since an RNN processes an input sequence one step at a time, the input space partition is gradually built up and reﬁned through time.
Revised: Since an RNN processes an input sequence one step at a time, the input space partition is gradually built up and reﬁned through time.

Original: As a consequence, when seen through the MASO lens, the forward pass of an RNN is simply developing and reﬁning the partition code of the input sequence.
Revised: As a consequence, when seen through the MASO lens, the forward pass of an RNN is simply developing and reﬁning the partition code of the input sequence.

Original: Visualizing the evolution of the partition codes can be potentially beneﬁcial for diagnosing RNNs and understanding their dynamics.
Revised: Visualizing the evolution of the partition codes can be potentially beneﬁcial for diagnosing RNNs and understanding their dynamics.

Original: As an example, we demonstrate the evolution of the partition codes of a one-layer ReLU RNN trained on the MNIST dataset, with each image ﬂattened into a 1-dimensional sequence so that input at each time step is a single pixel.
Revised: As an example, we demonstrate the evolution of the partition codes of a one-layer ReLU RNN trained on the MNIST dataset, with each image ﬂattened into a 1-dimensional sequence so that input at each time step is a single pixel.

Original: Details of the model and experiments are in Appendix C.
Revised: Details of the model and experiments are in Appendix C.

Original: Since the ReLU activation partitions its input space into only 2 regions , we can retrieve the RNN partition codes of the input images simply by binarizing and concatenating all of the hidden states.
Revised: Since the ReLU activation partitions its input space into only 2 regions , we can retrieve the RNN partition codes of the input images simply by binarizing and concatenating all of the hidden states.

Original: Figure 2 visualizes how the partition codes of MNIST images evolve through time using t-SNE, a distance-preserving dimensionality reduction technique (van der Maaten & Hinton, 2008).
Revised: Figure 2 visualizes how the partition codes of MNIST images evolve through time using t-SNE, a distance-preserving dimensionality reduction technique (van der Maaten & Hinton, 2008).

Original: The ﬁgure clearly shows the evolution of the partition codes from hardly any separation between classes of digits to forming more and better separated clusters through time.
Revised: The ﬁgure clearly shows the evolution of the partition codes from hardly any separation between classes of digits to forming more and better separated clusters through time.

Original: We can also be assured that the model is well-behaved, since the ﬁnal partition shows that the images are well clustered based on their labels.
Revised: We can also be assured that the model is well-behaved, since the ﬁnal partition shows that the images are well clustered based on their labels.

Original: Additional visualizations are available in Section D.
Revised: Additional visualizations are available in Section D.

Original:  RNN S AS M ATCHED F ILTERBANKS
Revised:  RNN S AS M ATCHED F ILTERBANKS

Original: The MASO viewpoint enables us to connect RNNs to classical signal processing tools like the matched ﬁlter.
Revised: The MASO viewpoint enables us to connect RNNs to classical signal processing tools like the matched ﬁlter.

Original: Indeed, we can directly interpret an RNN as a matched ﬁlterbank , where the classiﬁcation decision is informed via the simple inner product between a “template” and the input sequence.
Revised: Indeed, we can directly interpret an RNN as a matched ﬁlterbank , where the classiﬁcation decision is informed via the simple inner product between a “template” and the input sequence.

Original: To see this, we follow an argument similar to that in Section 4.
Revised: To see this, we follow an argument similar to that in Section 4.

Original: First, note that the slope parameter A ( (cid:96),t ) for each RNN cell is a “locally optimal template” because it maximizes each of its output dimensions over the R regions that the nonlinearity induces (recall (3) and (7)).
Revised: First, note that the slope parameter A ( (cid:96),t ) for each RNN cell is a “locally optimal template” because it maximizes each of its output dimensions over the R regions that the nonlinearity induces (recall (3) and (7)).

Original: For a multi-layer RNN composed of many RNN cells, the overall “template” A RNN corresponds to the composition of the optimal templates from each RNN cell, which can be computed simply via d z / d x (recall (9)).
Revised: For a multi-layer RNN composed of many RNN cells, the overall “template” A RNN corresponds to the composition of the optimal templates from each RNN cell, which can be computed simply via d z / d x (recall (9)).

Original: Thus, we can view an RNN as a matched ﬁlterbank whose output is the maximum inner product between the input and the rows of the overall template A RNN (van Trees, 1992; 2013).
Revised: Thus, we can view an RNN as a matched ﬁlterbank whose output is the maximum inner product between the input and the rows of the overall template A RNN (van Trees, 1992; 2013).

Original: The overall template is also known in the machine learning community as a salience map ; see Li et al.
Revised: The overall template is also known in the machine learning community as a salience map ; see Li et al.

Original: for an example of using saliency maps to visualize RNNs.
Revised: for an example of using saliency maps to visualize RNNs.

Original: Our new insight here is that a good template produces a larger inner product with the input regardless of the visual quality of the template , thus complementing prior work.
Revised: Our new insight here is that a good template produces a larger inner product with the input regardless of the visual quality of the template , thus complementing prior work.

Original: The template matching view of RNNs thus provides a principled methodology to visualize and diagnose RNNs by examining the inner products between the inputs and the templates.
Revised: The template matching view of RNNs thus provides a principled methodology to visualize and diagnose RNNs by examining the inner products between the inputs and the templates.

Original: To illustrate the matched ﬁlter interpretation, we train a one-layer ReLU RNN on the polarized Stanford Sentiment Treebank dataset (SST-2) (Socher et al., 2013), which poses a binary classiﬁcation problem, and display in Figure 3 the templates corresponding to the correct and incorrect classes of an input where the correct class is a negative sentiment.
Revised: To illustrate the matched ﬁlter interpretation, we train a one-layer ReLU RNN on the polarized Stanford Sentiment Treebank dataset (SST-2) (Socher et al., 2013), which poses a binary classiﬁcation problem, and display in Figure 3 the templates corresponding to the correct and incorrect classes of an input where the correct class is a negative sentiment.

Original: We see that the input has a much larger inner product with the template corresponding to the correct class (left plot) than that corresponding to the incorrect class (right plot), which informs us that the model correctly classiﬁes this input.
Revised: We see that the input has a much larger inner product with the template corresponding to the correct class (left plot) than that corresponding to the incorrect class (right plot), which informs us that the model correctly classiﬁes this input.

Original: Additional experimental results are given in Appendix E.
Revised: Additional experimental results are given in Appendix E.

Original:  I MPROVING RNN S V IA R ANDOM I NITIAL H IDDEN S TATE
Revised:  I MPROVING RNN S V IA R ANDOM I NITIAL H IDDEN S TATE

Original: In this section, we provide a theoretical motivation for the use of a random initial hidden state in
Revised: In this section, we provide a theoretical motivation for the use of a random initial hidden state in

Original: RNNs.
Revised: RNNs.

Original: The initial hidden state needs to be set to some prior value to start the recursion (recall
Revised: The initial hidden state needs to be set to some prior value to start the recursion (recall

Original: Section 2).
Revised: Section 2).

Original: Little is understood regarding the best choice of initial hidden state other than Zimmermann et al.
Revised: Little is understood regarding the best choice of initial hidden state other than Zimmermann et al.

Original: (2012)’s dynamical system argument.
Revised: (2012)’s dynamical system argument.

Original: Consequently, it is typically simply set to zero.
Revised: Consequently, it is typically simply set to zero.

Original: Leveraging the MASO view of RNNs, we now demonstrate that one can improve signiﬁcantly over a zero initial hidden state by using a random initial hidden state.
Revised: Leveraging the MASO view of RNNs, we now demonstrate that one can improve signiﬁcantly over a zero initial hidden state by using a random initial hidden state.

Original: This choice regularizes the afﬁne slope parameter associated with the initial hidden state and molliﬁes the so-called exploding gradient problem (Pascanu et al., 2013).
Revised: This choice regularizes the afﬁne slope parameter associated with the initial hidden state and molliﬁes the so-called exploding gradient problem (Pascanu et al., 2013).

Original: Random Initial Hidden State as an Explicit Regularization.
Revised: Random Initial Hidden State as an Explicit Regularization.

Original: We ﬁrst state our theoretical result that using random initial hidden state corresponds to an explicit regularization and then discuss its impact on exploding gradients.
Revised: We ﬁrst state our theoretical result that using random initial hidden state corresponds to an explicit regularization and then discuss its impact on exploding gradients.

Original: Without loss of generality, we focus on one-layer ReLU
Revised: Without loss of generality, we focus on one-layer ReLU

Original: RNNs.
Revised: RNNs.

Original: Let N be the number of data points and C the number of classes.
Revised: Let N be the number of data points and C the number of classes.

Original: Deﬁne A h :=
Revised: Deﬁne A h :=

Original: A 1: T = (cid:81)s =
Revised: A 1: T = (cid:81)s =

Original: [Equation]
Revised: [Equation]

Original: r (recall (8)).
Revised: r (recall (8)).

Original: Theorem 3.
Revised: Theorem 3.

Original: Let L be an RNN loss function, and let (cid:101) L represent the modiﬁed loss function when the RNN initial hidden state is set to a Gaussian random vector (cid:15) ∼ N (0 , σ 2 (cid:15) I ) with small standard deviation σ (cid:15) .
Revised: Let L be an RNN loss function, and let (cid:101) L represent the modiﬁed loss function when the RNN initial hidden state is set to a Gaussian random vector (cid:15) ∼ N (0 , σ 2 (cid:15) I ) with small standard deviation σ (cid:15) .

Original: Then we have that E (cid:15) (cid:104)(cid:101) L (cid:105) =
Revised: Then we have that E (cid:15) (cid:104)(cid:101) L (cid:105) =

Original: L + R .
Revised: L + R .

Original: For the cross-entropy loss L with softmax output,
Revised: For the cross-entropy loss L with softmax output,

Original: R = σ 2 (cid:15) 2 N (cid:80) Nn
Revised: R = σ 2 (cid:15) 2 N (cid:80) Nn

Original: = (cid:13) (cid:13) (cid:13)(cid:13) diag (cid:18)(cid:104) d (cid:98)
Revised: = (cid:13) (cid:13) (cid:13)(cid:13) diag (cid:18)(cid:104) d (cid:98)

Original: y ni
Revised: y ni

Original: ∂z nj (cid:105) i =
Revised: ∂z nj (cid:105) i =

Original: j (cid:19)
Revised: j (cid:19)

Original: A h (cid:13) (cid:13) (cid:13)(cid:13) 2 , where (cid:98)
Revised: A h (cid:13) (cid:13) (cid:13)(cid:13) 2 , where (cid:98)

Original: y
Revised: y

Original: ni is the i th dimension of the softmax output of the n th data point and i, j ∈ { 1 , . . .
Revised: ni is the i th dimension of the softmax output of the n th data point and i, j ∈ { 1 , . . .

Original: , C } are the class indices.
Revised: , C } are the class indices.

Original: For the mean-squared error loss L ,
Revised: For the mean-squared error loss L ,

Original: R = σ 2 (cid:15) 2 N (cid:80) Nn
Revised: R = σ 2 (cid:15) 2 N (cid:80) Nn

Original: =1 (cid:107)
Revised: =1 (cid:107)

Original: A h (cid:107) 2 .
Revised: A h (cid:107) 2 .

Original: We prove this result for the cross-entropy loss in Appendix G.2.
Revised: We prove this result for the cross-entropy loss in Appendix G.2.

Original: The standard deviation σ (cid:15) controls the importance of the regularization term and recovers the case of standard zero initial hidden state when σ (cid:15) = 0 .
Revised: The standard deviation σ (cid:15) controls the importance of the regularization term and recovers the case of standard zero initial hidden state when σ (cid:15) = 0 .

Original: Connection to the Exploding Gradient Problem.
Revised: Connection to the Exploding Gradient Problem.

Original: Backpropagation through time (BPTT) is the default RNN training algorithm.
Revised: Backpropagation through time (BPTT) is the default RNN training algorithm.

Original: Updating the recurrent weight W r with its gradient using BPTT involves calculating the gradient of the RNN output with respect to the hidden state at each time step t = 0 , . .
Revised: Updating the recurrent weight W r with its gradient using BPTT involves calculating the gradient of the RNN output with respect to the hidden state at each time step t = 0 , . .

Original: , T
Revised: , T

Original: [Equation]
Revised: [Equation]

Original: (cid:89) s
Revised: (cid:89) s

Original: [Equation]
Revised: [Equation]

Original: (cid:33) .
Revised: (cid:33) .

Original: When (cid:107) A ( s ) σ W r (cid:107) 2 > 1 , the product term (cid:81) t
Revised: When (cid:107) A ( s ) σ W r (cid:107) 2 > 1 , the product term (cid:81) t

Original: +1 s =
Revised: +1 s =

Original: T A ( s ) σ W r in (10) blows up, which leads to unstable training.
Revised: T A ( s ) σ W r in (10) blows up, which leads to unstable training.

Original: This is known as the exploding gradient problem (Pascanu et al., 2013).
Revised: This is known as the exploding gradient problem (Pascanu et al., 2013).

Original: Our key realization is that the gradient of the RNN output with respect to the initial hidden state h (0) features the term A h from Theorem
Revised: Our key realization is that the gradient of the RNN output with respect to the initial hidden state h (0) features the term A h from Theorem

Original: [Equation]
Revised: [Equation]

Original: Of all the terms in (10), this one involves the most matrix products and hence is the most erratic.
Revised: Of all the terms in (10), this one involves the most matrix products and hence is the most erratic.

Original: Fortunately, Theorem 3 instructs us that introducing randomness into the initial hidden state effects a regularization on A h and hence tamps down the gradient before it can explode.
Revised: Fortunately, Theorem 3 instructs us that introducing randomness into the initial hidden state effects a regularization on A h and hence tamps down the gradient before it can explode.

Original: An interesting direction for future work is extending this analysis to every term in (10).
Revised: An interesting direction for future work is extending this analysis to every term in (10).

Original: Experiments.
Revised: Experiments.

Original: We now report on the results of a number of experiments that indicate the signiﬁcant performance gains that can be obtained using a random initial hidden state of properly chosen standard deviation σ (cid:15) .
Revised: We now report on the results of a number of experiments that indicate the signiﬁcant performance gains that can be obtained using a random initial hidden state of properly chosen standard deviation σ (cid:15) .

Original: Unless otherwise mentioned, in all experiments we use ReLU RNNs with 128-dimensional hidden states and with the recurrent weight matrix W ( (cid:96) )
Revised: Unless otherwise mentioned, in all experiments we use ReLU RNNs with 128-dimensional hidden states and with the recurrent weight matrix W ( (cid:96) )

Original: r initialized as an identity matrix (Le et al., 2015; Talathi & Vartak, 2016).
Revised: r initialized as an identity matrix (Le et al., 2015; Talathi & Vartak, 2016).

Original: We summarize the experimental results; experimental details and additional results are available in Appendices C and F.
Revised: We summarize the experimental results; experimental details and additional results are available in Appendices C and F.

Original: Visualizing the Regularizing Effect of a Random Initial Hidden State.
Revised: Visualizing the Regularizing Effect of a Random Initial Hidden State.

Original: We ﬁrst consider a simulated task of adding 2 sequences of length 100.
Revised: We ﬁrst consider a simulated task of adding 2 sequences of length 100.

Original: This is a ternary classiﬁcation problem with input X ∈ R 2 × T and target y ∈ { 0 , 1 , 2 } , y =
Revised: This is a ternary classiﬁcation problem with input X ∈ R 2 × T and target y ∈ { 0 , 1 , 2 } , y =

Original: (cid:80)
Revised: (cid:80)

Original: i 

Revised: i

Original: X 2 i =
Revised: X 2 i =

Original: 1 X 1 i .
Revised: 1 X 1 i .

Original: The ﬁrst row of X contains randomly chosen 0 ’s and 1 ’s; the second row of X contains 1 ’s at 2 randomly chosen indices and 0 ’s everywhere else.
Revised: The ﬁrst row of X contains randomly chosen 0 ’s and 1 ’s; the second row of X contains 1 ’s at 2 randomly chosen indices and 0 ’s everywhere else.

Original: Prior work treats this task as a regression task (Arjovsky et al., 2016); our regression results are provided in Appendix F.1.
Revised: Prior work treats this task as a regression task (Arjovsky et al., 2016); our regression results are provided in Appendix F.1.

Original: In Figure 4, we visualize the norm of A h , the norm of the recurrent weight gradient (cid:107)
Revised: In Figure 4, we visualize the norm of A h , the norm of the recurrent weight gradient (cid:107)

Original: d L d W r (cid:107) , and the validation loss against training epochs for various random initial state standard deviations.
Revised: d L d W r (cid:107) , and the validation loss against training epochs for various random initial state standard deviations.

Original: The top two plots clearly demonstrate the effect of the random initial hidden state in regularizing both A h and the norm of the recurrent weight gradient, since larger σ (cid:15) reduces the magnitudes of both A h and (cid:107)
Revised: The top two plots clearly demonstrate the effect of the random initial hidden state in regularizing both A h and the norm of the recurrent weight gradient, since larger σ (cid:15) reduces the magnitudes of both A h and (cid:107)

Original: d L d W r (cid:107) .
Revised: d L d W r (cid:107) .

Original: Notably, the reduced magnitude of the gradient term (cid:107) d L d W r (cid:107) empirically demonstrates the molliﬁcation of the exploding gradient problem.
Revised: Notably, the reduced magnitude of the gradient term (cid:107) d L d W r (cid:107) empirically demonstrates the molliﬁcation of the exploding gradient problem.

Original: The bottom plot shows that setting σ (cid:15) too large can negatively impact learning.
Revised: The bottom plot shows that setting σ (cid:15) too large can negatively impact learning.

Original: This can be explained as having too much regularization effect.
Revised: This can be explained as having too much regularization effect.

Original: This suggests the question of choosing the best value of σ (cid:15) in practice, which we now investigate.
Revised: This suggests the question of choosing the best value of σ (cid:15) in practice, which we now investigate.

Original: Choosing the Standard Deviation of the Random Initial Hidden State.
Revised: Choosing the Standard Deviation of the Random Initial Hidden State.

Original: We examine the effect on performance of different random initial state standard deviations σ (cid:15) in RNNs using RMSprop and
Revised: We examine the effect on performance of different random initial state standard deviations σ (cid:15) in RNNs using RMSprop and

Original: SGD with varying learning rates.
Revised: SGD with varying learning rates.

Original: We perform experiments on the MNIST dataset with each image ﬂattened to a length 784 sequence (recall Section 4).
Revised: We perform experiments on the MNIST dataset with each image ﬂattened to a length 784 sequence (recall Section 4).

Original: The full experimental results are included in
Revised: The full experimental results are included in

Original: Appendix F.2; here, we report two interesting ﬁndings.
Revised: Appendix F.2; here, we report two interesting ﬁndings.

Original: First, for both optimizers, using a random initial hidden state permits the use of higher learning rates that would lead to an exploding gradient when training without a random initial hidden state.
Revised: First, for both optimizers, using a random initial hidden state permits the use of higher learning rates that would lead to an exploding gradient when training without a random initial hidden state.

Original: Second, RMSprop is less sensitive to the choice of σ (cid:15) than SGD and achieves favorable accuracy even when σ (cid:15) is very large (e.g., σ (cid:15) = 5 ).
Revised: Second, RMSprop is less sensitive to the choice of σ (cid:15) than SGD and achieves favorable accuracy even when σ (cid:15) is very large (e.g., σ (cid:15) = 5 ).

Original: This might be due to the gradient smoothing that RMSprop performs during optimization.
Revised: This might be due to the gradient smoothing that RMSprop performs during optimization.

Original: We therefore recommend the use of RMSprop with a random initial hidden state to improve model performance.
Revised: We therefore recommend the use of RMSprop with a random initial hidden state to improve model performance.

Original: [Table]
Revised: [Table]

Original: We used RMSprop to train ReLU RNNs of one and two layers with and without random initial hidden state on the MNIST, permuted MNIST 4 and SST-2 datasets.
Revised: We used RMSprop to train ReLU RNNs of one and two layers with and without random initial hidden state on the MNIST, permuted MNIST 4 and SST-2 datasets.

Original: Table 1 shows the classiﬁcation accuracies of these models as well as a few state-of-the-art results using complicated models.
Revised: Table 1 shows the classiﬁcation accuracies of these models as well as a few state-of-the-art results using complicated models.

Original: It is surprising that a random initial hidden state elevates the performance of a simple ReLU RNN to near state-of-the-art performance.
Revised: It is surprising that a random initial hidden state elevates the performance of a simple ReLU RNN to near state-of-the-art performance.

Original: Random Initial Hidden State in Complex RNN Models.
Revised: Random Initial Hidden State in Complex RNN Models.

Original: Inspired by the results of the previous experiment, we integrated a random initial hidden state into some more complex RNN models.
Revised: Inspired by the results of the previous experiment, we integrated a random initial hidden state into some more complex RNN models.

Original: We ﬁrst evaluate a one-layer gated recurrent unit (GRU) on the MNIST and permuted MNIST datasets, with a random and zero initial hidden state.
Revised: We ﬁrst evaluate a one-layer gated recurrent unit (GRU) on the MNIST and permuted MNIST datasets, with a random and zero initial hidden state.

Original: Although the performance gains are not quite as impressive as those for ReLU RNNs, our results for GRUs still show worthwhile accuract improvements, from 0 .
Revised: Although the performance gains are not quite as impressive as those for ReLU RNNs, our results for GRUs still show worthwhile accuract improvements, from 0 .

Original: 986 to 0 .
Revised: 986 to 0 .

Original: 987 for MNIST and from 0 .
Revised: 987 for MNIST and from 0 .

Original: 888 to 0 .
Revised: 888 to 0 .

Original: 904 for permuted MNIST.
Revised: 904 for permuted MNIST.

Original: We continue our experiments with a more complex, convolutional-recurrent model composed ofconvolution layers followed by 2 GRU layers (Cakir et al., 2017) and the Bird Audio Detection Challenge dataset 5 .
Revised: We continue our experiments with a more complex, convolutional-recurrent model composed ofconvolution layers followed by 2 GRU layers (Cakir et al., 2017) and the Bird Audio Detection Challenge dataset.

Original: This binary classiﬁcation problem aims to detect whether or not an audio recording contains bird songs; see Appendix C for the details.
Revised: This binary classiﬁcation problem aims to detect whether or not an audio recording contains bird songs; see Appendix C for the details.

Original: We use the area under the ROC curve (AUC) as the evaluation metric, since the dataset is highly imbalanced.
Revised: We use the area under the ROC curve (AUC) as the evaluation metric, since the dataset is highly imbalanced.

Original: Simply switching from a zero to a random initial hidden state provides a signiﬁcant boost in the AUC: from 90.5% to 93.4%.
Revised: Simply switching from a zero to a random initial hidden state provides a signiﬁcant boost in the AUC: from 90.5% to 93.4%.

Original: These encouraging preliminary results suggest that, while more theoretical and empirical investigations are needed, a random initial hidden state can also boost the performance of complicated RNN models that are not piecewise afﬁne and convex.
Revised: These encouraging preliminary results suggest that, while more theoretical and empirical investigations are needed, a random initial hidden state can also boost the performance of complicated RNN models that are not piecewise afﬁne and convex.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: We have developed and explored a novel perspective of RNNs in terms of max-afﬁne spline operators (MASOs).
Revised: We have developed and explored a novel perspective of RNNs in terms of max-afﬁne spline operators (MASOs).

Original: RNNs with piecewise afﬁne and convex nonlinearities are piecewise afﬁne spline operators with a simple, elegant analytical form.
Revised: RNNs with piecewise afﬁne and convex nonlinearities are piecewise afﬁne spline operators with a simple, elegant analytical form.

Original: The connections to input space partitioning (vector quantization) and matched ﬁltering followed immediately.
Revised: The connections to input space partitioning (vector quantization) and matched ﬁltering followed immediately.

Original: The spline viewpoint also suggested that the typical zero initial hidden state be replaced with a random one that molliﬁes the exploding gradient problem and improves generalization performance.
Revised: The spline viewpoint also suggested that the typical zero initial hidden state be replaced with a random one that molliﬁes the exploding gradient problem and improves generalization performance.

Original: There remain abundant promising research directions.
Revised: There remain abundant promising research directions.

Original: First, we can extend the MASO RNN framework following (Balestriero & Baraniuk, 2018b) to cover more general networks like gated RNNs (e.g, GRUs, LSTMs) that employ the sigmoid nonlinearity, which is neither piecewise afﬁne nor convex.
Revised: First, we can extend the MASO RNN framework following (Balestriero & Baraniuk, 2018b) to cover more general networks like gated RNNs (e.g, GRUs, LSTMs) that employ the sigmoid nonlinearity, which is neither piecewise afﬁne nor convex.

Original: Second, we can apply recent random matrix theory results (Martin & Mahoney, 2018) to the afﬁne parameter A RNN (e.g., the change of the distribution of its singular values during training) to understand RNN training dynamics.
Revised: Second, we can apply recent random matrix theory results (Martin & Mahoney, 2018) to the afﬁne parameter A RNN (e.g., the change of the distribution of its singular values during training) to understand RNN training dynamics.

Original: A CKNOWLEDGEMENTS
Revised: A CKNOWLEDGEMENTS

Original: We thank the anonymous reviewers for their constructive feedback.
Revised: We thank the anonymous reviewers for their constructive feedback.

Original: This work was partially supported by NSF grants IIS-17-30574 and IIS-18-38177, AFOSR grant FA9550-18-1-0478, ARO grant W911NF-15-1-0316, ONR grants N00014-17-1-2551 and N00014-18-12571, DARPA grant
Revised: This work was partially supported by NSF grants IIS-17-30574 and IIS-18-38177, AFOSR grant FA9550-18-1-0478, ARO grant W911NF-15-1-0316, ONR grants N00014-17-1-2551 and N00014-18-12571, DARPA grant

Original: G001534- and DOD Vannevar Bush Faculty Fellowship (NSSEFF) grant N00014-18-1-2047.
Revised: G001534- and DOD Vannevar Bush Faculty Fellowship (NSSEFF) grant N00014-18-1-2047.

Original: A N OTATION
Revised: A N OTATION

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: example
Revised: example

Original: For classiﬁcation y n ∈ { 1 , . .
Revised: For classiﬁcation y n ∈ { 1 , . .

Original: , C } , C >
Revised: , C } , C >

Original: [Table]
Revised: [Table]

Original: For regression y n ∈ R C
Revised: For regression y n ∈ R C

Original: C
Revised: C

Original: [Table]
Revised: [Table]

Original: Output of an RNN cell at layer (cid:96) and time step t
Revised: Output of an RNN cell at layer (cid:96) and time step t

Original: [Table]
Revised: [Table]

Original: Alternatively, input to an RNN cell at layer (cid:96) + 1 and step t −
Revised: Alternatively, input to an RNN cell at layer (cid:96) + 1 and step t −

Original: [Table]
Revised: [Table]

Original: Concatenation of hidden state h ( (cid:96)
Revised: Concatenation of hidden state h ( (cid:96)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: RNN
Revised: RNN

Original: [Table]
Revised: [Table]

Original: be piecewise afﬁne and convex in this
Revised: be piecewise afﬁne and convex in this

Original: [Table]
Revised: [Table]

Original: MASO formula of the
Revised: MASO formula of the

Original: [Table]
Revised: [Table]

Original: B D ATASETS AND P REPROCESSING S TEPS
Revised: B D ATASETS AND P REPROCESSING S TEPS

Original: Below we describe the datasets and explain the preprocessing steps for each dataset.
Revised: Below we describe the datasets and explain the preprocessing steps for each dataset.

Original: MNIST .
Revised: MNIST .

Original: The dataset 6 consists of 60k images in the training set and 10k images in the test set.
Revised: The dataset 6 consists of 60k images in the training set and 10k images in the test set.

Original: We randomly select 10k images from the training set as validation set.
Revised: We randomly select 10k images from the training set as validation set.

Original: We ﬂatten each image to a 1- dimensional vector of size is also centered and normalized with mean and standard deviation of 0.
Revised: We ﬂatten each image to a 1- dimensional vector of size is also centered and normalized with mean and standard deviation of 0.

Original: (PyTorch default values).
Revised: (PyTorch default values).

Original: permuted MNIST .
Revised: permuted MNIST .

Original: We apply a ﬁxed permutation to all images in the MNIST dataset to obtain the permuted MNIST dataset.
Revised: We apply a ﬁxed permutation to all images in the MNIST dataset to obtain the permuted MNIST dataset.

Original: SST-2 .
Revised: SST-2 .

Original: The dataset 7 consists of 6920, 872, 1821 sentences in the training, validation and test set, respectively.
Revised: The dataset 7 consists of 6920, 872, 1821 sentences in the training, validation and test set, respectively.

Original: Total number of vocabulary is 17539, and average sentence length is 19.67.
Revised: Total number of vocabulary is 17539, and average sentence length is 19.67.

Original: Each sentence is minimally processed into sequences of words and use a ﬁxed-dimensional and trainable vector to represent each word.
Revised: Each sentence is minimally processed into sequences of words and use a ﬁxed-dimensional and trainable vector to represent each word.

Original: We initialize these vectors either randomly or using GloVe (Pennington et al., 2014).
Revised: We initialize these vectors either randomly or using GloVe (Pennington et al., 2014).

Original: Due to the small size of the dataset, the phrases in each sentence that have semantic labels are also used as part of the training set in addition to the whole sentence during training.
Revised: Due to the small size of the dataset, the phrases in each sentence that have semantic labels are also used as part of the training set in addition to the whole sentence during training.

Original: Dropout of 0.5 is applied to all experiments.
Revised: Dropout of 0.5 is applied to all experiments.

Original: Phrases are not used during validation and testing, i.e., we always use entire sentences during validation and testing.
Revised: Phrases are not used during validation and testing, i.e., we always use entire sentences during validation and testing.

Original: Bird Audio Dataset .
Revised: Bird Audio Dataset .

Original: The dataset 8 consists of 7 , 000 ﬁeld recording signals of 10 seconds sampled at 44 kHz from the Freesound Stowell & Plumbley (2014)
Revised: The dataset 8 consists of 7 , 000 ﬁeld recording signals of 10 seconds sampled at 44 kHz from the Freesound Stowell & Plumbley (2014)

Original: audio archive representing slightly less than 20 hours of audio signals.
Revised: audio archive representing slightly less than 20 hours of audio signals.

Original: The audio waveforms are extracted from diverse scenes such as city, nature, train, voice, water, etc., some of which include bird sounds.
Revised: The audio waveforms are extracted from diverse scenes such as city, nature, train, voice, water, etc., some of which include bird sounds.

Original: The labels regarding the bird detection task can be found in the ﬁle freefield1010 .
Revised: The labels regarding the bird detection task can be found in the ﬁle freefield1010 .

Original: Performance is measured by Area
Revised: Performance is measured by Area

Original: Under Curve (AUC) due to the unbalanced distribution of the classes.
Revised: Under Curve (AUC) due to the unbalanced distribution of the classes.

Original: We preprocess every audio clip by ﬁrst using short-time Fourier transform (STFT) with 40ms and 50% overlapping Hamming window to obtain audio spectrum and then by extracting 40 log mel-band energy features.
Revised: We preprocess every audio clip by ﬁrst using short-time Fourier transform (STFT) with 40ms and 50% overlapping Hamming window to obtain audio spectrum and then by extracting 40 log mel-band energy features.

Original: After preprocessing, each input is of dimension D = 96 and T = 999 .
Revised: After preprocessing, each input is of dimension D = 96 and T = 999 .

Original: C E XPERIMENTAL S ETUP
Revised: C E XPERIMENTAL S ETUP

Original: Experiment setup for various datasets is summarized in Table 2.
Revised: Experiment setup for various datasets is summarized in Table 2.

Original: Some of the experiments do not appear in the main text but in the appendix; we include setup for those experiments as well.
Revised: Some of the experiments do not appear in the main text but in the appendix; we include setup for those experiments as well.

Original: A setting common to all experiments is that we use learning rate scheduler so that when validation loss plateaus for 5 consecutive epochs, we reduce the current learning rate by a factor of 0.7.
Revised: A setting common to all experiments is that we use learning rate scheduler so that when validation loss plateaus for 5 consecutive epochs, we reduce the current learning rate by a factor of 0.7.

Original: Setup of the experiments on inﬂuence of various standard deviations in random initial hidden state under different settings .
Revised: Setup of the experiments on inﬂuence of various standard deviations in random initial hidden state under different settings .

Original: We use σ (cid:15) chosen in { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 5 } and learning rates in { 1 × 10 − 5 , 1 × 10 − 4 , 1 . 5 × 10 − 4 , 2 × 10 − 4 } for RMSprop and { 1 × 10 − 7 , 1 × 10 − 6 , 1 . 25 × 10 − 6 , 1 . 5 × 10 − 6 } plain SGD.
Revised: We use σ (cid:15) chosen in { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 5 } and learning rates in { 1 × 10 − 5 , 1 × 10 − 4 , 1 . 5 × 10 − 4 , 2 × 10 − 4 } for RMSprop and { 1 × 10 − 7 , 1 × 10 − 6 , 1 . 25 × 10 − 6 , 1 . 5 × 10 − 6 } plain SGD.

Original: Setup of input space partitioning experiments .
Revised: Setup of input space partitioning experiments .

Original: For the results in the main text, we use t-SNE visualization (van der Maaten & Hinton, 2008) with 2 dimensions and the default settings from the python sklearn package.
Revised: For the results in the main text, we use t-SNE visualization (van der Maaten & Hinton, 2008) with 2 dimensions and the default settings from the python sklearn package.

Original: Visualization is performed on the whole 10k test set images.
Revised: Visualization is performed on the whole 10k test set images.

Original: For ﬁnding the nearest neighbors of examples in the SST-2 dataset, since the examples are of varying lengths, we constrain the distance comparison to within +/-10 words of the target sentence.
Revised: For ﬁnding the nearest neighbors of examples in the SST-2 dataset, since the examples are of varying lengths, we constrain the distance comparison to within +/-10 words of the target sentence.

Original: When the sentence lengths are not the same, we simply pad the shorter ones to the longest one, then process it with
Revised: When the sentence lengths are not the same, we simply pad the shorter ones to the longest one, then process it with

Original: RNN and ﬁnally calculate the distance as the (cid:96) 2 distance of the partition codes (i.e., concatenation of all hidden states) that RNN computes.
Revised: RNN and ﬁnally calculate the distance as the (cid:96) 2 distance of the partition codes (i.e., concatenation of all hidden states) that RNN computes.

Original: We justify the comparison between examples of different lengths using padding by noting that batching examples and padding the examples to the longest example within a batch has been a common practice in modern natural language processing tasks.
Revised: We justify the comparison between examples of different lengths using padding by noting that batching examples and padding the examples to the longest example within a batch has been a common practice in modern natural language processing tasks.

Original: Setup of exploratory experiments .
Revised: Setup of exploratory experiments .

Original: We experimented with one-layer GRU with 128 hidden unites for MNIST and permuted MNIST datasets.
Revised: We experimented with one-layer GRU with 128 hidden unites for MNIST and permuted MNIST datasets.

Original: We use RMSprop optimizer with an initial learning rate of 0.001.
Revised: We use RMSprop optimizer with an initial learning rate of 0.001.

Original: We experimented with various standard deviations in random initial hidden state including { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 , 5 } .
Revised: We experimented with various standard deviations in random initial hidden state including { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 , 5 } .

Original: The optimal standard deviations that produce the results in the main text are σ (cid:15) = { 0 . 01 , 0 . 05 , 0 . 01 } , for MNIST, permuted MNIST and bird detection datasets, respectively.
Revised: The optimal standard deviations that produce the results in the main text are σ (cid:15) = { 0 . 01 , 0 . 05 , 0 . 01 } , for MNIST, permuted MNIST and bird detection datasets, respectively.

Original: D A DDITIONAL I NPUT S PACE P ARTITION V ISUALIZATIONS
Revised: D A DDITIONAL I NPUT S PACE P ARTITION V ISUALIZATIONS

Original: We provide ample additional visualizations to demonstrate the partition codes that an RNN computes on its input sequences.
Revised: We provide ample additional visualizations to demonstrate the partition codes that an RNN computes on its input sequences.

Original: Here, the results are focused more on the properties of the ﬁnal partition
Revised: Here, the results are focused more on the properties of the ﬁnal partition

Original: [Table]
Revised: [Table]

Original: Here, we visualize the codes computed after the RNN processes the entire input sequence rather than part of the input sequence.
Revised: Here, we visualize the codes computed after the RNN processes the entire input sequence rather than part of the input sequence.

Original: Several additional sets of experimental results are shown; the the
Revised: Several additional sets of experimental results are shown; the the

Original: First, we visualize the nearest and farthest neighbors of several MNIST digits in Figure 5.
Revised: First, we visualize the nearest and farthest neighbors of several MNIST digits in Figure 5.

Original: Distance is calculated using the partition codes of the images.
Revised: Distance is calculated using the partition codes of the images.

Original: The left column is the original image; The next ﬁve columns are the ﬁve nearest neighbors to the original image; The last ﬁve columns are ﬁve farthest neighbors.
Revised: The left column is the original image; The next ﬁve columns are the ﬁve nearest neighbors to the original image; The last ﬁve columns are ﬁve farthest neighbors.

Original: This ﬁgure shows that partition codes the images are well clustered.
Revised: This ﬁgure shows that partition codes the images are well clustered.

Original: Second, we show the two dimensional projection using t-SNE of the raw pixel and VQ representations of each data points in the MNIST dataset and visualize them in Figure 6.
Revised: Second, we show the two dimensional projection using t-SNE of the raw pixel and VQ representations of each data points in the MNIST dataset and visualize them in Figure 6.

Original: We clearly see a more distinct clustering using VQ representation of the data than using the raw pixel representation.
Revised: We clearly see a more distinct clustering using VQ representation of the data than using the raw pixel representation.

Original: This comparison demonstrate the ability of the RNN to extract useful information from the raw representation of the data in the form of VQ.
Revised: This comparison demonstrate the ability of the RNN to extract useful information from the raw representation of the data in the form of VQ.

Original: Third, we perform a KNN classiﬁcation with k ∈ { 1 , 2 , 5 , 10 } using 1) the RNN computed partition codes of the inputs and 2) raw pixel data representation the MNIST test set to illustrate that the data reparametrized by the RNN has better clustering property than the original data representations.
Revised: Third, we perform a KNN classiﬁcation with k ∈ { 1 , 2 , 5 , 10 } using 1) the RNN computed partition codes of the inputs and 2) raw pixel data representation the MNIST test set to illustrate that the data reparametrized by the RNN has better clustering property than the original data representations.

Original: We use 80% of the test set to train the classiﬁer and the rest for testing.
Revised: We use 80% of the test set to train the classiﬁer and the rest for testing.

Original: The results are reported in
Revised: The results are reported in

Original: Table 3.
Revised: Table 3.

Original: We see that the classiﬁcation accuracies when using RNN computed partition codes of the inputs are signiﬁcantly higher than those when using raw pixel representations.
Revised: We see that the classiﬁcation accuracies when using RNN computed partition codes of the inputs are signiﬁcantly higher than those when using raw pixel representations.

Original: This result again shows the superior quality of the input space partitioning that RNN produces, and may suggest a new way to improve classiﬁcation accuracy by just using the reparametrized data with a KNN classiﬁer.
Revised: This result again shows the superior quality of the input space partitioning that RNN produces, and may suggest a new way to improve classiﬁcation accuracy by just using the reparametrized data with a KNN classiﬁer.

Original: Finally, we visualize the 5 nearest and 5 farthest neighbors of a selected sentence from the SST2 dataset to demonstrate that the partitioning effect on dataset of another modality.
Revised: Finally, we visualize the 5 nearest and 5 farthest neighbors of a selected sentence from the SST2 dataset to demonstrate that the partitioning effect on dataset of another modality.

Original: Again, the distances are computed using the partition codes of the inputs.
Revised: Again, the distances are computed using the partition codes of the inputs.

Original: The results are shown in Figure 7.
Revised: The results are shown in Figure 7.

Original: [Table]
Revised: [Table]

Original: We can see that all sentences that are nearest neighbors are of similar sentiment to the target sentence, whereas all sentences that are farthest neighbors are of the opposite sentiment.
Revised: We can see that all sentences that are nearest neighbors are of similar sentiment to the target sentence, whereas all sentences that are farthest neighbors are of the opposite sentiment.

Original: E A DDITIONAL T EMPLATE V ISUALIZATIONS
Revised: E A DDITIONAL T EMPLATE V ISUALIZATIONS

Original: We provide here more templates on images and texts in Figures 8 and 9.
Revised: We provide here more templates on images and texts in Figures 8 and 9.

Original: Notice here that, although visually the templates may look similar or meaningless, they nevertheless have meaningful inner product with the input.
Revised: Notice here that, although visually the templates may look similar or meaningless, they nevertheless have meaningful inner product with the input.

Original: The class index of the template that produces the largest inner product with the input is typically the correct class, as can be seen in the two ﬁgures.
Revised: The class index of the template that produces the largest inner product with the input is typically the correct class, as can be seen in the two ﬁgures.

Original: F A DDITIONAL E XPERIMENTAL R ESULTS FOR R ANDOM I NITIAL H IDDEN S TATE F. R EGULARIZATION E FFECT FOR R EGRESSION P ROBLEM
Revised: F A DDITIONAL E XPERIMENTAL R ESULTS FOR R ANDOM I NITIAL H IDDEN S TATE F. R EGULARIZATION E FFECT FOR R EGRESSION P ROBLEM

Original: We present the regularization effect on adding task formulated as a regression problem, following setup in Arjovsky et al.
Revised: We present the regularization effect on adding task formulated as a regression problem, following setup in Arjovsky et al.

Original: Result is shown in Figure 10.
Revised: Result is shown in Figure 10.

Original: We see regularization effect similar to that presented in Figure 4, which demonstrates that the regularization effect does indeed happens for both classiﬁcation and regression problems, as Thm.
Revised: We see regularization effect similar to that presented in Figure 4, which demonstrates that the regularization effect does indeed happens for both classiﬁcation and regression problems, as Thm.

Original: 3 suggests
Revised: 3 suggests

Original: . F. C HOOSING S TANDARD D EVIATION IN R ANDOM I NITIAL H IDDEN S TATE
Revised: . F. C HOOSING S TANDARD D EVIATION IN R ANDOM I NITIAL H IDDEN S TATE

Original: Table 4 shows the classiﬁcation accuracies under various settings.
Revised: Table 4 shows the classiﬁcation accuracies under various settings.

Original: The discussion of the results is in
Revised: The discussion of the results is in

Original: Section 6.
Revised: Section 6.

Original: Original
Revised: Original

Original: [Table]
Revised: [Table]

Original: It is a ﬁlm that will have people walking out halfway through , will encourage others to stand up and applaud , and will , undoubtedly , leave both camps engaged in a ferocious debate for years to come .
Revised: It is a ﬁlm that will have people walking out halfway through , will encourage others to stand up and applaud , and will , undoubtedly , leave both camps engaged in a ferocious debate for years to come .

Original: ( + )
Revised: ( + )

Original: Nearest 5 neighbors
Revised: Nearest 5 neighbors

Original: Farthest 5 neighbors
Revised: Farthest 5 neighbors

Original: Well-written , nicely acted and beautifully shot and scored , the ﬁlm works on several levels , openly questioning social mores while ensnaring the audi- ence with its emotional pull .
Revised: Well-written , nicely acted and beautifully shot and scored , the ﬁlm works on several levels , openly questioning social mores while ensnaring the audi- ence with its emotional pull .

Original: ( + , 22.00)
Revised: ( + , 22.00)

Original: Marries the amateurishness of The Blair Witch Project with the illogic of Series 7 : The Contenders to create a completely crass and forgettable movie .
Revised: Marries the amateurishness of The Blair Witch Project with the illogic of Series 7 : The Contenders to create a completely crass and forgettable movie .

Original: A stunning piece of visual poetry that will , hope- fully , be remembered as one of the most important stories to be told in Australia ’s ﬁlm history .
Revised: A stunning piece of visual poetry that will , hope- fully , be remembered as one of the most important stories to be told in Australia ’s ﬁlm history .

Original: ( + , 22.23)
Revised: ( + , 22.23)

Original: K-19 may not hold a lot of water as a submarine epic , but it holds even less when it turns into an elegiacally soggy Saving Private Ryanovich .
Revised: K-19 may not hold a lot of water as a submarine epic , but it holds even less when it turns into an elegiacally soggy Saving Private Ryanovich .

Original: Cute , funny , heartwarming digitally animated fea- ture ﬁlm with plenty of slapstick humor for the kids , lots of in-jokes for the adults and heart enough for everyone .
Revised: Cute , funny , heartwarming digitally animated fea- ture ﬁlm with plenty of slapstick humor for the kids , lots of in-jokes for the adults and heart enough for everyone .

Original: ( + , 22.61)
Revised: ( + , 22.61)

Original: This is a great subject for a movie , but Hollywood has squandered the opportunity , using it as a prop for warmed-over melodrama and the kind of chore- ographed mayhem that director John Woo has built his career on .
Revised: This is a great subject for a movie , but Hollywood has squandered the opportunity , using it as a prop for warmed-over melodrama and the kind of chore- ographed mayhem that director John Woo has built his career on .

Original: Though it is by no means his best work , Laissez-
Revised: Though it is by no means his best work , Laissez-

Original: Passer is a distinguished and distinctive effort by a bona-ﬁde master , a fascinating ﬁlm replete with re- wards to be had by all willing to make the effort to reap them .
Revised: Passer is a distinguished and distinctive effort by a bona-ﬁde master , a fascinating ﬁlm replete with re- wards to be had by all willing to make the effort to reap them .

Original: ( + , 22.78)
Revised: ( + , 22.78)

Original: Flotsam in the sea of moviemaking , not big enough for us to worry about it causing signiﬁcant harm and not smelly enough to bother despising .
Revised: Flotsam in the sea of moviemaking , not big enough for us to worry about it causing signiﬁcant harm and not smelly enough to bother despising .

Original: An absorbing trip into the minds and motivations of people under stress as well as a keen , unsentimental look at variations on the theme of motherhood .
Revised: An absorbing trip into the minds and motivations of people under stress as well as a keen , unsentimental look at variations on the theme of motherhood .

Original: ( + , 22.89)
Revised: ( + , 22.89)

Original: If you ’re not a prepubescent girl , you ’ll be laugh- ing at Britney Spears ’ movie-starring debut whenever it does n’t have you impatiently squinting at your watch .
Revised: If you ’re not a prepubescent girl , you ’ll be laugh- ing at Britney Spears ’ movie-starring debut whenever it does n’t have you impatiently squinting at your watch .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: image 0, -6.
Revised: image 0, -6.

Original: [Figure]
Revised: [Figure]

Original: G P ROOFS G. P ROOF OF T HM .
Revised: G P ROOFS G. P ROOF OF T HM .

Original: To simplify notation, similar to the main text, in the proof here we drop the afﬁne parameters’ dependencies on the input, but keep in mind the input-dependency of these parameters.
Revised: To simplify notation, similar to the main text, in the proof here we drop the afﬁne parameters’ dependencies on the input, but keep in mind the input-dependency of these parameters.

Original: We ﬁrst derive the expression for a hidden state h ( (cid:96),t ) at a given time step t and layer (cid:96) .
Revised: We ﬁrst derive the expression for a hidden state h ( (cid:96),t ) at a given time step t and layer (cid:96) .

Original: Using
Revised: Using

Original: Prop. 1, we start with unrolling the RNN cell of layer (cid:96) at time step t for two time steps to t − 2 by recursively applying (1) as follows:
Revised: Prop. 1, we start with unrolling the RNN cell of layer (cid:96) at time step t for two time steps to t − 2 by recursively applying (1) as follows:

Original: [Table]
Revised: [Table]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: From (12) to (13), we use the result of Prop. 1.
Revised: From (12) to (13), we use the result of Prop. 1.

Original: From (13) to (14), we expand the term that involves the hidden state h ( (cid:96),t − 1) at the previous time step recursively using (13).
Revised: From (13) to (14), we expand the term that involves the hidden state h ( (cid:96),t − 1) at the previous time step recursively using (13).

Original: From (14) to (15), we group terms and write them compactly.
Revised: From (14) to (15), we group terms and write them compactly.

Original: Now deﬁne A ( (cid:96),s ) s : t := (cid:81) s
Revised: Now deﬁne A ( (cid:96),s ) s : t := (cid:81) s

Original: +1 k
Revised: +1 k

Original: = t A ( (cid:96),k ) σ
Revised: = t A ( (cid:96),k ) σ

Original: W ( (cid:96) )
Revised: W ( (cid:96) )

Original: r for s < t and A ( (cid:96),t ) t : t := I where I is the identity matrix.
Revised: r for s < t and A ( (cid:96),t ) t : t := I where I is the identity matrix.

Original: Using this deﬁnition
Revised: Using this deﬁnition

Original: [Equation]
Revised: [Equation]

Original: we rewrite (15) and proceed with the unrolling to the initial time step as follows:
Revised: we rewrite (15) and proceed with the unrolling to the initial time step as follows:

Original: [Equation]
Revised: [Equation]

Original: where B ( (cid:96)
Revised: where B ( (cid:96)

Original: Repeat the above derivation for t ∈ { 1 , · · · , T } and stack h ( (cid:96),t ) in decreasing time steps from top to bottom, we have:
Revised: Repeat the above derivation for t ∈ { 1 , · · · , T } and stack h ( (cid:96),t ) in decreasing time steps from top to bottom, we have:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: (cid:80) t =
Revised: (cid:80) t =

Original: T
Revised: T

Original: A ( (cid:96) ) t : T B ( (cid:96),t )
Revised: A ( (cid:96) ) t : T B ( (cid:96),t )

Original: + A ( (cid:96) ) 0: T h ( (cid:96), 0) ...
Revised: + A ( (cid:96) ) 0: T h ( (cid:96), 0) ...

Original: A ( (cid:96) ) 1:1 B ( (cid:96),t )
Revised: A ( (cid:96) ) 1:1 B ( (cid:96),t )

Original: + A ( (cid:96) )
Revised: + A ( (cid:96) )

Original: 0:1 h ( (cid:96), 0)    , h ( (cid:96) − 1) =   h ( (cid:96) − 1 ,T ) ...
Revised: 0:1 h ( (cid:96), 0)    , h ( (cid:96) − 1) =   h ( (cid:96) − 1 ,T ) ...

Original: h ( (cid:96) − 1 , 1)   , which concludes the proof.
Revised: h ( (cid:96) − 1 , 1)   , which concludes the proof.

Original: Thm. 2 follows from the recursive application of the above arguments for each layer (cid:96) ∈ { 1 , · · · , L } .
Revised: Thm. 2 follows from the recursive application of the above arguments for each layer (cid:96) ∈ { 1 , · · · , L } .

Original: G. P ROOF OF T HM .
Revised: G. P ROOF OF T HM .

Original: We prove for the case of multi-class classiﬁcation problem with softmax output.
Revised: We prove for the case of multi-class classiﬁcation problem with softmax output.

Original: The proof for the case of regression problems easily follows.
Revised: The proof for the case of regression problems easily follows.

Original: Let a i be the i th row of the input-dependent afﬁne parameter
Revised: Let a i be the i th row of the input-dependent afﬁne parameter

Original: A h where A h :
Revised: A h where A h :

Original: = A 1: T = (cid:81) 1 s = T A ( (cid:96),s ) σ
Revised: = A 1: T = (cid:81) 1 s = T A ( (cid:96),s ) σ

Original: W ( (cid:96) )
Revised: W ( (cid:96) )

Original: r (recall Section 6), x n =
Revised: r (recall Section 6), x n =

Original: [ x (1) (cid:62) n , · · · , x ( T ) (cid:62) n ] (cid:62) be the concatenation of the n th input sequence of length T and c be the index of the correct class.
Revised: [ x (1) (cid:62) n , · · · , x ( T ) (cid:62) n ] (cid:62) be the concatenation of the n th input sequence of length T and c be the index of the correct class.

Original: We assume the amplitude of random initial hidden state is small so that the input-dependent afﬁne parameter A h , which also depends on h (0) , does not change when using random h (0) .
Revised: We assume the amplitude of random initial hidden state is small so that the input-dependent afﬁne parameter A h , which also depends on h (0) , does not change when using random h (0) .

Original: Also, let z n = f RNN ( x n , h (0) ) be the overall RNN computation that represents (9).
Revised: Also, let z n = f RNN ( x n , h (0) ) be the overall RNN computation that represents (9).

Original: We ﬁrst rewrite the cross entropy loss with random initial hidden state (cid:101)
Revised: We ﬁrst rewrite the cross entropy loss with random initial hidden state (cid:101)

Original: L CE =
Revised: L CE =

Original: L CE (cid:0) softmax (cid:0) f RNN (cid:0) x n , h (0) + (cid:15) (cid:1)(cid:1)(cid:1) as follows:
Revised: L CE (cid:0) softmax (cid:0) f RNN (cid:0) x n , h (0) + (cid:15) (cid:1)(cid:1)(cid:1) as follows:

Original: [Equation]
Revised: [Equation]

Original: Taking expectation of the (cid:101) L with respect to the distribution of the random Gaussian vector that the initial hidden state is set to, we have
Revised: Taking expectation of the (cid:101) L with respect to the distribution of the random Gaussian vector that the initial hidden state is set to, we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: R = 1 N
Revised: R = 1 N

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We now simplify (21) using second order Taylor expansion on h (0) of the summation inside the log function.
Revised: We now simplify (21) using second order Taylor expansion on h (0) of the summation inside the log function.

Original: Deﬁne function u ( x (1: T ) n , h (0) ) :
Revised: Deﬁne function u ( x (1: T ) n , h (0) ) :

Original: = log ( (cid:80) j exp ( z nj ))
Revised: = log ( (cid:80) j exp ( z nj ))

Original: =
Revised: =

Original: log ( (cid:80) j exp ( f ( x (1: T ) n , h (0) ))) .
Revised: log ( (cid:80) j exp ( f ( x (1: T ) n , h (0) ))) .

Original: Then, we can approximate (21) as follows:
Revised: Then, we can approximate (21) as follows:

Original: [Equation]
Revised: [Equation]

Original: where d u ( x n , h (0) )
Revised: where d u ( x n , h (0) )

Original: d 2 h (0) is the Hessian matrix:
Revised: d 2 h (0) is the Hessian matrix:

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we can write the trace term in (22) as follows:
Revised: we can write the trace term in (22) as follows:

Original: [Equation]
Revised: [Equation]

Original: As a result, using the above approximations
Revised: As a result, using the above approximations

Original: [Equation]
Revised: [Equation]

Original: we can rewrite the loss with random initial state in (19) as:
Revised: we can rewrite the loss with random initial state in (19) as:

Original: [Equation]
Revised: [Equation]

Original: We see that this regularizer term does not dependent on the correct class index c of each data points.
Revised: We see that this regularizer term does not dependent on the correct class index c of each data points.

Original: H P RIOR W ORK ON THE E XPLODING G RADIENT IN RNN S
Revised: H P RIOR W ORK ON THE E XPLODING G RADIENT IN RNN S

Original: The problem of exploding gradients has been widely studied from different perspectives.
Revised: The problem of exploding gradients has been widely studied from different perspectives.

Original: First approaches have attempted to directly control the amplitude of the gradient through gradient clipping (Pascanu et al., 2013).
Revised: First approaches have attempted to directly control the amplitude of the gradient through gradient clipping (Pascanu et al., 2013).

Original: A more model driven approach has leveraged the analytical formula of the gradient when using speciﬁc nonlinearities and topologies in order to develop parametrization of the recurrent weights.
Revised: A more model driven approach has leveraged the analytical formula of the gradient when using speciﬁc nonlinearities and topologies in order to develop parametrization of the recurrent weights.

Original: This has led to various unitary reparametrizations of the recurrent weight (Arjovsky et al., 2016;
Revised: This has led to various unitary reparametrizations of the recurrent weight (Arjovsky et al., 2016;

Original: Wisdom et al., 2016;
Revised: Wisdom et al., 2016;

Original: Helfrich et al., 2018; Henaff et al., 2016;
Revised: Helfrich et al., 2018; Henaff et al., 2016;

Original: Jing et al., 2017; Mhammedi et al., 2017; Hyland & R¨atsch, 2017; Jose et al., 2018).
Revised: Jing et al., 2017; Mhammedi et al., 2017; Hyland & R¨atsch, 2017; Jose et al., 2018).

Original: A soft version of such parametrization lies in regularization of the DNNs.
Revised: A soft version of such parametrization lies in regularization of the DNNs.

Original: This includes dropout applied to either the output layer (Pham et al., 2014) or hidden state (Zaremba et al., 2014), noisin (Dieng et al., 2018), zoneout (Krueger et al., 2017) and recurrent batch normalization (Cooijmans et al., 2017).
Revised: This includes dropout applied to either the output layer (Pham et al., 2014) or hidden state (Zaremba et al., 2014), noisin (Dieng et al., 2018), zoneout (Krueger et al., 2017) and recurrent batch normalization (Cooijmans et al., 2017).

Original: Lastly, identity initialization of ReLU RNNs has been studied in Le et al.
Revised: Lastly, identity initialization of ReLU RNNs has been studied in Le et al.

Original: and Talathi & Vartak (2016).
Revised: and Talathi & Vartak (2016).

Original: Our results complements prior works in that simply using random initial hidden state instead of zero initial hidden state and without changing the RNN structure also relieves the exploding gradient problem by regularization the potentially largest term in the recurrent weight gradient.
Revised: Our results complements prior works in that simply using random initial hidden state instead of zero initial hidden state and without changing the RNN structure also relieves the exploding gradient problem by regularization the potentially largest term in the recurrent weight gradient.