Original: A M AX -A FFINE S PLINE P ERSPECTIVE OF R ECURRENT N EURAL N ETWORKS
Revised: A M AX -A FFINE S PLINE P ERSPECTIVE OF R ECURRENT N EURAL N ETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We develop a framework for understanding and improving recurrent neural networks (RNNs) using max-afﬁne spline operators (MASO).
Revised: We develop a framework for understanding and improving recurrent neural networks (RNNs) using max-afﬁne spline operators (MASOs).

Original: We prove that RNNs using piecewise afﬁne and convex nonlinearities can be written as a simple piecewise afﬁne spline operator.
Revised: We prove that RNNs using piecewise afﬁne and convex nonlinearities can be written as a simple piecewise afﬁne spline operator.

Original: The resulting representation provides several new perspectives for analyzing RNNs, three of which we study in this paper.
Revised: The resulting representation provides several new perspectives for analyzing RNNs, three of which we study in this paper.

Original: First, we show that an RNN internally partitions the input space during training and that it builds up the partition through time.
Revised: First, we show that an RNN internally partitions the input space during training and that it builds up the partition through time.

Original: Second, we show that the afﬁne  parameter of an RNN corresponds to an input-speciﬁc template, from which we can interpret an RNN as performing a simple template matching (matched ﬁltering) given the input.
Revised: Second, we show that the afﬁne slope parameter of an RNN corresponds to an input-speciﬁc template, from which we can interpret an RNN as performing a simple template matching (matched ﬁltering) given the input.

Original: Third, by closely examining the MASO RNN formula, we prove that injecting Gaussian noise in the initial hidden statein RNNs corresponds to an explicit (cid:96)regularization on the afﬁne parameters, which links to exploding gradient issues and improves generalization.
Revised: Third, by carefully examining the MASO RNN afﬁne mapping, we prove that using a random initial hidden state corresponds to an explicit (cid:96) 2 regularization of the afﬁne parameters, which can mollify exploding gradients and improve generalization.

Original: Extensive experiments on several datasets of various modalities demonstrate and validate each of the above analyses.
Revised: Extensive experiments on several datasets of various modalities demonstrate and validate each of the above conclusions.

Original: In particular, using  initial hidden states elevates simple RNNs to  state-of-the-art performance on these datasets.
Revised: In particular, using a random initial hidden states elevates simple RNNs to near state-of-the-art performers on these datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recurrent neural networks (RNNs) are a powerful class of models for processing sequential inputs and have been the basic building block for more advanced models that have found success in challenging problems, including  classiﬁcation (e g., sentiment analysis (Socher et al., 2013; Li et al., 2016; Teng et al., 2016)), sequence generation (e g., machine translation (Bahdanau et al., 2014)), speech recognition, and image captioning.
Revised: Recurrent neural networks (RNNs) are a powerful class of models for processing sequential inputs and a basic building block for more advanced models that have found success in challenging problems involving sequential data, including sequence classiﬁcation (e.g., sentiment analysis (Socher et al., 2013) , sequence generation (e.g., machine translation (Bahdanau et al., 2014)), speech recognition (Graves et al., 2013) and image captioning (Mao et al., 2015).

Original: Despite the successes, our understanding of how RNNs workstill remains limited.
Revised: Despite their success, however, our understanding of how RNNs work remains limited.

Original: An attractive theoretical result is the universal approximation property that states that an RNN can approximate an arbitrary function (Sch¨afer & Zimmermann, 2006; Siegelmann & Sontag, 1995; Hammer, 2000).
Revised: For instance, an attractive theoretical result is the universal approximation property that states that an RNN can approximate an arbitrary function (Sch ¨ afer & Zimmermann, 2006; Siegelmann & Sontag, 1995; Hammer, 2000).

Original: These classical theoretical results have been obtained primarily from a dynamical system (Siegelmann & Sontag, 1995; Sch ¨ afer & Zimmermann, 2006) and measure theory (Hammer, 2000) perspective.
Revised: These classical theoretical results have been obtained primarily from the dynamical system (Siegelmann & Sontag, 1995; Sch¨afer & Zimmermann, 2006) and measure theory (Hammer, 2000) perspectives.

Original: These theories provide approximation error bounds but unfortunately limited guidance on applying RNNs and understanding their performance and behaviors in practice.
Revised: These theories provide approximation error bounds but unfortunately limited guidance on applying RNNs and understanding their performance and behavior in practice.

Original: In this paper, we provide a new angle for understanding RNNs using max-afﬁne spline operators (MASOs) (Magnani & Boyd, 2009; Hannah & Dunson, 2013) from approximation theory.
Revised: In this paper, we provide a new angle for understanding RNNs using max-afﬁne spline operators (MASOs) (Magnani & Boyd, 2009; Hannah & Dunson, 2013) from approximation theory.

Original: MASOs are a piecewise afﬁne approximation to arbitrary functions that provide a  useful framework to study neural networks.
Revised: The piecewise afﬁne approximations made by compositions of MASOs provide a new and useful framework to study neural networks.

Original: For example, Balestriero & Baraniuk (2018); Balestriero & Baraniuk (2018a) have provided  detailed analysis in the context of feedforwardnetworks that highlights the practical value that spline operators bring to interpreting deep networks.
Revised: For example, Balestriero & Baraniuk (2018); Balestriero & Baraniuk (2018a) have provided a detailed analysis in the context of feedforward networks.

Original: Here, we go one step further and demonstrate the new insights and interpretations from the MASO perspective for RNNs.
Revised: Here, we go one step further and ﬁnd new insights and interpretations from the MASO perspective for RNNs.

Original: We will see that the input space partitioning and matched ﬁltering views extend to RNNs and yield interesting insights into the inner workings of RNNs.
Revised: We will see that the input space partitioning and matched ﬁltering links developed in Balestriero & Baraniuk (2018); Balestriero & Baraniuk (2018a) extend to RNNs and yield interesting insights into their inner workings.

Original: Moreover, the MASO formulation of RNNs allows us to theoretically justify the use of noise in initial hidden state and improve RNN performance.
Revised: Moreover, the MASO formulation of RNNs enables us to theoretically justify the use of a random initial hidden state to improve RNN performance.

Original: For concreteness, We focus our analysis on a speciﬁc class of simple RNNs (Elman, 1990) with piecewise afﬁne and convex nonlinearities such as the ReLU (Glorot et al., 2011).
Revised: For concreteness, we focus our analysis on a speciﬁc class of simple RNNs (Elman, 1990) with piecewise afﬁne and convex nonlinearities such as the ReLU (Glorot et al., 2011).

Original: RNNs withsuch nonlinearities have recently gained considerable attention due to their ability to combat the exploding gradient problem; with proper initialization (Le et al., 2015; Talathi & Vartak, 2016) and h ( (cid:96),t )
Revised: RNNs with h (2 , 2) h (2 , 1) h (2 , 3) h (1 , 2) h (1 , 1) h (1 , 3) W (2) r W (2) r W (1) r

Original: [Figure] ( (cid:96),t − 1) h ( (cid:96),t +1)
Revised: W (1) r W (1) r W (2) W (2) W (2) h (2 , 0) h (1 , 0)

Original: h ( (cid:96) − 1 ,t ) h ( (cid:96) − 1 ,t − 1) h ( (cid:96) − 1 ,t +1) W ( (cid:96) ) r
Revised: W (2) r

Original: W ( (cid:96) ) r W ( (cid:96) − 1) r W ( (cid:96) − 1) r W ( (cid:96) − 1)
Revised: W (1) r x (1) x (2) x (3)

Original: [Figure]

Original: ( (cid:96) ) [Figure] [Figure] W ( (cid:96)
Revised: W (1)

Original: [Figure] W ( (cid:96) [Figure] [Figure]
Revised: W (1)

Original: ( (cid:96) [Figure] W ( (cid:96) ) W ( (cid:96) ) clever parametrization of the recurrent weight (Arjovsky et al., 2016; Wisdom et al., 2016;
Revised: W (1) such nonlinearities have recently gained considerable attention due to their ability to combat the exploding gradient problem ; with proper initialization (Le et al., 2015; Talathi & Vartak, 2016) and clever parametrization of the recurrent weight (Arjovsky et al., 2016; Wisdom et al., 2016;

Original: Jing et al., 2017; Hyland & R¨atsch, 2017; Mhammedi et al., 2017; Helfrich et al., 2018), these RNNs achieve performance on par with more complex ones such as LSTMs.
Revised: Jing et al., 2017; Hyland & R¨atsch, 2017; Mhammedi et al., 2017; Helfrich et al., 2018), these RNNs achieve performance on par with more complex ones such as LSTMs.

Original: Below is a summary of our key contributions:
Revised: Below is a summary of our key contributions.

Original: Contribution 1.
Revised: Contribution 1.

Original: We prove that an RNN with piecewise afﬁne and convex nonlinearities can be rewritten as a composition of MASOs, making it a piecewise afﬁne spline operator that has an elegant analytical form (Section 2).
Revised: We prove that an RNN with piecewise afﬁne and convex nonlinearities can be rewritten as a composition of MASOs, making it a piecewise afﬁne spline operator with an elegant analytical form (Section 3).

Original: Contribution 2.
Revised: Contribution 2.

Original: We leverage the partitioning of piecewise afﬁne spline operators to analyze the input space partitioning that  RNN implicitly performs.
Revised: We leverage the partitioning of piecewise afﬁne spline operators to analyze the input space partitioning that an RNN implicitly performs.

Original: We show that  RNN calculates a new, high dimensional representation  of an input sequence that captures informative underlying characteristics of the input.
Revised: We show that an RNN calculates a new, high-dimensional representation (the partition code) of the input sequence that captures informative underlying characteristics of the input.

Original: We also provide a new perspective for RNN dynamics by visualizing the evolution of  RNN VQ partitioning through time. (Section 3).
Revised: We also provide a new perspective on RNN dynamics by visualizing the evolution of the RNN input space partitioning through time (Section 4).

Original: Contribution 3.
Revised: Contribution 3.

Original: We show the piecewise afﬁne mapping in an RNN associated with a given input sequence is an input-dependent template, from which we can interpret the RNN as performing greedy template matching  at every RNN cell. (Section 4).
Revised: We show the piecewise afﬁne mapping in an RNN associated with a given input sequence corresponds to an input-dependent template, from which we can interpret the RNN as performing greedy template matching (matched ﬁltering) at every RNN cell (Section 5).

Original: Contribution 4.
Revised: Contribution 4.

Original: We rigorously prove that adding noise to the initial hidden state of an RNN corresponds to an explicit regularizer that links to exploding gradient and empirically show  that such  regularization improves RNN performance  on four datasets of different modalities (Section 5).
Revised: We rigorously prove that using a random (rather than zero) initial hidden state in an RNN corresponds to an explicit regularizer that can mollify exploding gradients. We show empirically that such a regularization improves RNN performance (to state-of-the-art) on four datasets of different modalities (Section 6).

Original: 1.  B ACKGROUND: R ECURRENT N EURAL N ETWORKS (RNN S )
Revised:  B ACKGROUND

Revised: Recurrent Neural Networks (RNNs).

Original: A simple RNN unit (Elman, 1990) per layer (cid:96) and time step t , referred to as a “cell”, performs the following recursive computation:
Revised: A simple RNN unit (Elman, 1990) per layer (cid:96) and time step t , referred to as a “cell,” performs the following recursive computation

Original: [Equation]
Revised: [Equation]

Original: where h ( (cid:96),t ) is the hidden state at layer (cid:96) and time step t , h (0 ,t ) :
Revised: where h ( (cid:96),t ) is the hidden state at layer (cid:96) and time step t , h (0 ,t ) :

Original: = x ( t ) which is the input sequence, σ is an activation function and W ( (cid:96) ) , W ( (cid:96) ) r  and b ( (cid:96) ) are time-invariant parameters at layer (cid:96) .
Revised: = x ( t ) which is the input sequence, σ is an activation function and W ( (cid:96) ) , W ( (cid:96) ) r , and b ( (cid:96) ) are time-invariant parameters at layer (cid:96) .

Revised: h ( (cid:96), 0) is the initial hidden state at layer (cid:96) which needs to be set to some value beforehand to start the

Original:  Unrolling the RNN through time gives an intuitive view of the RNN dynamics, which we visualize in Figure 1.
Revised: RNN recursive computation. Unrolling the RNN through time gives an intuitive view of the RNN dynamics, which we visualize in Figure 1.

Original: The output of the overall RNN is usually an afﬁne transformation of  hidden state of the last layer L ateach time step t:
Revised: The output of the overall RNN is typically an afﬁne transformation of the hidden state of the last layer L at time step t

Original: [Equation]
Revised: [Equation]

Original:  In the special case when the RNN has  one output at the end of processing the entire input sequence, the RNN output is an afﬁne transformation of the hidden state at the last time step, i.e., z ( T ) =
Revised: [Equation] In the special case where the RNN has only one output at the end of processing the entire input sequence, the RNN output is an afﬁne transformation of the hidden state at the last time step, i.e., z ( T ) =

Original: W h ( L,T ) + b .
Revised: W h ( L,T ) + b .

Original: [Equation]
Revised: Max-Afﬁne Spline Operators (MASOs).

Original: -afﬁne spline operator (MASO) is piecewise afﬁne and convex with respect to each output dimension
Revised: A max-afﬁne spline operator (MASO) is piecewise afﬁne and convex with respect to each output dimension

Original: k
Revised: k

Original: = 1 , . .
Revised: = 1 , . .

Original: , K .
Revised: , K .

Original: It is deﬁned as a parametric function S : R D
Revised: It is deﬁned as a parametric function S : R D

Original: → R K with parameters A ∈ R K × R × D and B ∈ R K × R .
Revised: → R K with parameters A ∈ R K × R × D and B ∈ R K × R .

Original: A MASO leverages K independent max-afﬁne splines (Magnani & Boyd, 2009), each with R partition regions.
Revised: A MASO leverages K independent max-afﬁne splines (Magnani & Boyd, 2009), each with R partition regions.

Original: Its output foreach output dimension k is produced via
Revised: Its output for output dimension k is produced via

Original: [Equation]
Revised: [Equation]

Original: where x ∈ R D and y ∈ R K are dummy variables that respectively denote the input and output of the MASO S and (cid:104)· , ·(cid:105) denotes inner product.
Revised: where x ∈ R D and y ∈ R K are dummy variables that respectively denote the input and output of the MASO S and (cid:104)· , ·(cid:105) denotes inner product.

Original: The three subscripts of the “slope” parameter [ A ] k,r,d correspond to output k , partition region r , and input signal index d .
Revised: The three subscripts of the “slope” parameter [ A ] k,r,d correspond to output k , partition region r , and input signal index d .

Original: The two subscripts of the “bias” parameter [ B ] k,r correspond to output k and partition region r .
Revised: The two subscripts of the “bias” parameter [ B ] k,r correspond to output k and partition region r .

Original: We highlight two important and interrelated MASO properties relevant to the discussions throughout the paper.
Revised: We highlight two important and interrelated MASO properties relevant to the discussions throughout the paper.

Original: First, a MASO performs implicit input space partitioning, which is made explicit by rewriting Eq. 3 as
Revised: First, a MASO performs implicit input space partitioning, which is made explicit by rewriting (3) as

Original: [Equation]
Revised: [Equation]

Original: where Q ∈ R K × R is a partition selection matrix 1 calculated as 
Revised: where Q ∈ R K × R is a partition selection matrix 1 calculated as [ Q ] k,r = 1 ( r = [ r ∗ ] k ) , where [ r ∗ ]

Original: [Equation]
Revised: [Equation]

Original: where [ r ∗ ] k = arg
Revised: = arg

Original: [Equation]
Revised: [Equation]

Original: Namely, Q contains K stacked one-hot row vectors, each of which selects the [ r ∗ ] th k partition of the input space that maximizes Eq. 4 foreach output dimension k .
Revised: Namely, Q contains K stacked one-hot row vectors, each of which selects the [ r ∗ ] th k partition of the input space that maximizes (4) for output dimension k .

Original: As a consequence, knowing Q is equivalent to knowing the partition of an input x that the MASO implicitly computes.
Revised: As a consequence, knowing Q is equivalent to knowing the partition of an input x that the MASO implicitly computes.

Original: We will use this property in Section 3 to provide new insights into RNN dynamics.
Revised: We will use this property in Section 4 to provide new insights into RNN dynamics.

Original: Second, given the partition r ∗ that an input belongs to, as determined by Eq. 5, the output of  MASO ofeach dimension k (Eq. 3) reduces to a simple afﬁne transformation of the input:
Revised: Second, given the partition r ∗ that an input belongs to, as determined by (5), the output of the MASO of dimension k from (3) reduces to a simple afﬁne transformation of the input [ y ] k

Original: [Equation] k, [Equation]
Revised: =

Original:  k , where [ A ] k, · = [ A ] k, [ r ∗ ] k, · and [ B ] k, · = [ B ] k, [ r ∗ ] k .
Revised: [ A ] k, · x + [ B ] k , where [ A ] k, · = [ A ] k, [ r ∗ ] k and [ B ] k, · = [ B ] k, [ r ∗ ] k .

Original: Here, the selected afﬁne parameters A ∈ R K × D and B ∈ R K are speciﬁc to the input’s partition region [ r ∗ ] k , which are simply the [ r ∗ ] th k slice and [ r ∗ ] th k column of A and B , respectively, foreach output dimension k .
Revised: Here, the selected afﬁne parameters A ∈ R K × D and B ∈ R K are speciﬁc to the input’s partition region [ r ∗ ] k , which are simply the [ r ∗ ] th k slice and [ r ∗ ] th k column of A and B , respectively, for output dimension k .

Original: We emphasize that A and B are input-dependent: different input x gives rise to different A and B .
Revised: We emphasize that A and B are input-dependent; different inputs x induce different A and B .

Original: 2 We will use this property in Section 4 to link RNNs to matched ﬁlterbanks.
Revised: We will use this property in Section 5 to link RNNs to matched ﬁlterbanks.

Original:  RNN S AS P IECEWISE A FFINE S PLINE O PERATORS
Revised:  RNN S AS P IECEWISE A FFINE S PLINE O PERATORS

Original: We now leverage the MASO framework to rewrite, interpret and analyze RNNs.
Revised: We now leverage the MASO framework to rewrite, interpret, and analyze RNNs.

Original: We focus on RNNs with piecewise afﬁne and convex nonlinearities in order to derive rigorous analytical results.
Revised: We focus on RNNs with piecewise afﬁne and convex nonlinearities in order to derive rigorous analytical results.

Original: Our goal of this section is to show that an RNN has a simple, elegant analytical form when rewritten using MASOs.

Original: Analyses of RNNs with other nonlinearities are left for future work.
Revised: The analysis of RNNs with other nonlinearities is left for future work.

Original: We ﬁrst derive the MASO formula for an RNN cell (Eq. 1), then extend to one layer of time-unrolled RNN and ﬁnally to a multi-layer, time-unrolled RNN.
Revised: We ﬁrst derive the MASO formula for an RNN cell (1) and then extend to one layer of a timeunrolled RNN and ﬁnally to a multi-layer, time-unrolled RNN.

Original: Let z ( (cid:96),t ) =
Revised: Let z ( (cid:96),t ) =

Original: (cid:2) h [Equation] , h 
Revised: (cid:2) h ( (cid:96) − 1 ,t ) (cid:62) , h ( (cid:96),t

Original: [Equation]
Revised: [Equation]

Original:  (cid:62) be the input to an RNN cell which is the concatenation of the current input
Revised: 1) (cid:62) (cid:3) (cid:62) be the input to an RNN cell that is the concatenation of the current input

Original: h ( (cid:96) − 1 ,t ) and the previous hidden state h ( (cid:96),t − 1) .
Revised: h ( (cid:96) − 1 ,t ) and the previous hidden state h ( (cid:96),t − 1) .

Original: Then we have the following result, which is a straightforward extension of Prop.in Balestriero & Baraniuk (2018).
Revised: Then we have the following result, which is a straightforward extension of Proposition 4 in Balestriero & Baraniuk (2018).

Original: Proposition 1.
Revised: Proposition 1.

Original: An RNN cell of the form in Eq. 1 is a MASO deﬁned as follows:
Revised: An RNN cell of the form (1) is a MASO with

Original: [Equation]
Revised: [Equation]

Original:  where A ( (cid:96),t ) =
Revised: (7) where A ( (cid:96),t ) =

Original: A ( (cid:96),t ) σ [Equation] ) , W ( (cid:96) )
Revised: A ( (cid:96),t ) σ [ W ( (cid:96) ) , W ( (cid:96) )

Original: r ] and B ( t ) =
Revised: r ] and B ( t ) =

Original: A ( (cid:96),t ) σ b ( (cid:96) ) are the afﬁne parameters and A ( (cid:96),t ) σ is the afﬁne parameter corresponding to the  nonlinearity σ ( · )3 that depend on the cell input z
Revised: A ( (cid:96),t ) σ b ( (cid:96) ) are the afﬁne parameters and A ( (cid:96),t ) σ is the afﬁne parameter corresponding to the piecewise afﬁne and convex nonlinearity σ ( · ) that depends on the cell input z

Original: ( (cid:96),t ) .
Revised: ( (cid:96),t ) .

Original: We then proceed to derive the explicit afﬁne formula of a time-unrolled RNN at layer (cid:96) .
Revised: We now derive an explicit afﬁne formula for a time-unrolled RNN at layer (cid:96) .

Original: Let h ( (cid:96) − 1) =
Revised: Let h ( (cid:96) − 1) =

Original: (cid:2) h ( (cid:96) − 1 , 1) (cid:62) , · · · , h ( (cid:96) − 1 ,T ) (cid:62) (cid:3) (cid:62) be the entire input sequence to the RNN at layer (cid:96)  and  h ( (cid:96) )
Revised: (cid:2) h ( (cid:96) − 1 , 1) (cid:62) , · · · , h ( (cid:96) − 1 ,T ) (cid:62) (cid:3) (cid:62) be the entire input sequence to the RNN at layer (cid:96) , and let h ( (cid:96) )

Original: = (cid:2) h
Revised: = (cid:2) h

Original: ( (cid:96), 1) (cid:62) , · · · , h ( (cid:96),T ) (cid:62) (cid:3) (cid:62) be all the hidden states that are the outputs of the RNN at layer (cid:96) .
Revised: ( (cid:96), 1) (cid:62) , · · · , h ( (cid:96),T ) (cid:62) (cid:3) (cid:62) be all the hidden states that are output at layer (cid:96) .

Original: With some algebra and simpliﬁcation, we arrive at the following result.
Revised: After some algebra and simpliﬁcation, we arrive at the following result.

Original: Theorem 1.
Revised: Theorem 1.

Original: The (cid:96) th layer of an RNN is a piecewise afﬁne spline operator deﬁned as follows:
Revised: The (cid:96) th layer of an RNN is a piecewise afﬁne spline operator deﬁned as [Equation] [Equation] (cid:125)

Revised: [Equation]

Revised: 

Original: [Equation]
Revised: [Equation]

Revised: (cid:96)

Revised: [Equation]

Revised: T )

Revised: [Equation]

Original: where
Revised: h [Equation] (cid:96) [Equation]

Revised: [Equation]

Revised: 1) 

Revised: [Equation]

Revised: 

Revised: [Equation]

Revised: [Equation]

Revised: 

Original: [Equation]
Revised: [Equation]

Revised: (8) where A ( (cid:96) )

Revised: t : T (cid:48) =

Revised: (cid:16)(cid:81)

Revised: t +1 s

Original:  (cid:17) for t < T (cid:48) and identity otherwise, 4 h ( (cid:96), 0) is the initial hidden state of the RNN at layer (cid:96)  and A ( (cid:96) )RNN and B ( (cid:96) )
Revised: = T (cid:48) A ( (cid:96),s ) σ W ( (cid:96) ) r (cid:17) for t < T (cid:48) and identity otherwise, 3 h ( (cid:96), 0) is the initial hidden state of the RNN at layer (cid:96) , and A ( (cid:96) )RNN and B ( (cid:96) )

Original: RNN are afﬁne parameters that depend on the layer input h ( (cid:96) − 1) and the initial hidden state h ( (cid:96), 0) .
Revised: RNN are afﬁne parameters that depend on the layer input h ( (cid:96) − 1) and the initial hidden state h ( (cid:96), 0) .

Original: We present the proof for Thm. 1 in Appendix G.
Revised: We present the proof for Theorem 1 in Appendix G.

Original: The key point here is that, by leveraging MASOs, we can represent the time-unrolled RNN as a simple afﬁne transformation of the entire input sequence (Eq. 7).
Revised: The key point here is that, by leveraging MASOs, we can represent the time-unrolled RNN as a simple afﬁne transformation of the entire input sequence (8).

Original: Note that this afﬁne transformation changes depending on the partition region in which the input belongsto (recall Eqs. 4 and 5).
Revised: Note that this afﬁne transformation changes depending on the partition region in which the input belongs (recall (4) and (5)).

Original: Note also that the initial hidden state affects the layer output by inﬂuencing the afﬁne parameters and contributing a bias term A ( (cid:96) ) 0:
Revised: Note also that the initial hidden state affects the layer output by inﬂuencing the afﬁne parameters and contributing a bias term A ( (cid:96) ) 0:

Original: t h ( (cid:96), 0) to the bias parameter B ( (cid:96) )RNN .
Revised: t h ( (cid:96), 0) to the bias parameter B ( (cid:96) )RNN .

Original: We study the impact of the initial hidden state in more detail in Section 5.
Revised: We study the impact of the initial hidden state in more detail in Section 6.

Original: We are now ready to generalize the above result to multi-layer RNNs.
Revised: We are now ready to generalize the above result to multi-layer RNNs.

Original: Let x = (cid:2) x (1) (cid:62) , · · · , x ( T ) (cid:62) (cid:3) (cid:62) be the input sequence to a multi-layer RNN and and  z = (cid:2) z (1) (cid:62) , · · · , z ( T ) (cid:62) (cid:3) (cid:62)  the output sequence.
Revised: Let x = (cid:2) x (1) (cid:62) , · · · , x ( T ) (cid:62) (cid:3) (cid:62) be the input sequence to a multi-layer RNN, and let z = (cid:2) z (1) (cid:62) , · · · , z ( T ) (cid:62) (cid:3) (cid:62) be the output sequence.

Original: We state the following result for the overall mapping of a multi-layer RNN.
Revised: We state the following result for the overall mapping of a multi-layer RNN.

Original: Theorem 2.
Revised: Theorem 2.

Original: The output of an L -layer RNN is a piecewise afﬁne spline operator deﬁned as
Revised: The output of an L -layer RNN is a piecewise afﬁne spline operator deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where A RNN =
Revised: where A RNN =

Original: (cid:81) 1 (cid:96) =
Revised: (cid:81) 1 (cid:96) =

Original: L A ( (cid:96) )RNN and B RNN = (cid:80) L(cid:96)
Revised: L A ( (cid:96) )RNN and B RNN = (cid:80) L(cid:96)

Original: =1 (cid:16)(cid:81)
Revised: =1 (cid:16)(cid:81)

Original: L − 1 (cid:96) (cid:48) =
Revised: L − 1 (cid:96) (cid:48) =

Original: (cid:96) A ( (cid:96) (cid:48) )
Revised: (cid:96) A ( (cid:96) (cid:48) )

Original: RNN (cid:17) B ( (cid:96) )
Revised: RNN (cid:17) B ( (cid:96) )

Original: RNN are the afﬁne parameters of the L -layer RNN.
Revised: RNN are the afﬁne parameters of the L -layer RNN.

Original: W and b are parameters of the fully connected output layer where W =
Revised: W and b are parameters of the fully connected output layer, where W =

Original: [ W , W , . . .
Revised: [ W , W , . . .

Original: , W ] when the RNN outputs at every time step and W =
Revised: , W ] when the RNN outputs at every time step and W =

Original: [ W , 0 , . .
Revised: [ W , 0 , . .

Original: , 0] when the RNNonly outputs  at the last time step.
Revised: , 0] when the RNN outputs only at the last time step.

Original: Thm. 2 shows that, using MASOs, we have a simple, elegant analytical formula of a multi-layer RNN, and  that the output  is computedvia locally  very simple functions.
Revised: Theorem 2 shows that, using MASOs, we have a simple, elegant, and closed-form formula showing that the output of an RNN is computed locally via very simple functions.

Original: This theorem is proved by recursively applying the proof for Thm. 1.
Revised: This result is proved by recursively applying the proof for Theorem 1.

Original: Theabove afﬁne  formula  opens  doors for many analyses of RNNs by examining the afﬁne parameters becausenow the afﬁne parameters are simply a matrix and a vector, and we canuse many established matrix theories to shed light on RNNs through its afﬁne formula.
Revised: The afﬁne mapping formula (9) opens many doors for RNN analyses, because we can shed light on RNNs by applying established matrix results.

Original: In the remaining sections, we provide three analyses that leverage RNNs’ afﬁne formula.
Revised: In the next sections, we provide three analyses that follow this programme.

Original: First, we show that RNNs partition the input space and that they develop the partitions through time.
Revised: First, we show that RNNs partition the input space and that they develop the partitions through time.

Original: Second, we analyze the forms of theslope afﬁne  parameter and link RNNs to matched ﬁlterbanks.
Revised: Second, we analyze the forms of the afﬁne slope parameter and link RNNs to matched ﬁlterbanks.

Original: Third, we study the impact of  initial hidden state and justify the use of noise in initial hidden state.
Revised: Third, we study the impact of the initial hidden state to rigorously justify the use of randomness in initial hidden state.

Original: From now on, for simplicity, we will assume  zero initial hidden state unless otherwise stated.
Revised: From this point, for simplicity, we will assume a zero initial hidden state unless otherwise stated.

Original:  I NTERNAL I NPUT S PACE P ARTITIONING IN RNN S
Revised:  I NTERNAL I NPUT S PACE P ARTITIONING IN RNN S

Original: In this section, we show that RNNs implicitly partition the input sequences and do so through time, which provides a new perspective of the dynamics of RNNs.
Revised: The MASO viewpoint enables us to see how an RNN implicitly partitions its input sequence through time, which provides a new perspective of its dynamics.

Original: To see this, ﬁrst recall that, for an 
Revised: To see this, ﬁrst recall that, for an RNN

Original: RNN cell, the piecewise afﬁne and convex  nonlinearity partitions each dimension of the cell input z ( (cid:96),t ) into R regions (for ReLU, R = 2 ).
Revised: [Figure] cell, the piecewise afﬁne and convex activation nonlinearity partitions each dimension of the cell input z ( (cid:96),t ) into R regions (for ReLU, R = 2 ).

Original: Knowing the state of the nonlinearity (which region r is activated) is thus equivalent to knowing the partition of a cell input.
Revised: Knowing the state of the nonlinearity (which region r is activated) is thus equivalent to knowing the partition of the cell input.

Original: Then, for a multi-layer RNN composed of many RNN cells (recall Figure 1), the  partition  of an input sequence is retrieved by ti m e s t e p s ti m e s t e p s ti m e s t e p s
Revised: For a multi-layer RNN composed of many RNN cells (recall Figure 1), the input sequence partition can be retrieved by accessing the collection of the states of all of the nonlinearities; each input sequence can be represented by a partition “code” that determines the partition to which it belongs.

Original: A ll ti m

Original: e s t e p s accessing the collection of states of all nonlinearities.

Original: Since an RNN processes an input sequence one step at a time, the input space partition is gradually built up and reﬁned through time.
Revised: Since an RNN processes an input sequence one step at a time, the input space partition is gradually built up and reﬁned through time.

Original: As a consequence,  the forward pass of an RNN becomes simply developing partitions ofinput sequences, which is easier to interpret than the seemingly convoluted RNN dynamics when not viewed from a MASO perspective.
Revised: As a consequence, when seen through the MASO lens, the forward pass of an RNN is simply developing and reﬁning the partition code of the input sequence.

Original: Visualizing the evolution of partitioning can be potentially beneﬁcial for diagnosing RNNs and understanding their behaviors.
Revised: Visualizing the evolution of the partition codes can be potentially beneﬁcial for diagnosing RNNs and understanding their dynamics.

Original: As an example, we demonstrate the evolution of input space partitioning of a one-layer ReLU RNN trained on  MNIST dataset,  each image ﬂattened into a 1-dimensional sequence.
Revised: As an example, we demonstrate the evolution of the partition codes of a one-layer ReLU RNN trained on the MNIST dataset, with each image ﬂattened into a 1-dimensional sequence so that input at each time step is a single pixel.

Original: Details of  model and experiments are in Appendix C.Since
Revised: Details of the model and experiments are in Appendix C.

Original:  ReLU induces only 2 partition regions, we thus retrieve the partitioning of  input imagesthat the RNN computes by simply  binarizing and concatenating all  hidden states.
Revised: Since the ReLU activation partitions its input space into only 2 regions , we can retrieve the RNN partition codes of the input images simply by binarizing and concatenating all of the hidden states.

Original: Figure 2 visualizes how the partitioning ofimages in the MNIST test set develops through time using t-SNE .
Revised: Figure 2 visualizes how the partition codes of MNIST images evolve through time using t-SNE, a distance-preserving dimensionality reduction technique (van der Maaten & Hinton, 2008).

Original: The ﬁgure clearly shows the evolution of the RNN input space partitioning, from hardly any separation  to forming more clustered partitions through time.
Revised: The ﬁgure clearly shows the evolution of the partition codes from hardly any separation between classes of digits to forming more and better separated clusters through time.

Original: We can also know that the model is well behaved since the ﬁnal partition shows that the images are well clustered based on their labels.
Revised: We can also be assured that the model is well-behaved, since the ﬁnal partition shows that the images are well clustered based on their labels.

Original: Additional visualizations are available in Section D.
Revised: Additional visualizations are available in Section D.

Original:  RNN  ASA M ATCHED F ILTERBANK
Revised:  RNN S AS M ATCHED F ILTERBANKS

Revised: The MASO viewpoint enables us to connect RNNs to classical signal processing tools like the matched ﬁlter.

Original: In this section, weshow that RNNs can be regarded as  matched ﬁlterbanks and that the classiﬁcation decision is informed via  inner product between the “template” and the input sequence.
Revised: Indeed, we can directly interpret an RNN as a matched ﬁlterbank , where the classiﬁcation decision is informed via the simple inner product between a “template” and the input sequence.

Original: To see this, we apply similar argument  to that in Section 3.
Revised: To see this, we follow an argument similar to that in Section 4.

Original: First, note that the slope parameter A ( (cid:96),t ) for each RNN cell is a “locally optimal template” because it maximizes each  output dimension over  R regions that the nonlinearity induces (recall Eqs. 3 and 6).
Revised: First, note that the slope parameter A ( (cid:96),t ) for each RNN cell is a “locally optimal template” because it maximizes each of its output dimensions over the R regions that the nonlinearity induces (recall (3) and (7)).

Original: Then, for a multi-layer RNN composed of many RNN cells, the overall “template” A RNN corresponds to the composition of  optimal templates obtained at each RNN cell and can be computed simply via d z /d x (recall Eq. 8).
Revised: For a multi-layer RNN composed of many RNN cells, the overall “template” A RNN corresponds to the composition of the optimal templates from each RNN cell, which can be computed simply via d z / d x (recall (9)).

Original: Thus, we can view the RNN as a matched ﬁlterbank that computes the output by ﬁnding the maximum inner product between the  overall template A RNN and the input (Trees, 1992; 2013).
Revised: Thus, we can view an RNN as a matched ﬁlterbank whose output is the maximum inner product between the input and the rows of the overall template A RNN (van Trees, 1992; 2013).

Original: The overall template is also known in the machine learning community as  salience map; see Li et al.
Revised: The overall template is also known in the machine learning community as a salience map ; see Li et al.

Original: for an example of using saliency maps to visualize RNNs.
Revised: for an example of using saliency maps to visualize RNNs.

Original: Our new insight here is that a good template produces a larger inner product with the input regardless of the visual quality of the template , thus complementing prior work.
Revised: Our new insight here is that a good template produces a larger inner product with the input regardless of the visual quality of the template , thus complementing prior work.

Original: A template matching view of RNNs thus provides a principled and motivated way to visualize and diagnose RNNs by examining the inner products between  inputs and  templates.
Revised: The template matching view of RNNs thus provides a principled methodology to visualize and diagnose RNNs by examining the inner products between the inputs and the templates.

Original: As an example, we train a one-layer ReLU RNN on the polarized Stanford Sentiment Treebank  (SST-2) (Socher et al., 2013), which is a binary classiﬁcation problem, and show in Figure 3 the templates corresponding to the correct and incorrect classes of an input where the correct class is  negative sentiment.
Revised: To illustrate the matched ﬁlter interpretation, we train a one-layer ReLU RNN on the polarized Stanford Sentiment Treebank dataset (SST-2) (Socher et al., 2013), which poses a binary classiﬁcation problem, and display in Figure 3 the templates corresponding to the correct and incorrect classes of an input where the correct class is a negative sentiment.

Original: We see that the input has a much larger inner product with the template corresponding to the correct class (left plot) than that corresponding to the incorrect class (right plot), which informs us that the model correctly classiﬁes this input.
Revised: We see that the input has a much larger inner product with the template corresponding to the correct class (left plot) than that corresponding to the incorrect class (right plot), which informs us that the model correctly classiﬁes this input.

Original: Additional experimental results are  in Appendix E.
Revised: Additional experimental results are given in Appendix E.

Original:  I MPROVING RNN S V IA N OISY I NITIAL H IDDEN S TATE
Revised:  I MPROVING RNN S V IA R ANDOM I NITIAL H IDDEN S TATE

Original: In this section, we provide  theoretical insights on the use of noise in an RNN’s initial hidden state.
Revised: In this section, we provide a theoretical motivation for the use of a random initial hidden state in RNNs.

Revised: The initial hidden state needs to be set to some prior value to start the recursion (recall

Revised: Section 2).

Original: Little is understood about setting the value for the  initial hidden state except for Zimmermann et al.
Revised: Little is understood regarding the best choice of initial hidden state other than Zimmermann et al.

Original: (2012)’s dynamical system argument.
Revised: (2012)’s dynamical system argument.

Original: Therefore the initial hidden state is typically  set to zero without much justiﬁcation.
Revised: Consequently, it is typically simply set to zero.

Original: Leveraging the MASO formula of RNNs, we prove that noisy initial hidden state is equivalent to explicit regularizing the afﬁne parameter associated with the initial hidden state.
Revised: Leveraging the MASO view of RNNs, we now demonstrate that one can improve signiﬁcantly over a zero initial hidden state by using a random initial hidden state.

Revised: This choice regularizes the afﬁne slope parameter associated with the initial hidden state and molliﬁes the so-called exploding gradient problem (Pascanu et al., 2013).

Original: 5. N OISY I NITIAL H IDDEN S TATE AS AN E XPLICIT R EGULARIZER
Revised: Random Initial Hidden State as an Explicit Regularization.

Original: Let (cid:101) L be the loss with noise injected to the initial hidden state, N be the total number of data points,
Revised: We ﬁrst state our theoretical result that using random initial hidden state corresponds to an explicit regularization and then discuss its impact on exploding gradients. Without loss of generality, we focus on one-layer ReLU RNNs.

Original: C be thetotal number of classes and deﬁne A h :
Revised: Let N be the number of data points and C the number of classes.

Original: = A 1: T = (cid:81) 1 s = T A ( (cid:96),s ) σ
Revised: Deﬁne A h := A 1: T = (cid:81)s = [Equation]

Original: W ( (cid:96) )

Original: r (Recall Eq. 7).
Revised: r (recall (8)).

Original: We ﬁrst state the theoretical result and then link it to the exploding gradient problem.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let  (cid:15) ∼ N (0 , σ 2 (cid:15) I ) be a Gaussian noise where σ2 (cid:15) is small.
Revised: Let L be an RNN loss function, and let (cid:101) L represent the modiﬁed loss function when the RNN initial hidden state is set to a Gaussian random vector (cid:15) ∼ N (0 , σ 2 (cid:15) I ) with small standard deviation σ (cid:15) .

Original: We have that 
Revised: Then we have that E (cid:15) (cid:104)(cid:101) L (cid:105) =

Original: [Equation]
Revised: L + R .

Original: For cross entropy loss  with softmax output,R =
Revised: For the cross-entropy loss L with softmax output,

Original: [Equation]
Revised: R = σ 2 (cid:15) 2 N (cid:80) Nn

Original: [Table]
Revised: = (cid:13) (cid:13) (cid:13)(cid:13) diag (cid:18)(cid:104) d (cid:98)

Original:  ni
Revised: y ni

Original: [Table] nj (cid:105) i =
Revised: ∂z nj (cid:105) i =

Original: j (cid:19)
Revised: j (cid:19)

Original: A h (cid:13)(cid:13)(cid:13)(cid:13) 2  where (cid:98)
Revised: A h (cid:13) (cid:13) (cid:13)(cid:13) 2 , where (cid:98)

Original: y ni is the i th dimension of the softmax output of the n th data point and i, j ∈ { 1 , · · · , C } are the class indices .
Revised: y ni is the i th dimension of the softmax output of the n th data point and i, j ∈ { 1 , . . . , C } are the class indices.

Original: For mean squared error loss, R = σ 2 (cid:15) 2 N (cid:80) Nn =1 (cid:107)
Revised: For the mean-squared error loss L , R = σ 2 (cid:15) 2 N (cid:80) Nn =1 (cid:107)

Original: A h (cid:107) 2 .
Revised: A h (cid:107) 2 .

Original: We prove this result for the cross entropy case in Appendix G.2.
Revised: We prove this result for the cross-entropy loss in Appendix G.2.

Original: We can see that the noise standard deviation σ (cid:15) controls the importance of the regularization term and recovers thestandard case for σ (cid:15) 
Revised: The standard deviation σ (cid:15) controls the importance of the regularization term and recovers the case of standard zero initial hidden state when σ (cid:15) = 0 .

Original:  = 0  . 
Revised: Connection to the Exploding Gradient Problem. Backpropagation through time (BPTT) is the default RNN training algorithm. Updating the recurrent weight W r with its gradient using BPTT involves calculating the gradient of the RNN output with respect to the hidden state at each time step t = 0 , . . , T

Original: Additionally, the regularizer does not depend on the accuracy of the classiﬁer, which is a regularization scheme on the model itself (Wager et al., 2013).
Revised: [Equation] (cid:89) s [Equation] (cid:33) .

Original: Interpretation and Connections to Prior Work.
Revised: When (cid:107) A ( s ) σ W r (cid:107) 2 > 1 , the product term (cid:81) t +1 s = T A ( s ) σ W r in (10) blows up, which leads to unstable training.

Original: Closely inspecting the form of A h , we see that this term is  the gradient of the RNN output with respect to the initial hiddenstate. In fact, the gradient of the RNN output with respect to hidden state at any time step takes the following form:
Revised: This is known as the exploding gradient problem (Pascanu et al., 2013). Our key realization is that the gradient of the RNN output with respect to the initial hidden state h (0) features the term A h from Theorem

Original: [Equation]
Revised: [Equation]

Original: which is the basis of studying gradient exploding problems in RNNs, since it is clear that constraining this term will help limit the gradient in the backward pass. By introducing noise to the initial hidden state, we regularize the term with the most number of matrix multiplications in the gradient calculation of the recurrent weight.
Revised: Of all the terms in (10), this one involves the most matrix products and hence is the most erratic. Fortunately, Theorem 3 instructs us that introducing randomness into the initial hidden state effects a regularization on A h and hence tamps down the gradient before it can explode.

Original: While whether regularizing this term amounts toregularizing everyother term in the recurrent weight gradient calculation remains to be investigated, we provide empirical evidence that the regularization introduced by noisy initial hidden state already reduces the magnitude of the recurrent weight gradient and improves model performance. A detailed review of exploding gradient problem is provided in Appendix H.
Revised: An interesting direction for future work is extending this analysis to every term in (10).

Original: 5. E XPERIMENTS
Revised: Experiments.

Original: We provide extensive empirical evidence to advocate for the use of noisy initial hidden state and demonstrate that a properly chosennoise standard deviation σ (cid:15) improves performance for all the datasets that we experiment on.
Revised: We now report on the results of a number of experiments that indicate the signiﬁcant performance gains that can be obtained using a random initial hidden state of properly chosen standard deviation σ (cid:15) .

Original: Unless otherwise mentioned,  we useidentity-initialized ReLU RNNs of 128-dimensional hidden state (denoted as RNN) in all our experiments.
Revised: Unless otherwise mentioned, in all experiments we use ReLU RNNs with 128-dimensional hidden states and with the recurrent weight matrix W ( (cid:96) )

Revised: r initialized as an identity matrix (Le et al., 2015; Talathi & Vartak, 2016).

Original: We summarizein this section thegist of our experimental results; Details on models, datasets, experiment setup and additional results are available in Appendix C and F.
Revised: We summarize the experimental results; experimental details and additional results are available in Appendices C and F.

Original: Visualizing the Regularization Effect of the Noisy Initial Hidden State.
Revised: Visualizing the Regularizing Effect of a Random Initial Hidden State.

Original: We ﬁrst visualize the regularization effect when injecting noise into the initial hidden state on a simulation task of adding 2 sequences of length 100.
Revised: We ﬁrst consider a simulated task of adding 2 sequences of length 100.

Original: This is a ternary classiﬁcaion problem with input X ∈ R 2 × T and target y ∈ { 0 , 1 , 2 } , y =
Revised: This is a ternary classiﬁcation problem with input X ∈ R 2 × T and target y ∈ { 0 , 1 , 2 } , y =

Original: (cid:80)
Revised: (cid:80)

Original: i 

Revised: i

Original: X 2 i =1 · X 1 i .
Revised: X 2 i = 1 X 1 i .

Original: The ﬁrst row of X contains randomly chosen 0 ’s and 1 ’s, and the second row of X contains 1 ’s at 2 randomly chosen indices and 0 ’s everywhere else.
Revised: The ﬁrst row of X contains randomly chosen 0 ’s and 1 ’s; the second row of X contains 1 ’s at 2 randomly chosen indices and 0 ’s everywhere else.

Original: Note that prior work treats this task as a regression task (Arjovsky et al., 2016). Results for treating this task as a regression problem is provided in Appendix F.1, which are similar to those presented here.
Revised: Prior work treats this task as a regression task (Arjovsky et al., 2016); our regression results are provided in Appendix F.1.

Original: In Fig. 4, we visualize the norm of A h , the norm of  recurrent weight gradient (cid:107)
Revised: In Figure 4, we visualize the norm of A h , the norm of the recurrent weight gradient (cid:107)

Original: d L d W r (cid:107)  and  validation loss against training epochs for various noise standard deviations.
Revised: d L d W r (cid:107) , and the validation loss against training epochs for various random initial state standard deviations.

Original: The top two plots clearly demonstrate theregularization effect of noisy initial hidden state in regularize both A h and  norm of the recurrent weight gradient, since larger σ (cid:15) reduces the magnitudes of both A h and (cid:107)
Revised: The top two plots clearly demonstrate the effect of the random initial hidden state in regularizing both A h and the norm of the recurrent weight gradient, since larger σ (cid:15) reduces the magnitudes of both A h and (cid:107)

Original: d L d W r (cid:107) .
Revised: d L d W r (cid:107) .

Original: Notably,this regularization effect happens even when the regularizer does not directly regularize these two terms, which validates our intuition.
Revised: Notably, the reduced magnitude of the gradient term (cid:107) d L d W r (cid:107) empirically demonstrates the molliﬁcation of the exploding gradient problem.

Original: The  plotin the last row shows that settinga big σ (cid:15)  can negatively impact learning.
Revised: The bottom plot shows that setting σ (cid:15) too large can negatively impact learning.

Original: This can be explained by having too much regularization effect with a σ (cid:15) that is too large.
Revised: This can be explained as having too much regularization effect.

Original: This brings the question of choosing  σ (cid:15) in practice, which we investigate below.
Revised: This suggests the question of choosing the best value of σ (cid:15) in practice, which we now investigate.

Original: Choosing the Noise Parameter.
Revised: Choosing the Standard Deviation of the Random Initial Hidden State.

Original: We examine the  performance of different noise standard deviations  using RMSprop and SGD with varying learning rates to provide practical guidance on choosing the noise parameter σ (cid:15) .
Revised: We examine the effect on performance of different random initial state standard deviations σ (cid:15) in RNNs using RMSprop and SGD with varying learning rates.

Original: We perform experiments on  MNIST dataset, each image ﬂattened to a length 784 sequence (recall Section 3).
Revised: We perform experiments on the MNIST dataset with each image ﬂattened to a length 784 sequence (recall Section 4).

Original: Experimental results are included in Appendix F.2. Here, we report two interesting ﬁndings .
Revised: The full experimental results are included in Appendix F.2; here, we report two interesting ﬁndings.

Original: First, for both optimizers, using noisy initial hidden state permits the use of higher learning rates that typically lead to  exploding gradient when trainingRNNs without noisy initial hidden state.
Revised: First, for both optimizers, using a random initial hidden state permits the use of higher learning rates that would lead to an exploding gradient when training without a random initial hidden state.

Original: Second, RMSprop is more tolerable to the choice of σ (cid:15) than SGD and achieves favorable accuracies even when σ (cid:15) is very large (e.g., σ (cid:15) = 5 ).
Revised: Second, RMSprop is less sensitive to the choice of σ (cid:15) than SGD and achieves favorable accuracy even when σ (cid:15) is very large (e.g., σ (cid:15) = 5 ).

Original: This might be due to the gradient smoothing that RMSprop performs during optimization.
Revised: This might be due to the gradient smoothing that RMSprop performs during optimization.

Original: We therefore recommend the use of RMSprop with noisy initial hidden state to improve model performance.
Revised: We therefore recommend the use of RMSprop with a random initial hidden state to improve model performance.

Revised: [Table]

Original: We then use RMSprop to train ReLU RNNs of one or two layersand with or without noisy initial hidden state on  MNIST, permuted MNIST 5 and SST-2 datasets.
Revised: We used RMSprop to train ReLU RNNs of one and two layers with and without random initial hidden state on the MNIST, permuted MNIST 4 and SST-2 datasets.

Original: Table 1 shows the classiﬁcation accuracies of these models as well as a few state-of-the-artresults. These results further suggest case
Revised: Table 1 shows the classiﬁcation accuracies of these models as well as a few state-of-the-art results using complicated models.

Original: [Table] the use of noisy initial hidden state in improving performance of ReLU RNN, elevating such simple model to a strong competitor of a number of more complex, state-of-the-art models.
Revised: It is surprising that a random initial hidden state elevates the performance of a simple ReLU RNN to near state-of-the-art performance.

Original: Noisy Initial Hidden State in Complex  Models.
Revised: Random Initial Hidden State in Complex RNN Models.

Original: We perform exploratory experiments on the use of  initial hidden state in more complex  models.
Revised: Inspired by the results of the previous experiment, we integrated a random initial hidden state into some more complex RNN models.

Original: We ﬁrst evaluate a one-layer gated recurrent unit (GRU) on  MNIST and permuted MNIST datasets, with  and without noisy initial hidden state.
Revised: We ﬁrst evaluate a one-layer gated recurrent unit (GRU) on the MNIST and permuted MNIST datasets, with a random and zero initial hidden state.

Original: Although  not  as impressive as those for ReLU RNNs,  results for GRUsshow that we still obtain modestly improved accuracy with noisy initial hidden state, from 0 .
Revised: Although the performance gains are not quite as impressive as those for ReLU RNNs, our results for GRUs still show worthwhile accuract improvements, from 0 .

Original: 986 to 0 .
Revised: 986 to 0 .

Original: 987 for MNIST and  0 .
Revised: 987 for MNIST and from 0 .

Original: 888 to 0 .
Revised: 888 to 0 .

Original: 904 for permuted MNIST.
Revised: 904 for permuted MNIST.

Original: We then evaluate a more complex, convolutional-recurrent model composed of 4 convolution layers followed by 2 GRU layers (Cakir et al., 2017) on the Bird Audio Detection Challenge datasetwhich involves a binary classiﬁcation problem  to detect whether or not an audio recording contain bird songs; see Appendix C for  details.
Revised: We continue our experiments with a more complex, convolutional-recurrent model composed ofconvolution layers followed by 2 GRU layers (Cakir et al., 2017) and the Bird Audio Detection Challenge dataset 5 . This binary classiﬁcation problem aims to detect whether or not an audio recording contains bird songs; see Appendix C for the details.

Original: We use the area under the  curve (AUC) as the evaluation metric since the dataset is highly imbalanced.
Revised: We use the area under the ROC curve (AUC) as the evaluation metric, since the dataset is highly imbalanced.

Original: Results show a 3 points of percentage improvement in AUC when using noisy initial hidden state, from 90.5% to 93.4%.
Revised: Simply switching from a zero to a random initial hidden state provides a signiﬁcant boost in the AUC: from 90.5% to 93.4%.

Original: These encouraging preliminary results imply that, while more theoretical and empirical investigations are needed, noisy initial hidden state potentially improves more complex RNN models where the nonlinearities are no longer piecewise afﬁne and convex.
Revised: These encouraging preliminary results suggest that, while more theoretical and empirical investigations are needed, a random initial hidden state can also boost the performance of complicated RNN models that are not piecewise afﬁne and convex.

Original: Further experimentation and theoretical development to understand and characterize using noise in initial hidden state in these elaborate models are left for future work.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: We provide a novel perspective of RNNs using max-afﬁne spline operators (MASOs).We prove that
Revised: We have developed and explored a novel perspective of RNNs in terms of max-afﬁne spline operators (MASOs).

Original: RNNs with piecewise afﬁne and convex nonlinearities becomes piecewise afﬁne spline operators that have a simple, elegant analytical form.
Revised: RNNs with piecewise afﬁne and convex nonlinearities are piecewise afﬁne spline operators with a simple, elegant analytical form.

Original: Leveraging this formulation, we provide new insights and visualizations of RNNs from input space partitioning  and matched ﬁltering perspectives. Furthermore, we show thatinjecting noise in the  initial hidden state of an RNN corresponds to an explicit regularization and improves generalization. Extensive empirical studies demonstrate the utility of the new insights and the effectiveness of noisy initial hidden state.
Revised: The connections to input space partitioning (vector quantization) and matched ﬁltering followed immediately. The spline viewpoint also suggested that the typical zero initial hidden state be replaced with a random one that molliﬁes the exploding gradient problem and improves generalization performance.

Original: The MASO framework offers abundant promising research directions for understanding RNNs, two of which we brieﬂy discuss here. First,generalizing MASOs to probabilistic cases (Balestriero & Baraniuk, 2018b), we can extend our study to  RNNs with other nonlinearities such as GRUs and
Revised: There remain abundant promising research directions. First, we can extend the MASO RNN framework following (Balestriero & Baraniuk, 2018b) to cover more general networks like gated RNNs (e.g, GRUs, LSTMs) that employ the sigmoid nonlinearity, which is neither piecewise afﬁne nor convex.

Original: LSTMs. Second, we can apply  random matrix theory from a recent result (Martin & Mahoney, 2018) tostudy the afﬁne parameter A RNN , e.g., the change of  distribution of its singular values during training, to understand the training dynamics of RNNs.
Revised: Second, we can apply recent random matrix theory results (Martin & Mahoney, 2018) to the afﬁne parameter A RNN (e.g., the change of the distribution of its singular values during training) to understand RNN training dynamics. A CKNOWLEDGEMENTS We thank the anonymous reviewers for their constructive feedback.

Revised: This work was partially supported by NSF grants IIS-17-30574 and IIS-18-38177, AFOSR grant FA9550-18-1-0478, ARO grant W911NF-15-1-0316, ONR grants N00014-17-1-2551 and N00014-18-12571, DARPA grant

Revised: G001534- and DOD Vannevar Bush Faculty Fellowship (NSSEFF) grant N00014-18-1-2047.

Original: A N OTATION
Revised: A N OTATION

Original: [Table]
Revised: [Table]

Original: A dataset

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: variable) with the n serie example
Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  For classiﬁcation y n ∈ { 1 , . ..
Revised: example For classiﬁcation y n ∈ { 1 , . .

Original: , C } , C >1 ; For regression y n ∈ R C , C ≥
Revised: , C } , C >

Original: [Table]
Revised: [Table]

Revised: For regression y n ∈ R C

Revised: C

Original: [Table]
Revised: [Table]

Original: Output of an RNN cell at layer (cid:96) and time step t;
Revised: Output of an RNN cell at layer (cid:96) and time step t

Revised: [Table]

Original: Alternatively, input to an RNN cell at layer (cid:96) + 1 andtime step t −h
Revised: Alternatively, input to an RNN cell at layer (cid:96) + 1 and step t −

Original: [Table]
Revised: [Table]

Original: Concatenation of hidden state h ( 
Revised: Concatenation of hidden state h ( (cid:96)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Pointwise nonlinearity in an RNN (assumed to be piecewise afﬁne and convex in this paper)
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: noise injected
Revised: RNN

Original: [Table]
Revised: [Table]

Original: the initial hidden
Revised: be piecewise afﬁne and convex in this

Original: [Table]
Revised: [Table]

Original:  formula 
Revised: MASO formula of the

Original: [Table]
Revised: [Table]

Original: B D ATASETS AND P REPROCESSING S TEPS
Revised: B D ATASETS AND P REPROCESSING S TEPS

Original: Below we describe the datasets and explain the preprocessing steps for each dataset.
Revised: Below we describe the datasets and explain the preprocessing steps for each dataset.

Original: MNIST .
Revised: MNIST .

Original: The dataset 7 consists of 60k images in the training set and 10k images in the test set.
Revised: The dataset 6 consists of 60k images in the training set and 10k images in the test set.

Original: We randomly select 10k images from the training set as validation set.
Revised: We randomly select 10k images from the training set as validation set.

Original: We ﬂatten each image to a 1- dimensional vector of size784. Each image is also centered and normalized with mean  of 0.
Revised: We ﬂatten each image to a 1- dimensional vector of size is also centered and normalized with mean and standard deviation of 0.

Original: and standard deviation of 0.3081 (PyTorch default values).
Revised: (PyTorch default values).

Original: For permuted MNIST .
Revised: permuted MNIST .

Original: We apply a ﬁxed permutation to all images in the MNIST dataset to obtain the permuted MNIST dataset.
Revised: We apply a ﬁxed permutation to all images in the MNIST dataset to obtain the permuted MNIST dataset.

Original: SST-2 .
Revised: SST-2 .

Original: The dataset 8 consists of 6920, 872, 1821 sentences in the training, validation and test set, respectively.
Revised: The dataset 7 consists of 6920, 872, 1821 sentences in the training, validation and test set, respectively.

Original: Total number of vocabulary is 17539, and average sentence length is 19.67.
Revised: Total number of vocabulary is 17539, and average sentence length is 19.67.

Original: Each sentence is minimally processed into sequences of words and use a ﬁxed-dimensional and trainable vector to represent each word.
Revised: Each sentence is minimally processed into sequences of words and use a ﬁxed-dimensional and trainable vector to represent each word.

Original: We initialize these vectors either randomly or using GloVe (Pennington et al., 2014).
Revised: We initialize these vectors either randomly or using GloVe (Pennington et al., 2014).

Original: Due to the small size of the dataset, the phrases in each sentence that have semantic labels are also used as part of the training set in addition to the whole sentence during training.
Revised: Due to the small size of the dataset, the phrases in each sentence that have semantic labels are also used as part of the training set in addition to the whole sentence during training.

Original: Dropout of 0.5 is applied to all experiments.
Revised: Dropout of 0.5 is applied to all experiments.

Original: Phrases are not used during validation and testing, i.e., we always use entire sentences during validation and testing.
Revised: Phrases are not used during validation and testing, i.e., we always use entire sentences during validation and testing.

Original: Bird Audio Dataset .
Revised: Bird Audio Dataset .

Original: The dataset 9 consists of 7 , 000 ﬁeld recording signals of 10 sec. sampled at 44 kHz from the Freesound Stowell & Plumbley (2014) audio archive representing slightly less than 20 hours of audio signals.
Revised: The dataset 8 consists of 7 , 000 ﬁeld recording signals of 10 seconds sampled at 44 kHz from the Freesound Stowell & Plumbley (2014) audio archive representing slightly less than 20 hours of audio signals.

Original: The audio waveforms are extracted from diverse scenes such as city, nature, train, voice, water, etc., some of which include bird songs.
Revised: The audio waveforms are extracted from diverse scenes such as city, nature, train, voice, water, etc., some of which include bird sounds.

Original: The labels regarding the bird detection task can be found in the ﬁle freefield1010 .
Revised: The labels regarding the bird detection task can be found in the ﬁle freefield1010 .

Original: Performance is measured by AreaUnder
Revised: Performance is measured by Area

Original:  Curve (AUC) due to the unbalanced distribution of the classes ( 3 for 1 ).
Revised: Under Curve (AUC) due to the unbalanced distribution of the classes.

Original: We preprocess every audio clip by ﬁrst using short-time Fourier transform (STFT) with 40ms and 50% overlapping Hamming window to obtain audio spectrum and then by extracting 40 log mel-band energy features.
Revised: We preprocess every audio clip by ﬁrst using short-time Fourier transform (STFT) with 40ms and 50% overlapping Hamming window to obtain audio spectrum and then by extracting 40 log mel-band energy features.

Original: After preprocessing, each input is of dimension D = 96 and T = 999 .
Revised: After preprocessing, each input is of dimension D = 96 and T = 999 .

Original: C E XPERIMENTAL S ETUP
Revised: C E XPERIMENTAL S ETUP

Original: Experiment setup for various datasets is summarized in Table 2.
Revised: Experiment setup for various datasets is summarized in Table 2.

Original: Some of the experiments do not appear in the main text but in the appendix; we include setup for those experiments as well.
Revised: Some of the experiments do not appear in the main text but in the appendix; we include setup for those experiments as well.

Original: A setting common to all experiments is that we use learning rate scheduler so that when validation loss plateaus for 5 consecutive epochs, we reduce the current learning rate by a factor of 0.7.
Revised: A setting common to all experiments is that we use learning rate scheduler so that when validation loss plateaus for 5 consecutive epochs, we reduce the current learning rate by a factor of 0.7.

Original: Setup of the experiments on inﬂuence of variousnoise standard deviation under different settings .
Revised: Setup of the experiments on inﬂuence of various standard deviations in random initial hidden state under different settings .

Original: We use σ (cid:15) chosen in { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 5 } and learning rates in { 1 × 10 − 5 , 1 × 10 − 4 , 1 . 5 × 10 − 4 , 2 × 10 − 4 } for RMSprop and { 1 × 10 − 7 , 1 × 10 − 6 , 1 . 25 × 10 − 6 , 1 . 5 × 10 − 6 } plain SGD.
Revised: We use σ (cid:15) chosen in { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 5 } and learning rates in { 1 × 10 − 5 , 1 × 10 − 4 , 1 . 5 × 10 − 4 , 2 × 10 − 4 } for RMSprop and { 1 × 10 − 7 , 1 × 10 − 6 , 1 . 25 × 10 − 6 , 1 . 5 × 10 − 6 } plain SGD.

Original: Setup of input space partitioning  .
Revised: Setup of input space partitioning experiments .

Original: For the results in the main text, we use tSNE visualization (Maaten & Hinton, 2008) with 2 dimensions and the default settings from the python sklearn package.
Revised: For the results in the main text, we use t-SNE visualization (van der Maaten & Hinton, 2008) with 2 dimensions and the default settings from the python sklearn package.

Original: Visualization is performed on the whole 10k test set images.
Revised: Visualization is performed on the whole 10k test set images.

Original: For ﬁnding the nearest neighbors of examples in the SST-2 dataset, since the examples are of varying lengths, we constrain the distance comparison to within +/-10 words of the target sentence.
Revised: For ﬁnding the nearest neighbors of examples in the SST-2 dataset, since the examples are of varying lengths, we constrain the distance comparison to within +/-10 words of the target sentence.

Original: When the sentence lengths are not the same, we simply pad the shorter ones to the longest one, then process it with RNN and ﬁnally calculate the distance as the L-2 distance of the VQ (i.e., concatenation of all hidden states) that RNN computes.
Revised: When the sentence lengths are not the same, we simply pad the shorter ones to the longest one, then process it with RNN and ﬁnally calculate the distance as the (cid:96) 2 distance of the partition codes (i.e., concatenation of all hidden states) that RNN computes.

Original: We justify the comparison between examples of different lengths using padding by noting that batching examples, a common practice in modern NLP tasks, pad the examples to the longest example  in the batch.
Revised: We justify the comparison between examples of different lengths using padding by noting that batching examples and padding the examples to the longest example within a batch has been a common practice in modern natural language processing tasks.

Original: Setup of exploratory experiments .
Revised: Setup of exploratory experiments .

Original: We experimented with one-layer GRU with 128 hidden unites for MNIST and permuted MNIST datasets.
Revised: We experimented with one-layer GRU with 128 hidden unites for MNIST and permuted MNIST datasets.

Original: We use RMSprop optimizer with an initial learning rate of 0.001.
Revised: We use RMSprop optimizer with an initial learning rate of 0.001.

Original: We experimented with variousnoise standard deviations in the initial hidden state including { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 , 5 } .
Revised: We experimented with various standard deviations in random initial hidden state including { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 , 5 } .

Original: The optimalnoise standard deviations that produce the results in the main text are σ (cid:15) = { 0 . 01 , 0 . 05 , 0 . 01 } , for MNIST, permuted MNIST and bird detection datasets, respectively.
Revised: The optimal standard deviations that produce the results in the main text are σ (cid:15) = { 0 . 01 , 0 . 05 , 0 . 01 } , for MNIST, permuted MNIST and bird detection datasets, respectively.

Original: D A DDITIONAL I NPUT S PACE P ARTITION V ISUALIZATIONS
Revised: D A DDITIONAL I NPUT S PACE P ARTITION V ISUALIZATIONS

Original: We provide ample additional visualizations to demonstrate the internal input space partitioning that an RNN performs.
Revised: We provide ample additional visualizations to demonstrate the partition codes that an RNN computes on its input sequences.

Original: Here, the results are focused more on the properties of the ﬁnal input space
Revised: Here, the results are focused more on the properties of the ﬁnal partition

Original: [Table]
Revised: [Table]

Original: partitioning, i.e., partition of the entire input sequence, instead of part of the input sequence.
Revised: Here, we visualize the codes computed after the RNN processes the entire input sequence rather than part of the input sequence.

Original: Several additional sets of experimental results are shown; theﬁrst three on MNIST and thelast one on SST-2.
Revised: Several additional sets of experimental results are shown; the the

Original: First, we visualize the nearest and farthest neighbors of several MNIST digits in Figure 5.
Revised: First, we visualize the nearest and farthest neighbors of several MNIST digits in Figure 5.

Revised: Distance is calculated using the partition codes of the images.

Original: The left column is the original image; The next ﬁve columns are the ﬁve nearest neighbors to the original image; The last ﬁve columns are ﬁve farthest neighbors.
Revised: The left column is the original image; The next ﬁve columns are the ﬁve nearest neighbors to the original image; The last ﬁve columns are ﬁve farthest neighbors.

Original: This ﬁgure shows that  the images are well clustered.
Revised: This ﬁgure shows that partition codes the images are well clustered.

Original: Second, we visualize the nearest neighbors of a few wrongly classiﬁed images in Figure 6.

Original: This ﬁgure further shows the impressive input clustering that a simple RNN is able to achieve, in that even wrongly classiﬁed images have neighbors that are clearly of the same class as the wrongly classiﬁed ones.

Original: Third, we show the two dimensional projection using tSNE of the raw pixel and VQ representations of each data points in the MNIST dataset and visualize them in Figure 7.
Revised: Second, we show the two dimensional projection using t-SNE of the raw pixel and VQ representations of each data points in the MNIST dataset and visualize them in Figure 6.

Original: We clearly see a more distinct clustering using VQ representation of the data than using the raw pixel representation.
Revised: We clearly see a more distinct clustering using VQ representation of the data than using the raw pixel representation.

Original: This comparison demonstrate the ability of the RNN to extract useful information from the raw representation of the data in the form of VQ.
Revised: This comparison demonstrate the ability of the RNN to extract useful information from the raw representation of the data in the form of VQ.

Original: Fourth, we perform a KNN classiﬁcation with k ∈ { 1 , 2 , 5 , 10 } using 1) the RNN reparametrized data representation and 2) raw pixel data representation the MNIST test set to illustrate that the data reparametrized by the RNN has better clustering property than the original data representations.
Revised: Third, we perform a KNN classiﬁcation with k ∈ { 1 , 2 , 5 , 10 } using 1) the RNN computed partition codes of the inputs and 2) raw pixel data representation the MNIST test set to illustrate that the data reparametrized by the RNN has better clustering property than the original data representations.

Original: We use 80% of the test set to train the classiﬁer and the rest for testing.
Revised: We use 80% of the test set to train the classiﬁer and the rest for testing.

Original: The results are reported in
Revised: The results are reported in

Original: Table 3.
Revised: Table 3.

Original: We see that the classiﬁcation accuracies when using RNN reparametrized representations are signiﬁcantly higher than those when using raw pixel representations.
Revised: We see that the classiﬁcation accuracies when using RNN computed partition codes of the inputs are signiﬁcantly higher than those when using raw pixel representations.

Original: This result again shows the superior quality of the input space partitioning that RNN produces, and may suggest a new way to improve classiﬁcation accuracy by just using the reparametrized data with a KNN classiﬁer.[Table]
Revised: This result again shows the superior quality of the input space partitioning that RNN produces, and may suggest a new way to improve classiﬁcation accuracy by just using the reparametrized data with a KNN classiﬁer.

Original: Finally we visualize the 5 nearest and 5 farthest neighbors of a selected sentence from the SSTdataset to demonstrate that the partitioning effect on dataset of another modality.
Revised: Finally, we visualize the 5 nearest and 5 farthest neighbors of a selected sentence from the SST2 dataset to demonstrate that the partitioning effect on dataset of another modality.

Revised: Again, the distances are computed using the partition codes of the inputs.

Original: The results are shown in Figure 8.
Revised: The results are shown in Figure 7. [Table]

Original: We can see that all sentences that are nearest neighbors are of similar sentiment to the target sentence, whereas all sentences that are farthest neighbors are of the opposite sentiment.
Revised: We can see that all sentences that are nearest neighbors are of similar sentiment to the target sentence, whereas all sentences that are farthest neighbors are of the opposite sentiment.

Original: E A DDITIONAL T EMPLATE V ISUALIZATIONS
Revised: E A DDITIONAL T EMPLATE V ISUALIZATIONS

Original: We provide here more templates on images and texts in Figures 9 and 10.
Revised: We provide here more templates on images and texts in Figures 8 and 9.

Original: Notice here that although visually the templates may look similar or meaningless, they nevertheless have meaningful inner 
Revised: Notice here that, although visually the templates may look similar or meaningless, they nevertheless have meaningful inner product with the input.

Revised: The class index of the template that produces the largest inner product with the input is typically the correct class, as can be seen in the two ﬁgures.

Revised: F A DDITIONAL E XPERIMENTAL R ESULTS FOR R ANDOM I NITIAL H IDDEN S TATE F. R EGULARIZATION E FFECT FOR R EGRESSION P ROBLEM

Original: [Table]
Revised: We present the regularization effect on adding task formulated as a regression problem, following setup in Arjovsky et al. Result is shown in Figure 10. We see regularization effect similar to that presented in Figure 4, which demonstrates that the regularization effect does indeed happens for both classiﬁcation and regression problems, as Thm. 3 suggests

Revised: . F. C HOOSING S TANDARD D EVIATION IN R ANDOM I NITIAL H IDDEN S TATE

Revised: Table 4 shows the classiﬁcation accuracies under various settings.

Revised: The discussion of the results is in

Original:  It is a ﬁlm that will have people walking out halfway through , will encourage others to stand up and applaud , and will , undoubtedly , leave both camps engaged in a ferocious debate for years to come .
Revised: Section 6. Original [Table] It is a ﬁlm that will have people walking out halfway through , will encourage others to stand up and applaud , and will , undoubtedly , leave both camps engaged in a ferocious debate for years to come .

Original: ( + )
Revised: ( + )

Original: Nearest 2 neighbors
Revised: Nearest 5 neighbors

Original: Farthest 2 neighbors
Revised: Farthest 5 neighbors

Original: Well-written , nicely acted and beautifully shot and scored , the ﬁlm works on several levels , openly questioning social mores while ensnaring the audi- ence with its emotional pull .
Revised: Well-written , nicely acted and beautifully shot and scored , the ﬁlm works on several levels , openly questioning social mores while ensnaring the audi- ence with its emotional pull .

Original: ( + , 22.00)
Revised: ( + , 22.00)

Original: Marries the amateurishness of The Blair Witch Project with the illogic of Series 7 : The Contenders to create a completely crass and forgettable movie .
Revised: Marries the amateurishness of The Blair Witch Project with the illogic of Series 7 : The Contenders to create a completely crass and forgettable movie .

Original: A stunning piece of visual poetry that will , hope- fully , be remembered as one of the most important stories to be told in Australia ’s ﬁlm history .
Revised: A stunning piece of visual poetry that will , hope- fully , be remembered as one of the most important stories to be told in Australia ’s ﬁlm history .

Original: ( + , 22.23)
Revised: ( + , 22.23)

Original: K-19 may not hold a lot of water as a submarine epic , but it holds even less when it turns into an elegiacally soggy Saving Private Ryanovich .
Revised: K-19 may not hold a lot of water as a submarine epic , but it holds even less when it turns into an elegiacally soggy Saving Private Ryanovich .

Original: Cute , funny , heartwarming digitally animated fea- ture ﬁlm with plenty of slapstick humor for the kids , lots of in-jokes for the adults and heart enough for everyone .
Revised: Cute , funny , heartwarming digitally animated fea- ture ﬁlm with plenty of slapstick humor for the kids , lots of in-jokes for the adults and heart enough for everyone .

Original: ( + , 22.61)
Revised: ( + , 22.61)

Original: This is a great subject for a movie , but Hollywood has squandered the opportunity , using it as a prop for warmed-over melodrama and the kind of chore- ographed mayhem that director John Woo has built his career on .
Revised: This is a great subject for a movie , but Hollywood has squandered the opportunity , using it as a prop for warmed-over melodrama and the kind of chore- ographed mayhem that director John Woo has built his career on .

Original: Though it is by no means his best work , Laissez-
Revised: Though it is by no means his best work , Laissez-

Original: Passer is a distinguished and distinctive effort by a bona-ﬁde master , a fascinating ﬁlm replete with re- wards to be had by all willing to make the effort to reap them .
Revised: Passer is a distinguished and distinctive effort by a bona-ﬁde master , a fascinating ﬁlm replete with re- wards to be had by all willing to make the effort to reap them .

Original: ( + , 22.78)
Revised: ( + , 22.78)

Original: Flotsam in the sea of moviemaking , not big enough for us to worry about it causing signiﬁcant harm and not smelly enough to bother despising .
Revised: Flotsam in the sea of moviemaking , not big enough for us to worry about it causing signiﬁcant harm and not smelly enough to bother despising .

Original: An absorbing trip into the minds and motivations of people under stress as well as a keen , unsentimental look at variations on the theme of motherhood .
Revised: An absorbing trip into the minds and motivations of people under stress as well as a keen , unsentimental look at variations on the theme of motherhood .

Original: ( + , 22.89)
Revised: ( + , 22.89)

Original: If you ’re not a prepubescent girl , you ’ll be laugh- ing at Britney Spears ’ movie-starring debut whenever it does n’t have you impatiently squinting at your watch .
Revised: If you ’re not a prepubescent girl , you ’ll be laugh- ing at Britney Spears ’ movie-starring debut whenever it does n’t have you impatiently squinting at your watch .

Original: [Figure]
Revised: [Figure]

Original: image 0, -31.

Original: Input image 0, -6.

Original: Input image 0, -5.

Original: product with the input.

Original: The class index of the template that produces the largest inner product with the input is typically the correct class, as can be seen in the two ﬁgures.

Original: F A DDITIONAL E XPERIMENTAL R ESULTS FOR N OISY I NITIAL H IDDEN S TATE

Original: Here, we include additional experimental results for Section 5.

Original: We present the regularization effect on adding task formulated as a regression problem, following setup in Arjovsky et al. Result is shown in Figure 11. We see regularization effect similar to that presented in Figure 4, which demonstrates that the regularization effect does indeed happens for both classiﬁcation and regression problems, as Thm. 3 suggests.
Revised: [Figure]

Original: F. C HOOSING N OISE P ARAMETER

Original: Table 4 shows the classiﬁcation accuracies under various settings. The discussion of the results is in Section 5. G P ROOFS
Revised: image 0, -6.

Original: We present detailed proofs for several results in the main text.

Original: [Table]
Revised: [Figure]

Original: . P ROOF OF T HM .
Revised: G P ROOFS G. P ROOF OF T HM .

Original: To simplify notation

Original: [Table] similar to the main text [Table] in the proof here we drop the afﬁne parameters’ dependencies on the input, but keep in mind the input-dependency of these parameters.
Revised: To simplify notation, similar to the main text, in the proof here we drop the afﬁne parameters’ dependencies on the input, but keep in mind the input-dependency of these parameters.

Original: We ﬁrst derive the expression for a hidden state h ( (cid:96),t ) at a given time step t and layer (cid:96) .
Revised: We ﬁrst derive the expression for a hidden state h ( (cid:96),t ) at a given time step t and layer (cid:96) .

Original: Using
Revised: Using

Original: Prop. 1, we start with unrolling the RNN cell of layer (cid:96) at time step t for two time steps to t − 2 by recursively applying Eq. 1 as follows: 
Revised: Prop. 1, we start with unrolling the RNN cell of layer (cid:96) at time step t for two time steps to t − 2 by recursively applying (1) as follows: [Table] [Equation]

Original: [Equation]
Revised: [Equation]

Original: From Eq. 11 to Eq. 12, we use the result of Prop. 1.
Revised: From (12) to (13), we use the result of Prop. 1.

Original: From Eq. 12 to Eq. 13, we expand the term that involves the hidden state h ( (cid:96),t − 1) at the previous time step recursively using Eq.
Revised: From (13) to (14), we expand the term that involves the hidden state h ( (cid:96),t − 1) at the previous time step recursively using (13).

Original: From Eq. 13 to Eq. 14, we group terms and write them compactly.
Revised: From (14) to (15), we group terms and write them compactly.

Original: Now deﬁne A ( (cid:96),s ) s : t := (cid:81) s
Revised: Now deﬁne A ( (cid:96),s ) s : t := (cid:81) s

Original: +1 k
Revised: +1 k

Original: = t A ( (cid:96),k ) σ
Revised: = t A ( (cid:96),k ) σ

Original: W ( (cid:96) )
Revised: W ( (cid:96) )

Original: r for s < t and A ( (cid:96),t ) t : t := I where I is the identity matrix.
Revised: r for s < t and A ( (cid:96),t ) t : t := I where I is the identity matrix.

Original: Using this deﬁnition, we rewrite Eq.
Revised: Using this deﬁnition [Equation]

Original: 14 and proceed with the unrolling to the initial time step as follows:[Equation] [Equation]
Revised: we rewrite (15) and proceed with the unrolling to the initial time step as follows:

Original: [Equation]
Revised: [Equation]

Original: where B ( (cid:96),s ) =
Revised: where B ( (cid:96)

Original: A ( (cid:96),s ) σ b ( (cid:96) ) as deﬁned in Prop. 1.

Original: Repeat the above derivation for t ∈ { 1 , · · · , T } and stack h ( (cid:96),t ) in decreasing time steps from top to bottom, we have:
Revised: Repeat the above derivation for t ∈ { 1 , · · · , T } and stack h ( (cid:96),t ) in decreasing time steps from top to bottom, we have:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] [Equation] 
Revised: where

Original: [Equation]
Revised: [Equation]

Original: 
Revised: (cid:80) t = T A ( (cid:96) ) t : T B ( (cid:96),t )

Original: [Equation] where
Revised: + A ( (cid:96) ) 0: T h ( (cid:96), 0) ...

Original: [Equation]
Revised: A ( (cid:96) ) 1:1 B ( (cid:96),t ) + A ( (cid:96) ) 0:1 h ( (cid:96), 0)    , h ( (cid:96) − 1) =   h ( (cid:96) − 1 ,T ) ...

Original:  which concludes the proof.
Revised: h ( (cid:96) − 1 , 1)   , which concludes the proof.

Original: Thm. 2 follows from the recursive application of the above arguments for each layer (cid:96) ∈ { 1 , · · · , L } .
Revised: Thm. 2 follows from the recursive application of the above arguments for each layer (cid:96) ∈ { 1 , · · · , L } .

Original: G. P ROOF OF T HM .
Revised: G. P ROOF OF T HM .

Original: We prove for the case of multi-class classiﬁcation problem with softmax output.
Revised: We prove for the case of multi-class classiﬁcation problem with softmax output.

Original: The proof for the case of regression problems easily follows.
Revised: The proof for the case of regression problems easily follows.

Original: Let a i be the i th row of the input-dependent afﬁne parameter
Revised: Let a i be the i th row of the input-dependent afﬁne parameter

Original: A h where A h :
Revised: A h where A h :

Original: = A 1: T = (cid:81) 1 s = T A ( (cid:96),s ) σ
Revised: = A 1: T = (cid:81) 1 s = T A ( (cid:96),s ) σ

Original: W ( (cid:96) )
Revised: W ( (cid:96) )

Original: r (recall Section 5.1), x n =
Revised: r (recall Section 6), x n =

Original: [ x (1) (cid:62) n , · · · , x ( T ) (cid:62) n ] (cid:62) be the concatenation of the n th input sequence of length T and c be the index of the correct class.
Revised: [ x (1) (cid:62) n , · · · , x ( T ) (cid:62) n ] (cid:62) be the concatenation of the n th input sequence of length T and c be the index of the correct class.

Original: We assume thenoise amplitude  is small so that the input-dependent afﬁne parameter A h , which also depends on h (0) , does not change when noise is injected into h (0) .
Revised: We assume the amplitude of random initial hidden state is small so that the input-dependent afﬁne parameter A h , which also depends on h (0) , does not change when using random h (0) .

Original: Also, let z n = f RNN ( x n , h (0) ) be the overall RNN computation that represents Eq. 8.
Revised: Also, let z n = f RNN ( x n , h (0) ) be the overall RNN computation that represents (9).

Original: We ﬁrst rewrite the cross entropy loss with noise injected hidden state (cid:101)
Revised: We ﬁrst rewrite the cross entropy loss with random initial hidden state (cid:101)

Original: L CE 
Revised: L CE =

Original: [Equation] L [Equation] follows:
Revised: L CE (cid:0) softmax (cid:0) f RNN (cid:0) x n , h (0) + (cid:15) (cid:1)(cid:1)(cid:1) as follows:

Original: [Equation]
Revised: [Equation]

Original: Taking expectation of the (cid:101) L with respect to the error distribution, we have
Revised: Taking expectation of the (cid:101) L with respect to the distribution of the random Gaussian vector that the initial hidden state is set to, we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: R = 
Revised: R = 1 N

Original: [Equation]
Revised: [Equation]

Original: We note that similar forms of Eq. 20 have been previously derived by Wager et al.
Revised: [Equation]

Original: We now simplify Eq. 20 using second order Taylor expansion on h (0) of the summation inside the log function.
Revised: We now simplify (21) using second order Taylor expansion on h (0) of the summation inside the log function.

Original: Deﬁne function u ( x (1: T [Equation] n 
Revised: Deﬁne function u ( x (1: T ) n , h (0) ) :

Original: [Equation] Then
Revised: = log ( (cid:80) j exp ( z nj ))

Original: [Equation]
Revised: = log ( (cid:80) j exp ( f ( x (1: T ) n , h (0) ))) .

Original:  we can approximate Eq. 20 as follows:
Revised: Then, we can approximate (21) as follows:

Original: [Equation]
Revised: [Equation]

Original: where ∂u ( x n , h (0) 
Revised: where d u ( x n , h (0) )

Original: [Equation]

Original: d 2 h (0) is the Hessian matrix:
Revised: d 2 h (0) is the Hessian matrix:

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we can write the trace term in Eq. 21 as follows:
Revised: we can write the trace term in (22) as follows:

Original: [Equation]
Revised: [Equation]

Original: As a result, using the above approximations, we can rewrite the loss with noisy initial state in Eq.as:
Revised: As a result, using the above approximations

Original: [Equation]
Revised: [Equation]

Revised: we can rewrite the loss with random initial state in (19) as:

Original: [Equation]
Revised: [Equation]

Original: We see that this regularizer term does not dependent on the correct class index c of each data points.
Revised: We see that this regularizer term does not dependent on the correct class index c of each data points.

Original: H P RIOR W ORK ON THE E XPLODING G RADIENT IN RNN S
Revised: H P RIOR W ORK ON THE E XPLODING G RADIENT IN RNN S

Original: The problem of exploding gradients has been widely studied from different perspectives.
Revised: The problem of exploding gradients has been widely studied from different perspectives.

Original: First approaches have attempted to directly control the amplitude of the gradient through gradient clipping (Pascanu et al., 2013).
Revised: First approaches have attempted to directly control the amplitude of the gradient through gradient clipping (Pascanu et al., 2013).

Original: A more model driven approach has leveraged the analytical formula of the gradient when using speciﬁc nonlinearities and topologies in order to develop parametrization of the recurrent weights.
Revised: A more model driven approach has leveraged the analytical formula of the gradient when using speciﬁc nonlinearities and topologies in order to develop parametrization of the recurrent weights.

Original: This has led to various unitary reparametrizations of the recurrent weight (Arjovsky et al., 2016;
Revised: This has led to various unitary reparametrizations of the recurrent weight (Arjovsky et al., 2016;

Original: Wisdom et al., 2016;
Revised: Wisdom et al., 2016;

Original: Helfrich et al., 2018; Henaff et al., 2016;
Revised: Helfrich et al., 2018; Henaff et al., 2016;

Original: Jing et al., 2017; Mhammedi et al., 2017; Hyland & R¨atsch, 2017; Jose et al., 2018).
Revised: Jing et al., 2017; Mhammedi et al., 2017; Hyland & R¨atsch, 2017; Jose et al., 2018).

Original: A soft version of such parametrization lies in regularization of the DNNs.
Revised: A soft version of such parametrization lies in regularization of the DNNs.

Original: This includes dropout applied to either the output layer (Pham et al., 2014) or hidden state (Zaremba et al., 2014), noisin (Dieng et al., 2018), zoneout (Krueger et al., 2017) and recurrent batch normalization (Cooijmans et al., 2017).
Revised: This includes dropout applied to either the output layer (Pham et al., 2014) or hidden state (Zaremba et al., 2014), noisin (Dieng et al., 2018), zoneout (Krueger et al., 2017) and recurrent batch normalization (Cooijmans et al., 2017).

Original: Lastly, identity initialization of ReLU RNNs has been studied in Le et al.
Revised: Lastly, identity initialization of ReLU RNNs has been studied in Le et al.

Original: and Talathi & Vartak (2016).
Revised: and Talathi & Vartak (2016).

Original: Our results complements prior works in that simply injecting noise to the initial hidden state and without changing the RNN structure also relieves the exploding gradient problem by regularization the potentially largest term in the recurrent weight gradient.
Revised: Our results complements prior works in that simply using random initial hidden state instead of zero initial hidden state and without changing the RNN structure also relieves the exploding gradient problem by regularization the potentially largest term in the recurrent weight gradient.