Original: h - DETACH : M ODIFYING THE LSTM G RADIENT T OWARDS B ETTER O PTIMIZATION
Revised: h - DETACH : M ODIFYING THE LSTM G RADIENT T OWARDS B ETTER O PTIMIZATION

Original: Anonymous authors
Revised: gmail com

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP).
Revised: Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP).

Original: This problem becomes more evident in tasks where the information needed to correctly solve them exist over long time scales, because
Revised: This problem becomes more evident in tasks where the information needed to correctly solve them exist over long time scales, because

Original: EVGP prevents important gradient components from being back-propagated adequately over a large number of steps.
Revised: EVGP prevents important gradient components from being back-propagated adequately over a large number of steps.

Original: We introduce a simple stochastic algorithm ( h -detach) that is speciﬁc to LSTM optimization and targeted towards addressing this problem.
Revised: We introduce a simple stochastic algorithm ( h -detach) that is speciﬁc to LSTM optimization and targeted towards addressing this problem.

Original: Speciﬁcally, we show that when the LSTM weights are large, the gradient components through the linear path (cell state) in the LSTM computational graph get suppressed.
Revised: Speciﬁcally, we show that when the LSTM weights are large, the gradient components through the linear path (cell state) in the LSTM computational graph get suppressed.

Original: Based on the hypothesis that these components carry information about long term dependencies (which we show empirically), their suppression can prevent LSTMs from capturing them.
Revised: Based on the hypothesis that these components carry information about long term dependencies (which we show empirically), their suppression can prevent LSTMs from capturing them.

Original: Our algorithm  prevents gradients ﬂowing through this path from getting suppressed, thus allowing the LSTM to capture such dependencies better.
Revised: Our algorithm 1 prevents gradients ﬂowing through this path from getting suppressed, thus allowing the LSTM to capture such dependencies better.

Original: We show signiﬁcant improvements over vanilla LSTM gradient based training in terms of convergence speed, robustness to seed and learning rate, and generalization using our modiﬁcation of
Revised: We show signiﬁcant improvements over vanilla LSTM gradient based training in terms of convergence speed, robustness to seed and learning rate, and generalization using our modiﬁcation of

Original: LSTM gradient on various benchmark datasets.
Revised: LSTM gradient on various benchmark datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recurrent Neural Networks (RNNs) (Rumelhart et al.
Revised: Recurrent Neural Networks (RNNs) (Rumelhart et al.

Original: (1986); Elman (1990))
Revised: (1986); Elman (1990))

Original: are a class of neural network architectures used for modeling sequential data.
Revised: are a class of neural network architectures used for modeling sequential data.

Original: Compared to feed-forward networks, the loss landscape of recurrent neural networks are much harder to optimize.
Revised: Compared to feed-forward networks, the loss landscape of recurrent neural networks are much harder to optimize.

Original: Among others, this difﬁculty may be attributed to the exploding and vanishing gradient problem (Hochreiter, 1991; Bengio et al., 1994; Pascanu et al., 2013) which is more severe for recurrent networks and arises due to the highly ill-conditioned nature of their loss surface.
Revised: Among others, this difﬁculty may be attributed to the exploding and vanishing gradient problem (Hochreiter, 1991; Bengio et al., 1994; Pascanu et al., 2013) which is more severe for recurrent networks and arises due to the highly ill-conditioned nature of their loss surface.

Original: This problem becomes more evident in tasks where training data has dependencies that exist over long time scales.
Revised: This problem becomes more evident in tasks where training data has dependencies that exist over long time scales.

Original: Due to the aforementioned optimization difﬁculty, variants of RNN architectures have been proposed that aim at addressing these problems.
Revised: Due to the aforementioned optimization difﬁculty, variants of RNN architectures have been proposed that aim at addressing these problems.

Original: The most popular among such architectures that are used in a wide number of applications include long short term memory (LSTM, Hochreiter & Schmidhuber (1997)) and gated recurrent unit (GRU, Chung et al.
Revised: The most popular among such architectures that are used in a wide number of applications include long short term memory (LSTM, Hochreiter & Schmidhuber (1997)) and gated recurrent unit (GRU, Chung et al.

Original: networks, which is a variant of LSTM with forget gates (Gers et al., 1999).
Revised: networks, which is a variant of LSTM with forget gates (Gers et al., 1999).

Original: These architectures mitigate such difﬁculties by introducing a linear temporal path that allows gradients to ﬂow more freely across time steps.
Revised: These architectures mitigate such difﬁculties by introducing a linear temporal path that allows gradients to ﬂow more freely across time steps.

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: (2016) on the other hand try to address this problem by parameterizing a recurrent neural network to have unitary transition matrices based on the idea that unitary matrices have unit singular values which prevents gradients from exploding/vanishing.
Revised: (2016) on the other hand try to address this problem by parameterizing a recurrent neural network to have unitary transition matrices based on the idea that unitary matrices have unit singular values which prevents gradients from exploding/vanishing.

Original: Among the aforementioned RNN architectures, LSTMs are arguably most widely used (for instance they have more representational power compared with GRUs (Weiss et al., 2018)) and it remains a hard problem to optimize them on tasks that involve long term dependencies.
Revised: Among the aforementioned RNN architectures, LSTMs are arguably most widely used (for instance they have more representational power compared with GRUs (Weiss et al., 2018)) and it remains a hard problem to optimize them on tasks that involve long term dependencies.

Original: Examples of such tasks are copying problem (Bengio et al., 1994; Pascanuet al., 2013), andsequential MNIST (Le et al., 2015), which are designed in such a way that the only way to produce the correct output is for the model to retain information over long time scales.
Revised: Examples of such tasks are copying problem (Bengio al , 1994; Pascanu al., 2013), and MNIST (

Original: The goal of this paper is to introduce a simple trick that is speciﬁc to LSTM optimization and improves its training on tasks that involve long term dependencies.
Revised: The goal of this paper is to introduce a simple trick that is speciﬁc to LSTM optimization and improves its training on tasks that involve long term dependencies.

Original: To achieve this goal, we write out the full back-propagation gradient equation for LSTM parameters and split the composition of this gradient into its components resulting from different paths in the unrolled network.
Revised: To achieve this goal, we write out the full back-propagation gradient equation for LSTM parameters and split the composition of this gradient into its components resulting from different paths in the unrolled network.

Original: We then  when LSTM weights are large in magnitude, the gradients through the linear temporal path (cell state) get suppressed ( that this path was designed to allow smooth gradient ﬂow over many time steps).
Revised: We then show that when LSTM weights are large in magnitude, the gradients through the linear temporal path (cell state) get suppressed (recall that this path was designed to allow smooth gradient ﬂow over many time steps).

Original: We show empirical evidence that this path carries information about long term dependencies (see section 3.5) and hence gradients from this path getting suppressed is problematic for tasks ﬁx this problem, we introduce a simple stochastic algorithm that in expectation scales the individual gradient components, which prevents the gradients through the linear temporal path from being suppressed.
Revised: We show empirical evidence that this path carries information about long term dependencies (see section 3.5) and hence gradients from this path getting suppressed is problematic for such tasks. To ﬁx this problem, we introduce a simple stochastic algorithm that in expectation scales the individual gradient components, which prevents the gradients through the linear temporal path from being suppressed.

Original: In essence, the algorithm stochastically prevents gradient from ﬂowing through the h -state of the LSTM ( hence we call it h -detach.
Revised: In essence, the algorithm stochastically prevents gradient from ﬂowing through the h -state of the LSTM (see ﬁgure 1), hence we call it h -detach.

Original: Using this method, we show improvements in convergence/generalization over vanilla LSTM optimization on the copying task, transfer copying task, sequential and permuted MNIST and image captioning.
Revised: Using this method, we show improvements in convergence/generalization over vanilla LSTM optimization on the copying task, transfer copying task, sequential and permuted MNIST, and image captioning.

Original:  P ROPOSED M ETHOD : h - DETACH
Revised:  P ROPOSED M ETHOD : h - DETACH

Original: We begin by reviewing the LSTM roll-out equations.
Revised: We begin by reviewing the LSTM roll-out equations.

Original: We then derive the LSTM back-propagation equations and by studying its decomposition, identify the aforementioned problem.
Revised: We then derive the LSTM back-propagation equations and by studying its decomposition, identify the aforementioned problem.

Original: Based on this analysis we propose a simple stochastic algorithm to ﬁx this problem.
Revised: Based on this analysis we propose a simple stochastic algorithm to ﬁx this problem.

Original: 2. L ONG S HORT T ERM M EMORY N ETWORKS
Revised: 2. L ONG S HORT T ERM M EMORY N ETWORKS

Original: LSTM is a variant of traditional RNNs that was designed with the goal of improving the ﬂow of gradients over many time steps.
Revised: LSTM is a variant of traditional RNNs that was designed with the goal of improving the ﬂow of gradients over many time steps.

Original: The roll-out equations of an LSTM are as follows,
Revised: The roll-out equations of an LSTM are as follows,

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) denotes point-wise product and the gates f t , i t , o t and g t are deﬁned as,
Revised: where (cid:12) denotes point-wise product and the gates f t , i t , o t and g t are deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: Here c t and h t are the cell state and hidden state respectively.
Revised: Here c t and h t are the cell state and hidden state respectively.

Original: Usually a transformation φ ( h T ) is used as the output at time step t (Eg.
Revised: Usually a transformation φ ( h T ) is used as the output at time step t (Eg.

Original: next word prediction in language model) based on which we can compute the loss (cid:96) t :=
Revised: next word prediction in language model) based on which we can compute the loss (cid:96) t :=

Original: (cid:96) ( φ ( h t )) for that time step.
Revised: (cid:96) ( φ ( h t )) for that time step.

Original: An important feature of the LSTM architecture is the linear recursive relation between the cell states c t as shown in Eq.
Revised: An important feature of the LSTM architecture is the linear recursive relation between the cell states c t as shown in Eq.

Original: This linear path allows gradients to ﬂow easily over long time scales.
Revised: This linear path allows gradients to ﬂow easily over long time scales.

Original: This however is one of the components in the full composition of the LSTM gradient.
Revised: This however is one of the components in the full composition of the LSTM gradient.

Original: As we will show next, the remaining components that are a result of the other paths in the LSTM computational graph are polynomial in the weight matrices W gh , W fh , W ih , W oh whose order grows with the number of time steps.
Revised: As we will show next, the remaining components that are a result of the other paths in the LSTM computational graph are polynomial in the weight matrices W gh , W fh , W ih , W oh whose order grows with the number of time steps.

Original: These terms cause an imbalance in the order of magnitude of gradients from different paths, thereby suppressing gradients from linear paths of LSTM computational graph in cases where the weight matrices are large.
Revised: These terms cause an imbalance in the order of magnitude of gradients from different paths, thereby suppressing gradients from linear paths of LSTM computational graph in cases where the weight matrices are large.

Original: 2. B ACK - PROPAGATION E QUATIONS FOR LSTM
Revised: 2. B ACK - PROPAGATION E QUATIONS FOR LSTM

Original: In this section we derive the back-propagation equations for LSTM network and by studying its composition, we identify a problem in this composition.
Revised: In this section we derive the back-propagation equations for LSTM network and by studying its composition, we identify a problem in this composition.

Original: The back-propagation equation of an LSTM can be written in the following form.
Revised: The back-propagation equation of an LSTM can be written in the following form.

Original: Theorem 1 Fix w to be an element of the matrix W gh , W fh , W ih , W oh , W gx , W fx , W ix or
Revised: Theorem 1 Fix w to be an element of the matrix W gh , W fh , W ih , W oh , W gx , W fx , W ix or

Original: W ox .
Revised: W ox .

Original: Deﬁne,
Revised: Deﬁne,

Original: [Equation]
Revised: [Equation]

Original: Then z t =
Revised: Then z t =

Original: ( A t + B t )
Revised: ( A t + B t )

Original: z t − 1 .
Revised: z t − 1 .

Original: In other words,
Revised: In other words,

Original: [Equation]
Revised: [Equation]

Original:  where all the symbols used to deﬁne A t and B t are deﬁned in notation 1 in appendix.
Revised: (8) where all the symbols used to deﬁne A t and B t are deﬁned in notation 1 in appendix.

Original: To avoid unnecessary details, we use a compressed deﬁnitions of A t and B t in the above statement and write the detailed deﬁnitions of the symbols that constitute them in notation 1 in appendix.
Revised: To avoid unnecessary details, we use a compressed deﬁnitions of A t and B t in the above statement and write the detailed deﬁnitions of the symbols that constitute them in notation 1 in appendix.

Original: Nonetheless, we now provide some intuitive properties of the matrices A t and B t .
Revised: Nonetheless, we now provide some intuitive properties of the matrices A t and B t .

Original: The matrix A t contains components of parameter’s full gradient that arise due to the cell state (linear temporal path) described in Eq.
Revised: The matrix A t contains components of parameter’s full gradient that arise due to the cell state (linear temporal path) described in Eq.

Original: (1) (top most horizontal path in ﬁgure 1).
Revised: (1) (top most horizontal path in ﬁgure 1).

Original: Thus the terms in A t are a function of the LSTM gates and hidden and cell states.
Revised: Thus the terms in A t are a function of the LSTM gates and hidden and cell states.

Original: Note that all the gates and hidden states h t are bounded by deﬁnition because they are a result of sigmoid or tanh activation functions.
Revised: Note that all the gates and hidden states h t are bounded by deﬁnition because they are a result of sigmoid or tanh activation functions.

Original: The cell state c t on the other hand evolves through a linear recursive equation shown in Eq.
Revised: The cell state c t on the other hand evolves through a linear recursive equation shown in Eq.

Original: Thus it can grow at each time step by at most ± 1 (element-wise) and its value is bounded by the number of time steps t .
Revised: Thus it can grow at each time step by at most ± 1 (element-wise) and its value is bounded by the number of time steps t .

Original: Thus given a ﬁnite number of time steps and ﬁnite initialization of c 0 , the values in matrix A t are bounded.
Revised: Thus given a ﬁnite number of time steps and ﬁnite initialization of c 0 , the values in matrix A t are bounded.

Original: The matrix B t on the other hand contains components of parameter’s full gradient that arise due to the remaining paths.
Revised: The matrix B t on the other hand contains components of parameter’s full gradient that arise due to the remaining paths.

Original: The elements of B t are a linear function of the weights
Revised: The elements of B t are a linear function of the weights

Original: W gh , W fh , W ih , W
Revised: W gh , W fh , W ih , W

Original: oh .
Revised: oh .

Original: Thus the magnitude of elements in B t can become very large irrespective of the number of time steps if the weights are very large.
Revised: Thus the magnitude of elements in B t can become very large irrespective of the number of time steps if the weights are very large.

Original: This problem becomes worse when we multiply B t s in Eq.
Revised: This problem becomes worse when we multiply B t s in Eq.

Original: (8) because the product becomes polynomial in the weights which can become unbounded for large weights very quickly as the number of time steps grow.
Revised: (8) because the product becomes polynomial in the weights which can become unbounded for large weights very quickly as the number of time steps grow.

Original: Thus based on the above analysis, we identify the following problem with the LSTM gradient: when the LSTM weights are large, the gradient component through the cell state paths ( A t ) get suppressed compared to the gradient components through the other paths ( B t ) due to an imbalance in gradient component magnitudes.
Revised: Thus based on the above analysis, we identify the following problem with the LSTM gradient: when the LSTM weights are large, the gradient component through the cell state paths ( A t ) get suppressed compared to the gradient components through the other paths ( B t ) due to an imbalance in gradient component magnitudes.

Original: We recall that the linear recursion in the cell state path was introduced in the LSTM architecture (Hochreiter & Schmidhuber, 1997) as an important feature to allow gradients to ﬂow smoothly through time.
Revised: We recall that the linear recursion in the cell state path was introduced in the LSTM architecture (Hochreiter & Schmidhuber, 1997) as an important feature to allow gradients to ﬂow smoothly through time.

Original: As we show in our ablation studies (section 3.5), this path carries information about long term dependencies in the data.
Revised: As we show in our ablation studies (section 3.5), this path carries information about long term dependencies in the data.

Original: Hence it is problematic if the gradient components from this path get suppressed.
Revised: Hence it is problematic if the gradient components from this path get suppressed.

Original: 2. h - DETACH
Revised: 2. h - DETACH

Original: We now propose a simple ﬁx to the above problem.
Revised: We now propose a simple ﬁx to the above problem.

Original: Our goal is to manipulate the gradient components such that the components through the cell state path ( A t ) do not get suppressed when the components through the remaining paths ( B t ) are very large (described in the section 2.2).
Revised: Our goal is to manipulate the gradient components such that the components through the cell state path ( A t ) do not get suppressed when the components through the remaining paths ( B t ) are very large (described in the section 2.2).

Original: Thus it would be helpful to multiply B t by a positive number less than 1 to dampen its magnitude.
Revised: Thus it would be helpful to multiply B t by a positive number less than 1 to dampen its magnitude.

Original: In
Revised: In

Original: Algorithm 1 we propose a simple trick that achieves this goal.
Revised: Algorithm 1 we propose a simple trick that achieves this goal.

Original: A diagrammatic form of algorithm 1 is shown in Figure 1.
Revised: A diagrammatic form of algorithm 1 is shown in Figure 1.

Original: In simple words, our algorithm essentially blocks gradients from ﬂowing through each of the h t states independently with a probability 1 − p , where p ∈ [0 , 1] is a tunable hyper-parameter.
Revised: In simple words, our algorithm essentially blocks gradients from ﬂowing through each of the h t states independently with a probability 1 − p , where p ∈ [0 , 1] is a tunable hyper-parameter.

Original: Note the subtle detail in Algorithm 1 (line 9) that the loss (cid:96) t at any time step t is a function of h t which is not detached.
Revised: Note the subtle detail in Algorithm 1 (line 9) that the loss (cid:96) t at any time step t is a function of h t which is not detached.

Original: 1: INPUT: { x t } Tt =1 , h 0 , c 0 , p 2: (cid:96) =3: for 1 ≤ t ≤ T do 4: if bernoulli( p )=
Revised: 1: INPUT: { x t } Tt =1 , h 0 , c 0 , p 2: (cid:96) =3: for 1 ≤ t ≤ T do 4: if bernoulli( p )=

Original: =1 then 5: ˜h t − 1 ← stop-gradient ( h t − 1 ) 6: else 7: ˜h t − 1 ← h t −8: h t , c t ← LSTM( x t , ˜h t − 1 , c
Revised: =1 then 5: ˜h t − 1 ← stop-gradient ( h t − 1 ) 6: else 7: ˜h t − 1 ← h t −8: h t , c t ← LSTM( x t , ˜h t − 1 , c

Original: t − 1 ) (Eq.
Revised: t − 1 ) (Eq.

Original: 1- 6) 9: (cid:96) t ← loss( φ ( h t ) ) 

Revised: 1- 6) 9: (cid:96) t ← loss( φ ( h t ) )

Original: (cid:96) ← (cid:96) + (cid:96) t 11: return (cid:96)
Revised: (cid:96) ← (cid:96) + (cid:96) t 11: return (cid:96)

Original: We now show that the gradient of the loss function resulting from the LSTM forward pass shown in algorithm 1 has the property that the gradient components arising from B t get dampened.
Revised: We now show that the gradient of the loss function resulting from the LSTM forward pass shown in algorithm 1 has the property that the gradient components arising from B t get dampened.

Original: Theorem 2 Let z t =
Revised: Theorem 2 Let z t =

Original: [ d c t dw T ; d h t dw T ; 1 Tn ] T and ˜ z t be the analogue of z t when applying h -detach with probability 1 − p during back-propagation.
Revised: [ d c t dw T ; d h t dw T ; 1 Tn ] T and ˜ z t be the analogue of z t when applying h -detach with probability 1 − p during back-propagation.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where ξ t , ξ t − 1 , . . .
Revised: where ξ t , ξ t − 1 , . . .

Original: , ξ 2 are i.i.d.
Revised: , ξ 2 are i.i.d.

Original: Bernoulli random variables with probability p of being 1, and w ,
Revised: Bernoulli random variables with probability p of being 1, and w ,

Original: A t and B t and are same as deﬁned in theorem 1.
Revised: A t and B t and are same as deﬁned in theorem 1.

Original: The above theorem shows that by stochastically blocking gradients from ﬂowing through the h t states of an LSTM with probability 1 − p , we stochastically drop the B t term in the gradient components.
Revised: The above theorem shows that by stochastically blocking gradients from ﬂowing through the h t states of an LSTM with probability 1 − p , we stochastically drop the B t term in the gradient components.

Original: The corollary below shows that in expectation, this results in dampening the B t term compared to the original LSTM gradient.
Revised: The corollary below shows that in expectation, this results in dampening the B t term compared to the original LSTM gradient.

Original: Corollary
Revised: Corollary

Original: [Equation]
Revised: [Equation]

Original: Finally, we note that when training LSTMs with h -detach, we reduce the amount of computation needed.
Revised: Finally, we note that when training LSTMs with h -detach, we reduce the amount of computation needed.

Original: This is simply because by stochastically blocking the gradient from ﬂowing through the h t hidden states of LSTM, less computation needs to be done during back-propagation through time (BPTT).
Revised: This is simply because by stochastically blocking the gradient from ﬂowing through the h t hidden states of LSTM, less computation needs to be done during back-propagation through time (BPTT).

Original:  E XPERIMENTS 3. C OPYING T ASK
Revised:  E XPERIMENTS 3. C OPYING T ASK

Original: This task requires the recurrent network to memorize the network inputs provided at the ﬁrst few time steps and output them in the same order after a large time delay.
Revised: This task requires the recurrent network to memorize the network inputs provided at the ﬁrst few time steps and output them in the same order after a large time delay.

Original: Thus the only way to solve this task is for the network to capture the long term dependency between inputs and targets which requires gradient components carrying this information to ﬂow through many time steps.
Revised: Thus the only way to solve this task is for the network to capture the long term dependency between inputs and targets which requires gradient components carrying this information to ﬂow through many time steps.

Original: We follow the copying task setup identical to Arjovsky et al.
Revised: We follow the copying task setup identical to Arjovsky et al.

Original: (described in appendix).
Revised: (described in appendix).

Original: Using their data generation process, we sample 100,000 training input-target sequence pairs and 5, validation pairs.
Revised: Using their data generation process, we sample 100,000 training input-target sequence pairs and 5, validation pairs.

Original: We use cross-entropy as our loss to train an LSTM with hidden state size 128 for a maximum of 500-600 epochs.
Revised: We use cross-entropy as our loss to train an LSTM with hidden state size 128 for a maximum of 500-600 epochs.

Original: We use the ADAM optimizer with batch-size 100, learning rate 0.
Revised: We use the ADAM optimizer with batch-size 100, learning rate 0.

Original: and clip the gradient norms to 1.
Revised: and clip the gradient norms to 1.

Revised: [Figure]

Revised: T = 300 is a more interesting case because it involves longer term dependencies.

Revised: In this case we ﬁnd that h -detach leads to faster convergence and achieves ∼ 100% validation accuracy while being more robust to the choice of seed.

Original: Figure 2 shows the validation accuracy plots for copying task training for T = 100 (top row) and
Revised: Figure 2 shows the validation accuracy plots for copying task training for T = 100 (top row) and

Original: T = 300 (bottom row) without h -detach (left), and with h -detach (middle and right).
Revised: T = 300 (bottom row) without h -detach (left), and with h -detach (middle and right).

Original: Each plot contains runs from the same algorithm with multiple seeds to show a healthy sample of variations using these algorithms.
Revised: Each plot contains runs from the same algorithm with multiple seeds to show a healthy sample of variations using these algorithms.

Original: For T = 100 time delay, we see both vanilla LSTM and LSTM with hdetach converge to 100% accuracy.
Revised: For T = 100 time delay, we see both vanilla LSTM and LSTM with hdetach converge to 100% accuracy.

Original: For time delay 100 and the training setting used, vanilla LSTM is known to converge to optimal validation performance (for instance, see Arjovsky et al.
Revised: For time delay 100 and the training setting used, vanilla LSTM is known to converge to optimal validation performance (for instance, see Arjovsky et al.

Original: Nonetheless, we note that h -detach converges faster in this setting.
Revised: Nonetheless, we note that h -detach converges faster in this setting.

Original: A more interesting case is when [Figure] T = 300 is a more interesting case because it involves longer term dependencies.
Revised: A more interesting case is when time decay is set to 300 because it requires capturing longer term dependencies.

Original: In this case we ﬁnd that h -detach leads to faster convergence and achieves ∼ 100% validation accuracy while being more robust to the choice of seed.

Original: [Table]

Original: time decay is set to 300 because it requires capturing longer term dependencies.

Original: In this case, we ﬁnd that LSTM training without h -detach achieves a validation accuracy of ∼ 82% at best while a number of other seeds converge to much worse performance.
Revised: In this case, we ﬁnd that LSTM training without h -detach achieves a validation accuracy of ∼ 82% at best while a number of other seeds converge to much worse performance.

Original: On the other hand, we ﬁnd that using h -detach with detach probabilities 0.25 and 0.5 achieves the best performance of 100% and converging quickly while being reasonably robust to the choice of seed.
Revised: On the other hand, we ﬁnd that using h -detach with detach probabilities 0.25 and 0.5 achieves the best performance of 100% and converging quickly while being reasonably robust to the choice of seed.

Original: 3. T RANSFER COPYING TASK
Revised: 3. T RANSFER COPYING TASK

Original: Having shown the beneﬁt of h -detach in terms of training dynamics, we now extend the challenge of the copying task by evaluating how well an LSTM trained on data with a certain time delay generalizes when a larger time delay is used during inference.
Revised: Having shown the beneﬁt of h -detach in terms of training dynamics, we now extend the challenge of the copying task by evaluating how well an LSTM trained on data with a certain time delay generalizes when a larger time delay is used during inference.

Original: This task is referred as the transfer copying task (Hochreiter & Schmidhuber, 1997).
Revised: This task is referred as the transfer copying task (Hochreiter & Schmidhuber, 1997).

Original: Speciﬁcally, we train the LSTM architecture on copying task with delay T = 100 without h -detach and with h -detach with probability 0.25 and 0.5.
Revised: Speciﬁcally, we train the LSTM architecture on copying task with delay T = 100 without h -detach and with h -detach with probability 0.25 and 0.5.

Original: We then evaluate the accuracy of the trained model for each setting for various values of T > 100 .
Revised: We then evaluate the accuracy of the trained model for each setting for various values of T > 100 .

Original: The results are shown in table 1.
Revised: The results are shown in table 1.

Original: We ﬁnd that the function learned by LSTM when trained with h -detach generalize signiﬁcantly better on longer time delays during inference compared with the
Revised: We ﬁnd that the function learned by LSTM when trained with h -detach generalize signiﬁcantly better on longer time delays during inference compared with the

Original: LSTM trained without h -detach.
Revised: LSTM trained without h -detach.

Original: [Figure]

Original: 3. S EQUENTIAL MNIST
Revised: 3. S EQUENTIAL MNIST

Original: This task is a sequential version of the MNIST classiﬁcation task (LeCun & Cortes, 2010).
Revised: This task is a sequential version of the MNIST classiﬁcation task (LeCun & Cortes, 2010).

Original: In this task, an image is fed into the LSTM one pixel per time step and the goal is to predict the label after the last pixel is fed.
Revised: In this task, an image is fed into the LSTM one pixel per time step and the goal is to predict the label after the last pixel is fed.

Original: We consider two versions of the task: one is which the pixels are read in order (from left to right and top to bottom), and one where all the pixels are permuted in a random but ﬁxed order.
Revised: We consider two versions of the task: one is which the pixels are read in order [Table] [Figure]

Original: We callthe second version the permuted MNIST task or pMNIST in short. The setup usedfor this experiment is as follows.
Revised: We call used

Original: We use 50000 images for training, 10000 for validation and 10000 for testing.
Revised: We use 50000 images for training, 10000 for validation and 10000 for testing.

Original: We use the ADAM optimizer with different learning rates– 0.001,0.0005 and 0.0001, and a ﬁxed batch size of 100.
Revised: We use the ADAM optimizer with different learning rates– 0.001,0.0005 and 0.0001, and a ﬁxed batch size of 100.

Original: We train for 200 epochs and pick our ﬁnal model based on the best validation score.
Revised: We train for 200 epochs and pick our ﬁnal model based on the best validation score.

Original: We use an LSTM with 100 hidden units.
Revised: We use an LSTM with 100 hidden units.

Original: For h -detach, we do a hyperparameter search on the detach probability in { 0 . 1 , 0 . 25 , 0 . 4 , 0 . 5 } .
Revised: For h -detach, we do a hyperparameter search on the detach probability in { 0 . 1 , 0 . 25 , 0 . 4 , 0 . 5 } .

Original: For both pixel by pixel MNIST and pMNIST, we found the detach hyper-parameter of 0 .
Revised: For both pixel by pixel MNIST and pMNIST, we found the detach hyper-parameter of 0 .

Original: 25 to perform best on the validation set for both MNIST and pMNIST.
Revised: 25 to perform best on the validation set for both MNIST and pMNIST.

Original: On the sequential MNIST task, both vanilla LSTM and training with h -detach give an accuracy of 98 .
Revised: On the sequential MNIST task, both vanilla LSTM and training with h -detach give an accuracy of 98 .

Original: Here, we note that the convergence of our method is much faster and is more robust to the different learning rates of the ADAM optimizer as seen in Figure 3.
Revised: Here, we note that the convergence of our method is much faster and is more robust to the different learning rates of the ADAM optimizer as seen in Figure 3.

Original: Refer to appendix (ﬁgure 6) for experiments with multiple seeds that shows the robustness of our method to initialization.
Revised: Refer to appendix (ﬁgure 6) for experiments with multiple seeds that shows the robustness of our method to initialization.

Original: In the pMNIST task, we ﬁnd that training LSTM with h -detach gives a test accuracy of 92 . 3% which is an improvement over the regular LSTM training which reaches an accuracy of 91 .
Revised: In the pMNIST task, we ﬁnd that training LSTM with h -detach gives a test accuracy of 92 . 3% which is an improvement over the regular LSTM training which reaches an accuracy of 91 .

Original: A detailed comparison of test performance with existing algorithms is shown in table 2.
Revised: A detailed comparison of test performance with existing algorithms is shown in [Table] [Table] [Table]

Original: I MAGE C APTIONING
Revised: I MAGE C APTIONING

Original: We now evaluate h -detach on an image captioning task which involves using an RNN for generating captions for images.
Revised: We now evaluate h -detach on an image captioning task which involves using an RNN for generating captions for images.

Original: We use the Microsoft COCO dataset (Lin et al., 2014) which contains 82, training images and 40,504 validation images.
Revised: We use the Microsoft COCO dataset (Lin et al., 2014) which contains 82, training images and 40,504 validation images.

Original: Since this dataset does not have a standard split for training, validation and test, we follow the setting in Karpathy & Fei-Fei (2015) which suggests a split of 80,000 training images and 5,000 images each for validation and test set.[Table]
Revised: Since this dataset does not have a standard split for training, validation and test, we follow the setting in Karpathy & Fei-Fei (2015) which suggests a split of 80,000 training images and 5,000 images each for validation and test set.

Original: [Table] We use two models to test our approach– the Show&Tell encoder-decoder model (Vinyals et al., 2015) which does not employ any attention mechanism, and the Show, Attend and Tell model (Xu et al., 2015), which uses soft attention.
Revised: We use two models to test our approach– the Show&Tell encoder-decoder model (Vinyals et al., 2015) which does not employ any attention mechanism, and the ‘Show, Attend and Tell’ model (Xu et al., 2015), which uses soft attention.

Original: For feature extraction, we use the 2048-dimensional last layer feature vector of a residual network (Resnet He et al.
Revised: For feature extraction, we use the 2048-dimensional last layer feature vector of a residual network (Resnet He et al.

Original: with 152 layers which was pretrained on ImageNet for image classiﬁcation.
Revised: with 152 layers which was pretrained on ImageNet for image classiﬁcation.

Original: We use an LSTM with 512 hidden units for caption generation.
Revised: We use an LSTM with 512 hidden units for caption generation.

Original: We train both the Resnet and LSTM models using the ADAM optimizer (Kingma &
Revised: We train both the Resnet and LSTM models using the ADAM optimizer (Kingma &

Original: Ba, 2014) with a learning rate of 10 − 4 and leave the rest of the hyper-parameters as suggested in their paper.
Revised: Ba, 2014) with a learning rate of 10 − 4 and leave the rest of the hyper-parameters as suggested in their paper.

Original: We also perform a small hyperparameter search where we ﬁnd the optimial value of the h -detach parameter.
Revised: We also perform a small hyperparameter search where we ﬁnd the optimial value of the h -detach parameter.

Original: We considered values in the set { 0 . 1 , 0 .
Revised: We considered values in the set { 0 . 1 , 0 .

Original: and pick the optimal value based on the best validation score.
Revised: and pick the optimal value based on the best validation score.

Original: Similar to Serdyuk et al.
Revised: Similar to Serdyuk et al.

Original: (2018), we early stop based on the validation CIDEr scores and report BLEU-1 to BLEU-4, CIDEr, and Meteor scores.
Revised: (2018), we early stop based on the validation CIDEr scores and report BLEU-1 to BLEU-4, CIDEr, and Meteor scores.

Original: The results are presented in table 3.
Revised: The results are presented in table 3.

Original: Training the LSTM with h -detach outperforms the baseline
Revised: Training the LSTM with h -detach outperforms the baseline

Original: LSTM by a good margin for all the metrics and produces the best BLEU-1 to BLEU-3 scores among all the compared methods.
Revised: LSTM by a good margin for all the metrics and produces the best BLEU-1 to BLEU-3 scores among all the compared methods.

Original: Even for the other metrics, except for the results reported by Lu et al.
Revised: Even for the other metrics, except for the results reported by Lu et al.

Original: (2017), we beat all the other methods reported.
Revised: (2017), we beat all the other methods reported.

Original: We emphasize that compared to all the other reported methods, h -detach is extremely simple to implement and does not add any computational overhead (in fact reduces computation).
Revised: We emphasize that compared to all the other reported methods, h -detach is extremely simple to implement and does not add any computational overhead (in fact reduces computation).

Original: 3. A BLATION S TUDIES
Revised: 3. A BLATION S TUDIES

Original: In this section, we ﬁrst study the effect of removing gradient clipping in the LSTM training and compare how the training of vanilla LSTM and our method get affected.
Revised: In this section, we ﬁrst study the effect of removing gradient clipping in the LSTM training and compare how the training of vanilla LSTM and our method get affected.

Original: Getting rid of gradient clipping would be insightful because it would conﬁrm our claim that stochastically blocking gradients through the hidden states h t of the LSTM prevent the growth of gradient magnitude.
Revised: Getting rid of gradient clipping would be insightful because it would conﬁrm our claim that stochastically blocking gradients through the hidden states h t of the LSTM prevent the growth of gradient magnitude.

Original: We train both models on pixel by pixel MNIST using ADAM without any gradient clipping.
Revised: We train both models on pixel by pixel MNIST using ADAM without any gradient clipping.

Original: The validation accuracy curves are reported in ﬁgure 4 for two different learning rates.
Revised: The validation accuracy curves are reported in ﬁgure 4 for two different learning rates.

Original: We notice that removing gradient clipping causes the Vanilla LSTM training to become extremely unstable.
Revised: We notice that removing gradient clipping causes the Vanilla LSTM training to become extremely unstable.

Original: h -detach on the  other hand seems robust to removing gradient clipping for both the learning rates used.
Revised: h -detach on the [Figure] [Figure] suggests other hand seems robust to removing gradient clipping for both the learning rates used.

Original: Additional experiments with multiple seeds and learning rates can be found in ﬁgure 8 in appendix.
Revised: Additional experiments with multiple seeds and learning rates can be found in ﬁgure 8 in appendix.

Original: Second, we conduct experiments where we stochastically block gradients from ﬂowing through the cell state c t instead of the hidden state h t and observe how the LSTM behaves in such a scenario.
Revised: Second, we conduct experiments where we stochastically block gradients from ﬂowing through the cell state c t instead of the hidden state h t and observe how the LSTM behaves in such a scenario.

Original: We refer detaching the cell state as c -detach.
Revised: We refer detaching the cell state as c -detach.

Original: The goal of this experiment is to corroborate our hypothesis that the gradients through the cell state path carry information about long term dependencies.
Revised: The goal of this experiment is to corroborate our hypothesis that the gradients through the cell state path carry information about long term dependencies.

Original: Figure 5 shows the effect of c -detach (with probabilities shown) on copying task and pixel by pixel MNIST[Figure] [Figure] task.
Revised: Figure 5 shows the effect of c -detach (with probabilities shown) on copying task and pixel by pixel MNIST task.

Original: We notice in the copying task for T = 100 , learning becomes very slow (ﬁgure 5 (a)) and does not converge even after 500 epochs, whereas when not detaching the cell state, even the Vanilla
Revised: We notice in the copying task for T = 100 , learning becomes very slow (ﬁgure 5 (a)) and does not converge even after 500 epochs, whereas when not detaching the cell state, even the Vanilla

Original: LSTM converges in around 150 epochs for most cases for T=100 as shown in the experiments in section 3.1.
Revised: LSTM converges in around 150 epochs for most cases for T=100 as shown in the experiments in section 3.1.

Original: For pixel by pixel MNIST (which involves 784 time steps), there is a much larger detrimental effect on learning as we ﬁnd that none of the seeds cross 60% accuracy at the end of training (Figure 5 (b)).
Revised: For pixel by pixel MNIST (which involves 784 time steps), there is a much larger detrimental effect on learning as we ﬁnd that none of the seeds cross 60% accuracy at the end of training (Figure 5 (b)).

Original: This experiment corroborates our hypothesis that gradients through the cell state contain important components of the gradient signal as blocking them worsens the performance of these models when compared to Vanilla LSTM.
Revised: This experiment corroborates our hypothesis that gradients through the cell state contain important components of the gradient signal as blocking them worsens the performance of these models when compared to Vanilla LSTM.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Capturing long term dependencies in data using recurrent neural networks has been long known to be a hard problem (Hochreiter, 1991; Bengio et al., 1993).
Revised: Capturing long term dependencies in data using recurrent neural networks has been long known to be a hard problem (Hochreiter, 1991; Bengio et al., 1993).

Original: Therefore, there has been a considerable amount of work on addressing this issue.
Revised: Therefore, there has been a considerable amount of work on addressing this issue.

Original: Prior to the invention of the LSTM architecture (Hochreiter & Schmidhuber, 1997), another class of architectures called NARX (nonlinear autoregressive models with exogenous) recurrent networks (Lin et al., 1996) was popular for tasks involving long term dependencies.
Revised: Prior to the invention of the LSTM architecture (Hochreiter & Schmidhuber, 1997), another class of architectures called NARX (nonlinear autoregressive models with exogenous) recurrent networks (Lin et al., 1996) was popular for tasks involving long term dependencies.

Original: More recently gated recurrent unit (GRU) networks (Chung et al., 2014) was proposed that adapts some favorable properties of LSTM while requiring fewer parameters.
Revised: More recently gated recurrent unit (GRU) networks (Chung et al., 2014) was proposed that adapts some favorable properties of LSTM while requiring fewer parameters.

Revised: Other recent recurrent architecture designs that are aimed at preventing EVGP can be found in Zhang et al.

Original:  Work has also been done towards better optimization for such tasks (Martens & Sutskever, 2011; Kingma & Ba, 2014).
Revised: , Jose et al. (2017) and Li et al. Work has also been done towards better optimization for such tasks (Martens & Sutskever, 2011; Kingma & Ba, 2014).

Original: Since vanishing and exploding gradient problems (Hochreiter, 1991; Bengio et al., 1994) also hinder this goal, gradient clipping methods have been proposed to alleviate this problem (Tomas, 2012; Pascanu et al., 2013).
Revised: Since vanishing and exploding gradient problems (Hochreiter, 1991; Bengio et al., 1994) also hinder this goal, gradient clipping methods have been proposed to alleviate this problem (Tomas, 2012; Pascanu et al., 2013).

Original: Yet another line of work focuses on making use of unitary transition matrices in order to avoid loss of information as hidden states evolve over time.
Revised: Yet another line of work focuses on making use of unitary transition matrices in order to avoid loss of information as hidden states evolve over time.

Original: Le et al.
Revised: Le et al.

Original: (2015) propose to initialize recurrent networks with unitary weights while Arjovsky et al.
Revised: (2015) propose to initialize recurrent networks with unitary weights while Arjovsky et al.

Original: (2016) propose a new network parameterization that ensures that the state transition matrix remains unitary.
Revised: (2016) propose a new network parameterization that ensures that the state transition matrix remains unitary.

Revised: Extensions of the unitary RNNs have been proposed in Wisdom et al.

Revised: (2016), Mhammedi et al.

Revised: and Jing et al.

Original: Very recently, Ke et al.
Revised: Very recently, Ke et al.

Original: (2017) propose to learn an attention mechanism over past hidden states and sparsely back-propagate through paths with high attention weights in order to capture long term dependencies.
Revised: (2017) propose to learn an attention mechanism over past hidden states and sparsely back-propagate through paths with high attention weights in order to capture long term dependencies.

Original: Trinh et al.
Revised: Trinh et al.

Original: (2018) propose to add an unsupervised auxiliary loss to the original objective that is designed to encourage the network to capture such dependencies.
Revised: (2018) propose to add an unsupervised auxiliary loss to the original objective that is designed to encourage the network to capture such dependencies.

Original: We point out that our proposal in this paper is orthogonal to a number of the aforementioned papers and may even be applied in conjunction to some of them.
Revised: We point out that our proposal in this paper is orthogonal to a number of the aforementioned papers and may even be applied in conjunction to some of them.

Original: Further, our method is speciﬁc to LSTM optimization and reduces computation relative to the vanilla LSTM optimization which is in stark contrast to most of the aforementioned approaches which increase the amount of computation needed for training.
Revised: Further, our method is speciﬁc to LSTM optimization and reduces computation relative to the vanilla LSTM optimization which is in stark contrast to most of the aforementioned approaches which increase the amount of computation needed for training.

Original:  D ISCUSSION AND F UTURE W ORK
Revised:  D ISCUSSION AND F UTURE W ORK

Original: In section 3.5 we showed that LSTMs trained with h -detach are stable even without gradient clipping.
Revised: In section 3.5 we showed that LSTMs trained with h -detach are stable even without gradient clipping.

Original: We caution that while this is true, in general the gradient magnitude depends on the value of detaching probability used in h -detach.
Revised: We caution that while this is true, in general the gradient magnitude depends on the value of detaching probability used in h -detach.

Original: Hence for the general case, we do not recommend removing gradient clipping.
Revised: Hence for the general case, we do not recommend removing gradient clipping.

Original: When training stacked LSTMs, there are two ways in which h -detach can be used: 1) detaching the hidden state of all LSTMs simultaneously for a given time step t depending on the stochastic variable ξ t ) stochastically detaching the hidden state of each LSTM separately.
Revised: When training stacked LSTMs, there are two ways in which h -detach can be used: 1) detaching the hidden state of all LSTMs simultaneously for a given time step t depending on the stochastic variable ξ t ) stochastically detaching the hidden state of each LSTM separately.

Original: We leave this for future work.
Revised: We leave this for future work.

Original: h -detach stochastically blocks the gradient from ﬂowing through the hidden states of LSTM.
Revised: h -detach stochastically blocks the gradient from ﬂowing through the hidden states of LSTM.

Original: In corollary 1, we showed that in expectation, this is equivalent to dampening the gradient components from paths other than the cell state path.
Revised: In corollary 1, we showed that in expectation, this is equivalent to dampening the gradient components from paths other than the cell state path.

Original: We especially chose this strategy because of its ease of implementation in current auto-differentiation libraries.
Revised: We especially chose this strategy because of its ease of implementation in current auto-differentiation libraries.

Original: Another approach to dampen these gradient components would be to directly multiply these components with a dampening factor.
Revised: Another approach to dampen these gradient components would be to directly multiply these components with a dampening factor.

Original: This feature is currently unavailable in these libraries but may be an interesting direction to look into.
Revised: This feature is currently unavailable in these libraries but may be an interesting direction to look into.

Original: A downside of using this strategy though is that it will not reduce the amount of computation similar to h -detach
Revised: A downside of using this strategy though is that it will not reduce the amount of computation similar to h -detach

Original: (although it will not increase the amount of computation compared with vanilla LSTM either).
Revised: (although it will not increase the amount of computation compared with vanilla LSTM either).

Original: Regularizing the recurrent weight matrices to have small norm can also potentially prevent the gradient components from the cell state path from being suppressed but it may also restrict the representational power of the model.
Revised: Regularizing the recurrent weight matrices to have small norm can also potentially prevent the gradient components from the cell state path from being suppressed but it may also restrict the representational power of the model.

Original: Given the superﬁcial similarity of h -detach with dropout, we outline the difference between the two methods.
Revised: Given the superﬁcial similarity of h -detach with dropout, we outline the difference between the two methods.

Original: Dropout randomly masks the hidden units of a network during the forward pass (and can be seen as a variant of the stochastic delta rule (Hanson, 1990)).
Revised: Dropout randomly masks the hidden units of a network during the forward pass (and can be seen as a variant of the stochastic delta rule (Hanson, 1990)).

Original: Therefore, a common view of dropout is training an ensemble of networks (Warde-Farley et al., 2013).
Revised: Therefore, a common view of dropout is training an ensemble of networks (Warde-Farley et al., 2013).

Original: On the other hand, our method does not mask the hidden units during the forward pass.
Revised: On the other hand, our method does not mask the hidden units during the forward pass.

Original: It instead randomly blocks the gradient component through the h-states of the LSTM only during the backward pass and does not change the output of the network during forward pass.
Revised: It instead randomly blocks the gradient component through the h-states of the LSTM only during the backward pass and does not change the output of the network during forward pass.

Original: More speciﬁcally, our theoretical analysis shows the precise behavior of our method: the effect of h -detach is that it changes the update direction used for descent which prevents the gradients through the cell state path from being suppressed.
Revised: More speciﬁcally, our theoretical analysis shows the precise behavior of our method: the effect of h -detach is that it changes the update direction used for descent which prevents the gradients through the cell state path from being suppressed.

Original: We would also like to point out that even though we show improvements on the image captioning task, it does not ﬁt the proﬁle of a task involving long term dependencies that we focus on.
Revised: We would also like to point out that even though we show improvements on the image captioning task, it does not ﬁt the proﬁle of a task involving long term dependencies that we focus on.

Original: We believe the reason why our method leads to improvements on this task is that the gradient components from the cell state path are important for this task and our theoretical analysis shows that h -detach prevents these components from getting suppressed compared with the gradient components from the other paths.
Revised: We believe the reason why our method leads to improvements on this task is that the gradient components from the cell state path are important for this task and our theoretical analysis shows that h -detach prevents these components from getting suppressed compared with the gradient components from the other paths.

Original: On the same note, we also tried our method on language modeling tasks but did not notice any beneﬁt.
Revised: On the same note, we also tried our method on language modeling tasks but did not notice any beneﬁt.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a simple stochastic algorithm called h -detach aimed at improving LSTM performance on tasks that involve long term dependencies.
Revised: We proposed a simple stochastic algorithm called h -detach aimed at improving LSTM performance on tasks that involve long term dependencies.

Original: We provided a theoretical understanding of the method using a novel analysis of the back-propagation equations of the LSTM architecture.
Revised: We provided a theoretical understanding of the method using a novel analysis of the back-propagation equations of the LSTM architecture.

Original: We note that our method reduces the amount of computation needed during training compared to vanilla LSTM training.
Revised: We note that our method reduces the amount of computation needed during training compared to vanilla LSTM training.

Original: Finally, we empirically showed that h -detach is robust to initialization, makes the convergence of LSTM faster, and/or improves generalization compared to vanilla LSTM (and other existing methods) on various benchmark datasets. 
Revised: Finally, we empirically showed that h -detach is robust to initialization, makes the convergence of LSTM faster, and/or improves generalization compared to vanilla LSTM (and other existing methods) on various benchmark datasets. A CKNOWLEDGMENTS

Revised: We thank Stanisław Jastrz˛ebski, David Kruger and Isabela Albuquerque for helpful discussions.

Revised: DA was supported by IVADO.

Original: A PPENDIX A A DDITIONAL I NFORMATION
Revised: A PPENDIX A A DDITIONAL I NFORMATION

Original: Copying Experiment setup - We deﬁne 10 tokens, { a i } 9 i =0 .
Revised: Copying Experiment setup - We deﬁne 10 tokens, { a i } 9 i =0 .

Original: The input to the LSTM is a sequence of length T + 20 formed using one of the ten tokens at each time step.
Revised: The input to the LSTM is a sequence of length T + 20 formed using one of the ten tokens at each time step.

Original: Input for the ﬁrst 10 time steps are sampled i.i.d.
Revised: Input for the ﬁrst 10 time steps are sampled i.i.d.

Original: (uniformly) from { a i } 7 i =0 .
Revised: (uniformly) from { a i } 7 i =0 .

Original: The next T − 1 entries are set to a 8 , which constitutes a delay.
Revised: The next T − 1 entries are set to a 8 , which constitutes a delay.

Original: The next single entry is a 9 , which represents a delimiter, which should indicate to the algorithm that it is now required to reproduce the initial 10 input tokens as output.
Revised: The next single entry is a 9 , which represents a delimiter, which should indicate to the algorithm that it is now required to reproduce the initial 10 input tokens as output.

Original: The remaininginput entries are set to a 8 .
Revised: The remaininginput entries are set to a 8 .

Original: The target sequence consists of T + 10 repeated entries of a 8 , followed by the ﬁrst 10 entries of the input sequence in exactly the same order.
Revised: The target sequence consists of T + 10 repeated entries of a 8 , followed by the ﬁrst 10 entries of the input sequence in exactly the same order.

Original: [Figure]
Revised: [Figure]

Original: B D ERIVATION OF B ACK - PROPAGATION E QUATION FOR LSTM
Revised: B D ERIVATION OF B ACK - PROPAGATION E QUATION FOR LSTM

Original: Let us recall the equations from an LSTM
Revised: Let us recall the equations from an LSTM

Original: [Equation]
Revised: [Equation]

Original: Here (cid:12) denotes the element-wise product, also called the Hadamard product.
Revised: Here (cid:12) denotes the element-wise product, also called the Hadamard product.

Original: σ denotes the sigmoid activation function.
Revised: σ denotes the sigmoid activation function.

Original: W o =
Revised: W o =

Original: [ W oh ; W ox ] .
Revised: [ W oh ; W ox ] .

Original: W i
Revised: W i

Original: =
Revised: =

Original: [ W ih ; W ix ] .
Revised: [ W ih ; W ix ] .

Original: W g =
Revised: W g =

Original: [ W gh ; W gx ] .
Revised: [ W gh ; W gx ] .

Original: W f =
Revised: W f =

Original: [ W fh ; W fx ] .
Revised: [ W fh ; W fx ] .

Original: Notation
Revised: Notation

Original: [Equation]
Revised: [Equation]

Original: For any (cid:63) ∈ { f, g, o, i } , deﬁne E (cid:63) ( w ) to be a matrix of size dim( h t ) × dim( [ h t ; x t ] ).
Revised: For any (cid:63) ∈ { f, g, o, i } , deﬁne E (cid:63) ( w ) to be a matrix of size dim( h t ) × dim( [ h t ; x t ] ).

Original: We set all the elements of this matrix to 0 s if if w is not an element of W (cid:63) .
Revised: We set all the elements of this matrix to 0 s if if w is not an element of W (cid:63) .

Original: Further, if w 
Revised: Further, if w =

Original: [Equation]
Revised: [Equation]

Original: Lemma 1 Let us assume w is an entry of the matrix W f , W i , W g or W o , then
Revised: Lemma 1 Let us assume w is an entry of the matrix W f , W i , W g or W o , then

Original: [Equation]
Revised: [Equation]

Original: Proof By chain rule of total differentiation,
Revised: Proof By chain rule of total differentiation,

Original: [Equation]
Revised: [Equation]

Original: We note that,
Revised: We note that,

Original: [Equation]
Revised: [Equation]

Original: and,
Revised: and,

Original: [Equation]
Revised: [Equation]

Original: which proves the claim for d f t dw .
Revised: which proves the claim for d f t dw .

Original: The derivation for d o t dw , d i t dw , d g t dw are similar.
Revised: The derivation for d o t dw , d i t dw , d g t dw are similar.

Original: Now let us establish recursive formulas for d h t dw and d c t dw , using the above formulas
Revised: Now let us establish recursive formulas for d h t dw and d c t dw , using the above formulas

Original: Corollary 1 Considering the above notations, we have
Revised: Corollary 1 Considering the above notations, we have

Original: [Equation]
Revised: [Equation]

Original: Proof Recall that h t =
Revised: Proof Recall that h t =

Original: o t (cid:12) tanh( c t ) , and thus
Revised: o t (cid:12) tanh( c t ) , and thus

Original: [Equation]
Revised: [Equation]

Original: Using the previous Lemma as well as the above notation
Revised: Using the previous Lemma as well as the above notation

Original: [Equation]
Revised: [Equation]

Original: we get d h t
Revised: we get d h t

Original: [Equation]
Revised: [Equation]

Original: Corollary 2 Considering the above notations
Revised: Corollary 2 Considering the above notations

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Proof Recall that c t = f t (cid:12) c
Revised: Proof Recall that c t = f t (cid:12) c

Original: t − 1 + i t (cid:12) g t , and thus
Revised: t − 1 + i t (cid:12) g t , and thus

Original: [Equation]
Revised: [Equation]

Original: Using the previous Lemma as well as the above notation
Revised: Using the previous Lemma as well as the above notation

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Let us now combine corollary 1 and 2 to get a recursive expression of dh t dw in terms of d h t − 1 dw and d c t − 1 dw
Revised: Let us now combine corollary 1 and 2 to get a recursive expression of dh t dw in terms of d h t − 1 dw and d c t − 1 dw

Original: Corollary 3 Considering the above notations, we have
Revised: Corollary 3 Considering the above notations, we have

Original: [Equation]
Revised: [Equation]

Original: Proof From Corollary 1, we know that
Revised: Proof From Corollary 1, we know that

Original: [Equation]
Revised: [Equation]

Original: Using Corollary
Revised: Using Corollary

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 Fix w to be an element of the matrix W gh , W fh , W ih or W
Revised: Theorem 1 Fix w to be an element of the matrix W gh , W fh , W ih or W

Original: oh .
Revised: oh .

Original: Deﬁne,
Revised: Deﬁne,

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: In other words,
Revised: In other words,

Original: [Equation]
Revised: [Equation]

Original: where all the symbols used to deﬁne A t and B t are deﬁned in notation 1.
Revised: where all the symbols used to deﬁne A t and B t are deﬁned in notation 1.

Original: Proof By Corollary 2, we get
Revised: Proof By Corollary 2, we get

Original: [Equation]
Revised: [Equation]

Original: Similarly by Corollary 3, we get
Revised: Similarly by Corollary 3, we get

Original: [Equation]
Revised: [Equation]

Original: Thus we havez
Revised: Thus we have

Original: [Equation]
Revised: [Equation]

Original: Applying this formula recursively proves the claim.
Revised: Applying this formula recursively proves the claim.

Original: Note: Since A t has 0 n ’s in the second column of the block matrix representation, it ignores the contribution of z t coming from h t − 1 , whereas B t (having non-zero block matrices only in the second column of the block matrix representation) only takes into account the contribution coming from h t − 1 .
Revised: Note: Since A t has 0 n ’s in the second column of the block matrix representation, it ignores the contribution of z t coming from h t − 1 , whereas B t (having non-zero block matrices only in the second column of the block matrix representation) only takes into account the contribution coming from h t − 1 .

Original: Hence A t captures the contribution of the gradient coming from the cell state c t − 1 .
Revised: Hence A t captures the contribution of the gradient coming from the cell state c t − 1 .

Original: C D ERIVATION OF B ACK - PROPAGATION E QUATION FOR LSTM WITH h - DETACH
Revised: C D ERIVATION OF B ACK - PROPAGATION E QUATION FOR LSTM WITH h - DETACH

Original: Theorem 2 Let z t =
Revised: Theorem 2 Let z t =

Original: [ d c t dw T ; d h t dw T ; 1 Tn ] T and ˜ z t be the analogue of z t when applying h -detach with probability p during back-propagation.
Revised: [ d c t dw T ; d h t dw T ; 1 Tn ] T and ˜ z t be the analogue of z t when applying h -detach with probability p during back-propagation.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where ξ t , ξ t − 1 , . . .
Revised: where ξ t , ξ t − 1 , . . .

Original: , ξ 2 are i.i.d.
Revised: , ξ 2 are i.i.d.

Original: Bernoulli random variables with probability p of being 1, A t and
Revised: Bernoulli random variables with probability p of being 1, A t and

Original: B t and are same as deﬁned in theorem 1.
Revised: B t and are same as deﬁned in theorem 1.

Original: Proof Replacing ∂ ∂
Revised: Proof Replacing ∂ ∂

Original: h t − 1 by ξ t ∂ ∂ h t − 1 in lemma 1 and therefore in Corollaries 2 and 3, we get the following analogous equations
Revised: h t − 1 by ξ t ∂ ∂ h t − 1 in lemma 1 and therefore in Corollaries 2 and 3, we get the following analogous equations

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Similarly as in the proof of previous theorem, we can rewrite
Revised: Similarly as in the proof of previous theorem, we can rewrite

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Thus˜z [Equation] diag ( ˜k [Equation]
Revised: Thus

Original: 0 n 0 n

Original: Id n   · ˜z

Original: [Equation]

Original: −

Original: [Equation]
Revised: [Equation]

Original:    

Original: F t 0 n diag ( k t ) ˜F t 0 n diag ( ˜k t ) 0 n 0 n

Original: Id n   + ξ t   0 n ψ t 0 n 0 n ˜ ψ t

Original: [Equation]
Revised: [Equation]

Original: Iterating this formula gives,
Revised: Iterating this formula gives,

Original: [Equation]
Revised: [Equation]

Original: Corollary
Revised: Corollary

Original: [Equation]
Revised: [Equation]

Original: It sufﬁces to take the expectation both sides, and use independence of ξ t ’s
Revised: It sufﬁces to take the expectation both sides, and use independence of ξ t ’s

Original: [Equation]
Revised: [Equation]