Original: R EDUCING C OMPUTATION IN R ECURRENT N ETWORKS BY S ELECTIVELY U PDATING S TATE N EURONS
Revised: R EDUCING C OMPUTATION IN R ECURRENT N ETWORKS BY S ELECTIVELY U PDATING S TATE N EURONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recurrent Neural Networks (RNNs) are the state-of-the-art approach to sequential learning.
Revised: Recurrent Neural Networks (RNNs) are the state-of-the-art approach to sequential learning.

Original: However, standard RNNs use the same amount of computation at each timestep, regardless of the input data.
Revised: However, standard RNNs use the same amount of computation at each timestep, regardless of the input data.

Original: As a result, even for high-dimensional hidden states, all dimensions are updated at each timestep no matter the choice of recurrent memory cell.
Revised: As a result, even for high-dimensional hidden states, all dimensions are updated at each timestep no matter the choice of recurrent memory cell.

Original: Reducing this rigid assumption could allow for models with large hidden states to perform inference more quickly.
Revised: Reducing this rigid assumption could allow for models with large hidden states to perform inference more quickly.

Original: Intuitively, not all hidden state dimensions need to be recomputed from scratch at each timestep.
Revised: Intuitively, not all hidden state dimensions need to be recomputed from scratch at each timestep.

Original: Thus, recent methods have begun studying this problem by imposing mainly a priori-determined patterns for updating the states at each step.
Revised: Thus, recent methods have begun studying this problem by imposing mainly a priori-determined patterns for updating the states at each step.

Original: In contrast, we now design a fullylearned approach, SA-RNN, that augments any RNN by predicting discrete update patterns at the ﬁne granularity of independent hidden state dimensions.
Revised: In contrast, we now design a fullylearned approach, SA-RNN, that augments any RNN by predicting discrete update patterns at the ﬁne granularity of independent hidden state dimensions.

Original: This is achieved through the parameterization of a distribution of update-likelihoods driven by the input data.
Revised: This is achieved through the parameterization of a distribution of update-likelihoods driven by the input data.

Original: Notably, through this approach we impose no assumptions on the structure of the update pattern.
Revised: Notably, through this approach we impose no assumptions on the structure of the update pattern.

Original: Better yet, our method adapts the update patterns online, allowing different dimensions to be updated conditional to the input.
Revised: Better yet, our method adapts the update patterns online, allowing different dimensions to be updated conditional to the input.

Original: To learn which dimensions to update, the model solves a multi-objective optimization problem, maximizing task performance while minimizing the number of updates based on a uniﬁed control.
Revised: To learn which dimensions to update, the model solves a multi-objective optimization problem, maximizing task performance while minimizing the number of updates based on a uniﬁed control.

Original: Using publicly-available datasets we demonstrate that our method consistently achieves higher accuracy with fewer updates compared to state-of-the-art alternatives.
Revised: Using publicly-available datasets we demonstrate that our method consistently achieves higher accuracy with fewer updates compared to state-of-the-art alternatives.

Original: Additionally, our method can be directly applied to a wide variety of models containing RNN architectures.
Revised: Additionally, our method can be directly applied to a wide variety of models containing RNN architectures.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recurrent Neural Networks (RNN) are the state-of-the-art approach to many sequential learning problems including speech recognition (Graves et al., 2013), machine translation (Bahdanau et al., 2015), and sequence generation (Graves, 2013;
Revised: Recurrent Neural Networks (RNN) are the state-of-the-art approach to many sequential learning problems including speech recognition (Graves et al., 2013), machine translation (Bahdanau et al., 2015), and sequence generation (Graves, 2013;

Original: Xu et al., 2015).
Revised: Xu et al., 2015).

Original: However, RNNs typically rely on computationally-taxing updates to their entire hidden state at each timestep, a cost that grows with hidden state size.
Revised: However, RNNs typically rely on computationally-taxing updates to their entire hidden state at each timestep, a cost that grows with hidden state size.

Original: As demonstrated by the success of gating mechanisms such as the GRU (Cho et al., 2014) and LSTM (Hochreiter & Schmidhuber, 1997), all dimensions rarely need to be re-computed from scratch at each timestep.
Revised: As demonstrated by the success of gating mechanisms such as the GRU (Cho et al., 2014) and LSTM (Hochreiter & Schmidhuber, 1997), all dimensions rarely need to be re-computed from scratch at each timestep.

Original: By discretely selecting which dimensions to update at each timestep via a learned update pattern , RNNs with a large hidden state can be trained with lower computational requirements (Bengio et al., 2013), inference in long RNNs can be expedited (Campos et al., 2018), and hidden representations may be made more robust to misleading inputs such as outliers or noise.
Revised: By discretely selecting which dimensions to update at each timestep via a learned update pattern , RNNs with a large hidden state can be trained with lower computational requirements (Bengio et al., 2013), inference in long RNNs can be expedited (Campos et al., 2018), and hidden representations may be made more robust to misleading inputs such as outliers or noise.

Original: Selective neuron activation in RNNs has recently gained attention in the literature (Koutnik et al., 2014; Neil et al., 2016;
Revised: Selective neuron activation in RNNs has recently gained attention in the literature (Koutnik et al., 2014; Neil et al., 2016;

Original: Shen et al., 2019; Jernite et al., 2017; Campos et al., 2018).
Revised: Shen et al., 2019; Jernite et al., 2017; Campos et al., 2018).

Original: The most popular methods hand-craft speciﬁc update patterns , dictating which dimensions of the hidden state will update at which timesteps according to prior knowledge of a task (Koutnik et al., 2014; Neil et al., 2016).
Revised: The most popular methods hand-craft speciﬁc update patterns , dictating which dimensions of the hidden state will update at which timesteps according to prior knowledge of a task (Koutnik et al., 2014; Neil et al., 2016).

Original: This imposes undue challenges in implementation, limits extensibility, and ignores the data-driven curation of information-ﬂow through the RNN, a signature property of recurrent memory cells (Hochreiter & Schmidhuber, 1997; Cho et al., 2014).
Revised: This imposes undue challenges in implementation, limits extensibility, and ignores the data-driven curation of information-ﬂow through the RNN, a signature property of recurrent memory cells (Hochreiter & Schmidhuber, 1997; Cho et al., 2014).

Original: More recent methods learn to react to input data but impose strict relationships between the update patterns across both hidden dimensions and time (Shen et al., 2019; Jernite et al., 2017; Campos et al., 2018).
Revised: More recent methods learn to react to input data but impose strict relationships between the update patterns across both hidden dimensions and time (Shen et al., 2019; Jernite et al., 2017; Campos et al., 2018).

Original: While applicable to tasks with clear hierarchical components, such as modeling character-level text (Chung et al., 2017), these assumptions limit the expressiveness of the learned update patterns.
Revised: While applicable to tasks with clear hierarchical components, such as modeling character-level text (Chung et al., 2017), these assumptions limit the expressiveness of the learned update patterns.

Original: Speciﬁcally, we study the problem of generating a binary update-pattern for the hidden states learned by an RNN.
Revised: Speciﬁcally, we study the problem of generating a binary update-pattern for the hidden states learned by an RNN.

Original: The learned update-pattern deﬁnes which dimensions of the hidden state to update at each timestep, similar to the motivation for Residual Networks (He et al., 2016; Wang & Tian, 2016) and Highway Networks (Srivastava et al., 2015; Zilly et al., 2017).
Revised: The learned update-pattern deﬁnes which dimensions of the hidden state to update at each timestep, similar to the motivation for Residual Networks (He et al., 2016; Wang & Tian, 2016) and Highway Networks (Srivastava et al., 2015; Zilly et al., 2017).

Original: Ideally, only a small subset of the hidden state’s dimensions needs to be updated at each timestep, especially with high-dimensional hidden states.
Revised: Ideally, only a small subset of the hidden state’s dimensions needs to be updated at each timestep, especially with high-dimensional hidden states.

Original: In this way, representations can be learned while both solving a sequential learning task and minimizing the number of updates.
Revised: In this way, representations can be learned while both solving a sequential learning task and minimizing the number of updates.

Original: This results in a reduction of the compute time.
Revised: This results in a reduction of the compute time.

Original: A solution to this multi-objective optimization problem should have a comparable accuracy to a traditional constantly-updating RNN but save the majority of computation steps along the way, ultimately accelerating inference and training (Neil et al., 2016).
Revised: A solution to this multi-objective optimization problem should have a comparable accuracy to a traditional constantly-updating RNN but save the majority of computation steps along the way, ultimately accelerating inference and training (Neil et al., 2016).

Original: Despite the potential for reducing computation required by RNNs, learning said update patterns is a challenging problem.
Revised: Despite the potential for reducing computation required by RNNs, learning said update patterns is a challenging problem.

Original: First, binary-output neurons making discrete decisions (whether or not to update a hidden state dimension, for example) in the interior of a neural network is a classic challenge to gradient-based learning.
Revised: First, binary-output neurons making discrete decisions (whether or not to update a hidden state dimension, for example) in the interior of a neural network is a classic challenge to gradient-based learning.

Original: This is because such decisions are non-differentiable by nature and therefore backpropagation cannot be directly used to update the weights.
Revised: This is because such decisions are non-differentiable by nature and therefore backpropagation cannot be directly used to update the weights.

Original: Second, the quality of a learned update pattern is unsupervised and thus the only feedback is task-speciﬁc.
Revised: Second, the quality of a learned update pattern is unsupervised and thus the only feedback is task-speciﬁc.

Original: This discourages a priori assumptions of the update patterns.
Revised: This discourages a priori assumptions of the update patterns.

Original: To address the aforementioned challenges, we propose the selective activation RNN, or SA-RNN, which parameterizes a distribution of update-likelihoods, one per hidden state dimension, from which update-decisions can be made at each timestep.
Revised: To address the aforementioned challenges, we propose the selective activation RNN, or SA-RNN, which parameterizes a distribution of update-likelihoods, one per hidden state dimension, from which update-decisions can be made at each timestep.

Original: We augment an RNN with an update coordinator that adaptively controls which coordinate directions to update in the hidden state on the ﬂy.
Revised: We augment an RNN with an update coordinator that adaptively controls which coordinate directions to update in the hidden state on the ﬂy.

Original: The coordinator is modeled as a lightweight neural network that observes incoming data at each timestep and makes a discrete decision of whether or not enough information is stored in each individual hidden dimension to warrant an update.
Revised: The coordinator is modeled as a lightweight neural network that observes incoming data at each timestep and makes a discrete decision of whether or not enough information is stored in each individual hidden dimension to warrant an update.

Original: Subsequently, each hidden dimension is either computed by the RNN or copied from the previous timestep.
Revised: Subsequently, each hidden dimension is either computed by the RNN or copied from the previous timestep.

Original: The coordinator ’s architecture is kept as simple as possible so the complexity of the RNN can scale without simply outsourcing computation to another network, similar to the controller in Ha & Schmidhuber (2018).
Revised: The coordinator ’s architecture is kept as simple as possible so the complexity of the RNN can scale without simply outsourcing computation to another network, similar to the controller in Ha & Schmidhuber (2018).

Original: Most notably, in contrast to other recent approaches (Koutnik et al., 2014; Jernite et al., 2017; Neil et al., 2016; Campos et al., 2018;
Revised: Most notably, in contrast to other recent approaches (Koutnik et al., 2014; Jernite et al., 2017; Neil et al., 2016; Campos et al., 2018;

Original: Shen et al., 2019; Liu et al., 2018)
Revised: Shen et al., 2019; Liu et al., 2018)

Original: we impose no assumptions of which individual hidden dimensions should update (or not update) together.
Revised: we impose no assumptions of which individual hidden dimensions should update (or not update) together.

Original: Instead, we show that using an entirely-learned approach still results in complex task-speciﬁc update patterns.
Revised: Instead, we show that using an entirely-learned approach still results in complex task-speciﬁc update patterns.

Original: On three publicly-available datasets, we show that our low-bias approach achieves higher accuracy with far fewer updates than recent state-of-the-art methods (Koutnik et al., 2014; Jernite et al., 2017; Neil et al., 2016; Campos et al., 2018).
Revised: On three publicly-available datasets, we show that our low-bias approach achieves higher accuracy with far fewer updates than recent state-of-the-art methods (Koutnik et al., 2014; Jernite et al., 2017; Neil et al., 2016; Campos et al., 2018).

Original: These results indicate that predicting RNN update-patterns solely with respect to a task is not only feasible and low-bias, but is also favorable in a variety of settings.
Revised: These results indicate that predicting RNN update-patterns solely with respect to a task is not only feasible and low-bias, but is also favorable in a variety of settings.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Recurrent neuron update patterns have gained much interest in recent literature (Koutnik et al., 2014;
Revised: Recurrent neuron update patterns have gained much interest in recent literature (Koutnik et al., 2014;

Original: Jernite et al., 2017; Neil et al., 2016; Chung et al., 2017;
Revised: Jernite et al., 2017; Neil et al., 2016; Chung et al., 2017;

Original: Shen et al., 2019; Liu et al., 2018).
Revised: Shen et al., 2019; Liu et al., 2018).

Original: All of these methods boast fewer updates to the hidden states than standard RNN architectures.
Revised: All of these methods boast fewer updates to the hidden states than standard RNN architectures.

Original: However, there are several limitations of these methods, two of which are summarized as follows.
Revised: However, there are several limitations of these methods, two of which are summarized as follows.

Original: First, the most popular methods rely on extensively-handcrafted update patterns consisting of periodic neuron activations (Koutnik et al., 2014; Neil et al., 2016; Liu et al., 2018).
Revised: First, the most popular methods rely on extensively-handcrafted update patterns consisting of periodic neuron activations (Koutnik et al., 2014; Neil et al., 2016; Liu et al., 2018).

Original: This requires either prior knowledge of sampling frequencies or seasonal patterns present in the data, reducing the potential extension to many sequential learning problems.
Revised: This requires either prior knowledge of sampling frequencies or seasonal patterns present in the data, reducing the potential extension to many sequential learning problems.

Original: Additionally, these input-agnostic periodic updates are ﬁxed prior to learning.
Revised: Additionally, these input-agnostic periodic updates are ﬁxed prior to learning.

Original: The choice of update periods heavily impacts the performance of the model, and sequences with irregular information ﬂow cannot be modeled without massive state representations.
Revised: The choice of update periods heavily impacts the performance of the model, and sequences with irregular information ﬂow cannot be modeled without massive state representations.

Original: Second, the most recent works allow for data-reactive update patterns (Jernite et al., 2017; Shen et al., 2019; Campos et al., 2018) but assume temporal hierarchies in the input sequences and study settings where this effect is exceedingly obvious (for example, character-level sentence modeling (Chung et al., 2017)).
Revised: Second, the most recent works allow for data-reactive update patterns (Jernite et al., 2017; Shen et al., 2019; Campos et al., 2018) but assume temporal hierarchies in the input sequences and study settings where this effect is exceedingly obvious (for example, character-level sentence modeling (Chung et al., 2017)).

Original: In many real-world settings, temporal hierarchies are often subtle and forcing this assumption into the architectural design may limit its applications.
Revised: In many real-world settings, temporal hierarchies are often subtle and forcing this assumption into the architectural design may limit its applications.

Original: [Figure]
Revised: [Figure]

Original: Additionally, our approach is related to conditional computation , which predicts subsets of neural networks to activate depending on input data (Bengio et al., 2015; Shazeer et al., 2017; Cheng et al., 2017).
Revised: Additionally, our approach is related to conditional computation , which predicts subsets of neural networks to activate depending on input data (Bengio et al., 2015; Shazeer et al., 2017; Cheng et al., 2017).

Original: In many cases, when a particular concept can be represented using only the sub-network of a large neural network, computation can be preserved by learning the structure of said sub-network (Schmidhuber, 2012) and activating it accordingly.
Revised: In many cases, when a particular concept can be represented using only the sub-network of a large neural network, computation can be preserved by learning the structure of said sub-network (Schmidhuber, 2012) and activating it accordingly.

Original:  S ELECTIVE N EURON A CTIVATION FOR RNN S
Revised:  S ELECTIVE N EURON A CTIVATION FOR RNN S

Original: We introduce the S
Revised: We introduce the S

Original: elective-
Revised: elective-

Original: A ctivation RNN , or SA-RNN, a broadly-applicable augmentation to
Revised: A ctivation RNN , or SA-RNN, a broadly-applicable augmentation to

Original: RNNs which minimizes the computation required for RNNs by facilitating unimpeded informationﬂow across timesteps for individual dimensions of the hidden state.
Revised: RNNs which minimizes the computation required for RNNs by facilitating unimpeded informationﬂow across timesteps for individual dimensions of the hidden state.

Original: At its core, SA-RNN learns a data-driven strategy for discretely reading and writing information to the latent state space through the learned parameterization of an update-likelihood distribution.
Revised: At its core, SA-RNN learns a data-driven strategy for discretely reading and writing information to the latent state space through the learned parameterization of an update-likelihood distribution.

Original: Despite leaving hidden dimension update patterns independent from one another, complex strategies still arise naturally depending on the sequential learning task at hand.
Revised: Despite leaving hidden dimension update patterns independent from one another, complex strategies still arise naturally depending on the sequential learning task at hand.

Original: In this section, we describe the training process of SA-RNN with D -dimensional hidden states on sequences of length T for input data x with V variables.
Revised: In this section, we describe the training process of SA-RNN with D -dimensional hidden states on sequences of length T for input data x with V variables.

Original: We omit biases from afﬁne transformation equations and use notation for one training instance for ease of readability.
Revised: We omit biases from afﬁne transformation equations and use notation for one training instance for ease of readability.

Original: An overview of the forward pass through SA-RNN is shown in Figure 1.
Revised: An overview of the forward pass through SA-RNN is shown in Figure 1.

Original: 3. C OMPUTING H IDDEN S TATES
Revised: 3. C OMPUTING H IDDEN S TATES

Original: RNNs compute a sequence of hidden states one timestep at a time (Elman, 1990), each computed by a parametric recurrence function R ( · ) : h t =
Revised: RNNs compute a sequence of hidden states one timestep at a time (Elman, 1990), each computed by a parametric recurrence function R ( · ) : h t =

Original: R ( h t − 1 , x t | θ r ) .
Revised: R ( h t − 1 , x t | θ r ) .

Original: The result is a sequence of vector representations H
Revised: The result is a sequence of vector representations H

Original: = { h 1 , . .
Revised: = { h 1 , . .

Original: . , h T } where each h t ∈ R D represents temporal dynamics of the time series up to timestep t with respect to a task, preserving not only temporal dependencies but also the ordering of the inputs.
Revised: . , h T } where each h t ∈ R D represents temporal dynamics of the time series up to timestep t with respect to a task, preserving not only temporal dependencies but also the ordering of the inputs.

Original: A popular and powerful augmentation to the RNN, as it was originally proposed, is the Gated
Revised: A popular and powerful augmentation to the RNN, as it was originally proposed, is the Gated

Original: Recurrent Unit (GRU) (Cho et al., 2014), which adds a series of gates between h t − 1 and h t to alleviate the vanishing gradient problem (Bengio et al., 1994):
Revised: Recurrent Unit (GRU) (Cho et al., 2014), which adds a series of gates between h t − 1 and h t to alleviate the vanishing gradient problem (Bengio et al., 1994):

Original: [Equation]
Revised: [Equation]

Original: where W s and U s are matrices of learnable parameters of shape D × D and D × V respectively, x t ∈ R V is the input data at timestep t , (cid:12) represents the element-wise multiplication, σ represents the sigmoid function, and φ represents a non-linearity (traditionally the hyperbolic tangent function).
Revised: where W s and U s are matrices of learnable parameters of shape D × D and D × V respectively, x t ∈ R V is the input data at timestep t , (cid:12) represents the element-wise multiplication, σ represents the sigmoid function, and φ represents a non-linearity (traditionally the hyperbolic tangent function).

Original: Its design is motivated heavily by the LSTM (Hochreiter & Schmidhuber, 1997).
Revised: Its design is motivated heavily by the LSTM (Hochreiter & Schmidhuber, 1997).

Original: The GRU performs soft read/write operations, recomputing the entire vector h t at each timestep since gate z ∈ [0 , 1] D , the space of vectors with values inclusively between 0 and 1.
Revised: The GRU performs soft read/write operations, recomputing the entire vector h t at each timestep since gate z ∈ [0 , 1] D , the space of vectors with values inclusively between 0 and 1.

Original: Instead, we propose that all dimensions do not need to be updated at each timestep, as the position of the hidden state in many dimensions may often encode enough of the modeled input.
Revised: Instead, we propose that all dimensions do not need to be updated at each timestep, as the position of the hidden state in many dimensions may often encode enough of the modeled input.

Original: Note that the output of the recurrence function is referred to as ˜ h t .
Revised: Note that the output of the recurrence function is referred to as ˜ h t .

Original: In the next section, we describe how to compute h t , the ﬁnal hidden state for timestep t which is subsequently used for computing h t +1 or the task.
Revised: In the next section, we describe how to compute h t , the ﬁnal hidden state for timestep t which is subsequently used for computing h t +1 or the task.

Original: 3. S ELECTIVE N EURON A CTIVATION
Revised: 3. S ELECTIVE N EURON A CTIVATION

Original: To reduce computation required to generate state representations, we assume updating representations to be a sequence of binary decisions – at each timestep a neuron will either be updated or not.
Revised: To reduce computation required to generate state representations, we assume updating representations to be a sequence of binary decisions – at each timestep a neuron will either be updated or not.

Original: Thus, we propose a learned update coordinator , which generates a binary mask for each hidden dimension, forecasting which dimensions need to be updated at the next timestep.
Revised: Thus, we propose a learned update coordinator , which generates a binary mask for each hidden dimension, forecasting which dimensions need to be updated at the next timestep.

Original: First, an update-likelihood ˜ u t is computed for each neuron, informed by both the data observed at the current timestep and the previous update-likelihoods: ˜ u t = σ ( W u h t − 1 + W i x t ) where W u ∈ R D
Revised: First, an update-likelihood ˜ u t is computed for each neuron, informed by both the data observed at the current timestep and the previous update-likelihoods: ˜ u t = σ ( W u h t − 1 + W i x t ) where W u ∈ R D

Original: × D is a diagonal matrix of trainable parameters which dictate the linear relationship between h t − 1 and ˜ u t .
Revised: × D is a diagonal matrix of trainable parameters which dictate the linear relationship between h t − 1 and ˜ u t .

Original: W u is kept diagonal to maintain relationships between update-decisions of the same dimension while avoiding the extensive computation of a fully-connected layer, similar to the hidden state decay in Che et al.
Revised: W u is kept diagonal to maintain relationships between update-decisions of the same dimension while avoiding the extensive computation of a fully-connected layer, similar to the hidden state decay in Che et al.

Original: W i ∈
Revised: W i ∈

Original: R D × V encodes the inﬂuence of the input data on the current update-likelihood and σ ( · ) represents the  sigmoid function, bounding ˜ u  .
Revised: R D × V encodes the inﬂuence of the input data on the current update-likelihood and σ ( · ) represents the hard sigmoid function, bounding ˜ u according to a slope α .

Original: Thus ˜ u t ∈ [0 , 1] D , with one update-likelihood per dimension of the hidden state.
Revised: Thus ˜ u t ∈ [0 , 1] D , with one update-likelihood per dimension of the hidden state.

Original: To discretize ˜ u t ,thus allowing information to ﬂow unimpeded,an element-wise binarizationfunction is applied: 
Revised: To discretize ˜ u t , allowing information to ﬂow unimpeded, element-wise binarization is applied: u t = binarize (

Original: [Equation]
Revised: [Equation]

Original:  , where
Revised: t ) , where

Original: [Equation]
Revised: [Equation]

Original: We apply this ﬁnal discrete update decision as a binary gating mechanism since u t ∈ { 0 , 1 } D : 
Revised: We apply this ﬁnal discrete update decision as a binary gating mechanism since u t ∈ { 0 , 1 } D : [Equation]

Original: [Equation]
Revised: [Equation]

Original: As written, this equation requires the pre-computation of ˜ h t .
Revised: As written, this equation requires the pre-computation of ˜ h t .

Original: However, through masking the computation can be directed at only the needed updates upon calculation of u t .
Revised: However, through masking the computation can be directed at only the needed updates upon calculation of u t .

Original: Thus when ˜ u nt , the update decision for the n -th dimension in h , is 1 , h nt is updated according to the new information present in ˜ h
Revised: Thus when ˜ u nt , the update decision for the n -th dimension in h , is 1 , h nt is updated according to the new information present in ˜ h

Original: nt .
Revised: nt .

Original: We note that this update-decision strategy does not impose the inter-neuron assumptions of
Revised: We note that this update-decision strategy does not impose the inter-neuron assumptions of

Original: Jernite et al.
Revised: Jernite et al.

Original: ; Shen et al.
Revised: ; Shen et al.

Original: ; Koutnik et al.
Revised: ; Koutnik et al.

Original: (2014) while still allowing such strategies to be learned if they are found to be optimal by the model since decisions are made with respect to previous decisions, similar to Campos et al.
Revised: (2014) while still allowing such strategies to be learned if they are found to be optimal by the model since decisions are made with respect to previous decisions, similar to Campos et al.

Original: We hypothesize that updating neurons together may generally be beneﬁcial since complex temporal dependencies often require representations evolving in blocks of multiple neurons, as discussed in Koutnik et al.
Revised: We hypothesize that updating neurons together may generally be beneﬁcial since complex temporal dependencies often require representations evolving in blocks of multiple neurons, as discussed in Koutnik et al.

Original: Since binary-output neurons are inherently non-differentiable, barring the direct use of backpropagation, we approximate the gradient of the binarization function using the straight-through gradient estimator (Bengio et al., 2013) trained with slope-annealing (Chung et al., 2017):
Revised: Since binary-output neurons are inherently non-differentiable, barring the direct use of backpropagation, we approximate the gradient of the binarization function using the straight-through gradient estimator (Bengio et al., 2013) trained with slope-annealing (Chung et al., 2017):

Original: [Equation]
Revised: [Equation]

Revised: binarize (

Original:  By estimating the gradient in this way we avoid additional loss terms and end up with empiricallyreasonable approximations in comparison to other high-variance methods, such as REINFORCE (Williams, 1992; Chung et al., 2017; Campos et al., 2018).
Revised: [Equation] [Equation] [Equation] By estimating the gradient in this way we avoid additional loss terms and end up with empiricallyreasonable approximations in comparison to other high-variance methods, such as REINFORCE (Williams, 1992; Chung et al., 2017; Campos et al., 2018).

Original: After computing the sequence of state representations H , they are projected into the output space depending on the task at hand.
Revised: After computing the sequence of state representations H , they are projected into the output space depending on the task at hand.

Original: 3. T RAINING
Revised: 3. T RAINING

Original: All weights of SA-RNN are updated together using back-propagation to minimize one loss function.
Revised: All weights of SA-RNN are updated together using back-propagation to minimize one loss function.

Original: For readability we gather all weights into one parameter matrix θ .
Revised: For readability we gather all weights into one parameter matrix θ .

Original: Our loss function J ( θ ) consists of two parts: a task-driven loss (denoted as L task ) and an update-budget.
Revised: Our loss function J ( θ ) consists of two parts: a task-driven loss (denoted as L task ) and an update-budget.

Original: The task-driven component may be be cross entropy for classiﬁcation or possibly mean squared error for regression.
Revised: The task-driven component may be be cross entropy for classiﬁcation or possibly mean squared error for regression.

Original: The updatebudget encourages sparser updates to the hidden dimensions, as shown in Equation 9 where N is the number of training examples, ˆ y is the model’s prediction, and y is the label.
Revised: The updatebudget encourages sparser updates to the hidden dimensions, as shown in Equation 9 where N is the number of training examples, ˆ y is the model’s prediction, and y is the label.

Original: [Equation]
Revised: [Equation]

Original: λ tunes the emphasis on the minimization of ˜ u , the update probability , and in practice it is reasonable to set λ = 0 , encouraging the model to make update decisions solely with respect to the learning task .
Revised: λ tunes the emphasis on the minimization of ˜ u , the update probability , and in practice it is reasonable to set λ = 0 , encouraging the model to make update decisions solely with respect to the learning task .

Original: As λ is increased and update likelihoods decrease, the hidden dimension is update fewer times until eventually
Revised: As λ is increased and update likelihoods decrease, the hidden dimension is update fewer times until eventually

Original: h 

Revised: h

Original: = h T where the hidden state is not updated at all.
Revised: = h T where the hidden state is not updated at all.

Original:  E XPERIMENTS 4. D ATASETS
Revised:  E XPERIMENTS 4. D ATASETS

Original: We conduct our evaluation using three classiﬁcation tasks on the following datasets, each being publicly-available.
Revised: We conduct our evaluation using three classiﬁcation tasks on the following datasets, each being publicly-available.

Original: Seizures 1 (Andrzejak et al., 2001):
Revised: Seizures 1 (Andrzejak et al., 2001):

Original: From 11,800 178-timestep time series, the task is to detect which EEGs contain evidence of epileptic seizure activity.
Revised: From 11,800 178-timestep time series, the task is to detect which EEGs contain evidence of epileptic seizure activity.

Original: Since there are only 2,300 cases of such activity, we down-sample an equal number from the negative class, resulting in a balanced dataset with 4,600 time series.
Revised: Since there are only 2,300 cases of such activity, we down-sample an equal number from the negative class, resulting in a balanced dataset with 4,600 time series.

Original: Finally, we center the time series around zero and compute the mean value of every 10-timestep chunk, summarizing each series into 17 ﬁnal timesteps.
Revised: Finally, we center the time series around zero and compute the mean value of every 10-timestep chunk, summarizing each series into 17 ﬁnal timesteps.

Original: TwitterBuzz 2 (Kawala et al., 2013):
Revised: TwitterBuzz 2 (Kawala et al., 2013):

Original: To predict buzz events on Twitter, we work with 77dimensional time series with labels indicating whether or not a spike in tweets on a particular topic is observed.
Revised: To predict buzz events on Twitter, we work with 77dimensional time series with labels indicating whether or not a spike in tweets on a particular topic is observed.

Original: Starting with over 140,000 timesteps, we compute the mean of every ﬁve steps, center the time series around zero, and break the resulting 28,000 timesteps into 2,800 lengthsequences.
Revised: Starting with over 140,000 timesteps, we compute the mean of every ﬁve steps, center the time series around zero, and break the resulting 28,000 timesteps into 2,800 lengthsequences.

Original: We then extract the 776 time series containing any buzz events and balance the dataset by randomly selecting an equal number of no-buzz time series, resulting in 1552 labeled time series.
Revised: We then extract the 776 time series containing any buzz events and balance the dataset by randomly selecting an equal number of no-buzz time series, resulting in 1552 labeled time series.

Original: Yahoo 3 : We re-frame this outlier detection dataset as a classiﬁcation problem: whether or not a sequence contains an anomaly.
Revised: Yahoo 3 : We re-frame this outlier detection dataset as a classiﬁcation problem: whether or not a sequence contains an anomaly.

Original: We begin by chunking the time series into subsequences of length 25.
Revised: We begin by chunking the time series into subsequences of length 25.

Original: Then, we create a balanced dataset by selecting all subsequences with anomalies present along with an equal number of randomly-selected subsequences with no anomalies to serve as our negative class.
Revised: Then, we create a balanced dataset by selecting all subsequences with anomalies present along with an equal number of randomly-selected subsequences with no anomalies to serve as our negative class.

Original: Thus we end up with a dataset containing 418 length-25 time series.
Revised: Thus we end up with a dataset containing 418 length-25 time series.

Original: 4. S ETTINGS Baselines.
Revised: 4. S ETTINGS Baselines.

Original: To evaluate the performance of SA-RNN, we compare with ﬁve recent state-of-the-art related methods:
Revised: To evaluate the performance of SA-RNN, we compare with ﬁve recent state-of-the-art related methods:

Original: • Random Updates : This method is effectively the random version of our proposed method.
Revised: • Random Skips : This method is effectively the random version of our proposed method

Original: At each step, random hidden dimensions are updated.
Revised: At each step, random hidden dimensions are updated.

Original: This is similar to Zoneout (Krueger et al., 2017) however we maintain random updating during testing.
Revised: This is similar to Zoneout (Krueger et al., 2017) however we maintain random updating during testing.

Original: • Clockwork RNN (Koutnik et al., 2014): Hidden dimensions are updated in groups at predetermined “clock” rates.
Revised: • Clockwork RNN (Koutnik et al., 2014): Hidden dimensions are updated in groups at predetermined “clock” rates.

Original: For example, the ﬁrst ﬁve dimensions in the hidden state may update every step while the second ﬁve neurons update every 3 steps.
Revised: For example, the ﬁrst ﬁve dimensions in the hidden state may update every step while the second ﬁve neurons update every 3 steps.

Original: • Phased LSTM (Neil et al., 2016): Hidden dimensions are updated independently at sampled “clock” rates where the user deﬁnes the distribution from which to sample.
Revised: • Phased LSTM (Neil et al., 2016): Hidden dimensions are updated independently at sampled “clock” rates where the user deﬁnes the distribution from which to sample.

Original: Each hidden dimension has its own update pattern.
Revised: Each hidden dimension has its own update pattern.

Original: • SkipRNN (Campos et al., 2018):
Revised: • VC-GRU (Jernite et al., 2017):

Original: Hidden dimensions update in lock-step with one another so the entire hidden state is either modiﬁed or left unchanged at each step. Additionally, the update likelihoods increase monotonically, regardless of input data.
Revised: A value p ∈ [0 , 1] is predicted at each step indicating the proportion of the representation to update. Then, the ﬁrst p ∗ D dimensions are updated, imposing a hierarchical structure to the update patterns. php?

Original: =s&did=
Revised: =s&did=

Original: [Figure]
Revised: [Figure]

Original: • VC-GRU (Jernite et al., 2017):
Revised: • SkipRNN (Campos et al., 2018):

Original: A value p ∈ [0 , is predicted at each step indicating the proportion of the representation to update.
Revised: Hidden dimensions update in lock-step with one so the entire hidden state is either modiﬁed or left unchanged at each step.

Original: Then, the ﬁrst p ∗ D dimensions are updated, imposing a hierarchical structure to the update patterns.
Revised: Additionally, the update likelihoods increase monotonically, regardless of input data

Original: Implementation Details.
Revised: Implementation Details.

Original: For all experiments, we use an 80% training, 10% validation, and 10% testing dataset split.
Revised: For all experiments, we use an 80% training, 10% validation, and 10% testing dataset split.

Original: The training data are used to tune the parameters of the models; the validation data are used to validate hyperparameter selection; the testing data are used to report ﬁnal performances.
Revised: The training data are used to tune the parameters of the models; the validation data are used to validate hyperparameter selection; the testing data are used to report ﬁnal performances.

Original: We randomly repeat this process ten times to compute conﬁdence intervals for performance metrics.
Revised: We randomly repeat this process ten times to compute conﬁdence intervals for performance metrics.

Original: Across all models we ﬁx the size of state representations to be ﬁfty neurons.
Revised: Across all models we ﬁx the size of state representations to be ﬁfty neurons.

Original: Keeping this number ﬁxed allows us to purely compare performance of the update-patterns observed in alternative algorithms.
Revised: Keeping this number ﬁxed allows us to purely compare performance of the update-patterns observed in alternative algorithms.

Original: For λ selection in SkipRNN and our proposed method, SA-RNN, we search in a log-space ranging from 0 . 

Revised: For λ selection in SkipRNN and our proposed method, SA-RNN, we search in a log-space ranging from 0 .

Original: to 0 .
Revised: to 0 .

Original: 1 in 11 steps.
Revised: 1 in 11 steps.

Original: For all methods, we choose the same learning rate of 1 e − 03 after a log-space search ranging from 1 e − 05 to 1 e − 01 , likely due to similarities behind the core of the sequential learning (RNNs with equal state representation sizes).
Revised: For all methods, we choose the same learning rate of 1 e − 03 after a log-space search ranging from 1 e − 05 to 1 e − 01 , likely due to similarities behind the core of the sequential learning (RNNs with equal state representation sizes).

Original: We optimize all models using Adam (Kingma & Ba, 2014).
Revised: We optimize all models using Adam (Kingma & Ba, 2014).

Original: While we describe our method in the case of the GRU, it may also be directly applied to other gating mechanisms such as the LSTM.
Revised: While we describe our method in the case of the GRU, it may also be directly applied to other gating mechanisms such as the LSTM.

Revised: We implement slope annealing following the setting described in

Revised: Chung et al.

Original: The code for our method is available at https://hiddenforreview.com .
Revised: (2017).The code for our method is available at https://hiddenforreview.com .

Original: 4. E XPERIMENTAL R ESULTS
Revised: 4. E XPERIMENTAL R ESULTS

Original: Contrasting update patterns of alternative algorithms.
Revised: Contrasting update patterns of alternative algorithms.

Original: First, we inspect the update-patterns generated by each baseline algorithm, as shown in Figure 4.
Revised: First, we inspect the update-patterns generated by each baseline algorithm, as shown in Figure 4.

Original: Both PhasedLSTM and VC-GRU use partial updates, and do not make fully-binary update decisions.
Revised: Both PhasedLSTM and VC-GRU use partial updates, and do not make fully-binary update decisions.

Original: For this visualization we binarize their update patterns using a ceiling function on non-zero update decisions since the neurons are still updated.
Revised: For this visualization we binarize their update patterns using a ceiling function on non-zero update decisions since the neurons are still updated.

Original: Importantly, all algorithms are subject to update-budgets, which clearly effect the observed patterns.
Revised: Importantly, all algorithms are subject to update-budgets, which clearly effect the observed patterns.

Original: For example, since the SkipRNN updates all neurons at the same time, deciding to update has a large cost, eating through the update budget quickly.
Revised: For example, since the SkipRNN updates all neurons at the same time, deciding to update has a large cost, eating through the update budget quickly.

Original: Meanwhile, other methods such as our own SA-RNN and the PhasedLSTM take much smaller bites into the update budget since the neurons are updated independently.
Revised: Meanwhile, other methods such as our own SA-RNN and the PhasedLSTM take much smaller bites into the update budget since the neurons are updated independently.

Original: From these visuals, it is clear that ours is the only method which can adaptively learn to update independent neurons for many steps at a time, according to the data.
Revised: From these visuals, it is clear that ours is the only method which can adaptively learn to update independent neurons for many steps at a time, according to the data.

Original: Thus, some neurons may update a few times in the middle of the sequence instead of at the beginning or end, a decision which is driven directly by the data.
Revised: Thus, some neurons may update a few times in the middle of the sequence instead of at the beginning or end, a decision which is driven directly by the data.

Original: This allows our method to spend its computational budget in a more informed way, choosing to activate some neurons many times in a row, while leaving others unchanged.
Revised: This allows our method to spend its computational budget in a more informed way, choosing to activate some neurons many times in a row, while leaving others unchanged.

Original: Comparing accuracy and update-frequency across methods.
Revised: Comparing accuracy and update-frequency across methods.

Original: Second, we show results from the three tasks described in Section 4.1.
Revised: Second, we show results from the three tasks described in Section 4.1.

Original: We measure four properties of
Revised: We measure four properties of

Original: [Table] [Table]
Revised: [Table] [Table]

Original: each method.
Revised: each method.

Original: 1) Skip Percent – the proportion of neurons which are not updated, computed across all timesteps.
Revised: 1) Skip Percent – the proportion of neurons which are not updated, computed across all timesteps.

Original: To make a fair comparison between all methods, we tune their hyperparameters so that their skip percents are as close together as possible.
Revised: To make a fair comparison between all methods, we tune their hyperparameters so that their skip percents are as close together as possible.

Original: This allows us to better analyze the effects of the differences in update patterns instead of changing the hidden state dimension sizes.
Revised: This allows us to better analyze the effects of the differences in update patterns instead of changing the hidden state dimension sizes.

Original: 2) Accuracy – the accuracy achieved on the task.
Revised: 2) Accuracy – the accuracy achieved on the task.

Original: For each task we use balanced datasets, so the lower bound is 50%. 3) (cid:112)
Revised: For each task we use balanced datasets, so the lower bound is 50%. 3) (cid:112)

Original: Accuracy × Skip % – the geometric mean of a model’s accuracy and skip percent.
Revised: Accuracy × Skip % – the geometric mean of a model’s accuracy and skip percent.

Original: The measure allows comparison of all methods since they do not all update the exact same number of times.
Revised: The measure allows comparison of all methods since they do not all update the exact same number of times.

Original: Thus, this is the key performance metric, the upper and lower limits of which are 1.0 and 0.0, respectively.
Revised: Thus, this is the key performance metric, the upper and lower limits of which are 1.0 and 0.0, respectively.

Original: 4) FLOPs – as in Campos et al.
Revised: 4) FLOPs – as in Campos et al.

Original: we compute the FLOPs for each method as a surrogate for wall-clock time, which is hardware-dependent and often ﬂuctuates dramatically in practice.
Revised: we compute the FLOPs for each method as a surrogate for wall-clock time, which is hardware-dependent and often ﬂuctuates dramatically in practice.

Original: Across all tasks, SA-RNN achieves on average higher accuracy with fewer updates, as shown in
Revised: Across all tasks, SA-RNN achieves on average higher accuracy with fewer updates, as shown in

Original: Tables 1, 2, and 3.
Revised: Tables 1, 2, and 3.

Original: In Yahoo and TwitterBuzz , SA-RNN maintains by far the closest accuracy to that of the baseline GRU, which updates every hidden dimension at every timestep.
Revised: In Yahoo and TwitterBuzz , SA-RNN maintains by far the closest accuracy to that of the baseline GRU, which updates every hidden dimension at every timestep.

Original: In Seizures , both SA-RNN and PhasedLSTM are similar to the GRU, possibly due to periodic dynamics in the data.
Revised: In Seizures , both SA-RNN and PhasedLSTM are similar to the GRU, possibly due to periodic dynamics in the data.

Original: We also show in Tables 1 and 3, the accuracy of Random Updates is far worse than SA-RNN, indicating that the beneﬁts of our proposed update-strategy comes strongly from the learning .
Revised: We also show in Tables 1 and 3, the accuracy of Random Skips is far worse than SA-RNN, indicating that the beneﬁts of our proposed update-strategy comes strongly from the learning .

Original: The beneﬁts of adaptive update patterns are not present in the Clockwork RNN or PhasedLSTM.
Revised: The beneﬁts of adaptive update patterns are not present in the Clockwork RNN or PhasedLSTM.

Original: SA-RNN has nearly-equivalent FLOPs to the other methods that learn update patterns since each adds another afﬁne transformation to map data to update decisions.
Revised: SA-RNN has nearly-equivalent FLOPs to the other methods that learn update patterns since each adds another afﬁne transformation to map data to update decisions.

Original: As shown in Table 3, adapting the update decisions according to the input data adds a signiﬁcant amount of computation with highdimensional inputs.
Revised: As shown in Table 3, adapting the update decisions according to the input data adds a signiﬁcant amount of computation with highdimensional inputs.

Original: To improve this, update patterns may be predicted for multiple steps concurrently, depending on the input data.
Revised: To improve this, update patterns may be predicted for multiple steps concurrently, depending on the input data.

Original: Given the same number of updates, the FLOPs are roughly equivalent between our method, VC-GRU, and SkipRNN.
Revised: Given the same number of updates, the FLOPs are roughly equivalent between our method, VC-GRU, and SkipRNN.

Original: As in Campos et al.
Revised: As in Campos et al.

Original: (2018), it may be possible to only observe the previous hidden state, however this results in perpetual lag as information ﬁlls the hidden state.
Revised: (2018), it may be possible to only observe the previous hidden state, however this results in perpetual lag as information ﬁlls the hidden state.

Original: CW-RNN consistently has extremely low FLOPs since there is no data-driven decision making in the update patterns, instead hard-coding them beforehand.
Revised: CW-RNN consistently has extremely low FLOPs since there is no data-driven decision making in the update patterns, instead hard-coding them beforehand.

Original: Additionally, their recurrence function is not a gated recurrent memory cell.
Revised: Additionally, their recurrence function is not a gated recurrent memory cell.

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure]
Revised: [Figure]

Original: [Table][Figure] u [Figure]
Revised: [Table]

Original: cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: Effects of budgeting neuron updates.
Revised: Effects of budgeting neuron updates.

Original: Finally, we assess how the performance of SA-RNN depends on its hyperparameter λ , which budgets the number of permitted updates.
Revised: Finally, we assess how the performance of SA-RNN depends on its hyperparameter λ , which budgets the number of permitted updates.

Original: We investigate λ values from 0 to 0 .
Revised: We investigate λ values from 0 to 0 .

Original: 1 on a log-scale, since empirically all updates stopped when λ > 0 .
Revised: 1 on a log-scale, since empirically all updates stopped when λ > 0 .

Original: Interestingly, the relationship between accuracy and λ depends heavily on the task, as demonstrated in in the ﬁrst row of Figure 3.
Revised: Interestingly, the relationship between accuracy and λ depends heavily on the task, as demonstrated in in the ﬁrst row of Figure 3.

Original: For example, on the
Revised: For example, on the

Original: Twitter task, we observe a smooth transition from random guessing (when no updates are allowed) to our peak accuracy (no constraint on updates).
Revised: TwitterBuzz task, we observe a smooth transition from random guessing (when no updates are allowed) to our peak accuracy (no constraint on updates).

Original: Meanwhile, on the Seizures task, there is a sharp increase from random predictions to near-peak performance.
Revised: Meanwhile, on the Seizures task, there is a sharp increase from random predictions to near-peak performance.

Original: This could be a feature of the parameter search space, but with already-small changes to λ this indicates high-sensitivity.
Revised: This could be a feature of the parameter search space, but with already-small changes to λ this indicates high-sensitivity.

Original: Additionally, we investigate how accuracy changes as a function of the number of neurons skipped, as shown in the second row of Figure 3.
Revised: We also investigate how accuracy changes as a function of the number of neurons skipped, as shown in the second row of Figure 3.

Original: Interestingly, there are steep elbows where very few updates are needed to observe near-peak accuracy.
Revised: Interestingly, there are steep elbows where very few updates are needed to observe near-peak accuracy.

Original: This bolsters the intuition behind Residual Networks and related methods: many hidden dimensions often capture enough information to warrant direct copying.
Revised: This bolsters the intuition behind Residual Networks and related methods: many hidden dimensions often capture enough information to warrant direct copying.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we study the problem of reducing the number of updates to the state representations learned by RNNs.
Revised: In this paper, we study the problem of reducing the number of updates to the state representations learned by RNNs.

Original: This allows for less computation at each timestep.
Revised: This allows for less computation at each timestep.

Original: We propose an augmentation to general RNN models, called SA-RNN, which is carefully crafted to skip neurons while maintaining accuracy through the parameterization of a distribution of neuron update-likelihoods, making binary decisions for each neuron at each step.
Revised: We propose an augmentation to general RNN models, called SA-RNN, which is carefully crafted to skip neurons while maintaining accuracy through the parameterization of a distribution of neuron update-likelihoods, making binary decisions for each neuron at each step.

Original: We conduct extensive experiments on three real-world publicly-available datasets and show that our method achieves on average higher accuracy with fewer neuron updates compared to recent state-of-the-art alternatives.
Revised: We conduct extensive experiments on three real-world publicly -available datasets and show that our method achieves on average higher accuracy with fewer neuron updates compared to recent state-of-the-art alternatives.

Original: Our results demonstrate that updating state representations without imposing high-bias decisions on the update-patterns is not only easier to implement and train, but preferable in terms of performance.
Revised: Our results demonstrate that updating state representations without imposing high-bias decisions on the update-patterns is not only easier to implement and train, but preferable in terms of performance.

Original:  A PPENDIX 6. H YPERPARAMETER SELECTION
Revised:  A PPENDIX 6. A DDING T ASK

Revised: In this experiment, the network must learn to use a binary mask to sum the corresponding values in a real-valued sequence.

Revised: This task was classically used to validate the LSTM’s effectiveness and is now commonly used to evaluate the robustness of memory in RNNs in the presence of extremely long-term dependencies (Hochreiter & Schmidhuber, 1997; Neil et al., 2016; Campos et al., 2018).

Revised: The task is as follows: given a data sequence x of T values sampled uniformly between 0 and 1 and a binary masking sequence m containing T − 2 zeros and 2 ones which are located at indices i and j , the ground truth y is computed as y

Revised: = x i + x j .

Revised: Given a new data sequence x and a masking sequence m , the mean squared error between y and f θ ( x, m ) should be approximately 0.

Revised: We set T = 500 to stress-test long-term dependencies and use 128-dimensional hidden states, as suggested in Henaff et al.

Revised: [Table]

Revised: Random

Revised: [Table]

Revised: Random

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: As shown in Table 4, SA-RNN consistently solves the adding task with far higher skip percentage and lower FLOPs than the baseline GRU and the SkipRNN.

Revised: Even though SkipRNN theoretically can and should ﬁnd the proper timesteps to observe given the mask, it did not solve the problem consistently, only performing well under a couple of choice settings which updated the vast majority of the hidden states.

Revised: However, we note that the sequence-length T used in this experiment is ten times greater than was originally explored in Campos et al.

Original: We employ the validation set to select hyperparameters for compared models, as described in Section 3.2 of the submitted paper.
Revised: We employ the validation set to select hyperparameters for compared models described in Section 4.2.

Original: • Random Updates : When training Random Updates, the one hyperparameter to tune is the probability of updating each hidden dimension .
Revised: • Random Updates : When training Random Updates, the one hyperparameter to tune is the probability of updating each hidden dimension .

Original: Our goal is to match the number updates in Random Updates to the number observed in our proposed method, SA-RNN.
Revised: Our goal is to match the number updates in Random Updates to the number observed in our proposed method, SA-RNN.

Original: So after observing how many times SA-RNN updated, we simply create random binary masks with the same update proportions.
Revised: So after observing how many times SA-RNN updated, we simply create random binary masks with the same update proportions.

Original: • Clockwork RNN : This method relies on hand-crafted update patterns.
Revised: • Clockwork RNN : This method relies on hand-crafted update patterns.

Original: In order to have roughly 20 % of the updates (to be similar to the other methods) with 50-dimension state representations, we use block-sizes of 5 (the number of dimensions whose updates are tied together) with exponentially-increasing clock rates as proposed in the original paper: { 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 , 264 , 512 } .
Revised: In order to have roughly 20 % of the updates (to be similar to the other methods) with 50-dimension state representations, we use block-sizes of 5 (the number of dimensions whose updates are tied together) with exponentially-increasing clock rates as proposed in the original paper: { 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 , 264 , 512 } .

Original: We choose this as it is the originally-intended use of the model and resulted in the proper number of updates.
Revised: We choose this as it is the originally-intended use of the model and resulted in the proper number of updates.

Original: There is no variance in this method as it does not react to the input data.
Revised: There is no variance in this method as it does not react to the input data.

Original: • PhasedLSTM : This method also hand-crafted update patterns, sampling update periods for individual hidden dimensions.
Revised: • PhasedLSTM : This method also hand-crafted update patterns, sampling update periods for individual hidden dimensions.

Original: The distribution from which to sample update periods is left up to the user (along with the parameters of such distribution).
Revised: The distribution from which to sample update periods is left up to the user (along with the parameters of such distribution).

Original: As proposed in the paper, we sample update periods from a uniform distribution U .
Revised: As proposed in the paper, we sample update periods from a uniform distribution U .

Original: To determine the lower and upper bounds of the distribution, we tried sampling update patterns from a uniform distribution of all combinations of integers between 0 and 6, ﬁnally choosing U (1 , 3) .
Revised: To determine the lower and upper bounds of the distribution, we tried sampling update patterns from a uniform distribution of all combinations of integers between 0 and 6, ﬁnally choosing U (1 , 3) .

Original: We sample D values from this distribution, where D is the dimension of the state representation.
Revised: We sample D values from this distribution, where D is the dimension of the state representation.

Original: We choose the shift as described in the original PhasedLSTM paper, sampling a value with an upper limit of the period for each neuron.
Revised: We choose the shift as described in the original PhasedLSTM paper, sampling a value with an upper limit of the period for each neuron.

Original: These two settings, paired with r on = 0 .
Revised: These two settings, paired with r on = 0 .

Original: 3 led to a comparable number of updates.
Revised: 3 led to a comparable number of updates.

Original: • VC-GRU : This method predicts proportions of the state representation to update at each timestep.
Revised: • VC-GRU : This method predicts proportions of the state representation to update at each timestep.

Original: During the optimization, there is a loss term which adds a “target” proportion, penalizing deviations from this target value.
Revised: During the optimization, there is a loss term which adds a “target” proportion, penalizing deviations from this target value.

Original: As suggested in the original paper, we tried values of 0 . 1 , 0 . 2 , and 0 .
Revised: As suggested in the original paper, we tried values of 0 . 1 , 0 . 2 , and 0 .

Original: 5 and selected 0 .
Revised: 5 and selected 0 .

Original: 1 , which resulted in a comparable number of updates.
Revised: 1 , which resulted in a comparable number of updates.

Original: • SkipRNN : This method predicts likelihoods of state updates at each timestep.
Revised: • SkipRNN : This method predicts likelihoods of state updates at each timestep.

Original: In the loss function, there is one hyperparameter, λ , which penalizes the number of updates.
Revised: In the loss function, there is one hyperparameter, λ , which penalizes the number of updates.

Original: We use a log-space search, ranging from 0 . 0 to 0 .
Revised: We use a log-space search, ranging from 0 . 0 to 0 .

Original: 1 in 11 steps, settling on 1 e − 05 for the Twitter task, and 1 e − 04 for the Seizures and Yahoo tasks.
Revised: 1 in 11 steps, settling on 1 e − 05 for the Twitter task, and 1 e − 04 for the Seizures and Yahoo tasks.

Original: We found the number of updates to be extremely sensitive, skipping from roughly 13 % to 100 % between single log-space steps. 
Revised: We found the number of updates to be extremely sensitive, skipping from roughly 13 % to 100 % between single log-space steps. [Figure]

Original: 6. V ISUALIZING UPDATE PATTERNS
Revised: 6. V ISUALIZING UPDATE PATTERNS

Original: In Section 4.3 we visualize and contrast the update patterns between all described methods.
Revised: In Section 4.3 we visualize and contrast the update patterns between all described methods.

Original: PhasedLSTM and VC-GRU, however, do not propose fully-binary update patterns.
Revised: PhasedLSTM and VC-GRU, however, do not propose fully-binary update patterns.

Original: PhasedLSTM softly opens and closes neurons, and VC-GRU approximates binary update-decisions using a soft mask, changing the proportion of updated-neurons at each timestep.
Revised: PhasedLSTM softly opens and closes neurons, and VC-GRU approximates binary update-decisions using a soft mask, changing the proportion of updated-neurons at each timestep.

Original: In order to fairly compare them to all other methods, we employ ceiling functions to raise their update predictions to 1 (prior to this, their updates are between 0 and 1, similar to LSTM and GRU).
Revised: In order to fairly compare them to all other methods, we employ ceiling functions to raise their update predictions to 1 (prior to this, their updates are between 0 and 1, similar to LSTM and GRU).

Original: In Figure 4, we display their update patterns prior to this ceiling function, to show their use as originally intended.
Revised: In Figure 4, we display their update patterns prior to this ceiling function, to show their use as originally intended.

Original: We believe that this use of the ceiling function is fair, since such soft updates are still updates, which is the focus of our work.
Revised: We believe that this use of the ceiling function is fair, since such soft updates are still updates, which is the focus of our work.

Original: 6. I MPLEMENTING SA-RNN
Revised: 6. I MPLEMENTING SA-RNN

Original: We implement our method (and all other methods) in PyTorch 1.0.
Revised: We implement our method (and all other methods) in PyTorch 1.0.

Original: Below, we provide some pseudocode to give a taste of how to approach implementing SA-RNN (at a high level).
Revised: Below, we provide some pseudocode to give a taste of how to approach implementing SA-RNN (at a high level).

Original: The pseudocode is loosely formatted similar to PyTorch for readability.
Revised: The pseudocode is loosely formatted similar to PyTorch for readability.

Original: In the pseudocode, Discriminator is simply[Figure] a network which maps the hidden state to the latent space for the ﬁnal prediction.
Revised: In the pseudocode, Discriminator is simply a network which maps the hidden state to the latent space for the ﬁnal prediction.

Original: Upon acceptance of the paper, the full code will be released along with our entire training process and many examples.
Revised: Upon acceptance of the paper, the full code will be released along with our entire training process and many examples.

Original: In practice the next state does not need to be fully computed once the update decisions are computed.
Revised: In practice the next state does not need to be fully computed once the update decisions are computed.

Original: def forward(sequence): u_likelihoods=
Revised: def forward(sequence): u_likelihoods=

Original: zeros
Revised: zeros

Original: # Initialize update likelihoods as zeros state =
Revised: # Initialize update likelihoods as zeros state =

Original: intializeState() # Create the initial state for t in range(len(sequence)): # Loop through sequence.
Revised: intializeState() # Create the initial state for t in range(len(sequence)): # Loop through sequence.

Original: x = sequence[t] # Extract one timestep of data u_likelihoods =
Revised: x = sequence[t] # Extract one timestep of data u_likelihoods =

Original: Coordinator.forward(x, u_likelihoods) u_decisions =
Revised: Coordinator.forward(x, u_likelihoods) u_decisions =

Original: Binarize.forward(u_likelihoods)
Revised: Binarize.forward(u_likelihoods)

Original: state_tilde = RNN.forward(x, state) state =
Revised: state_tilde = RNN.forward(x, state) state =

Original: u_decisions*state_tilde + (1-u_decisions)*state prediction = Discriminator.forward(state) return prediction
Revised: u_decisions*state_tilde + (1-u_decisions)*state prediction = Discriminator.forward(state) return prediction