Original: T RAINING R ECURRENT N EURAL N ETWORKS O NLINE BY L EARNING E XPLICIT S TATE V ARIABLES
Revised: T RAINING R ECURRENT N EURAL N ETWORKS O NLINE BY L EARNING E XPLICIT S TATE V ARIABLES

Original: A BSTRACT
Revised: A BSTRACT

Original: Recurrent neural networks (RNNs) allow an agent to construct a staterepresentation from a stream of experience, which is essential in partially observable problems.
Revised: Recurrent neural networks (RNNs) allow an agent to construct a staterepresentation from a stream of experience, which is essential in partially observable problems.

Original: However, there are two primary issues one must overcome when training an RNN: the sensitivity of the learning algorithm’s performance to truncation length and and long training times.
Revised: However, there are two primary issues one must overcome when training an RNN: the sensitivity of the learning algorithm’s performance to truncation length and and long training times.

Original: There are variety of strategies to improve training in RNNs, the mostly notably Backprop Through Time (BPTT) and by Real-Time Recurrent Learning.
Revised: There are variety of strategies to improve training in RNNs, the mostly notably Backprop Through Time (BPTT) and by Real-Time Recurrent Learning.

Original: These strategies, however, are typically computationally expensive and focus computation on computing gradients back in time.
Revised: These strategies, however, are typically computationally expensive and focus computation on computing gradients back in time.

Original: In this work, we reformulate the RNN training objective to explicitly learn state vectors; this breaks the dependence across time and so avoids the need to estimate gradients far back in time.
Revised: In this work, we reformulate the RNN training objective to explicitly learn state vectors; this breaks the dependence across time and so avoids the need to estimate gradients far back in time.

Original: We show that for a ﬁxed buffer of data, our algorithm—called Fixed Point Propagation (FPP)—is sound: it converges to a stationary point of the new objective.
Revised: We show that for a ﬁxed buffer of data, our algorithm—called Fixed Point Propagation (FPP)—is sound: it converges to a stationary point of the new objective.

Original: We investigate the empirical performance of our online FPP algorithm, particularly in terms of computation compared to truncated BPTT with varying truncation levels.
Revised: We investigate the empirical performance of our online FPP algorithm, particularly in terms of computation compared to truncated BPTT with varying truncation levels.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many online prediction problems are partially observable: the most recent observation is typically insufﬁcient to make accurate predictions about the future.
Revised: Many online prediction problems are partially observable: the most recent observation is typically insufﬁcient to make accurate predictions about the future.

Original: Augmenting the inputs with a history can improve accuracy, but can require a long history when there are long-term dependencies back in time.
Revised: Augmenting the inputs with a history can improve accuracy, but can require a long history when there are long-term dependencies back in time.

Original: Recurrent Neural Networks (RNNs) (Elman, 1990; Hopﬁeld, 1982) learn a state which summarizes this history.
Revised: Recurrent Neural Networks (RNNs) (Elman, 1990; Hopﬁeld, 1982) learn a state which summarizes this history.

Original: Speciﬁcally, RNNs contain recurrent connections to their hidden layers which allow past information to propagate through time.
Revised: Speciﬁcally, RNNs contain recurrent connections to their hidden layers which allow past information to propagate through time.

Original: This state need not correspond to a true underlying state; rather, the state is subjective and constructed to facilitate prediction.
Revised: This state need not correspond to a true underlying state; rather, the state is subjective and constructed to facilitate prediction.

Original: RNNs have been widely used, in speech recognition (Hinton et al. , 2012; Graves et al. , 2013; Miao et al. , 2015; Chan et al. , 2016), image captioning (Mao et al. , 2014;
Revised: RNNs have been widely used, in speech recognition (Hinton et al. , 2012; Graves et al. , 2013; Miao et al. , 2015; Chan et al. , 2016), image captioning (Mao et al. , 2014;

Original: Lu et al. , 2016; Vinyals et al. , 2014), speech synthesis (Mehri et al. , 2016) and reinforcement learning (Hochreiter and Schmidhuber, 1997; Düll et al. , 2012).
Revised: Lu et al. , 2016; Vinyals et al. , 2014), speech synthesis (Mehri et al. , 2016) and reinforcement learning (Hochreiter and Schmidhuber, 1997; Düll et al. , 2012).

Original: Despite these success, there are signiﬁcant stability and computational issues in training RNNs online (Pascanu et al. , 2013; Tallec and Ollivier, 2017).
Revised: Despite these success, there are signiﬁcant stability and computational issues in training RNNs online (Pascanu et al. , 2013; Tallec and Ollivier, 2017).

Original: In the online setting, the agent faces an unending stream of data and on each step the agent must update its parameters to make a new prediction.
Revised: In the online setting, the agent faces an unending stream of data and on each step the agent must update its parameters to make a new prediction.

Original: RNNs are typically trained either using Backpropagation-through-time (BPTT)
Revised: RNNs are typically trained either using Backpropagation-through-time (BPTT)

Original: (Werbos, 1990) or approximations to an algorithm called Real-Time Recurrent Learning (RTRL)
Revised: (Werbos, 1990) or approximations to an algorithm called Real-Time Recurrent Learning (RTRL)

Original: (Williams and Zipser, 1989a; Pearlmutter, 1995), although there are methods that appeal to other principles (see Murray (2019) for instance).
Revised: (Williams and Zipser, 1989a; Pearlmutter, 1995), although there are methods that appeal to other principles (see Murray (2019) for instance).

Original: The update for BPTT is a variant of standard backpropagation, computing gradients all the way back in time.
Revised: The update for BPTT is a variant of standard backpropagation, computing gradients all the way back in time.

Original: This approach is problematic because the computational cost scales linearly with the number of time-steps.
Revised: This approach is problematic because the computational cost scales linearly with the number of time-steps.

Original: A more common alternative is truncated BPTT (T-BPTT)
Revised: A more common alternative is truncated BPTT (T-BPTT)

Original: (Williams and Peng, 1990) which only computes the gradient up to some maximum number of steps: we truncate how far back in time we unroll the network to update the parameters.
Revised: (Williams and Peng, 1990) which only computes the gradient up to some maximum number of steps: we truncate how far back in time we unroll the network to update the parameters.

Original: This approximation, though, is not robust to long-term dependencies (Tallec and Ollivier, 2017).
Revised: This approximation, though, is not robust to long-term dependencies (Tallec and Ollivier, 2017).

Original: Approximate gradients can also be computed online by RTRL (Williams and Zipser, 1989b).
Revised: Approximate gradients can also be computed online by RTRL (Williams and Zipser, 1989b).

Original: This online algorithm, however, has high computational complexity per step and therefore is not commonly used in practice.
Revised: This online algorithm, however, has high computational complexity per step and therefore is not commonly used in practice.

Original: Recently, there have been some efforts towards approximating gradients for back-propagation, both for feedforward NNs and RNNs.
Revised: Recently, there have been some efforts towards approximating gradients for back-propagation, both for feedforward NNs and RNNs.

Original: Synthetic gradients and BP( λ ) (Jaderberg et al. , 2017) use an idea similar to returns from reinforcement learning: they approximate gradients by bootstrapping off estimated gradients in later layers (Jaderberg et al. , 2017; Czarnecki et al. , 2017).
Revised: Synthetic gradients and BP( λ ) (Jaderberg et al. , 2017) use an idea similar to returns from reinforcement learning: they approximate gradients by bootstrapping off estimated gradients in later layers (Jaderberg et al. , 2017; Czarnecki et al. , 2017).

Original: There are several methods that approximate RTRL—which is itself an approximation of the true gradient back in time—including NoBackTrack (Ollivier and Charpiat, 2015), Unbiased Online Recurrent
Revised: There are several methods that approximate RTRL—which is itself an approximation of the true gradient back in time—including NoBackTrack (Ollivier and Charpiat, 2015), Unbiased Online Recurrent

Original: Optimization (UORO) (Tallec and Ollivier, 2017) which uses an unbiased rank-1 approximation to the full matrix gradient, and Kronecker Factored RTRL (Mujika et al. , 2018) which uses a Kronecker product decomposition to approximate the RTRL update for a class of RNNs.
Revised: Optimization (UORO) (Tallec and Ollivier, 2017) which uses an unbiased rank-1 approximation to the full matrix gradient, and Kronecker Factored RTRL (Mujika et al. , 2018) which uses a Kronecker product decomposition to approximate the RTRL update for a class of RNNs.

Original: Finally, there are some methods that use selective memory back in time to compute gradients for the most pertinent samples, using skip connections (Ke et al. , 2017).
Revised: Finally, there are some methods that use selective memory back in time to compute gradients for the most pertinent samples, using skip connections (Ke et al. , 2017).

Original: All of these methods, however, attempt to approximate the gradient back in time, for the current observation and state, and so suffer to some extent from the same issues as BPTT and RTRL.
Revised: All of these methods, however, attempt to approximate the gradient back in time, for the current observation and state, and so suffer to some extent from the same issues as BPTT and RTRL.

Original: In this paper, we investigate an alternative optimization strategy that does not attempt to approximate the gradient back in time.
Revised: In this paper, we investigate an alternative optimization strategy that does not attempt to approximate the gradient back in time.

Original: Instead we learn the state variables in the RNN explicitly.
Revised: Instead we learn the state variables in the RNN explicitly.

Original: These new variables are optimized to both improve prediction accuracy, and to maintain consistency in producing the next learned state variables.
Revised: These new variables are optimized to both improve prediction accuracy, and to maintain consistency in producing the next learned state variables.

Original: This second constraint is a ﬁxed-point formula for the states under the given RNN dynamics.
Revised: This second constraint is a ﬁxed-point formula for the states under the given RNN dynamics.

Original: 1 We develop a provably sound stochastic update for the new ﬁxed-point objective, which we then use to develop an online algorithm for training RNNs.
Revised: 1 We develop a provably sound stochastic update for the new ﬁxed-point objective, which we then use to develop an online algorithm for training RNNs.

Original: The algorithm explicitly optimizes state vectors and RNN parameters with many efﬁcient one-step—or short term multi-step updates—across a buffer.
Revised: The algorithm explicitly optimizes state vectors and RNN parameters with many efﬁcient one-step—or short term multi-step updates—across a buffer.

Original: Instead of focusing computation to get a more accurate gradient estimates for this time-step, our algorithm, called Fixed Point Propagation (FPP), can more effectively use computation to update prediction accuracy across states.
Revised: Instead of focusing computation to get a more accurate gradient estimates for this time-step, our algorithm, called Fixed Point Propagation (FPP), can more effectively use computation to update prediction accuracy across states.

Original: We demonstrate that the algorithm is effective on several problems with long-term dependencies, and improves over T-BPTT, particularly in terms of stability and computation.
Revised: We demonstrate that the algorithm is effective on several problems with long-term dependencies, and improves over T-BPTT, particularly in terms of stability and computation.

Original:  P ROBLEM S ETTING AND B ACKGROUND
Revised:  P ROBLEM S ETTING AND B ACKGROUND

Original: We consider a partially observable online setting, where an immediate observation is not sufﬁcient for prediction.
Revised: We consider a partially observable online setting, where an immediate observation is not sufﬁcient for prediction.

Original: More formally, assume there is a sequence of n observations, o 1 , . . .
Revised: More formally, assume there is a sequence of n observations, o 1 , . . .

Original: , o n , which provide only partial information about an unknown underlying sequence of states.
Revised: , o n , which provide only partial information about an unknown underlying sequence of states.

Original: After obtaining an observation o i , the agent makes a prediction ˆ y i and sees the actual outcome y i .
Revised: After obtaining an observation o i , the agent makes a prediction ˆ y i and sees the actual outcome y i .

Original: The goal is to minimize this prediction error.
Revised: The goal is to minimize this prediction error.

Original: Given only o
Revised: Given only o

Original: i , the agent is unlikely to make accurate predictions about y i , because o i is not a sufﬁcient statistic to predict y
Revised: i , the agent is unlikely to make accurate predictions about y i , because o i is not a sufﬁcient statistic to predict y

Original: i : p ( y | o i , o i − 1 , o i − 2 , . . . )
Revised: i : p ( y | o i , o i − 1 , o i − 2 , . . . )

Original: (cid:54) =
Revised: (cid:54) =

Original: p ( y | o i ) .
Revised: p ( y | o i ) .

Original: The agent could obtain a lower prediction error using a history of observations.
Revised: The agent could obtain a lower prediction error using a history of observations.

Original: Unfortunately, the agent may require a prohibitively long history, even if this history could have been summarized compactly.
Revised: Unfortunately, the agent may require a prohibitively long history, even if this history could have been summarized compactly.

Original: An alternative is to construct state using a Recurrent Neural Network (RNN), by learning a stateupdate function.
Revised: An alternative is to construct state using a Recurrent Neural Network (RNN), by learning a stateupdate function.

Original: Given the current (constructed) state s t − 1 ∈ R k , and a new observation o t ∈ R d , the parameterized state-update function f W : R k × R d
Revised: Given the current (constructed) state s t − 1 ∈ R k , and a new observation o t ∈ R d , the parameterized state-update function f W : R k × R d

Original: →
Revised: →

Original: R k , with parameters W , produces the next (constructed) state s t = f W ( s t − 1 , o t ) .
Revised: R k , with parameters W , produces the next (constructed) state s t = f W ( s t − 1 , o t ) .

Original: For example, f W could be a linear weighting of s t − 1 and o t , with a ReLu activation: f W ( s t − 1 , o t )
Revised: For example, f W could be a linear weighting of s t − 1 and o t , with a ReLu activation: f W ( s t − 1 , o t )

Original: =
Revised: =

Original: max([ s t − 1 , o t ] W , 0 ) .
Revised: max([ s t − 1 , o t ] W , 0 ) .

Original: More complex state-updates are possible, like the gating in Long Short-Term Memory (LSTM)
Revised: More complex state-updates are possible, like the gating in Long Short-Term Memory (LSTM)

Original: (Hochreiter and Schmidhuber, 1997).
Revised: (Hochreiter and Schmidhuber, 1997).

Original: The objective is to adapt these parameters W for the state-update to minimize prediction error.
Revised: The objective is to adapt these parameters W for the state-update to minimize prediction error.

Original: For the current state s t , a prediction is made by parameterized function g β : R k
Revised: For the current state s t , a prediction is made by parameterized function g β : R k

Original: → R m with learned parameters β .
Revised: → R m with learned parameters β .

Original: For example, the prediction could be a linear weighting of the state, g β ( s ) =
Revised: For example, the prediction could be a linear weighting of the state, g β ( s ) =

Original: [Equation]
Revised: [Equation]

Original: We denote the prediction error as (cid:96) β : R k × R m → R for a given β .
Revised: We denote the prediction error as (cid:96) β : R k × R m → R for a given β .

Original: For example, this loss could be
Revised: For example, this loss could be

Original: [Equation]
Revised: [Equation]

Original: The goal in RNNs training is to minimize, for some start state s 0 ,
Revised: The goal in RNNs training is to minimize, for some start state s 0 ,

Original: [Equation]
Revised: [Equation]

Original: Computing gradients for this objective, however, can be prohibitively expensive.
Revised: Computing gradients for this objective, however, can be prohibitively expensive.

Original: A large literature on optimizing RNNs focuses on approximating this gradient, either through approximations to RTRL or improvements to BPTT.
Revised: A large literature on optimizing RNNs focuses on approximating this gradient, either through approximations to RTRL or improvements to BPTT.

Original: RTRL (Williams and Zipser, 1989b) uses a recursive gradient form, which can take advantage of gradients computed up until the last observation to compute the gradient for the next observation.
Revised: RTRL (Williams and Zipser, 1989b) uses a recursive gradient form, which can take advantage of gradients computed up until the last observation to compute the gradient for the next observation.

Original: This estimate, however, is only exact in the ofﬂine case and thus RTRL is an approximation of the true gradient in our online setting.
Revised: This estimate, however, is only exact in the ofﬂine case and thus RTRL is an approximation of the true gradient in our online setting.

Original: Further, in either online or ofﬂine,
Revised: Further, in either online or ofﬂine,

Original: RTRL requires O ( k 4 ) computation per observation (recall k is the dimension of the state).
Revised: RTRL requires O ( k 4 ) computation per observation (recall k is the dimension of the state).

Original: In BPTT, gradients are computed back in time, by unrolling the network.
Revised: In BPTT, gradients are computed back in time, by unrolling the network.

Original: In the online setting, it is infeasible to compute gradients all the way back to the beginning of time.
Revised: In the online setting, it is infeasible to compute gradients all the way back to the beginning of time.

Original: Instead, this procedure is truncated T steps back in time.
Revised: Instead, this procedure is truncated T steps back in time.

Original: T-BPTT is suitable for the online setting, and requires O ( Tk 2 ) computation per step, i.e., for each observation.
Revised: T-BPTT is suitable for the online setting, and requires O ( Tk 2 ) computation per step, i.e., for each observation.

Original: Arguably the most widely-used strategy is T-BPTT, because of its simplicity.
Revised: Arguably the most widely-used strategy is T-BPTT, because of its simplicity.

Original: Unfortunately, T-BPTT has been shown to fail in settings where dependencies back in time are further than T (Tallec and
Revised: Unfortunately, T-BPTT has been shown to fail in settings where dependencies back in time are further than T (Tallec and

Original: Ollivier, 2017), as we afﬁrm in our experiments.
Revised: Ollivier, 2017), as we afﬁrm in our experiments.

Original: Yet, the need for simple algorithms remains.
Revised: Yet, the need for simple algorithms remains.

Original: In this work, we investigate an alternative direction for optimizing RNNs that does not attempt to estimate the gradients of (1).
Revised: In this work, we investigate an alternative direction for optimizing RNNs that does not attempt to estimate the gradients of (1).

Original: Note that in addition to a variety of optimization strategies, different architectures have also been proposed to facilitate learning long-term dependencies with RNNs.
Revised: Note that in addition to a variety of optimization strategies, different architectures have also been proposed to facilitate learning long-term dependencies with RNNs.

Original: The most commonly used are
Revised: The most commonly used are

Original: LSTMs (Hochreiter and Schmidhuber, 1997), which use gates to remember and forget components of the state.
Revised: LSTMs (Hochreiter and Schmidhuber, 1997), which use gates to remember and forget components of the state.

Original: Other architectures include clockwork RNNs (Koutník et al. , 2014), phased LSTMs (Neil et al. , 2016), hierarchical multi-scale RNNs (Chung et al. , 2016), dilated RNNs (Chang et al. , 2017), and skip RNNs (Campos et al. , 2017).
Revised: Other architectures include clockwork RNNs (Koutník et al. , 2014), phased LSTMs (Neil et al. , 2016), hierarchical multi-scale RNNs (Chung et al. , 2016), dilated RNNs (Chang et al. , 2017), and skip RNNs (Campos et al. , 2017).

Original: In this work, we focus on a general purpose RNN algorithm, that could be combined with each of these architectures for further improvement.
Revised: In this work, we focus on a general purpose RNN algorithm, that could be combined with each of these architectures for further improvement.

Original:  A N EW F IXED -P OINT O BJECTIVE FOR RNN S
Revised:  A N EW F IXED -P OINT O BJECTIVE FOR RNN S

Original: In this section we introduce our new formulation for training RNNs.
Revised: In this section we introduce our new formulation for training RNNs.

Original: We begin with an idealized setting to introduce and explain the ideas.
Revised: We begin with an idealized setting to introduce and explain the ideas.

Original: Later we will generalize our approach to partially observable online training tasks.
Revised: Later we will generalize our approach to partially observable online training tasks.

Original: First, assume the observations are produced by an underly Markov Chain with a discrete set of states, and the agent has access to a set of observations that are deterministic function of the state.
Revised: First, assume the observations are produced by an underly Markov Chain with a discrete set of states, and the agent has access to a set of observations that are deterministic function of the state.

Original: We denote the set of states H = { 1 , . . . , n } , and the observations from each state as o 1 , . . .
Revised: We denote the set of states H = { 1 , . . . , n } , and the observations from each state as o 1 , . . .

Original: , o n .
Revised: , o n .

Original: The goal is to ﬁnd state vectors s 1 , . . .
Revised: The goal is to ﬁnd state vectors s 1 , . . .

Original: , s n ∈ R k that satisfy two goals.
Revised: , s n ∈ R k that satisfy two goals.

Original: One is to enable the state to be updated
Revised: One is to enable the state to be updated

Original: [Equation]
Revised: [Equation]

Original: for P : H × H → [0 , 1] the transition dynamics.
Revised: for P : H × H → [0 , 1] the transition dynamics.

Original: Another criterion is for these state vectors to facilitate accurate predictions.
Revised: Another criterion is for these state vectors to facilitate accurate predictions.

Original: In particular, the learned state should minimize (cid:96) β
Revised: In particular, the learned state should minimize (cid:96) β

Original: ( s j ; y j ) for all h , where y j ∈ R is the expected target for a true state j .
Revised: ( s j ; y j ) for all h , where y j ∈ R is the expected target for a true state j .

Original: Together, this results in the following optimization, with the relationship between states encoded as a constraint
Revised: Together, this results in the following optimization, with the relationship between states encoded as a constraint

Original: [Equation]
Revised: [Equation]

Original: The satisﬁability of this will depend on f W and if s i and o j can uniquely determine s j .
Revised: The satisﬁability of this will depend on f W and if s i and o j can uniquely determine s j .

Original: More generally, we will not know the underlying state, nor is it necessarily discrete.
Revised: More generally, we will not know the underlying state, nor is it necessarily discrete.

Original: But, we can consider a similar objective for observed data.
Revised: But, we can consider a similar objective for observed data.

Original: Assume n observations have been observed, o 1 , . . .
Revised: Assume n observations have been observed, o 1 , . . .

Original: o n , with corresponding targets y 1 , . . .
Revised: o n , with corresponding targets y 1 , . . .

Original: , y n .
Revised: , y n .

Original: Let the state variables be stacked in a matrix
Revised: Let the state variables be stacked in a matrix

Original: S ∈ R k × n and observations as a matrix O ∈ R d × n , with S
Revised: S ∈ R k × n and observations as a matrix O ∈ R d × n , with S

Original: =
Revised: =

Original: [ s 0 , . . .
Revised: [ s 0 , . . .

Original: , s n ] and O =
Revised: , s n ] and O =

Original: [ o 1 , . . .
Revised: [ o 1 , . . .

Original: , o n ] .
Revised: , o n ] .

Original: The constraint on the states becomes S = F W ( S , O ) for operator
Revised: The constraint on the states becomes S = F W ( S , O ) for operator

Original: [Equation]
Revised: [Equation]

Original: We call this the ﬁxed-point constraint, since a solution S to the constraint is a ﬁxed point of the system deﬁned by F W ( · , O ) .
Revised: We call this the ﬁxed-point constraint, since a solution S to the constraint is a ﬁxed point of the system deﬁned by F W ( · , O ) .

Original: The resulting optimization, for this batch, is
Revised: The resulting optimization, for this batch, is

Original: [Equation]
Revised: [Equation]

Original: The solution to this new optimization corresponds to the solution for the original RNN problem in (1)—when also optimizing over s 0 in (1)—because the ﬁxed-point constraint forces variables s i to be representable by f W .
Revised: The solution to this new optimization corresponds to the solution for the original RNN problem in (1)—when also optimizing over s 0 in (1)—because the ﬁxed-point constraint forces variables s i to be representable by f W .

Original: Therefore, the reformulation as a ﬁxed point problem has not changed the solution; rather, it has only made explicit that the goal is to learn these states and facilitates the use of alternative optimization strategies.
Revised: Therefore, the reformulation as a ﬁxed point problem has not changed the solution; rather, it has only made explicit that the goal is to learn these states and facilitates the use of alternative optimization strategies.

Original: Reformulations like the one in (4) have been widely considered in optimization, because (4) is actually an auxiliary variable reformulation of (1).
Revised: Reformulations like the one in (4) have been widely considered in optimization, because (4) is actually an auxiliary variable reformulation of (1).

Original: In this case, the auxiliary variables are the states
Revised: In this case, the auxiliary variables are the states

Original: S .
Revised: S .

Original: Using auxiliary variables is a standard strategy in optimization—under the general term method of multipliers —to decouple terms in an optimization and so facilitate decentralized optimization.
Revised: Using auxiliary variables is a standard strategy in optimization—under the general term method of multipliers —to decouple terms in an optimization and so facilitate decentralized optimization.

Original: Such auxiliary variable methods have even been previously considered for optimizing neural networks.
Revised: Such auxiliary variable methods have even been previously considered for optimizing neural networks.

Original: Carreira-Perpiñán and Wang (2014) introduced the Method of Auxiliary Coordinates (MAC), which explicitly optimize hidden vectors in the neural network.
Revised: Carreira-Perpiñán and Wang (2014) introduced the Method of Auxiliary Coordinates (MAC), which explicitly optimize hidden vectors in the neural network.

Original: Taylor et al.
Revised: Taylor et al.

Original: (2016) proposed a similar strategy, but introduced an additional set of auxiliary variables to obtain further decoupling and a particularly efﬁcient algorithm for the batch setting.
Revised: (2016) proposed a similar strategy, but introduced an additional set of auxiliary variables to obtain further decoupling and a particularly efﬁcient algorithm for the batch setting.

Original: Scellier and Bengio (2017) introduced
Revised: Scellier and Bengio (2017) introduced

Original: Equilibrium Propagation for symmetric neural networks, where the state of the network is explicitly optimized to obtain a stationary point in terms of the energy function.
Revised: Equilibrium Propagation for symmetric neural networks, where the state of the network is explicitly optimized to obtain a stationary point in terms of the energy function.

Original: Gotmare et al.
Revised: Gotmare et al.

Original: (2018) built on these previous ideas to obtain a stochastic gradient descent update for distributed updates to blocks of weights in a neural network.
Revised: (2018) built on these previous ideas to obtain a stochastic gradient descent update for distributed updates to blocks of weights in a neural network.

Original: Our proposed optimization can be seen as a variation of the objective considered for MAC (Carreira-Perpiñán and Wang, 2014, Equation 1), though we arrived at it from a different perspective: with the goal to learn explicit state vectors.
Revised: Our proposed optimization can be seen as a variation of the objective considered for MAC (Carreira-Perpiñán and Wang, 2014, Equation 1), though we arrived at it from a different perspective: with the goal to learn explicit state vectors.

Original: The objective in (4) still has two issues.
Revised: The objective in (4) still has two issues.

Original: First, it is not amenable to online updating: it is a batch optimization with a constraint.
Revised: First, it is not amenable to online updating: it is a batch optimization with a constraint.

Original: Second, it does not allow for any training back in time.
Revised: Second, it does not allow for any training back in time.

Original: But, this stringent computational restriction is unnecessary.
Revised: But, this stringent computational restriction is unnecessary.

Original: We could have instead asked: learn states so that when iterated twice through the RNN, the resulting state enables accurate predictions on the target two steps in the future.
Revised: We could have instead asked: learn states so that when iterated twice through the RNN, the resulting state enables accurate predictions on the target two steps in the future.

Original: We develop a more general objective below to address both issues.
Revised: We develop a more general objective below to address both issues.

Original: We can rewrite the objective so that it is clear how to stochastically sample it, and so enable online updating.
Revised: We can rewrite the objective so that it is clear how to stochastically sample it, and so enable online updating.

Original: As in MAC-QP (Carreira-Perpiñán and Wang, 2014), we reformulate this constrained objective into an unconstrained objective with a quadratic penalty, with λ >
Revised: As in MAC-QP (Carreira-Perpiñán and Wang, 2014), we reformulate this constrained objective into an unconstrained objective with a quadratic penalty, with λ >

Original: [Equation]
Revised: [Equation]

Original: Once in this unconstrained form, we can perform stochastic gradient descent on this objective in terms of β , W and S to reach a stationary point.
Revised: Once in this unconstrained form, we can perform stochastic gradient descent on this objective in terms of β , W and S to reach a stationary point.

Original: To use stochastic gradient descent, the objective needs to break up into a sum of losses, L ( β , W , S )
Revised: To use stochastic gradient descent, the objective needs to break up into a sum of losses, L ( β , W , S )

Original: = 1 n (cid:80) ni
Revised: = 1 n (cid:80) ni

Original: =1 L i ( β , W , S ) , where we deﬁne
Revised: =1 L i ( β , W , S ) , where we deﬁne

Original: [Equation]
Revised: [Equation]

Original: We can stochastically sample i from our buffer of n samples and update our variables with ∇ L i .
Revised: We can stochastically sample i from our buffer of n samples and update our variables with ∇ L i .

Original: Fortunately, because the state variables break connections across time, this gradient is zero for most variables, except β , W , s i − 1 and s i .
Revised: Fortunately, because the state variables break connections across time, this gradient is zero for most variables, except β , W , s i − 1 and s i .

Original: Therefore, each stochastic update can be computed efﬁciently.
Revised: Therefore, each stochastic update can be computed efﬁciently.

Original: Second
Revised: Second

Original: [Equation]
Revised: [Equation]

Original: we can generalize this objective to incorporate more than one step of propagation back in time, simply by generalizing the ﬁxed-point operator.
Revised: we can generalize this objective to incorporate more than one step of propagation back in time, simply by generalizing the ﬁxed-point operator.

Original: Consider the more general T -step ﬁxed point problem S = F T, W ( S , O ) where
Revised: Consider the more general T -step ﬁxed point problem S = F T, W ( S , O ) where

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: +1YEBT6jLT80dOk3nonQtKIv6lxTLohGnAaUIyUpnrWh5emnh9AmfVSdcGzCpym0Sw9dzUxnsNF6Zz4EVfn4gm0bRvCyq+F5tRSl+UeuX9pUAa9rGeVHduZBvwL3ByUQR71nvXp9SOchIQrzJCUHdeJVTdFQlHMSFb0EklihEdoQDoachQS2U2n28zgqWb6MIiEPlzBKbvoSFEo5Tj0tTJEaihXaxPyv1onUcFNN6U8ThTheDYoSBhUEZx8DexTQbBiYw0QFlTfFeIhEggr/YFFvQR39cl/QbNqu47tvl6Wa4/5OgrgGJyAM+CCa1ADL6AOGgAbRcMxbo07s2Remffmw0xqGrnnCCyF+fwNWELBIw==</latexit
Revised: +1YEBT6jLT80dOk3nonQtKIv6lxTLohGnAaUIyUpnrWh5emnh9AmfVSdcGzCpym0Sw9dzUxnsNF6Zz4EVfn4gm0bRvCyq+F5tRSl+UeuX9pUAa9rGeVHduZBvwL3ByUQR71nvXp9SOchIQrzJCUHdeJVTdFQlHMSFb0EklihEdoQDoachQS2U2n28zgqWb6MIiEPlzBKbvoSFEo5Tj0tTJEaihXaxPyv1onUcFNN6U8ThTheDYoSBhUEZx8DexTQbBiYw0QFlTfFeIhEggr/YFFvQR39cl/QbNqu47tvl6Wa4/5OgrgGJyAM+CCa1ADL6AOGgAbRcMxbo07s2Remffmw0xqGrnnCCyF+fwNWELBIw==</latexit

Original: [Figure]
Revised: [Figure]

Original: For T = 1 , we recover the operator provided in (3).
Revised: For T = 1 , we recover the operator provided in (3).

Original: This generalization mimics the use of T -step methods for learning value functions in reinforcement learning.
Revised: This generalization mimics the use of T -step methods for learning value functions in reinforcement learning.

Original: This generalization provides more ﬂexibility in using the allocated computation per step.
Revised: This generalization provides more ﬂexibility in using the allocated computation per step.

Original: For example, for a budget of T updates per step, we could use T 1-step updates, T/ 2 2-step updates, all the way up to one T -step update.
Revised: For example, for a budget of T updates per step, we could use T 1-step updates, T/ 2 2-step updates, all the way up to one T -step update.

Original: The loss for general T similarly decomposes into a sum
Revised: The loss for general T similarly decomposes into a sum

Original: [Equation]
Revised: [Equation]

Original: 6) where ˆ s i ( s i − T , W ) def = f W ( . . .
Revised: 6) where ˆ s i ( s i − T , W ) def = f W ( . . .

Original: f W ( f W ( s i − T − 1 , o i − T ) ,
Revised: f W ( f W ( s i − T − 1 , o i − T ) ,

Original: o i − T +1 ) , . . . )
Revised: o i − T +1 ) , . . . )

Original: , o i ) .
Revised: , o i ) .

Original: For each stochastic sample i , ∇ L i is only non-zero for β , W , s i − T and s i .
Revised: For each stochastic sample i , ∇ L i is only non-zero for β , W , s i − T and s i .

Original: Though these updates can simply be computed using automatic differentiation on L i , the explicit updates are simple so we include them here, using shorthand ˆ s i for ˆ s i ( s i − T , W ) :
Revised: Though these updates can simply be computed using automatic differentiation on L i , the explicit updates are simple so we include them here, using shorthand ˆ s i for ˆ s i ( s i − T , W ) :

Original: [Equation]
Revised: [Equation]

Original: The online algorithm uses these updates on a sliding window buffer, instead of a ﬁxed buffer.
Revised: The online algorithm uses these updates on a sliding window buffer, instead of a ﬁxed buffer.

Original: This algorithm—called Fixed Point Propagation (FPP)— is summarized in Figure 1 and Algorithm 1.
Revised: This algorithm—called Fixed Point Propagation (FPP)— is summarized in Figure 1 and Algorithm 1.

Original: As alluded to, the advantage of FPP over T-BPTT is that we are not restricted to focusing all computation to estimate the gradient T -steps back in time for one state-observation pair.
Revised: As alluded to, the advantage of FPP over T-BPTT is that we are not restricted to focusing all computation to estimate the gradient T -steps back in time for one state-observation pair.

Original: Rather, instead of sweeping all the way back, we spread value by using updates on random transitions in the buffer.
Revised: Rather, instead of sweeping all the way back, we spread value by using updates on random transitions in the buffer.

Original: This has three advantages.
Revised: This has three advantages.

Original: First, it updates more states per step, including updates towards their targets.
Revised: First, it updates more states per step, including updates towards their targets.

Original: Second, this ensures that targets for older transitions are constantly being reinforced, and spends gradient computation resources towards this goal, rather than spending all computation on computing a more exact gradient for the recent time step.
Revised: Second, this ensures that targets for older transitions are constantly being reinforced, and spends gradient computation resources towards this goal, rather than spending all computation on computing a more exact gradient for the recent time step.

Original: This distributes updates better across time, and should likely also result in a more stable state.
Revised: This distributes updates better across time, and should likely also result in a more stable state.

Original: Third, the formulation as stochastic gradient descent on the ﬁxed point objective makes it a sound strategy—as opposed to truncation which is not sound.
Revised: Third, the formulation as stochastic gradient descent on the ﬁxed point objective makes it a sound strategy—as opposed to truncation which is not sound.

Original: FPP, therefore, maintains the simplicity of T-BPTT, but provides a more viable direction to obtain sound algorithms for training RNNs.
Revised: FPP, therefore, maintains the simplicity of T-BPTT, but provides a more viable direction to obtain sound algorithms for training RNNs.

Original:  C ONVERGENCE R ESULTS
Revised:  C ONVERGENCE R ESULTS

Original: In this section we show two theoretical results.
Revised: In this section we show two theoretical results.

Original: First, we show that the FPP algorithm converges to a stationary point, for a ﬁxed buffer.
Revised: First, we show that the FPP algorithm converges to a stationary point, for a ﬁxed buffer.

Original: This result is a relatively straightforward application of recent theory for nonconvex optimization (Ghadimi et al. , 2016), mainly requiring us to show that
Revised: This result is a relatively straightforward application of recent theory for nonconvex optimization (Ghadimi et al. , 2016), mainly requiring us to show that

Original: our
Revised: our

Original: Algorithm
Revised: Algorithm

Original: Input : a truncation parameter T , mini-batch size B , and number of updates per step M
Revised: Input : a truncation parameter T , mini-batch size B , and number of updates per step M

Original: Initialize weights W and β randomly
Revised: Initialize weights W and β randomly

Original: Initialize state s 0 ← 0 ∈ R d Initialize an empty buffer B of size N for t ← 1 , 2 , ... do if B is full then
Revised: Initialize state s 0 ← 0 ∈ R d Initialize an empty buffer B of size N for t ← 1 , 2 , ... do if B is full then

Original: Remove the oldest transition end if
Revised: Remove the oldest transition end if

Original: Observe o t , y t , and compute s t = f W ( s t − 1 , o t )
Revised: Observe o t , y t , and compute s t = f W ( s t − 1 , o t )

Original: Add ( s t , o t , y t ) to buffer B if t ≥ T then for j ← 1 , · · · , M do (cid:46) Multiple updates
Revised: Add ( s t , o t , y t ) to buffer B if t ≥ T then for j ← 1 , · · · , M do (cid:46) Multiple updates

Original: Sample a mini-batch of size B , of trajectories of length T , from the buffer: { ( s i l − T , o i l − T , . . .
Revised: Sample a mini-batch of size B , of trajectories of length T , from the buffer: { ( s i l − T , o i l − T , . . .

Original: , s i l , o i l , y i l ) }
Revised: , s i l , o i l , y i l ) }

Original: Bl =1 where i l is the index of the l -th mini-batch
Revised: Bl =1 where i l is the index of the l -th mini-batch

Original: Compute the average mini-batch loss and update { s i l − T , s i l } Bl =1 , W and β
Revised: Compute the average mini-batch loss and update { s i l − T , s i l } Bl =1 , W and β

Original: Update { s i l − T , s i l } Bl =1 in the buffer end for end if end for algorithm can be written as an instance of that framework and to show that each stochastic gradient update is unbiased.
Revised: Update { s i l − T , s i l } Bl =1 in the buffer end for end if end for algorithm can be written as an instance of that framework and to show that each stochastic gradient update is unbiased.

Original: This convergence result, nonetheless, is key, as it suggests that FPP is a sound strategy for using replay with RNNs.
Revised: This convergence result, nonetheless, is key, as it suggests that FPP is a sound strategy for using replay with RNNs.

Original: Previous attempts to use replay for RNNs, in reinforcement learning, were not able to show convergence (Kapturowski et al. , 2019), which is to be expected as truncated BPTT updates on a buffer may not be sound.
Revised: Previous attempts to use replay for RNNs, in reinforcement learning, were not able to show convergence (Kapturowski et al. , 2019), which is to be expected as truncated BPTT updates on a buffer may not be sound.

Original: Additionally, we show that as λ approaches inﬁnity, the set of stationary points of the FPP objective approaches the set of stationary points for the RNN objective.
Revised: Additionally, we show that as λ approaches inﬁnity, the set of stationary points of the FPP objective approaches the set of stationary points for the RNN objective.

Original: In our experiments, we use λ = 1 , as obtaining precisely the same solutions as the RNN objective is not our goal.
Revised: In our experiments, we use λ = 1 , as obtaining precisely the same solutions as the RNN objective is not our goal.

Original: We include this theoretical result nonetheless for completeness to characterize the relationship between the stationary points of FPP objective and the RNN objective.
Revised: We include this theoretical result nonetheless for completeness to characterize the relationship between the stationary points of FPP objective and the RNN objective.

Original: The proof is similar to that for MAC-QP (CarreiraPerpiñán and Wang, 2014), with the main novelty in checking the KKT conditions for our objective and for linear independence in the Jacobian.
Revised: The proof is similar to that for MAC-QP (CarreiraPerpiñán and Wang, 2014), with the main novelty in checking the KKT conditions for our objective and for linear independence in the Jacobian.

Original: Full proofs for both results are in Appendix A.
Revised: Full proofs for both results are in Appendix A.

Original: 4. C ONVERGENCE OF FPP TO A S TATIONARY P OINT FOR A F IXED B UFFER
Revised: 4. C ONVERGENCE OF FPP TO A S TATIONARY P OINT FOR A F IXED B UFFER

Original: Recent work uses the idea of randomized gradient descent to show convergence to a stationary point for nonconvex objectives (Ghadimi et al. , 2016), as opposed to typical restrictions such as convexity or the PL condition (Karimi et al. , 2016).
Revised: Recent work uses the idea of randomized gradient descent to show convergence to a stationary point for nonconvex objectives (Ghadimi et al. , 2016), as opposed to typical restrictions such as convexity or the PL condition (Karimi et al. , 2016).

Original: The randomized approach uses a random stopping time
Revised: The randomized approach uses a random stopping time

Original: R , and characterizes the norm of the expected gradient for the variables at this random time.
Revised: R , and characterizes the norm of the expected gradient for the variables at this random time.

Original: The variables we learn are ( W , β , S ) ∈
Revised: The variables we learn are ( W , β , S ) ∈

Original: R z , where z is the appropriate dimension.
Revised: R z , where z is the appropriate dimension.

Original: For the proof we also require the variables to remain in a closed, convex set, to ensure that our objective is Lipschitz.
Revised: For the proof we also require the variables to remain in a closed, convex set, to ensure that our objective is Lipschitz.

Original: To do so, we will analyze our update with the addition of a projection operator onto a closed ball C in R z of radius r > 0 about the origin.
Revised: To do so, we will analyze our update with the addition of a projection operator onto a closed ball C in R z of radius r > 0 about the origin.

Original: r can be very large, and we emphasize that C is only a convenience used for theoretical analysis.
Revised: r can be very large, and we emphasize that C is only a convenience used for theoretical analysis.

Original: In practice, we do not project our iterates.
Revised: In practice, we do not project our iterates.

Original: Since R d is a Hilbert space and C is closed and convex, we have the existence of a unique projection operator Γ
Revised: Since R d is a Hilbert space and C is closed and convex, we have the existence of a unique projection operator Γ

Original: [Equation]
Revised: [Equation]

Original: Our objective is L ( W , β , S ) def =
Revised: Our objective is L ( W , β , S ) def =

Original: [Equation]
Revised: [Equation]

Original: ni =
Revised: ni =

Original: [Equation]
Revised: [Equation]

Original: L i ( W , β , S ) , for L i deﬁned in Equation (6), for n > T samples.
Revised: L i ( W , β , S ) , for L i deﬁned in Equation (6), for n > T samples.

Original: Each time we perform an update, we randomly sample k t ∼ uniform- ( T, n ) , inclusive of both endpoints.
Revised: Each time we perform an update, we randomly sample k t ∼ uniform- ( T, n ) , inclusive of both endpoints.

Original: The update to parameters at time t , for stepsize α t , is
Revised: The update to parameters at time t , for stepsize α t , is

Original: [Equation]
Revised: [Equation]

Original: t )) .
Revised: t )) .

Original: Theorem 1.
Revised: Theorem 1.

Original: Let D be a Lipschitz constant of ∇ L ( W , β , S ) .
Revised: Let D be a Lipschitz constant of ∇ L ( W , β , S ) .

Original: Deﬁne probability mass functions
Revised: Deﬁne probability mass functions

Original: [Equation]
Revised: [Equation]

Original: for each N ∈ N .
Revised: for each N ∈ N .

Original: Let R be distributed according to P N .
Revised: Let R be distributed according to P N .

Original: Assume α t = 12 D for all t and that we perform N stochastic updates.
Revised: Assume α t = 12 D for all t and that we perform N stochastic updates.

Original: Write x R =
Revised: Write x R =

Original: ( W R , β R , S R ) .
Revised: ( W R , β R , S R ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: 4. R ECOVERING RNN S OLUTIONS
Revised: 4. R ECOVERING RNN S OLUTIONS

Original: Consider the standard RNN problem, min
Revised: Consider the standard RNN problem, min

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where we also optimize over s 0 .
Revised: where we also optimize over s 0 .

Original: Our goal is to show that for increasing λ , the set of stationary points of the FPP objective in Equation (5) approach stationary points of the RNN objective in Equation (10).
Revised: Our goal is to show that for increasing λ , the set of stationary points of the FPP objective in Equation (5) approach stationary points of the RNN objective in Equation (10).

Original: We assume T = 1 in our analysis of FPP.
Revised: We assume T = 1 in our analysis of FPP.

Original: Theorem 2.
Revised: Theorem 2.

Original: Assume we have a positive, increasing sequence { λ k } → ∞ , a non-negative sequence { (cid:15) k } → 0 , and a sequence of points { ( W k , β k , S k ) } such that (cid:107)∇ L ( W k , β k , S k ); λ k ) (cid:107) ≤ (cid:15) k for
Revised: Assume we have a positive, increasing sequence { λ k } → ∞ , a non-negative sequence { (cid:15) k } → 0 , and a sequence of points { ( W k , β k , S k ) } such that (cid:107)∇ L ( W k , β k , S k ); λ k ) (cid:107) ≤ (cid:15) k for

Original: L ( W k , β k , S k ); λ k ) def = 1 n n
Revised: L ( W k , β k , S k ); λ k ) def = 1 n n

Original: [Equation]
Revised: [Equation]

Original: Assume further that { ( W k , β k , S k ) } has a convergent subsequence { ( W k i ,
Revised: Assume further that { ( W k , β k , S k ) } has a convergent subsequence { ( W k i ,

Original: β k i , S k i ) } with limit ( W ∗ , β ∗ , S ∗ ) .
Revised: β k i , S k i ) } with limit ( W ∗ , β ∗ , S ∗ ) .

Original: Then ( W ∗ , β ∗ , S ∗ ) is a KKT point of the constrained FPP objective (see (12) ) and ( W ∗ , β ∗ , s ∗ 0 ) is a KKT point of the RNN objective (10) .
Revised: Then ( W ∗ , β ∗ , S ∗ ) is a KKT point of the constrained FPP objective (see (12) ) and ( W ∗ , β ∗ , s ∗ 0 ) is a KKT point of the RNN objective (10) .

Original: Further, if ( W ∗ , β ∗ , S ∗ ) is a local min of the constrained FPP objective, then ( W ∗ , β ∗ , s ∗ 0 ) is a local min of (10) .
Revised: Further, if ( W ∗ , β ∗ , S ∗ ) is a local min of the constrained FPP objective, then ( W ∗ , β ∗ , s ∗ 0 ) is a local min of (10) .

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We designed a sequence of experiments in real and synthetic problems to evaluate our new method compared with several common baselines and to highlight the robustness of our method to different truncation lengths, buffer sizes and number of updates.
Revised: We designed a sequence of experiments in real and synthetic problems to evaluate our new method compared with several common baselines and to highlight the robustness of our method to different truncation lengths, buffer sizes and number of updates.

Original: In particular we compare (1) against T-BPTT with a variety of truncation lengths greater and lesser than the temporal delay required to solve each problem; (2) No-Overlap T-BPTT, a common a variant of T-BPTT that updates on disjoint partitions of the data; and (3) FPP without the state update, which is similar to the Stored State T-BPTT algorithm (Kapturowski et al. , 2019).
Revised: In particular we compare (1) against T-BPTT with a variety of truncation lengths greater and lesser than the temporal delay required to solve each problem; (2) No-Overlap T-BPTT, a common a variant of T-BPTT that updates on disjoint partitions of the data; and (3) FPP without the state update, which is similar to the Stored State T-BPTT algorithm (Kapturowski et al. , 2019).

Original: We begin by describing the problems we used to evaluate our methods, and why they were chosen.
Revised: We begin by describing the problems we used to evaluate our methods, and why they were chosen.

Original: Unless otherwise stated, we report average performance over all training steps (online performance), averaged over 30 independent runs.
Revised: Unless otherwise stated, we report average performance over all training steps (online performance), averaged over 30 independent runs.

Original: Simulation Problems We used two small simulation problems to highlight the robustness of each method to increasing temporal delay in online training.
Revised: Simulation Problems We used two small simulation problems to highlight the robustness of each method to increasing temporal delay in online training.

Original: The ﬁrst tasks is a simple ring of states, called
Revised: The ﬁrst tasks is a simple ring of states, called

Original: Cycle World .
Revised: Cycle World .

Original: On each timestep the agent deterministically transitions to the next state in the chain.
Revised: On each timestep the agent deterministically transitions to the next state in the chain.

Original: The agent’s observation is zero in every state, except the last.
Revised: The agent’s observation is zero in every state, except the last.

Original: The agent’s objective is to predict the next observation, which is difﬁcult without a memory of length equal to the length of the cycle.
Revised: The agent’s objective is to predict the next observation, which is difﬁcult without a memory of length equal to the length of the cycle.

Original: With a shorter memory, the agent cannot tell when the last non-zero observation, which is essential for predicting the next observation.
Revised: With a shorter memory, the agent cannot tell when the last non-zero observation, which is essential for predicting the next observation.

Original: This task has been used exclusively in benchmarking k-Markov methods, POMDPs, and predictive state representations (Tanner and Sutton, 2005).
Revised: This task has been used exclusively in benchmarking k-Markov methods, POMDPs, and predictive state representations (Tanner and Sutton, 2005).

Original: The complexity of the task can be easily varied, and yet the determinism ensures the variance does not introduce confounding factors.
Revised: The complexity of the task can be easily varied, and yet the determinism ensures the variance does not introduce confounding factors.

Original: At each time step, we measure the prediction accuracy for the next observation.
Revised: At each time step, we measure the prediction accuracy for the next observation.

Original: We also experimented with a stochastic prediction task, where correct prediction requires remembering two independent observation’s from the past.
Revised: We also experimented with a stochastic prediction task, where correct prediction requires remembering two independent observation’s from the past.

Original: In particular, the target on the next timestep is probabilistically dependent on the one-dimensional observation 15 timesteps ago and 30 timesteps ago.
Revised: In particular, the target on the next timestep is probabilistically dependent on the one-dimensional observation 15 timesteps ago and 30 timesteps ago.

Original: The dynamics are summarised in Table 1, in Appendix C.
Revised: The dynamics are summarised in Table 1, in Appendix C.

Original: This task is called Stochastic World .
Revised: This task is called Stochastic World .

Original: For this problem, a cross-entropy loss of 0.66 or higher indicates that the learned state did not capture the observation from either 15 or 30 steps in the past.
Revised: For this problem, a cross-entropy loss of 0.66 or higher indicates that the learned state did not capture the observation from either 15 or 30 steps in the past.

Original: If the state captures the observation fromtime-steps ago the cross entropy loss is about 0.51.
Revised: If the state captures the observation fromtime-steps ago the cross entropy loss is about 0.51.

Original: Optimal performance in this problem results in a cross-entropy loss is about 0.46.
Revised: Optimal performance in this problem results in a cross-entropy loss is about 0.46.

Original: Like Cycle World, Stochastic World requires a long and detailed memory of past observations, but the stochastic nature of the target pose an additional challenge.
Revised: Like Cycle World, Stochastic World requires a long and detailed memory of past observations, but the stochastic nature of the target pose an additional challenge.

Original: Real DataSets We also performed experiments on two ﬁxed datasets, to gain insights into how each method performed on better known benchmark tasks.
Revised: Real DataSets We also performed experiments on two ﬁxed datasets, to gain insights into how each method performed on better known benchmark tasks.

Original: In both cases the data was processed and performance evaluated in an online fashion.
Revised: In both cases the data was processed and performance evaluated in an online fashion.

Original: The ﬁrst problem is Sequential MNIST.
Revised: The ﬁrst problem is Sequential MNIST.

Original: The objective is to classify numerical digits based on a stream of pixel inputs.
Revised: The objective is to classify numerical digits based on a stream of pixel inputs.

Original: On each timestep the input is one row (1x28) of the image, and the target is the label of the image.
Revised: On each timestep the input is one row (1x28) of the image, and the target is the label of the image.

Original: We used an RNN architecture withhidden units as in previous work (Arjovsky et al. , 2015).
Revised: We used an RNN architecture withhidden units as in previous work (Arjovsky et al. , 2015).

Original: It is not possible to predict the target image base on a few samples, so we wait until 15 steps (corresponding to 15x28 pixels) to begin measuring the error.
Revised: It is not possible to predict the target image base on a few samples, so we wait until 15 steps (corresponding to 15x28 pixels) to begin measuring the error.

Original: Here, we report these incorrect predictions for the last 15 time-steps for every image.
Revised: Here, we report these incorrect predictions for the last 15 time-steps for every image.

Original: We ran this on 1000 images, which correspond to 28000 steps.
Revised: We ran this on 1000 images, which correspond to 28000 steps.

Original: Finally, we also include results on a a character prediction problem called Penn Tree Bank dataset.
Revised: Finally, we also include results on a a character prediction problem called Penn Tree Bank dataset.

Original: This problem is relevant because language modelling remains an important application of recurrent learning systems, and robust performance on this dataset can provide insight into the utility of our new method in application.
Revised: This problem is relevant because language modelling remains an important application of recurrent learning systems, and robust performance on this dataset can provide insight into the utility of our new method in application.

Original: We used a vocabulary size of 10000.
Revised: We used a vocabulary size of 10000.

Original: The Target Loss function used here is a weighted cross-entropy loss for a sequence of logits.
Revised: The Target Loss function used here is a weighted cross-entropy loss for a sequence of logits.

Original: We used an LSTM with 200 hidden nodes as this architecture was found to perform well in previous work (Zaremba et al. , 2014).
Revised: We used an LSTM with 200 hidden nodes as this architecture was found to perform well in previous work (Zaremba et al. , 2014).

Original: Comparison to T-BPTT
Revised: Comparison to T-BPTT

Original: We compare FPP to T-BPTT for varying truncation levels.
Revised: We compare FPP to T-BPTT for varying truncation levels.

Original: For all the algorithms, we used a constant buffer size of 100 and the trajectory length T for both T-BPTT(overlap and no overlap versions) and FPP.
Revised: For all the algorithms, we used a constant buffer size of 100 and the trajectory length T for both T-BPTT(overlap and no overlap versions) and FPP.

Original: All algorithms use O ( T ) computation per step.
Revised: All algorithms use O ( T ) computation per step.

Original: For overlap
Revised: For overlap

Original: T-BPTT, we employ T-BPTT online by taking each observation and updating with respect to the loss at that time-step, using a T-step truncated gradient.
Revised: T-BPTT, we employ T-BPTT online by taking each observation and updating with respect to the loss at that time-step, using a T-step truncated gradient.

Original: The no-overlap version of T-BPTT performs a batch update for every T observations, such that these observations do not overlap.
Revised: The no-overlap version of T-BPTT performs a batch update for every T observations, such that these observations do not overlap.

Original: Additionally, we include UORO (Tallec and Ollivier, 2017) as another baseline.
Revised: Additionally, we include UORO (Tallec and Ollivier, 2017) as another baseline.

Original: UORO uses an unbiased rank-1 approximation to approximate RTRL.
Revised: UORO uses an unbiased rank-1 approximation to approximate RTRL.

Original: It is a relatively new method for training RNN online, and has only been tested on small scale datasets.
Revised: It is a relatively new method for training RNN online, and has only been tested on small scale datasets.

Original: In our experiment, we include memoryand rank- 1 UORO in Cycleworld and StochasticWorld.
Revised: In our experiment, we include memoryand rank- 1 UORO in Cycleworld and StochasticWorld.

Original: We ﬁrst compare the performances of FPP and T-BPTT on Cycleworld with varying p.
Revised: We ﬁrst compare the performances of FPP and T-BPTT on Cycleworld with varying p.

Original: We expect
Revised: We expect

Original: T-BPTT to degrade with T less than the dependence back in time (the length of the cycle p ); we therefore test both T = p and T =
Revised: T-BPTT to degrade with T less than the dependence back in time (the length of the cycle p ); we therefore test both T = p and T =

Original: p/ 2 for increasing p .
Revised: p/ 2 for increasing p .

Original: To make the results comparable across p , we report performance as the ratio to a simple baseline of predicting 0 at every time step.
Revised: To make the results comparable across p , we report performance as the ratio to a simple baseline of predicting 0 at every time step.

Original: From Figure 2, we can see that FPP is more robust to T , whereas T-BPTT with T =
Revised: From Figure 2, we can see that FPP is more robust to T , whereas T-BPTT with T =

Original: p/ 2 performs poorly even when given more data (Figure 2(b)).
Revised: p/ 2 performs poorly even when given more data (Figure 2(b)).

Original: In early learning, with fewer samples, FPP has an even more clear advantage.
Revised: In early learning, with fewer samples, FPP has an even more clear advantage.

Original: Even though T-BPTT can eventually learn optimal predictions for T = p , it takes longer than FPP which learns near optimal predictions in early learning (Figure 2(a)).
Revised: Even though T-BPTT can eventually learn optimal predictions for T = p , it takes longer than FPP which learns near optimal predictions in early learning (Figure 2(a)).

Original: We additionally compare FPP and the two variants of T-BPTT across all four problems, under different settings of T , shown in Figure 3.
Revised: We additionally compare FPP and the two variants of T-BPTT across all four problems, under different settings of T , shown in Figure 3.

Original: Across all problems, FPP outperforms the other two for every T , except T = 1 in CycleWorld where all three methods perform similarly.
Revised: Across all problems, FPP outperforms the other two for every T , except T = 1 in CycleWorld where all three methods perform similarly.

Original: The performance of FPP is notably better for smaller T , as compared to T-BPTT.
Revised: The performance of FPP is notably better for smaller T , as compared to T-BPTT.

Original: For example, in Figure 3(b) 20-BPTT has a high loss and is unable to learn both the dependencies, whereas FPP with T=20, performs almost as well as 40-BPTT.
Revised: For example, in Figure 3(b) 20-BPTT has a high loss and is unable to learn both the dependencies, whereas FPP with T=20, performs almost as well as 40-BPTT.

Original: Similar conclusions can be made for T ∈ { 3 , 5 } in (a), T ∈
Revised: Similar conclusions can be made for T ∈ { 3 , 5 } in (a), T ∈

Original: Beneﬁts of mini-batch updates and multiple updates per step One of the advantages of using a buffer is the ability to perform mini-batch updates and multiple updates per step.
Revised: Beneﬁts of mini-batch updates and multiple updates per step One of the advantages of using a buffer is the ability to perform mini-batch updates and multiple updates per step.

Original: We evaluate the performance of FPP with and without state updates using M updates per step and a mini-batch of size B. We show the performance with varying T .
Revised: We evaluate the performance of FPP with and without state updates using M updates per step and a mini-batch of size B. We show the performance with varying T .

Original: To show the effect of multiple update, we ﬁx B and vary M ∈ { 1 , 2 , 4 , 8 , 16 } .
Revised: To show the effect of multiple update, we ﬁx B and vary M ∈ { 1 , 2 , 4 , 8 , 16 } .

Original: To show the effect of mini-batch update, we ﬁx M and vary
Revised: To show the effect of mini-batch update, we ﬁx M and vary

Original: B ∈ { 1 , 2 , 4 , 8 , 16 } .
Revised: B ∈ { 1 , 2 , 4 , 8 , 16 } .

Original: We use a buffer size of 1000 and 10000 training steps.
Revised: We use a buffer size of 1000 and 10000 training steps.

Original: We also include FPP without state updating, to determine if the beneﬁts of FPP are mainly due to using a buffer rather than due to the new objective to learn explicit state variables.
Revised: We also include FPP without state updating, to determine if the beneﬁts of FPP are mainly due to using a buffer rather than due to the new objective to learn explicit state variables.

Original: We particularly expect FPP to outperform FPP without state updating under more updates per step, because we showed converge for FPP on a ﬁxed buffer whereas no such result exists for FPP without state
Revised: We particularly expect FPP to outperform FPP without state updating under more updates per step, because we showed converge for FPP on a ﬁxed buffer whereas no such result exists for FPP without state

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: updating.
Revised: updating.

Original: Here, the buffer is not ﬁxed, but performing more updates per step should move the FPP solution closer to a stationary point of the current buffer.
Revised: Here, the buffer is not ﬁxed, but performing more updates per step should move the FPP solution closer to a stationary point of the current buffer.

Original: Figure 4 (a) and (b) shows the effect of multiple updates and (c) and (d) the effect of mini-batch updates.
Revised: Figure 4 (a) and (b) shows the effect of multiple updates and (c) and (d) the effect of mini-batch updates.

Original: For both, increasing the number of updates and the size of the mini-batch improves performance, except for a bit of overﬁtting we observed in Stochastic World for increasing updates ( B = 1 , M = 16 ).
Revised: For both, increasing the number of updates and the size of the mini-batch improves performance, except for a bit of overﬁtting we observed in Stochastic World for increasing updates ( B = 1 , M = 16 ).

Original: However, in general, FPP can better take advantage of both multiple updates and mini-batch updating.
Revised: However, in general, FPP can better take advantage of both multiple updates and mini-batch updating.

Original: The most noticeable gaps are for T = 16 and T = 32 in StochasticWorld and T = 1 and T = 2 in CycleWorld.
Revised: The most noticeable gaps are for T = 16 and T = 32 in StochasticWorld and T = 1 and T = 2 in CycleWorld.

Original: The theory suggests that more updates, even with T = 1 , should allow FPP to converge to a reasonable solution.
Revised: The theory suggests that more updates, even with T = 1 , should allow FPP to converge to a reasonable solution.

Original: We test this on CycleWorld (with Figure 7 in
Revised: We test this on CycleWorld (with Figure 7 in

Original: Appendix C), and ﬁnd that for both larger mini-batch and number of updates FPP can get the error down to zero, whereas FPP without state updating cannot.
Revised: Appendix C), and ﬁnd that for both larger mini-batch and number of updates FPP can get the error down to zero, whereas FPP without state updating cannot.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The main objective of this paper is to reformulate RNN training to explicitly learn state variables.
Revised: The main objective of this paper is to reformulate RNN training to explicitly learn state variables.

Original: In particular, the goal is to investigate methods that can better distribute computation, and improve state updating without having to compute expensive—and potentially unstable—gradients back in time for each state.
Revised: In particular, the goal is to investigate methods that can better distribute computation, and improve state updating without having to compute expensive—and potentially unstable—gradients back in time for each state.

Original: We introduce a new objective to explicitly learn state variables for RNNs, which breaks gradient dependence back in time.
Revised: We introduce a new objective to explicitly learn state variables for RNNs, which breaks gradient dependence back in time.

Original: The choice of T to compute gradients back in time is used only to improve training speed, rather than to effectively approximate gradients.
Revised: The choice of T to compute gradients back in time is used only to improve training speed, rather than to effectively approximate gradients.

Original: We found that our algorithm, called FPP, was indeed more robust to T , than truncated BPTT was to its truncation level.
Revised: We found that our algorithm, called FPP, was indeed more robust to T , than truncated BPTT was to its truncation level.

Original: We proved that our algorithm converges to a stationary point, under a ﬁxed buffer, and so is a sound approach to using a buffer to train RNNs.
Revised: We proved that our algorithm converges to a stationary point, under a ﬁxed buffer, and so is a sound approach to using a buffer to train RNNs.

Original: Further, we chose simple optimization choices in this work; there are clear next steps for beneﬁting more from the decoupled update, such as by parallelizing updates across state variables.
Revised: Further, we chose simple optimization choices in this work; there are clear next steps for beneﬁting more from the decoupled update, such as by parallelizing updates across state variables.

Original: Overall, this work provides evidence that FPP could be a promising direction for robustly training RNNs, without the need to compute or approximate long gradients back in time.
Revised: Overall, this work provides evidence that FPP could be a promising direction for robustly training RNNs, without the need to compute or approximate long gradients back in time.

Original: A F ULL P ROOFS A. C ONVERGENCE ON A F IXED B UFFER
Revised: A F ULL P ROOFS A. C ONVERGENCE ON A F IXED B UFFER

Original: At ﬁrst glance, the update (9) is different than the update in Ghadimi et al.
Revised: At ﬁrst glance, the update (9) is different than the update in Ghadimi et al.

Original: (2016, p. 276).
Revised: (2016, p. 276).

Original: Nevertheless, the following lemma guarantees that they are indeed the same.
Revised: Nevertheless, the following lemma guarantees that they are indeed the same.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let f be L or a stochastic sample of L and let α > 0 .
Revised: Let f be L or a stochastic sample of L and let α > 0 .

Original: Write x =
Revised: Write x =

Original: ( W , β , S ) .
Revised: ( W , β , S ) .

Original: Then arg min u
Revised: Then arg min u

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is a straightforward calculation.
Revised: The proof is a straightforward calculation.

Original: [Equation]
Revised: [Equation]

Original: Our goal is to apply Corollary 3 of Ghadimi et al.
Revised: Our goal is to apply Corollary 3 of Ghadimi et al.

Original: (2016, p. 282).
Revised: (2016, p. 282).

Original: We must show that ∇ g is Lipschitz on C and demonstrate that Assumption A1 in Ghadimi et al.
Revised: We must show that ∇ g is Lipschitz on C and demonstrate that Assumption A1 in Ghadimi et al.

Original: (2016, p. 268) holds.
Revised: (2016, p. 268) holds.

Original: Lemma 2.
Revised: Lemma 2.

Original: ∇ L is Lipschitz on C .
Revised: ∇ L is Lipschitz on C .

Original: Proof.
Revised: Proof.

Original: A function is Lipschitz if it gradient is bounded.
Revised: A function is Lipschitz if it gradient is bounded.

Original: Since L is smooth and C is compact (continuous functions on compact sets are bounded), this lemma follows.
Revised: Since L is smooth and C is compact (continuous functions on compact sets are bounded), this lemma follows.

Original: Lemma 3.
Revised: Lemma 3.

Original: ∇ L k is an unbiased estimate of ∇ L , where k ∼ uniform-
Revised: ∇ L k is an unbiased estimate of ∇ L , where k ∼ uniform-

Original: ( T, n ) .
Revised: ( T, n ) .

Original: Proof.
Revised: Proof.

Original: The terms in ∇ L k corresponding to the gradients of W and β are exactly ∇ W L and ∇ β L in expectation, given that k ∼ uniform- ( T, n ) .
Revised: The terms in ∇ L k corresponding to the gradients of W and β are exactly ∇ W L and ∇ β L in expectation, given that k ∼ uniform- ( T, n ) .

Original: Let us consider the gradient elements corresponding to the parameters s 0: n .
Revised: Let us consider the gradient elements corresponding to the parameters s 0: n .

Original: For shorthand, deﬁne [ a : b ] :=
Revised: For shorthand, deﬁne [ a : b ] :=

Original: { a, a + 1 , · · · , b − 1 , b } .
Revised: { a, a + 1 , · · · , b − 1 , b } .

Original: Deﬁne P 0 :=
Revised: Deﬁne P 0 :=

Original: [0 : n − T ] , P 1 :=
Revised: [0 : n − T ] , P 1 :=

Original: [ T : n ] .
Revised: [ T : n ] .

Original: If j ∈ P 0 , then s j predicts future states.
Revised: If j ∈ P 0 , then s j predicts future states.

Original: If s j ∈ P 1 , then s j is predicted by other states in the regularizer terms of L .
Revised: If s j ∈ P 1 , then s j is predicted by other states in the regularizer terms of L .

Original: Note that P 0 and P 1 are not disjoint.
Revised: Note that P 0 and P 1 are not disjoint.

Original: First, we calculate.
Revised: First, we calculate.

Original: [Equation]
Revised: [Equation]

Original: 1 , then s j does not show up as the target (i.e., the term that is not ˆ s k ) in any regularizer term of L .
Revised: 1 , then s j does not show up as the target (i.e., the term that is not ˆ s k ) in any regularizer term of L .

Original: Hence, ∇ s j L k is zero with probability 1 − 1 n − T +1 , and is ( ∇ ˆ s j + T (cid:96) β
Revised: Hence, ∇ s j L k is zero with probability 1 − 1 n − T +1 , and is ( ∇ ˆ s j + T (cid:96) β

Original: (ˆ s j + T ; y j + T )
Revised: (ˆ s j + T ; y j + T )

Original: − λ
Revised: − λ

Original: ( s j + T − ˆ s j + T ))
Revised: ( s j + T − ˆ s j + T ))

Original: (cid:62) ∇ s j ˆ s j with probability 1 n − T +1 .
Revised: (cid:62) ∇ s j ˆ s j with probability 1 n − T +1 .

Original: If j ∈ P (cid:123) 0 ∩ P 1 , then s j only shows up as a target in a regularizer term, so ∇ s j L k is zero with probability 1 − 1 n − T
Revised: If j ∈ P (cid:123) 0 ∩ P 1 , then s j only shows up as a target in a regularizer term, so ∇ s j L k is zero with probability 1 − 1 n − T

Original: +

Revised: +

Original: and is otherwise 1 n − T +

Revised: and is otherwise 1 n − T +

Original: λ ( s j − ˆ s j ) .
Revised: λ ( s j − ˆ s j ) .

Original: If j ∈ P 0 ∩ P 1 , then ∇ s j L k is zero with probability 1 − 2 n − T +1 , λ ( s j − ˆ s j ) with probability 1 n − T +1 , and ( ∇ ˆ s j + T (cid:96) β ( ˆ s j + T ; y j + T ) − λ
Revised: If j ∈ P 0 ∩ P 1 , then ∇ s j L k is zero with probability 1 − 2 n − T +1 , λ ( s j − ˆ s j ) with probability 1 n − T +1 , and ( ∇ ˆ s j + T (cid:96) β ( ˆ s j + T ; y j + T ) − λ

Original: ( s j + T − ˆ s j + T ))
Revised: ( s j + T − ˆ s j + T ))

Original: (cid:62) ∇ s j ˆ s j with probability 1 n − T +1 .
Revised: (cid:62) ∇ s j ˆ s j with probability 1 n − T +1 .

Original: The case for j ∈ P (cid:123) 

Revised: The case for j ∈ P (cid:123)

Original: ∩ P (cid:123) 1 is trivial.
Revised: ∩ P (cid:123) 1 is trivial.

Original: Consequently, E [ ∇ s j L k ] = ∇ s j L for all j ∈ { 0 , · · · , n } .
Revised: Consequently, E [ ∇ s j L k ] = ∇ s j L for all j ∈ { 0 , · · · , n } .

Original: Lemma 4.
Revised: Lemma 4.

Original: The variance of ∇ L k is bounded on C .
Revised: The variance of ∇ L k is bounded on C .

Original: Proof.
Revised: Proof.

Original: This follows because ∇ L k and ∇ L are both smooth functions on a compact set C , and thus bounded.
Revised: This follows because ∇ L k and ∇ L are both smooth functions on a compact set C , and thus bounded.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let D be a Lipschitz constant of ∇ L ( W , β , S ) .
Revised: Let D be a Lipschitz constant of ∇ L ( W , β , S ) .

Original: Deﬁne probability mass functions
Revised: Deﬁne probability mass functions

Original: [Equation]
Revised: [Equation]

Original: for each N ∈ N .
Revised: for each N ∈ N .

Original: Let R be distributed according to P N .
Revised: Let R be distributed according to P N .

Original: Assume α t = 12 D for all t and that we perform N stochastic updates.
Revised: Assume α t = 12 D for all t and that we perform N stochastic updates.

Original: Write x R =
Revised: Write x R =

Original: ( W R , β R , S R ) .
Revised: ( W R , β R , S R ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The g X,R (deﬁned in Ghadimi et al.
Revised: The g X,R (deﬁned in Ghadimi et al.

Original: (2016, p. 271, 274))
Revised: (2016, p. 271, 274))

Original: in Corollary 3 of Ghadimi et al.
Revised: in Corollary 3 of Ghadimi et al.

Original: (2016, p. 282) corresponds in our case to the following.
Revised: (2016, p. 282) corresponds in our case to the following.

Original: [Equation]
Revised: [Equation]

Original: In the last line, we use Lemma 1.
Revised: In the last line, we use Lemma 1.

Original: Since we project based on squared norm distance in (8) (corresponding to ω ( x ) = 12 (cid:107) x (cid:107) 22 in Ghadimi et al.
Revised: Since we project based on squared norm distance in (8) (corresponding to ω ( x ) = 12 (cid:107) x (cid:107) 22 in Ghadimi et al.

Original: (2016)), the α in Ghadimi et al.
Revised: (2016)), the α in Ghadimi et al.

Original: (2016, p. 271) (not our step-size α t ) can be set to 1.
Revised: (2016, p. 271) (not our step-size α t ) can be set to 1.

Original: After applying our Lemma 2, Lemma 3, and Lemma 4, we have from Corollary 3 of Ghadimi et al.
Revised: After applying our Lemma 2, Lemma 3, and Lemma 4, we have from Corollary 3 of Ghadimi et al.

Original: (2016, p. 282)
Revised: (2016, p. 282)

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: The only thing left to check is that Γ( x R − α R ∇ L ( x R
Revised: The only thing left to check is that Γ( x R − α R ∇ L ( x R

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: x R
Revised: x R

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: A. R ECOVERY OF RNN S OLUTIONS
Revised: A. R ECOVERY OF RNN S OLUTIONS

Original: Recall our goal is to compare to the RNN solutions of (10).
Revised: Recall our goal is to compare to the RNN solutions of (10).

Original: [Equation]
Revised: [Equation]

Original: Let us also write a constrained version of the above problem, which we will use in the analysis of
Revised: Let us also write a constrained version of the above problem, which we will use in the analysis of

Original: FPP.
Revised: FPP.

Original: [Equation]
Revised: [Equation]

Original: The idea is that FPP can be viewed as a way to solve the problem (12) and thus (10) through quadratic regularization.
Revised: The idea is that FPP can be viewed as a way to solve the problem (12) and thus (10) through quadratic regularization.

Original: We will use s 0: n as shorthand for { s 0 , · · · , s n } , which in the main paper we labeled as S , but for this proof it will be convenient to use explicit variables.
Revised: We will use s 0: n as shorthand for { s 0 , · · · , s n } , which in the main paper we labeled as S , but for this proof it will be convenient to use explicit variables.

Original: Deﬁne the feasible set of (12) as
Revised: Deﬁne the feasible set of (12) as

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: Let ( W ∗ , β ∗ , s ∗ 0 ) be a local min of (10) .
Revised: Let ( W ∗ , β ∗ , s ∗ 0 ) be a local min of (10) .

Original: For 1 ≤ i ≤ n , deﬁne recursively s ∗ i :=
Revised: For 1 ≤ i ≤ n , deﬁne recursively s ∗ i :=

Original: f W ∗ ( s ∗ i − 1 , o i ) .
Revised: f W ∗ ( s ∗ i − 1 , o i ) .

Original: Then ( W ∗ , β ∗ , s ∗ 0: n ) is a local min of (12) .
Revised: Then ( W ∗ , β ∗ , s ∗ 0: n ) is a local min of (12) .

Original: Let ( W ∗ , β ∗ , s ∗ 0: n ) be a local min of (12) .
Revised: Let ( W ∗ , β ∗ , s ∗ 0: n ) be a local min of (12) .

Original: Then ( W ∗ , β ∗ , s ∗ 0 ) is a local min of (10) .
Revised: Then ( W ∗ , β ∗ , s ∗ 0 ) is a local min of (10) .

Original: Proof.
Revised: Proof.

Original: First, let N ⊂ R w + b + k be a neighbourhood of ( W ∗ , β ∗ , s ∗ 0 ) such that ∀ ( W , β , s 0 ) ∈
Revised: First, let N ⊂ R w + b + k be a neighbourhood of ( W ∗ , β ∗ , s ∗ 0 ) such that ∀ ( W , β , s 0 ) ∈

Original: N , we have
Revised: N , we have

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, we may take N to be open.
Revised: Without loss of generality, we may take N to be open.

Original: Otherwise, by deﬁnition of a neighbourhood, we may take a smaller open set around ( W ∗ , β ∗ , s ∗ 0 ) by deﬁnition of a neighbourhood and call that set N .
Revised: Otherwise, by deﬁnition of a neighbourhood, we may take a smaller open set around ( W ∗ , β ∗ , s ∗ 0 ) by deﬁnition of a neighbourhood and call that set N .

Original: Let s ∗ i be deﬁned as above.
Revised: Let s ∗ i be deﬁned as above.

Original: Deﬁne M :=
Revised: Deﬁne M :=

Original: N × R nk , which is an open neighbourhood of ( W ∗ , β ∗ , s ∗ 0 ) since N is open.
Revised: N × R nk , which is an open neighbourhood of ( W ∗ , β ∗ , s ∗ 0 ) since N is open.

Original: Let ( W , β , s 0: n ) ∈ M ∩ Ω .
Revised: Let ( W , β , s 0: n ) ∈ M ∩ Ω .

Original: Note that ( W , β , s 0 ) ∈ N .
Revised: Note that ( W , β , s 0 ) ∈ N .

Original: By deﬁnition of Ω , we have that f W ( s i − 1 , o i ) =
Revised: By deﬁnition of Ω , we have that f W ( s i − 1 , o i ) =

Original: s i .
Revised: s i .

Original: Hence, E fpp ( W , β , s 0: n ) =
Revised: Hence, E fpp ( W , β , s 0: n ) =

Original: E ( W , β , s 0 ) .
Revised: E ( W , β , s 0 ) .

Original: By deﬁnition of (12), (10), we have E ( W ∗ , β ∗ , s ∗ 0 ) =
Revised: By deﬁnition of (12), (10), we have E ( W ∗ , β ∗ , s ∗ 0 ) =

Original: E fpp ( W ∗ , β ∗ , s ∗ 0: n ) .
Revised: E fpp ( W ∗ , β ∗ , s ∗ 0: n ) .

Original: Finally,
Revised: Finally,

Original: [Equation]
Revised: [Equation]

Original: For the second part of the proof, assume ( W ∗ , β ∗ , s ∗ 0: n ) is a local min of (12), meaning there is a neigbourhood M ⊂ R w + b +( n +1) k of ( W ∗ , β ∗ , s ∗ 0: n ) such that for every ( W , β , s 0: n ) ∈ M ∩ Ω ,
Revised: For the second part of the proof, assume ( W ∗ , β ∗ , s ∗ 0: n ) is a local min of (12), meaning there is a neigbourhood M ⊂ R w + b +( n +1) k of ( W ∗ , β ∗ , s ∗ 0: n ) such that for every ( W , β , s 0: n ) ∈ M ∩ Ω ,

Original: [Equation]
Revised: [Equation]

Original: Similarly, without loss of generality, we can assume that M is an open ball, so we may write for some (cid:15) > 0 , M = B (cid:15) ( W ∗ , β ∗ , s ∗ 0: n ) .
Revised: Similarly, without loss of generality, we can assume that M is an open ball, so we may write for some (cid:15) > 0 , M = B (cid:15) ( W ∗ , β ∗ , s ∗ 0: n ) .

Original: We will construct an open set N ⊂ R w + b + k such that ( W ∗ , β ∗ , s ∗ 0 ) is a local min with respect to N .
Revised: We will construct an open set N ⊂ R w + b + k such that ( W ∗ , β ∗ , s ∗ 0 ) is a local min with respect to N .

Original: Deﬁne the projection π onto the ﬁrst w + b + k indices.
Revised: Deﬁne the projection π onto the ﬁrst w + b + k indices.

Original: Deﬁne N := π ( M ∩ Ω) .
Revised: Deﬁne N := π ( M ∩ Ω) .

Original: Let us show that
Revised: Let us show that

Original: N is open.
Revised: N is open.

Original: We will write f W ( s 0: n − 1 , o 1: n ) to mean { f W ( s 0 , o 1 ) , · · · , f W ( f W ( · · · ( s 0 , o 1 ) , o 2 ) , · · · , o n )) } .
Revised: We will write f W ( s 0: n − 1 , o 1: n ) to mean { f W ( s 0 , o 1 ) , · · · , f W ( f W ( · · · ( s 0 , o 1 ) , o 2 ) , · · · , o n )) } .

Original: We can write N as
Revised: We can write N as

Original: [Equation]
Revised: [Equation]

Original: On the second line, we used the fact that s i =
Revised: On the second line, we used the fact that s i =

Original: f W ( s i − 1 , o i ) in Ω .
Revised: f W ( s i − 1 , o i ) in Ω .

Original: Since the norm and f are continuous and ( −∞ , (cid:15) ) is open, we have that N , a continuous preimage of an open set, is open.
Revised: Since the norm and f are continuous and ( −∞ , (cid:15) ) is open, we have that N , a continuous preimage of an open set, is open.

Original: Now, let ( W , β , s 0 ) ∈
Revised: Now, let ( W , β , s 0 ) ∈

Original: N such that ∃ s 1: n with ( W , β , s 0: n ) ∈ M ∩ Ω .
Revised: N such that ∃ s 1: n with ( W , β , s 0: n ) ∈ M ∩ Ω .

Original: [Equation]
Revised: [Equation]

Original: The claim follows.
Revised: The claim follows.

Original: Proposition 2.
Revised: Proposition 2.

Original: The ﬁrst order KKT equations for (10) and for (12) are the same.
Revised: The ﬁrst order KKT equations for (10) and for (12) are the same.

Original: Proof.
Revised: Proof.

Original: Given ( W ∗ , β ∗ , s ∗ 0 ) , for 1 ≤ i ≤ n deﬁne ˜
Revised: Given ( W ∗ , β ∗ , s ∗ 0 ) , for 1 ≤ i ≤ n deﬁne ˜

Original: s i :
Revised: s i :

Original: = f W (˜ s i − 1 , o i ) , where ˜ s 0 :
Revised: = f W (˜ s i − 1 , o i ) , where ˜ s 0 :

Original: =
Revised: =

Original: s ∗ 0 .
Revised: s ∗ 0 .

Original: If we write ∂f
Revised: If we write ∂f

Original: W ∗ (˜ s l , o l +1 ) ∂
Revised: W ∗ (˜ s l , o l +1 ) ∂

Original: s l for instance, this is taken to mean the gradient of f W ∗ (˜ s l , o l +1 ) with respect to the function arguments corresponding to ˜ s l .
Revised: s l for instance, this is taken to mean the gradient of f W ∗ (˜ s l , o l +1 ) with respect to the function arguments corresponding to ˜ s l .

Original: Furthermore, when writing ∂f W ∗ (˜ s j , o j +1 ) ∂ W , we only mean the gradient with respect to the parameters of the outer f W ∗ , and not with respect to any of the parameters of ˜ s j .
Revised: Furthermore, when writing ∂f W ∗ (˜ s j , o j +1 ) ∂ W , we only mean the gradient with respect to the parameters of the outer f W ∗ , and not with respect to any of the parameters of ˜ s j .

Original: Using the chain rule for the ﬁrst and third equations below, the ﬁrst order KKT conditions for (10) are given by
Revised: Using the chain rule for the ﬁrst and third equations below, the ﬁrst order KKT conditions for (10) are given by

Original: [Equation]
Revised: [Equation]

Original: The Lagrangian for (12) is
Revised: The Lagrangian for (12) is

Original: [Equation]
Revised: [Equation]

Original: where λ i ∈
Revised: where λ i ∈

Original: R k for 1 ≤ i ≤ n are Lagrange multipliers.
Revised: R k for 1 ≤ i ≤ n are Lagrange multipliers.

Original: We deﬁne λ 0 := 0 for convenience.
Revised: We deﬁne λ 0 := 0 for convenience.

Original: The
Revised: The

Original: KKT equations for (16) are
Revised: KKT equations for (16) are

Original: [Equation]
Revised: [Equation]

Original: First, let us ﬁnd a closed-form expression for λ
Revised: First, let us ﬁnd a closed-form expression for λ

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Lemma 1.
Revised: Lemma 1.

Original: Let 0 ≤ j ≤ n .
Revised: Let 0 ≤ j ≤ n .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: Proof.
Revised: Proof.

Original: We proceed by induction.
Revised: We proceed by induction.

Original: The base case and the case j = n − 1 are trivial.
Revised: The base case and the case j = n − 1 are trivial.

Original: Assume the claim is true for m + 1 > 0 .
Revised: Assume the claim is true for m + 1 > 0 .

Original: We will show the claim for j = m .
Revised: We will show the claim for j = m .

Original: Using the KKT equations (17) and the induction hypothesis,
Revised: Using the KKT equations (17) and the induction hypothesis,

Original: [Equation]
Revised: [Equation]

Original: Tm :=
Revised: Tm :=

Original: [Equation]
Revised: [Equation]

Original: (cid:18) ∂(cid:96) β
Revised: (cid:18) ∂(cid:96) β

Original: [Equation]
Revised: [Equation]

Original: ( f W ∗ ( s ∗ m , o m +1 ); y m +1 )
Revised: ( f W ∗ ( s ∗ m , o m +1 ); y m +1 )

Original: ∂f W −
Revised: ∂f W −

Original: [Equation]
Revised: [Equation]

Original: Tm
Revised: Tm

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: W ∗ ( s ∗ m , o m +1 ) ∂ s m = − (cid:32) ∂(cid:96)
Revised: W ∗ ( s ∗ m , o m +1 ) ∂ s m = − (cid:32) ∂(cid:96)

Original: β ∗ ( f W ∗ ( s ∗ m , o m +1 ); y m +1 )
Revised: β ∗ ( f W ∗ ( s ∗ m , o m +1 ); y m +1 )

Original: ∂f W + n − 1 (cid:88) i
Revised: ∂f W + n − 1 (cid:88) i

Original: = m
Revised: = m

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: β ∗ ( f W ∗ ( s ∗
Revised: β ∗ ( f W ∗ ( s ∗

Original: i , o i +1 )
Revised: i , o i +1 )

Original: ; y i +1 )
Revised: ; y i +1 )

Original: [Equation]
Revised: [Equation]

Original: W i (cid:89) l
Revised: W i (cid:89) l

Original: = m + ∂f W ∗ ( s ∗ l , o l +1 ) ∂ s l (cid:33)
Revised: = m + ∂f W ∗ ( s ∗ l , o l +1 ) ∂ s l (cid:33)

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: m , o m +1 )
Revised: m , o m +1 )

Original: [Equation]
Revised: [Equation]

Original: s m =
Revised: s m =

Original: [Equation]
Revised: [Equation]

Original: ( f
Revised: ( f

Original: [Equation]
Revised: [Equation]

Original: ( s ∗ m
Revised: ( s ∗ m

Original: [Equation]
Revised: [Equation]

Original: o m
Revised: o m

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: y m
Revised: y m

Original: [Equation]
Revised: [Equation]

Original: 1 ) ∂f W ∂f W
Revised: 1 ) ∂f W ∂f W

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: o m +1 )
Revised: o m +1 )

Original: ∂
Revised: ∂

Original: s m − n
Revised: s m − n

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i
Revised: (cid:88) i

Original: = m
Revised: = m

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ( f
Revised: ( f

Original: [Equation]
Revised: [Equation]

Original: ∗ ( s
Revised: ∗ ( s

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: o
Revised: o

Original: [Equation]
Revised: [Equation]

Original: +1 )
Revised: +1 )

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: 1 ) ∂f W i (cid:89) l
Revised: 1 ) ∂f W i (cid:89) l

Original: [Equation]
Revised: [Equation]

Original: m +
Revised: m +

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: o l +1 ) ∂
Revised: o l +1 ) ∂

Original: s l
Revised: s l

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: o m +1 )
Revised: o m +1 )

Original: ∂ s m
Revised: ∂ s m

Original: = − (cid:32) n
Revised: = − (cid:32) n

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:88) i = m ∂(cid:96) β ∗ ( f W ∗ ( s ∗
Revised: 1 (cid:88) i = m ∂(cid:96) β ∗ ( f W ∗ ( s ∗

Original: i , o i +1 )
Revised: i , o i +1 )

Original: ; y i +1 )
Revised: ; y i +1 )

Original: ∂f W i (cid:89) l
Revised: ∂f W i (cid:89) l

Original: = m ∂f W ∗ ( s ∗ l , o l +1 ) ∂ s l (cid:33)
Revised: = m ∂f W ∗ ( s ∗ l , o l +1 ) ∂ s l (cid:33)

Original: Now, we will show that the sets of equations are the same.
Revised: Now, we will show that the sets of equations are the same.

Original: First, it is clear that the two equations involving gradients of β in (13) and (17) are the same given that the constraint must be satisﬁed in (17).
Revised: First, it is clear that the two equations involving gradients of β in (13) and (17) are the same given that the constraint must be satisﬁed in (17).

Original: Now consider the equations involving gradients with respect to W .
Revised: Now consider the equations involving gradients with respect to W .

Original: [Equation]
Revised: [Equation]

Original: ∂f W
Revised: ∂f W

Original: [Equation]
Revised: [Equation]

Original: W ∗ ( s ∗ i −
Revised: W ∗ ( s ∗ i −

Original: [Equation]
Revised: [Equation]

Original: o i ) ∂
Revised: o i ) ∂

Original: W + n (cid:88) i =   n − 1 (cid:88)
Revised: W + n (cid:88) i =   n − 1 (cid:88)

Original: j = i ∂(cid:96) β ∗ ( f W ∗ ( s ∗ j , o j +1 ); y j +1 )
Revised: j = i ∂(cid:96) β ∗ ( f W ∗ ( s ∗ j , o j +1 ); y j +1 )

Original: ∂f W j (cid:89) l
Revised: ∂f W j (cid:89) l

Original: = i ∂f W ∗ ( s ∗ l , o l +1 ) ∂
Revised: = i ∂f W ∗ ( s ∗ l , o l +1 ) ∂

Original: s l 
Revised: s l 

Original: 
Revised: 

Original: ∂f W ∗ ( s ∗ i − 1 , o i )
Revised: ∂f W ∗ ( s ∗ i − 1 , o i )

Original: ∂ W = n (cid:88) j =
Revised: ∂ W = n (cid:88) j =

Original: [Equation]
Revised: [Equation]

Original: y j )
Revised: y j )

Original: ∂f W
Revised: ∂f W

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: o
Revised: o

Original: [Equation]
Revised: [Equation]

Original: ) ∂ W + n
Revised: ) ∂ W + n

Original: [Equation]
Revised: [Equation]

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: i = ∂(cid:96)
Revised: i = ∂(cid:96)

Original: [Equation]
Revised: [Equation]

Original: ∗ ( f W ∗ ( s ∗ j , o j +1 ); y j +1 )
Revised: ∗ ( f W ∗ ( s ∗ j , o j +1 ); y j +1 )

Original: ∂f W j (cid:89) l
Revised: ∂f W j (cid:89) l

Original: = i ∂f W ∗ ( s ∗ l
Revised: = i ∂f W ∗ ( s ∗ l

Original: [Equation]
Revised: [Equation]

Original: o l +1 ) ∂ s l (cid:33)
Revised: o l +1 ) ∂ s l (cid:33)

Original: ∂f W
Revised: ∂f W

Original: [Equation]
Revised: [Equation]

Original: ( s ∗ i −
Revised: ( s ∗ i −

Original: [Equation]
Revised: [Equation]

Original: o i ) ∂ W = ∂(cid:96) β ∗ ( f W ∗ ( s ∗ 0 , o 1 ); y 1 )
Revised: o i ) ∂ W = ∂(cid:96) β ∗ ( f W ∗ ( s ∗ 0 , o 1 ); y 1 )

Original: ∂f W ∂f W ∗ ( s ∗
Revised: ∂f W ∂f W ∗ ( s ∗

Original: [Equation]
Revised: [Equation]

Original: o 1 ) ∂ W + n − 1 (cid:88)
Revised: o 1 ) ∂ W + n − 1 (cid:88)

Original: j = ∂(cid:96) β ∗ ( f W ∗ ( s ∗ j , o j +1 ); y j +1 )
Revised: j = ∂(cid:96) β ∗ ( f W ∗ ( s ∗ j , o j +1 ); y j +1 )

Original: ∂f W (cid:32)
Revised: ∂f W (cid:32)

Original: ∂f W ∗ ( s ∗ j , o j
Revised: ∂f W ∗ ( s ∗ j , o j

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: By substituting in the constraint equations s ∗ i = f W ∗ ( s ∗
Revised: By substituting in the constraint equations s ∗ i = f W ∗ ( s ∗

Original: i − 1 , o i )
Revised: i − 1 , o i )

Original: , this recovers exactly the gradient with respect to W in (13).
Revised: , this recovers exactly the gradient with respect to W in (13).

Original: Finally, consider the gradient with respect to s 0 .
Revised: Finally, consider the gradient with respect to s 0 .

Original: [Equation]
Revised: [Equation]

Original: This matches the corresponding equation in (13).
Revised: This matches the corresponding equation in (13).

Original: Proposition 3.
Revised: Proposition 3.

Original: Let ( W ∗ , β ∗ , s ∗ 0: n ) be a local min of (12) .
Revised: Let ( W ∗ , β ∗ , s ∗ 0: n ) be a local min of (12) .

Original: Write the constraints of (12) as a vector:
Revised: Write the constraints of (12) as a vector:

Original: [Equation]
Revised: [Equation]

Original: Index each element of h by h i .
Revised: Index each element of h by h i .

Original: Then the vectors ∇ h i ( W ∗ , s ∗ 0: n ) are linearly independent.
Revised: Then the vectors ∇ h i ( W ∗ , s ∗ 0: n ) are linearly independent.

Original: Proof.
Revised: Proof.

Original: In the following, we will write ∂ [ g ] l ∂x ij to mean the derivative of the l -th component of g with respect to the i -th component of x j .
Revised: In the following, we will write ∂ [ g ] l ∂x ij to mean the derivative of the l -th component of g with respect to the i -th component of x j .

Original: For compactness, write g ( i ) :
Revised: For compactness, write g ( i ) :

Original: = f W ∗ ( s ∗
Revised: = f W ∗ ( s ∗

Original: i − 1 , o i ) − s ∗ i .
Revised: i − 1 , o i ) − s ∗ i .

Original: We can write the Jacobian ∇ h ( W ∗ , s ∗ 0: n ) as
Revised: We can write the Jacobian ∇ h ( W ∗ , s ∗ 0: n ) as

Original: [Equation]
Revised: [Equation]

Original: =  
Revised: =  

Original: ∂ [ g (1)] 1 ∂ W· · · ∂ [ g (1)] 1 ∂ W w ∂ [ g (1)] 1 ∂s· · · ∂ [ g (1)] 1 ∂s k· · · ∂ [ g (1)] 1 ∂ s kn ...
Revised: ∂ [ g (1)] 1 ∂ W· · · ∂ [ g (1)] 1 ∂ W w ∂ [ g (1)] 1 ∂s· · · ∂ [ g (1)] 1 ∂s k· · · ∂ [ g (1)] 1 ∂ s kn ...

Original: ∂ [ g (1)] k ∂ W· · · ∂ [ g (1)] k ∂ W w ∂
Revised: ∂ [ g (1)] k ∂ W· · · ∂ [ g (1)] k ∂ W w ∂

Original: [ g (1)]
Revised: [ g (1)]

Original: k ∂s· · · ∂ [ g (1)] k ∂s k· · · ∂ [ g (1)] k ∂ s kn ∂ [ g (2)] 1 ∂ W· · · ∂ [ g (2)] 1 ∂ W w ∂ [ g (2)] 1 ∂s· · · ∂ [ g (2)] 1 ∂s k· · · ∂ [ g (2)] 1 ∂ s kn ...
Revised: k ∂s· · · ∂ [ g (1)] k ∂s k· · · ∂ [ g (1)] k ∂ s kn ∂ [ g (2)] 1 ∂ W· · · ∂ [ g (2)] 1 ∂ W w ∂ [ g (2)] 1 ∂s· · · ∂ [ g (2)] 1 ∂s k· · · ∂ [ g (2)] 1 ∂ s kn ...

Original: ∂ [ g ( n )] k ∂ W· · · ∂ [ g ( n )] k ∂ W w ∂
Revised: ∂ [ g ( n )] k ∂ W· · · ∂ [ g ( n )] k ∂ W w ∂

Original: [ g ( n )] k ∂s· · · ∂ [ g ( n )]
Revised: [ g ( n )] k ∂s· · · ∂ [ g ( n )]

Original: k ∂s k· · · ∂ [ g ( n )] k ∂ s kn 
Revised: k ∂s k· · · ∂ [ g ( n )] k ∂ s kn 

Original: 
Revised: 

Original: We will show that the rows of ∇ h ( W ∗ , s ∗ 0: n ) are linearly independent.
Revised: We will show that the rows of ∇ h ( W ∗ , s ∗ 0: n ) are linearly independent.

Original: To this end, let λ ij ∈ R for i ∈ { 1 , · · · , n } , j ∈ { 1 , · · · , k } be such that:
Revised: To this end, let λ ij ∈ R for i ∈ { 1 , · · · , n } , j ∈ { 1 , · · · , k } be such that:

Original: [Equation]
Revised: [Equation]

Original: In particular, for 1 ≤ a ≤ n , 1 ≤ b ≤ k , k (cid:88)
Revised: In particular, for 1 ≤ a ≤ n , 1 ≤ b ≤ k , k (cid:88)

Original: j
Revised: j

Original: = n
Revised: = n

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: By setting a = n , we have that λ nb = 0 for all 1 ≤ b ≤ k .
Revised: By setting a = n , we have that λ nb = 0 for all 1 ≤ b ≤ k .

Original: Setting a = n − 1 , we similarly have that λ n − 1 ,b = 0 .
Revised: Setting a = n − 1 , we similarly have that λ n − 1 ,b = 0 .

Original: Proceeding in this fashion, we have that λ ab = 0 for all 1 ≤ a ≤ n , 1 ≤ b ≤ k .
Revised: Proceeding in this fashion, we have that λ ab = 0 for all 1 ≤ a ≤ n , 1 ≤ b ≤ k .

Original: Actually, we did not at any point use the fact that ( W ∗ , β ∗ , s ∗ 0: n ) is a local min, so that the constraint gradients are linearly independent everywhere, and in particular at ( W ∗ , β ∗ , s ∗ 0: n ) .
Revised: Actually, we did not at any point use the fact that ( W ∗ , β ∗ , s ∗ 0: n ) is a local min, so that the constraint gradients are linearly independent everywhere, and in particular at ( W ∗ , β ∗ , s ∗ 0: n ) .

Original: Theorem 2.
Revised: Theorem 2.

Original: Assume we have a positive, increasing sequence { λ k } → ∞ , a non-negative sequence { (cid:15) k } → 0 , and a sequence of points { ( W k , β k , S k ) } such that (cid:107)∇ L ( W k , β k , S k ); λ k ) (cid:107) ≤ (cid:15) k for
Revised: Assume we have a positive, increasing sequence { λ k } → ∞ , a non-negative sequence { (cid:15) k } → 0 , and a sequence of points { ( W k , β k , S k ) } such that (cid:107)∇ L ( W k , β k , S k ); λ k ) (cid:107) ≤ (cid:15) k for

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: k ,
Revised: k ,

Original: [Equation]
Revised: [Equation]

Original: k , S
Revised: k , S

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) def
Revised: ) def

Original: [Equation]
Revised: [Equation]

Original: Assume further that { ( W k , β k , S k ) } has a convergent subsequence { ( W k i ,
Revised: Assume further that { ( W k , β k , S k ) } has a convergent subsequence { ( W k i ,

Original: β k i , S k i ) } with limit ( W ∗ , β ∗ , S ∗ ) .
Revised: β k i , S k i ) } with limit ( W ∗ , β ∗ , S ∗ ) .

Original: Then ( W ∗ , β ∗ , S ∗ ) is a KKT point of the constrained FPP objective (see (12) ) and ( W ∗ , β ∗ , s ∗ 0 ) is a KKT point of the RNN objective (10) .
Revised: Then ( W ∗ , β ∗ , S ∗ ) is a KKT point of the constrained FPP objective (see (12) ) and ( W ∗ , β ∗ , s ∗ 0 ) is a KKT point of the RNN objective (10) .

Original: Further, if ( W ∗ , β ∗ , S ∗ ) is a local min of the constrained FPP objective, then ( W ∗ , β ∗ , s ∗ 0 ) is a local min of (10) .
Revised: Further, if ( W ∗ , β ∗ , S ∗ ) is a local min of the constrained FPP objective, then ( W ∗ , β ∗ , s ∗ 0 ) is a local min of (10) .

Original: Proof.
Revised: Proof.

Original: By Proposition 3 and Proposition 2.3 from Bertsekas (1982), We have the existence of a
Revised: By Proposition 3 and Proposition 2.3 from Bertsekas (1982), We have the existence of a

Original: Lagrange multiplier vector λ such that
Revised: Lagrange multiplier vector λ such that

Original: [Equation]
Revised: [Equation]

Original: where h ( W ∗ , s ∗ 0: n ) is as in Proposition 3.
Revised: where h ( W ∗ , s ∗ 0: n ) is as in Proposition 3.

Original: Hence, ( W ∗ , β ∗ , s ∗ 0: n ) is a KKT point of (12).
Revised: Hence, ( W ∗ , β ∗ , s ∗ 0: n ) is a KKT point of (12).

Original: By Proposition 2, ( W ∗ , β ∗ , s ∗ 0 ) is a KKT point for (10).
Revised: By Proposition 2, ( W ∗ , β ∗ , s ∗ 0 ) is a KKT point for (10).

Original: Finally, if ( W ∗ , β ∗ , s ∗ 0: n ) is a local min of (12), then by Proposition 1 we have that ( W ∗ , β ∗ , s ∗ 0 ) is a local min of (10).
Revised: Finally, if ( W ∗ , β ∗ , s ∗ 0: n ) is a local min of (12), then by Proposition 1 we have that ( W ∗ , β ∗ , s ∗ 0 ) is a local min of (10).

Original: B P ARAMETER S TUDY
Revised: B P ARAMETER S TUDY

Original: We investigate the sensitivity of FPP to its two key parameters: the length of the trajectory T , and the buffer size N .
Revised: We investigate the sensitivity of FPP to its two key parameters: the length of the trajectory T , and the buffer size N .

Original: Overall, the losses on y-axis of Figure 5 show that FPP is robust to buffer size and truncation length.
Revised: Overall, the losses on y-axis of Figure 5 show that FPP is robust to buffer size and truncation length.

Original: As expected, for very small T, performance degrades, but otherwise the move from T= 10 to T= 50 does not result in a large difference.
Revised: As expected, for very small T, performance degrades, but otherwise the move from T= 10 to T= 50 does not result in a large difference.

Original: The algorithm was quite invariant to buffer size, starting from a reasonable size of 100.
Revised: The algorithm was quite invariant to buffer size, starting from a reasonable size of 100.

Original: For too large a buffer with a small number of updates, performance did degrade somewhat.
Revised: For too large a buffer with a small number of updates, performance did degrade somewhat.

Original: Overall, though, across this wide range of settings, FPP performed consistently well.
Revised: Overall, though, across this wide range of settings, FPP performed consistently well.

Original: [Figure]
Revised: [Figure]

Original: We also investigated how performance changes when changing λ .
Revised: We also investigated how performance changes when changing λ .

Original: Throughout all previous experiments, we simply set λ = 1 , to avoid unfairly tuning our method to each problem.
Revised: Throughout all previous experiments, we simply set λ = 1 , to avoid unfairly tuning our method to each problem.

Original: Interestingly, tuning λ does enable further performance improvements, though the algorithm worked well for quite a large range of λ .
Revised: Interestingly, tuning λ does enable further performance improvements, though the algorithm worked well for quite a large range of λ .

Original: C E XPERIMENTAL D ETAILS
Revised: C E XPERIMENTAL D ETAILS

Original: The dynamics for the Stochastic World environment are in Table 1.
Revised: The dynamics for the Stochastic World environment are in Table 1.

Original: For all experiments, we use RMSprop optimizer and the learning rate is chosen over the set { 0 . 0001 , 0 . 0003 , 0 .
Revised: For all experiments, we use RMSprop optimizer and the learning rate is chosen over the set { 0 . 0001 , 0 . 0003 , 0 .

Original: based on the average accuracy/loss.
Revised: based on the average accuracy/loss.

Original: For real datasets, we use multiple trajectories to speed upthe training.
Revised: For real datasets, we use multiple trajectories to speed up training.

Original: The details of each task are provided below:
Revised: The details of each task are provided below:

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C.
Revised: C.

Original: C YCLE W ORLD
Revised: C YCLE W ORLD

Original: Network Type = simple RNN
Revised: Network Type = simple RNN

Original: Hidden Units =
Revised: Hidden Units =

Original: C.
Revised: C.

Original: S TOCHASTIC W ORLD
Revised: S TOCHASTIC W ORLD

Original: Network Type = simple RNN
Revised: Network Type = simple RNN

Original: Hidden Units =
Revised: Hidden Units =

Original: C.
Revised: C.

Original: S EQUENTIAL MNIST
Revised: S EQUENTIAL MNIST

Original: Network Type = simple RNN
Revised: Network Type = simple RNN

Original: Hidden Units =
Revised: Hidden Units =

Original: Image size = 784 pixels
Revised: Image Size = 784 pixels

Original: Input dimension = 28 pixels
Revised: Input Dimension = 28 pixels

Original: Number of Steps = 28000 (1000 images of 28 steps)
Revised: Number of Steps = 28000 (1000 images of 28 steps)

Original: [Table]
Revised: [Table]

Original: [Table]