Original: I NTEGRATING E PISODIC M EMORY INTO A R EINFORCEMENT L EARNING A GENT USING R ESERVOIR S AMPLING
Revised: I NTEGRATING E PISODIC M EMORY INTO A R EINFORCEMENT L EARNING A GENT USING R ESERVOIR S AMPLING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Episodic memory is a psychology term which refers to the ability to recall speciﬁc events from the past.
Revised: Episodic memory is a psychology term which refers to the ability to recall speciﬁc events from the past.

Original: We suggest one advantage of this particular type of memory is the ability to easily assign credit to a speciﬁc state when remembered information is found to be useful.
Revised: We suggest one advantage of this particular type of memory is the ability to easily assign credit to a speciﬁc state when remembered information is found to be useful.

Original: Inspired by this idea we propose a novel algorithm which uses a reservoir sampling procedure to maintain a memory consisting of a ﬁxed number of past states.
Revised: Inspired by this idea we propose a novel algorithm which uses a reservoir sampling procedure to maintain a memory consisting of a ﬁxed number of past states.

Original: The algorithm allows a deep reinforcement learning agent to learn online to preferentially remember those states which are found to be useful to recall later on.
Revised: The algorithm allows a deep reinforcement learning agent to learn online to preferentially remember those states which are found to be useful to recall later on.

Original: Much of reinforcement learning (RL) theory is based on the assumption that the environment has the
Revised: Much of reinforcement learning (RL) theory is based on the assumption that the environment has the

Original: Markov property, meaning that future states are independent of past states given the present state.
Revised: Markov property, meaning that future states are independent of past states given the present state.

Original: This implies the agent has all the information it needs to make an optimal decision at each time and therefore has no need to remember the past.
Revised: This implies the agent has all the information it needs to make an optimal decision at each time and therefore has no need to remember the past.

Original: This is however not realistic in general, realistic problems often require signiﬁcant information from the past to make an informed decision in the present, and there is often no obvious way to incorporate the relevant information into an expanded present state.
Revised: This is however not realistic in general, realistic problems often require signiﬁcant information from the past to make an informed decision in the present, and there is often no obvious way to incorporate the relevant information into an expanded present state.

Original: It is thus desirable to establish techniques for learning a representation of the relevant details of the past (e.g. a memory, or learned state) to facilitate decision making in the present.
Revised: It is thus desirable to establish techniques for learning a representation of the relevant details of the past (e.g. a memory, or learned state) to facilitate decision making in the present.

Original: A popular approach to integrate information from the past into present decision making is to use some variant of a recurrent neural network, possibly coupled to some form of external memory, trained with backpropagation through time.
Revised: A popular approach to integrate information from the past into present decision making is to use some variant of a recurrent neural network, possibly coupled to some form of external memory, trained with backpropagation through time.

Original: This can work well for many tasks, but generally requires backpropagating many steps into the past which is not practical in an online RL setting.
Revised: This can work well for many tasks, but generally requires backpropagating many steps into the past which is not practical in an online RL setting.

Original: In this work we investigate an alternative approach, which we liken to idea of episodic memory from psychology.
Revised: In this work we investigate an alternative approach, which we liken to idea of episodic memory from psychology.

Original: In this approach the information stored in memory is constrained to consist of a ﬁnite set of past states experienced by the agent.
Revised: In this approach the information stored in memory is constrained to consist of a ﬁnite set of past states experienced by the agent.

Original: In this work, by states we mean observations explicitly provided by the environment.
Revised: In this work, by states we mean observations explicitly provided by the environment.

Original: In general, states could be more abstract, such as the internal state of an RNN or predictions generated by something like the Horde architecture of Sutton et al.
Revised: In general, states could be more abstract, such as the internal state of an RNN or predictions generated by something like the Horde architecture of Sutton et al.

Original: By storing states explicitly we enforce that the information recorded also provides the context in which it was recorded.
Revised: By storing states explicitly we enforce that the information recorded also provides the context in which it was recorded.

Original: We can therefore assign credit to the recorded state without explicitly backpropagating through time between when the information proves useful and when it was recorded.
Revised: We can therefore assign credit to the recorded state without explicitly backpropagating through time between when the information proves useful and when it was recorded.

Original: If a recorded state is found to be useful we train the agent to preferentially remember similar states in the future.
Revised: If a recorded state is found to be useful we train the agent to preferentially remember similar states in the future.

Original: In our approach the set of states in memory at a given time is drawn from a distribution over all n -subsets (subsets of size n ) of visited states, parameterized by a weight value assigned to each state by a trained model.
Revised: In our approach the set of states in memory at a given time is drawn from a distribution over all n -subsets (subsets of size n ) of visited states, parameterized by a weight value assigned to each state by a trained model.

Original: To allow us to draw from such a distribution without maintaining all visited states in memory we introduce a reservoir sampling technique.
Revised: To allow us to draw from such a distribution without maintaining all visited states in memory we introduce a reservoir sampling technique.

Original: Reservoir sampling refers to a class of algorithms for sampling from a distribution over n -subsets of items from a larger set streamed one item at a time.
Revised: Reservoir sampling refers to a class of algorithms for sampling from a distribution over n -subsets of items from a larger set streamed one item at a time.

Original: The goal is to ensure, through speciﬁc add and drop probabilities, that the n items in the reservoir at each time-step correspond to a sample from the desired distribution over n -subsets of all observed items.
Revised: The goal is to ensure, through speciﬁc add and drop probabilities, that the n items in the reservoir at each time-step correspond to a sample from the desired distribution over n -subsets of all observed items.

Original: Two important examples which sample from different distributions are found in Chao (1982) and Efraimidis & Spirakis (2006).
Revised: Two important examples which sample from different distributions are found in Chao (1982) and Efraimidis & Spirakis (2006).

Original: In this work we will deﬁne our own distribution and sampling procedure to suit our needs.
Revised: In this work we will deﬁne our own distribution and sampling procedure to suit our needs.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Deep learning systems which make use of an external memory have received a lot of interest lately.
Revised: Deep learning systems which make use of an external memory have received a lot of interest lately.

Original: Two prototypical examples are found in Graves et al.
Revised: Two prototypical examples are found in Graves et al.

Original: (2014) and the follow-up Graves et al.
Revised: (2014) and the follow-up Graves et al.

Original: These systems use an LSTM controller attached to read and write heads of a fully differentiable external memory and train the combined system to perform algorithmic tasks.
Revised: These systems use an LSTM controller attached to read and write heads of a fully differentiable external memory and train the combined system to perform algorithmic tasks.

Original: Contrary to our approach, training is done entirely by backpropagation through time.
Revised: Contrary to our approach, training is done entirely by backpropagation through time.

Original: See also Zaremba & Sutskever (2015), Joulin & Mikolov (2015), Sukhbaatar et al.
Revised: See also Zaremba & Sutskever (2015), Joulin & Mikolov (2015), Sukhbaatar et al.

Original: (2015), Gulcehre et al.
Revised: (2015), Gulcehre et al.

Original: (2017) and Kaiser et al.
Revised: (2017) and Kaiser et al.

Original: (2017) for more examples of deep learning systems with integrated external memory.
Revised: (2017) for more examples of deep learning systems with integrated external memory.

Original: More directly related to the present work is the application of deep RL to non-markov tasks, in particular Oh et al.
Revised: More directly related to the present work is the application of deep RL to non-markov tasks, in particular Oh et al.

Original: They experiment with architectures using a combination of key-value memory and a recurrent neural networks.
Revised: They experiment with architectures using a combination of key-value memory and a recurrent neural networks.

Original: The memory saves keys and values corresponding to the last N observations for some integer N , thus it is inherently limited in temporal extent but does not require any mechanism for information triage.
Revised: The memory saves keys and values corresponding to the last N observations for some integer N , thus it is inherently limited in temporal extent but does not require any mechanism for information triage.

Original: They test on problems in the Minecraft domain which could provide compelling testbeds for a potential follow-up to the present work.
Revised: They test on problems in the Minecraft domain which could provide compelling testbeds for a potential follow-up to the present work.

Original: See also
Revised: See also

Original: Bakker et al.
Revised: Bakker et al.

Original: , Wierstra et al.
Revised: , Wierstra et al.

Original: (2010), Zhang et al.
Revised: (2010), Zhang et al.

Original: and Hausknecht & Stone (2015) for more examples of applying deep RL to non-markov tasks.
Revised: and Hausknecht & Stone (2015) for more examples of applying deep RL to non-markov tasks.

Original:  A RCHITECTURE
Revised:  A RCHITECTURE

Original: Our model is based around an advantage actor critic architecture (Mnih et al., 2016) consisting of separate value and policy networks.
Revised: Our model is based around an advantage actor critic architecture (Mnih et al., 2016) consisting of separate value and policy networks.

Original: In addition we include an external memory M consisting of a set of n past visited states ( S t 0 , .., S t n − 1 ) with associated importance weights ( w t 0 , ..., w t n − 1 ) .
Revised: In addition we include an external memory M consisting of a set of n past visited states ( S t 0 , .., S t n − 1 ) with associated importance weights ( w t 0 , ..., w t n − 1 ) .

Original: The query network q ( S t ) outputs a vector of size equal to the state size with tanh activation.
Revised: The query network q ( S t ) outputs a vector of size equal to the state size with tanh activation.

Original: At each time step a single item S t i is drawn from the memory to condition the policy according to:
Revised: At each time step a single item S t i is drawn from the memory to condition the policy according to:

Original: [Equation]
Revised: [Equation]

Original: (1) where τ is a positive learnable temperature parameter.
Revised: (1) where τ is a positive learnable temperature parameter.

Original: The state, m t , selected from memory is given as input to the policy network along with the current state, both of which condition the resulting policy.
Revised: The state, m t , selected from memory is given as input to the policy network along with the current state, both of which condition the resulting policy.

Original: Finally the write network takes the current state as input and outputs a single value with sigmoid activation.
Revised: Finally the write network takes the current state as input and outputs a single value with sigmoid activation.

Original: This value is used to determine how likely the present state is to be written to and subsequently retained in the memory according to the distribution in equation 7 which will be throughly explained in section 3.
Revised: This value is used to determine how likely the present state is to be written to and subsequently retained in the memory according to the distribution in equation 7 which will be throughly explained in section 3.

Original: An illustration of this architecture is shown in ﬁgure 1.
Revised: An illustration of this architecture is shown in ﬁgure 1.

Original: [Figure]
Revised: [Figure]

Original:  A LGORITHM
Revised:  A LGORITHM

Original: For the most part our model is trained using standard stochastic gradient descent on common RL loss functions.
Revised: For the most part our model is trained using standard stochastic gradient descent on common RL loss functions.

Original: The value network is trained by gradient descent on the squared one step temporal different error δ 2 t where δ t =
Revised: The value network is trained by gradient descent on the squared one step temporal different error δ 2 t where δ t =

Original: r t +

Revised: r t +

Original: + V ( S t +1 )
Revised: + V ( S t +1 )

Original: − V ( S t ) , and the gradient is passed only through
Revised: − V ( S t ) , and the gradient is passed only through

Original: V ( S t ) .
Revised: V ( S t ) .

Original: The policy is trained using the advantage loss − δ t log( π ( a t | S t , m t )) with gradients passed only through π ( a t | S t , m t ) .
Revised: The policy is trained using the advantage loss − δ t log( π ( a t | S t , m t )) with gradients passed only through π ( a t | S t , m t ) .

Original: The query network is trained similarly on the loss − δ t log( Q ( m t | S t )) with gradients passed only through Q ( m t | S t ) .
Revised: The query network is trained similarly on the loss − δ t log( Q ( m t | S t )) with gradients passed only through Q ( m t | S t ) .

Original: We train online, performing one update per timestep with no experience replay.
Revised: We train online, performing one update per timestep with no experience replay.

Original: The main innovation of this work is in the training method for the write network which is described in sections 3.1 and 3.2.
Revised: The main innovation of this work is in the training method for the write network which is described in sections 3.1 and 3.2.

Original: There are two main desiderata we wish to satisfy with the write network.
Revised: There are two main desiderata we wish to satisfy with the write network.

Original: First we want to use the weights w ( S t ) generated by the network in a reservoir sampling algorithm such that the probability of a particular state S
Revised: First we want to use the weights w ( S t ) generated by the network in a reservoir sampling algorithm such that the probability of a particular state S

Original: ˆ t being present in memory at any given future time t > ˆ t is proportional to the associated weight w ( S ˆ t ) .
Revised: ˆ t being present in memory at any given future time t > ˆ t is proportional to the associated weight w ( S ˆ t ) .

Original: Second we want to obtain estimates of the gradient of the return with respect to the weight of each item in memory such that we can perform approximate gradient descent on the generated weights.
Revised: Second we want to obtain estimates of the gradient of the return with respect to the weight of each item in memory such that we can perform approximate gradient descent on the generated weights.

Original: 3. G RADIENT E STIMATE
Revised: 3. G RADIENT E STIMATE

Original: For brevity, in this section we will use the notation E t [ x ] to denote E [ x | S 0 , ..., S t ] i.e. the expectation conditioned on the entire history of state visitation up to time t .
Revised: For brevity, in this section we will use the notation E t [ x ] to denote E [ x | S 0 , ..., S t ] i.e. the expectation conditioned on the entire history of state visitation up to time t .

Original: Similarly P t ( x ) will represent probability conditioned on the entire history of state visitation.
Revised: Similarly P t ( x ) will represent probability conditioned on the entire history of state visitation.

Original: All expectations and probabilities are assumed to be with respect to the current policy, query and write network.
Revised: All expectations and probabilities are assumed to be with respect to the current policy, query and write network.

Original: Let A represent the set of available actions and A t the action selected at time t .
Revised: Let A represent the set of available actions and A t the action selected at time t .

Original: 3.1. O NE -S TATE M EMORY C ASE
Revised: 3.1. O NE -S TATE M EMORY C ASE

Original: To introduce the idea we ﬁrst present our gradient estimation procedure for the case when our memory can store just one state, and thus there is no need to query.
Revised: To introduce the idea we ﬁrst present our gradient estimation procedure for the case when our memory can store just one state, and thus there is no need to query.

Original: Here m t represents the state in memory at time t and thus, in the one-state memory case, the state read from memory by the agent at time t .
Revised: Here m t represents the state in memory at time t and thus, in the one-state memory case, the state read from memory by the agent at time t .

Original: Assume the stored memory is drawn from a distribution parameterized as follows by a set of weights { w i | i ∈ { 0 , ..., t − 1 }} associated with each state S i when the state is ﬁrst visited:
Revised: Assume the stored memory is drawn from a distribution parameterized as follows by a set of weights { w i | i ∈ { 0 , ..., t − 1 }} associated with each state S i when the state is ﬁrst visited:

Original: [Equation]
Revised: [Equation]

Original: We can then write the expected return R t =
Revised: We can then write the expected return R t =

Original: r t +1 + γr t +

Revised: r t +1 + γr t +

Original: + ... as follows:
Revised: + ... as follows:

Original: [Equation]
Revised: [Equation]

Original: In order to perform gradient descent on the weights w i we wish to estimate the gradient of this expectation value with respect to each weight.
Revised: In order to perform gradient descent on the weights w i we wish to estimate the gradient of this expectation value with respect to each weight.

Original: In particular we will derive an estimate of this gradient using an actor critic method which is unbiased if the critic’s evaluation is correct.
Revised: In particular we will derive an estimate of this gradient using an actor critic method which is unbiased if the critic’s evaluation is correct.

Original: Additionally our estimate will be non-zero only for the w i associated with the index i such that m t =
Revised: Additionally our estimate will be non-zero only for the w i associated with the index i such that m t =

Original: S i .
Revised: S i .

Original: This means if our weights w i are generated by a neural network, we will only have to propagate gradients through the single stored state.
Revised: This means if our weights w i are generated by a neural network, we will only have to propagate gradients through the single stored state.

Original: This is crucial to allow our algorithm to run online, as otherwise we would require storing every visited state to compute the gradient estimate.
Revised: This is crucial to allow our algorithm to run online, as otherwise we would require storing every visited state to compute the gradient estimate.

Original: [Equation]
Revised: [Equation]

Original: We can rewrite this as:
Revised: We can rewrite this as:

Original: [Equation]
Revised: [Equation]

Original: See appendix A for a detailed derivation of this expression.
Revised: See appendix A for a detailed derivation of this expression.

Original: We will use a policy gradient approach, similar to REINFORCE (Williams, 1992), to estimate the ﬁrst term in this gradient using an estimator G i such that E t
Revised: We will use a policy gradient approach, similar to REINFORCE (Williams, 1992), to estimate the ﬁrst term in this gradient using an estimator G i such that E t

Original: [ G i ]
Revised: [ G i ]

Original: ≈ ∂E t [ R t ] ∂w i .
Revised: ≈ ∂E t [ R t ] ∂w i .

Original: The second term is then estimated recursively on subsequent time-steps.
Revised: The second term is then estimated recursively on subsequent time-steps.

Original: In the present work we will focus on the undiscounted episodic case with the start-state value objective, for which it sufﬁces to follow the ﬁrst term in the above gradient expression for each visited state.
Revised: In the present work we will focus on the undiscounted episodic case with the start-state value objective, for which it sufﬁces to follow the ﬁrst term in the above gradient expression for each visited state.

Original: This is also true in the continuing case with an average-reward objective.
Revised: This is also true in the continuing case with an average-reward objective.

Original: See Sutton et al.
Revised: See Sutton et al.

Original: (2000) for further discussion of this distinction.
Revised: (2000) for further discussion of this distinction.

Original: Consider the gradient estimator:
Revised: Consider the gradient estimator:

Original: [Equation]
Revised: [Equation]

Original: which has expectation:
Revised: which has expectation:

Original: [Equation]
Revised: [Equation]

Original: Where the approximation is limited by the accuracy of our value function.
Revised: Where the approximation is limited by the accuracy of our value function.

Original: In conventional policy gradient subtracting the state value (e.g. using δ t =
Revised: In conventional policy gradient subtracting the state value (e.g. using δ t =

Original: r t +1 + γV
Revised: r t +1 + γV

Original: ( S t +1 )
Revised: ( S t +1 )

Original: − V ( S t ) instead of r t +1 + γV
Revised: − V ( S t ) instead of r t +1 + γV

Original: S t +1 ) ) is a means of variance reduction.
Revised: S t +1 ) ) is a means of variance reduction.

Original: Here it is critical to avoid computing gradients with respect to the denominator of equation A, which allows our algorithm to run online while computing the gradient with respect to only the weight stored in memory.
Revised: Here it is critical to avoid computing gradients with respect to the denominator of equation A, which allows our algorithm to run online while computing the gradient with respect to only the weight stored in memory.

Original: Given these estimated gradients with respect to w i we apply the chain rule to compute ∂R t ∂θ w = ∂R
Revised: Given these estimated gradients with respect to w i we apply the chain rule to compute ∂R t ∂θ w = ∂R

Original: t ∂w i ∂w
Revised: t ∂w i ∂w

Original: [Equation]
Revised: [Equation]

Original: ∂θ w ≈ G i ∂w ( S i )
Revised: ∂θ w ≈ G i ∂w ( S i )

Original: ∂θ w , for each parameter θ w of the write network.
Revised: ∂θ w , for each parameter θ w of the write network.

Original: This gradient estimate is used in a gradient descent procedure to emphasize retention of states which improve the return.
Revised: This gradient estimate is used in a gradient descent procedure to emphasize retention of states which improve the return.

Original: Gradient estimates are generated based on the stored values of w i in memory but applied to the parameters of the network at the present time.
Revised: Gradient estimates are generated based on the stored values of w i in memory but applied to the parameters of the network at the present time.

Original: With online updating, this introduces a multiple timescale issue which perhaps warrants further investigation.
Revised: With online updating, this introduces a multiple timescale issue which perhaps warrants further investigation.

Original: We assume this should be acceptable as long as the step size is sufﬁciently small, but leave further investigation to future work.
Revised: We assume this should be acceptable as long as the step size is sufﬁciently small, but leave further investigation to future work.

Original: There are a number of ways to extend the distribution deﬁned in equation A to the case where multiple elements of a set must be selected (see for example Efraimidis & Spirakis (2006)).
Revised: There are a number of ways to extend the distribution deﬁned in equation A to the case where multiple elements of a set must be selected (see for example Efraimidis & Spirakis (2006)).

Original: We will focus on a generalization which is less explored but which we will see in the following section results in gradient estimates which are an elegant generalization of the single-state memory case.
Revised: We will focus on a generalization which is less explored but which we will see in the following section results in gradient estimates which are an elegant generalization of the single-state memory case.

Original: 3.1. M ULTIPLE -S TATE M EMORY C ASE
Revised: 3.1. M ULTIPLE -S TATE M EMORY C ASE

Original: In this section and those that follow we will routinely use the notation (cid:0) Zn (cid:1) where Z is a set and n an integer to indicate the set of all n -subsets of Z .
Revised: In this section and those that follow we will routinely use the notation (cid:0) Zn (cid:1) where Z is a set and n an integer to indicate the set of all n -subsets of Z .

Original: Note that (cid:0) Z 0 (cid:1) = {∅} and we adopt the convention (cid:81) x
Revised: Note that (cid:0) Z 0 (cid:1) = {∅} and we adopt the convention (cid:81) x

Original: ∈∅ x = 1 thus (cid:80) ˆ Z ∈ ( Z 0 ) (cid:81) x
Revised: ∈∅ x = 1 thus (cid:80) ˆ Z ∈ ( Z 0 ) (cid:81) x

Original: ∈
Revised: ∈

Original: ˆ Z x = 1 which will be important in a few places in what follows.
Revised: ˆ Z x = 1 which will be important in a few places in what follows.

Original: We will introduce some notation to facilitate reasoning about sets of states.
Revised: We will introduce some notation to facilitate reasoning about sets of states.

Original: Let T t = { t (cid:48) : 0 ≤ t (cid:48) ≤ t − 1 } be the set of all time indices from 0 to t − 1 .
Revised: Let T t = { t (cid:48) : 0 ≤ t (cid:48) ≤ t − 1 } be the set of all time indices from 0 to t − 1 .

Original: Let ˆ T ∈ (cid:0) T t n (cid:1) be a set of n indices chosen from
Revised: Let ˆ T ∈ (cid:0) T t n (cid:1) be a set of n indices chosen from

Original: T t where n is the memory size.
Revised: T t where n is the memory size.

Original: Let S ˆ T be the set of states { S ˆ t : ˆ t ∈ ˆ T } .
Revised: Let S ˆ T be the set of states { S ˆ t : ˆ t ∈ ˆ T } .

Original: Let M t be the set of states in memory at time t .
Revised: Let M t be the set of states in memory at time t .

Original: Let Q ( S ˆ t | S ˆ T ) be the probability of querying S ˆ t given M t =
Revised: Let Q ( S ˆ t | S ˆ T ) be the probability of querying S ˆ t given M t =

Original: S ˆ T .
Revised: S ˆ T .

Original: The probability for a particular set of states being contained in memory is deﬁned to be the following:
Revised: The probability for a particular set of states being contained in memory is deﬁned to be the following:

Original: [Equation]
Revised: [Equation]

Original: A straightforward extension of the derivation of the equation 5 shows that this choice results in a gradient estimate which is an elegant extension of the one-state memory case.
Revised: A straightforward extension of the derivation of the equation 5 shows that this choice results in a gradient estimate which is an elegant extension of the one-state memory case.

Original: The derivation is given in appendix B, the result for ∂∂w
Revised: The derivation is given in appendix B, the result for ∂∂w

Original: i E t [ R t ] is:
Revised: i E t [ R t ] is:

Original: [Equation]
Revised: [Equation]

Original: As in the single-state memory case we recursively handle the second term.
Revised: As in the single-state memory case we recursively handle the second term.

Original: To estimate the ﬁrst term we could choose the following estimator G i :
Revised: To estimate the ﬁrst term we could choose the following estimator G i :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This estimator is unbiased under the assumption the critic is perfect, however it scales poorly in terms of both variance and computation time as the memory size increases.
Revised: This estimator is unbiased under the assumption the critic is perfect, however it scales poorly in terms of both variance and computation time as the memory size increases.

Original: This is because it requires updating every state in memory regardless of whether it was queried, spreading credit assignment and requiring compute time proportional to the product of the number of states in memory with the number of parameters in the write network.
Revised: This is because it requires updating every state in memory regardless of whether it was queried, spreading credit assignment and requiring compute time proportional to the product of the number of states in memory with the number of parameters in the write network.

Original: Instead we will further approximate the second term and perform an update only for the queried item.
Revised: Instead we will further approximate the second term and perform an update only for the queried item.

Original: We rewrite the ﬁrst term of equation 8 as follows:
Revised: We rewrite the ﬁrst term of equation 8 as follows:

Original: [Equation]
Revised: [Equation]

Original: This approximation is accurate to the extent that our query network is able to accurately select useful states.
Revised: This approximation is accurate to the extent that our query network is able to accurately select useful states.

Original: To see this, note that if querying a state when it’s in memory helps to generate a better expected return a well trained query network should do it with high probability and hence
Revised: To see this, note that if querying a state when it’s in memory helps to generate a better expected return a well trained query network should do it with high probability and hence

Original: P ( m t
Revised: P ( m t

Original: [Equation]
Revised: [Equation]

Original: = S i |M t
Revised: = S i |M t

Original: [Equation]
Revised: [Equation]

Original: S i ) will be low.
Revised: S i ) will be low.

Original: On the other hand if querying a state in memory is unhelpful (cid:18) (cid:80) a ∈A P t
Revised: On the other hand if querying a state in memory is unhelpful (cid:18) (cid:80) a ∈A P t

Original: [Equation]
Revised: [Equation]

Original: will generally be small.
Revised: will generally be small.

Original: With this approximation the gradient estimate becomes identical to the one-state memory case:
Revised: With this approximation the gradient estimate becomes identical to the one-state memory case:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: While this justiﬁcation is not rigorous, this approximation should signiﬁcantly improve computational and sample efﬁciency, and is used in our experiments in section 4.
Revised: While this justiﬁcation is not rigorous, this approximation should signiﬁcantly improve computational and sample efﬁciency, and is used in our experiments in section 4.

Original: 3. R ESERVOIR S AMPLING P ROCEDURE
Revised: 3. R ESERVOIR S AMPLING P ROCEDURE

Original: In the previous section we derived a gradient estimator for our desired memory distribution.
Revised: In the previous section we derived a gradient estimator for our desired memory distribution.

Original: in this section we introduce a method for sampling from this distribution online.
Revised: in this section we introduce a method for sampling from this distribution online.

Original: Speciﬁcally
Revised: Speciﬁcally

Original: we will
Revised: we will

Original: 1: Ω ← Zeros(n+1) 2: ˜Ω ← Zeros(n) 3: W ← Zeros(n) 4: ˆ T ← Zeros(n) 5: for time 0 ≤ t ≤ n − 1 do 6:
Revised: 1: Ω ← Zeros(n+1) 2: ˜Ω ← Zeros(n) 3: W ← Zeros(n) 4: ˆ T ← Zeros(n) 5: for time 0 ≤ t ≤ n − 1 do 6:

Original: Receive w t 7:
Revised: Receive w t 7:

Original: W [ t ] ← w t 8: ˆ T [ t ] ← t 9: end for 10: Apply equivalent random permutation to W and ˆ T 11: Ω[ n ] ←12: ˜Ω[ n − 1] ←13: for n − 1 ≥ i ≥ 0 do 14: Ω
Revised: W [ t ] ← w t 8: ˆ T [ t ] ← t 9: end for 10: Apply equivalent random permutation to W and ˆ T 11: Ω[ n ] ←12: ˜Ω[ n − 1] ←13: for n − 1 ≥ i ≥ 0 do 14: Ω

Original: [ i ] =
Revised: [ i ] =

Original: W [ i ] · Ω[ i + 1] 15: end for 16: for n − 2 ≥ i ≥ 0 do 17: ˜Ω[ i ] =
Revised: W [ i ] · Ω[ i + 1] 15: end for 16: for n − 2 ≥ i ≥ 0 do 17: ˜Ω[ i ] =

Original: Ω[ i + 1] + W [ i ] · ˜Ω[ i + 1] 18: end for 19: for all time t ≥ n do 20:
Revised: Ω[ i + 1] + W [ i ] · ˜Ω[ i + 1] 18: end for 19: for all time t ≥ n do 20:

Original: Receive w t 21:
Revised: Receive w t 21:

Original: U PDATE ( w t , t ) 22: end for 1: function U PDATE ( w , t ) 2: ω ← w 3: τ ← t 4: for 0 ≤ i ≤ n − 1 do 5: Ω (cid:48) ← Ω
Revised: U PDATE ( w t , t ) 22: end for 1: function U PDATE ( w , t ) 2: ω ← w 3: τ ← t 4: for 0 ≤ i ≤ n − 1 do 5: Ω (cid:48) ← Ω

Original: [ i ] + ω · ˜Ω[ i ] 6: if i=n-1 then 7: Ω (cid:48)(cid:48) ← Ω
Revised: [ i ] + ω · ˜Ω[ i ] 6: if i=n-1 then 7: Ω (cid:48)(cid:48) ← Ω

Original: [ i + 1] 8: else 9: Ω (cid:48)(cid:48) ← Ω
Revised: [ i + 1] 8: else 9: Ω (cid:48)(cid:48) ← Ω

Original: [ i + 1] + ω · ˜Ω[ i + 1] 10: end if 11:
Revised: [ i + 1] + ω · ˜Ω[ i + 1] 10: end if 11:

Original: P ← 1 − Ω (cid:48)(cid:48)
Revised: P ← 1 − Ω (cid:48)(cid:48)

Original: Ω
Revised: Ω

Original: [ i ] Ω (cid:48)
Revised: [ i ] Ω (cid:48)

Original: Ω[ i +1] 12:
Revised: Ω[ i +1] 12:

Original: Swap ω with W [ i ] and τ with ˆ T [ i ] with probability P 13: Ω[ i ] ← Ω (cid:48) 14: end for 15: for n − 2 ≥ i ≥ 0 do 16:
Revised: Swap ω with W [ i ] and τ with ˆ T [ i ] with probability P 13: Ω[ i ] ← Ω (cid:48) 14: end for 15: for n − 2 ≥ i ≥ 0 do 16:

Original: ˜ Ω[ i ] =
Revised: ˜ Ω[ i ] =

Original: Ω[ i + 1] + W
Revised: Ω[ i + 1] + W

Original: [Equation]
Revised: [Equation]

Original: 17: end for 18: end function formulate a reservoir sampling algorithm for drawing a subset ˆ T of size n from a set of indices
Revised: 17: end for 18: end function formulate a reservoir sampling algorithm for drawing a subset ˆ T of size n from a set of indices

Original: T = { 0 , ..., t − 1 } according to a distribution parameterized by a weight w i for each index i ∈ T .
Revised: T = { 0 , ..., t − 1 } according to a distribution parameterized by a weight w i for each index i ∈ T .

Original: Following equation 7 the probability for a given subset ˆ T is deﬁned as follows:
Revised: Following equation 7 the probability for a given subset ˆ T is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: This distribution can be sampled from by selecting members sequentially for i ∈ { 0 , .., n − 1 } with the following conditional probabilities:
Revised: This distribution can be sampled from by selecting members sequentially for i ∈ { 0 , .., n − 1 } with the following conditional probabilities:

Original: [Equation]
Revised: [Equation]

Original: We abuse notation slightly and use ˆ T to refer to both an ordered vector and the set of its elements.
Revised: We abuse notation slightly and use ˆ T to refer to both an ordered vector and the set of its elements.

Original: Lemma 1.
Revised: Lemma 1.

Original: Selecting elements sequentially according to equation 12 will result in a vector ˆ T whose elements correspond to a sample drawn from equation 11.
Revised: Selecting elements sequentially according to equation 12 will result in a vector ˆ T whose elements correspond to a sample drawn from equation 11.

Original: Proof.
Revised: Proof.

Original: See appendix C.
Revised: See appendix C.

Original: We will use lemma 1 to derive a reservoir sampling procedure which works online to update the reservoir ˆ T at each time-step when a new index is added to T along with an associated weight.
Revised: We will use lemma 1 to derive a reservoir sampling procedure which works online to update the reservoir ˆ T at each time-step when a new index is added to T along with an associated weight.

Original: The result is algorithm 1.
Revised: The result is algorithm 1.

Original: At each time-step UPDATE moves through ˆ T starting from index 0 and chooses whether to swap the item and weight at each index with the ones currently contained in a buffer ( ω and τ , initially set to contain the newly added item and associated weight).
Revised: At each time-step UPDATE moves through ˆ T starting from index 0 and chooses whether to swap the item and weight at each index with the ones currently contained in a buffer ( ω and τ , initially set to contain the newly added item and associated weight).

Original: The probability of swapping is chosen such that it corrects the conditional probability of the item at each index (conditioned on the items before it) to compensate for the item in the buffer being added to the set of possible items for that index.
Revised: The probability of swapping is chosen such that it corrects the conditional probability of the item at each index (conditioned on the items before it) to compensate for the item in the buffer being added to the set of possible items for that index.

Original: After doing this sequentially at each index the overall probability of ˆ T will be correct with the newly added item.
Revised: After doing this sequentially at each index the overall probability of ˆ T will be correct with the newly added item.

Original: Computing the necessary swap probabilities is nontrivial in itself, however we show that it is possible to do this in O ( n ) time per time-step (where here n is the memory size) by iteratively updating two vectors Ω and ˜Ω .
Revised: Computing the necessary swap probabilities is nontrivial in itself, however we show that it is possible to do this in O ( n ) time per time-step (where here n is the memory size) by iteratively updating two vectors Ω and ˜Ω .

Original: Theorem 1.
Revised: Theorem 1.

Original: In algorithm 1 let t refer to the parameter of the call to UPDATE, ˆ T t [ i ] refer to the value of ˆ T [ i ] when that call is made, and T t = { t (cid:48) : 0 ≤ t (cid:48) ≤ t − 1 } refer to the set of all time indices from 0 to t − 1 .
Revised: In algorithm 1 let t refer to the parameter of the call to UPDATE, ˆ T t [ i ] refer to the value of ˆ T [ i ] when that call is made, and T t = { t (cid:48) : 0 ≤ t (cid:48) ≤ t − 1 } refer to the set of all time indices from 0 to t − 1 .

Original: ∀ t ≥ n, 0 ≤ i ≤ n − 1 :
Revised: ∀ t ≥ n, 0 ≤ i ≤ n − 1 :

Original: P
Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where [ t 0 , ..., t i ] is any arbitrary vector of unique elements of T t .
Revised: ) where [ t 0 , ..., t i ] is any arbitrary vector of unique elements of T t .

Original: Proof.
Revised: Proof.

Original: See appendix D.
Revised: See appendix D.

Original: Corollary 1.1.
Revised: Corollary 1.1.

Original: At the call to UPDATE with parameter t , ∀ t ≥ n, ˆ T ∈ (cid:0) T t n (cid:1) :
Revised: At the call to UPDATE with parameter t , ∀ t ≥ n, ˆ T ∈ (cid:0) T t n (cid:1) :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof follows from theorem 1 and lemma 1.
Revised: The proof follows from theorem 1 and lemma 1.

Original: Thus algorithm 1 produces reservoirs which are indeed valid samples form the desired distribution at each time-step.
Revised: Thus algorithm 1 produces reservoirs which are indeed valid samples form the desired distribution at each time-step.

Original: Note that it runs in O ( n ) time per time-step where n is the size of the memory.
Revised: Note that it runs in O ( n ) time per time-step where n is the size of the memory.

Original: We use this algorithm along with the weights generated by our write network to manage updating the memory on each new state visitation.
Revised: We use this algorithm along with the weights generated by our write network to manage updating the memory on each new state visitation.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: We test our algorithm on a toy problem we call “the secret informant problem”.
Revised: We test our algorithm on a toy problem we call “the secret informant problem”.

Original: The problem is intended to highlight the kinds of sharp, long-term dependencies that are often difﬁcult for recurrent models but which our algorithm is intended to deal with.
Revised: The problem is intended to highlight the kinds of sharp, long-term dependencies that are often difﬁcult for recurrent models but which our algorithm is intended to deal with.

Original: The problem is such that in order to behave optimally an agent must remember speciﬁc, initially unknown past states.
Revised: The problem is such that in order to behave optimally an agent must remember speciﬁc, initially unknown past states.

Original: An instance of the problem is shown in ﬁgure 2 and a detailed explanation of the problem structure is available in the associated caption.
Revised: An instance of the problem is shown in ﬁgure 2 and a detailed explanation of the problem structure is available in the associated caption.

Original: In each training episode a new random instance of the problem is created (with the chain length, number of actions and number of decisions held ﬁxed for a particular training run).
Revised: In each training episode a new random instance of the problem is created (with the chain length, number of actions and number of decisions held ﬁxed for a particular training run).

Original: This consists of randomly choosing the ﬁnal action sequence, the location of the informative state for each decision, and the implied action and decision state for each of the uninformative states.
Revised: This consists of randomly choosing the ﬁnal action sequence, the location of the informative state for each decision, and the implied action and decision state for each of the uninformative states.

Original: All experiments are run for 3 repetitions with error bars indicating standard error in the mean over these 3 runs.
Revised: All experiments are run for 3 repetitions with error bars indicating standard error in the mean over these 3 runs.

Original: The architecture and hyper-parameters used in each experiment are identical.
Revised: The architecture and hyper-parameters used in each experiment are identical.

Original: We use the architecture from section 2 with 1 hidden layer for the value, query and write networks andhidden layers for policy.
Revised: We use the architecture from section 2 with 1 hidden layer for the value, query and write networks andhidden layers for policy.

Original: The value network and query network outputs each uses tanh activation, the policy uses softmax, and the write network uses sigmoid.
Revised: The value network and query network outputs each uses tanh activation, the policy uses softmax, and the write network uses sigmoid.

Original: Each hidden layer has 10 units.
Revised: Each hidden layer has 10 units.

Original: We train using ordinary stochastic gradient descent with learning rate 0 .
Revised: We train using ordinary stochastic gradient descent with learning rate 0 .

Original: 005 and gradient estimates generated as speciﬁed in section 3.
Revised: 005 and gradient estimates generated as speciﬁed in section 3.

Original: We also ran a recurrent baseline with full backpropagation through time which we found required more ﬁne-tuning to train with online updates.
Revised: We also ran a recurrent baseline with full backpropagation through time which we found required more ﬁne-tuning to train with online updates.

Original: To make comparison as meaningful as possible the recurrent baseline used essentially the same architecture but with the entire memory module replaced by a basic GRU (Cho et al., 2014) network of 10 units.
Revised: To make comparison as meaningful as possible the recurrent baseline used essentially the same architecture but with the entire memory module replaced by a basic GRU (Cho et al., 2014) network of 10 units.

Original: Stochastic gradient descent alone was found to give very poor results with the recurrent learner so RMSProp was used instead with learning rate 0.0005.
Revised: Stochastic gradient descent alone was found to give very poor results with the recurrent learner so RMSProp was used instead with learning rate 0.0005.

Original: Additionally to obtain reasonable results with the recurrent learner, and avoid catastrophic looping behavior, it was necessary to add a discount factor ( γ = 0 .
Revised: Additionally to obtain reasonable results with the recurrent learner, and avoid catastrophic looping behavior, it was necessary to add a discount factor ( γ = 0 .

Original: 9 , applied only for learning purposes and not used in computing the plotted return) as well as entropy regularization on the policy with a relatively low weight of 0 . 0005 .
Revised: 9 , applied only for learning purposes and not used in computing the plotted return) as well as entropy regularization on the policy with a relatively low weight of 0 . 0005 .

Original: Perhaps surprisingly neither of these were necessary with the episodic memory based system as it tended to proceed quickly through the chain without signiﬁcant looping even without discounting or entropy regularization.
Revised: Perhaps surprisingly neither of these were necessary with the episodic memory based system as it tended to proceed quickly through the chain without signiﬁcant looping even without discounting or entropy regularization.

Original: This baseline is not intended to be representative of the performance of all possible architectures based on RNN variants trained with backpropagation through time, but merely to provide context for the main experimental results of this work.
Revised: This baseline is not intended to be representative of the performance of all possible architectures based on RNN variants trained with backpropagation through time, but merely to provide context for the main experimental results of this work.

Original: Results and descriptions of the experiments are shown in ﬁgures 3, 4 and 5, in each plot the x-axis shows number of training episodes.
Revised: Results and descriptions of the experiments are shown in ﬁgures 3, 4 and 5, in each plot the x-axis shows number of training episodes.

Original: One additional experiment with twice the environment length
Revised: One additional experiment with twice the environment length

Original: [Figure]
Revised: [Figure]

Original: correct thing an to do in each decision state is to take the action suggested by the informative state with the same values of bits 6 and 7.
Revised: correct thing an to do in each decision state is to take the action suggested by the informative state with the same values of bits 6 and 7.

Original: The next bit is a decision state indicator and will be 1 if and only if the state is a decision state.
Revised: The next bit is a decision state indicator and will be 1 if and only if the state is a decision state.

Original: The ﬁnal bit is a correct path indicator and indicates for a decision state whether all the decisions made so far have been correct.
Revised: The ﬁnal bit is a correct path indicator and indicates for a decision state whether all the decisions made so far have been correct.

Original: This is necessary for our current system because without it the ﬁnal decision states all look the same and it is not possible to learn via one step updates which decision is correct at the ﬁrst decision state, in future work we would like to investigate eliminating the need for information like this by using multi-step updates or eligibility traces.
Revised: This is necessary for our current system because without it the ﬁnal decision states all look the same and it is not possible to learn via one step updates which decision is correct at the ﬁrst decision state, in future work we would like to investigate eliminating the need for information like this by using multi-step updates or eligibility traces.

Original: The particular state shown above is informative, it indicates that the correct action for the second decision state will be the top action.
Revised: The particular state shown above is informative, it indicates that the correct action for the second decision state will be the top action.

Original: Versions of this problem can be created with variable length (which we use to refer to the number of informative states plus the number of uninformative states), number of actions, and number of decision states by modifying the above description in the obvious way.
Revised: Versions of this problem can be created with variable length (which we use to refer to the number of informative states plus the number of uninformative states), number of actions, and number of decision states by modifying the above description in the obvious way.

Original: [Figure]
Revised: [Figure]

Original: is shown in appendix E. Notice that in each case the episodic memory learner was is able to learn a good query policy, drive the write weight of the uninformative states to near 0 while keeping the value for informative states much larger, and obtain close to perfect average return.
Revised: is shown in appendix E. Notice that in each case the episodic memory learner was is able to learn a good query policy, drive the write weight of the uninformative states to near 0 while keeping the value for informative states much larger, and obtain close to perfect average return.

Original: Comparing
Revised: Comparing

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ﬁgures 3 and 4 it appears that the addition of redundant memory size may accelerate initial learning though it has little effect on the overall convergence time.
Revised: ﬁgures 3 and 4 it appears that the addition of redundant memory size may accelerate initial learning though it has little effect on the overall convergence time.

Original: Comparing ﬁgures 4 and 5 the number of episodes to converge appears to roughly double from approximately 25 , 000 to 50 , 

Revised: Comparing ﬁgures 4 and 5 the number of episodes to converge appears to roughly double from approximately 25 , 000 to 50 ,

Original: with the addition of the extra decision state but the training remains quite stable.
Revised: with the addition of the extra decision state but the training remains quite stable.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present a novel algorithm for integrating a form of episodic memory with trainable reading and writing into a RL agent.
Revised: We present a novel algorithm for integrating a form of episodic memory with trainable reading and writing into a RL agent.

Original: The method depends on the observation that if we restrict the information stored in memory to be a set of past visited states, the information recorded also provides the context in which it was recorded.
Revised: The method depends on the observation that if we restrict the information stored in memory to be a set of past visited states, the information recorded also provides the context in which it was recorded.

Original: This means it is possible to assign credit to useful information without needing to backpropagate through time to when it was recorded.
Revised: This means it is possible to assign credit to useful information without needing to backpropagate through time to when it was recorded.

Original: To achieve this we devise a reservoir sampling technique which uses a sampling procedure we introduce to generate a distribution over memory conﬁgurations for which we can derive gradient estimates.
Revised: To achieve this we devise a reservoir sampling technique which uses a sampling procedure we introduce to generate a distribution over memory conﬁgurations for which we can derive gradient estimates.

Original: The whole algorithm is
Revised: The whole algorithm is

Original: O(n) in both the number of trainable parameters and the size of the memory.
Revised: O(n) in both the number of trainable parameters and the size of the memory.

Original: In particular neither memory required nor computation time increase with history length, making it feasible to run in an online RL setting.
Revised: In particular neither memory required nor computation time increase with history length, making it feasible to run in an online RL setting.

Original: We show that the resulting algorithm is able to achieve good performance on a toy problem we introduce designed to have sharp long-term dependencies which can be problematic for recurrent models.
Revised: We show that the resulting algorithm is able to achieve good performance on a toy problem we introduce designed to have sharp long-term dependencies which can be problematic for recurrent models.

Original: A CKNOWLEDGEMENTS
Revised: A CKNOWLEDGEMENTS

Original: We acknowledge the support of the Natural Sciences and Engineering Council of Canada (NSERC).
Revised: We acknowledge the support of the Natural Sciences and Engineering Council of Canada (NSERC).

Original: A D ERIVATION OF E XPRESSION FOR G RADIENT FOR O NE -S TATE M EMORY
Revised: A D ERIVATION OF E XPRESSION FOR G RADIENT FOR O NE -S TATE M EMORY

Original: [Equation]
Revised: [Equation]

Original: Working out the ﬁrst term: t
Revised: Working out the ﬁrst term: t

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:88) k
Revised: 1 (cid:88) k

Original: = ∂P
Revised: = ∂P

Original: [Equation]
Revised: [Equation]

Original: m t
Revised: m t

Original: [Equation]
Revised: [Equation]

Original: k ) ∂w
Revised: k ) ∂w

Original: [Equation]
Revised: [Equation]

Original: (cid:88) a ∈A π ( a | S t
Revised: (cid:88) a ∈A π ( a | S t

Original: [Equation]
Revised: [Equation]

Original: S k ) E t [ R t | A t
Revised: S k ) E t [ R t | A t

Original: [Equation]
Revised: [Equation]

Original: a ] = t − 1 (cid:88) k
Revised: a ] = t − 1 (cid:88) k

Original: = ∂ log( P t ( m t =
Revised: = ∂ log( P t ( m t =

Original: S k ))
Revised: S k ))

Original: ∂w
Revised: ∂w

Original: i
Revised: i

Original: P t ( m t = S k ) (cid:88) a ∈A π ( a | S t , S k ) E t
Revised: P t ( m t = S k ) (cid:88) a ∈A π ( a | S t , S k ) E t

Original: [ R t | A t = a ] = t − 1 (cid:88) k
Revised: [ R t | A t = a ] = t − 1 (cid:88) k

Original: = ∂
Revised: = ∂

Original: ∂w i (log( w k ) − log( t − 1 (cid:88)
Revised: ∂w i (log( w k ) − log( t − 1 (cid:88)

Original: j = w j ))
Revised: j = w j ))

Original: P t ( m t =
Revised: P t ( m t =

Original: S k ) (cid:88)
Revised: S k ) (cid:88)

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: Working out the second term:
Revised: Working out the second term:

Original: [Equation]
Revised: [Equation]

Original: Where we are able to drop ∂E t [ r t +1 | A t = a ] ∂w i because the immediate reward is independent of the state in memory once conditioned on the action.
Revised: Where we are able to drop ∂E t [ r t +1 | A t = a ] ∂w i because the immediate reward is independent of the state in memory once conditioned on the action.

Original: Thus we ﬁnally arrive at: ∂
Revised: Thus we ﬁnally arrive at: ∂

Original: [Equation]
Revised: [Equation]

Original: B D ERIVATION OF E XPRESSION FOR G RADIENT FOR M ULTIPLE -S TATE M EMORY
Revised: B D ERIVATION OF E XPRESSION FOR G RADIENT FOR M ULTIPLE -S TATE M EMORY

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Working out the ﬁrst term:
Revised: Working out the ﬁrst term:

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: ( Ttn )
Revised: ( Ttn )

Original: [Equation]
Revised: [Equation]

Original: )   (cid:81)
Revised: )   (cid:81)

Original: [Equation]
Revised: [Equation]

Original: ˆ T \{ i }
Revised: ˆ T \{ i }

Original: [Equation]
Revised: [Equation]

Original: ˜ T
Revised: ˜ T

Original: [Equation]
Revised: [Equation]

Original: ( Ttn ) (cid:81) j ∈ ˜ T w j   (cid:32)(cid:88)
Revised: ( Ttn ) (cid:81) j ∈ ˜ T w j   (cid:32)(cid:88)

Original: j ∈ ˆ T
Revised: j ∈ ˆ T

Original: Q ( S j | S ˆ T ) (cid:88) a ∈A π ( a | S t , S j ) E t [ R t | A t = a ] − E t
Revised: Q ( S j | S ˆ T ) (cid:88) a ∈A π ( a | S t , S j ) E t [ R t | A t = a ] − E t

Original: [ R t ] (cid:33) =
Revised: [ R t ] (cid:33) =

Original: (cid:88)
Revised: (cid:88)

Original: ˆ T ∈ ( ( Ttn ) (cid:51) i )
Revised: ˆ T ∈ ( ( Ttn ) (cid:51) i )

Original: w i P t ( M t =
Revised: w i P t ( M t =

Original: S ˆ T ) (cid:32)(cid:88)
Revised: S ˆ T ) (cid:32)(cid:88)

Original: j ∈ ˆ T
Revised: j ∈ ˆ T

Original: Q ( S j | S ˆ T ) (cid:88) a ∈A π ( a | S t , S j ) E t [ R t | A t = a ] − E t
Revised: Q ( S j | S ˆ T ) (cid:88) a ∈A π ( a | S t , S j ) E t [ R t | A t = a ] − E t

Original: [ R t ] (cid:33)
Revised: [ R t ] (cid:33)

Original: Working out the second term:
Revised: Working out the second term:

Original: [Equation]
Revised: [Equation]

Original: So all together we get: ∂
Revised: So all together we get: ∂

Original: [Equation]
Revised: [Equation]

Original: C P ROOF OF L EMMA
Revised: C P ROOF OF L EMMA

Original: Lemma 1.
Revised: Lemma 1.

Original: Selecting elements sequentially according to equation 12 will result in a vector ˆ T whose elements correspond to a sample drawn from equation 11.
Revised: Selecting elements sequentially according to equation 12 will result in a vector ˆ T whose elements correspond to a sample drawn from equation 11.

Original: Proof.
Revised: Proof.

Original: Selecting elements sequentially according to equation 12 gives the following probability for a given vector ˆ T .
Revised: Selecting elements sequentially according to equation 12 gives the following probability for a given vector ˆ T .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: T \
Revised: T \

Original: [Equation]
Revised: [Equation]

Original: T [
Revised: T [

Original: [Equation]
Revised: [Equation]

Original: (cid:81)
Revised: (cid:81)

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: To complete the proof note that this is one of n ! vectors with the same set of elements in different order, each of which will have equal probability, hence to obtain the probability for the corresponding set we simply have to multiply by n ! which gives us equation 11.
Revised: To complete the proof note that this is one of n ! vectors with the same set of elements in different order, each of which will have equal probability, hence to obtain the probability for the corresponding set we simply have to multiply by n ! which gives us equation 11.

Original: D P ROOF OF THEOREM
Revised: D P ROOF OF THEOREM

Original: We divide the proof into a series of lemmas.
Revised: We divide the proof into a series of lemmas.

Original: First we will deﬁne some notation to facilitate referencing algorithm 1 in the proofs below.
Revised: First we will deﬁne some notation to facilitate referencing algorithm 1 in the proofs below.

Original: Let T t be the set { 0 , ..., t − 1 } and ˆ T t [0 : i − 1] be the value of ˆ T [0 : i − 1] at the call to UPDATE with parameter t .
Revised: Let T t be the set { 0 , ..., t − 1 } and ˆ T t [0 : i − 1] be the value of ˆ T [0 : i − 1] at the call to UPDATE with parameter t .

Original: Note that ˆ T t [0 : − 1] = ∅ .
Revised: Note that ˆ T t [0 : − 1] = ∅ .

Original: Let Ω t
Revised: Let Ω t

Original: [ i ] and ˜Ω t [ i ] be the values of Ω
Revised: [ i ] and ˜Ω t [ i ] be the values of Ω

Original: [ i ] and ˜Ω[ i ] respectively, at the call to UPDATE with parameter t .
Revised: [ i ] and ˜Ω[ i ] respectively, at the call to UPDATE with parameter t .

Original: Let P t,i be the value of P when it is set in loop index i of the loop starting at line 4 within the call to UPDATE with parameter t. Let Ω (cid:48)(cid:48) t,i and Ω (cid:48)
Revised: Let P t,i be the value of P when it is set in loop index i of the loop starting at line 4 within the call to UPDATE with parameter t. Let Ω (cid:48)(cid:48) t,i and Ω (cid:48)

Original: t,i be the values of Ω (cid:48)(cid:48) and Ω (cid:48) respectively after they are set within index i of the loop starting at line 4 within the UPDATE call with parameter t .
Revised: t,i be the values of Ω (cid:48)(cid:48) and Ω (cid:48) respectively after they are set within index i of the loop starting at line 4 within the UPDATE call with parameter t .

Original: Let ω t,i and τ t,i be the values of ω and τ respectively at the beginning of loop index i of the loop starting at line 4 within the call to UPDATE with parameter t. Note that ω t,i =
Revised: Let ω t,i and τ t,i be the values of ω and τ respectively at the beginning of loop index i of the loop starting at line 4 within the call to UPDATE with parameter t. Note that ω t,i =

Original: w τ t,i and that:
Revised: w τ t,i and that:

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof follows from noting that the ﬁrst sum on the right side includes every term of the left sum where ˜ T does not contain ˆ t , while the second term on the right side is equivalent to summing over those ˜ T in the left sum that do contain ˆ t .
Revised: The proof follows from noting that the ﬁrst sum on the right side includes every term of the left sum where ˜ T does not contain ˆ t , while the second term on the right side is equivalent to summing over those ˜ T in the left sum that do contain ˆ t .

Original: Lemma 3.
Revised: Lemma 3.

Original: For t ≥ n :
Revised: For t ≥ n :

Original: [Equation]
Revised: [Equation]

Original: and also
Revised: and also

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Towards a proof by induction, assume the lemma holds at time t , a quick trace through algorithm 1 will show that the update leading to Ω t +

Revised: Towards a proof by induction, assume the lemma holds at time t , a quick trace through algorithm 1 will show that the update leading to Ω t +

Original: [ i ] for i ∈ { 0 , ..., n − 1 } is always:
Revised: [ i ] for i ∈ { 0 , ..., n − 1 } is always:

Original: [Equation]
Revised: [Equation]

Original: From here we can apply lemma 2, along with equation 13 as follows to get the desired result:
Revised: From here we can apply lemma 2, along with equation 13 as follows to get the desired result:

Original: [Equation]
Revised: [Equation]

Original: Now note that ˜Ω t [ n ] = 1 for all t which is also the correct value thus we have completed the induction step for the ﬁrst half of the lemma.
Revised: Now note that ˜Ω t [ n ] = 1 for all t which is also the correct value thus we have completed the induction step for the ﬁrst half of the lemma.

Original: On the other hand the update leading to ˜Ω t +1 [ i ] is as follows:
Revised: On the other hand the update leading to ˜Ω t +1 [ i ] is as follows:

Original: [Equation]
Revised: [Equation]

Original: As a second level of induction assume the lemma holds for ˜Ω t
Revised: As a second level of induction assume the lemma holds for ˜Ω t

Original: +

Revised: +

Original: [ i + 1] then applying lemma 2 we get:
Revised: [ i + 1] then applying lemma 2 we get:

Original: [Equation]
Revised: [Equation]

Original: As the base case for this second level of induction, note that ˜Ω t +1 [ n − 1] = 1 for all t which is also the correct value, thus assuming Ω t +1 is correct we will also get the correct values for ˜Ω t
Revised: As the base case for this second level of induction, note that ˜Ω t +1 [ n − 1] = 1 for all t which is also the correct value, thus assuming Ω t +1 is correct we will also get the correct values for ˜Ω t

Original: +1 .
Revised: +1 .

Original: This completes the induction step for the lemma, it remains to prove the base case.
Revised: This completes the induction step for the lemma, it remains to prove the base case.

Original: Note that Ω[ n ] and ˜Ω[ n − 1] each start at 1 .
Revised: Note that Ω[ n ] and ˜Ω[ n − 1] each start at 1 .

Original: The beginning of the algorithm before line 19 is then intended to initialize all Ω and ˜ Ω values to have the correct value at time t = n , to see that this is the case, ﬁrst note ∀ i ∈ { 0 , ..., n } :
Revised: The beginning of the algorithm before line 19 is then intended to initialize all Ω and ˜ Ω values to have the correct value at time t = n , to see that this is the case, ﬁrst note ∀ i ∈ { 0 , ..., n } :

Original: [Equation]
Revised: [Equation]

Original: ] =
Revised: ] =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Tn
Revised: Tn

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) (cid:89) ˜ t ∈ ˜ T w ˜ t
Revised: ) (cid:89) ˜ t ∈ ˜ T w ˜ t

Original: And also, by induction on the trivial i = n − 1 case, ∀ i ∈ { 0 , ..., n − 2 } : ˜Ω
Revised: And also, by induction on the trivial i = n − 1 case, ∀ i ∈ { 0 , ..., n − 2 } : ˜Ω

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: ˜ T
Revised: ˜ T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:88) ˜
Revised: (cid:88) ˜

Original: [Equation]
Revised: [Equation]

Original: ( Tn \ ˆ Tn [0: i ] n − i
Revised: ( Tn \ ˆ Tn [0: i ] n − i

Original: [Equation]
Revised: [Equation]

Original: ) (cid:89) ˜ t
Revised: ) (cid:89) ˜ t

Original: [Equation]
Revised: [Equation]

Original: ˜ T w ˜ t
Revised: ˜ T w ˜ t

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Tn
Revised: Tn

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) (cid:89) ˜ t ∈ ˜ T w ˜ t
Revised: ) (cid:89) ˜ t ∈ ˜ T w ˜ t

Original: Thus indeed the lemma holds at time n which serves as a base case for the rest of the proof.
Revised: Thus indeed the lemma holds at time n which serves as a base case for the rest of the proof.

Original: Lemma 4.
Revised: Lemma 4.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Substituting the deﬁnition from equation 12, along with the value assigned to P t,i and simplifying slightly what we wish to show is:
Revised: Substituting the deﬁnition from equation 12, along with the value assigned to P t,i and simplifying slightly what we wish to show is:

Original: [Equation]
Revised: [Equation]

Original: i −) (cid:81) j ∈ ˜ T
Revised: i −) (cid:81) j ∈ ˜ T

Original: w j (cid:80) ˜ T ∈ ( Tt \ ˆ Tt [0: i − 1] n − i ) (cid:81) j ∈ ˜ T
Revised: w j (cid:80) ˜ T ∈ ( Tt \ ˆ Tt [0: i − 1] n − i ) (cid:81) j ∈ ˜ T

Original: w j (cid:80) ˜ T ∈ ( Tt +1 \ ˆ Tt +1[0:
Revised: w j (cid:80) ˜ T ∈ ( Tt +1 \ ˆ Tt +1[0:

Original: i − 1] n − i ) (cid:81) j ∈ ˜ T w j (cid:80) ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:81) j ∈ ˜ T w j
Revised: i − 1] n − i ) (cid:81) j ∈ ˜ T w j (cid:80) ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:81) j ∈ ˜ T w j

Original: Substituting in the values of Ω t [ i ] and Ω t +1 [ i ] from lemma 3 into the above formula, it will sufﬁce to apply lemma 2 and lemma 3 to additionally show:
Revised: Substituting in the values of Ω t [ i ] and Ω t +1 [ i ] from lemma 3 into the above formula, it will sufﬁce to apply lemma 2 and lemma 3 to additionally show:

Original: [Equation]
Revised: [Equation]

Original: = (cid:88) ˜
Revised: = (cid:88) ˜

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) (cid:89) ˜ t ∈
Revised: ) (cid:89) ˜ t ∈

Original: [Equation]
Revised: [Equation]

Original: T w ˜ t + ω t,i · (cid:88)
Revised: T w ˜ t + ω t,i · (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i − 1] n − i − 1 ) (cid:89)
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i − 1] n − i − 1 ) (cid:89)

Original: ˜ t ∈ ˜ T w ˜ t = (cid:88)
Revised: ˜ t ∈ ˜ T w ˜ t = (cid:88)

Original: ˜ T ∈ ( T t +1 \ ˆ T t +1 [0: i − 1] n − i ) (cid:89) j ∈ ˜ T w j and for 0 ≤ i ≤ n −Ω (cid:48)(cid:48) t,i =
Revised: ˜ T ∈ ( T t +1 \ ˆ T t +1 [0: i − 1] n − i ) (cid:89) j ∈ ˜ T w j and for 0 ≤ i ≤ n −Ω (cid:48)(cid:48) t,i =

Original: Ω t [ i + 1] + ω t,i · ˜Ω t [ i + 1] = (cid:88)
Revised: Ω t [ i + 1] + ω t,i · ˜Ω t [ i + 1] = (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) ˜
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) ˜

Original: t ∈ ˜ T w ˜ t + ω t,i · (cid:88)
Revised: t ∈ ˜ T w ˜ t + ω t,i · (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 2 ) (cid:89) ˜
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 2 ) (cid:89) ˜

Original: t ∈ ˜ T w ˜ t = (cid:88)
Revised: t ∈ ˜ T w ˜ t = (cid:88)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst is a simple application of equation 13.
Revised: The ﬁrst is a simple application of equation 13.

Original: The second follows from a similar observation in addition to noting that ˆ T t [ i ] has been removed from each term.
Revised: The second follows from a similar observation in addition to noting that ˆ T t [ i ] has been removed from each term.

Original: For i = n − 1  Ω t,i = 1 which trivially obeys the same formula.
Revised: For i = n − 1 , Ω t,i = 1 which trivially obeys the same formula.

Original: Lemma 5.
Revised: Lemma 5.

Original: P t,i ≥ 0 .
Revised: P t,i ≥ 0 .

Original: Proof.
Revised: Proof.

Original: P t,i = 1 − (cid:80) ˜
Revised: P t,i = 1 − (cid:80) ˜

Original: T ∈ ( Tt +1 \ (ˆ Tt +1[0:
Revised: T ∈ ( Tt +1 \ (ˆ Tt +1[0:

Original: i − 1] ∪{ ˆ Tt [ i ] } ) n − i −) (cid:81) j ∈ ˜ T w j (cid:80) ˜ T ∈ ( Tt \ ˆ Tt [0: i − 1] n − i ) (cid:81) j ∈ ˜ T w j (cid:80) ˜ T ∈ ( Tt +1 \
Revised: i − 1] ∪{ ˆ Tt [ i ] } ) n − i −) (cid:81) j ∈ ˜ T w j (cid:80) ˜ T ∈ ( Tt \ ˆ Tt [0: i − 1] n − i ) (cid:81) j ∈ ˜ T w j (cid:80) ˜ T ∈ ( Tt +1 \

Original: ˆ Tt +1[0:
Revised: ˆ Tt +1[0:

Original: i − 1] n − i ) (cid:81) j ∈ ˜ T w j (cid:80) ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:81) j ∈ ˜ T w j
Revised: i − 1] n − i ) (cid:81) j ∈ ˜ T w j (cid:80) ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:81) j ∈ ˜ T w j

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (cid:88) ˜ T ∈ ( Tt +1 \
Revised: (cid:88) ˜ T ∈ ( Tt +1 \

Original: ˆ Tt +1[0:
Revised: ˆ Tt +1[0:

Original: i − 1] n − i )
Revised: i − 1] n − i )

Original: (cid:89) j ∈ ˜ T w j (cid:88)
Revised: (cid:89) j ∈ ˜ T w j (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) j ∈ ˜ T
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) j ∈ ˜ T

Original: w j ≥ (cid:88)
Revised: w j ≥ (cid:88)

Original: ˜ T ∈ ( Tt +1 \ (ˆ Tt +1[0:
Revised: ˜ T ∈ ( Tt +1 \ (ˆ Tt +1[0:

Original: i − 1] ∪{ ˆ Tt [ i ] } ) n − i −)
Revised: i − 1] ∪{ ˆ Tt [ i ] } ) n − i −)

Original: (cid:89) j ∈ ˜ T w j (cid:88)
Revised: (cid:89) j ∈ ˜ T w j (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i − 1] n − i ) (cid:89) j ∈ ˜ T
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i − 1] n − i ) (cid:89) j ∈ ˜ T

Original: w j lem.
Revised: w j lem.

Original: [Table]
Revised: [Table]

Original:   (cid:88)
Revised:   (cid:88)

Original: [Table]
Revised: [Table]

Original: T ∈ ( Tt +1 \ (ˆ
Revised: T ∈ ( Tt +1 \ (ˆ

Original: [Table]
Revised: [Table]

Original: +1[0:
Revised: +1[0:

Original: i − 1] ∪{ ˆ Tt [ i ] } ) n − i ) (cid:89) j ∈ ˜ T w j + w ˆ T t
Revised: i − 1] ∪{ ˆ Tt [ i ] } ) n − i ) (cid:89) j ∈ ˜ T w j + w ˆ T t

Original: [ i ] (cid:88) ˜ T ∈ ( Tt +1 \ (ˆ Tt +1[0:
Revised: [ i ] (cid:88) ˜ T ∈ ( Tt +1 \ (ˆ Tt +1[0:

Original: i − 1] ∪{ ˆ Tt [ i ] } ) n − i −)
Revised: i − 1] ∪{ ˆ Tt [ i ] } ) n − i −)

Original: (cid:89) j ∈ ˜ T w j   (cid:88)
Revised: (cid:89) j ∈ ˜ T w j   (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) j ∈ ˜ T
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) j ∈ ˜ T

Original: w j ≥   (cid:88)
Revised: w j ≥   (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i ) (cid:89) j ∈ ˜ T w j + w ˆ T t [ i ] (cid:88) ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) j ∈ ˜ T w j   (cid:88)
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i ) (cid:89) j ∈ ˜ T w j + w ˆ T t [ i ] (cid:88) ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) j ∈ ˜ T w j   (cid:88)

Original: ˜ T ∈ ( Tt +1 \ (ˆ Tt +1[0:
Revised: ˜ T ∈ ( Tt +1 \ (ˆ Tt +1[0:

Original: i − 1] ∪{ ˆ Tt [ i ] } ) n − i −)
Revised: i − 1] ∪{ ˆ Tt [ i ] } ) n − i −)

Original: (cid:89) j ∈ ˜ T w j
Revised: (cid:89) j ∈ ˜ T w j

Original: [Table]
Revised: [Table]

Original: (cid:88)
Revised: (cid:88)

Original: [Table]
Revised: [Table]

Original: T ∈ ( Tt +1 \ (ˆ Tt +1[0:
Revised: T ∈ ( Tt +1 \ (ˆ Tt +1[0:

Original: i − 1] ∪{ ˆ Tt [ i ] } ) n − i ) (cid:89) j ∈ ˜ T w j (cid:88)
Revised: i − 1] ∪{ ˆ Tt [ i ] } ) n − i ) (cid:89) j ∈ ˜ T w j (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) j ∈ ˜ T
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) j ∈ ˜ T

Original: w j ≥ (cid:88)
Revised: w j ≥ (cid:88)

Original: [Table]
Revised: [Table]

Original: ∈ (
Revised: ∈ (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [0: i ] n − i ) (cid:89) j ∈ ˜ T w j (cid:88)
Revised: [0: i ] n − i ) (cid:89) j ∈ ˜ T w j (cid:88)

Original: ˜ T ∈ ( Tt +1 \ (ˆ Tt +1[0:
Revised: ˜ T ∈ ( Tt +1 \ (ˆ Tt +1[0:

Original: i − 1] ∪{ ˆ Tt [ i ] } ) n − i −)
Revised: i − 1] ∪{ ˆ Tt [ i ] } ) n − i −)

Original: (cid:89) j ∈ ˜ T w j
Revised: (cid:89) j ∈ ˜ T w j

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  (cid:88)
Revised:  (cid:88)

Original: [Table]
Revised: [Table]

Original: T ∈ ( Tt \ ˆ Tt [0: i ] n − i ) (cid:89) ˜
Revised: T ∈ ( Tt \ ˆ Tt [0: i ] n − i ) (cid:89) ˜

Original: t ∈ ˜ T w ˜ t + ω t,i · (cid:88)
Revised: t ∈ ˜ T w ˜ t + ω t,i · (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) ˜
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) ˜

Original: t ∈ ˜ T w ˜ t   (cid:88)
Revised: t ∈ ˜ T w ˜ t   (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) j ∈ ˜ T
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) j ∈ ˜ T

Original: w j ≥   (cid:88)
Revised: w j ≥   (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) ˜
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) ˜

Original: t ∈ ˜ T w ˜ t + ω t,i · (cid:88)
Revised: t ∈ ˜ T w ˜ t + ω t,i · (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 2 ) (cid:89) ˜
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 2 ) (cid:89) ˜

Original: t ∈ ˜ T w ˜ t   (cid:88)
Revised: t ∈ ˜ T w ˜ t   (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i ) (cid:89) j ∈ ˜ T
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i ) (cid:89) j ∈ ˜ T

Original: w j ⇐⇒   (cid:88)
Revised: w j ⇐⇒   (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) j ∈ ˜ T w j   ≥ (cid:88) ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i ) (cid:89) j ∈ ˜ T w j (cid:88)
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 1 ) (cid:89) j ∈ ˜ T w j   ≥ (cid:88) ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i ) (cid:89) j ∈ ˜ T w j (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 2 ) (cid:89) j ∈ ˜ T
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i ] n − i − 2 ) (cid:89) j ∈ ˜ T

Original: w j ⇐⇒ (cid:88)
Revised: w j ⇐⇒ (cid:88)

Original: ˜ T ∈ ( Tt \ ˆ Tt [0: i − 1] n − i − 1 ) (cid:88)
Revised: ˜ T ∈ ( Tt \ ˆ Tt [0: i − 1] n − i − 1 ) (cid:88)

Original: ˜ T (cid:48) ∈
Revised: ˜ T (cid:48) ∈

Original: ( Tt \ ˆ Tt [0: i − 1] n − i − 1 ) (cid:89) j ∈ ˜ T w j (cid:89) j ∈ ˜ T (cid:48)
Revised: ( Tt \ ˆ Tt [0: i − 1] n − i − 1 ) (cid:89) j ∈ ˜ T w j (cid:89) j ∈ ˜ T (cid:48)

Original: w j ≥ (cid:88)
Revised: w j ≥ (cid:88)

Original: ˜
Revised: ˜

Original: T ∈ ( Tt \ ˆ Tt [0: i − 1] n − i − 2 ) (cid:88)
Revised: T ∈ ( Tt \ ˆ Tt [0: i − 1] n − i − 2 ) (cid:88)

Original: ˜ T (cid:48) ∈
Revised: ˜ T (cid:48) ∈

Original: ( Tt \ ˆ Tt [0: i − 1] n − i ) (cid:89) j ∈ ˜ T w j (cid:89) j ∈ ˜ T (cid:48)
Revised: ( Tt \ ˆ Tt [0: i − 1] n − i ) (cid:89) j ∈ ˜ T w j (cid:89) j ∈ ˜ T (cid:48)

Original: [Equation]
Revised: [Equation]

Original: It is relatively straightforward to show that the lemma holds in this last form.
Revised: It is relatively straightforward to show that the lemma holds in this last form.

Original: To do so note that except for terms for which ˜ T and ˜ T (cid:48) are identical on the left (which are not possible on the right),
Revised: To do so note that except for terms for which ˜ T and ˜ T (cid:48) are identical on the left (which are not possible on the right),

Original: Since (cid:0) 2 NN (cid:1) > (cid:0) 2 NN − 1 (cid:1) , ∀ N every term which is present on the right side is present on the left with more repetitions and thus the left side must be greater than the right and the lemma holds.
Revised: Since (cid:0) 2 NN (cid:1) > (cid:0) 2 NN − 1 (cid:1) , ∀ N every term which is present on the right side is present on the left with more repetitions and thus the left side must be greater than the right and the lemma holds.

Original: This lemma shows that P is indeed a valid probability, which means that swapping according to it in algorithm 1 is admissible.
Revised: This lemma shows that P is indeed a valid probability, which means that swapping according to it in algorithm 1 is admissible.

Original: Theorem 1.
Revised: Theorem 1.

Original: ∀ t ≥ n, 0 ≤ i ≤ n − 1 :
Revised: ∀ t ≥ n, 0 ≤ i ≤ n − 1 :

Original: [Equation]
Revised: [Equation]

Original: where [ t 0 , ..., t i ] is an arbitrary vector of unique elements of T t .
Revised: where [ t 0 , ..., t i ] is an arbitrary vector of unique elements of T t .

Original: Proof.
Revised: Proof.

Original: Let ˆ T i,t be the value of ˆ T right before making the swap decision on line 12 in loop index i of the loop starting at line 4 within the call to UPDATE with parameter t. Towards a proof by induction assume that at time index t directly prior to making the swap decision for ˆ T [ i ] at line 12 of UPDATE we have for any arbitrary vector [ t 0 , ...,
Revised: Let ˆ T i,t be the value of ˆ T right before making the swap decision on line 12 in loop index i of the loop starting at line 4 within the call to UPDATE with parameter t. Towards a proof by induction assume that at time index t directly prior to making the swap decision for ˆ T [ i ] at line 12 of UPDATE we have for any arbitrary vector [ t 0 , ...,

Original: t i − 1 ] of unique elements of T t :
Revised: t i − 1 ] of unique elements of T t :

Original: P ( ˆ T
Revised: P ( ˆ T

Original: i,t [ j ] =
Revised: i,t [ j ] =

Original: t j | ˆ T t [0 : j − 1] =
Revised: t j | ˆ T t [0 : j − 1] =

Original: [ t 0 , ..., t j − 1 ]) =
Revised: [ t 0 , ..., t j − 1 ]) =

Original: ˆ P ( t j | [ t 0 , ..., t j − 1 ]; T t , n ) , ∀
Revised: ˆ P ( t j | [ t 0 , ..., t j − 1 ]; T t , n ) , [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: s.t.
Revised: n

Original: [Equation]
Revised: [Equation]

Original: n −and for any arbitrary vector [ t (cid:48) 0 , ..., t (cid:48)
Revised: and for any arbitrary vector [ t (cid:48) 0 , ..., t (cid:48)

Original: i − 1 ] of unique elements of T t +1 :
Revised: i − 1 ] of unique elements of T t +1 :

Original: [Equation]
Revised: [Equation]

Original: That is all elements of ˆ T from 0 to i − 1 have the desired probability conditioned on ˆ T t +1 while all elements from i to n − 1 still have the desired probability when conditioned on ˆ T t .
Revised: That is all elements of ˆ T from 0 to i − 1 have the desired probability conditioned on ˆ T t +1 while all elements from i to n − 1 still have the desired probability when conditioned on ˆ T t .

Original: This is a natural inductive assumption given we have already made our swap decisions up to but not including i and are just about to make our decision for i .
Revised: This is a natural inductive assumption given we have already made our swap decisions up to but not including i and are just about to make our decision for i .

Original: Consider two mutually possible selections of ˆ T t +1 [0 : i − 1] =
Revised: Consider two mutually possible selections of ˆ T t +1 [0 : i − 1] =

Original: [ t (cid:48) 0 , ..., t (cid:48)
Revised: [ t (cid:48) 0 , ..., t (cid:48)

Original: j − 1 ] and ˆ T t [0 : i − 1] =
Revised: j − 1 ] and ˆ T t [0 : i − 1] =

Original: [ t 0 , ..., t j − 1 ] and note that together these uniquely determine the value τ t,i .
Revised: [ t 0 , ..., t j − 1 ] and note that together these uniquely determine the value τ t,i .

Original: Fixing these values, the only possible way to end up with ˆ T t +1 [ i ]
Revised: Fixing these values, the only possible way to end up with ˆ T t +1 [ i ]

Original: = ˆ t for ˆ t ∈ T t \ { t 0 , ..., t j − 1 } is to have ˆ T t
Revised: = ˆ t for ˆ t ∈ T t \ { t 0 , ..., t j − 1 } is to have ˆ T t

Original: [ i ] = ˆ t and then choose not to swap ˆ T
Revised: [ i ] = ˆ t and then choose not to swap ˆ T

Original: [ i ] at time t .
Revised: [ i ] at time t .

Original: Thus, substituting in 1 − P t,i using the expression for P t,i obtained in lemma 4 we get:
Revised: Thus, substituting in 1 − P t,i using the expression for P t,i obtained in lemma 4 we get:

Original: [Equation]
Revised: [Equation]

Original: On the other hand to end up with ˆ T t +

Revised: On the other hand to end up with ˆ T t +

Original: [ i ] =
Revised: [ i ] =

Original: τ t,i we may start with any ˆ t ∈ T t \ ( { t 0 , ..., t j − 1 } ) and then choose to swap ˆ T [ i ] at time t , in this case we get:
Revised: τ t,i we may start with any ˆ t ∈ T t \ ( { t 0 , ..., t j − 1 } ) and then choose to swap ˆ T [ i ] at time t , in this case we get:

Original: [Equation]
Revised: [Equation]

Original: Thus for any ﬁxed mutually possible selections of ˆ T t +1 [0 : i − 1] =
Revised: Thus for any ﬁxed mutually possible selections of ˆ T t +1 [0 : i − 1] =

Original: [ t (cid:48) 0 , ..., t (cid:48)
Revised: [ t (cid:48) 0 , ..., t (cid:48)

Original: j − 1 ] and ˆ T t [0 : i − 1] =
Revised: j − 1 ] and ˆ T t [0 : i − 1] =

Original: [ t 0 , ..., t j − 1 ] for both τ t,i and all other ˆ t ∈ T t \ { t (cid:48) 0 , ..., t (cid:48)
Revised: [ t 0 , ..., t j − 1 ] for both τ t,i and all other ˆ t ∈ T t \ { t (cid:48) 0 , ..., t (cid:48)

Original: j − 1 } we end up with correct conditional probabilities for ˆ T t +

Revised: j − 1 } we end up with correct conditional probabilities for ˆ T t +

Original: [ i ] assuming they are correct for ˆ T t
Revised: [ i ] assuming they are correct for ˆ T t

Original: [ i ] .
Revised: [ i ] .

Original: Now note that the resulting probabilities are ultimately independent of [ t 0 , ..., t j − 1 ] , hence we would get the same values by conditioning on [ t (cid:48) 0 , ..., t (cid:48)
Revised: Now note that the resulting probabilities are ultimately independent of [ t 0 , ..., t j − 1 ] , hence we would get the same values by conditioning on [ t (cid:48) 0 , ..., t (cid:48)

Original: j − 1 ] alone.
Revised: j − 1 ] alone.

Original: Thus if our inductive assumption holds we have for any arbitrary vector [ t (cid:48) 0 , ..., t (cid:48)
Revised: Thus if our inductive assumption holds we have for any arbitrary vector [ t (cid:48) 0 , ..., t (cid:48)

Original: i − 1 ] of unique elements of T t +1 :
Revised: i − 1 ] of unique elements of T t +1 :

Original: [Equation]
Revised: [Equation]

Original: Which completes the inductive step.
Revised: Which completes the inductive step.

Original: To prove the base case note that we initialize W such that at time n it is ﬁlled with all available items in random order.
Revised: To prove the base case note that we initialize W such that at time n it is ﬁlled with all available items in random order.

Original: It is easy to show that equation 12 implies that the probability of any ordering of a given set ˆ T is equal thus if the items in T exactly ﬁll ˆ T then ordering them at random will give the desired probabilities ˆ P ( ˆ T n [ i ] | ˆ T t +1 [0 : i − 1]; T t +1 , n ) , ∀ i s.t. 

Revised: It is easy to show that equation 12 implies that the probability of any ordering of a given set ˆ T is equal thus if the items in T exactly ﬁll ˆ T then ordering them at random will give the desired probabilities ˆ P ( ˆ T n [ i ] | ˆ T t +1 [0 : i − 1]; T t +1 , n ) , ∀ i s.t.

Original: ≤ i ≤ n − 1 , which gives us the base case to complete the inductive proof.
Revised: ≤ i ≤ n − 1 , which gives us the base case to complete the inductive proof.

Original: E L ENGTH 20 E NVIRONMENT E XPERIMENT
Revised: E L ENGTH 20 E NVIRONMENT E XPERIMENT

Original: Figure 6 shows the results of an experiment on the secret informant problem with environment length 20.
Revised: Figure 6 shows the results of an experiment on the secret informant problem with environment length 20.

Original: Comparing ﬁgures 5 and 6 the number of episodes to convergence increases from 50 ,to around 80 , 000 with a doubling of the environment length while training still remains stable
Revised: Comparing ﬁgures 5 and 6 the number of episodes to convergence increases from 50 ,to around 80 , 000 with a doubling of the environment length while training still remains stable

Original: [Figure]
Revised: [Figure]