Original: B EYOND P RIORITIZED R EPLAY : S AMPLING S TATES IN M ODEL -B ASED RL VIA S IMULATED P RIORITIES
Revised: B EYOND P RIORITIZED R EPLAY : S AMPLING S TATES IN M ODEL -B ASED RL VIA S IMULATED P RIORITIES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Model-based reinforcement learning (MBRL) can signiﬁcantly improve sample efﬁciency, particularly when carefully choosing the states from which to sample hypothetical transitions.

Original: Such prioritization has been empirically shown to be useful for both experience replay (ER) and Dyna-style planning. However, there isas yet little theoretical understanding in RL about such prioritization strategies, and why they help.
Revised: The prioritized Experience Replay (ER) method has attracted great attention; however, there is little theoretical understanding of such prioritization strategy and why they help.

Original: In this work, we revisit prioritized ER and, in an ideal setting, show equivalence to minimizing cubic loss, providing theoretical insight into why it improves upon uniform sampling.
Revised: In this work, we revisit prioritized ER and, in an ideal setting, show equivalence to minimizing cubic loss, providing theoretical insight into why it improves upon uniform sampling.

Original: This theoretical equivalence highlights two limitations of current prioritized experience replay methods: insufﬁcient coverage of the sample space and outdated priorities of training samples.
Revised: This theoretical equivalence highlights two limitations of current prioritized experience replay methods: insufﬁcient coverage of the sample space and outdated priorities of training samples.

Original: This motivates our model-based approach, which does not suffer from these limitations.
Revised: This motivates our model-based approach, which does not suffer from these limitations.

Original: Our key idea is to actively search for high priority states using gradient ascent.
Revised: Our key idea is to actively search for high priority states using gradient ascent.

Original: Under certain conditions, we prove that the hypothetical experiences generated from these states are sampled proportionally to approximately true priorities.
Revised: Under certain conditions, we prove that the hypothetical experiences generated from these states are sampled proportionally to approximately true priorities.

Original: We also characterize the distance between the sampling distribution of our method and the true prioritized sampling distribution.
Revised: We also characterize the distance between the sampling distribution of our method and the true prioritized sampling distribution.

Original: Our experiments on both benchmark and application-oriented domains show that our approach achieves superior performance over baselines.
Revised: Our experiments on both benchmark and application-oriented domains show that our approach achieves superior performance over baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Using hypothetical experience simulated from an environment model can signiﬁcantly improve sample efﬁciency of RL agents (Ha & Schmidhuber, 2018; Holland et al., 2018; Pan et al., 2018;
Revised: Using hypothetical experience simulated from an environment model can signiﬁcantly improve sample efﬁciency of RL agents (Ha & Schmidhuber, 2018; Holland et al., 2018; Pan et al., 2018;

Original: Janner et al., 2019; van Hasselt et al., 2019).
Revised: Janner et al., 2019; van Hasselt et al., 2019).

Original: Dyna (Sutton, 1991) is a classical MBRL architecture where the agent uses real experience to updates its policy as well as its reward and dynamics models.
Revised: Dyna (Sutton, 1991) is a classical MBRL architecture where the agent uses real experience to updates its policy as well as its reward and dynamics models.

Original: In-between taking actions, the agent can get hypothetical experience from the model to further improve the policy.
Revised: In-between taking actions, the agent can get hypothetical experience from the model to further improve the policy.

Original: An important question for effective Dyna-style planning is search-control : from what states should the agent simulate hypothetical transitions?
Revised: An important question for effective Dyna-style planning is search-control : from what states should the agent simulate hypothetical transitions?

Original: On each planning step in Dyna, the agent has to select a state and action from which to query the model for the next state and reward.
Revised: On each planning step in Dyna, the agent has to select a state and action from which to query the model for the next state and reward.

Original: This question, in fact, already arises in what is arguably the simplest variant of Dyna: Experience Replay (ER) (Lin, 1992).
Revised: This question, in fact, already arises in what is arguably the simplest variant of Dyna: Experience Replay (ER) (Lin, 1992).

Original: In ER, visited transitions are stored in a buffer and at each time step, a mini-batch of experiences is sampled to update the value function.
Revised: In ER, visited transitions are stored in a buffer and at each time step, a mini-batch of experiences is sampled to update the value function.

Original: ER can be seen as an instance of Dyna, using a (limited) non-parametric model given by the buffer (see van Seijen & Sutton (2015) for a deeper discussion).
Revised: ER can be seen as an instance of Dyna, using a (limited) non-parametric model given by the buffer (see van Seijen & Sutton (2015) for a deeper discussion).

Original: Performance can be signiﬁcantly improved by sampling proportionally to priorities based on errors, as in prioritized ER (Schaul et al., 2016; de Bruin et al., 2018), as well as specialized sampling for the off-policy setting (Schlegel et al., 2019).
Revised: Performance can be signiﬁcantly improved by sampling proportionally to priorities based on errors, as in prioritized ER (Schaul et al., 2016; de Bruin et al., 2018), as well as specialized sampling for the off-policy setting (Schlegel et al., 2019).

Original: Search-control strategies in Dyna similarly often rely on using priorities, though they can be more ﬂexible in leveraging the model rather than being limited to only retrieving visited experiences.
Revised: Search-control strategies in Dyna similarly often rely on using priorities, though they can be more ﬂexible in leveraging the model rather than being limited to only retrieving visited experiences.

Original: For example, a model enables the agent to sweep backwards by generating predecessors, as in prioritized sweeping (Moore & Atkeson, 1993; Sutton et al., 2008; Pan et al., 2018; Corneil et al., 2018).
Revised: For example, a model enables the agent to sweep backwards by generating predecessors, as in prioritized sweeping (Moore & Atkeson, 1993; Sutton et al., 2008; Pan et al., 2018; Corneil et al., 2018).

Original: Other methods have tried alternatives to error-based prioritization, such as searching for states with high reward (Goyal et al., 2019), high value (Pan et al., 2019) or states that are difﬁcult to learn (Pan et al., 2020).
Revised: Other methods have tried alternatives to error-based prioritization, such as searching for states with high reward (Goyal et al., 2019), high value (Pan et al., 2019) or states that are difﬁcult to learn (Pan et al., 2020).

Original: Another strategy is to directly generate hypothetical experiences from trajectory optimization algorithms (Gu et al., 2016).
Revised: Another strategy is to directly generate hypothetical experiences from trajectory optimization algorithms (Gu et al., 2016).

Original: These methods are all supported by nice intuition, but as yet lack solid theoretical reasons for why they can improve sample efﬁciency.
Revised: These methods are all supported by nice intuition, but as yet lack solid theoretical reasons for why they can improve sample efﬁciency.

Original: In this work, we provide new insights about how to choose the sampling distribution over states from which we generate hypothetical experience.
Revised: In this work, we provide new insights about how to choose the sampling distribution over states from which we generate hypothetical experience.

Original: In particular, we theoretically motivate why errorbased prioritization is effective, and provide a mechanism to generate states according to more accurate error estimates.
Revised: In particular, we theoretically motivate why errorbased prioritization is effective, and provide a mechanism to generate states according to more accurate error estimates.

Original: We ﬁrst prove that l 2 regression with error-based prioritized sampling is equivalent to minimizing a cubic objective with uniform sampling in an ideal setting.
Revised: We ﬁrst prove that l 2 regression with error-based prioritized sampling is equivalent to minimizing a cubic objective with uniform sampling in an ideal setting.

Original: We then show that minimizing the cubic power objective has a faster convergence rate during early learning stage, providing theoretical motivation for error-based prioritization.
Revised: We then show that minimizing the cubic power objective has a faster convergence rate during early learning stage, providing theoretical motivation for error-based prioritization.

Original: The theoretical understanding illuminates two issues of prioritized ER: insufﬁcient sample space coverage and outdated priorities.
Revised: The theoretical understanding illuminates two issues of prioritized ER: insufﬁcient sample space coverage and outdated priorities.

Original: To overcome the limitations, we propose a search-control strategy in Dyna that leverages a model to simulate errors and to ﬁnd states with high expected error.
Revised: To overcome the limitations, we propose a search-control strategy in Dyna that leverages a model to simulate errors and to ﬁnd states with high expected error.

Original: Finally, we demonstrate the efﬁcacy of our method on various benchmark domains and an autonomous driving application.
Revised: Finally, we demonstrate the efﬁcacy of our method on various benchmark domains and an autonomous driving application.

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: We formalize the problem as a Markov Decision Process (MDP), a tuple ( S , A , P , R, γ ) including state space S , action space A , probability transition kernel P , reward function R , and discount rate γ ∈ [0 , 1] .
Revised: We formalize the problem as a Markov Decision Process (MDP), a tuple ( S , A , P , R, γ ) including state space S , action space A , probability transition kernel P , reward function R , and discount rate γ ∈ [0 , 1] .

Original: At each environment time step t , an RL agent observes a state s t ∈ S , and takes an action a t ∈ A .
Revised: At each environment time step t , an RL agent observes a state s t ∈ S , and takes an action a t ∈ A .

Original: The environment transitions to the next state s t +1 ∼ P ( ·| s t , a t ) , and emits a scalar reward signal r t +

Revised: The environment transitions to the next state s t +1 ∼ P ( ·| s t , a t ) , and emits a scalar reward signal r t +

Original: = R ( s t , a t , s t +1 ) .
Revised: = R ( s t , a t , s t +1 ) .

Original: A policy is a mapping π : S × A → [0 , 1] that determines the probability of choosing an action at a given state. 
Revised: A policy is a mapping π : S × A → [0 , 1] that determines the probability of choosing an action at a given state. Algorithm

Original: Input: hill climbing crit.
Revised: Input: hill climbing crit.

Original: h : S (cid:55)→ R , batch-size b
Revised: h : S (cid:55)→ R , batch-size b

Original: Initialize empty search-control queue B sc ; empty
Revised: Initialize empty search-control queue B sc ; empty

Original: ER buffer B er ; initialize policy and model P for t = 1 , 2 , . . .
Revised: ER buffer B er ; initialize policy and model P for t = 1 , 2 , . . .

Original: do
Revised: do

Original: Add ( s t , a t , s t +1 , r t +1 ) to B er while within some budget time steps do s ← s + α a ∇ s h ( s ) //hill climbing
Revised: Add ( s t , a t , s t +1 , r t +1 ) to B er while within some budget time steps do s ← s + α a ∇ s h ( s ) //hill climbing

Original: Add s into B sc for n times do
Revised: Add s into B sc for n times do

Original: B ← ∅ for b/ 2 times do
Revised: B ← ∅ for b/ 2 times do

Original: Sample s ∼ B sc , on-policy action a
Revised: Sample s ∼ B sc , on-policy action a

Original: Sample s (cid:48) , r ∼ P ( s, a )
Revised: Sample s (cid:48) , r ∼ P ( s, a )

Original: B ← ( s, a, s (cid:48) , r )
Revised: B ← ( s, a, s (cid:48) , r )

Original: Sample b/ 2 experiences from B er , add to B
Revised: Sample b/ 2 experiences from B er , add to B

Original: Update policy on the mixed mini-batch B
Revised: Update policy on the mixed mini-batch B

Original: The agent’s objective is to ﬁnd an optimal policy.
Revised: The agent’s objective is to ﬁnd an optimal policy.

Original: A popular algorithm is Qlearning (Watkins & Dayan, 1992), where parameterized action-values Q θ are updated using θ = θ + αδ t ∇ θ Q θ ( s t , a t ) for stepsize α > 0 with TD-error δ t def = r t +1 + γ max a (cid:48) ∈A Q θ ( s t +1 , a (cid:48) )
Revised: A popular algorithm is Qlearning (Watkins & Dayan, 1992), where parameterized action-values Q θ are updated using θ = θ + αδ t ∇ θ Q θ ( s t , a t ) for stepsize α > 0 with TD-error δ t def = r t +1 + γ max a (cid:48) ∈A Q θ ( s t +1 , a (cid:48) )

Original: − Q θ ( s t , a t ) .
Revised: − Q θ ( s t , a t ) .

Original: The policy is deﬁned by acting greedily w.r.t.
Revised: The policy is deﬁned by acting greedily w.r.t.

Original: these action-values.
Revised: these action-values.

Original: ER is critical when using neural networks to estimate Q θ , as used in
Revised: ER is critical when using neural networks to estimate Q θ , as used in

Original: DQN (Mnih et al., 2015), both to stabilize and speed up learning.
Revised: DQN (Mnih et al., 2015), both to stabilize and speed up learning.

Original: MBRL has the potential to provide even further sample efﬁciency improvements.
Revised: MBRL has the potential to provide even further sample efﬁciency improvements.

Original: We build on the Dyna formalism (Sutton, 1991) for MBRL, and more speciﬁcally the recently proposed HC-Dyna (Pan et al., 2019) as shown in Algorithm 1.
Revised: We build on the Dyna formalism (Sutton, 1991) for MBRL, and more speciﬁcally the recently proposed HC-Dyna (Pan et al., 2019) as shown in Algorithm 1.

Original: HC-Dyna provides a particular approach to search-control—the mechanism of generating states or state-action pairs from which to query the model to get next states and rewards (i.e. hypothetical experiences).
Revised: HC-Dyna provides a particular approach to search-control—the mechanism of generating states or state-action pairs from which to query the model to get next states and rewards (i.e. hypothetical experiences).

Original: It is characterized the fact that it generates states by hill climbing on some criterion function h ( · ) .
Revised: It is characterized the fact that it generates states by hill climbing on some criterion function h ( · ) .

Original: The term Hill Climbing (HC) is used for generality as the vanilla gradient ascent procedure is modiﬁed to resolve certain challenges (Pan et al., 2019).
Revised: The term Hill Climbing (HC) is used for generality as the vanilla gradient ascent procedure is modiﬁed to resolve certain challenges (Pan et al., 2019).

Original: Two particular choices have been proposed for h ( · ) : the value function v ( s ) from Pan et al.
Revised: Two particular choices have been proposed for h ( · ) : the value function v ( s ) from Pan et al.

Original: and the gradient magnitude ||∇ s v ( s ) || from Pan et al.
Revised: and the gradient magnitude ||∇ s v ( s ) || from Pan et al.

Original: The former is used as measure of the utility of visiting a state and the latter is considered as a measure of value approximation difﬁculty.
Revised: The former is used as measure of the utility of visiting a state and the latter is considered as a measure of value approximation difﬁculty.

Revised: The hypothetical experience is obtained by ﬁrst selecting a state s , then typically selecting the action a according to the current policy, and then querying the model to get next state s (cid:48) and reward r .

Original: These hypothetical transitions are treated just like real transitions.
Revised: These hypothetical transitions are treated just like real transitions.

Original: For this reason, HC-Dyna combines both real experience and hypothetical experience into mini-batch updates.
Revised: For this reason, HC-Dyna combines both real experience and hypothetical experience into mini-batch updates.

Original: These n updates, performed before taking the next action, are called planning updates, as they improve the action-value estimates— and so the policy—using a model.
Revised: These n updates, performed before taking the next action, are called planning updates, as they improve the action-value estimates— and so the policy—using a model.

Revised: However, it should be noted that there are several limitations to the two previous works.

Revised: First, the

Revised: HC method proposed by Pan et al.

Revised: (2019) is mostly supported by intuitions, without any theoretical justiﬁcation to use the stochastic gradient ascent trajectories for search-control.

Revised: Second, the HC on gradient norm and Hessian norm of the learned value function Pan et al.

Original:   A D EEPER L OOK AT E RROR - BASED P RIORITIZED S AMPLING
Revised: (2020) is supported by some suggestive theoretical evidence, but it suffers from great computation cost and zero gradient due to the high order differentiation (i.e., ∇ s ||∇ s v ( s ) || ) as suggested by the authors. This paper will introduce our novel HC search-control method motivated by overcoming the limitations of the prioritized ER method, which has stronger theoretical support than the work by Pan et al. and improved computational cost comparing with the existed work by Pan et al.  A D EEPER L OOK AT E RROR - BASED P RIORITIZED S AMPLING

Original: In this section, we provide theoretical motivation for error-based prioritized sampling.
Revised: In this section, we provide theoretical motivation for error-based prioritized sampling.

Original: We show that prioritized sampling can be reformulated as optimizing a cubic power objective with uniform sampling.
Revised: We show that prioritized sampling can be reformulated as optimizing a cubic power objective with uniform sampling.

Original: We prove that optimizing the cubic objective provides a faster convergence rate during early learning.
Revised: We prove that optimizing the cubic objective provides a faster convergence rate during early learning.

Original: Based on these results, we highlight that prioritized ER has two limitations 1) outdated priorities and 2) insufﬁcient coverage of the sample space.
Revised: Based on these results, we highlight that prioritized ER has two limitations 1) outdated priorities and 2) insufﬁcient coverage of the sample space.

Original: This motivates our method in the next section to address the two limitations.
Revised: This motivates our method in the next section to address the two limitations.

Original: 3. P RIORITIZED S AMPLING AS A C UBIC O BJECTIVE
Revised: 3. P RIORITIZED S AMPLING AS A C UBIC O BJECTIVE

Original: In  l 2 regression, we minimize the mean squared error min θ 12 n (cid:80) ni
Revised: In the l 2 regression, we minimize the mean squared error min θ 12 n (cid:80) ni

Original: =

Revised: =

Original: ( f θ ( x i ) − y i ) 2 , for training set T = { ( x i , y i ) } ni =1 and function approximator f θ , such as a neural network.
Revised: ( f θ ( x i ) − y i ) 2 , for training set T = { ( x i , y i ) } ni =1 and function approximator f θ , such as a neural network.

Original: In error-based prioritized sampling, we deﬁne the priority of a sample ( x, y ) ∈ T as | f θ ( x ) − y | ; the probability of drawing a sample ( x, y ) ∈ T is typically q ( x, y ; θ ) ∝ | f θ ( x ) − y | .
Revised: In error-based prioritized sampling, we deﬁne the priority of a sample ( x, y ) ∈ T as | f θ ( x ) − y | ; the probability of drawing a sample ( x, y ) ∈ T is typically q ( x, y ; θ ) ∝ | f θ ( x ) − y | .

Original: We employ the following form to compute the probabilities:
Revised: We employ the following form to compute the probabilities:

Original: [Equation]
Revised: [Equation]

Original: We can show an equivalence between the gradients of the squared objective with this prioritization and the cubic power objective 13 n (cid:80) ni
Revised: We can show an equivalence between the gradients of the squared objective with this prioritization and the cubic power objective 13 n (cid:80) ni

Original: =1 | f θ ( x i ) − y
Revised: =1 | f θ ( x i ) − y

Original: i | 3 .
Revised: i | 3 .

Original: See Appendix A.3 for the proof.
Revised: See Appendix A.3 for the proof.

Original: Theorem 1.
Revised: Theorem 1.

Original: For a constant c determined by θ, T , we have
Revised: For a constant c determined by θ, T , we have

Original: [Equation]
Revised: [Equation]

Original: This simple theorem provides an intuitive reason for why prioritized sampling can help improve sample efﬁciency: the gradient direction of cubic function is sharper than that of the square function when the error is relatively large (Figure 1).
Revised: This simple theorem provides an intuitive reason for why prioritized sampling can help improve sample efﬁciency: the gradient direction of cubic function is sharper than that of the square function when the error is relatively large (Figure 1).

Original: Theorem 2 further characterizes the difference between the convergence rates by optimizing the mean square error and the cubic power objective, providing a solid motivation for using error-based prioritized sampling.
Revised: Theorem 2 further characterizes the difference between the convergence rates by optimizing the mean square error and the cubic power objective, providing a solid motivation for using error-based prioritized sampling.

Original: Theorem 2 (Fast early learning) .
Revised: Theorem 2 (Fast early learning) .

Original: Consider the following two objectives: (cid:96) 2 ( x, y ) def = 12 ( x − y ) 2 , and (cid:96) 3 ( x, y ) def = 13 | x − y | 3 .
Revised: Consider the following two objectives: (cid:96) 2 ( x, y ) def = 12 ( x − y ) 2 , and (cid:96) 3 ( x, y ) def = 13 | x − y | 3 .

Original: Denote δ t def = | x t − y | , and ˜ δ t def = | ˜ x t − y | .
Revised: Denote δ t def = | x t − y | , and ˜ δ t def = | ˜ x t − y | .

Original: Deﬁne the functional gradient ﬂow updates on these two objectives:
Revised: Deﬁne the functional gradient ﬂow updates on these two objectives:

Original: [Equation]
Revised: [Equation]

Original: Given error threshold (cid:15) ≥ 0 , deﬁne the hitting time t (cid:15) def =
Revised: Given error threshold (cid:15) ≥ 0 , deﬁne the hitting time t (cid:15) def =

Original: min t { t : δ t ≤ (cid:15) } and ˜ t (cid:15) def =
Revised: min t { t : δ t ≤ (cid:15) } and ˜ t (cid:15) def =

Original: min t { t : ˜ δ t ≤ (cid:15) } .
Revised: min t { t : ˜ δ t ≤ (cid:15) } .

Original: For any initial function value x 0 s.t.
Revised: For any initial function value x 0 s.t.

Original: δ 0 > 1 , ∃ (cid:15) 0 ∈ (0 , 1) such that ∀ (cid:15) > (cid:15) 0 , t (cid:15) ≥ ˜ t (cid:15) .
Revised: δ 0 > 1 , ∃ (cid:15) 0 ∈ (0 , 1) such that ∀ (cid:15) > (cid:15) 0 , t (cid:15) ≥ ˜ t (cid:15) .

Original: Proof.
Revised: Proof.

Original: Please see Appendix A.4.
Revised: Please see Appendix A.4.

Original: Given the same (cid:15) and the same initial value of x , ﬁrst we derive t (cid:15) =
Revised: Given the same (cid:15) and the same initial value of x , ﬁrst we derive t (cid:15) =

Original: 1 η · ln (cid:8) δ 0 (cid:15) (cid:9) , ˜ t (cid:15) =
Revised: 1 η · ln (cid:8) δ 0 (cid:15) (cid:9) , ˜ t (cid:15) =

Original: 1 η · (cid:16) 1 (cid:15) − 1 δ 0 (cid:17) .
Revised: 1 η · (cid:16) 1 (cid:15) − 1 δ 0 (cid:17) .

Original: Then we analyze the condition on (cid:15) to see when t (cid:15) ≥ ˜ t (cid:15) , i.e. minimizing the square error is slower than minimizing the cubic error.
Revised: Then we analyze the condition on (cid:15) to see when t (cid:15) ≥ ˜ t (cid:15) , i.e. minimizing the square error is slower than minimizing the cubic error.

Original: The above theorem says that when the initial error is relatively large, it is faster to get to a certain low error point with the cubic objective.
Revised: The above theorem says that when the initial error is relatively large, it is faster to get to a certain low error point with the cubic objective.

Original: We can test this in simulation, with the following minimization problems: min x ≥ 0 x 2 and min x ≥ 0 x 3 .
Revised: We can test this in simulation, with the following minimization problems: min x ≥ 0 x 2 and min x ≥ 0 x 3 .

Original: We use the hitting time formulae t (cid:15) =
Revised: We use the hitting time formulae t (cid:15) =

Original: 1 η · ln (cid:8) δ 0 (cid:15) (cid:9) , ˜ t (cid:15) =
Revised: 1 η · ln (cid:8) δ 0 (cid:15) (cid:9) , ˜ t (cid:15) =

Original: 1 η · (cid:16) 1 (cid:15) − 1 δ 0 (cid:17) derived in the proof, to compute the hitting time ratio t (cid:15)
Revised: 1 η · (cid:16) 1 (cid:15) − 1 δ 0 (cid:17) derived in the proof, to compute the hitting time ratio t (cid:15)

Original: ˜ t (cid:15) under different initial values x 0 and ﬁnal error value (cid:15) .
Revised: ˜ t (cid:15) under different initial values x 0 and ﬁnal error value (cid:15) .

Original: In Figure 1(c)(d), we can see that it usually takes a signiﬁcantly shorter time for the cubic loss to reach a certain x t with various x 0 values.
Revised: In Figure 1(c)(d), we can see that it usually takes a signiﬁcantly shorter time for the cubic loss to reach a certain x t with various x 0 values.

Original: Implications of the above theory.
Revised: Implications of the above theory.

Original: The equivalence from Theorem 1 inspires us to identify two limitations of the current prioritized ER method: 1)
Revised: The equivalence from Theorem 1 inspires us to identify two limitations of the current prioritized ER method: 1)

Original: The equivalence requires the priorities of all samples to get updated after the training parameters get updated at each time step.
Revised: The equivalence requires the priorities of all samples to get updated after the training parameters get updated at each time step.

Original: The equivalence requires the prioritized sampling distribution to be calculated on the whole training set; in an online
Revised: The equivalence requires the prioritized sampling distribution to be calculated on the whole training set; in an online

Original: RL setting, at the current time step t , we only have visited samples.
Revised: RL setting, at the current time step t , we only have visited samples.

Original: These visited samples provide a biased training set w.r.t.
Revised: These visited samples provide a biased training set w.r.t.

Original: current policy which likely does not reasonably cover the state space.
Revised: current policy which likely does not reasonably cover the state space.

Original: We will present our approach to overcome the limitations in Section 4.
Revised: We will present our approach to overcome the limitations in Section 4.

Original: In the next section, we will empirically verify our theoretical ﬁndings.
Revised: In the next section, we will empirically verify our theoretical ﬁndings.

Original: 3. E MPIRICAL D EMONSTRATION
Revised: 3. E MPIRICAL D EMONSTRATION

Original: In this section, we empirically show: 1) the practical performance of the cubic objective; 2) the importance of having sufﬁcient sample space coverage and of updating the priorities of all the training samples; and 3) the reasons for why we should not directly use a high power objective in general.
Revised: In this section, we empirically show: 1) the practical performance of the cubic objective; 2) the importance of having sufﬁcient sample space coverage and of updating the priorities of all the training samples; and 3) the reasons for why we should not directly use a high power objective in general.

Original: We refer readers to A.6 for missing details and to A.7 for additional experiments.
Revised: We refer readers to A.6 for missing details and to A.7 for additional experiments.

Original: We conduct experiments on a supervised learning task.
Revised: We conduct experiments on a supervised learning task.

Original: We generate a training set T , |T | =by uniformly sampling x ∈ [ − 2 , 2] and adding zero-mean Gaussian noise with standard deviation σ to the target f sin ( x ) values, where f sin ( x ) = sin(8 πx ) if x ∈ [ − 2 , 0) and f sin ( x ) = sin( πx ) if x ∈ [0 , 2] .
Revised: We generate a training set T , |T | =by uniformly sampling x ∈ [ − 2 , 2] and adding zero-mean Gaussian noise with standard deviation σ to the target f sin ( x ) values, where f sin ( x ) = sin(8 πx ) if x ∈ [ − 2 , 0) and f sin ( x ) = sin( πx ) if x ∈ [0 , 2] .

Original: The testing set contains 1 k samples and the targets are not noise-contaminated.
Revised: The testing set contains 1 k samples and the targets are not noise-contaminated.

Original: Pan et al.
Revised: Pan et al.

Original: (2020) show that the high frequency region [ − 2 , 0] is the main source of prediction error.
Revised: (2020) show that the high frequency region [ − 2 , 0] is the main source of prediction error.

Original: Hence we expect prioritized sampling to make a clear difference in terms of sample efﬁciency on this dataset.
Revised: Hence we expect prioritized sampling to make a clear difference in terms of sample efﬁciency on this dataset.

Original: We compare the following algorithms.
Revised: We compare the following algorithms.

Original: L2 : the l 2 regression with uniformly sampling from T .
Revised: L2 : the l 2 regression with uniformly sampling from T .

Original: Full-PrioritizedL2 : the l 2 regression with prioritized sampling according to the distribution deﬁned in equation 1, the priorities of all samples in the training set are updated after each mini-batch update.
Revised: Full-PrioritizedL2 : the l 2 regression with prioritized sampling according to the distribution deﬁned in equation 1, the priorities of all samples in the training set are updated after each mini-batch update.

Original: PrioritizedL2 : the only difference with Full-PrioritizedL2 is that only the priorities of those training examples sampled in the mini-batch are updated at each iteration, the rest of the training samples use the original priorities.
Revised: PrioritizedL2 : the only difference with Full-PrioritizedL2 is that only the priorities of those training examples sampled in the mini-batch are updated at each iteration, the rest of the training samples use the original priorities.

Original: This resembles the approach taken by vanilla Prioritized ER in the
Revised: This resembles the approach taken by vanilla Prioritized ER in the

Original: RL setting (Schaul et al., 2016).
Revised: RL setting (Schaul et al., 2016).

Original: Cubic : minimizing the cubic objective with uniformly sampling.
Revised: Cubic : minimizing the cubic objective with uniformly sampling.

Original: Power4 : min θ 1 n (cid:80) [-ni =
-] ( f θ ( x i ) − y i ) 4 with uniformly sampling.
Revised: Power4 : min θ 1 n (cid:80) [-ni =
-]n i =1 ( f θ ( x i ) − y i ) 4 with uniformly sampling.

Original: We include it to show that there is almost no gain and potential harm by using higher powers.
Revised: We include it to show that there is almost no gain and potential harm by using higher powers.

Original: We use 32 × 32 tanh layers for all algorithms and optimize the learning rate from the range { 0 . 01 , 0 . 001 , 0 . 0001 } .
Revised: We use 32 × 32 tanh layers for all algorithms and optimize the learning rate from the range { 0 . 01 , 0 . 001 , 0 . 0001 } .

Original: Figure 2 (a)-(d) shows the learning curves in terms of testing error of all the above algorithms with various settings.
Revised: Figure 2 (a)-(d) shows the learning curves in terms of testing error of all the above algorithms with various settings.

Original: 2 We identify ﬁve important observations: 1) with a small mini-batch size 128 , there is a signiﬁcant difference between Full-PrioritizedL2 and Cubic ; 2) with increased mini-batch size, although all algorithms perform better, Cubic achieves largest improvement and its behavior tends to approximate the prioritized sampling algorithm; 3) as shown in Figure 2 (a), the prioritized sampling does not show advantage when the training set is small; 4) Prioritized l 2 without updating all priorities can be signiﬁcantly worse than vanilla l 2 regression (uniform sampling); 5) when increasing the noise standard deviation σ from 0 .
Revised: 2 We identify ﬁve important observations: 1) with a small mini-batch size 128 , there is a signiﬁcant difference between Full-PrioritizedL2 and Cubic ; 2) with increased mini-batch size, although all algorithms perform better, Cubic achieves largest improvement and its behavior tends to approximate the prioritized sampling algorithm; 3) as shown in Figure 2 (a), the prioritized sampling does not show advantage when the training set is small; 4) Prioritized l 2 without updating all priorities can be signiﬁcantly worse than vanilla l 2 regression (uniform sampling); 5) when increasing the noise standard deviation σ from 0 .

Original: 1 to 0 .
Revised: 1 to 0 .

Original: 5 , all algorithms perform worse and the objectives with higher power get more hurt.
Revised: 5 , all algorithms perform worse and the objectives with higher power get more hurt.

Original: The importance of sample space coverage .
Revised: The importance of sample space coverage .

Original: Observation 1) and 2) show that a high power objective has to use a much larger mini-batch size to achieve comparable performance with the l 2 with prioritized sampling.
Revised: Observation 1) and 2) show that a high power objective has to use a much larger mini-batch size to achieve comparable performance with the l 2 with prioritized sampling.

Original: A possible reason is that prioritized sampling allows us to immediately get many samples from those high error region.
Revised: A possible reason is that prioritized sampling allows us to immediately get many samples from those high error region.

Original: Uniformly sampling, on the other hand, can get fewer of2 We those samples with a limited mini-batch size.
Revised: Uniformly sampling, on the other hand, can get fewer of those samples with a limited mini-batch size.

Original: This motivates us to test prioritized sampling with a small training set where both algorithms get fewer samples.
Revised: This motivates us to test prioritized sampling with a small training set where both algorithms get fewer samples.

Original: Figure 2(a) together with (b) indicate that prioritized sampling needs sufﬁcient samples across the sample space to maintain advantage.
Revised: Figure 2(a) together with (b) indicate that prioritized sampling needs sufﬁcient samples across the sample space to maintain advantage.

Original: This requirement is intuitive but it illuminates a serious limitation of prioritized ER in RL : only those visited real experiences from the ER buffer can get sampled.
Revised: This requirement is intuitive but it illuminates a serious limitation of prioritized ER in RL : only those visited real experiences from the ER buffer can get sampled.

Original: If the state space is large, the ER buffer likely contains only a small subset of the state space, indicating a very small training set.
Revised: If the state space is large, the ER buffer likely contains only a small subset of the state space, indicating a very small training set.

Original: Thorough priority updating.
Revised: Thorough priority updating.

Original: Observation 4) highlights the importance of using an up-to-date sampling distribution at each time step.
Revised: Observation 4) highlights the importance of using an up-to-date sampling distribution at each time step.

Original: Outdated priorities change the sampling distribution in an unpredictable manner and the learning performance can degrade.
Revised: Outdated priorities change the sampling distribution in an unpredictable manner and the learning performance can degrade.

Original: We further verify this phenomenon on the classical Mountain Car domain (Sutton & Barto, 2018; Brockman et al., 2016).
Revised: We further verify this phenomenon on the classical Mountain Car domain (Sutton & Barto, 2018; Brockman et al., 2016).

Original: Figure 2(e) shows the evaluation learning curves of different variants of Deep Q networks (DQN) corresponding to the supervised learning algorithms.
Revised: Figure 2(e) shows the evaluation learning curves of different variants of Deep Q networks (DQN) corresponding to the supervised learning algorithms.

Original: We use a small 16 × 16 ReLu NN as the Q -function.
Revised: We use a small 16 × 16 ReLu NN as the Q -function.

Original: We expect that a small NN should highlight the issue of priority updating: every mini-batch update potentially perturbs the values of many other states.
Revised: We expect that a small NN should highlight the issue of priority updating: every mini-batch update potentially perturbs the values of many other states.

Original: Hence it is likely that many experiences in the ER buffer have the wrong priorities without thorough priority updating.
Revised: Hence it is likely that many experiences in the ER buffer have the wrong priorities without thorough priority updating.

Original: We do indeed ﬁnd this to be the case, with Full-PrioritizedER performing signiﬁcantly better.
Revised: We do indeed ﬁnd this to be the case, with Full-PrioritizedER performing signiﬁcantly better.

Original: Regarding high power objectives.
Revised: Regarding high power objectives.

Original: As we discussed above, observation 1) and 2) tell us that that high power objective likely requires a large mini-batch size.
Revised: As we discussed above, observation 1) and 2) tell us that that high power objective likely requires a large mini-batch size.

Original: Ideally, it would use a batch algorithm, i.e. the whole training set, for the improved convergence rate to manifest.
Revised: Ideally, it would use a batch algorithm, i.e. the whole training set, for the improved convergence rate to manifest.

Original: This requirement makes the algorithm not scalable to larger training dataset.
Revised: This requirement makes the algorithm not scalable to larger training dataset.

Original: Observation 5) indicates another reason for why a high power objective should not be preferred: it augments the effect of noise added to the target variables.
Revised: Observation 5) indicates another reason for why a high power objective should not be preferred: it augments the effect of noise added to the target variables.

Original: In Figure 2(d), the Power4 objective suffers most from the increased target noise.
Revised: In Figure 2(d), the Power4 objective suffers most from the increased target noise.

Original:  A DDRESSING THE LIMITATIONS P RIORITIZED R EPLAY : A CQUIRING S AMPLES VIA SIMULATED PRIORITIES ON C ONTINUOUS D OMAINS
Revised:  A DDRESSING THE LIMITATIONS P RIORITIZED R EPLAY : A CQUIRING S AMPLES VIA SIMULATED PRIORITIES ON C ONTINUOUS D OMAINS

Original: In this section, we propose a method to mitigate the limitations of the conventional prioritized ER method mentioned in the above section.
Revised: In this section, we propose a method to mitigate the limitations of the conventional prioritized ER method mentioned in the above section.

Original: We start by the following theorem.
Revised: We start by the following theorem.

Original: We denote P
Revised: We denote P

Original: π ( s (cid:48) , r | s ) as the transition probability given a policy π .
Revised: π ( s (cid:48) , r | s ) as the transition probability given a policy π .

Original: Theorem 3.
Revised: Theorem 3.

Original: Sampling method.
Revised: Sampling method.

Original: Given the state s ∈ S , let v π ( ·
Revised: Given the state s ∈ S , let v π ( ·

Original: ; θ ) : S (cid:55)→ R be a differentiable value function under policy π parameterized by θ . Deﬁne: y ( s ) def = E r,s (cid:48)
Revised: ; θ ) : S (cid:55)→ R be a differentiable value function under policy π parameterized by θ . Deﬁne: y ( s ) def = E r,s (cid:48)

Original: ∼P
Revised: ∼P

Original: π ( s (cid:48) ,r | s )
Revised: π ( s (cid:48) ,r | s )

Original: [ r + γv π ( s (cid:48) ; θ )] , and denote the TD error as δ ( s, y ; θ t ) def =
Revised: [ r + γv π ( s (cid:48) ; θ )] , and denote the TD error as δ ( s, y ; θ t ) def =

Original: y ( s ) − v ( s ; θ t ) .
Revised: y ( s ) − v ( s ; θ t ) .

Original: Given some initial state s 0 ∈ S , deﬁne the state sequence { s i } as the one generated by state updating rule s i +1 ← s i
Revised: Given some initial state s 0 ∈ S , deﬁne the state sequence { s i } as the one generated by state updating rule s i +1 ← s i

Original: + α a ∇ s log | δ
Revised: + α a ∇ s log | δ

Original: ( s i , y ( s i ); θ t ) | +
Revised: ( s i , y ( s i ); θ t ) | +

Original: X i , where α a is a sufﬁciently small stepsize and X i is a Gaussian random variable with some constant variance.
Revised: X i , where α a is a sufﬁciently small stepsize and X i is a Gaussian random variable with some constant variance.

Original: Then the sequence { s i } converges to the distribution p ( s ) ∝ | δ ( s, y ( s ) | .
Revised: Then the sequence { s i } converges to the distribution p ( s ) ∝ | δ ( s, y ( s )) | .

Original: The proof is a direct consequence of the convergent behavior of Langevin dynamics stochastic differential equation (SDE) (Roberts, 1996; Welling & Teh, 2011; Zhang et al., 2017).
Revised: The proof is a direct consequence of the convergent behavior of Langevin dynamics stochastic differential equation (SDE) (Roberts, 1996; Welling & Teh, 2011; Zhang et al., 2017).

Original: We include a brief discussion and background knowledge in the Appendix A.2.
Revised: We include a brief discussion and background knowledge in the Appendix A.2.

Original: In practice, we can compute the state value estimate by v ( s ) = max a Q ( s, a ; θ t ) as suggested by Pan et al.
Revised: In practice, we can compute the state value estimate by v ( s ) = max a Q ( s, a ; θ t ) as suggested by Pan et al.

Original: In the case that a true environment model is not available, we have to compute an estimate ˆ y ( s ) of y ( s ) by a learned model.
Revised: In the case that a true environment model is not available, we have to compute an estimate ˆ y ( s ) of y ( s ) by a learned model.

Original: Then at each time step t , states approximately following the distribution p ( s ) ∝ | δ ( s, y ( s )) | can be generated by
Revised: Then at each time step t , states approximately following the distribution p ( s ) ∝ | δ ( s, y ( s )) | can be generated by

Original: [Equation]
Revised: [Equation]

Original: where X is a Gaussian random variable with zero-mean and reasonably small variance.
Revised: where X is a Gaussian random variable with zero-mean and some small variance.

Original: In the implementation, observing that α a is small, we consider ˆ y ( s ) as a constant given a state s without backpropagating through it.
Revised: In the implementation, observing that α a is small, we consider ˆ y ( s ) as a constant given a state s without backpropagating through it.

Original: We provide an upper bound in the below theorem for the difference between the sampling distribution acquired by the true model and the learned model.
Revised: We provide an upper bound in the below theorem for the difference between the sampling distribution acquired by the true model and the learned model.

Original: We denote the transition probability distribution under policy π and the true model as P π
Revised: We denote the transition probability distribution under policy π and the true model as P π

Original: ( r, s (cid:48) | s ) , and the learned model as ˆ P π ( r, s (cid:48) | s ) .
Revised: ( r, s (cid:48) | s ) , and the learned model as ˆ P π ( r, s (cid:48) | s ) .

Original: Let p ( s ) and ˆ p ( s ) be the convergent distributions described in Theorem 3 by using the true and learned models respectively.
Revised: Let p ( s ) and ˆ p ( s ) be the convergent distributions described in Theorem 3 by using the true and learned models respectively.

Original: Let d tv ( · , · ) be the total variation distance between the two probability distributions.
Revised: Let d tv ( · , · ) be the total variation distance between the two probability distributions.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then we have the following bound.
Revised: Then we have the following bound.

Original: Please see Appendix A.5 for the proof and further interpretations.
Revised: Please see Appendix A.5 for the proof and further interpretations.

Original: Theorem 4.
Revised: Theorem 4.

Original: Assume: 1) the reward magnitude is bounded | r | ≤ R max and deﬁne V max def
Revised: Assume: 1) the reward magnitude is bounded | r | ≤ R max and deﬁne V max def

Original: = R max 1 − γ ; 2) the
Revised: = R max [Equation]

Original: largest model error for a single
Revised: γ ;

Original:  state is some small value: (cid:15)s def
Revised: 2) the largest model error for a single state is some small value: (cid:15)

Original: [Equation]
Revised: [Equation]

Original: max

Original: s d tv

Original: P π

Original: ·| s ) , ˆ P π ( ·| s )) and the total model error is bounded, i e. (cid:15) def = (cid:82) s ∈S (cid:15)
Revised: ) , ˆ P π ( ·| s )) and the total model error is bounded, i.e. (cid:15) def = (cid:82) s ∈S (cid:15)

Original: s ds < ∞ .
Revised: s ds < ∞ .

Original: Then ∀ s ∈ S , | p ( s ) − ˆ p ( s ) | ≤ min( V max ( p ( s ) (cid:15) + (cid:15) s )
Revised: Then ∀ s ∈ S , | p ( s ) − ˆ p ( s ) | ≤ min( V max ( p ( s ) (cid:15) + (cid:15) s )

Original: ˆ Z [Table]
Revised: ˆ Z , V max (ˆ p ( s ) (cid:15) + (cid:15) s ) Z ) .

Original: Algorithmic details.
Revised: Algorithmic details.

Original: We present our algorithm called Dyna-TD (Temporal Difference error) in the Algorithm 3 in Appendix A.6.
Revised: We present our algorithm called Dyna-TD (Temporal Difference error) in the Algorithm 3 in Appendix A.6.

Original: Our algorithm follows Algorithm 1, particularly, we choose the function h ( s ) def = log | ˆ y ( s ) − max a Q ( s, a ; θ t ) | , i.e. run the updating rule 3 to generate states.
Revised: Our algorithm follows Algorithm 1, particularly, we choose the function h ( s ) def = log | ˆ y ( s ) − max a Q ( s, a ; θ t ) | , i.e. run the updating rule 3 to generate states.

Original: Empirical veriﬁcation of sampling distribution.
Revised: Empirical veriﬁcation of sampling distribution.

Original: We validate the efﬁcacy of our sampling method by empirically examining the distance between the sampling distribution acquired by our gradient ascent rule in equation 3 (denoted as p 1 ( · ) ) and the desired distribution computed by thorough priority updating p ∗ ( · ) of all states under the current parameter on the GridWorld domain (Pan et al., 2019)
Revised: We validate the efﬁcacy of our sampling method by empirically examining the distance between the sampling distribution acquired by our gradient ascent rule in equation 3 (denoted as p 1 ( · ) ) and the desired distribution computed by thorough priority updating p ∗ ( · ) of all states under the current parameter on the GridWorld domain (Pan et al., 2019)

Original: (Figure 3(a)), where the probability density can be conveniently approximated by discretization.
Revised: (Figure 3(a)), where the probability density can be conveniently approximated by discretization.

Original: We record the distance change when we train our Algorithm 3.
Revised: We record the distance change when we train our Algorithm 3.

Original: The distance between the sampling distribution fo Prioritized ER (denoted as p 2 ( · ) ) is also included for comparison.
Revised: The distance between the sampling distribution fo Prioritized ER (denoted as p 2 ( · ) ) is also included for comparison.

Original: All those distributions are computed by normalizing visitation counts on the discretized 50 × 50 GridWorld.
Revised: All those distributions are computed by normalizing visitation counts on the discretized 50 × 50 GridWorld.

Original: We compute the distances of p 1 , p 2 to p ∗ by two sensible weighting schemes: 1) on-policy weighting: (cid:80)j =1 d π ( s j ) | p
Revised: We compute the distances of p 1 , p 2 to p ∗ by two sensible weighting schemes: 1) on-policy weighting: (cid:80)j =1 d π ( s j ) | p

Original: i
Revised: i

Original: ( s j ) − p ∗ ( s j ) | , i ∈ { 1 , 2 } , where d π is approximated by uniformly sample 3 k states from a recency buffer; 2) uniform weighting:[-1 2500 (cid:80) 
-]
Revised: ( s j ) − p ∗ ( s j ) | , i ∈ { 1 , 2 } , where d π is approximated by uniformly sample 3 k states from a recency buffer; 2) uniform weighting:[-1 2500 (cid:80) 
-]

Original: j =1 | p i ( s j ) − p ∗ ( s j ) | , i ∈ { 1 , 2 } .
Revised: [Equation]

Original: All details are in Appendix A.
Revised: All details are in Appendix A.

Original: Figure 3(b)(c) shows that our algorithm Dyna-TD, either with a true or an online learned model, maintains a signiﬁcantly closer distance to the desired sampling distribution p ∗ than PrioritizedER under both weighting schemes.
Revised: Figure 3(b)(c) shows that our algorithm Dyna-TD, either with a true or an online learned model, maintains a signiﬁcantly closer distance to the desired sampling distribution p ∗ than PrioritizedER under both weighting schemes.

Original: Furthermore, despite the mismatch between implementation and our above Theorem 3—namely that Dyna-TD may not run enough gradient steps to reach stationary distribution—the induced sampling distribution is quite close to the one by running long gradient steps (Dyna-TD-Long), which we expect to reach stationary behavior.
Revised: Furthermore, despite the mismatch between implementation and our above Theorem 3—namely that Dyna-TD may not run enough gradient steps to reach stationary distribution—the induced sampling distribution is quite close to the one by running long gradient steps (Dyna-TD-Long), which we expect to reach stationary behavior.

Original: This indicates that the we can reduce the time cost by lowering the number of gradient steps, while keep the sampling distribution similar.
Revised: This indicates that the we can reduce the time cost by lowering the number of gradient steps, while keep the sampling distribution similar.

Original: In Figure 3(d), we further verify that given the same time budget, our algorithm achieves better performance, despite the fact that DQN and PrioritizedER are able to process many more samples.
Revised: In Figure 3(d), we further verify that given the same time budget, our algorithm achieves better performance, despite the fact that DQN and PrioritizedER are able to process many more samples.

Original: This makes the additional time spent on search-control worth it.
Revised: This makes the additional time spent on search-control worth it.

Original: E XPERIMENTS
Revised: Sample space coverage. To further illuminate that our method indeed enables a broader coverage of the model-free ER method, we visualize the DQN’s ER state distributions trained with and without prioritization respectively and our algorithm’s search-control queue state distribution. Figure 4 shows that there is a signiﬁcant difference between ER’s and our queue’s distributions. Speciﬁcally, our search-control queue distribution looks more uniformly distributed across the whole state space with slightly higher density in the middle.

Revised: This concentration corresponds to Figure 5 from the work

Revised: [Figure]

Revised: by Pan et al.

Revised: ), as the agent must learn to pass the small hole (i

Revised: [Figure]

Revised: e., a bottleneck region) to get to the goal area.

Revised: The signiﬁcantly broader coverage of our search-control queue distribution possibly explains the superior performance of our algorithm in Figure 5 (c)(d)

Revised: [Figure]

Revised:  E XPERIMENTS

Original: In this section, we empirically show that our algorithm achieves stable and consistent performance across different settings.
Revised: In this section, we empirically show that our algorithm achieves stable and consistent performance across different settings.

Original: We ﬁrst show the overall comparative performance on various benchmark domains.
Revised: We ﬁrst show the overall comparative performance on various benchmark domains.

Original: We then show that our algorithm Dyna-TD is more robust to environment noise than
Revised: We then show that our algorithm Dyna-TD is more robust to environment noise than

Original: PrioritizedER.
Revised: PrioritizedER.

Original: Last, we demonstrate the practical utility of our algorithm on an autonomous driving[Figure] vehicle application.
Revised: Last, we demonstrate the practical utility of our algorithm on an autonomous driving vehicle application.

Original: Note that Dyna-TD uses the same hill climbing parameter settings across all benchmark domains.
Revised: Note that Dyna-TD uses the same hill climbing parameter settings across all benchmark domains.

Original: We refer readers to the Appendix A.6 for any missing details.
Revised: We refer readers to the Appendix A.6 for any missing details.

Original: Baselines.
Revised: Baselines.

Original: We include the following baseline competitors.
Revised: We include the following baseline competitors.

Original: ER is DQN with a regular ER buffer without prioritized sampling.
Revised: ER is DQN with a regular ER buffer without prioritized sampling.

Original: PrioritizedER uses a priority queue to store visited experiences and each experience is sampled proportionally to its TD error magnitude.
Revised: PrioritizedER uses a priority queue to store visited experiences and each experience is sampled proportionally to its TD error magnitude.

Original: Note that, as per the original paper (Schaul et al., 2016), after each mini-batch update, only the priorities of those samples in the mini-batch are updated.
Revised: Note that, as per the original paper (Schaul et al., 2016), after each mini-batch update, only the priorities of those samples in the mini-batch are updated.

Original: Dyna-Value
Revised: Dyna-Value

Original: (Pan et al., 2019) is the Dyna variant which performs hill climbing on value function to acquire states to populate the search-control queue.
Revised: (Pan et al., 2019) is the Dyna variant which performs hill climbing on value function to acquire states to populate the search-control queue.

Original: Dyna-Frequency (Pan et al., 2020) is the Dyna variant which performs hill climbing on the norm of the gradient of the value function to acquire states to populate the search-control queue.
Revised: Dyna-Frequency (Pan et al., 2020) is the Dyna variant which performs hill climbing on the norm of the gradient of the value function to acquire states to populate the search-control queue.

Original: Overall Performance.
Revised: Overall Performance.

Original: Figure 4 shows the overall performance of different algorithms on Acrobot,
Revised: Figure 5 shows the overall performance of different algorithms on Acrobot,

Original: CartPole, GridWorld (Figure 3(a)) and MazeGridWorld (Figure 4(g)).
Revised: CartPole, GridWorld (Figure 3(a)) and MazeGridWorld (Figure 5(g)).

Original: Our key observations are: 1)
Revised: Our key observations are: 1)

Original: Dyna-Value or Dyna-Frequency may converge to a sub-optimal policy when using a large number of planning steps; 2) Dyna-Frequency has clearly inconsistent performance across different domains; 3) our algorithm performs the best in most cases: even with an online learned model, our algorithm outperforms others on most of the tasks.
Revised: Dyna-Value or Dyna-Frequency may converge to a sub-optimal policy when using a large number of planning steps; 2) Dyna-Frequency has clearly inconsistent performance across different domains; 3) our algorithm performs the best in most cases: even with an online learned model, our algorithm outperforms others on most of the tasks.

Original: Our interpretations of those observations are as follows.
Revised: Our interpretations of those observations are as follows.

Original: First, we can think about the case where some states have high value but low TD error.
Revised: First, we can think about the case where some states have high value but low TD error.

Original: Dyna-Value may still frequently generate those states; this can waste samples and even incur sampling distribution bias, which can lead to a sub-optimal policy.
Revised: Dyna-Value may still frequently generate those states; this can waste samples and even incur sampling distribution bias, which can lead to a sub-optimal policy.

Original: This sub-optimality can be clearly observed on Acrobot, GridWorld and MazeGridWorld.
Revised: This sub-optimality can be clearly observed on Acrobot, GridWorld and MazeGridWorld.

Original: Similar reasoning applies to Dyna-Frequency.
Revised: Similar reasoning applies to Dyna-Frequency.

Original: Second, for Dyna-Frequency, as indicated by the original paper Pan et al.
Revised: Second, for Dyna-Frequency, as indicated by the original paper Pan et al.

Original: (2020), the gradient or Hessian norm have very different numerical scales and highly depend on the choice of the function approximator or domain.
Revised: (2020), the gradient or Hessian norm have very different numerical scales and highly depend on the choice of the function approximator or domain.

Original: This indicates that the algorithm requires ﬁnely tuned parameter settings, as the testing domain is varied, which possibly explains its inconsistent performances across domains.
Revised: This indicates that the algorithm requires ﬁnely tuned parameter settings, as the testing domain is varied, which possibly explains its inconsistent performances across domains.

Original: Third, notice that even though each algorithm runs the same number of planning steps, the model-based algorithms perform signiﬁcantly better.
Revised: Third, notice that even though each algorithm runs the same number of planning steps, the model-based algorithms perform signiﬁcantly better.

Original: This provides evidence for the beneﬁts of leveraging the generalization power of the learned value function and a model.
Revised: This provides evidence for the beneﬁts of leveraging the generalization power of the learned value function and a model.

Original: In contrast, model-free methods can only utilize visited states.
Revised: In contrast, model-free methods can only utilize visited states.

Original: Robustness to Noise.
Revised: Robustness to Noise.

Original: As a corresponding experiment to the supervised learning setting in Section 3, we show that our algorithm is more robust to increased noise variance than PrioritizedER.
Revised: As a corresponding experiment to the supervised learning setting in Section 3, we show that our algorithm is more robust to increased noise variance than PrioritizedER.

Original: Figureshows the evaluation learning curves on Mountain Car with planning steps 10 , 30 and reward noise standard deviation σ ∈ { 0 , 0 . 1 } .
Revised: Figureshows the evaluation learning curves on Mountain Car with planning steps 10 , 30 and reward noise standard deviation σ ∈ { 0 , 0 . 1 } .

Original: We would like to identify three key observations.
Revised: We would like to identify three key observations.

Original: First, our algorithm’s relative performance to PrioritizedER resembles the Full-PrioritizedL2 to PrioritizedL2 from the supervised learning setting, as Full-PrioritizedL2 is more robust to target noise than PrioritizedL2.
Revised: First, our algorithm’s relative performance to PrioritizedER resembles the Full-PrioritizedL2 to PrioritizedL2 from the supervised learning setting, as Full-PrioritizedL2 is more robust to target noise than PrioritizedL2.

Original: Second, our algorithm achieves almost the same performance as Dyna-Frequency which is claimed to be robust to noise by Pan et al.
Revised: Second, our algorithm achieves almost the same performance as Dyna-Frequency which is claimed to be robust to noise by Pan et al.

Original: Last, as observed on other environments, usually all algorithms can beneﬁt from the increased number of planning steps; however, PrioritizedER and ER clearly degrade when using more planning steps with noise present.
Revised: Last, as observed on other environments, usually all algorithms can beneﬁt from the increased number of planning steps; however, PrioritizedER and ER clearly degrade when using more planning steps with noise present.

Original: Practical Utility in Autonomous Driving Application.
Revised: Practical Utility in Autonomous Driving Application.

Original: We study the practical utility of our method in an autonomous driving application (Leurent, 2018) with an online learned model.
Revised: We study the practical utility of our method in an autonomous driving application (Leurent, 2018) with an online learned model.

Original: As shown in
Revised: As shown in

Original: Figure 6 (a), we test on the roundabout-v0 domain, where the agent (i.e. the green car) should learn[Figure] to go through a roundabout without collisions while maintaining as high speed as possible.
Revised: Figure 7 (a), we test on the roundabout-v0 domain, where the agent (i.e. the green car) should learn to go through a roundabout without collisions while maintaining as high speed as possible.

Original: We would like to emphasize that there is a signiﬁcantly lower number of car crashes with the policy learned by our algorithm on both domains as we show in Figure 6(b).
Revised: We would like to emphasize that there is a signiﬁcantly lower number of car crashes with the policy learned by our algorithm on both domains as we show in Figure 7(b).

Original: This coincides with our intuition.
Revised: This coincides with our intuition.

Original: The crash should incur high temporal difference error and our method of actively searching such states by gradient ascent.
Revised: The crash should incur high temporal difference error and our method of actively searching such states by gradient ascent.

Original: This ensures the agent gets sufﬁcient training in this states during the planning stage, so that it learns to avoid them
Revised: This ensures the agent gets sufﬁcient training in this states during the planning stage, so that it learns to avoid them.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work we provide theoretical justiﬁcation for why prioritized ER can help improve sample efﬁciency.
Revised: In this work, we provide theoretical justiﬁcation for why prioritized ER can help improve sample efﬁciency.

Original: We identify crucial factors for it to be effective: sample space coverage and thorough priority updating.
Revised: We identify crucial factors for it to be effective: sample space coverage and thorough priority updating.

Original: We then propose to sample states by Langevine dynamics and conduct experiments to showthe efﬁcacy of our method.
Revised: We then propose to sample states by Langevin dynamics and conduct experiments to show our method’s efﬁcacy.

Original: There are several interesting directions for future work.
Revised: There are several interesting directions for future work.

Original: One is to study the effects of model error on sample efﬁciency with our search control strategy.
Revised: One is to study the effects of model error on sample efﬁciency with our search control strategy.

Original: Another is to apply our method with a feature-to-feature model, which can improvescalability of our method.
Revised: Another is to apply our method with a feature-to-feature model, which can improve our method’s scalability.

Revised: On the theory side, our cubic objective explains the original TD-error based prioritized ER (Schaul et al., 2016).

Original: A A PPENDIX
Revised: However, there are other types of choices beyond TD-error based prioritization, such as distribution location or reward-based prioritization (Lambert et al., 2020). Whether these alternative prioritizations can also be formulated as surrogate objectives are interesting future directions. In a concurrent work, Fujimoto et al. (2020) established an equivalence between loss functions and sampling distributions, which bears similarities to our Theorem 1.

Revised: However, it is not clear if similar optimization beneﬁts shown in our Theorem 2 are enjoyed by more general loss functions and sampling distributions, which requires further investigations

Original: In Section A.1, we introduce some background in Dyna architecture.
Revised: In Section A.1, we introduce some background in Dyna architecture.

Original: We brieﬂy discuss Langevin dynamics and its computation cost in our case in Section A.2.
Revised: We brieﬂy discuss Langevin dynamics and its computation cost in our case in Section A.2.

Original: We then provide the full proof of
Revised: We then provide the full proof of

Original: Theorem 2 in Section A.4.
Revised: Theorem 2 in Section A.4.

Original: We present the proof for Theorem 4 in Section A.5.
Revised: We present the proof for Theorem 4 in Section A.5.

Original: Details for reproducible research are in Section A.6.
Revised: Details for reproducible research are in Section A.6.

Original: We provide supplementary experimental results in Section A.7.
Revised: We provide supplementary experimental results in Section A.7.

Original: A.
Revised: A.

Original: B ACKGROUND IN D YNA
Revised: B ACKGROUND IN D YNA

Original: Dyna integrates model-free and model-based policy updates in an online RL setting (Sutton, 1990).
Revised: Dyna integrates model-free and model-based policy updates in an online RL setting (Sutton, 1990).

Original: As shown in Algorithm 2, at each time step, a Dyna agent uses the real experience to learn a model and performs model-free policy update.
Revised: As shown in Algorithm 2, at each time step, a Dyna agent uses the real experience to learn a model and performs model-free policy update.

Original: During the planning stage, simulated experiences are acquired from the model to further improve the policy.
Revised: During the planning stage, simulated experiences are acquired from the model to further improve the policy.

Original: It should be noted that the concept of planning refers to any computational process which leverages a model to improve policy, according to Sutton & Barto (2018).
Revised: It should be noted that the concept of planning refers to any computational process which leverages a model to improve policy, according to Sutton & Barto (2018).

Original: The mechanism of generating states or state-action pairs from which to query the model is called search-control , which is of critical importance to the sample efﬁciency.
Revised: The mechanism of generating states or state-action pairs from which to query the model is called search-control , which is of critical importance to the sample efﬁciency.

Original: There are abundant existing works (Moore & Atkeson, 1993;
Revised: There are abundant existing works (Moore & Atkeson, 1993;

Original: Sutton et al., 2008; Gu et al., 2016; Pan et al., 2018; Corneil et al., 2018; Goyal et al., 2019; Janner et al., 2019; Pan et al., 2019) report different level of sample efﬁciency improvements by using different way of generating hypothetical experiences during the planning stage.
Revised: Sutton et al., 2008; Gu et al., 2016; Pan et al., 2018; Corneil et al., 2018; Goyal et al., 2019; Janner et al., 2019; Pan et al., 2019) report different level of sample efﬁciency improvements by using different way of generating hypothetical experiences during the planning stage.

Original: Algorithm 2 Tabular Dyna
Revised: Algorithm 2 Tabular Dyna

Original: Initialize Q ( s, a ) ; initialize model M ( s, a ) , ∀ ( s, a ) ∈ S
Revised: Initialize Q ( s, a ) ; initialize model M ( s, a ) , ∀ ( s, a ) ∈ S

Original: × A while true do observe s , take action a by (cid:15) -greedy w.r.t Q ( s, · ) execute a , observe reward R and next State s (cid:48)
Revised: × A while true do observe s , take action a by (cid:15) -greedy w.r.t Q ( s, · ) execute a , observe reward R and next State s (cid:48)

Original: Q-learning update for Q ( s, a ) update model M ( s, a ) (i.e. by counting) store ( s, a ) into search-control queue for i=1:
Revised: Q-learning update for Q ( s, a ) update model M ( s, a ) (i.e. by counting) store ( s, a ) into search-control queue for i=1:

Original: d do sample (˜ s, ˜ a ) from search-control queue (˜ s (cid:48) , ˜ R ) ← M (˜ s, ˜ a ) //
Revised: d do sample (˜ s, ˜ a ) from search-control queue (˜ s (cid:48) , ˜ R ) ← M (˜ s, ˜ a ) //

Original: simulated transition
Revised: simulated transition

Original: Q-learning update for Q (˜ s, ˜ a ) // planning update
Revised: Q-learning update for Q (˜ s, ˜ a ) // planning update

Original: A. D ISCUSSION ON THE L ANGEVIN D YNAMICS
Revised: A. D ISCUSSION ON THE L ANGEVIN D YNAMICS

Original: Deﬁne a SDE: d W ( t ) = ∇ U ( W t )
Revised: Deﬁne a SDE: d W ( t ) = ∇ U ( W t )

Original: d t + √ 2d B t , where B t ∈ R
Revised: d t + √ 2d B t , where B t ∈ R

Original: d is a d -dimensional Brownian motion and U is a continuous differentiable function.
Revised: d is a d -dimensional Brownian motion and U is a continuous differentiable function.

Original: It turns out that the Langevin diffusion ( W t ) t ≥ 0 converges to a unique invariant distribution p ( x ) ∝ exp ( U ( x ))
Revised: It turns out that the Langevin diffusion ( W t ) t ≥ 0 converges to a unique invariant distribution p ( x ) ∝ exp ( U ( x ))

Original: (Chiang et al., 1987).
Revised: (Chiang et al., 1987).

Original: By applying the Euler-Maruyama discretization scheme to the SDE, we acquire the discretized version
Revised: By applying the Euler-Maruyama discretization scheme to the SDE, we acquire the discretized version

Original: Y k +

Revised: Y k +

Original: =
Revised: =

Original: Y k
Revised: Y k

Original: + α k +1 ∇ U
Revised: + α k +1 ∇ U

Original: ( Y k )
Revised: ( Y k )

Original: + √ 

Revised: + √

Original: α k +1 Z k +1 where ( Z k ) k ≥ 1 is an i.i.d.
Revised: α k +1 Z k +1 where ( Z k ) k ≥ 1 is an i.i.d.

Original: sequence of standard ddimensional Gaussian random vectors and ( α k ) k ≥ 1 is a sequence of step sizes.
Revised: sequence of standard ddimensional Gaussian random vectors and ( α k ) k ≥ 1 is a sequence of step sizes.

Original: It has been proved that the limiting distribution of the sequence ( Y k ) k ≥ 1 converges to the invariant distribution of the underlying SDE Roberts (1996); Durmus & Moulines (2017).
Revised: It has been proved that the limiting distribution of the sequence ( Y k ) k ≥ 1 converges to the invariant distribution of the underlying SDE Roberts (1996); Durmus & Moulines (2017).

Original: As a result, considering U ( · ) as δ ( · ) ,
Revised: As a result, considering U ( · ) as δ ( · ) ,

Original: Y as s completes the proof for Theorem 3.
Revised: Y as s completes the proof for Theorem 3.

Original: Computational time cost.
Revised: Computational time cost.

Original: It should be noted that the Langevin Dynamics Monte Carlo method we used for generating states does introduce additional computation time cost.
Revised: It should be noted that the Langevin Dynamics Monte Carlo method we used for generating states does introduce additional computation time cost.

Original: However, in the main body of the paper, we already show that the time cost worths 3(d).
Revised: However, in the main body of the paper, we already show that the time cost worths 3(d).

Original: In theory, the computational time is reasonably small.
Revised: In theory, the computational time is reasonably small.

Original: Although each gradient ascent step takes one backpropagation, this backpropagation is w.r.t.
Revised: Although each gradient ascent step takes one backpropagation, this backpropagation is w.r.t.

Original: a single state, not a mini-batch.
Revised: a single state, not a mini-batch.

Original: Let the mini-batch size of updating DQN be b , and the number of gradient steps be k .
Revised: Let the mini-batch size of updating DQN be b , and the number of gradient steps be k .

Original: If we assume one mini-batch update takes O ( c ) , then the time cost of Dyna-TD is O ( kc/b ) .
Revised: If we assume one mini-batch update takes O ( c ) , then the time cost of Dyna-TD is O ( kc/b ) .

Original: We would like to highlight that, though our approach incur higher computational cost, but it is able use fewer samples (i.e. fewer physical interactions with real environment) to achieve better performance.
Revised: We would like to highlight that, though our approach incur higher computational cost, but it is able use fewer samples (i.e. fewer physical interactions with real environment) to achieve better performance.

Original: The cost of physical interactions could be higher in practical situations.
Revised: The cost of physical interactions could be higher in practical situations.

Original: When really needed, there are intuitive engineering tricks to reduce the computational cost.
Revised: When really needed, there are intuitive engineering tricks to reduce the computational cost.

Original: For example, we can learn a low dimensional embedding ﬁrst and build a model in such space; and then the gradient ascent can be done w.r.t.
Revised: For example, we can learn a low dimensional embedding ﬁrst and build a model in such space; and then the gradient ascent can be done w.r.t.

Original: the feature instead of the raw input/observation itself.
Revised: the feature instead of the raw input/observation itself.

Original: To further save time, we can also sacriﬁce a bit the accuracy and do the gradient ascent every certain number of time steps to lower down the amortized cost.
Revised: To further save time, we can also sacriﬁce a bit the accuracy and do the gradient ascent every certain number of time steps to lower down the amortized cost.

Original: Reducing time cost is not our current focus.
Revised: Reducing time cost is not our current focus.

Original: A. P ROOF FOR T HEOREM
Revised: A. P ROOF FOR T HEOREM

Original: Theorem 1.
Revised: Theorem 1.

Original: For a constant c determined by θ, T , we have
Revised: For a constant c determined by θ, T , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is very intuitive.
Revised: The proof is very intuitive.

Original: The expected gradient of the uniform sampling method is
Revised: The expected gradient of the uniform sampling method is

Original: [Equation]
Revised: [Equation]

Original: Setting c =
Revised: Setting c =

Original: [Equation]
Revised: [Equation]

Original: completes the proof.
Revised: completes the proof.

Original: A. P ROOF FOR T HEOREM
Revised: A. P ROOF FOR T HEOREM

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider the following two objectives: (cid:96) 2 ( x, y ) def = 12 ( x − y ) 2 , and (cid:96) 3 ( x, y ) def = 13 | x − y | 3 .
Revised: Consider the following two objectives: (cid:96) 2 ( x, y ) def = 12 ( x − y ) 2 , and (cid:96) 3 ( x, y ) def = 13 | x − y | 3 .

Original: Denote δ t def = | x t − y | , and ˜ δ t def = | ˜ x t − y | .
Revised: Denote δ t def = | x t − y | , and ˜ δ t def = | ˜ x t − y | .

Original: Deﬁne the functional gradient ﬂow updates on these two objectives:
Revised: Deﬁne the functional gradient ﬂow updates on these two objectives:

Original: [Equation]
Revised: [Equation]

Original: Given error threshold (cid:15) ≥ 0 , deﬁne the hitting time t (cid:15) def =
Revised: Given error threshold (cid:15) ≥ 0 , deﬁne the hitting time t (cid:15) def =

Original: min t { t : δ t ≤ (cid:15) } and ˜ t (cid:15) def =
Revised: min t { t : δ t ≤ (cid:15) } and ˜ t (cid:15) def =

Original: min t { t : ˜ δ t ≤ (cid:15) } .
Revised: min t { t : ˜ δ t ≤ (cid:15) } .

Original: For any initial function value x 0 s.t.
Revised: For any initial function value x 0 s.t.

Original: δ 0 > 1 , ∃ (cid:15) 0 ∈ (0 , 1) such that ∀ (cid:15) > (cid:15) 0 , t (cid:15) ≥ ˜ t (cid:15) .
Revised: δ 0 > 1 , ∃ (cid:15) 0 ∈ (0 , 1) such that ∀ (cid:15) > (cid:15) 0 , t (cid:15) ≥ ˜ t (cid:15) .

Original: Proof.
Revised: Proof.

Original: For the gradient ﬂow update on the (cid:96) 2 objective, we have,
Revised: For the gradient ﬂow update on the (cid:96) 2 objective, we have,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (8) which implies,
Revised: (8) which implies,

Original: [Equation]
Revised: [Equation]

Original: Taking integral
Revised: Taking integral

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to (letting δ t = (cid:15) ),
Revised: which is equivalent to (letting δ t = (cid:15) ),

Original: [Equation]
Revised: [Equation]

Original: On the other hand, for the gradient ﬂow update on the (cid:96) 3 objective, we have,
Revised: On the other hand, for the gradient ﬂow update on the (cid:96) 3 objective, we have,

Original: [Equation]
Revised: [Equation]

Original: which implies,
Revised: which implies,

Original: [Equation]
Revised: [Equation]

Original: Taking integral
Revised: Taking integral

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to (letting ˜ δ t
Revised: which is equivalent to (letting ˜ δ t

Original: = (cid:15) ), ˜
Revised: = (cid:15) ), ˜

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then we have,
Revised: Then we have,

Original: [Equation]
Revised: [Equation]

Original: Deﬁne the function f ( x ) =
Revised: Deﬁne the function f ( x ) =

Original: ln 1 x − 1 x , x > 0 is continuous and max x> 0 f ( x ) =
Revised: ln 1 x − 1 x , x > 0 is continuous and max x> 0 f ( x ) =

Original: f (1) = − 1 .
Revised: f (1) = − 1 .

Original: We have lim x → 0 f ( x ) =
Revised: We have lim x → 0 f ( x ) =

Original: lim x →∞ f ( x ) =
Revised: lim x →∞ f ( x ) =

Original: −∞ , and f ( · ) is monotonically increasing for x ∈ (0 , 1] and monotonically decreasing for x ∈ (1 , ∞ ) .
Revised: −∞ , and f ( · ) is monotonically increasing for x ∈ (0 , 1] and monotonically decreasing for x ∈ (1 , ∞ ) .

Original: Given δ 0 > 1 , we have f ( δ 0 ) <
Revised: Given δ 0 > 1 , we have f ( δ 0 ) <

Original: f (1) = − 1 .
Revised: f (1) = − 1 .

Original: Using the intermediate value theorem for f ( · ) on (0 , 1] , we have ∃ (cid:15) 0 < 1 , such that f ( (cid:15) 0 ) =
Revised: Using the intermediate value theorem for f ( · ) on (0 , 1] , we have ∃ (cid:15) 0 < 1 , such that f ( (cid:15) 0 ) =

Original: f ( δ 0 ) .
Revised: f ( δ 0 ) .

Original: Since f ( · ) is monotonically increasing on (0 , 1] and monotonically decreasing on (1 , ∞ ) , for any (cid:15) ∈ [ (cid:15) 0 , δ 0 ] , we have f ( (cid:15) ) ≥ f ( δ 0 ) .
Revised: Since f ( · ) is monotonically increasing on (0 , 1] and monotonically decreasing on (1 , ∞ ) , for any (cid:15) ∈ [ (cid:15) 0 , δ 0 ] , we have f ( (cid:15) ) ≥ f ( δ 0 ) .

Original: Hence we have,
Revised: Hence we have,

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: Figure 7 shows the function f ( x ) =
Revised: Figure 8 shows the function f ( x ) =

Original: ln 1 x − 1 x , x > 0 .
Revised: ln 1 x − 1 x , x > 0 .

Original: Fix arbitrary x (cid:48) > 1 , there will be another root (cid:15) 0 < 1 s.t. f
Revised: Fix arbitrary x (cid:48) > 1 , there will be another root (cid:15) 0 < 1 s.t. f

Original: ( (cid:15) 0 )
Revised: ( (cid:15) 0 )

Original: =
Revised: =

Original: f ( x (cid:48) ) .
Revised: f ( x (cid:48) ) .

Original: However, there is no real-valued solution for (cid:15) 0 .
Revised: However, there is no real-valued solution for (cid:15) 0 .

Original: The solution in C is (cid:15) 

Revised: The solution in C is (cid:15)

Original: = −
Revised: = −

Original: [Equation]
Revised: [Equation]

Original: ) , where W ( · ) is a Wright Omega function.
Revised: ) , where W ( · ) is a Wright Omega function.

Original: Hence, ﬁnding the exact value of (cid:15) 0 would require a deﬁnition of ordering on complex plane.
Revised: Hence, ﬁnding the exact value of (cid:15) 0 would require a deﬁnition of ordering on complex plane.

Original: Our current theorem statement is sufﬁcient for the purpose of characterizing convergence rate.
Revised: Our current theorem statement is sufﬁcient for the purpose of characterizing convergence rate.

Original: The theorem states that there always exists some desired low error level < 1 , minimizing the square loss converges slower than the cubic loss.
Revised: The theorem states that there always exists some desired low error level < 1 , minimizing the square loss converges slower than the cubic loss.

Original: A. P ROOF FOR T HEOREM
Revised: A. P ROOF FOR T HEOREM

Original: We now provide the error bound for Theorem 4.
Revised: We now provide the error bound for Theorem 4.

Original: We denote the transition probability distribution under policy π with the true model as P π ( r, s (cid:48) | s ) ; denote that with the learned model as ˆ P π ( r, s (cid:48) | s ) .
Revised: We denote the transition probability distribution under policy π with the true model as P π ( r, s (cid:48) | s ) ; denote that with the learned model as ˆ P π ( r, s (cid:48) | s ) .

Original: Let p ( s ) and ˆ p ( s ) be the convergent distributions described in Theorem 3 by using true model and learned model respectively.
Revised: Let p ( s ) and ˆ p ( s ) be the convergent distributions described in Theorem 3 by using true model and learned model respectively.

Original: Let d tv ( · , · ) be the total variation distance between two probability distributions.
Revised: Let d tv ( · , · ) be the total variation distance between two probability distributions.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then we have the following bound.
Revised: Then we have the following bound.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Theorem 4.
Revised: Theorem 4.

Original: Assume: 1) the reward magnitude is bounded | r | ≤ R max and deﬁne V max def
Revised: Assume: 1) the reward magnitude is bounded | r | ≤ R max and deﬁne V max def

Original: =
Revised: =

Original: R max 1 − γ ; 2) the largest model error for a single state is some small value: (cid:15) s def = max s d tv ( P π ( ·| s ) , ˆ P π ( ·| s )) and the total model error is bounded, i.e. (cid:15) def = (cid:82) s ∈S (cid:15)
Revised: R max 1 − γ ; 2) the largest model error for a single state is some small value: (cid:15) s def = max s d tv ( P π ( ·| s ) , ˆ P π ( ·| s )) and the total model error is bounded, i.e. (cid:15) def = (cid:82) s ∈S (cid:15)

Original: s ds < ∞ .
Revised: s ds < ∞ .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, we bound the estimated temporal difference error.
Revised: First, we bound the estimated temporal difference error.

Original: Fix an arbitrary state s ∈ S , it is sufﬁcient the consider the case u ( s ) >
Revised: Fix an arbitrary state s ∈ S , it is sufﬁcient the consider the case u ( s ) >

Original: ˆ u ( s ) , then
Revised: ˆ u ( s ) , then

Original: [Equation]
Revised: [Equation]

Original: Now, we show that | Z − ˆ Z | ≤ V max (cid:15) .
Revised: Now, we show that | Z − ˆ Z | ≤ V max (cid:15) .

Original: [Equation]
Revised: [Equation]

Original: ds
Revised: ds

Original: [Equation]
Revised: [Equation]

Original: Consider the case p ( s ) >
Revised: Consider the case p ( s ) >

Original: ˆ p ( s ) ﬁrst.
Revised: ˆ p ( s ) ﬁrst.

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, below inequality should also hold:
Revised: Meanwhile, below inequality should also hold:

Original: [Equation]
Revised: [Equation]

Original: Because both the two inequalities must hold, when p ( s ) − ˆ p ( s ) > 0 , we have:
Revised: Because both the two inequalities must hold, when p ( s ) − ˆ p ( s ) > 0 , we have:

Original: [Equation]
Revised: [Equation]

Original: It turns out that the bound is the same when p ( s ) ≤
Revised: It turns out that the bound is the same when p ( s ) ≤

Original: ˆ p ( s ) .
Revised: ˆ p ( s ) .

Original: This completes the proof.
Revised: This completes the proof.

Original: Remark.
Revised: Remark.

Original: This bound actually indicates that | p ( s ) − ˆ p ( s ) | should be small.
Revised: This bound actually indicates that | p ( s ) − ˆ p ( s ) | should be small.

Original: Because if p ( s ) is much larger ˆ p , then we may expect the second term in the min function would be chosen.
Revised: Because if p ( s ) is much larger ˆ p , then we may expect the second term in the min function would be chosen.

Original: During early learning, although the model error can be large, but ˆ Z, Z should be also very large.
Revised: During early learning, although the model error can be large, but ˆ Z, Z should be also very large.

Original: The total model error is scaled by p ( s ) or ˆ p ( s ) and it should be small.
Revised: The total model error is scaled by p ( s ) or ˆ p ( s ) and it should be small.

Original: We may expect a nice approximation even when the model is not that perfectly learned.
Revised: We may expect a nice approximation even when the model is not that perfectly learned.

Original: A. R EPRODUCIBLE R ESEARCH
Revised: A. R EPRODUCIBLE R ESEARCH

Original: Our implementations are based on tensorﬂow with version 1 . 13 . 

Revised: Our implementations are based on tensorﬂow with version 1 . 13 .

Original: (Abadi et al., 2015).
Revised: (Abadi et al., 2015).

Original: We use Adam optimizer (Kingma & Ba, 2014) for all experiments.
Revised: We use Adam optimizer (Kingma & Ba, 2014) for all experiments.

Original: A.6. R EPRODUCE EXPERIMENTS BEFORE S ECTION
Revised: A.6. R EPRODUCE EXPERIMENTS BEFORE S ECTION

Original: Supervised learning experiment.
Revised: Supervised learning experiment.

Original: For the supervised learning experiment shown in section 3, we use 32 × 32 tanh units neural network, with learning rate swept from { 0 . 01 , 0 . 001 , 0 . 0001 , 0 . 00001 } for all algorithms.
Revised: For the supervised learning experiment shown in section 3, we use 32 × 32 tanh units neural network, with learning rate swept from { 0 . 01 , 0 . 001 , 0 . 0001 , 0 . 00001 } for all algorithms.

Original: We compute the constant c as speciﬁed in the Theorem 1 at each time step for
Revised: We compute the constant c as speciﬁed in the Theorem 1 at each time step for

Original: Cubic loss.
Revised: Cubic loss.

Original: We compute the testing error every 500 iterations/mini-batch updates and our evaluation learning curves are plotted by averaging 50 random seeds.
Revised: We compute the testing error every 500 iterations/mini-batch updates and our evaluation learning curves are plotted by averaging 50 random seeds.

Original: For each random seed, we randomly split the dataset to testing set and training set and the testing set has 1 k data points.
Revised: For each random seed, we randomly split the dataset to testing set and training set and the testing set has 1 k data points.

Original: Note that the testing set is not noise-contaminated.
Revised: Note that the testing set is not noise-contaminated.

Original: Reinforcement Learning experiments in Section 3.
Revised: Reinforcement Learning experiments in Section 3.

Original: We use a particularly small neural network 16 × 16 to highlight the issue of incomplete priority updating.
Revised: We use a particularly small neural network 16 × 16 to highlight the issue of incomplete priority updating.

Original: Intuitively, a large neural network may be able to memorize each state’s value and thus updating one state’s value is less likely to affect others.
Revised: Intuitively, a large neural network may be able to memorize each state’s value and thus updating one state’s value is less likely to affect others.

Original: We choose a small neural network, in which case a complete priority updating for all states should be very important.
Revised: We choose a small neural network, in which case a complete priority updating for all states should be very important.

Original: We set the maximum ER buffer size as 10 k and mini-batch size as 32 .
Revised: We set the maximum ER buffer size as 10 k and mini-batch size as 32 .

Original: The learning rate is 0 .
Revised: The learning rate is 0 .

Original: 001 and the target network is updated every 1 k steps.
Revised: 001 and the target network is updated every 1 k steps.

Original: Distribution distance computation in Section 4.
Revised: Distribution distance computation in Section 4.

Original: We now introduce the implementation details for Figure 3.
Revised: We now introduce the implementation details for Figure 3.

Original: The distance is estimated by the following steps.
Revised: The distance is estimated by the following steps.

Original: First, in order to compute the desired sampling distribution, we discretize the domain into 50 × 50 grids and calculate the absolute TD error of each grid (represented by the left bottom vertex coordinates) by using the true environment model and the current learned Q function.
Revised: First, in order to compute the desired sampling distribution, we discretize the domain into 50 × 50 grids and calculate the absolute TD error of each grid (represented by the left bottom vertex coordinates) by using the true environment model and the current learned Q function.

Original: We then normalize these priorities to get probability distribution p ∗ .
Revised: We then normalize these priorities to get probability distribution p ∗ .

Original: Note that this distribution is considered as the desired one since we have access to all states across the state space with priorities computed by current Q-function at each time step.
Revised: Note that this distribution is considered as the desired one since we have access to all states across the state space with priorities computed by current Q-function at each time step.

Original: Second, we estimate our sampling distribution by randomly sampling 3 k states from search-control queue and count the number of states falling into each discretized grid and normalize these counts to get p 1 .
Revised: Second, we estimate our sampling distribution by randomly sampling 3 k states from search-control queue and count the number of states falling into each discretized grid and normalize these counts to get p 1 .

Original: Third, for comparison, we estimate the sampling distribution of the conventional prioritized
Revised: Third, for comparison, we estimate the sampling distribution of the conventional prioritized

Original: ER (Schaul et al., 2016) by sampling 3 k states from the prioritized ER buffer and count the states falling into each grid and compute its corresponding distribution p 2 by normalizing the counts.
Revised: ER (Schaul et al., 2016) by sampling 3 k states from the prioritized ER buffer and count the states falling into each grid and compute its corresponding distribution p 2 by normalizing the counts.

Original: Then we compute the distances of p 1 , p 2 to p ∗ by two weighting schemes: 1) on-policy weighting: (cid:80)
Revised: Then we compute the distances of p 1 , p 2 to p ∗ by two weighting schemes: 1) on-policy weighting: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: 2 } , where d π is approximated by uniformly sample 3 k states from a recency buffer and normalizing their visitation counts on the discretized GridWorld; 2) uniform weighting:
Revised: 2 } , where d π is approximated by uniformly sample 3 k states from a recency buffer and normalizing their visitation counts on the discretized GridWorld; 2) uniform weighting:

Original: [Equation]
Revised: [Equation]

Original: We examine the two weighting schemes because of two considerations: for the on-policy weighting, we concern about the asymptotic convergent behavior and want to down-weight those states with relatively high TD error but get rarely visited as the policy gets close to optimal; uniform weighting makes more sense during early learning stage, where we consider all states are equally important and want the agents to sufﬁciently explore the whole state space.
Revised: We examine the two weighting schemes because of two considerations: for the on-policy weighting, we concern about the asymptotic convergent behavior and want to down-weight those states with relatively high TD error but get rarely visited as the policy gets close to optimal; uniform weighting makes more sense during early learning stage, where we consider all states are equally important and want the agents to sufﬁciently explore the whole state space.

Original: A.6. R EPRODUCE EXPERIMENTS IN S ECTION
Revised: A.6. R EPRODUCE EXPERIMENTS IN S ECTION

Original: For our algorithm, the pseudo-code with concrete parameter settings is presented in Algorithm 4.
Revised: For our algorithm, the pseudo-code with concrete parameter settings is presented in Algorithm 4.

Original: Common settings.
Revised: Common settings.

Original: For all domains other than roundabout-v0, we use 32 × 32 neural network with
Revised: For all domains other than roundabout-v0, we use 32 × 32 neural network with

Original: ReLu hidden units except the Dyna-Frequency which uses tanh units as suggested by the author (Pan et al., 2020).
Revised: ReLu hidden units except the Dyna-Frequency which uses tanh units as suggested by the author (Pan et al., 2020).

Original: Except the output layer parameters which were initialized from a uniform distribution [ − 0 .
Revised: Except the output layer parameters which were initialized from a uniform distribution [ − 0 .

Original: 003 , 0 . 003] , all other parameters are initialized using Xavier initialization (Glorot & Bengio, 2010).
Revised: 003 , 0 . 003] , all other parameters are initialized using Xavier initialization (Glorot & Bengio, 2010).

Original: We use mini-batch size b = 32 and maximum ER buffer size 50 k. All algorithms use target network moving frequency 1000 and we sweep learning rate from { 0 . 001 , 0 . 0001 } .
Revised: We use mini-batch size b = 32 and maximum ER buffer size 50 k. All algorithms use target network moving frequency 1000 and we sweep learning rate from { 0 . 001 , 0 . 0001 } .

Original: We use warm up steps = 5000 (i.e. random action is taken in the ﬁrst 5 k time steps) to populate the ER buffer before learning starts.
Revised: We use warm up steps = 5000 (i.e. random action is taken in the ﬁrst 5 k time steps) to populate the ER buffer before learning starts.

Original: We keep exploration noise as 0 .
Revised: We keep exploration noise as 0 .

Original: 1 without decaying.
Revised: 1 without decaying.

Original: Meta-parameter settings.
Revised: Meta-parameter settings.

Original: For our algorithm Dyna-TD, we are able to keep the same parameter setting across all benchmark domains: α a = 0 .
Revised: For our algorithm Dyna-TD, we are able to keep the same parameter setting across all benchmark domains: α a = 0 .

Original: 1 , c = 20 and learning rate 0 .
Revised: 1 , c = 20 and learning rate 0 .

Original: For all Dyna variants, we fetch the same number of states ( m = 20 ) from hill climbing (i.e. search-control process) as Dyna-TD does, and use (cid:15) a = 0 .
Revised: For all Dyna variants, we fetch the same number of states ( m = 20 ) from hill climbing (i.e. search-control process) as Dyna-TD does, and use (cid:15) a = 0 .

Original: 1 and set the maximum number of gradient step as k = 100 unless otherwise speciﬁed.
Revised: 1 and set the maximum number of gradient step as k = 100 unless otherwise speciﬁed.

Original: Our Prioritized ER is implemented as the proportional version with sum tree data structure.
Revised: Our Prioritized ER is implemented as the proportional version with sum tree data structure.

Original: To ensure fair comparison, since all model-based methods are using mixed mini-batch of samples, we use prioritized ER without importance ratio but half of mini-batch samples are uniformly sampled from the ER buffer as a strategy for bias correction.
Revised: To ensure fair comparison, since all model-based methods are using mixed mini-batch of samples, we use prioritized ER without importance ratio but half of mini-batch samples are uniformly sampled from the ER buffer as a strategy for bias correction.

Original: For Dyna-Value and Dyna-Frequency, we use the setting as described by the original papers.
Revised: For Dyna-Value and Dyna-Frequency, we use the setting as described by the original papers.

Original: For the purpose of learning an environment model, we use a 64 × 64 ReLu units neural network to predict s (cid:48) − s and reward given a state-action pair s, a ; and we use mini-batch size 128 and learning rate 0 . 0001 to minimize the mean squared error objective for training the environment model.
Revised: For the purpose of learning an environment model, we use a 64 × 64 ReLu units neural network to predict s (cid:48) − s and reward given a state-action pair s, a ; and we use mini-batch size 128 and learning rate 0 . 0001 to minimize the mean squared error objective for training the environment model.

Original: Environment-speciﬁc settings.
Revised: Environment-speciﬁc settings.

Original: All of the environments are from OpenAI (Brockman et al., 2016) except that: 1) the GridWorld envirnoment is taken from Pan et al.
Revised: All of the environments are from OpenAI (Brockman et al., 2016) except that: 1) the GridWorld envirnoment is taken from Pan et al.

Original: and the MazeGridWorld is from Pan et al.
Revised: and the MazeGridWorld is from Pan et al.

Original: (2020); 2) Roundabout-v0 is from Leurent et al.
Revised: (2020); 2) Roundabout-v0 is from Leurent et al.

Original: For all OpenAI environments, we use the default setting except on Mountain Car where we set the episodic length limit to 2 k.
Revised: For all OpenAI environments, we use the default setting except on Mountain Car where we set the episodic length limit to 2 k.

Original: The
Revised: The

Original: GridWorld has state space S =
Revised: GridWorld has state space S =

Original: [0 , 1] 2 and each episode starts from the left bottom and the goal area is at the top right [0 .
Revised: [0 , 1] 2 and each episode starts from the left bottom and the goal area is at the top right [0 .

Original: There is a wall in the middle with a hole to allow the agent to pass.
Revised: There is a wall in the middle with a hole to allow the agent to pass.

Original: MazeGridWorld is a more complicated version where the state and action spaces are the same as
Revised: MazeGridWorld is a more complicated version where the state and action spaces are the same as

Original: GridWorld, but there are two walls in the middle and it takes a long time for model-free methods to be successful.
Revised: GridWorld, but there are two walls in the middle and it takes a long time for model-free methods to be successful.

Original: On the this domain, we use the same setting as the original paper for all Dyna variants.
Revised: On the this domain, we use the same setting as the original paper for all Dyna variants.

Original: We use exactly the same setting as described above except that we change the Q − network size to 64 × 64 ReLu units, and number of search-control samples is m = 50 as used by the original paper.
Revised: We use exactly the same setting as described above except that we change the Q − network size to 64 × 64 ReLu units, and number of search-control samples is m = 50 as used by the original paper.

Original: We refer readers to the original paper (Pan et al., 2020) for more details.
Revised: We refer readers to the original paper (Pan et al., 2020) for more details.

Original: On roundabout-v0 domain, we use 64 × 64 ReLu units for all algorithms and set mini-batch size as 64 .
Revised: On roundabout-v0 domain, we use 64 × 64 ReLu units for all algorithms and set mini-batch size as 64 .

Original: For Dyna-TD, we start using the model after 5 k steps and set m = 100 , α a = 1 . 0 , k = 

Revised: For Dyna-TD, we start using the model after 5 k steps and set m = 100 , α a = 1 . 0 , k =

Original: and we do search-control every 10 environment time steps to reduce computational cost.
Revised: and we do search-control every 10 environment time steps to reduce computational cost.

Original: To alleviate the effect of model error, we use only 16 out of 64 samples from the search-control queue in a mini-batch.
Revised: To alleviate the effect of model error, we use only 16 out of 64 samples from the search-control queue in a mini-batch.

Original: A. A DDITIONAL E XPERIMENTS
Revised: A. A DDITIONAL E XPERIMENTS

Original: The main purpose of the additional experiments here is to strengthen our claims from Section 3.
Revised: The main purpose of the additional experiments here is to strengthen our claims from Section 3.

Original: Learning curve in terms of training error corresponding to Figure 2 .
Revised: Learning curve in terms of training error corresponding to Figure 2 .

Original: In Section 3.1, we show the learning curve in terms of testing error.
Revised: In Section 3.1, we show the learning curve in terms of testing error.

Original: We now show training error to closely match our theoretical result 1.
Revised: We now show training error to closely match our theoretical result 1.

Original: As a supplement, we include the learning curve in terms of testing error in Figure 8.
Revised: As a supplement, we include the learning curve in terms of testing error in Figure 9.

Original: Input: m : number of states to fetch through search-control; B sc : empty search-control queue;
Revised: Input: m : number of states to fetch through search-control; B sc : empty search-control queue;

Original: B er : ER buffer; (cid:15) a : threshold for accepting a state; initialize Q -network Q θ for t = 1 , 2 , . . .
Revised: B er : ER buffer; (cid:15) a : threshold for accepting a state; initialize Q -network Q θ for t = 1 , 2 , . . .

Original: do
Revised: do

Original: Observe ( s t , a t , s t +1 , r t +1 ) and add it to B er //
Revised: Observe ( s t , a t , s t +1 , r t +1 ) and add it to B er //

Original: Hill climbing on absolute TD error
Revised: Hill climbing on absolute TD error

Original: Sample s from B er , c ← 0 , ˜ s ← s while c < m do ˆ y ← E s (cid:48) ,r ∼ ˆ P ( ·| s,a ) [ r + γ max a Q θ ( s (cid:48) , a )]
Revised: Sample s from B er , c ← 0 , ˜ s ← s while c < m do ˆ y ← E s (cid:48) ,r ∼ ˆ P ( ·| s,a ) [ r + γ max a Q θ ( s (cid:48) , a )]

Original: Update s by rule equationif s is out of the state space then
Revised: Update s by rule equationif s is out of the state space then

Original: Sample s from B er , ˜ s ← s // restart continue if || ˜ s − s || 2 / √ d ≥ (cid:15)
Revised: Sample s from B er , ˜ s ← s // restart continue if || ˜ s − s || 2 / √ d ≥ (cid:15)

Original: a then //
Revised: a then //

Original: d is the number of state variables, i.e. S ⊂ R d
Revised: d is the number of state variables, i.e. S ⊂ R d

Original: Add s into B sc , ˜ s ← s , c ← c +// n planning updates for n times do
Revised: Add s into B sc , ˜ s ← s , c ← c +// n planning updates for n times do

Original: Sample a mixed mini-batch with half samples from B sc and half from B er
Revised: Sample a mixed mini-batch with half samples from B sc and half from B er

Original: Update Q -network parameters by using the mixed mini-batch
Revised: Update Q -network parameters by using the mixed mini-batch

Original: Learning curve with a larger neural network on the sin dataset.
Revised: Learning curve with a larger neural network on the sin dataset.

Original: We try to eliminate the effect of neural network size.
Revised: We try to eliminate the effect of neural network size.

Original: Hence we use a larger neural network size ( 128 × 128 tanh units) on the same sin dataset.
Revised: Hence we use a larger neural network size ( 128 × 128 tanh units) on the same sin dataset.

Original: As one can see from Figure 9, FullPrioritizedL2 still performs the best and when we increase the mini-batch size from 128 to 512 , the high power objective versions still moves closer to
Revised: As one can see from Figure 10, FullPrioritizedL2 still performs the best and when we increase the mini-batch size from 128 to 512 , the high power objective versions still moves closer to

Original: FullPrioritizedL2, as we saw in Figure 2.
Revised: FullPrioritizedL2, as we saw in Figure 2.

Original: Learning curve with on a real-world dataset.
Revised: Learning curve with on a real-world dataset.

Original: To illustrate the generality of our Theorem 1, we also conduct tests on a frequently cited regression Bike sharing dataset Fanaee-T & Gama (2013).
Revised: To illustrate the generality of our Theorem 1, we also conduct tests on a frequently cited regression Bike sharing dataset Fanaee-T & Gama (2013).

Original: The data preprocessing is as follows.
Revised: The data preprocessing is as follows.

Original: We remove attributes: date, index, year, weather situation 4, weekday 7, registered, casual.
Revised: We remove attributes: date, index, year, weather situation 4, weekday 7, registered, casual.

Original: We use one-hot encoding for all categorical variables.
Revised: We use one-hot encoding for all categorical variables.

Original: We scale the target to [0 , 1] and scale it back when computing training errors.
Revised: We scale the target to [0 , 1] and scale it back when computing training errors.

Original: We use a 64 × 64 ReLu units neural network with mini-batch size 128 and learning rate 0 . 0001 for training.
Revised: We use a 64 × 64 ReLu units neural network with mini-batch size 128 and learning rate 0 . 0001 for training.

Original: It should be noted that the behavior of Cubic is consistent on the previous sin example and on this real world dataset: it gets closer to FullPrioritizedL2 as we increase the mini-batch size.
Revised: It should be noted that the behavior of Cubic is consistent on the previous sin example and on this real world dataset: it gets closer to FullPrioritizedL2 as we increase the mini-batch size.

Original: Another observation is that the Power4 objective is highly variant on this domain, because the real world data should be noisy and the high order objective suffers.
Revised: Another observation is that the Power4 objective is highly variant on this domain, because the real world data should be noisy and the high order objective suffers.

Original: This observation corresponds to what we observed in Section 3.1, where we show that high power objective is sensitive to the noise.
Revised: This observation corresponds to what we observed in Section 3.1, where we show that high power objective is sensitive to the noise.

Original: Input or notations: k = 20 : number search-control states to acquire by hill climbing, k b = 100 : the budget of maximum number of hill climbing steps; ρ = 0 .
Revised: Input or notations: k = 20 : number search-control states to acquire by hill climbing, k b = 100 : the budget of maximum number of hill climbing steps; ρ = 0 .

Original: 5 : percentage of samples from search-control queue, d : S ⊂ R d ; empty search-control queue B sc and ER buffer B er empirical covariance matrix: ˆΣ s ← I µ ss ← 0 ∈ R d × d , µ s ← 0 ∈ R d (auxiliary variables for computing empirical covariance matrix, sample average will be maintained for µ ss , µ s ) n τ ← 0 : count for parameter updating times, τ ← 1000 target network updating frequency (cid:15) a ← 0 : threshold for accepting a state
Revised: 5 : percentage of samples from search-control queue, d : S ⊂ R d ; empty search-control queue B sc and ER buffer B er empirical covariance matrix: ˆΣ s ← I µ ss ← 0 ∈ R d × d , µ s ← 0 ∈ R d (auxiliary variables for computing empirical covariance matrix, sample average will be maintained for µ ss , µ s ) n τ ← 0 : count for parameter updating times, τ ← 1000 target network updating frequency (cid:15) a ← 0 : threshold for accepting a state

Original: Initialize Q network Q θ and target Q network Q θ (cid:48)
Revised: Initialize Q network Q θ and target Q network Q θ (cid:48)

Original: for t = 1 , 2 , . . .
Revised: for t = 1 , 2 , . . .

Original: do
Revised: do

Original: Observe ( s, a, s (cid:48) , r ) and add it to B er µ
Revised: Observe ( s, a, s (cid:48) , r ) and add it to B er µ

Original: [Equation]
Revised: [Equation]

Original: , µ
Revised: , µ

Original: [Equation]
Revised: [Equation]

Original: t ˆΣ s ← µ
Revised: t ˆΣ s ← µ

Original: [Equation]
Revised: [Equation]

Original: (cid:15) a ← (1 − β ) (cid:15) a + β || s (cid:48) − s || 2 for β = 0 .//
Revised: (cid:15) a ← (1 − β ) (cid:15) a + β || s (cid:48) − s || 2 for β = 0 .//

Original: Hill climbing on absolute TD error
Revised: Hill climbing on absolute TD error

Original: Sample s from B er , c ← 0 , ˜ s ← s, i ←while c < k and i < k b do // since environment is deterministic, the environment model becomes a Dirac-delta distribution and we denote it as a deterministic function M :
Revised: Sample s from B er , c ← 0 , ˜ s ← s, i ←while c < k and i < k b do // since environment is deterministic, the environment model becomes a Dirac-delta distribution and we denote it as a deterministic function M :

Original: S × A (cid:55)→ S
Revised: S × A (cid:55)→ S

Original: × R s (cid:48) , r ← M ( s, a ) ˆ y ← r + γ max a Q θ ( s (cid:48) , a ) // add a smooth constant 10 − 5 inside the logarithm s ← s + α a ∇ s log( | ˆ y − max a Q ( s, a ; θ t ) | + 10 − 5 )
Revised: × R s (cid:48) , r ← M ( s, a ) ˆ y ← r + γ max a Q θ ( s (cid:48) , a ) // add a smooth constant 10 − 5 inside the logarithm s ← s + α a ∇ s log( | ˆ y − max a Q ( s, a ; θ t ) | + 10 − 5 )

Original: + X, X ∼ N (0 , 0 .
Revised: + X, X ∼ N (0 , 0 .

Original: 01ˆΣ s ) if s is out of the state space
Revised: 01ˆΣ s ) if s is out of the state space

Original: then // restart hill climbing
Revised: then // restart hill climbing

Original: Sample s from B er , ˜ s ← s continue if || ˜ s − s || 2 / √ d ≥ (cid:15)
Revised: Sample s from B er , ˜ s ← s continue if || ˜ s − s || 2 / √ d ≥ (cid:15)

Original: a then
Revised: a then

Original: Add s into B sc
Revised: Add s into B sc

Original: ˜ s ← s c ← c +i ← i +for n times do
Revised: ˜ s ← s c ← c +i ← i +for n times do

Original: Sample a mixed mini-batch b with proportion  ρ er
Revised: Sample a mixed mini-batch b with proportion − ρ er

Original: Update parameters θ (
Revised: Update parameters θ (

Original: DQN update) with b n
Revised: DQN update) with b n

Original: ← n
Revised: ← n

Original: +if mod ( n τ , τ ) =
Revised: +if mod ( n τ , τ ) =

Original: = 

Revised: =

Original: then
Revised: then

Original: Q θ (cid:48) ← Q θ
Revised: Q θ (cid:48) ← Q θ