Original: Understanding and Mitigating the Limitations of Prioritized Experience Replay
Revised: Understanding and Mitigating the Limitations of Prioritized Experience Replay

Original: Abstract
Revised: Abstract

Original: Prioritized Experience Replay (ER) has been empirically shown to improve sample efﬁciency across many domains and attracted great attention; however, there is little theoretical understanding of why such prioritized sampling helps and its limitations.
Revised: Prioritized Experience Replay (ER) has been empirically shown to improve sample efﬁciency across many domains and attracted great attention; however, there is little theoretical understanding of why such prioritized sampling helps and its limitations.

Original: In this work, we take a deep look at the prioritized ER.
Revised: In this work, we take a deep look at the prioritized ER.

Original: In a supervised learning setting, we show the equivalence between the error-based prioritized sampling method for  mean squared error and  uniform sampling for cubic power loss.
Revised: In a supervised learning setting, we show the equivalence between the error-based prioritized sampling method for minimizing mean squared error and the uniform sampling for cubic power loss.

Original: We then provide theoretical insight into why it improves convergence rate upon uniform sampling  during early learning.
Revised: We then provide theoretical insight into why error-based prioritized sampling improves convergence rate upon uniform sampling when minimizing mean squared error during early learning.

Original: Based on the insight, we further point out two limitations of the prioritized ER method: 1) outdated priorities and 2) insufﬁcient coverage of the sample space.
Revised: Based on the insight, we further point out two limitations of the prioritized ER method: 1) outdated priorities and 2) insufﬁcient coverage of the sample space.

Original: To mitigate the limitations, we propose our model-based stochastic gradient
Revised: To mitigate the limitations, we propose our model-based stochastic gradient

Original: Langevin dynamics sampling method.
Revised: Langevin dynamics sampling method.

Original: We show that our method does provide states distributed close to an ideal prioritized sampling distribution estimated by the brute-force method, which does not suffer from the two limitations.
Revised: We show that our method does provide states distributed close to an ideal prioritized sampling distribution estimated by the brute-force method, which does not suffer from the two limitations.

Original: We conduct experiments on both discrete and continuous control problems to show our approach’s efﬁcacy and examine the practical implication of our method in an autonomous driving application.
Revised: We conduct experiments on both discrete and continuous control problems to show our approach’s efﬁcacy and examine the practical implication of our method in an autonomous driving application.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Experience Replay (ER)
Revised: Experience Replay (ER)

Original: [Lin, 1992] has been a popular method for training large-scale modern Reinforcement
Revised: [Lin, 1992] has been a popular method for training large-scale modern Reinforcement

Original: Learning (RL) systems [Degris et al., 2012, Adam and Busoniu, 2012, Mnih et al., 2015a, Hessel et al., 2018, François- Lavet et al., 2018].
Revised: Learning (RL) systems [Degris et al., 2012, Adam and Busoniu, 2012, Mnih et al., 2015a, Hessel et al., 2018, FrançoisLavet et al., 2018].

Original: In ER, visited experiences are stored in a buffer, and at each time step, a mini-batch of experiences is uniformly sampled to update the training parameters in the value or policy function.
Revised: In ER, visited experiences are stored in a buffer, and at each time step, a mini-batch of experiences is uniformly sampled to update the training parameters in the value or policy function.

Original: Such a method is empirically shown to effectively stabilize the training and improve the sample efﬁciency of deep RL algorithms.
Revised: Such a method is empirically shown to effectively stabilize the training and improve the sample efﬁciency of deep RL algorithms.

Original: Several follow-up works propose to improve upon it by designing non-uniform sampling distributions or re-weighting mechanisms of experiences [Schaul et al., 2016, Andrychowicz et al., 2017,
Revised: Several follow-up works propose to improve upon it by designing non-uniform sampling distributions or re-weighting mechanisms of experiences [Schaul et al., 2016, Andrychowicz et al., 2017,

Original: Oh et al., 2018, de Bruin et al., 2018, Horgan et al., 2018,
Revised: Oh et al., 2018, de Bruin et al., 2018, Horgan et al., 2018,

Original: Zha et al., 2019, Novati and Koumoutsakos, 2019, Kumar et al., 2020, Sun et al., 2020, Liu et al., 2021, Lee et al., 2021, Sinha et al., 2022].
Revised: Zha et al., 2019, Novati and Koumoutsakos, 2019, Kumar et al., 2020, Sun et al., 2020, Liu et al., 2021, Lee et al., 2021, Sinha et al., 2022].

Original: The most relevant one to our work is prioritized ER [Schaul et al., 2016], which attempts to improve the vanilla ER method by sampling those visited experiences proportional to their absolute Temporal Difference (TD) errors.
Revised: The most relevant one to our work is prioritized ER [Schaul et al., 2016], which attempts to improve the vanilla ER method by sampling those visited experiences proportional to their absolute Temporal Difference (TD) errors.

Original: Empirically, it can signiﬁcantly improve sample efﬁciency upon vanilla ER on many domains.
Revised: Empirically, it can signiﬁcantly improve sample efﬁciency upon vanilla ER on many domains.

Original: ER methods have a close connection to Model-Based RL (MBRL) methods [Kaelbling et al., 1996, Bertsekas, 2009,
Revised: ER methods have a close connection to Model-Based RL (MBRL) methods [Kaelbling et al., 1996, Bertsekas, 2009,

Original: Sutton and Barto, 2018].
Revised: Sutton and Barto, 2018].

Original: ER can be thought of as an instance of a classical model-based RL architecture—Dyna [Sutton, 1991], using a(limited) non-parametric model given by the buffer [van Seijen and Sutton, 2015, van Hasselt et al., 2019].
Revised: ER can be thought of as an instance of a classical model-based RL architecture—Dyna [Sutton, 1991], using a non-parametric model given by the buffer [van Seijen and Sutton, 2015, van Hasselt et al., 2019].

Original: A Dyna agent uses real experience to update its policy as well as its reward and dynamics model.
Revised: A Dyna agent uses real experience to update its policy as well as its reward and dynamics model.

Original: In-between taking actions, the agent can get hypothetical experiences from the model and use them just like the real experiences to further improve the policy.
Revised: In-between taking actions, the agent can get hypothetical experiences from the model and use them just like the real experiences to further improve the policy.

Original: The mechanism of generating those hypothetical experiences is termed search-control . When deﬁning the search-control as sampling experiences from an ER buffer, the Dyna agent reduces to an ER agent.
Revised: How to generate those hypothetical experiences is largely dependent on search-control —the mechanism of generating states or state-action pairs from which to query the model to get the next states and rewards.

Original: Existing works show that smart search-control strategies can further improve sample efﬁciency of a Dyna agent [Sutton et al., 2008, Gu et al., 2016, Goyal et al., 2019, Holland et al., 2018, Pan et al., 2018, Corneil et al., 2018, Janner et al., 2019, Chelu et al., 2020].
Revised: Existing works show that smart search-control strategies can further improve sample efﬁciency of a Dyna agent [Sutton et al., 2008, Gu et al., 2016, Goyal et al., 2019, Holland et al., 2018, Pan et al., 2018, Corneil et al., 2018, Janner et al., 2019, Chelu et al., 2020].

Original: Particularly, prioritized sweeping [Moore and Atkeson, 1993] is among the earliest work that improves upon vanilla Dyna.
Revised: Particularly, prioritized sweeping [Moore and Atkeson, 1993] is among the earliest work that improves upon vanilla Dyna.

Original: The idea behind prioritized sweeping is quite intuitive: we should give high priority to states whose absolute TD errors are large because they are likely to cause the most change in value estimates.
Revised: The idea behind prioritized sweeping is quite intuitive: we should give high priority to states whose absolute TD errors are large because they are likely to cause the most change in value estimates.

Original: Hence, the prioritized ER by Schaul et al.
Revised: Hence, the prioritized ER by Schaul et al.

Original: [2016], which applies TD error-based prioritized sampling to ER, is a natural idea in a model-free RL setting.
Revised: [2016], which applies TD error-based prioritized sampling to ER, is a natural idea in a model-free RL setting.

Original: However, there is little rigorous understanding towards prioritized ER method about why it can help and its limitations.
Revised: However, there is little rigorous understanding towards prioritized ER method about why it can help and its limitations.

Original: This work provides a theoretical insight into the prioritized
Revised: This work provides a theoretical insight into the prioritized

Original: ER’s advantage and points out its two drawbacks: outdated priorities and insufﬁcient sample space coverage, which may signiﬁcantly weaken its efﬁcacy.
Revised: ER’s advantage and points out its two drawbacks: outdated priorities and insufﬁcient sample space coverage, which may signiﬁcantly weaken its efﬁcacy.

Original: To mitigate the two issues, we propose to use the Stochastic Gradient Langevin Dynamics (SGLD) sampling method to acquire states.
Revised: To mitigate the two issues, we propose to use the Stochastic Gradient Langevin Dynamics (SGLD) sampling method to acquire states.

Original: Our method relies on applying an environment model to 1) simulate priorities of states and 2) acquire hypothetical experiences.
Revised: Our method relies on applying an environment model to 1) simulate priorities of states and 2) acquire hypothetical experiences.

Original: Then these experiences are used for further improving the policy.
Revised: Then these experiences are used for further improving the policy.

Original: We demonstrate that, comparing with the conventional prioritized ER method, the hypothetical experiences generated by our method are distributed closer to the ideal
Revised: We demonstrate that, comparing with the conventional prioritized ER method, the hypothetical experiences generated by our method are distributed closer to the ideal

Original: TD error-based sampling distribution, which does not suffer from the two drawbacks.
Revised: TD error-based sampling distribution, which does not suffer from the two drawbacks.

Original: Finally, we demonstrate the utility of our approach on various benchmark discrete and continuous control domains and an autonomous driving application.
Revised: Finally, we demonstrate the utility of our approach on various benchmark discrete and continuous control domains and an autonomous driving application.

Original:  BACKGROUND
Revised:  BACKGROUND

Original: In this section, we ﬁrstly review basic concepts in RL.
Revised: In this section, we ﬁrstly review basic concepts in RL.

Original: Then we brieﬂy introduce the prioritized ER method, which will be examined in-depth in the next section.
Revised: Then we brieﬂy introduce the prioritized ER method, which will be examined in-depth in the next section.

Original: We conclude this section by discussing a classic MBRL architecture called
Revised: We conclude this section by discussing a classic MBRL architecture called

Original: Dyna [Sutton, 1991] and its recent variants, which are most relevant to our work.
Revised: Dyna [Sutton, 1991] and its recent variants, which are most relevant to our work.

Original: Basic notations.
Revised: Basic notations.

Original: We consider a discounted Markov Decision Process (MDP) framework [Szepesvári, 2010].
Revised: We consider a discounted Markov Decision Process (MDP) framework [Szepesvári, 2010].

Original: An
Revised: An

Original: MDP can be denoted as a tuple ( S , A , P , R,  ) including state space S , action space A , probability transition kernel P , reward function R , and discount rate  2 [0 , 1] .
Revised: MDP can be denoted as a tuple ( S , A , P , R,  ) including state space S , action space A , probability transition kernel P , reward function R , and discount rate  2 [0 , 1] .

Original: At each environment time step t , an RL agent observes a state s t 2 S , takes an action a t 2 A , and moves to the next state s t +1 ⇠ P ( ·| s t , a t ) , and receives a scalar reward signal r t +1 =
Revised: At each environment time step t , an RL agent observes a state s t 2 S , takes an action a t 2 A , and moves to the next state s t +1 ⇠ P ( ·| s t , a t ) , and receives a scalar reward signal r t +1 =

Original: R ( s t , a t , s t +1 ) .
Revised: R ( s t , a t , s t +1 ) .

Original: A policy is a mapping ⇡ : S ⇥ A !
Revised: A policy is a mapping ⇡ : S ⇥ A !

Original: [0 , 1] that determines the probability of choosing an action at a given state.
Revised: [0 , 1] that determines the probability of choosing an action at a given state.

Original: A popular algorithm to ﬁnd an optimal policy is Qlearning [Watkins and Dayan, 1992].
Revised: A popular algorithm to ﬁnd an optimal policy is Qlearning [Watkins and Dayan, 1992].

Original: With function approximation, parameterized action-values Q ✓ are updated using ✓ = ✓ + ↵ t r ✓ Q ✓ ( s t , a t ) for stepsize ↵ > 0 with TD-error  t def = r t +1 +  max a 0 2 A Q ✓ ( s t +1 , a 0 )
Revised: With function approximation, parameterized action-values Q ✓ are updated using ✓ = ✓ + ↵ t r ✓ Q ✓ ( s t , a t ) for stepsize ↵ > 0 with TD-error  t def = r t +1 +  max a 0 2 A Q ✓ ( s t +1 , a 0 )

Original:  Q ✓ ( s t , a t ) .
Revised:  Q ✓ ( s t , a t ) .

Original: The policy is deﬁned by acting greedily w.r.t.
Revised: The policy is deﬁned by acting greedily w.r.t.

Original: these action-values.
Revised: these action-values.

Original: ER methods.
Revised: ER methods.

Original: ER is critical when using neural networks to estimate Q ✓ , as used in DQN [Mnih et al., 2015b], both to stabilize and speed up learning.
Revised: ER is critical when using neural networks to estimate Q ✓ , as used in DQN [Mnih et al., 2015b], both to stabilize and speed up learning.

Original: The vanilla ER method uniformly samples a mini-batch of experiences from those visited ones in the form of ( s t , a t , s t +1 , r t +1 ) to update neural network parameters.
Revised: The vanilla ER method uniformly samples a mini-batch of experiences from those visited ones in the form of ( s t , a t , s t +1 , r t +1 ) to update neural network parameters.

Original: Prioritized ER
Revised: Prioritized ER

Original: [Schaul et al., 2016] improves upon it by sampling prioritized experiences, where the probability of sampling a certain experience is proportional to its TD error magnitude, i.e., p ( s t , a t , s t +1 , r t +1 ) /
Revised: [Schaul et al., 2016] improves upon it by sampling prioritized experiences, where the probability of sampling a certain experience is proportional to its TD error magnitude, i.e., p ( s t , a t , s t +1 , r t +1 ) /

Original: |  t | .
Revised: |  t | .

Original: However, the underlying theoretical mechanism behind this method is still not well understood.
Revised: However, the underlying theoretical mechanism behind this method is still not well understood.

Original: MBRL and Dyna.
Revised: MBRL and Dyna.

Original: With a model, an agent has more ﬂexibility to sample hypothetical experiences.
Revised: With a model, an agent has more ﬂexibility to sample hypothetical experiences.

Original: We consider a one-step model which takes a state-action pair as input and provides a distribution over the next state and reward.
Revised: We consider a one-step model which takes a state-action pair as input and provides a distribution over the next state and reward.

Original: We build on the Dyna formalism [Sutton, 1991] for MBRL, and more speciﬁcally, the recently proposed (Hill Climbing) HCDyna
Revised: We build on the Dyna formalism [Sutton, 1991] for MBRL, and more speciﬁcally, the recently proposed (Hill Climbing) HCDyna

Original: [Pan et al., 2019] as shown in Algorithm 1.
Revised: [Pan et al., 2019] as shown in Algorithm 1.

Original: HC-Dyna provides some smart approach to Search-Control (SC)—the mechanism of generating states or state-action pairs from which to query the model to get the next states and rewards.
Revised: HC-Dyna provides some smart approach to Search-Control (SC).

Original: HC-Dyna proposes to employ stochastic gradient Langevin dynamics (SGLD) for sampling states, which relies on hill climbing onperforming some criterion function h ( · ) .
Revised: HC-Dyna proposes to employ stochastic gradient Langevin dynamics (SGLD) for sampling states, which relies on hill climbing on some criterion function h ( · ) .

Original: The term “Hill Climbing (HC)” is used for generality as the SGLD sampling process can be thought of as doing some modiﬁed gradient ascent [Pan et al., 2019, 2020].
Revised: The term “Hill Climbing (HC)” is used for generality as the SGLD sampling process can be thought of as doing some modiﬁed gradient ascent [Pan et al., 2019, 2020].

Original: The algorithmic framework maintains two buffers: the conventional ER buffer storing experiences (i.e., an experience/transition has the form of ( s t , a t , s t +1 , r t +1 ) ) and a search-control queue storing the states acquired by searchcontrol mechanisms (i.e., SLGD sampling).
Revised: The algorithmic framework maintains two buffers: the conventional ER buffer storing experiences (i.e., an experience/transition has the form of ( s t , a t , s t +1 , r t +1 ) ) and a search-control queue storing the states acquired by searchcontrol mechanisms (i.e., SLGD sampling).

Original: At each time step t , a real experience ( s t , a t , s t +1 , r t +1 ) is collected and stored into the ER buffer.
Revised: At each time step t , a real experience ( s t , a t , s t +1 , r t +1 ) is collected and stored into the ER buffer.

Original: Then the HC search-control process starts to collect states and store them into the searchcontrol queue.
Revised: Then the HC search-control process starts to collect states and store them into the searchcontrol queue.

Original: A hypothetical experience is obtained by ﬁrst selecting a state s from the search-control queue, then selecting an action a according to the current policy, and then querying the model to get the next state s 0 and reward r to form an experience ( s, a, s 0 , r ) .
Revised: A hypothetical experience is obtained by ﬁrst selecting a state s from the search-control queue, then selecting an action a according to the current policy, and then querying the model to get the next state s 0 and reward r to form an experience ( s, a, s 0 , r ) .

Original: These hypothetical transitions are combined with real experiences into a single mini-batch to update the training parameters.
Revised: These hypothetical transitions are combined with real experiences into a single mini-batch to update the training parameters.

Original: The n updates, performed before taking the next action, are called planning updates [Sutton and Barto, 2018], as they improve the value/policy by using a model.
Revised: The n updates, performed before taking the next action, are called planning updates [Sutton and Barto, 2018], as they improve the value/policy by using a model.

Original: The choice of pairing states with on-policy actions to form hypothetical experiences has been reported to be beneﬁcial
Revised: The choice of pairing states with on-policy actions to form hypothetical experiences has been reported to be beneﬁcial

Original: [Gu et al., 2016, Pan et al., 2018, Janner et al., 2019].
Revised: [Gu et al., 2016, Pan et al., 2018, Janner et al., 2019].

Original: Two instances have been proposed for h ( · ) : the value function v ( s )
Revised: Two instances have been proposed for h ( · ) : the value function v ( s )

Original: [Pan et al., 2019] and the  gradient magnitude || r
Revised: [Pan et al., 2019] and the sum of gradient and

Revised: Hessian magnitude || r s v ( s ) ||

Revised: + ||

Original: s v ( s ) ||
Revised: H v ( s ) ||

Original: [Pan et al., 2020].
Revised: [Pan et al., 2020].

Original: The former is used as a measure of the utility of a state: doing HC on the learned value function should ﬁnd high-value states without being constrained by the physical environment dynamics.
Revised: The former is used as a measure of the utility of a state: doing HC on the learned value function should ﬁnd high-value states without being constrained by the physical environment dynamics.

Original: The latter is considered as a measure ofthe value approximation difﬁ-
Revised: The latter is considered as a measure of

Original: Input: Hill Climbing (HC) criterion function h : S 7!
Revised: Input: Hill Climbing (HC) criterion function h : S 7!

Original: R ; batch-size b ; Initialize empty search-control queue B sc ; empty ER buffer B er ; initialize policy and model P ; HC stepsize ↵ h ; mini-batch size b ; environment P ; mixing rate ⇢ decides the proportion of hypothetical experiences in a mini-batch.
Revised: R ; batch-size b ; initialize empty search-control queue B sc ; empty ER buffer B er ; initialize policy and model P ; HC stepsize ↵ h ; mini-batch size b ; environment P ; mixing rate ⇢ decides the proportion of hypothetical experiences in a mini-batch.

Original: for t = 1 , 2 , . . .
Revised: for t = 1 , 2 , . . .

Original: do
Revised: do

Original: Add ( s t , a t , s t +1 , r t +1 ) to B er while within some budget time steps do // SGLD sampling for states s s + ↵ h r s h
Revised: Add ( s t , a t , s t +1 , r t +1 ) to B er while within some budget time steps do // SGLD sampling for states s s + ↵ h r s h

Original: ( s ) + Gaussian noise // Searchcontrol, see Section 4 for details about SGLD
Revised: ( s ) + Gaussian noise // Searchcontrol, see Section 4 for details about SGLD

Original: samplingAdd s into B sc // n planning updates/steps for n times do
Revised: samplingAdd s into B sc // n planning updates/steps for n times do

Original: B ; // initialize an empty mini-batch B for b ⇢ times do
Revised: B ; // initialize an empty mini-batch B for b ⇢ times do

Original: Sample s ⇠ B sc , on-policy action
Revised: Sample s ⇠ B sc , on-policy action

Original: a
Revised: a

Original: Sample s 0 , r ⇠ P ( s, a )
Revised: Sample s 0 , r ⇠ P ( s, a )

Original: Add ( s, a, s 0 , r ) into B
Revised: Add ( s, a, s 0 , r ) into B

Original: Sample b (1  ⇢ ) experiences from B er , add to B // NOTE: if ⇢ = 0 , then we only uniformly sample b experiences from B er and use them as B , and the algorithm reduces to ER
Revised: Sample b (1  ⇢ ) experiences from B er , add to B // NOTE: if ⇢ = 0 , then we only uniformly sample b experiences from B er and use them as B , and the algorithm reduces to ER

Original: Update policy/value on mixed mini-batch B culty, then doing HC provides additional states whose values are difﬁcult to learn.
Revised: Update policy/value on mixed mini-batch B the value approximation difﬁculty, then doing HC provides additional states whose values are difﬁcult to learn.

Original: The two suffer from several issues as we discuss in the Appendix A.1.
Revised: The two suffer from several issues as we discuss in the Appendix A.1.

Original: This paper will introduce a HC search-control method motivated by overcoming the limitations of the prioritized ER.
Revised: This paper will introduce a HC search-control method motivated by overcoming the limitations of the prioritized ER.

Original:  A DEEPER LOOK AT ERROR-BASED PRIORITIZED SAMPLING
Revised:  A DEEPER LOOK AT ERROR-BASED PRIORITIZED SAMPLING

Original: In this section, we provide theoretical motivation for errorbased prioritized sampling by showing its equivalence to optimizing a cubic power objective with uniform sampling in a supervised learning setting.
Revised: In this section, we provide theoretical motivation for errorbased prioritized sampling by showing its equivalence to optimizing a cubic power objective with uniform sampling in a supervised learning setting.

Original: We prove that optimizing the cubic objective provides a faster convergence rate during early learning.
Revised: We prove that optimizing the cubic objective provides a faster convergence rate during early learning.

Original: Based on the insight, we discuss two limitations of the prioritized ER: 1) outdated priorities and 2) insufﬁcient coverage of the sample space.
Revised: Based on the insight, we discuss two limitations of the prioritized ER: 1) outdated priorities and 2) insufﬁcient coverage of the sample space.

Original: We then empirically study the limitations.
Revised: We then empirically study the limitations.

Original: 3.1 THEORETICAL INSIGHT INTO ERROR-BASED PRIORITIZED SAMPLING
Revised: 3.1 THEORETICAL INSIGHT INTO ERROR-BASED PRIORITIZED SAMPLING

Original: In the l 2 regression, we minimize the mean squared error min ✓ 12 n
Revised: In the l 2 regression, we minimize the mean squared error min ✓ 12 n

Original: P ni =1 ( f ✓ ( x i )  y i ) 2 , for training set T
Revised: P ni =1 ( f ✓ ( x i )  y i ) 2 , for training set T

Original: = { ( x i , y i ) } n i =1 and function approximator f ✓ , such as a neural network.
Revised: = { ( x i , y i ) } n i =1 and function approximator f ✓ , such as a neural network.

Original: In error-based prioritized sampling, we deﬁne the priority of a sample ( x, y ) 2 T as | f ✓ ( x )  y | ; the probability of drawing a sample ( x, y ) 2 T is typically q ( x, y ; ✓ ) / | f ✓ ( x )  y | .
Revised: In error-based prioritized sampling, we deﬁne the priority of a sample ( x, y ) 2 T as | f ✓ ( x )  y | ; the probability of drawing a sample ( x, y ) 2 T is typically q ( x, y ; ✓ ) / | f ✓ ( x )  y | .

Original: We employ the following form to compute the probability of a point ( x, y ) 2 T :
Revised: We employ the following form to compute the probability of a point ( x, y ) 2 T :

Original: [Equation]
Revised: [Equation]

Original: We can show an equivalence between the gradients of the squared objective with this prioritization and the cubic power objective 13 n
Revised: We can show an equivalence between the gradients of the squared objective with this prioritization and the cubic power objective 13 n

Original: [Equation]
Revised: [Equation]

Original:  Theorem 1 below.
Revised: in Theorem 1 below.

Original: See Appendix A.3 for the proof.
Revised: See Appendix A.3 for the proof.

Original: Theorem 1.
Revised: Theorem 1.

Original: For a constant c determined by ✓ , T , we have
Revised: For a constant c determined by ✓ , T , we have

Original: [Equation]
Revised: [Equation]

Original: We empirically verify this equivalence in the Appendix A.7.
Revised: We empirically verify this equivalence in the Appendix A.7.

Original: This simple theorem provides an intuitive reason for why prioritized sampling can help improve sample efﬁciency: the gradient direction of the cubic function is sharper than that of the square function when the error is relatively large (Figure 8).
Revised: This simple theorem provides an intuitive reason for why prioritized sampling can help improve sample efﬁciency: the gradient direction of the cubic function is sharper than that of the square function when the error is relatively large (Figure 8).

Original: We refer readers to the work by Fujimoto et al.
Revised: We refer readers to the work by Fujimoto et al.

Original: [2020] regarding more discussions about the equivalence between prioritized sampling and of uniform sampling.
Revised: [2020] regarding more discussions about the equivalence between prioritized sampling and of uniform sampling.

Original: Theorembelow further proves that optimizing the cubic power objective by gradient descent has faster convergence rate than the squared objective, and this provides a solid motivation for using error-based prioritized sampling.
Revised: Theorembelow further proves that optimizing the cubic power objective by gradient descent has faster convergence rate than the squared objective, and this provides a solid motivation for using error-based prioritized sampling.

Original: See Appendix A. for a detailed version of the theorem below, and its proof and empirical simulations.
Revised: See Appendix A. for a detailed version of the theorem below, and its proof and empirical simulations.

Original: Theorem 2 (Fast early learning, concise version) .
Revised: Theorem 2 (Fast early learning, concise version) .

Original: Let n be a positive integer (i.e., the number of training samples).
Revised: Let n be a positive integer (i.e., the number of training samples).

Original: Let x t , ˜ x t 2 R n be the target estimates of all samples at time t, t  0 , and x ( i )( i 2 [ n ] , [ n ] def = { 1 , 2 , ..., n } ) be the i th element in the vector.
Revised: Let x t , ˜ x t 2 R n be the target estimates of all samples at time t, t  0 , and x ( i )( i 2 [ n ] , [ n ] def = { 1 , 2 , ..., n } ) be the i th element in the vector.

Original: We deﬁne the objectives:
Revised: We deﬁne the objectives:

Original: [Equation]
Revised: [Equation]

Original: Let { x t } t  0 and { ˜ x t } t  0 be generated by using ` 2 , ` 3 objec- tives respectively.
Revised: Let { x t } t  0 and { ˜ x t } t  0 be generated by using ` 2 , ` 3 objec- tives respectively.

Original: Then deﬁne the total absolute prediction errors respectively:
Revised: Then deﬁne the total absolute prediction errors respectively:

Original: [Equation]
Revised: [Equation]

Original: where y ( i ) 2 R is the training target for the i th training sample.
Revised: where y ( i ) 2 R is the training target for the i th training sample.

Original: That is, 8 i 2 [ n ] ,
Revised: That is, 8 i 2 [ n ] ,

Original: [Equation]
Revised: [Equation]

Original: Given any 0 < ✏   0 =
Revised: Given any 0 < ✏   0 =

Original: P ni =1  0 ( i ) , deﬁne the following hitting time,
Revised: P ni =1  0 ( i ) , deﬁne the following hitting time,

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: Assume the same initialization x 0 = ˜ x 0 .
Revised: Assume the same initialization x 0 = ˜ x 0 .

Original: We have the following conclusion .
Revised: We have the following conclusion .

Original: If there exists  0 2 R and 0 < ✏   0 such that
Revised: If there exists  0 2 R and 0 < ✏   0 such that

Original: [Equation]
Revised: [Equation]

Original: then we have t ✏  ˜ t ✏ , which means gradient descent using the cubic loss function will achieve the total absolute error threshold ✏ faster than using the squared objective function.
Revised: then we have t ✏  ˜ t ✏ , which means gradient descent using the cubic loss function will achieve the total absolute error threshold ✏ faster than using the squared objective function.

Original: This theorem illustrates that when the total loss of all training examples is greater than some threshold, cubic power learns faster.
Revised: This theorem illustrates that when the total loss of all training examples is greater than some threshold, cubic power learns faster.

Original: For example, let the number of samples n = 1000 , and each sample has initial loss  0 ( i )
Revised: For example, let the number of samples n = 1000 , and each sample has initial loss  0 ( i )

Original: = 2 .
Revised: = 2 .

Original: Then  0 = 2000 .
Revised: Then  0 = 2000 .

Original: Setting ✏ = 570 (i.e., ✏ ( i ) ⇡ 0 . 57 ) satisﬁes the inequality (2).
Revised: Setting ✏ = 570 (i.e., ✏ ( i ) ⇡ 0 . 57 ) satisﬁes the inequality (2).

Original: This implies  using the cubic objective is faster when reducing the total loss from 2000 to 570 .
Revised: This implies that using the cubic objective is faster in reducing the total loss from 2000 to 570 .

Original: Though it is not our focus here to investigate the practical utility of the high power objectives, we include some empirical results and discuss the practical utilities of such objectives in Appendix A.6.
Revised: Though it is not our focus here to investigate the practical utility of the high power objectives, we include some empirical results and discuss the practical utilities of such objectives in Appendix A.6.

Original: Note that, although the original PER raises the importance ratio to a certain power, which is annealing from 1 at the beginning to 

Revised: Note that, although the original prioritized ER raises the importance ratio to a certain power, which is annealing from 1 at the beginning to

Original: [Schaul et al., 2016]; our theorem still explains the improvement of sample efﬁciency during the early learning stage.
Revised: [Schaul et al., 2016]; our theorem still explains the improvement of sample efﬁciency during the early learning stage.

Original: It is because, the power is close to one and hence it is equivalent to using a higher power loss.
Revised: It is because, the power is close to one and hence it is equivalent to using a higher power loss.

Original: This point has also been conﬁrmed by a concurrent work [Fujimoto et al., 2020, Sec 5.1, Theorem 3].
Revised: This point has also been conﬁrmed by a concurrent work [Fujimoto et al., 2020, Sec 5.1, Theorem 3].

Original: 3.2 LIMITATIONS OF THE PRIORITIZED ER
Revised: 3.2 LIMITATIONS OF THE PRIORITIZED ER

Original: Inspired by the above theorems, we now discuss two drawbacks of prioritized sampling: outdated priorities and insufﬁcient sample space coverage .
Revised: Inspired by the above theorems, we now discuss two drawbacks of prioritized sampling: outdated priorities and insufﬁcient sample space coverage .

Original: Then we empirically examine their importance and effects in the next section.
Revised: Then we empirically examine their importance and effects in the next section.

Original: The above two theorems show that the advantage of prioritized sampling comes from the faster convergence rate of cubic power objective during early learning.
Revised: The above two theorems show that the advantage of prioritized sampling comes from the faster convergence rate of cubic power objective during early learning.

Original: By Theorem 1, such advantage requires to update the priorities of all training samples by using the updated training parameters ✓ at each time step.
Revised: By Theorem 1, such advantage requires to update the priorities of all training samples by using the updated training parameters ✓ at each time step.

Original: In RL, however, at the each time step t , the original prioritized ER method only updates the priorities of those experiences from the sampled mini-batch, leaving the priorities of the rest of experiences unchanged [Schaul et al., 2016].
Revised: In RL, however, at the each time step t , the original prioritized ER method only updates the priorities of those experiences from the sampled mini-batch, leaving the priorities of the rest of experiences unchanged [Schaul et al., 2016].

Original: We call this limitation outdated priorities .
Revised: We call this limitation outdated priorities .

Original: It is typically infeasible to update the priorities of all experiences at each time step.
Revised: It is typically infeasible to update the priorities of all experiences at each time step.

Original: In fact, in RL, “ all training samples ” in RL are restricted to those visited experiences in the ER buffer, which may only contain a small subset of the whole state space, making the estimate of the prioritized sampling distribution inaccurate.
Revised: In fact, in RL, “ all training samples ” in RL are restricted to those visited experiences in the ER buffer, which may only contain a small subset of the whole state space, making the estimate of the prioritized sampling distribution inaccurate.

Original: There can be many reasons for the small coverage: the exploration is difﬁcult, the state space is huge, or the memory resource of the buffer is quite limited, etc.
Revised: There can be many reasons for the small coverage: the exploration is difﬁcult, the state space is huge, or the memory resource of the buffer is quite limited, etc.

Original: We call this issue insufﬁcient sample space coverage , which is also noted by Fedus et al.
Revised: We call this issue insufﬁcient sample space coverage , which is also noted by Fedus et al.

Original: Note that insufﬁcient sample space coverage should not be considered equivalent to off-policy distribution issue.
Revised: Note that insufﬁcient sample space coverage should not be considered equivalent to off-policy distribution issue.

Original: The latter refers to some old experiences in the ER buffer may be unlikely to appear under the current policy [Novati and
Revised: The latter refers to some old experiences in the ER buffer may be unlikely to appear under the current policy [Novati and

Original: Koumoutsakos, 2019, Zha et al., 2019, Sun et al., 2020, Oh et al., 2021].
Revised: Koumoutsakos, 2019, Zha et al., 2019, Sun et al., 2020, Oh et al., 2021].

Original: In contrast, the issue of insufﬁcient sample space coverage can raise naturally.
Revised: In contrast, the issue of insufﬁcient sample space coverage can raise naturally.

Original: For example, the state space is large and an agent is only able to visit a small subset of the state space during early learning stage.
Revised: For example, the state space is large and an agent is only able to visit a small subset of the state space during early learning stage.

Original: We visualize the state space coverage issue on a RL domain in Section 4.
Revised: We visualize the state space coverage issue on a RL domain in Section 4.

Original: 3.3 NEGATIVE EFFECTS OF THE LIMITATIONS
Revised: 3.3 NEGATIVE EFFECTS OF THE LIMITATIONS

Original: In this section, we empirically show that the outdated priorities and insufﬁcient sample space coverage signiﬁcantly blur the advantage of the prioritized sampling method.
Revised: In this section, we empirically show that the outdated priorities and insufﬁcient sample space coverage signiﬁcantly blur the advantage of the prioritized sampling method.

Original: Experiment setup .
Revised: Experiment setup .

Original: We conduct experiments on a supervised learning task.
Revised: We conduct experiments on a supervised learning task.

Original: We generate a training set T by uniformly sampling x 2 [  2 , 2] and adding zero-mean Gaussian noise with standard deviation  = 0 .
Revised: We generate a training set T by uniformly sampling x 2 [  2 , 2] and adding zero-mean Gaussian noise with standard deviation  = 0 .

Original: 5 to the target f sin ( x ) values.
Revised: 5 to the target f sin ( x ) values.

Original: Deﬁne f sin ( x ) def = sin(8 ⇡ x ) if x 2 [  2 , 0) and f sin ( x ) = sin( ⇡ x ) if x 2 [0 , 2] .
Revised: Deﬁne f sin ( x ) def = sin(8 ⇡ x ) if x 2 [  2 , 0) and f sin ( x ) = sin( ⇡ x ) if x 2 [0 , 2] .

Original: The testing set contains 1 k samples where the targets are not noise-contaminated.
Revised: The testing set contains 1 k samples where the targets are not noise-contaminated.

Original: Previous work [Pan et al., 2020] shows that the high frequency region [  2 , 0] usually takes long time to learn.
Revised: Previous work [Pan et al., 2020] shows that the high frequency region [  2 , 0] usually takes long time to learn.

Original: Hence we expect error-based prioritized sampling to make a clear difference in terms of sample efﬁciency on this dataset.
Revised: Hence we expect error-based prioritized sampling to make a clear difference in terms of sample efﬁciency on this dataset.

Original: We use 32 ⇥ 32 tanh layers neural network for all algorithms.
Revised: We use 32 ⇥ 32 tanh layers neural network for all algorithms.

Original: We refer to Appendix A.8 for missing details and A.7 for additional experiments.
Revised: We refer to Appendix A.8 for missing details and A.7 for additional experiments.

Original: Naming of algorithms .
Revised: Naming of algorithms .

Original: L2 : the l 2 regression with uniformly sampling from T .
Revised: L2 : the l 2 regression with uniformly sampling from T .

Original: Full-PrioritizedL2 : the l 2 regression with prioritized sampling according to the distribution deﬁned in (1), the priorities of all samples in the training set are updated after each mini-batch update.
Revised: Full-PrioritizedL2 : the l 2 regression with prioritized sampling according to the distribution deﬁned in (1), the priorities of all samples in the training set are updated after each mini-batch update.

Original: PrioritizedL2 : the only difference with Full-PrioritizedL2 is that only the priorities of those training examples sampled in the minibatch are updated at each iteration, the rest of the training samples use the original priorities.
Revised: PrioritizedL2 : the only difference with Full-PrioritizedL2 is that only the priorities of those training examples sampled in the minibatch are updated at each iteration, the rest of the training samples use the original priorities.

Original: This resembles the ap- proach taken by the prioritized ER in RL [Schaul et al., 2016].
Revised: This resembles the ap- proach taken by the prioritized ER in RL [Schaul et al., 2016].

Original: We show the learning curves in Figure 1.
Revised: We show the learning curves in Figure 1.

Original: Outdated priorities.
Revised: Outdated priorities.

Original: Figure 1 (a) shows that PrioritizedL without updating all priorities can be signiﬁcantly worse than Full-PrioritizedL2.
Revised: Figure 1 (a) shows that PrioritizedL without updating all priorities can be signiﬁcantly worse than Full-PrioritizedL2.

Original: Correspondingly, we further verify this phenomenon on the classical Mountain Car domain [Brockman et al., 2016].
Revised: Correspondingly, we further verify this phenomenon on the classical Mountain Car domain [Brockman et al., 2016].

Original: Figure 1(c) shows the evaluation learning curves of different DQN variants in an RL setting.
Revised: Figure 1(c) shows the evaluation learning curves of different DQN variants in an RL setting.

Original: We use a small 16 ⇥ 16 ReLu NN as the Q -function, which should highlight the issue of priority updating: every mini-batch update potentially perturbs the values of many other states.
Revised: We use a small 16 ⇥ 16 ReLu NN as the Q -function, which should highlight the issue of priority updating: every mini-batch update potentially perturbs the values of many other states.

Original: Hence many experiences in the ER buffer have the wrong priorities.
Revised: Hence many experiences in the ER buffer have the wrong priorities.

Original: Full-PrioritizedER does perform signiﬁcantly better.
Revised: Full-PrioritizedER does perform signiﬁcantly better.

Original: Sample space coverage.
Revised: Sample space coverage.

Original: To check the effect of insufﬁcient sample space coverage, we examine how the relative performances of L2 and Full-PrioritizedL2 change when we train them on a smaller training dataset with only 400 examples as shown in Figure 1(b).
Revised: To check the effect of insufﬁcient sample space coverage, we examine how the relative performances of L2 and Full-PrioritizedL2 change when we train them on a smaller training dataset with only 400 examples as shown in Figure 1(b).

Original: The small training set has a small coverage of the sample space.
Revised: The small training set has a small coverage of the sample space.

Original: Unsurprisingly, using a small training set makes all algorithms perform worse; however, it signiﬁcantly narrows the gap between Full-PrioritizedL and L2.
Revised: Unsurprisingly, using a small training set makes all algorithms perform worse; however, it signiﬁcantly narrows the gap between Full-PrioritizedL and L2.

Original: This indicates that prioritized sampling needs sufﬁcient samples across the sample space to estimate the prioritized sampling distribution reasonably accurate.
Revised: This indicates that prioritized sampling needs sufﬁcient samples across the sample space to estimate the prioritized sampling distribution reasonably accurate.

Original: We further verify the sample space coverage issue in prioritized ER on a RL problem in the next section.
Revised: We further verify the sample space coverage issue in prioritized ER on a RL problem in the next section.

Original:  ADDRESSING THE LIMITATIONS
Revised:  ADDRESSING THE LIMITATIONS

Original: In this section, we propose a Stochastic Gradient Langevin
Revised: In this section, we propose a Stochastic Gradient Langevin

Original: Dynamics (SGLD) sampling method to mitigate the limitations of the prioritized ER method mentioned in the above section.
Revised: Dynamics (SGLD) sampling method to mitigate the limitations of the prioritized ER method mentioned in the above section.

Original: Then we empirically examine our sampling distribution.
Revised: Then we empirically examine our sampling distribution.

Original: We also describe how our sampling method is used for the search-control component in Dyna.
Revised: We also describe how our sampling method is used for the search-control component in Dyna.

Original: 4.1 SAMPLING METHOD
Revised: 4.1 SAMPLING METHOD

Original: SGLD sampling method.
Revised: SGLD sampling method.

Original: Let v ⇡ ( · ; ✓ ) :
Revised: Let v ⇡ ( · ; ✓ ) :

Original: S 7!
Revised: S 7!

Original: R be a differentiable value function under policy ⇡ parameterized by ✓ .
Revised: R be a differentiable value function under policy ⇡ parameterized by ✓ .

Original: For s 2 S , deﬁne y ( s ) def = E r,s 

Revised: For s 2 S , deﬁne y ( s ) def = E r,s

Original: ⇠ P ⇡ ( s 0 ,r | s )
Revised: ⇠ P ⇡ ( s 0 ,r | s )

Original: [ r +  v ⇡ ( s 0 ; ✓ )] , and denote the TD error as  ( s, y ; ✓ t )
Revised: [ r +  v ⇡ ( s 0 ; ✓ )] , and denote the TD error as  ( s, y ; ✓ t )

Original: def =
Revised: def =

Original: y ( s )  v ( s ; ✓ t ) .
Revised: y ( s )  v ( s ; ✓ t ) .

Original: Given some initial state s 0 2 S , let the state sequence { s i } be the one generated by updating rule s i
Revised: Given some initial state s 0 2 S , let the state sequence { s i } be the one generated by updating rule s i

Original: + s i
Revised: + s i

Original: + ↵ h r s log |  ( s i , y ( s i ); ✓ t ) | + X
Revised: + ↵ h r s log |  ( s i , y ( s i ); ✓ t ) | + X

Original: i , where ↵ h is a stepsize and X i is a Gaussian random variable with some constant variance.
Revised: i , where ↵ h is a stepsize and X i is a Gaussian random variable with some constant variance.

Original: Then { s i } converges to the distribution p ( s ) /
Revised: Then { s i } converges to the distribution p ( s ) /

Original: |  ( s, y ( s ))
Revised: |  ( s, y ( s ))

Original: | as i ! 1 .
Revised: | as i ! 1 .

Original: The proof is a direct consequence of the convergent behavior of Langevin dynam- ics stochastic differential equation (SDE)
Revised: The proof is a direct consequence of the convergent behavior of Langevin dynam- ics stochastic differential equation (SDE)

Original: [Roberts, 1996,
Revised: [Roberts, 1996,

Original: Welling and Teh, 2011, Zhang et al., 2017].
Revised: Welling and Teh, 2011, Zhang et al., 2017].

Original: We include a brief background knowledge in Appendix A.2.
Revised: We include a brief background knowledge in Appendix A.2.

Original: It should be noted that, this sampling method enables us to acquire states 1) whose absolute TD errors are estimated by using  current parameter ✓ t and 2) that are not restricted to those visited ones .
Revised: It should be noted that, this sampling method enables us to acquire states 1) whose absolute TD errors are estimated by using the current parameter ✓ t and 2) that are not restricted to those visited ones .

Original: We empirically verify the two points in Section 4.2.
Revised: We empirically verify the two points in Section 4.2.

Original: Implementation .
Revised: Implementation .

Original: In practice, we can compute the state value estimate by v ( s ) = max a Q ( s, a ; ✓ t ) as suggested by Pan et al.
Revised: In practice, we can compute the state value estimate by v ( s ) = max a Q ( s, a ; ✓ t ) as suggested by Pan et al.

Original: In the case that a true environment model is not available, we compute an estimate ˆ y ( s ) of y ( s ) by a learned model.
Revised: In the case that a true environment model is not available, we compute an estimate ˆ y ( s ) of y ( s ) by a learned model.

Original: Then at each time step t , states approximately following the distribution p ( s ) / |  ( s, y ( s )) | can be generated by
Revised: Then at each time step t , states approximately following the distribution p ( s ) / |  ( s, y ( s )) | can be generated by

Original: [Equation]
Revised: [Equation]

Original: where X is a Gaussian random variable with zero-mean and some small variance.
Revised: where X is a Gaussian random variable with zero-mean and some small variance.

Original: Observing that ↵ h is small, we consider ˆ y ( s ) as a constant given a state s without backpropagating through it.
Revised: Observing that ↵ h is small, we consider ˆ y ( s ) as a constant given a state s without backpropagating through it.

Original: Though this updating rule introduces bias due to the usage of a learned model, fortunately, the difference between the sampling distribution acquired by the true model and the learned model can be upper bounded as we show in Theorem 3 in Appendix A.5.
Revised: Though this updating rule introduces bias due to the usage of a learned model, fortunately, the difference between the sampling distribution acquired by the true model and the learned model can be upper bounded as we show in Theorem 3 in Appendix A.5.

Original: Algorithmic details.
Revised: Algorithmic details.

Original: We present our algorithm called DynaTD in the Algorithm 3 in Appendix A.8.
Revised: We present our algorithm called DynaTD in the Algorithm 3 in Appendix A.8.

Original: Our algorithm follows the general steps in Algorithm 1.
Revised: Our algorithm follows the general steps in Algorithm 1.

Original: Particularly, we choose the function h ( s ) def = log | ˆ y ( s )  max a Q ( s, a ; ✓ t ) | for HC search-control process, i.e., run the updating ruleto generate states.
Revised: Particularly, we choose the function h ( s ) def = log | ˆ y ( s )  max a Q ( s, a ; ✓ t ) | for HC search-control process, i.e., run the updating ruleto generate states.

Original: 4.2 EMPIRICAL VERIFICATION OF TD ERROR-BASED SAMPLING METHOD
Revised: 4.2 EMPIRICAL VERIFICATION OF TD ERROR-BASED SAMPLING METHOD

Original: We visualize the distribution of the sampled states by our method and those from the buffer of the prioritized ER, verifying that our sampled states have an obviously larger coverage of the state space.
Revised: We visualize the distribution of the sampled states by our method and those from the buffer of the prioritized ER, verifying that our sampled states have an obviously larger coverage of the state space.

Original: We then empirically verify that our sampling distribution is closer to a brute-force calculated prioritized sampling distribution—which does not suffer from the two limitations—than the prioritized ER method.
Revised: We then empirically verify that our sampling distribution is closer to a brute-force calculated prioritized sampling distribution—which does not suffer from the two limitations—than the prioritized ER method.

Original: Finally, we discuss concerns regarding computational cost.
Revised: Finally, we discuss concerns regarding computational cost.

Original: Please see Appendix A.8 for any missing details.
Revised: Please see Appendix A.8 for any missing details.

Original: Large sample space coverage .
Revised: Large sample space coverage .

Original: During early learning, we visualize 2 k states sampled from 1) DQN’s buffer trained by prioritized ER and 2) our algorithm Dyna-TD’s SearchControl (SC) queue on the continuous state GridWorld (Figure 2(a)).
Revised: During early learning, we visualize 2 k states sampled from 1) DQN’s buffer trained by prioritized ER and 2) our algorithm Dyna-TD’s SearchControl (SC) queue on the continuous state GridWorld (Figure 2(a)).

Original: Figure 2 (b-c) visualize state distributions with different sampling methods via heatmap.
Revised: Figure 2 (b-c) visualize state distributions with different sampling methods via heatmap.

Original: Darker color indicates higher density.
Revised: Darker color indicates higher density.

Original: (b)(c) show that DQN’s ER buffer, no matter with or without prioritized sampling, does not
Revised: (b)(c) show that DQN’s ER buffer, no matter with or without prioritized sampling, does not

Original: [Figure]
Revised: [Figure]

Original: cover well the top-left part and the right half part on the
Revised: cover well the top-left part and the right half part on the

Original: GridWorld.
Revised: GridWorld.

Original: In contrast, Figure 2 (d) shows that states from our SC queue are more diversely distributed on the square.
Revised: In contrast, Figure 2 (d) shows that states from our SC queue are more diversely distributed on the square.

Original: These visualizations verify that our sampled states cover better the sample space than the prioritized ER does.
Revised: These visualizations verify that our sampled states cover better the state space than the prioritized ER does.

Original: Notations and experiment setting .
Revised: Notations and experiment setting .

Original: We denote our sampling distribution as p 1 ( · ) , the one acquired by conventional prioritized ER as p 2 ( · ) , and the one computed by thorough priority updating of enumerating all states in the state space as p ⇤ ( · ) (this one should be unrealistic in practice and we call it the ideal distribution as it does not suffer from the two limitations we discussed).
Revised: We denote our sampling distribution as p 1 ( · ) , the one acquired by conventional prioritized ER as p 2 ( · ) , and the one computed by thorough priority updating of enumerating all states in the state space as p ⇤ ( · ) (this one should be unrealistic in practice and we call it the ideal distribution as it does not suffer from the two limitations we discussed).

Original: We visualize how well p 1 ( · ) and p 2 ( · ) can approximate p ⇤ ( · ) on the GridWorld domain, where the state distributions can be conveniently estimated by discretizing the continuous state GridWorld to a 50 ⇥ 50 one.
Revised: We visualize how well p 1 ( · ) and p 2 ( · ) can approximate p ⇤ ( · ) on the GridWorld domain, where the state distributions can be conveniently estimated by discretizing the continuous state GridWorld to a 50 ⇥ 50 one.

Original: We compute the distances of p 1 , p 2 to p ⇤ by two sensible weighting schemes: 1) on-policy weighting:
Revised: We compute the distances of p 1 , p 2 to p ⇤ by two sensible weighting schemes: 1) on-policy weighting:

Original: [Equation]
Revised: [Equation]

Original: { 1 , 2 } , where d ⇡ is approximated by uniformly sample 3 k states from a recency buffer; 2) uniform weighting
Revised: { 1 , 2 } , where d ⇡ is approximated by uniformly sample 3 k states from a recency buffer; 2) uniform weighting

Original: [Equation]
Revised: [Equation]

Original: Sampling distribution is close to the ideal one .
Revised: Sampling distribution is close to the ideal one .

Original: We plot the distances change when we train our Algorithm 3 and the prioritized ER in Figure 3(a)(b).
Revised: We plot the distances change when we train our Algorithm 3 and the prioritized ER in Figure 3(a)(b).

Original: They show that the
Revised: They show that the

Original: HC procedure in our algorithm Dyna-TD-Long produces a state distribution with a signiﬁcantly closer distance to the desired sampling distribution p ⇤ than PrioritizedER under both weighting schemes.
Revised: HC procedure in our algorithm Dyna-TD-Long produces a state distribution with a signiﬁcantly closer distance to the desired sampling distribution p ⇤ than PrioritizedER under both weighting schemes.

Original: In contrast, the state distribution acquired from PrioritizedER, which suffers from the two limitations, is far away from p ⇤ .
Revised: In contrast, the state distribution acquired from PrioritizedER, which suffers from the two limitations, is far away from p ⇤ .

Original: Note that the sufﬁx “-Long” of Dyna-TD-Long indicates that we run a large number of
Revised: Note that the sufﬁx “-Long” of Dyna-TD-Long indicates that we run a large number of

Original: SGLD steps (i.e., 1 k) to reach stationary behavior.
Revised: SGLD steps (i.e., 1 k) to reach stationary behavior.

Original: This is a sanity check but impractical; hence, we test the version with only a few SGLD steps.
Revised: This is a sanity check but impractical; hence, we test the version with only a few SGLD steps.

Original: Sampling distribution with much fewer SGLD steps .
Revised: Sampling distribution with much fewer SGLD steps .

Original: In practice, we probably only want to run a small number of
Revised: In practice, we probably only want to run a small number of

Original: SGLD steps to save time.
Revised: SGLD steps to save time.

Original: As a result, we include a practical version of Dyna-TD, which only runs 30 SGLD steps, with either a true or learned model.
Revised: As a result, we include a practical version of Dyna-TD, which only runs 30 SGLD steps, with either a true or learned model.

Original: Figure 3(a)(b) show that even a few SGLD steps can give better sampling distribution than the conventional PrioritizedER does.
Revised: Figure 3(a)(b) show that even a few SGLD steps can give better sampling distribution than the conventional PrioritizedER does.

Original: Computational cost.
Revised: Computational cost.

Original: Let the mini-batch size be b , and the number of HC steps be k HC .
Revised: Let the mini-batch size be b , and the number of HC steps be k HC .

Original: If we assume one mini-batch update takes O ( c ) , then the time cost of our sampling is
Revised: If we assume one mini-batch update takes O ( c ) , then the time cost of our sampling is

Original: O ( ck HC /b ) , which is reasonable.
Revised: O ( ck HC /b ) , which is reasonable.

Original: On the GridWorld, Figure 3(c) shows that given the same time budget, our algorithm achieves better performance.
Revised: On the GridWorld, Figure 3(c) shows that given the same time budget, our algorithm achieves better performance.

Original: This makes the additional time spent on search-control worth it.
Revised: This makes the additional time spent on search-control worth it.

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: In this section, we ﬁrstly introduce baselines and the basic experimental setup.
Revised: In this section, we ﬁrstly introduce baselines and the basic experimental setup.

Original: Then we design experiments in the three paragraphs 1) performances on benchmarks , 2) Dyna variants comparison , and 3) a demo for continuous control to answerbelow three corresponding questions.
Revised: Then we design experiments in the three paragraphs 1) performances on benchmarks , 2) Dyna variants comparison , and 3) a demo for continuous control to answer three following questions correspondingly.

Original: By mitigating the limitations of the conventional prioritized ER method, can Dyna-TD outperform the prioritized ER under various planning budgets in different environments? 2.
Revised: By mitigating the limitations of the conventional prioritized ER method, can Dyna-TD outperform the prioritized ER under various planning budgets in different environments? 2.

Original: Can Dyna-TD outperform the existing Dyna variants?
Revised: Can Dyna-TD outperform the existing Dyna variants?

Original: How effective is Dyna-TD under an online learned model, particularly for more realistic applications where actions are continuous?
Revised: How effective is Dyna-TD under an online learned model, particularly for more realistic applications where actions are continuous?

Original: Baselines and basic setup.
Revised: Baselines and basic setup.

Original: ER is DQN with a regular ER buffer without prioritized sampling.
Revised: ER is DQN with a regular ER buffer without prioritized sampling.

Original: PrioritizedER is the one by Schaul et al.
Revised: PrioritizedER is the one by Schaul et al.

Original: [2016], which has the drawbacks as discussed in our paper.
Revised: [2016], which has the drawbacks as discussed in our paper.

Original: Dyna-Value
Revised: Dyna-Value

Original: [Pan et al., 2019] is the Dyna variant which performs HC on the learned value function to acquire states to populate the SC queue.
Revised: [Pan et al., 2019] is the Dyna variant which performs HC on the learned value function to acquire states to populate the SC queue.

Original: DynaFrequency [Pan et al., 2020] is the Dyna variant which performs HC on the norm of the gradient of the value function to acquire states to populate the SC queue.
Revised: DynaFrequency [Pan et al., 2020] is the Dyna variant which performs HC on the norm of the gradient of the value function to acquire states to populate the SC queue.

Original: For fair
Revised: For fair

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: comparison, at each environment time step, we stochastically sample the same number of mini-batches to train those model-free baselines as the number of planning updates in Dyna variants.
Revised: comparison, at each environment time step, we stochastically sample the same number of mini-batches to train those model-free baselines as the number of planning updates in Dyna variants.

Original: We are able to ﬁx the same HC hyperparameter setting across all environments.
Revised: We are able to ﬁx the same HC hyperparameter setting across all environments.

Original: Whenever it involves an online learned model, we use the mean squared error to learn a deterministic model, which we found to be reasonably good on those tested domains in this paper.
Revised: Whenever it involves an online learned model, we use the mean squared error to learn a deterministic model, which we found to be reasonably good on those tested domains in this paper.

Original: Please see Appendix A.8 for experiment details 2 .
Revised: Please see Appendix A.8 for experiment details.

Original:  We also refer readers to Appendix A.7.4 for experiments on the autonomous driving domain.
Revised: 2 We also refer readers to Appendix A.7.4 for experiments on the autonomous driving domain.

Original: Performances on benchmarks.
Revised: Performances on benchmarks.

Original: Figure 4 shows the performances of different algorithms on MountainCar, Acrobot,
Revised: Figure 4 shows the performances of different algorithms on MountainCar, Acrobot,

Original: GridWorld (Figure 2(a)), and CartPole.
Revised: GridWorld (Figure 2(a)), and CartPole.

Original: On these small domains, we focus on studying our sampling distribution and hence we need to isolate the effect of model errors (by using a true environment model), though we include our algorithm
Revised: On these small domains, we focus on studying our sampling distribution and hence we need to isolate the effect of model errors (by using a true environment model), though we include our algorithm

Original: Dyna-TD with an online learned model for curiosity.
Revised: Dyna-TD with an online learned model for curiosity.

Original: We have the following observations.
Revised: We have the following observations.

Original: First, our algorithm Dyna-
Revised: First, our algorithm Dyna-

Original: TD consistently outperforms PrioritizedER across domains and planning updates.
Revised: TD consistently outperforms PrioritizedER across domains and planning updates.

Original: In contrast, the PrioritizedER may not even outperform regular ER, as occurred in the previous supervised learning experiment.
Revised: In contrast, the PrioritizedER may not even outperform regular ER, as occurred in the previous supervised learning experiment.

Original: Second, Dyna-TD’s performance signiﬁcantly improves and even outperforms other Dyna variants when increasing the planning budget (i.e., planning updates n ) from 10 to 30 .
Revised: Second, Dyna-TD’s performance signiﬁcantly improves and even outperforms other Dyna variants when increasing the planning budget (i.e., planning updates n ) from 10 to 30 .

Original: This validates the utility of those additional hypothetical experiences acquired by our sampling method.
Revised: This validates the utility of those additional hypothetical experiences acquired by our sampling method.

Original: In contrast, both ER and PrioritizedER show limited gain when increasing the planning budget (i.e., number of mini-batch updates), which implies the limited utility of those visited experiences.
Revised: In contrast, both ER and PrioritizedER show limited gain when increasing the planning budget (i.e., number of mini-batch updates), which implies the limited utility of those visited experiences.

Original: Dyna variants comparison .
Revised: Dyna variants comparison .

Original: Dyna-Value occasionally ﬁnds a sub-optimal policy when using a large number of planning updates, while Dyna-TD always ﬁnds a better policy.
Revised: Dyna-Value occasionally ﬁnds a sub-optimal policy when using a large number of planning updates, while Dyna-TD always ﬁnds a better policy.

Original: We hypothesize that Dyna-Value results in a heavy sampling distribution bias even during the late learning stage, with density always concentrated around the high-value regions.
Revised: We hypothesize that Dyna-Value results in a heavy sampling distribution bias even during the late learning stage, with density always concentrated around the high-value regions.

Original: We veriﬁed our hypothesis by checking the entropy of the sampling distribution in the late training stage, as shown in Figure 5.
Revised: We veriﬁed our hypothesis by checking the entropy of the sampling distribution in the late training stage, as shown in Figure 5.

Original: A high entropy indicates the sampling distri-
Revised: A high entropy indicates the sampling distri-

Original: [Figure]
Revised: [Figure]

Original: bution is more dispersed than the with low entropy
Revised: bution is more dispersed than the with low entropy

Original: We found that the sampling distribution of Dyna-Value has lower entropy than Dyna-TD.
Revised: We found that the sampling distribution of Dyna-Value has lower entropy than Dyna-TD.

Original: Dyna-Frequency suffers from explosive or zero gradients.
Revised: Dyna-Frequency suffers from explosive or zero gradients.

Original: It requires computing third-order differentiation r s
Revised: It requires computing third-order differentiation r s

Original: ||
Revised: ||

Original: H v ( s ) ||
Revised: H v ( s ) ||

Original: (i.e., taking the gradient of the Hessian).
Revised: (i.e., taking the gradient of the Hessian).

Original: It is hence sensitive to domains and parameter settings such as learning rate choice and activation type.
Revised: It is hence sensitive to domains and parameter settings such as learning rate choice and activation type.

Original: This observation is consistent with the description from Pan et al.
Revised: This observation is consistent with the description from Pan et al.

Original: [Figure]
Revised: [Figure]

Original: (a) has entropy around 4 . 5 and (b) has entropy around 3 .
Revised: (a) has entropy around 4 . 5 and (b) has entropy around 3 .

Original: A demo for continuous control.
Revised: A demo for continuous control.

Original: We demonstrate that our approach can be applied for Mujoco [Todorov et al., 2012] continuous control problems with an online learned model and still achieve superior performance.
Revised: We demonstrate that our approach can be applied for Mujoco [Todorov et al., 2012] continuous control problems with an online learned model and still achieve superior performance.

Original: We use DDPG (Deep
Revised: We use DDPG (Deep

Original: Deterministic Policy Gradient)
Revised: Deterministic Policy Gradient)

Original: [Lillicrap et al., 2016, Silver et al., 2014] as an example for use inside our Dyna-TD.
Revised: [Lillicrap et al., 2016, Silver et al., 2014] as an example for use inside our Dyna-TD.

Original: Let
Revised: Let

Original: [Figure]
Revised: [Figure]

Original: A be the actor, then we set the HC function as h ( s ) def = log | ˆ y  Q ✓ ( s, ⇡ ✓ 0 ( s )) | where ˆ y is the TD target.
Revised: A be the actor, then we set the HC function as h ( s ) def = log | ˆ y  Q ✓ ( s, ⇡ ✓ 0 ( s )) | where ˆ y is the TD target.

Original: Figure 6 (a)(b) shows the learning curves of DDPG trained with ER, PrioritizedER, and our Dyna-TD on Hopper and
Revised: Figure 6 (a)(b) shows the learning curves of DDPG trained with ER, PrioritizedER, and our Dyna-TD on Hopper and

Original: Walker2d respectively.
Revised: Walker2d respectively.

Original: Since other Dyna variants never show an advantage and are not relevant to the purpose of this experiment, we no longer include them.
Revised: Since other Dyna variants never show an advantage and are not relevant to the purpose of this experiment, we no longer include them.

Original: Dyna-TD shows quick improvement as before.
Revised: Dyna-TD shows quick improvement as before.

Original: This indicates our sampled hypothetical experiences could be helpful for actor-critic algorithms that are known to be prone to local optimums.
Revised: This indicates our sampled hypothetical experiences could be helpful for actor-critic algorithms that are known to be prone to local optimums.

Original: Additionally, we note again that ER outperforms PrioritizedER, as occurred in thediscrete control and supervised learning (PrioritizedL2 is worse than L2) experiments.
Revised: Additionally, we note again that ER outperforms PrioritizedER, as occurred in the supervised learning (PrioritizedL2 is worse than L2) experiments.

Original: [Figure]
Revised: [Figure]

Original:  DISCUSSION
Revised:  DISCUSSION

Original: We provide theoretical insight into the error-based prioritized sampling by establishing its equivalence to the uniform sampling for a cubic power objective in a supervised learning setting.
Revised: We provide theoretical insight into the error-based prioritized sampling by establishing its equivalence to the uniform sampling for a cubic power objective in a supervised learning setting.

Original: Then we identify two drawbacks of prioritized
Revised: Then we identify two drawbacks of prioritized

Original: ER: outdated priorities and insufﬁcient sample space coverage.
Revised: ER: outdated priorities and insufﬁcient sample space coverage.

Original: We mitigate the two limitations by SGLD sampling method with empirical veriﬁcation.
Revised: We mitigate the two limitations by SGLD sampling method with empirical veriﬁcation.

Original: Our empirical results on both discrete and continuous control domains show the efﬁcacy of our method.
Revised: Our empirical results on both discrete and continuous control domains show the efﬁcacy of our method.

Original: There are several promising future directions.
Revised: There are several promising future directions.

Original: First, a natural question is how a model should be learned to beneﬁt a particular sampling method, as this work mostly focuses on sampling hypothetical experiences without considering model learning algorithms.
Revised: First, a natural question is how a model should be learned to beneﬁt a particular sampling method, as this work mostly focuses on sampling hypothetical experiences without considering model learning algorithms.

Original: Existing results show that learning a model while considering how to use it should make the policy robust to model errors [Farahmand et al., 2017, Farahmand, 2018].
Revised: Existing results show that learning a model while considering how to use it should make the policy robust to model errors [Farahmand et al., 2017, Farahmand, 2018].

Original: Second, one may apply our approach with a model in latent space [Hamilton et al., 2014, Wahlström et al., 2015, Ha and Schmidhuber, 2018, Hafner et al., 2019,
Revised: Second, one may apply our approach with a model in latent space [Hamilton et al., 2014, Wahlström et al., 2015, Ha and Schmidhuber, 2018, Hafner et al., 2019,

Original: Schrittwieser et al., 2020], which enables our method to scale to large domains.
Revised: Schrittwieser et al., 2020], which enables our method to scale to large domains.

Original: Third, since there are existing works examining how ER is affected by bootstrap return [Daley and Amato, 2019], by buffer or mini-batch size [Zhang and
Revised: Third, since there are existing works examining how ER is affected by bootstrap return [Daley and Amato, 2019], by buffer or mini-batch size [Zhang and

Original: Sutton, 2017, Liu and Zou, 2017], and by environment steps taken per gradient step [Fu et al., 2019, van Hasselt et al., 2018, Fedus et al., 2020], it is worth studying the theoretical implications of those design choices and their effects on prioritized ER’s efﬁcacy.
Revised: Sutton, 2017, Liu and Zou, 2017], and by environment steps taken per gradient step [Fu et al., 2019, van Hasselt et al., 2018, Fedus et al., 2020], it is worth studying the theoretical implications of those design choices and their effects on prioritized ER’s efﬁcacy.

Original: Last, as our cubic objective explains only one version of the error-based prioritization, efforts should also be made to theoretically interpret other sampling distributions, such as distribution location or reward-based prioritization [Lambert et al., 2020].
Revised: Last, as our cubic objective explains only one version of the error-based prioritization, efforts should also be made to theoretically interpret other sampling distributions, such as distribution location or reward-based prioritization [Lambert et al., 2020].

Original: It is interesting to explore whether these alternatives can be formulated as surrogate objectives.
Revised: It is interesting to explore whether these alternatives can be formulated as surrogate objectives.

Original: Furthermore, a recent work by Fujimoto et al.
Revised: Furthermore, a recent work by Fujimoto et al.

Original: [2020] establishes an equivalence between various distributions and uniform sampling for different loss functions.
Revised: [2020] establishes an equivalence between various distributions and uniform sampling for different loss functions.

Original: Studying if those general loss functions have faster convergence rate as shown in our Theorem 2 could help illuminate their beneﬁts
Revised: Studying if those general loss functions have faster convergence rate as shown in our Theorem 2 could help illuminate their beneﬁts.

Original: . Acknowledgements
Revised: Acknowledgements

Original: We would like to thank all anonymous reviewers for their helpful feedback during multiple submissions of this paper.
Revised: We would like to thank all anonymous reviewers for their helpful feedback during multiple submissions of this paper.

Original: We acknowledge the funding from the Canada CIFARAI
Revised: We acknowledge the funding from the Canada CIFAR

Original:  Chairs program and Alberta Machine Intelligence Institute.
Revised: AI Chairs program, Alberta Machine Intelligence Institute, and Natural Sciences and Engineering Council of Canada (NSERC) Discovery Grant