Original: R EMEMBER AND F ORGET FOR E XPERIENCE R EPLAY
Revised: R EMEMBER AND F ORGET FOR E XPERIENCE R EPLAY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Experience replay (ER) is crucial for attaining high data-efﬁciency in off-policy  reinforcement learning (RL).
Revised: Experience replay (ER) is crucial for attaining high data-efﬁciency in off-policy deep reinforcement learning (RL).

Original: ER entails the recall of experiences obtained in past iterations to compute gradient estimates for the current policy.
Revised: ER entails the recall of experiences obtained in past iterations to compute gradient estimates for the current policy.

Original: However, the accuracy of such updates may deteriorate when the policy diverges from past behaviors, possibly undermining the effectiveness of ER.
Revised: However, the accuracy of such updates may deteriorate when the policy diverges from past behaviors, possibly undermining the effectiveness of ER.

Original: Previous off-policy RL algorithms mitigated this issue by tuning their hyper-parameters in order to abate policy changes.
Revised: Previous off-policy RL algorithms mitigated this issue by tuning their hyper-parameters in order to abate policy changes.

Original: We propose  a method for ER that relies on systematically Re membering and F orgetting past behaviors ( ReF-ER ).
Revised: We propose ReF-ER , a method for active management of experiences in the Replay Memory (RM).

Original: ReF-ER forgets experiences that would be too unlikely with the current policy and constrains policy changes within a trust region of the behaviors in the replay memory.
Revised: ReF-ER forgets experiences that would be too unlikely with the current policy and constrains policy changes within a trust region of the behaviors in the RM.

Original: We couple ReF-ER with Q-learning, deterministic policy gradient and off-policy gradient methods and we show that ReF-ER reliably improves the performance of continuous-action off-policy RL.
Revised: We couple ReF-ER with Q-learning, deterministic policy gradient and off-policy gradient methods to show that ReF-ER reliably improves the performance of continuous-action off-policy RL.

Original: We complement ReF-ER with a novel off-policy actor-critic algorithm ( RACER ) for continuous-action control.
Revised: We complement ReF-ER with a novel off-policy actor-critic algorithm ( RACER ) for continuous-action control.

Original: RACER employs a computationally efﬁcient closedform approximation of the action values and is shown to be highly competitive with state-of-the-art algorithms on benchmark problems, while being robust to large hyper-parameter variations.
Revised: RACER employs a computationally efﬁcient closed-form approximation of the action values and is shown to be highly competitive with state-of-the-art algorithms on benchmark problems, while being robust to large hyper-parameter variations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Reinforcement Learning (RL) has an ever increasing number of success stories ranging from realistic simulated environments (Schulman et al., 2015;
Revised: Deep Reinforcement Learning (RL) has an ever increasing number of success stories ranging from realistic simulated environments (Schulman et al., 2015;

Original: Mnih et al., 2016), robotics (Levine et al., 2016) and games (Mnih et al., 2015; Silver et al., 2016).
Revised: Mnih et al., 2016), robotics (Levine et al., 2016) and games (Mnih et al., 2015; Silver et al., 2016).

Original: Experience Replay (ER) (Lin, 1992) enhances deep RL algorithms by using information collected in past policy ( µ ) iterations to compute updates for the current policy ( π ).
Revised: Experience Replay (ER) (Lin, 1992) enhances deep RL algorithms by using information collected in past policy ( µ ) iterations to compute updates for the current policy ( π ).

Original: ER has become one of the mainstay techniques to improve the sample-efﬁciency of off-policy RL.
Revised: ER has become one of the mainstay techniques to improve the sample-efﬁciency of off-policy RL.

Original: Sampling from a Replay Memory (RM) stabilizes stochastic gradient descent (SGD) by disrupting temporal correlations and extracts information from useful experiences over multiple updates (Schaul et al., 2015b).
Revised: Sampling from a Replay Memory (RM) stabilizes stochastic gradient descent (SGD) by disrupting temporal correlations and extracts information from useful experiences over multiple updates (Schaul et al., 2015b).

Original: However, when π is parameterized by a deep
Revised: However, when π is parameterized by a deep

Original: Neural Network (NN), SGD updates may result in signiﬁcant changes to the policy, thereby shifting the distribution of states observed from the environment.
Revised: Neural Network (NN), SGD updates may result in signiﬁcant changes to the policy, thereby shifting the distribution of states observed from the environment.

Original: In this case sampling the RM for further updates may lead to incorrect gradient estimates, therefore RL methods must account for and limit the dissimilarity between π and past behaviors in the RM.
Revised: In this case sampling the RM for further updates may lead to incorrect gradient estimates, therefore RL methods must account for and limit the dissimilarity between π and past behaviors in the RM.

Original: Previous works employed trust region methods to bound policy updates (Schulman et al., 2015; Wang et al., 2017).
Revised: Previous works employed trust region methods to bound policy updates (Schulman et al., 2015; Wang et al., 2017).

Original: Despite several successes, deep
Revised: Despite several successes, deep

Original: RL algorithms are known to suffer from instabilities and exhibit high-variance of outcomes (Islam et al., 2017; Henderson et al., 2017), especially continuous-action methods employing the stochastic (Sutton et al., 2000) or deterministic (Silver et al., 2014) Policy Gradients (PG or DPG).
Revised: RL algorithms are known to suffer from instabilities and exhibit high-variance of outcomes (Islam et al., 2017; Henderson et al., 2017), especially continuous-action methods employing the stochastic (Sutton et al., 2000) or deterministic (Silver et al., 2014) Policy Gradients (PG or DPG).

Original: In this work we redesign ER in order to control the distance between the behaviors µ used to compute the update and the current policy π .
Revised: In this work we redesign ER in order to control the distance between the behaviors µ used to compute the update and the current policy π .

Original: More speciﬁcally, we classify experiences either as “near-policy" or “far-policy", depending on the ratio ρ of probabilities of selecting the associated action with π and that with µ .
Revised: More speciﬁcally, we classify experiences either as “near-policy" or “far-policy", depending on the ratio ρ of probabilities of selecting the associated action with π and that with µ .

Original: The weight ρ appears in many estimators that are used with ER such as the off-policy policy gradients (off-PG) (Degris et al., 2012) and the off-policy return-based evaluation algorithm
Revised: The weight ρ appears in many estimators that are used with ER such as the off-policy policy gradients (off-PG) (Degris et al., 2012) and the off-policy return-based evaluation algorithm

Original: Retrace (Munos et al., 2016).
Revised: Retrace (Munos et al., 2016).

Original: From this classiﬁcation, we introduce measures to limit the fraction of “far-policy" samples in the RM, as well as computing gradient estimates only from “near-policy" experiences.
Revised: From this classiﬁcation, we introduce measures to limit the fraction of “far-policy" samples in the RM, as well as computing gradient estimates only from “near-policy" experiences.

Original: Furthermore, these hyper-parameters can be gradually annealed during training to obtain increasingly accurate updates from nearly on-policy experiences.
Revised: Furthermore, these hyper-parameters can be gradually annealed during training to obtain increasingly accurate updates from nearly on-policy experiences.

Original: Remember and Forget Experience
Revised: Remember and Forget Experience

Original: Replay ( ReF-ER ) is a simple algorithm that can be applied to virtually any off-policy RL algorithm with parameterized policies.
Revised: Replay ( ReF-ER ) is a simple algorithm that can be applied to virtually any off-policy RL algorithm with parameterized policies.

Original: We show that ReF-ER allows better stability and performance than conventional ER in all three main classes of continuous-actions off-policy RL algorithms: methods based on the DPG (ie.
Revised: We show that ReF-ER allows better stability and performance than conventional ER in all three main classes of continuous-actions off-policy RL algorithms: methods based on the DPG (ie.

Original: DDPG (Lillicrap et al., 2016)), methods based on Q-learning (ie.
Revised: DDPG (Lillicrap et al., 2016)), methods based on Q-learning (ie.

Original: NAF (Gu et al., 2016)), and with off-PG (Degris et al., 2012; Wang et al., 2017).
Revised: NAF (Gu et al., 2016)), and with off-PG (Degris et al., 2012; Wang et al., 2017).

Original: In recent years, there has been a growing interest in coupling RL with high-ﬁdelity physics simulations (Reddy et al., 2016; Novati et al., 2017; Colabrese et al., 2017; Verma et al., 2018).
Revised: In recent years, there has been a growing interest in coupling RL with high-ﬁdelity physics simulations (Reddy et al., 2016; Novati et al., 2017; Colabrese et al., 2017; Verma et al., 2018).

Original: The computational cost of these simulations calls for data-efﬁcient RL methods that are reliable and do not require problem-speciﬁc tweaks to the hyper-parameters.
Revised: The computational cost of these simulations calls for data-efﬁcient RL methods that are reliable and do not require problem-speciﬁc tweaks to the hyper-parameters.

Original: Moreover, while on-policy training of simple policy architectures has been shown to be sufﬁcient in some benchmark environments (Rajeswaran et al., 2017), agents aiming to solve complex problems with highly non-linear dynamics might require deep or recurrent models that can be trained more efﬁciently with off-policy methods.
Revised: Moreover, while on-policy training of simple policy architectures has been shown to be sufﬁcient in some benchmark environments (Rajeswaran et al., 2017), agents aiming to solve complex problems with highly non-linear dynamics might require deep or recurrent models that can be trained more efﬁciently with off-policy methods.

Original: We address these issues by introducing a simple and computationally efﬁcient off-policy actor-critic architecture ( RACER ) for continuous-action control problems.
Revised: We address these issues by introducing a simple and computationally efﬁcient off-policy actor-critic architecture ( RACER ) for continuous-action control problems.

Original: We systematically analyze a wide range of hyper-parameters on the OpenAI Gym (Brockman et al., 2016) robotic tasks, and show that
Revised: We systematically analyze a wide range of hyper-parameters on the OpenAI Gym (Brockman et al., 2016) robotic tasks, and show that

Original: RACER combined with ReF-ER reliably obtains results that are competitive with the state-of-the-art.
Revised: RACER combined with ReF-ER reliably obtains results that are competitive with the state-of-the-art.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Consider the sequential decision process of an agent aiming to optimize its interaction with the environment.
Revised: Consider the sequential decision process of an agent aiming to optimize its interaction with the environment.

Original: At each step t , the agent observes its state s t ∈ R d S , performs an action by sampling a policy a t ∼ µ ( a | s t ) ∈ R d A , and transitions to a new state s t +1 ∼ D ( s | a t , s t ) with reward r t +1 ∈ R .
Revised: At each step t , the agent observes its state s t ∈ R d S , performs an action by sampling a policy a t ∼ µ ( a | s t ) ∈ R d A , and transitions to a new state s t +1 ∼ D ( s | a t , s t ) with reward r t +1 ∈ R .

Original: These interactions { s t , a t , r t , µ t } are stored in a RM, which constitutes the data used by off-policy RL to train the parametric policy π w ( a | s ) .
Revised: These interactions { s t , a t , r t , µ t } are stored in a RM, which constitutes the data used by off-policy RL to train the parametric policy π w ( a | s ) .

Original: The importance weight ρ t = π w ( a t | s t ) /µ
Revised: The importance weight ρ t = π w ( a t | s t ) /µ

Original: t is the ratio between the probability of selecting a t with the current π w and with the behavior µ t , which gradually becomes dissimilar from π w as the latter is trained.
Revised: t is the ratio between the probability of selecting a t with the current π w and with the behavior µ t , which gradually becomes dissimilar from π w as the latter is trained.

Original: The on-policy state-action value Q π ( s, a ) measures the expected returns from ( s, a ) following the policy π w :
Revised: The on-policy state-action value Q π ( s, a ) measures the expected returns from ( s, a ) following the policy π w :

Original: [Equation]
Revised: [Equation]

Original: Here γ is a discount factor.
Revised: Here γ is a discount factor.

Original: The value of state s is the on-policy expectation: V π ( s ) =
Revised: The value of state s is the on-policy expectation: V π ( s ) =

Original: E a ∼ π [ Q π ( s, a )] and the action advantage is A π ( s, a )
Revised: E a ∼ π [ Q π ( s, a )] and the action advantage is A π ( s, a )

Original: =
Revised: =

Revised: [Equation]

Revised: π

Revised: [Equation]

Revised: s,

Original: Q π ( s, a ) − V π ( s ) , such that E a ∼ π [ A π ( s, a )] :=0 .
Revised: [Equation] ) − [Equation] π ( s ) , such that E a ∼ π [ A π ( s, a )] :=0 .

Original: We consider algorithms that train parametric approximators Q w from off-policy data.
Revised: We consider algorithms that train parametric approximators Q w from off-policy data.

Original: The Q-learning target is ˆ q t =
Revised: The Q-learning target is ˆ q t =

Original: r t +1 + γ E a (cid:48) ∼ π Q w ( s t +1 , a (cid:48) ) .
Revised: r t +1 + γ E a [Equation] ∼ π Q w ( s t +1 , a (cid:48) ) .

Original: The Retrace algorithm (Munos et al., 2016) includes all the rewards obtained by the behavior µ t in the value estimation:
Revised: The Retrace algorithm (Munos et al., 2016) includes all the rewards obtained by the behavior µ t in the value estimation:

Original: [Equation]
Revised: [Equation]

Original: The off-PG (Degris et al., 2012) can be used to update π w by ER: ˆ g off-PG t
Revised: The off-PG (Degris et al., 2012) can be used to update π w by ER: ˆ g off-PG t

Original: [Equation]
Revised: [Equation]

Original: Here, ˆ A πt is an estimator for the on-policy advantage, such as ˆ A ret t :=
Revised: Here, ˆ A πt is an estimator for the on-policy advantage, such as ˆ A ret t :=

Original: ˆ Q ret t − V w ( s t ) .
Revised: ˆ Q ret t − V w ( s t ) .

Original:  R EMEMBER AND F ORGET E XPERIENCE R EPLAY
Revised:  R EMEMBER AND F ORGET E XPERIENCE R EPLAY

Original: In off-policy RL it is common to maximize on-policy returns averaged over the distribution of states in a RM (Degris et al., 2012).
Revised: In off-policy RL it is common to maximize on-policy returns averaged over the distribution of states in a RM (Degris et al., 2012).

Original: However, as π w gradually shifts away from previous behaviors µ t , the distribution of states in the RM is increasingly dissimilar from the on-policy distribution, and trying to increase an off-policy performance metric may not improve on-policy outcomes.
Revised: However, as π w gradually shifts away from previous behaviors µ t , the distribution of states in the RM is increasingly dissimilar from the on-policy distribution, and trying to increase an off-policy performance metric may not improve on-policy outcomes.

Original: This issue may be compounded with algorithm-speciﬁc concerns.
Revised: This issue may be compounded with algorithm-speciﬁc concerns.

Original: For example, in ACER (Wang et al., 2017) the dissimilarity between µ t and π w may cause vanishing or diverging importance weights ρ t , thereby increasing the variance of the off-PG and deteriorating the convergence speed of Retrace by inducing “trace-cutting” (Munos et al., 2016).
Revised: For example, in ACER (Wang et al., 2017) the dissimilarity between µ t and π w may cause vanishing or diverging importance weights ρ t , thereby increasing the variance of the off-PG and deteriorating the convergence speed of Retrace by inducing “trace-cutting” (Munos et al., 2016).

Original: As a remedy, ACER tunes the learning rate and uses a target network (Mnih et al., 2015), updated as a delayed copy of the policy network, to constrain policy updates.
Revised: As a remedy, ACER tunes the learning rate and uses a target network (Mnih et al., 2015), updated as a delayed copy of the policy network, to constrain policy updates.

Original: Target networks are also employed in DDPG (Lillicrap et al., 2016) in order to slow down the feedback loop between value-network and policy-network optimizations.
Revised: Target networks are also employed in DDPG (Lillicrap et al., 2016) in order to slow down the feedback loop between value-network and policy-network optimizations.

Original: This feedback loop causes overestimated action values that can only be corrected by acquiring new on-policy samples.
Revised: This feedback loop causes overestimated action values that can only be corrected by acquiring new on-policy samples.

Original: When using non-linear networks there is no guarantee that small weight changes correspond to small changes in the output and recent works (Henderson et al., 2017) have shown the often opaque variability of outcomes of continuous-action RL algorithms depending on hyper-parameters.
Revised: When using non-linear networks there is no guarantee that small weight changes correspond to small changes in the output and recent works (Henderson et al., 2017) have shown the often opaque variability of outcomes of continuous-action RL algorithms depending on hyper-parameters.

Original: Here we propose a set of three simple rules, collectively referred to as Remember and Forget ER ( ReF-ER ), to directly control the degree of “off-policyness” of the samples in the RM: 
Revised: Here we propose a set of three simple rules, collectively referred to as Remember and Forget ER ( ReF-ER ), to directly control the degree of “off-policyness” of the samples in the RM: 1.

Original: Updates are computed from mini-batches drawn uniformly from the RM.
Revised: Updates are computed from mini-batches drawn uniformly from the RM.

Original: We compute the importance weight ρ t of each sample and classify it as “near-policy" if 1 /c max <ρ t <c max with c max > 1 .
Revised: We compute the importance weight ρ t of each sample and classify it as “near-policy" if 1 /c max <ρ t <c max with c max > 1 .

Original: The samples with vanishing ( ρ t < 1 /c max ) or exploding ( ρ t >c max ) importance weights are classiﬁed as “far-policy".When computing
Revised: The samples with vanishing ( ρ t < 1 /c max ) or exploding ( ρ t >c max ) importance weights are classiﬁed as “far-policy".

Original: off -policy estimators with ﬁnite batch-sizes, such as ˆ
Revised: When computing off-policy estimators with ﬁnite batch-sizes, such as ˆ

Original: Q ret or the off-PG, “far-policy" samples may either be irrelevant or increase the variance.
Revised: Q ret or the off-PG, “far-policy" samples may either be irrelevant or increase the variance.

Original: Therefore, ReF-ER allows computing updates exclusively from “near -policy" samples.
Revised: A RL method employing ReF-ER computes updates exclusively from “near-policy" samples.

Original: In order to efﬁciently approximate the number of far-policy samples in the RM, we store for each step its most recent ρ t .2.
Revised: In order to efﬁciently approximate the number of far-policy samples in the RM, we store for each step its most recent ρ t .

Original: When acquiring samples from the environment, older episodes with the highest fraction of far-policy samples are removed until the number n obs of samples in the RM is at most N .3.
Revised: When acquiring samples from the environment, older episodes with the highest fraction of far-policy samples are removed until the number n obs of samples in the RM is at most N .

Original: Policy updates are penalized in order to attract the current policy π w towards µ t :
Revised: Policy updates are penalized in order to attract the current policy π w towards µ t :

Original: [Equation]
Revised: [Equation]

Original: otherwise [Equation] Here D KL is the Kullback–Leibler divergence and the coefﬁcient β ∈ [0 , 1] is updated after each gradient step such that a ﬁxed fraction D ∈ (0 , 1) of the RM are far-policy samples:
Revised: Here D KL is the Kullback–Leibler divergence and the coefﬁcient β ∈ [0 , 1] is updated after each gradient step such that a ﬁxed fraction D ∈ (0 , 1) of the RM are far-policy samples:

Original: [Equation]
Revised: [Equation]

Original: where η is the learning rate and n far is the number of far-policy samples.
Revised: where η is the learning rate and n far is the number of far-policy samples.

Original: Note that iteratively updating β with Eq.
Revised: Note that iteratively updating β with Eq.

Original: 4 has ﬁxed points in 0 for n far /n obs >D and in 1 otherwise.
Revised: 4 has ﬁxed points in 0 for n far /n obs >D and in 1 otherwise.

Original: For c max → 1 and D → 0 , ReF-ER becomes asymptotically equivalent to computing gradient estimates from on-policy experiences.
Revised: For c max → 1 and D → 0 , ReF-ER becomes asymptotically equivalent to computing gradient estimates from on-policy experiences.

Original: ReF-ER reduces the sensitivity on the network architecture and training hyper-parameters, because the rate at which the policy is allowed to change is directly linked to its similarity with the behaviors in the training set.
Revised: ReF-ER reduces the sensitivity on the network architecture and training hyper-parameters, because the rate at which the policy is allowed to change is directly linked to its similarity with the behaviors in the training set.

Original:  RACER : R EGULARIZED A CTOR -C RITIC WITH E XPERIENCE R EPLAY
Revised:  RACER : R EGULARIZED A CTOR -C RITIC WITH E XPERIENCE R EPLAY

Original: This work analyzesthe effect of ReF-ER on the three main classes of continuous-actions off-policy  RL algorithms: methods based on the DPG (ie. DDPG ), based on Q-learning (ie. NAF ), and with off-PG. Both ACER (Wang et al., 2017) and IMPALA (Espeholt et al., 2018) train their policy-network with the off-PG.
Revised: This work analyzes ReF-ER with three types of continuous-actions off-policy deep RL algorithms: methods based on the DPG ( i.e. DDPG ), based on Q-learning ( i.e. NAF ), and based on the off-PG (e.g. ACER (Wang et al., 2017) and IMPALA (Espeholt et al., 2018)).

Original: In order to compare each class of methods on equal footing, here we present RACER , an off-PG based architecturethat shares similarities with  ACER and IMPALA and adds some beneﬁts.
Revised: Here we introduce RACER , an off-PG based architecture with some improvements over ACER and IMPALA which aid our analysis of ReF-ER .

Original: First, it does not require expensive network architectures (easing reproducibility and exploration of the hyper-parameters).
Revised: First, RACER does not require expensive network architectures (easing reproducibility and exploration of the hyper-parameters).

Original: Second, like DDPG and NAF, and unlike ACER and IMPALA , it samples time steps rather than episodes, further reducing its cost.
Revised: Second, it samples time steps rather than episodes (like DDPG and NAF and unlike ACER and IMPALA ), further reducing its cost.

Original: Third, it does not introduce any heuristic remedy
Revised: Third, it does not introduce any technique that would interfere with ReF-ER and affect its analysis. Speciﬁcally, ACER uses

Original: [Figure]
Revised: [Figure]

Original: to deal with unbounded ρ which would interfere with ReF-ER and affect its analysis ( ACER uses the variance truncation and bias correction trick  and modiﬁes Retrace to use  ρ1 /d A instead of ρ ).
Revised: ﬂow of the error signals. the variance truncation and bias correction trick (TBC), employs a target network to bound policy updates with a trust-region scheme, and modiﬁes Retrace to use dA √ ρ instead of ρ . Lacking these techniques, we expect RACER to require ReF-ER to deal with unbounded importance weights.

Original: RACER ’s network architecture consists of a single Multilayer Perceptron (MLP) with weights w which receives the state s and outputs as shown in Fig.
Revised: RACER ’s network architecture consists of a single Multilayer Perceptron (MLP) with weights w which receives the state s and outputs as shown in Fig.

Original: 1b.
Revised: 1b.

Original: From the ﬁnal layer, the ﬁrst 2 d A outputs are taken as the mean m and diagonal covariance Σ of the Gaussian policy π w ( a | s ) .
Revised: From the ﬁnal layer, the ﬁrst 2 d A outputs are taken as the mean m and diagonal covariance Σ of the Gaussian policy π w ( a | s ) .

Original: One output estimates the state value V w , and the remaining outputs are the parameters of a closed-form approximator for the action advantage A w .
Revised: One output estimates the state value V w , and the remaining outputs are the parameters of a closed-form approximator for the action advantage A w .

Original: RACER can be extended for problems where the state is received as a visual feed (by inserting convolutional layers), or where the state is only partially observable (by substituting the MLP with a recurrent model).
Revised: RACER can be extended for problems where the state is received as a visual feed (by inserting convolutional layers), or where the state is only partially observable (by substituting the MLP with a recurrent model).

Original: However, these extensions are outside the scope of this paper.
Revised: However, these extensions are outside the scope of this paper.

Original: The parameters w are updated from mini-batches of off-policy time steps with ReF-ER .
Revised: The parameters w are updated from mini-batches of off-policy time steps with ReF-ER .

Original: A separate gradient is deﬁned for each component of the actor-critic framework, as sketched in ﬁgure 1b.
Revised: A separate gradient is deﬁned for each component of the actor-critic framework, as sketched in ﬁgure 1b.

Original: The policy statistics m ( s ) and Σ ( s ) are updated with the off-PG (Degris et al., 2012):
Revised: The policy statistics m ( s ) and Σ ( s ) are updated with the off-PG (Degris et al., 2012):

Original: [Equation]
Revised: [Equation]

Original: Weuse Retrace (Eq. 2) to estimate the on-policy advantage.
Revised: We estimate the on-policy advantage with Retrace.

Original: From the “variance truncation and bias correction (TBC) method (Wang et al., 2017) and Retrace, we obtain an estimator for the on-policy state value ˆ V tbc t
Revised: From TBC and Retrace, we obtain an estimator for the on-policy state value ˆ V tbc t

Original: [Equation]
Revised: = V w ( s t )

Original: ) , used as a target for V w :
Revised: + min { 1 , ρ t } ( ˆ Q ret t − Q w ( s t , a t )) , used as a target for V w :

Original: [Equation]
Revised: [Equation]

Original: Rather than having a separate MLP with inputs ( s, a ) to parameterize Q w (as in ACER or DDPG ), whose expected value under the policy would be computationally demanding to compute, we employ a closed-form equation for A w inspired by NAF (Gu et al., 2016).
Revised: Rather than having a separate MLP with inputs ( s, a ) to parameterize Q w (as in ACER or DDPG ), whose expected value under the policy would be computationally demanding to compute, we employ a closed-form equation for A w inspired by NAF (Gu et al., 2016).

Original: The network outputs the coefﬁcients of a concave function f w ( s, a ) which is chosen such that its maximum coincides with the mean of the policy m ( s ) , and such that it is possible to derive analytical expectations for a ∼ π w .
Revised: The network outputs the coefﬁcients of a concave function f w ( s, a ) which is chosen such that its maximum coincides with the mean of the policy m ( s ) , and such that it is possible to derive analytical expectations for a ∼ π w .

Original: In App.
Revised: In App.

Original: B we explore multiple choices of f w ( s, a ) .
Revised: B we explore multiple choices of f w ( s, a ) .

Original: For all results in the main text of the manuscript we used:
Revised: For all results in the main text of the manuscript we used:

Original: [Equation]
Revised: [Equation]

Original: Here a − = min [ a − m ( s ) , 0 ] and a + = max [ a − m ( s ) , 0 ] (both element-wise operations).
Revised: Here a − = min [ a − m ( s ) , 0 ] and a + = max [ a − m ( s ) , 0 ] (both element-wise operations).

Original: This parameterization requires one MLP output for K ( s ) and d A outputs for each diagonal matrix L + and
Revised: This parameterization requires one MLP output for K ( s ) and d A outputs for each diagonal matrix L + and

Original: L − .
Revised: L − .

Original: In total, given a state s , the MLP computes m , Σ , V , K , L + and L − .
Revised: In total, given a state s , the MLP computes m , Σ , V , K , L + and L − .

Original: From Eq.7, for any action a the advantage is uniquely deﬁned: A w ( s, a ):= f w ( s, a ) − E a
Revised: From Eq.7, for any action a the advantage is uniquely deﬁned: A w ( s, a ):= f w ( s, a ) − E a

Original: [Equation]
Revised: [Equation]

Original: ∼ π [ f w ( s, a (cid:48) )] .
Revised: ∼ π [ f w ( s, a (cid:48) )] .

Original: Like the exact on-policy advantage A π , A w has by design expectation zero under the policy.
Revised: Like the exact on-policy advantage A π , A w has by design expectation zero under the policy.

Original: The parameterization coefﬁcients K and L are updated to minimize the L2 error from ˆ A ret t :
Revised: The parameterization coefﬁcients K and L are updated to minimize the L2 error from ˆ A ret t :

Original: [Equation]
Revised: [Equation]

Original: Here, ρ t reduces the weight of estimation errors for unlikely actions, where A w is expected to be less accurate.
Revised: Here, ρ t reduces the weight of estimation errors for unlikely actions, where A w is expected to be less accurate.

Original: To ensure that Σ , K , L + and L − are positive deﬁnite, the respective outputs are mapped onto R + by a Softplus rectiﬁer.
Revised: To ensure that Σ , K , L + and L − are positive deﬁnite, the respective outputs are mapped onto R + by a Softplus rectiﬁer.

Original: The analytical derivation of E a (cid:48) ∼ π
Revised: The analytical derivation of E a (cid:48) ∼ π

Original: [ f w ( s, a (cid:48) )] can be found in App. A.
Revised: [ f w ( s, a (cid:48) )] can be found in App. A.

Original: Collectively, ˆ g πt ( w ) , ˆ g V t ( w ) , and ˆ g
Revised: Collectively, ˆ g πt ( w ) , ˆ g Vt ( w ) , and ˆ g

Original: At ( w ) form a vector ˆ g AC t ( w ) with the same size as the MLP output.
Revised: At ( w ) form a vector ˆ g AC t ( w ) with the same size as the MLP output.

Original: ˆ g AC t ( w ) is weighted with the ReF-ER penalization (Eq. 3) and then back-propagated to the MLP.
Revised: ˆ g AC t ( w ) is weighted with the ReF-ER penalization (Eq. 3) and then back-propagated to the MLP.

Original: In order to estimate ˆ Q ret t for a sampled time step t , Retrace (Eq. 2) requires V w and Q w for all future steps in sample t ’s episode.
Revised: In order to estimate ˆ Q ret t for a sampled time step t , Retrace (Eq. 2) requires V w and Q w for all future steps in sample t ’s episode.

Original: These are naturally computed when training from batches of episodes (as in ACER ) rather than time steps.
Revised: These are naturally computed when training from batches of episodes (as in ACER ) rather than time steps.

Original: However, the information contained in consecutive steps is highly correlated, worsening the quality of the gradient estimate, and episodes can be composed of thousands of time steps, increasing the computational cost.
Revised: However, the information contained in consecutive steps is highly correlated, worsening the quality of the gradient estimate, and episodes can be composed of thousands of time steps, increasing the computational cost.

Original: In order to efﬁciently train from uncorrelated time steps, RACER stores for each sample in the RM the most recently computed estimations of V w ( s t ) ,
Revised: In order to efﬁciently train from uncorrelated time steps, RACER stores for each sample in the RM the most recently computed estimations of V w ( s t ) ,

Original: A w ( s t , a t ) , ρ t and Q ret t .
Revised: A w ( s t , a t ) , ρ t and Q ret t .

Original: When a batch of time steps is sampled, the stored Q ret t are used to compute the gradient update.
Revised: When a batch of time steps is sampled, the stored Q ret t are used to compute the gradient update.

Original: At the same time, the stored V w ( s t ) , A w ( s t , a t ) and ρ t are updated with the current NN outputs and used to correct Q ret for all prior time-steps in the respective episodes with
Revised: At the same time, the stored V w ( s t ) , A w ( s t , a t ) and ρ t are updated with the current NN outputs and used to correct Q ret for all prior time-steps in the respective episodes with

Original: Eq.
Revised: Eq.

Original: A description of all remaining implementation details is provided in App.
Revised: A description of all remaining implementation details is provided in App.

Original: D.
Revised: D.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The rules that determine which samples are kept in the RM and how they are used for training can be designed to address several objectives.
Revised: The rules that determine which samples are kept in the RM and how they are used for training can be designed to address several objectives.

Original: For example, it may be necessary to deliberately design ER to prevent lifelong learning agents from forgetting previously mastered tasks (Isele & Cosgun, 2018).
Revised: For example, it may be necessary to deliberately design ER to prevent lifelong learning agents from forgetting previously mastered tasks (Isele & Cosgun, 2018).

Original: Prioritized Experience Replay (Schaul et al., 2015b)
Revised: Prioritized Experience Replay (Schaul et al., 2015b)

Original: ( PER ) improves the performance of DQN (Mnih et al., 2015) by biasing sampling in favor of experiences that cause large temporal-difference (TD) errors.
Revised: ( PER ) improves the performance of DQN (Mnih et al., 2015) by biasing sampling in favor of experiences that cause large temporal-difference (TD) errors.

Original: TD errors may signal rare events that would convey useful information to the learner, especially if the rewards are rare.
Revised: TD errors may signal rare events that would convey useful information to the learner, especially if the rewards are rare.

Original: Alternatively ER can be used to train transition models in planning-based RL (Pan et al., 2018), or to train off-policy learners on auxiliary tasks (Schaul et al., 2015a; Jaderberg et al., 2017) helping to shape the network features.
Revised: Alternatively ER can be used to train transition models in planning-based RL (Pan et al., 2018), or to train off-policy learners on auxiliary tasks (Schaul et al., 2015a; Jaderberg et al., 2017) helping to shape the network features.

Original: Finally, when rewards are extremely rare, exploration may be guided by training the agent to reach rarely-visited states (Andrychowicz et al., 2017).
Revised: Finally, when rewards are extremely rare, exploration may be guided by training the agent to reach rarely-visited states (Andrychowicz et al., 2017).

Original: de Bruin et al.
Revised: de Bruin et al.

Original: (2015) proposes a modiﬁcation to ER that increases the diversity of behaviors contained in the RM, which is the opposite of what ReF-ER achieves.
Revised: (2015) proposes a modiﬁcation to ER that increases the diversity of behaviors contained in the RM, which is the opposite of what ReF-ER achieves.

Original: The ideas proposed by de Bruin et al.
Revised: The ideas proposed by de Bruin et al.

Original: (2015) cannot readily be applied to complex tasks (they consider a low-dimensional problem which can be learned with the RM holding a handful of episodes) and the authors concede that their method is not suitable when the policy is advanced for many iterations.
Revised: (2015) cannot readily be applied to complex tasks (they consider a low-dimensional problem which can be learned with the RM holding a handful of episodes) and the authors concede that their method is not suitable when the policy is advanced for many iterations.

Original: For these reasons, we compare
Revised: For these reasons, we compare

Original: ReF-ER only to PER and conventional ER.
Revised: ReF-ER only to PER and conventional ER.

Original: We assume that if increasing the diversity of experiences in the RM were beneﬁcial to off-policy RL then either PER or ER would outperform ReF-ER .
Revised: We assume that if increasing the diversity of experiences in the RM were beneﬁcial to off-policy RL then either PER or ER would outperform ReF-ER .

Original: ReF-ER is inspired by the techniques developed for on-policy RL to bound policy updates of
Revised: ReF-ER is inspired by the techniques developed for on-policy RL to bound policy updates of

Original: PPO (Schulman et al., 2017).
Revised: PPO (Schulman et al., 2017).

Original: Rule 1 of ReF-ER is similar to the clipped objective function of PPO (gradients are zero if ρ lies outside of some range).
Revised: Rule 1 of ReF-ER is similar to the clipped objective function of PPO (gradients are zero if ρ lies outside of some range).

Original: However, Rule 1 is not affected by the sign of the advantage estimate and clips both policy and value gradients.
Revised: However, Rule 1 is not affected by the sign of the advantage estimate and clips both policy and value gradients.

Original: Like Rule 3, one variant of PPO penalizes D KL ( µ t || π w ) (also Schulman et al.
Revised: Like Rule 3, one variant of PPO penalizes D KL ( µ t || π w ) (also Schulman et al.

Original: (2015) and Wang et al.
Revised: (2015) and Wang et al.

Original: (2017) employ trust-region schemes in the on- and off-policy setting respectively).
Revised: (2017) employ trust-region schemes in the on- and off-policy setting respectively).

Original: While PPO picks one of the two techniques, in ReF-ER Rules 1 and 3 complement each other and can be applied to most off-policy RL methods with parametric policies.
Revised: While PPO picks one of the two techniques, in ReF-ER Rules 1 and 3 complement each other and can be applied to most off-policy RL methods with parametric policies.

Original: In the ER setting, samples remain in the RM over many iterations.
Revised: In the ER setting, samples remain in the RM over many iterations.

Original: If only
Revised: If only

Original: Rule 1 is included, any PG could push the sample’s ρ outside of the clipping range without means to recover, leading to zero-valued gradients.
Revised: Rule 1 is included, any PG could push the sample’s ρ outside of the clipping range without means to recover, leading to zero-valued gradients.

Original: Including only Rule 3 would not reduce the number of hyper-parameters (a target D KL would be needed), and would not prevent unbounded ρ .
Revised: Including only Rule 3 would not reduce the number of hyper-parameters (a target D KL would be needed), and would not prevent unbounded ρ .

Original: The ﬁrst method to combine ER and PG was the Off-Policy Actor Critic (Degris et al., 2012).
Revised: The ﬁrst method to combine ER and PG was the Off-Policy Actor Critic (Degris et al., 2012).

Original: Further contributions were introduced with ACER (Wang et al., 2017), such as extension to NN, estimation of on-policy returns with Retrace, and the "variance truncation and bias correction trick".
Revised: Further contributions were introduced with ACER (Wang et al., 2017), such as extension to NN, estimation of on-policy returns with Retrace, and the variance truncation and bias correction trick (TBC).

Original: In the continuous-action domain, the proposed RACER offers several advantages over ACER : (a) reduced complexity by employing a single NN and closed-form A w (continuous- ACER uses 9 NN evaluations per gradient).
Revised: In the continuous-action domain, the proposed RACER offers several advantages over ACER : (a) reduced complexity by employing a single NN and closed-form A w (continuous- ACER uses 9 NN evaluations per gradient).

Original: (b) relies on ReF-ER rather than constraining policy updates around a target network.
Revised: (b) relies on ReF-ER rather than constraining policy updates around a target network.

Original: (c) samples time steps rather than episodes (which may consist of thousands of steps).
Revised: (c) samples time steps rather than episodes (which may consist of thousands of steps).

Original: (d) uses the original Retrace estimator, which has better convergence guarantees than the value-estimators used by continuous- ACER .
Revised: (d) uses the original Retrace estimator, which has better convergence guarantees than the value-estimators used by continuous- ACER .

Original: Because of (a) and (c), RACER is two orders of magnitude faster than ACER .
Revised: Because of (a) and (c), RACER is two orders of magnitude faster than ACER .

Original:  R ESULTS
Revised:  R ESULTS

Original: In this section we couple ReF-ER , conventional ER and PER with one method from each of the three main classes of continuous-action RL algorithms: DDPG , NAF , and RACER .
Revised: In this section we couple ReF-ER , conventional ER and PER with one method from each of the three main classes of continuous-action RL algorithms: DDPG , NAF , and RACER .

Original: The performance of each combination of algorithms is measured on the MuJoCo (Todorov et al., 2012) tasks of OpenAI
Revised: The performance of each combination of algorithms is measured on the MuJoCo (Todorov et al., 2012) tasks of OpenAI

Original: Gym (Brockman et al., 2016) by plotting the mean cumulative reward R = (cid:80) t r t .
Revised: Gym (Brockman et al., 2016) by plotting the mean cumulative reward R = (cid:80) t r t .

Original: Each plot tracks the average R among all episodes entering the RM within intervals of 2 · 10 5 time steps and averaged again among ﬁve differently seeded training trials.
Revised: Each plot tracks the average R among all episodes entering the RM within intervals of 2 · 10 5 time steps and averaged again among ﬁve differently seeded training trials.

Original: In the appendix we include contours of the 20 th to 80 th percentiles of R obtained by DDPG and RACER on the OpenAI Gym and on the DeepMind
Revised: In the appendix we include contours of the 20 th to 80 th percentiles of R obtained by DDPG and RACER on the OpenAI Gym and on the DeepMind

Original: Control Suite (Tassa et al., 2018).
Revised: Control Suite (Tassa et al., 2018).

Original: The code to reproduce all present results is available on GitHub.
Revised: The code to reproduce all present results is available on GitHub.

Original: 6. R ESULTS FOR DDPG
Revised: 6. R ESULTS FOR DDPG

Original: DDPG (Lillicrap et al., 2016) trains two networks by ER.
Revised: DDPG (Lillicrap et al., 2016) trains two networks by ER.

Original: The value network outputs Q w  ( s, a ) and is trained to minimize the L2 distance from the Q-learning target (Sec. 2).
Revised: The value network outputs Q w (cid:48) ( s, a ) and is trained to minimize the L2 distance from the Q-learning target (Sec. 2).

Original: The policy network outputs a deterministic policy m w(cid:48) ( st +1 ) and is trained with the DPG (Silver et al., 2014):
Revised: The policy network outputs a deterministic policy m w ( s ) and is trained with the DPG (Silver et al., 2014):

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original:  Noise is added to the deterministic policy π w(cid:48) = m w(cid:48) + N ( 0 , σ 2 I ) for exploration.
Revised: [Figure] Noise is added to the deterministic policy π w = m w + N ( 0 , σ 2 I ) for exploration.

Original: We consider two[Figure] [Figure] variants of DDPG : one with m w(cid:48) bounded to the unit box [ − 1 , 1] d A (as in the original) and one without bounds (referred to as u-DDPG , implementation details can be found in App. D).
Revised: We consider two variants of DDPG : one with m w bounded to the unit box [ − 1 , 1] d A (as in the original) and one without bounds (referred to as u-DDPG , implementation details can be found in App. D).

Original: Bounding the policy may lead to lower returns in the OpenAI MuJoCo benchmarks, which are deﬁned for unbounded actions.
Revised: Bounding the policy may lead to lower returns in the OpenAI MuJoCo benchmarks, which are deﬁned for unbounded actions.

Original: We note that, without measures to constrain policy updates or without careful tuning of the hyper-parameters (we found the critic’s weight decay and temporally-correlated action noise to be necessary), u-DDPG is unstable.
Revised: We note that, without measures to constrain policy updates or without careful tuning of the hyper-parameters (we found the critic’s weight decay and temporally-correlated action noise to be necessary), u-DDPG is unstable.

Original: The returns for u-DDPG can fall to large negative values, especially in tasks that include a control-cost penalty in the reward such as Ant.
Revised: The returns for u-DDPG can fall to large negative values, especially in tasks that include a control-cost penalty in the reward such as Ant.

Original: This behavior is explained by the value network not having learned local maxima with respect to the action (Silver et al., 2014).
Revised: This behavior is explained by the value network not having learned local maxima with respect to the action (Silver et al., 2014).

Original: By replacing ER with ReF-ER we can stabilize u-DDPG and greatly improve its performance, especially for tasks with complex dynamics such as Humanoid or Ant.
Revised: By replacing ER with ReF-ER we can stabilize u-DDPG and greatly improve its performance, especially for tasks with complex dynamics such as Humanoid or Ant.

Original: The hyper-parameter c max determines how much the policy is allowed to change from the behaviors in the RM.
Revised: The hyper-parameter c max determines how much the policy is allowed to change from the behaviors in the RM.

Original: By annealing c max we can allow faster improvements at the beginning of training, when an inaccurate policy gradient might be sufﬁcient to estimate a good direction for the update.
Revised: By annealing c max we can allow faster improvements at the beginning of training, when an inaccurate policy gradient might be sufﬁcient to estimate a good direction for the update.

Original: Conversely, during the later stages of training, precise updates can be computed from almost on-policy samples.
Revised: Conversely, during the later stages of training, precise updates can be computed from almost on-policy samples.

Original: We anneal c max and the learning rate according to:
Revised: We anneal c max and the learning rate according to:

Original: [Equation]
Revised: [Equation]

Original: Here k is the gradient step index, η is the initial learning rate ( 10 − 4 and 10 − 5 for the value and policy networks respectively, as when using regular ER).
Revised: Here k is the gradient step index, η is the initial learning rate ( 10 − 4 and 10 − 5 for the value and policy networks respectively, as when using regular ER).

Original: We found that annealing η worsened the instability of u-DDPG with regular ER.
Revised: We found that annealing η worsened the instability of u-DDPG with regular ER.

Original: Lower values of C reduce the speed of policy improvements, but after 10 7 time steps C = 2 achieves the best performance in most tasks.
Revised: Lower values of C reduce the speed of policy improvements, but after 10 7 time steps C = 2 achieves the best performance in most tasks.

Original: In Fig.
Revised: In Fig.

Original: 3 we report for a subset of problems the average D KL ( µ t || π w ) .
Revised: 3 we report for a subset of problems the average D KL ( µ t || π w ) .

Original: D KL decreases for lower values of C and further decreases during training due to the annealing process.
Revised: D KL decreases for lower values of C and further [Figure]

Original: With regular ER, even after lowering η by one
Revised: a t i v e

Original: [Figure]
Revised: [Figure]

Revised: e w a r d decreases during training due to the annealing process.

Original: a t i v e r e w a r d order of magnitude from that of the original paper, the distance between π and µ may span the entire action space.
Revised: With regular ER, even after lowering η by one order of magnitude from that of the original paper, the distance between π and µ may span the entire action space.

Original: In fact, in most tasks shown in Fig.
Revised: In fact, in most tasks shown in Fig.

Original: 3 the D KL ( µ t || π w ) of DDPG is of similar order of magnitude as its maximum d A 2 (2 /σ ) 2 (for example, since σ = 0 .
Revised: 3 the D KL ( µ t || π w ) of DDPG is of similar order of magnitude as its maximum d A 2 (2 /σ ) 2 (for example, since σ = 0 .

Original: 2 , the maximum D KL is 850 for
Revised: 2 , the maximum D KL is 850 for

Original: Humanoid and 300 for Walker2d and it oscillates during training around 100 and 50 respectively).
Revised: Humanoid and 300 for Walker2d and it oscillates during training around 100 and 50 respectively).

Original: ReF-ER maintains a RM of mostly near-policy samples, providing the value-network with multiple examples of trajectories that are possible with the current policy.
Revised: ReF-ER maintains a RM of mostly near-policy samples, providing the value-network with multiple examples of trajectories that are possible with the current policy.

Original: This focuses the predictive capabilities of the value-network, enabling it to extrapolate the effect of a marginal change of action on the expected returns, and therefore increasing the accuracy of the DPG.
Revised: This focuses the predictive capabilities of the value-network, enabling it to extrapolate the effect of a marginal change of action on the expected returns, and therefore increasing the accuracy of the DPG.

Original: Any misstep of the DPG is weighted with a penalization term that attracts the policy towards past behaviors.
Revised: Any misstep of the DPG is weighted with a penalization term that attracts the policy towards past behaviors.

Original: This allows time for the learner to gather experiences with the new policy, improve the value-network, and correct the misstep.
Revised: This allows time for the learner to gather experiences with the new policy, improve the value-network, and correct the misstep.

Original: This reasoning is almost diametrically opposed to that behind PER .
Revised: This reasoning is almost diametrically opposed to that behind PER .

Original: In PER observations that are associated with larger TD errors are sampled more frequently.
Revised: In PER observations that are associated with larger TD errors are sampled more frequently.

Original: In the continuous-action setting, however, TD errors may be result from actions that are farther from the current policy.
Revised: In the continuous-action setting, however, TD errors may be result from actions that are farther from the current policy.

Original: Therefore, precisely estimating their value might not help the value network in yielding an accurate estimate of the DPG.
Revised: Therefore, precisely estimating their value might not help the value network in yielding an accurate estimate of the DPG.

Original: We obtained better results with the rank-based variant of PER , which has similar performance to that of DDPG with ER.
Revised: We obtained better results with the rank-based variant of PER , which has similar performance to that of DDPG with ER.

Original: The main beneﬁts over ReF-ER arise in tasks that require more exploration, such as Swimmer and HumanoidStandup.
Revised: The main beneﬁts over ReF-ER arise in tasks that require more exploration, such as Swimmer and HumanoidStandup.

Original: 6. R ESULTS FOR NAF
Revised: 6. R ESULTS FOR NAF

Original: Normalized Advantage Functions ( NAF ) is the state-of-the-art of Q-learning based algorithms for continuous-action problems.
Revised: Normalized Advantage Functions ( NAF ) is the state-of-the-art of Q-learning based algorithms for continuous-action problems.

Original: It employs a quadratic-form approximation of the advantage
Revised: It employs a quadratic-form approximation of the advantage

Original: A w , analogous to the one employed by RACER :
Revised: A w , analogous to the one employed by RACER :

Original: [Equation]
Revised: [Equation]

Original: NAF trains A w with the Q-learning target (Sec. 2) and uses the location m w of its maximum as the mean of the policy, with added Gaussian noise for exploration.
Revised: NAF trains A w with the Q-learning target (Sec. 2) and uses the location m w of its maximum as the mean of the policy, with added Gaussian noise for exploration.

Original: When the actual A π is not well approximated by a quadratic (e.g. when the return landscape is multi-modal), NAF may fail to choose good actions.
Revised: When the actual A π is not well approximated by a quadratic (e.g. when the return landscape is multi-modal), NAF may fail to choose good actions.

Original: In contrast, RACER updates π w with the off-PG, which is independent of the error in approximating A π ( s t , a t ) (but depends on the error at t +1 ).
Revised: In contrast, RACER updates π w with the off-PG, which is independent of the error in approximating A π ( s t , a t ) (but depends on the error at t +1 ).

Original: Figure 4 shows how NAF is affected by the choice of ER algorithm.
Revised: Figure 4 shows how NAF is affected by the choice of ER algorithm.

Original: While Q-learning based methods are thought to be less sensitive than PG-based methods to the dissimilarity between policy and stored behaviors owing to the bootstrapped Q-learning target, NAF beneﬁts from REF-ER .
Revised: While Q-learning based methods are thought to be less sensitive than PG-based methods to the dissimilarity between policy and stored behaviors owing to the bootstrapped Q-learning target, NAF beneﬁts from REF-ER .

Original: This is because A π is likely to be approximated well by a quadratic in a small neighborhood near its local maxima.
Revised: This is because A π is likely to be approximated well by a quadratic in a small neighborhood near its local maxima.

Original: ReF-ER constrains learning from actions within this neighborhood and prevents large TD errors from disrupting the locally-accurate approximation of A w .
Revised: ReF-ER constrains learning from actions within this neighborhood and prevents large TD errors from disrupting the locally-accurate approximation of A w .

Original: This intuition is supported by observing that rank-based PER (the better performing variant of PER also in this case), often worsens the performance of NAF .
Revised: This intuition is supported by observing that rank-based PER (the better performing variant of PER also in this case), often worsens the performance of NAF .

Original: PER aims at biasing sampling in favor of larger TD errors, which are more likely to be farther from m ( s ) c

Original:  u m u l a t
Revised: PER c u m u l a t

Original: i v e r e w a r d
Revised: i v e r e w a r d

Original: [Figure]
Revised: [Figure]

Original: c u m u l a t i v e r e w a r d
Revised: c u m u l a t i v e r e w a r d

Original: [Figure]
Revised: [Figure]

Original:  (note that f w NAF is unbounded), and their accurate prediction might not help the learner in ﬁne-tuning the policy by improving a local approximation of the advantage.
Revised: aims at biasing sampling in favor of larger TD errors, which are more likely to be farther from m ( s ) (note that f w NAF is unbounded), and their accurate prediction might not help the learner in ﬁne-tuning the policy by improving a local approximation of the advantage.

Original: 6. R ESULTS FOR RACER
Revised: 6. R ESULTS FOR RACER

Original: In this section we present state-of-the-art results in the OpenAI MuJoCo tasks obtained by coupling
Revised: In this section we present state-of-the-art results in the OpenAI MuJoCo tasks obtained by coupling

Original: RACER with ReF-ER .
Revised: RACER with ReF-ER .

Original: We compare RACER to ACER and PPO , an algorithm that owing to its simplicity and good performance on MuJoCo tasks is often used as baseline.
Revised: We compare RACER to ACER and PPO , an algorithm that owing to its simplicity and good performance on MuJoCo tasks is often used as baseline.

Original: Without ReF-ER , or the various remedies introduced by ACER , the unbounded importance weights ρ t cause off-policy PG estimates to diverge, disrupting any prior learning progress.
Revised: Without ReF-ER , or the various remedies introduced by ACER , the unbounded importance weights ρ t cause off-policy PG estimates to diverge, disrupting any prior learning progress.

Original: Therefore, we omit results from coupling
Revised: Therefore, we omit results from coupling

Original: RACER with ER or PER as it yields large negative R values that would not ﬁt the graphs.
Revised: RACER with ER or PER as it yields large negative R values that would not ﬁt the graphs.

Original: The two hyper-parameters that most strongly affect the performance of RACER are the RM size N and c max , annealed with Eq.
Revised: The two hyper-parameters that most strongly affect the performance of RACER are the RM size N and c max , annealed with Eq.

Original: 10 (for a discussion of the other parameters see App. B).
Revised: 10 (for a discussion of the other parameters see App. B).

Original: RACER uses the Retrace estimator Q ret which is expected to converge to the on-policy Q π (Munos et al., 2016).
Revised: RACER uses the Retrace estimator Q ret which is expected to converge to the on-policy Q π (Munos et al., 2016).

Original: Because π w is gradually changing during training, it is crucial to maximize the convergence speed of
Revised: Because π w is gradually changing during training, it is crucial to maximize the convergence speed of

Original: Retrace to obtain accurate estimates of the off-PG.
Revised: Retrace to obtain accurate estimates of the off-PG.

Original: Large values of c max increase the variance of the policy gradient and increase the amount of “trace-cutting” as the importance weights are allowed to diverge from 1.
Revised: Large values of c max increase the variance of the policy gradient and increase the amount of “trace-cutting” as the importance weights are allowed to diverge from 1.

Original: The cumulative rewards reported in Fig.
Revised: The cumulative rewards reported in Fig.

Original: 5 show that “stable” tasks, where the agent’s success is less predicated on avoiding mistakes that would cause it to trip, are more tolerant to high values of c max (e.g. HalfCheetah).
Revised: 5 show that “stable” tasks, where the agent’s success is less predicated on avoiding mistakes that would cause it to trip, are more tolerant to high values of c max (e.g. HalfCheetah).

Original: In most other tasks, especially those that require precise controls (e.g. Walker), the best results are obtained for values of c max that strike a balance between strict penalty terms, which slow down policy improvements, and high-variance gradient estimates.
Revised: In most other tasks, especially those that require precise controls (e.g. Walker), the best results are obtained for values of c max that strike a balance between strict penalty terms, which slow down policy improvements, and high-variance gradient estimates.

Original: The RM size N has a parallel effect.
Revised: The RM size N has a parallel effect.

Original: A small RM may not contain enough diversity of samples for the learner to accurately estimate the gradients.
Revised: A small RM may not contain enough diversity of samples for the learner to accurately estimate the gradients.

Original: Conversely, a large RM is composed of episodes obtained with increasingly older versions of π w .
Revised: Conversely, a large RM is composed of episodes obtained with increasingly older versions of π w .

Original: In this case, the penalty terms required to preserve a sufﬁcient fraction of near-policy samples may prevent the policy from improving.
Revised: In this case, the penalty terms required to preserve a sufﬁcient fraction of near-policy samples may prevent the policy from improving.

Original: A discussion of all the secondary hyper-parameters and the choice of function parameterizing A w can be found in App.
Revised: A discussion of all the secondary hyper-parameters and the choice of function parameterizing A w can be found in App.

Original: B.
Revised: B.

Original: For most combinations of hyper-parameters and tasks presented in this section, RACER outperforms the best result from DDPG (Sec. 6.1), PPO , and is competitive with the best results found in the published literature, which to our knowledge were achieved by the on-policy algorithms
Revised: For most combinations of hyper-parameters and tasks presented in this section, RACER outperforms the best result from DDPG (Sec. 6.1), PPO , and is competitive with the best results found in the published literature, which to our knowledge were achieved by the on-policy algorithms

Original: TRPO (Schulman et al., 2015) and Policy Search with Natural Gradient (Rajeswaran et al., 2017).
Revised: TRPO (Schulman et al., 2015) and Policy Search with Natural Gradient (Rajeswaran et al., 2017).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original:  We present  a novel Experience Replay algorithm ( ReF-ER ) that is shown toenhance the stability and performance of off-policy RL. ReF-ER regulates the pace at which the policy π w is allowed to deviate from previous behaviors µ , and how past experiences are used to update π w .
Revised: Many RL algorithms update a policy π w from past experiences collected with off-policy behaviors µ . We present evidence that off-policy continuous-action deep RL methods beneﬁt from actively maintaining similarity between policy and replay behaviors. We propose a novel ER algorithm ( ReF-ER ) that extends these beneﬁts to off-policy PG, deterministic PG and Q-learning methods.

Original: The relevance of past behaviors to the current policy is quantiﬁed by the  importance weight ρ = π  ( a | s ) /µ
Revised: ReF-ER characterizes past behaviors either as “near-policy" or “far-policy" by the deviation from one of the importance weight ρ = π w ( a | s ) /µ

Original: ( a | s ) .
Revised: ( a | s ) .

Original: We characterize samples as either “near-policy" or “far-policy" by the deviation of ρ from unity.

Original: The proposed ReF-ERalgorithm consists of: 1) Computing gradient estimates only from near-policy experiences.
Revised: ReF-ER consists of: 1) Computing gradients only from near-policy experiences.

Original: 2) Forgetting far-policy samples when new observations are obtained from the environment.
Revised: 2) Forgetting far-policy samples when new observations are obtained from the environment.

Original: 3) “Remembering” past policies througha penalty term that reduces the Kullback–Leibler divergence between π w and µ .
Revised: 3) Regulating the pace at which π w is allowed to deviate from µ through penalty terms that reduce D KL ( µ || π w ) .

Original: This allows time for the learner to gather experiences with the new policy, improve the value estimators, and increase the accuracy of the next steps.
Revised: This allows time for the learner to gather experiences with the new policy, improve the value estimators, and increase the accuracy of the next steps.

Original: We show that ReF-ERcan be applied to most off-policy RL algorithm with parametric policies by coupling it with off-policy PG, deterministic PG and Q-learning methods for continuous-action problems. The present results provide evidence that most off-policy RL methods may beneﬁt from being trained with a replay memory composed of near-policy experiences.
Revised: The extension of ReF-ER to discrete-action methods is subject of ongoing work.

Original: Finally, ReF-ER is combined with a novel offpolicy actor critic algorithm ( RACER ) with simpliﬁed closed-form parameterization of theon-policy advantage approximator.
Revised: ReF-ER is combined with a novel method based on the off-policy PG ( RACER ). RACER uses a closed-form parameterization of the advantage and it estimates on-policy returns with a bootstrapped formulation of Retrace.

Original: RACER emphasizes a simple architecture, computational efﬁciency and minimal hyper-parameters.
Revised: These innovations greatly increase the computational efﬁciency of RACER compared to similar methods.

Original: The application of ReF-ER and RACER to OpenAI
Revised: The application of ReF-ER and RACER to OpenAI

Original: Gym benchmarks produces state-of-the-art results while being robust to large variations in their hyper-parameters.
Revised: Gym benchmarks produces state-of-the-art results while being robust to large variations in their hyper-parameters.

Original: A E XPECTATIONS OF THE PARAMETERIZED ADVANTAGE FUNCTIONS
Revised: A E XPECTATIONS OF THE PARAMETERIZED ADVANTAGE FUNCTIONS

Original: We considered three parameterizations for the action advantage:
Revised: We considered three parameterizations for the action advantage:

Original: [Equation]
Revised: [Equation]

Original: We recall a − =
Revised: We recall a − =

Original: min [ a − m ( s ) , 0 ] and a + = max [ a − m ( s ) , 0 ] .
Revised: min [ a − m ( s ) , 0 ] and a + = max [ a − m ( s ) , 0 ] .

Original: The ﬁrst parameterization f w Q is identical to the one employed by NAF (Gu et al., 2016) and requires training ( d 2 A + d A ) / 2 coefﬁcients of the lower triangular matrix L Q .
Revised: The ﬁrst parameterization f w Q is identical to the one employed by NAF (Gu et al., 2016) and requires training ( d 2 A + d A ) / 2 coefﬁcients of the lower triangular matrix L Q .

Original: Its quadratic complexity makes the choice of f w Q unfavorable for high-dimensional action spaces (e.g. it requires 153 parameters for the 17-dimensional Humanoid tasks of OpenAI Gym, against the 35 of f w DG ).
Revised: Its quadratic complexity makes the choice of f w Q unfavorable for high-dimensional action spaces (e.g. it requires 153 parameters for the 17-dimensional Humanoid tasks of OpenAI Gym, against the 35 of f w DG ).

Original: In order to preserve bijection between L Q and L Q L (cid:124) Q , the diagonal terms are mapped to R + with a Softplus function.
Revised: In order to preserve bijection between L Q and L Q L (cid:124) Q , the diagonal terms are mapped to R + with a Softplus function.

Original: The expectation under a Gaussian policy can be computed as (Petersen et al., 2008):
Revised: The expectation under a Gaussian policy can be computed as (Petersen et al., 2008):

Original: [Equation]
Revised: [Equation]

Original: Here Tr denotes the trace of a matrix.
Revised: Here Tr denotes the trace of a matrix.

Original: The second parameterization requires training the 1 + d A coefﬁcients for K ( s ) and the diagonal matrix L ( s ) .
Revised: The second parameterization requires training the 1 + d A coefﬁcients for K ( s ) and the diagonal matrix L ( s ) .

Original: The expectation can be easily derived from the properties of products of Gaussian densities:
Revised: The expectation can be easily derived from the properties of products of Gaussian densities:

Original: [Equation]
Revised: [Equation]

Original: Here | · | denotes a determinant.
Revised: Here | · | denotes a determinant.

Original: To derive the expectation of the third parameterization, which was used for most results in the paper, we recall that the expectation of a product of independent variables is the product of the expectations.
Revised: To derive the expectation of the third parameterization, which was used for most results in the paper, we recall that the expectation of a product of independent variables is the product of the expectations.

Original: For one component i of the action vector:
Revised: For one component i of the action vector:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here we exploited the symmetry of the Gaussian policy around the mean.
Revised: Here we exploited the symmetry of the Gaussian policy around the mean.

Original: Since Σ , L + , and L − are all diagonal, we can compute the expectation:
Revised: Since Σ , L + , and L − are all diagonal, we can compute the expectation:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Finally, we note that all these parameterizations are differentiable.
Revised: Finally, we note that all these parameterizations are differentiable.

Original: B S ENSITIVITY TO HYPER - PARAMETERS
Revised: B S ENSITIVITY TO HYPER - PARAMETERS

Original: Figure 6 shows the robustness of RACER to various hyper-parameters and to the choice of the advantage parameterizations introduced in App.
Revised: Figure 6 shows the robustness of RACER to various hyper-parameters and to the choice of the advantage parameterizations introduced in App.

Original: A. Moreover, we can bypass the advantage approximator (i.e. A w :=0 ) and rely exclusively on the state value approximator.
Revised: A. Moreover, we can bypass the advantage approximator (i.e. A w :=0 ) and rely exclusively on the state value approximator.

Original: We recall the on-policy value
Revised: We recall the on-policy value

Original: [Figure]
Revised: [Figure]

Original: estimator obtained with the “variance truncation and bias correction trick” (Wang et al., 2017)
Revised: estimator obtained with the “variance truncation and bias correction trick” (Wang et al., 2017)

Original: (Sec. 4): ˆ
Revised: (Sec. 4): ˆ

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: From Eq.
Revised: From Eq.

Original: 2 and 

Revised: 2 and

Original: we obtain ˆ Q ret ( s t , a t ) =
Revised: we obtain ˆ Q ret ( s t , a t ) =

Original: r t +1 + γ ˆ V tbc ( s t +1 ) .
Revised: r t +1 + γ ˆ V tbc ( s t +1 ) .

Original: By neglecting A w , from this relation and Eq.
Revised: By neglecting A w , from this relation and Eq.

Original: 19, we obtain a recursive Retrace-based estimator for the on-policy state value that depends on V w alone:
Revised: 19, we obtain a recursive Retrace-based estimator for the on-policy state value that depends on V w alone:

Original: [Equation]
Revised: [Equation]

Original: This target is equivalent to the recently proposed V-trace estimator (Espeholt et al., 2018) when all importance weights are clipped at 1, which was empirically found by the authors to be the best-performing solution.
Revised: This target is equivalent to the recently proposed V-trace estimator (Espeholt et al., 2018) when all importance weights are clipped at 1, which was empirically found by the authors to be the best-performing solution.

Original: As expected, lacking a model for A w ( s, a ) , this architecture (denoted as V-RACER in the ﬁrst column of results of Fig. 6) yielded worse performance than the original
Revised: As expected, lacking a model for A w ( s, a ) , this architecture (denoted as V-RACER in the ﬁrst column of results of Fig. 6) yielded worse performance than the original

Original: RACER .
Revised: RACER .

Original: However the difference is often minor and the increased simplicity of this architecture might justify its adoption for some problems.
Revised: However the difference is often minor and the increased simplicity of this architecture might justify its adoption for some problems.

Original: The tolerance D for far-policy samples in the RM has a similar effect as c max : low values tend to delay learning while high values reduce the fraction of the RM that is used to compute updates and may decrease the accuracy of gradient estimates.
Revised: The tolerance D for far-policy samples in the RM has a similar effect as c max : low values tend to delay learning while high values reduce the fraction of the RM that is used to compute updates and may decrease the accuracy of gradient estimates.

Original: Increasing the number F of time steps per gradient step could either cause a rightward shift of the expected returns because the learner computes fewer updates for the same budget of observations, or it could aid the learner by providing more on-policy samples.
Revised: Increasing the number F of time steps per gradient step could either cause a rightward shift of the expected returns because the learner computes fewer updates for the same budget of observations, or it could aid the learner by providing more on-policy samples.

Original: The actual outcomes depend on the problem: tasks with simpler dynamics (e.g.
Revised: The actual outcomes depend on the problem: tasks with simpler dynamics (e.g.

Original: HalfCheetah) can be learned more quickly by performing more gradient steps, while problems with more complex dynamics (e.g. Ant, Humanoid) beneﬁt from more on-policy samples.
Revised: HalfCheetah) can be learned more quickly by performing more gradient steps, while problems with more complex dynamics (e.g. Ant, Humanoid) beneﬁt from more on-policy samples.

Original: The batch-size
Revised: The batch-size

Original: B and learning rate η have a minor effect on performance.
Revised: B and learning rate η have a minor effect on performance.

Original: C S TATE , ACTION AND REWARD PREPROCESSING
Revised: C S TATE , ACTION AND REWARD PREPROCESSING

Original: Several authors have employed state (Henderson et al., 2017) and reward (Duan et al., 2016)
Revised: Several authors have employed state (Henderson et al., 2017) and reward (Duan et al., 2016)

Original: (Gu et al., 2017) rescaling to improve the learning results.
Revised: (Gu et al., 2017) rescaling to improve the learning results.

Original: This technique has not systematically been studied but reﬂects a deeper challenge of RL.
Revised: This technique has not systematically been studied but reﬂects a deeper challenge of RL.

Original: The success of any RL algorithm is intrinsically linked to the design of the control problem.
Revised: The success of any RL algorithm is intrinsically linked to the design of the control problem.

Original: For example, the stability of DDPG is affected by the L2 weight decay of the value-network.
Revised: For example, the stability of DDPG is affected by the L2 weight decay of the value-network.

Original: Depending on the numerical values of the distribution of rewards provided by the environment and the choice of weight decay coefﬁcient, the L2 penalization can be either negligible or dominate the Bellman error.
Revised: Depending on the numerical values of the distribution of rewards provided by the environment and the choice of weight decay coefﬁcient, the L2 penalization can be either negligible or dominate the Bellman error.

Original: Similarly, the distribution of values describing the state variables can increase the challenge of learning by gradient descent.
Revised: Similarly, the distribution of values describing the state variables can increase the challenge of learning by gradient descent.

Original: We propose partially addressing these issues by rescaling both rewards and state vectors depending on the the experiences contained in the RM.
Revised: We propose partially addressing these issues by rescaling both rewards and state vectors depending on the the experiences contained in the RM.

Original: At the beginning of training we prepare the RM by collecting N start observations and then we compute:
Revised: At the beginning of training we prepare the RM by collecting N start observations and then we compute:

Original: [Equation]
Revised: [Equation]

Original: Throughout training, µ s and σ s are used to standardize all state vectors ˆ s t =
Revised: Throughout training, µ s and σ s are used to standardize all state vectors ˆ s t =

Original: ( s t − µ s ) / ( σ s + (cid:15) ) before feeding them to the network approximators.
Revised: ( s t − µ s ) / ( σ s + (cid:15) ) before feeding them to the network approximators.

Original: Moreover, every 1000 steps, chosen as the smallest power of ten that did not affect the run time, we loop over the n obs observations stored in the RM to compute:
Revised: Moreover, every 1000 steps, chosen as the smallest power of ten that did not affect the run time, we loop over the n obs observations stored in the RM to compute:

Original: [Equation]
Revised: [Equation]

Original: This value is used to scale the rewards ˆ r t =
Revised: This value is used to scale the rewards ˆ r t =

Original: r t / ( σ r + (cid:15) ) used by the Q-learning target of DDPG and the Retrace algorithm for RACER .
Revised: r t / ( σ r + (cid:15) ) used by the Q-learning target of DDPG and the Retrace algorithm for RACER .

Original: We use (cid:15) =
Revised: We use (cid:15) =

Original: 10 − 7 to ensure numerical stability.
Revised: 10 − 7 to ensure numerical stability.

Original: The actions sampled by the learner may need to be rescaled or bounded to some interval depending on the environment.
Revised: The actions sampled by the learner may need to be rescaled or bounded to some interval depending on the environment.

Original: For the OpenAI Gym tasks this amounts to a linear scaling a (cid:48)
Revised: For the OpenAI Gym tasks this amounts to a linear scaling a (cid:48)

Original: = a ( upper_value − lower_value ) / 2 , where the values speciﬁed by the Gym library are ± 0 .
Revised: = a ( upper_value − lower_value ) / 2 , where the values speciﬁed by the Gym library are ± 0 .

Original: 4 for the Humanoid tasks, ± 8 for the Pendulum tasks, and ± 1 for all others.
Revised: 4 for the Humanoid tasks, ± 8 for the Pendulum tasks, and ± 1 for all others.

Original: The tasks of
Revised: The tasks of

Original: DeepMind Control Suite are deﬁned with all actions bounded do the interval [ − 1 , 1] d A .
Revised: DeepMind Control Suite are deﬁned with all actions bounded do the interval [ − 1 , 1] d A .

Original: In this case we can learn Gaussian policies, as if the action space were unbounded, and then map the actions sent to the environment with an hyperbolic tangent: a (cid:48) t
Revised: In this case we can learn Gaussian policies, as if the action space were unbounded, and then map the actions sent to the environment with an hyperbolic tangent: a (cid:48) t

Original: = tanh a t .
Revised: = tanh a t .

Original: This approach, however, might prevent efﬁciently learning policies that behave like bang-bang controllers.
Revised: This approach, however, might prevent efﬁciently learning policies that behave like bang-bang controllers.

Original: Close to the bounds of the control space, actions that should be similar may be mapped to distant positions in the unbounded action space.
Revised: Close to the bounds of the control space, actions that should be similar may be mapped to distant positions in the unbounded action space.

Original: Therefore, learning bang-bang controls is more likely to be hindered by the D KL penalties of
Revised: Therefore, learning bang-bang controls is more likely to be hindered by the D KL penalties of

Original: ReF-ER .
Revised: ReF-ER .

Original: This issue may explain some poor results on the DeepMind Control Suite shown in Fig.and 10.
Revised: This issue may explain some poor results on the DeepMind Control Suite shown in Fig.and 10.

Original: These results show that successfully tackling control problems with bounded action spaces may require policies parameterized as Beta distribution (Chou et al., 2017).
Revised: These results show that successfully tackling control problems with bounded action spaces may require policies parameterized as Beta distribution (Chou et al., 2017).

Original: D I MPLEMENTATION AND NETWORK ARCHITECTURE DETAILS
Revised: D I MPLEMENTATION AND NETWORK ARCHITECTURE DETAILS

Original: We implemented all presented learning algorithms within smarties , 2 our open source C++ RL framework, and optimized for high CPU-level efﬁciency through ﬁne-grained multi-threading, strict control of cache-locality, and computation-communication overlap.
Revised: We implemented all presented learning algorithms within smarties , 2 our open source C++ RL framework, and optimized for high CPU-level efﬁciency through ﬁne-grained multi-threading, strict control of cache-locality, and computation-communication overlap.

Original: On every step, we asynchronously obtain on-policy data by sampling the environment with π , which advances the index t of observed time steps, and we compute updates by sampling from the Replay Memory (RM), which advances the index k of gradient steps (Fig. 1a).
Revised: On every step, we asynchronously obtain on-policy data by sampling the environment with π , which advances the index t of observed time steps, and we compute updates by sampling from the Replay Memory (RM), which advances the index k of gradient steps (Fig. 1a).

Original: During training, the ratio of time and update steps is usually equal to a constant: t/k = F .
Revised: During training, the ratio of time and update steps is usually equal to a constant: t/k = F .

Original: This parameter affects the data efﬁciency of the algorithm; by lowering F each sample is used more times to improve the policy before being replaced by newer samples.
Revised: This parameter affects the data efﬁciency of the algorithm; by lowering F each sample is used more times to improve the policy before being replaced by newer samples.

Original: Upon completion of all tasks, we apply the gradient update and proceed to the next step.
Revised: Upon completion of all tasks, we apply the gradient update and proceed to the next step.

Original: The pseudo-codes in App.
Revised: The pseudo-codes in App.

Original: E neglect parallelization details as they do not affect execution.
Revised: E neglect parallelization details as they do not affect execution.

Original: In order to evaluate all algorithms on equal footing, we use the same baseline network architecture for
Revised: In order to evaluate all algorithms on equal footing, we use the same baseline network architecture for

Original: RACER , DDPG and NAF , consisting of an MLP with two hidden layers of 128 units each.
Revised: RACER , DDPG and NAF , consisting of an MLP with two hidden layers of 128 units each.

Original: For the sake of computational efﬁciency, we employed Softsign activation functions.
Revised: For the sake of computational efﬁciency, we employed Softsign activation functions.

Original: The weights of the hidden layers are initialized according to U (cid:2) − 6 / √ f
Revised: The weights of the hidden layers are initialized according to U (cid:2) − 6 / √ f

Original: i + f o , 6 / √ f
Revised: i + f o , 6 / √ f

Original: i + f o (cid:3) , where f i and f o are respectively the layer’s fan-in and fan-out (Glorot & Bengio, 2010).
Revised: i + f o (cid:3) , where f i and f o are respectively the layer’s fan-in and fan-out (Glorot & Bengio, 2010).

Original: The weights of the linear output layer are initialized from the distribution U (cid:2) − 0 .
Revised: The weights of the linear output layer are initialized from the distribution U (cid:2) − 0 .

Original: 1 / √ f i , 0 . 1 / √ f i (cid:3) , such that the MLP has near-zero outputs at the beginning of training.
Revised: 1 / √ f i , 0 . 1 / √ f i (cid:3) , such that the MLP has near-zero outputs at the beginning of training.

Original: When sampling the components of the action vectors, the policies are
Revised: When sampling the components of the action vectors, the policies are

Original: [Table]
Revised: [Table]

Original: treated as truncated normal distributions with symmetric bounds at three standard deviations from the mean.
Revised: treated as truncated normal distributions with symmetric bounds at three standard deviations from the mean.

Original: Finally, we optimize the network weights with the Adam algorithm (Kingma & Ba, 2015).
Revised: Finally, we optimize the network weights with the Adam algorithm (Kingma & Ba, 2015).

Original: RACER We note that the values of the diagonal covariance matrix are shared among all states and initialized to Σ = 0 .
Revised: RACER We note that the values of the diagonal covariance matrix are shared among all states and initialized to Σ = 0 .

Original: 2 I .
Revised: 2 I .

Original: The remaining hyper-parameters of RACER are listed in table 1.
Revised: The remaining hyper-parameters of RACER are listed in table 1.

Original: DDPG In its original formulation, DDPG transforms the policy-network’s output onto the bounded interval [ − 1 , 1] d A with an hyperbolic tangent function.
Revised: DDPG In its original formulation, DDPG transforms the policy-network’s output onto the bounded interval [ − 1 , 1] d A with an hyperbolic tangent function.

Original: We found that bounding the action space may limit the performance of DDPG with the OpenAI MuJoCo tasks.
Revised: We found that bounding the action space may limit the performance of DDPG with the OpenAI MuJoCo tasks.

Original: In order to stabilize the unboundedaction version of DDPG ( u-DDPG ) we set the learning rate for the policy-network to 1 · 10 − 5 and that of the value-network to 1 · 10 − 4 with L2 weight decay coefﬁcient of 1 · 10 − 4 .
Revised: In order to stabilize the unboundedaction version of DDPG ( u-DDPG ) we set the learning rate for the policy-network to 1 · 10 − 5 and that of the value-network to 1 · 10 − 4 with L2 weight decay coefﬁcient of 1 · 10 − 4 .

Original: These changes lead to better performance also with the bounded-action DDPG and therefore were used for all numerical experiments.
Revised: These changes lead to better performance also with the bounded-action DDPG and therefore were used for all numerical experiments.

Original: The RM is set to contain N =2 19 observations and we follow Henderson et al.
Revised: The RM is set to contain N =2 19 observations and we follow Henderson et al.

Original: (2017) for the remaining hyper-parameters: mini-batches of B = 128 samples, γ =0 .
Revised: (2017) for the remaining hyper-parameters: mini-batches of B = 128 samples, γ =0 .

Original: 995 , soft target network update coefﬁcient 0 .
Revised: 995 , soft target network update coefﬁcient 0 .

Original: We note that while DDPG is the only algorithm employing two networks, choosing half the batch-size as RACER and NAF makes the compute cost roughly equal among the three methods.
Revised: We note that while DDPG is the only algorithm employing two networks, choosing half the batch-size as RACER and NAF makes the compute cost roughly equal among the three methods.

Original: Finally, when using ReF-ER we add exploratory Gaussian noise to the deterministic policy: π
Revised: Finally, when using ReF-ER we add exploratory Gaussian noise to the deterministic policy: π

Original: [Equation]
Revised: [Equation]

Original: + N ( 0 , σ 2 I ) with σ =0 .
Revised: + N ( 0 , σ 2 I ) with σ =0 .

Original: When performing regular ER or PER we sample the exploratory noise from an Ornstein–Uhlenbeck process with σ =0 .
Revised: When performing regular ER or PER we sample the exploratory noise from an Ornstein–Uhlenbeck process with σ =0 .

Original: 2 and θ =0 .
Revised: 2 and θ =0 .

Original: NAF We use the same baseline MLP architecture and learning rate η = 10 − 4 , batch-size B = 256 , discount γ = 0 .
Revised: NAF We use the same baseline MLP architecture and learning rate η = 10 − 4 , batch-size B = 256 , discount γ = 0 .

Original: 995 , RM size N = 2 19 , and soft target network update coefﬁcient 0 .
Revised: 995 , RM size N = 2 19 , and soft target network update coefﬁcient 0 .

Original: Gaussian noise is added to the deterministic policy π w (cid:48)
Revised: Gaussian noise is added to the deterministic policy π w (cid:48)

Original: = m w (cid:48)
Revised: = m w (cid:48)

Original: + N ( 0 , σ 2 I ) with σ =0 .
Revised: + N ( 0 , σ 2 I ) with σ =0 .

Original: PPO We use the same baseline architecture as Henderson et al.
Revised: PPO We use the same baseline architecture as Henderson et al.

Original: (2017): γ = 0 .
Revised: (2017): γ = 0 .

Original: 995 , GAE λ = 0 .
Revised: 995 , GAE λ = 0 .

Original: 97 , policy clipping at ∆ ρ t = 0 . 2 , and we alternate performing 2048 environment steps and 10 optimizer epochs with batch-size 64 on the obtained data.
Revised: 97 , policy clipping at ∆ ρ t = 0 . 2 , and we alternate performing 2048 environment steps and 10 optimizer epochs with batch-size 64 on the obtained data.

Original: Both the policy network and the value network are 2-layer MLPs with 64 units per layer.
Revised: Both the policy network and the value network are 2-layer MLPs with 64 units per layer.

Original: Finally, we achieved the best results by having separate learning rates ( 1 · 10 − 4 for the policy and 3 · 10 − 4 for the critic) with the same annealing as used in the other experiments.
Revised: Finally, we achieved the best results by having separate learning rates ( 1 · 10 − 4 for the policy and 3 · 10 − 4 for the critic) with the same annealing as used in the other experiments.

Original: ACER We kept most hyper-parameters as described in the original paper (Wang et al., 2017): the "variance truncation and bias correction trick" clipping parameter is set to c = 5 , the trust-region update parameter is δ = 1 , ﬁve samples of the A w network are used to compute the advantage estimate under π , and the soft target-network update coefﬁcient is 0 .
Revised: ACER We kept most hyper-parameters as described in the original paper (Wang et al., 2017): the "variance truncation and bias correction trick" clipping parameter is set to c = 5 , the trust-region update parameter is δ = 1 , ﬁve samples of the A w network are used to compute the advantage estimate under π , and the soft target-network update coefﬁcient is 0 .

Original: We use a RM of 1 e 5 samples,we perform one gradient step per environment step, and each gradient is computed from 24 uniformly sampled episodes.
Revised: We use a RM of 1 e 5 samples, each gradient is computed from 24 uniformly sampled episodes and we perform one gradient step per environment step.

Original:  Wang et al.
Revised: Conversely, Wang et al.

Original: (2017) performs one update every 50 steps, but that leads to a far worse sample-efﬁciency compared to the other two methods in Fig.
Revised: (2017) performs one update every 50 steps, but that leads to a far worse sample-efﬁciency compared to the other two methods in Fig.

Original: Each network (value, advantage, and policy) is an MLP with 2 layers and 64 units per layer (we tried shorter training runs with 96 units per layer and did not observe improvements).
Revised: Each network (value, advantage, and policy) is an MLP with 2 layers and 64 units per layer (we tried shorter training runs with 96 units per layer and did not observe improvements).

Original: The learning rate for the value networks is η = 3 · 10 −and for the policy network η = 1 · 10 − 4 .
Revised: The learning rate for the value networks is η = 3 · 10 − 4 and for the policy network η = 1 · 10 − 4 .

Original: E P SEUDO - CODES
Revised: E P SEUDO - CODES

Original: Remarks on algorithm 2: 1) The reward for an episode’s initial state, before having performed any action, is zero by deﬁnition.
Revised: Remarks on algorithm 1: 1) It describes the general structure of the ER-based off-policy RL algorithms implemented for this work (i.e. RACER , DDPG , and NAF ).

Original: The value V w ( s t ) for the last state of an episode is computed if the
Revised: This algorithm can be adapted to

Original: Algorithmmaster algorithm.
Revised: Algorithm algorithm.

Original: t = 0 , k = 0 ,
Revised: t = 0 , k = 0 ,

Original: Initialize an empty RM, network weights w , and Adam algorithm’s
Revised: Initialize an empty RM, network weights w , and Adam algorithm’s

Original: (Kingma & Ba, 2015) moments estimates.
Revised: (Kingma & Ba, 2015) moments estimates.

Original: while n obs < N start do
Revised: while n obs < N start do

Original: Advance the environment according algorithm 2.
Revised: Advance the environment according algorithm 2.

Original: end while
Revised: end while

Original: Compute the initial statistics used to standardize the state vectors (App. C).
Revised: Compute the initial statistics used to standardize the state vectors (App. C).

Original: Compute the initial statistics used to rescale the rewards (App. C).
Revised: Compute the initial statistics used to rescale the rewards (App. C).

Original: while t < T max do while t < F · k do
Revised: while t < T max do while t < F · k do

Original: Advance the environment according  algorithm 2.
Revised: Advance the environment according to algorithm 2.

Revised: while n obs > N start do

Revised: Remove an episode from the RM (either ﬁrst in ﬁrst out or as in Sec. 3).

Original:  t ← t +end while
Revised: end while t ← t +end while

Original: Compute a gradient estimate (for RACER with algorithm 3).
Revised: Compute a gradient estimate (e.g. for RACER with algorithm 3).

Original: Perform the gradient step with the Adam algorithm.
Revised: Perform the gradient step with the Adam algorithm.

Original: If applicable, update the ReF-ER penalization coefﬁcient β .
Revised: If applicable, update the ReF-ER penalization coefﬁcient β .

Original: if modulo ( k, 1000) is 0 then
Revised: if modulo ( k, 1000) is 0 then

Original: Update the statistics used to rescale the rewards (App. C).
Revised: Update the statistics used to rescale the rewards (App. C).

Original: end if k ← k +end while 
Revised: end if k ← k +end while conventional ER, PER (by modifying the sampling algorithm to compute the gradient estimates), or ReF-ER (by following Sec. 3)). The algorithm requires 3 hyper-parameters: the ratio of time step to gradient steps F (usually set to 1 as in DDPG), the maximal size of the RM N , and the minimal size of the RM before we begin gradient updates N start .

Original: Algorithm 
Revised: Algorithm 2 Environment sampling

Original: Observe s t and r t .
Revised: Observe s t and r t .

Original: if s t concludes an episode then
Revised: if s t concludes an episode then

Original: Store data for t into the RM: { s t , r t , V w ( s t ) }
Revised: Store data for t into the RM: { s t , r t , V w ( s t ) }

Original: Compute and store Q ret for all steps of the episode else
Revised: Compute and store Q ret for all steps of the episode else

Original: Sample the current policy a t ∼ π w ( a | s t ) = µ t
Revised: Sample the current policy a t ∼ π w ( a | s t ) = µ t

Original: Store data for t into the RM: { s t , r t , a t , µ t , V w ( s t ) , A w ( s t , a t ) }
Revised: Store data for t into the RM: { s t , r t , a t , µ t , V w ( s t ) , A w ( s t , a t ) }

Original: Advance the environment by performing a t end if
Revised: Advance the environment by performing a t end if

Original: [Figure]
Revised: Remarks on algorithm 2: 1) The reward for an episode’s initial state, before having performed any action, is zero by deﬁnition.

Original:  episode has been truncated due the task’s time limits or is set to zero if s t is a terminal state.
Revised: The value V w ( s t ) for the last state of an episode is computed if the episode has been truncated due the task’s time limits or is set to zero if s t is a terminal state.

Original: Each time step we use the learner’s updated policy network and we store µ t = { m ( s t ) , Σ ( s t ) } .
Revised: Each time step we use the learner’s updated policy network and we store µ t = { m ( s t ) , Σ ( s t ) } .

Revised: Remarks on algorithm 3: 1)

Revised: In order to compute the gradients ˆ g ACt i ( w ) , we rely on advantage estimates

Original: Sample mini-batch of B time steps from RM.
Revised: Q ret t i that were computed when subsequent time steps in t i ’s episode were previously drawn by ER. Not having to compute the quantities A w , V w , and ρ for all following steps comes with clear computational efﬁciency beneﬁts, at the risk of employing an incorrect estimate for Q ret t i . In practice, we ﬁnd that the Retrace values incur only minor changes between updates (even when large RM sizes decrease the frequency of updates to the Retrace estimator) and that relying on previous estimates has no evident effect on performance.

Revised: This could be attributed to the gradual policy changes enforced by ReF-ER .

Revised: 2) With a little abuse of the notation, with π (or µ ) we denote the statistics (mean, covariance) of the multivariate normal policy, with π ( a | s ) we denote the probability of performing action a given state s , and with π ( ·| s ) we denote the probability density function over actions given state s .

Revised: Algorithm

Revised: Sample mini-batch of B time steps from the RM.

Original: for i = 0 to B do
Revised: for i = 0 to B do

Original: Fetch all relevant information: s t
Revised: Fetch all relevant information: s t

Original: i , a t i , Q ret t i , and µ t i = { m t i , Σ t i } .
Revised: i , a t i , Q ret t i , and µ t i = { m t i , Σ t i } .

Original: Call the approximator to compute π w , V w ( s t i ) , A w ( s t
Revised: Call the approximator to compute π w , V w ( s t i ) , A w ( s t

Original: i , a t i )
Revised: i , a t i )

Original: Update Q ret for all prior steps in t i ’s episode with V w ( s t i ) , A w ( s t
Revised: Update Q ret for all prior steps in t i ’s episode with V w ( s t i ) , A w ( s t

Original: i , a t i )
Revised: i , a t i )

Original: Update the importance weight ρ t i = π w ( a t i | s t i )
Revised: Update the importance weight ρ t i = π w ( a t i | s t i )

Original: /µ
Revised: /µ

Original: t i ( a t i | s t i ) if 1 /c max < ρ t i <
Revised: t i ( a t i | s t i ) if 1 /c max < ρ t i <

Original: c max then Compute ˆ g ACt i ( w ) according to Sec.else ˆ g ACt i ( w )
Revised: c max then Compute ˆ g ACt i ( w ) according to Sec.else ˆ g ACt i ( w )

Original: =end if
Revised: =end if

Original: ReF-ER penalization: ˆ g ReF-ER t i ( 
Revised: ReF-ER penalization: ˆ g ReF-ER t i ( w ) =

Original: [Equation]
Revised: β ˆ g ACt i w ) − (1 − β ) ∇ D KL [ µ t i ( ·| s t i )

Original: [Equation] end for
Revised: || π w ( ·| s t i )] end for

Original: Accumulate the gradients (cid:80) Ti =
 
Revised: Compute the gradient estimate over the mini-batch 1 B (cid:80) Bi =
 ˆ g ReF-ER t i ( w )

Revised: Algorithm 4 DDPG ’s gradient update with ReF-ER

Original: ˆ
Revised: Update the target policy- ( ˜ w ← (1 − α ) ˜ w + α w ) and target value-networks ( ˜ w (cid:48) ← (1 − α )

Original: [Equation]
Revised: ˜ w (cid:48) + α w (cid:48) ). Sample mini-batch of B time steps from the RM. for i = 0 to B do

Revised: Fetch all relevant information: s t

Original: ReF
Revised: i , a t i , and µ t i = { m t i , Σ t i } . The policy-network computes m w ( s t i ) and the value-network computes Q w (cid:48) ( s t i , a t i ) . Deﬁne a stochastic policy with Gaussian exploration noise: π w ( a | s t i ) = m w ( s t i ) + N Update the importance weight ρ t i =

Original: [Equation]
Revised: [Equation]

Original: ER t i ( w )
Revised: w [Equation] i

Original: Remarks on algorithm 3: 1)
Revised: [Equation] s t [Equation] ) /µ

Original: In order to compute the gradients ˆ g ACt i ( w ) , we rely on advantage estimates
Revised: t i ( [Equation] i [Equation]

Original: Q ret t i that were computed when subsequent time steps in t i ’s episode were previously drawn by ER.
Revised: t i ) if 1 /c max < ρ t i < c max then Compute the policy at t i +1 with the target-network: m ˜ w ( s t i +1 )

Original: Not having to compute the quantities A w , V w , and ρ for all following steps comes with clear computational efﬁciency beneﬁts, at the risk of employing an incorrect estimate for Q ret t i .
Revised: Compute the Q-learning target: ˆ q t i = r t i +1 + γQ ˜ w (cid:48) (cid:0) s t

Original: In practice, we ﬁnd that the
Revised: i +1 , m ˜ w ( s t i +1 ) (cid:1)

Original: Retrace values incur only minor changes between updates (even when large RM sizes decrease the frequency ofupdates to the Retrace estimator) and that relying on previous estimates has no evident effect on performance.
Revised: The gradient g Qt i ( w (cid:48) ) of the value-network minimizes the squared distance from ˆ q t i . The gradient g DPG t i ( w ) of the policy-network is the deterministic PG (Eq. 9). else [Equation]

Original: This could be attributed to the gradual policy changes enforced by ReF-ER .
Revised: end if ReF-ER penalization: ˆ g ReF-ER [Equation] end for

Original: 2) With a little abuse of the notation, with π (or µ ) we denote the statistics (mean, covariance) of the multivariate normal policy, with π ( a | s ) we denote the probability of performing action a given state s , and with π ( ·| s ) we denote the probability density function over actions given state s .
Revised: Compute the gradient estimates over the mini-batch for both networks. Remarks on algorithm 4: 1) It assumes that weights and Adam are initialized for both policy-network and value-network. The “target” weights are initialized as identical to the “trained” weights. 3) For the sake of brevity, we omit the algorithm for NAF , whose structure would be very similar to this one.

Original: [Figure]
Revised: The key difference is that NAF employs only one network and all the gradients are computed from the Q-learning target.