Original: A N E MPIRICAL S TUDY OF N ON -U NIFORM S AMPLING IN O FF -P OLICY R EINFORCEMENT L EARNING FOR C ONTINUOUS C ONTROL Anonymous authors Paper under double-blind review
Revised: An Empirical Study of Non-Uniform Sampling in Off-Policy Reinforcement Learning for Continuous Control

Original: A BSTRACT
Revised: Abstract

Original: Off-policy reinforcement learning (RL) algorithms can take advantage of samples generated from all previous interactions with the environment through ”experience replay”.
Revised: Off-policy reinforcement learning (RL) algorithms can take advantage of samples generated from all previous interactions with the environment through "experience replay".

Original: Such methods outperform almost all on-policy and model based alternatives in complex tasks where a structured or well parameterized model of the world does not exist.
Revised: Such methods outperform almost all on-policy and model based alternatives in complex tasks where a structured or well parameterized model of the world does not exist.

Original: This makes them desirable for practitioners who lack domain speciﬁc knowledge, but who still require high sample efﬁciency.
Revised: This makes them desirable for practitioners who lack domain speciﬁc knowledge, but who still require high sample efﬁciency.

Original: However this high performance can come at a cost.
Revised: However this high performance can come at a cost.

Original: Because of additional hyperparameters introduced to efﬁciently learn function approximators, off-policy RL can perform poorly on new problems.
Revised: Because of additional hyperparameters introduced to efﬁciently learn function approximators, off-policy RL can perform poorly on new problems.

Original: To address parameter sensitivity, we show how the correct choice of non-uniform sampling for experience replay can stabilize model performance under varying environmental conditions and hyper-parameters.
Revised: To address parameter sensitivity, we show how the correct choice of non-uniform sampling for experience replay can stabilize model performance under varying environmental conditions and hyper-parameters.

Original:  I NTRODUCTION
Revised:  Introduction

Original: Deep reinforcement learning (RL) represents a unique approach on how to produce programs that can take gathered experience, and turn it into policies able to complete complex tasks.
Revised: Deep reinforcement learning (RL) represents a unique approach on how to produce programs that can take gathered experience and turn it into policies able to complete complex tasks.

Original: What’s more, its practical applications in autonomous vehicles, video-games and compiler optimization have proven its economic utility.
Revised: What’s more, its practical applications in autonomous vehicles, video-games and compiler optimization have proven its economic utility.

Original: However, alongside these potential applications lay a host of practical issues which stem from complicated algorithms, poor theoretical understanding, and the fundamental difﬁculty of the problem.
Revised: However, alongside these potential applications lay a host of practical issues which stem from complicated algorithms, poor theoretical understanding and the fundamental difﬁculty of the problem.

Original: To take full advantage of these methods as a research community, we need to better understand how these methods work, and why.
Revised: To take full advantage of these methods as a research community, we need to better understand how these methods work and why.

Original: To make this question more tractable, in this work we review a special class of RL algorithms known as “off-policy” algorithms, which have in recent years proven to be the most performant class of methods in deep RL (Haarnoja et al., 2019; Fujimoto et al., 2018; Lee et al., 2021a).
Revised: To make this question more tractable, in this work we review a special class of RL algorithms known as “off-policy“ algorithms which have in recent years proven to be the most performant class of methods in deep RL (Haarnoja et al., 2019; Fujimoto et al., 2018; Lee et al., 2021a).

Original: These algorithms use experience gathered from all interactions with the environment to produce a better estimate of the expected cumulative reward; this contrasts with on-policy methods which only uses the most recent environment interaction or model based approaches which simulate examples using an internal model of the world.
Revised: These algorithms use experience gathered from all interactions with the environment to produce a better estimate of the expected cumulative reward; this contrasts with on-policy methods which only uses the most recent environment interaction or model based approaches which simulate examples using an internal model of the world.

Original: However, using this cache of experience effectively can be difﬁcult in practice due to computational constraints and the bootstrapping error introduced by off-policy algorithms.
Revised: However, using this cache of experience effectively can be difﬁcult in practice due to computational constraints and the bootstrapping error introduced by off-policy algorithms.

Original: These sources of error make changes to how interactions are added to replay buffer, and how previous interactions are sampled, important questions when presented with a new control problem; particularly in continuous control where both simulation and real world interactions can quickly become expensive.
Revised: These sources of error make changes to how interactions are added to replay buffer, and how previous interactions are sampled, important questions when presented with a new control problem; particularly in continuous control where both simulation and real world interactions can quickly become expensive.

Original: We therefore consider how the sampling scheme affects performance of one of the most popular off-policy algorithms, Soft Actor Critic (SAC) (Haarnoja et al., 2018).
Revised: We therefore consider how the sampling scheme affects performance of one of the most popular off-policy algorithms, Soft Actor-Critic (SAC) (Haarnoja et al., 2018).

Original: SAC is one of the most common baselines in deep reinforcement learning algorithms tailored to continuous control, and as we show below, can often be improved by properly chosen combinations of sampling methods and their hyper-parameters.
Revised: SAC is one of the most common baselines in deep reinforcement learning algorithms tailored to continuous control, and as we show below, can often be improved by properly chosen combinations of sampling methods and their hyper-parameters.

Original: We have chosen continuous control, because unlike discrete problems such as Atari, which have seen extensive experimentation across different sampling schemes, the effects of NUS on continuous control have remained largely ignored.
Revised: We have chosen continuous control, because unlike discrete problems such as Atari, which have seen extensive experimentation across different sampling schemes, the effects of non-uniform sampling (NUS) on continuous control have remained largely ignored.

Original: In fact outside of broader sensitivity analyses which either exclude non-uniform sampling (Islam et al., 2017; Henderson et al., 2018; Duan et al., 2016), or have focused speciﬁcally on on-policy methods (Engstrom et al., 2020;
Revised: In fact outside of broader sensitivity analyses which either exclude NUS (Islam et al., 2017; Henderson et al., 2018; Duan et al., 2016), or have focused speciﬁcally on on-policy methods (Engstrom et al., 2020;

Original: Andrychowicz et al., 2021), a modern practical survey on non-uniform sampling for continuous control seems non-existent.
Revised: Andrychowicz et al., 2021), a modern practical survey on NUS for continuous control seems nonexistent.

Original: Generally, the community has focused on the exploration method in the continuous control setting and assumes samples will be taken uniformly at random from the memory induced by the behavioral policy.
Revised: Generally, the community has focused on the exploration method in the continuous control setting and assumes samples will be taken uniformly at random from the memory induced by the behavioral policy.

Original: Where to contrast with discrete control, PER has been shown to outperform other sampling schemes (Hessel et al., 2018; Schaul et al., 2016) across almost all standard benchmarks.
Revised: Where to contrast with discrete control, PER has been shown to outperform other sampling schemes (Hessel et al., 2018; Schaul et al., 2016) across almost all standard benchmarks.

Original: To tackle this inconsistency, this paper covers the two broad classes of sampling algorithms for continuous control: generic priority based algorithms such as priority experience replay (PER) (Schaul et al., 2016), and recency based algorithms such emphasized experiencereplay (ERE) (Wang & Ross, 2019).
Revised: To tackle this inconsistency, this paper covers the two broad classes of sampling algorithms for continuous control: generic priority based algorithms such as prioritized experience replay (PER) (Schaul et al., 2016), and recency based algorithms such emphasizing recent experience (ERE) (Wang & Ross, 2019).

Original: Picking the most popular variants of these two approaches, we answer the following questions:
Revised: Picking the most popular variants of these two approaches, we answer the following questions:

Original: Which sampling methods are most effective in the deterministic continuous control setting, and are any robust in the stochastic feedback setting?
Revised: Which sampling methods are most effective in the deterministic continuous control setting, and are any robust in the stochastic feedback setting?

Original: How does the memory capacity, the maximum number of examples stored in the replay buffer, affect the performance of the algorithm.
Revised: How does the memory capacity, the maximum number of examples stored in the replay buffer, affect the performance of the algorithm.

Original: And can non-uniform sampling improve performance in the low-capacity setting?
Revised: And can NUS improve performance in the low-capacity setting?

Original: How do code-level design choices in non-uniform sampling methods affect performance?
Revised: How do code-level design choices in NUS methods affect performance?

Original: Which sampling methods are most robust to changes in hyper-parameters?
Revised: Which sampling methods are most robust to changes in hyper-parameters?

Original: Can classic non-uniform sampling algorithms like priority experience replay (Schaul et al., 2016) be improved through the use of different priority metrics?
Revised: Can classic NUS algorithms like prioritized experience replay (Schaul et al., 2016) be improved through the use of different priority metrics?

Original: In answering these questions, this paper ﬁrst shows that while no individual sampler performs better then others across tasks in the deterministic setting, PER outperforms both uniform and ERE in most stochastic settings.
Revised: In answering these questions, this paper ﬁrst shows that while no individual sampler performs better then others across tasks in the deterministic setting, PER outperforms both uniform and ERE in most stochastic settings.

Original: Next we show that the memory capacity, independent of the non-uniform sampling algorithm chosen, can destroy performance.
Revised: Next we show that the memory capacity, independent of the NUS algorithm chosen, can destroy performance.

Original: And ﬁnally, we show that careful tuning of parameters like mini-batch size and number of policy updates can yield non-trivial improvement at a small increase in computational cost, while mixed sampling approaches for both the actor and critic networks beat the standard combined sampling approaches most commonly used in practice.
Revised: And ﬁnally, we show that careful tuning of parameters like minibatch size and number of policy updates can yield non-trivial improvement at a small increase in computational cost, while mixed sampling approaches for both the actor and critic networks beat the standard combined sampling approaches most commonly used in practice.

Original:  B ACKGROUND 2. M ARKOV D ECISION P ROCESSES AND R EINFORCEMENT L EARNING
Revised:  Background 2.1 Markov Decision Processes and Reinforcement Learning

Original: We deﬁne a Markov decision process (MDP), M Φ ( S , A , R , T 0 , T , Π Φ ) , as a stochastic process which produces a sequence of tuples τ t := { a t , s t , s t +1 , r t } , for a particular set of states s t ∈ S , actions a t ∈ A , initial state distribution p ( s 0 ) ∈ T 0 , transition distribution p ( s t +1 | s t , a t ) ∈ T , reward function r t : S ×
Revised: We deﬁne a Markov decision process (MDP), M Φ ( S , A , R , T 0 , T , Π Φ ) , as a stochastic process which produces a sequence of tuples τ t := { a t , s t , s t +1 , r t } , for a particular set of states s t ∈ S , actions a t ∈ A , initial state distribution p ( s 0 ) ∈ T 0 , transition distribution p ( s t +1 | s t , a t ) ∈ T , reward function r t : S ×

Original: A × S → R , and policy π φ ∈ Π φ : S →
Revised: A × S → R , and policy π φ ∈ Π φ : S →

Original: A parameterized by φ ∈ Φ .
Revised: A parameterized by φ ∈ Φ .

Original: The generative model for this ﬁnite horizon process is deﬁned:
Revised: The generative model for this ﬁnite horizon process is deﬁned:

Original: [Equation]
Revised: [Equation]

Original: We denote the marginal distribution with respect to a state s , under the trajectory distribution given in equation 1, as q π φ ( s ) .
Revised: We denote the marginal distribution with respect to a state s , under the trajectory distribution given in equation 1, as q π φ ( s ) .

Original: In the ﬁnite horizon MDP setting, RL seeks to recover the policy π φ , which maximizes expected cumulative reward over the trajectory induced, through seeking the solution of φ ∗ = argmax φ ∈
Revised: In the ﬁnite horizon MDP setting, RL seeks to recover the policy π φ , which maximizes expected cumulative reward over the trajectory induced, through seeking the solution of φ ∗ = arg

Original: [-Φ E q πφ ( τ ) (cid:104)(cid:80) Tt =
 r t ( s t , a t ) (cid:105) .-]
Revised: [-Φ E q πφ ( τ ) (cid:104)(cid:80) Tt =
 r t ( s t , a t ) (cid:105) .-][Equation]

Original: We can extend this problem class to an inﬁnite horizon setting through the use of a discount factor which signiﬁes the non-zero probability of termination after every time-step.
Revised: We can extend this problem class to an inﬁnite horizon setting through the use of a discount factor which signiﬁes the non-zero probability of termination after every time-step.

Original: In this setting it is common to arrive at a policy through the solution of a ﬁxed point equation known as the bellman equation (Bertsekas, 2019;
Revised: In this setting it is common to arrive at a policy through the solution of a ﬁxed point equation known as the bellman equation (Bertsekas, 2019;

Original: Sutton & Barto, 2018), by solving the following optimization problem:
Revised: Sutton & Barto, 2018), by solving the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: and Q

Original: [Equation]
Revised: [Equation]

Original: and Q π φ ( a, s 
Revised: a, s ) =

Original: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Here the distribution over states, d π φ ( s ) in Equation 2, deﬁnes the state occupancy , while the function in Equation 3 deﬁnes the Q function.
Revised: Here the distribution over states, d π φ ( s ) in Equation 2, deﬁnes the state occupancy , while the function in Equation 3 deﬁnes the Q function.

Original: For practical purposes, the Q function is generally parameterized, and learned in an ofﬂine manner.
Revised: For practical purposes, the Q function is generally parameterized, and learned in an ofﬂine manner.

Original: In this ofﬂine setting the target of the Q function shifts from directly targeting the expected reward ahead of the current policy, and instead targets the expected reward ahead of a pessimistic approximation which is deﬁned as the mixture over all previous behavior policies used to interact with the environment.
Revised: In this ofﬂine setting the target of the Q function shifts from directly targeting the expected reward ahead of the current policy, and instead targets the expected reward ahead of a pessimistic approximation which is deﬁned as the mixture over all previous behavior policies used to interact with the environment.

Original: [Equation]
Revised: 2.2 Soft Actor-Critic

Original: One popular variant of this framework is known as soft actor-critic (SAC).
Revised: One popular variant of this framework is known as soft actor-critic (SAC).

Original: This algorithm is similar to most off-policy actor critic algorithms, in that it learns to estimate the expected reward ahead given the state and action (the Q-function), using the bellman error:
Revised: This algorithm is similar to most off-policy actor critic algorithms, in that it learns to estimate the expected reward ahead given the state and action (the Q-function), using the bellman error:

Original: [Equation]
Revised: [Equation]

Original: Here, D deﬁnes the set of examples gathered from the environment under a behavioral policy, or a sequence of behavioral policies.
Revised: Here, D deﬁnes the set of examples gathered from the environment under a behavioral policy, or a sequence of behavioral policies.

Original: The primary distinction between SAC and other AC algorithms (Mnih et al., 2016; Fujimoto et al., 2018) however, is that r t is replaced with a surrogate ˆ r t =
Revised: The primary distinction between SAC and other AC algorithms (Mnih et al., 2016; Fujimoto et al., 2018) however, is that r t is replaced with a surrogate ˆ r t =

Original: r t − H t , where H t indicates the policy entropy at state s t .
Revised: r t − H t , where H t indicates the policy entropy at state s t .

Original: Additionally in this setting, the bar over Q indicates a target network, generally deﬁned by the use of parameter averaging (Mnih et al., 2013; Wang et al., 2016), a model ensemble (Chen et al., 2017), or both (Hessel et al., 2018).
Revised: Additionally in this setting, the bar over Q indicates a target network, generally deﬁned by the use of parameter averaging (Mnih et al., 2013; Wang et al., 2016), a model ensemble (Chen et al., 2017), or both (Hessel et al., 2018).

Original: Using this Q function estimate, we can iteratively update our policy to match the so called “softmax-optimal” policy (Haarnoja et al., 2018; 2019; Levine, 2018).
Revised: Using this Q function estimate, we can iteratively update our policy to match the so called “softmax-optimal" policy (Haarnoja et al., 2018, 2019; Levine, 2018).

Original: This approach deﬁnes its policy objective based on a divergence between the current policy and a normalized target Q function:
Revised: This approach deﬁnes its policy objective based on a divergence between the current policy and a normalized target Q function:

Original: [Equation]
Revised: [Equation]

Original: Here the gradient is evaluated using the reparametrization trick (Le et al., 2017) in cases where the distribution over actions is reparameterizable (e.g. normal, gamma ect.), and with reinforce (Williams, 1992) otherwise.
Revised: Here the gradient is evaluated using the reparametrization trick (Le et al., 2017) in cases where the distribution over actions is reparameterizable (e.g. normal, gamma ect.), and with reinforce (Williams, 1992) otherwise.

Original: A description of SAC is included in Algorithm 1.
Revised: A description of SAC is included in Algorithm 1.

Original: P RIORITY E XPERIENCE R EPLAY
Revised: 2.3 Prioritized Experience Replay

Original: Prioritized Experience Replay (PER)
Revised: Prioritized Experience Replay (PER)

Original: (Schaul et al., 2016) is one of the most well known sampling methods in off-policy RL (Hessel et al., 2018), and was originally proposed for deep Qlearning (Hessel et al., 2018).
Revised: (Schaul et al., 2016) is one of the most well known sampling methods in off-policy RL (Hessel et al., 2018), and was originally proposed for deep Q-learning (Hessel et al., 2018).

Original: PER assigns priorities to transition examples using absolute temporal difference(TD) error.
Revised: PER assigns priorities to transition examples using absolute temporal difference(TD) error.

Original: The larger the absolute TD-error, the more likely the associated example is to be sampled.
Revised: The larger the absolute TD-error, the more likely the associated example is to be sampled.

Original: More speciﬁcally, the probability assigned to a transition is:
Revised: More speciﬁcally, the probability assigned to a transition is:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: In the original paper by Schaul et al.
Revised: In the original paper by Schaul et al.

Original: (2016), two prioritization schemes based on the above function are considered: a proportional prioritization, and a rank-based based prioritization.
Revised: (2016), two prioritization schemes based on the above function are considered: a proportional prioritization, and a rank-based based prioritization.

Original: For the proportional prioritization p ( i ) =
Revised: For the proportional prioritization p ( i ) =

Original: | δ i | + (cid:15) for the i th transition, where | δ
Revised: | δ i | + (cid:15) for the i th transition, where | δ

Original: i | is the absolute TD-error and (cid:15) a small positive constant that ensures
Revised: i | is the absolute TD-error and (cid:15) a small positive constant that ensures

Original: that p ( i ) > 0 ∀ i .
Revised: that p ( i ) > 0 ∀ i .

Original: In rank-based prioritization p ( i ) = [Equation] where the rank ( i ) is sorted based on the absolute TD-error | δ i | .
Revised: In rank-based prioritization p ( i ) = 1 rank ( i ) , where the rank ( i ) is sorted based on the absolute TD-error | δ i | .

Original: This work only considers proportional scoring scheme, as (Schaul et al., 2016) showed that, on average, it performed better than its ranked counterpart.
Revised: This work only considers proportional scoring scheme, as (Schaul et al., 2016) showed that, on average, it performed better than its ranked counterpart.

Original: Lastly, PER incorporates importance sampling to make the method more robust to noisy data, this is done by re-weighting the loss function by w i =
Revised: Lastly, PER incorporates importance sampling to make the method more robust to noisy data, this is done by re-weighting the loss function by w i =

Original: ( 1 N · 1 P ( i ) ) β .
Revised: ( 1 N · 1 P ( i ) ) β .

Original: Here N is the number of samples stored in the buffer and β is an annealing parameter which determines how close to uniform the sampler will be.
Revised: Here N is the number of samples stored in the buffer and β is an annealing parameter which determines how close to uniform the sampler will be.

Original: The full PER algorithm is described in Algorithm 2.
Revised: The full PER algorithm is described in Algorithm 2.

Original: 2. E MPHASIZED E XPERIENCE R EPLAY
Revised: 2.4 Emphasizing Recent Experience

Original: Emphasized ExperienceReplay (ERE) (Wang & Ross, 2019) increases the importance of recently observed data by sampling more aggressively from recent experiences.
Revised: Emphasizing Recent Experience (ERE) (Wang & Ross, 2019) increases the importance of recently observed data by sampling more aggressively from recent experiences.

Original: This is done by uniformly sampling from the most recent c k data points in the replay buffer, where for the implementation discussed in this paper, c k = max( N · η k 1000 K , M ) .
Revised: This is done by uniformly sampling from the most recent c k data points in the replay buffer, where for the implementation discussed in this paper, c k = max( N · η k 1000 K , M ) .

Original: Here, N deﬁnes the number of observations stored in the replay buffer total, while η ∈ (0 , 1] determines how aggressively to sample from recent data.
Revised: Here, N deﬁnes the number of observations stored in the replay buffer total, while η ∈ (0 , 1] determines how aggressively to sample from recent data.

Original: K is included in cases where multiple ( K ) mini-batch updates are taken per environment interaction.
Revised: K is included in cases where multiple ( K ) mini-batch updates are taken per environment interaction.

Original: The parameter M provides a ﬁxed range of η values per update cycle, and helps to avoid over-ﬁtting in small replay buffers.
Revised: The parameter M provides a ﬁxed range of η values per update cycle, and helps to avoid over-ﬁtting in small replay buffers.

Original: A more detailed description of the version of ERE used in this paper is included in Algorithm 3.
Revised: A more detailed description of the version of ERE used in this paper is included in Algorithm 3.

Original:  R ELATED W ORK
Revised:  Related Work

Original: Investigation into the effects of design decisions in experience replay has in recent years been sparse, with some exceptions.
Revised: Investigation into the effects of design decisions in experience replay has in recent years been sparse, with some exceptions.

Original: Notably for tabular and discrete settings, Zhang & Sutton (2017) explored the relationship of the size of the buffer capacity and model performance, while Liu & Zou (2018) explored the both effects of the buffer capacity and mini-batch size.
Revised: Notably for tabular and discrete settings, Zhang & Sutton (2017) explored the relationship of the size of the buffer capacity and model performance, while Liu & Zou (2018) explored the both effects of the buffer capacity and mini-batch size.

Original: Additionally, Fedus et al.
Revised: Additionally, Fedus et al.

Original: (2020) studied the buffer capacity, and the effect of the model update frequency and demonstrated that larger buffers with higher update to sample ratio performed better on average in the Atari Arcade Learning
Revised: (2020) studied the buffer capacity, and the effect of the model update frequency and demonstrated that larger buffers with higher update to sample ratio performed better on average in the Atari Arcade Learning

Original: Environment (Bellemare et al., 2013).
Revised: Environment (Bellemare et al., 2013).

Original: To our knowledge, only a few extensions to continuous control have been made.
Revised: To our knowledge, only a few extensions to continuous control have been made.

Original: Of these works, Wang & Ross (2019) studied the effects of replay buffer capacity on SAC (Haarnoja et al., 2018) where they found that decreased capacity again decreased in performance, while Islam et al.
Revised: Of these works, Wang & Ross (2019) studied the effects of replay buffer capacity on SAC (Haarnoja et al., 2018) where they found that decreased capacity again decreased in performance, while Islam et al.

Original: (2017) varied the batch size and found inconclusive results with respect to performance.
Revised: (2017) varied the batch size and found inconclusive results with respect to performance.

Original: Importantly, these studies were limited only to the uniform sampling, where we consider NUS methods.
Revised: Importantly, these studies were limited only to the uniform sampling, where we consider NUS methods.

Original: Crucially for this paper, signiﬁcant work has been done to ﬁnd more suitable sampling schemes for different environmental settings.
Revised: Crucially for this paper, signiﬁcant work has been done to ﬁnd more suitable sampling schemes for different environmental settings.

Original: Hessel et al.
Revised: Hessel et al.

Original: (2018) reported PER (Schaul et al., 2016) as being one of the most signiﬁcant modular improvements for DQN-like algorithms (Hessel et al., 2018; Mnih et al., 2013), with variations of it found in Lee et al.
Revised: (2018) reported PER (Schaul et al., 2016) as being one of the most signiﬁcant modular improvements for DQN-like algorithms (Hessel et al., 2018; Mnih et al., 2013), with variations of it found in Lee et al.

Original: (2021b)
Revised: (2021b)

Original: ; Fujimoto et al.
Revised: ; Fujimoto et al.

Original: Other work like
Revised: Other work like

Original: Ramicic & Bonarini (2017) deﬁnes new priority metrics for PER based on the perceived entropy of the state space, while Zha et al.
Revised: Ramicic & Bonarini (2017) deﬁnes new priority metrics for PER based on the perceived entropy of the state space, while Zha et al.

Original: and Oh et al.
Revised: and Oh et al.

Original: (2021) employed deep neural networks as a prioritization function to score stored transitions, learning both from local and global features.
Revised: (2021) employed deep neural networks as a prioritization function to score stored transitions, learning both from local and global features.

Original: Other sampling schemes incorporate recency prioritization:
Revised: Other sampling schemes incorporate recency prioritization:

Original: Wang & Ross (2019) takes subsets of the buffer during mini-batch updates to increase the sampling frequency of more recent data, while Kong et al.
Revised: Wang & Ross (2019) takes subsets of the buffer during mini-batch updates to increase the sampling frequency of more recent data, while Kong et al.

Original:  samples over a half-normal distribution centered at the latest observation, and Zhang & Sutton (2017) includes the latest observation during model updates.
Revised: (2021) samples over a half-normal distribution centered at the latest observation, and Zhang & Sutton (2017) includes the latest observation during model updates.

Original: More specialized sampling methods include Andrychowicz et al.
Revised: More specialized sampling methods include Andrychowicz et al.

Original: (2017) and Fang et al.
Revised: (2017) and Fang et al.

Original: (2019), which are aimed to improve performance in challenging sparse environments using goal conditioning.
Revised: (2019), which are aimed to improve performance in challenging sparse environments using goal conditioning.

Original:  E XPERIMENTS
Revised:  Experiments

Original: Using Deepmind’s Control Suite(Bellemare et al., 2013) we test Uniform, PER, and ERE sampling methods with SAC on 6 different tasks (WalkerWalk, CheetahRun, HumanoidStand, HumanoidRun,
Revised: Using Deepmind’s Control Suite(Bellemare et al., 2013) we test Uniform, PER, and ERE sampling methods with SAC on 6 different tasks (WalkerWalk, CheetahRun, HumanoidStand, HumanoidRun,

Original: QuadrupedRun, HopperHop) using 5 random seeds for 1 · 10 6 environment interactions.
Revised: QuadrupedRun, HopperHop) using 5 random seeds for 1 · 10 6 environment interactions.

Original: The experiments are broken down into the following four subsections, which start with creating a baseline of algorithm performance for each sampling method (4.1).
Revised: The experiments are broken down into the following four subsections, which start with creating a baseline of algorithm performance for each sampling method (4.1).

Original: Next we consider sampling methods in the stochastic setting (4.2) where noise is added to the reward to measure relative performance from deterministic baselines.
Revised: Next we consider sampling methods in the stochastic setting (4.2) where noise is added to the reward to measure relative performance from deterministic baselines.

Original: We then explore the sensitivity of standard replay buffer hyper-parameters(4.3) by varying: the buffer capacity, batch size, initial exploration steps, and number of policy updates.
Revised: We then explore the sensitivity of standard replay buffer hyperparameters(4.3) by varying: the buffer capacity, batch size, initial exploration steps, and number of policy updates.

Original: Lastly, we consider one possible degree of freedom with respect to implementation by varying priority metrics and splitting sampling procedures between actor and critic networks(4.4).
Revised: Lastly, we consider one possible degree of freedom with respect to implementation by varying priority metrics and splitting sampling procedures between actor and critic networks(4.4).

Original: 4. B ASELINE PERFORMANCE IN C ONTINUOUS C ONTROL
Revised: 4.1 Baseline performance in Continuous Control

Original: Starting with performance tests for Uniform, PER, and ERE with SAC on the Deepmind Control
Revised: Starting with performance tests for Uniform, PER, and ERE with SAC on the Deepmind Control

Original: Suite Tassa et al.
Revised: Suite Tassa et al.

Original: (2018) using standard hypeparameters found in Table 2.
Revised: (2018) using standard hypeparameters found in Table 2.

Original: As shown in Figure 1, no single sampling method outperforms all the others across environments, though both ERE and
Revised: As shown in Figure 1, no single sampling method outperforms all the others across environments, though both ERE and

Original: PER seem to provide examples in which they signiﬁcantly out-perform the other two samplers on a per-environment basis.
Revised: PER seem to provide examples in which they signiﬁcantly out-perform the other two samplers on a per-environment basis.

Original: In this setting, it seems clear that provided you search over non-uniform sampling schemes, you can get non-trivial increases in performance over the uniform baseline. 
Revised: In this setting, it seems clear that provided you search over NUS schemes, you can get non-trivial increases in performance over the uniform baseline. [Figure] L o g A v e r a g e

Original: 4. S AMPLING M ETHODS IN S TOCHASTIC S ETTING
Revised: R e w a r d [Figure] 4.2 Sampling Methods in Stochastic Setting

Original: Following the work of Romoff et al.
Revised: Following the work of Romoff et al.

Original: we introduce noise in the reward function to each environment by sampling from three different noise distributions: Gaussian, Uniform and Sparse.
Revised: we introduce noise in the reward function to each environment by sampling from three different noise distributions: Gaussian, Uniform and Sparse.

Original: For the Gaussian Noise example, we sample from a normal distribution Ψ ∼ N (0 , σ 2 ) with increasing variance of σ ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } such that the new reward r gt at step t is calculated r gt =
Revised: For the Gaussian Noise example, we sample from a normal distribution Ψ ∼ N (0 , σ 2 ) with increasing variance of σ ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } such that the new reward r gt at step t is calculated r gt =

Original: r t + ψ .
Revised: r t + ψ .

Original: CheetahRun

Original: HumanoidStand

Original: WalkerWalk

Original: QuadrupedRun

Original: HumanoidRun

Original: HopperHop Environments

Original: L o g

Original: A v e r a g e

Original: R e w a r d

Original: Uniform PER ERE For the Uniform and Sparse noise the new rewards r ut and r st are calculated respectively: 
Revised: For the Uniform and Sparse noise the new rewards r ut and r st are calculated respectively: r u t = (cid:26) ξ, with probability (cid:15)

Original: [Equation]
Revised: u r t , with probability (1 − (cid:15) u ) , r s t = (cid:26) 0 , with probability (cid:15) s r t , with probability (1 − (cid:15) s ) (7)

Original: With ξ ∼ U ( − 1 , 1) and probabilities (cid:15) u ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } and (cid:15) s ∈ { 0 . 6 , 0 .
Revised: With ξ ∼ U ( − 1 , 1) and probabilities (cid:15) u ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } and (cid:15) s ∈ { 0 . 6 , 0 .

Original: We compare the results by measuring the relative gain from the baseline which is calculated as 
Revised: We compare the results by measuring the relative gain from the baseline which is calculated as 100 × [ Stochastic Environment Performance ] − [ Deterministic

Original: [Equation]
Revised: [Equation]

Original: Each of the noise levels across 6 environments are included in Figure 2.
Revised: Each of the noise levels across 6 environments are included in Figure 2.

Original: This experiment, as expected, shows that the performance of each sampling method deteriorates as the noise becomes larger in magnitude.
Revised: This experiment, as expected, shows that the performance of each sampling method deteriorates as the noise becomes larger in magnitude.

Original: Interestingly, within the Gaussian Noise experiment QuadrupedRun environment σ = 0 . 1 , and Sparsity experiment HopperHop environment with (cid:15) = 0 .
Revised: Interestingly, within the Gaussian Noise experiment QuadrupedRun environment σ = 0 . 1 , and Sparsity experiment HopperHop environment with (cid:15) = 0 .

Original: 6 , PER and ERE respectively report a net positive gain, actually performing better than their respective noise-free baseline shown in Figure 1.
Revised: 6 , PER and ERE respectively report a net positive gain, actually performing better than their respective noise-free baseline shown in Figure 1.

Original: This could indicate that some regularization might beneﬁt learning in certain environments depending on the reward structure, or that additional seeds are required.
Revised: This could indicate that some regularization might beneﬁt learning in certain environments depending on the reward structure, or that additional seeds are required.

Original: Other than these anomalies, almost any noise appears to be catastrophic to the learning performance, though amongst sampling methods PER seems to perform the best in general.
Revised: Other than these anomalies, almost any noise appears to be catastrophic to the learning performance, though amongst sampling methods PER seems to perform the best in general.

Original: It should be noted however that PER does seem to be especially susceptible to the uniform noise regime, while both ERE and uniform are not.
Revised: It should be noted however that PER does seem to be especially susceptible to the uniform noise regime, while both ERE and uniform are not.

Original: However the difference in performance in this one instance is somewhat negligible in comparison to the performance gains we see in other distributions of noise.
Revised: However the difference in performance in this one instance is somewhat negligible in comparison to the performance gains we see in other distributions of noise.

Original: This experiment therefore implies that without any prior knowledge of the distribution of noise which might apply to your environment, PER is the best choice of NUS algorithm.
Revised: This experiment therefore implies that without any prior knowledge of the distribution of noise which might apply to your environment, PER is the best choice of NUS algorithm.

Original: 4. R EPLAY B UFFER H YPERPARAMETER S ENSITIVITY A NALYSIS
Revised: 4.3 Replay Buffer Hyperparameter Sensitivity Analysis

Original: We now conduct a hyperparameter sensitivity analysis of replay buffers and their effect in Uniform,
Revised: We now conduct a hyperparameter sensitivity analysis of replay buffers and their effect in Uniform,

Original: PER and ERE sampling methods.
Revised: PER and ERE sampling methods.

Original: More speciﬁcally we will be considering the following settings:
Revised: More speciﬁcally we will be considering the following settings:

Original:  Decreasing Replay Buffer Capacity N b to 10 5 , 10 4 , and 10• Varying exploration steps E s to 1000, 5000, 10000 ,and• Varying mini-batch size B s to 128, 256, 512, and• Varying number of Policy updates P u to 1, 3, 5, and These experiments conﬁrm some of the existing hypothesis within the literature for continuous control in off-policy rl, while showing that these hypothesis do not quite hold for all non-uniform sampling algorithms. Looking at the mini-batch size ﬁrst, we see that while ERE and Uniform see monotonic improvement in the batch-size, there is a very negligible effect in the context of PER. This indicates that PER might be best suited to large models for which low batch sizes are required.{+
+}
Revised: • Decreasing Replay Buffer Capacity N b to 10 5 , 10 4 , and {+
+}

Original: WalkerWalk CheetahRun
Revised: WalkerWalk CheetahRun

Original: QuadrupedRun HopperHop
Revised: QuadrupedRun HopperHop

Original: HumanoidStand HumanoidRun U n i f
Revised: HumanoidStand HumanoidRun U n i f

Original: o r m
Revised: o r m

Original: WalkerWalk CheetahRun
Revised: WalkerWalk CheetahRun

Original: QuadrupedRun HopperHop
Revised: QuadrupedRun HopperHop

Original: HumanoidStand HumanoidRun
Revised: HumanoidStand HumanoidRun

Original: P E R 0.
Revised: P E R 0.

Original: 0.4 Gaussian
Revised: 0.4 Gaussian

Original: WalkerWalk CheetahRun
Revised: WalkerWalk CheetahRun

Original: QuadrupedRun HopperHop
Revised: QuadrupedRun HopperHop

Original: HumanoidStand HumanoidRun
Revised: HumanoidStand HumanoidRun

Original: E R E 0.
Revised: E R E 0.

Original: [Figure] [Figure] Uniform 0.
Revised: 0.4 Uniform 0.

Original: 0.9 Sparse −−
Revised: 0.9 Sparse −−

Original: Average Relative Gain
Revised: Average Relative Gain

Original: Parameter Value
Revised: Parameter Value

Original: Uniform
Revised: Uniform

Original: PER
Revised: PER

Original: ERE
Revised: ERE

Original: Gaussian Reward Noise σ = 0 .-36.
Revised: Gaussian Reward Noise σ = 0 .-36.

Original: σ = 0 .-53. σ = 0 .-63.
Revised: σ = 0 .-53. σ = 0 .-52.

Original: σ = 0 .-62.
Revised: σ = 0 .-62.

Original: Uniform Reward Noise (cid:15) = 0 .-36.
Revised: Uniform Reward Noise (cid:15) = 0 .-49.

Original: (cid:15)
Revised: (cid:15)

Original: = 0 .-53.
Revised: = 0 .-55.

Original: (cid:15)
Revised: (cid:15)

Original: = 0 .-63.
Revised: = 0 .-63.

Original: (cid:15) = 0 .-62.{+
+}
Revised: (cid:15) = {+
+}

Original: Sparse Reward Noise (cid:15) = 0 .-36.{+.-22. (cid:15) = 
+}
Revised: Sparse Reward Noise (cid:15) = 0 {+.-22. (cid:15) = 
+}

Original: (cid:15)
Revised: (cid:15)

Original: = 0 .-53.{+
+}
Revised: = {+
+}

Original: (cid:15)
Revised: (cid:15)

Original: = 0 .-63.
Revised: = 0 .-55.

Original: (cid:15) =
Revised: • Varying exploration steps E s to 1000, 5000, 10000 ,and• Varying mini-batch size B s to 128, 256, 512, and• Varying number of Policy updates P u to 1, 3, 5, and

Original: [Table] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: These experiments conﬁrm some of the existing hypothesis within the literature for continuous control in off-policy RL, while showing that these hypothesis do not quite hold for all NUS algorithms. Looking at the mini-batch size ﬁrst

Original: [Figure]
Revised: [Figure]

Revised: we see that while ERE and Uniform see monotonic improvement in the batch-size

Original: [Figure]
Revised: [Figure]

Revised: there is a very negligible effect in the context of PER.

Revised: This indicates that PER might be best suited to large models for which low batch sizes are required.

Original: Next, considering the size of the replay buffer, we can see that this is clearly one of the most important hyper-parameters of SAC independent of the sampling algorithm used.
Revised: Next, considering the size of the replay buffer, we can see that this is clearly one of the most important hyper-parameters of SAC independent of the sampling algorithm used.

Original: We also consider the effect of multiple policy steps per update where we ﬁnd that there is approximately monotonic improvement as one increases the number of policy updates, though the improvement is generally negligible.
Revised: We also consider the effect of multiple policy steps per update where we ﬁnd that there is approximately monotonic improvement as one increases the number of policy updates, though the improvement is generally negligible.

Original: This means that  throughout training, we are both targeting a useful objective, and that our policy remains close to the optimum (for any intermediate Q function) at any given step.
Revised: This means that [Figure] [Figure] throughout training, we are both targeting a useful objective, and that our policy remains close to the optimum (for any intermediate Q function) at any given step.

Original: The last plot, which varies exploration steps does not exhibit any major trends, indicating that tuning this parameter will be situation dependent.
Revised: The last plot, which varies exploration steps does not exhibit any major trends, indicating that tuning this parameter will be situation dependent.

Original: 4. S EPARATE S AMPLING M ETHODS AND V ARYING P RIORITY M ETRICS
Revised: 4.4 Separate Sampling Methods and Varying Priority Metrics

Original: In the ﬁnal experiment we explore different variations of sampling method pairs.
Revised: In the ﬁnal experiment we explore different variations of sampling method pairs.

Original: A central replay buffer is used for all variations, while each method per pair samples from this buffer independently from the other.
Revised: A central replay buffer is used for all variations, while each method per pair samples from this buffer independently from the other.

Original: We also explore different priority metrics for PER, like the mean-squared error from the critic’s loss function, and the KL divergence from the actor’s loss function.
Revised: We also explore different priority metrics for PER, like the mean-squared error from the critic’s loss function, and the KL divergence from the actor’s loss function.

Original: The results can be found in Figure 4 and are sorted by performance.
Revised: The results can be found in Figure 4 and are sorted by performance.

Original: Interestingly enough, a combination of ERE in the policy and uniform in the critic performs the best on average across standard benchmarks.[Figure] r [Figure]
Revised: Interestingly enough, a combination of ERE in the policy and uniform in the critic performs the best on average across standard benchmarks.

Original: - P E R

Original: E R E - P E

Original: R P E R

Original: P E R

Original: [Figure]

Original: U n i f

Original: o r m E R E

Original: U n i f

Original: o r m - E R E

Original: P E R - E R

Original: E

Original: E R E - U n i f

Original: o r m

Original: [Figure]

Original: CheetahRun

Original: HumanoidStand

Original: QuadrupedRun

Original: HopperHop

Original: HumanoidRun

Original: [Figure]
Revised: [Figure]

Original: varying priority metrics for PER instead of TD-errror:

Original: PER using KL-Divergence (PER+KL), PER using Mean Squared Error (PER+MSE).

Original: Tested on 6 Deepmind Control tasks (WalkerWalk, CheetahRun, HumanoidStand, QuadrupedRun, HopperHop, HumanoidRun) over 10 6 environment interactions for 5 random seeds.

Original: Sampling methods are sorted by ascending order based on their average learning performance over the 6 tasks.

Original:  C ONCLUSION
Revised:  Conclusion

Original: In this paper we investigated the behavior of non-uniform sampling methods in off-policy RL.
Revised: In this paper we investigated the behavior of NUS methods in off-policy RL.

Original: Our experiments showed the following results: ﬁrst, in the deterministic setting no individual sampling method performed signiﬁcantly better for all tasks.
Revised: Our experiments showed the following results: ﬁrst, in the deterministic setting no individual sampling method performed signiﬁcantly better for all tasks.

Original: Second, in the stochastic setting PER performed signiﬁcantly better then all other sampling methods considered.
Revised: Second, in the stochastic setting PER performed signiﬁcantly better then all other sampling methods considered.

Original: Third, changing hyper-parameters, particularly the memory capacity, can drastically change the relative performance of each algorithm.
Revised: Third, changing hyper-parameters, particularly the memory capacity, can drastically change the relative performance of each algorithm.

Original: Lastly, by varying the types of samplers used for the actor and critic networks we gain non-trivial improvement in each of the environments over the single sampler baselines, indicating the optimal sampler used to update both the actor and critic networks might not be the same.
Revised: Lastly, by varying the types of samplers used for the actor and critic networks we gain non-trivial improvement in each of the environments over the single sampler baselines, indicating the optimal sampler used to update both the actor and critic networks might not be the same.