Original: L OOK B ACK W HEN S URPRISED : S TABILIZING R EVERSE E XPERIENCE R EPLAY FOR N EURAL A PPROXIMATION Anonymous authors Paper under double-blind review
Revised: Look Back When Surprised: Stabilizing Reverse Experience Replay for Neural Approximation

Original: A BSTRACT
Revised: Abstract

Original: Experience replay-based sampling techniques are essential to several reinforcement learning (RL) algorithms since they aid in convergence by breaking spurious correlations.
Revised: Experience replay-based sampling techniques are essential to several reinforcement learning (RL) algorithms since they aid in convergence by breaking spurious correlations.

Original: The most popular techniques, such as uniform experience replay (UER) and prioritized experience replay (PER), seem to suffer from sub-optimal convergence and signiﬁcant bias error, respectively.
Revised: The most popular techniques, such as uniform experience replay (UER) and prioritized experience replay (PER), seem to suffer from sub-optimal convergence and signiﬁcant bias error, respectively.

Original: To alleviate this, we introduce a new experience replay method for reinforcement learning, called Introspective
Revised: To alleviate this, we introduce a new experience replay method for reinforcement learning, called Introspective

Original: Experience Replay (IER).
Revised: Experience Replay (IER).

Original: IER picks batches corresponding to data points consecutively before the ‘surprising’ points.
Revised: IER picks batches corresponding to data points consecutively before the ‘surprising’ points.

Original: Our proposed approach is based on the theoretically rigorous reverse experience replay (RER), which can be shown to remove bias in the linear approximation setting but can be sub-optimal with neural approximation.
Revised: Our proposed approach is based on the theoretically rigorous reverse experience replay (RER), which can be shown to remove bias in the linear approximation setting but can be sub-optimal with neural approximation.

Original: We show empirically that IER is stable with neural function approximation and has a superior performance compared to the state-of-the-art techniques like uniform experience replay (UER), prioritized experience replay (PER), and hindsight experience replay (HER) on the majority of tasks.
Revised: We show empirically that IER is stable with neural function approximation and has a superior performance compared to the state-of-the-art techniques like uniform experience replay (UER), prioritized experience replay (PER), and hindsight experience replay (HER) on the majority of tasks.

Original:  I NTRODUCTION
Revised:  Introduction

Original: Reinforcement learning (RL) involves learning with dependent data, and algorithms designed for independent data might behave poorly coupled with the Markovian trajectories encountered in this setting.
Revised: Reinforcement learning (RL) involves learning with dependent data, and algorithms designed for independent data might behave poorly coupled with the Markovian trajectories encountered in this setting.

Original: Experience replay (Lin, 1992) involves storing the received data points in a large buffer and producing a random sample from this buffer whenever the learning algorithm requires it.
Revised: Experience replay (Lin, 1992) involves storing the received data points in a large buffer and producing a random sample from this buffer whenever the learning algorithm requires it.

Original: Therefore experience replay is usually deployed with popular algorithms like DQN and TD3 to achieve state-ofthe-art performance (Mnih et al., 2015; Lillicrap et al., 2015).
Revised: Therefore experience replay is usually deployed with popular algorithms like DQN, DDPG and TD3 to achieve state-of-the-art performance (Mnih et al., 2015; Lillicrap et al., 2015).

Original: It has been shown experimentally (Mnih et al., 2015) and theoretically (Nagaraj et al., 2020) that these learning algorithms for Markovian data behave sub-optimally without experience replay.
Revised: It has been shown experimentally (Mnih et al., 2015) and theoretically (Nagaraj et al., 2020) that these learning algorithms for Markovian data behave sub-optimally without experience replay.

Original: Note that we use the term “sub-optimal” when consistently observing a sub-par performance compared to the oracle.
Revised: Note that we use the term “sub-optimal” when consistently observing a sub-par performance compared to the oracle.

Original: In contrast, the term “instability” refers to the setting where there is a high variance in our experiments, where only a few seeds work well.
Revised: In contrast, the term “instability” refers to the setting where there is a high variance in our experiments, where only a few seeds work well.

Original: We maintain this distinction throughout our paper.
Revised: We maintain this distinction throughout our paper.

Original: The simplest and most widely used experience replay method is the uniform experience replay (UER), where the data points stored in the buffer are sampled uniformly at random every time a data point is queried (Mnih et al., 2015).
Revised: The simplest and most widely used experience replay method is the uniform experience replay (UER), where the data points stored in the buffer are sampled uniformly at random every time a data point is queried (Mnih et al., 2015).

Original: However, UER might pick uninformative data points most of the time, which may slow down the convergence.
Revised: However, UER might pick uninformative data points most of the time, which may slow down the convergence.

Original: For this reason, optimistic experience replay (OER) and prioritized experience replay (PER) (Schaul et al., 2015) were introduced, where samples with higher TD error (i.e., ‘surprise’) are sampled more often from the buffer.
Revised: For this reason, optimistic experience replay (OER) and prioritized experience replay (PER) (Schaul et al., 2015) were introduced, where samples with higher TD error (i.e., ‘surprise’) are sampled more often from the buffer.

Original: Optimistic experience replay  was shown to have a high bias, and Prioritized experience replay was proposed to solve this issue (Schaul et al., 2015).
Revised: Optimistic experience replay (originally called “greedy TD-error prioritisation”) was shown to have a high bias, and Prioritized experience replay was proposed to solve this issue (Schaul et al., 2015).

Original: However, as shown in our experiments outside of the Atari environments, PER still suffers from the problem of high bias.
Revised: However, as shown in our experiments outside of the Atari environments, PER still suffers from the problem of high bias.

Original: Although this speeds up the learning process in many cases, there can be signiﬁcant biases due to picking and choosing only speciﬁc data points, which can make this method sub-optimal.
Revised: Although this speeds up the learning process in many cases, there can be signiﬁcant biases due to picking and choosing only speciﬁc data points, which can make this method sub-optimal.

Original: The design of experience replay continues to be an active ﬁeld of research.
Revised: The design of experience replay continues to be an active ﬁeld of research.

Original: Several other experience replay techniques like Hindsight experience replay (HER) (Andrychowicz et al., 2017), Reverse Experience Replay (RER) (Rotinov, 2019), and Topological Experience Replay (TER) (Hong et al., 2022) have been proposed.
Revised: Several other experience replay techniques like Hindsight experience replay (HER) (Andrychowicz et al., 2017), Reverse Experience Replay (RER) (Rotinov, 2019), and Topological Experience Replay (TER) (Hong et al., 2022) have been proposed.

Original: An overview of these methods in discussed in Section 2.
Revised: An overview of these methods in discussed in Section 2.

Original: Even though these methods are widely deployed in practice, theoretical analyses have been very limited.
Revised: Even though these methods are widely deployed in practice, theoretical analyses have been very limited.

Original: Recent results on learning dynamical systems (Kowshik et al., 2021b;a) showed rigorously in a theoretical setting that RER is the conceptually-grounded algorithm when learning from Markovian data.
Revised: Recent results on learning dynamical systems (Kowshik et al., 2021b,a) showed rigorously in a theoretical setting that RER is the conceptually-grounded algorithm when learning from Markovian data.

Original: Furthermore, this work was extended to the RL setting in Agarwal et al.
Revised: Furthermore, this work was extended to the RL setting in Agarwal et al.

Original: (2021) to achieve efﬁcient Q learning with linear function approximation.
Revised: (2021) to achieve efﬁcient Q learning with linear function approximation.

Original: The RER technique achieves good performance since reverse order sampling of the data points prevents the build-up of spurious correlations in the learning algorithm.
Revised: The RER technique achieves good performance since reverse order sampling of the data points prevents the build-up of spurious correlations in the learning algorithm.

Original: In this paper, we build on this line of work and introduce Introspective
Revised: In this paper, we build on this line of work and introduce Introspective

Original: Experience Replay (IER).
Revised: Experience Replay (IER).

Original: Roughly speaking, IER ﬁrst picks top k ‘pivot’ points from a large buffer according to their TD error.
Revised: Roughly speaking, IER ﬁrst picks top k ‘pivot’ points from a large buffer according to their TD error.

Original: It then returns batches of data formed by selecting the consecutive points temporally before these pivot points.
Revised: It then returns batches of data formed by selecting the consecutive points temporally before these pivot points.

Original: In essence, the algorithm looks back when surprised .
Revised: In essence, the algorithm looks back when surprised .

Original: The intuition behind our approach is linked to the fact that the agent should always associate outcomes to its past actions, just like in RER.
Revised: The intuition behind our approach is linked to the fact that the agent should always associate outcomes to its past actions, just like in RER.

Original: The summary of our approach is shown in Figure 1.
Revised: The summary of our approach is shown in Figure 1.

Original: This technique is an amalgamation of Reverse Experience Replay (RER), and Optimistic Experience Replay (OER), which only picks the points with the highest TD error.
Revised: This technique is an amalgamation of Reverse Experience Replay (RER), and Optimistic Experience Replay (OER), which only picks the points with the highest TD error.

Original: [Figure]
Revised: [Figure]

Original: Our main ﬁndings are summarized below:
Revised: Our main ﬁndings are summarized below:

Original: Better Performance Against SOTA: Our proposed methodology (IER) outperforms previous stateof-the-art baselines such as PER, UER and HER on most environments (see Table 1, Section 5).
Revised: Better Performance Against SOTA: Our proposed methodology (IER) outperforms previous stateof-the-art baselines such as PER, UER and HER on most environments (see Table 1, Section 5).

Original: Conceptual Understanding: We consider a simple toy example where we understand the differences between UER, RER, OER and IER (Section 3.2).
Revised: Conceptual Understanding: We consider a simple toy example where we understand the differences between UER, RER, OER and IER (Section 3.2).

Original: This study illustrates the better performance of our proposed method by showing a) why naive importance sampling, like in OER incurs a signiﬁcant bias and b) why techniques like UER and RER are slow to learn.
Revised: This study illustrates the better performance of our proposed method by showing a) why naive importance sampling, like in OER incurs a signiﬁcant bias and b) why techniques like UER and RER are slow to learn.

Original: Forward vs. Reverse: We show that the temporal direction (forward/reverse) to consider after picking the pivot is non-trivial.
Revised: Forward vs. Reverse: We show that the temporal direction (forward/reverse) to consider after picking the pivot is non-trivial.

Original: We show empirically that IER performs much better than its forward counterpart, IER (F) (see Table 3).
Revised: We show empirically that IER performs much better than its forward counterpart, IER (F) (see Table 3).

Original: This gives evidence towards causality playing a role in the success of IER.
Revised: This gives evidence towards causality playing a role in the success of IER.

Original: Whole vs.
Revised: Whole vs.

Original: Component Parts: Our method (IER) is obtained by combining two methods
Revised: Component Parts: Our method (IER) is obtained by combining two methods

Original: RER (which picks the samples in the reverse order as received) and OER (which greedily picks the samples with the largest TD error).
Revised: RER (which picks the samples in the reverse order as received) and OER (which greedily picks the samples with the largest TD error).

Original: We show that neither of these components performs well compared to their amalgamation, IER (Table 4).
Revised: We show that neither of these components performs well compared to their amalgamation, IER (Table 4).

Original: Minimal hyperparameter tuning: Our proposed methodology uses minimal hyperparameter tuning.
Revised: Minimal hyperparameter tuning: Our proposed methodology uses minimal hyperparameter tuning.

Original: We use the same policy network architecture, learning rate, batch size, and all other parameters across all our runs for a given environment.
Revised: We use the same policy network architecture, learning rate, batch size, and all other parameters across all our runs for a given environment.

Original: These hyperparameters are selected based on the setting required to achieve SOTA performance on UER.
Revised: These hyperparameters are selected based on the setting required to achieve SOTA performance on UER.

Original: However, we have few options available, a Hindsight ﬂag, leading to the H-IER sampler, and the Uniform Mixing fraction leading to the U-IER sampler.
Revised: However, we have few options available, a Hindsight ﬂag, leading to the H-IER sampler, and the Uniform Mixing fraction leading to the U-IER sampler.

Original: Furthermore, for most of our experiments, we use a mixing fraction of 0 .
Revised: Furthermore, for most of our experiments, we use a mixing fraction of 0 .

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORKS AND C OMPARISON 2. E XPERIENCE R EPLAY T ECHNIQUES
Revised:  Related Works and Comparison 2.1 Experience Replay Techniques

Original: Experience replay involves storing consecutive temporally dependent data in a (large) buffer in a
Revised: Experience replay involves storing consecutive temporally dependent data in a (large) buffer in a

Original: FIFO order.
Revised: FIFO order.

Original: Whenever a learning algorithm queries for batched data, the experience replay algorithm returns a sub-sample from this buffer such that this data does not hinder the learning algorithms due to spurious correlations.
Revised: Whenever a learning algorithm queries for batched data, the experience replay algorithm returns a sub-sample from this buffer such that this data does not hinder the learning algorithms due to spurious correlations.

Original: The most basic form of experience replay is UER (Lin, 1992) which samples the data in the replay buffer uniformly at random.
Revised: The most basic form of experience replay is UER (Lin, 1992) which samples the data in the replay buffer uniformly at random.

Original: This approach has signiﬁcantly improved the performance of off-policy RL algorithms like DQN (Mnih et al., 2015).
Revised: This approach has signiﬁcantly improved the performance of off-policy RL algorithms like DQN (Mnih et al., 2015).

Original: Several other methods of sampling from the buffer have been proposed since; PER (Schaul et al., 2015)
Revised: Several other methods of sampling from the buffer have been proposed since; PER (Schaul et al., 2015)

Original: samples experiences from a probability distribution which assigns higher probability to experiences with signiﬁcant TD error and is shown to boost the convergence speed of the algorithm.
Revised: samples experiences from a probability distribution which assigns higher probability to experiences with signiﬁcant TD error and is shown to boost the convergence speed of the algorithm.

Original: This out-performs UER in most Atari environments.
Revised: This out-performs UER in most Atari environments.

Original: HER (Andrychowicz et al., 2017) works in the "what if" scenario, where even a sub-optimal policy can lead the agent to learn what not to do and nudge the agent towards the correct action. 
Revised: HER (Andrychowicz et al., 2017) works in the "what if" scenario, where even a sub-optimal policy can lead the agent to learn what not to do and nudge the agent towards the correct action. There have also been other approaches such as Liu et al. ; Fang et al.

Revised: (2018, 2019) have adapted HER in order to improve the overall performance with varying intuition.

Original: RER processes the data obtained in a buffer in the reverse temporal order.
Revised: RER processes the data obtained in a buffer in the reverse temporal order.

Original: We refer to the following sub-section for a detailed review of this and related techniques.
Revised: We refer to the following sub-section for a detailed review of this and related techniques.

Original: We will also consider ‘optimistic experience replay’ (OER), the naive version of PER, where at each step, only top B elements in the buffer are returned when batched data is queried.
Revised: We will also consider ‘optimistic experience replay’ (OER), the naive version of PER, where at each step, only top B elements in the buffer are returned when batched data is queried.

Original: This approach is known to suffer from high bias error mitigated by a sophisticated sampling procedure employed in PER.
Revised: This approach is known to suffer from high bias error mitigated by a sophisticated sampling procedure employed in PER.

Revised: Other works such as Fujimoto et al.

Original: 2. R EVERSE S WEEP T ECHNIQUES
Revised: Pan et al. (2022) attempt to study PER and address some of its shortcomings and Lahire et al. (2021) introduces ’large batch experience replay’ (LaBER) which reduces the stochastic noise in gradients, while keeping them unbiased. 2.2 Reverse Sweep Techniques

Original: Reverse sweep or backward value iteration refers to methods that process the data as received in reverse order.
Revised: Reverse sweep or backward value iteration refers to methods that process the data as received in reverse order.

Original: This has been studied in the context of planning tabular MDPs (Dai & Hansen, 2007;
Revised: This has been studied in the context of planning tabular MDPs (Dai & Hansen, 2007;

Original: Grze´s & Hoey, 2013).
Revised: Grze´s & Hoey, 2013).

Original: We refer to Section 4 for a brief overview of why these methods are considered.
Revised: We refer to Section 4 for a brief overview of why these methods are considered.

Original: However, this line of work assumes that the MDP and the transition functions are known.
Revised: However, this line of work assumes that the MDP and the transition functions are known.

Original: Inspired by the behavior of biological networks, Rotinov (2019) proposed reverse experience replay where the experience replay buffer is replayed in a LIFO order.
Revised: Inspired by the behavior of biological networks, Rotinov (2019) proposed reverse experience replay where the experience replay buffer is replayed in a LIFO order.

Original: Since RER forms mini-batches with consecutive data points, it is unstable with Neural approximation (i.e, it does not learn consistently across different environments).
Revised: Since RER forms mini-batches with consecutive data points, it is unstable with Neural approximation (i.e, it does not learn consistently across different environments).

Original: Therefore, the iterations are stabilized by ‘mixing’ RER with UER.
Revised: Therefore, the iterations are stabilized by ‘mixing’ RER with UER.

Original: However, the experiments are limited and do not demonstrate that this method outperforms even UER.
Revised: However, the experiments are limited and do not demonstrate that this method outperforms even UER.

Original: A similar procedure called Episodic Backward Update (EBU) is introduced in Lee et al.
Revised: A similar procedure called Episodic Backward Update (EBU) is introduced in Lee et al.

Original: However, to stabilize the pure RER , the EBU method seeks to also change the target for Q learning instead of just changing the sampling scheme in the replay buffer.
Revised: However, to stabilize the pure RER , the EBU method seeks to also change the target for Q learning instead of just changing the sampling scheme in the replay buffer.

Original: The reverse sweep was independently rediscovered as RER in the context of streaming linear system identiﬁcation in Kowshik et al.
Revised: The reverse sweep was independently rediscovered as RER in the context of streaming linear system identiﬁcation in Kowshik et al.

Original: (2021b), where SGD with reverse experience replay was shown to achieve near-optimal performance.
Revised: (2021b), where SGD with reverse experience replay was shown to achieve near-optimal performance.

Original: In contrast, naive SGD was signiﬁcantly sub-optimal due to bias caused by Markovian data.
Revised: In contrast, naive SGD was signiﬁcantly sub-optimal due to bias caused by Markovian data.

Original: The follow-up work Agarwal et al.
Revised: The followup work Agarwal et al.

Original: (2021) analyzed off-policy Q learning with linear function approximation and reverse experience replay to provide near-optimal convergence guarantees using the unique super martingale structure endowed by reverse experience replay.
Revised: (2021) analyzed off-policy Q learning with linear function approximation and reverse experience replay to provide near-optimal convergence guarantees using the unique super martingale structure endowed by reverse experience replay.

Original: Hong et al.
Revised: Hong et al.

Original: (2022) considers topological experience replay, which executes reverse replay over a directed graph of observed transitions.
Revised: (2022) considers topological experience replay, which executes reverse replay over a directed graph of observed transitions.

Original: Mixed with PERenables non-trivial learning in some challenging environments.
Revised: Mixed with PER enables non-trivial learning in some challenging environments.

Original: Another line of work (Florensa et al., 2017; Moore & Atkeson, 1993; Goyal et al., 2018; Schroecker et al., 2019) considers reverse sweep with access to a simulator or using a ﬁtted generative model.
Revised: Another line of work (Florensa et al., 2017; Moore & Atkeson, 1993; Goyal et al., 2018; Schroecker et al., 2019) considers reverse sweep with access to a simulator or using a ﬁtted generative model.

Original: Our work on the other hand, only seeks on-policy access to the MDP.
Revised: Our work on the other hand, only seeks on-policy access to the MDP.

Original:  B ACKGROUND AND P ROPOSED M ETHODOLOGY
Revised:  Background and Proposed Methodology

Original: We consider episodic reinforcement learning (Sutton & Barto, 2018), where at each time step an agent takes actions a t in an uncertain environment with state s t , and receives a reward r t .
Revised: We consider episodic reinforcement learning (Sutton & Barto, 2018), where at each time step an agent takes actions a t in an uncertain environment with state s t , and receives a reward r t .

Original: The environment then evolves into a new state s t +1 whose law depends only on s t , a t .
Revised: The environment then evolves into a new state s t +1 whose law depends only on s t , a t .

Original: Our goal is to (approximately)
Revised: Our goal is to (approximately)

Original: ﬁnd the policy π ∗ which maps the environmental state s to an action a such that when the agent takes the action a t = π ∗ ( s t ) , the discounted reward E [ (cid:80) ∞ t =

Revised: ﬁnd the policy π ∗ which maps the environmental state s to an action a such that when the agent takes the action a t = π ∗ ( s t ) , the discounted reward E [ (cid:80) ∞ t =

Original: γ t r t ] is maximized.
Revised: γ t r t ] is maximized.

Original: To achieve this, we consider algorithms like DQN (Mnih et al.
Revised: To achieve this, we consider algorithms like DQN (Mnih et al.

Revised: , DDPG (Lillicrap et al.

Original: and TD3 (Fujimoto et al.
Revised: and TD (Fujimoto et al.

Original: (2018)), which routinely use experience replay buffers.
Revised: (2018)), which routinely use experience replay buffers.

Original: In this paper, we introduce a new experience replay method, IER, and investigate the performance of the aforementioned RL algorithms with this modiﬁcation.
Revised: In this paper, we introduce a new experience replay method, IER, and investigate the performance of the aforementioned RL algorithms with this modiﬁcation.

Original: In this work, when we say “return”, we mean discounted episodic reward.
Revised: In this work, when we say “return”, we mean discounted episodic reward.

Original: 3. M ETHODOLOGY
Revised: 3.1 Methodology

Original: We now describe our main method in a general way where we assume that we have access to a data collection mechanism T which samples new data points.
Revised: We now describe our main method in a general way where we assume that we have access to a data collection mechanism T which samples new data points.

Original: This then appends the sampled data points to a buffer H and discards some older data points.
Revised: This then appends the sampled data points to a buffer H and discards some older data points.

Original: The goal is to run an iterative learning algorithm A , which learns from batched data of batch size B in every iteration.
Revised: The goal is to run an iterative learning algorithm A , which learns from batched data of batch size B in every iteration.

Original: We also consider an important metric I associated with the problem.
Revised: We also consider an important metric I associated with the problem.

Original: At each step, the data collection mechanism T collects a new episode and appends it to the buffer H and discards some old data points, giving us the new buffer as
Revised: At each step, the data collection mechanism T collects a new episode and appends it to the buffer H and discards some old data points, giving us the new buffer as

Original: H ← T ( H ) .
Revised: H ← T ( H ) .

Original: We then sort the entries of H based on the importance metric I and store the indices of the top G data points in an array P =
Revised: We then sort the entries of H based on the importance metric I and store the indices of the top G data points in an array P =

Original: [ P [0] , . .
Revised: [ P [0] , . .

Original: , P [ G − 1]] .
Revised: , P [ G − 1]] .

Original: Then for every index in P , we run the learning algorithm
Revised: Then for every index in P , we run the learning algorithm

Original: A with the batch D =
Revised: A with the batch D =

Original: ( H ( P [ i ]) , . . .
Revised: ( H ( P [ i ]) , . . .

Original: , H ( P [ i ] − B + 1)) .
Revised: , H ( P [ i ] − B + 1)) .

Original: In some cases, we can ‘mix’ 1 this with the standard UER sampling mechanism to reduce bias, as shown below.
Revised: In some cases, we can ‘mix’ 1 this with the standard UER sampling mechanism to reduce bias, as shown below.

Original: This amalgamation help mitigate bias that could arise from our important metric sampling curriculum.
Revised: This amalgamation help mitigate bias that could arise from our important metric sampling curriculum.

Original: We describe this procedure in Algorithm 1.
Revised: We describe this procedure in Algorithm 1.

Original: In the reinforcement learning setting, T runs an environment episode with the current policy and appends the transitions and corresponding rewards to the buffer H in the FIFO order, maintaining a total of 1 E 6 data points, usually.
Revised: In the reinforcement learning setting, T runs an environment episode with the current policy and appends the transitions and corresponding rewards to the buffer H in the FIFO order, maintaining a total of 1 E 6 data points, usually.

Original: We choose A to be an RL algorithm like TD3 or DQN.
Revised: We choose A to be an RL algorithm like TD3 or DQN or DDPG.

Original: The importance function I is the magnitude of the TD error with respect to the current Q-value estimate provided by the algorithm A .
Revised: The importance function I is the magnitude of the TD error with respect to the current Q-value estimate provided by the algorithm A .

Original: When the data collection mechanism ( T ) is the same as in UER, we will call this method IER .
Revised: When the data collection mechanism ( T ) is the same as in UER, we will call this method IER .

Original: In optimistic experience replay (OER), we take T to be the same as in UER.
Revised: In optimistic experience replay (OER), we take T to be the same as in UER.

Original: However, we query top BG data points from the buffer H and return G disjoint batches each of size B from these ‘important’ points.
Revised: However, we query top BG data points from the buffer H and return G disjoint batches each of size B from these ‘important’ points.

Original: It is clear that IER is a combination of OER and RER.
Revised: It is clear that IER is a combination of OER and RER.

Original: Notice that we can also consider the data collection mechanism like that of HER, where examples are labeled with different goals, i.e. T has now been made different, keeping the sampling process exactly same as before.
Revised: Notice that we can also consider the data collection mechanism like that of HER, where examples are labeled with different goals, i.e. T has now been made different, keeping the sampling process exactly same as before.

Original: In this case, we will call our algorithm H-IER .
Revised: In this case, we will call our algorithm H-IER .

Original: Our experiment in Enduro and Acrobat depicts an example of this successful coalition.
Revised: Our experiment in Enduro and Acrobat depicts an example of this successful coalition.

Original: We also consider the RER method, which served as a motivation for our proposed approach.
Revised: We also consider the RER method, which served as a motivation for our proposed approach.

Original: Under this sampling methodology, the batches are drawn from H in the temporally reverse direction.
Revised: Under this sampling methodology, the batches are drawn from H in the temporally reverse direction.

Original: This approach is explored in the works mentioned in Section 2.2.
Revised: This approach is explored in the works mentioned in Section 2.2.

Original: We discuss this methodology in more detail in Appendix F.
Revised: We discuss this methodology in more detail in Appendix F.

Original: 3. D IDACTIC T OY E XAMPLE
Revised: 3.2 Didactic Toy Example

Original: In this section, we discuss the working of IER on a simple environment such as GridWorld-1D, and compare this with some of our baselines such as UER, OER, RER, and IER (F).
Revised: In this section, we discuss the working of IER on a simple environment such as GridWorld-1D, and compare this with some of our baselines such as UER, OER, RER, and IER (F).

Original: In this environment, the agent lives on a discrete 1-dimensional grid of size 40 with a max-timestep of 1000 steps, and at each time step, the agent can either move left or right by one step.
Revised: In this environment, the agent lives on a discrete 1-dimensional grid of size 40 with a max-timestep of 1000 steps, and at Input: Data collection mechanism T , Data buffer H , Batch size B , grad steps per Epoch G , number of episodes N , Importance function I , learning procedure A , Uniform Mixing fraction p n ← 0 ; while n < N do n ← n + 1 ; H ← T ( H ) I ← I

Original:  The agent starts from the staring state (S; [6] ), the goal of the agent is to reach goal state (G; [40] ) getting a reward of +1 , and there is also a trap state (T; [3] ), where the agents gets a reward of − 2 .
Revised: P ; while do end each time step, the agent can either move left or right by one step. The agent starts from the starting state (S; [6] ), the goal of the agent is to reach goal state (G; [40] ) getting a reward of +1 , and there is also a trap state (T; [3] ), where the agents gets a reward of − 2 .

Original: The reward in every other state is 0 .
Revised: The reward in every other state is 0 .

Original: For simplicity, we execute an ofﬂine exploratory policy where the agent moves left or right with a probability of half and obtain a buffer of size 30000.
Revised: For simplicity, we execute an ofﬂine exploratory policy where the agent moves left or right with a probability of half and obtain a buffer of size 30000.

Original: The rewarding states occur very rarely in the buffer since it is hard to reach for this exploration policy.
Revised: The rewarding states occur very rarely in the buffer since it is hard to reach for this exploration policy.

Original: The episode ends upon meeting either of two conditions: (i) the agent reaches the terminal state, which is the goal state , or (ii) the agent has exhausted the max-timestep condition and has not succeeded in reaching any terminal state.
Revised: The episode ends upon meeting either of two conditions: (i) the agent reaches the terminal state, which is the goal state , or (ii) the agent has exhausted the max-timestep condition and has not succeeded in reaching any terminal state.

Original: An overview of our toy environment is depicted in Figure 2(a).
Revised: An overview of our toy environment is depicted in Figure 2(a).

Revised: Other hyperparameters crucial to replicating this experiment are described in Appendix B.

Original: In this example, reaching the goal state as quickly as possible is vital to receive a positive reward and avoid the fail state.
Revised: In this example, reaching the goal state as quickly as possible is vital to receive a positive reward and avoid the fail state.

Original: Therefore, it is essential to understand the paths which reach the goal state.
Revised: Therefore, it is essential to understand the paths which reach the goal state.

Original: Figure 2(b) depicts the number of times each state occurs in the buffer.
Revised: Figure 2(b) depicts the number of times each state occurs in the buffer.

Original: Furthermore, the remaining subplots of Figure 2 depict the Absolute Frequency of our off-policy algorithm trained in this environment.
Revised: Furthermore, the remaining subplots of Figure 2 depict the Absolute Frequency of our off-policy algorithm trained in this environment.

Original: A state’s “absolute frequency” is the number of times the replay technique samples a given state during the algorithm’s run.
Revised: A state’s “absolute frequency” is the number of times the replay technique samples a given state during the algorithm’s run.

Original: The experiments on this simple didactic toy environment do highlight a few interesting properties:
Revised: The experiments on this simple didactic toy environment do highlight a few interesting properties:

Original: Comparison of UER and IER:
Revised: Comparison of UER and IER:

Original: Since the goal state appears very rarely in buffer, UER and
Revised: Since the goal state appears very rarely in buffer, UER and

Original: RER rarely sample the goal state and hence do not manage to learn effectively.
Revised: RER rarely sample the goal state and hence do not manage to learn effectively.

Original: While RER naturally propagates the information about the reward back in time to the states that led to the reward, it does not often sample the rewarding state.
Revised: While RER naturally propagates the information about the reward back in time to the states that led to the reward, it does not often sample the rewarding state.

Original: Limitation of OER: While OER samples a lot from the states close to the goal state, the information about the reward does not propagate to the start state.
Revised: Limitation of OER: While OER samples a lot from the states close to the goal state, the information about the reward does not propagate to the start state.

Original: We refer to the bottleneck in Figure 2(e) where some intermediate states are not sampled.
Revised: We refer to the bottleneck in Figure 2(e) where some intermediate states are not sampled.

Original: Advantage of IER: IER prioritizes sampling from the goal state and propagates the reward backward so that the entire path leading to the reward is now aware of how to reach the reward.
Revised: Advantage of IER: IER prioritizes sampling from the goal state and propagates the reward backward so that the entire path leading to the reward is now aware of how to reach the reward.

Original: Therefore, a combination of RER and OER reduces the sampling bias in OER by preventing the bottlenecks seen in Figure 2(e).
Revised: Therefore, a combination of RER and OER reduces the sampling bias in OER by preventing the bottlenecks seen in Figure 2(e).

Original: Bottleneck of IER (F): IER (F) has a more signiﬁcant bottleneck when compared to RER and chooses to sample the non-rewarding middle states most often.
Revised: Bottleneck of IER (F): IER (F) has a more signiﬁcant bottleneck when compared to RER and chooses to sample the non-rewarding middle states most often.

Original: Also, note that whenever IER (F) chooses the goal state as the pivot, it selects the rest of the batch to overﬂow into the next episode, which begins at the starting state.
Revised: Also, note that whenever IER (F) chooses the goal state as the pivot, it selects the rest of the batch to overﬂow into the next episode, which begins at the starting state.

Original: This does not allow the algorithm to effectively learn the path which led to the goal state.
Revised: This does not allow the algorithm to effectively learn the path which led to the goal state.

Original: The toy example above models several salient features in more complicated RL environments. 
Revised: The toy example above models several salient features in more complicated RL environments. [Figure]

Original: (i
Revised: (i)

Original: In the initial stages of learning, the exploratory policy is essentially random, and such a naive exploratory policy does not often lead to non-trivial rewards.
Revised: In the initial stages of learning, the exploratory policy is essentially random, and such a naive exploratory policy does not often lead to non-trivial rewards.

Original: [Figure] Large positive  negative reward states (the goal and trap ), and their neighbors provide 
Revised: (ii) Large positive and negative reward states (the goal and trap states), and their neighbors provide the pivot state for IER and OER.

Original: We show empirically that this holds in more complicated environments as well.
Revised: We show empirically that this holds in more complicated environments as well.

Original: Figure 3 depicts the surprise vs. reward for the Ant environment.
Revised: Figure 3 depicts the surprise vs. reward for the Ant environment.

Original: Here we see a strong correlation betweenhigh absolute reward and TD error (“Surprise factor”)
Revised: Here we see a strong correlation between absolute reward and TD error (“Surprise factor”).

Original: [Figure]
Revised: [Figure]

Original: 3.5 TD Error
Revised: [Figure] [Figure] [Figure] [Figure]

Original: R
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: r d
Revised: e d

Original: [Figure]
Revised: R e w a r d (a) UER 0. 3.0 Normalized TD Error 0.

Original: [Figure]
Revised: N o r m a li z e d R e w a r d (b) RER    4 TD Error 0. R e w a r d (c) IER

Original:  U NDERSTANDING R EVERSE R EPLAY
Revised:  Understanding Reverse Replay

Original: There are various conceptual ways we can look at RER and IER.
Revised: There are various conceptual ways we can look at RER and IER.

Original: This section outlines some of the motivations behind using this technique.
Revised: This section outlines some of the motivations behind using this technique.

Original: Theoretical works such as Agarwal et al.
Revised: Theoretical works such as Agarwal et al.

Original: ; Kowshik et al.
Revised: ; Kowshik et al.

Original: (2021b) have established rigorous theoretical guarantees for RER by utilizing super martingale structures.
Revised: (2021b) have established rigorous theoretical guarantees for RER by utilizing super martingale structures.

Original: This structure is not present in forward replay techniques (i.e., the opposite of reverse replay) as shown in Kowshik et al.
Revised: This structure is not present in forward replay techniques (i.e., the opposite of reverse replay) as shown in Kowshik et al.

Original: (2021a).
Revised: (2021a).

Original: We refer to Appendix D where we show via an ablation study that going forward in time instead of reverse does not work very well.
Revised: We refer to Appendix D where we show via an ablation study that going forward in time instead of reverse does not work very well.

Original: We give the following explanations for the success of IER.
Revised: We give the following explanations for the success of IER.

Original: Propogation of Sparse Rewards: In many RL problems, non-trivial rewards are sparse and only received at the goal states.
Revised: Propogation of Sparse Rewards: In many RL problems, non-trivial rewards are sparse and only received at the goal states.

Original: Therefore, processing the data backward in time from such goal states helps the algorithm learn about the states that led to this non-trivial reward.
Revised: Therefore, processing the data backward in time from such goal states helps the algorithm learn about the states that led to this non-trivial reward.

Original: Our study (see Figureand Appendix E for further details) shows that in many environments IER picks pivots which are the states with large (positive or negative) rewards, enabling effective learning.
Revised: Our study (see Figureand Appendix E for further details) shows that in many environments IER picks pivots which are the states with large (positive or negative) rewards, enabling effective learning.

Original: Bias Reduction in OER: OER, which greedily chooses the examples with the largest TD error to learn from, performs very poorly due to bias.
Revised: Bias Reduction in OER: OER, which greedily chooses the examples with the largest TD error to learn from, performs very poorly due to bias.

Original: To illustrate this phenomenon, we refer to the didactic example in Section 3.2.
Revised: To illustrate this phenomenon, we refer to the didactic example in Section 3.2.

Original: One possible way of viewing IER is that RER is used to reduce the bias in OER.
Revised: One possible way of viewing IER is that RER is used to reduce the bias in OER.

Original: Indeed, the theoretical analysis in Agarwal et al.
Revised: Indeed, the theoretical analysis in Agarwal et al.

Original: (2021) shows that RER removes bias in
Revised: (2021) shows that RER removes bias in

Original: Q-learning type algorithms, albeit in a different sense.
Revised: Q-learning type algorithms, albeit in a different sense.

Original: PER achieves a similar bias reduction for
Revised: PER achieves a similar bias reduction for

Original: OER with a sophisticated and expensive sampling scheme over the buffer.
Revised: OER with a sophisticated and expensive sampling scheme over the buffer.

Original: Causality: MDPs have a natural causal structure: actions and events in the past inﬂuence the events in the future.
Revised: Causality: MDPs have a natural causal structure: actions and events in the past inﬂuence the events in the future.

Original: Therefore, whenever we see a surprising or an unexpected event, we can understand why or how it happened by looking into the past.
Revised: Therefore, whenever we see a surprising or an unexpected event, we can understand why or how it happened by looking into the past.

Original: Further theoretical work is needed to realize IER better.
Revised: Further theoretical work is needed to realize IER better.

Original: We further illustrate the same by referring to the straightforward didactic example (Section 3.2) where we can see the effects of each of the experience replay methods.
Revised: We further illustrate the same by referring to the straightforward didactic example (Section 3.2) where we can see the effects of each of the experience replay methods.

Original: We also demonstrate superior performance on more complicated environments (Section 5) showcasing the robustness of our approach with minimal hyperparameter tuning.
Revised: We also demonstrate superior performance on more complicated environments (Section 5) showcasing the robustness of our approach with minimal hyperparameter tuning.

Original:  E XPERIMENTAL R ESULTS
Revised:  Experimental Results

Original: In this section, we brieﬂy discuss our experimental setup as well as the results from our experiments.
Revised: In this section, we brieﬂy discuss our experimental setup as well as the results from our experiments.

Original: Environments: We evaluate our approach on a diverse class of environments, such as (i) Environments with low-dimensional state space (including classic control and Box-2D environments), (ii) Multiple joint dynamic simulation and robotics environments (including Mujoco and Robotics environments), and (iii) Human-challenging environments (such as Atari environments).
Revised: Environments: We evaluate our approach on a diverse class of environments, such as (i) Environments with low-dimensional state space (including classic control and Box-2D environments), (ii) Multiple joint dynamic simulation and robotics environments (including Mujoco and Robotics environments), and (iii) Human-challenging environments (such as Atari environments).

Original: Note that previous seminal papers in the ﬁeld of experience replay such as Mnih et al.
Revised: Note that previous seminal papers in the ﬁeld of experience replay such as Mnih et al.

Original: (2013), Schaul et al.
Revised: (2013), Schaul et al.

Original: (2015), and Andrychowicz et al.
Revised: (2015), and Andrychowicz et al.

Original: (2017) showed the efﬁcacy of their approach on a subset of these classes.
Revised: (2017) showed the efﬁcacy of their approach on a subset of these classes.

Original: For instance, UERand PER was shown to work well on Atari games.
Revised: For instance, UERand PER was shown to work well on Atari games.

Original: Furthermore, HER was effective in the Robotics environments such as FetchReach.
Revised: Furthermore, HER was effective in the Robotics environments such as FetchReach.

Original: In this work, we perform a more extensive study to show the robustness and effectiveness of our model not just in Atari and Robotics environments but also in Mujoco, Box2D, and Classic Control environments.
Revised: In this work, we perform a more extensive study to show the robustness and effectiveness of our model not just in Atari and Robotics environments but also in Mujoco, Box2D, and Classic Control environments.

Original: Due to computational limitations and the non-reproducibility of some baselines, we could not extend our experiments to some Atari environments.
Revised: Due to computational limitations and the non-reproducibility of some baselines, we could not extend our experiments to some Atari environments.

Original: We refer to Appendix A for a brief description of the environments used.
Revised: We refer to Appendix A for a brief description of the environments used.

Original: Hyperparameters: Refer to Appendix B for the exact hyperparameters used.
Revised: Hyperparameters: Refer to Appendix B for the exact hyperparameters used.

Original: Across all our experiments on various environments, we use a standard setting for all the different experience replay buffers.
Revised: Across all our experiments on various environments, we use a standard setting for all the different experience replay buffers.

Original: This classic setting is set so we can reproduce state-of-the-art performance using UER on the respective environment.
Revised: This classic setting is set so we can reproduce state-of-the-art performance using UER on the respective environment.

Original: For most of our experiments, we set the uniform mixing fraction ( p ) from
Revised: For most of our experiments, we set the uniform mixing fraction ( p ) from

Original: Algorithm 1 to be 0.
Revised: Algorithm 1 to be 0.

Original: We use a non-zero p value only for a few environments to overcome the bias while training, as described in Appendix B.
Revised: We use a non-zero p value only for a few environments to overcome the bias while training, as described in Appendix B.

Original: For PER, we use the same hyperparameters used in the
Revised: For PER, we use the same hyperparameters used in the

Original: Schaul et al.
Revised: Schaul et al.

Original: (2015) paper which had been proven robust across 50 different environments of Atari.
Revised: (2015) paper which had been proven robust across 50 different environments of Atari.

Original: Metric: To compare the different models, we use the top-k seeds moving average return as the evaluation metric across all our runs.
Revised: Metric: To compare the different models, we use the top-k seeds moving average return as the evaluation metric across all our runs.

Original: Top-k seeds here mean we take the average of k = 3 seeds that gave the best performance.
Revised: Top-k seeds here mean we take the average of k = 3 seeds that gave the best performance.

Original: It is common to use top-k trials to be selected from among many trials in the reinforcement learning literature (see Schaul et al.
Revised: It is common to use top-k trials to be selected from among many trials in the reinforcement learning literature (see Schaul et al.

Original: (2015);Schaff et al.
Revised: (2015);Schaff et al.

Original: (2019);Sarmad et al.
Revised: (2019);Sarmad et al.

Original: (2019);Wu et al.
Revised: (2019);Wu et al.

Original: (2017);Mnih et al. (2016)).
Revised: (2017);Mnih et al. (2016)).

Original: This factors in the seed sensitivity.
Revised: This factors in the seed sensitivity.

Original: Moving average with a given window size is taken for learning curves (with a window size of 20 for FetchReach and 50 for all others) to reduce the variation in return which is inherently present in each epoch.
Revised: Moving average with a given window size is taken for learning curves (with a window size of 20 for FetchReach and 50 for all others) to reduce the variation in return which is inherently present in each epoch.

Original: We argue that taking a moving average is essential since, usually, pure noise can be leveraged to pick a time instant where a given method performs best (Henderson et al., 2018).
Revised: We argue that taking a moving average is essential since, usually, pure noise can be leveraged to pick a time instant where a given method performs best (Henderson et al., 2018).

Original: Considering the top-k seed averaging of the last step performance of the moving average of the learning curves gives our metric - the top-k seeds moving average return .
Revised: Considering the top-k seed averaging of the last step performance of the moving average of the learning curves gives our metric - the top-k seeds moving average return .

Original: Comparison with SOTA: Table 2 depicts our results in various environments upon using different
Revised: Comparison with SOTA: Table 2 depicts our results in various environments upon using different

Original: SOTA replay sampler mechanisms (UER, PER and HER).
Revised: SOTA replay sampler mechanisms (UER, PER and HER).

Original: Our proposed sampler outperforms all other baselines in most tasks and compares favorably in others.
Revised: Our proposed sampler outperforms all other baselines in most tasks and compares favorably in others.

Original: Our experiments on various environments across various classes, such as classic control, Atari, etc., show that our proposed methodology consistently outperforms all other baselines in most environments, as summarized in Table 1.
Revised: Our experiments on various environments across various classes, such as classic control, Atari, etc., show that our proposed methodology consistently outperforms all other baselines in most environments, as summarized in Table 1.

Original: Furthermore, our proposed methodology is robust across various environments, as highlighted in Table 2.
Revised: Furthermore, our proposed methodology is robust across various environments, as highlighted in Table 2.

Original: The learning curves for our experiments have been depicted in the Appendix.
Revised: The learning curves for our experiments have been depicted in the Appendix.

Original: (see Appendix C)
Revised: (see Appendix C)

Original: [Table]
Revised: [Table]

Original: Forward vs. Reverse: The intuitive limitation to the "looking forward" approach is that in many
Revised: Forward vs. Reverse: The intuitive limitation to the "looking forward" approach is that in many

Original: RL problems, the objective for the agent is to reach a ﬁnal goal state, where the non-trivial reward is obtained.
Revised: RL problems, the objective for the agent is to reach a ﬁnal goal state, where the non-trivial reward is obtained.

Original: Since non-trivial rewards are only offered in this goal state, it is informative to look back from here to learn about the states that lead to this.
Revised: Since non-trivial rewards are only offered in this goal state, it is informative to look back from here to learn about the states that lead to this.

Original: When the goals are sparse, the TD error is more likely to be large upon reaching the goal state.
Revised: When the goals are sparse, the TD error is more likely to be large upon reaching the goal state.

Original: Our algorithm selects these as pivots [Table] and IER (F) might select batches overﬂowing into the next episode.
Revised: Our algorithm selects these as pivots, and IER (F) might select batches overﬂowing into the next episode.

Original: Our studies on many environments (see
Revised: Our studies on many environments (see

Original: Figure 3 and Appendix E) show that the pivot points selected based on importance indeed have large (positive or negative) rewards.
Revised: Figure 3 and Appendix E) show that the pivot points selected based on importance indeed have large (positive or negative) rewards.

Original: Our experiments depicted in Table 3 show that IER outperforms IER (F) in most environments.[Table]
Revised: Our experiments depicted in Table 3 show that IER outperforms IER (F) in most environments.

Original: [Table]
Revised: [Table]

Original: Whole vs. Component Parts: Our approach is anefﬁcient amalgamation of OER and RER.
Revised: Whole vs. Component Parts: Our approach is an amalgamation of OER and RER.

Original: Here we compare these individual parts with IER.
Revised: Here we compare these individual parts with IER.

Original: Table 4 describes the Top-k seeds Moving Average Return across various environments in this domain.
Revised: Table 4 describes the Top-k seeds Moving Average Return across various environments in this domain.

Original: As demonstrated, neither OER nor RER perform well compared to their amalgamation, IER.
Revised: As demonstrated, IER outperforms its component parts OER nor RER .

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION AND C ONCLUSIONS
Revised:  Discussion and Conclusions

Original: This section summarizes some of the exciting properties of IER discovered from our series of experiments in this paper.

Original: We also discusssome possible future steps.
Revised: We summarize our results and discuss possible future steps.

Original: Speedup:
Revised: Speedup:

Original: IER shows a signiﬁcant speedup in terms of time complexity overother state-of-the-art samplers such as PER as depicted in Table 5.
Revised: IER shows a signiﬁcant speedup in terms of time complexity over PER as depicted in Table 5.

Original: On average IER achieves a speedup improvement of 26 .
Revised: On average IER achieves a speedup improvement of 26 .

Original: 20% over PER across a large umbrella of environment classes.
Revised: 20% over PER across a large umbrella of environment classes.

Original: As the network becomes more extensive, our approach does have a higher overhead (especially computing TD error).
Revised: As the network becomes more extensive, our approach does have a higher overhead (especially computing TD error).

Original: Future work can investigate how to further reduce the computational complexity of our method by computing the TD error fewer times at the cost of operating with an older TD error.
Revised: Future work can investigate how to further reduce the computational complexity of our method by computing the TD error fewer times at the cost of operating with an older TD error.

Original: We also notice a speedup of convergence towards an optimal policy of our proposed approach, as shown on few environments.
Revised: We also notice a speedup of convergence towards an optimal policy of our proposed approach, as shown on few environments.

Original: Furthermore, the lack of speedup in some of the other experiments (even if they offer an overall performance improvement) could be due to the fact that the "surprised" pivot cannot be successfully utilized to teach the agent rapidly in the initial stages.
Revised: Furthermore, the lack of speedup in some of the other experiments (even if they offer an overall performance improvement) could be due to the fact that the "surprised" pivot cannot be successfully utilized to teach the agent rapidly in the initial stages.

Original: We refer to Appendix C for the learning curves.
Revised: We refer to Appendix C for the learning curves.

Original: [Table]
Revised: [Table]

Original: Issues with stability and consistency Picking pivot points by looking at the TD error might cause more biases and instability compared to UER as seen in some environments like
Revised: Issues with stability and consistency Picking pivot points by looking at the TD error might cause more biases and instability compared to UER as seen in some environments like

Original: HalfCheetah, LunarLander, and Ant, where there is a sudden drop in performance for some episodes (see Appendix C).
Revised: HalfCheetah, LunarLander, and Ant, where there is a sudden drop in performance for some episodes (see Appendix C).

Original: We observe that our strategy IER corrects itself quickly, unlike
Revised: We observe that our strategy IER corrects itself quickly, unlike

Original: RER, which cannot do this (see Figure 5(c)).
Revised: RER, which cannot do this (see Figure 5(e)).

Original: Increasing the number of pivot points per episode (the parameter G ) and the uniform mixing probability p usually mitigates this.
Revised: Increasing the number of pivot points per episode (the parameter G ) and the uniform mixing probability p usually mitigates this.

Original: However, these steps might slow the initial convergence to a locally optimal policy.
Revised: However, these steps might slow the initial convergence to a locally optimal policy.

Original: In this work, we do not focus on tuning these hyper-parameters since our objective was to obtain methods that require minimal hyper-parameter tuning.
Revised: In this work, we do not focus on tuning these hyper-parameters since our objective was to obtain methods that require minimal hyper-parameter tuning.

Original: However, future work can systematically investigate the signiﬁcance of these parameters in various environments.
Revised: However, future work can systematically investigate the signiﬁcance of these parameters in various environments.

Original: Why does IER outperform the traditional RER?
Revised: Why does IER outperform the traditional RER?

Original: The instability of pure RER with neural approximation has been noted in various works (Rotinov, 2019; Lee et al., 2019), where RER is stabilized by mixing it with UER .
Revised: The instability of pure RER with neural approximation has been noted in various works (Rotinov, 2019; Lee et al., 2019), where RER is stabilized by mixing it with UER .

Original: Hong et al.
Revised: Hong et al.

Original: (2022) stabilizes reverse sweep by mixing it with PER.
Revised: (2022) stabilizes reverse sweep by mixing it with PER.

Original: This is an interesting phenomenon since RER is near-optimal in the tabular and linear approximation settings (Agarwal et al., 2021).
Revised: This is an interesting phenomenon since RER is near-optimal in the tabular and linear approximation settings (Agarwal et al., 2021).

Original: Two explanations of this are i)
Revised: Two explanations of this are i)

Original: The loss function used to train the neural network is highly non-convex, which hinders the working of RER and ii)
Revised: The loss function used to train the neural network is highly non-convex, which hinders the working of RER and ii)

Original: The proof given in
Revised: The proof given in

Original: Agarwal et al.
Revised: Agarwal et al.

Original: (2021) relies extensively on ‘coverage’ of the entire state-action space - that is, the entire state-action space is visited enough number of times - which might not hold, as shown in the toy example in Section 3.2.
Revised: (2021) relies extensively on ‘coverage’ of the entire state-action space - that is, the entire state-action space is visited enough number of times - which might not hold, as shown in the toy example in Section 3.2.

Original: R EPRODUCIBILITY S TATEMENT
Revised: Reproducibility Statement

Original: In this paper we work with ten datasets, all of which are open-sourced in gym ( https://github. com/openai/gym ).
Revised: In this paper, we work with thirteen datasets, all of which are open-sourced in gym ( https:// github.com/openai/gym ).

Original: More information about the environments is available in Appendix A. We predominantly use DQN and TD3 algorithms in our research, both of which have been adapted from their open-source code.
Revised: More information about the environments is available in Appendix A. We predominantly use DQN, DDPG and TD3 algorithms in our research, both of which have been adapted from their open-source code.

Original: We also experimented with seven different replay buffer methodologies, all of which have been adapted from their source code.
Revised: We also experimented with seven different replay buffer methodologies, all of which have been adapted from their source code 2 .

Original: Most of these resources are publicly available at their open-source code 2 . More details about the models and hyperparameters are described in Appendix B. All runs have been run using the A100-SXM4-40GB, TITAN RTX, and V100 GPUs. 
Revised: More details about the models and hyperparameters are described in Appendix B. All runs have been run using the A100SXM4-40GB, TITAN RTX and V100 GPUs. Our source code is made available for additional reference 3 .

Original: A E NVIRONMENTS
Revised: A Environments

Original: For all OpenAI environments, data is summarized from https://github.com/openai/gym , and more information is provided in the wiki https://github.com/openai/gym/wiki .
Revised: For all OpenAI environments, data is summarized from https://github.com/openai/gym , and more information is provided in the wiki https://github.com/openai/gym/wiki .

Original: Below we brieﬂy describe some of the tasks we experimented on in this paper.
Revised: Below we brieﬂy describe some of the tasks we experimented on in this paper.

Original: A. C ART P OLE - V 
Revised: A.1 CartPole-v

Original: CartPole, as introduced in Barto et al.
Revised: CartPole, as introduced in Barto et al.

Original: (1983), is a task of balancing a pole on top of the cart.
Revised: (1983), is a task of balancing a pole on top of the cart.

Original: The cart has access to position and velocity as its state vector.
Revised: The cart has access to position and velocity as its state vector.

Original: Furthermore, it can go either left or right for each action.
Revised: Furthermore, it can go either left or right for each action.

Original: The task is over when the agent achieves 200 timesteps without a positive reward (balancing the pole) which is the goal state or has failed, either when (i) the cart goes out of boundaries ( ± 2.
Revised: The task is over when the agent achieves 200 timesteps without a positive reward (balancing the pole) which is the goal state or has failed, either when (i) the cart goes out of boundaries ( ± 2.

Original: units off the center), or (ii) the pole falls over (less than ± 12 deg).
Revised: units off the center), or (ii) the pole falls over (less than ± 12 deg).

Original: The agent is given a continuous 4-dimensional space describing the environment and can respond by returning one of two values, pushing the cart either right or left.
Revised: The agent is given a continuous 4-dimensional space describing the environment and can respond by returning one of two values, pushing the cart either right or left.

Original: A. A CROBOT - V 
Revised: A.2 Acrobot-v

Original: Acrobot, as introduced in Sutton (1995), is a task where the agent is given rewards for swinging a double-jointed pendulum up from a stationary position.
Revised: Acrobot, as introduced in Sutton (1995), is a task where the agent is given rewards for swinging a double-jointed pendulum up from a stationary position.

Original: The agent can actuate the second joint by one of three actions: left, right, or no torque.
Revised: The agent can actuate the second joint by one of three actions: left, right, or no torque.

Original: The agent is given a six-dimensional vector comprising the environment’s angles and velocities.
Revised: The agent is given a six-dimensional vector comprising the environment’s angles and velocities.

Original: The episode terminates when the end of the second pole is over the base.
Revised: The episode terminates when the end of the second pole is over the base.

Original: Each timestep that the agent does not reach this state gives a -1 reward, and the episode length is 500 timesteps.  
Revised: Each timestep that the agent does not reach this state gives a -1 reward, and the episode length is 500 timesteps. A.3 Pendulum-v

Revised: The inverted pendulum swingup problem, as introduced in Lillicrap et al.

Revised: (2015) is based on the classic problem in control theory.

Original: A. L UNAR L ANDER - V 
Revised: The system consists of a pendulum attached ﬁxed at one end, and free at the other end. The pendulum starts in a random position and the goal is to apply torque on the free end to swing it into an upright position, with its center of gravity right above the ﬁxed point. The episode length is 200 timesteps, and the maximum reward possible is 0, when no torque is being applied, and the object has 0 velocity remaining at an upright conﬁguration. A.4 LunarLander-v

Original: The LunarLander environment introduced in Brockman et al.
Revised: The LunarLander environment introduced in Brockman et al.

Original: (2016) is a classic rocket trajectory optimization problem.
Revised: (2016) is a classic rocket trajectory optimization problem.

Original: The environment has four discrete actions - do nothing, ﬁre the left orientation engine, ﬁre the right orientation engine, and ﬁre the main engine.
Revised: The environment has four discrete actions - do nothing, ﬁre the left orientation engine, ﬁre the right orientation engine, and ﬁre the main engine.

Original: This scenario is per Pontryagin’s maximum principle, as it is optimal to ﬁre the engine at full throttle or turn it off.
Revised: This scenario is per Pontryagin’s maximum principle, as it is optimal to ﬁre the engine at full throttle or turn it off.

Original: The landing coordinates (goal) is always at (0 , 0) .
Revised: The landing coordinates (goal) is always at (0 , 0) .

Original: The coordinates are the ﬁrst two numbers in the state vector.
Revised: The coordinates are the ﬁrst two numbers in the state vector.

Original: There are a total of 8 features in the state vector.
Revised: There are a total of 8 features in the state vector.

Original: The episode terminates if (i) the lander crashes, (ii) the lander gets outside the window, or (iii) the lander does not move nor collide with any other body.
Revised: The episode terminates if (i) the lander crashes, (ii) the lander gets outside the window, or (iii) the lander does not move nor collide with any other body.

Original: A. H ALF C HEETAH - V 
Revised: A.5 HalfCheetah-v

Original: HalfCheetah is an environment based on the work by Wawrzy´nski (2009) adapted by Todorov et al.
Revised: HalfCheetah is an environment based on the work by Wawrzy´nski (2009) adapted by Todorov et al.

Original: The HalfCheetah is a 2-dimensional robot with nine links and eight joints connecting them (including two paws).
Revised: The HalfCheetah is a 2-dimensional robot with nine links and eight joints connecting them (including two paws).

Original: The goal is to apply torque on the joints to make the cheetah run forward (right) as fast as possible, with a positive reward allocated based on the distance moved forward and a negative reward is given for moving backward.
Revised: The goal is to apply torque on the joints to make the cheetah run forward (right) as fast as possible, with a positive reward allocated based on the distance moved forward and a negative reward is given for moving backward.

Original: The torso and head of the cheetah are ﬁxed, and the torque can only be applied to the other six joints over the front and back thighs (connecting to the torso), shins (connecting to the thighs), and feet (connecting to the shins).
Revised: The torso and head of the cheetah are ﬁxed, and the torque can only be applied to the other six joints over the front and back thighs (connecting to the torso), shins (connecting to the thighs), and feet (connecting to the shins).

Original: The reward obtained by the agent is calculated as follows:
Revised: The reward obtained by the agent is calculated as follows:

Original: [Equation]
Revised: [Equation]

Revised: A.6 Ant-v

Original: Ant is an environment based on the work by Schulman et al.
Revised: Ant is an environment based on the work by Schulman et al.

Original: and adapted by Todorov et al.
Revised: and adapted by Todorov et al.

Original: The ant is a 3D robot with one torso, a free rotational body, and four legs.
Revised: The ant is a 3D robot with one torso, a free rotational body, and four legs.

Original: The task is to coordinate the four legs to move in the forward direction by applying torques on the eight hinges connecting the two links of each leg and the torso.
Revised: The task is to coordinate the four legs to move in the forward direction by applying torques on the eight hinges connecting the two links of each leg and the torso.

Original: Observations consist of positional values of different body parts of the ant, followed by the velocities of those individual parts (their derivatives), with all the positions ordered before all the velocities.
Revised: Observations consist of positional values of different body parts of the ant, followed by the velocities of those individual parts (their derivatives), with all the positions ordered before all the velocities.

Original: The reward obtained by the agent is calculated as follows:
Revised: The reward obtained by the agent is calculated as follows:

Original: [Table]
Revised: [Equation]

Original: A. H OPPER - V 
Revised: A.7 Reacher-v

Revised: The Reacher environment, as introduced in Todorov et al.

Revised: (2012), is a two-jointed robot arm.

Revised: The goal is to move the robot’s end effector (called *ﬁngertip*) close to a target that is spawned at a random positions.

Revised: The action space is a two-dimensional vector representing the torque to be applied at the two joints.

Revised: The state space consists of angular positions (in terms of cosine and sine of the angle formed by the two moving arms), coordinates, and velocity states for different body parts followed by the distance from target for the whole object.

Revised: A.8 Hopper-v

Original: The Hopper environment, as introduced in Todorov et al.
Revised: The Hopper environment, as introduced in Todorov et al.

Original: (2012), sets out to increase the number of independent state and control variables compared to classic control environments.
Revised: (2012), sets out to increase the number of independent state and control variables compared to classic control environments.

Original: The hopper is a two-dimensional ﬁgure with one leg that consists of four main body parts - the torso at the top, the thigh in the middle, the leg at the bottom, and a single foot on which the entire body rests.
Revised: The hopper is a two-dimensional ﬁgure with one leg that consists of four main body parts - the torso at the top, the thigh in the middle, the leg at the bottom, and a single foot on which the entire body rests.

Original: The goal of the environment is to make hops that move in the forward (right) direction by applying torques on the three hinges connecting the body parts.
Revised: The goal of the environment is to make hops that move in the forward (right) direction by applying torques on the three hinges connecting the body parts.

Original: The action space is a three-dimensional element vector.
Revised: The action space is a three-dimensional element vector.

Original: The state space consists of positional values for different body parts followed by the velocity states of individual parts.
Revised: The state space consists of positional values for different body parts followed by the velocity states of individual parts.

Original: A. I NVERTED D OUBLE -P ENDULUM - V 
Revised: A.9 Walker-v

Revised: The Walker environment, as builds on top of the Hopper environment introduced in Todorov et al. (2012), by adding another set of legs making it possible for the robot to walker forward instead of hop. The hopper is a two-dimensional ﬁgure with two legs that consists of four main body parts - the torso at the top, two thighs in the middle, two legs at the bottom, and two feet on which the entire body rests.

Revised: The goal of the environment is to coordinate both feel and move in the forward (right) direction by applying torques on the six hinges connecting the body parts.

Revised: The action space is a six-dimensional element vector.

Revised: The state space consists of positional values for different body parts followed by the velocity states of individual parts.

Revised: A.10 Inverted Double-Pendulum-v

Original: Inverted Double-Pendulum as introduced in Todorov et al.
Revised: Inverted Double-Pendulum as introduced in Todorov et al.

Original: (2012) is built upon the CartPole environment as introduced in Barto et al.
Revised: (2012) is built upon the CartPole environment as introduced in Barto et al.

Original: (1983), with the infusion of Mujoco.
Revised: (1983), with the infusion of Mujoco.

Original: This environment involves a cart that can be moved linearly, with a pole ﬁxed and a second pole on the other end of the ﬁrst one (leaving the second pole as the only one with one free end).
Revised: This environment involves a cart that can be moved linearly, with a pole ﬁxed and a second pole on the other end of the ﬁrst one (leaving the second pole as the only one with one free end).

Original: The cart can be pushed either left or right.
Revised: The cart can be pushed either left or right.

Original: The goal is to balance the second pole on top of the ﬁrst pole, which is on top of the cart, by applying continuous forces on the cart.
Revised: The goal is to balance the second pole on top of the ﬁrst pole, which is on top of the cart, by applying continuous forces on the cart.

Original: The agent takes a one-dimensional continuous action space in the range [-1,1], denoting the force applied to the cart and the sign depicting the direction of the force.
Revised: The agent takes a one-dimensional continuous action space in the range [-1,1], denoting the force applied to the cart and the sign depicting the direction of the force.

Original: The state space consists of positional values of different body parts of the pendulum system, followed by the velocities of those individual parts (their derivatives) with all the positions ordered before all the velocities.
Revised: The state space consists of positional values of different body parts of the pendulum system, followed by the velocities of those individual parts (their derivatives) with all the positions ordered before all the velocities.

Original: The goal is to balance the double-inverted pendulum on the cart while maximizing its height off the ground and having minimum disturbance in its velocity
Revised: The goal is to balance the double-inverted pendulum on the cart while maximizing its height off the ground and having minimum disturbance in its velocity.

Original: . A. F ETCH R EACH - V 
Revised: A.11 FetchReach-v

Original: The FetchReach environment introduced in Plappert et al.
Revised: The FetchReach environment introduced in Plappert et al.

Original: (2018) was released as part of OpenAI
Revised: (2018) was released as part of OpenAI

Original: Gym and used the Mujoco physics engine for fast and accurate simulation.
Revised: Gym and used the Mujoco physics engine for fast and accurate simulation.

Original: The goal is 3-dimensional and describes the desired position of the object.
Revised: The goal is 3-dimensional and describes the desired position of the object.

Original: Rewards in this environment are sparse and binary.
Revised: Rewards in this environment are sparse and binary.

Original: The agent obtains a reward of 0 if the target location is at the target location (within a tolerance ofcm) and − 1 otherwise.
Revised: The agent obtains a reward of 0 if the target location is at the target location (within a tolerance ofcm) and − 1 otherwise.

Original: Actions are four-dimensional, where 3 speciﬁes desired gripper movement, and the last dimension controls the opening and closing of the gripper.
Revised: Actions are four-dimensional, where 3 speciﬁes desired gripper movement, and the last dimension controls the opening and closing of the gripper.

Original: The FetchReach aims to move the gripper to a target position.
Revised: The FetchReach aims to move the gripper to a target position.

Original: A. P ONG - V 
Revised: A.12 Pong-v

Original: Pong, also introduced in Mnih et al.
Revised: Pong, also introduced in Mnih et al.

Original: (2013), is comparatively more accessible than other Atari games such as Enduro.
Revised: (2013), is comparatively more accessible than other Atari games such as Enduro.

Original: Pong is a two-dimensional sports game that simulates table tennis.
Revised: Pong is a two-dimensional sports game that simulates table tennis.

Original: The player controls an in-game paddle by moving vertically across the left and right sides of the screen.
Revised: The player controls an in-game paddle by moving vertically across the left and right sides of the screen.

Original: Players use this paddle to hit the ball back and forth.
Revised: Players use this paddle to hit the ball back and forth.

Original: The goal is for each player to reach eleven points before the opponent, where the point is earned for each time the agent returns the ball and the opponent misses.
Revised: The goal is for each player to reach eleven points before the opponent, where the point is earned for each time the agent returns the ball and the opponent misses.

Original: A. E NDURO - V 
Revised: A.13 Enduro-v

Original: Enduro, introduced in Mnih et al.
Revised: Enduro, introduced in Mnih et al.

Original: (2013), is a hard environment involving maneuvering a race car in the National Enduro, a long-distance endurance race.
Revised: (2013), is a hard environment involving maneuvering a race car in the National Enduro, a long-distance endurance race.

Original: The goal of the race is to pass a certain number of cars each day.
Revised: The goal of the race is to pass a certain number of cars each day.

Original: The agent must pass 200 cars on the ﬁrst day and 300 cars on all subsequent days.
Revised: The agent must pass 200 cars on the ﬁrst day and 300 cars on all subsequent days.

Original: Furthermore, as time passes, the visibility changes as well.
Revised: Furthermore, as time passes, the visibility changes as well.

Original: At night in the game, the player can only see the oncoming cars’ taillights.
Revised: At night in the game, the player can only see the oncoming cars’ taillights.

Original: As the days’ progress, cars will become more challenging to avoid.
Revised: As the days’ progress, cars will become more challenging to avoid.

Original: Weather and time of day are factors in how to play.
Revised: Weather and time of day are factors in how to play.

Original: During the day, the player may drive through an icy patch on the road, which would limit control of the vehicle, or a patch of fog may reduce visibility.
Revised: During the day, the player may drive through an icy patch on the road, which would limit control of the vehicle, or a patch of fog may reduce visibility.

Original: B M ODEL AND H YPERPARAMETERS
Revised: B Model and Hyperparameters

Original: In this paper [Table] we work with two classes of algorithms: DQN and TD3.
Revised: In this paper, we work with two classes of algorithms: DQN, DDPG and TD3.

Original: The hyperparameters used for training our DQN algorithms in various environments are described in Table 6. 
Revised: The hyperparameters used for training our DQN algorithms in various environments are described in Table 6. The hyperparameters used for training our DDPG algorithms in various environments are described in Table 7. The hyperparameters used for training DDPG are described in Table 7.

Original: Furthermore, the hyperparameters used for training TD3 are described in Table
Revised: Furthermore, the hyperparameters used for training TD3 are described in Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: 5 e −5 e −5 e −1 e

Original: [Figure] [Table] [Figure] [Table]{+−
+}
Revised: {+−
+}

Original: p
Revised: e

Original: [Table] [Figure]
Revised: −lr IERSpeciﬁcSettings

Original: [Table] [Figure] [Table] [Figure]
Revised: UseHindsightforstoringstates − (cid:88) − − (cid:88) use_hindsight MixingFactor(p)     p

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: OptimizerSettings PolicyLearningrate

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table]
Revised: QFLearningrate

Original: [Figure]
Revised: [Figure]

Original: −3 e −1 e −pol_lr QF Learning rate 1 e −1 e −1 e −1 e −1 e −qf_lr

Original: IER Speciﬁc Settings
Revised: IERSpeciﬁcSettings

Original: Use Hindsight for storing states − (cid:88) − − − use_hindsight Mixing Factor (p)  0.0 . 
Revised: UseHindsightforstoringstates − − − (cid:88) − − − use_hindsight MixingFactor(p)  0 . [Figure] [Figure] [Figure]

Original: p
Revised: p

Original: C A DDITIONAL R ESULTS
Revised: Additionally, we use Tabular MDPs for learning a policy in our toy example. Since the environment is fairly simpler, and has very few states, function approximation is unnecessary. For all the agents trained on GridWorld, we use a common setting as described in Table 9. [Table]

Original: This section discusses a few additional results from our experiments on various environments.
Revised: [Figure] [Table] [Figure] [Table]

Original: C. P ERFORMANCE OF IER WITH L OW -D IMENSIONAL S TATE S PACE
Revised: C Additional Results C.1 Performance of IER with Low-Dimensional State Space

Original: This section brieﬂy discusses our results on environments with a low-dimensional state space, such as classic control environments (CartPole and Acrobot) and Box-2D environments (LunarLander).
Revised: This section brieﬂy discusses our results on environments with a low-dimensional state space, such as classic control environments (CartPole, Acrobot, and Pendulum) and Box-2D environments (LunarLander).

Original: Figure 4 depicts the learning curves of our DQN agents in these environments.
Revised: Figure 4 depicts the learning curves of our DQN/DDPG agents in these environments.

Original: [Figure]
Revised: [Figure]

Original: We note that our proposed methodology can signiﬁcantly outperform other baselines for the classic control algorithms.
Revised: We note that our proposed methodology can signiﬁcantly outperform other baselines for the classic control algorithms.

Original: Furthermore, Figure 4(a) shows excellent promise as we achieve a near-perfect score across all seeds in one-tenth of the time it took to train PER.
Revised: Furthermore, Figure 4(a) shows excellent promise as we achieve a near-perfect score across all seeds in one-tenth of the time it took to train PER.

Original: C. P ERFORMANCE IN M ULTIPLE J OINT D YNAMICS S IMULATION AND R OBOTICS E NVIRONMENTS
Revised: C.2 Performance in Multiple Joint Dynamics Simulation and Robotics Environments

Original: Multiple joint dynamic simulation environments (mujoco physics environments) and robotics environments such as HalfCheetah, Ant, Inverted Double-Pendulum, and FetchReach (Todorov et al.
Revised: Multiple joint dynamic simulation environments (mujoco physics environments) and robotics environments such as HalfCheetah, Ant, Inverted Double-Pendulum, and FetchReach (Todorov et al.

Original: Plappert et al.
Revised: Plappert et al.

Original: (2018)) are more complex and enable us to study whether the agent can understand the physical phenomenon of real-world environments.
Revised: (2018)) are more complex and enable us to study whether the agent can understand the physical phenomenon of real-world environments.

Original: Figure 5 depicts the learning curves of our
Revised: Figure 5 depicts the learning curves of our

Original: TD3 agents in these environments.
Revised: TD3 agents in these environments.

Original: Again, our proposed methodology outperforms all other baselines signiﬁcantly in most of the environments studied in this section.
Revised: Again, our proposed methodology outperforms all other baselines signiﬁcantly in most of the environments studied in this section.

Original: Additionally, it is essential to point out that our proposed method shows an impressive speedup of convergence in Inverted Double
Revised: Additionally, it is essential to point out that our proposed method shows an impressive speedup of convergence in Inverted Double

Original: Pendulum and convergence to a much better policy in Ant.
Revised: Pendulum and convergence to a much better policy in Ant.

Original: C. P ERFORMANCE OF IER IN H UMAN C HALLENGING E NVIRONMENTS
Revised: C.3 Performance of IER in Human Challenging Environments

Original: This section brieﬂy discusses our results on human-challenging environments such as Atari environments (Pong and Enduro).
Revised: This section brieﬂy discusses our results on human-challenging environments such as Atari environments (Pong and Enduro).

Original: These environments are highly complex, and our algorithms take millions of steps to converge to a locally optimal policy.
Revised: These environments are highly complex, and our algorithms take millions of steps to converge to a locally optimal policy.

Original: Figure 6 depicts the learning curves of our
Revised: Figure 6 depicts the learning curves of our

Original: DQN agents in these environments.
Revised: DQN agents in these environments.

Original: We note that our proposed methodology can perform favorably when compared to other baselines for the Atari environments and can reach large reward policies signiﬁcantly faster than UER. 
Revised: We note that our proposed methodology can perform favorably when compared to other baselines for the Atari environments and can reach large reward policies signiﬁcantly faster than UER. [Figure]

Original: C. W HOLE VS . C OMPONENT P ARTS
Revised: C.4 Whole vs. Component Parts

Original: This section brieﬂy presents the learning curves of our models on three different sampling schemes:
Revised: This section brieﬂy presents the learning curves of our models on three different sampling schemes:

Original: IER, OER and RER.
Revised: IER, OER and RER.

Revised: C.5 Buffer Batch size sensitivity of IER

Revised: This section brieﬂy presents the sensitivity to the buffer batch size hyperparameter for our proposed approach (IER).

Revised: To analyze this, we run our experiments on the CartPole environment with varying batch size of the range 2-256.

Revised: Table ??

Revised: and Figure 8 depict the buffer batch size sensitivity results from our proposed sampler.

Original: [Figure] 
Revised: [Figure] [Figure] [Table]

Revised: C.6 How important is sampling pivots?

Revised: This section brieﬂy presents the ablation study to analyze the importance of sampling “surprising” states as pivots.

Revised: As a baseline, we build a experience replay where these pivots are randomly sampled from the buffer.

Revised: The "looking back" approach is used to create batches of data.

Revised: For nomenclature, we refer to our proposed approach (IER) to use the “TD Metric” sampling of pivots, and the baseline that uses “Uniform” sampling of pivots.

Revised: Table 11 and Figure 9 depict the buffer batch size sensitivity results from our proposed sampler.

Revised: [Table]

Revised: [Table]

Revised: C.7 How important is “looking back”?

Revised: This section brieﬂy presents the ablation study to analyze the importance of “looking back” after sampling pivots.

Revised: As a baseline, we build a experience replay where we sample uniformly instead of looking back.

Revised: For nomenclature, we refer to our proposed approach (IER) to use the “Looking Back” approach (similar to IER), and the baseline that uses “Uniform” approach.

Revised: We refer to these two approaches as possible ﬁlling schemes, i.e. ﬁll the buffer with states once the pivot state is sampled.

Revised: Table 12 and Figure 10 depict the buffer batch size sensitivity results from our proposed sampler.

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: D A BLATION STUDY OF T EMPORAL EFFECTS
Revised: D Ablation study of Temporal effects

Original: This section studies the ablation effects of going temporally forward and backward once we choose a pivot/surprise point.
Revised: This section studies the ablation effects of going temporally forward and backward once we choose a pivot/surprise point.

Original: Furthermore, Figure 8 depicts the learning curves of the two proposed methodologies.
Revised: Furthermore, Figure 11 depicts the learning curves of the two proposed methodologies.

Original: We notice that against theoretical intuition, the unbiased forward sampling scheme is worse in most environments compared to the reverse sampling scheme. [Figure]
Revised: We notice that against theoretical intuition, the unbiased forward sampling scheme is worse in most environments compared to the reverse sampling scheme

Original: E S PARSITY AND R EWARDS OF S URPRISING S TATES E. S URPRISING S TATES H AVE L ARGE R EWARDS
Revised: E Sparsity and Rewards of Surprising States E.1 Surprising States Have Large Rewards

Original: In this section, we study the “learning from sparse reward” intuition provided in Section 4 – i.e., we want to check if the states corresponding to large TD error correspond to states with large (positive or negative) rewards.
Revised: In this section, we study the “learning from sparse reward” intuition provided in Section 4 – i e., we want to check if the states corresponding to large TD error correspond to states with large (positive or negative) rewards.

Original: To test the hypothesis, we consider a sampled buffer and plot the TD error of these  points in the buffer against the respective reward.
Revised: To test the hypothesis, we consider a sampled buffer and plot the TD error of these [Figure] points in the buffer against the respective reward.

Original: Figure 3 shows the distribution of TD error against reward for the sampled buffers in the Ant environment.
Revised: Figure 3 shows the distribution of TD error against reward for the sampled buffers in the Ant environment.

Original: We see that high reward states (positive or negative) also have higher TD errors.
Revised: We see that high reward states (positive or negative) also have higher TD errors.

Original: Therefore, our algorithm picks large reward states as endpoints to learn in such environments.
Revised: Therefore, our algorithm picks large reward states as endpoints to learn in such environments.

Original: E. S URPRISING S TATES ARE S PARSE AND I SOLATED
Revised: E.2 Surprising States are Sparse and Isolated

Original: Figure 9 and Figure 10 depict the distribution of “surprise”/TD error in a sampled batch for CartPole and Ant environments respectively.
Revised: Figure 12 and Figure 13 depict the distribution of “surprise”/TD error in a sampled batch for CartPole and Ant environments respectively.

Original: These two ﬁgures help show that the states with a large “surprise” factor are few and that even though the pivot of a buffer has a large TD error, the rest of the buffer typically does not.
Revised: These two ﬁgures help show that the states with a large “surprise” factor are few and that even though the pivot of a buffer has a large TD error, the rest of the buffer typically does not.

Original: Figure 9(d) and Figure 10(d) show a magniﬁed view of Figure 9(c) and Figure 10(d) where the pivot point selected is dropped.
Revised: Figure 12(d) and Figure 13(d) show a magniﬁed view of Figure 12(c) and Figure 13(d) where the pivot point selected is dropped.

Original: This helps with a uniform comparison with the remaining timesteps within the sampled buffer.
Revised: This helps with a uniform comparison with the remaining timesteps within the sampled buffer.

Original: Again, we notice little correlation between the timesteps within the sampled buffer. 
Revised: Again, we notice little correlation between the timesteps within the sampled buffer. [Figure]

Original: [Figure]
Revised: [Figure]

Original: F R EVERSE E XPERIENCE R EPLAY (RER)
Revised: F Reverse Experience Replay (RER)

Original: This section discusses our implementation of Reverse Experience Replay (RER), which served as a motivation for our proposed approach.
Revised: This section discusses our implementation of Reverse Experience Replay (RER), which served as a motivation for our proposed approach.

Original: The summary of the RER approach is shown in Figure 11.
Revised: The summary of the RER approach is shown in Figure 14.

Original: Furthermore, an overview of our implemented approach to RER is described brieﬂy in Algorithm 2.[Figure] [Figure]
Revised: Furthermore, an overview of our implemented approach to RER is described brieﬂy in Algorithm 2.

Original: [Figure]
Revised: [Figure]

Original: Input collection mechanism  buffer  Batch  B  grad  , number  n ← N ;
Revised: Input: Data collection mechanism T , Data buffer H , Batch size B , grad steps per Epoch G , number of episodes N , learning procedure A n ← N ;

Original: P ← len ( H ) ; // Set index to last element of Buffer H while n < N do n ← n + 1 ;
Revised: P ← len ( H ) ; // Set index to last element of Buffer H while n < N do n ← n + 1 ;

Original: H ← T ( H ) ; // Add a new episode to the buffer g ← 0 ; while g < G do if P − B < 0 then
Revised: H ← T ( H ) ; // Add a new episode to the buffer g ← 0 ; while g < G do if P − B < 0 then

Original: P ← len ( H ) ; // Set index to last element of Buffer H else
Revised: P ← len ( H ) ; // Set index to last element of Buffer H else

Original: P ← P − B ; end
Revised: P ← P − B ; end

Original: D ← H
Revised: D ← H

Original: [ P − B, P ] ; // Load batch of previous B samples from index P g ←  ; A  ; // Run the learning algorithm with batch data D end end
Revised: [ P − B, P ] ; // Load batch of previous B samples from index P g ← g + 1 ; A ( D ) ; // Run the learning algorithm with batch data D end end