Original: L EARNING FROM DEMONSTRATIONS WITH SACR2: S OFT A CTOR -C RITIC WITH R EWARD R ELABELING
Revised: L EARNING FROM DEMONSTRATIONS WITH SACR2: S OFT A CTOR -C RITIC WITH R EWARD R ELABELING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: During recent years, deep reinforcement learning (DRL) has made successful incursions into complex decision-making applications such as robotics, autonomous driving or video games.
Revised: During recent years, deep reinforcement learning (DRL) has made successful incursions into complex decision-making applications such as robotics, autonomous driving or video games.

Original: However, a well-known caveat of DRL algorithms is their inefﬁciency, requiring huge amounts of data to converge.

Original: Off-policy algorithms tend to be more sample-efﬁcient, and can additionally beneﬁt from any off-policy data stored in the replay buffer.
Revised: Off-policy algorithms tend to be more sample-efficient than their on-policy counterparts, and can additionally benefit from any off-policy data stored in the replay buffer.

Original: Expert demonstrations are a popular source for such data: the agent is exposed to successful states and actions early on, which can accelerate the learning process and improve performance.
Revised: Expert demonstrations are a popular source for such data: the agent is exposed to successful states and actions early on, which can accelerate the learning process and improve performance.

Original: In the past, multiple ideas have been proposed to make good use of the demonstrations in the buffer, such as pretraining on demonstrations only or minimizing additional cost functions.
Revised: In the past, multiple ideas have been proposed to make good use of the demonstrations in the buffer, such as pretraining on demonstrations only or minimizing additional cost functions.

Original: We carry on a study to evaluate several of these ideas in isolation, to see which of them have the most signiﬁcant impact.
Revised: We carry on a study to evaluate several of these ideas in isolation, to see which of them have the most significant impact.

Original: We also present a new method, based on a reward bonus given to demonstrations and successful episodes.
Revised: We also present a new method, based on a reward bonus given to demonstrations and successful episodes.

Original: First, we give a reward bonus to the transitions coming from demonstrations to encourage the agent to match the demonstrated behaviour.
Revised: First, we give a reward bonus to the transitions coming from demonstrations to encourage the agent to match the demonstrated behaviour.

Original: Then, upon collecting a successful episode, we relabel its transitions with the same bonus before adding them to the replay buffer, encouraging the agent to also match its previous successes.
Revised: Then, upon collecting a successful episode, we relabel its transitions with the same bonus before adding them to the replay buffer, encouraging the agent to also match its previous successes.

Original: The base algorithm for our experiments is the popular Soft Actor-Critic (SAC), a state-of-the-art off-policy algorithm for continuous action spaces.
Revised: The base algorithm for our experiments is the popular Soft Actor-Critic (SAC), a state-of-the-art off-policy algorithm for continuous action spaces.

Original: Our experiments focus on  robotics, speciﬁcally on a  reaching task for a robotic arm in simulation.
Revised: Our experiments focus on manipulation robotics, specifically on a 3D reaching task for a robotic arm in simulation.

Original: We show that our method SACR2 based on reward relabeling improves the performance on this task, even in the absence of demonstrations.
Revised: We show that our method SACR2 based on reward relabeling improves the performance on this task, even in the absence of demonstrations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite having known great success, reinforcement learning algorithms have yet to prove that they can consistently produce good results across different domains.
Revised: Despite having known great success, reinforcement learning algorithms have yet to prove that they can consistently produce good results across different domains.

Original: Works like Engstrom et al.
Revised: Works like Engstrom et al.

Original: (2020) show that we still don’t know with certainty what are the things that matter the most in current algorithms.
Revised: (2020) show that we still don’t know with certainty what are the things that matter the most in current algorithms.

Original: Other challenges such as reproducibility issues have brought some scepticism to the ﬁeld.
Revised: Other challenges such as reproducibility issues have brought some scepticism to the field.

Original: Two of the main challenges in reinforcement learning are reward shaping and sample efﬁciency.
Revised: Two of the main challenges in reinforcement learning are reward shaping and sample efficiency.

Original: Reward shaping makes it very difﬁcult to translate results from one task to another, and often relies on the intuition of the designer rather than a robust methodology.
Revised: Reward shaping makes it very difficult to translate results from one task to another, and often relies on the intuition of the designer rather than a robust methodology.

Original: Sample efﬁcient algorithms are required to obtain faster and more reliable results, particularly in robotics where it is much harder to deploy an algorithm outside of simulation.
Revised: Sample efficient algorithms are required to obtain faster and more reliable results, particularly in robotics where it is much harder to deploy an algorithm outside of simulation.

Original: Recent progress has enabled some deployment to real robots, but there are still issues to navigate such as safety orheavy human intervention (Gupta et al., 2021).
Revised: Recent progress has enabled some deployment to real robots, but there are still issues to consider such as safety or human intervention (Gupta et al., 2021).

Original: The recent trend towards more data-driven algorithms could be a solution to both of these problems.
Revised: The recent trend towards more data-driven algorithms could be a solution to both of these problems.

Original: Indeed, additional data can alleviate the need for online data collection, and demonstration data can guide the agent to good behaviours with a simple task-agnostic reward function.
Revised: Indeed, additional data can alleviate the need for online data collection, and demonstration data can guide the agent to good behaviours with a simple task-agnostic reward function.

Original: In this work, we focus on a reaching task with sparse rewards, and aim to identify what are the best methods that leverage expert demonstrations to improve sample efﬁciency.
Revised: In this work, we focus on a reaching task with sparse rewards, and aim to identify what are the best methods that leverage expert demonstrations to improve sample efficiency.

Original: We focus on generic task-agnostic methods that can be applied to any off-policy algorithm with some minor modiﬁcations.
Revised: We focus on generic task-agnostic methods that can be applied to any off-policy algorithm with some minor modifications.

Original: We also present a new such method, which is based on the observation that, in hindsight, a successful episode of collected experience is in fact a demonstration, so it should receive the same treatment.
Revised: We also present a new such method, which is based on the observation that, in hindsight, a successful episode of collected experience is in fact a demonstration, so it should receive the same treatment.

Original: In particular, we propose to add a reward bonus to all transitions coming from both demonstrations and succesful episodes.
Revised: In particular, we propose to add a reward bonus to all transitions coming from both demonstrations and succesful episodes.

Original: We instantiate our approach with Soft Actor-Critic (SAC) (Haarnoja et al., 2018), and compare it to two other algorithms, SACfD (Vecer´ık et al., 2017) and
Revised: We instantiate our approach with Soft Actor-Critic (SAC) (Haarnoja et al., 2018), and compare it to two other algorithms, SACfD (Vecer´ık et al., 2017) and

Original: SACBC (Nair et al., 2018), presented in detail in section 2.
Revised: SACBC (Nair et al., 2018), presented in detail in section 2.

Original: The contributions of this paper are the following:
Revised: The contributions of this paper are the following:

Original: • We present an ablation study of existing methods that improve sample efﬁciency by leveraging demonstrations in off-policy reinforcement learning algorithms.
Revised: • We present an ablation study of existing methods that improve sample efficiency by leveraging demonstrations in off-policy reinforcement learning algorithms.

Original: • We introduce a new such method that consists on giving a reward bonus to demonstrations and relabeling successful episodes as demonstrations.
Revised: • We introduce a new such method that consists on giving a reward bonus to demonstrations and relabeling successful episodes as demonstrations.

Original: Our approach is among the methods with the highest performance increase, and is a component of the ﬁnal algorithm that achieves the best results.
Revised: Our approach is among the methods with the highest performance increase, and is a component of the final algorithm that achieves the best results.

Original: It also consistently solves the task without demonstrations, whereas SAC fails to do so quite often.
Revised: It also consistently solves the task without demonstrations, whereas SAC often fails.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Reinforcement Learning (RL) and Imitation Learning (IL) are the two most popular paradigms to solve decision-making tasks within machine learning.
Revised: Reinforcement Learning (RL) and Imitation Learning (IL) are the two most popular paradigms to solve decision-making tasks within machine learning.

Original: In classical RL, the data is collected during training by interacting with the environment, which provides a reward signal that we try to maximize.
Revised: In classical RL, the data is collected during training by interacting with the environment, which provides a reward signal that we try to maximize.

Original: In IL, the data is generally collected before training, and it consists of expert trajectories of a behaviour that we wish to copy or imitate.
Revised: In IL, the data is generally collected before training, and it consists of expert trajectories of a behaviour that we wish to copy or imitate.

Original: Learning for robotics.
Revised: Learning for robotics.

Original: In this paper, we focus on general-purpose RL algorithms that can be applied to any problem out of the box.
Revised: In this paper, we focus on general-purpose RL algorithms that can be applied to any problem out of the box.

Original: However, roboticists have for a long time pursued more sample-efﬁcient algorithms that could be deployed in the real world.
Revised: However, roboticists have for a long time pursued more sample-efficient algorithms that could be deployed in the real world.

Original: Outside of RL, Transporters Networks (Zeng et al., 2020) exploit spatial symmetries and RGB-D sensors to solve a variety of tasks from just a few samples, and NTP (Xu et al., 2017) and the more recent NTG (Huang et al., 2019) use clever representations to execute a hierarchy of movement primitives that solve complex sequential tasks from demonstrations.
Revised: Outside of RL, Transporters Networks (Zeng et al., 2020) exploit spatial symmetries and RGB-D sensors to solve a variety of tasks from just a few samples, and NTP (Xu et al., 2017) and the more recent NTG (Huang et al., 2019) use clever representations to execute a hierarchy of movement primitives that solve complex sequential tasks from demonstrations.

Original: The recent C2F-ARM from James et al.
Revised: The recent C2F-ARM from James et al.

Original: (2021) relies on RL, point cloud inputs, and a clever pre-processing pipeline to achieve impressive results from just a handful of demonstrations.
Revised: (2021) relies on RL, point cloud inputs, and a clever pre-processing pipeline to achieve impressive results from just a handful of demonstrations.

Original: Many successful algorithms for robotics have come from model-based RL, since they can learn from taskagnostic data.
Revised: Many successful algorithms for robotics have come from model-based RL, since they can learn from taskagnostic data.

Original: Earlier works proposed dynamics models over latent representation spaces, which could be learnt from a reconstruction objective (Finn et al., 2016b), or directly from their ability to produce models that accurately explain the observed data (Zhang et al., 2019).
Revised: Earlier works proposed dynamics models over latent representation spaces, which could be learnt from a reconstruction objective (Finn et al., 2016b), or directly from their ability to produce models that accurately explain the observed data (Zhang et al., 2019).

Original: More recently, we have also seen models learnt directly in image space, such as Finn & Levine (2017), or Schmeckpeper et al. 
Revised: More recently, works such as Finn & Levine (2017) or Schmeckpeper et al. (2020) successfully learnt a model directly in image space.

Original: More generally, the recent trend of ofﬂine RL has brought algorithms able to leverage huge amounts of data, expert or not, such as QT-Opt (Kalashnikov et al., 2018) or the most recent multi-task version MT-Opt (Kalashnikov et al., 2021).
Revised: More generally, the recent trend of offline RL has brought algorithms able to leverage huge amounts of data, expert or not, such as QT-Opt (Kalashnikov et al., 2018) or the most recent multi-task version MT-Opt (Kalashnikov et al., 2021).

Original: Relabeling past experience.
Revised: Relabeling past experience.

Original: Since off-policy RL algorithms can theoretically use data coming from any policy, a natural idea was to share data between tasks in a multi-task setting.
Revised: Since off-policy RL algorithms can theoretically use data coming from any policy, a natural idea was to share data between tasks in a multi-task setting.

Original: An even better idea came in Hindsight
Revised: An even better idea came in Hindsight

Original: Experience Replay (HER) (Andrychowicz et al., 2017), where the authors pointed out that if we accidentally solve one task when trying to perform another task, that experience is still optimal if we relabel the goal that was initially intended.
Revised: Experience Replay (HER) (Andrychowicz et al., 2017), where the authors pointed out that if we accidentally solve one task when trying to perform another task, that experience is still optimal if we relabel the goal that was initially intended.

Original: Similar and more general works followed, such as
Revised: Similar and more general works followed, such as

Original: GCSL (Ghosh et al., 2019), Generalized Hindsight (Li et al., 2020), and HIPI (Eysenbach et al., 2020), which reframes the relabeling problem as inverse RL.
Revised: GCSL (Ghosh et al., 2019), Generalized Hindsight (Li et al., 2020), and HIPI (Eysenbach et al., 2020), which reframes the relabeling problem as inverse RL.

Original: RCP (Kumar et al., 2019) extended the idea to the single-task setting, by learning a policy conditioned on the trajectory return: trajectories collected from sub-optimal policies can be viewed as optimal supervision for matching the reward of the given trajectory.
Revised: RCP (Kumar et al., 2019) extended the idea to the single-task setting, by learning a policy conditioned on the trajectory return: trajectories collected from sub-optimal policies can be viewed as optimal supervision for matching the reward of the given trajectory.

Original: Learning from demonstrations.
Revised: Learning from demonstrations.

Original: The most straight-forward variant of IL is behaviour cloning (BC), where we directly look for a policy that acts like the expert by solving a supervised learning problem.
Revised: The most straight-forward variant of IL is behaviour cloning (BC), where we directly look for a policy that acts like the expert by solving a supervised learning problem.

Original: This approach suffers from a series of issues, namely compounding errors that can put the agent into states where it can no longer recover, because they lie outside the training state distribution.
Revised: This approach suffers from a series of issues, namely compounding errors that can put the agent into states where it can no longer recover, because they lie outside the training state distribution.

Original: However, BC has seen success in a variety of complex applications such as autonomous driving (Bojarski et al algorithms have been proposed, like Dagger from Ross et al.
Revised: However, BC has seen success in a variety of complex applications such as autonomous driving (Bojarski et al., 2016). More robust algorithms have been proposed, like Dagger from Ross et al.

Original: (2011), which does however require that the expert is available during training.
Revised: (2011), which does however require that the expert is available during training.

Original: Another variant of IL which has known great success is inverse RL (IRL), where we try to infer the reward function that the expert was most likely trying to maximize, while optionally jointly learning a policy.
Revised: Another variant of IL which has known great success is inverse RL (IRL), where we try to infer the reward function that the expert was most likely trying to maximize, while optionally jointly learning a policy.

Original: IRL algorithms beneﬁt from online data collection, but assume that the reward signal from the environment is unknown.
Revised: IRL algorithms benefit from online data collection, but assume that the reward signal from the environment is unknown.

Original: The most recent
Revised: The most recent

Original: IRL algorithms that can handle complex behaviours are based on adversarial optimization, such as
Revised: IRL algorithms that can handle complex behaviours are based on adversarial optimization, such as

Original: GCL from Finn et al.
Revised: GCL from Finn et al.

Original: (2016a), which presents a similar idea to the well-known GAIL from Ho &
Revised: (2016a), which presents a similar idea to the well-known GAIL from Ho &

Original: Ermon (2016).
Revised: Ermon (2016).

Original: Learning from both demonstrations and reinforcement learning.
Revised: Learning from both demonstrations and reinforcement learning.

Original: Demonstrations can be used to design the reward, guide exploration, augment the training data, initialize policies, etc.
Revised: Demonstrations can be used to design the reward, guide exploration, augment the training data, initialize policies, etc.

Original: In NAC (Gao et al., 2018), the demonstrations, which can be sub-optimal, are used as the only training data during the ﬁrst iterations.
Revised: In NAC (Gao et al., 2018), the demonstrations, which can be sub-optimal, are used as the only training data during the first iterations.

Original: In Zhu et al.
Revised: In Zhu et al.

Original: the demonstrations are used to augment the manually designed task reward with an imitation-based reward.
Revised: the demonstrations are used to augment the manually designed task reward with an imitation-based reward.

Original: In DAPG (Rajeswaran et al., 2017) the demonstrations are used twice: to pretrain with behavior cloning, and to augment the policy gradient equation.
Revised: In DAPG (Rajeswaran et al., 2017) the demonstrations are used twice: to pretrain with behavior cloning, and to augment the policy gradient equation.

Original: In DAC (Liu et al., 2020), they introduce a novel objective based on an augmented reward, the larger the closer the policy to the expert policy.
Revised: In DAC (Liu et al., 2020), they introduce a novel objective based on an augmented reward, the larger the closer the policy to the expert policy.

Original: We will focus on three algorithms that can be applied to any continous-action off-policy algorithm with very minor modiﬁcations.
Revised: We will focus on three algorithms that can be applied to any continous-action off-policy algorithm with very minor modifications.

Original: SACfD (Vecer ´ ık et al., 2017) (originally DDPGfD based on DDPG) introduces three main ideas: transitions from demonstrations are added to the replay buffer, prioritized replay is used for sampling transitions (demonstration data is given a bonus to be sampled more often), and a mix of 1-step and n-step return losses are used.
Revised: SACfD (Vecer ´ ık et al., 2017) (originally DDPGfD based on DDPG) introduces three main ideas: transitions from demonstrations are added to the replay buffer, prioritized replay is used for sampling transitions (demonstration data is given a bonus to be sampled more often), and a mix of 1-step and n-step return losses are used.

Original: SACBC (Nair et al., 2018) (has no name and originally based on DDPG) also introduces three main ideas: transitions from demonstrations are added to a separate additional replay buffer, an auxiliary behaviour cloning loss is applied to samples from this buffer, and some episodes are reset to a state sampled uniformly from a demonstration.
Revised: SACBC (Nair et al., 2018) (has no name and originally based on DDPG) also introduces three main ideas: transitions from demonstrations are added to a separate additional replay buffer, an auxiliary behaviour cloning loss is applied to samples from this buffer, and some episodes are reset to a state sampled uniformly from a demonstration.

Original: The authors additionally present other ideas regarding the multi-goal setting which we won’t cover in this work.
Revised: The authors additionally present other ideas regarding the multi-goal setting which we won’t cover in this work.

Original: SQIL (Reddy et al., 2020) is actually a pure IL algorithm, since the reward signal is supposed unknown, but we present it here since it also incorporates demonstration data into the replay buffer.
Revised: SQIL (Reddy et al., 2020) is actually a pure IL algorithm, since the reward signal is supposed unknown, but we present it here since it also incorporates demonstration data into the replay buffer.

Original: The replay buffer is initially ﬁlled with demonstrations where the rewards are always r = 1 , and new experiences collected by the agent are added with reward r = 0 .
Revised: The replay buffer is initially filled with demonstrations where the rewards are always r = 1 , and new experiences collected by the agent are added with reward r = 0 .

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Reinforcement learning.
Revised: Reinforcement learning.

Original: In reinforcement learning an agent interacts with an environment by performing an action and observing a feedback signal (reward r ) and the new state of the environment.
Revised: In reinforcement learning, an agent interacts with an environment by performing an action and observing a feedback signal (reward r ) and the new state of the environment.

Original: The goal is to ﬁnd the policy (function mapping states s to actions a ) that maximizes the discounted cumulative reward:
Revised: The goal is to find the policy (function mapping states s to actions a ) that maximizes the discounted cumulative reward:

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the Q-function Q π ( s t , a t )
Revised: We define the Q-function Q π ( s t , a t )

Original: =
Revised: =

Original: T (cid:80)
Revised: T (cid:80)

Original: [Equation]
Revised: [Equation]

Original:  as the reward-to-go from the state s t if we pick the action a t and then follow π .
Revised: t r ( s t ′ , [Equation] t ′ ) | s t , a t (cid:105) as the reward-to-go from the state s t if we pick the action a t and then follow π .

Original: Soft Actor-Critic.
Revised: Soft Actor-Critic.

Original: In reinforcement learning, the optimal policy is always deterministic under full observability, but stochastic policies have interesting properties: better exploration and robustness (due to wider coverage of states), and multi-modality.
Revised: In reinforcement learning, the optimal policy is always deterministic under full observability, but stochastic policies have interesting properties: better exploration and robustness (due to wider cover- age of states), and multi-modality.

Original: We need an objective that promotes stochasticity by maximizing the entropy  of the policy (we omit γ here for simplicity):
Revised: We need an objective that promotes stochasticity by maximizing the entropy H of the policy (we omit γ here for simplicity):

Original: [Equation]
Revised: [Equation]

Original: We can deﬁne a new Q-function (slightly different to accommodate the entropy term) which follows the soft Bellman equation:
Revised: We can define a new Q-function (slightly different to accommodate the entropy term) which follows the soft Bellman equation:

Original: [Equation]
Revised: [Equation]

Original: To train the critic, we can approximate the right-hand expectation with samples, set it equal to y , and minimize the MSBE loss on a parameterized Q φ :
Revised: To train the critic, we can approximate the right-hand expectation with samples, set it equal to y , and minimize the MSBE loss on a parameterized Q ϕ (in practice, two approximators Q ϕ 1 and Q ϕare trained):

Original: [Equation]
Revised: [Equation]

Original: To train the actor, the actor loss is derived from the reparameterization trick.
Revised: To train the actor π θ , the actor loss is derived from the reparameterization trick to compute samples ˜ a θ ( s, ϵ ) = µ θ ( s ) + σ θ ( s ) ϵ , where ϵ is some random noise: [Equation]

Original:  M ETHOD
Revised:  M ETHOD

Original: We propose SACR2, Soft Actor-Critic with Reward Relabeling, a straight-forward method that can be implemented to any off-policy reinforcement learning algorithm with sparse rewards.
Revised: We propose SACR2, Soft Actor-Critic with Reward Relabeling, a straight-forward method that can be implemented to any off-policy reinforcement learning algorithm with sparse rewards.

Original: First, we add demonstration data to the buffer.
Revised: First, we add demonstration data to the buffer.

Original: The last transition of each expert trajectory is given the sparse reward from the environment.
Revised: The last transition of each expert trajectory is given the sparse reward from the environment.

Original: The other transitions are given a reward equal to b , typically smaller than the sparse reward.
Revised: The other transitions are given a reward equal to b , typically smaller than the sparse reward.

Original: Then, every time we collect a successful episode, we relabel the last N −transitions leading to the sparse reward, where N is the average length of the demonstrations, by assigning to them a reward equal to b .
Revised: Then, every time we collect a successful episode, we relabel the last N −transitions leading to the sparse reward, where N is the average length of the demonstrations, by assigning to them a reward equal to b .

Original: The ﬁrst part of our algorithm is most similar to SQIL.
Revised: The first part of our algorithm is most similar to SQIL.

Original: Intuitively, it gives the agent an incentive to imitate the expert.
Revised: Intuitively, it gives the agent an incentive to imitate the expert.

Original: Their paper shows theoretical connections between SQIL and regularized behaviour cloning.
Revised: Their paper shows theoretical connections between SQIL and regularized behaviour cloning.

Original: One important difference is that SQIL is a pure imitation learning algorithm, while our method learns from both the reward bonuses and the reward from the environment.
Revised: One important difference is that SQIL is a pure imitation learning algorithm, while our method learns from both the reward bonuses and the reward from the environment.

Original: This also means that our reward bonus b should be carefully tuned so that it has an impact without completely swallowing the environment reward.
Revised: This also means that our reward bonus b should be carefully tuned so that it has an impact without completely swallowing the environment reward.

Original: Empirically we found that b ≈ RN − 1 , where R is the value of the sparse reward, provided good results (see section 6), meaning that an expert demonstration has on average a return twice as big as a successful episode.
Revised: Empirically we found that b ≈ RN − 1 , where R is the value of the sparse reward, provided good results (see section 6), meaning that an expert demonstration has on average a return twice as big as a successful episode.

Original: Intuitively, SQIL also gives an incentive to avoid states that weren’t in the demonstration data, which could potentially be harmful if those states led to successful behaviour.
Revised: Intuitively, SQIL also gives an incentive to avoid states that weren’t in the demonstration data, which could potentially be harmful if those states led to successful behaviour.

Original: The relabeling part of our algorithm tries to mitigate this issue and is most similar to Self-Imitation
Revised: The relabeling part of our algorithm tries to mitigate this issue and is most similar to Self-Imitation

Original: Learning (SIL) from Oh et al.
Revised: Learning (SIL) from Oh et al.

Original: In SIL, the self-imitation is achieved by an additional loss function that pushes the agent to imitate its own decisions in the past only when they resulted in larger returns than expected.
Revised: In SIL, the self-imitation is achieved by an additional loss function that pushes the agent to imitate its own decisions in the past only when they resulted in larger returns than expected.

Original: In our method, the self-imitation is achieved by effectively treating successful episodes as if they were demonstrations.
Revised: In our method, the self-imitation is achieved by effectively treating successful episodes as if they were demonstrations.

Original: In order to avoid rewarding poor trajectories that solve the task by chance, we only reward the last N − 1 transitions leading to the sparse reward, where N is the average length of the demonstrations.
Revised: In order to avoid rewarding poor trajectories that solve the task by chance, we only reward the last N − 1 transitions leading to the sparse reward, where N is the average length of the demonstrations.

Original: Algorithm

Original: Require: b reward bonus, N average length of demonstrations;

Original: Initialize buffer with demonstrations, set r = b for all non-ﬁnal transitions;

Original: Initialize empty episode; while not converged do do a SAC update; if len(episode) =

Original: = 0 then collect one episode; if episode is successful then set r = b for the last N − 1 non-ﬁnal transitions; end end pop a transition ( s, a, s (cid:48) , r ) the episode and add it to the buffer; end

Original:  E XPERIMENTAL SETUP
Revised:  E XPERIMENTAL SETUP

Original: We evaluate our method in a simulated reaching task for a 6 degrees-of-freedom robot manipulator.
Revised: We evaluate our method in a simulated reaching task for a 6 degrees-of-freedom robot manipulator.

Original: The goal is to reach a target ball that appears randomly in the 3D space within the reach of the robot.
Revised: The goal is to reach a target ball that appears randomly in the 3D space within the reach of the robot.

Original: The initial position of the target changes after each episode, but it doesn’t move during an episode.
Revised: The initial position of the target changes after each episode, but it doesn’t move during an episode.

Original: The initial state of the robot is always the same, close to an upright position.
Revised: The initial state of the robot is always the same, close to an upright position.

Original: An episode ends once the robot has reached the ball, or after 100 time-steps.
Revised: An episode ends once the robot has reached the ball, or after 100 time-steps.

Original: The reward is fully sparse, and is equal to +100 if the robot reaches the ball and 0 otherwise.
Revised: The reward is fully sparse, and is equal to +100 if the robot reaches the ball and 0 otherwise.

Original: The state has 22 dimensions, 19 from the robot proprioceptive state (joint angles, joint speeds, gripper pose) and 3 from the task-related information (3D coordinates of the ball).
Revised: The state has 22 dimensions, 19 from the robot Require: b reward bonus, N average length of demonstrations; Initialize buffer with demonstrations, set reward r = b for all non-final transitions;

Revised: Initialize empty episode; while not converged do do a SAC update; if len(episode) =

Revised: = 0 then collect one episode; if episode is successful then set r = b for the last N − 1 non-final transitions; end end pop a transition ( s, a, s ′ , r ) from the episode and add it to the buffer; end proprioceptive state (joint angles, joint speeds, gripper pose) and 3 from the task-related information (3D coordinates of the ball).

Original: Simulator and demonstration data collection .
Revised: Simulator and demonstration data collection .

Original: The reaching task is part of the benchmark and learning environment RLBench from James et al.
Revised: The reaching task is part of the benchmark and learning environment RLBench from James et al.

Original: (2020), which is built around CoppeliaSim (Rohmer et al., 2013).
Revised: (2020), which is built around CoppeliaSim (Rohmer et al., 2013).

Original: The backend physics engine is the
Revised: The backend physics engine is the

Original: Bullet physics library (Coumans, 2015), and the expert demonstrations are provided by RLBench and rely on OMPL (Sucan et al., 2012) for motion planning.
Revised: Bullet physics library (Coumans, 2015), and the expert demonstrations are provided by RLBench and rely on OMPL (Sucan et al., 2012) for motion planning.

Original: Experiments .
Revised: Experiments .

Original: The basic SAC algorithm without demonstrations is able to solve this task (with  100 % accuracy) on some runs (around half), so our goal is to reduce the amount of training steps required to get there.
Revised: The basic SAC algorithm without demonstrations is able to solve this task (with close to 100 % accuracy) on some runs, so our goal is to reduce the amount of training steps required to get there.

Original: Each experiment is averaged over 4 runs.
Revised: Each experiment is averaged over 4 runs.

Original: We want to answer four questions: Which of the ideas presented in SACBC and SACfD have the most signiﬁcant impact ?
Revised: We want to answer four questions: Which of the ideas presented in SACBC and SACfD have the most significant impact ?

Original: How does SACR2 perform ?
Revised: How does SACR2 perform ?

Original: What is the conﬁguration that achieves the best performance ?
Revised: What is the configuration that achieves the best performance ?

Original: Does SACR2 make a difference when no demonstrations are available ?
Revised: Does SACR2 make a difference when no demonstrations are available ?

Original: In order to answer the ﬁrst two questions, we test each method in isolation on top of a baseline
Revised: In order to answer the first two questions, we test each method in isolation on top of a baseline

Original: SAC+Demo algorithm, which we deﬁne as SAC with  the following modiﬁcations:
Revised: SAC+Demo, which we define as SAC with demonstrations in the buffer. We apply the following modifications to both SAC+Demo and SACR2:

Original: • Single buffer initially ﬁlled with 200 demonstrations, and kept thereafter at a ratio of 10% demonstration data.
Revised: • Single buffer initially filled with 200 demonstrations, and kept thereafter at a ratio of 10% demonstration data.

Original: After 130000 training iterations, around 800 more demonstrations have been added to the buffer.
Revised: After 130000 training iterations, around 800 more demonstrations have been added to the buffer.

Original: • We additionally add 1000 random interactions to the buffer before training, and we pretrain during 3000 iterations before collecting any data.
Revised: • We additionally add 1000 random interactions to the buffer before training, and we pretrain during 3000 iterations before collecting any data.

Original: • The data is sampled from the buffer according to prioritized experience replay (PER) (Schaul et al., 2016).
Revised: • The data is sampled from the buffer according to prioritized experience replay (PER) (Schaul et al., 2016).

Original: • The replay ratio on the collected data is set to 32.
Revised: • The replay ratio on the collected data is set to 32.

Original: Since the batch size is set to 64, the agent takes two environment steps per training step.
Revised: Since the batch size is set to 64, the agent takes two environment steps per training step.

Original: • As suggested in both SACfD and SACBC, we use L2 regularization losses on the weights of the critic and the actor.
Revised: • As suggested in both SACfD and SACBC, we use L2 regularization losses on the weights of the critic and the actor.

Original: We evaluate  four loss functions, two buffer conﬁgurations, SACR2, pre-training on demonstrations, resetting some episodes to a demonstration state, and modifying the sampling probabilities of the replay buffer.
Revised: We evaluate SACR2, four loss functions, two buffer configurations, pre-training on demonstrations, resetting some episodes to a demonstration state, and modifying the sampling probabilities of the replay buffer.

Original:  R ESULTS
Revised:  R ESULTS

Original: Loss function .
Revised: Loss function .

Original: Following SACfD, we try a n-step return loss L n to train the critic.
Revised: Following SACfD, we try a n-step return loss L n to train the critic.

Original: This loss is a modiﬁed version of the loss L 1 (see 4) with n-step returns replacing the immediate reward.
Revised: This loss is a modified version of the loss L 1 (see 4) with n-step returns replacing the immediate reward.

Original: Using a larger lookahead can be particularly useful for tasks with sparse rewards, since it increases the chances of encountering a reward.
Revised: Using a larger lookahead can be particularly useful for tasks with sparse rewards, since it increases the chances of encountering a reward.

Original: We set λ n = 1 and n = 5 .
Revised: We set λ n = 1 and n = 5 .

Original: [Equation]
Revised: [Equation]

Original: Following SACBC, we try a behaviour cloning loss L BC to train the actor.
Revised: Following SACBC, we try a behaviour cloning loss L BC to train the actor.

Original: The loss prevents the policy from deviating too much from the demonstrations, and accounts for sub-optimimality of the demonstrations by ﬁltering out updates where the critic under-performs.
Revised: The loss prevents the policy from deviating too much from the demonstrations, and accounts for sub-optimimality of the demonstrations by filtering out updates where the critic under-performs.

Original: However, since our demonstrations come from an expert motion planner, we decide to not include the ﬁltering term (we ran some experiments with it and it performed signiﬁcantly worse).
Revised: However, since our demonstrations come from an expert motion planner, we decide to not include the filtering term (we ran some experiments with it and it performed significantly worse).

Original: We set λ BC = 2 .
Revised: We set λ BC = 2 .

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The results (Figure 2 bottom-left) show a signiﬁcant increase in performance for both methods, and an even bigger increase when combined.
Revised: The results (Figure 2 bottom-left) show a significant increase in performance for both methods, and an even bigger increase when combined.

Original: One vs two buffers .
Revised: One vs two buffers .

Original: Both SQIL and SACBC use a separate buffer to store the demonstrations, while SACfD stores them in the same buffer as the collected data.
Revised: Both SQIL and SACBC use a separate buffer to store the demonstrations, while SACfD stores them in the same buffer as the collected data.

Original: The main difference between these two approaches is the ratio of demonstrations in the sampled batches.
Revised: The main difference between these two approaches is the ratio of demonstrations in the sampled batches.

Original: With two buffers, this ratio can be ﬁxed (SQIL uses 50 % and SACBC roughly 10% ).
Revised: With two buffers, this ratio can be fixed (SQIL uses 50 % and SACBC roughly 10% ).

Original: With one buffer, the ratio varies from batch to batch and is on average equal to the ratio of demonstration data in the buffer if the sampling is uniform.
Revised: With one buffer, the ratio varies from batch to batch and is on average equal to the ratio of demonstration data in the buffer if the sampling is uniform.

Original: Since we sample according to PER, the ratio is actually slightly higher, but the results (Figure 2 top-left) show that two buffers perform better, albeit by a small margin.
Revised: Since we sample according to PER, the ratio is actually slightly higher, but the results (Figure 2 top-left) show that two buffers perform better, albeit by a small margin.

Original: One implementation advantage of using two buffers over one is that we can add as many demonstrations as we wish to the demonstration buffer in order to have a different replay ratio, but we introduced an equivalent amount of demonstrations to keep things fair.
Revised: One practical advantage of using two buffers over one is that we can add as many demonstrations as we wish to the demonstration buffer in order to have a different replay ratio, but we introduced an equivalent amount of demonstrations to keep things fair.

Original: Reset to demonstrations .
Revised: Reset to demonstrations .

Original: Following SACBC, we try to reset some episodes (10 % ) to a demonstration: the position of the target ball is the same as in the demonstration, and the initial state of the robot is randomly chosen from the demonstration.
Revised: Following SACBC, we try to reset some episodes (10 % ) to a demonstration: the position of the target ball is the same as in the demonstration, and the initial state of the robot is randomly chosen from the demonstration.

Original: This should act as a form of curriculum learning and lead to more successful episodes early on.
Revised: This should act as a form of curriculum learning and lead to more successful episodes early on.

Original: The results (Figure 2 middle-left) do show an initial boost in the learning process, but the improvement isn’t too signiﬁcant.
Revised: The results (Figure 2 middle-left) do show an initial boost in the learning process, but the improvement isn’t too significant.

Original: Pre-training on demonstrations .
Revised: Pre-training on demonstrations .

Original: Pre-training on demonstrations is one of the most common approaches to leverage demonstrations in the literature, and does seem like a good idea according to the results (Figure 2 middle-right).
Revised: Pre-training on demonstrations is one of the most common approaches to leverage demonstrations in the literature, and does seem like a good idea according to the results (Figure 2 middle-right).

Original: However, too much pre-training (10000 iterations on 800 demonstrations rather than 3000 iterations on 200 demonstrations) also decreases performance.
Revised: However, too much pre-training (10000 iterations on 800 demonstrations rather than 3000 iterations on 200 demonstrations) also decreases performance.

Original: Intuitively, the agents winds up forgetting what it initially learnt from the demonstrations when it ﬁrst encounters subpar trajectories from its collected experience.
Revised: Intuitively, the agents winds up forgetting what it initially learnt from the demonstrations when it first encounters subpar trajectories from its collected experience.

Original: This is much more clear in Figure 5 of the appendix where we do an even more drastic pre-training (20000 iterations on 2000 demonstrations).
Revised: This is much more clear in Figure 5 of the appendix where we do an even more drastic pre-training (20000 iterations on 2000 demonstrations).

Original: Prioritized replay .
Revised: Prioritized replay .

Original: Following SACfD, we try to modify the PER strategy with two additional terms: a term representing the actor loss, and a constant bonus applied to all transitions coming from demonstrations.
Revised: Following SACfD, we try to modify the PER strategy with two additional terms: a term representing the actor loss, and a constant bonus applied to all transitions coming from demonstrations.

Original: From our limited experiments, this new strategy didn’t have a great impact in terms of ratio of demonstration data in the sampled batches.
Revised: From our limited experiments, this new strategy didn’t have a great impact in terms of ratio of demonstration data in the sampled batches.

Original: With both PER and the modiﬁed PER the ratio is close to 11% (we recall that 10% of the transitions in the buffer come from demonstrations).
Revised: With both PER and the modified PER the ratio is close to 11% (we recall that 10% of the transitions in the buffer come from demonstrations).

Original: However, the results (Figure 2 top-right) do show a major initial boost during training, although the overall improvement isn’t too signiﬁcant.
Revised: However, the results (Figure 2 top-right) do show a major initial boost during training, although the overall improvement isn’t too significant.

Original: We recall that in PER, the probability of sampling a particular transition is proportional to its priority p i , which is commonly computed from the transition’s temporal difference (TD) error δ i , for instance p i =
Revised: We recall that in PER, the probability of sampling a particular transition is proportional to its priority p i , which is commonly computed from the transition’s temporal difference (TD) error δ i , for instance p i =

Original: δ 2 i + (cid:15) where (cid:15) is a bonus given to all transitions.
Revised: δ 2 i + ϵ where ϵ is a bonus given to all transitions.

Original: SACfD adds a square term representing the actor loss and a second bonus (cid:15) D given only to the demonstrations.
Revised: SACfD adds a square term representing the actor loss and a second bonus ϵ D given only to the demonstrations.

Original: We use the same hyperparameters as in SACfD.
Revised: We use the same hyperparameters as in SACfD.

Original: SACR2 (ours) .
Revised: SACR2 (ours) .

Original: The results (Figure 2 bottom-right) show an initial boost during training, both with and without relabeling, probably coming from the agent imitating the demonstrations more aggressively.
Revised: The results (Figure 2 bottom-right) show an initial boost during training, both with and without relabeling, probably coming from the agent imitating the demonstrations more aggressively.

Original: However, without relabeling, the learning process becomes unstable, probably due to the lack of consistency on the rewards once the agent has collected enough sucessful episodes.
Revised: However, without relabeling, the learning process becomes unstable, probably due to the lack of consistency on the rewards once the agent has collected enough sucessful episodes.

Original: Overall, SACR2 increases the performance by a wide margin.
Revised: Overall, SACR2 increases the performance by a wide margin.

Original: More experiments with different values are presented in Figure 3.
Revised: More experiments with different values are presented in Figure 3.

Original: We can see that the higher b , the more unstable the training process without relabeling.
Revised: We can see that the higher b , the more unstable the training process without relabeling.

Original: With relabeling, there doesn’t seem to be any signiﬁcant difference between b = 5 and b = 10 .
Revised: With relabeling, there doesn’t seem to be any significant difference between b = 5 and b = 10 .

Original: We also tested b = 1 but its impact was practically unnoticeable with respect to b = 0 .
Revised: We also tested b = 1 but its impact was practically unnoticeable with respect to b = 0 .

Revised: Best configuration .

Original: Based on our ablation results, we present a second baseline SAC+Demo*, which is deﬁned as SAC+Demo with the method that provided the greatest increase in performance: the addition of the loss functions L BC and L n .
Revised: Based on our ablation results, we introduce SAC+Demo* and SACR2*, which are defined as the previous algorithms plus the method that provided the greatest increase in performance: the addition of the loss functions L BC and L n .

Original: To answer the third question from section 5, we carry additional experiments shown in the appendix
Revised: To answer the third question from section 5, we carry additional experiments shown in the appendix

Original: A, and the only method that furthers improves upon SAC+Demo* is SACR2, as shown in Figure 3.
Revised: A, and the only method that furthers improves upon SAC+Demo* is SACR2*, as shown in Figure 3.

Original: However, the improvement is very minor as SAC+Demo* with SACR2 seems to be the fastest method to reach an acuracy of 90% , but both methods (SAC+Demo*with and without SACR2) look pretty much equal afterwards.
Revised: However, the improvement is very minor as SACR2* seems to be the fastest method to reach an accuracy of 90% , but both methods (SAC+Demo* and SACR2*) look pretty much equal afterwards.

Original: Learning without demonstrations .
Revised: Learning without demonstrations .

Original: One ﬁnal interesting experiment is to see whether SACR2 can also improve the performance when no demonstrations are available, by just relabeling successful episodes.
Revised: One final interesting experiment is to see whether SACR2 can also improve the performance when no demonstrations are available, by just relabeling successful episodes.

Original: Figure 4 shows that SACR actually solves the task consistently, while SAC only solves it on 2  of the 4 runs.
Revised: Figure 4 shows that SACR actually solves the task consistently, while SAC only solves it on 2 out of the 4 runs.

Original: However, over the runs where SAC solved the task, its sample efﬁciency was comparable to SACR2.
Revised: However, over the runs where SAC solved the task, its sample efficiency was comparable to SACR2.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: We propose SACR2, a generic method that can be applied to any off-policy reinforcement learning.
Revised: We propose SACR2, a generic method that can be applied to any off-policy reinforcement learning.

Original: It encourages two behaviours: imitate the expert demonstrations (if available), and imitate the past successful trajectories.
Revised: It encourages two behaviours: imitate the expert demonstrations (if available), and imitate the past successful trajectories.

Original: From our limited experiments, we identiﬁed three methods where the results were strong enough to advocate their use on other tasks: relabeling rewards with SACR2, the behaviour cloning loss from SACBC (Nair et al., 2018), and the n-step loss from SACfD (Vecer´ık et al., 2017).
Revised: From our limited experiments, we identified three methods where the results were strong enough to advocate their use on other tasks: relabeling rewards with SACR2, the behaviour cloning loss from SACBC (Nair et al., 2018), and the n-step loss from SACfD (Vecer ´ ık et al., 2017).

Original: We show that this methods stack together, as the best results were obtained with the three methods combined.
Revised: We show that these methods stack together, as the best results were obtained with the three methods combined.

Original: Regarding SACR2, further analysis needs to be done on the impact of the hyper-parameters b and N across different tasks.
Revised: Regarding SACR2, further analysis needs to be done on the impact of the hyper-parameters b and N across different tasks.

Original: Many improvements could be brought to the method, such as using a more principled value rather than a constant reward bonus, or implementing a decay or other strategy to switch the focus to the environment reward once it becomes more common.
Revised: Many improvements could be brought to the method, such as using a more principled value rather than a constant reward bonus, or implementing a decay or other strategy to switch the focus to the environment reward once it becomes more common.

Original: The main limitation of our method is that it assumes that the expert demonstrations are optimal.
Revised: The main limitation of our method is that it assumes that the expert demonstrations are optimal.

Original: Other methods like NAC and SACBC are able to handle sub-optimal demonstrations, which is very useful for most tasks outside of robotics, and even for more complex robotics tasks where an optimal planner is not available.
Revised: Other methods like NAC and SACBC are able to handle sub-optimal demonstrations, which is very useful for most tasks outside of robotics, and even for more complex robotics tasks where an optimal planner is not available.

Original: Our method also requires episodic tasks, preferably over a short horizon, since it would be difﬁcult to relabel a continuous ﬂow of experiences.
Revised: Our method also requires episodic tasks, preferably over a short horizon, since it would be difficult to relabel a continuous flow of experiences.

Original: Although our results show that SACR2 can greatly improve performance, even when no demonstrations are available and the only supervision comes from a sparse reward, the truth is that we only tested our method on a single task.
Revised: Our results show that SACR2 can greatly improve performance, even when no demonstrations are available and the only supervision comes from a sparse reward.

Original: We don’t know if the results presented here would apply to other tasks, in particular more complex tasks where we try to increase the success rate rather than the sample efﬁciency.
Revised: However, these results come from a single task, and we don’t know how they would translate to other tasks, in particular more complex tasks where we try to increase the success rate rather than the sample efficiency.

Original: Figures 2 and 5 show that some of the tested methods, like the modiﬁed
Revised: Further evaluation is also needed to clarify the impact of some of the other methods.

Original: PER, resetting to demonstrations, and having a separate demonstration buffer, improve slightly  upon our ﬁrst baseline SAC+Demo, but actually hurt our second stronger baseline SAC+Demo*, which doesn’t allow us to draw any signiﬁcant conclusions.
Revised: As shown in Figures 2 and 5, some of them slightly improve upon our first baseline SAC+Demo, but actually hurt our second stronger baseline SAC+Demo*, which doesn’t allow us to draw any significant conclusions.

Original: It wouldalso be interesting to test SACR with another base algorithm different to SAC, for instance a Q-learning-type algorithm on a task with discrete actions.
Revised: Finally, it would be interesting to test SACR2 with another base algorithm different to SAC, for instance a Q-learning-type algorithm on a task with discrete actions.