Original: L EARNING S ELF -I MITATING D IVERSE P OLICIES
Revised: L EARNING S ELF -I MITATING D IVERSE P OLICIES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The success of popular algorithms for deep reinforcement learning, such as policygradients and Q-learning, relies heavily on the availability of an informative reward signal at each timestep of the sequential decision-making process.
Revised: The success of popular algorithms for deep reinforcement learning, such as policygradients and Q-learning, relies heavily on the availability of an informative reward signal at each timestep of the sequential decision-making process.

Original: When rewards are only sparsely available during an episode, or a rewarding feedback is provided only after episode termination, these algorithms perform sub-optimally due to the difﬁcultly in credit assignment.
Revised: When rewards are only sparsely available during an episode, or a rewarding feedback is provided only after episode termination, these algorithms perform sub-optimally due to the difﬁcultly in credit assignment.

Original: Alternatively, trajectory-based policy optimization methods, such as cross-entropy method and evolution strategies, do not require per-timestep rewards, but have been found to suffer from high sample complexity by completing forgoing the temporal nature of the problem.
Revised: Alternatively, trajectory-based policy optimization methods, such as cross-entropy method and evolution strategies, do not require per-timestep rewards, but have been found to suffer from high sample complexity by completing forgoing the temporal nature of the problem.

Original: Improving the efﬁciency of RL algorithms in real-world problems with sparse or episodic rewards is therefore a pressing need.
Revised: Improving the efﬁciency of RL algorithms in real-world problems with sparse or episodic rewards is therefore a pressing need.

Original: In this work, we introduce a self-imitation learning algorithm that exploits and explores well in the sparse and episodic reward settings.
Revised: In this work, we introduce a self-imitation learning algorithm that exploits and explores well in the sparse and episodic reward settings.

Original: We view each policy as a state-action visitation distribution and formulate policy optimization as a divergence minimization problem.
Revised: We view each policy as a state-action visitation distribution and formulate policy optimization as a divergence minimization problem.

Original: We show that with Jensen-Shannon divergence, this divergence minimization problem can be reduced into a policy-gradient algorithm with shaped rewards learned from experience replays.
Revised: We show that with Jensen-Shannon divergence, this divergence minimization problem can be reduced into a policy-gradient algorithm with shaped rewards learned from experience replays.

Original: Experimental results indicate that our algorithm works comparable to existing algorithms in environments with dense rewards, and signiﬁcantly better in environments with sparse and episodic rewards.
Revised: Experimental results indicate that our algorithm works comparable to existing algorithms in environments with dense rewards, and signiﬁcantly better in environments with sparse and episodic rewards.

Original: We then discuss limitations of self-imitation learning, and propose to solve them by using Stein variational policy gradient descent with the Jensen-Shannon kernel to learn multiple diverse policies.
Revised: We then discuss limitations of self-imitation learning, and propose to solve them by using Stein variational policy gradient descent with the Jensen-Shannon kernel to learn multiple diverse policies.

Original: We demonstrate its effectiveness on a challenging variant of continuous-control MuJoCo locomotion tasks.
Revised: We demonstrate its effectiveness on a challenging variant of continuous-control MuJoCo locomotion tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep reinforcement learning (RL) has demonstrated signiﬁcant applicability and superior performance in many problems outside the reach of traditional algorithms, such as computer and board games (Mnih et al., 2015; Silver et al., 2016), continuous control (Lillicrap et al., 2015), and robotics (Levine et al., 2016).
Revised: Deep reinforcement learning (RL) has demonstrated signiﬁcant applicability and superior performance in many problems outside the reach of traditional algorithms, such as computer and board games (Mnih et al., 2015; Silver et al., 2016), continuous control (Lillicrap et al., 2015), and robotics (Levine et al., 2016).

Original: Using deep neural networks as functional approximators, many classical RL algorithms have been shown to be very effective in solving sequential decision problems.
Revised: Using deep neural networks as functional approximators, many classical RL algorithms have been shown to be very effective in solving sequential decision problems.

Original: For example, a policy that selects actions under certain state observation can be parameterized by a deep neural network that takes the current state observation as input and gives an action or a distribution over actions as output.
Revised: For example, a policy that selects actions under certain state observation can be parameterized by a deep neural network that takes the current state observation as input and gives an action or a distribution over actions as output.

Original: Value functions that take both state observation and action as inputs and predict expected future reward can also be parameterized as neural networks.
Revised: Value functions that take both state observation and action as inputs and predict expected future reward can also be parameterized as neural networks.

Original: In order to optimize such neural networks, policy gradient methods (Mnih et al., 2016; Schulman et al., 2015; 2017a) and Q-learning algorithms (Mnih et al., 2015) capture the temporal structure of the sequential decision problem and decompose it to a supervised learning problem, guided by the immediate and discounted future reward from rollout data.
Revised: In order to optimize such neural networks, policy gradient methods (Mnih et al., 2016; Schulman et al., 2015; 2017a) and Q-learning algorithms (Mnih et al., 2015) capture the temporal structure of the sequential decision problem and decompose it to a supervised learning problem, guided by the immediate and discounted future reward from rollout data.

Original: Unfortunately, when the reward signal becomes sparse or delayed, these RL algorithms may suffer from inferior performance and inefﬁcient sample complexity, mainly due to the scarcity of the immediate supervision when training happens in single-timestep manner.
Revised: Unfortunately, when the reward signal becomes sparse or delayed, these RL algorithms may suffer from inferior performance and inefﬁcient sample complexity, mainly due to the scarcity of the immediate supervision when training happens in single-timestep manner.

Original: This is known as the temporal credit assignment problem (Sutton, 1984).
Revised: This is known as the temporal credit assignment problem (Sutton, 1984).

Original: For instance, consider the Atari Montezuma’s revenge game – a reward is received after collecting certain items or arriving at the ﬁnal destination in the lowest level, while no reward is received as the agent is trying to reach these goals.
Revised: For instance, consider the Atari Montezuma’s revenge game – a reward is received after collecting certain items or arriving at the ﬁnal destination in the lowest level, while no reward is received as the agent is trying to reach these goals.

Original: The sparsity of the reward makes the neural network training very inefﬁcient and also poses challenges in exploration.
Revised: The sparsity of the reward makes the neural network training very inefﬁcient and also poses challenges in exploration.

Original: It is not hard to see that many of the real-world problems tend to be of the form where rewards are either only sparsely available during an episode, or the rewards are episodic, meaning that a non-zero reward is only provided at the end of the trajectory or episode.
Revised: It is not hard to see that many of the real-world problems tend to be of the form where rewards are either only sparsely available during an episode, or the rewards are episodic, meaning that a non-zero reward is only provided at the end of the trajectory or episode.

Original: In addition to policy-gradient and Q-learning, alternative algorithms, such as those for global- or stochastic-optimization, have recently been studied for policy search.
Revised: In addition to policy-gradient and Q-learning, alternative algorithms, such as those for global- or stochastic-optimization, have recently been studied for policy search.

Original: These algorithms do not decompose trajectories into individual timesteps, but instead apply zeroth-order ﬁnite-difference gradient or gradient-free methods to learn policies based on the cumulative rewards of the entire trajectory.
Revised: These algorithms do not decompose trajectories into individual timesteps, but instead apply zeroth-order ﬁnite-difference gradient or gradient-free methods to learn policies based on the cumulative rewards of the entire trajectory.

Original: Usually, trajectory samples are ﬁrst generated by running the current policy and then the distribution of policy parameters is updated according to the trajectory-returns.
Revised: Usually, trajectory samples are ﬁrst generated by running the current policy and then the distribution of policy parameters is updated according to the trajectory-returns.

Original: The cross-entropy method (CEM, Rubinstein & Kroese (2016)) and evolution strategies (Salimans et al., 2017) are two nominal examples.
Revised: The cross-entropy method (CEM, Rubinstein & Kroese (2016)) and evolution strategies (Salimans et al., 2017) are two nominal examples.

Original: Although their sample efﬁciency is often not comparable to the policy gradient methods when dense rewards are available from the environment, they are more widely applicable in the sparse or episodic reward settings as they are agnostic to task horizon, and only the trajectorybased cumulative reward is needed.
Revised: Although their sample efﬁciency is often not comparable to the policy gradient methods when dense rewards are available from the environment, they are more widely applicable in the sparse or episodic reward settings as they are agnostic to task horizon, and only the trajectorybased cumulative reward is needed.

Original: Our contribution is the introduction of a new algorithm based on policy-gradients, with the objective of achieving better performance than existing RL algorithms in sparse and episodic reward settings.
Revised: Our contribution is the introduction of a new algorithm based on policy-gradients, with the objective of achieving better performance than existing RL algorithms in sparse and episodic reward settings.

Original: Using the equivalence between the policy function and its state-action visitation distribution, we formulate policy optimization as a divergence minimization problem between the current policy’s visitation and the distribution induced by a set of experience replay trajectories with high returns.
Revised: Using the equivalence between the policy function and its state-action visitation distribution, we formulate policy optimization as a divergence minimization problem between the current policy’s visitation and the distribution induced by a set of experience replay trajectories with high returns.

Original: We show that with the Jensen-Shannon divergence ( D JS ), this divergence minimization problem can be reduced into a policy-gradient algorithm with shaped, dense rewards learned from these experience replays.
Revised: We show that with the Jensen-Shannon divergence ( D JS ), this divergence minimization problem can be reduced into a policy-gradient algorithm with shaped, dense rewards learned from these experience replays.

Original: This algorithm can be seen as self-imitation learning , in which the expert trajectories in the experience replays are self-generated by the agent during the course of learning, rather than using some external demonstrations.
Revised: This algorithm can be seen as self-imitation learning , in which the expert trajectories in the experience replays are self-generated by the agent during the course of learning, rather than using some external demonstrations.

Original: We combine the divergence minimization objective with the standard RL objective, and empirically show that the shaped, dense rewards signiﬁcantly help in sparse and episodic settings by improving credit assignment.
Revised: We combine the divergence minimization objective with the standard RL objective, and empirically show that the shaped, dense rewards signiﬁcantly help in sparse and episodic settings by improving credit assignment.

Original: Following that, we qualitatively analyze the shortcomings of the self-imitation algorithm.
Revised: Following that, we qualitatively analyze the shortcomings of the self-imitation algorithm.

Original: Our second contribution is the application of Stein variational policy gradient (SVPG) with the Jensen-Shannon kernel to simultaneously learn multiple diverse policies.
Revised: Our second contribution is the application of Stein variational policy gradient (SVPG) with the Jensen-Shannon kernel to simultaneously learn multiple diverse policies.

Original: We demonstrate the beneﬁts of this addition to the self-imitation framework by considering difﬁcult exploration tasks with sparse and deceptive rewards.
Revised: We demonstrate the beneﬁts of this addition to the self-imitation framework by considering difﬁcult exploration tasks with sparse and deceptive rewards.

Original: Related Works.
Revised: Related Works.

Original: Divergence minimization has been used in various policy learning algorithms.
Revised: Divergence minimization has been used in various policy learning algorithms.

Original: Relative Entropy Policy Search (REPS) (Peters et al., 2010) restricts the loss of information between policy updates by constraining the KL-divergence between the state-action distribution of old and new policy.
Revised: Relative Entropy Policy Search (REPS) (Peters et al., 2010) restricts the loss of information between policy updates by constraining the KL-divergence between the state-action distribution of old and new policy.

Original: Policy search can also be formulated as an EM problem, leading to several interesting algorithms, such as RWR (Peters & Schaal, 2007) and PoWER (Kober & Peters, 2009).
Revised: Policy search can also be formulated as an EM problem, leading to several interesting algorithms, such as RWR (Peters & Schaal, 2007) and PoWER (Kober & Peters, 2009).

Original: Here the
Revised: Here the

Original: M-step minimizes a KL-divergence between trajectory distributions, leading to an update rule which resembles return-weighted imitation learning.
Revised: M-step minimizes a KL-divergence between trajectory distributions, leading to an update rule which resembles return-weighted imitation learning.

Original: Please refer to Deisenroth et al.
Revised: Please refer to Deisenroth et al.

Original: for a comprehensive exposition.
Revised: for a comprehensive exposition.

Original: MATL (Wulfmeier et al., 2017) uses adversarial training to bring state occupancy from a real and simulated agent close to each other for efﬁcient transfer learning.
Revised: MATL (Wulfmeier et al., 2017) uses adversarial training to bring state occupancy from a real and simulated agent close to each other for efﬁcient transfer learning.

Original: In Guided
Revised: In Guided

Original: Policy Search (GPS, Levine & Koltun (2013)), a parameterized policy is trained by constraining the divergence between the current policy and a controller learnt via trajectory optimization.
Revised: Policy Search (GPS, Levine & Koltun (2013)), a parameterized policy is trained by constraining the divergence between the current policy and a controller learnt via trajectory optimization.

Original: Learning from Demonstrations (LfD).
Revised: Learning from Demonstrations (LfD).

Original: The objective in LfD, or imitation learning, is to train a control policy to produce a trajectory distribution similar to the demonstrator.
Revised: The objective in LfD, or imitation learning, is to train a control policy to produce a trajectory distribution similar to the demonstrator.

Original: Approaches for self-driving cars (Bojarski et al., 2016) and drone manipulation (Ross et al., 2013) have used human-expert data, along with Behavioral Cloning algorithm to learn good control policies.
Revised: Approaches for self-driving cars (Bojarski et al., 2016) and drone manipulation (Ross et al., 2013) have used human-expert data, along with Behavioral Cloning algorithm to learn good control policies.

Original: Deep Q-learning has been combined with human demonstrations to achieve performance gains in Atari (Hester et al., 2017) and robotics tasks (Veˇcer´ık et al., 2017; Nair et al., 2017).
Revised: Deep Q-learning has been combined with human demonstrations to achieve performance gains in Atari (Hester et al., 2017) and robotics tasks (Veˇcer´ık et al., 2017; Nair et al., 2017).

Original: Human data has also been used in the maximum entropy IRL framework to learn cost functions under which the demonstrations are optimal (Finn et al., 2016).
Revised: Human data has also been used in the maximum entropy IRL framework to learn cost functions under which the demonstrations are optimal (Finn et al., 2016).

Original: Ho & Ermon (2016) use the same framework to derive an imitation-learning algorithm (GAIL) which is motivated by minimizing the divergence between agent’s rollouts and external expert demonstrations.
Revised: Ho & Ermon (2016) use the same framework to derive an imitation-learning algorithm (GAIL) which is motivated by minimizing the divergence between agent’s rollouts and external expert demonstrations.

Original: Besides humans, other sources of expert supervision include planningbased approaches such as iLQR (Levine et al., 2016) and MCTS (Silver et al., 2016).
Revised: Besides humans, other sources of expert supervision include planningbased approaches such as iLQR (Levine et al., 2016) and MCTS (Silver et al., 2016).

Original: Our algorithm departs from prior work in forgoing external supervision, and instead using the past experiences of the learner itself as demonstration data.
Revised: Our algorithm departs from prior work in forgoing external supervision, and instead using the past experiences of the learner itself as demonstration data.

Original: Exploration and Diversity in RL.
Revised: Exploration and Diversity in RL.

Original: Count-based exploration methods utilize state-action visitation counts N ( s, a ) , and award a bonus to rarely visited states (Strehl & Littman, 2008).
Revised: Count-based exploration methods utilize state-action visitation counts N ( s, a ) , and award a bonus to rarely visited states (Strehl & Littman, 2008).

Original: In large statespaces, approximation techniques (Tang et al., 2017), and estimation of pseudo-counts by learning density models (Bellemare et al., 2016; Fu et al., 2017) has been researched.
Revised: In large statespaces, approximation techniques (Tang et al., 2017), and estimation of pseudo-counts by learning density models (Bellemare et al., 2016; Fu et al., 2017) has been researched.

Original: Intrinsic motivation has been shown to aid exploration, for instance by using information gain (Houthooft et al., 2016) or prediction error (Stadie et al., 2015) as a bonus.
Revised: Intrinsic motivation has been shown to aid exploration, for instance by using information gain (Houthooft et al., 2016) or prediction error (Stadie et al., 2015) as a bonus.

Original: Hindsight Experience Replay (Andrychowicz et al., 2017) adds additional goals (and corresponding rewards) to a Q-learning algorithm.
Revised: Hindsight Experience Replay (Andrychowicz et al., 2017) adds additional goals (and corresponding rewards) to a Q-learning algorithm.

Original: We also obtain additional rewards, but from a discriminator trained on past agent experiences, to accelerate a policy-gradient algorithm.
Revised: We also obtain additional rewards, but from a discriminator trained on past agent experiences, to accelerate a policy-gradient algorithm.

Original: Prior work has looked at training a diverse ensemble of agents with good exploratory skills (Liu et al., 2017; Conti et al., 2017; Florensa et al., 2017).
Revised: Prior work has looked at training a diverse ensemble of agents with good exploratory skills (Liu et al., 2017; Conti et al., 2017; Florensa et al., 2017).

Original: To enjoy the beneﬁts of diversity, we incorporate a modiﬁcation of SVPG (Liu et al., 2017) in our ﬁnal algorithm.
Revised: To enjoy the beneﬁts of diversity, we incorporate a modiﬁcation of SVPG (Liu et al., 2017) in our ﬁnal algorithm.

Original: In very recent work, Oh et al.
Revised: In very recent work, Oh et al.

Original: (2018) propose exploiting past good trajectories to drive exploration.
Revised: (2018) propose exploiting past good trajectories to drive exploration.

Original: Their algorithm buffers ( s, a ) and the corresponding return for each transition in rolled trajectories, and reuses them for training if the stored return value is higher than the current state-value estimate.
Revised: Their algorithm buffers ( s, a ) and the corresponding return for each transition in rolled trajectories, and reuses them for training if the stored return value is higher than the current state-value estimate.

Original: Our approach presents a different objective for self-imitation based on divergence-minimization.
Revised: Our approach presents a different objective for self-imitation based on divergence-minimization.

Original: With this view, we learn shaped, dense rewards which are then used for policy optimization.
Revised: With this view, we learn shaped, dense rewards which are then used for policy optimization.

Original: We further improve the algorithm with SVPG.
Revised: We further improve the algorithm with SVPG.

Original: Reusing high-reward trajectories has also been explored for program synthesis and semantic parsing tasks (Liang et al., 2016; 2018; Abolaﬁa et al., 2018).
Revised: Reusing high-reward trajectories has also been explored for program synthesis and semantic parsing tasks (Liang et al., 2016; 2018; Abolaﬁa et al., 2018).

Original:  M AIN M ETHODS
Revised:  M AIN M ETHODS

Original: We start with a brief introduction to RL in Section 2.1, and then introduce our main algorithm of self-imitating learning in Section 2.2.
Revised: We start with a brief introduction to RL in Section 2.1, and then introduce our main algorithm of self-imitating learning in Section 2.2.

Original: Section 2.3 further extends our main method to learn multiple diverse policies using Stein variational policy gradient with Jensen-Shannon kernel.
Revised: Section 2.3 further extends our main method to learn multiple diverse policies using Stein variational policy gradient with Jensen-Shannon kernel.

Original: 2. R EINFORCEMENT L EARNING B ACKGROUND
Revised: 2. R EINFORCEMENT L EARNING B ACKGROUND

Original: A typical RL setting involves an environment modeled as a Markov Decision Process with an unknown system dynamics model p ( s t +1 | s t , a t ) and an initial state distribution p 0 ( s 0 ) .
Revised: A typical RL setting involves an environment modeled as a Markov Decision Process with an unknown system dynamics model p ( s t +1 | s t , a t ) and an initial state distribution p 0 ( s 0 ) .

Original: An agent interacts sequentially with the environment in discrete time-steps using a policy π which maps the an observation s t ∈ S to either a single action a t (deterministic policy), or a distribution over the action space
Revised: An agent interacts sequentially with the environment in discrete time-steps using a policy π which maps the an observation s t ∈ S to either a single action a t (deterministic policy), or a distribution over the action space

Original: A (stochastic policy).
Revised: A (stochastic policy).

Original: We consider the scenario of stochastic policies over high-dimensional, continuous state and action spaces.
Revised: We consider the scenario of stochastic policies over high-dimensional, continuous state and action spaces.

Original: The agent receives a per-step reward r t ( s t , a t ) ∈ R , and the RL objective involves maximization of the expected discounted sum of rewards, η ( π θ ) =
Revised: The agent receives a per-step reward r t ( s t , a t ) ∈ R , and the RL objective involves maximization of the expected discounted sum of rewards, η ( π θ ) =

Original: E p 0 ,p,π (cid:2) (cid:80) ∞
Revised: E p 0 ,p,π (cid:2) (cid:80) ∞

Original: t =0 γ t r ( s t , a t ) (cid:3) , where γ ∈ (0 , 1] is the discount factor.
Revised: t =0 γ t r ( s t , a t ) (cid:3) , where γ ∈ (0 , 1] is the discount factor.

Original: The action-value function is
Revised: The action-value function is

Original: Q π ( s t , a t ) =
Revised: Q π ( s t , a t ) =

Original: E p 0 ,p,π (cid:2) (cid:80) ∞
Revised: E p 0 ,p,π (cid:2) (cid:80) ∞

Original: t (cid:48)
Revised: t (cid:48)

Original: =
Revised: =

Original: t γ t (cid:48) − t r ( s t (cid:48) , a t (cid:48) ) (cid:3) .
Revised: t γ t (cid:48) − t r ( s t (cid:48) , a t (cid:48) ) (cid:3) .

Original: We deﬁne the unnormalized γ -discounted statevisitation distribution for a policy π by ρ π ( s ) =
Revised: We deﬁne the unnormalized γ -discounted statevisitation distribution for a policy π by ρ π ( s ) =

Original: (cid:80) ∞ t =0 γ t P ( s t = s | π ) , where P ( s t = s | π ) is the probability of being in state s at time t , when following policy π and starting state s 0 ∼ p 0 .
Revised: (cid:80) ∞ t =0 γ t P ( s t = s | π ) , where P ( s t = s | π ) is the probability of being in state s at time t , when following policy π and starting state s 0 ∼ p 0 .

Original: The expected policy return η ( π θ ) can then be written as E ρ π
Revised: The expected policy return η ( π θ ) can then be written as E ρ π

Original: ( s,a ) [ r ( s, a )] , where ρ π ( s, a ) =
Revised: ( s,a ) [ r ( s, a )] , where ρ π ( s, a ) =

Original: ρ π ( s ) π ( a | s ) is the state-action visitation distribution.
Revised: ρ π ( s ) π ( a | s ) is the state-action visitation distribution.

Original: Using the policy gradient theorem (Sutton et al., 2000), we can get the direction of ascent ∇ θ η ( π θ ) =
Revised: Using the policy gradient theorem (Sutton et al., 2000), we can get the direction of ascent ∇ θ η ( π θ ) =

Original: E ρ π
Revised: E ρ π

Original: ( s,a ) (cid:2) ∇ θ log π θ ( a | s ) Q π ( s, a ) (cid:3) .
Revised: ( s,a ) (cid:2) ∇ θ log π θ ( a | s ) Q π ( s, a ) (cid:3) .

Original: 2. P OLICY O PTIMIZATION AS D IVERGENCE M INIMIZATION WITH S ELF -I MITATION
Revised: 2. P OLICY O PTIMIZATION AS D IVERGENCE M INIMIZATION WITH S ELF -I MITATION

Original: Although the policy π ( a | s ) is given as a conditional distribution, its behavior is better characterized by the corresponding state-action visitation distribution
Revised: Although the policy π ( a | s ) is given as a conditional distribution, its behavior is better characterized by the corresponding state-action visitation distribution

Original: ρ π ( s, a ) , which wraps the MDP dynamics and fully decides the expected return via η ( π ) =
Revised: ρ π ( s, a ) , which wraps the MDP dynamics and fully decides the expected return via η ( π ) =

Original: E ρ π
Revised: E ρ π

Original: [ r ( s, a )] .
Revised: [ r ( s, a )] .

Original: Therefore, distance metrics on a policy π should be deﬁned with respect to the visitation distribution ρ π , and the policy search should be viewed as ﬁnding policies with good visitation distributions ρ π that yield high reward.
Revised: Therefore, distance metrics on a policy π should be deﬁned with respect to the visitation distribution ρ π , and the policy search should be viewed as ﬁnding policies with good visitation distributions ρ π that yield high reward.

Original: Suppose we have access to a good policy π ∗ , then it is natural to consider ﬁnding a π such that its visitation distribution ρ π matches ρ π ∗ .
Revised: Suppose we have access to a good policy π ∗ , then it is natural to consider ﬁnding a π such that its visitation distribution ρ π matches ρ π ∗ .

Original: To do so, we can deﬁne a divergence measure D ( ρ π , ρ π ∗ ) that captures the similarity between two distributions, and minimize this divergence for policy improvement.
Revised: To do so, we can deﬁne a divergence measure D ( ρ π , ρ π ∗ ) that captures the similarity between two distributions, and minimize this divergence for policy improvement.

Original: Assume there exists an expert policy π E , such that policy optimization can be framed as minimizing the divergence min π D ( ρ π , ρ π E ) , that is, ﬁnding a policy π to imitate π E .
Revised: Assume there exists an expert policy π E , such that policy optimization can be framed as minimizing the divergence min π D ( ρ π , ρ π E ) , that is, ﬁnding a policy π to imitate π E .

Original: In practice, however, we do not have access to any real guiding expert policy.
Revised: In practice, however, we do not have access to any real guiding expert policy.

Original: Instead, we can maintain a selected subset M E of highly-rewarded trajectories from the previous rollouts of policy π , and optimize the policy π to minimize the divergence between ρ π and the empirical state-action pair distribution { ( s i , a i ) } M E :min [Equation]
Revised: Instead, we can maintain a selected subset M E of highly-rewarded trajectories from the previous rollouts of policy π , and optimize the policy π to minimize the divergence between ρ π and the empirical state-action pair distribution { ( s i , a i ) } M E :

Original: [Equation]
Revised: [Equation]

Original: Since it is not always possible to explicitly formulate ρ π even with the exact functional form of π , we generate rollouts from π in the environment and obtain an empirical distribution of ρ π .
Revised: Since it is not always possible to explicitly formulate ρ π even with the exact functional form of π , we generate rollouts from π in the environment and obtain an empirical distribution of ρ π .

Original: To measure the divergence between two empirical distributions, we use the Jensen-Shannon divergence, with the following variational form (up to a constant shift) as exploited in GANs (Goodfellow et al., 2014):
Revised: To measure the divergence between two empirical distributions, we use the Jensen-Shannon divergence, with the following variational form (up to a constant shift) as exploited in GANs (Goodfellow et al., 2014):

Original: [Equation]
Revised: [Equation]

Original: where d ( s, a ) and d E ( s, a ) are empirical density estimators of ρ π and ρ π E , respectively.
Revised: where d ( s, a ) and d E ( s, a ) are empirical density estimators of ρ π and ρ π E , respectively.

Original: Under certain assumptions, we can obtain an approximate gradient of D JS w.r.t the policy parameters, thus enabling us to optimize the policy.
Revised: Under certain assumptions, we can obtain an approximate gradient of D JS w.r.t the policy parameters, thus enabling us to optimize the policy.

Original: Gradient Approximation: Let ρ π
Revised: Gradient Approximation: Let ρ π

Original: ( s, a ) and ρ π E ( s, a ) be the state-action visitation distributions induced by two policies π and π E respectively.
Revised: ( s, a ) and ρ π E ( s, a ) be the state-action visitation distributions induced by two policies π and π E respectively.

Original: Let d π and d π E be the surrogates to ρ π and ρ π E , respectively, obtained by solving Equation 2.
Revised: Let d π and d π E be the surrogates to ρ π and ρ π E , respectively, obtained by solving Equation 2.

Original: Then, if the policy π is parameterized by θ , the gradient of D JS ( ρ π , ρ π E ) with respect to policy parameters ( θ ) can be approximated as :
Revised: Then, if the policy π is parameterized by θ , the gradient of D JS ( ρ π , ρ π E ) with respect to policy parameters ( θ ) can be approximated as :

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) Q 
Revised: where (cid:101) Q π ( s t , a t

Original: [Equation]
Revised: [Equation]

Revised: = (cid:101) E ρ π

Original: ( s
Revised: ( s,a

Original: [Equation]
Revised: [Equation]

Original: , a
Revised: (cid:2) ∞ (cid:88)

Original: [Equation]
Revised: [Equation]

Original:  )=
Revised: , a t (cid:48) ) d π ( s t (cid:48) , a t (cid:48) )

Original: [Equation] , a t (cid:48) ) (cid:3) ..
Revised: + d π E ( s t (cid:48) , a t (cid:48) ) (cid:3) ..

Original: The derivation of the approximation and the underlying assumptions are in Appendix 5.1.
Revised: The derivation of the approximation and the underlying assumptions are in Appendix 5.1.

Original: Next, we introduce a simple and inexpensive approach to construct the replay memory M E using highreturn past experiences during training.
Revised: Next, we introduce a simple and inexpensive approach to construct the replay memory M E using highreturn past experiences during training.

Original: In this way, ρ π E can be seen as a mixture of deterministic policies, each representing a delta point mass distribution in the trajectory space or a ﬁnite discrete visitation distribution of state-action pairs.
Revised: In this way, ρ π E can be seen as a mixture of deterministic policies, each representing a delta point mass distribution in the trajectory space or a ﬁnite discrete visitation distribution of state-action pairs.

Original: At each iteration, we apply the current policy π θ to sample b trajectories { τ } b 1 .
Revised: At each iteration, we apply the current policy π θ to sample b trajectories { τ } b 1 .

Original: We hope to include in M E , the top- k trajectories (or trajectories with returns above a threshold) generated thus far during the training process.
Revised: We hope to include in M E , the top- k trajectories (or trajectories with returns above a threshold) generated thus far during the training process.

Original: For this, we use a priorityqueue list for M E which keeps the trajectories sorted according to the total trajectory reward.
Revised: For this, we use a priorityqueue list for M E which keeps the trajectories sorted according to the total trajectory reward.

Original: The reward for each newly sampled trajectory in { τ } b 1 is compared with the current threshold of the priority-queue, updating M E accordingly.
Revised: The reward for each newly sampled trajectory in { τ } b 1 is compared with the current threshold of the priority-queue, updating M E accordingly.

Original: The frequency of updates is impacted by the exploration capabilities of the agent and the stochasticity in the environment.
Revised: The frequency of updates is impacted by the exploration capabilities of the agent and the stochasticity in the environment.

Original: We ﬁnd that simply sampling noisy actions from Gaussian policies is sufﬁcient for several locomotion tasks (Section 3).
Revised: We ﬁnd that simply sampling noisy actions from Gaussian policies is sufﬁcient for several locomotion tasks (Section 3).

Original: To handle more challenging environments, in the next sub-section, we augment our policy optimization procedure to explicitly enhance exploration and produce an ensemble of diverse policies.
Revised: To handle more challenging environments, in the next sub-section, we augment our policy optimization procedure to explicitly enhance exploration and produce an ensemble of diverse policies.

Original: In the usual imitation learning framework, expert demonstrations of trajectories—from external sources—are available as the empirical distribution of ρ π E of an expert policy π E .
Revised: In the usual imitation learning framework, expert demonstrations of trajectories—from external sources—are available as the empirical distribution of ρ π E of an expert policy π E .

Original: In our approach, since the agent learns by treating its own good past experiences as the expert, we can view the algorithm as self-imitation learning from experience replay.
Revised: In our approach, since the agent learns by treating its own good past experiences as the expert, we can view the algorithm as self-imitation learning from experience replay.

Original: As noted in Equation 3, the gradient estimator of D JS has a form similar to policy gradients, but
Revised: As noted in Equation 3, the gradient estimator of D JS has a form similar to policy gradients, but

Original: for replacing the true reward function with per-timestep reward deﬁned as log( d π ( s, a ) / ( d π ( s, a ) + d π E
Revised: for replacing the true reward function with per-timestep reward deﬁned as log( d π ( s, a ) / ( d π ( s, a ) + d π E

Original: ( s, a ))) .
Revised: ( s, a ))) .

Original: Therefore, it is possible to interpolate the gradient of D JS and the standard policy gradient.
Revised: Therefore, it is possible to interpolate the gradient of D JS and the standard policy gradient.

Original: We would highlight the beneﬁt of this interpolation soon.
Revised: We would highlight the beneﬁt of this interpolation soon.

Original: The net gradient on the policy parameters is:
Revised: The net gradient on the policy parameters is:

Original: [Equation]
Revised: [Equation]

Original: where Q r is the Q function with true rewards, and π E is the mixture policy represented by the samples in M E .
Revised: where Q r is the Q function with true rewards, and π E is the mixture policy represented by the samples in M E .

Original: Let r φ ( s, a ) =
Revised: Let r φ ( s, a ) =

Original: d π ( s, a ) /
Revised: d π ( s, a ) /

Original: [ d π ( s, a ) + d π E
Revised: [ d π ( s, a ) + d π E

Original: ( s, a )] .
Revised: ( s, a )] .

Original: r φ ( s, a ) can be computed using parameterized networks for densities d π and d π E , which are trained by solving the D JS optimization (Eq 2) using the current policy rollouts and M E , where φ includes the parameters for d π and d π E .
Revised: r φ ( s, a ) can be computed using parameterized networks for densities d π and d π E , which are trained by solving the D JS optimization (Eq 2) using the current policy rollouts and M E , where φ includes the parameters for d π and d π E .

Original: Using Equation 3, the interpolated gradient can be further simpliﬁed to:
Revised: Using Equation 3, the interpolated gradient can be further simpliﬁed to:

Original: [Equation]
Revised: [Equation]

Original:  where Q r φ ( s t , a t ) = − E p 0 ,p,π (cid:2) (cid:80) ∞ t (cid:48) =
Revised: (5) where Q r φ ( s t , a t ) = − E p 0 ,p,π (cid:2) (cid:80) ∞ t [Equation]

Revised: =

Original: t γ t (cid:48) − t log r φ ( s t (cid:48) , a t (cid:48) ) (cid:3) is the Q function calculated using − log r φ ( s, a ) as the reward.
Revised: t γ t [Equation] − t log r φ ( s t (cid:48) , a t (cid:48) ) (cid:3) is the Q function calculated using − log r φ ( s, a ) as the reward.

Original: This reward is high in the regions of the S ×
Revised: This reward is high in the regions of the S ×

Original: A space frequented more by the expert than the learner, and low in regions visited more by the learner than the expert.
Revised: A space frequented more by the expert than the learner, and low in regions visited more by the learner than the expert.

Original: The effective Q in Equation 5 is therefore an interpolation between Q r obtained with true environment rewards, and Q r φ obtained with rewards which are implicitly shaped to guide the learner towards expert behavior.
Revised: The effective Q in Equation 5 is therefore an interpolation between Q r obtained with true environment rewards, and Q r φ obtained with rewards which are implicitly shaped to guide the learner towards expert behavior.

Original: In environments with sparse or deceptive rewards, where the signal from
Revised: In environments with sparse or deceptive rewards, where the signal from

Original: Q r is weak or sub-optimal, a higher weight on Q r φ enables successful learning by imitation.
Revised: Q r is weak or sub-optimal, a higher weight on Q r φ enables successful learning by imitation.

Original: We show this empirically in our experiments.
Revised: We show this empirically in our experiments.

Original: We further ﬁnd that even in cases with dense environment rewards, the two gradient components can be successfully combined for policy optimization.
Revised: We further ﬁnd that even in cases with dense environment rewards, the two gradient components can be successfully combined for policy optimization.

Original: The complete algorithm for self-imitation is outlined in Appendix 5.2 (Algorithm 1).
Revised: The complete algorithm for self-imitation is outlined in Appendix 5.2 (Algorithm 1).

Original: Limitations of self-imitation.
Revised: Limitations of self-imitation.

Original: We now elucidate some shortcomings of the self-imitation approach.
Revised: We now elucidate some shortcomings of the self-imitation approach.

Original: Since the replay memory M E is only constructed from the past training rollouts, the quality of the trajectories in M E is hinged on good exploration by the agent.
Revised: Since the replay memory M E is only constructed from the past training rollouts, the quality of the trajectories in M E is hinged on good exploration by the agent.

Original: Consider a maze environment where the robot is only rewarded when it arrives at a goal G placed in a far-off corner.
Revised: Consider a maze environment where the robot is only rewarded when it arrives at a goal G placed in a far-off corner.

Original: Unless the robot reaches G once, the trajectories in M E always have a total reward of zero, and the learning signal from Q
Revised: Unless the robot reaches G once, the trajectories in M E always have a total reward of zero, and the learning signal from Q

Original: r φ is not useful.
Revised: r φ is not useful.

Original: Secondly, self-imitation can lead to sub-optimal policies when there are local minima in the policy optimization landscape; for example, assume the maze has a second goal
Revised: Secondly, self-imitation can lead to sub-optimal policies when there are local minima in the policy optimization landscape; for example, assume the maze has a second goal

Original: G (cid:48) in the opposite direction of G , but with a much smaller reward.
Revised: G (cid:48) in the opposite direction of G , but with a much smaller reward.

Original: With simple exploration, the agent may ﬁll M E with below-par trajectories leading to G (cid:48) , and the reinforcement from Q r φ would drive it further to G (cid:48) .
Revised: With simple exploration, the agent may ﬁll M E with below-par trajectories leading to G (cid:48) , and the reinforcement from Q r φ would drive it further to G (cid:48) .

Original: Thirdly, stochasticity in the environment may make it difﬁcult to recover the optimal policy just by imitating the past top- k rollouts.
Revised: Thirdly, stochasticity in the environment may make it difﬁcult to recover the optimal policy just by imitating the past top- k rollouts.

Original: For instance, in a 2-armed bandit problem with reward distributions Bernoulli (p) and Bernoulli (p+ (cid:15) ), rollouts from both the arms get conﬂated in
Revised: For instance, in a 2-armed bandit problem with reward distributions Bernoulli (p) and Bernoulli (p+ (cid:15) ), rollouts from both the arms get conﬂated in

Original: M E during training with high probability, making it hard to imitate the action of picking the arm with the higher expected reward.
Revised: M E during training with high probability, making it hard to imitate the action of picking the arm with the higher expected reward.

Original: We propose to overcome these pitfalls by training an ensemble of self-imitating agents, which are explicitly encouraged to visit different, non-overlapping regions of the state-space.
Revised: We propose to overcome these pitfalls by training an ensemble of self-imitating agents, which are explicitly encouraged to visit different, non-overlapping regions of the state-space.

Original: This helps to discover useful rewards in sparse settings, avoids deceptive reward traps, and in environments with reward-stochasticity like the 2-armed bandit, increases the probability of the optimal policy being present in the ﬁnal trained ensemble.
Revised: This helps to discover useful rewards in sparse settings, avoids deceptive reward traps, and in environments with reward-stochasticity like the 2-armed bandit, increases the probability of the optimal policy being present in the ﬁnal trained ensemble.

Original: We detail the enhancements next.
Revised: We detail the enhancements next.

Original: 2. I MPROVING E XPLORATION WITH S TEIN V ARIATIONAL G RADIENT
Revised: 2. I MPROVING E XPLORATION WITH S TEIN V ARIATIONAL G RADIENT

Original: One approach to achieve better exploration in challenging cases like above is to simultaneously learn multiple diverse policies and enforce them to explore different parts of the high dimensional space.
Revised: One approach to achieve better exploration in challenging cases like above is to simultaneously learn multiple diverse policies and enforce them to explore different parts of the high dimensional space.

Original: This can be achieved based on the recent work by Liu et al.
Revised: This can be achieved based on the recent work by Liu et al.

Original: (2017) on Stein variational policy gradient (SVPG).
Revised: (2017) on Stein variational policy gradient (SVPG).

Original: The idea of SVPG is to ﬁnd an optimal distribution q ( θ ) over the policy parameters θ which maximizes the expected policy returns, along with an entropy regularization
Revised: The idea of SVPG is to ﬁnd an optimal distribution q ( θ ) over the policy parameters θ which maximizes the expected policy returns, along with an entropy regularization

Original: that enforces diversity on the parameter space, i.e.
Revised: that enforces diversity on the parameter space, i.e.

Original: [Equation]
Revised: [Equation]

Original: Without a parametric assumption on q , this problem admits a challenging functional optimization problem.
Revised: Without a parametric assumption on q , this problem admits a challenging functional optimization problem.

Original: Stein variational gradient descent (SVGD, Liu & Wang (2016)) provides an efﬁcient solution for solving this problem, by approximating q with a delta measure
Revised: Stein variational gradient descent (SVGD, Liu & Wang (2016)) provides an efﬁcient solution for solving this problem, by approximating q with a delta measure

Original: q = (cid:80) ni
Revised: q = (cid:80) ni

Original: =1 δ
Revised: =1 δ

Original: θ i
Revised: θ i

Original: /n , where { θ i } ni =1 is an ensemble of policies, and iteratively update { θ i } with
Revised: /n , where { θ i } n i =1 is an ensemble of policies, and iteratively update { θ i } with

Original: [Equation]
Revised: [Equation]

Original: 6) where k ( θ j , θ i ) is a positive deﬁnite kernel function.
Revised: where k ( θ j , θ i ) is a positive deﬁnite kernel function.

Original: The ﬁrst term in ∆ θ i moves the policy to regions with high expected return (exploitation), while the second term creates a repulsion pressure between policies in the ensemble and encourages diversity (exploration).
Revised: The ﬁrst term in ∆ θ i moves the policy to regions with high expected return (exploitation), while the second term creates a repulsion pressure between policies in the ensemble and encourages diversity (exploration).

Original: The choice of kernel is critical.
Revised: The choice of kernel is critical.

Original: Liu et al.
Revised: Liu et al.

Original: (2017) used a simple Gaussian RBF kernel k ( θ j , θ i ) =
Revised: (2017) used a simple Gaussian RBF kernel k ( θ j , θ i ) =

Original: exp( −(cid:107) θ j − θ i (cid:107) 22 /h ) , with the bandwidth h dynamically adapted.
Revised: exp( −(cid:107) θ j − θ i (cid:107) 22 /h ) , with the bandwidth h dynamically adapted.

Original: This, however, assumes a ﬂat Euclidean distance between θ j and θ i , ignoring the structure of the entities deﬁned by them, which are probability distributions.
Revised: This, however, assumes a ﬂat Euclidean distance between θ j and θ i , ignoring the structure of the entities deﬁned by them, which are probability distributions.

Original: A statistical distance, such as D JS , serves as a better metric for comparing policies (Amari, 1998; Kakade, 2002).
Revised: A statistical distance, such as D JS , serves as a better metric for comparing policies (Amari, 1998; Kakade, 2002).

Original: Motivated by this, we propose to improve SVPG using JS kernel k ( θ j , θ i ) =
Revised: Motivated by this, we propose to improve SVPG using JS kernel k ( θ j , θ i ) =

Original: exp( − D JS ( ρ π θj , ρ π θi ) /T ) , where ρ π θ ( s, a ) is the state-action visitation distribution obtained by running policy π θ , and T is the temperature.
Revised: exp( − D JS ( ρ π θj , ρ π θi ) /T ) , where ρ π θ ( s, a ) is the state-action visitation distribution obtained by running policy π θ , and T is the temperature.

Original: The second exploration term in SVPG involves the gradient of the kernel w.r.t policy parameters.
Revised: The second exploration term in SVPG involves the gradient of the kernel w.r.t policy parameters.

Original: With the JS kernel, this requires estimating gradient of D JS , which as shown in Equation 3, can be obtained using policy gradients with an appropriately trained reward function.
Revised: With the JS kernel, this requires estimating gradient of D JS , which as shown in Equation 3, can be obtained using policy gradients with an appropriately trained reward function.

Original: Our full algorithm is summarized in Appendix 5.3 (Algorithm 2).
Revised: Our full algorithm is summarized in Appendix 5.3 (Algorithm 2).

Original: In each iteration, we apply the
Revised: In each iteration, we apply the

Original: SVPG gradient to each of the policies, where the ∇ θ η ( π θ ) in Equation 6 is the interpolated gradient from self-imitation (Equation 5).
Revised: SVPG gradient to each of the policies, where the ∇ θ η ( π θ ) in Equation 6 is the interpolated gradient from self-imitation (Equation 5).

Original: We also utilize state-value function networks as baselines to reduce the variance in sampled policy-gradients.
Revised: We also utilize state-value function networks as baselines to reduce the variance in sampled policy-gradients.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our goal in this section is to answer the following questions: 1) How does self-imitation fare against standard policy gradients under various reward distributions from the environment, namely episodic, noisy and dense?
Revised: Our goal in this section is to answer the following questions: 1) How does self-imitation fare against standard policy gradients under various reward distributions from the environment, namely episodic, noisy and dense?

Original: 2) How far does the SVPG exploration go in overcoming the limitations of selfimitation, such as susceptibility to local-minimas?
Revised: 2) How far does the SVPG exploration go in overcoming the limitations of selfimitation, such as susceptibility to local-minimas?

Original: We benchmark high-dimensional, continuous-control locomotion tasks based on the MuJoCo physics simulator by extending the OpenAI Baselines (Dhariwal et al., 2017) framework.
Revised: We benchmark high-dimensional, continuous-control locomotion tasks based on the MuJoCo physics simulator by extending the OpenAI Baselines (Dhariwal et al., 2017) framework.

Original: Our control policies ( θ i ) are modeled as unimodal Gaussians.
Revised: Our control policies ( θ i ) are modeled as unimodal Gaussians.

Original: All feed-forward networks have two layers of 64 hidden units each with tanh non-linearity.
Revised: All feed-forward networks have two layers of 64 hidden units each with tanh non-linearity.

Original: For policy-gradient, we use the clipped-surrogate based PPO algorithm (Schulman et al., 2017b).
Revised: For policy-gradient, we use the clipped-surrogate based PPO algorithm (Schulman et al., 2017b).

Original: Further implementation details are in the Appendix.
Revised: Further implementation details are in the Appendix.

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Original: 3. S ELF -I MITATION WITH D IFFERENT R EWARD D ISTRIBUTIONS
Revised: 3. S ELF -I MITATION WITH D IFFERENT R EWARD D ISTRIBUTIONS

Original: We evaluate the performance of self-imitation with a single agent in this sub-section; combination with SVPG exploration for multiple agents is discussed in the next.
Revised: We evaluate the performance of self-imitation with a single agent in this sub-section; combination with SVPG exploration for multiple agents is discussed in the next.

Original: We consider the locomotion tasks in OpenAI Gym under 3 separate reward distributions: Dense refers to the default reward function in Gym, which provides a reward for each simulation timestep.
Revised: We consider the locomotion tasks in OpenAI Gym under 3 separate reward distributions: Dense refers to the default reward function in Gym, which provides a reward for each simulation timestep.

Original: In episodic reward setting, rather than providing r ( s t , a t ) at each timestep of an episode, we provide (cid:80) t r ( s t , a t ) at the last timestep of the episode, and zero reward at other timesteps.
Revised: In episodic reward setting, rather than providing r ( s t , a t ) at each timestep of an episode, we provide (cid:80) t r ( s t , a t ) at the last timestep of the episode, and zero reward at other timesteps.

Original: This is the case for many practical settings where the reward function is hard to design, but scoring each trajectory, possibly by a human (Christiano et al., 2017), is feasible.
Revised: This is the case for many practical settings where the reward function is hard to design, but scoring each trajectory, possibly by a human (Christiano et al., 2017), is feasible.

Original: In noisy reward setting, we probabilistically mask out each out each per-timestep reward r ( a t , s t ) in an episode.
Revised: In noisy reward setting, we probabilistically mask out each out each per-timestep reward r ( a t , s t ) in an episode.

Original: Reward masking is done independently for every new episode, and therefore, the agent receives non-zero feedback at different—albeit only  few—timesteps in different episodes.
Revised: Reward masking is done independently for every new episode, and therefore, the agent receives non-zero feedback at different—albeit only [Figure] few—timesteps in different episodes.

Original: The probability of masking-out or suppressing the rewards is denoted by p m .
Revised: The probability of masking-out or suppressing the rewards is denoted by p m .

Original: [Figure]

Original: In Figure 1, we plot the learning curves on three tasks with episodic rewards.
Revised: In Figure 1, we plot the learning curves on three tasks with episodic rewards.

Original: Recall that ν is the hyper-parameter controlling the weight distribution between gradients with environment rewards and the gradients with shaped reward from r φ (Equation 5).
Revised: Recall that ν is the hyper-parameter controlling the weight distribution between gradients with environment rewards and the gradients with shaped reward from r φ (Equation 5).

Original: The baseline PPO agents use ν = 0 , meaning that the entire learning signal comes from the environment.
Revised: The baseline PPO agents use ν = 0 , meaning that the entire learning signal comes from the environment.

Original: We compare them with selfimitating (SI) agents using a constant value ν = 0 . 8 .
Revised: We compare them with selfimitating (SI) agents using a constant value ν = 0 . 8 .

Original: The capacity of M E is ﬁxed at 10 trajectories.
Revised: The capacity of M E is ﬁxed at 10 trajectories.

Original: We didn’t observe our method to be particularly sensitive to the choice of ν and the capacity value.
Revised: We didn’t observe our method to be particularly sensitive to the choice of ν and the capacity value.

Original: For instance, ν = 1 works equally well.
Revised: For instance, ν = 1 works equally well.

Original: Further ablation on these two hyper-parameters can be found in the Appendix.
Revised: Further ablation on these two hyper-parameters can be found in the Appendix.

Original: In Figure 1, we see that the PPO agents are unable to make any tangible progress on these tasks with episodic rewards, possibly due to difﬁculty in credit assignment – the lumped rewards at the end of the episode can’t be properly attributed to the individual state-action pairs during the episode.
Revised: In Figure 1, we see that the PPO agents are unable to make any tangible progress on these tasks with episodic rewards, possibly due to difﬁculty in credit assignment – the lumped rewards at the end of the episode can’t be properly attributed to the individual state-action pairs during the episode.

Original: In case of Self-Imitation, the algorithm has access to the shaped rewards for each timestep, derived from the high-return trajectories in M E .
Revised: In case of Self-Imitation, the algorithm has access to the shaped rewards for each timestep, derived from the high-return trajectories in M E .

Original: This makes credit-assignment easier, leading to successful learning even for very high-dimensional control tasks such as Humanoid.
Revised: This makes credit-assignment easier, leading to successful learning even for very high-dimensional control tasks such as Humanoid.

Original: Table 1 summarizes the ﬁnal performance, averaged over 5 runs with random seeds, under the various reward settings.
Revised: Table 1 summarizes the ﬁnal performance, averaged over 5 runs with random seeds, under the various reward settings.

Original: For the noisy rewards, we compare performance with two different reward masking values - suppressing each reward r ( s t , a t ) with 90% probability ( p m = 0 . 9 ), and with 50% probability ( p m = 0 . 5 ).
Revised: For the noisy rewards, we compare performance with two different reward masking values - suppressing each reward r ( s t , a t ) with 90% probability ( p m = 0 . 9 ), and with 50% probability ( p m = 0 . 5 ).

Original: The density of rewards increases across the reward settings from left to right in Table 1.
Revised: The density of rewards increases across the reward settings from left to right in Table 1.

Original: We ﬁnd that SI agents ( ν = 0 . 8 ) achieve higher average score than the baseline
Revised: We ﬁnd that SI agents ( ν = 0 . 8 ) achieve higher average score than the baseline

Original: PPO agents ( ν = 0 ) in majority of the tasks for all the settings.
Revised: PPO agents ( ν = 0 ) in majority of the tasks for all the settings.

Original: This indicates that not only does self-imitation vastly help when the environment rewards are scant, it can readily be incorporated with the standard policy gradients via interpolation, for successful learning across reward settings.
Revised: This indicates that not only does self-imitation vastly help when the environment rewards are scant, it can readily be incorporated with the standard policy gradients via interpolation, for successful learning across reward settings.

Original: For completion, we include performance of CEM and ES since these algorithms depend only on the total trajectory rewards and don’t exploit the temporal structure.
Revised: For completion, we include performance of CEM and ES since these algorithms depend only on the total trajectory rewards and don’t exploit the temporal structure.

Original: CEM perform poorly in most of the cases.
Revised: CEM perform poorly in most of the cases.

Original: ES, while being able to solve the tasks, is sample-inefﬁcient.
Revised: ES, while being able to solve the tasks, is sample-inefﬁcient.

Original: We include ES performance from Salimans et al.
Revised: We include ES performance from Salimans et al.

Original: (2017) after 5M timesteps of training for a fair comparison with our algorithm.
Revised: (2017) after 5M timesteps of training for a fair comparison with our algorithm.

Original: 3. C HARACTERIZING E NSEMBLE OF D IVERSE S ELF -I MITATING P OLICIES
Revised: 3. C HARACTERIZING E NSEMBLE OF D IVERSE S ELF -I MITATING P OLICIES

Original: We now conduct experiments to show how self-imitation can lead to sub-optimal policies in certain cases, and how the SVPG objective, which trains an ensemble with an explicit D JS repulsion between policies, can improve performance.
Revised: We now conduct experiments to show how self-imitation can lead to sub-optimal policies in certain cases, and how the SVPG objective, which trains an ensemble with an explicit D JS repulsion between policies, can improve performance.

Original: 2D-Navigation.
Revised: 2D-Navigation.

Original: Consider a simple Maze environment where the start location of the agent (blue particle) is shown in the ﬁgure on the right, along with two regions – the red region is closer to agent’s starting location but has a per-timestep reward of only 1 point if the agent hovers over it; the green region is on the other side of the wall but has a per-timestep reward of 10 points.
Revised: Consider a simple Maze environment where the start location of the agent (blue particle) is shown in the ﬁgure on the right, along with two regions – the red region is closer to agent’s starting location but has a per-timestep reward of only 1 point if the agent hovers over it; the green region is on the other side of the wall but has a per-timestep reward of 10 points.

Original: We run 8 independent, non-interacting, self-imitating (with ν = 0 . 8 ) agents on this task.
Revised: We run 8 independent, non-interacting, self-imitating (with ν = 0 . 8 ) agents on this task.

Original: This ensemble is denoted as SI-independent .
Revised: This ensemble is denoted as SI-independent .

Original: Figures 2a plots the state-visitation density for SI-independent after training, from which it is evident that the agents get trapped in the local minima.
Revised: Figures 2a plots the state-visitation density for SI-independent after training, from which it is evident that the agents get trapped in the local minima.

Original: The red-region is relatively easily explored and trajectories leading to it ﬁll the M E , causing sub-optimal imitation.
Revised: The red-region is relatively easily explored and trajectories leading to it ﬁll the M E , causing sub-optimal imitation.

Original: We contrast this with an instantiation of our full algorithm, which is referred to as SI-interact-JS .
Revised: We contrast this with an instantiation of our full algorithm, which is referred to as SI-interact-JS .

Original: It is composed of 8 self-imitating agents which share information for gradient calculation with the SVPG objective (Equation 6).
Revised: It is composed of 8 self-imitating agents which share information for gradient calculation with the SVPG objective (Equation 6).

Original: The temperature
Revised: The temperature

Original: T = 0 . 5 is held constant, and the weight on exploration-facilitating repulsion term ( α ) is linearly decayed over time.
Revised: T = 0 . 5 is held constant, and the weight on exploration-facilitating repulsion term ( α ) is linearly decayed over time.

Original: Figure 2b depicts the state-visitation density for this ensemble.
Revised: Figure 2b depicts the state-visitation density for this ensemble.

Original: SI-interact-JS explores wider portions of the maze, with multiple agents reaching the green zone of high reward.
Revised: SI-interact-JS explores wider portions of the maze, with multiple agents reaching the green zone of high reward.

Original: Figures 2c and 2d show the kernel matrices for the two ensembles after training.
Revised: Figures 2c and 2d show the kernel matrices for the two ensembles after training.

Original: Cell ( i, j ) in the matrix corresponds to the kernel value k
Revised: Cell ( i, j ) in the matrix corresponds to the kernel value k

Original: ( θ i , θ j ) =
Revised: ( θ i , θ j ) =

Original: exp( − JS ( ρ i , ρ j ) /T ) .
Revised: exp( − JS ( ρ i , ρ j ) /T ) .

Original: For SI-independent, many darker cells indicate that policies are closer (low JS).
Revised: For SI-independent, many darker cells indicate that policies are closer (low JS).

Original: For SI-interact-JS, which explicitly tries to decrease k
Revised: For SI-interact-JS, which explicitly tries to decrease k

Original: ( θ i , θ j ) , the cells are noticeably lighter, indicating dissimilar policies (high JS).
Revised: ( θ i , θ j ) , the cells are noticeably lighter, indicating dissimilar policies (high JS).

Original: Behavior of PPO-independent ( ν = 0 ) is similar to SI-independent ( ν = 0 . 8 ) for the Maze task.
Revised: Behavior of PPO-independent ( ν = 0 ) is similar to SI-independent ( ν = 0 . 8 ) for the Maze task.

Original: Locomotion.
Revised: Locomotion.

Original: To explore the limitations of self-imitation in harder exploration problems in highdimensional, continuous state-action spaces, we modify 3 MuJoCo tasks as follows – SparseHalfCheetah , SparseHopper and SparseAnt yield a forward velocity reward only when the centerof-mass of the corresponding bot is beyond a certain threshold distance.
Revised: To explore the limitations of self-imitation in harder exploration problems in highdimensional, continuous state-action spaces, we modify 3 MuJoCo tasks as follows – SparseHalfCheetah , SparseHopper and SparseAnt yield a forward velocity reward only when the centerof-mass of the corresponding bot is beyond a certain threshold distance.

Original: At all timesteps, there is an energy penalty to move the joints, and a survival bonus for bots that can fall over causing premature episode termination (Hopper, Ant).
Revised: At all timesteps, there is an energy penalty to move the joints, and a survival bonus for bots that can fall over causing premature episode termination (Hopper, Ant).

Original: Figure 3 plots the performance of PPO-independent,
Revised: Figure 3 plots the performance of PPO-independent,

Original: SI-independent, SI-interact-JS and SI-interact-RBF (which uses RBF-kernel from Liu et al.
Revised: SI-independent, SI-interact-JS and SI-interact-RBF (which uses RBF-kernel from Liu et al.

Original: (2017) instead of the JS-kernel) on the tasks.
Revised: (2017) instead of the JS-kernel) on the tasks.

Original: Each of these 4 algorithms is an ensemble of 8 agents using the same amount of simulation timesteps.
Revised: Each of these 4 algorithms is an ensemble of 8 agents using the same amount of simulation timesteps.

Original: The results are averaged over 3 separate runs, where for each run, the best agent from the ensemble after training is selected.
Revised: The results are averaged over 3 separate runs, where for each run, the best agent from the ensemble after training is selected.

Original: The SI-independent agents rely solely on action-space noise from the Gaussian policy parameterization to ﬁnd high-return trajectories which are added to M E as demonstrations.
Revised: The SI-independent agents rely solely on action-space noise from the Gaussian policy parameterization to ﬁnd high-return trajectories which are added to M E as demonstrations.

Original: This is mostly inadequate or slow for sparse environments.
Revised: This is mostly inadequate or slow for sparse environments.

Original: Indeed, we ﬁnd that all demonstrations in M E for
Revised: Indeed, we ﬁnd that all demonstrations in M E for

Original: SparseHopper are with the bot standing upright (or tilted) and gathering only the survival bonus, as action-space noise alone can’t discover hopping behavior.
Revised: SparseHopper are with the bot standing upright (or tilted) and gathering only the survival bonus, as action-space noise alone can’t discover hopping behavior.

Original: Similarly, for SparseHalfCheetah , M E has trajectories with the bot haphazardly moving back and forth.
Revised: Similarly, for SparseHalfCheetah , M E has trajectories with the bot haphazardly moving back and forth.

Original: On the other hand, in SI-interactJS, the D JS repulsion term encourages the agents to be diverse and explore the state-space much more effectively.
Revised: On the other hand, in SI-interactJS, the D JS repulsion term encourages the agents to be diverse and explore the state-space much more effectively.

Original: This leads to faster discovery of quality trajectories, which then provide good reinforcement through self-imitation, leading to higher overall score.
Revised: This leads to faster discovery of quality trajectories, which then provide good reinforcement through self-imitation, leading to higher overall score.

Original: SI-interact-RBF doesn’t perform as well, suggesting that the JS-kernel is more formidable for exploration.
Revised: SI-interact-RBF doesn’t perform as well, suggesting that the JS-kernel is more formidable for exploration.

Original: PPO-independent gets stuck in the local optimum for SparseHopper and SparseHalfCheetah – the bots stand still after training, avoiding energy penalty.
Revised: PPO-independent gets stuck in the local optimum for SparseHopper and SparseHalfCheetah – the bots stand still after training, avoiding energy penalty.

Original: For SparseAnt , the bot can cross our preset distance threshold using only action-space noise, but learning is slow due to na¨ıve exploration.
Revised: For SparseAnt , the bot can cross our preset distance threshold using only action-space noise, but learning is slow due to na¨ıve exploration.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: We approached policy optimization for deep RL from the perspective of JS-divergence minimization between state-action distributions of a policy and its own past good rollouts.
Revised: We approached policy optimization for deep RL from the perspective of JS-divergence minimization between state-action distributions of a policy and its own past good rollouts.

Original: This leads to a self-imitation algorithm which improves upon standard policy-gradient methods via the addition of a simple gradient term obtained from implicitly shaped dense rewards.
Revised: This leads to a self-imitation algorithm which improves upon standard policy-gradient methods via the addition of a simple gradient term obtained from implicitly shaped dense rewards.

Original: We observe substantial performance gains over the baseline for high-dimensional, continuous-control tasks with episodic and noisy rewards.
Revised: We observe substantial performance gains over the baseline for high-dimensional, continuous-control tasks with episodic and noisy rewards.

Original: Further, we discuss the potential limitations of the self-imitation approach, and propose ensemble training with the SVPG objective and JS-kernel as a solution.
Revised: Further, we discuss the potential limitations of the self-imitation approach, and propose ensemble training with the SVPG objective and JS-kernel as a solution.

Original: Through experi- mentation, we demonstrate the beneﬁts of a self-imitating, diverse ensemble for efﬁcient exploration and avoidance of local minima.
Revised: Through experimentation, we demonstrate the beneﬁts of a self-imitating, diverse ensemble for efﬁcient exploration and avoidance of local minima.

Original: An interesting future work is improving our algorithm using the rich literature on exploration in
Revised: An interesting future work is improving our algorithm using the rich literature on exploration in

Original: RL.
Revised: RL.

Original: Since ours is a population-based exploration method, techniques for efﬁcient single agent exploration can be readily combined with it.
Revised: Since ours is a population-based exploration method, techniques for efﬁcient single agent exploration can be readily combined with it.

Original: For instance, parameter-space noise or curiosity-driven exploration can be applied to each agent in the SI-interact-JS ensemble.
Revised: For instance, parameter-space noise or curiosity-driven exploration can be applied to each agent in the SI-interact-JS ensemble.

Original: Secondly, our algorithm for training diverse agents could be used more generally.
Revised: Secondly, our algorithm for training diverse agents could be used more generally.

Original: In Appendix 5. we show preliminary results for two cases: a) hierarchical RL, where a diverse group of Swimmer bots is trained for downstream use in a complex Swimming+Gathering task; b) RL without environment rewards, relying solely on diversity as the optimization objective.
Revised: In Appendix 5.6, we show preliminary results for two cases: a) hierarchical RL, where a diverse group of Swimmer bots is trained for downstream use in a complex Swimming+Gathering task; b) RL without environment rewards, relying solely on diversity as the optimization objective.

Original: Further investigation is left for future work
Revised: Further investigation is left for future work

Original:  A PPENDIX 5. D ERIVATION OF G RADIENT A PPROXIMATION
Revised:  A PPENDIX 5. D ERIVATION OF G RADIENT A PPROXIMATION

Original: Let d ∗ π ( s, a ) and d ∗ E ( s, a ) be the exact state-action densities for the current policy ( π θ ) and the expert, respectively.
Revised: Let d ∗ π ( s, a ) and d ∗ E ( s, a ) be the exact state-action densities for the current policy ( π θ ) and the expert, respectively.

Original: Therefore, by deﬁnition, we have (up to a constant shift):
Revised: Therefore, by deﬁnition, we have (up to a constant shift):

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: d ∗ π ( s, a ) is a local surrogate to ρ π θ ( s, a ) .
Revised: d ∗ π ( s, a ) is a local surrogate to ρ π θ ( s, a ) .

Original: By approximating it to be constant in an (cid:15) − ball neighborhood around θ
Revised: By approximating it to be constant in an (cid:15) − ball neighborhood around θ

Original: [Equation]
Revised: [Equation]

Original: we get the following after taking gradient of the above equation w.r.t θ :
Revised: we get the following after taking gradient of the above equation w.r.t θ :

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) 
Revised: where (cid:101) Q

Original: [Equation]
Revised: [Equation]

Original: The last step follows directly from the policy gradient theorem (Sutton et al., 2000).
Revised: The last step follows directly from the policy gradient theorem (Sutton et al., 2000).

Original: Since we do not have the exact densities d ∗ π ( s, a ) and d ∗ E ( s, a ) , we substitute them with the optimized density estimators d
Revised: Since we do not have the exact densities d ∗ π ( s, a ) and d ∗ E ( s, a ) , we substitute them with the optimized density estimators d

Original: π ( s, a ) and d E ( s, a ) from the maximization in Equation 2 for computing D JS .
Revised: π ( s, a ) and d E ( s, a ) from the maximization in Equation 2 for computing D JS .

Original: This gives us the gradient approximation mentioned in Section 2.2.
Revised: This gives us the gradient approximation mentioned in Section 2.2.

Original: A similar approximation is also used by Ho & Ermon (2016) for Generative Adversarial Imitation Learning (GAIL).
Revised: A similar approximation is also used by Ho & Ermon (2016) for Generative Adversarial Imitation Learning (GAIL).

Original: 5. A LGORITHM FOR S ELF -I MITATION Notation:
Revised: 5. A LGORITHM FOR S ELF -I MITATION Notation:

Original:  parameters parameters
Revised: θ = Policy parameters φ =

Revised: Discriminator parameters r ( a

Revised: = Environment reward

Original: 2 M E ← empty replay memory 3 for each iteration do
Revised: 2 M E ← empty replay memory 3 for each iteration do

Original: Generate batch of trajectories { τ } b 1 with two rewards for each transition: r 1 = r
Revised: Generate batch of trajectories { τ } b 1 with two rewards for each transition: r 1 = r

Original: ( s, a ) and r 2 = − log r φ ( s, a )
Revised: ( s, a ) and r 2 = − log r φ ( s, a )

Original: Update M E using priory queue threshold /* Update policy θ */ for each minibatch do
Revised: Update M E using priory queue threshold /* Update policy θ */ for each minibatch do

Original: Calculate g 1 = ∇ θ η r 1 ( π θ ) with PPO objective using r 1 reward
Revised: Calculate g 1 = ∇ θ η r 1 ( π θ ) with PPO objective using r 1 reward

Original: Calculate g 2 = ∇ θ η r 2 ( π θ ) with PPO objective using r 2 reward
Revised: Calculate g 2 = ∇ θ η r 2 ( π θ ) with PPO objective using r 2 reward

Original: Update θ with (1 − ν ) g 1 + νg 2 using ADAM end /
Revised: Update θ with (1 − ν ) g 1 + νg 2 using ADAM end /

Original: * Update self-imitation discriminator φ */ for each epoch do s 1 ← Sample mini-batch of (s,a) from M E s 2 ← Sample mini-batch of (s,a) from { τ } b
Revised: * Update self-imitation discriminator φ */ for each epoch do s 1 ← Sample mini-batch of (s,a) from M E s 2 ← Sample mini-batch of (s,a) from { τ } b

Original: Update φ with log-loss objective using s 1 , s end 16 end
Revised: Update φ with log-loss objective using s 1 , s end 16 end

Original: 5. A LGORITHM FOR S ELF -I MITATING D IVERSE P OLICIES
Revised: 5. A LGORITHM FOR S ELF -I MITATING D IVERSE P OLICIES

Original: Notation:
Revised: Notation:

Original: Algorithm 2: /
Revised: Algorithm 2: /

Original: * This is run for every rank i ∈ 1 . . .
Revised: * This is run for every rank i ∈ 1 . . .

Original: n */ 1 θ i , φ i , ψ i ∼ some initial distributions 2 M E ← empty replay memory local to rank i 3 k
Revised: n */ 1 θ i , φ i , ψ i ∼ some initial distributions 2 M E ← empty replay memory local to rank i 3 k

Original: ( i, j ) ← 0 , ∀ j (cid:54) =
Revised: ( i, j ) ← 0 , ∀ j (cid:54) =

Original: i 4 for each iteration do
Revised: i 4 for each iteration do

Original: Generate batch of trajectories { τ i } b
Revised: Generate batch of trajectories { τ i } b

Original: Update M E using priory queue threshold /* Update policy θ i */ for each minibatch do
Revised: Update M E using priory queue threshold /* Update policy θ i */ for each minibatch do

Original: Calculate ∇ θ i η
Revised: Calculate ∇ θ i η

Original: ( π θ i ) using self-imitation (as in Algorithm 1)
Revised: ( π θ i ) using self-imitation (as in Algorithm 1)

Original: MPI send: ∇ θ i η
Revised: MPI send: ∇ θ i η

Original: ( π θ i ) to other ranks
Revised: ( π θ i ) to other ranks

Original: MPI recv: ∇ θ j η ( π θ j ) from other ranks
Revised: MPI recv: ∇ θ j η ( π θ j ) from other ranks

Original: Calculate ∇ θ i
Revised: Calculate ∇ θ i

Original: k
Revised: k

Original: ( i, j ) using ψ i , ψ j
Revised: ( i, j ) using ψ i , ψ j

Original: Use k
Revised: Use k

Original: ( i, j ) and lines 8, 10, 11 in SVPG to get ∆ θ i
Revised: ( i, j ) and lines 8, 10, 11 in SVPG to get ∆ θ i

Original: Update θ i with ∆ θ i using ADAM end /
Revised: Update θ i with ∆ θ i using ADAM end /

Original: * Update self-imitation discriminator φ i */ for each epoch do s 1 ← Sample mini-batch of s,a) from M E s 2 ← Sample mini-batch s,a { τ i }b
Revised: * Update self-imitation discriminator φ i */ for each epoch do s 1 ← Sample mini-batch of s,a) from M E s 2 ← Sample mini-batch of s,a) i }

Original: Update φ i objective s 1 , s end /
Revised: Update φ i objective s 1 , s end /

Original: * Update state-action visitation network ψ i */
Revised: * Update state-action visitation network ψ i */

Original: MPI send: ψ i to other ranks
Revised: MPI send: ψ i to other ranks

Original: MPI send: { τ i } b 1 to other ranks
Revised: MPI send: { τ i } b 1 to other ranks

Original: MPI recv: ψ j from other ranks
Revised: MPI recv: ψ j from other ranks

Original: MPI recv: { τ j } b 1 from other ranks
Revised: MPI recv: { τ j } b 1 from other ranks

Original: Update ψ i with objective j , { τ
Revised: Update ψ i with objective j , { τ

Original: } b  , { τ j } b
Revised: } b 1 , { τ j } b

Original: Update k  i, j ) 26 end
Revised: Update k ( i, j ) 26 end

Original: 5. A BLATION S TUDIES
Revised: 5. A BLATION S TUDIES

Original: We show the sensitivity of self-imitation to ν and the capacity of M E , denoted by C .
Revised: We show the sensitivity of self-imitation to ν and the capacity of M E , denoted by C .

Original: The experiments in this subsection are done on Humanoid and Hopper tasks with episodic rewards.
Revised: The experiments in this subsection are done on Humanoid and Hopper tasks with episodic rewards.

Original: The tables show the average performance over 5 random seeds.
Revised: The tables show the average performance over 5 random seeds.

Original: For ablation on ν , C is ﬁxed at 10; for ablation on C , ν is ﬁxed at 0.8.
Revised: For ablation on ν , C is ﬁxed at 10; for ablation on C , ν is ﬁxed at 0.8.

Original: With episodic rewards, a higher value of ν helps boost performance since the RL signal from the environment is weak.
Revised: With episodic rewards, a higher value of ν helps boost performance since the RL signal from the environment is weak.

Original: With ν = 0 .
Revised: With ν = 0 .

Original: 8 , there isn’t a single best choice for C , though all values of C give better results than baseline PPO ( ν = 0 ).
Revised: 8 , there isn’t a single best choice for C , though all values of C give better results than baseline PPO ( ν = 0 ).

Original: Humanoid
Revised: Humanoid

Original: Hopper ν =
Revised: Hopper ν =

Original: [Table]
Revised: [Table]

Original: =0 .
Revised: =

Original: [Table]
Revised: [Table]

Original:  ν = 0 .
Revised: [Table] ν = [Table] [Table]

Original: ν =
Revised: =

Original: Humanoid
Revised: Humanoid

Original: Hopper
Revised: Hopper

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table]

Original:  = C =
Revised: C = C =

Original: [Table] =[Table]
Revised: C =

Original:  =
Revised: C =

Original: 5. H YPERPARAMETERS
Revised: 5. H YPERPARAMETERS

Original: - Horizon (T) =
Revised: - Horizon (T) =

Original: 1000 (locomotion), 250 (Maze), 5000 (Swimming+Gathering) - Discount ( γ ) = 0.
Revised: 1000 (locomotion), 250 (Maze), 5000 (Swimming+Gathering) - Discount ( γ ) = 0.

Original: - GAE parameter ( λ ) = 0. - PPO internal epochs =- PPO learning rate =
Revised: - GAE parameter ( λ ) = 0. - PPO internal epochs =- PPO learning rate =

Original: 1e- - PPO mini
Revised: 1e- - PPO mini

Original: batch =
Revised: batch =

Original: 5. L EVERAGING D IVERSE P OLICIES
Revised: 5. L EVERAGING D IVERSE P OLICIES

Original: The diversity-promoting D JS repulsion can be used for various other purposes apart from aiding exploration in the sparse environments considered thus far.
Revised: The diversity-promoting D JS repulsion can be used for various other purposes apart from aiding exploration in the sparse environments considered thus far.

Original: First, we consider the paradigm of hierarchical reinforcement learning wherein multiple sub-policies (or skills) are managed by a highlevel policy, which chooses the most apt sub-policy to execute at any given time.
Revised: First, we consider the paradigm of hierarchical reinforcement learning wherein multiple sub-policies (or skills) are managed by a highlevel policy, which chooses the most apt sub-policy to execute at any given time.

Original: In Figure 4, we use the Swimmer environment from Gym and show that diverse skills (movements) can be acquired in a pre-training phase when D JS repulsion is used.
Revised: In Figure 4, we use the Swimmer environment from Gym and show that diverse skills (movements) can be acquired in a pre-training phase when D JS repulsion is used.

Original: The skills can then be used in a difﬁcult downstream task.
Revised: The skills can then be used in a difﬁcult downstream task.

Original: During pre-training with SVPG, exploitation is done with policy-gradients calculated using the norm of the velocity as dense rewards, while the exploration term uses the JS-kernel.
Revised: During pre-training with SVPG, exploitation is done with policy-gradients calculated using the norm of the velocity as dense rewards, while the exploration term uses the JS-kernel.

Original: As before, we compare an ensemble of 8 interacting agents with 8 independent agents.
Revised: As before, we compare an ensemble of 8 interacting agents with 8 independent agents.

Original: Figures 4a and 4b depict the paths taken by the Swimmer after training with independent and interacting agents, respectively.
Revised: Figures 4a and 4b depict the paths taken by the Swimmer after training with independent and interacting agents, respectively.

Original: The latter exhibit variety.
Revised: The latter exhibit variety.

Original: Figure 4c is the downstream task of Swimming+Gathering (Duan et al., 2016) where the bot has to swim and collect the green dots, whilst avoiding the red ones.
Revised: Figure 4c is the downstream task of Swimming+Gathering (Duan et al., 2016) where the bot has to swim and collect the green dots, whilst avoiding the red ones.

Original: The utility of pre-training a diverse ensemble is shown in Figure 4d, which plots the performance on this task while training a higher-level categorical manager policy ( |A| = 8 ).
Revised: The utility of pre-training a diverse ensemble is shown in Figure 4d, which plots the performance on this task while training a higher-level categorical manager policy ( |A| = 8 ).

Original: Diversity can sometimes also help in learning a skill without any rewards from the environment, as observed by Eysenbach et al.
Revised: Diversity can sometimes also help in learning a skill without any rewards from the environment, as observed by Eysenbach et al.

Original: (2018) in recent work.
Revised: (2018) in recent work.

Original: We consider a Hopper task with no rewards, but we do require weak supervision in form of the length of each trajectory L .
Revised: We consider a Hopper task with no rewards, but we do require weak supervision in form of the length of each trajectory L .

Original: Using policy-gradient
Revised: Using policy-gradient

Original: [Figure]
Revised: [Figure]

Original: with L as reward and D JS repulsion, we see the emergence of hopping behavior within an ensemble of 8 interacting agents.
Revised: with L as reward and D JS repulsion, we see the emergence of hopping behavior within an ensemble of 8 interacting agents.

Original: Videos of the skills acquired can be found here 1 .
Revised: Videos of the skills acquired can be found here 1 .

Original: 5. P ERFORMANCE ON MORE M U J O C O TASKS
Revised: 5. P ERFORMANCE ON MORE M U J O C O TASKS

Original: Episodic rewards
Revised: Episodic rewards

Original: Noisy rewards
Revised: Noisy rewards

Original: Each r t suppressed w/ 90% prob.
Revised: Each r t suppressed w/ 90% prob.

Original: ( p m = 0 . 9 )
Revised: ( p m = 0 . 9 )

Original: Noisy rewards
Revised: Noisy rewards

Original: Each r t suppressed w/ 50% prob.
Revised: Each r t suppressed w/ 50% prob.

Original: ( p m = 0 . 5 )
Revised: ( p m = 0 . 5 )

Original: Dense rewards (Gym default) ν = 0 .(SI)
Revised: Dense rewards (Gym default) ν = 0 .(SI)

Original: ν =(PPO) ν
Revised: ν =(PPO) ν

Original: = 0 .(SI)
Revised: = 0 .(SI)

Original: ν =(PPO) ν
Revised: ν =(PPO) ν

Original: = 0 .(SI)
Revised: = 0 .(SI)

Original: ν =(PPO) ν
Revised: ν =(PPO) ν

Original: = 0 .(SI)
Revised: = 0 .(SI)

Original: ν =(PPO)
Revised: ν =(PPO)

Original: Half

Original: [Table]
Revised: [Table]

Original: Cheetah
Revised: Cheetah

Original: [Table]
Revised: [Table]

Original: 5. A DDITIONAL DETAILS ON SVPG EXPLORATION WITH JS- KERNEL 5.8. SVPG FORMULATION
Revised: 5. A DDITIONAL DETAILS ON SVPG EXPLORATION WITH JS- KERNEL 5.8. SVPG FORMULATION

Original: Let the policy parameters be parameterized by θ .
Revised: Let the policy parameters be parameterized by θ .

Original: To achieve diverse, high-return policies, we seek to obtain the distribution q ∗ ( θ ) which is the solution of the optimization problem: max q E θ ∼ q [ η ( θ )]
Revised: To achieve diverse, high-return policies, we seek to obtain the distribution q ∗ ( θ ) which is the solution of the optimization problem: max q E θ ∼ q [ η ( θ )]

Original: + αH ( q ) , where H ( q ) =
Revised: + αH ( q ) , where H ( q ) =

Original: E θ ∼ q [ − log q ( θ )] is the entropy of q .
Revised: E θ ∼ q [ − log q ( θ )] is the entropy of q .

Original: Solving the above equation by setting derivative to zero yields the an energy-based formulation for the optimal policy-parameter distribution: q ∗ ( θ ) ∝ exp( η ( θ ) α ) .
Revised: Solving the above equation by setting derivative to zero yields the an energy-based formulation for the optimal policy-parameter distribution: q ∗ ( θ ) ∝ exp( η ( θ ) α ) .

Original: Drawing samples from this posterior using traditional methods such as MCMC is computationally intractable.
Revised: Drawing samples from this posterior using traditional methods such as MCMC is computationally intractable.

Original: Stein variational gradient descent (SVGD; Liu & Wang (2016)) is an efﬁcient method for generating samples and also converges to the posterior of the energy-based model.
Revised: Stein variational gradient descent (SVGD; Liu & Wang (2016)) is an efﬁcient method for generating samples and also converges to the posterior of the energy-based model.

Original: Let { θ } n 1 be the n particles that constitute the policy ensemble.
Revised: Let { θ } n 1 be the n particles that constitute the policy ensemble.

Original: SVGD provides appropriate direction for perturbing each particle such that induced KL-divergence between the particles and the target distribution q ∗ ( θ ) is reduced.
Revised: SVGD provides appropriate direction for perturbing each particle such that induced KL-divergence between the particles and the target distribution q ∗ ( θ ) is reduced.

Original: The perturbation (gradient) for particle θ i is given by (please see Liu & Wang (2016) for derivation):
Revised: The perturbation (gradient) for particle θ i is given by (please see Liu & Wang (2016) for derivation):

Original: [Equation]
Revised: [Equation]

Original: where k ( θ j , θ i ) is a positive deﬁnite kernel function.
Revised: where k ( θ j , θ i ) is a positive deﬁnite kernel function.

Original: Using q ∗ ( θ ) ∝ exp( η ( θ ) α ) as target distribution, and k ( θ j , θ i ) =
Revised: Using q ∗ ( θ ) ∝ exp( η ( θ ) α ) as target distribution, and k ( θ j , θ i ) =

Original: exp( − D JS ( ρ π θj , ρ π θi ) /T )
Revised: exp( − D JS ( ρ π θj , ρ π θi ) /T )

Original: as the JS-kernel, we get the gradient direction for ascent:
Revised: as the JS-kernel, we get the gradient direction for ascent:

Original: [Equation]
Revised: [Equation]

Original: where ρ π θ ( s, a ) is the state-action visitation distribution for policy π θ , and T is the temperature.
Revised: where ρ π θ ( s, a ) is the state-action visitation distribution for policy π θ , and T is the temperature.

Original: Also, for our case, ∇ θ j η ( π θ j ) is the interpolated gradient from self-imitation (Equation 5).
Revised: Also, for our case, ∇ θ j η ( π θ j ) is the interpolated gradient from self-imitation (Equation 5).

Original: 5.8. I MPLEMENTATION DETAILS
Revised: 5.8. I MPLEMENTATION DETAILS

Original: The −∇ θ j D JS ( ρ π θj , ρ π θi ) gradient in the above equation is the repulsion factor that pushes π θ i away from π θ j .
Revised: The −∇ θ j D JS ( ρ π θj , ρ π θi ) gradient in the above equation is the repulsion factor that pushes π θ i away from π θ j .

Original: Similar repulsion can be achieved by using the gradient + ∇ θ i D JS ( ρ π θj , ρ π θi ) ; note that this gradient is w.r.t θ i instead of θ j and the sign is reversed.
Revised: Similar repulsion can be achieved by using the gradient + ∇ θ i D JS ( ρ π θj , ρ π θi ) ; note that this gradient is w.r.t θ i instead of θ j and the sign is reversed.

Original: Empirically, we ﬁnd that the latter results in slightly better performance.
Revised: Empirically, we ﬁnd that the latter results in slightly better performance.

Original: Estimation of ∇ θ i D JS ( ρ j , ρ i ) :
Revised: Estimation of ∇ θ i D JS ( ρ j , ρ i ) :

Original: This can be done in two ways - using implicit and explicit distributions.
Revised: This can be done in two ways - using implicit and explicit distributions.

Original: In the implicit method, we could train a parameterized discriminator network ( φ ) using state-actions pairs from π i and π j to implicitly approximate the ratio r φij =
Revised: In the implicit method, we could train a parameterized discriminator network ( φ ) using state-actions pairs from π i and π j to implicitly approximate the ratio r φij =

Original: ρ π i ( s, a ) /
Revised: ρ π i ( s, a ) /

Original: [ ρ π i ( s, a ) + ρ π j ( s, a )] .
Revised: [ ρ π i ( s, a ) + ρ π j ( s, a )] .

Original: We could then use the policy gradient theorem to obtain the gradient of D JS as explained in Section 2.2.
Revised: We could then use the policy gradient theorem to obtain the gradient of D JS as explained in Section 2.2.

Original: This, however, requires us to learn O ( n 2 ) discriminator networks for a population of size n , one for each policy pair ( i, j ) .
Revised: This, however, requires us to learn O ( n 2 ) discriminator networks for a population of size n , one for each policy pair ( i, j ) .

Original: To reduce the computational and memory resource burden to O ( n ) , we opt for explicit modeling of ρ π i .
Revised: To reduce the computational and memory resource burden to O ( n ) , we opt for explicit modeling of ρ π i .

Original: Speciﬁcally, we train a network ρ ψ i to approximate the state-action visitation density for each policy
Revised: Speciﬁcally, we train a network ρ ψ i to approximate the state-action visitation density for each policy

Original: π i .
Revised: π i .

Original: The ρ ψ 1 . . .
Revised: The ρ ψ 1 . . .

Original: ρ ψ n networks are learned using the D JS optimization (Equation 2), and we can easily obtain the ratio r ij ( s, a )
Revised: ρ ψ n networks are learned using the D JS optimization (Equation 2), and we can easily obtain the ratio r ij ( s, a )

Original: =
Revised: =

Original: ρ ψ i ( s, a ) /
Revised: ρ ψ i ( s, a ) /

Original: [ ρ ψ i ( s, a ) + ρ ψ j ( s, a )] .
Revised: [ ρ ψ i ( s, a ) + ρ ψ j ( s, a )] .

Original: The agent then uses log r ij ( s, a ) as the SVPG exploration rewards in the policy gradient theorem.
Revised: The agent then uses log r ij ( s, a ) as the SVPG exploration rewards in the policy gradient theorem.

Original: State-value baselines: We use state-value function networks as baselines to reduce the variance in sampled policy-gradients.
Revised: State-value baselines: We use state-value function networks as baselines to reduce the variance in sampled policy-gradients.

Original: Each agent θ i in a population of size n trains n + 1 state-value networks corresponding to real environment rewards r ( s, a ) , self-imitation rewards − log r φ ( s, a ) , and n −
Revised: Each agent θ i in a population of size n trains n + 1 state-value networks corresponding to real environment rewards r ( s, a ) , self-imitation rewards − log r φ ( s, a ) , and n −

Original: SVPG exploration rewards log r ij ( s, a ) .
Revised: SVPG exploration rewards log r ij ( s, a ) .

Original: 5. C OMPARISON TO O H ET AL . (2018)
Revised: 5. C OMPARISON TO O H ET AL . (2018)

Original: In this section, we provide evaluation for a recently proposed method for self-imitation learning (SIL;
Revised: In this section, we provide evaluation for a recently proposed method for self-imitation learning (SIL;

Original: Oh et al.
Revised: Oh et al.

Original: The SIL loss function take the form:
Revised: The SIL loss function take the form:

Original: [Equation]
Revised: [Equation]

Original: In words, the algorithm buffers ( s, a ) and the corresponding return ( R ) for each transition in rolled trajectories, and reuses them for training if the stored return value is higher than the current statevalue estimate V θ ( s ) .
Revised: In words, the algorithm buffers ( s, a ) and the corresponding return ( R ) for each transition in rolled trajectories, and reuses them for training if the stored return value is higher than the current statevalue estimate V θ ( s ) .

Original: We use the code provided by the authors 2 .
Revised: We use the code provided by the authors 2 .

Original: As per our understanding, PPO+SIL does not use a single set of hyper-parameters for all the MuJoCo tasks (Appendix A; Oh et al.
Revised: As per our understanding, PPO+SIL does not use a single set of hyper-parameters for all the MuJoCo tasks (Appendix A; Oh et al.

Original: We follow their methodology and report numbers for the best conﬁguration for each task.
Revised: We follow their methodology and report numbers for the best conﬁguration for each task.

Original: This is different from our experiments since we run all tasks on a single ﬁx hyper-parameter set (Appendix 5.5), and therefore a direct comparison of the average scores between the two approaches is tricky.
Revised: This is different from our experiments since we run all tasks on a single ﬁx hyper-parameter set (Appendix 5.5), and therefore a direct comparison of the average scores between the two approaches is tricky.

Original: [Table]
Revised: [Table]

Original: Table 3 shows the performance of PPO+SIL on MuJoCo tasks under the various reward distributions explained in Section 3.1 - dense, episodic and noisy.
Revised: Table 3 shows the performance of PPO+SIL on MuJoCo tasks under the various reward distributions explained in Section 3.1 - dense, episodic and noisy.

Original: We observe that, compared to the dense rewards setting (default Gym rewards), the performance suffers under the episodic case and when the rewards are masked out with p m = 0 .
Revised: We observe that, compared to the dense rewards setting (default Gym rewards), the performance suffers under the episodic case and when the rewards are masked out with p m = 0 .

Original: Our intuition is as follows.
Revised: Our intuition is as follows.

Original: PPO+SIL makes use of the cumulative return ( R ) from each transition of a past good rollout for the update.
Revised: PPO+SIL makes use of the cumulative return ( R ) from each transition of a past good rollout for the update.

Original: When rewards are provided only at the end of the episode, for instance, cumulative return does not help with the temporal credit assignment problem and hence is not a strong learning signal.
Revised: When rewards are provided only at the end of the episode, for instance, cumulative return does not help with the temporal credit assignment problem and hence is not a strong learning signal.

Original: Our approach, on the other hand, derives dense, per-timestep rewards using an objective based on divergence-minimization.
Revised: Our approach, on the other hand, derives dense, per-timestep rewards using an objective based on divergence-minimization.

Original: This is useful for credit assignment, and as indicated in Table 1.
Revised: This is useful for credit assignment, and as indicated in Table 1.

Original: (Section 3.1) leads to learning good policies even under the episodic and noisy p m = 0 .
Revised: (Section 3.1) leads to learning good policies even under the episodic and noisy p m = 0 .

Original: 9 settings.
Revised: 9 settings.

Original: 5. C OMPARISON TO OFF - POLICY RL (Q- LEARNING )
Revised: 5. C OMPARISON TO OFF - POLICY RL (Q- LEARNING )

Original: Our approach makes use of replay memory M E to store the past good rollouts of the agent.
Revised: Our approach makes use of replay memory M E to store the past good rollouts of the agent.

Original: Offpolicy RL methods such as DQN (Mnih et al., 2015) also accumulate agent experience in a replay buffer and reuse them for learning (e.g. by reducing TD-error).
Revised: Offpolicy RL methods such as DQN (Mnih et al., 2015) also accumulate agent experience in a replay buffer and reuse them for learning (e.g. by reducing TD-error).

Original: In this section, we evaluate the performance of one
Revised: In this section, we evaluate the performance of one

Original: such recent algorithm - Twin Delayed Deep Deterministic policy gradient (TD3; Fujimoto et al.
Revised: such recent algorithm - Twin Delayed Deep Deterministic policy gradient (TD3; Fujimoto et al.

Original: on tasks with episodic and noisy rewards.
Revised: on tasks with episodic and noisy rewards.

Original: TD3 builds on DDPG (Lillicrap et al., 2015) and surpasses its performance on all the MuJoCo tasks evaluated by the authors.
Revised: TD3 builds on DDPG (Lillicrap et al., 2015) and surpasses its performance on all the MuJoCo tasks evaluated by the authors.

Original: [Table]
Revised: [Table]

Original:  shows that the performance of TD3 suffers appreciably with the episodic and noisy p m =
Revised: Table 4 shows that the performance of TD3 suffers appreciably with the episodic and noisy p m =

Original: .reward
Revised: .reward

Original: settings, indicating that popular off-policy algorithms (DDPG, TD3) do not exploit the past experience in a manner that accelerates learning when rewards are scarce during an episode.
Revised: settings, indicating that popular off-policy algorithms (DDPG, TD3) do not exploit the past experience in a manner that accelerates learning when rewards are scarce during an episode.

Original: * For 3 tasks used in our paper—Swimmer and the high-dimensional Humanoid, HumanoidStandup—the TD3 code from the authors 3 is unable to learn a good policy even in presence of dense rewards (default Gym rewards).
Revised: * For 3 tasks used in our paper—Swimmer and the high-dimensional Humanoid, HumanoidStandup—the TD3 code from the authors 3 is unable to learn a good policy even in presence of dense rewards (default Gym rewards).

Original: These tasks are also not included in the evaluation by Fujimoto et al.
Revised: These tasks are also not included in the evaluation by Fujimoto et al.

Original: 5. C OMPARING SVPG EXPLORATION TO A NOVELTY - BASED BASELINE
Revised: 5. C OMPARING SVPG EXPLORATION TO A NOVELTY - BASED BASELINE

Original: We run a new exploration baseline - EX 2 (Fu et al., 2017) and compare its performance to SIinteract-JS on the hard exploration MuJoCo tasks considered in Section 3.2.
Revised: We run a new exploration baseline - EX 2 (Fu et al., 2017) and compare its performance to SIinteract-JS on the hard exploration MuJoCo tasks considered in Section 3.2.

Original: The EX 2 algorithm does implicit state-density ρ ( s ) estimation using discriminative modeling, and uses it for noveltybased exploration by adding − log ρ ( s ) as the bonus.
Revised: The EX 2 algorithm does implicit state-density ρ ( s ) estimation using discriminative modeling, and uses it for noveltybased exploration by adding − log ρ ( s ) as the bonus.

Original: We used the author provided code 4 and hyperparameter settings.
Revised: We used the author provided code 4 and hyperparameter settings.

Original: TRPO is used as the policy gradient algorithm.
Revised: TRPO is used as the policy gradient algorithm.

Original: EX
Revised: EX

Original: SI-interact-JS
Revised: SI-interact-JS

Original: SparseHalfCheetahSparseHopper
Revised: SparseHalfCheetah -

Revised: [Table]

Original: SparseAnt -3.
Revised: [Table]