Original: N OVEL P OLICY S EEKING WITH C ONSTRAINED O PTIMIZATION
Revised: N OVEL P OLICY S EEKING WITH C ONSTRAINED O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We address the problem of seeking novel policies in reinforcement learning tasks.
Revised: We address the problem of seeking novel policies in reinforcement learning tasks.

Original: Instead of following the multi-objective framework commonly used in existing methods, we propose to rethink the problem under a novel perspective of constrained optimization.
Revised: Instead of following the multi-objective framework commonly used in existing methods, we propose to rethink the problem under a novel perspective of constrained optimization.

Original: Weat ﬁrst introduce a new metric to evaluate the difference between policies, and then design two practical novel policy seeking methods following the new perspective, namely the Constrained Task Novel Bisector (CTNB), and the Interior Policy Differentiation (IPD), corresponding to the feasible direction method and the interior point method commonly known in the constrained optimization literature.
Revised: We ﬁrst introduce a new metric to evaluate the difference between policies, and then design two practical novel policy seeking methods following the new perspective, namely the Constrained Task Novel Bisector (CTNB), and the Interior Policy Differentiation (IPD), corresponding to the feasible direction method and the interior point method commonly known in the constrained optimization literature.

Original: Experimental comparisons on the MuJuCo control suite show our methods can achieve substantial improvements over previous novelty-seeking methods in terms of both the novelty of policies and their performances in the primal task.
Revised: Experimental comparisons on the MuJuCo control suite show our methods can achieve substantial improvement over previous novelty-seeking methods in terms of both the novelty of policies and their performances in the primal task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Inthe paradigm of Reinforcement Learning, an agent interacts with the environment to learn a policy that can maximize a certain form of cumulative rewards (Sutton & Barto, 1998). Modeling the policy function with a Deep Neural Network, the policy gradient method can be applied to optimize current policy  (Sutton et al., 2000).
Revised: In Reinforcement Learning, an agent interacts with the environment to learn a policy that can maximize a certain form of cumulative rewards (Sutton & Barto, 1998), while the policy gradient method can be applied to optimize parametric policy functions (Sutton et al., 2000).

Original: However, direct optimization with respect to the reward function is prone to get stuck in sub-optimal solutions and therefore hinders the policy optimization (Liepins & Vose, 1991; Lehman & Stanley, 2011; Plappert et al., 2018).
Revised: However, direct optimization with respect to the reward function is prone to get stuck in sub-optimal solutions and therefore hinders the policy optimization (Liepins & Vose, 1991; Lehman & Stanley, 2011; Plappert et al., 2018).

Original: Consequently, an appropriate exploration strategy is crucial for the success of policy learning (Auer, 2002;
Revised: Consequently, an appropriate exploration strategy is crucial for the success of policy learning (Auer, 2002;

Original: Bellemare et al., 2016; Houthooft et al., 2016;
Revised: Bellemare et al., 2016; Houthooft et al., 2016;

Original: Tang et al., 2017; Ostrovski et al., 2017; Tessler et al., 2019; Ciosek et al., 2019).
Revised: Tang et al., 2017; Ostrovski et al., 2017; Tessler et al., 2019; Ciosek et al., 2019).

Original: Recently many works have shown that incorporating curiosity in the policy learning leads to better exploration strategies (Pathak et al., 2017; Burda et al., 2018a;b; Liu et al., 2019).
Revised: Recently many works have shown that incorporating curiosity in the policy learning leads to better exploration strategies (Pathak et al., 2017; Burda et al., 2018a;b; Liu et al., 2019).

Original: In these works, visiting a previous unseen or infrequent state is assigned with an extra curiosity bonus reward.
Revised: In these works, visiting a previously unseen or infrequent state is assigned with an extra curiosity bonus reward.

Original: Different from those curiosity-driven methods which focus on the discovery of new states within the learning procedure of a repeated single policy, another direction Novel Policy Seeking (Lehman & Stanley, 2011; Zhang et al., 2019; Pugh et al., 2016) focuses on learning different policies with diverse or the so-called novel behaviors to solve the primal task.
Revised: Different from those curiosity-driven methods which focus on discovering new states within the learning procedure of a repeated single policy, the alternative approach of Novel Policy Seeking (Lehman & Stanley, 2011; Zhang et al., 2019; Pugh et al., 2016) focuses on learning different policies with diverse or the so-called novel behaviors to solve the primal task.

Original: In the process of novel policy seeking, policies in new iterations are usually encouraged to be different from previous policies.
Revised: In the process of novel policy seeking, policies in new iterations are usually encouraged to be different from previous policies.

Original: Therefore novel policy seeking can be viewed as an extrinsic curiosity-driven method at the level of policies, as well as an exploration strategy for a population of agents.
Revised: Therefore novel policy seeking can be viewed as an extrinsic curiosity-driven method at the level of policies, as well as an exploration strategy for a population of agents.

Original: Besides encouraging exploration (Eysenbach et al., 2018; Gangwani et al., 2018; Liu et al., 2017), novel policy seeking is also related to policy ensemble (Osband et al., 2018; 2016; Florensa et al., 2017) and evolution strategies (ES) (Salimans et al., 2017; Conti et al., 2018).
Revised: Besides encouraging exploration (Eysenbach et al., 2018; Gangwani et al., 2018; Liu et al., 2017), novel policy seeking is also related to policy ensemble (Osband et al., 2018; 2016; Florensa et al., 2017) and evolution strategies (ES) (Salimans et al., 2017; Conti et al., 2018).

Revised: In this work, we aim at generating a set of policies that behave differently from all previous given policies while trying to keep their primal task performance.

Original: In order to generate novel policies, previous work often deﬁnes a heuristic metric for novelty estimation, e.g., differences of state distributions estimated by neural networks are used in (Zhang et al., 2019), and tries to solve the problem under the formulation of multi-objective optimization.
Revised: In order to generate novel policies, previous work often deﬁnes a heuristic metric for novelty estimation, e.g., differences of state distributions estimated by neural networks are used in (Zhang et al., 2019), and tries to solve the problem under the formulation of multi-objective optimization.

Original: However, most of these metrics suffer from the difﬁculty when dealing with episodic novelty reward, i.e., the difﬁculty of episodic credit assignment (Sutton et al., 1998), thus their effectiveness in learning novel policies is limited.
Revised: However, most of these metrics suffer from the difﬁculty when dealing with episodic novelty reward, i.e., the difﬁculty of episodic credit assignment (Sutton et al., 1998), thus their effectiveness in learning novel policies is limited.

Original: Moreover, the difﬁculty of balancing different objectives impedes the agent to ﬁnd a well-performing policy for the primal task, as shown by Fig.

Original: 1 which compares the policy gradients of three cases, namely the one without novel policy seeking,

Original: novelty seeking with multi-objective optimization and novelty seeking with constrained optimization methods, respectively.

Original: [Figure]
Revised: [Figure]

Original: The multi-objective optimization method may impede the learning procedure when the novelty gradient is being applied all the time (Zhang et al., 2019), e.g., a random initialized policy will be penalized from getting closer to previous policy due to the conﬂict of gradients, which limits the learning efﬁciency and the ﬁnal performance. On the contrary, the novelty gradient of our constrained optimization approach will only be considered within a certain region to keep the  policy being optimized away from highly similar solutions. Such an approach is more ﬂexible and includes the multi-objective optimization method as its special case.
Revised: Moreover, the difﬁculty of balancing different objectives impedes the agent to ﬁnd a well-performing policy for the primal task, as shown by Fig. 1 which compares the policy gradients of three cases, namely the one without novel policy seeking, novelty seeking with multi-objective optimization and novelty seeking with constrained optimization methods, respectively.

Original: In this work, weintend to take into consideration both the novelty of learned policies as well as their performances in terms of the primal task when addressing the problem of novel policy seeking.
Revised: In this work we take into consideration both the novelty of the learning policies and the performance of the primal task when addressing the problem of novel policy seeking.

Original: Toachieve this goal, we propose to seek novel policies with a  constrained optimization formulation.
Revised: To this end we propose to seek novel policies under a formulation of constrained optimization.

Original: Twospeciﬁc algorithms under such a formulation are designed to seek novel policies while keeping their performances in the primal task, avoiding excessive novelty seeking.
Revised: Two algorithms under such a formulation are designed to seek novel policies while keeping their performances of the primal task, avoiding excessive novelty seeking.

Original: As a consequence, with these two algorithms, the performances of our learned novel policies can be guaranteed and even further improved.
Revised: As a result, the performances of our learned novel policies can be guaranteed and even further improved.

Original: Our contributions can be summarized in three-folds.
Revised: Our contributions can be summarized in three-folds.

Original: Firstly, we introduce a new metric to compute the difference between policies with instant feedback at every timestep; Secondly, we propose a constrained optimization formulation for novel policy seeking and design two practical algorithms resembling two approaches in constrained optimization literature; Thirdly, we evaluate our proposed algorithms on the MuJoCo locomotion environments, showing the advantages of these constrained optimization novelty-seeking methods which can generate a series of diverse and well-performing policies over previous multi-objective novelty seeking methods.
Revised: Firstly, we introduce a new metric to compute the difference between policies with instant feedback at every timestep; Secondly, we propose a constrained optimization formulation for novel policy seeking and design two practical algorithms resembling two approaches in constrained optimization literature; Thirdly, we evaluate our proposed algorithms on the MuJoCo locomotion environments, showing the advantages of these constrained optimization novelty-seeking methods which can generate a series of diverse and well-performing policies over previous multi-objective novelty seeking methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Intrinsic motivation methods In previous work, different approaches are proposed to provide intrinsic motivation or intrinsic reward as a supplementary to the primal task reward for better exploration (Houthooft et al., 2016;
Revised: Intrinsic motivation methods In previous work, different approaches are proposed to provide intrinsic motivation or intrinsic reward as a supplementary to the primal task reward for better exploration (Houthooft et al., 2016;

Original: Pathak et al., 2017; Burda et al., 2018a;b; Liu et al., 2019).
Revised: Pathak et al., 2017; Burda et al., 2018a;b; Liu et al., 2019).

Original: All those approaches leverage the weighted sum of two rewards, the primal rewards provided by environments, and intrinsic rewards that provided by different heuristics.
Revised: All those approaches leverage the weighted sum of two rewards, the primal rewards provided by environments, and intrinsic rewards that provided by different heuristics.

Original: On the other hand, the work of DIAYN and DADS (Eysenbach et al., 2018; Sharma et al., 2019) learn diverse skills without extrinsic reward.
Revised: On the other hand, the work of DIAYN and DADS (Eysenbach et al., 2018; Sharma et al., 2019) learn diverse skills without extrinsic reward.

Original: Those approaches focus on decomposing diverse skills of a single policy, while our work focuses on learning diverse behaviors among a batch of policies for the same task.
Revised: Those approaches focus on decomposing diverse skills of a single policy, while our work focuses on learning diverse behaviors among a batch of policies for the same task.

Original: Diverse policy seeking methods The work of Such et al. shows that different RL algorithms may converge to different policies for the same task (Such et al., 2018).
Revised: Diverse policy seeking methods The work of Such et al. shows that different RL algorithms may converge to different policies for the same task (Such et al., 2018).

Original: On the contrary, we are interested in how to learn different policies through a single learning algorithm with the capability of avoiding local optimum.
Revised: On the contrary, we are interested in how to learn different policies through a single learning algorithm with the capability of avoiding local optimum.

Original: The work of Pugh et al.
Revised: The work of Pugh et al.

Original: establishes a standard framework for understanding and comparing different approaches to search for quality diversity (QD)
Revised: establishes a standard framework for understanding and comparing different approaches to search for quality diversity (QD)

Original: (Pugh et al., 2016).
Revised: (Pugh et al., 2016).

Original: Conti et al.
Revised: Conti et al.

Original: proposes a solution which avoids local optima as well as achieves higher performance by adding novelty search and QD to evolution strategies (Conti et al., 2018).
Revised: proposes a solution which avoids local optima as well as achieves higher performance by adding novelty search and QD to evolution strategies (Conti et al., 2018).

Original: The Task-Novelty Bisector (TNB) (Zhang et al., 2019) aims to solve novelty seeking problem by jointly optimize the extrinsic rewards and novelty rewards deﬁned by an auto-encoder.
Revised: The Task-Novelty Bisector (TNB) (Zhang et al., 2019) aims to solve novelty seeking problem by jointly optimize the extrinsic rewards and novelty rewards deﬁned by an auto-encoder.

Original: In this work, one of the two proposed methods is closely related to TNB, but is adapted to the constrained optimization formulation.
Revised: In this work, one of the two proposed methods is closely related to TNB, but is adapted to the constrained optimization formulation.

Original: Constrained Markov Decision Process The Constrained Markov Decision Process (CMDP) (Altman, 1999) considers the situation where an agent interact with the environment under certain constraints.
Revised: Constrained Markov Decision Process The Constrained Markov Decision Process (CMDP) (Altman, 1999) considers the situation where an agent interact with the environment under certain constraints.

Original: Formally, the CMDP can be deﬁned as a tuple ( S , A , γ, r, c, C, P, s 0 ) , where S and A are the state and action space; γ ∈ [0 , 1) is a discount factor; r :
Revised: Formally, the CMDP can be deﬁned as a tuple ( S , A , γ, r, c, C, P, s 0 ) , where S and A are the state and action space; γ ∈ [0 , 1) is a discount factor; r :

Original: S ×A×S
Revised: S ×A×S

Original: → R and c : S ×A×S → R denote the reward function and cost function; C ∈ R + is the upper bound of permitted expected cumulative cost; P ( ·| s, a ) : S ×
Revised: → R and c : S ×A×S → R denote the reward function and cost function; C ∈ R + is the upper bound of permitted expected cumulative cost; P ( ·| s, a ) : S ×

Original: A → S denotes the transition dynamics, and s 0 is the initial state.
Revised: A → S denotes the transition dynamics, and s 0 is the initial state.

Original: Denote the Markovian policy class as Π , where Π = { π : S ×
Revised: Denote the Markovian policy class as Π , where Π = { π : S ×

Original: A → [0 , 1] , (cid:80) a π ( a | π ) = 1 } The learning objective of a policy for CMDP is to ﬁnd a π ∗ ∈ Π , such that
Revised: A → [0 , 1] , (cid:80) a π ( a | π ) = 1 } The learning objective of a policy for CMDP is to ﬁnd a π ∗ ∈ Π , such that

Original: [Equation]
Revised: [Equation]

Original: where τ indicates a trajectory ( s 0 , a 0 , s 1 , ... ) and τ ∼ π represents the distribution over trajectories following policy π : a t ∼ π ( ·| s t ) , s t +1 ∼ P ( ·| s t , a t ); t = 0 , 1 , 2 , ... .
Revised: where τ indicates a trajectory ( s 0 , a 0 , s 1 , ... ) and τ ∼ π represents the distribution over trajectories following policy π : a t ∼ π ( ·| s t ) , s t +1 ∼ P ( ·| s t , a t ); t = 0 , 1 , 2 , ... .

Original: Previous literature provide several approaches to solve CMDP (Achiam et al., 2017;
Revised: Previous literature provide several approaches to solve CMDP (Achiam et al., 2017;

Original: Chow et al., 2018; Ray et al., 2019), and in this work we include the CPO (Achiam et al., 2017) as baseline according to Ray et al.
Revised: Chow et al., 2018; Ray et al., 2019), and in this work we include the CPO (Achiam et al., 2017) as baseline according to Ray et al.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In Sec.3.1, we start with deﬁning a metric space that measures the difference between policies, which is the fundamental ingredient for the methods introduced later.
Revised: In Sec.3.1, we start with deﬁning a metric space that measures the difference between policies, which is the fundamental element for the proposed methods.

Original: In Sec.3.2, we develop a practical estimation method for this metric.
Revised: In Sec.3.2, we develop a practical estimation method for this metric.

Original: Sec.3.3 describes the formulation of constrained optimization on novel policy seeking.
Revised: Sec.3.3 describes the formulation of constrained optimization on novel policy seeking.

Original: The implementations of two practical algorithms are  introduced in Sec.3.4.
Revised: The implementations of two practical algorithms are further introduced in Sec.3.4.

Original: We denote the policies as { π θ i ; θ i ∈
Revised: We denote the policies as { π θ i ; θ i ∈

Original: Θ , i = 1 , 2 , ... } , wherein θ i represents parameters of the i -th policy, Θ denotes the whole parameter space.
Revised: Θ , i = 1 , 2 , ... } , wherein θ i represents parameters of the i -th policy, Θ denotes the whole parameter space.

Original: In this work, we focus on improving the behavior diversity of policies from PPO (Schulman et al., 2017), thus we use Θ to represent Θ PPO in this paper.
Revised: In this work, we focus on improving the behavior diversity of policies from PPO (Schulman et al., 2017), thus we use Θ to represent Θ PPO in this paper.

Original: It is worth noting that the proposed methods can be easily extended to other RL algorithms (Schulman et al., 2015; Lillicrap et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018).
Revised: It is worth noting that the proposed methods can be easily extended to other RL algorithms (Schulman et al., 2015; Lillicrap et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018).

Original: To simplify the notation, we omit π and denote a policy π θ i as θ i unless stated otherwise.
Revised: To simplify the notation, we omit π and denote a policy π θ i as θ i unless stated otherwise.

Original: 3. M EASURING THE D IFFERENCE BETWEEN P OLICIES
Revised: 3. M EASURING THE D IFFERENCE BETWEEN P OLICIES

Original: In this work, we use the Wasserstein metric W p (R ¨ uschendorf, 1985; Villani, 2008; Arjovsky et al., 2017) to measure the distance between policies.
Revised: In this work, we use the Wasserstein metric W p (R ¨ uschendorf, 1985; Villani, 2008; Arjovsky et al., 2017) to measure the distance between policies.

Original: Concretely, in this work we consider the
Revised: Concretely, in this work we consider the

Original: Gaussian-parameterized policies, where the W p over two policies can be written in the closed form W 22 ( N ( m 1 , Σ 1 ) , N ( m 2 , Σ 2 ))
Revised: Gaussian-parameterized policies, where the W p over two policies can be written in the closed form W 22 ( N ( m 1 , Σ 1 ) , N ( m 2 , Σ 2 ))

Original: =
Revised: =

Original: ||
Revised: ||

Original: m 1 − m 

Revised: m 1 − m

Original: || 2 + tr [Σ 1 + Σ 2 − 2(Σ 1 /
Revised: || 2 + tr [Σ 1 + Σ 2 − 2(Σ 1 /

Original: 2 1 Σ 2 Σ 1 / 2 1 ) 1 / 2 ] as p
Revised: 2 1 Σ 2 Σ 1 / 2 1 ) 1 / 2 ] as p

Original: = 2 , where m 1 , Σ 1 , m 2 , Σ 2 are mean and covariance metrics of the two normal distributions.
Revised: = 2 , where m 1 , Σ 1 , m 2 , Σ 2 are mean and covariance metrics of the two normal distributions.

Original: In the following of this paper, we use D W to denote the W 2 and it is worth noting that when the covariance matrix is identical, the trace term disappears and only the term involving the means remains, i.e.,
Revised: In the following of this paper, we use D W to denote the W 2 and it is worth noting that when the covariance matrix is identical, the trace term disappears and only the term involving the means remains, i.e.,

Original: D W = | m 1 − m 2 | for Dirac delta distributions located at points m 1 and m 2 .
Revised: D W = | m 1 − m 2 | for Dirac delta distributions located at points m 1 and m 2 .

Original: This diversity metric satisﬁes the three properties of a metric, namely identity, symmetry as well as triangle inequality.
Revised: This diversity metric satisﬁes the three properties of a metric, namely identity, symmetry as well as triangle inequality.

Original: Proposition 1 (Metric Space (Θ , D qW ) ) .
Revised: Proposition 1 (Metric Space (Θ , D qW ) ) .

Original: The expectation of D W ( · , · ) of two policies over any state distribution q ( s ) :
Revised: The expectation of D W ( · , · ) of two policies over any state distribution q ( s ) :

Original: [Equation]
Revised: [Equation]

Original: is a metric on Θ , thus (Θ , D qW ) is a metric space.
Revised: is a metric on Θ , thus (Θ , D qW ) is a metric space.

Original: The proof of Proposition 1 is straightforward.
Revised: The proof of Proposition 1 is straightforward.

Original: It is worth mentioning that Jensen Shannon divergence
Revised: It is worth mentioning that Jensen Shannon divergence

Original: D JS or Total Variance Distance D TV (Endres & Schindelin, 2003; Fuglede & Topsoe, 2004; Schulman et al., 2015) can also be applied as alternative metric spaces, we choose D W in our work for that the Wasserstein metric better preserves the continuity (Arjovsky et al., 2017).
Revised: D JS or Total Variance Distance D TV (Endres & Schindelin, 2003; Fuglede & Topsoe, 2004; Schulman et al., 2015) can also be applied as alternative metric spaces, we choose D W in our work for that the Wasserstein metric better preserves the continuity (Arjovsky et al., 2017).

Original: On top of the metric space (Θ , D qW ) , we can then compute the novelty of a policy as follows.
Revised: On top of the metric space (Θ , D qW ) , we can then compute the novelty of a policy as follows.

Original: Deﬁnition 1 (Novelty of Policy) .
Revised: Deﬁnition 1 (Novelty of Policy) .

Original: Given a reference policy set Θ ref such that Θ ref = { θ refi , i = 1 , 2 , ... } , Θ ref ⊂ Θ , the novelty U( θ | Θ ref ) of policy θ is the minimal difference between θ and all policies in the reference policy set, i.e.,
Revised: Given a reference policy set Θ ref such that Θ ref = { θ refi , i = 1 , 2 , ... } , Θ ref ⊂ Θ , the novelty U( θ | Θ ref ) of policy θ is the minimal difference between θ and all policies in the reference policy set, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Consequently, to encourage the discovery of novel policies discovery, typical novelty-seeking methods tend to directly maximize the novelty of a new policy, i.e., max θ U( θ | Θ ref ) , where the Θ ref includes all existing policies.
Revised: Consequently, to encourage the discovery of novel policies discovery, typical novelty-seeking methods tend to directly maximize the novelty of a new policy, i.e., max θ U( θ | Θ ref ) , where the Θ ref includes all existing policies.

Original: 3. E STIMATION OF D qW ( θ i , θ j ) AND THE S ELECTION OF q ( s )
Revised: 3. E STIMATION OF D qW ( θ i , θ j ) AND THE S ELECTION OF q ( s )

Original: In practice, the calculation of D qW ( θ i , θ j ) is based on Monte Carlo estimation where we need to sample s from q ( s ) .
Revised: In practice, the calculation of D qW ( θ i , θ j ) is based on Monte Carlo estimation where we need to sample s from q ( s ) .

Original: Although in Eq.(2) q ( s ) can be selected simply as a uniform distribution over the state space, there remains two obstacles: ﬁrst, in a ﬁnite state space we can get precise estimation after establishing ergodicity, but problem arises when facing continuous state spaces due to the difﬁculty of efﬁciently obtaining enough samples; second, when s is sampled from a uniform distribution q , we can only get sparse episodic reward instead of dense online reward which is more useful in learning.
Revised: Although in Eq.(2) q ( s ) can be selected simply as a uniform distribution over the state space, there remains two obstacles: ﬁrst, in a ﬁnite state space we can get precise estimation after establishing ergodicity, but problem arises when facing continuous state spaces due to the difﬁculty of efﬁciently obtaining enough samples; second, when s is sampled from a uniform distribution q , we can only get sparse episodic reward instead of dense online reward which is more useful in learning.

Original: Therefore, we make an approximation here based on importance sampling.
Revised: Therefore, we make an approximation here based on importance sampling.

Original: Formally, we denote the domain of q ( s ) as S q ⊂ S and assume q ( s ) to be a uniform distribution over
Revised: Formally, we denote the domain of q ( s ) as S q ⊂ S and assume q ( s ) to be a uniform distribution over

Original: S q , without loss of generality in later analysis.
Revised: S q , without loss of generality in later analysis.

Original: Notice S q is closely related to the algorithm being used in generating trajectories (Henderson et al., 2018).
Revised: Notice S q is closely related to the algorithm being used in generating trajectories (Henderson et al., 2018).

Original: As we only care about the reachable regions of a certain algorithm (in this work, PPO), the domain S q can be decomposed by S q = lim N →∞ (cid:83) N i =1 S θ i , where
Revised: As we only care about the reachable regions of a certain algorithm (in this work, PPO), the domain S q can be decomposed by S q = lim N →∞ (cid:83) N i =1 S θ i , where

Original: S θ
Revised: S θ

Original: i denotes all the possible states a policy θ i can visit given a starting state distribution.
Revised: i denotes all the possible states a policy θ i can visit given a starting state distribution.

Original: In order to get online-reward, we estimate Eq.(2) with
Revised: In order to get online-reward, we estimate Eq.(2) with

Original: [Equation]
Revised: [Equation]

Original: where we use ρ θ ( s ) to denote the stationary state visitation frequency under policy θ , i.e.
Revised: where we use ρ θ ( s ) to denote the stationary state visitation frequency under policy θ , i.e.

Original: , ρ θ ( s ) =
Revised: , ρ θ ( s ) =

Original: P ( s 0 = s | θ )
Revised: P ( s 0 = s | θ )

Original: + P ( s 1 = s | θ ) + ...
Revised: + P ( s 1 = s | θ ) + ...

Original: + P ( s T = s | θ ) in ﬁnite horizon problems.
Revised: + P ( s T = s | θ ) in ﬁnite horizon problems.

Original: We propose to use the averaged stationary visitation frequency as q ( s ) , e.g., for PPO, q ( s ) =
Revised: We propose to use the averaged stationary visitation frequency as q ( s ) , e.g., for PPO, q ( s ) =

Original: ρ ( s ) =
Revised: ρ ( s ) =

Original: E θ ∼ Θ PPO [ ρ θ ( s )] .
Revised: E θ ∼ Θ PPO [ ρ θ ( s )] .

Original: Clearly, choosing q ( s ) =
Revised: Clearly, choosing q ( s ) =

Original: ρ ( s ) will be much better than choosing a uniform distribution as the importance weight will be closer to 1 .
Revised: ρ ( s ) will be much better than choosing a uniform distribution as the importance weight will be closer to 1 .

Original: Such an importance sampling process requires a necessary condition that ρ θ i ( s ) and q ( s ) have the same domain, which can be guaranteed by applying a sufﬁcient exploration noise on θ .
Revised: Such an importance sampling process requires a necessary condition that ρ θ i ( s ) and q ( s ) have the same domain, which can be guaranteed by applying a sufﬁcient exploration noise on θ .

Original: Another difﬁculty lies in the estimation of ρ ( s ) , which is always intractable given a limited number of trajectories.
Revised: Another difﬁculty lies in the estimation of ρ ( s ) , which is always intractable given a limited number of trajectories.

Original: However, during training, θ i is a policy to be optimized and θ j ∈ Θ ref is a ﬁxed reference policy.
Revised: However, during training, θ i is a policy to be optimized and θ j ∈ Θ ref is a ﬁxed reference policy.

Original: The error introduced by approximating the importance weight as 1 will get larger when θ i becomes more distinct from normal policies, at least in terms of the state visitation frequency.
Revised: The error introduced by approximating the importance weight as 1 will get larger when θ i becomes more distinct from normal policies, at least in terms of the state visitation frequency.

Original: We may just regard increasing of the approximation error as the discovery of novel policies.
Revised: We may just regard increasing of the approximation error as the discovery of novel policies.

Original: Proposition 2 (Unbiased Single Trajectory Estimation) .
Revised: Proposition 2 (Unbiased Single Trajectory Estimation) .

Original: The estimation of ρ θ ( s ) using a single trajectory τ is unbiased.
Revised: The estimation of ρ θ ( s ) using a single trajectory τ is unbiased.

Original: The Proposition 2 follows the usual trick in RL that uses a single trajectory to estimate the stationary state visitation frequency.
Revised: The Proposition 2 follows the usual trick in RL that uses a single trajectory to estimate the stationary state visitation frequency.

Original: Given the deﬁnition of novelty and a practically unbiased sampling method, the next step is to develop an efﬁcient learning algorithm.
Revised: Given the deﬁnition of novelty and a practically unbiased sampling method, the next step is to develop an efﬁcient learning algorithm.

Original: 3. C ONSTRAINED O PTIMIZATION F ORMULATION FOR N OVEL P OLICY S EEKING
Revised: 3. C ONSTRAINED O PTIMIZATION F ORMULATION FOR N OVEL P OLICY S EEKING

Original: In the traditional RL paradigm, maximizing the expectation of cumulative rewards is commonly used as the objective.
Revised: In the traditional RL paradigm, maximizing the expectation of cumulative rewards is commonly used as the objective.

Original: i.e., max θ ∈ Θ E τ ∼ θ [ g ] , where g = (cid:80) t =

Revised: i.e., max θ ∈ Θ E τ ∼ θ [ g ] , where g = (cid:80) t =

Original: γ t r t and τ ∼ θ denotes a trajectory τ sampled from the policy θ .
Revised: γ t r t and τ ∼ θ denotes a trajectory τ sampled from the policy θ .

Original: To improve the diversity of different agents’ behaviors, the learning objective must take both the reward from the primal task and the policy novelty into consideration.
Revised: To improve the diversity of different agents’ behaviors, the learning objective must take both the reward from the primal task and the policy novelty into consideration.

Original: Previous approaches (Houthooft et al., 2016; Pathak et al., 2017; Burda et al., 2018a;b; Liu et al., 2019) often directly use the weighted sum of these two terms as the objective:
Revised: Previous approaches (Houthooft et al., 2016; Pathak et al., 2017; Burda et al., 2018a;b; Liu et al., 2019) often directly use the weighted sum of these two terms as the objective:

Original: [Equation]
Revised: [Equation]

Original: where 0 < α < 1 is a weight hyper-parameter, g task is the reward from the primary task, and g int = (cid:80) t =

Revised: where 0 < α < 1 is a weight hyper-parameter, g task is the reward from the primary task, and g int = (cid:80) t =

Original: γ t r int ,t is the cumulative intrinsic reward of the intrinsic reward r int ,t .
Revised: γ t r int ,t is the cumulative intrinsic reward of the intrinsic reward r int ,t .

Original: In our case, the intrinsic reward is the novelty reward r int = min θ j ∈ Θ ref D ρ W ( θ, θ j ) .
Revised: In our case, the intrinsic reward is the novelty reward r int = min θ j ∈ Θ ref D ρ W ( θ, θ j ) .

Original: These methods can be summarized as Weighted Sum Reward (WSR) methods (Zhang et al., 2019).
Revised: These methods can be summarized as Weighted Sum Reward (WSR) methods (Zhang et al., 2019).

Original: Such an objective is sensitive to the selection of α as well as the formulation of r int .
Revised: Such an objective is sensitive to the selection of α as well as the formulation of r int .

Original: For example, in our case formulating the novelty reward r int as min θ j D ρW
Revised: For example, in our case formulating the novelty reward r int as min θ j D ρW

Original: ( θ, θ j ) , exp [min θ j D ρW ( θ, θ j )] and − exp [ − min θ
Revised: ( θ, θ j ) , exp [min θ j D ρW ( θ, θ j )] and − exp [ − min θ

Original: j D ρW
Revised: j D ρW

Original: ( θ, θ j )] will lead to signiﬁcantly different results as they determine the trade-offs in the two terms given α .
Revised: ( θ, θ j )] will lead to signiﬁcantly different results as they determine the trade-offs in the two terms given α .

Original: Besides, dilemma also arises in the selection of α : while a large α may undermine the contribution of intrinsic reward, a small α could ignore the importance of the primal task, leading to the failure of an agent in solving the task.
Revised: Besides, dilemma also arises in the selection of α : while a large α may undermine the contribution of intrinsic reward, a small α could ignore the importance of the primal task, leading to the failure of an agent in solving the task.

Original: To tackle such an issue, the crux is to deal with the conﬂict between different objectives.
Revised: To tackle such an issue, the crux is to deal with the conﬂict between different objectives.

Original: The work of Zhang et al.
Revised: The work of Zhang et al.

Original: proposes the TNB, where the task reward is regarded as the dominant one while the novelty reward is regarded as subordinate Zhang et al.
Revised: proposes the TNB, where the task reward is regarded as the dominant one while the novelty reward is regarded as subordinate Zhang et al.

Original: However, as TNB considers the novelty gradient all the time, it may hinder the learning process, e.g., Intuitively, well-performing policies should be more similar to each other than to random initialized policies.
Revised: However, as TNB considers the novelty gradient all the time, it may hinder the learning process, e.g., Intuitively, well-performing policies should be more similar to each other than to random initialized policies.

Original: As a new random initialized policy is different enough from previous policies, considering the novelty gradient at beginning of training will result in a much slower learning process.
Revised: As a new random initialized policy is different enough from previous policies, considering the novelty gradient at beginning of training will result in a much slower learning process.

Original: In order to tackle the above problems and adjust the extent of novelty in new policies, we propose to solve the novelty-seeking problem under the perspective of constrained optimization.
Revised: In order to tackle the above problems and adjust the extent of novelty in new policies, we propose to solve the novelty-seeking problem under the perspective of constrained optimization.

Original: The basic idea is as follows: while the task reward is considered as a learning objective, the novelty reward should be considered as a bonus instead of another objective, and should not impede the learning of the primal task.
Revised: The basic idea is as follows: while the task reward is considered as a learning objective, the novelty reward should be considered as a bonus instead of another objective, and should not impede the learning of the primal task.

Original: Fig.
Revised: Fig.

Original: 1 illustrates how novelty gradients impede the learning of a policy: at the beginning of learning, a random initialized policy should in total learn to be more similar to a well-performing policy rather than be different.
Revised: 1 illustrates how novelty gradients impede the learning of a policy: at the beginning of learning, a random initialized policy should in total learn to be more similar to a well-performing policy rather than be different.

Original: The seeking of novelty should not be taken into consideration all the time during learning.
Revised: The seeking of novelty should not be taken into consideration all the time during learning.

Original: With such an insight, we change the multi-objective optimization problem in
Revised: With such an insight, we change the multi-objective optimization problem in

Original: Eq.(5) into a constrained optimization problem as:
Revised: Eq.(5) into a constrained optimization problem as:

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: where r 0 is a threshold indicating minimal permitted novelty, and r int ,t denotes a moving average of r int ,t .
Revised: where r 0 is a threshold indicating minimal permitted novelty, and r int ,t denotes a moving average of r int ,t .

Original: as we need not force every single action of a new agent to be different from others.
Revised: as we need not force every single action of a new agent to be different from others.

Original: Instead, we care more about the long-term differences.
Revised: Instead, we care more about the long-term differences.

Original: Therefore, we use cumulative novelty terms as constraints.
Revised: Therefore, we use cumulative novelty terms as constraints.

Original: Moreover, the constraints can be ﬂexibly applied after the ﬁrst t S timesteps (e.g., t S = 20 ) for the consideration of similar starting sequences, so that the constraints can be written as g t ( θ ) ≥ 0 , ∀ t = t S , ..., T .
Revised: Moreover, the constraints can be ﬂexibly applied after the ﬁrst t S timesteps (e.g., t S = 20 ) for the consideration of similar starting sequences, so that the constraints can be written as g t ( θ ) ≥ 0 , ∀ t = t S , ..., T .

Original: 3. P RACTICAL N OVEL P OLICY S EEKING M ETHODS
Revised: 3. P RACTICAL N OVEL P OLICY S EEKING M ETHODS

Original: We note here, WSR and TNB proposed in previous work (Zhang et al., 2019) can correspond to different approaches in constrained optimization problems, yet some important ingredients are missing.
Revised: We note here, WSR and TNB proposed in previous work (Zhang et al., 2019) can correspond to different approaches in constrained optimization problems, yet some important ingredients are missing.

Original: We improve TNB according to the Feasible Direction Method in constrained optimization and then propose the Interior Policy Differentiation (IPD) method according to the Interior Point
Revised: We improve TNB according to the Feasible Direction Method in constrained optimization and then propose the Interior Policy Differentiation (IPD) method according to the Interior Point

Original: Method in constrained optimization.
Revised: Method in constrained optimization.

Original: WSR: Penalty Method The Penalty Method considers the constraints of Eq.(6) by putting constraint g ( θ ) into a penalty term, followed by solving the unconstrained problem
Revised: WSR: Penalty Method The Penalty Method considers the constraints of Eq.(6) by putting constraint g ( θ ) into a penalty term, followed by solving the unconstrained problem

Original: [Equation]
Revised: [Equation]

Original: in an iterative manner.
Revised: in an iterative manner.

Original: The limit of the above unconstrained problem when α → 0 then leads to the solution of the original constrained problem.
Revised: The limit of the above unconstrained problem when α → 0 then leads to the solution of the original constrained problem.

Original: As an approximation, WSR chooses a ﬁxed weight α , and uses the gradient of ∇ θ f + 1 − αα ∇ θ g instead of ∇ θ f + 1 − αα ∇ θ min { g ( θ ) , 0 } , thus the ﬁnal solution will intensely rely on the selection of α .
Revised: As an approximation, WSR chooses a ﬁxed weight α , and uses the gradient of ∇ θ f + 1 − αα ∇ θ g instead of ∇ θ f + 1 − αα ∇ θ min { g ( θ ) , 0 } , thus the ﬁnal solution will intensely rely on the selection of α .

Original: TNB:
Revised: TNB:

Original: Feasible Direction Method The Feasible Direction Method (FDM) (Ruszczy´nski, 1980;
Revised: Feasible Direction Method The Feasible Direction Method (FDM) (Ruszczy´nski, 1980;

Original: Herskovits, 1998) solves the constrained optimization problem by ﬁnding a direction (cid:126)p where taking gradient upon will lead to increment of the objective function as well as constraints satisfaction, i.e., ∇ θ f T · (cid:126)p > 0 , if g > 0 and ∇ θ g T · (cid:126)p > 0 otherwise.
Revised: Herskovits, 1998) solves the constrained optimization problem by ﬁnding a direction (cid:126)p where taking gradient upon will lead to increment of the objective function as well as constraints satisfaction, i.e., ∇ θ f T · (cid:126)p > 0 , if g > 0 and ∇ θ g T · (cid:126)p > 0 otherwise.

Original: The TNB proposes to use a revised bisector of gradients ∇ θ f and ∇ θ g as (cid:126)p ,
Revised: The TNB proposes to use a revised bisector of gradients ∇ θ f and ∇ θ g as (cid:126)p ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Clearly, Eq.(8) satisﬁes the constraints but it is more strict than it as the ∇ θ g term always exists during the optimization of TNB.
Revised: Clearly, Eq.(8) satisﬁes the constraints but it is more strict than it as the ∇ θ g term always exists during the optimization of TNB.

Original: Based on TNB, we provide a revised approach, named Constrained
Revised: Based on TNB, we provide a revised approach, named Constrained

Original: Task Novel Bisector (CTNB), which resembles better with FDM.
Revised: Task Novel Bisector (CTNB), which resembles better with FDM.

Original: Speciﬁcally, when g > 0 , CTNB will not apply ∇ θ g on g .
Revised: Speciﬁcally, when g > 0 , CTNB will not apply ∇ θ g on g .

Original: It is clear that TNB is a special case of CTNB when the novelty threshold ris set to inﬁnity.
Revised: It is clear that TNB is a special case of CTNB when the novelty threshold ris set to inﬁnity.

Original: We note that in both TNB and CTNB, the learning stride is ﬁxed to be |∇ θ f | + |∇ θ g |and may lead to problem when ∇ θ f → 0 , where the ﬁnal optimization result will rely heavily on the selection of g , i.e., the shape of g is crucial for the success of this approach.
Revised: We note that in both TNB and CTNB, the learning stride is ﬁxed to be |∇ θ f | + |∇ θ g |and may lead to problem when ∇ θ f → 0 , where the ﬁnal optimization result will rely heavily on the selection of g , i.e., the shape of g is crucial for the success of this approach.

Original: IPD: Interior Point
Revised: IPD: Interior Point

Original: Method The Interior Point Method (Potra & Wright, 2000; Dantzig & Thapa, 2006) is another approach used to solve the constrained optimization problem.
Revised: Method The Interior Point Method (Potra & Wright, 2000; Dantzig & Thapa, 2006) is another approach used to solve the constrained optimization problem.

Original: Thus here we solve Eq.(6) using the Interior Policy Differentiation (IPD), which can be regarded as an analogy of the Interior Point Method.
Revised: Thus here we solve Eq.(6) using the Interior Policy Differentiation (IPD), which can be regarded as an analogy of the Interior Point Method.

Original: In the vanilla Interior Point Method, the constrained optimization problem in Eq.(6) is solved by reforming it to an unconstrained form with an additional barrier term − α log g ( θ ) in the objective as max θ ∈ Θ f ( θ ) − α log g ( θ ) , or more precisely in our problem with the formulation with Eq.(6) we have max θ ∈ Θ
Revised: In the vanilla Interior Point Method, the constrained optimization problem in Eq.(6) is solved by reforming it to an unconstrained form with an additional barrier term − α log g ( θ ) in the objective as max θ ∈ Θ f ( θ ) − α log g ( θ ) , or more precisely in our problem with the formulation with Eq.(6) we have max θ ∈ Θ

Original: E τ ∼ θ [ g task − (cid:80) Tt =
Revised: E τ ∼ θ [ g task − (cid:80) Tt =

Original: α log ( r int ,t − r 0 )] , where α > 0 is the barrier factor.
Revised: α log ( r int ,t − r 0 )] , where α > 0 is the barrier factor.

Original: Besides the log barrier term, there are other choices like α 1 g ( θ ) can be used and the objective becomes max θ ∈ Θ f ( θ ) + α 1 g ( θ ) .
Revised: Besides the log barrier term, there are other choices like α 1 g ( θ ) can be used and the objective becomes max θ ∈ Θ f ( θ ) + α 1 g ( θ ) .

Original: As α is small, the barrier term will introduce only minuscule inﬂuence on the objective.
Revised: As α is small, the barrier term will introduce only minuscule inﬂuence on the objective.

Original: On the other hand, when θ get closer to the barrier, the objective will increase rapidly.
Revised: On the other hand, when θ get closer to the barrier, the objective will increase rapidly.

Original: The limits when α → 0 then lead to the solution of Eq.(6).
Revised: The limits when α → 0 then lead to the solution of Eq.(6).

Original: The convergence of such methods are provided in previous works Conn et al.
Revised: The convergence of such methods are provided in previous works Conn et al.

Original: ; Wright (2001).
Revised: ; Wright (2001).

Original: However, directly applying IPM is computationally expensive and numerically unstable.
Revised: However, directly applying IPM is computationally expensive and numerically unstable.

Original: In this work, we propose a simple yet novel heuristic method that resembles the idea of barrier methods: we implicitly apply such barrier terms by providing termination signals in interactions with the environments.
Revised: In this work, we propose a simple yet novel heuristic method that resembles the idea of barrier methods: we implicitly apply such barrier terms by providing termination signals in interactions with the environments.

Original: Our method can be regarded as revising the primal task MDP into a new one in which the behaviors of agents must satisfy novelty constraints.
Revised: Our method can be regarded as revising the primal task MDP into a new one in which the behaviors of agents must satisfy novelty constraints.

Original: Speciﬁcally, in the RL paradigm, the learning procedure of an agent is determined by the experiences collected during interactions with the environment and the sampling strategy used to ﬁlter experiences in the calculation of policy gradients.
Revised: Speciﬁcally, in the RL paradigm, the learning procedure of an agent is determined by the experiences collected during interactions with the environment and the sampling strategy used to ﬁlter experiences in the calculation of policy gradients.

Original: Since the learning process is based on sampled transitions, a more natural way can thus be used to perform the constrained optimization.
Revised: Since the learning process is based on sampled transitions, a more natural way can thus be used to perform the constrained optimization.

Original: We can simply bound the collected transitions in the feasible region by permitting previously trained M policies θ i ∈ Θ ref , i = 1 , 2 , ..., M sending termination signals during the training process of new agents.
Revised: We can simply bound the collected transitions in the feasible region by permitting previously trained M policies θ i ∈ Θ ref , i = 1 , 2 , ..., M sending termination signals during the training process of new agents.

Original: In other words, we implicitly bound the feasible region by terminating any new agent that steps outside it.
Revised: In other words, we implicitly bound the feasible region by terminating any new agent that steps outside it.

Original: Consequently, during the training process, all valid samples we collected are inside the feasible region, which means these samples are less likely to appear in previously trained policies.
Revised: Consequently, during the training process, all valid samples we collected are inside the feasible region, which means these samples are less likely to appear in previously trained policies.

Original: At the end of the training, wethen naturally obtain a new policy that has sufﬁcient novelty.
Revised: At the end of the training, we obtain a new policy that has sufﬁcient novelty.

Original: In this way, we no longer need to consider the trade-off between intrinsic and extrinsic rewards deliberately.
Revised: In this way, we no longer need to consider the trade-off between intrinsic and extrinsic rewards deliberately.

Original: The learning process of IPD is thus more robust and no longer suffers from  objective inconsistency.
Revised: The learning process of IPD is thus more robust and no longer suffers from the objective inconsistency.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: According to Proposition 2, the novelty reward r int in Eq.(6) under our novelty metric can be unbiasedly approximated by r int = min θ j ∈ Θ ref D ρ θ W ( θ ( a | s t ) , θ j ( a j | s t )) .
Revised: According to Proposition 2, the novelty reward r int in Eq.(6) under our novelty metric can be unbiasedly approximated by r int = min θ j ∈ Θ ref D ρ θ W ( θ ( a | s t ) , θ j ( a j | s t )) .

Original: We thus utilize this novelty metric directly throughout our experiments.
Revised: We thus utilize this novelty metric directly throughout our experiments.

Original: We apply different novel policy seeking methods, namely WSR, TNB, CTNB, and IPD, to the backbone RL algorithm PPO (Schulman et al., 2017).
Revised: We apply different novel policy seeking methods, namely WSR, TNB, CTNB, and IPD, to the backbone RL algorithm PPO (Schulman et al., 2017).

Original: The extension to other popular RL algorithms is straightforward.
Revised: The extension to other popular RL algorithms is straightforward.

Original: More implementation details are depicted in Appendix D. Experiments in the work of Henderson et al.
Revised: More implementation details are depicted in Appendix D. Experiments in the work of Henderson et al.

Original: show that one can simply change the random seeds before training to get policies that perform differently Henderson et al.
Revised: show that one can simply change the random seeds before training to get policies that perform differently Henderson et al.

Original: Therefore, we also use PPO with varying random seeds as a baseline method for novel policy seeking.
Revised: Therefore, we also use PPO with varying random seeds as a baseline method for novel policy seeking.

Original: And we use the averaged differences between policies learned by this baseline as the default threshold in CTNB and IPD.
Revised: And we use the averaged differences between policies learned by this baseline as the default threshold in CTNB and IPD.

Original: Algorithm 1 and Algorithm 2 show the pseudo code of IPD and CTNB based on PPO, where the blue lines show the addition to the primal PPO algorithm.
Revised: Algorithm 1 and Algorithm 2 show the pseudo code of IPD and CTNB based on PPO, where the blue lines show the addition to the primal PPO algorithm.

Original: 4. T HE M U J O C O E NVIRONMENT
Revised: 4. T HE M U J O C O E NVIRONMENT

Original: We evaluate our proposed method on the OpenAI Gym based on the MuJoCo engine (Brockman et al., 2016; Todorov et al., 2012).
Revised: We evaluate our proposed method on the OpenAI Gym based on the MuJoCo engine (Brockman et al., 2016; Todorov et al., 2012).

Original: Concretely, we test on three locomotion environments, the Hopper-v
Revised: Concretely, we test on three locomotion environments, the Hopper-v

Revised: Input:

Original: Initialize θ for iteration
Revised: Initialize θ for iteration

Original: ... dofor do  the environment  taking action a t ∼ θ  transitions; if U τ
Revised: do do Step the environment by taking action a t ∼ θ old and collect transitions; if U ( θ old

Original:  − r 0 <  t > t S then Break this episode; end end
Revised: { θ j }| τ ) − r 0 < AND t > t S then Break this episode; end end

Original: Update policy parameters based on sampled data; enddo do
Revised: Update policy parameters based on sampled data; end

Original: ; end
Revised: Initialize do do Step the environment taking action ∼ θ old and collect transitions; end

Original: Compute advantage of reward ˆ A r, 1 , ..., ˆ A r,T
Revised: Compute advantage of reward ˆ A r, 1 , ..., ˆ A r,T

Original: Compute advantage of cost ˆ A c, 1 , ..., ˆ A c,T
Revised: Compute advantage of cost ˆ A c, 1 , ..., ˆ A c,T

Original: Optimize surrogate loss related to reward L CLIP r in
Revised: Optimize surrogate loss related to reward L CLIP r in

Original: PPO w.r.t.
Revised: PPO w.r.t.

Original: θ , with gradient g r = ∇ θ L CLIP r
Revised: θ , with gradient g r = ∇ θ L CLIP r

Original: Optimize surrogate loss related to cost L CLIP c in PPO w.r.t.
Revised: Optimize surrogate loss related to cost L CLIP c in PPO w.r.t.

Original: θ , with gradient g c = −∇ θ L CLIP c if U  then
Revised: θ , with gradient g c = −∇ θ L CLIP c if U θ − r then

Original: Calculate (cid:126)p according to Eq.(8) with g r and g c else
Revised: Calculate (cid:126)p according to Eq.(8) with g r and g c else

Original: Calculate (cid:126)p with g r end
Revised: Calculate (cid:126)p with g r end

Original: Update policy parameters end
Revised: Update policy parameters end

Original: [Figure]
Revised: [Figure]

Original: (11 observations and 3 actions), Walker2d-v3 (11 observations and 6 actions), and HalfCheetah-v (17 observations and 6 actions).
Revised: (11 observations and 3 actions), Walker2d-v3 (11 observations and 6 actions), and HalfCheetah-v (17 observations and 6 actions).

Original: Although relaxing the healthy termination thresholds in Hopper and
Revised: Although relaxing the healthy termination thresholds in Hopper and

Original: Walker may permit more visible behavior diversity, all the environment parameters are set as default values in our experiments to demonstrate the generality of our method.
Revised: Walker may permit more visible behavior diversity, all the environment parameters are set as default values in our experiments to demonstrate the generality of our method.

Original: 4.1. C OMPARISON ON N OVELTY AND P ERFORMANCE
Revised: 4.1. C OMPARISON ON N OVELTY AND P ERFORMANCE

Original: We implement WSR, TNB, CTNB, and IPD using the same hyper-parameter settings per environment.
Revised: We implement WSR, TNB, CTNB, and IPD using the same hyper-parameter settings per environment.

Original: And we also apply CPO Achiam et al.
Revised: And we also apply CPO Achiam et al.

Original: (2017) as a baseline as a solution of CMDP.
Revised: (2017) as a baseline as a solution of CMDP.

Original: For each method, we ﬁrst train 10 policies using PPO with different random seeds.
Revised: For each method, we ﬁrst train 10 policies using PPO with different random seeds.

Original: Those PPO policies are used as the primal reference policies, and then we train 10 novel policies that try to be different from previous reference policies.
Revised: Those PPO policies are used as the primal reference policies, and then we train 10 novel policies that try to be different from previous reference policies.

Original: Concretely, in each method, the 1 st novel policy is trained to be different from the previous 10 PPO policies, and the 2 nd should be different from the previous 11 policies, and so on.
Revised: Concretely, in each method, the 1 st novel policy is trained to be different from the previous 10 PPO policies, and the 2 nd should be different from the previous 11 policies, and so on.

Original: More implementation details are depicted in Appendix D.
Revised: More implementation details are depicted in Appendix D.

Original: [Table]
Revised: [Table]

Original: Fig.
Revised: Fig.

Original: 2 shows our experimental results in terms of novelty (the x-axis) and the performance (the y-axis).
Revised: 2 shows our experimental results in terms of novelty (the x-axis) and the performance (the y-axis).

Original: Policies close to the upper right corner are the more novel ones with higher performance.
Revised: Policies close to the upper right corner are the more novel ones with higher performance.

Original: In all environments, the performance of CTNB, IPD and CPO outperforms WSR and TNB, showing the advantage of constrained optimization approaches in novel policy seeking.
Revised: In all environments, the performance of CTNB, IPD and CPO outperforms WSR and TNB, showing the advantage of constrained optimization approaches in novel policy seeking.

Original: Speciﬁcally, the results of CTNB are all better than their multi-objective counterparts, i
Revised: Speciﬁcally, the results of CTNB are all better than their multi-objective counterparts, i

Original: [Table]
Revised: [Table]

Original: e., the results from TNB, showing the superiority of seeking novel policies with constrained optimization.
Revised: e., the results from TNB, showing the superiority of seeking novel policies with constrained optimization.

Original: Moreover, the IPD method provides more novelty than CTNB and CPO, while the primal task performances are still guaranteed.
Revised: Moreover, the IPD method provides more novelty than CTNB and CPO, while the primal task performances are still guaranteed.

Original: Comparisons of the task-related rewards are carried out in Table 1, where among all the four methods,
Revised: Comparisons of the task-related rewards are carried out in Table 1, where among all the four methods,

Original: IPD provides sufﬁcient diversity with minimum loss of performance.
Revised: IPD provides sufﬁcient diversity with minimum loss of performance.

Original: Instead of performance decay, we ﬁnd IPD is able to ﬁnd better policies in the environment of Hopper and HalfCheetah.
Revised: Instead of performance decay, we ﬁnd IPD is able to ﬁnd better policies in the environment of Hopper and HalfCheetah.

Original: Moreover, in the Hopper environment, while the agents trained with PPO tend to fall into the same local minimum.
Revised: Moreover, in the Hopper environment, while the agents trained with PPO tend to fall into the same local minimum.

Original: (e.g., they all jump as far as possible and then terminate this episode.
Revised: (e.g., they all jump as far as possible and then terminate this episode.

Original: On the contrary, PPO with IPD keeps new agents away from falling into the same local minimum, because once an agent has reached some local minimum, agents learned later will try to avoid this region due to the novelty constraints.
Revised: On the contrary, PPO with IPD keeps new agents away from falling into the same local minimum, because once an agent has reached some local minimum, agents learned later will try to avoid this region due to the novelty constraints.

Original: Such property shows that IPD can enhance the traditional RL schemes to tackle the local exploration challenge (Tessler et al., 2019; Ciosek et al., 2019).
Revised: Such property shows that IPD can enhance the traditional RL schemes to tackle the local exploration challenge (Tessler et al., 2019; Ciosek et al., 2019).

Original: A similar feature brings about reward growth in the environment of HalfCheetah.
Revised: A similar feature brings about reward growth in the environment of HalfCheetah.

Original: Detailed analysis and discussions are developed in Appendix E.
Revised: Detailed analysis and discussions are developed in Appendix E.

Original: 4.1. S UCCESS R ATE OF E ACH M ETHOD
Revised: 4.1. S UCCESS R ATE OF E ACH M ETHOD

Original: In addition to averaged reward, we also use the success rate as another metric to compare the performance of different approaches.
Revised: In addition to averaged reward, we also use the success rate as another metric to compare the performance of different approaches.

Original: Roughly speaking, the success rate evaluates the stability of each method in terms of generating a policy that performs as good as the policies PPO generates.
Revised: Roughly speaking, the success rate evaluates the stability of each method in terms of generating a policy that performs as good as the policies PPO generates.

Original: In this work, we regard a policy successful when its performance achieves at least as good as the median performance of policies trained with PPO.
Revised: In this work, we regard a policy successful when its performance achieves at least as good as the median performance of policies trained with PPO.

Original: To be speciﬁc, we use the median of the ﬁnal performance of PPO as the baseline, and if a novel policy, which aims at performing differently to solve the same task, surpasses the baseline during its training process, it will be regarded as a successful policy.
Revised: To be speciﬁc, we use the median of the ﬁnal performance of PPO as the baseline, and if a novel policy, which aims at performing differently to solve the same task, surpasses the baseline during its training process, it will be regarded as a successful policy.

Original: By deﬁnition, the success rate of PPO is 0 .
Revised: By deﬁnition, the success rate of PPO is 0 .

Original: 5 as a baseline for every environment.
Revised: 5 as a baseline for every environment.

Original: Table 1 shows the success rate of all the methods.
Revised: Table 1 shows the success rate of all the methods.

Original: The results show that all constrained novelty seeking methods (CTNB,
Revised: The results show that all constrained novelty seeking methods (CTNB,

Original: IPD, CPO) can surpass the average baseline during training, while the multi-objective optimization approaches normally can not.
Revised: IPD, CPO) can surpass the average baseline during training, while the multi-objective optimization approaches normally can not.

Original: Thus the performance of constrained novelty seeking methods can always be insured.
Revised: Thus the performance of constrained novelty seeking methods can always be insured.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we rethink the novel policy seeking problem under the perspective of constrained optimization.
Revised: In this work, we rethink the novel policy seeking problem under the perspective of constrained optimization.

Original: We  introduce a new metric to measure the distances between policies, and then we introduce the deﬁnition of policy novelty.
Revised: We ﬁrst introduce a new metric to measure the distances between policies, and then give a deﬁnition of policy novelty.

Original: We propose a new perspective by connecting the domain of constrained optimization to the domain of RL, and come up with two methods forseeking novel policies, namely the Constrained Task Novel Bisector (CTNB), and the Interior Policy Differentiation (IPD).
Revised: Based on the formulation of constrained optimization, we come up with two practical algorithms for novel policy seeking, namely the Constrained Task Novel Bisector (CTNB), and the Interior Policy Differentiation (IPD).

Original: Our experimental results demonstrate that the proposed method can effectively learn various well-behaved yet diverse policies, outperforming previous methods following the multi-objective formulation.
Revised: Our experimental results demonstrate that the proposed method can effectively learn various well-performing yet diverse policies, outperforming previous methods which are under the multi-objective formulation.

Original: A PPENDIX A M ETRIC S PACE
Revised: A PPENDIX A M ETRIC S PACE

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: A metric space is an ordered pair ( M, d ) where M is a set and d is a metric on M , i.e., a function d : M × M → R such that for any x, y, z ∈ M , the following holds:
Revised: A metric space is an ordered pair ( M, d ) where M is a set and d is a metric on M , i.e., a function d : M × M → R such that for any x, y, z ∈ M , the following holds:

Original: d ( x, y ) ≥ 0 , d ( x, y )
Revised: d ( x, y ) ≥ 0 , d ( x, y )

Original: = 

Revised: =

Original: ⇔ x = y , 2.
Revised: ⇔ x = y , 2.

Original: d ( x, y ) =
Revised: d ( x, y ) =

Original: d ( y, x ) , 3.
Revised: d ( y, x ) , 3.

Original: d ( x, z ) ≤ d ( x, y ) + d ( y, z ) .
Revised: d ( x, z ) ≤ d ( x, y ) + d ( y, z ) .

Original: B P ROOF OF P ROPOSITION
Revised: B P ROOF OF P ROPOSITION

Original: The ﬁrst two properties are obviously guaranteed by D ρW .
Revised: The ﬁrst two properties are obviously guaranteed by D ρW .

Original: As for the triangle inequality,
Revised: As for the triangle inequality,

Original: E s ∼ ρ ( s ) [ D W ( θ i ( s ) , θ k
Revised: E s ∼ ρ ( s ) [ D W ( θ i ( s ) , θ k

Original: ( s )]
Revised: ( s )]

Original: =
Revised: =

Original: E s ∼ ρ ( s ) [ |A| (cid:88) l
Revised: E s ∼ ρ ( s ) [ |A| (cid:88) l

Original: = | θ i ( s ) − θ k
Revised: = | θ i ( s ) − θ k

Original: ( s ) | ] =
Revised: ( s ) | ] =

Original: E s ∼ ρ ( s ) [ |A| (cid:88)
Revised: E s ∼ ρ ( s ) [ |A| (cid:88)

Original: l
Revised: l

Original: = | θ i ( s ) − θ j
Revised: = | θ i ( s ) − θ j

Original: ( s ) + θ j ( s ) − θ k
Revised: ( s ) + θ j ( s ) − θ k

Original: ( s ) | ] ≤ E s ∼ ρ ( s ) [ ( |A| (cid:88) l
Revised: ( s ) | ] ≤ E s ∼ ρ ( s ) [ ( |A| (cid:88) l

Original: = | θ i ( s ) − θ j ( s ) | + | θ j ( s ) − θ k
Revised: = | θ i ( s ) − θ j ( s ) | + | θ j ( s ) − θ k

Original: ( s ) | )]
Revised: ( s ) | )]

Original: =
Revised: =

Original: E s ∼ ρ ( s ) [ |A| (cid:88) l
Revised: E s ∼ ρ ( s ) [ |A| (cid:88) l

Original: = | θ i ( s ) − θ j ( s ) | ] + E s ∼ ρ ( s ) [ |A| (cid:88) l = | θ j ( s ) − θ k
Revised: = | θ i ( s ) − θ j ( s ) | ] + E s ∼ ρ ( s ) [ |A| (cid:88) l = | θ j ( s ) − θ k

Original: ( s ) | ] =
Revised: ( s ) | ] =

Original: E s ∼ ρ ( s ) [ D W ( θ i ( s ) , θ j ( s )]
Revised: E s ∼ ρ ( s ) [ D W ( θ i ( s ) , θ j ( s )]

Original: + E s ∼ ρ ( s )
Revised: + E s ∼ ρ ( s )

Original: [ D W ( θ j ( s ) , θ k
Revised: [ D W ( θ j ( s ) , θ k

Original: ( s )]
Revised: ( s )]

Original: C P ROOF OF P ROPOSITION
Revised: C P ROOF OF P ROPOSITION

Original: ρ θ ( s ) =
Revised: ρ θ ( s ) =

Original: P ( s 0 = s | θ )
Revised: P ( s 0 = s | θ )

Original: + P ( s 1 = s | θ ) + ...
Revised: + P ( s 1 = s | θ ) + ...

Original: + P ( s T = s | θ )
Revised: + P ( s T = s | θ )

Original: L.L.N. = lim N →∞ (cid:80) Ni
Revised: L.L.N. = lim N →∞ (cid:80) Ni

Original: =

Revised: =

Original: I ( s 

Revised: I ( s

Original: = s | τ i )
Revised: = s | τ i )

Original: N + (cid:80) Ni
Revised: N + (cid:80) Ni

Original: =

Revised: =

Original: I ( s 1 = s | τ i )
Revised: I ( s 1 = s | τ i )

Original: N + ... + (cid:80) Ni =

Revised: N + ... + (cid:80) Ni =

Original: I ( s T = s |
Revised: I ( s T = s |

Original: τ i )
Revised: τ i )

Original: N = lim N →∞ (cid:80) T j =0 (cid:80)
Revised: N = lim N →∞ (cid:80) T j =0 (cid:80)

Original: Ni =

Revised: Ni =

Original: I ( s j = s | τ i )
Revised: I ( s j = s | τ i )

Original: N ρ θ ( s ) =
Revised: N ρ θ ( s ) =

Original: N (cid:88) i =
Revised: N (cid:88) i =

Original: T (cid:88)
Revised: T (cid:88)

Original: j =
Revised: j =

Original: I ( s j = s | τ i )
Revised: I ( s j = s | τ i )

Original: N
Revised: N

Original: E [
Revised: E [

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: D I MPLEMENTATION D ETAILS Calculation of D W
Revised: D I MPLEMENTATION D ETAILS Calculation of D W

Original: We use deterministic part of policies in the calculation of D W , i.e., we remove the Gaussian noise on the action space in PPO and use D W ( a 1 , a 2 ) = | a 1 − a 2 | .
Revised: We use deterministic part of policies in the calculation of D W , i.e., we remove the Gaussian noise on the action space in PPO and use D W ( a 1 , a 2 ) = | a 1 − a 2 | .

Original: Network Structure We use MLP with 2 hidden layers as our actor models in PPO.
Revised: Network Structure We use MLP with 2 hidden layers as our actor models in PPO.

Original: The ﬁrst hidden layer is ﬁxed to have 32 units.
Revised: The ﬁrst hidden layer is ﬁxed to have 32 units.

Original: We choose to use 10 , 64 and 256 hidden units for the three tasks respectively in all of the main experiments, after taking the success rate, performance and computation expense (i.e. the preference to use less unit when the other two factors are similar) into consideration.
Revised: We choose to use 10 , 64 and 256 hidden units for the three tasks respectively in all of the main experiments, after taking the success rate, performance and computation expense (i.e. the preference to use less unit when the other two factors are similar) into consideration.

Original: [Figure]
Revised: [Figure]

Original: Training Timesteps We ﬁx the training timesteps in our experiments.
Revised: Training Timesteps We ﬁx the training timesteps in our experiments.

Original: The timesteps are ﬁxed to be 1 M in Hopper-v3, 1 . 6 M for Walker2d-v3 and 3 M for HalfCheetah-v3.
Revised: The timesteps are ﬁxed to be 1 M in Hopper-v3, 1 . 6 M for Walker2d-v3 and 3 M for HalfCheetah-v3.

Original: E D ISCUSSION E. N OVEL P OLICY S EEKING WITHOUT P ERFORMANCE D ECAY
Revised: E D ISCUSSION E. N OVEL P OLICY S EEKING WITHOUT P ERFORMANCE D ECAY

Original: Multi-objective formulation of novel policy seeking has the risk of sacriﬁcing the primal performance as the overall objective needs to consider both novelty and primal task rewards.
Revised: Multi-objective formulation of novel policy seeking has the risk of sacriﬁcing the primal performance as the overall objective needs to consider both novelty and primal task rewards.

Original: On the contrary, under the perspective of constrained optimization, there will be no more trade-off between novelty and ﬁnal reward as the only objective is the task reward.
Revised: On the contrary, under the perspective of constrained optimization, there will be no more trade-off between novelty and ﬁnal reward as the only objective is the task reward.

Original: Given a certain novelty threshold, the algorithms tend to ﬁnd the optimal solution in terms of task reward under constraints, thus the learning process becomes more controllable and reliable, i.e., one can utilize the novelty threshold to control the degree of novelty.
Revised: Given a certain novelty threshold, the algorithms tend to ﬁnd the optimal solution in terms of task reward under constraints, thus the learning process becomes more controllable and reliable, i.e., one can utilize the novelty threshold to control the degree of novelty.

Original: Intuitively, the proper magnitude of the novelty threshold will lead to more exploration among a population of policies, thus the performance of latter found policies may be better than or at least as good as those trained without novelty seeking.
Revised: Intuitively, the proper magnitude of the novelty threshold will lead to more exploration among a population of policies, thus the performance of latter found policies may be better than or at least as good as those trained without novelty seeking.

Original: However, when a larger magnitude of novelty threshold is applied, the performance of found novel policies will decrease because ﬁnding a feasible solution will get harder under more strict constraints.
Revised: However, when a larger magnitude of novelty threshold is applied, the performance of found novel policies will decrease because ﬁnding a feasible solution will get harder under more strict constraints.

Original: Fig.
Revised: Fig.

Original: 3 shows our ablation study on adjusting the thresholds, which veriﬁes our intuition.
Revised: 3 shows our ablation study on adjusting the thresholds, which veriﬁes our intuition.

Original: E. C URRICULUM L EARNING IN H ALF C HEETAH
Revised: E. C URRICULUM L EARNING IN H ALF C HEETAH

Original: Moreover, we observe a kind of auto-curriculum learning behavior in the learning of HalfCheetah, which may also help to understand the performance improvement in this environment.
Revised: Moreover, we observe a kind of auto-curriculum learning behavior in the learning of HalfCheetah, which may also help to understand the performance improvement in this environment.

Original: The environment of HalfCheetah is different from the other two in that there is no explicit early termination signal in its default setting (i.e., there is no explicit threshold for the states, exceeds which would trigger a termination).
Revised: The environment of HalfCheetah is different from the other two in that there is no explicit early termination signal in its default setting (i.e., there is no explicit threshold for the states, exceeds which would trigger a termination).

Original: At the beginning of the learning, a PPO agent always acts randomly and keep twitching without moving, resulting in massive repeated and trivial samples and large control costs.
Revised: At the beginning of the learning, a PPO agent always acts randomly and keep twitching without moving, resulting in massive repeated and trivial samples and large control costs.

Original: Contrarily, in the learning of IPD, the agent can receive termination signals since repeated behaviors break the novelty constraint, preventing it from wasting too much effort acting randomly.
Revised: Contrarily, in the learning of IPD, the agent can receive termination signals since repeated behaviors break the novelty constraint, preventing it from wasting too much effort acting randomly.

Original: Moreover, such termination signals also encourage the agent to imitate previous policies to get out of random explorations at the starting stage, avoiding heavy control costs while receiving less negative rewards.
Revised: Moreover, such termination signals also encourage the agent to imitate previous policies to get out of random explorations at the starting stage, avoiding heavy control costs while receiving less negative rewards.

Original: After that, the agent begins to learn to behave differently to pursue higher positive rewards.
Revised: After that, the agent begins to learn to behave differently to pursue higher positive rewards.

Original: From this point of view, the learning process can be interpreted as a kind of implicit curriculum, which saves lots of interactions with the environment, improves the sample efﬁciency and therefore achieves better performance in the given learning timesteps.
Revised: From this point of view, the learning process can be interpreted as a kind of implicit curriculum, which saves lots of interactions with the environment, improves the sample efﬁciency and therefore achieves better performance in the given learning timesteps.

Original: + + + +
Revised: + + + +

Original: WSR, Reward: 47.6 ± 12.
Revised: WSR, Reward: 47.6 ± 12.

Original: Traj1Traj2Traj3Traj4Traj
Revised: Traj1Traj2Traj3Traj4Traj

Original: TNB, Reward: 120.6 ± 32.
Revised: TNB, Reward: 120.6 ± 32.

Original: Traj1Traj2Traj3Traj4Traj
Revised: Traj1Traj2Traj3Traj4Traj

Original: CTNB, Reward: 147.3 ± 32.
Revised: CTNB, Reward: 147.3 ± 32.

Original: Traj1Traj2Traj3Traj4Traj
Revised: Traj1Traj2Traj3Traj4Traj

Original: IPD, Reward: 144.0 ± 21.
Revised: IPD, Reward: 144.0 ± 21.

Original: Traj1Traj2Traj3Traj4Traj
Revised: Traj1Traj2Traj3Traj4Traj

Original: F V ISUALIZE D IVERSITYIN T OY M ODEL
Revised: F V ISUALIZE D IVERSITY

Original: [Figure]
Revised: [Figure]

Original: We ﬁrst utilize a basic 2-D environment named Four Reward Maze as a diagnostic environment where we can visualize learned policies directly.
Revised: We ﬁrst utilize a basic 2-D environment named Four Reward Maze as a diagnostic environment where we can visualize learned policies directly.

Original: In this environment, four positive rewards of different values (e.g., +5 , +5 , +10 , +1 for top, down, left and right respectively) are assigned to four middle points with radius 1 on each edge in a 2-D N × N square map.
Revised: In this environment, four positive rewards of different values (e.g., +5 , +5 , +10 , +1 for top, down, left and right respectively) are assigned to four middle points with radius 1 on each edge in a 2-D N × N square map.

Original: We use N = 16 in our experiments.
Revised: We use N = 16 in our experiments.

Original: The observation of a policy is the current position and the agent will receive a negative reward of − 0 .
Revised: The observation of a policy is the current position and the agent will receive a negative reward of − 0 .

Original: 01 at each timestep except stepping into the reward regions.
Revised: 01 at each timestep except stepping into the reward regions.

Original: Each episode starts from a randomly initialized position and the action space is limited to [ − 1 , 1] .
Revised: Each episode starts from a randomly initialized position and the action space is limited to [ − 1 , 1] .

Original: The performance of each agent is evaluated by the averaged performances over 100 trials.
Revised: The performance of each agent is evaluated by the averaged performances over 100 trials.

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Original: 4, where the behaviors of the PPO agents are quite similar, suggesting the diversity provided by random seeds is limited.
Revised: 4, where the behaviors of the PPO agents are quite similar, suggesting the diversity provided by random seeds is limited.

Original: WSR and TNB solve the novelty-seeking problem from the multi-objective optimization formulation, they thus suffer from the unbalance between performance and novelty.
Revised: WSR and TNB solve the novelty-seeking problem from the multi-objective optimization formulation, they thus suffer from the unbalance between performance and novelty.

Original: While WSR and TNB both provide sufﬁcient novelty, performances of agents learned by WSR decay signiﬁcantly, so did TNB due to an encumbered learning process, as we analyzed in Sec.3.3.
Revised: While WSR and TNB both provide sufﬁcient novelty, performances of agents learned by WSR decay signiﬁcantly, so did TNB due to an encumbered learning process, as we analyzed in Sec.3.3.

Original: Both CTNB and IPD, solving the task with novelty-seeking from the constrained optimization formulation, provide evident behavior diversity and perform recognizably better than TNB and WSR. 
Revised: Both CTNB and IPD, solving the task with novelty-seeking from the constrained optimization formulation, provide evident behavior diversity and perform recognizably better than TNB and WSR. F. M UJOCO L OCOMOTION

Revised: In this section, we provide some qualitative results of IPD on the Mujoco locomotion tasks.

Revised: In all of our experiments we use the vanilla Mujoco locomotion benchmarks, with the default settings on deﬁning healthy states.

Revised: Although otherwise the visualization of learned policies might become more diverse (e.g., a Hopper agent may learn to stand-up after falling down while another agent may learn to move forward on the ground if we set the z -axis healthy threshold as 0 ).

Revised: With the method of IPD, the Hopper policies (Figure 5) learns to jump further and avoids falling down rather instead of just jumping and falling down (Figure 6).

Revised: In the Walker2d environment, the color of purple indicates the left leg is visible.

Revised: It can be seen that the IPD policies (Figure 7) learn to use both left and right legs in walking, while the PPO policies usually learn jumping.

Revised: (Figure 8).

Revised: In

Revised: HalfCheetah, the IPD policies (Figure 9) perform much better than the PPO policies (Figure 10).

Revised: The

Revised: IPD policies leran to run with head-downward (Figure 9 line 1), head-upward (Figure 9 line 3), and forward (Figure 9 line 5) while the PPO policies are always head-downward.

Revised: In Hopper and HalfCheetah, IPD is able to improve the primal task performance by avoiding always getting trapped in some certain sub-optimal behaviors.

Revised: Most agents learn a policy that can be described as Jump as far as possible and fall down , leading to relative poor performance.