Original: A TTRACTION -R EPULSION A CTOR -C RITIC FOR C ONTINUOUS C ONTROL R EINFORCEMENT L EARNING
Revised: A TTRACTION -R EPULSION A CTOR -C RITIC FOR C ONTINUOUS C ONTROL R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In reinforcement learning, continuous control tasks are often useful for understanding how agents perform in environments with deceptive rewards where the agent can easily become trapped into suboptimal solutions.
Revised: In reinforcement learning, robotic control tasks are often useful for understanding how agents perform in environments with deceptive rewards where the agent can easily become trapped into suboptimal solutions.

Original: One way to avoid these local optima is to use a population of agents to ensure coverage of the policy space (a form of exploration ), yet learning a population with the “best” coverage is still an open problem.
Revised: One way to avoid these local optima is to use a population of agents to ensure coverage of the policy space (a form of exploration ), yet learning a population with the “best” coverage is still an open problem.

Original: In this work, we present a novel approach to population-based RL in continuous control that leverages properties of normalizing ﬂows to perform attractive and repulsive operations between current members of the population and previously observed policies.
Revised: In this work, we present a novel approach to population-based RL in continuous control that leverages properties of normalizing ﬂows to perform attractive and repulsive operations between current members of the population and previously observed policies.

Original: Empirical results on the MuJoCo suite demonstrate a high performance gain for our algorithm compared to prior work, including
Revised: Empirical results on the MuJoCo suite demonstrate a high performance gain for our algorithm compared to prior work, including

Original: Soft-Actor Critic (SAC).
Revised: Soft-Actor Critic (SAC).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many important reinforcement learning (RL) tasks, such as those in robotics and self-driving cars, are challenging due to large action and state spaces (Lee et al., 2018).
Revised: Many important reinforcement learning (RL) tasks, such as those in robotics and self-driving cars, are challenging due to large action and state spaces (Lee et al., 2018).

Original: In particular, environments with largenon-convex continuous action spaces are prone to deceptive rewards, i.e. fall into local optima in learning (Conti et al., 2018).
Revised: In particular, environments with large continuous action spaces are prone to deceptive rewards, i.e. fall into local optima in learning (Conti et al., 2018).

Original: Applying traditional policy optimization algorithms to these domains often leads to locally optimal, yet globally sub-optimal policies.
Revised: Applying traditional policy optimization algorithms to these domains often leads to locally optimal, yet globally sub-optimal policies.

Original: The agent should then explore the reward landscape more thoroughly in order to avoid falling into these local optima.
Revised: The agent should then explore the reward landscape more thoroughly in order to avoid falling into these local optima.

Original: Not all RL domains that require exploration are suitable for understanding how to train agents that are robust to deceptive rewards.
Revised: Not all RL domains that require exploration are suitable for understanding how to train agents that are robust to deceptive rewards.

Original: For example, Montezuma’s Revenge, a game in the Atari Learning
Revised: For example, Montezuma’s Revenge, a game in the Atari Learning

Original: Environment (Bellemare et al., 2013), has sparse rewards ; algorithms that perform the best on this task encourage exploration by providing a denser intrinsic reward to the agent to encourage exploration (Tang et al., 2017).
Revised: Environment (Bellemare et al., 2013), has sparse rewards ; algorithms that perform the best on this task encourage exploration by providing a denser intrinsic reward to the agent to encourage exploration (Tang et al., 2017).

Original: On the other hand, many continuous control problems, such as those found in MuJoCo (Todorov et al., 2012), provide the agent with a dense reward signal, yet their high-dimensional action spaces induce a multimodal, often deceptive , reward landscape.
Revised: On the other hand, many robotic control problems, such as those found in MuJoCo (Todorov et al., 2012), provide the agent with a dense reward signal, yet their high-dimensional action spaces induce a multimodal, often deceptive , reward landscape.

Original: For example, in the biped environments, coordinating both arms and legs is crucial for performing well on even simple tasks such as forward motion.
Revised: For example, in the biped environments, coordinating both arms and legs is crucial for performing well on even simple tasks such as forward motion.

Original: However, simply learning to maximize the reward can be detrimental across training: agents will tend to run and fall further away from the start point rather than discovering stable and efﬁcient walking motion.
Revised: However, simply learning to maximize the reward can be detrimental across training: agents will tend to run and fall further away from the start point rather than discovering stable and efﬁcient walking motion.

Original: In this setting, exploration serves to provide a more reliable learning signal for the agent by covering more different types of actions during learning.
Revised: In this setting, exploration serves to provide a more reliable learning signal for the agent by covering more different types of actions during learning.

Original: One way to maximize action space coverage is the maximum entropy RL framework (Ziebart, 2010), which prevents variance collapse by adding a policy entropy auxiliary objective.
Revised: One way to maximize action space coverage is the maximum entropy RL framework (Ziebart, 2010), which prevents variance collapse by adding a policy entropy auxiliary objective.

Original: One such prominent algorithm, Soft Actor-Critic (SAC,Haarnoja et al.
Revised: One such prominent algorithm, Soft Actor-Critic (SAC,Haarnoja et al.

Original: (2018)), has been shown to excel in large continuous action spaces.
Revised: (2018)), has been shown to excel in large continuous action spaces.

Original: To further improve on exploration properties of SAC, one can maintain a population of agents that cover non-identical sections of the policy space.
Revised: To further improve on exploration properties of SAC, one can maintain a population of agents that cover non-identical sections of the policy space.

Original: To prevent premature convergence, a diversity-preserving mechanism is typically put in place; balancing the objective and the diversity term becomes key to converging to a global optimum (Hong et al., 2018).
Revised: To prevent premature convergence, a diversity-preserving mechanism is typically put in place; balancing the objective and the diversity term becomes key to converging to a global optimum (Hong et al., 2018).

Original: This paper studies a particular family of population-based exploration methods, which conduct coordinated local search in the policy space.
Revised: This paper studies a particular family of population-based exploration methods, which conduct coordinated local search in the policy space.

Original: Prior work on population-based strategies improves performance on continuous control domains through stochastic perturbation on a single actor’s parameter (Pourchot & Sigaud, 2019) or a set of actor’s parameters (Conti et al., 2018; Khadka & Tumer, 2018; Liu et al., 2017).
Revised: Prior work on population-based strategies improves performance on robotic control domains through stochastic perturbation on a single actor’s parameter (Pourchot & Sigaud, 2019) or a set of actor’s parameters (Conti et al., 2018; Khadka & Tumer, 2018; Liu et al., 2017).

Original: We hypothesize that exploring directly in the policy space will be more effective than perturbing the parameters of the policy, as the latter does not guarantee diversity (i.e., different neural network parameterizations can approximately represent the same function).
Revised: We hypothesize that exploring directly in the policy space will be more effective than perturbing the parameters of the policy, as the latter does not guarantee diversity (i.e., different neural network parameterizations can approximately represent the same function).

Original: Given a population of RL agents, we enforce local exploration using an Attraction-Repulsion (AR) mechanism.
Revised: Given a population of RL agents, we enforce local exploration using an Attraction-Repulsion (AR) mechanism.

Original: The later consists in adding an auxiliary loss to encourage pairwise attraction or repulsion between members of a population, as measured by a divergence term.
Revised: The later consists in adding an auxiliary loss to encourage pairwise attraction or repulsion between members of a population, as measured by a divergence term.

Original: We make use of the KullbackLeibler (KL) divergence because of its desirable statistical properties and its easiness of computation.
Revised: We make use of the KullbackLeibler (KL) divergence because of its desirable statistical properties and its easiness of computation.

Original: However, naively maximizing the KL term between two Gaussian policies can be detrimental (e.g. drives both means apart).
Revised: However, naively maximizing the KL term between two Gaussian policies can be detrimental (e.g. drives both means apart).

Original: Because of this, we parametrize the policy with a general family of distributions called Normalizing Flows (NFs, Rezende & Mohamed, 2015); this modiﬁcation allows to improve upon AR+Gaussian (see Appendix Figure 6).
Revised: Because of this, we parametrize the policy with a general family of distributions called Normalizing Flows (NFs, Rezende & Mohamed, 2015); this modiﬁcation allows to improve upon AR+Gaussian (see Appendix Figure 6).

Original: NFs are shown to improve the expressivity of the policies using invertible mappings while maintaining entropy guarantees (Mazoure et al., 2019;
Revised: NFs are shown to improve the expressivity of the policies using invertible mappings while maintaining entropy guarantees (Mazoure et al., 2019;

Original: Tang & Agrawal, 2018).
Revised: Tang & Agrawal, 2018).

Original: Nonlinear density estimators have also been previously used for deep RL problems in contexts of distributional RL (Doan et al., 2018) and reward shaping (Tang et al., 2017).
Revised: Nonlinear density estimators have also been previously used for deep RL problems in contexts of distributional RL (Doan et al., 2018) and reward shaping (Tang et al., 2017).

Original: The AR objective blends particularly well with SAC, since computing the KL requires stochastic policies with tractable densities for each agent.
Revised: The AR objective blends particularly well with SAC, since computing the KL requires stochastic policies with tractable densities for each agent.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We ﬁrst formalize the RL setting in a Markov decision process (MDP).
Revised: We ﬁrst formalize the RL setting in a Markov decision process (MDP).

Original: A discrete-time, ﬁnite-horizon,
Revised: A discrete-time, ﬁnite-horizon,

Original: MDP (Bellman, 1957; Puterman, 2014) is described by a state space S , an action space A , a transition function P : S ×
Revised: MDP (Bellman, 1957; Puterman, 2014) is described by a state space S , an action space A , a transition function P : S ×

Original: A × S (cid:55)→ R + , and a reward function r : S × A (cid:55)→ R .
Revised: A × S (cid:55)→ R + , and a reward function r : S × A (cid:55)→ R .

Original: On each round t , an agent interacting with this MDP observes the current state s t ∈ S , selects an action a t ∈ A , and observes a reward r ( s t , a t ) ∈ R upon transitioning to a new state s t +1 ∼ P ( s t , a t ) .
Revised: On each round t , an agent interacting with this MDP observes the current state s t ∈ S , selects an action a t ∈ A , and observes a reward r ( s t , a t ) ∈ R upon transitioning to a new state s t +1 ∼ P ( s t , a t ) .

Original: Let γ ∈ [0 , 1] be a discount factor.
Revised: Let γ ∈ [0 , 1] be a discount factor.

Original: The goal of an agent evolving in a discounted MDP is to learn a policy π :
Revised: The goal of an agent evolving in a discounted MDP is to learn a policy π :

Original: S × A (cid:55)→ [0 , 1] such as taking action a t ∼ π ( ·| s t ) would maximize the expected sum of discounted returns,
Revised: S × A (cid:55)→ [0 , 1] such as taking action a t ∼ π ( ·| s t ) would maximize the expected sum of discounted returns,

Original: [Equation]
Revised: [Equation]

Original: In the following, we use ρ π to denote the trajectory distribution induced by following policy π .
Revised: In the following, we use ρ π to denote the trajectory distribution induced by following policy π .

Original: If S or A are vector spaces, action and space vectors are respectively denoted by a and s .
Revised: If S or A are vector spaces, action and space vectors are respectively denoted by a and s .

Original: 2. D ISCOVERING NEW SOLUTIONS THROUGH POPULATION - BASED A TTRACTION -R EPULSION
Revised: 2. D ISCOVERING NEW SOLUTIONS THROUGH POPULATION - BASED A TTRACTION -R EPULSION

Original: Consider evolving a population of M agents, also called individuals , { π θ m } Mm =1 , each agent corresponding to a policy with its own parameters.
Revised: Consider evolving a population of M agents, also called individuals , { π θ m } Mm =1 , each agent corresponding to a policy with its own parameters.

Original: In order to discover new solutions, we aim to generate agents that can mimic some target policy while following a path different from those of other policies.
Revised: In order to discover new solutions, we aim to generate agents that can mimic some target policy while following a path different from those of other policies.

Original: Let G denote an archive of policies encountered in previous generations of the population.
Revised: Let G denote an archive of policies encountered in previous generations of the population.

Original: A natural way of enforcing π to be different from or similar to the policies contained in G is by augmenting the loss of the agent with an Attraction-Repulsion (AR) term:
Revised: A natural way of enforcing π to be different from or similar to the policies contained in G is by augmenting the loss of the agent with an Attraction-Repulsion (AR) term:

Original: [Equation]
Revised: [Equation]

Original: (1) where π (cid:48) is an archived policy and β π (cid:48) is a coefﬁcient weighting the relative importance of the
Revised: (1) where π (cid:48) is an archived policy and β π (cid:48) is a coefﬁcient weighting the relative importance of the

Original: Kullback-Leibler (KL) divergence between π and π (cid:48) , which we will choose to be a function of the average reward (see Sec.
Revised: Kullback-Leibler (KL) divergence between π and π (cid:48) , which we will choose to be a function of the average reward (see Sec.

Original: 3.2 below).
Revised: 3.2 below).

Original: Intuitively, Eq. 1 adds to the agent objective a weighted average distance between the current and the archived policies.
Revised: Intuitively, Eq. 1 adds to the agent objective a weighted average distance between the current and the archived policies.

Original: For β π (cid:48) ≥ 0 , the agent tends to move away from the archived policy’s behavior (i.e. repulsion , see Figure 1) a).
Revised: For β π (cid:48) ≥ 0 , the agent tends to move away from the archived policy’s behavior (i.e. repulsion , see Figure 1) a).

Original: On the other hand, β π (cid:48) <encourages the agent π to imitate π (cid:48) (i.e. attraction ).
Revised: On the other hand, β π (cid:48) <encourages the agent π to imitate π (cid:48) (i.e. attraction ).

Original: Requirements for AR In order for agents within a population to be trained using the proposed
Revised: Requirements for AR In order for agents within a population to be trained using the proposed

Original: AR-based loss (Eq. 1), we have the following requirements:
Revised: AR-based loss (Eq. 1), we have the following requirements:

Original: Their policies should be stochastic, so that the KL-divergence between two policies is well-deﬁned.
Revised: Their policies should be stochastic, so that the KL-divergence between two policies is well-deﬁned.

Original: Their policies should have tractable distributions, so that the KL-divergence can be computed easily, either with closed-form solution or Monte Carlo estimation.
Revised: Their policies should have tractable distributions, so that the KL-divergence can be computed easily, either with closed-form solution or Monte Carlo estimation.

Original: Several RL algorithms enjoy such properties (Haarnoja et al., 2018;
Revised: Several RL algorithms enjoy such properties (Haarnoja et al., 2018;

Original: Schulman et al., 2015; 2017).
Revised: Schulman et al., 2015; 2017).

Original: In particular, the soft actor-critic (SAC, Haarnoja et al., 2018) is a straightforward choice, as it currently outperforms other candidates and is off-policy, thus maintains a single critic shared among all agents (instead of one critic per agent), which reduces computation costs.
Revised: In particular, the soft actor-critic (SAC, Haarnoja et al., 2018) is a straightforward choice, as it currently outperforms other candidates and is off-policy, thus maintains a single critic shared among all agents (instead of one critic per agent), which reduces computation costs.

Original: 2. S OFT ACTOR - CRITIC
Revised: 2. S OFT ACTOR - CRITIC

Original: SAC (Haarnoja et al., 2018) is an off-policy learning algorithm which ﬁnds the information projection of the Boltzmann Q-function onto the set of diagonal Gaussian policies Π :
Revised: SAC (Haarnoja et al., 2018) is an off-policy learning algorithm which ﬁnds the information projection of the Boltzmann Q-function onto the set of diagonal Gaussian policies Π :

Original: [Equation]
Revised: [Equation]

Original: where α ∈ (0 , 1) controls the temperature, i.e. the peakedness of the distribution.
Revised: where α ∈ (0 , 1) controls the temperature, i.e. the peakedness of the distribution.

Original: The policy π , critic
Revised: The policy π , critic

Original: Q , and value function V are optimized according to the following loss functions:
Revised: Q , and value function V are optimized according to the following loss functions:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where B is the replay buffer.
Revised: where B is the replay buffer.

Original: The policy used in SAC as introduced in Haarnoja et al.
Revised: The policy used in SAC as introduced in Haarnoja et al.

Original: (2018) is
Revised: (2018) is

Original: Gaussian, which is both stochastic and tractable, thus compatible with our AR loss function in Eq.
Revised: Gaussian, which is both stochastic and tractable, thus compatible with our AR loss function in Eq.

Original: Together with the AR loss in Eq.
Revised: Together with the AR loss in Eq.

Original: 1, the ﬁnal policy loss becomes:
Revised: 1, the ﬁnal policy loss becomes:

Original: [Equation]
Revised: [Equation]

Original: However, Gaussian policies are arguably of limited expressibility; we can improve on the family of policy distributions without sacriﬁcing qualities necessary for AR or SAC by using Normalizing
Revised: However, Gaussian policies are arguably of limited expressibility; we can improve on the family of policy distributions without sacriﬁcing qualities necessary for AR or SAC by using Normalizing

Original: Flows (NFs, Rezende & Mohamed, 2015).
Revised: Flows (NFs, Rezende & Mohamed, 2015).

Original: 2. N ORMALIZING FLOWS
Revised: 2. N ORMALIZING FLOWS

Original: NFs (Rezende & Mohamed, 2015) were introduced as a means of transforming simple distributions into more complex distributions using learnable and invertible functions.
Revised: NFs (Rezende & Mohamed, 2015) were introduced as a means of transforming simple distributions into more complex distributions using learnable and invertible functions.

Original: Given a random variable zwith density q 0 , they deﬁne a set of differentiable and invertible functions, { f i } Ni =1 , which generate a sequence of d -dimensional random variables, { z i } Ni =1 .
Revised: Given a random variable zwith density q 0 , they deﬁne a set of differentiable and invertible functions, { f i } Ni =1 , which generate a sequence of d -dimensional random variables, { z i } Ni =1 .

Original: Because SAC uses explicit, yet simple parametric policies, NFs can be used to transform the SAC policy into a richer one (e.g., multimodal) without risk loss of information.
Revised: Because SAC uses explicit, yet simple parametric policies, NFs can be used to transform the SAC policy into a richer one (e.g., multimodal) without risk loss of information.

Original: For example, Mazoure et al.
Revised: For example, Mazoure et al.

Original: (2019) enhanced SAC using a family of radial contractions around a point z 0 ∈ R d ,
Revised: (2019) enhanced SAC using a family of radial contractions around a point z 0 ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: (6) for α ∈ R + and β ∈ R .
Revised: (6) for α ∈ R + and β ∈ R .

Original: This results in a rich set of policies comprised of an initial noise sample a 0 , a state-noise embedding h θ ( a 0 , s t ) , and a ﬂow { f φ i } Ni =1 of arbitrary length N , parameterized by φ = { φ i } Ni =1 .
Revised: This results in a rich set of policies comprised of an initial noise sample a 0 , a state-noise embedding h θ ( a 0 , s t ) , and a ﬂow { f φ i } Ni =1 of arbitrary length N , parameterized by φ = { φ i } Ni =1 .

Original: Sampling from the policy π φ,θ ( a t | s t ) can be described by the following set of equations:
Revised: Sampling from the policy π φ,θ ( a t | s t ) can be described by the following set of equations:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where h θ = a 0 σ I + µ ( s t ) depends on the state and the noise variance σ > 0 .
Revised: where h θ = a 0 σ I + µ ( s t ) depends on the state and the noise variance σ > 0 .

Original: Different SAC policies can thus be crafted by parameterizing their NFs layers.
Revised: Different SAC policies can thus be crafted by parameterizing their NFs layers.

Original:  ARAC: A TTRACTION -R EPULSION A CTOR -C RITIC
Revised:  ARAC: A TTRACTION -R EPULSION A CTOR -C RITIC

Original: We now detail the general procedure for training a population of agents using the proposed diversityseeking AR mechanism.
Revised: We now detail the general procedure for training a population of agents using the proposed diversityseeking AR mechanism.

Original: More speciﬁcally, we consider here SAC agents enhanced with NFs (Mazoure et al., 2019).
Revised: More speciﬁcally, we consider here SAC agents enhanced with NFs (Mazoure et al., 2019).

Original: Figure 1 displays the general ﬂow of the procedure.
Revised: Figure 1 displays the general ﬂow of the procedure.

Original: Algorithm 1 (Appendix) provides the pseudo-code of the proposed ARAC strategy, where sub-procedures for rollout and archive update can be found in the Appendix.
Revised: Algorithm 1 (Appendix) provides the pseudo-code of the proposed ARAC strategy, where sub-procedures for rollout and archive update can be found in the Appendix.

Original: Overview ARAC works by evolving a population of M SAC agents { π mφ,θ } Mm =1 with radial NFs policies (Eq. 7) and shared critic Q ω , and by maintaining an archive of policies encountered in previous generations of the population.
Revised: Overview ARAC works by evolving a population of M SAC agents { π mφ,θ } Mm =1 with radial NFs policies (Eq. 7) and shared critic Q ω , and by maintaining an archive of policies encountered in previous generations of the population.

Original: After performing T steps per agent on the environment (Alg.
Revised: After performing T steps per agent on the environment (Alg.

Original: L8-12), individuals are evaluated by performing R rollouts 2 on the environment (Alg. 1 L26-28).
Revised: L8-12), individuals are evaluated by performing R rollouts 2 on the environment (Alg. 1 L26-28).

Original: This allows to identify the top- K best agents (Alg. 1 L29), also called elites , which will be used to update the critic as they provide the most meaningful feedback (Alg. 1 L13-17).
Revised: This allows to identify the top- K best agents (Alg. 1 L29), also called elites , which will be used to update the critic as they provide the most meaningful feedback (Alg. 1 L13-17).

Original: The archive is ﬁnally updated in a diversity-seeking fashion using the current population (Alg. 1 L30).
Revised: The archive is ﬁnally updated in a diversity-seeking fashion using the current population (Alg. 1 L30).

Original: The core component of the proposed approach lies within the update of the agents (Alg. 1 L18-25).
Revised: The core component of the proposed approach lies within the update of the agents (Alg. 1 L18-25).

Original: During this phase, elite individuals are updated using AR operations w.r.t.
Revised: During this phase, elite individuals are updated using AR operations w.r.t.

Original: policies sampled from the archive (Eq. 5), whereas non-elites are updated regularly (Eq. 2).
Revised: policies sampled from the archive (Eq. 5), whereas non-elites are updated regularly (Eq. 2).

Original: 3. E NHANCING DIVERSITY IN THE ARCHIVE
Revised: 3. E NHANCING DIVERSITY IN THE ARCHIVE

Original: Throughout the training process, we maintain an archive G of maximum capacity G , which contains some previously encountered policies.
Revised: Throughout the training process, we maintain an archive G of maximum capacity G , which contains some previously encountered policies.

Original: The process goes as follow: until reaching full capacity, the archive saves a copy of the parameters of every individual in the population after the evaluation step.
Revised: The process goes as follow: until reaching full capacity, the archive saves a copy of the parameters of every individual in the population after the evaluation step.

Original: However, by naively adding all individuals as if the archive were just a heap, the archive could end up ﬁlled with policies leading to similar rewards, which would result in a loss of diversity (Mauldin, 1984).
Revised: However, by naively adding all individuals as if the archive were just a heap, the archive could end up ﬁlled with policies leading to similar rewards, which would result in a loss of diversity (Mauldin, 1984).

Original: We mitigate this issue by keeping track of two ﬁtness clusters (low and high) using the partition formed by running a k -means algorithm on the ﬁtness value.
Revised: We mitigate this issue by keeping track of two ﬁtness clusters (low and high) using the partition formed by running a k -means algorithm on the ﬁtness value.

Original: Hence, when |G| = G is reached and a new individual is added to the archive, it randomly replaces an archived policy from its respective cluster.
Revised: Hence, when |G| = G is reached and a new individual is added to the archive, it randomly replaces an archived policy from its respective cluster.

Original: This approach, also known as niching , has proved itself effective at maintaining high diversity levels (Gupta & Ghaﬁr, 2012; Mahfoud, 1995).
Revised: This approach, also known as niching , has proved itself effective at maintaining high diversity levels (Gupta & Ghaﬁr, 2012; Mahfoud, 1995).

Original: 3. D ISCOVERING NEW POLICIES THROUGH A TTRACTION -R EPULSION
Revised: 3. D ISCOVERING NEW POLICIES THROUGH A TTRACTION -R EPULSION

Original: The crux of this work lies in the explicit search for diversity in the policy space achieved using the
Revised: The crux of this work lies in the explicit search for diversity in the policy space achieved using the

Original: AR mechanism.
Revised: AR mechanism.

Original: Since the KL between two base policies (i.e. input of the ﬁrst ﬂow layer) can be trivially maximized by driving their means apart, we apply attraction-repulsion only on the ﬂow layers, while holding the mean of the base policy constant.
Revised: Since the KL between two base policies (i.e. input of the ﬁrst ﬂow layer) can be trivially maximized by driving their means apart, we apply attraction-repulsion only on the ﬂow layers, while holding the mean of the base policy constant.

Original: This ensures that the KL term doesn’t depend on the difference in means and hence controls the magnitude of the AR mechanism.
Revised: This ensures that the KL term doesn’t depend on the difference in means and hence controls the magnitude of the AR mechanism.

Original: Every time the AR operator is applied (Alg. 1 L20-21), n policies are sampled from the archive and are used for estimating the AR loss (Eq. 1).
Revised: Every time the AR operator is applied (Alg. 1 L20-21), n policies are sampled from the archive and are used for estimating the AR loss (Eq. 1).

Original: As in Hong et al.
Revised: As in Hong et al.

Original: , we consider two possible strategies to dictate the value of β π (cid:48) coefﬁcients for policies π (cid:48) ∼ G :
Revised: , we consider two possible strategies to dictate the value of β π (cid:48) coefﬁcients for policies π (cid:48) ∼ G :

Original: [Equation]
Revised: [Equation]

Original: where f ( π ) 3 represents the ﬁtness function of policy π (average reward in our case), and f min and f max are estimated based on the n sampled archived policies.
Revised: where f ( π ) 3 represents the ﬁtness function of policy π (average reward in our case), and f min and f max are estimated based on the n sampled archived policies.

Original: The proactive strategy aims to mimic high reward archived policies, while the reactive strategy is more cautious, only repulsing away the current policy from low ﬁtness archived policies.
Revised: The proactive strategy aims to mimic high reward archived policies, while the reactive strategy is more cautious, only repulsing away the current policy from low ﬁtness archived policies.

Original: Using this approach, the current agent policy will be attracted to some sampled policies ( β π (cid:48) < 0 ) and will be repulsed from others ( β π (cid:48) ≥ 0 ) in a more or less aggressive way, depending on the strategy.
Revised: Using this approach, the current agent policy will be attracted to some sampled policies ( β π (cid:48) < 0 ) and will be repulsed from others ( β π (cid:48) ≥ 0 ) in a more or less aggressive way, depending on the strategy.

Original: Unlike Hong et al.
Revised: Unlike Hong et al.

Original: (2018) who applied proactive and reactive strategies on policies up to 5 timesteps back, we maintain an archive consisting of two clusters seen so far: policies with low and high ﬁtness, respectively.
Revised: (2018) who applied proactive and reactive strategies on policies up to 5 timesteps back, we maintain an archive consisting of two clusters seen so far: policies with low and high ﬁtness, respectively.

Original: Having this cluster allows to attract/repulse from a set of diverse agents, replacing high-reward policies by policies with similar performance.
Revised: Having this cluster allows to attract/repulse from a set of diverse agents, replacing high-reward policies by policies with similar performance.

Original: Indeed, without this process, elements of the archive would collapse on the most frequent policy, from which all agents would attract/repulse.
Revised: Indeed, without this process, elements of the archive would collapse on the most frequent policy, from which all agents would attract/repulse.

Original: To avoid performing AR against a single "average policy" , we separate low-reward and high-reward agents via clustering.
Revised: To avoid performing AR against a single "average policy" , we separate low-reward and high-reward agents via clustering.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The challenges of exploration are well studied in the RL literature.
Revised: The challenges of exploration are well studied in the RL literature.

Original: Previously proposed approaches for overcoming hard exploration domains tend to either increase the capacity of the state-action value function (Gal & Ghahramani, 2016; Henderson et al., 2017) or the policy expressivity (Mazoure et al., 2019; Tang & Agrawal, 2018; Touati et al., 2018).
Revised: Previously proposed approaches for overcoming hard exploration domains tend to either increase the capacity of the state-action value function (Gal & Ghahramani, 2016; Henderson et al., 2017) or the policy expressivity (Mazoure et al., 2019; Tang & Agrawal, 2018; Touati et al., 2018).

Original: This work rather tackles exploration from a diverse multi-agent perspective.
Revised: This work rather tackles exploration from a diverse multi-agent perspective.

Original: Unlike prior population-based approaches for exploration (Conti et al., 2018; Khadka & Tumer, 2018; Pourchot & Sigaud, 2019), which seek diversity through the parameters space, we directly promote diversity in the policy space.
Revised: Unlike prior population-based approaches for exploration (Conti et al., 2018; Khadka & Tumer, 2018; Pourchot & Sigaud, 2019), which seek diversity through the parameters space, we directly promote diversity in the policy space.

Original: The current work was inspired by Hong et al.
Revised: The current work was inspired by Hong et al.

Original: (2018), who relied on the KL divergence to attract/repulse from a set of previous policies to discover new solutions.
Revised: (2018), who relied on the KL divergence to attract/repulse from a set of previous policies to discover new solutions.

Original: However, in their work, the archive is time-based (they restrict themselves to the 5 most recent policies), while our archive is built following a diversity-seeking strategy (i.e., niching and policies come from multiple agents).
Revised: However, in their work, the archive is time-based (they restrict themselves to the 5 most recent policies), while our archive is built following a diversity-seeking strategy (i.e., niching and policies come from multiple agents).

Original: Notably, ARAC is different of previously discussed works in that it explores the action space in multiple regions simultaneously, a property enforced through the AR mechanism.
Revised: Notably, ARAC is different of previously discussed works in that it explores the action space in multiple regions simultaneously, a property enforced through the AR mechanism.

Original: The proposed approach bears some resemblance with Liu et al.
Revised: The proposed approach bears some resemblance with Liu et al.

Original: (2017), who took advantage of a multi-agent framework in order to perform repulsion operations among agents using of similarity kernels between parameters of the agents.
Revised: (2017), who took advantage of a multi-agent framework in order to perform repulsion operations among agents using of similarity kernels between parameters of the agents.

Original: The AR mechanism gives rise to exploration through structured policy rather than randomized policy.
Revised: The AR mechanism gives rise to exploration through structured policy rather than randomized policy.

Original: This strategy has also been employed in multi-task learning (Gupta et al., 2018), where experience on previous tasks was used to explore on new tasks.
Revised: This strategy has also been employed in multi-task learning (Gupta et al., 2018), where experience on previous tasks was used to explore on new tasks.

Original:  E XPERIMENTS 5. D IDACTIC EXAMPLE
Revised:  E XPERIMENTS 5. D IDACTIC EXAMPLE

Original: Consider a 2-dimensional multi-armed bandit problem where the actions lie in the real square [ − 6 , 6] 2 .
Revised: Consider a 2-dimensional multi-armed bandit problem where the actions lie in the real square [ − 6 , 6] 2 .

Original: We illustrate the example of using a proactive strategy where a SAC agent with radial ﬂows policy imitates a desirable (expert) policy while simultaneously repelling from a less desirable policy.
Revised: We illustrate the example of using a proactive strategy where a SAC agent with radial ﬂows policy imitates a desirable (expert) policy while simultaneously repelling from a less desirable policy.

Original: The task consists in matching the expert’s policy (blue density) while avoiding taking actions from a repulsive policy π (cid:48) (red).
Revised: The task consists in matching the expert’s policy (blue density) while avoiding taking actions from a repulsive policy π (cid:48) (red).

Original: We illustrate the properties of radial ﬂows in Figure 2 by increasing the number of ﬂows (where 0 ﬂow corresponds to a Gaussian distribution).
Revised: We illustrate the properties of radial ﬂows in Figure 2 by increasing the number of ﬂows (where 0 ﬂow corresponds to a Gaussian distribution).

Original: We observe that increasing the number of ﬂows (bottom to top) leads to more complex policy’s shapes and multimodality unlike the Gaussian policy which has its variance shrinked (the KL divergence the ﬁtness depending on the context
Revised: We observe that increasing the number of ﬂows (bottom to top) leads to more complex policy’s shapes and multimodality unlike the Gaussian policy which has its variance shrinked (the KL divergence the ﬁtness depending on the context

Original: Increasing the number of ﬂows leads to more complex policy’s shape.
Revised: Increasing the number of ﬂows leads to more complex policy’s shape.

Original: is proportional to the ratio of the two variances, hence maximizing it can lead to a reduction in the variance which can be detrimental for exploration purpose).
Revised: is proportional to the ratio of the two variances, hence maximizing it can lead to a reduction in the variance which can be detrimental for exploration purpose).

Original: Details are provided in Appendix.
Revised: Details are provided in Appendix.

Original: 5. M U J O C O LOCOMOTION BENCHMARKS
Revised: 5. M U J O C O LOCOMOTION BENCHMARKS

Original: We now compare ARAC against the CEM-TD3 (Pourchot & Sigaud, 2019), ERL (Khadka &
Revised: We now compare ARAC against the CEM-TD3 (Pourchot & Sigaud, 2019), ERL (Khadka &

Original: Tumer, 2018) and CERL (Khadka et al., 2019)  baselines on seven continuous control tasks from the MuJoco suite (Duan et al., 2016):
Revised: Tumer, 2018) and CERL (Khadka et al., 2019) multi-agent baselines on seven continuous control tasks from the MuJoco suite (Duan et al., 2016):

Original: Ant-v2 , HalfCheetah-v2 , Humanoid-v2 ,
Revised: Ant-v2 , HalfCheetah-v2 , Humanoid-v2 ,

Original: HumanoidStandup-v2 , Hopper-v2 , Walker2d-v2 and Humanoid (rllab) .
Revised: HumanoidStandup-v2 , Hopper-v2 , Walker2d-v2 and Humanoid (rllab) .

Original: We also designed a sparse reward environment SparseHumanoid-v2 .
Revised: We also designed a sparse reward environment SparseHumanoid-v2 .

Original: All algorithms are run over 1 M time steps on each environment, except Humanoid (rllab) which gets 2 M time steps and
Revised: All algorithms are run over 1 M time steps on each environment, except Humanoid (rllab) which gets 2 M time steps and

Original: SparseHumanoid-v2 on 0 .
Revised: SparseHumanoid-v2 on 0 .

Original: 6 M time steps.
Revised: 6 M time steps.

Revised: We also include comparison against single-agent baselines.

Original: ARAC performs R = 10 rollouts for evaluation steps every 10 , 000 interaction steps with the environment.
Revised: ARAC performs R = 10 rollouts for evaluation steps every 10 , 000 interaction steps with the environment.

Original: We consider a small population of N = 5 individuals with K = 2 as elites.
Revised: We consider a small population of N = 5 individuals with K = 2 as elites.

Original: Every SAC agent has one feedforward hidden layer of 256 units acting as state embedding, followed by a radial ﬂow of length ∈ { 3 , 4 } .
Revised: Every SAC agent has one feedforward hidden layer of 256 units acting as state embedding, followed by a radial ﬂow of length ∈ { 3 , 4 } .

Original: A temperature of α = 0 .
Revised: A temperature of α = 0 .

Original: 05 or 0 . 2 is used across all the environments (See appendix for more details).
Revised: 05 or 0 . 2 is used across all the environments (See appendix for more details).

Original: AR operations are carried out by sampling uniformly n = 5 archived policies from G .
Revised: AR operations are carried out by sampling uniformly n = 5 archived policies from G .

Original: Parameters details are provided in the Appendix (Table 4).
Revised: Parameters details are provided in the Appendix (Table 4).

Original: All networks are trained with Adam optimizer (Kingma & Ba, 2015) using a learning rate of 3 E − 4 .
Revised: All networks are trained with Adam optimizer (Kingma & Ba, 2015) using a learning rate of 3 E − 4 .

Original: Baselines CEM-TD3 4 ,
Revised: Baselines CEM-TD3 4 ,

Original: ERL 5 , CERL 6 use the code contained in their respective repositories.
Revised: ERL 5 , CERL 6 use the code contained in their respective repositories.

Original: [Table]
Revised: [Table]

Original: Figure 4 displays the performance of all algorithms on three environments over time steps (see
Revised: Figure 4 displays the performance of all algorithms on three environments over time steps (see

Original: Appendix Figure 7 for all environments).
Revised: Appendix Figure 7 for all environments).

Original: Results  averaged  the best observed reward for method
Revised: Results are averaged over 5 random seeds. Table 1 reports the best observed reward for each method.

Revised: Curves are smoothed using Savitzky-Golay ﬁltering with window size of

Original: Small state space environments HalfCheetah-v2 , Hopper-v2 , and Walker2d-v2 are low-dimensional state space environments ( d ≤ 17 ).
Revised: Small state space environments HalfCheetah-v2 , Hopper-v2 , and Walker2d-v2 are low-dimensional state space environments ( d ≤ 17 ).

Original: Except for HalfCheetah-v2 , the proposed approach shows comparable results with its concurrent.
Revised: Except for HalfCheetah-v2 , the proposed approach shows comparable results with its concurrent.

Original: Those results match the ﬁndings of (Plappert et al., 2018) that some environments with well-structured dynamics require little exploration.
Revised: Those results match the ﬁndings of (Plappert et al., 2018) that some environments with well-structured dynamics require little exploration.

Original: Full learning curves can be found in the Appendix.
Revised: Full learning curves can be found in the Appendix.

Original: Deceptive reward and
Revised: Deceptive reward and

Original: Large state space environments Humanoid-v2 ,
Revised: Large state space environments Humanoid-v2 ,

Original: HumanoidStandup-v2 and Humanoid (rllab) belong to bipedal environments with high-dimensional state space ( d = 376 and d = 147 ), and are known to trap algorithms into suboptimal solutions.
Revised: HumanoidStandup-v2 and Humanoid (rllab) belong to bipedal environments with high-dimensional state space ( d = 376 and d = 147 ), and are known to trap algorithms into suboptimal solutions.

Original: In addition to the legs, the agent also needs to control the arms, which may inﬂuence the walking way and hence induce deceptive rewards (Conti et al., 2018).
Revised: In addition to the legs, the agent also needs to control the arms, which may inﬂuence the walking way and hence induce deceptive rewards (Conti et al., 2018).

Original: Figure 4 shows the learning curves on MuJoCo tasks.
Revised: Figure 4 shows the learning curves on MuJoCo tasks.

Original: We observe that ARAC beats both baselines in performance as well as in convergence rate.
Revised: We observe that ARAC beats both baselines in performance as well as in convergence rate.

Original: Ant-v2 is another high-dimensional state space environment ( d ≥ 100 ).
Revised: Ant-v2 is another high-dimensional state space environment ( d ≥ 100 ).

Original: In an unstable setup, a naive algorithm implementing an unbalanced fast walk could still generate high reward, the reward taking into account the distance from start, instead of learning to stand, stabilize, and walk (as expected).
Revised: In an unstable setup, a naive algorithm implementing an unbalanced fast walk could still generate high reward, the reward taking into account the distance from start, instead of learning to stand, stabilize, and walk (as expected).

Original: Sparse reward environment To test ARAC in a sparse reward environment, we created
Revised: Sparse reward environment To test ARAC in a sparse reward environment, we created

Original: SparseHumanoid-v2 .
Revised: SparseHumanoid-v2 .

Original: The dynamic is the same as Humanoid-v2 but rewards of +1 is granted only given is the center of mass of the agent is above a threshold (set to 0.6 unit in our case).
Revised: The dynamic is the same as Humanoid-v2 but rewards of +1 is granted only given is the center of mass of the agent is above a threshold (set to 0.6 unit in our case).

Original: The challenge not only lies in the sparse reward property but also on the complex body dynamic that can make the agent falling down and terminating the episode.
Revised: The challenge not only lies in the sparse reward property but also on the complex body dynamic that can make the agent falling down and terminating the episode.

Original: As shown in Figure 4, ARAC is the only method that can achieve non zero performance.
Revised: As shown in Figure 4, ARAC is the only method that can achieve non zero performance.

Original: A comparison against single agent methods in the
Revised: A comparison against single agent methods in the

Original: Appendix also shows better performance for ARAC.
Revised: Appendix also shows better performance for ARAC.

Original: Sample efﬁciency compared with single agent methods Figure 8 (in Appendix) also shows that the sample efﬁciency of the population-based ARAC compares to a single SAC agent (with and without NFs) and other baselines methods (SAC, TD3).
Revised: Sample efﬁciency compared with single agent methods Figure 3 (in Appendix) also shows that the sample efﬁciency of the population-based ARAC compares to a single SAC agent (with and without NFs) and other baselines methods (SAC, TD3).

Original: On Humanoid-v2 and Ant-v2 ARAC converges faster, reaching the 6 k ( 4 k, respectively) milestone performance after only 1 M steps, while a single SAC agent requires 4 M ( 3 M, respectively) steps according to (Haarnoja et al., 2018).
Revised: On Humanoid-v2 and Ant-v2 ARAC converges faster, reaching the 6 k ( 4 k, respectively) milestone performance after only 1 M steps, while a single SAC agent requires 4 M ( 3 M, respectively) steps according to (Haarnoja et al., 2018).

Original: In general, ARAC achieves competitive results (no ﬂat curves) and makes the most difference (faster convergence and better performance) in the biped environments.
Revised: In general, ARAC achieves competitive results (no ﬂat curves) and makes the most difference (faster convergence and better performance) in the biped environments.

Original: Attraction-repulsion ablation study To illustrate the impact of repulsive forces, we introduce a hyperparameter λ in the overall loss (Eq. 5):
Revised: Attraction-repulsion ablation study To illustrate the impact of repulsive forces, we introduce a hyperparameter λ in the overall loss (Eq. 5):

Original: [Equation]
Revised: [Equation]

Original: We ran an ablation analysis on Humanoid-v2 by varying that coefﬁcient.
Revised: We ran an ablation analysis on Humanoid-v2 by varying that coefﬁcient.

Original: For two random states, we sampled 500 actions from all agents and mapped these actions onto a two-dimensional space (via t-SNE).
Revised: For two random states, we sampled 500 actions from all agents and mapped these actions onto a two-dimensional space (via t-SNE).

Original:  Figure 3 shows that without repulsion ( λ = 0 ), actions from all agents are entangled, while repulsion ( λ > 0) forces agents to behave differently and hence explore different regions of the action space.
Revised: Appendix Figure 5 shows that without repulsion ( λ = 0 ), actions from all agents are entangled, while repulsion ( λ > 0) forces agents to behave differently and hence explore different regions of the action space.

Original: The second ablation study is dedicated to highlight the differences between a Gaussian policy (similar to Hong et al.
Revised: The second ablation study is dedicated to highlight the differences between a Gaussian policy (similar to Hong et al.

Original: (2018) and an NF policy under AR operators.
Revised: (2018) and an NF policy under AR operators.

Original: As one can observe in Figure 6, using a
Revised: As one can observe in Figure 6, using a

Original: Gaussian policy deteriorates the solution as the repulsive KL term drives apart the means of agents and blows up/ shrinks the variance of the Gaussian policy.
Revised: Gaussian policy deteriorates the solution as the repulsive KL term drives apart the means of agents and blows up/ shrinks the variance of the Gaussian policy.

Original: On the other hand, applying the AR term on the NF layers maximizes the KL conditioned on the mean and variance of both base policies, resulting in a solution which allows sufﬁcient exploration.
Revised: On the other hand, applying the AR term on the NF layers maximizes the KL conditioned on the mean and variance of both base policies, resulting in a solution which allows sufﬁcient exploration.

Original: More details are provided in the Appendix.
Revised: More details are provided in the Appendix.

Original: Finally, through a toy example subject to AR, we characterize the policy’s shape when increasing the number of the radial ﬂow policy in Figure 2 (experimental setup in Appendix).
Revised: Finally, through a toy example subject to AR, we characterize the policy’s shape when increasing the number of the radial ﬂow policy in Figure 2 (experimental setup in Appendix).

Original: Unlike the diagonal
Revised: Unlike the diagonal

Original: Gaussian policy (SAC) that has symmetry constraints, increasing the number of ﬂows allows the radial policy to adopt more complex shapes (from bottom to top).
Revised: Gaussian policy (SAC) that has symmetry constraints, increasing the number of ﬂows allows the radial policy to adopt more complex shapes (from bottom to top).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we addressed the issue of RL domains with deceptive rewards by introducing a population-based search model for optimal policies using attraction-repulsion operators.
Revised: In this paper, we addressed the issue of RL domains with deceptive rewards by introducing a population-based search model for optimal policies using attraction-repulsion operators.

Original: Our method relies on powerful density estimators (normalizing ﬂows), to let policies exploit the reward landscape under AR constraints.
Revised: Our method relies on powerful density estimators (normalizing ﬂows), to let policies exploit the reward landscape under AR constraints.

Original: Our ablation studies showed that (1) the strength of AR and (2) the number of ﬂows are the two factors which predominantly affect the shape of the policy.
Revised: Our ablation studies showed that (1) the strength of AR and (2) the number of ﬂows are the two factors which predominantly affect the shape of the policy.

Original: Selecting the correct
Revised: Selecting the correct

Original: AR coefﬁcient is therefore important to obtain good performance, while at the same time preventing premature convergence.
Revised: AR coefﬁcient is therefore important to obtain good performance, while at the same time preventing premature convergence.

Original: Empirical results on the MuJoCo suite demonstrate high performance of the proposed method in most settings, including with sparse rewards.
Revised: Empirical results on the MuJoCo suite demonstrate high performance of the proposed method in most settings, including with sparse rewards.

Original: Moreover, in biped environments that are known to trap algorithms into suboptimal solutions, ARAC enjoys higher sample efﬁciency and better performance compared to its competitors which conﬁrms our intuitions on using AR with normalizing ﬂows.
Revised: Moreover, in biped environments that are known to trap algorithms into suboptimal solutions, ARAC enjoys higher sample efﬁciency and better performance compared to its competitors which conﬁrms our intuitions on using AR with normalizing ﬂows.

Original: As future steps, borrowing from multi-objective optimization literature methods could allow one to combine other diversity metrics with the performance objective, to in turn improve the coverage of the solution space among the individuals by working with the corresponding Pareto front (Horn et al., 1994).
Revised: As future steps, borrowing from multi-objective optimization literature methods could allow one to combine other diversity metrics with the performance objective, to in turn improve the coverage of the solution space among the individuals by working with the corresponding Pareto front (Horn et al., 1994).

Original: A PPENDIX R EPRODUCIBILITY C HECKLIST
Revised: A PPENDIX R EPRODUCIBILITY C HECKLIST

Original: We follow the reproducibility checklist (Pineau, 2018) and point to relevant sections explaining them here.
Revised: We follow the reproducibility checklist (Pineau, 2018) and point to relevant sections explaining them here.

Original: Forall algorithms presented, check if you include:
Revised: Forall algorithms presented, check if you include:

Original: • A clear description of the algorithm, see main paper and included codebase.
Revised: • A clear description of the algorithm, see main paper and included codebase.

Original: The proposed approach is completely described by Alg.
Revised: The proposed approach is completely described by Alg.

Original: 1 (main paper), 2 (Appendix), and(Appendix).
Revised: 1 (main paper), 2 (Appendix), and(Appendix).

Original: The proposed population-based method uses attraction-repulsion operators in order to enforce a better policy space coverage by different agents.
Revised: The proposed population-based method uses attraction-repulsion operators in order to enforce a better policy space coverage by different agents.

Original: • An analysis of the complexity (time, space, sample size) of the algorithm.
Revised: • An analysis of the complexity (time, space, sample size) of the algorithm.

Original: See Appendix Figure 7 and 8.
Revised: See Appendix Figure 7 and 3.

Original: Experimentally, we demonstrate improvement in sample complexity as discussed in our main paper.
Revised: Experimentally, we demonstrate improvement in sample complexity as discussed in our main paper.

Original: In term of computation time, the proposed method scales linearly with the population size if agents are evaluated sequentially (as presented in Alg.for clarity).
Revised: In term of computation time, the proposed method scales linearly with the population size if agents are evaluated sequentially (as presented in Alg.for clarity).

Original: However, this as mentioned in the paper, can be parallelized.
Revised: However, this as mentioned in the paper, can be parallelized.

Original: All our results are obtained using M small network architectures with 1 × 256 -units hidden layer followed by f layers of | A | + 2 units each ( f being the number of radial ﬂows and | A | being the action space dimension).
Revised: All our results are obtained using M small network architectures with 1 × 256 -units hidden layer followed by f layers of | A | + 2 units each ( f being the number of radial ﬂows and | A | being the action space dimension).

Original: • A link to a downloadable source code, including all dependencies.
Revised: • A link to a downloadable source code, including all dependencies.

Original: The code is included with the Appendix as a zip ﬁle; all dependencies can be installed using Python’s package manager.
Revised: The code is included with the Appendix as a zip ﬁle; all dependencies can be installed using Python’s package manager.

Original: Upon publication, the code would be available on Github.
Revised: Upon publication, the code would be available on Github.

Original: For all ﬁgures and tables that present empirical results, check if you include:
Revised: For all ﬁgures and tables that present empirical results, check if you include:

Original: • A complete description of the data collection process, including sample size.
Revised: • A complete description of the data collection process, including sample size.

Original: We use standard benchmarks provided in OpenAI Gym (Brockman et al., 2016).
Revised: We use standard benchmarks provided in OpenAI Gym (Brockman et al., 2016).

Original: • A link to downloadable version of the dataset or simulation environment.
Revised: • A link to downloadable version of the dataset or simulation environment.

Original: See: https://github.com/ • An explanation of how samples were allocated for training / validation / testing.
Revised: See: https://github.com/ • An explanation of how samples were allocated for training / validation / testing.

Original: We do not use a training-validation-test split, but instead report the mean performance (and one standard deviation) of the policy at evaluation time, openai/gym for OpenAI Gym benchmarks and https://www.roboti.us/index.html for MuJoCo suite.
Revised: We do not use a training-validation-test split, but instead report the mean performance (and one standard deviation) of the policy at evaluation time, openai/gym for OpenAI Gym benchmarks and https://www.roboti.us/index.html for MuJoCo suite.

Original: obtained withrandom seeds.
Revised: obtained withrandom seeds.

Original: An explanation of any data that were excluded.
Revised: An explanation of any data that were excluded.

Original: We did not compare on easy environments (e.g. Reacher-v2 ) because all existing methods perform well on them.
Revised: We did not compare on easy environments (e.g. Reacher-v2 ) because all existing methods perform well on them.

Original: In that case, the improvement of our method upon baselines is incremental and not worth mentioning.
Revised: In that case, the improvement of our method upon baselines is incremental and not worth mentioning.

Original: • The exact number of evaluation runs.
Revised: • The exact number of evaluation runs.

Original: 5 seeds for MuJoCo experiments, 1M, 2M or 3M environment steps depending on the domain.
Revised: 5 seeds for MuJoCo experiments, 1M, 2M or 3M environment steps depending on the domain.

Original: • A description of how experiments were run.
Revised: • A description of how experiments were run.

Original: See Section 5 in the main paper and didactic example details in Appendix.
Revised: See Section 5 in the main paper and didactic example details in Appendix.

Original: • A clear deﬁnition of the speciﬁc measure or statistics used to report results.
Revised: • A clear deﬁnition of the speciﬁc measure or statistics used to report results.

Original: Undiscounted returns across the whole episode are reported, and in turn averaged across 5 seeds.
Revised: Undiscounted returns across the whole episode are reported, and in turn averaged across 5 seeds.

Original: • Clearly deﬁned error bars.
Revised: • Clearly deﬁned error bars.

Original: Conﬁdence intervals and table values are always mean ±standard deviation over 5 seeds.
Revised: Conﬁdence intervals and table values are always mean ±standard deviation over 5 seeds.

Original: • A description of results with central tendency (e.g. mean) and variation (e.g. stddev) .
Revised: • A description of results with central tendency (e.g. mean) and variation (e.g. stddev) .

Original: All results use the mean and standard deviation.
Revised: All results use the mean and standard deviation.

Original: • A description of the computing infrastructure used.
Revised: • A description of the computing infrastructure used.

Original: All runs used 1 CPU for all experiments (toy and MuJoCo) with 8 Gb of memory.
Revised: All runs used 1 CPU for all experiments (toy and MuJoCo) with 8 Gb of memory.

Original: [Figure]
Revised: [Figure]

Original: To illustrate the impact of the repulsive force coefﬁcient λ , we ran an ablation analysis by varying that coefﬁcient (recall that the overall loss function is L π =
Revised: To illustrate the impact of the repulsive force coefﬁcient λ , we ran an ablation analysis by varying that coefﬁcient (recall that the overall loss function is L π =

Original: L π, SAC + λ L AR where λ
Revised: L π, SAC + λ L AR where λ

Original: = 1 in our experiment).
Revised: = 1 in our experiment).

Original: For two random states, we sampled 500 actions from all agents and mapped theses actions in a common 2-dimensional space (t-SNE).
Revised: For two random states, we sampled 500 actions from all agents and mapped theses actions in a common 2-dimensional space (t-SNE).

Original: As shown in the Figure 5, policies trained without AR ( λ = 0 ) result in entangled actions, while increasing the repulsive coefﬁcient λ forces agents to have different actions and hence explore different regions of the policy space.
Revised: As shown in the Figure above, policies trained without AR ( λ = 0 ) result in entangled actions, while increasing the repulsive coefﬁcient λ forces agents to have different actions and hence explore different regions of the policy space.

Original: Note that due to the speciﬁc nature of t-SNE , the policies are shown as Gaussians in a lower-dimensional embedding, while it is not necessarily the case in the true space.
Revised: Note that due to the speciﬁc nature of t-SNE , the policies are shown as Gaussians in a lower-dimensional embedding, while it is not necessarily the case in the true space.

Original: S TABILIZING A TTRACTION -R EPULSION WITH N ORMALIZING F LOW
Revised: S TABILIZING A TTRACTION -R EPULSION WITH N ORMALIZING F LOW

Original: In this section, we illustrate the consequence of the AR operators with a Gaussian policy (as in Hong et al.
Revised: In this section, we illustrate the consequence of the AR operators with a Gaussian policy (as in Hong et al.

Original: and our Normalizing ﬂow policy for Ant-v2 , Humanoid-v2 and
Revised: and our Normalizing ﬂow policy for Ant-v2 , Humanoid-v2 and

Original: HalfCheetah-v2 .
Revised: HalfCheetah-v2 .

Original: As shown in the ﬁgure below, AR with Gaussian policies yield worse results.
Revised: As shown in the ﬁgure below, AR with Gaussian policies yield worse results.

Original: One reason is that the KL term drives apart the mean and variance of the Gaussian policy which deteriorates the main objective of maximizing the reward.
Revised: One reason is that the KL term drives apart the mean and variance of the Gaussian policy which deteriorates the main objective of maximizing the reward.

Original: On the other side, our method applies the AR only on the NF layers allows enough exploration by deviating sufﬁciently from the main objective function.
Revised: On the other side, our method applies the AR only on the NF layers allows enough exploration by deviating sufﬁciently from the main objective function.

Original: C OMPARING ARAC AGAINST BASELINES ON M UJOCO TASKS
Revised: C OMPARING ARAC AGAINST BASELINES ON M UJOCO TASKS

Original: Figure 7 shows the performance of ARAC and baselines (CEM-TD3, CERL and ERL) over time steps.
Revised: Figure 7 shows the performance of ARAC and baselines (CEM-TD3, CERL and ERL) over time steps.

Original: Learning curves are averaged over 5 random seeds and displayed with one standard deviation.
Revised: Learning curves are averaged over 5 random seeds and displayed with one standard deviation.

Original: Evaluation is done every 10 , 000 environment steps using 10 rollouts per agent.
Revised: Evaluation is done every 10 , 000 environment steps using 10 rollouts per agent.

Original: Overall, ARAC has reasonable performance on all tasks (no ﬂat curves) and demonstrates high performance, especially in humanoid tasks.
Revised: Overall, ARAC has reasonable performance on all tasks (no ﬂat curves) and demonstrates high performance, especially in humanoid tasks.

Original: B ENEFITS OF POPULATION - BASED STRATEGIES : ARAC AGAINST SINGLE AGENTS
Revised: B ENEFITS OF POPULATION - BASED STRATEGIES : ARAC AGAINST SINGLE AGENTS

Original: In this section, we highlight the beneﬁts of the proposed population-based strategy by comparing with single agents.
Revised: In this section, we highlight the beneﬁts of the proposed population-based strategy by comparing with single agents.

Original: Figure 8 shows the performance of ARAC against a single SAC agent (with and without normalizing ﬂows).
Revised: Figure 3 shows the performance of ARAC against a single SAC agent (with and without normalizing ﬂows).

Original: Learning curves are averaged over 5 random seeds and displayed with one standard deviation.
Revised: Learning curves are averaged over 5 random seeds and displayed with one standard deviation.

Original: Evaluation is done every 10 , 000 environment steps using 10 rollouts per agent.
Revised: Evaluation is done every 10 , 000 environment steps using 10 rollouts per agent.

Original: We observe a high beneﬁcial impact on the convergence rate as well as on the performance.
Revised: We observe a high beneﬁcial impact on the convergence rate as well as on the performance.

Original: ARAC outperforms single agents in almost all tasks (except for HalfCheetah-v2 and
Revised: ARAC outperforms single agents in almost all tasks (except for HalfCheetah-v2 and

Original: Walker-v2 ) with large improvement.
Revised: Walker-v2 ) with large improvement.

Original: Note the high sample efﬁciency on humanoid environments ( Humanoid-v2 and Humanoid (rllab) ), where it shows faster convergence in addition to better performance.
Revised: Note the high sample efﬁciency on humanoid environments ( Humanoid-v2 and Humanoid (rllab) ), where it shows faster convergence in addition to better performance.

Original: Indeed, on Humanoid (rllab) a single SAC agent reaches the 4 k milestone after 4 M steps (Haarnoja et al., 2018) while ARAC achieves this performance in less than 2 M steps.
Revised: Indeed, on Humanoid (rllab) a single SAC agent reaches the 4 k milestone after 4 M steps (Haarnoja et al., 2018) while ARAC achieves this performance in less than 2 M steps.

Original: Also, in SparseHumanoid-v2 , due to its better coordinated exploration, ARAC could ﬁnd a good solution faster than SAC-NF.
Revised: Also, in SparseHumanoid-v2 , due to its better coordinated exploration, ARAC could ﬁnd a good solution faster than SAC-NF.

Original: O VERALL
Revised: O VERALL

Original: [Table]
Revised: [Table]

Original: CLEAR
Revised: ARAC

Original: TRPO
Revised: TRPO

Original: PPO
Revised: PPO

Original: Trust-PCL
Revised: Trust-PCL

Original: Plappert
Revised: Plappert

Original: [Table]
Revised: [Table]

Original: Touati
Revised: Touati

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Hong
Revised: Hong

Original: [Table]
Revised: [Table]

Original: HalfCheetah

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: N/
Revised: N/A

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: N/A
Revised: N/A

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] N/
Revised: 6 ,−

Original: [Table]
Revised: [Table]

Original: N/A
Revised: N/A

Original: N/A Humanoid
Revised: N/A N/A

Original: [Table]
Revised: [Table]

Original: N/A N/A
Revised: N/A N/A

Original: N/A 1 ,
Revised: N/A 1 ,

Original: HumanoidStandup

Original: [Table]
Revised: [Table]

Original: N/
Revised: N/A

Original: [Table]

Original: N/
Revised: N/A

Original: [Table]

Original: N/A N/A
Revised: N/A N/A

Original: N/A Humanoid
Revised: N/A Humanoid

Original: [Table]
Revised: [Table]

Revised: rllab) 4 ,

Original: N/A N/A
Revised: N/A N/A

Original: N/A N/A
Revised: N/A N/A

Original: E XPERIMENTAL PARAMETERS
Revised: E XPERIMENTAL PARAMETERS

Original: Table 4 provides the hyperparameters of ARAC used to obtain results in the MuJoCo domains.
Revised: Table 4 provides the hyperparameters of ARAC used to obtain results in the MuJoCo domains.

Original: The noise input for normalizing ﬂows in SAC policies (see Sec. 2.3) is sampled from N (0 , σ ) , where the variance σ is a function of the state (either ﬁxed at a given value or learned)
Revised: The noise input for normalizing ﬂows in SAC policies (see Sec. 2.3) is sampled from N (0 , σ ) , where the variance σ is a function of the state (either ﬁxed at a given value or learned)

Original: [Table]
Revised: [Table]

Original: I MPACT OF NUMBER OF FLOWS ON THE POLICY SHAPE
Revised: I MPACT OF NUMBER OF FLOWS ON THE POLICY SHAPE

Original: We used a single SAC agent with different radial ﬂows numbers and randomly initialized weights, starting with actions centered at (0 , 0) .
Revised: We used a single SAC agent with different radial ﬂows numbers and randomly initialized weights, starting with actions centered at (0 , 0) .

Original: All ﬂow parameters are (cid:96) 1 regularized with hyperparameter 2 .
Revised: All ﬂow parameters are (cid:96) 1 regularized with hyperparameter 2 .

Original: The agent is trained with the classical evidence lower bound (ELBO) objective augmented with the AR loss (Eq. 1), where the coefﬁcient of the repulsive policy π (cid:48) is given by β t = 10 t +1 .
Revised: The agent is trained with the classical evidence lower bound (ELBO) objective augmented with the AR loss (Eq. 1), where the coefﬁcient of the repulsive policy π (cid:48) is given by β t = 10 t +1 .

Original: Fig.shows how both the NF and learned variance Gaussian policies manage to recover the target policy.
Revised: Fig.shows how both the NF and learned variance Gaussian policies manage to recover the target policy.

Original: We see that NF takes advantage of its ﬂexible parametrization to adjust its density and can show asymmetric properties unlike the Gaussian distribution.
Revised: We see that NF takes advantage of its ﬂexible parametrization to adjust its density and can show asymmetric properties unlike the Gaussian distribution.

Original: This indeed can have advantage in some non symmetric environment where the Gaussian policy would be trapped into a suboptimal behavior.
Revised: This indeed can have advantage in some non symmetric environment where the Gaussian policy would be trapped into a suboptimal behavior.

Original: Finally, increasing the number of ﬂows (from bottom to top) can lead to more complex policy’s shape.
Revised: Finally, increasing the number of ﬂows (from bottom to top) can lead to more complex policy’s shape.

Revised: 6. V ARIANCE OF FITNESS IN THE ARCHIVE

Revised: Due to the high computation time for behavioral-diversity baselines such as DIYAN, we propose to use the agent’s ﬁtness (i.e. undiscounted returns) as a candidate to repulse/attract from.

Revised: Figure ??

Original:  18: for do 19: if policy π m is in top
Revised: shows the variance of the archive across three MuJoCo domains: Ant, Humanoid and HumanoidStandup. As training progresses, the clustering approach allows to maintain a high variance in the archive, preventing mode collapse to a single, "average" ﬁtness. 18: for do 19: if policy π m is in top

Original: K
Revised: K

Original: Sample policies uniformly
Revised: Sample policies uniformly

Original: G 21: mini
Revised: G 21: mini

Original: 22: else 23:
Revised: 22: else 23:

Original: 24: end
Revised: 24: end

Original: 25: end
Revised: 25: end

Original: 26: for do 27:
Revised: 26: for do 27:

Original: 28: end
Revised: 28: end

Original: Rank
Revised: Rank

Original: 30: update
Revised: 30: update

Original: 31: end
Revised: 31: end

Original: Collect samples
Revised: Collect samples

Original: Update critic
Revised: Update critic

Original: Update actors
Revised: Update actors

Original: Evaluate actors
Revised: Evaluate actors

Original: C OMPLEMENTARY PSEUDO - CODE FOR ARAC
Revised: C OMPLEMENTARY PSEUDO - CODE FOR ARAC

Original: Algorithms 2 and 3 respectively provide the pseudo-code of functions rollout and update_archive used in Algorithm 1.for episode
Revised: Algorithms 2 and 3 respectively provide the pseudo-code of functions rollout and update_archive used in Algorithm 1.

Original: , E do s ← Initial state s 0 from the environment for step t
Revised: episode do s ← Initial state s 0 from the environment for step t

Original: termination
Revised: termination

Original: do if with noise then
Revised: do if with noise then

Original: Sample noise z else
Revised: Sample noise z else

Original: Set z ←end if a t ∼ π ( .
Revised: Set z ←end if a t ∼ π ( .

Original: | s t , z )
Revised: | s t , z )

Original: Observe s t +1 ∼ P ( ·| s t , a t ) and obtain reward r t
Revised: Observe s t +1 ∼ P ( ·| s t , a t ) and obtain reward r t

Original: Fitness ← Fitness + r t
Revised: Fitness ← Fitness + r t

Original: Store transition ( s t , a t , r t , s t +1 ) in B end for end for
Revised: Store transition ( s t , a t , r t , s t +1 ) in B end for end for

Original: Fitness ← Fitness /E return Average ﬁtness per episode and number of steps performed
Revised: Fitness ← Fitness /E return Average ﬁtness per episode and number of steps performed

Original: Input: archive G ; population of size M maximal archive capacity G .
Revised: Input: archive G ; population of size M maximal archive capacity G .

Original: if |G| < G then
Revised: if |G| < G then

Original: Add all agents of current population to G else c 1 , c 2 ← 2 -means ( ﬁtness of individuals in G ) for agent m = 1 , . .
Revised: Add all agents of current population to G else c 1 , c 2 ← 2 -means ( ﬁtness of individuals in G ) for agent m = 1 , . .

Original: , M do
Revised: , M do

Original: Assign agent m to closest cluster c ∈ { c 1 , c 2 } based on its ﬁtness
Revised: Assign agent m to closest cluster c ∈ { c 1 , c 2 } based on its ﬁtness

Original: Sample an archived agent j ∼ Uniform ( c )
Revised: Sample an archived agent j ∼ Uniform ( c )

Original: Replace archived individual j by m end for end if return Updated archive G
Revised: Replace archived individual j by m end for end if return Updated archive G