D IVERSITY P OLICY G RADIENT FOR S AMPLE E FFICIENT Q UALITY -D IVERSITY O PTIMIZATION
A BSTRACT
A fascinating aspect of nature lies in its ability to produce a large and diverse collection of high-performing organisms in an open-ended way.
By contrast, most
AI algorithms seek convergence and focus on finding a single efficient solution to a given problem.
Aiming for diversity through divergent search in addition to performance is a convenient way to deal with the exploration-exploitation tradeoff that plays a central role in learning.
It also allows for increased robustness when the returned collection contains several working solutions to the considered problem, making it well-suited for real applications such as robotics.
QualityDiversity (QD) methods are evolutionary algorithms designed for this purpose.
This paper proposes a novel algorithm, QD - PG , which combines the strength of
Policy Gradient algorithms and Quality Diversity approaches to produce a collection of diverse and high-performing neural policies in continuous control environments.
The main contribution of this work is the introduction of a Diversity
Policy Gradient (DPG) that drives policies towards more diversity in a sampleefficient and open-ended manner.
Specifically, QD - PG selects neural controllers from a MAP - ELITES grid and uses two gradient-based mutation operators to improve both quality and diversity.
Our results demonstrate that QD - PG is significantly more sample-efficient than its evolutionary competitors.
is rewarded for running forward as fast as possible, following the reward signal without further exploration leads the agent into a trap.
Our method QD - PG allows deeper exploration, necessary to solve such deceptive control problems.
QD - PG builds upon the MAPELITES and leverages the data-efficiency of reinforcement learning by using a quality critic and a diversity critic to derive policy gradient based mutations in order to seek for high performance and new behaviors.
I NTRODUCTION
Natural evolution has the fascinating ability to produce diverse organisms that are all well adapted to their respective niche.
It is a creative and open-ended (OE) process that continuously produces new solutions without explicitely optimizing for a fitness objective.
Such open-endedness could be the key to solving problems where the required stepping stones cannot be anticipated and planned, emphasizing the need for a serendipitous search process.
Inspired by this ability to produce a tremendous diversity of living systems, Quality-Diversity (QD) algorithms aim at searching for a collection of both diverse and high-performing solutions (Pugh et al., 2016; Cully & Demiris, 2017).
While classic optimization algorithms focus on finding a single efficient solution, QD algorithms aim to cover the range of possible solution types and to return the best solution for each type.
By seeking diversity, QD algorithms can be viewed as partially open-ended tools that balance openendedness with practical use (Mouret, 2020).
Diversity search is the core component that allows QD algorithms to generate large collections of diverse solutions in an open-ended manner.
By continuously encouraging the emergence of novel solutions never seen before, the search for diversity is only limited by the very nature of the considered problem (or environment).
QD methods have been useful in autonomous skill discovery (Cully, 2019), demonstrating their ability to create solutions of increasing complexity from the void, but also as a part of broader open-ended systems (Wang et al., 2019).
Building large and efficient controllers that work with continuous actions has been a long-standing goal in Artificial Intelligence and particularly in robotics.
Deep reinforcement learning (RL), and especially Policy Gradient (PG) methods have proven efficient at training such large controllers (Schulman et al., 2017;
Lillicrap et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018).
One of the keys to this success lies in the fact that PG methods exploit the structure of the objective function when the problem can be formalized as a Markov Decision Process (MDP).
Moreover, they also exploit the analytical structure of the controller when known, which allows the sample complexity of these methods to be independent of the parameter space dimensionality (Vemula et al., 2019).
Although exploration is very important to reach optimal policies, PG methods usually rely on simple exploration mechanisms, like adding Gaussian noise (Fujimoto et al., 2018) or maximizing entropy (Haarnoja et al., 2018) to explore the action space, which happens to be insufficient in hard exploration tasks where the reward signal is sparse or deceptive (Colas et al., 2018; Nasiriany et al., 2019).
These techniques only focus on building high-performing solutions and do not explicitly encourage diversity within the set of produced solutions.
In this regard, they fail when confronted with hard exploration problems.
Contributions.
In this work, we introduce the idea of a diversity policy gradient ( D - PG ) that drives the search process towards unseen and novel solutions in an open-ended way.
We show that the
D - PG can be used in combination with the standard policy gradient, dubbed quality policy gradient ( Q - PG ), to produce high-performing and diverse solutions.
Our algorithm, called QD - PG , builds on
MAP - ELITES and PGA - ME (Nilsson & Cully, 2021) replacing random diversity search by D - PG , and demonstrates remarkable sample efficiency brought by off-policy PG methods.
We compare QD - PG to state-of-the-art policy gradient methods algorithms (including SAC , TD 3, RND , AGAC , DIAYN and
PGA - ME ), and to several evolutionary methods known as Evolution Strategies (ESs) augmented with a diversity objective (namely the NS - ES family (Conti et al., 2018) and the ME - ES algorithm (Colas et al., 2020))
on a set of challenging continuous control tasks with deceptive reward signals.
B ACKGROUND 2. P ROBLEM STATEMENT
We consider an MDP ( S , A , R , T , γ ) where S is the state space,
A the action space, R : S×A
→ R the reward function, T : S ×
A → S the dynamics transition function and γ a discount factor.
We assume that both S and A are continuous and consider a controller, or policy, π θ : S → A , a neural network parameterized by θ ∈ Θ , which is called a solution to the problem.
The fitness F : Θ → R of a solution measures its performance, defined as the expectation over the sum of rewards obtained by controller π θ .
A solution with high fitness is said to be high-performing .
We introduce a behavior descriptor (BD) space B , a behavior descriptor extraction function ξ : Θ →
B , and define a distance metric || .
|| B over B .
The diversity of a set of K solutions { θ k } k =1 ,...,K is defined as d : Θ K → R + :
[Equation]
meaning that a set of solutions is diverse if the solutions are distant with respect to each other in the sense of || .
|| B .
Following the objective of the family of Quality Diversity algorithms, we are trying to evolve a population of diverse and high-performing solutions.
2. T HE MAP-E LITES ALGORITHM
MAP - ELITES (Mouret & Clune, 2015) is a simple yet state-of-the-art QD algorithm that has been successfully applied to a wide range of challenging problems such as robot damage recovery (Cully et al., 2015), molecular robotic control (Cazenille et al., 2019) and game design (Alvarez et al., 2019).
In MAP - ELITES , the behavior descriptor space B is discretized into a grid of cells, also called niches, with the aim of filling each cell with a high-performing solution.
A variant, called CVT MAPELITES uses Centroidal Voronoi Tesselations (Vassiliades et al., 2016) to initially divide the grid into the desired number of cells.
The algorithm starts with an empty grid and an initial random set of
K solutions that are evaluated and added to the grid by following simple insertion rules.
If the cell corresponding to the behavior descriptors of a solution is empty, then the solution is added to this cell.
If there is already a solution in the cell, the new solution replaces it only if it has greater fitness.
At each iteration, P existing solutions are sampled uniformly from the grid and randomly mutated to create P new solutions, encouraging the emergence of serendipity throughout the search process: a new high-performing solution is often created from a distant stepping stone in the behavior space.
These new solutions are then evaluated and added to the grid following the same insertion rules.
This cycle is repeated until convergence or for a given budget of iterations.
2. R EINFORCEMENT L EARNING AND TD
Deep Reinforcement learning (DRL) is a paradigm to learn high-performing policies implemented by neural networks in MDPs.
In this work, we focus on a class of policy search methods called policy gradient methods.
In opposition to standard evolutionary methods that rely on random updates, policy gradient methods exploit the structure of the MDP under the form of the Bellman equations to compute efficient performance-driven updates to improve the policy.
Among many algorithms in the reinforcement learning community, TD 3 (Fujimoto et al., 2018) shows state-of-the-art performance to train controllers in environments with continuous action space and large state space.
TD 3 relies on the deterministic policy gradient update Silver et al.
(2014) to train deterministic policies, as defined in Section 2.1.
In most policy gradients methods, a critic
Q π : S ×
A → R implemented by a neural network is introduced.
The critic evaluates the expected fitness of policy π when performing a rollout starting from a state-action pair ( s, a ) .
The policy is updated to take the actions that will maximise the critic’s value estimation in each state.
Both the actor and the critic have an associated target network and use delayed policy updates to stabilize the training.
More details can be found in the supplementary material in Section B.
K EY P RINCIPLE : D IVERSITY P OLICY G RADIENT 3. G ENERAL PRINCIPLE
In this work, we introduce the Diversity Policy Gradient, which aims at using time-step level information in order to mutate controllers towards diversity.
The DPG acts in addition to the structural pressure for diversity induced by the MAP - ELITES selection and insertion mechanisms.
It can be viewed as a divergent search (Lehman & Miikkulainen, 2015), open-ended oriented component, which continuously drives the search process towards creating solutions that are novel and different from the so far created ones.
We are trying to characterize the behavior of policies thanks to the states they visit and hence construct a diversity reward at the time-step-level that is correlated with the diversity at the episode-level.
This reward is computed based on distance of a state compared to the nearest states visited by other controllers in the MAP - ELITES grid.
Aiming at maximizing the accumulated diversity rewards will induce an increase of the diversity of the population we are evolving.
The following section provides a mathematical motivation for this approach.
3. M ATHEMATICAL FORMULATION
In this section, we motivate and introduce formally the D - PG computations.
Let us assume that we have a MAP - ELITES grid containing K solutions ( θ 1 , . .
, θ K ) and that we sampled θ 1 from the grid to evolve it.
We want to update θ 1 in such a way that the population’s diversity as defined in Equation equation 1 will increase.
For this purpose, we aim to compute the gradient of the diversity with respect to θ 1 and update θ 1 in its direction using standard gradient ascent techniques.
Proposition 1 The gradient of diversity with respect to θ 1 can be written as
[Equation]
and θ 2 is θ 1 closest neighbour and ( θ j ) with j = 3 , . .
, J are the elements in the population for which θ 1 is the nearest neighbour.
Proof in Appendix F.
We call n the novelty of θ 1 with respect to its nearest elements.
This proposition means that we can increase the diversity of the population by increasing the novelty of θ 1 with respect to the solutions for which it is the nearest neighbor.
Under this form, the diversity gradient cannot benefit from the variance reduction methods in the
RL literature to efficiently compute policy gradients Sutton et al.
To this end, we need to express it as a gradient over the expectation of a sum of scalar quantities obtained by policy π θ 1 at each step when interacting with the environment.
For this purpose, we introduce a novel space D , dubbed state descriptor space and a state descriptor extraction function ψ : S → D .
We assume D and B have the same dimension.
The notion of state descriptor will be used in the following to link diversity at the time step level to the diversity at the trajectory level.
In this context, if the following compatibility equation is satisfied:
[Equation]
then the diversity policy gradient can be computed as:
[Equation]
See the Appendix F for more details about the motivation behind this assumption.
The obtained expression corresponds to the classical policy gradient setting where γ = 1 and where the corresponding reward signal, here dubbed diversity reward, is computed as r Dt = n ( s t , ( θ j ) 2 ≤ j ≤ J ) .
Therefore, this gradient can be computed using any PG estimation technique replacing the environment reward by the diversity reward r D t .
Equation equation 3 enforces a relation between B and D and between extraction functions ψ and ξ .
In practice, it may be hard to define the behavior descriptor and state descriptor of a solution that satisfy this relation while being meaningful to the problem at hand and tractable.
Nevertheless, a strict equality is not necessary: a positive correlation between the two hand-size is sufficient for diversity seeking.
R ELATED W ORK
Simultaneously maximizing diversity and performance is the central goal of QD methods (Pugh et al., 2016; Cully & Demiris, 2017).
Among the various possible combinations offered by the QD framework (Cully & Demiris, 2017), Novelty Search with Local Competition ( NSLC ) (Lehman &
Stanley, 2011b) and MAP - ELITES (Mouret & Clune, 2015) are the two most popular algorithms.
NSLC builds on the Novelty Search (NS) algorithm (Lehman & Stanley, 2011a) and maintains an unstructured archive of solutions selected for their local performance while MAP - ELITES uniformly samples individuals from a structured grid that discretizes the BD space.
Although very efficient in small parameter spaces, those methods struggle to scale to bigger spaces which limits their application to neuro-evolution.
Gradients in QD.
Algorithms such as NSR - ES and NSRA - ES have been applied to challenging continuous control environments (Conti et al., 2018).
But, as outlined by Colas et al. (2020), they still suffer from poor sample efficiency and the diversity and environment reward functions could be mixed in a more efficient way.
ME - ES (Colas et al., 2020) went one step further in that direction, achieving a better mix of quality and diversity seeking through the use of a MAP - ELITES grid and two specialized ES populations.
Using these methods was shown to be critically more successful than population-based GA algorithms (Salimans et al., 2017), but they rely on heavy computation resources.
Differentiable QD (Fontaine & Nikolaidis, 2021) improves data-efficiency in QD with an efficient search in the descriptor space but does not tackle neuro-evolution and is limited to problems where the fitness and behavior descriptor functions are differentiable.
Exploration and diversity in RL.
RL methods generally seek for diversity either in the state space or in the action space.
This is the case of algorithms maintaining a population of RL agents for exploration without an explicit diversity criterion (Jaderberg et al., 2017) or algorithms explicitly looking for diversity but in the action space rather than in the state space like ARAC (Doan et al., 2019), AGAC (Flet-Berliac et al.), P 3 S - TD 3 (Jung et al., 2020) and D v D (Parker-Holder et al., 2020).
An exception is Curiosity Search Stanton & Clune (2016) which defines a notion of intra-life novelty that is similar to our state novelty defined in Section 3.
However, their novelty relies on skills rather than states.
Our work is also related to algorithms using RL mechanisms to search for diversity only like DIAYN (Eysenbach et al., 2018) and others (Pong et al., 2019; Lee et al., 2019; Islam et al., 2019).
These methods have proven useful in sparse reward situations, but they are inherently limited when the reward signal can orient exploration, as they ignore it.
Other works sequentially combine diversity seeking and RL (Colas et al., 2018; Ecoffet et al., 2019).
Mixing policy gradient and evolution.
The fruitful synergy between evolutive and RL methods has been explored in many recent methods, notably ERL (Khadka & Tumer, 2018), CERL (Khadka et al.), CEM - RL (Pourchot & Sigaud, 2018) and PGA - ME (Nilsson & Cully, 2021).
ERL and CEM - RL mix Evolution Strategies and RL to evolve a population of agents to maximize quality but ignores the diversity of the population.
Policy Gradient Assisted MAP-Elites ( PGA - ME ) successfully combines
QD and RL.
This algorithm scales MAP - ELITES to neuroevolution by evolving half of its offsprings with a quality policy gradient update instead of using a genetic mutation alone.
Nevertheless, the seek for diversity is only explicitly done with the genetic mutation.
To the best of our knowledge, QD - PG is the first algorithm optimizing both diversity and performance in the solution space and in the state space, using a sample-efficient policy gradient computation method for the latter.
M ETHODS
Our full algorithm is called QD - PG , its pseudo code and general architecture are given in Apendix A.
QD - PG is an iterative algorithm based on MAP - ELITES that replaces random mutations with policy gradient updates.
As we consider a continuous action space and want to improve sample efficiency by using an off-policy policy gradient method, we rely on TD 3.
QD - PG maintains three permanent structures.
On the QD side, a CVT MAP - ELITES grid stores the most novel and performing solutions.
On the RL side, a replay buffer contains all transitions collected when evaluating solutions and an archive A stores all state descriptors obtained so far.
QD - PG starts with an initial population of random solutions, evaluates them and inserts them into the MAPELITES grid.
At each iteration, solutions are sampled from the grid, copied, and updated.
The updated solutions are then evaluated through one rollout in the environment and inserted into the grid according to the usual insertion rules.
Transitions collected during evaluation are stored in the replay buffer, and state descriptors are stored in the archive A .
Note that these state descriptors are first filtered to avoid insertion in the archive of multiple state descriptors that are too close to each other.
During the update step, half the population is updated with Q - PG ascent and the other half with
D - PG ascent.
The choice of whether an agent is updated for quality or diversity is random, meaning that it can be updated for quality and later for diversity if selected again.
To justify this design, we show in Section 7 that updating consecutively for quality and diversity outperforms updating based on joint criteria.
Both gradients are computed from batches of transitions sampled from the replay buffer.
The Q - PG is computed from the usual environment rewards (similar to TD 3) whereas for
D - PG , we get ”fresh” novelty rewards as the sum of the distances between state s t descriptor and its
J nearest neighbors in the archive A .
Diversity rewards must be recomputed at each update because A changes during training.
Following Equation equation 3, diversity rewards should be computed as the sum of the distances between the descriptor of s t and the descriptors of all the states visited by a list of J solutions.
In practice, we consider the J nearest neighbors of s t .
This choice simplifies the algorithm, is faster to compute and works well in practice.
TD 3 relies on a parameterized critic to reduce the variance of its policy gradient estimate.
In QDPG , we maintain two parameterized critics Q Dw and Q Qv , respectively dubbed diversity and quality critics.
Every time a policy gradient is computed, QD - PG also updates the corresponding critic.
The critics are hence trained with all agents in the population instead of a specialised agent.
This helps avoiding local minima in exploration environments where the specialised actor could get stuck and hence mislead the values learned by the critic.
However, having critics trained with multiple agents can destabilize the process, which is why we avoid using QD - PG with big grids.
In our benchmarks, our grids usually contain 3 times less cells than PGA - ME .
As in TD 3, we use pairs of critics and target critics to fight the overestimation bias.
We share the critic parameters among the population as in CEM Pourchot & Sigaud (2018).
Reasons for doing so come from the fact that diversity is not stationary, as it depends on the current population.
If each agent had its own diversity critic, since an agent may not be selected for a large number of generations before being selected again, its critic would convey an outdated picture of the evolving diversity.
We tried this solution, and it failed.
A side benefit of critic sharing is that both critics become accurate faster as they combine experience from all agents.
Additional details on QD - PG implementation are available in Appendix C.
E XPERIMENTS
In this section, we intend to answer the following questions: 1.
Can QD - PG produce collections of diverse and high-performing neural policies and what are the advantages to do so?
Is QD - PG more sample efficient than its evolutionary competitors?
How difficult are the considered benchmarks for classical policy gradients methods?
6. E NVIRONMENTS
We assess QD - PG capabilities in continuous control environments that exhibit high dimensional observation and action spaces as well as deceptive rewards.
The size of the state/action space makes exploration difficult for Genetic Algorithms and Evolution Strategies.
Deceptive rewards creates exploration difficulties which is particularly challenging for classical RL methods.
We consider three
OpenAI Gym environments based on the M U J O C O physics engine that all exhibit strong deceptive rewards (illustrated in the Appendix in Figure 5), POINT - MAZE , ANT - MAZE and ANT - TRAP .
Such environments have also been widely used in previous works (Parker-Holder et al., 2020; Colas et al., 2020; Frans et al., 2018; Shi et al., 2020).
In the POINT - MAZE environment, an agent represented as a green sphere must find the exit of the maze depicted in Figure 2a, represented as a red sphere.
The reward is expressed as the negative
Euclidean distance between the center of gravity of the agent and the exit center.
The ANT - MAZE environment is modified from OpenAI Gym ANT - V 2 (Brockman et al., 2016) and also used in (Colas et al., 2020; Frans et al., 2018).
In ANT - MAZE , a four-legged ant starts in the bottom left of a maze and has to reach a goal zone located in the lower right part of it (green area in Figure 2a).
As in
POINT - MAZE , the reward is expressed as the negative distance the goal zone.
Finally, the ANT - TRAP environment also derives from ANT - V 2 and is inspired from (Colas et al., 2020; Parker-Holder et al., 2020).
In ANT - TRAP , the four-legged ant initially appears in front of a trap and must bypass it to run as fast as possible in the forward direction (see Figure 2a), as in ANTV 2, the reward is computed as the ant velocity on the x-axis.
Additional details on environments can be found in Appendix D
R ESULTS
Can QD-PG produce collections of neural policies and what are the advantages to do so?
Table 1a presents QD - PG performances.
In all environments, our algorithm manages to find working solutions that avoid local minima and reach the overall objective.
In addition to its exploration capabilities, QD - PG generates collections of high performing solutions in a single run.
During the
ANT - TRAP experiment, the final collection of solutions returned by QD - PG contained, among others, 5 solutions that were within a 10% performance margin from the best one.
even after setting new randomly located goals, the MAP - ELITES grid still contains solutions that are suited for the new objectives
Generating a collection of diverse solutions comes with the benefit of having a repertoire of diverse solutions that can be used as alternatives when the MDP changes (Cully et al., 2015).
We show that QD - PG is more robust than conventional policy gradient methods by changing the reward signal of the ANT - MAZE environment.
We replace the original goal in the bottom right part of the maze (see Figure 3) with a new randomly located goal in the maze.
Instead of running QD - PG to optimize for this new objective, we run a Bayesian optimization process to quickly find a good solution among the ones already stored in the grid.
With a budget of only 20 solutions to be tested during the Bayesian optimization process, we are able to quickly recover a good solution for the new objective.
We repeat this experiment 100 times, each time with a different random goal, and obtain an average performance of − 10 with a standard deviation of 9 .
In other words, 20 interaction episodes (corresponding to 60 .
000 time steps) suffice for the adaptation process to find a solution that performs well for the new objective without the need to re-train agents.
More detailed results can be found in Appendix E. [Table] [Table]
Is QD - PG more sample efficient than its evolutionary competitors?
Table 1b compares QD - PG to Deep Neuroevolution algorithms with a diversity seeking component ( ME - ES , NSR - ES , NSRAES , CEM - RL ) in terms of sample efficiency.
QD - PG runs on 10 CPU cores for 2 days while its competitors used 1000 CPU cores for the same duration.
We see three reasons for the improved sample efficiency of QD - PG : 1) QD - PG leverages a replay buffer and can re-use each sample several times. 2) QD - PG leverages novelty at the state level and can exploit all collected transitions to maximize quality and diversity.
For instance, in ANT - MAZE , a trajectory brings 3000 samples to QD - PG while standard QD methods would consider it a unique sample.
3) PG exploits the analytical gradient between the neural network weights and the resulting policy action distribution and estimates only the impact of the distribution on the return.
By contrast, standard QD methods directly estimate the impact on the return of randomly modifying the weights.
How challenging are the considered benchmarks for policy gradients methods?
Table 1a compares QD - PG to state-of-the-art policy gradient algorithms and validates that classical policy gradient ( TD 3 and SAC ) methods fail to find optimal solutions in deceptive environments.
Besides, despite its exploration mechanism based on CEM , CEM - RL also quickly converges to local optima in all benchmarks, confirming the need for a dedicated diversity seeking component.
RND , which adds an exploration bonus used as an intrinsic reward, also demonstrates performances inferior to QD - PG in all environments but manages to solve POINT - MAZE .
In ANT - TRAP , RND extensively explores the BD space but fails to obtain high returns.
Although maintaining diversity into a population of reinforcement learning agents, AGAC and P 3 S - TD 3 are not able to explore enough the environments to solve them, showing some limits of exploration through the action space.
DIAYN is able to explore in POINT - MAZE but does not fully explore the environment and hence does not reach the goal.
DIAYN ensures that its skills explore states different enough to be discriminated, but once they are, there is no incentive to further explore.
Moreover, DIAYN in ANT - TRAP shows a limit of unsupervised learning methods: In ANT - TRAP the behavior descriptor chosen is not perfectly aligned with the performance measured, as a matter of fact, the objective is not to reach a point but to achieve the best forward speed with some control costs.
But as those rewards are completely ignored by DIAYN , the produced controllers have very low fitness.
C ONCLUSION
We introduced the diversity policy gradient to efficiently implement divergent search for diversity seeking.
Based on this component we proposed a novel algorithm, QD - PG , that builds upon MAPElites algorithm and significantly improve its sample efficiency.
We believe that in future work, the DPG could be part of and enhance open-ended algorithms that seek diversity in a more sample efficient manner.
A QD-PG C ODE AND ARCHITECTURE
The full algorithm can be found on pseudocode 1 and its architecture in Figure 4.
[Figure]
B D ETAILED E QUATIONS OF TD
The Twin Delayed Deep Deterministic ( TD 3) agent Fujimoto et al.
(2018) builds upon the Deep
Deterministic Policy Gradient ( DDPG ) agent Lillicrap et al.
It trains a deterministic actor π ϕ : S
→ A directly mapping observations to continuous actions and a critic Q θ : S ×
A → R which takes a state s and an action a and estimates the average return from selecting action a in state s and then following policy π ϕ .
As DDPG , TD 3 alternates between policy evaluation and policy improvement so as to maximise the average discounted return.
In DDPG , the critic is updated to minimize a temporal difference error during the policy evaluation step which induces an overestimation bias.
TD 3 corrects for this bias by introducing two critics Q θ 1 and Q θ 2 .
TD 3 plays one step in the environment using its deterministic policy and then stores the observed transition ( s t , a t , r t , s t +1 ) into a replay buffer M .
Then, it samples a batch of transitions from M and updates the critic networks.
Half the time it also samples another batch of transitions to update the actor network.
Both critics are updated so as to minimize a loss function which is expressed as a mean squared error between their predictions and a target:
[Equation]
where the common target y t is computed as:
[Equation]
where ϵ ∼ N (0 , I ) .
Given: N, max steps, gradient steps ratio, BD extraction function ξ , state descriptor extraction function ψ Initialize: MAP-Elites grid M , Replay Buffer R , N actors { π θ i } i = { 1 ,...,N } , 2 critics Q Dw and Q Qw , state descriptors archive A total steps, actor steps = 0 , 0 //
Step counters //
Parallel evaluation of the j ← 1 to N do
Play one episode with actor π θ j and store all transitions in R
Get episode length T , discounted return R and state descriptors { ψ ( s 1 ) , . .
. , ψ ( s T ) } Store state descriptors { ψ ( s 1 ) , . . .
, ψ ( s T ) } in A
Compute ξ ( θ j ) and add the tuple ( R , ξ ( θ j ) , θ j ) in the MAP-Elites grid M actor steps ← actor steps + T end //
Main loop while total steps < max steps do // Select new generation
Get N actors π θ i , i ∈ { 1 , . .
. , N } from M gradient steps = int ( actor steps × gradient steps ratio ) actor steps =// Perform in parallel population update and evaluation for j ← 1 to N do // Update the population for i ← 1 to gradient steps do
Sample batch of ( s t , a t , r t , s t +1 , ψ ( s t )) from R //
First half is updated to maximise diversity if j ≤ N // 2 then
Compute novelty reward as r Dt from ψ ( s t ) and A
Update π θ j for diversity
Compute the novelty critic gradient locally Average novelty critic gradients between threads
Update novelty critic Q Dw
end //
Second half is updated to maximise quality else
Update π θ j for quality
Compute the quality critic gradient locally Average quality critic gradients between threads
Update quality critic Q Qv end end //
Evaluate the updated actors
Play one episode with actor π θ j and store all transitions in R
Get episode length T , discounted return R and state descriptors { ψ ( s 1 ) , . .
. , ψ ( s T ) } Store state descriptors { ψ ( s 1 ) , . . .
, ψ ( s T ) } in A
Compute ξ ( θ j ) and add the tuple ( R , ξ ( θ j ) , θ j ) in the MAP-Elites grid M actor steps ← actor steps + T end total steps ← total steps + actor steps // Update total time steps end
The Q-value estimation used to compute target y t is taken as minimum between both critic predictions thus reducing the overestimation bias.
TD 3 also adds a small perturbation ϵ to the action π ϕ ( s t +1 ) so as to smooth the value estimate by bootstrapping similar state-action value estimates.
Every two critics updates, the actor π ϕ is updated using the deterministic policy gradient also used in DDPG Silver et al.
For a state s , DDPG updates the actor so as to maximise the critic estimation for this state s and the action a = π ϕ ( s ) selected by the actor.
As there are two critics in TD 3, the authors suggest to take the first critic as an arbitrary choice.
The actor is updated by minimizing the following loss function:
[Equation]
Policy evaluation and policy improvement steps are repeated until convergence.
TD 3 demonstrates state of the art performance on several M U J O C
O benchmarks.
C QD-PG D ETAILS C. C OMPUTATIONAL DETAILS
We consider populations of N = 4 actors for the POINT - MAZE environment and N = 10 actors for
ANT - MAZE and ANT - TRAP .
We use 1 CPU thread per actor and parallelization is implemented with the Message Passing Interface (MPI) library.
Our experiments are run on a standard computer with 10 CPU cores and 100 GB of RAM, although the maximum RAM consumption per experiment at any time never exceeds 10GB due to an efficient and centralized management of the MAP - ELITES grid which stores all solutions.
An experiment on POINT - MAZE typically takes between 2 andhours while an experiment on ANT - MAZE or ANT - TRAP takes about 2 days.
Note that these durations can vary significantly depending on the type of CPU used.
We did not use any GPU .
Computational costs of QD - PG mainly come from backpropagation during the update of each agent, and to the interaction between agents and the environment.
These costs scale linearly with the population size but, as many other population-based methods, the structure of QD - PG lends itself very well to parallelization.
We leverage this property and parallelize our implementation to assign one agent per CPU thread.
Memory consumption also scales linearly with the number of agents.
To reduce this consumption, we centralize the MAP - ELITES grid on a master worker and distribute data among workers when needed.
With these implementation choices, QD - PG only needs a very accessible computational budget for all experiments.
C. MAP-E LITES I MPLEMENTATION DETAILS
QD - PG uses a MAP - ELITES grid as archive of solutions.
We assume that the BD space is bounded and can be discretized into an Cartesian grid.
We discretize each dimension into m meshes, see
Table 2 for the value of m depending on the environment.
Hence, the number of cells in the MAPELITES grid equals m times the number of dimensions of the BD space.
When a new solution θ is obtained after the mutation phase, we look for the cell corresponding to its BD, ξ ( θ ) .
If the cell is empty, the solution is added, otherwise the new solution replaces the solution already contained in the cell if its score F ( θ ) is better than the score of the already contained solution.
During selection, we sample solutions uniformly from the MAP - ELITES grid.
C. D IVERSITY REWARD COMPUTATION
QD - PG optimizes solutions for quality but also for diversity at the state level.
The diversity policy gradient updates the solutions so as to encourage them to visit states with novel state descriptors.
The novelty of a state descriptor ψ ( s t ) is expressed through a diversity reward r
Dt .
In practice, we maintain a FIFO archive
A of the state descriptors encountered so far.
When a transition ( s t , a t , r t , s t +1 , ψ ( s t )) is stored in the replay buffer, we also add ψ ( s t ) to A .
We only add a state descriptor in A if its mean Euclidean distance to its K nearest neighbors is greater than an acceptance threshold.
This filtering step enables to keep the archive size reasonable and to facilitate the computation of the K nearest neighbors.
The values of K and of the threshold are given in Table 2.
When a batch of transitions is collected during the update phase, we recompute fresh diversity rewards r
Dt as the mean Euclidean distance between the sampled state descriptors ψ ( s t ) and their K nearest neighbors in A .
These diversity rewards are used instead of standard rewards in sampled transitions ( s t , a t , r Dt , s t +1 , ψ ( s t )) to compute the diversity policy gradient.
C. QD-PG H YPER - PARAMETERS
Table 2 summarizes hyper-parameters used in experiments.
Most of these hyper-parameter values are taken from TD 3.
[Table]
D E NVIRONMENTS D. E NVIRONMENTS DETAILS
In POINT - MAZE an observation contains the agent position at time t
[Table]
and an action corresponds to position increments along the x and y axes.
The trajectory length cannot exceed 200 steps.
In ANT - MAZE the final performance is the maximum reward received during an episode.
The environment is considered solved when an agent obtains a score superior to − 10 .
An episode consists of 3000 time steps, this horizon is three times larger than in usual M U J O C O environments, making this environment particularly challenging for RL based methods (Vemula et al., 2019).
In ANT - TRAP , the trap consists of three walls forming a dead-end directly in front of the ant.
In this environment, the trajectory length cannot exceed 1000 steps.
In the three environments, the state descriptor is defined as the agent center of gravity position at time step t while the solution descriptor is the center of gravity position at the end of the trajectory.
As opposed to POINT - MAZE and ANT - MAZE , where the objective is to reach the exit area, there is no unique way to solve ANTTRAP and we expect a QD algorithm to generate various effective solutions as depicted in Figure 1.
Furthermore, the behavior descriptor is not aligned with the fitness which makes it more difficult for
MAP - ELITES and also for pure novelty seeking approaches as simply exploring the BD space is not enough to find a performing solution.
D. E NVIRONMENTS ANALYSIS
In POINT - MAZE , the state and action spaces are two-dimensional.
By contrast, in ANT - MAZE and
ANT - TRAP , the dimensions of their observation spaces are respectively equal to 29 and 113 while the dimensions of their action spaces are both equal to 8, making these two environments much more challenging as they require larger controllers.
The ANT - TRAP environment also differs from mazes as it is open-ended, i.e., the space to be explored by the agent is unlimited, unlike mazes where this space is restricted by the walls.
In this case, a state descriptor corresponds to the ant position that is clipped to remain in a given range.
On the y -axis, this range is defined as three times the width of the trap.
On the x -axis, this range begins slightly behind the starting position of the ant and is large enough to let it accelerate along this axis.
Figure 8b depicts the BD space in ANT - TRAP .
In all environments, state descriptors ψ ( s t ) are defined as the agent’s position at time step t and behavior descriptors ξ ( θ ) are defined as the agent’s position at the end of a trajectory.
Therefore, we have B = D
= R 2 , ψ ( s t ) =
( x t , y t ) and ξ ( θ ) =
( x T , y T ) where T is the trajectory length.
We also take || .
|| B
and || .
|| D
as Euclidean distances.
This choice does not always satisfy Equation equationbut is convenient in practice and led to satisfactory results.
The peculiarity of ANT - TRAP lies in the fact that the reward is expressed as the forward velocity of the ant, thus making the descriptors not totally aligned with the task.
Figure 5 highlights the deceptive nature of the POINT - MAZE and the ANT - MAZE objective functions by depicting gradient fields in both environments.
Similarly, the reward is also deceptive in ANTTRAP .
[Figure]
E D ETAILED RESULTS
In this section, we provide performance charts corresponding to Table 1b and Table 1a, coverage maps highlighting the exploration capabilities of QD - PG , and detailed results of the fast adaptation experiment.
Table 3 summarizes the different components present in QD - PG , its ablations and all baselines.
E. P ERFORMANCE CHARTS
Figure 7 shows supplementary results, with an ablation study and a comparison to evolutionary competitors in ANT - MAZE .
In QD - PG , the current population of solutions is evaluated every 150.
time steps in ANT - MAZE and ANT - TRAP , and every 5000 time steps in POINT - MAZE .
At evaluation time, agents are set to be deterministic and stop exploring.
Figure 7 reports the performance obtained by the best agent in the population at a given time step.
[Table]
[Figure]
E. A BLATION S TUDY
In addition to Section we wanted to answer the following question: What is the usefulness of the different components of QD - PG ?
To answer this question we propose to investigate the following matters: Can we replace alternating quality and diversity updates by a single update that optimizes for the sum of both criteria?
Are quality (resp.
diversity) gradients updates alone enough to fill the MAP - ELITES grid?
Consequently, we consider the following ablations of QD - PG : QD - PG SUM computes a gradient to optimize the sum of the quality and diversity rewards, D - PG applies only diversity gradients to the solutions and Q - PG applies only quality gradients but both D - PG and
Q - PG still use QD selection.
The ablation study in Table 1a and Figure 7 shows that when maximising quality only, Q - PG fails due to the deceptive nature of the reward and when maximizing diversity only, D - PG sufficiently explores to solve POINT - MAZE but requires more steps and finds lower-performing solutions.
When optimizing simultaneously for quality and diversity, QD - PG SUM fails to learn in ANT - TRAP and manages to solve the task in ANT - MAZE but requires more samples than QD - PG .
We hypothesize that quality and diversity rewards may give rise to conflicting gradients.
For instance, at the beginning of training in
ANT - TRAP , the quality reward drives the ant forward whereas the diversity reward drives it back to escape the trap and explore the environment.
Therefore, both rewards cancel each other, preventing any learning.
This study validates the usefulness of QD - PG components: 1) optimizing for diversity is required to overcome the deceptive nature of the reward; 2) adding quality optimization provides better asymptotic performance; 3) it is better to disentangle quality and diversity updates.
[Figure] [Figure]
Figure 8a shows coverage maps of the POINT - MAZE environment obtained with one representative seed by the different algorithms presented in the ablation study (see Table 1a).
A dot in the figure corresponds to the final position of an agent after an episode.
The color spectrum highlights the course of training: agents evaluated early in training are in blue while newer ones are represented in purple.
[Figure]
QD - PG and D - PG almost cover the whole BD space including the objective.
Unsurprisingly, Q - PG and TD 3 present very poor coverage maps, both algorithms optimize only for quality and the MAPELITES selection mechanism in Q - PG contributes nothing in this setting.
By contrast, algorithms optimizing for diversity ( QD - PG and D - PG ) find the maze exit.
However, as shown in Table 1a,
QD - PG which also optimizes for quality, is able to refine trajectories through the maze and obtains significantly better performance.
Figure 8b depicts the coverage maps of the ANT - TRAP environment by QD - PG and TD 3.
Only QD - PG is able to bypass the trap and to cover a large part of the BD space.
E. F AST ADAPTATION
The fast adaptation experiment described in Section 7 uses a Bayesian optimization process to quickly find a high-performing solution for a new randomly sampled goal.
Browsing the MAPELITES grid in an exhaustive way is another option to find a good solution for a new objective.
However, the number of solutions to be tested with this option increases quadratically w.r.t.
the number of meshes used to discretize the dimensions of the BD space.
As shown in Table 2, we use a 7 × 7 grid to train QD - PG in the ANT - MAZE environment, containing a maximum of 49 solutions.
In this setting, the difference in computation cost between exhaustive search and Bayesian optimization is negligible.
To ensure that fast adaptation scales to finely discretized MAP - ELITES grids, we reproduce this experiment with a 100 × 100 grid, thus containing thousands of solutions.
We first train QD - PG again on the standard objective of ANT - MAZE and obtain a 100 × 100 grid of solutions.
Then, we repeat the fast adaptation experiment described in Section 7 using this large grid.
With a budget of only 50 solutions to be tested during the Bayesian optimization process among the thousands of solutions contained in the grid, we are able to recover a good solution for the new objective.
Figure 9 maps these 100 fast adaptation experiments to their respective goal location and performance.
In each square, we display the score of the best experiment whose goal was sampled in this region of the maze.
For instance, the square in the top left corner of the performance map corresponds to one of the 100 fast adaptation experiments that sampled its goal in this part of the maze, and obtained a performance of − 12 after testing 50 solutions from the MAP - ELITES grid during the
Bayesian optimization process.
Some squares do not have a score when no experiment sampled its goal in this region of the maze.
F D - PG DERIVATIONS EXTENDED F. P ROOF OF P ROPOSITION 1.
The diversity of a set of K solutions { θ k } k =1 ,...,K is defined as d : Θ K → R + : d [Equation]
[Equation]
The diversity can be split into three terms: the distance of θ 1 to its nearest neighbor (defined as θ 2 ), the distance of θ 1 to the θ j for which θ 1 is the nearest neighbor (defined { θ j } j =3 ,...,K 1 ) and a third term that does not depend on θ 1 .
Thus:
[Equation]
where M = (cid:80) i ̸∈{ 1 ,...,J } min k ̸
= i
||
ξ ( θ i ) , ξ ( θ k )
|| B does not depend on θ 1 .
Hence, the gradient with respect to θ 1 is:
[Equation]
As the remaining term is precisely defined as the novelty n :
[Equation]
[Equation]
We get the final relation :
[Equation]
F. M OTIVATION BEHIND E QUATION
The idea behind this equation is to link novelty defined at the solution level to a notion of novelty defined at the time step level.
The information that we use at the time step level is the current state in the environment, so we describe a solution by the states it visits and hence want to define state novelty such that :
[Equation]
Furthermore, we assume that a state is novel w.r.t.
some solutions if it is novel w.r.t.
to the states visited by these solutions.
To be able to do so, we introduce the notion of state descriptor Ψ , which enables to define the novelty between states.
Hence, we can define the novelty of a state w.r.t.
to a set of solutions by:
[Equation]
This state descriptor Ψ constrains the link between the novelty at the state level with the novelty at the solution level through a link with the behavior descriptor ξ .
That being said, we can now compute the diversity gradient thanks to the novelty at the state level.
As a matter of fact, proposition 1 links diversity gradient to novelty at solution level:
[Equation]
Then we link novelty at the solution level to novelty at the state level with the following equation, which is satisfied thanks to the relevant choice of the state descriptor Ψ .
[Equation]
Finally, by replacing the novelty at the solution level by the novelty at the state level in proposition 1, we get the formulation of the diversity policy gradient given in equation 4.
[Equation]