D IFFERENTIABLE Q UALITY D IVERSITY FOR R EINFORCEMENT L EARNING BY A PPROXIMATING G RADIENTS
A BSTRACT
Consider the problem of training robustly capable agents.
One approach is to generate a diverse collection of agent polices.
Training can then be viewed as a quality diversity (QD) optimization problem, where we search for a collection of performant policies that are diverse with respect to quantiﬁed behavior.
Recent work shows that differentiable quality diversity (DQD) algorithms greatly accelerate QD optimization when exact gradients are available.
However, agent policies typically assume that the environment is not differentiable.
To apply DQD algorithms to training agent policies, we must approximate gradients for performance and behavior. We propose two variants of the current state-of-the-art DQD algorithm that compute gradients via approximation methods common in reinforcement learning (RL).
We evaluate our approach on four simulated locomotion tasks.
One variant achieves results comparable to the current state-of-the-art in combining QD and RL, while the other performs comparably in two locomotion tasks.
These results provide insight into the limitations of current DQD algorithms in domains where gradients must be approximated. Source code is available at https://github.com/icaros-usc/dqd-rl
I NTRODUCTION
We focus on the problem of extending differentiable quality diversity (DQD) to reinforcement learning (RL) domains.
We propose to approximate gradients for the objective and measure functions, resulting in two variants of the DQD algorithm CMA-MEGA (Fontaine and Nikolaidis, 2021b).
Consider a half-cheetah agent (Fig. 2) trained for locomotion, where the agent must continue walking forward even when one foot is damaged.
If we frame this challenge as an RL problem, two approaches to design a robustly capable agent would be to (1) design a reward function and (2) apply domain randomization (Tobin et al., 2017; Peng et al., 2018).
However, prior work (Irpan, 2018; Clark and Amodei, 2016) suggests that designing such a reward function is difﬁcult, while domain randomization may require manually selecting hundreds of environment parameters (Peng et al., 2018; OpenAI et al., 2019).
As an alternative approach, consider that we have intuition on what behaviors would be useful for adapting to damage.
For instance, we can measure how often each foot is used during training, and we can pre-train a collection of policies that are diverse in how the agent uses its feet.
When one of the agent’s feet is damaged during deployment, the agent can adapt to the damage by selecting a policy that did not move the damaged foot during training (Cully et al., 2015; Colas et al., 2020).
Pre-training such a collection of policies may be viewed as a quality diversity (QD) optimization problem (Pugh et al., 2016; Cully et al., 2015; Mouret and Clune, 2015; Colas et al., 2020).
Formally,
QD assumes an objective function f and one or more measure functions m .
The goal of QD is to
[Figure]
The recently proposed DQD algorithm CMA-MEGA (Fontaine and Nikolaidis, 2021b) outperforms
QD algorithms by orders of magnitude when exact gradients are available, such as when searching the latent space of a generative model.
However, RL problems like the half-cheetah lack these gradients because the environment is typically non-differentiable, thus limiting the applicability of
DQD.
To address this limitation, we draw inspiration from how evolution strategies (ES) (Akimoto et al., 2010; Wierstra et al., 2014; Salimans et al., 2017; Mania et al., 2018) and deep RL actorcritic methods (Schulman et al., 2015; 2017; Lillicrap et al., 2016; Fujimoto et al., 2018) optimize a reward objective by approximating gradients for gradient descent.
Our key insight is to approximate objective and measure gradients for DQD algorithms by adapting ES and actor-critic methods.
Our work makes three contributions.
(1) We formalize the problem of quality diversity for reinforcement learning (QD-RL) and reduce it to an instance of DQD.
(2) We develop two QD-RL variants of the DQD algorithm CMA-MEGA, where each algorithm approximates objective and measure gradients with a different combination of ES and actor-critic methods.
(3) We benchmark our variants on four PyBullet locomotion tasks from QDGym (Ellenberger, 2019; Nilsson, 2021).
One variant performs comparably (in terms of QD score; Sec. 5.1.2) to the state-of-the-art PGA-MAP-Elites (Nilsson and Cully, 2021) in two tasks.
The other variant achieves comparable QD score with PGAMAP-Elites in all tasks 1 but is less efﬁcient than PGA-MAP-Elites in two tasks.
These results contrast with prior work (Fontaine and Nikolaidis, 2021b) where CMA-MEGA vastly outperforms a DQD algorithm inspired by PGA-MAP-Elites on benchmark functions where gradient information is available.
Overall, we shed light on the limitations of CMA-MEGA in QD domains where the main challenge comes from optimizing the objective rather than from exploring measure space.
At the same time, since we decouple gradient estimates from QD optimization, our work opens a path for future research that would beneﬁt from independent improvements to either DQD or RL. [Figure]
P ROBLEM S TATEMENT 2. Q UALITY D IVERSITY (QD)
We adopt the deﬁnition of QD from prior work (Fontaine and Nikolaidis, 2021b).
For a solution vector φ ∈ R n , QD considers an objective function f ( φ ) and k measures 2 m i ( φ ) ∈ R (for i ∈ 1 ..k ) or, as a joint measure, m ( φ ) ∈ R k .
These measures form a k -dimensional measure space X .
For every x ∈ X , the QD objective is to ﬁnd solution φ such that m ( φ ) =
x and f ( φ ) is maximized.
Since X is continuous, it would require inﬁnite memory to solve the QD problem, so algorithms in the MAP-Elites family (Mouret and Clune, 2015; Cully et al., 2015)
discretize X by forming a tesselation Y consisting of M cells.
Thus, we relax the QD problem to one of searching for an archive A consisting of M elites φ i , one for each cell in Y .
Then, the QD objective is to maximize the performance f ( φ i ) of all elites:
[Equation]
2.1. D IFFERENTIABLE Q UALITY D IVERSITY (DQD)
In DQD, we assume f and m are ﬁrst-order differentiable.
We denote the objective gradient as ∇ f ( φ ) , or abbreviated as ∇ f , and the measure gradients as ∇ m ( φ ) or ∇ m .
2. Q UALITY D IVERSITY FOR R EINFORCEMENT L EARNING (QD-RL)
We deﬁne QD-RL as an instance of the QD problem in which each solution φ parameterizes an RL policy π φ . Then, the objective f ( φ ) is the expected discounted return of π φ , and the measures m ( φ ) are functions of π φ .
Formally, drawing on the Markov Decision Process (MDP) formulation (Sutton and Barto, 2018), we represent QD-RL as a tuple ( S , U , p, r, γ, m ) .
On discrete timesteps t in an episode of interaction, an agent observes state s ∈ S and takes action a ∈ U according to a policy π φ ( a | s ) .
The agent then receives scalar reward r ( s, a ) and observes next state s (cid:48) ∈ S according to s (cid:48) ∼ p ( ·| s, a ) .
Each episode thus has a trajectory ξ = { s 0 , a 0 , s 1 , a 1 , .., s T } , where T is the number of timesteps in the episode, and the probability that policy π φ takes trajectory ξ is p φ [Equation]
) = p
[Equation]
s 0 )
[Equation]
t[-=
-]
[Equation] ) p [Equation] s t
[Equation]
1 | s t , a t ) .
Now, we deﬁne the expected discounted return of policy π φ as
[Equation]
where the discount factor γ ∈ (0 , 1) trades off between short- and long-term rewards.
Finally, we quantify the behavior of policy π φ via a k -dimensional measure function m ( φ ) .
2.2. QD-RL AS AN INSTANCE OF DQD
We reduce QD-RL to a DQD problem.
Since the exact gradients ∇ f and ∇ m usually do not exist in QD-RL, we must instead approximate them.
B ACKGROUND 3. S INGLE -O BJECTIVE R EINFORCEMENT L EARNING
We review algorithms which train a policy to maximize a single objective, i.e. f ( φ ) in Eq. 2, with the goal of applying these algorithms’ gradient approximations to DQD in Sec. 4.
3.1. E VOLUTION STRATEGIES (ES)
ES (Beyer and Schwefel, 2002) is a class of evolutionary algorithms which optimizes the objective by sampling a population of solutions and moving the population towards areas of higher performance.
Natural Evolution Strategies (NES)
(Wierstra et al., 2014; 2008) is a type of ES which updates the sampling distribution of solutions by taking steps on distribution parameters in the direction of the natural gradient (Amari, 1998).
For example, with a Gaussian sampling distribution, each iteration of an NES would compute natural gradients to update the mean µ and covariance Σ .
We consider an NES-inspired algorithm (Salimans et al., 2017) which has demonstrated success in RL domains.
This algorithm, which we refer to as OpenAI-ES, samples λ es solutions from an isotropic Gaussian but only computes a gradient step for the mean φ .
Each solution sampled by OpenAI-ES is represented as φ + σ (cid:15) i , where σ is the ﬁxed standard deviation of the Gaussian and (cid:15) i ∼ N ( 0 , I ) .
Once these solutions are evaluated, OpenAI-ES estimates the gradient as
[Equation]
OpenAI-ES then passes this estimate to an Adam optimizer (Kingma and Ba, 2015) which outputs a gradient ascent step for φ .
To make the estimate more accurate, OpenAI-ES further includes techniques such as mirror sampling and rank normalization (Brockhoff et al., 2010; Ha, 2017; Wierstra et al., 2014).
3.1. A CTOR - CRITIC METHODS
While ES treats the objective as a black box, actor-critic methods leverage the MDP structure of the objective, i.e. the fact that f ( φ ) is a sum of Markovian values.
We are most interested in Twin
Delayed Deep Deterministic policy gradient (TD3) (Fujimoto et al., 2018), an off-policy actor-critic method.
TD3 maintains (1) an actor consisting of the policy π φ and (2) a critic consisting of stateaction value functions Q θ 1 ( s, a ) and Q θ 2 ( s, a ) which differ only in random initialization.
Through interactions in the environment, the actor generates experience which is stored in a replay buffer B .
This experience is sampled to train Q θ 1 and Q θ 2 .
Simultaneously, the actor improves by maximizing
Q θ 1 via gradient ascent ( Q θ 2 is only used during critic training).
Speciﬁcally, for an objective f (cid:48) which is based on the critic and approximates f , TD3 estimates a gradient ∇ f (cid:48) ( φ ) and passes it to an Adam optimizer.
Notably, TD3 never updates network weights directly, instead accumulating weights into target networks π φ (cid:48) , Q θ (cid:48) 1 , Q θ (cid:48) 2 via an exponentially weighted moving average with update rate τ .
3. Q UALITY D IVERSITY A LGORITHMS 3.2. MAP-E LITES EXTENSIONS FOR QD-RL
One of the simplest QD algorithms is MAP-Elites (Mouret and Clune, 2015; Cully et al., 2015).
MAP-Elites creates an archive A by tesselating the measure space X into a grid of evenly-sized cells.
Then, it draws λ initial solutions from a multivariate Gaussian N ( φ 0 , σ I ) centered at some φ 0 .
Next, for each sampled solution φ , MAP-Elites computes f ( φ ) and m ( φ ) and inserts φ into A .
In subsequent iterations, MAP-Elites randomly selects λ solutions from A and adds Gaussian noise, i.e. solution φ becomes φ + N ( 0 , σ I ) .
Solutions are placed into cells based on their measures; if a solution has higher f than the solution currently in the cell, it replaces that solution.
Once inserted into A , solutions are known as elites .
Due to the high dimensionality of neural network parameters, direct policy optimization with MAPElites has not proven effective in QD-RL (Colas et al., 2020), although indirect encodings have been shown to scale to large policy networks (Rakicevic et al., 2021; Gaier et al., 2020).
For direct search, several extensions merge MAP-Elites with actor-critic methods and ES.
For instance, Policy Gradient Assisted MAP-Elites (PGA-MAP-Elites) (Nilsson and Cully, 2021) combines MAP-Elites with TD3.
Each iteration, PGA-MAP-Elites evaluates λ solutions for insertion into the archive.
λof these are created by selecting random solutions from the archive and taking gradient ascent steps with a TD3 critic.
The other λ 2 solutions are created with a directional variation operator (Vassiliades and Mouret, 2018) which selects two solutions φ 1 and φ 2 from the archive and creates a new one according to φ (cid:48) =
φ 1 + σ 1 N ( 0 , I ) + σ 2 ( φ 2 − φ 1 ) N (0 , 1) .
Finally, PGA-MAP-Elites maintains a “greedy actor” which provides actions when training the critics (identically to the actor in TD3).
Every iteration, PGA-MAP-Elites inserts this greedy actor into the archive.
PGA-MAP-Elites achieves state-of-the-art performance on locomotion tasks in the QDGym benchmark (Nilsson, 2021).
Another MAP-Elites extension is ME-ES (Colas et al., 2020), which combines MAP-Elites with an OpenAI-ES optimizer.
In the “explore-exploit” variant, ME-ES alternates between two phases.
In the “exploit” phase, ME-ES restarts OpenAI-ES at a mean φ and optimizes the objective for k iterations, inserting the current φ into the archive in each iteration.
In the “explore” phase, ME-ES repeats this process, but OpenAI-ES instead optimizes for novelty, where novelty is the distance in measure space from a new solution to previously encountered solutions.
ME-ES also has an “exploit” variant and an “explore” variant, which each execute only one type of phase.
Our work is related to ME-ES in that we also adapt OpenAI-ES, but instead of alternating between following a novelty gradient and objective gradient, we compute all objective and measure gradients and allow a CMA-ES (Hansen, 2016) instance to decide which gradients to follow by sampling gradient coefﬁcients from a multivariate Gaussian updated over time (Sec. 3.2.2).
We include MAP-Elites, PGA-MAP-Elites, and ME-ES as baselines in our experiments.
Refer to Fig.
3 for a diagram which compares these algorithms to our approach.
3.2. C OVARIANCE M ATRIX A DAPTATION MAP-E LITES VIA A G RADIENT A RBORESCENCE (CMA-MEGA)
We directly extend CMA-MEGA (Fontaine and Nikolaidis, 2021b) to address QD-RL.
CMAMEGA is a DQD algorithm based on the QD algorithm CMA-ME (Fontaine et al., 2020).
The intuition behind CMA-MEGA is that if we knew which direction the current solution point φ ∗ should move in objective-measure space, then we could calculate that change in search space via a linear combination of objective and measure gradients.
From CMA-ME, we know a good direction is one that results in the largest archive improvement.
Each iteration, CMA-MEGA ﬁrst calculates objective and measure gradients for a solution point φ ∗ .
Next, it generates λ new solutions by sampling gradient coefﬁcients c ∼ N ( µ , Σ ) and computing φ (cid:48) ← φ ∗ + c 0 ∇ f ( φ ∗ )
+ (cid:80) kj =
1 c j ∇ m j ( φ ∗ ) .
CMA-MEGA inserts these solutions into the archive and computes their improvement, ∆ .
∆ is deﬁned as f ( φ (cid:48) )
if φ (cid:48) populates a new cell, and f ( φ (cid:48) )
− f ( φ (cid:48)E ) if φ (cid:48) improves an existing cell (replaces a previous solution φ (cid:48)E ).
After CMA-MEGA inserts the solutions, it ranks them by ∆ .
If a solution populates a new cell, its ∆ always ranks higher than that of a solution which only improves an existing cell.
CMA-MEGA then moves the solution point φ ∗ towards the largest archive improvement, but also adapts the distribution N ( µ , Σ ) towards better gradient coefﬁcients by the same ranking.
By leveraging gradient information, CMA-MEGA solves
QD benchmarks with orders of magnitude fewer solution evaluations than previous QD algorithms.
A PPROXIMATING G RADIENTS FOR DQD
Since DQD algorithms require exact objective and measure gradients, we cannot directly apply
CMA-MEGA to QD-RL.
To address this limitation, we replace exact gradients with gradient approximations (Sec. 4.1) and develop two CMA-MEGA variants (Sec. 4.2).
4. A PPROXIMATING O BJECTIVE AND M EASURE G RADIENTS
We adapt gradient approximations from ES and actor-critic methods.
Since the objective has an
MDP structure, we estimate objective gradients ∇ f with ES and actor-critic methods.
Since the measures are black boxes, we estimate measure gradients ∇ m with ES.
4.1. A PPROXIMATING OBJECTIVE GRADIENTS WITH ES AND ACTOR - CRITIC METHODS
We estimate objective gradients with two methods.
First, we treat the objective as a black box and estimate its gradient with a black box method, namely the OpenAI-ES gradient estimate in Eq. 3.
Since OpenAI-ES performs well in RL domains (Salimans et al., 2017;
Pagliuca et al., 2020; Lehman et al., 2018), we believe this estimate is suitable for approximating gradients for CMA-MEGA in QD-RL settings.
Importantly, this estimate requires environment interaction by evaluating λ es solutions.
Since the objective has a well-deﬁned structure, i.e. it is a sum of rewards from an MDP (Eq. 2), we also estimate its gradient with an actor-critic method, TD3.
TD3 is well-suited for this purpose because it efﬁciently estimates objective gradients for the multiple policies that CMA-MEGA and other QD-RL algorithms generate.
In particular, once the critic is trained, TD3 can provide a gradient estimate for any policy without additional environment interaction.
Among actor-critic methods, we select TD3 since it achieves high performance while optimizing primarily for the RL objective.
Prior work (Fujimoto et al., 2018) shows that TD3 outperforms onpolicy actor-critic methods (Schulman et al., 2015; 2017).
While the off-policy Soft Actor-Critic (Haarnoja et al., 2018) algorithm can outperform TD3, it optimizes a maximum-entropy objective designed to encourage exploration.
In our work, we explore by ﬁnding policies with different measures. Thus, we leave for future work the problem of integrating QD-RL with the action diversity encouraged by entropy maximization.
4.1. A PPROXIMATING MEASURE GRADIENTS WITH ES
Since measures do not have special properties such as an MDP structure (Sec. 2.2), we only estimate their gradient with black box methods.
Thus, similar to the objective, we approximate each measure’s gradient ∇ m i with the OpenAI-ES gradient estimate, replacing f with m i in Eq. 3.
Since the OpenAI-ES gradient estimate requires additional environment interaction, all of our CMAMEGA variants require environment interaction to estimate gradients.
However, the environment interaction required to estimate measure gradients remains constant even as the number of measures increases, since we can reuse the same λ es solutions to estimate each ∇ m i .
In problems where the measures have an MDP structure similar to the objective, it may be feasible to estimate each ∇ m i with its own TD3 instance.
In the environments in our work (Sec. 5.1), each measure is non-Markovian since it calculates the proportion of time a walking agent’s foot spends on the ground.
This calculation depends on the entire agent trajectory rather than on one state.
4. CMA-MEGA V ARIANTS
Our choice of gradient approximations leads to two CMA-MEGA variants.
CMA-MEGA (ES) approximates objective and measure gradients with OpenAI-ES, while CMA-MEGA (TD3, ES) approximates the objective gradient with TD3 and measure gradients with OpenAI-ES.
Fig.shows an overview of both algorithms, and Algorithm 1 shows their pseudocode.
As CMA-MEGA (TD3, ES) builds on CMA-MEGA (ES), we present only CMA-MEGA (TD3, ES) and highlight lines that CMA-MEGA (TD3, ES) additionally executes. Identically to CMA-MEGA, the two variants maintain three primary components: a solution point φ ∗ , a multivariate Gaussian distribution N ( µ , Σ ) for sampling gradient coefﬁcients, and a MAPElites archive A for storing solutions. We initialize the archive and solution point on line 3, and we initialize the coefﬁcient distribution as part of a CMA-ES instance on line
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
f ( φ ∗ ) + (cid:80) kj =
1 c
i,j ∇ m j ( φ ∗ ) φ (cid:48)
i ← φ ∗ + ∇ i f ( φ (cid:48) i ) , m (cid:48)
( φ (cid:48) i ) ← evaluate ( φ (cid:48) i ) ∆ i ← U PDATE _
A RCHIVE
( A , φ (cid:48)
i , f ( φ (cid:48) i ) , m ( φ (cid:48) i ) ) end
Rank c i , ∇ i by ∆ i
Adapt CMA-ES parameters µ , Σ , p based on rankings of c i φ ∗ ← φ ∗ + η (cid:80) λi =1 w i ∇ rank[i] //
w i is of p if there is no change in A then
Restart CMA-ES with µ = 0 , Σ
= σ g I
Set φ ∗ to a randomly selected elite from A end f
( φ q ) , m ( φ q ) ← evaluate ( φ q )
U PDATE _ A RCHIVE ( A , φ q , f ( φ q ) , m ( φ q ) ) Add experience from all calls to evaluate into B
T RAIN _TD3( Q θ 1 , Q θ 2 , π φ q , Q θ (cid:48) 1 , Q θ (cid:48) 2 , π φ (cid:48) q , B ) end
In the main loop (line 6), we follow the workﬂow shown in Fig.
First, after evaluating φ ∗ and inserting it into the archive (line 7-8), we approximate its gradients with either ES or TD3 (line 9-10).
This gradient approximation forms the key difference between our variants and the original CMAMEGA algorithm
(Fontaine and Nikolaidis, 2021b).
Next, we branch from φ ∗ to create solutions φ (cid:48)
i by sampling c
i from the coefﬁcient distribution and computing perturbations ∇ i (line 13-15).
We then evaluate each φ (cid:48) i and insert it into the archive (line 16-17).
Finally, we update the solution point and the coefﬁcient distribution’s CMA-ES instance by forming an improvement ranking based on the improvement ∆ i (Sec. 3.2.2; line 19-21).
Importantly, since we rank based on improvement, this update enables the CMA-MEGA variants to maximize the QD objective (Eq. 1)
(Fontaine and
Nikolaidis, 2021b).
Our CMA-MEGA variants have two additional components.
First, we check if no solutions were inserted into the archive at the end of the iteration, which would indicate that we should reset the coefﬁcient distribution and the solution point (line 22-24).
Second, in the case of CMA-MEGA (TD3, ES), we manage a TD3 instance similar to how PGA-MAP-Elites does (Sec. 3.2.1).
This
TD3 instance consists of a replay buffer B , critic networks Q θ 1 and Q θ 2 , a greedy actor π φ q , and target networks Q θ (cid:48) 1 , Q θ (cid:48) 2 , π φ (cid:48) q (all initialized on line 5).
At the end of each iteration, we use the greedy actor to train the critics, and we also insert it into the archive (line 26-29).
E XPERIMENTS
We compare our two proposed CMA-MEGA variants (CMA-MEGA (ES), CMA-MEGA (TD3,
ES)) with three baselines (PGA-MAP-Elites, ME-ES, MAP-Elites) in four locomotion tasks.
We implement MAP-Elites as described in Sec.
3.2.1, and we select the explore-exploit variant for MEES since it has performed at least as well as both the explore variant and the exploit variant in several domains (Colas et al., 2020).
5. E VALUATION D OMAINS 5.1. QDG YM
We evaluate our algorithms in four locomotion environments from QDGym (Nilsson, 2021), a library built on PyBullet Gym (Coumans and Bai, 2020; Ellenberger, 2019) and OpenAI Gym (Brockman et al., 2016).
Appendix D lists all environment details.
In each environment, the QD algorithm outputs an archive of walking policies for a simulated agent.
The agent is primarily rewarded for its forward speed.
There are also reward shaping (Ng et al., 1999) signals, such as a punishment for applying higher joint torques, intended to guide policy optimization.
The measures compute the proportion of time (number of timesteps divided by total timesteps in an episode) that each of the agent’s feet contacts the ground.
QDGym is challenging because the objective in each environment does not “align” with the measures, in that ﬁnding policies with different measures (i.e. exploring the archive) does not necessarily lead to optimization of the objective.
While it may be trivial to ﬁll the archive with low-performing policies which stand in place and lift the feet up and down to achieve different measures, the agents’ complexity (high degrees of freedom) makes it difﬁcult to learn a high-performing policy for each value of the measures.
5.1. M ETRICS
Our primary metric is QD score (Pugh et al., 2016), which provides a holistic view of algorithm performance.
QD score is the sum of the objective values of all elites in the archive, i.e. (cid:80)
Mi =1 1 φ i exists f ( φ i ) , where M is the number of archive cells.
We note that the contribution of a cell to the QD score is 0 if the cell is unoccupied.
We set the objective f to be the expected undiscounted return , i.e. we set γ = 1 in Eq. 2.
Since objectives may be negative, an algorithm’s QD score may be penalized when adding a new solution.
To prevent this, we deﬁne a minimum objective in each environment by taking the lowest objective value that was inserted into the archive in any experiment in that environment.
We subtract this minimum from every solution, such that every solution that was inserted into an archive has an objective value of at least 0.
Thus, we use QD score deﬁned as (cid:80)
Mi =1 1 φ i exists ( f ( φ i ) − min objective) .
We also deﬁne a maximum objective equivalent to each environment’s “reward threshold” in PyBullet Gym.
This threshold is the objective value at which an agent is considered to have successfully learned to walk.
We report two metrics in addition to QD score.
Archive coverage , the proportion of cells for which the algorithm found an elite, gauges how well the QD algorithm explores measure space, and best performance , the highest objective of any elite in the archive, gauges how well the QD algorithm exploits the objective.
5. E XPERIMENTAL D ESIGN
We follow a between-groups design, where the two independent variables are environment (QD
Ant, QD Half-Cheetah, QD Hopper, QD Walker) and algorithm (CMA-MEGA (ES), CMA-MEGA (TD3, ES), PGA-MAP-Elites, ME-ES, MAP-Elites).
The dependent variable is the QD score.
In we seeds and test
H1: CMA-MEGA (ES) will outperform all baselines (PGA-MAP-Elites, ME-ES, MAP-Elites).
H2: CMA-MEGA (TD3, ES) will outperform all baselines.
H3:
CMA-MEGA (TD3, ES) will outperform CMA-MEGA (ES).
H1 and H2 are based on prior work (Fontaine and Nikolaidis, 2021b) which showed that in QD benchmark domains, CMA-MEGA outperforms algorithms that do not leverage both objective and measure gradients.
H3 is based on results (Pagliuca et al., 2020) which suggest that actor-critic methods outperform ES in PyBullet Gym.
Thus, we expect the TD3 objective gradient to be more accurate than the ES objective gradient, leading to more efﬁcient traversal of objective-measure space and higher QD score.
5. I MPLEMENTATION
We implement all QD algorithms with the pyribs library (Tjanaka et al., 2021) except for ME-ES, which we adapt from the authors’ implementation.
We run each experiment with 100 CPUs on a high-performance cluster.
We allocate one NVIDIA Tesla P100 GPU to algorithms that train TD (CMA-MEGA (TD3, ES) and PGA-MAP-Elites).
Depending on the algorithm and environment, each experiment lasts 4-20 hours; refer to Table 12, Appendix E for mean runtimes.
R ESULTS
We ran 5 trials of each algorithm in each environment.
In each trial, we allocated 1 million evaluations and recorded the QD score, archive coverage, and best performance.
Fig.
4 plots these metrics, and Appendix E lists ﬁnal values of all metrics.
Appendix H shows example heatmaps and histograms of each archive, and the supplemental material contains videos of generated agents.
Refer to Appendix G for a discussion of our results.
C ONCLUSION
To extend DQD to RL settings, we adapted gradient approximations from actor-critic methods and
ES.
By integrating these approximations with CMA-MEGA, we proposed two novel variants that we evaluated on four locomotion tasks from QDGym.
CMA-MEGA (TD3, ES) performed comparably to the state-of-the-art PGA-MAP-Elites in all tasks but was less efﬁcient in two of the tasks.
CMAMEGA (ES) performed comparably in two tasks.
Our results contrast prior work (Fontaine and Nikolaidis, 2021b) where CMA-MEGA outperformed a baseline algorithm inspired by PGA-MAP-Elites in QD benchmark domains.
The difference seems to be that difﬁculty in the benchmarks arises from a hard-to-explore measure space, whereas difﬁculty in QDGym arises from an objective which requires rigorous optimization.
As such, future work could formalize the notions of “exploration difﬁculty” of a measure space and “optimization difﬁculty” of an objective and evaluate algorithms in benchmarks that cover a spectrum of these metrics.
For practitioners looking to apply DQD in RL settings, we recommend estimating objective gradients with an off-policy actor-critic method such as TD3 instead of with an ES.
Due to the difﬁculty of modern control benchmarks, it is important to efﬁciently optimize the objective — TD3 beneﬁts over ES since it can compute the objective gradient without further environment interaction.
Furthermore, reward signals in these benchmarks are designed for deep RL methods, making TD gradients more useful than ES gradients. By reducing QD-RL to DQD, we have decoupled QD-RL into DQD optimization and RL gradient approximations. In the future, we envision algorithms which beneﬁt from advances in either more efﬁcient DQD or more accurate RL gradient approximations.
A R ELATED W ORK A. B EYOND MAP-E LITES
Several QD-RL algorithms have been developed outside the MAP-Elites family.
NS-ES (Conti et al., 2018) builds on Novelty Search (NS) (Lehman and Stanley, 2011a;b), a family of QD algorithms which add solutions to an unstructured archive only if they are far away from existing archive solutions in measure space.
Using OpenAI-ES, NS-ES concurrently optimizes several agents for novelty.
Its variants NSR-ES and NSRA-ES optimize for a linear combination of novelty and objective.
Meanwhile, the QD-RL algorithm (Cideron et al., 2020) (distinct from the QD-RL problem we deﬁne) maintains an archive with all past solutions and optimizes agents along a Pareto front of the objective and novelty.
Finally, Diversity via Determinants (DvD) (Parker-Holder et al., 2020) leverages a kernel method to maintain diversity in a population of solutions.
As NS-ES, QD-RL, and
DvD do not output a MAP-Elites grid archive, we leave their investigation for future work.
A. D IVERSITY IN R EINFORCEMENT L EARNING
Here we distinguish QD-RL from prior work which also applies diversity to RL.
One area of work is in latent- and goal-conditioned policies.
For latent-conditioned policy π φ ( a | s, z ) (Eysenbach et al., 2019; Kumar et al., 2020;
Li et al., 2017) or goal-conditioned policy π φ ( a | s, g ) (Schaul et al., 2015;
Andrychowicz et al., 2017), varying the latent variable z or goal g results in different behaviors, e.g. different walking gaits or walking to a different location.
While QD-RL also seeks a range of behaviors, the measures m ( φ ) are computed after evaluating φ , rather than before the evaluation.
In general, QD-RL focuses on ﬁnding a variety of policies for a single task, rather than attempting to solve a variety of tasks with a single conditioned policy.
Another area of work combines evolutionary and actor-critic algorithms to solve single-objective hard-exploration problems (Colas et al., 2018; Khadka and Tumer, 2018; Pourchot and Sigaud, 2019;
Tang, 2021; Khadka et al., 2019).
In these methods, an evolutionary algorithm such as cross-entropy method (de Boer et al., 2005) facilitates exploration by generating a diverse population of policies, while an actor-critic algorithm such as TD3 trains high-performing policies with this population’s environment experience.
QD-RL differs from these methods in that it views diversity as a component of the output, while these methods view diversity as a means for environment exploration.
Hence,
QD-RL measures policy behavior via a measure function and collects diverse policies in an archive.
In contrast, these RL exploration methods assume that trajectory diversity, rather than targeting speciﬁc behavioral diversity, is enough to drive exploration to discover a single optimal policy.
B H ELPER F UNCTIONS FOR CMA-MEGA V ARIANTS
1 U PDATE _
A RCHIVE
( A , φ , f ( φ ) , m ( φ )) : //
E contains φ E , f ( φ E ) , m ( φ E )
E ← cell in A corresponding to m if E is empty then φ E , f ( φ E ) , m ( φ E ) ← φ , f ( φ ) , m ( φ ) return ( NEW _ CELL , f ( φ ) ) else if f ( φ ) >
f ( φ E ) then φ E , f ( φ E ) , m ( φ E ) ← φ , f ( φ ) , m ( φ ) return ( IMPROVE _ EXISTING _
CELL , f ( φ ) − f ( φ E ) ) else return ( NOT _ ADDED , f ( φ ) − f ( φ E ) ) end
1 I NITIALIZE _TD3:
B ← initialize_replay_buffer() //
As done in the TD3 author implementation (Fujimoto et al., 2018), we initialize these networks with the default
PyTorch weights.
Q θ 1 , Q θ 2 , π φ q ← initialize_networks()
Q θ (cid:48) 1 , Q θ (cid:48) 2 , π φ (cid:48)
q ← Q θ 1 , Q θ 2 , π φ q return B , Q θ 1 , Q θ 2 , π φ q , Q θ (cid:48) 1 , Q θ (cid:48) 2 , π φ (cid:48)
q 7 TD3_G RADIENT ( φ , Q θ 1 , B ) :
Sample n pg transitions ( s t , a t , r ( s t , a t ) , s t +1 ) from B ∇ φ J ( φ )
= 1 n pg (cid:80) ∇ φ π φ ( s t ) ∇ a Q θ 1 ( s t , a ) | a = π φ ( s t ) return ∇ φ J ( φ ) 11 T RAIN [-_TD
-]_TD3 ( Q θ 1 , Q θ 2 , π φ q Q θ (cid:48) 1 , Q θ (cid:48) 2 , π φ (cid:48) q , B ) : //
Trains the critic and the greedy actor.
for i ← 1 ..n crit do
Sample n q transitions ( s t , a t , r ( s t , a t ) , s t +1 ) from B //
Sample smoothing noise.
(cid:15) ∼ clip ( N (0 , σ p ) , − c clip , c clip ) y
= r ( s t , a t )
+ γ min i
=1 , 2 Q θ (cid:48)
i ( s t +1 , π φ (cid:48) q ( s t +1 ) + (cid:15) ) // Update critics.
θ
i ← arg min θ i 1 n q (cid:80) ( y − Q
θ i ( s t , a t ))
if t mod d = 0 then // Update greedy actor.
∇ φ q J ( φ q ) = 1 n q (cid:80) ∇ φ q π φ q ( s t ) ∇ a Q θ 1 ( s t , a ) |
a = π φ q ( s t ) //
Update targets.
θ (cid:48)
i ← τ θ i + (1 − τ ) θ (cid:48)
i φ (cid:48)
q ← τ φ q + (1 − τ ) φ (cid:48) q end end
∇ f ( φ ) , ∇ m ( φ )
C A LGORITHM H YPERPARAMETERS
Each agent’s policy is a neural network which takes in states and outputs actions.
There are two hidden layers of 128 nodes, and the hidden and output layers have tanh activation. We initialize weights with Xavier initialization (Glorot and Bengio, 2010).
For the archive, we tesselate each environment’s measure space into a grid of evenly-sized cells (see Table 6 for grid dimensions). Each measure is bound to the range [0 , 1] , the min and max proportion of time that one foot can contact the ground.
Each algorithm evaluates 1 million solutions in the environment.
Due to computational limits, we evaluate each solution once instead of averaging multiple episodes, so each algorithm runs 1 million episodes total.
Below we list additional parameters for each algorithm in our experiments.
[Table] [Table]
[Table] [Table]
[Table]
D E NVIRONMENT D ETAILS
[Table]
Table 6 lists all environment details.
The measures in QDGym are the proportions of time that each foot contacts the ground.
In each environment, the feet are ordered as follows:
• QD Ant: front left foot, front right foot, back left foot, back right foot • QD Half-Cheetah: front foot, back foot • QD Hopper: single foot • QD Walker: right foot, left foot
E F INAL M ETRICS
Tables 7-12 show the QD score (Sec. 5.1.2), QD score AUC (Sec. G.2.1), archive coverage (Sec.
5.1.2), best performance (Sec. 5.1.2), mean elite robustness (Sec. G.2.4), and runtime in hours for all algorithms in all environments.
The tables show the value of each metric after 1 million evaluations, averaged over 5 trials.
Due to its magnitude, QD score AUC is expressed as a multiple of 10 12 .
Though CMA-MEGA (TD3, ES) and PGA-MAP-Elites perform best overall, they rely on specialized hardware (a GPU) and require the most computation. As shown in Table 12, the TD3 training in these algorithms leads to long runtimes. When runtime is dominated by the algorithm itself (as opposed to solution evaluations), CMA-MEGA (ES) offers a viable alternative that may achieve reasonable performance.
[Table] [Table]
[Table] [Table] [Table]
[Table]
F F ULL S TATISTICAL A NALYSIS
To compare a metric such as QD score between two or more algorithms across all four QDGym environments, we performed a two-way ANOVA where environment and algorithm were the independent variables and the metric was the dependent variable.
When there was a signiﬁcant interaction effect (note that all of our analyses found signiﬁcant interaction effects), we followed up this
ANOVA with a simple main effects analysis in each environment.
Finally, we ran pairwise comparisons (two-sided t-tests) to determine which algorithms had a signiﬁcant difference on the metric.
We applied Bonferroni corrections within each environment / simple main effect.
For example, in
Table 13 we compared CMA-MEGA (ES) with three algorithms in each environment, so we applied a Bonferroni correction with n = 3 .
This section lists the ANOVA and pairwise comparison results for each of our analyses.
We have bolded all signiﬁcant p -values, where signiﬁcance is determined at the α = 0 .
05 threshold.
For pairwise comparisons, some p -values are marked as “1” because the Bonferroni correction caused the p -value to exceed 1.
p -values less than 0.001 have been marked as “ < 0.001 ”.
F. QD S CORE A NALYSIS (S EC . G.1)
To test the hypotheses we deﬁned in Sec. 5.2, we performed a two-way ANOVA for QD scores.
Since the ANOVA requires scores in all environments to have the same scale, we normalized the
QD score in all environments by dividing by the maximum QD score, deﬁned in Sec.
G.1 as grid cells * ( max objective - min objective ).
The results of the ANOVA were as follows:
• Interaction effect: F (12 , 80) = 16 .
82 , p < 0 .•
Simple main effects: – QD Ant: F (4 , 80) = 23 .
87 , p < 0 .
– QD Half-Cheetah: F (4 , 80) = 44 .
15 , p < 0 .
– QD Hopper: F (4 , 80) = 57 .
35 , p < 0 .
– QD Walker: F (4 , 80) = 90 .
84 , p < 0 .
Since the ANOVA showed a signiﬁcant interaction effect and signiﬁcant simple main effects, we performed pairwise comparisons for each hypothesis (Tables 13-15).
F. QD S CORE AUC A NALYSIS (S EC . G.2.1)
In this followup analysis, we hypothesized that PGA-MAP-Elites would have greater QD score AUC than CMA-MEGA (ES) and CMA-MEGA (TD3, ES).
Thus, we performed a two-way ANOVA which compared QD score AUC for PGA-MAP-Elites, CMA-MEGA (ES), and CMA-MEGA (TD3,
ES).
As we did for QD score, we normalized QD score AUC by the maximum QD score.
The
ANOVA results were as follows:
• Interaction effect: F (12 , 80) = 17 .
55 , p < 0 .•
Simple main effects: – QD Ant: F (4 , 80) = 31 .
77 , p < 0 .
– QD Half-Cheetah: F (4 , 80)
= 89 .
38 , p < 0 .
– QD Hopper: F (4 , 80) = 82 .
34 , p < 0 .
– QD Walker: F (4 , 80) = 71 .
64 , p < 0 .
As the interaction and simple main effects were signiﬁcant, we performed pairwise comparisons (Table 16).
F. M EAN E LITE R OBUSTNESS A NALYSIS (S EC . G.2.4)
In this followup analysis, we hypothesized that MAP-Elites would have lower mean elite robustness than CMA-MEGA (ES) and CMA-MEGA (TD3, ES).
Thus, we performed a two-way ANOVA which compared mean elite robustness for MAP-Elites, CMA-MEGA (ES), and CMA-MEGA
(TD3, ES).
We normalized by the score range, i.e. max objective - min objective .
The ANOVA results were as follows:
• Interaction effect: F (12 , 80) = 8 .
75 , p < 0 .•
Simple main effects: – QD Ant: F (4 , 80)
= 3 .
17 , p = 0 .
– QD Half-Cheetah: F (4 , 80)
= 9 .
60 , p < 0 .
– QD Hopper: F (4 , 80) = 21 .
07 , p < 0 .
– QD Walker: F (4 , 80)
= 3 .
70 , p = 0 .
As the interaction and simple main effects were signiﬁcant, we performed pairwise comparisons (Table 17).
T a b l e 13 :
H 1C o m p a r
i ng
QD s c o r e b e t w ee n C
M AM E
GA ( E S ) a nd b a s e li n e s
QD
A n t
QD
H a l fC h ee t a h
Q D
H opp e r
QD
W a l k e r
A l go r it h m
A l g o r it h m
C M
A - M
E
GA ( E S )
P GA - M
A P - E lit e s 0 .
< 0 .
M EE S < 0 .
< 0 . < 0 .
M A
P - E lit e s 0 .
T a b l e 14 :
H 2C
o m p
a r
i ng
QD s c o r e b e t w ee n C
M AM E
GA ( T D 3 , E
S ) a nd b a s e li n e s
QD
A n t
QD
H a l fC h ee t a h
Q D
H opp e r
QD
W a l k e r
A l go r it h m
A l g o r it h m
C M
A - M
E
GA ( T D 3 , E
S ) P
GAM A
P - E lit e s 0 .
M EE S < 0 .
< 0 .
< 0 . < 0 .
M A
P - E lit e s 0 .
< 0 .
T a b l e 15 :
H 3C o m p a r
i ng
QD s c o r e b e t w ee n C
M AM E
GA ( E S ) a nd
C M
A - M
E
GA ( T D 3 , E
S )
QD
A n t
QD
H a l fC h ee t a h
Q D
H opp e r
QD
W a l k e r
A l go r it h m
A l g o r it h m
C M
A - M
E
GA ( E S )
C M
A - M
E
GA ( T D 3 ,
E S ) 0 .
< 0 .
T a b l e 16 :
C o m p a r i ng
QD s c o r e
AU
C b e t w ee n P
GAM E a nd
C M
A - M
E
GA v a r i a n t s
QD
A n t
QD
H a l fC h ee t a h
Q D
H opp e r
QD
W a l k e r
A l go r it h m
A l g o r it h m
P GA - M
A P - E lit e s
C M
A - M
E
GA ( E S ) 0 .
< 0 . < 0 .
C M
A - M
E
GA ( T D 3 ,
E S ) 0 .
T a b l e 17 :
C o m p a r i ng m ea n e lit e r obu s t n e ss b e t w ee n
M A
P - E lit e s a nd
C M
A - M
E
GA v a r i a n t s
QD
A n t
QD
H a l fC h ee t a h
Q D
H opp e r
QD
W a l k e r
A l go r it h m
A l g o r it h m
M A
P - E lit e s
C M
A - M
E
GA ( E S ) < 0 .
< 0 .
C M
A - M
E
GA ( T D 3 ,
E S ) < 0 .
< 0 . < 0 .
[Figure]
G R ESULTS G. A NALYSIS
To test our hypotheses [Figure] we conducted a two-way ANOVA which examined the effect of algorithm and environment on the QD score.
We note that the ANOVA requires QD scores to have the same scale, but each environment’s QD score has a different scale by default.
Thus, for this analysis, we normalized QD scores by dividing by each environment’s maximum QD score, deﬁned as grid cells * ( max objective - min objective ) (see Appendix D for these quantities).
We found a statistically signiﬁcant interaction between algorithm and environment on QD score,
F (12 , 80) = 16 .
82 , p < 0 .
Simple main effects analysis indicated that the algorithm had a signiﬁcant effect on QD score in each environment, so we ran pairwise comparisons (two-sided t-tests) with Bonferroni corrections (Appendix F).
Our results are as follows:
H1:
There is no signiﬁcant difference in QD score between CMA-MEGA (ES) and PGA-MAPElites in QD Ant and QD Half-Cheetah, but in QD Hopper and QD Walker, CMA-MEGA (ES) attains signiﬁcantly lower QD score than PGA-MAP-Elites.
CMA-MEGA (ES) achieves signiﬁcantly higher QD score than ME-ES in all environments except QD Hopper, where there is no signiﬁcant difference.
There is no signiﬁcant difference between CMA-MEGA (ES) and MAP-Elites in all domains except QD Hopper, where CMA-MEGA (ES) attains signiﬁcantly lower QD score.
H2:
In all environments, there is no signiﬁcant difference in QD score between CMA-MEGA (TD3, ES) and PGA-MAP-Elites.
CMA-MEGA (TD3, ES) achieves signiﬁcantly higher QD score than ME-ES in all environments.
CMA-MEGA (TD3, ES) achieves signiﬁcantly higher QD score than MAP-Elites in QD Half-Cheetah and Walker, with no signiﬁcant difference in QD Ant and QD Hopper.
H3:
CMA-MEGA (TD3, ES) achieves signiﬁcantly higher QD score than CMA-MEGA (ES) in QD Hopper and QD Walker, but there is no signiﬁcant difference in QD Ant and QD Half-Cheetah. G. D ISCUSSION We discuss how the CMA-MEGA variants differ from the baselines (Sec.
G.2.1-G.2.4) and how they differ from each other (Sec. G.2.5).
G.2. PGA-MAP-E LITES AND OBJECTIVE - MEASURE SPACE EXPLORATION
Of the CMA-MEGA variants, CMA-MEGA (TD3, ES) performed the closest to PGA-MAP-Elites, with no signiﬁcant QD score difference in any environment.
This result differs from prior work (Fontaine and Nikolaidis, 2021b) in QD benchmark domains, where CMA-MEGA outperformed
OG-MAP-Elites, a baseline DQD algorithm inspired by PGA-MAP-Elites.
We attribute this difference to the difﬁculty of exploring objective-measure space in the benchmark domains.
For example, the linear projection benchmark domain is designed to be “distorted” (Fontaine et al., 2020).
Values in the center of its measure space are easy to obtain with random sampling, while values at the edges are unlikely to be sampled.
Hence, high QD score arises from exploring measure space and ﬁlling the archive.
Since CMA-MEGA adapts its sampling distribution, it is able to perform this exploration, while OG-MAP-Elites remains “stuck” in the center of the measure space.
In contrast, as discussed in Sec.
5.1.1, it is relatively easy to ﬁll the archive in QDGym.
We see this empirically: in all environments, all algorithms achieve nearly 100% archive coverage, usually within the ﬁrst 250k evaluations (Fig. 4).
Hence, the best QD score is achieved by increasing the objective value of solutions after ﬁlling the archive.
PGA-MAP-Elites achieves this by optimizing half of its generated solutions with respect to its TD3 critic. The genetic operator likely further enhances the efﬁcacy of this optimization, by taking previously-optimized solutions and combining them to obtain high-performing solutions in other parts of the archive.
On the other hand, the CMA-MEGA variants place less emphasis on maximizing the performance of each solution, compared to PGA-MAP-Elites: in each trial, PGA-MAP-Elites takes 5 million objective gradient steps with respect to its TD3 critic, while the CMA-MEGA variants only compute 5k objective gradients, because they dedicate a large part of the evaluation to estimating the measure gradients.
This difference suggests a possible extension to CMA-MEGA (TD3, ES) in which solutions are optimized with respect to the TD3 critic before being evaluated in the environment.
G.2. PGA-MAP-E LITES AND OPTIMIZATION EFFICIENCY
While there was no signiﬁcant difference in the ﬁnal QD scores of CMA-MEGA (TD3, ES) and
PGA-MAP-Elites, CMA-MEGA (TD3, ES) was less efﬁcient than PGA-MAP-Elites in some environments.
For instance, in QD Hopper, PGA-MAP-Elites reached 1.5M QD score after 100k evaluations, but CMA-MEGA (TD3, ES) required 400k evaluations.
We can quantify optimization efﬁciency with QD score AUC , the area under the curve (AUC) of the QD score plot.
For a QD algorithm which executes N iterations and evaluates λ solutions per iteration, we deﬁne QD score AUC as a Riemann sum:
[Equation]
After computing QD score AUC, we ran statistical analysis similar to Sec.
G.1 and found CMAMEGA (TD3, ES) had signiﬁcantly lower QD score AUC than PGA-MAP-Elites in QD Ant and QD
Hopper.
There was no signiﬁcant difference in QD Half-Cheetah and QD Walker.
As such, while
CMA-MEGA (TD3, ES) obtained comparable ﬁnal QD scores to PGA-MAP-Elites in all tasks, it was less efﬁcient at achieving those scores in QD Ant and QD Hopper.
G.2. ME-ES AND ARCHIVE INSERTIONS
With one exception (CMA-MEGA (ES) in QD Hopper), both CMA-MEGA variants achieved signiﬁcantly higher QD score than ME-ES in all environments.
We attribute this result to the number of solutions each algorithm inserts into the archive.
Each iteration, ME-ES evaluates 200 solutions (Appendix C) but only inserts one into the archive, for a total of 5000 solutions inserted during each run.
Given that each archive has at least 1000 cells, ME-ES has, on average, 5 opportunities to insert a solution that improves each cell.
In contrast, the CMA-MEGA variants have 100 times more insertions.
Though the CMA-MEGA variants evaluate 200 solutions per iteration, they insert 100 of these into the archive.
This totals to 500k insertions per run, allowing the CMA-MEGA variants to gradually improve archive cells.
G.2. MAP-E LITES AND ROBUSTNESS
In most cases, both CMA-MEGA variants had signiﬁcantly higher QD score than MAP-Elites or no signiﬁcant difference, but in QD Hopper, MAP-Elites achieved signiﬁcantly higher QD score than CMA-MEGA (ES).
However, when we visualized solutions found by MAP-Elites, their performance was lower than the performance recorded in the archive.
The best MAP-Elites solution in
QD Hopper hopped forward a few steps and fell down, despite recording an excellent performance of 2,648.31 (see supplemental videos).
One explanation for this behavior is that since we only evaluate solutions for one episode before inserting into the archive, a solution with noisy performance may be inserted because of a single high-performing episode, even if it performs poorly on average.
Prior work (Nilsson and Cully, 2021) has also encountered this issue when running MAP-Elites with a directional variation operator (Vassiliades and Mouret, 2018) in QDGym, and has suggested measuring robustness as a proxy for how much noise is present in an archive’s solutions.
Robustness is deﬁned as the difference between the mean performance of the solution over n episodes (we use n = 10 ) and the performance recorded in the archive.
The larger (more negative) this difference, the more noisy and less robust the solution.
To compare the robustness of the solutions output by the CMA-MEGA variants and MAP-Elites, we computed mean elite robustness , the average robustness of all elites in each experiment’s ﬁnal archive.
We then ran statistical analysis similar to Sec.
G.1.
In all environments, both CMA-MEGA (ES) and CMA-MEGA (TD3, ES) had signiﬁcantly higher mean elite robustness than MAP-Elites (Appendix E & F).
Overall, though MAP-Elites achieves high QD score, its solutions are less robust.
G.2. CMA-MEGA VARIANTS AND GRADIENT ESTIMATES
In QD Hopper and QD Walker, CMA-MEGA (TD3, ES) had signiﬁcantly higher QD score than
CMA-MEGA (ES).
One potential explanation is that PyBullet Gym (and hence QDGym) augments rewards with reward shaping signals intended to promote optimal solutions for deep RL algorithms.
In prior work (Pagliuca et al., 2020), these signals led PPO (Schulman et al., 2017) to train successful walking agents, while they led OpenAI-ES into local optima.
For instance, OpenAI-ES trained agents which stood still so as to maximize only the reward signal for staying upright.
Due to these signals, TD3’s objective gradient seems more useful than that of OpenAI-ES in QD
Hopper and QD Walker.
In fact, the algorithms which performed best in QD Hopper and QD Walker were ones that calculated objective gradients with TD3, i.e. PGA-MAP-Elites and CMA-MEGA (TD3, ES).
Prior work (Pagliuca et al., 2020) found that rewards could be tailored for ES, such that OpenAI-ES outperformed PPO.
Extensions of our work could investigate whether there is a similar effect for QD algorithms, where tailoring the reward leads CMA-MEGA (ES) to outperform PGA-MAP-Elites and
CMA-MEGA (TD3, ES).
H A RCHIVE V ISUALIZATIONS
We visualize “median” archives in Fig.
5 and Fig.
To determine these median archives, we selected the trial which achieved the median QD score out of the 5 trials of each algorithm in each environment.
Fig.
5 visualizes heatmaps of median archives in QD Half-Cheetah and QD Walker, while Fig.
6 shows the distribution (histogram) of objective values for median archives in all environments.
Refer to the supplemental material for videos of how these ﬁgures develop across iterations.
[Figure]
These heatmaps have several notable features
First, we can see that MAP-Elites primarily discovers low-performing solutions.
Second, from looking at the heatmap videos, we can see that
PGA-MAP-Elites gradually improves the entire archive “all at once” — this happens because
PGA-MAP-Elites samples solutions uniformly from the archive and applies variations to them, so the entire archive appears to improve simultaneously.
Finally, again based on the heatmap videos, we see that the CMA-MEGA variants improve the archive with “paintbrush strokes.” This happens because the CMA-MEGA variants gradually move the solution point φ ∗ around the archive while generating solutions around it. [Figure] in Table 6.
Note that in some plots, the number of items overﬂows the y-axis bounds (e.g. ME-ES in QD Walker).
Refer to the supplemental material for a video of how these distributions develop across iterations.