Effective Diversity in Population-Based Reinforcement Learning
Abstract
Exploration is a key problem in reinforcement learning, since agents can only learn from data they acquire in the environment.
With that in mind, maintaining a population of agents is an attractive method, as it allows data be collected with a diverse set of behaviors.
This behavioral diversity is often boosted via multi-objective loss functions.
However, those approaches typically leverage mean ﬁeld updates based on pairwise distances, which makes them susceptible to cycling behaviors and increased redundancy.
In addition, explicitly boosting diversity often has a detrimental impact on optimizing already fruitful behaviors for rewards.
As such, the reward-diversity trade off typically relies on heuristics.
Finally, such methods require behavioral representations, often handcrafted and domain speciﬁc.
In this paper, we introduce an approach to optimize all members of a population simultaneously.
Rather than using pairwise distance, we measure the volume of the entire population in a behavioral manifold, deﬁned by task-agnostic behavioral embeddings.
In addition, our algorithm Diversity via Determinants (DvD), adapts the degree of diversity during training using online learning techniques.
We introduce both evolutionary and gradient-based instantiations of DvD and show they effectively improve exploration without reducing performance when better exploration is not required.
1. Introduction
Reinforcement Learning (RL) considers the problem of an agent taking actions in an environment to maximize total (discounted/expected) reward (Sutton and Barto, 1998).
An agent can typically only learn from experience acquired in the environment, making exploration crucial to learn high performing behaviors.
Training a population of agents provides is a promising approach to gathering a diverse range of experiences, often with the same (wall-clock) training time (Recht et al., 2011;
Li et al., 2019).
Population-based methods are particularly prominent in the Neuroevolution community (Stanley et al., 2019), but have recently been of increased interest in RL (Jung et al., 2020; Liu et al., 2019; Khadka and Tumer, 2018; Jung et al., 2020; Khadka et al., 2019; Pourchot and
Sigaud, 2019).
One particularly exciting class of neuroevolution methods is Quality Diversity (QD, (Pugh et al., 2016)) where algorithms explicitly seek high performing, yet diverse behaviors (Cully et al., 2015).
However, these methods have a few key shortcomings.
Typically, each agent is optimized with a mean ﬁeld assumption, only considering its individual contribution to the population’s joint reward-diversity objective.
Consequently, cycles may arise, whereby different members of the population constantly switch between behaviors.
This may prevent any single agent exploiting a promising behavior.
This phenomenon motivates the MAP-Elites algorithm (Mouret and
Clune, 2015; Cully et al., 2015), whereby only one solution may lie in each quadrant of a pre-deﬁned space.
This pre-deﬁned space is a common issue with QD (Kistemaker and Whiteson, 2011).
Behavioral characterizations (BCs) often have to be provided to the agent, for example, in locomotion it is common to use the ﬁnal ( x, y ) coordinates.
As such, automating the discovery of BCs is an active research area (Gaier et al., 2020).
In RL, gradients are usually taken with respect to the actions taken by an agent for a set of sampled states.
This provides a natural way to embed a policy behavior (Jung et al., 2020;
Hong et al., 2018;
Doan et al., 2019).
Incidentally, the geometry induced by such embeddings has also been used in popular trust region algorithms (Schulman et al., 2018; 2015).
In this paper we formalize this approach and deﬁne behavioral embeddings as the actions taken by policies.
We measure the diversity of the entire population as the volume of the inter-agent kernel (or similarity) matrix, which we show has theoretical beneﬁts compared to pairwise distances.
In our approach agents are still optimizing for their local rewards and this signal is a part of their hybrid objective that also takes into account the global goal of the populationits diversity.
However, we note that it is remains a challenge to set the diversity-reward trade off.
If misspeciﬁed, we may see fruitful behaviors disregarded.
Existing methods either rely on a (potentially brittle) hyperparameter, or introduce an annealing-based heuristic (Conti et al., 2018).
To ensure diversity is promoted effectively , our approach is to adapt the diversity-reward objective using Thompson sampling (Russo et al., 2017; Russo and Van Roy, 2016).
This provides us with a principled means to trade-off reward vs. diversity through the lens of multi-armed bandits (Slivkins, 2019).
Combining these insights, we introduce Diversity via Determinants (DvD).
DvD is a general approach, and we introduce two implementations: one building upon Evolution Strategies, DvDES, which is able to discover diverse solutions in multimodal environments, and an off-policy RL algorithm, DvDTD3, which leads to greater performance on the challenging
Humanoid task.
Crucially, we show DvD still performs well when diversity is not required.
This paper is organized as follows.
(1) We begin by introducing main concepts in Sec. 2 and providing strong motivation for our DvD algorithm in Sec. 3.
(2) We then present our algorithm in Sec. 4.
In Sec.
5 we discuss related work in more detail.
In Sec.
6 we provide empirical evidence of the effectiveness of DvD. (5) We conclude and explain broader impact in Sec. ??
and provide additional technical details in the Appendix (including ablation studies and proofs).
2. Preliminaries
A Markov Decision Process ( MDP ) is a tuple ( S , A , P , R) .
Here S and A stand for the sets of states and actions respectively, such that for s t , s t +1 ∈ S and a t ∈ A : P( s t +1 | s t , a t ) is the probability that the system/agent transitions from s t to s t +1 given action a t and r t = r
( s t , a t , s t +1 ) is a reward obtained by an agent transitioning from s t to s t +1 via a t .
A policy π θ : S → A is a (possibly randomized) mapping (parameterized by θ ∈ R d ) from S to A .
Policies π θ are typically represented by neural networks, encoded by parameter vectors θ ∈ R d (their ﬂattened representations).
Model free RL methods are either on-policy, focusing on gradients of the policy from samples (Schulman et al., 2015; 2018), or off-policy, targeting learning a value function (Mnih et al., 2013; Fujimoto et al., 2018; Haarnoja et al., 2018).
In on-policy RL, the goal is to optimize parameters θ of π θ such that an agent deploying policy π θ in the environment given by a ﬁxed MDP maximizes R ( τ ) =
(cid:80) Tt =1 r t , the total (expected/discounted) reward over a rollout time-step horizon T (assumed to be ﬁnite).
The typical objective is as follows:
[Equation]
where P ( τ | θ )
=
ρ ( s 0 ) (cid:81)
Tt =1 P ( s t
+1 | s t , a t )
π θ ( a t | s t ) , for initial state probability ρ ( s 0 ) and transition dynamics
P ( s t +1 | s t , a t ) , which is often deterministic.
Policy gradient (PG) methods (Schulman et al., 2018; 2015), consider objectives of the following form:
[Equation]
which can be approximated with samples (in the action space) by sampling a t ∼ π θ ( a t
| s t ) .
An alternative formulation is Evolution Strategies (ES, (Salimans et al., 2017)), which has recently shown to be effective (Such et al., 2017; Choromanski et al., 2018; Mania et al., 2018).
ES methods optimize Equation 1 by considering
J ( π θ ) to be a blackbox function F : R d → R as taking as input parameters θ ∈ R d of a policy π θ and outputting R ( τ ) .
One beneﬁt of this approach is potentially achieving deep exploration (Plappert et al., 2018;
Fortunato et al., 2018).
A key beneﬁt of ES methods is they naturally allow us to maintain a population of solutions, which has been used to boost diversity.
Novelty search methods (Conti et al., 2018; Lehman and Stanley, 2011) go further and explicitly augment the loss function with an additional term, as follows:
[Equation]
where λ > 0 is the reward-diversity trade-off.
We assume the policy and environment are deterministic, so τ θ is the only possible trajectory, and d
[Equation]
for some l -dimensional behavioral mapping BC
[Equation]
This formulation has been shown to boost exploration, and as such, there have been a variety of attempts to incorporate novelty-based metrics into RL algorithms (Jung et al., 2020;
Doan et al., 2019; Hong et al., 2018).
Ideally, we would guide optimization in a way which would evenly distribute policies in areas of the embedding space, which correspond to high rewards.
However, the policy embeddings (BCs) used are often based on heuristics which may not generalize to new environments (Kistemaker and Whiteson, 2011).
In addition, the single sequential updates may lead high performing policies away from an improved reward (as is the case in the Humanoid experiment in (Conti et al., 2018)).
Finally, cycles may become present, whereby the population moves from one area of the feature space to a new area and back again (Mouret and Clune, 2015).
Below we address these issues, and introduce a new objective which updates all agents simultaneously.
3. Diversity via Determinants
Here we introduce our task agnostic embeddings and formalize our approach to optimize for population-wide diversity.
3.1. Task Agnostic Behavioral Embeddings
Many popular policy gradient algorithms (Schulman et al., 2015; 2018) consider a setting whereby a new policy π θ t + is optimized with a constraint, typically of the following form:
[Equation]
for δ ≥ 0 and where KL represents the KL-divergence.
This requirement can be considered as a constraint on the behavior of the new policy (Pacchiano et al., 2019).
Despite the remarkable success of these algorithms, there has been little consideration for action-based behavior embeddings for novelty search methods.
Inspired by this approach, we choose to represent policies as follows:
Deﬁnition 3.1.
Let θ i be a vector of neural network parameters encoding a policy π θ i and let S be a ﬁnite set of states.
The Behavioral Embedding of θ i is deﬁned as:
[Equation]
This approach allows us to represent the behavior of policies in vectorized form, as φ : θ → R l , where l = | a | × N , where | a | is the dimensionality of each action a ∈ A and
N is the total number of states.
When N is the number of states in a ﬁnite MDP, the policies are determimistic and the embedding is the as in Deﬁnition 3.1, we have:
[Equation]
where θ i , θ j are vectorized parameters.
In other words, the policies are the same since they always take the same action in every state of the ﬁnite MDP.
Note that this does not imply that θ i
= θ j .
3.2. Joint Population Update
Equipped with this notion of a behavioral embedding, we can now consider a means to compare two policies.
Consider a smooth kernel k , such that k ( x 1 , x 2 ) ≤ 1 , for x 1 , x 2 ∈ R d .
A popular choice of kernel is the squared exponential (SE), deﬁned as follows:
[Equation]
for some length-scale l > 0 .
Now moving back to policy embeddings, we extend our previous analysis to the kernel or similarity between two embeddings, as follows:
[Equation]
We consider two policies to be behaviorally orthogonal if k ( φ ( θ i ) , φ ( θ j ))
= 0 .
With a ﬂexible way of measuring inter-policy similarity at hand, we can deﬁne the populationwide diversity as follows:
Deﬁnition 3.2.
(Population Diversity) Consider a ﬁnite set of M policies, parameterized by Θ = { θ 1 , ..., θ M } , with θ i ∈ R d .
We denote Div(Θ) def = det( K ( φ ( θ it ) , φ ( θ jt ))
Mi,j =1 ) = det( K ) , where K : R l ×
R l → R is a given kernel function.
Matrix K is positive semideﬁnite since all principal minors of det( K ) are nonnegative.
This formulation is heavily inspired by Determinantal Point
Processes (DPPs, (Kulesza and Taskar, 2012)), a mechanism which produces diverse subsets by sampling proportionally to the determinant of the kernel matrix of points within the subset.
From a geometric perspective, the determinant of the kernel matrix represents the volume of a parallelepiped spanned by feature maps corresponding to the kernel choice.
We seek to maximize this volume, effectively “ﬁlling” the feature (or behavior) space.
Now consider the DvD loss function, as follows:
[Equation]
where λ t ∈ (0 , 1) is the trade-off between reward and diversity.
This fundamentally differs from Equation 3, since we directly optimize for Θ t rather than separately considering { θ i } Mi =1 .
Theorem 3.3.
Let M be a ﬁnite, tabular MDP with ˜ M ≥
M distinct optimal policies { π i } ˜ Mi
=1 all achieving a cumulative reward of R and such that the reward value R ( π ) of any suboptimal policy π satisﬁes R ( π ) + ∆ <
R for some ∆ > 0 .
There exists λ t > 0 , such that the objective in
Equation 8 can only be maximized if the population contains
M distinct optimal solutions.
The proof of Theorem 3.3 is in the Appendix (Sec. 9), where we also show that for the case of the squared exponential kernel, the ﬁrst order approximation to the determinant is in fact related to the mean pairwise L2-distance.
However, for greater population sizes, this ﬁrst order approximation is zero, implying the determinant comes from higher order terms.
We have now shown theoretically that our formulation for diversity can recover diverse high performing solutions.
This result shows the importance of utilizing the determinant to measure diversity rather than the pairwise distance.
Using the determinant to quantify diversity prevents the undesirable clustering phenomenon, where a population evolves to a collection of conjugate classes.
To illustrate this point, consider a simple scenario, where all M agents are partitioned into k clusters of size Mk each for k
= o ( M ) .
By increasing the distance between the clusters one can easily make the novelty measured as an average distance between agents’ embedded policies as large as desired, but that is not true for the corresponding determinants which will be zero if the similarity matrix is low rank.
Furthermore, even if all pairwise distances are large, the determinants can be still close to zero if spaces where agents’ high-dimensional policy embeddings live can be accurately approximated by much lower dimensional ones.
Standard methods are too crude to measure novelty in such a way (see: Fig. 1).
Next we provide a simple concrete example to demonstrate this phenomenon.
An Illustrative Example: Tabular MDP
[Equation]
+ −+ −+
Consider the simple MDP in Fig.
There are four states S
=
{ s 1 , s 2 , s 3 , s 4 } , each has three actions, A = { - 1 , 0 , 1 } corresponding to left , down and right respectively.
In addition, there are ﬁve terminal states, three of which achieve the maximum reward ( +1 ).
Let φ ∗ =
{ φ ( θ i ) } 5 i =
= { [ - 1 , - 1] , [ - 1 , 1] , [0 , 0] , [1 , - 1] , [1 , 1] } , be the set of optimal policies.
If we have a population of ﬁve agents, each achieving a positive reward, the determinant of the 5 × 5 kernel matrix is only > 0 if the population of agents is exactly φ ∗ .
This may seem trivial, but note the same is not true for the pairwise distance.
If we let d (Θ) = 1 n (cid:80) ni
=1 (cid:80) j>i
|| φ ( θ i ) − φ ( θ j ) ||
2 , then φ ∗ does not maximize d .
One such example which achieves a higher value would be φ (cid:48)
= { [ - 1 , - 1] , [ - 1 , 1] , [1 , - 1] , [1 , 1] , [1 , 1] } .
See the following link for a colab demonstration of this example: https://bit.ly/2XAlirX .
4. DvD Algorithm 4.1. Approximate Embeddings
In most practical settings the state space is intractably or inﬁnitely large.
Therefore, we must sample the states { s i } ni =1 , where n < N , and compute the embedding as an expectation as follows:
[Equation]
In our experiments we choose to randomly sample the states s , which corresponds to frequency weights.
Other possibilities include selecting diverse ensembles of states via
DPP-driven sampling or using probabilistic models to select less frequently visited states.
We explore each of these options in the experiments where we show the representative power of this action-based embedding is not overly sensitive to these design choices (see: Fig. 7).
However, it remains a potential degree of freedom for future improvement in our method, potentially allowing a far smaller number of states to be used.
4.2. Adaptive Exploration
Optimizing Equation 8 relies on a user-speciﬁed degree of priority for each of the two objectives ( λ t ).
We formalize this problem through the lens of multi-armed bandits, and adaptively select λ t such that we encourage favoring the reward or diversity at different stages of optimization.
Speciﬁcally, we use Thompson Sampling (Thompson, 1933;
Srinivas et al., 2010; Agrawal and Goyal, 2012; Russo et al., 2017; Russo and Van Roy, 2016).
Let K = { 1 , · · · , K } denote a set of arms available to the decision maker (learner) who is interested in maximizing its expected cumulative reward.
The optimal strategy for the learner is to pull the arm with the largest mean reward.
At the beginning of each round the learner produces a sample mean from its mean reward model for each arm, and pulls the arm from which it obtained the largest sample.
After observing the selected arm’s reward, it updates its mean reward model.
Let π it be the learner’s reward model for arm i at time t .
When t =
the learner initializes each of its mean reward models to prior distributions { π i 0 } Ki =1 .
At any other time t > 0 , the learner starts by sampling mean reward candidates µ i ∼ π
it − 1 and pulling the arm: i t = arg max i ∈K µ i .
After observing a true reward sample r t from arm i t , the learner updates its posterior distribution for arm i t .
All the posterior distributions over arms
i (cid:54) =
i t remain unchanged.
In this paper we make use of a Bernoulli model for the reward signal corresponding to the two arms ( λ = 0 , λ = 0 . 5) .
At any time t , the chosen arm’s sample reward is the indicator variable r t =
( R t +1 > R t ) where R t denotes the reward observed at θ t and R t +1 that at θ t +1 .
We make a simplifying stationarity assumption and disregard the changing nature of the arms’ means in the course of optimization.
We use beta distributions to model both the priors and the posteriors of the arms’ means.
For a more detailed description of the speciﬁcs of our methodology please see the Appendix (Sec. 10.1).
We believe this adaptive mechanism could also be used for count-based exploration methods or intrinsic rewards (Schmidhuber, 2010), and note very recent work using a similar approach to vary exploration in off-policy methods (Schaul et al., 2019; Badia et al., 2020) and model-based
RL (Ball et al., 2020).
Combining these insights, we obtain the DvD algorithm.
Next we describe two practical implementations of DvD.
4.3. DvD-ES Algorithm
At each timestep, the set of policies Θ t = { θ it } Mi =1 are simultaneously perturbed, with rewards computed locally and diversity computed globally.
These two objectives are combined to produce a blackbox function with respect to Θ t .
At every iteration we sample Mk Gaussian perturbation vectors { g mi } m =1 ,...,M i =1 ,...,k .
We use two partitionings of this Mk -element subset that illustrate our dual objectivehigh local rewards and large global diversity.
The ﬁrst partitioning assigns to m th worker a set { g m 1 , ..., g mk } .
These are the perturbations used by the worker to compute its local rewards.
The second partitioning splits { g mi } m =1 ,...,M i =1 ,...,k into subsets: D i = { g 1 i , ..., g Mi } .
Instead of measuring the contribution of an individual g mi to the diversity, we measure the contribution of the entire D i .
This motivates the following deﬁnition of diversity:
[Equation]
Thus, the DvD-ES gradient update is the following:
[Equation]
where σ > 0 is the smoothing parameter (Nesterov and
Spokoiny, 2017; Salimans et al., 2017), k is the number of
ES-sensings, η is the learning rate, and the embeddings are computed by sampling states from the most recent iteration.
4.4. DvD-TD3 Algorithm
It is also possible to compute analytic gradients for the diversity term in Equation 8.
This means we can update policies with respect to the joint diversity using automatic differentiation.
Lemma 4.1.
The gradient of log (det( K )) with respect to Θ = θ 1 , · · · , θ M equals:
[Equation]
, where
[Equation]
The proof of this lemma is in the Appendix, Sec. 9.2.
Inspired by (Jung et al., 2020), we introduce DvD-TD3, using multiple policies to collect data for a shared replay buffer.
This is done by dividing the total data collection by the number of policies.
When optimizing the policies, we use an augmented loss function, as in Equation 8, and make use of the samples in the existing batch for the embedding.
5. Related Work
Neuroevolution methods (Stanley et al., 2019), seek to maximize the reward of a policy through approaches strongly motivated by natural biological processes.
They typically work by perturbing a policy, and either computing a gradient (as in Evolution Strategies) or selecting the top performing perturbations (as in Genetic Algorithms).
The simplicity and scalability of these methods have led to their increased popularity in solving RL tasks (Salimans et al., 2017; Choromanski et al., 2018; 2019;
Conti et al., 2018; Such et al., 2017).
Neuroevolution methods often make use of behavioral representations (Conti et al., 2018; Gajewski et al., 2019).
In (Conti et al., 2018) it is proposed to use of a population of agents, each of which would seek to jointly maximize the reward and difference/ novelty in comparison to other policies, quantiﬁed as the mean pairwise distance.
Another approach, Evolvabilty ES (Gajewski et al., 2019) seeks to learn policies which can quickly adapt to a new task, by maximizing the variance or entropy of behaviors generated by a small perturbation.
The MAP-Elites (Mouret and Clune, 2015) algorithm is conceptually similar to ours, the authors seek to ﬁnd quality solutions in differing dimensions.
However, these dimensions need to be pre-speciﬁed, whereas our method can be considered a learned version of this approach.
To the best of our knowledge, only one recent neuroevolutionary approach (Jackson and Daley, 2019) uses the actions of a policy to represent behaviors, albeit in a genetic context over discrete actions.
There has recently been interest in unsupervised learning of diverse behaviors (Eysenbach et al., 2019; Hartikainen et al., 2020).
These methods are similar in principle to novelty search without a reward signal, but instead focus on diversity in behaviors deﬁned by the states they visit.
Another class of algorithms making use of behavioral representations (Gupta et al., 2018) focuses on meta learning in the behavioral space, however they require pre-training on similar tasks in order to learn a new one.
A meta learning approach from (Rusu et al., 2019) proposes using a latent generative representation of model parameters, which could be thought of as a behavioral embedding of the policy.
Finally, (Wang and Liu, 2019) propose a functional approach for learning diversiﬁed parameters.
As an iterative method, it is still subject to the undesirable cycling phenomenon.
6. Experiments
Here evaluate DvD-ES and DvD-TD3 in a variety of challenging settings.
6.1. Finding Diverse Solutions with DvD-ES
We compare DvD-ES against vanilla ES (referred to as ES), as well as NSR-ES from (Conti et al., 2018), both of which updates each population member sequentially.
For both
DvD and NSR-ES, we use the same embedding, withrandomly selected states.
We use this for all all DvD-ES experiments.
We parameterize our policies with two hidden layer neural networks, with tanh activations (more details are in the
Appendix, Section 8.2).
All x-axes are presented in terms of iterations.
Comparisons are made fair by dividing the number of iterations for two sequential algorithms (ES and
NSR-ES) by the population size.
All experiments made use of the ray (Moritz et al., 2017) library for parallel computing, with experiments run on a 32 -core machine.
[Figure]
Exploration We begin with a simple environment, whereby a two dimensional point agent is given a reward equal to the negative distance away from a goal.
The agent is separated from its goal by the wall (see: Fig. 3a)).
We ran ten seeds, with hyperparameters described in detail in the
Appendix (Table 4) and with population of size M = 5 .
As we see, both vanilla ES and NSR-ES fail to get past the wall (a reward of -800), yet DvD is able to solve the environment for all 10 seeds .
Multi-Modal Environments
A key attribute of a QD algorithm is the ability to learn a diverse set of high performing solutions.
This is often demonstrated qualitatively, through videos of learned gaits, and thus hard to scientiﬁcally prove.
To test this we create environments where an agent can be rewarded for multiple different behaviors, as is typically done in multi-objective RL (Yang et al., 2019).
Concretely, our environments are based on the Cheetah and
Ant , where we assign rewards for both Forward and Backward tasks, which commonly used in meta-RL (Finn et al., 2017; Rothfuss et al., 2019).
We can then evaluate the population of agents on both individual tasks, to quantitatively evaluate the diversity of the solutions.
[Figure]
In both settings we used a population size of M = 3 , which is sufﬁcient to learn both tasks.
In Fig.
4 we see that DvD is able to learn both modes in both environments.
For the
Cheetah , the Backward task appears simpler to learn, and with no diversity term vanilla ES learns this task quickly, but subsequently performs poorly in the Forward task (with inverse reward function).
For Ant , the noise from two separate tasks makes it impossible for vanilla ES to learn at all.
Single Mode Environments Now we consider the problem of optimizing tasks which have only one optimal solution or at least, with a smaller distance between optimal solutions (in the behavioral space), and an informative reward function.
In this case, overly promoting diversity may lead to worse performance on the task at hand, as seen in
NSR-ES ((Conti et al., 2018), Fig. 1.(c)).
We test this using four widely studied continuous control tasks from OpenAI
Gym .
In all cases we use a population size of M = 5 , we provide additional experimental details in the Appendix (see
Section 8.2).
[Figure]
As we see in Fig.
5, in all four environments NSR-ES fails to train (which is consistent with the vanilla Humanoid experiment from (Conti et al., 2018)) in contrast to DvD that shows only a minimal drop in performance vs. the vanilla
ES method which is solely focusing on the reward function.
This promising result implies that we gain the beneﬁt of diversity without compromising on tasks, where it is not required.
We note that DvD outperforms ES in Walker2d , which is known to have a deceptive local optimum atinduced by the survival bonus (Mania et al., 2018).
These experiments enable us also to demonstrate the cyclic behaviour that standard novelty search approaches suffer from (see: Section 1).
NSR-ES often initially performs well, before subsequently abandoning successful behaviors in the pursuit of novelty.
6.2. Teaching a Humanoid to Run with DvD-TD
We now evaluate DvD-TD3 on the challenging Humanoid environment from the Open AI Gym (Brockman et al., 2016).
The Humanoid - v2 task requires a signiﬁcant degree of exploration, as there is a well-known local optimum at 5000 , since the agent is provided with a survival bonus of 5 per timestep (Mania et al., 2018).
We train DvD-TD3 with M = 5 agents, where each agent has its own neural network policy, but a shared Q-function.
We benchmark against both a single agent ( M = 1 ), which is vanilla TD3, and then what we call ensemble TD3 (ETD3) where M =
but there is no diversity term.
We initialize all methods with 25 , 000 random timesteps, where we set λ t = 0 for DvD-TD3.
We train each for a total of one million timesteps, and repeat for 7 seeds.
The results are shown in Fig.
[Figure]
As we see, DvD-TD3 achieves better sample efﬁciency, as well as stronger maximum performance.
For comparison, the median best agents for each algorithm were: DvDTD3: 6091, E-TD3: 5654 and TD3: 5727.
This provides t-statistics of 2.35 and 2.29 for DvD-TD3 vs. E-TD3 and
TD3, using Welch’s unequal variances t-test.
Both of these are statistically signiﬁcant ( p < 0 . 05 ).
As far as we are aware, DvD-TD3 is the ﬁrst algorithm to get over 6000 on
Humanoid - v2 in one million timesteps.
This means the agent has achieved forward locomotion, rather than simply standing still (at 5000 ).
Previous results for Soft Actor Critic (SAC) have only reached 6000 at three million timesteps (Haarnoja et al., 2018; Ciosek et al., 2019).
We note that while we maintain full policy networks for each member of the population, it is likely possible to implement the population with multiple heads (as in (Osband et al., 2016)).
Space was not an issue in our case, but may be more important when using DvD for larger tasks (e.g. training from pixels).
7. Conclusion
In this paper we introduced DvD, a novel method for promoting diversity in population-based methods for reinforcement learning.
DvD addresses the issue of cycling by utilizing a joint population update via determinants corresponding to ensembles of policies’ embeddings.
Furthermore, DvD adapts the reward-diversity trade off in an online fashion, which facilitates ﬂexible and effective diversity.
We demonstrated across a variety of challenging tasks that DvD not only ﬁnds diverse, high quality solutions but also manages to maintain strong performances in one-good-solution settings.
8. Additional Experiment Details 8.1. Ablation Studies
Here we seek to analyze the sensitivity of DvD to design choices made, in order to gain conﬁdence surrounding the robustness of our method.
How sensitive is the embedding to the choice of states?
One of the crucial elements of this work is task-agnostic behavioral embedding.
Similar to what has been used in all trust-region based policy gradient algorithms (Schulman et al., 2015; 2018), we use a concatenation of actions to represent policy behavior.
In all experiments we used this behavioral embedding for both DvD and NSR, thus rendering the only difference between the two methods to be adaptive vs. ﬁxed diversity and joint vs. individual updates.
However, there is still a question whether the design choices we made had an impact on performance.
As such, we conducted ablation studies over the number n of states chosen and different strategies for choosing states (for n = 20 ) used to construct behavioral embeddings.
[Figure]
As we see in Fig.
7, both the number of states chosen and the mechanism for choosing them appear to have minimal impact on the performance.
In all cases the point escapes the local maximum (of moving into the area surrounded by walls) and
Swimmer reaches high rewards ( > 300 ).
Do we need to adapt?
In Fig.
8 we evaluate the effectiveness of the adaptive mechanism, bny running ﬁve experiments with the DvD algorithm with ﬁxed λ .
While notably we still see strong performance from the joint diversity score (vs. NSR
ES in Fig 5), it is clear the adaptive mechanism boosts performance in all cases.
[Figure]
Choice of DPP kernel For this work, we used the Squared Exponential (or RBF) kernel for all experiments.
This decision was made due to its widespread use in the machine learning community and many desirable properties.
However, in order to assess the quality of our method it is important to consider the sensitivity to the choice of kernel.
In Fig.
9 and Table 8.1 we show the result from 5 seeds for a variety of kernels.
As we see, in almost all cases the performance is strong, and similar to the Squared Exponential kernel used in the main experiments.
[Figure]
[Table]
What if the population size is much larger than the number of modes?
We also studied the relation between population size and the number of modes learned by the population.
Both the tasks considered here have two modes, and intuitively a population of size M = 3 should be capable of learning both of them.
However, we also considered the case for M =(Fig. 10), and found that in fact a larger population was harmful for the performance.
This leads to the interesting future work of adapting not only the degree of diversity but the size of the population.
[Figure]
8.2. Hyperparameters 8.3. DvD-ES
In Table 2 we show the method for generating behavioral embeddings used for NSR-ES and our method.
We used these settings across all experiments, and did not tune them.
Number of states corresponds to the number of states used for the embedding, which is the concatenation of the actions from a policy on those states.
State selection refers to the mechanism for selecting the states from the buffer of all states seen on the previous iteration.
The update frequency is how frequently we re-sample states to use for the embeddings.
We do not want this to be too frequent or we will have differnet embeddings every iteration simply as a result of the changing states.
In Table 3 we show the hyprparameters used for the multi-modal experiments.
We note the main difference between the two environments is the Ant requires more ES sensings (300 vs. 100) and a larger neural network (64-64 vs. 32-32) than the
[Table]
Cheetah .
These settings were used for all three algorithms studied.
[Table]
In Table 4 we show the hyperparameters used for the uni-model and deceptive reward experiments.
The main difference is the size of the neural network for point and Swimmer is smaller (16-16 vs. 32-32) since these environments have smaller state and action dimensions than the others.
In addition, we note the horizon H for the point is smaller, as 50 timesteps is sufﬁcient to reach the goal.
These settings were used for all three algorithms considered.
[Table]
8.4. DvD-TD
Our TD3 implementation comes from an open source repository.
Since it is a new library from a close collaborator, we will provide the link once anonymity has been lifted.
All parameters are the same as in the original TD3 paper (Fujimoto et al., 2018), apart from neural network architectures and choice of learning rate and batch size, which mirror SAC (Haarnoja et al., 2018).
9. Theoretical Results 9.1. Proof of Theorem 3.
Proof.
We start by recalling that Div(Θ)
= det( K ) .
Let α 1 , · · · , α M be the eigenvalues of K .
Since K is PSD, α i ≥ 0 for all i .
The following bounds hold:
[Equation]
Inequality ( i ) follows since K is a PSD matrix.
Inequality ( ii ) is a consequence of the AM-GM inequality and Equality ( iii ) follows because all the diagonal entries of K equal 1 .
Let { ˜ π i } Mi =1 be a set of policies with at least one suboptimal policy and parametrized by ˜Θ t .
Wlog let R (˜ π 1 ) + ∆ < R .
The following holds:
[Equation]
Now observe that for any set of optimal policies { π i } Mi =1 parametrised by Θ t the objective in Equation 8 satisﬁes:
[Equation]
Therefore if λ t < ∆ , then:
[Equation]
Thus we conclude the objective in Equation 8 can only be maximised when all policies parametrised by Θ t are optimal.
Since λ t > 0 and Div(Θ t ) is nonzero only when { π i } Mi =1 are distinct, and there exist at least M distinct solutions, we conclude that
whenever 0 < λ t < ∆ , the maximizer for Equation 8 corresponds to M distinct optimal policies.
9.2. Proof of Lemma 3.
In the case of deterministic behavioral embeddings, it is possible to compute gradients through the whole objective.
Notice that det( K ) is then differentiable as a function of the policy parameters.
In the case of trajectory based embeddings, differentiating through the determinant is not that simple.
Actually it may make sense in this case to use a log(det( K )) score instead.
This is because of the following lemma:
Lemma 9.1.
The gradient of log (det( K )) with respect to θ = θ 1 , · · · , θ M equals:
[Equation]
Where Φ( θ ) =
Φ( θ 1 ) · · · Φ( θ M )
Proof.
We start with:
[Equation]
This is a known result 1 .
Consequently,
[Equation]
Each of the other gradients can be computed exactly.
9.3. Determinants vs. Distances
In this section we consider the following question.
Let k ( x , y ) =
exp( −(cid:107) x − y (cid:107) 2 ) .
And K ∈ R M × M be the kernel matrix corresponding to M agents and resulting of computing the Kernel dot products between their corresponding embeddings { x 1 , · · · , x M } : how-to-calculate-the-gradient-of-log-det-matrix-inverse
Theorem 9.2.
For M ≤ 3 , the ﬁrst order approximation of det( K ) is proportional to the sum of the pairwise distances between { x 1 , · · · , x M } .
For M > 3 this ﬁrst order approximation equals 0 .
Proof.
Consider the case of a population size M = [Equation] some policy embedding φ i and the exponentiated quadratic kernel.
In this setting
[Equation]
the diversity, measured by the determinant of the kernel (or similarity ) matrix is as follows:
[Equation]
So if we take k to be the squared exponential kernel:
[Equation]
Recall that for | x | << 1 small enough, exp( x ) ≈ 1 + x .
Substituting this approximation in the expression above we see:
[Equation]
which is essentially the mean pairwise l 2 distance.
What can we say about these differences (e.g. exp vs. not)?
Does this same difference generalize to M > 3 ?
Approximation for M >
Recall that for a matrix A ∈ R M × M , the determinant can be written as:
[Equation]
Where S M denotes the symmetric group over M elements.
Lets identify A i,j = exp (cid:16) −
[Equation]
x
[Equation]
(cid:17) .
Notice that for any σ ∈ S M , we have the following approximation:
[Equation]
Whenever for all i, j ∈ [ M ] the value of (cid:107) x i − x j (cid:107) 2 is small.
We are interested in using this termwise approximation to compute an estimate of det ( A ) .
Plugging the approximation in
Equation 12 into the formula for the determinant yields the following:
[Equation]
= (cid:88) σ ∈ S M sign ( σ ) (cid:124) (cid:123)(cid:122) (cid:125)
I − (cid:88) σ ∈ S M sign ( σ )
M (cid:88) i =
(cid:107) x
i − x σ ( i ) (cid:107) (cid:124) (cid:123)(cid:122) (cid:125) II
Term I equals zero as it is the sum of all signs of the permutations of S n and n > 1 .
In order to compute the value of II we observe that by symmetry:
[Equation]
For some B ∈ R .
We show that B = 0 for M > 3 .
Let’s consider the set B 1 , 2 of permutations σ ∈ S M for which the sum
[Equation]
contains the term (cid:107) x 1 − x 2 (cid:107) 2.
Notice that B = 12 (cid:80) σ ∈ B 1 , 2 sign ( σ ) .
Let’s characterize B 1 , 2 more exactly.
Recall every permutation σ ∈ S M can be thought of as a product of cycles.
For more background on the cycle decomposition of permutations see (Stanley, 2011).
The term (cid:107) [Equation] appears whenever the cycle decomposition of σ contains a transition of the form 1 → 2 or 1 ← 2 .
It appears twice if the cycle decomposition of σ has the cycle corresponding to a single transposition 1 ↔ 2 .
Let −→ B 1 , 2 be the set of permutations containing a transition of the form 1 → 2 (and no transition of the form 1 ← 2 )
←− B 1 ,be the set of permutations containing a transition of the form 1 ← 2 (and no transition of the form 1 → 2 ) and ﬁnally ←→ B 1 ,be the set of permutations containing the transition 1 ↔ 2 .
Notice that:
[Equation]
We start by showing that for M > 3 , O 3 = 0 .
Indeed, any σ ∈ ←→ B 1 , 2 has the form σ = (1 , 2) σ (cid:48) where σ (cid:48) is the cycle decomposition of a permutation over 3 , · · · , M .
Consequently sign ( σ ) =
− sign ( σ (cid:48) ) .
Iterating over all possible σ (cid:48)
∈ S M −permutations over [3 , · · · , M ] yields the set ←→ B 1 , 2 and therefore:
[Equation]
The last equality holds because M − 2 ≥ 2 .
We proceed to analyze the terms O 1 and O 2 .
By symmetry it is enough to focus on O 1 .
Let c be a ﬁxed cycle structure containing the transition 1 → 2 .
Any σ ∈ −→ B 1 , 2 containing c can be written as σ = cσ (cid:48) where σ (cid:48) is a permutation over the remaining elements of { 1 , · · · , M }\ c and therefore sign ( σ )
= ( − 1)
| c |− 1 sign ( σ (cid:48) ) .
Let −→
B c 1 , 2 be the subset of −→ B 1 , 2 containing c .
Notice that:
[Equation]
Let’s analyze O c 1 :
[Equation]
[Equation]
If |{ 1 , · · · , M }\ c | ≥ 2 this quantity equals zero.
Otherwise it equals ( − 1) | c |− 1 .
We recognize two cases, when |{ 1 , · · · , M }\ c | = 0 and when |{ 1 , · · · , M }\ c | = 1 .
The following decomposition holds 2 :
[Equation]
A simple combinatorial argument shows that:
[Equation]
Roughly speaking this is because in order to build a size M cycle containing the transition 1 → 2 , we only need to decide on the positions of the next M − 2 elements, which can be shufﬂed in M − 2 ways.
Similarly, a simple combinatorial argument shows that:
[Equation]
A similar counting argument yields this result.
First, there are M − 2 ways of choosing the element that will not be in the cycle.
Second, there are ( M − 3)!
ways of arranging the remaining elements to ﬁll up the M − 3 missing slots of the M −sized cycle c .
We conclude that in this case O 1 = 0 .
This result implies two things:
When M ≤ 3 .
If the gradient of the embedding vectors is sufﬁciently small, the determinant penalty is up to ﬁrst order terms equivalent to a pairwise distances score.
This may not be true if the embedding vector’s norm is large.
When M >
The determinant diversity penalty variability is given by its higher order terms
It is therefore not equivalent to a pairwise distances score.
10. Extended Background
For completeness, we provide additional context for existing methods used in the paper.
10.1. Thompson Sampling
Let’s start by deﬁning the Thompson Sampling updates for Bernoulli random variables.
We borrow the notation from
Section 4.2.
Let K = { 1 , · · · , K } be a set of Bernoulli arms with mean parameters { µ i } Ki =1 .
Denote by π it the learner’s mean reward model for arm i at time t .
We let the learner begin with an independent prior belief over each µ i , which we denote π io .
These priors are beta-distributed with parameters α oi = 1 , β oi = 1 :
[Equation]
Where Γ denotes the gamma function.
It is convenient to use beta distributions because of their conjugacy properties.
It can be shown that whenever we use a beta prior, the posterior distribution is also a beta distribution.
Denote α ii , β ti as the values of parameters
α i , β i at time t .
Let i t be the arm selected by Thomson Sampling, as we explained in main body, at time t .
After observing reward r t ∈ { 0 , 1 } the arms posteriors are updated as follows:
( α t +1 i , β t +1 i ) = (cid:40) ( α ti + r t , β ti
+ (1 − r t )) if i = i t ( α ti , β ti ) o.w.
10.2. Reinforcement Learning Algorithms 10.2.1. E VOLUTION S TRATEGIES
ES methods cast RL as a blackbox optimization problem.
Since a blackbox function F : R d → R may not even be differentiable, in practice its smoothed variants are considered.
One of the most popular ones, the Gaussian smoothing (Nesterov and Spokoiny, 2017) F σ of a function F is deﬁned as:
[Equation]
where σ > 0 is a hyperparameter quantifying the smoothing level.
Even if F is nondifferentiable, we can easily obtain stochastic gradients for F σ .
The gradient of the Gaussian smoothing of F is given by the formula:
[Equation]
This equation leads to several Monte Carlo gradient estimators used successfully in Evolution Strategies (ES, (Salimans et al., 2017; Choromanski et al., 2018))
algorithms for blackbox optimization in RL.
Consequently, it provides gradient-based policy update rules such as:
[Equation]
(14) where R i =
F ( θ t + σ g i ) is the reward for perturbed policy θ t + σ g
i and η > 0 stands for the step size.
In practice Gaussian independent perturbations can be replaced by dependent ensembles to further reduce variance of the
Monte Carlo estimator of ∇ F σ ( θ ) via quasi Monte Carlo techniques.
10.2.2. N OVELTY S EARCH
In the context of population-based Reinforcement Learning, one prominent approach is the class of novelty search methods for RL (Lehman and Stanley, 2008; 2011; Conti et al., 2018).
The NSR-ES algorithm (Conti et al., 2018) maintains a metapopulation of M agents, and at each iteration t sequentially samples and an individual member θ mt .
This agent is perturbed with samples g m 1 , · · · , g m k ∼ N (0 , I d ) , and then the rewards R m i =
F ( θ mt + σ g m i ) and embeddings Φ( θ mt + σ g m i ) are computed in parallel.
The novelty of a perturbed policy is then computed as the mean Euclidean distance of its embedding to the embeddings Φ( θ it ) of the remaining members of the population for i (cid:54)
= m .
In order to update the policy, the rewards and novelty scores are normalized (denoted (cid:98) R mi and (cid:98) N mi ), and the policy is updated as follows:
[Equation]
[Equation]
where the novelty weight λ > 0 is a hyperparameter.
A value λ = 0 corresponds to the standard ES approach (see: Eq.14) whereas the algorithm with λ = 1 neglects the reward-signal and optimizes solely for diversity.
A simple template of this approach, with a ﬁxed population size, appears in Alg. 1.
Despite encouraging results on hard exploration environments, these algorithms contain several ﬂaws.
They lack a rigorous means to evaluate the diversity of the population as a whole, this means that M policies may fall into N < M conjugacy classes, leading to the illusion of a diverse population on a mean pairwise Euclidean distance metric.
Input: : learning rate η , noise standard deviation σ , number of policies to maintain M , number of iterations T , embedding Φ , novelty weight λ .
Initialize: { θ 10 , . . .
, θ M 0 } .
for t = 0 , 1 , . . .
, T − 1 do
Let (cid:98) R k and (cid:98) N k be the normalized reward and novelty for each perturbation g k .