CEM-RL: C OMBINING EVOLUTIONARY AND GRADIENT - BASED METHODS FOR POLICY SEARCH
fr
A BSTRACT
Deep neuroevolution and deep reinforcement learning (deep RL) algorithms are two popular approaches to policy search.
The former is widely applicable and rather stable but suffers from low sample efﬁciency.
By contrast, the latter is more sample efﬁcient but the most sample efﬁcient variants are also rather unstable and highly sensitive to hyper-parameter setting.
So far, these families of methods have mostly been compared as competing tools.
However, an emerging approach consists in combining them so as to get the best of both worlds.
Two previously existing combinations use either an ad hoc evolutionary algorithm or a goal exploration process together with the Deep Deterministic Policy Gradient ( DDPG ) algorithm, a sample efﬁcient off-policy deep RL algorithm.
In this paper, we propose a different combination scheme using the simple cross-entropy method ( CEM ) and Twin Delayed Deep Deterministic policy gradient ( TD 3), another off-policy deep RL algorithm which improves over DDPG .
We evaluate the resulting method, CEM - RL on a set of benchmarks classically used in deep RL.
We show that CEM - RL beneﬁts from several advantages over its competitors and offers a satisfactory trade-off between performance and sample efﬁciency
I NTRODUCTION
Policy search is the problem of ﬁnding a policy or controller maximizing some unknown utility function.
Recently, research on policy search methods has witnessed a surge of interest due to the combination with deep neural networks, making it possible to ﬁnd good enough continuous action policies in large domains.
From one side, this combination gave rise to the emergence of efﬁcient deep reinforcement learning (deep RL) techniques (Lillicrap et al., 2015; Schulman et al., 2015; 2017).
From the other side, evolutionary methods, and particularly deep neuroevolution methods applying evolution strategies (ESs) to the parameters of a deep network have emerged as a competitive alternative to deep RL due to their higher parallelization capability (Salimans & Kingma, 2016;
Conti et al., 2017; Petroski Such et al., 2017).
Both families of techniques have clear distinguishing properties.
Evolutionary methods are signiﬁcantly less sample efﬁcient than deep RL methods because they learn from complete episodes, whereas deep RL methods use elementary steps of the system as samples, and thus exploit more information (Sigaud & Stulp, 2018).
In particular, off-policy deep RL algorithms can use a replay buffer to exploit the same samples as many times as useful, greatly improving sample efﬁciency.
Actually, the sample efﬁciency of ESs can be improved using the ”importance mixing” mechanism, but a recent study has shown that the capacity of importance mixing to improve sample efﬁciency by a factor of ten is still not enough to compete with off-policy deep RL (Pourchot et al., 2018).
From the other side, sample efﬁcient off-policy deep RL methods such as the DDPG algorithm (Lillicrap et al., 2015) are known to be unstable and highly sensitive to hyper-parameter setting.
Rather than opposing both families as competing solutions to the policy search problem, a richer perspective consists in combining them so as to get the best of both worlds.
As covered in Section 2, there are very few attempts in this direction so far.
After presenting some background in Section 3, we propose in Section 4 a new combination method that combines the cross-entropy method ( CEM ) with DDPG or TD 3, an off-policy deep RL algorithm which improves over DDPG .
In Section 5, we investigate experimentally the properties of this CEMRL method, showing its advantages both over the components taken separately and over a competing approach.
Beyond the results of CEM - RL , the conclusion of this work is that there is still a lot of unexplored potential in new combinations of evolutionary and deep RL methods.
R ELATED WORK
Policy search is an extremely active research domain.
The realization that evolutionary methods are an alternative to continuous action reinforcement learning and that both families share some similarity is not new (Stulp & Sigaud, 2012a;b; 2013) but so far most works have focused on comparing them (Salimans et al., 2017; Petroski Such et al., 2017; Conti et al., 2017).
Under this perspective, it was shown in (Duan et al., 2016) that, despite its simplicity with respect to most deep RL methods, the Cross-Entropy Method ( CEM ) was a strong baseline in policy search problems.
Here, we focus on works which combine both families of methods.
Synergies between evolution and reinforcement learning have already been investigated in the context of the so-called Baldwin effect (Simpson, 1953).
This literature is somewhat related to research on meta-learning, where one seeks to evolve an initial policy from which a self-learned reinforcement learning algorithm will perform efﬁcient improvement (Wang et al., 2016; Houthooft et al., 2018; Gupta et al., 2018).
The key difference with respect to the methods investigated here is that in this literature, the outcome of the RL process is not incorporated back into the genome of the agent, whereas here evolution and reinforcement learning update the same parameters in iterative sequences.
Closer to ours, the work of Colas et al.
(2018) sequentially applies a goal exploration process ( GEP ) to ﬁll a replay buffer with purely exploratory trajectories and then applies DDPG to the resulting data.
The GEP shares many similarities with evolutionary methods, though it focuses on diversity rather than on performance of the learned policies.
The authors demonstrate on the Continuous Mountain
Car and HALF - CHEETAH - V 2 benchmarks that their combination, GEP - PG , is more sample-efﬁcient than DDPG , leads to better ﬁnal solutions and induces less variance during learning.
However, due to the sequential nature of the combination, the GEP part does not beneﬁt from the efﬁcient gradient steps of the deep RL part.
Another approach related to ours is the work of Maheswaranathan et al.
(2018), where the authors introduce optimization problems with a surrogate gradient, i.e. a direction which is correlated with the real gradient.
They show that by modifying the covariance matrix of an ES to incorporate the informations contained in the surrogate, a hybrid algorithm can be constructed.
They provide a thorough theoretical investigation of their procedure, which they experimentally show capable of outperforming both a standard gradient descent method and a pure ES on several simple benchmarks.
They argue that this method could be useful in RL, since surrogate gradients appear in Q-learning and actor-critic methods.
However, a practical demonstration of those claims remains to be performed.
Their approach resembles ours in that they use a gradient method to enhance an ES.
But a notable difference is that they use the gradient information to directly change the distribution from which samples are drawn, whereas we use gradient information on the samples themselves, impacting the distribution only indirectly.
The work which is the closest to ours is Khadka & Tumer (2018b).
The authors introduce an algorithm called ERL (for Evolutionary Reinforcement Learning), which is presented as an efﬁcient combination of a deep RL algorithm, DDPG , and a population-based evolutionary algorithm.
It takes the form of a population of actors, which are constantly mutated and selected in tournaments based on their ﬁtness.
In parallel, a single DDPG agent is trained from the samples generated by the evolutionary population.
This single agent is then periodically inserted into the population.
When the gradient-based policy improvement mechanism of DDPG is efﬁcient, this individual outperforms its evolutionary siblings, it gets selected into the next generation and draws the whole population towards higher performance.
Through their experiments, Khadka & Tumer demonstrate that this setup beneﬁts from an efﬁcient transfer of information between the RL algorithm and the evolutionary algorithm, and vice versa.
However, their combination scheme does not make proﬁt of the search efﬁciency of ESs.
This is unfortunate because ESs are generally efﬁcient evolutionary methods, and importance mixing can only be applied in their context to bring further sample efﬁciency improvement.
By contrast with the works outlined above, the method presented here combines CEM and TD 3 in such a way that our algorithm beneﬁts from the gradient-based policy improvement mechanism of
TD 3, from the better stability of ESs, and may even beneﬁt from the better sample efﬁciency brought by importance mixing, as described in Appendix B.
B ACKGROUND
In this section, we provide a quick overview of the evolutionary and deep RL methods used throughout the paper.
3. E VOLUTIONARY ALGORITHMS , EVOLUTION STRATEGIES AND EDA S
Evolutionary algorithms manage a limited population of individuals, and generate new individuals randomly in the vicinity of the previous elite individuals.
There are many variants of such algorithms, some using tournament selection as in Khadka & Tumer (2018b), niche-based selection or more simply taking a fraction of elite individuals, see Back (1996) for a broader view.
Evolution strategies can be seen as speciﬁc evolutionary algorithms where only one individual is retained from one generation to the next, this individual being the mean of the distribution from which new individuals are drawn.
More speciﬁcally, an optimum individual is computed from the previous samples and the next samples are obtained by adding Gaussian noise to the current optimum.
Finally, among
ESs, Estimation of Distribution Algorithms (EDAs) are a speciﬁc family where the population is represented as a distribution using a covariance matrix Σ (Larra˜naga & Lozano, 2001).
This covariance matrix deﬁnes a multivariate Gaussian function and samples at the next iteration are drawn according to Σ .
Along iterations, the ellipsoid deﬁned by Σ is progressively adjusted to the top part of the hill corresponding to the local optimum θ ∗ .
Various instances of EDAs, such as the CrossEntropy Method ( CEM ), Covariance Matrix Adaptation Evolution Strategy ( CMA - ES ) and PI 2 - CMA , are covered in Stulp & Sigaud (2012a;b; 2013).
Here we focus on the ﬁrst two.
3. T HE C ROSS -E NTROPY M ETHOD AND CMA - ES
The Cross-Entropy Method ( CEM ) is a simple EDA where the number of elite individuals is ﬁxed to a certain value
K e (usually set to half the population).
After all individuals of a population are evaluated, the K e ﬁttest individuals are used to compute the new mean and variance of the population, from which the next generation is sampled after adding some extra variance (cid:15) to prevent premature convergence.
In more details, each individual x i is sampled by adding Gaussian noise around the mean of the distribution µ , according to the current covariance matrix Σ , i.e. x i ∼ N ( µ, Σ) .
The problemdependent ﬁtness of these new individuals ( f i ) i =1 ,...,λ is computed, and the top-performing K e individuals, ( z i ) i
=1 ,...,K e are used to update the parameters of the distribution as follows:
[Equation]
where ( λ i ) i =1 ,...,K e are weights given to the individuals, commonly chosen with λ i = 1 K e or λ i
= log (1+ K e ) /i
(cid:80) Kei =1 log(1+ K e ) /i
(Hansen, 2016).
In the former, each individual is given the same importance, whereas the latter gives more importance to better individuals.
Similarly to CEM , Covariance Matrix Adaptation Evolution Strategy ( CMA - ES ) is an EDA where the number of elite individuals is ﬁxed to a certain value K e .
The mean and covariance of the new generation are constructed from those individuals.
However this construction is more elaborate than in CEM .
The top K e individuals are ranked according to their performance, and are assigned weights based on this ranking.
Those weights in turn impact the construction of the new mean and covariance.
Quantities called ”Evolutionary paths” are also used to accumulate the search directions of successive generations.
In fact, the updates in CMA - ES are shown to approximate the natural gradient, without explicitly modeling the Fisher information matrix (Arnold et al., 2011).
A minor difference between CEM and CMA - ES can be found in the update of the covariance matrix.
In its standard formulation, CEM uses the new estimate of the mean µ to compute the new Σ , whereas
CMA - ES uses the current µ (the one that was used to sample the current generation) as is the case in (2).
We used the latter as Hansen (2016) shows it to be more efﬁcient.
The algorithm we are using can thus be described either as CEM using the current µ for the estimation of the new Σ , or as CMA - ES without evolutionary paths.
The difference being minor, we still call the resulting algorithm CEM .
Besides, we add some noise in the form of (cid:15)
I to the usual covariance update to prevent premature convergence.
We choose to have an exponentially decaying (cid:15) , by setting an initial and a ﬁnal standard deviation, respectively σ init and σ end , initializing (cid:15) to σ init and updating (cid:15) at each iteration with (cid:15)
=
τ cem (cid:15) + (1 − τ cem ) σ end .
Note that, in practice Σ can be too large for computing the updates and sampling new individuals.
Indeed, if n denotes the number of actor parameters, simply sampling from Σ scales at least in
O ( n 2 . 3 ) , which becomes quickly intractable.
Instead, we constrain Σ to be diagonal.
This means that in our computations, we replace the update in (2) by
[Equation]
where the square of the vectors denote the vectors of the square of the coordinates.
3. DDPG AND TD
The Deep Deterministic Policy Gradient ( DDPG ) (Lillicrap et al., 2015) and Twin Delayed Deep
Deterministic policy gradient ( TD 3) (Fujimoto et al., 2018) algorithms are two off-policy, actorcritic and sample efﬁcient deep RL algorithms.
The DDPG algorithm suffers from instabilities partly due to an overestimation bias in critic updates, and is known to be difﬁcult to tune given its sensitivity to hyper-parameter settings.
The availability of properly tuned code baselines incorporating several advanced mechanisms improves on the latter issue (Dhariwal et al., 2017).
The TD 3 algorithm rather improves on the former issue, limiting the over-estimation bias by using two critics and taking the lowest estimate of the action values in the update mechanisms.
M ETHODS
As shown in Figure 1a, the CEM - RL method combines CEM with either DDPG or TD 3, giving rise to two algorithms named CEM - DDPG and CEM - TD 3.
The mean actor of the CEM population, referred to as π µ , is ﬁrst initialized with a random actor network.
A unique critic network Q π managed by TD 3 or DDPG is also initialized.
At each iteration, a population of actors is sampled by adding
Gaussian noise around the current mean π µ , according to the current covariance matrix Σ .
Half of the resulting actors are directly evaluated.
The corresponding ﬁtness is computed as the cumulative reward obtained during an episode in the environment.
Then, for each actor of the other half of the population, the critic is updated using this actor and, reciprocally, the actor follows the direction of the gradient given by the critic Q π for a ﬁxed number of steps.
The resulting actors are evaluated after this process.
The CEM algorithm then takes the top-performing half of the resulting global population to compute its new π µ and Σ .
The steps performed in the environment used to evaluate all actors in the population are fed into the replay buffer.
The critic is trained from that buffer pro rata to the quantity of new information introduced in the buffer at the current generation.
For instance, if the population contains 10 individuals, and if each episode lasts 1000 time steps, then 10 , 000 new samples are introduced in the replay buffer at each generation. The critic is thus trained Population Replay Buffer Environment
CEM DDPG/TD ReturnFitness ProvideExperience
Sample
Fill Buffer
Actor N
ApplyGradientSteps Actor
Actor N/
Evaluate
Critic
Learn Critic (a)
Population
Replay Buffer Environment
Selection
Mutation
Actor
Evolutionary Algorithm
DDPG
Actor
Actor N
ReturnFitness
ProvideExperience
PolicyGradient
Insert
Critic
Fill Buffer Evaluate (
[Figure]
for 10 , 000 mini-batches, which are divided into 2000 mini-batches per learning actor.
This is a common practice in deep RL algorithms, where one mini-batch update is performed for each step of the actor in the environment.
We also choose this number of steps ( 10 , 000 ) to be the number of gradient steps taken by half of the population at the next iteration.
A pseudo-code of CEM - RL is provided in Algorithm 1.
In cases where applying the gradient increases the performance of the actor, CEM beneﬁts from this increase by incorporating the corresponding actors in its computations.
By contrast, in cases where the gradient steps decrease performance, the resulting actors are ignored by CEM , which instead focuses on standard samples around π µ .
Those poor samples do not bring new insight on the current distribution of the CEM algorithm, since the gradient steps takes them away from the current distribution.
However, since all evaluated actors are ﬁlling the replay buffer, the resulting experience is still fed to the critic and the future learning actors, providing some supplementary exploration.
This approach generates a beneﬁcial ﬂow of information between the deep RL part and the evolutionary part.
Indeed, on one hand, good actors found by following the current critic directly improve the evolutionary population.
On the other hand, good actors found through evolution ﬁll the replay buffer from which the RL algorithm learns.
In that respect, our approach beneﬁts from the same properties as the ERL algorithm (Khadka &
Tumer, 2018b) depicted in Figure 1b.
But, by contrast with Khadka & Tumer (2018b), gradient steps are directly applied to several samples, and using the CEM algorithm makes it possible to use importance mixing, as described in Appendix B. Another difference is that in CEM - RL gradient steps are applied at each iteration whereas in ERL , a deep RL actor is only injected to the population from time to time.
One can also see from Figure 1 that, in contrast to ERL , CEM - RL does not use any deep RL actor.
Other distinguishing properties between ERL and CEM - RL are discussed in the light of empirical results in Section 5.2.
Finally, given that CMA - ES is generally considered as more sophisticated than CEM , one may wonder why we did not use CMA - ES instead of CEM into the CEM - RL algorithm.
Actually, the key contribution of CMA - ES with respect to CEM consists of the evolutionary path mechanism (see Section 3.2), but this mechanism results in some inertia in Σ updates, which resists to the beneﬁcial effect of applying RL gradient steps.
E XPERIMENTAL STUDY
In this section, we study the CEM - RL algorithm to answer the following questions:
• How does the performance of CEM - RL compare to that of CEM and TD 3 taken separately?
What if we remove the CEM mechanism, resulting in a multi-actor TD 3?
Draw the current population pop
N ( π
Σ) importance mixing (see Algorithmin Appendix
to Set the policy to i ] 10:
Initialize a with the weights of
Train π for
Reintroduce the weights of π in pop 14: 15: for i ← 1 to size : 16:
Set the current policy π to [ i ] 17: (ﬁtness f s ← evaluate( π ) 18:
Fill R with the collected experiences 19: top
) 21: end while • How does CEM - RL perform compared to ERL ? What are the main factors explaining the difference between both algorithms? Additionally, in Appendices B to E, we investigate other aspects of the performance of CEM - RL such as the impact of importance mixing, the addition of action noise or the use of the tanh non-linearity
. 5. E XPERIMENTAL SETUP
In order to investigate the above questions, we evaluate the corresponding algorithms in several continuous control tasks simulated with the MUJOCO physics engine and commonly used as policy search benchmarks: HALF - CHEETAH - V 2, HOPPER - V 2, WALKER 2 D - V 2, SWIMMER - V 2 and ANTV 2 (Brockman et al., 2016).
We implemented CEM - RL with the P Y T ORCH library 1 .
We built our code around the DDPG and
TD 3 implementations given by the authors of the TD 3 algorithm 2 .
For the ERL implementation, we used one given by the authors 3 .
Unless speciﬁed otherwise, each curve represents the average over 10 runs of the corresponding quantity, and the variance corresponds to the 68% conﬁdence interval for the estimation of the mean.
In all learning performance ﬁgures, dotted curves represent medians and the x-axis represents the total number of steps actually performed in the environment, to highlight potential sample efﬁciency effects, particularly when using importance mixing (see Appendix B).
Architectures of the networks are described in Appendix A. Most TD 3 and DDPG hyper-parameters were reused from Fujimoto et al.
The only notable difference is the use of tanh non linearities instead of RELU in the actor network, after we spotted that the latter performs better than the former on several environments.
We trained the networks with the Adam optimizer (Kingma &
Ba, 2014), with a learning rate of 1 e − 3 for both the actor and the critic.
The discount rate γ was set to 0.99, and the target weight τ to 5 e − 3 .
All populations contained 10 actors, and the standard deviations σ init , σ end and the constant τ cem of the CEM algorithm were respectively set to 1 e − 3 , 1 e − 5 and 0.95.
Finally, the size of the replay buffer was set to 1 e 6 , and the batch size to 100 .
5. R ESULTS
We ﬁrst compare CEM - TD 3 to TD 3, TD 3 and a multi-actor variant of TD 3, then CEM - RL to ERL based on several benchmarks.
A third section is devoted to additional results which have been rejected in appendices to comply with space constraints.
5.2. C OMPARISON TO CEM , TD 3 AND A MULTI - ACTOR TD
[Figure]
In this section, we compare CEM - TD 3 to three baselines: our variant of CEM , TD 3 and a multi-actor variant of TD 3.
For TD 3 and its multi-actor variant, we report the average of the score of the agent over 10 episodes for every 5000 steps performed in the environment.
For CEM and CEM - TD 3, we report after each generation the average of the score of the new average individual of the population over 10 episodes.
From Figure 2, one can see that CEM - TD 3 outperforms CEM and TD 3 on HALFCHEETAH - V 2, HOPPER - V 2 and WALKER 2 D - V 2.
On most benchmarks, CEM - TD 3 also displays slightly less variance than TD 3.
Further results in Appendix G show that on ANT - V 2, CEM - TDoutperforms CEM and is on par with TD 3.
More surprisingly, CEM outperforms all other algorithms on SWIMMER - V 2, as covered in Appendix E.
One may wonder whether the good performance of CEM - TD 3 mostly comes from its ”ensemble method” nature (Osband et al., 2016).
Indeed, having a population of actors improves exploration and stabilizes performances by ﬁltering out instabilities that can appear during learning.
To answer this question, we performed an ablative study where we removed the CEM mechanism.
We considered a population of 5 actors initialized as in CEM - TD 3, but then just following the gradient given by the TD 3 critic.
This algorithm can be seen as a multi-actor TD 3 where all actors share the same critic.
We reused the hyper-parameters described in Section 5.2.
From Figure 2, one can see that
CEM - TD 3 outperforms more or less signiﬁcantly multi-actor TD 3 on all benchmarks, which clearly suggests that the evolutionary part contributes to the performance of CEM - TD 3.
As a summary, Table 1 gives the ﬁnal performance of methods compared in this Section.
We conclude that CEM - TD 3 is generally superior to CEM , TD 3 and multi-actor TD 3.
More precisely, in environments where TD 3 provides a useful gradient information, CEM - TD 3 enhances CEM by accelerating updates towards better actors, and it enhances TD 3 by reducing variance in the learning process. [Table]
[Figure]
C OMPARISON TO ERL
In this section, we compare CEM - RL to ERL .
The ERL method using DDPG rather than TD 3, we compare it to both CEM - DDPG and CEM - TD 3.
This makes it possible to isolate the effect of the combination scheme from the improvement brought by TD 3 itself.
Results are shown in Figure 3.
We let ERL learn for the same number of steps as in Khadka & Tumer, namely 2 millions on HALFCHEETAH - V 2 and SWIMMER - V 2, 4 millions on HOPPER - V 2, 6 millions on ANT - V 2 and 10 millions on WALKER 2 D - V 2.
However, due to limited computational resources, we stop learning with both
CEM - RL methods after 1 million steps, hence the constant performance after 1 million steps.
Our results slightly differ from those of the ERL paper (Khadka & Tumer, 2018b).
We explain this difference by two factors.
First, the authors only average their results over 5 different seeds, whereas we used 10 seeds.
Second, the released implementation of ERL may be slightly different from the one used to produce the published results 4 , raising again the reproducibility issue recently discussed in the reinforcement learning literature (Henderson et al., 2017).
Figure 3 shows that after performing 1 million steps, both CEM - RL methods outperform ERL on
HALF - CHEETAH - V 2, HOPPER - V 2 and WALKER 2 D - V 2.
We can also see that CEM - TD 3 outperforms
CEM - DDPG on WALKER 2 D - V 2.
On ANT - V 2, CEM - DDPG and ERL being on par after 1 million steps, we increased the number of learning steps in CEM - DDPG to 6 millions.
The corresponding results are shown in Figure 10b in Appendix G. Results on SWIMMER - V 2 are covered in Appendix E. [Table]
One can see that, beyond outperforming ERL , CEM - TD 3 outperforms CEM - DDPG on most benchmarks, in terms of ﬁnal performance, convergence speed, and learning stability.
This is especially true for hard environments such as WALKER 2 D - V 2 and ANT - V 2.
The only exception in SWIMMERV 2, as studied in Appendix E.
Table 2 gives the ﬁnal best results of methods used in this Section.
The overall conclusion is that
CEM - RL generally outperforms ERL .
5.2. A DDITIONAL RESULTS
In this section [Table] we outline the main messages arising from further studies that have been rejected in appendices in order to comply with space constraints.
In Appendix B [Table] we investigate the inﬂuence of the importance mixing mechanism over the evolution of performance, for CEM and CEM - RL .
Results show that importance mixing has a limited impact on the sample efﬁciency of CEM - TD 3 on the benchmarks studied here, in contradiction with results from Pourchot et al.
(2018) obtained using various standard evolutionary strategies.
The fact that the covariance matrix Σ moves faster with CEM - RL may explain this result, as it prevents the reuse of samples.
In Appendix C, we analyze the effect of adding Gaussian noise to the actions of CEM - TD 3.
Unlike what Khadka & Tumer (2018b) suggested using ERL , we did not ﬁnd any conclusive evidence that action space noise improves performance with CEM - TD 3.
This may be due to the fact that, as further studied in Appendix D, the evolutionary algorithm in ERL tends to converge to a unique individual, hence additional noise is welcome, whereas evolutionary strategies like CEM more easily maintain some exploration.
Indeed, we further investigate the different dynamics of parameter space exploration provided by the ERL and CEM - TD 3 algorithms in Appendix D. Figure 6 and 7 show that the evolutionary population in ERL tends to collapse towards a single individual, which does not happen with the CEM population due to the sampling method.
In Appendix E, we highlight the fact that, on the SWIMMER - V 2 benchmark, the performance of the algorithms studied in this paper varies a lot from the performance obtained on other benchmarks.
The most likely explanation is that, in SWIMMER - V 2, any deep RL method provides a deceptive gradient information which is detrimental to convergence towards efﬁcient actor parameters.
In this particular context, ERL better resists to detrimental gradients than CEM - RL , which suggests to design a version of ERL using CEM to improve the population instead of its ad hoc evolutionary algorithm.
Finally, in Appendix F, we show that using a tanh non-linearity in the architecture of actors often results in signiﬁcantly stronger performance than using RELU .
This strongly suggests performing ”neural architecture search” (Zoph & Le, 2016; Elsken et al., 2018) in the context of RL.
C ONCLUSION AND FUTURE WORK
We advocated in this paper for combining evolutionary and deep RL methods rather than opposing them.
In particular, we have proposed such a combination, the CEM - RL method, and showed that in most cases it was outperforming not only some evolution strategies and some sample efﬁcient offpolicy deep RL algorithms, but also another combination, the ERL algorithm.
Importantly, despite being mainly an evolutionary method, CEM - RL is competitive to the state-of-the-art even when considering sample efﬁciency, which is not the case of other deep neuroevolution methods (Salimans et al., 2017; Petroski Such et al., 2017).
Beyond these positive performance results, our study raises more fundamental questions.
First, why does the simple CEM algorithm perform so well on the SWIMMER - V 2 benchmark?
Then, our empirical study of importance mixing did not conﬁrm a clear beneﬁt of using it, neither did the effect of adding noise on actions.
We suggest explanations for these phenomena, but nailing down the fundamental reasons behind them will require further investigations.
Such deeper studies will also help understand which properties are critical in the performance and sample efﬁciency of policy search algorithms, and deﬁne even more efﬁcient policy search algorithms in the future.
As suggested in Section 5.2.3, another avenue for future work will consist in designing an ERL algorithm based on CEM rather than on an ad hoc evolutionary algorithm.
Finally, given the impact of the neural architecture on our results, we believe that a more systemic search of architectures through techniques such as neural architecture search (Zoph & Le, 2016; Elsken et al., 2018) may provide important progress in performance of deep policy search algorithms.
A CKNOWLEDGMENTS
This work was supported by the European Commission, within the DREAM project, and has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement N o 640891.
We would like to thank Thomas Pierrot for fruitful discussions.
A A RCHITECTURE OF THE NETWORKS
Our network architectures are very similar to the ones described in Fujimoto et al.
In particular, the size of the layers remains the same.
The only differences resides in the non-linearities.
We use tanh operations in the actor between each layer, where Fujimoto et al.
use RELU and we use leaky RELU in the critic, where Fujimoto et al.
use simple RELU .
Reasons for this choice are presented in Appendix F.
[Table]
B I MPORTANCE MIXING
Importance mixing is a speciﬁc mechanism designed to improve the sample efﬁciency of evolution strategies.
It was initially introduced in Sun et al.
and consisted in reusing some samples from the previous generation into the current one, to avoid the cost of re-evaluating the corresponding policies in the environment.
The mechanism was recently extended in Pourchot et al.
(2018) to reusing samples from any generation stored into an archive.
Empirical results showed that importance sampling can improve sample efﬁciency by a factor of ten, and that most of these savings just come from using the samples from the previous generation, as performed by the initial mechanism.
A pseudo-code of the importance mixing mechanism is given in Algorithm 2.
In CEM , importance mixing is implemented as described in (Pourchot et al., 2018).
By contrast, some adaptation is required in CEM - RL .
Actors which take gradient steps can no longer be regarded as sampled from the current distribution of the CEM algorithm.
We thus choose to apply importance mixing only to the half of the population which does not receive gradient steps from the RL critic.
In practice, only actors which do not take gradient steps are inserted into the actor archive and can be replaced with samples from previous generations.
From Figure 4, one can see that in the CEM case, importance mixing introduces some minor instability, without noticeably increasing sample efﬁciency.
On HALF - CHEETAH - V 2, SWIMMER - Vand WALKER 2 D - V 2, performance even decreases when using importance mixing.
For CEM - RL , the effect varies greatly from environment to environment, but the gain in sample reuse is almost null Require: p ( z, θ new ) : current probability density function (pdf), p ( z, θ old ) : old pdf, g old : old generation 1: g new ← ∅ 2: for i ← 1 to N 3: Draw rand1 and rand2 uniformly from [0 , 1] 4: Let z i be the i th individual of the old generation g old 5: if min (1 , p ( z i , θ new ) p ( z i , θ old ) ) > rand1: 6:
Append z i to the current generation g new 7: Draw z (cid:48) i according to the current pdf p ( ., θ new ) 8: if max (0 , 1 − p ( z (cid:48) i , θ old ) p ( z (cid:48)
i , θ new ) ) > rand2: 9:
Append z (cid:48)
i to the current generation g new 10: size = | g new | 11: if size ≥ N : go to12: if size > N : remove a randomly chosen sample 13: if size < N : ﬁll the generation sampling from p ( ., θ new ) 14: return g new
[Figure]
as well, though an increase in performance can be seen on SWIMMER - V 2.
The latter fact is consistent with the ﬁnding that the gradient steps are not useful in this environment (see Appendix E).
On HOPPER - V 2 and HALF - CHEETAH - V 2, results with and without importance mixing seem to be equivalent.
On WALKER 2 D - V 2, importance mixing decreases ﬁnal performance.
On ANT - V 2, importance mixing seems to accelerate learning in the beginning, but ﬁnal performances are equivalent to those of CEM - RL .
Thus importance mixing seems to have a limited impact in CEM - TD 3.
This conclusion seems to contradict the results obtained in Pourchot et al.
This may be due to different things.
First, the dimensions of the search spaces in the experiments here are much larger than those studied in Pourchot et al.
(2018), which might deteriorate the estimation of the covariance matrices when samples are too correlated.
On top of this, the MUJOCO environments are harder than [Table] the ones used in Pourchot et al.
In particular, we can see from Figure 2 that CEM is far from solving the environments over one million steps.
Perhaps a study over a longer time period would make importance mixing relevant again.
Besides, by reusing old samples, the importance mixing mechanism somehow hinders exploration (since we evaluate less new individuals), which might be detrimental in the case of MUJOCO environments.
Finally, and most importantly, the use of RL gradient steps accelerates the displacement of the covariance matrix, resulting in fewer opportunities for sample reuse.
C E FFECT OF ACTION NOISE
In Khadka & Tumer (2018b) [Table] the authors indicate that one reason for the efﬁciency of their approach is that the replay buffer of DDPG gets ﬁlled with two types of noisy experiences.
On one hand, the buffer gets ﬁlled with noisy interactions of the DDPG actor with the environment.
This is usually referred to as action space noise .
On the other hand, actors with different parameters also ﬁll the buffer, which is more similar to parameter space noise (Plappert et al., 2017).
In CEM - RL , we only use parameter space noise, but it would also be possible to add action space noise.
To explore this direction, each actor taking gradient steps performs a noisy episode in the environment.
We report ﬁnal results after 1 million steps in Table 5.
Learning curves are available in Figure 5.
[Figure]
Unlike what Khadka & Tumer (2018b) suggested, we did not ﬁnd any conclusive evidence that action space noise improves performance.
In CEM - TD 3, the CEM part seems to explore enough of the action space on its own.
It seems that sampling performed in CEM results in sufﬁcient exploration and performs better than adding simple Gaussian noise to the actions.
This highlights a difference between using an evolutionary strategy like CEM and an evolutionary algorithm as done in ERL .
Evolutionary algorithms tend to converge to a unique individual whereas evolutionary strategies more easily maintain some exploration.
These aspects are further studied in Appendix D
[Table]
D P ARAMETER SPACE EXPLORATION IN CEM - RL AND ERL
In this section, we highlight the difference in policy parameter update dynamics in CEM - RL and
ERL .
Figure 6 displays the evolution of the ﬁrst two parameters of actor networks during training with CEM - RL and ERL on HALF - CHEETAH - V 2.
For ERL , we plot the chosen parameters of the DDPG actor with a continuous line, and represent those of the evolutionary actors with dots.
For CEM - RL , we represent the chosen parameters of sampled actors with dots, and the gradient steps based on the
TD 3 critic with continuous lines.
The same number of dots is represented for both algorithms.
[Figure]
One can see that, in ERL the evolutionary population tends to be much less diverse that in CEM - RL .
There are many redundancies in the parameters (dots with the same coordinates), and the population seems to converge to a single individual.
On the other hand, there is no such behavior in CEM - RL where each generation introduces completely new samples.
As a consequence, parameter space exploration looks better in the CEM - RL algorithm.
To further study this loss of intra-population diversity in ERL , we perform 10 ERL runs and report in
Figure 7 an histogram displaying the distribution of the population-wise similarity with respect to the populations encountered during learning.
We measure this similarity as the average percentage of parameters shared between two different individuals of the said population.
The results indicate that around 55% of populations encountered during a run of ERL display a population-similarity of above 80% .
One can also see the difference in how both methods use the gradient information of their respective deep RL part.
In the case of ERL , the parameters of the population concentrate around those of the DDPG actor.
Each 10 generations, its parameters are introduced into the population, and since
DDPG is already efﬁcient alone on HALF - CHEETAH - V 2, those parameters quickly spread into the population.
Indeed, according to Khadka & Tumer (2018b), the resulting DDPG actor is the elite of the population 80% of the time, and is introduced into the population 98% of the time.
This integration is however passive: the direction of exploration does not vary much after introducing the DDPG agent.
CEM - RL integrates this gradient information differently.
The short lines emerging from dots, which represent gradient steps performed by half of the actors, act as scouts.
Once CEM becomes aware of better solutions that can be found in a given direction, the sampling of the next population is modiﬁed so as to favor this promising direction.
CEM is thus pro-actively exploring in the good directions it has been fed with.
E T HE CASE OF THE SWIMMER - V 2 BENCHMARK
Experiments on the SWIMMER - V 2 benchmark give results that differ a lot from the results on other benchmarks, hence they are covered separately here.
Figure8a shows that CEM outperforms TD 3,
CEM - TD 3, multi-actor TD 3.
Besides, as shown in Figure 8b, ERL outperforms CEM - DDPG , which itself outperforms CEM - TD 3.
[Figure]
All these ﬁndings seem to show that being better at RL makes you worse at SWIMMER - V 2.
The most likely explanation is that, in SWIMMER - V 2, any deep RL method provides a deceptive gradient information which is detrimental to convergence towards efﬁcient actor parameters.
This conclusion could already be established from the results of Khadka & Tumer (2018b), where the evolution algorithm alone produced results on par with the ERL algorithm, showing that RL-based actors were just ignored.
In this particular context, the actors using TD 3 gradient being deteriorated by the deceptive gradient effect, CEM - RL is behaving as a CEM with only half a population, thus it is less efﬁcient than the standard CEM algorithm.
By contrast, ERL better resists than CEM - RL to the same issue.
Indeed, if the actor generated by DDPG does not perform better than the evolutionary population, then this actor is just ignored, and the evolutionary part behaves as usual, without any loss in performance.
In practice, Khadka & Tumer note that on SWIMMER - V 2, the DDPG actor was rejected 76% of the time.
Finally, by comparing CEM and ERL from Figure 8a and Figure 8b, one can conclude that on this benchmark, the evolutionary part of ERL behaves on par with CEM alone.
This is at odds with premature convergence effects seen in the evolutionary part of ERL , as studied in more details in Appendix D. From all these insights, the SWIMMER - V 2 environment appears particularly interesting, as we are not aware of any deep RL method capable of solving it quickly and reliably.
F U SING THE RELU OR TANH NON - LINEARITY
In this section, we explore the impact on performance of the type of non-linearities used in the actor of CEM - TD 3.
Table 5 reports the results of CEM - TD 3 using RELU non-linearities between the linear layers, instead of tanh .
[Figure]
Figure 9 displays the learning performance of CEM - TD 3 and CEM on benchmarks, using either the
RELU or the tanh nonlinearity in the actors.
Results indicate that on some benchmarks, changing from tanh to RELU can cause a huge drop in performance.
This is particularly obvious in the
ANT - V 2 benchmark, where the average performance drops by 46 % .
Figure 9(f) shows that, for the
CEM algorithm on the SWIMMER - V 2 benchmark, using RELU also causes a 60 % performance drop.
As previously reported in the literature (Henderson et al., 2017), this study suggests that network architectures can have a large impact on performance.
G A DDITIONAL RESULTS ON ANT - V
Figure 10 represents the learning performance of CEM , TD 3, multi-actor TD 3, CEM - DDPG and CEMTD 3 on the ANT - V 2 benchmark.
It is discussed in the main text.
[Figure]