C ONTINUOUS C ONTROL W ITH E NSEMBLE D EEP D ETERMINISTIC P OLICY G RADIENTS
Mateusz mk.olko@student.uw.edu krolikowski2@student uw ´Swi ˛atkowski jakub.swiatkowski@mimuw.edu pl Google Research marcina@google.com
Łukasz Kuci´nski Polish Academy of Sciences lkucinski@impan.pl Piotr Miło´s Polish Academy of Sciences pmilos@impan.pl
A BSTRACT
The growth of deep reinforcement learning (RL) has brought multiple exciting tools and methods to the field.
This rapid expansion makes it important to understand the interplay between individual elements of the RL toolbox.
We approach this task from an empirical perspective by conducting a study in the continuous control setting.
We present multiple insights of fundamental nature, including: an average of multiple actors trained from the same data boosts performance; the existing methods are unstable across training runs, epochs of training, and evaluation runs; a commonly used additive action noise is not required for effective training; a strategy based on posterior sampling explores better than the approximated UCB combined with the weighted Bellman backup; the weighted Bellman backup alone cannot replace the clipped double Q-Learning; the critics’ initialization plays the major role in ensemble-based actor-critic exploration.
As a conclusion, we show how existing tools can be brought together in a novel way, giving rise to the Ensemble
Deep Deterministic Policy Gradients (ED2) method, to yield state-of-the-art results on continuous control tasks from OpenAI Gym MuJoCo.
From the practical side,
ED2 is conceptually straightforward, easy to code, and does not require knowledge outside of the existing RL toolbox.
I NTRODUCTION
Recently, deep reinforcement learning (RL) has achieved multiple breakthroughs in a range of challenging domains (e.g. Silver et al.
; Berner et al.
; Andrychowicz et al.
(2020b);
Vinyals et al.
A part of this success is related to an ever-growing toolbox of tricks and methods that were observed to boost the RL algorithms’ performance (e.g. Hessel et al.
Haarnoja et al.
(2018b)
; Fujimoto et al.
; Wang et al.
Osband et al.
This state of affairs benefits the field but also brings challenges related to often unclear interactions between the individual improvements and the credit assignment related to the overall performance of the algorithm (Andrychowicz et al., 2020a; Ilyas et al., 2020).
In this paper, we present a comprehensive empirical study of multiple tools from the RL toolbox applied to the continuous control in the OpenAI Gym MuJoCo setting.
These are presented in Section 4 and Appendix B.
Our insights include:
• The ensemble of actors boost the agent performance.
• The current state-of-the-art methods are unstable under several stability criteria.
• The normally distributed action noise, commonly used for exploration, can hinder training.
• The approximated posterior sampling exploration (Osband et al., 2013) outperforms approximated UCB exploration combined with weighted Bellman backup (Lee et al., 2020).
• The weighted Bellman backup (Lee et al., 2020) can not replace the clipped double QLearning (Fujimoto et al., 2018).
• The critics’ initialization plays a major role in ensemble-based actor-critic exploration, while the training is mostly invariant to the actors’ initialization.
To address some of the issues listed above, we introduce the Ensemble Deep Deterministic Policy
Gradient (ED2) algorithm 1 , see Section 3.
ED2 brings together existing RL tools in a novel way: it is an off-policy algorithm for continuous control, which constructs an ensemble of streamlined versions of TD3 agents and achieves the state-of-the-art performance in OpenAI Gym MuJoCo, substantially improving the results on the two hardest tasks – Ant and Humanoid.
Consequently, ED2 does not require knowledge outside of the existing RL toolbox, is conceptually straightforward, and easy to code.
B ACKGROUND
We model the environment as a Markov Decision Process (MDP).
It is defined by the tuple ( S , A , R, P, γ, p 0 ) , where S is a continuous multi-dimensional state space, A denotes a continuous multi-dimensional action space, P is a transition kernel, γ ∈ [0 , 1) stands for a discount factor, p 0 refers to an initial state distribution, and R is a reward function.
The agent learns a policy from sequences of transitions τ =
[( s t , a t , r t , s t +1 , d )]
Tt =0 , called episodes or trajectories, where a t ∼ π ( ·| s t ) , s t +1 ∼ P ( ·| s t , a t ) , r t =
R ( s t , a t , s t +1 ) , d is a terminal signal, and T is the terminal time-step.
A stochastic policy π ( a | s ) maps each state to a distribution over actions.
A deterministic policy µ : S −→ A assigns each state an action.
All algorithms that we consider in this paper use a different policy for collecting data (exploration) and a different policy for evaluation (exploitation).
In order to keep track of the progress, the evaluation runs are performed every ten thousand environment interactions.
Because of the environments’ stochasticity, we run the evaluation policy multiple times.
Let { R i } Ni =1 be a set of (undiscounted) returns from N evaluation episodes { τ i } Ni =1 , i.e. R i = (cid:80) r t ∈ τ i r t .
We evaluate the policy using the average test return ¯ R = 1 N (cid:80) Ni =1 R i and the standard deviation of the test returns σ = (cid:113) 1 N − 1 (cid:80) Ni
{+=
 ( R i − ¯ R ) 2 .+}
We run experiments on four continuous control tasks and their variants, introduced in the appropriate sections, from the OpenAI Gym MuJoCo suite (Brockman et al., 2016).
The agent observes vectors that describe the kinematic properties of the robot and its actions specify torques to be applied on the robot joints.
See Appendix D for the details on the experimental setup.
E NSEMBLE D EEP D ETERMINISTIC P OLICY G RADIENTS
For completeness of exposition, we present ED2 before the experimental section.
The ED2 architecture is based on an ensemble of Streamlined Off-Policy (SOP) agents (Wang et al., 2020), meaning that our agent is an ensemble of TD3-like agents (Fujimoto et al., 2018) with the action normalization and the ERE replay buffer.
The pseudo-code listing can be found in Algorithm 1, while the implementation details, including a more verbose version of pseudo-code (Algorithm 3), can be found in Appendix E.
In the data collection phase (Lines 1-9), ED2 selects one actor from the ensemble uniformly at random (Lines 1 and 9) and run its deterministic policy for the course of one episode (Line 4).
In the evaluation phase (not shown in Algorithm 1), the evaluation policy averages all the actors’ output actions.
We train the ensemble every 50 environment steps withstochastic gradient descent updates (Lines 10-13).
ED2 concurrently learns K · 2 Q-functions, Q ϕ k,and Q ϕ k, 2 where k ∈ K , by mean square Bellman error minimization, in almost the same way that
SOP learns its two Q-functions.
The only difference is that we have K critic pairs that are initialized with different random weights and then trained independently with the same batches of data.
Because of the different initial weights, each Q-function has a different bias in its Q-values.
The K actors, π θ k , train maximizing their corresponding first critic, Q ϕ k, 1 , just like SOP.
Utilizing the ensembles requires several design choices, which we summarize below.
The ablation study of ED2 elements is provided in Appendix C.
Set s ← s ′ 7: Reset environment 10: if then 11: for as many as steps done in the environment do 12: Sample batch transitions
D
Update the parameters k , ϕ k, ϕ k, one
14: until convergence
Used: We train the ensemble of 5 actors and 5 critics; each actor learns from its own critic and the whole ensemble is trained on the same data.
Not used: We considered different actor-critic configurations, initialization schemes and relations, as well as the use of random prior networks (Osband et al., 2018), data bootstrap (Osband et al., 2016), and different ensemble sizes.
We also change the SOP network sizes and training intensity instead of using the ensemble.
Besides the prior networks in some special cases, these turn out to be inferior as shown in Section 4 and Appendix B.1.
Exploration
Used: We pick one actor uniformly at random to collect the data for the course of one episode.
The actor is deterministic (no additive action noise is applied).
These two choices ensure coherent and temporally-extended exploration similarly to Osband et al.
Not used: We tested several approaches to exploration: using the ensemble of actors, UCB (Lee et al., 2020), and adding the action noise in different proportions.
These experiments are presented in
Appendix B.2.
Exploitation
Used: The evaluation policy averages all the actors’ output actions to provide stable performance.
Not used: We tried picking an action with the biggest value estimate (average of the critics’ Qfunctions) in evaluation (Huang et al.,
Interestingly, both policies had similar results, see Appendix B.3.
Action normalization
Used: We use the action normalization introduced by Wang et al.
Not used: We experimented with the observations and rewards normalization, which turned out to be unnecessary.
The experiments are presented in Appendix B.4.
Q-function updates
Used: We do 50 SGD updates (ADAM optimizer (Kingma and Ba, 2015), MSE loss) to the actors and the critics every 50 environment interactions, use Clipped Double Q-Learning (Fujimoto et al., 2018).
Not used: We also examined doing the updates at the end of each episode (with the proportional number of updates), using the Hubert loss, and doing weighted Bellman backups (Lee et al., 2020).
However, we found them to bring no improvement to our method, as presented in Appendix B.5.
E XPERIMENTS
In this section, we present our comprehensive study and the resulting insights.
The rest of the experiments verifying that our design choices perform better than alternatives are in Appendix B.
Unless stated otherwise, a solid line in the figures represents an average, while a shaded region shows a 95% bootstrap confidence interval.
We used 30 seeds for ED2 and the baselines, and 7 seeds for the
ED2 variants.
4. E NSEMBLE OF ACTORS BOOST THE AGENT PERFORMANCE
ED2 achieves state-of-the-art performance on the OpenAI Gym MuJoCo suite.
Figure 1 shows the results of ED2 contrasted with three strong baselines:
SUNRISE (Lee et al., 2020) – which is also the ensemble-based method – SOP (Wang et al., 2020), and SAC (Haarnoja et al., 2018b).
Figure 2 shows ED2 with different ensemble sizes. As can be seen, the ensemble of size 5 (which we use in ED2) achieves good results, striking a balance between performance and computational overhead.
4. T HE CURRENT STATE - OF - THE - ART METHODS ARE UNSTABLE UNDER SEVERAL STABILITY CRITERIA
We consider three notions of stability: inference stability, asymptotic performance stability, and training stability.
ED2 outperforms baselines in each of these notions, as discussed below.
Similar metrics were also studied in Chan et al.
Inference stability We say that an agent is inference stable if, when run multiple times, it achieves similar test performance every time.
We measure inference stability using the standard deviation of test returns explained in Section 2.
We found that that the existing methods train policies that are surprisingly sensitive to the randomness in the environment initial conditions 2 .
Figure 1 and Figureshow that ED2 successfully mitigates this problem.
By the end of the training, ED2 produces results within 1% of the average performance on Humanoid, while the performance of SUNRISE, SOP, and
SAC may vary as much as 11% .
Asymptotic performance stability We say that an agent achieves asymptotic performance stability if it achieves similar test performance across multiple training runs starting from different initial networks weights.
Figure 4 shows that ED2 has a significantly smaller variance than the other methods while maintaining high performance.
Training stability We will consider training stable if performance does not severely deteriorate from one evaluation to the next.
We define the root mean squared deterioration metric (RMSD) as follows:
[Equation]
where M is the number of the evaluation phases during training and ¯ R i is the average test return at the i -th evaluation phase (described in Section 2).
We compare returns 20 evaluation phases apart to ensure that the deterioration in performance doesn’t stem from the evaluation variance.
ED2 has the lowest RMSD across all tasks, see Figure 5.
4. T HE NORMALLY DISTRIBUTED ACTION NOISE , COMMONLY USED FOR EXPLORATION , CAN HINDER TRAINING
In this experiment, we deprive SOP of its exploration mechanism, namely additive normal action noise with the standard deviation of 0 . 29 , and call this variant deterministic SOP (det.
SOP).
The lack of the action noise, while simplifying the algorithm, causes relatively minor deterioration in the Humanoid performance, has no significant influence on the Hopper or Walker performance, and substantially improves the Ant performance, see Figure 6.
This result shows that no additional exploration mechanism, often in a form of an exploration noise (Lillicrap et al., 2016; Fujimoto et al., 2018; Wang et al., 2020), is required for the diverse data collection and, in case of Ant, it even hinders training. ED2 leverages this insight and constructs an ensemble of deterministic SOP agents presented in Section 3.
Figure 7 shows that ED2 exhibit the same effects coming from the exploration without the action noise. In Figure 8 we present the more refined experiment where we vary the noise level. With more noise the Humanoid results get better, whereas the Ant results get much worse.
4. T HE APPROXIMATED POSTERIOR SAMPLING EXPLORATION OUTPERFORMS APPROXIMATED UCB EXPLORATION COMBINED WITH WEIGHTED B ELLMAN BACKUP
The posterior sampling is proved to be theoretically superior to the OFU strategy (Osband and
Van Roy, 2017).
We prove this empirically for the approximate methods.
ED2 uses posterior sampling exploration approximated with the bootstrap (Osband et al., 2016).
SUNRISE, on the other hand, approximates the Upper Confidence Bound (UCB) exploration technique and does weighted
Bellman backups (Lee et al., 2020).
For the fair comparison between ED2 and SUNRISE, we substitute the SUNRISE base algorithm SAC for the SOP algorithm used by ED2.
We call this variant
SUNRISE-SOP.
We test both methods on the standard MuJoCo benchmarks as well as delayed (Zheng et al., 2018a) and sparse (Plappert et al., 2018) rewards variants.
Both variations make the environments harder from the exploration standpoint.
In the delayed version, the rewards are accumulated and returned to the agent only every 10 time-steps.
In the sparse version, the reward for the forward motion is returned to the agent only after it crosses the threshold of one unit on the x-axis.
For a better perspective, a fully trained Humanoid is able to move to around five units until the end of the episode.
All the other reward components (living reward, control cost, and contact cost) remain unchanged.
The results are presented in Table 1.
[Equation]
Hopper-v    +5% +14%
Walker-v    +14%
+9%
Ant-v
[Table]
The performance in MuJoCo environments benefits from the ED2 approximate Bayesian posterior sampling exploration (Osband et al., 2013) in contrast to the approximated UCB in SUNRISE, which follows the OFU principle. Moreover, ED2 outperforms the non-ensemble method SOP, supporting the argument of coherent and temporally-extended exploration of ED2.
The experiment where the ED2’s exploration mechanism is replaced for UCB is in Appendix B.2.
This variant also achieves worse results than ED2.
The additional exploration efficiency experiment in the custom Humanoid environment, where an agent has to find and reach a goal position, is in
Appendix A.
4. T HE WEIGHTED B ELLMAN BACKUP CAN NOT REPLACE THE CLIPPED DOUBLE Q-L EARNING
We applied the weighted Bellman backups proposed by Lee et al.
(2020) to our method.
It is suggested that the method mitigates error propagation in Q-learning by re-weighting the Bellman backup based on uncertainty estimates from an ensemble of target Q-functions (i.e. variance of predictions).
Interestingly, Figure 9 does not show this positive effect on ED2.
Our method uses clipped double Q -Learning to mitigate overestimation in Q -functions (Fujimoto et al., 2018).
We wanted to check if it is required and if it can be exchanged for the weighted Bellman backups used by Lee et al.
Figure 10 shows that clipped double Q-Learning is required and that the weighted Bellman backups can not replace it.
4. T HE CRITICS ’ INITIALIZATION PLAYS A MAJOR ROLE IN ENSEMBLE - BASED ACTOR - CRITIC EXPLORATION , WHILE THE TRAINING IS MOSTLY INVARIANT TO THE ACTORS ’ INITIALIZATION
In this experiment, actors’ weights are initialized with the same random values (contrary to the standard case of different initialization). Moreover, we test a corresponding case with critics’ weights initialized with the same random values or simply training only a single critic. Figure 11 indicates that the choice of actors initialization does not matter in all tasks but Humanoid. Although the average performance on Humanoid seems to be better, it is also less stable.
This is quite interesting because the actors are deterministic. Therefore, the exploration must come from the fact that each actor is trained to optimize his own critic.
On the other hand, Figure 11 shows that the setup with the single critic severely impedes the agent performance.
We suspect that using the single critic impairs the agent exploration capabilities as its actors’ policies, trained to maximize the same critic’s Q -function, become very similar.
R ELATED WORK
Off-policy RL Recently, multiple deep RL algorithms for continuous control have been proposed, e.g. DDPG (Lillicrap et al., 2016), TD3 (Fujimoto et al., 2018), SAC (Haarnoja et al., 2018b), SOP (Wang et al., 2020), SUNRISE (Lee et al., 2020).
They provide a variety of methods for improving training quality, including double-Q bias reduction (van Hasselt et al., 2016), target policy smoothing or different update frequencies for actor and critic (Fujimoto et al., 2018), entropy regularization (Haarnoja et al., 2018b), action normalization (Wang et al., 2020), prioritized experience replay (Wang et al., 2020), weighted Bellman backups (Kumar et al., 2020; Lee et al., 2020), and use of ensembles (Osband et al., 2019; Lee et al., 2020; Kurutach et al., 2018;
Chua et al., 2018).
Ensembles Deep ensembles are a practical approximation of a Bayesian posterior, offering improved accuracy and uncertainty estimation (Lakshminarayanan et al., 2017; Fort et al., 2019).
They inspired a variety of methods in deep RL.
They are often used for temporally-extended exploration; see the next paragraph.
Other than that, ensembles of different TD-learning algorithms were used to calculate better Q -learning targets (Chen et al., 2018).
Others proposed to combine the actions and value functions of different RL algorithms (Wiering and van Hasselt, 2008) or the same algorithm with different hyper-parameters (Huang et al., 2017).
For mixing the ensemble components, complex self-adaptive confidence mechanisms were proposed in Zheng et al.
(2018b).
Our method is simpler: it uses the same algorithm with the same hyper-parameters without any complex or learnt mixing mechanism.
Lee et al.
(2020) proposed a unified framework for ensemble learning in deep RL (SUNRISE) which uses bootstrap with random initialization (Osband et al., 2016) similarly to our work.
We achieve better results than SUNRISE and show in Appendix B that their UCB exploration and weighted Bellman backups do not aid our algorithm performance.
Exploration Various frameworks have been developed to balance exploration and exploitation in
RL.
The optimism in the face of uncertainty principle (Lai and Robbins, 1985; Bellemare et al., 2016) assigns an overly optimistic value to each state-action pair, usually in the form of an exploration bonus reward, to promote visiting unseen areas of the environment.
The maximum entropy method (Haarnoja et al., 2018a) encourages the policy to be stochastic, hence boosting exploration.
In the parameter space approach (Plappert et al., 2018; Fortunato et al., 2018), noise is added to the network weights, which can lead to temporally-extended exploration and a richer set of behaviours.
Posterior sampling (Strens, 2000; Osband et al., 2016; 2018)
methods have similar motivations.
They stem from the Bayesian perspective and rely on selecting the maximizing action among sampled and statistically plausible set of action values.
The ensemble approach (Lowrey et al., 2018; Miło´s et al., 2019; Lee et al., 2020) trains multiple versions of the agent, which yields a diverse set of behaviours and can be viewed as an instance of posterior sampling RL.
C ONCLUSIONS
We conduct a comprehensive empirical analysis of multiple tools from the RL toolbox applied to the continuous control in the OpenAI Gym MuJoCo setting.
We believe that the findings can be useful to
RL researchers.
Additionally, we propose Ensemble Deep Deterministic Policy Gradients (ED2), an ensemble-based off-policy RL algorithm, which achieves state-of-the-art performance and addresses several issues found during the aforementioned study.
R EPRODUCIBILITY S TATEMENT
We have made a significant effort to make our results reproducible.
We use 30 random seeds, which is above the currently popular choice in the field (up to 5 seeds).
Furthermore, we systematically explain our design choices in Section 3 and we provide a detailed pseudo-code of our method in
Algorithm 3 in the Appendix B. Additionally, we open-sourced the code for the project 3 together with examples of how to reproduce the main experiments.
The implementation details are explained in Appendix E and extensive information about the experimental setup is given in Appendix D
A E XPLORATION EFFICIENCY IN THE CUSTOM H UMANOID ENVIRONMENT
To check the exploration capabilities of our method, we constructed two environments based on
Humanoid where the goal is not only to move forward as fast as possible but to find and get to the specific region.
The environments are described in Figure 12.
Because the Humanoid initial state is slightly perturbed every run, we compare solved rates over multiple runs, see details in Appendix D. Figure 13 compares the solved rates of our method and the three baselines.
Our method outperforms the baselines.
For this experiment, our method uses the prior networks (Osband et al., 2018).
B D ESIGN CHOICES
In this section, we summarize the empirical evaluation of various design choices grouped by topics related to an ensemble of agents (B.1), exploration (B.2), exploitation (B.3), normalization (B.4), and Q -function updates (B.5).
In the plots, a solid line and a shaded region represent an average and a 95% bootstrap confidence interval over 30 seeds in case of ED2 (ours) and 7 seeds otherwise.
All of these experiments test ED2 presented in Section 3 with Algorithm 2 used for evaluation (the ensemble critic variant).
We call Algorithm 2 a ’vote policy’.
Algorithm 2 Vote policy
1: Input: ensemble size K ; policy θ k and Q -function ϕ k, 1 parameters where k ∈ [1 , . .
, K ] ; max action scale M ; 2: function VOTE _ POLICY
[Equation]
s , c )
[Equation]
3: if
use arbitrary critic then
[Equation]
k = Q ϕ
[Equation]
[Equation]
s, a k ) for k ∈ [1 , . . .
, K ] (2) 4: else use ensemble critic
[Equation]
[Equation]
5: return a k for k = argmax k q k
B. E NSEMBLE
Prior networks We tested if our algorithm can benefit from prior networks (Osband et al., 2018).
It turned out that the results are very similar on OpenAI Gym MuJoCo tasks, see Figure 14.
However, the prior networks are useful on our crafted hard-exploration Humanoid environments, see Figure 15.
Moreover, we tested if the deterministic SOP variant can benefit from prior networks.
It turned out that the results are very similar or worse, see Figure 16.
Ensemble size Figure 17 shows ED2 with different ensemble sizes.
As can be seen, the ensemble of size 5 (which we use in ED2) achieves good results, striking a balance between performance and computational overhead.
Data bootstrap Osband et al.
(2016) and Lee et al.
(2020) remark that training an ensemble of agents using the same training data but with different initialization achieves, in most cases, better performance than applying different training samples to each agent.
We confirm this observation in
Figure 18.
Data bootstrap assigned each transition to each agent in the ensemble with 50% probability.
SOP bigger networks and training intensity We checked if simply training SOP with bigger networks or with higher training intensity (a number of updates made for each collected transition) can get it close to the ED2 results.
Figure 19 compares ED2 to SOP with different network sizes, while Figure 20 compares ED2 to SOP with one or five updates per environment step.
It turns out that bigger networks or higher training intensity does not improve SOP performance.
B. E XPLORATION
Vote policy In this experiment, we used the so-called "vote policy" described in Algorithm 2.
We use it for action selection in step 5 of Algorithm 3 in two variations: (1) where the random critic, chosen for the duration of one episode, evaluates each actor’s action or (2) with the full ensemble of critics for actors actions evaluation.
Figure 21 shows that the arbitrary critic is not much different from our method.
However, in the case of the ensemble critic, we observe a significant performance drop suggesting deficient exploration.
UCB We tested the UCB exploration method from Lee et al.
This method defines an upper-confidence bound (UCB) based on the mean and variance of Q-functions in an ensemble and selects actions with the highest UCB for efficient exploration.
Figure 22 shows that the UCB exploration method makes the results of our algorithm worse.
Gaussian noise While our method uses ensemble-based temporally coherent exploration, the most popular choice of exploration is injecting i.i.d.
noise (Fujimoto et al., 2018; Wang et al., 2020).
We evaluate if these two approaches can be used together.
We used Gaussian noise with the standard deviation of 0 .
29 , it is the default value in Wang et al.
We found that the effects are taskspecific, barely visible for Hopper and Walker, positive in the case of Humanoid, and negative for
Ant – see Figure 23.
In a more refined experiment, we varied the noise level.
With more noise the
Humanoid results are better, whereas the And results are worse – see Figure 24.
B. E XPLOITATION
We used the vote policy, see Algorithm 2, as the evaluation policy in step 21 of Algorithm 3.
Figure 25 shows that the vote policy does worse on the OpenAI Gym MuJoCo tasks.
However, on our custom Humanoid tasks introduced in Section 4, it improves our agent performance – see Figure 26.
B. N ORMALIZATION
We validated if rewards or observations normalization (Andrychowicz et al., 2020a) help our method.
In both cases, we keep the empirical mean and standard deviation of each reward/observation coordinate, based on all rewards/observations seen so far, and normalize rewards/observations by subtracting the empirical mean and dividing by the standard deviation.
It turned out that only the observations normalization significantly helps the agent on Humanoid, see Figures 27 and 28.
The action normalization influence is tested in Appendix C.
B. Q - FUNCTION UPDATES
Huber loss We tried using the Huber loss for the Q -function training.
It makes the results on all tasks worse, see Figure 29.
C A BLATION STUDY
In this section, we ablate the ED2 components to see their impact on performance and stability.
We start with the ensemble exploration and exploitation and then move on to the action normalization and the ERE replay buffer.
In all plots, a solid line and a shaded region represent an average and a 95% bootstrap confidence interval over 30 seeds in all but action normalization and ERE replay buffer experiments, where we run 7 seeds.
Exploration & Exploitation In the first experiment we wanted to isolate the effect of ensemblebased temporally coherent exploration on the performance and stability of ED2.
Figures 30compare the performance and stability of ED2 and one baseline, SOP, to ED2 with the single actor (the first one) used for evaluation in step 21 of Algorithm 3.
It is worth noting that the action selection during the data collection, step 5 in Algorithm 3, is left unchanged – the ensemble of actors is used for exploration and each actor is trained on all the data.
This should isolate the effect of exploration on the test performance of every actor.
The results show that the performance improvement and stability of ED2 does not come solely from the efficient exploration.
ED2 ablation performs comparably to the baseline and is even less stable.
In the next experiment, we wanted to check if the ensemble evaluation is all we need in that event.
Figure 34 compares the performance of ED2 and one baseline, SOP, to ED2 with the single actor (the first one) used for the data collection in step 5 of Algorithm 3.
The action selection during the evaluation, step 21 in Algorithm 3, is left unchanged – the ensemble of actors is trained on the data collected only by one of the actors.
We add Gaussian noise to the single actor’s actions for exploration as described in Appendix B.2.
The results show that the ensemble actor test performance collapses, possibly because of training on the out of distribution data.
This implies that the ensemble of actors, used for evaluation, improves the test performance and stability.
However, it is required that the same ensemble of actors is also used for exploration, during the data collection.
Action normalization The implementation details of the action normalization are described in
Appendix E. Figure 35 shows that the action normalization is especially required on the Ant and
Humanoid environments, while not disrupting the training on the other tasks.
ERE replay buffer The implementation details of the ERE replay buffer are described in Appendix
E. In Figure 36 we observe that it improves the final performance of ED2 on all tasks, especially on
Walker2d and Humanoid.
D E XPERIMENTAL SETUP
Plots In all evaluations, we used 30 evaluation episodes to better access the average performance of each policy, as described in Section 2.
For a more pleasant look and easier visual assessment, we smoothed the lines using an exponential moving average with a smoothing factor equal 0 .
OpenAI Gym MuJoCo
In MuJoCo environments, presented in Figure 37, a state is defined by ( x, y, z ) position and velocity of the robot’s root, and angular position and velocity of each of its joints.
The observation holds almost all information from the state except the x and y position of the robot’s root.
The action is a torque that should be applied to each joint of the robot.
Sizes of those spaces for each environment are summarised in Table 2.
MuJoCo is a deterministic physics engine thus all simulations conducted inside it are deterministic.
This includes simulations of our environments.
However, to simplify the process of data gathering and to counteract over-fitting the authors of OpenAI Gym decided to introduce some stochasticity.
Each episode starts from a slightly different state - initial positions and velocities are perturbed with random noise (uniform or normal depending on the particular environment).
[Table]
E I MPLEMENTATION DETAILS
Architecture and hyper-parameters
In our experiments, we use deep neural networks with two hidden layers, each of them with 256 units.
All of the networks use ReLU as an activation, except on the final output layer, where the activation used varies depending on the model: critic networks use no activation, while actor networks use tanh () multiplied by the max action scale.
Table 3 shows the hyper-parameters used for the tested algorithms.
Set
12: for as many as steps done in the environment do 13:
Sample transitions B = { ( s, a, r, s ′ , d ) } ⊂ D ▷ uses
Compute targets y
[Equation]
i =
[Equation]
[Equation]
) a
[Equation]
[Equation]
Update the Q -functions by one step of gradient descent using ∇ ϕ k,i | B | · K (cid:88) ( s,a,r,s ′ ,d ) ∈ B (cid:0) Q ϕ k,i ( s, a ) − y k
( r, s ′ , d ) (cid:1)for i ∈ { 1 , 2 } , k ∈ [1 , . . .
, K ] 16:
Update the policies by one step of gradient ascent using
[Equation]
Update target parameters with
[Equation]
18: if time to evaluate then 19: for specified number of evaluation runs do 20:
Reset the environment and observe the state s .
Execute policy a = 1 K (cid:80) Ki =1 M tanh ( µ θ i ( s )) until the terminal state.
Record and log the return.
23: until convergence
[Table]
Action normalization Our algorithm employs action normalization proposed by Wang et al.
It means that before applying the squashing function (e.g. tanh () ), the outputs of each actor network are normalized in the following way: let µ = ( µ 1 , . . . , µ A ) be the output of the actor’s network and let G = (cid:80) Ai =1 | µ i | /A be the average magnitude of this output, where A is the action’s dimensionality.
If G > 1 then we normalize the output by setting µ i to µ i /G for all i = 1 , . . .
, A .
Otherwise, we leave the output unchanged.
Each actor’s outputs are normalized independently from other actors in the ensemble.
Emphasizing Recent Experience
We implement the Emphasizing Recent Experience (ERE) mechanism from Wang et al.
ERE samples non-uniformly from the most recent experiences stored in the replay buffer.
Let B be the number of mini-batch updates and |D| be the size of the replay buffer.
When performing the gradient updates, we sample from the most recent c b data points stored in the replay buffer, where c b = |D| · η b 1000 B for b = 1 , . . .
, B .
The hyper-parameter η starts off with a set value of η 0 and is later adapted based on the improvements in the agent training performance.
Let I recent be the improvement in terms of training episode returns made over the last |D| / 2 time-steps
and I max be the maximum of such improvements over the course of the training.
We adapt η according to the formula:
[Equation]
Our implementation uses the exponentially weighted moving average to store the value of I recent .
More concretely, we define I recent based on two additional parameters R recent and R prev so that
I recent =
R recent − R prev .
Those parameters are then updated whenever we receive a new training episode return ep _ ret :
[Equation]
where λ prev =
T/
⌊ |D| 2 ⌋ , λ recent = 10 · λ prev and T is the maximum length of an episode.
Hardware During the training of our models, we employ only CPUs using a cluster where each node has 28 available cores of 2 . 6 GHz, alongside at least 64 GB of memory.
The running time of a typical experiment did not exceed 24 hours.