A Simple Decentralized Cross-Entropy Method
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Cross-Entropy Method (CEM) is commonly used for planning in model-basedreinforcement learning (MBRL) where a centralized approach is typically utilizedto update the sampling distribution based on only the top- k operations’ results onsamples.
In this paper, we show that such a centralized approach makes CEMvulnerable to local optima, thus impairing its sample efﬁciency.
To tackle thisissue, we propose Decent ralized CEM (DecentCEM) , a simple but effectiveimprovement over classical CEM, by using an ensemble of CEM instances runningindependently from one another, and each performing a local improvement of itsown sampling distribution.
We provide both theoretical and empirical analysis todemonstrate the effectiveness of this simple decentralized approach.
We empiricallyshow that, compared to the classical centralized approach using either a single oreven a mixture of Gaussian distributions, our DecentCEM ﬁnds the global optimummuch more consistently thus improves the sample efﬁciency.
Furthermore, we plugin our DecentCEM in the planning problem of MBRL, and evaluate our approach inseveral continuous control environments, with comparison to the state-of-art CEMbased MBRL approaches (PETS and POPLIN).
Results show sample efﬁciencyimprovement by simply replacing the classical CEM module with our DecentCEMmodule, while only sacriﬁcing a reasonable amount of computational cost, whichcan be alleviated by using the advantage of parallelism of our ensemble method.
Lastly, we conduct ablation studies for more in-depth analysis.
Introduction
Model-based reinforcement learning (MBRL) uses a model as a proxy of the environment for planningactions in multiple steps.
This paper studies planning in MBRL with a speciﬁc focus on the CrossEntropy Method (CEM)
[De Boer et al., 2005, Mannor et al., 2003], which is popular in MBRL dueto its ease of use and strong empirical performance [Chua et al., 2018, Hafner et al., 2019, Wang and
Ba, 2020, Zhang et al., 2021, Yang et al., 2020].
CEM is a stochastic, derivative-free optimizationmethod.
It uses a sampling distribution to generate imaginary trajectories of environment-agentinteractions with the model.
These trajectories are then ranked based on their returns computed fromthe rewards given by the model.
The sampling distribution is updated to increase the likelihood ofproducing the top- k trajectories with higher returns.
These steps are iterated and eventually yield animproved distribution over the action sequences to guide the action execution in the real environment.
Despite the strong empirical performance of CEM for planning, it is prone to two problems: (1) lowersample efﬁciency as the dimensionality of solution space increases, and (2) the Gaussian distributionthat is commonly used for sampling may cause the optimization to get stuck in local optima ofmulti-modal solution spaces commonly seen in real-world problems.
Previous works addressingthese problems either add gradient-based updates of the samples to optimize the parameters of CEM,or adopt more expressive sampling distributions, such as using Gaussian Mixture Model [Okada and
Taniguchi, 2020] or masked auto-regressive neural network [Hakhamaneshi et al., 2020].
Nevertheless, all CEM implementations to date are limited to a centralized formulation where the ranking stepinvolves all samples .
As analyzed below and in Section 3, such a centralized design makes CEMvulnerable to local optima and impairs its sample efﬁciency.
We propose Decent ralized CEM (DecentCEM) , a simple but effective improvement overclassical CEM, to address the above problems.
Rather than ranking all samples , as in thecentralized design, our method distribute the sampling budget across an ensemble of CEMinstances.
These instances run independently from one another, and each performs a local improvement of its own sampling distribution based on the ranking of its generatedsamples.
The best action is then aggregated by taking an arg max among the solutionof the instances.
It recovers the conventional CEM when the number of instance is one.
[Figure]
We hypothesize that by shifting to this decentralizeddesign, CEM can be less susceptible to prematureconvergence caused by the centralized ranking step.
As illustrated in Fig.
1, the centralized samplingdistribution exhibits a bias toward the sub-optimalsolutions near top right, due to the global top- k ranking.
This bias would occur regardless of the familyof distributions used.
In comparison, a decentralizedapproach could maintain enough diversity thanks toits local top-
k ranking in each sampling instance.
Through a one-dimensional multi-modal optimization problem in Section 3, we show that DecentCEMempirically ﬁnds the global optimum more consistently than centralized CEM approaches that use either a single or a mixture of Gaussian distributions.
Also we show that DecentCEM is theoretically soundthat it converges almost surely to a local optimum.
We further apply it to sequential decision makingproblems and use neural networks to parameterizethe sampling distributions.
Empirical results in several continuous control environments suggest that DecentCEM offers an effective mechanism to improvethe sample efﬁciency over the baseline CEM under the same sample budget for planning.
Preliminaries
We consider a Markov Decision Process (MDP) speciﬁed by ( S , A , R , P ,  , d 0 , T ).
S ⇢ R d s isthe state space, A ⇢ R d a is the action space.
d s , d a are scalars denoting the dimensionality.
R : S ⇥ A !
R is the reward function that maps a state and action pair to a real-valued reward.
P ( s 0 | s, a ) :
S ⇥
A ⇥ S !
R + is the transition probability from a state and action pair s, a to the nextstate s 0 .
2 [0 , 1] is the discount factor.
d 0 denotes the distribution of the initial state s 0 .
At timestep t , the agent receives a state s t 1 and takes an action a t according to a policy ⇡ ( ·| s ) that maps thestate to a probability distribution over the action space.
The environment transitions to the next states t +1 ⇠ P ( ·| s t , a t ) and gives a reward r t =
R ( s t , a t ) to the agent 2 .
The return G t =
P
Ti =
i r t + i ,is the sum of discounted reward within an episode length of T .
The agent aims to ﬁnd a policy ⇡that maximizes the expected return.
We denote the learned model in MBRL as f !
( ·| s, a ) , which isparameterized by !
and approximates P ( ·| s, a ) .
Planning with the Cross Entropy Method Planning in MBRL is about leveraging the model toﬁnd the best action in terms of its return.
Model-Predictive-Control (MPC) performs decision-timeplanning at each time step up to a horizon to ﬁnd the optimal action sequence:
[Equation]
Following the settings from CEM-based MBRL papers (Sec. 6.1), we assume that the reward function is deterministic (a mild assumption [Agarwal et al., 2019]) and known.
Note that they are not fundamental limitations of CEM and are adopted here so as to be consistent with the literature.
where H is the planning horizon, a t : t + H  1 denotes the action sequence from time step t to t + H  1 ,and V ( s H ) is the terminal value function at the end of the horizon.
The ﬁrst action in this sequenceis executed and the rest are discarded.
The agent then re-plans at the next time step.
The Cross-Entropy Method (CEM) is a gradient-free optimization method that can be used forsolving Eq.
The workﬂow is shown in Fig.
CEM planning starts by generating N samples{ ⌧ j } Nj =
= { (ˆ a j, 0 , ˆ a j, 1 , · · · , ˆ a j,H  1 ) } Nj =1 from an initial sampling distribution g  ( ⌧ ) parameterized 92 by  , where each sample ⌧ j is an action sequence from the current time step up to the planninghorizon H .
The domain of g  ( ⌧ ) has a dimension of d ⌧ = d a H .
Using a model f , CEM generates imaginary rollouts based on the action sequence { ⌧ j } (in the caseof a stochastic model) and estimate the associated value v ( ⌧ j ) =
E [ ⌃ H  1 i =
i r ( s j,i , a j,i )] where s j, 0is the current state s and s j,i +
⇠ f ( ·| s j,i , a j,i ) .
The terminal value  H V ( s j,H ) is omitted here 97 following the convention in the CEM planning literature but the MPC performance can be furtherimproved if paired with an accurate value predictor [Bertsekas, 2005, Lowrey et al., 2019].
Thesampling distribution is then updated by ﬁtting to the current top- k samples in terms of their valueestimates v ( ⌧ j ) , using the Maximum Likelihood Estimation (MLE) which solves:
[Equation] [Equation]
where v th is the threshold equal to the value of the k -th best sample and ( · ) is the indicator function.
In practice, the update to the distribution parameters are smoothed by
l +1 =
↵ 0 + (1  ↵ )  l where↵ 2 [0 , 1] is a smoothing parameter that balances between the solution to Eq.
(2) and the parameterat the current internal iteration l .
CEM repeats this process of sampling and distribution update in aninner-loop, until it reaches the stopping condition.
In practice, it is stopped when either a maximumnumber of iterations has been reached or the parameters have not changed for a few iterations.
Theoutput of CEM is an action sequence, typically set as the expectation 3 of the most recent samplingdistribution for uni-modal distributions such as Gaussians ˆ µ = E ( g  ) =
(ˆ a 0 , ˆ a 1 , · · · , ˆ a H  1 ) .
Choices of Sampling Distributions in CEM:
A common choice is a multivariate Gaussiandistribution under which Eq.(2) has an analytical solution.
But the uni-modal nature of Gaussianmakes it inadequate in solving multi-modal optimization that often occurs in MBRL.
To increase thecapacity of the distribution, a Gaussian Mixture Model (GMM) can be used [Okada and Taniguchi,2020].
We denote such an approach as CEM-GMM .
Going forward, we use CEM to refer to thevanilla version that employs a Gaussian distribution.
Computationally, the major difference between
CEM and CEM-GMM is that the distribution update in CEM-GMM is more intensive since it solvesfor more parameters.
Detailed steps can be found in Okada and Taniguchi [2020].
Decentralized CEM
In this section, we ﬁrst introduce the formulation of the proposed decentralized approach called the
Decent ralized CEM (DecentCEM) .
Then we illustrate the intuition behind the proposed approachusing a one-dimensional synthetic multi-modal optimization example where we show the issues ofthe existing CEM methods and how they can be addressed by DecentCEM.
Formulation of DecentCEM DecentCEM is composed of an ensemble of M CEM instancesindexed by i , each having its own sampling distributions g  i .
They can be described by a set ofdistribution parameters
= {  i } Mi =1 .
Each instance i manages its own sampling and distribution update by the steps described in Section 2, independently from other instances.
Note that the N samples and k elites are evenly split among the M instances.
The top- kM samplesets are decentralized and managed by each instance independently whereas the centralized approachonly keeps one set of top- k samples regardless of the distribution family used.
After the stopping condition is reached for all instances, the ﬁnal sampling distribution is taken as thebest distribution in the set  according to (the arg max uses a deterministic tie-breaking):
[Equation]
where E  i [ v ( x )] denotes the expectation with respect to the distribution
g  i , approximated by thesample mean of NM samples.
When M = 1 , it recovers the conventional CEM.
Motivational Example Considera one dimensional multi-modaloptimization problem shown in
Fig.3 (Left): arg min x sin( x ) +sin(10 x/ 3) ,  7 .
 x  7 . 5 .
Thereare eight local optima, including oneglobal optimum f ( x ⇤ ) =
1 .
9 wherex ⇤ = 5 .
This objective functionmimics the RL value landscape thathas many local optima, as shownby Wang and Ba [2020].
Thisoptimization problem is “easy” inthe sense that a grid search over thedomain can get us a solution close tothe global optimum.
However, onlyour proposed DecentCEM methodsuccessfully converges to the global optimum consistently under varying population size (i.e., numberof samples) and random runs, as shown in Fig.3 (Right) 4 .
Both CEM-GMM and the proposed DecentCEM are equipped with multiple sampling distributions.
The fact that CEM-GMM is outperformed by DecentCEM may appear surprising.
To gain someinsights, we illustrate in Fig.
4 how the sampling distribution evolves during the iterative update (moredetails in Fig. 10 in Appendix).
CEM updated the unimodal distribution toward a local optimumdespite seeing the global optimum.
CEM-GMM appears to have a similar issue.
During MLE on thetop- k samples, it moved most distribution components towards the same local optimum which quicklyled to mode collapse.
On the contrary, DecentCEM successfully escaped the local optima thanksto its independent distribution update over decentralized top- k samples and was able to maintain adecent diversity among the distributions.
GMM suits density estimation problems like distribution-based clustering where the samples aredrawn from a ﬁxed true distribution that can be represented by multi-modal Gaussians.
However,in CEM for optimization, exploration is coupled with density estimation: the sampling distributionin CEM is not ﬁxed but rather gets updated iteratively toward the top- k samples.
And the “true”distribution in optimization puts uniform non-zero densities to the global optima and zero densities everywhere else.
When there is a unique global optimum, it degenerates into a Dirac measure thatassigns the entire density to the optimum.
Density estimation of such a distribution only needs one
Gaussian but the exploration is challenging.
In other words, the conditions for GMM to work wellare not necessarily met when used as the sampling distribution in CEM .
CEM-GMM is subject tomode collapse during the iterative top- k greediﬁcation, causing premature convergence, as observedin Fig 4.
In comparison, our proposed decentralized approach takes care of the exploration aspect byrunning multiple CEM instances independently, each performing its own local improvement.
Thisis shown to be effective from this optimization example and the benchmark results in Section 6.
CEM-GMM only consistently converge to the global optimum when we increase the population sizeto the maximum 1,000 which causes expensive computations.
Our proposed DecentCEM runs morethan 100 times faster than CEM-GMM at this population size, shown in Table A.3 in Appendix.
Convergence of DecentCEM
The convergence of DecentCEM requires the following assumption:
Assumption 1.
Let M be the number of instances in DecentCEM and each instance has a sample sizeof N k M where N k is the total number of samples that satisﬁes N k = ⇥ ( k  ) where  > max { 0 , 1  2  } .
M and  are some positive constant and 0 < M < N k 8 k .
Here the assumption of the total sample size follows the same one as in the CEM literature.
Othersuch standard assumptions are summarized in the appendix.
Theorem 3.1 (Convergence of DecentCEM) .
If a CEM instance described in Algorithm 3 converges,and we decentralize it by evenly dividing its sample size N k into M CEM instances which satisﬁesthe Assumption 1, then the resulting DecentCEM converges almost surely to the best solution of theindividual instances.
Proof.
(sketch)
We show that the previous convergence result of CEM [Hu et al., 2011] extends to
DecentCEM under the same sample budget.
The key observation is that the convergence propertyof each CEM instance still holds since the number of samples in each instance is only changed bya constant factor, i.e., number of instances.
Each CEM instance converges to local optima.
Theconvergence of DecentCEM to the best solution comes from the argmax operator and applying thestrong law of large numbers.
The detailed proof is left to the appendix H.
DecentCEM for Planning
In this section, we develop two instantiations of DecentCEM for planning in MBRL wherethe sampling distributions are parameterized by policy networks.
For the dynamics modellearning, we adopt the ensemble of probabilistic neural networks proposed in [Chua et al.,2018].
Each network predicts the mean and diagonal covariance of a Gaussian distribution and is trained by minimizing the negative log likelihood.
Our work focuses onthe planning aspect and refer to Chua et al.
[2018] for further details on the model.
[Figure]
CEM Planning with a Policy Network In MBRL,
CEM is applied to every state separately to solve theoptimization problem stated in Eq.
The samplingdistribution is typically initialized to a ﬁxed distribution at the beginning of every episode [Okada and
Taniguchi, 2020, Pinneri et al., 2020], or more frequently at every time step [Hafner et al., 2019].
Suchinitialization schemes are sample inefﬁcient sincethere is no mechanism that allows the informationof the high-value region in the value space of onestate to generalize to nearby states.
Also, the information is discarded after the initialization.
It is hencedifﬁcult to scale the approach to higher dimensionalsolution spaces, present in many continuous control environments.
Wang and
They developed two methods: POPLIN-A that plans in the action space, and POPLIN-P that plansin the parameter space of the policy network.
In POPLIN-A , the policy network is used to learn tooutput the mean of a Gaussian sampling distribution of actions.
In POPLIN-P , the policy networkparameters serve as the initialization of the mean of the sampling distribution of parameters.
Theimproved policy network can then be used to generate an action.
They show that when compared tothe vanilla method of using a ﬁxed sampling distribution in the action space, both modes of CEMplanning with such a learned distribution perform better.
The same principle of combining a policynetwork with CEM can be applied to the DecentCEM approach as well, which we describe next.
DecentCEM Planning with an Ensemble of Policy Networks For better sample efﬁciency in
MBRL setting, we extend DecentCEM to use an ensemble of policy networks to learn the samplingdistributions in the CEM instances.
Similar to the POPLIN variations, we develop two instantiationsof DecentCEM , namely DecentCEM-A and DecentCEM-P .
The architecture of the proposed algorithmis illustrated in Fig.
DecentCEM-A plans in the action space.
It consists of an ensemble of policynetworks followed by CEM instances.
Each policy network takes the current state s t as input, outputsthe parameters ✓ i of the sampling distribution for CEM instance i .
There is no fundamental differencefrom the DecentCEM formulation in Section 3 except that the initialization of sampling distributionsis learned by the policy networks rather than a ﬁxed distribution.
The second instantiation DecentCEM-P plans in the parameter space of the policy network.
Theinitial sampling distribution is a Gaussian distribution over the policy parameter space with the meanat the current parameter values.
In the arg max operation in Eq.
(3), the sample ⌧ i,j denotes theparameters of the policy network.
Its value is obtained by computing the value of the action sequencegenerated from the policy network with the parameters ⌧ i,j .
The ensemble of policy networks in both instantiations DecentCEM-A and DecentCEM-P are initialized with random weights, which is empirically found to be adequate to ensure that the output of thenetworks do not collapse into the same distribution (Sec. 6.2 and Appendix F).
Training the Policy Networks in DecentCEM When planning in action space, the policy networksare trained by behavior cloning, similar to the scheme in POPLIN [Wang and Ba, 2020].
Denote theﬁrst action in the planned action sequence at time step t by the i -th CEM instance as ˆ a t,i , the i -thpolicy network is trained to mimic ˆ a t,i and the training objective is min ✓ i E s t , ˆ a t,i ⇠ D i k a ✓ i ( s t )  ˆ a t,i k
2where D i denotes the replay buffer with the state and action pairs ( s t , ˆ a t,i ) .
a ✓ i ( s t ) is the actionprediction at state s t from the policy network parameterized by ✓ i .
While the above training scheme can be applied to both planning in action space and parameter space,we follow the setting parameter average (AVG)
[Wang and Ba, 2020] training scheme when planningin parameter space.
The parameter is updated as ✓ i = ✓ i + 1 |
D i |
P  i
2 D i  i where D i =
{  i } is adataset of policy network parameter updates planned from the i -th CEM instance previously.
It ismore effective than behavior cloning based on the experimental result reported by Wang and Ba[2020] and our own preliminary experiments.
Note that each policy network in the ensemble is trained independently from the data observed byits corresponding CEM instance rather than from the aggregated result after taking the arg max .
This allows for enough diversity among the instances.
More importantly, it increases the size of thetraining dataset for the policy networks compared to the approach taken in POPLIN .
For example,with an ensemble of M instances, there would be M training data samples available from one realenvironment interaction, compared to the one data sample in POPLIN-A/P .
As a result, DecentCEMis able to train larger policy networks than is otherwise possible, shown in Sec.
6.2 and Appendix F.
Related Work
We limit the scope of related works to CEM planning methods, which is one of the broad class ofplanning algorithms in MBRL.
For a review of different families of planning algorithms, the readersare referred to Wang et al.
Vanilla CEM planning in action space with a single Gaussiandistribution has been adopted as the planning method for both simulated and real-world robot control[Chua et al., 2018, Finn and Levine, 2017, Ebert et al., 2018, Hafner et al., 2019, Yang et al., 2020,
Zhang et al., 2021].
Previous attempts to improving the performance of CEM-based planning can begrouped into two types of approaches.
The ﬁrst type includes CEM in a hybrid of CEM+X where “X” is some other component or algorithm.
POPLIN [Wang and Ba, 2020] is a prominent example where“X” is a policy network that learns a state conditioned distribution that initializes the subsequent CEMprocess.
Another common choice of “X” is gradient-based adjustment of the samples drawn in CEM.
GradCEM [Bharadhwaj et al., 2020] adjusts the samples in each iteration of CEM by taking gradientascent of the return estimate w.r.t the actions.
CEM-RL [Pourchot and Sigaud, 2019] also combines
CEM with gradient based updates from RL algorithms but the samples are in the parameter space ofthe actor network.
To improve computational efﬁciency, Lee et al.
[2020] proposes an asynchronousversion of CEM-RL where each CEM instance updates the sampling distribution asynchronously.
The second type of approach aims at improving CEM itself.
Amos and Yarats [2020] proposes afully-differentiable version of CEM called DCEM .
The key is to make the top- k selection in CEMdifferentiable such that the entire CEM module can be trained in an end-to-end fashion.
Despitecutting down the number of samples needed in CEM, this method does not beat the vanilla CEMin benchmark test.
GACEM [Hakhamaneshi et al., 2020] increase the capacity of the samplingdistribution by replacing the Gaussian distribution with an auto-regressive neural network.
Thischange allows CEM to perform search in multi-modal solution space but it is only veriﬁed in toyexamples and its computation seems too high to be scaled to MBRL tasks.
Another method thatincreases the capacity of the sampling distribution is PaETS [Okada and Taniguchi, 2020] that usesa GMM with CEM.
It is the approach that we followed for our CEM-GMM implementation.
Therunning time results in the optimization task in Sec.3 shows that it is computationally heavier thanthe CEM and DecentCEM methods, limiting its use in complex environments.
Macua et al.
[2015]proposed a “distributed” CEM that is similar in spirit to our method in that they used multiple samplingdistributions and applied the top- k selection locally to samples from each instance.
However, theirinstances are cooperative as opposed to being independent as in our work.
They applied “collaborativesmoothed projection steps” to update each sampling distribution as an average of its neighboringinstances including itself.
The updating procedure is more complicated than our proposed methodand proper network topology of the instances is needed: a naive approach of updating according toall instances will lead to mode collapse since the resulting sampling distributions will be identical.
The method was tested in toy optimization examples only.
Overall, this second type of approach didnot outperform vanilla CEM, a situation that motivated our move to a decentralized formulation.
Experiments
We evaluate the proposed DecentCEM methods in simulated environments with continuous actionspace.
The experimental evaluation is mainly setup to understand if DecentCEM improves theperformance and sample efﬁciency over conventional CEM approaches.
6.1 Benchmark Setup
Environments We run the benchmark in a set of OpenAI Gym environments commonly used inthe MBRL literature: Pendulum, InvertedPendulum, Cartpole, Acrobot, FixedSwimmer 5 , Reacher,
Hopper, Walker2D, HalfCheetah, PETS-Reacher3D, PETS-HalfCheetah, PETS-Pusher, Ant.
Thethree environments preﬁxed by “PETS” are proposed by Chua et al.
Note that MBRLalgorithms often make different assumptions about the dynamics model or the reward function.
Theirbenchmark environments are often modiﬁed from the original OpenAI gym environments such thatthe respective algorithm is runnable.
Whenever possible, we inherit the same environment setup fromthat of the respective baseline methods.
This is so that the comparison against the baselines is fair.
More details on the environments and their reward functions are in Appendix B.
Algorithms The baseline algorithms are PETS
[Chua et al., 2018] and POPLIN [Wang and Ba,2020].
PETS uses CEM with a single Gaussian distribution for planning.
The POPLIN algorithmcombines a single policy network with CEM.
As described in Sec.4, POPLIN comes with twomodes: POPLIN-A and POPLIN-P with the sufﬁx “A” denotes planning in action space and “P”for the network parameter space.
We reuse default hyperparameters for these algorithms from theoriginal papers if not mentioned speciﬁcally.
For our proposed methods, we include two variations
DecentCEM-A and DecentCEM-P as described in Sec. 4 where the sufﬁx carries the same meaningas in POPLIN-A/P .
The ensemble size of DecentCEM-A/P as well as detailed hyperparameters for all algorithms are listed in the Appendix D.2.
We also included Decentralized PETS.
All MBRLalgorithms studied in this benchmark uses the same ensemble networks proposed by Chua et al.[2018] for model learning.
We also include a model-free RL baseline SAC [Haarnoja et al., 2018].
Evaluation Protocol
The learning curve shows the mean and standard error of the test performanceout of 5 independent training runs.
The test performance is an average return of 5 episodes ofthe evaluation environment, evaluated at every training episode.
At the beginning of each trainingrun, the evaluation environment is initialized with a ﬁxed random seed such that the evaluationenvironments are consistent across different methods and multiple runs to make it a fair comparison.
All experiments were conducted using Tesla V100-PCIE-16GB GPUs.
[Figure]
6.2 Results
[Figure]
Learning Curves
The learning curves of the algorithms are shown in Fig.
6 for InvertedPendulum,
Acrobot, Reacher, Walker2D, PETS-Pusher and Ant, sorted by the state and action dimensionality oftask.
The curves are smoothed with a sliding window of 10 data points.
The full results for all environments are included in
Appendix E.
We can observe two main patterns from the results.
One pattern wasthat in most environments, the decentralized methods DecentPETS , DecentCEM-A/P either matchedor outperformed their counterpart that took a centralized approach.
In fact, the former can be seen asa generalization of the later, by an additional hyperparameter that controls the ensemble size with sizeone recovering the centralized approach.
The optimal ensemble size depends on the task.
This additional hyperparameter offers ﬂexibility in ﬁne-tuning CEM for individual domains.
For instance in
Fig.
6, for the P-mode where the planning is performed in policy parameter space, an ensemble size oflarger than one works better in most environments while an ensemble size of one works better in Ant.
The other pattern was that using policy networks to learn the sampling distribution in general helped improvingthe performance of centralized CEMbut not necessarily in decentralizedformulation.
This is perhaps due tothat the added exploration from multiple instances makes it possible toidentify good solutions in some environments.
Using a policy net in suchcase may hinder the exploration dueto overﬁtting to previous actions.
Ablation Study A natural questionto ask about the DecentCEM-A/Pmethods is whether the increased performance is from the larger numberof neural network parameters.
Weadded two variations of the POPLINbaselines where a bigger policy network was used.
The number of thenetwork parameters was equivalent tothat of the ensemble of policy networks in DecentCEM-A/P .We
showthe comparison in Reacher(2) and
PETS-Pusher(7) (action dimension inparenthesis) in Fig.
In both action space and parameter space planning, a bigger policy network in
POPLIN either did not help or signiﬁcantly impaired the performance (see the POPLIN-P results inreacher and PETS-Pusher).
This is expected since unlike DecentCEM , the training data in POPLINdo not scale with the size of the policy network, as explained at the end of Sec.
Fig.
8 shows the impact of ensemble size under the action space planning (left) and parameter spaceplanning (right).
Since we ﬁx the total number of samples the same across the methods, the larger theensemble size is, the fewer samples that each instance has access to.
As the ensemble size goes up,we are trading off the accuracy of the sample mean for better exploration by more instances.
Varyingthis hyperparameter allows us to ﬁnd the best trade-off, as can be observed from the ﬁgure whereincreasing the ensemble size beyond a sweet spot yields diminishing or worse returns.
Figure 9 (Left) shows the cumulativeselection ratio of each CEM instanceduring training of DecentCEM-A withan ensemble size of 5.
It suggeststhat the random initialization of thepolicy network is sufﬁcient to avoidmode collapse.
We also plot the actionstatistics of the instances in Figure 9(Right).
For visual clarity, we showa time segment toward the end of thetraining rather than all the 10k steps.
DecentCEM has maintained enoughdiversity in the instances even towardthe end of the training.
DecentCEM-Pis excluded from both plots since itshows a similar trend as DecentCEM-A .
More ablations results are included in Appendix F.
Conclusion and Future Work
In this paper, we study CEM planning in the context of continuous-action MBRL.
We propose anovel decentralized formulation of CEM named DecentCEM , which generalizes CEM to run multipleindependent instances and recovers the conventional CEM when the number of instances is one.
Weillustrate the strengths of the proposed DecentCEM approach in a motivational one-dimensionaloptimization task and show how it fundamentally differs from the CEM approach that uses a Gaussianor GMM.
We also show that DecentCEM has almost sure convergence to a local optimum.
We extendthe proposed approach to MBRL by instantiating two decentralized CEM methods that combine withpolicy networks.
We show their efﬁcacy in benchmark control tasks and ablations studies.
There is a gap between the convergence result and practice where the theory assumes that the numberof samples grow polynomially with the iterations whereas a constant sample size is commonly usedin practice including our work.
Investigating the convergence properties of CEM under a constantsample size makes an interesting direction for future work.
Another interesting direction to pursue isﬁnite-time analysis of both CEM and DecentCEM .
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu-
tions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Limitations are discussed in Sec.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 6, Appendix A and Appendix B.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments
multiple times)?
[Yes] See Section 6 and
Appendix A.
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs,
internal cluster, or cloud provider)?
[Yes] See section 6.1.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
The assets (Gym, mujoco) is cited and well
known that their licences do not need to mention separately.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
informa-
tion or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on
participant compensation?
[N/A]