Specializing Versatile Skill Libraries using Local Mixture of Experts
Anonymous Author(s)
AfﬁliationAddress email
Abstract: A long-cherished vision in robotics is to equip robots with skills thatmatch the versatility and precision of humans.
For example, when playing tabletennis, a robot should be capable of returning the ball in various ways while precisely placing it at a desired location.
A common approach to model such versatilebehavior is to use a Mixture of Experts (MoE) model, where each expert is a contextual motion primitive.
However, learning such MoEs is challenging as mostobjectives force the model to cover the entire context spaces, which prevents specialization of the primitives resulting in rather low quality components.
Startingfrom maximum entropy reinforcement learning (RL), we decompose the objective into optimizing an individual lower bound per mixture component.
Further,we introduce a curriculum by allowing the components to focus on a local context region, enabling the model to learn highly accurate skill representations.
Tothis end, we use local context distributions that are adapted jointly with the expertprimitives.
Our lower bound advocates an iterative addition of new components,where new components will concentrate on local context regions not covered bythe current MoE. This local and incremental learning results in a modular MoEmodel of high accuracy and versatility, where both properties can be scaled byadding more components on the ﬂy.
We demonstrate this by an extensive ablation and on two challenging simulated robot skill learning tasks.
We compare ourachieved performance to HiREPS, a known hierarchical policy search method forlearning diverse skills and LaDiPS.
Introduction
Human motor skills are precise and versatile, which allows us to perform motor tasks in differentways while achieving a consistent movement quality.
For example, when playing table tennis, wecan hit the ball in various ways while still targeting a speciﬁc landing point of the ball on the opponent’s side of the table.
Another example is grasping an object which lies behind an obstacle.
Wecan grasp even small objects precisely with different grasp types while avoiding the obstacle.
Suchversatile skills are crucial if we want to employ robots in unstructured and dynamically changingenvironments.
Such skills, often represented as movement primitives, were already successfullylearned for challenging robot learning tasks, e.g., the ball-in-a-cup [1, 2] task, by a variety of policy search algorithms [3].
Yet, most of these algorithms cannot ﬁnd multiple, versatile, and precisesolutions to the multi-modal solution space, as they usually assume a Gaussian policy [4, 2, 3, 5].
In this paper, we model versatile behavior with contextual skill libraries of motion primitives [6, 7],formalized by Mixtures of Experts (MoEs).
Here the context deﬁnes task properties, e.g., reachingdifferent goal positions or different friction parameters of an object to manipulate [5]. Our goal isto learn versatile skills, i.e. different movement styles to solve a given context.
Given a context, the MoE ﬁrst selects a component, i.e., a motion primitive, to execute.
Subsequently, the componentadjusts the primitive’s parameters and the primitive is executed.
Such models are already commonlyused [8, 9].
Yet, the quality and versatility of the learned skill libraries remains limited using existing algorithms.
Most algorithms are based on expectation-maximization (EM) [10] techniques [8], and suffer from local optima and mode averaging [10] which prevents the single components fromspecializing in a local context region.
Moreover, existing algorithms [8, 9] do not explicitly optimizethe versatility of the library.
Hence, they often yield degenerated libraries with only a single movement style.
We propose a new objective for learning contextual, precise, and versatile MoE modelsbased on a maximum entropy formulation.
We also introduce a learnable context distribution, whichprovides a curriculum for each component of the MoE model. We use a variational lower bound [11]to decompose the objective into individual updates for the components and their related local context distributions, allowing the components to specialize in local regions of the context space andpreventing mode averaging.
Due to the curriculum, the MoE does not have to cover the whole context space during training, which prevents the averaging effects that negatively affect most otherapproaches.
Yet, not covering the whole context space leads to poor performance for some contexts,which is also undesirable.
Thus, we introduce a heuristic-free mechanism to add new componentsduring training until the whole context space is covered.
Hence our algorithm provides a modularapproach that learns highly precise and versatile skills that cover the whole context space.
We evaluate our approach in a simulated beer pong and a table tennis environment. Both environments allow different motion styles to solve the task, which are discovered by our algorithm.
Moreover, we present ablation studies showing the importance of the single elements and hyperparameters of our algorithm.
Related Work
Contextual Episodic Policy Search.
Episodic policy search [3] aims at maximizing the expectedreturn by optimizing the parameters θ of a controller, e.g., a motion primitive [12, 6].
Most approaches use a stochastic search distribution π ( θ ) over the parameter space and aim to optimize theexpected return under this search distribution [3], i.e., [Equation] where R ( θ ) = (cid:80) t r t is the summed reward over a whole episode obtained when executing controller parameter θ . As it is common in the literature, we will denote π ( θ ) as our policy even thoughit only indirectly chooses the control actions of the agent by selecting the controller parameters θ .
Different optimization schemes such as policy gradients Sehnke et al.
[13], natural gradients [14],stochastic search strategies [15, 16] or trust-region optimization techniques [8, 17] have been used. Researchers extended these approaches to the contextual setting [18, 4], where the search distribution π ( θ | c ) now depends on a context vector c which describes the task, e.g., a goal location toreach. The contextual objective is typically formalized as [Equation]
where p ( c ) is the given distribution over context vector and the rewards now also depends on thecontext.
Klink et al.
[2] introduce a curriculum into contextual policy search.
By having an adaptabledistribution over the contexts, they allow the agent to concentrate on easy-to-solve contexts ﬁrst andthen generalize to the whole context space.
Versatile Skill Learning.
The Hierarchical Relative Entropy Policy Search (HiREPS) algorithm[8] extends the classical Relative Entropy Policy Search (REPS) approach [19] to MoEs, whichallows learning versatile skills in a contextual episodic policy search setting.
In a similar approach,
Layered Direct Policy Search (LaDiPS) [9] also uses MoE policies, but builds on Model-Based
Relative Entropy Policy Search (MORE) [17] instead of REPS.
Both HiREPS and LaDiPS addressthe same problems as our approach, yet there are also considerable differences.
First, HiREPSjointly optimizes the whole mixture model and introduces an additional constraint, which boundsthe entropy loss of the responsibilities in each iteration. This constraint is crucial to obtain versatileand well-performing solutions.
LaDiPS uses separate updates for the different parts of the mixturebut also relies on additional constraints, where the entropy of the gating is lower bounded with aconstant value.
In contrast, for our approach, the objective and separate updates of the individualmixture parts follow naturally from the maximum entropy formulation.
Second, neither HiREPSnor LaDiPS uses a curriculum for training.
Thus, in both approaches, the MoE always has to coverthe whole context space and, hence, the components cannot specialize.
Variational Inference.
Our algorithm is also based on several recent advances in variational inference [11, 20, 21].
It is well known that maximum entropy RL is equivalent to inference in an appropriate probabilistic model [22].
Similar to previous works [23, 24], we exploit this relation anddraw inspiration from recent research into variational inference and density estimation for Gaussianmixture models and MoEs [11, 20, 21].
We reformulate the lower bound objectives introduced inthose approaches for our maximum entropy RL setting and extend them with a curriculum for themixture components.
Related Step-Based Approaches.
In the step-based setting, the policy does not learn a functionfrom contexts to parameters of an episodic controller but directly maps from system states to actionsand the policy updates are performed with the information from each time-step.
Practitioners oftenuse deep neural networks to parameterize step-based policies, giving rise to the ﬁeld of deep RL.
In this context, versatile policy learning is also a very active research area [25, 26, 27, 28].
Theseapproaches use a similar MoE model where the mixture component is only chosen in the beginning of an episode. Yet, the component is chosen randomly without conditioning on a context orstate variable. Moreover, these approaches reformulate a mutual information based objective into amaximum entropy objective while we develop a more direct maximum entropy maximization.
Curriculum Learning.
Researchers also worked on introducing curricula into deep RL.
In a ﬁrstapproach Ghosh et al. [29] proposed partitioning the initial state distribution using clustering.
Theythen learn individual policies for each partition while keeping the partitioning ﬁxed.
Strictly, this isnot a curriculum as the partitioning is not adjusted, yet it still allows specialization of the individualpolicies in different regions of the state space.
To automatically generate and adapt a curriculumfor deep RL approaches, Klink et al.
[30] extended their approach from the episodic setting [2] tothe step-based setting.
Yet, neither of these approaches is addressing versatility.
While we followan episode-based approach, both methodologies have its own beneﬁts and limitations [3] which are,however, not the focus of this paper and therefore further discussed in the appendix where we alsooffer a quantitative comparison to a step-based approach for a common benchmark task.
Options.
A related hierarchical approach is the options-framework [31, 32, 33]. The optionsframework extends the standard MDP to a semi-MDP to include a temporal abstraction of low-levelcontrol policies.
Given a termination condition, the executed low-level policy can be terminatedand another can be turned on.
Our policy structure can be seen as a simpliﬁcation of the optionsframework where the option is only selected in the beginning of each episode. Yet, the optionframework does not explicitly address learning versatile skills.
Specializing Versatile Mixture of Expert Models
In this section, we derive a lower bound to optimize each component and its corresponding contextdistribution independently.
To allow versatile solutions, we employ a MoE model as policy representation which is given as π ( θ | c ) =
(cid:80) o
π ( o | c )
π
( θ | c , o ) , where π ( o | c ) is the gating distribution,assigning a probability to component o given the context c and π ( θ | c , o ) is the component distribution for component o , which adapts the motion primitive
’s parameters θ to the given context c .
Inorder to implement a curriculum over the context c , we also introduce a learned context distributionπ ( c )
=
(cid:80) o
π ( c | o ) π
( o ) , which is also a mixture model speciﬁed by the component-wise contextdistribution π
( c | o ) and the component weights π ( o ) .
By applying Bayes’ Rule to replace the gatingdistribution π
( o | c ) , we can now rewrite the general mixture of experts model as [Equation] This policy deﬁnition allows each component o to adjust its curriculum by explicitly optimizing forπ ( c | o ) (Section 3.3) and thus concentrating on a local region in the context space.
We model thecomponents π ( θ | c , o ) as a linear conditional Gaussian distribution and the component-wise state distribution π ( c | o ) as a Gaussian.
The prior weights π ( o ) deﬁne a categorical distribution.
Throughoutthe next sections, we denote every probability distribution which is adaptable through the optimization process with π as it is part of our policy π ( θ | c ) . Furthermore we show the full derivations forthe next sections in the Appendix A.
3.1 Maximum Entropy Skill Learning with Curriculum
We consider a maximum entropy objective [22] for episodic policy search, i.e.,
[Equation]
where p ( c ) is the task speciﬁc context distribution, R ( c , θ ) is the reward function, H ( π ( θ | c )) =− (cid:82) θ π ( θ | c ) log π ( θ | c ) d θ the entropy and π ( θ | c ) is our MoE model.
The reward maximizationenforces preciseness while the entropy bonus enforces versatility.
However, the standard maximumentropy objective does not allow each mixture component to create its own curriculum, since anoptimization over the component-wise context distributions π ( c | o ) is not given.
Inspired by thework from Klink et al.
[2], we extend and modify the objective to
[Equation]
[Equation]
where α and β are scaling parameters, H ( π ( θ | c ))
= − (cid:82) θ
π ( θ | c ) log π ( θ | c ) d θ is the entropy and KL ( π ( c ) (cid:107) p ( c )) = (cid:82) c π ( c ) log π ( c ) p ( c )
d c denotes the KL-divergence. Note the difference in the optimization variables compared to Eq. The Kullback-Leibler (KL) term ensures that the contextdistribution π ( c ) is close to the task speciﬁc context distribution p ( c ) while π ( c ) can choose to havelow probability in regions of the context space where the MoE model is performing poorly.
Notethat this objective is similar to the negative I-projection of the joint distribution π ( c , θ ) used in variational inference.
Here, we also exploit the properties of the I-projection for learning the contextdistribution – the I-projection is mode seeking instead of mode-averaging and therefore allows fora specialization on a local context area.
Yet, the given objective is difﬁcult to optimize for mixturemodels as the sum over the mixture components is appearing inside the log terms of the entropy andthe KL.
However, similar to [11], we can replace π ( θ | c ) in Obj. (3) with our mixture model andapply Bayes theorem to arrive at
[Equation]
The exact derivations are given in the Appendix A. Note that Eq.
(4) is equivalent to Eq.
(3),yet, instead of the entropy for the whole mixture model, it now contains entropy terms for eachhierarchy layer of the MoE model, i.e., H [ π ( θ | c , o )] , H [ π ( c | o )] and H [ π ( o )] , which are muchsimpler to compute.
We also introduced log responsibilities π ( o | c ,
[Equation]
| c ) /π ( θ | c )and π ( o | c ) = π ( c | o ) π ( o ) /π ( c ) , occurring in the augmented rewards.
They return a high negativereward for component o , if the context-parameter pair ( c , θ ) or the context sample c is alreadycovered by another component, pushing the component to uncovered regions of the parameter spaceor context space respectively. Yet, the regions for the components will still overlap due the entropybonuses for π ( θ | c , o ) and π ( c | o ) . Without this reward augmentation, each component could beoptimized completely independently in Eq. (4) using a max-entropy objective.
However, in thiscase, all components would concentrate on learning the best solution irrespective of whether thissolution has already been covered by another component.
Yet, the log responsibilities still hinderus from optimizing each component π
( θ | c , o ) and its corresponding context distribution π ( c | o )independently, since the sum over o from the mixture models π ( c ) , π ( θ | c ) respectively appears inthe log term.
In the following sections we show, how we can overcome this limitation by introducinga
lower bound inspired by variational inference [11]. As we consider each possible context as equallyimportant, p ( c ) is assumed uniformly distributed in a given interval in the following and thus, canbe neglected in the objective.
3.2 Lower-Bound Decomposition for Component Distributions
In order to maximize the objective in Eq. (4) for each component π ( θ | c , o ) individually, we can ﬁrstextract the terms which only depend on π ( θ | c , o ) for a speciﬁc o as
[Equation]
The responsibilities are still hindering us to optimize each component π
( θ | c , o ) independently. However, similar to [11], we can obtain a tight lower-bound by introducing a variational distribution
˜ π ( o | c , θ ) and replacing the responsibilities in Eq. (5) with ˜ π
( o | c , θ ) .
This variational distribution isﬁxed during the optimization and can be computed according to the last policy model allowing usto update each component independently.
It is easy to show that after the update of the variationaldistribution ˜ π ( o | c , θ ) = π old ( o | c , θ ) , the introduced lower bound is tight.
Please refer to the appendix. The resulting lower bound is a standard maximum entropy RL objective with an additionalreward augmentation of α log ˜ π
( o | c , θ ) and thus can be optimized with any suitable Policy Searchalgorithm. Here we use a maximum-entropy-adjusted version of contextual MORE [18].
3.3 Lower-Bound Decomposition for Context Distributions and Prior Weights
To update π ( c | o ) for a speciﬁc o , we extract the relevant terms from Obj. [Equation]
[Equation]
where L c ( o, c ) = E π ( θ | c ,o ) (cid:2) R ( c , θ )
+ α log ˜ π ( o | c , θ ) (cid:3) + α H [ π ( θ | c , o )]
d θ corresponds to the expected augmented maximum entropy objective of component π
( θ | c , o ) in context c and ˜ π ( o | c ) =
π old ( o | c ) is a second variational distribution, which we introduced to be able to optimize for eachπ ( c | o ) individually.
Similarly to the previous section, the objective given in Eq. (6) is a tight lowerbound to the original objective where the responsibilities π ( o | c ) are used instead of ˜ π ( o | c ) .
Weapproximate the integral over θ with a single sample, as we typically only have a single parametersample per context available.
Yet, as we still have the outer expectation E π ( c | o ) in Eq.
(6) whichis approximated by multiple context samples, the whole monte-carlo estimation of the expectationsis still unbiased and with low variance.
After the optimization step (Eq. 6), we obtain the optimalsolution π ∗ ( c | o ) and tighten the bound by setting ˜ π
( o | c ) =
π ∗ ( o | c ) and ˜ π
( o | c , θ ) =
π ∗ ( o | c , θ ) .
Again, this lower bound corresponds to an augmented maximum entropy RL objective and thus canbe updated with any suitable policy search method.
We use an adjusted version of MORE [17],which is also used in [11].
Finally, we can formulate the objective for updating the component weights π ( o ) , which resemblesa lower bound of the original Objective (4) and corresponds to the highest hierarchy in our updatescheme.
The objective is given as
[Equation]
which is a maximum entropy RL objective for categorical distributions.
Here, we use REPS [19].
To summarize, we split the initial objective in Eq. (4) into different hierarchies, allowing us to optimize the different terms in our mixture model individually.
Starting by ﬁrst updating the componentsπ ( θ | c , o ) using the maximization problem in Eq. (5), we can optimize for the component-wise context distributions π ( c | o ) with Obj.
(6) after tightening the bound.
The components π ( θ | c , o ) adjustthe movement primitive parameters θ given a context c , while the component-wise context distributions π ( c | o ) ensure that the components only see context samples from a local context region. Finally, we update the weight distribution π ( o ) using Eq.
Algorithmic Details and Addition of Components.
We initialize our algorithm with only onecomponent and incrementally add components, and its corresponding context distribution, randomly.
We ﬁx all components except for the newly added one and optimize it for K iterations to let itdiscover new solutions in yet undiscovered context regions. For updating π ( θ | c , o ) (Obj. wesample from the local context distribution π ( c | o ) .
By also updating π ( c | o ) according to Obj.
(6),the components can adjust their curriculum and search for their favored context regions.
After Kiterations we add a new component and repeat the procedure.
Due to the augmented rewards thenew component will focus on undiscovered solutions and the local context distribution will coveruncovered areas of the context space.
Note that such a simple adding procedure is only possibledue to the mode-seeking properties of the I-projection, as the components do not need to averageover multiple modes but can specialize on a local context region.
We also ﬁx the weights π ( o ) tobe uniformly distributed among all components during learning, since otherwise components whichare already fully trained might dominate the optimization.
By allowing to ﬁne-tune all componentsevery H iterations, the already added components can adjust to the newly added ones.
After ﬁnishingadding components, we update the weights π ( o ) at the end of our optimization procedure.
Thevariables K and the number of components added in total are task-dependent.
The algorithm isdescribed in more detail in the Appendix B. [Figure]
Experiments
We start by investigating the importance of the different terms of the objectives derived in Section3 and subsequently evaluate the versatility and precision of the learned skills on simulated robotbeer pong and table tennis experiments.
In Appendix C we report all hyperparameters and providea comparison and analysis to a step-based policy search strategy (PPO [34]) in different scenarios.
4.1 Ablation Studies
We investigate the importance of the augmented rewards on a 10-link planar reaching task with twodimensional context space.
For this purpose, we update the component distributions π ( θ | c , o ) andthe corresponding context distributions π ( c | o ) by optimizing the objectives in (5, 6) with i) considering the responsibilities (as given by our algorithm) and ii) by setting log ˜ π ( o | c , θ ) and log ˜ π
( o | c )to zero.
Furthermore we compare the solutions found by our method to the solutions found by the SOTA method HiREPS.
Versatility is Induced by the Augmented Rewards.
In the 10-link planar reaching task (see Figure1a) the robot has to reach the red dots with its end-effector in a 2d context-space, while avoiding therectangle-shaped obstacles.
By adding 60 components, we have trained both versions (i and ii) over15 seeds/trials and have chosen the best performing parameter constellations (Fig. 1a) for i), Fig.1d) for ii)).
We then picked the ﬁrst model and sampled for each of the shown six contexts vectors(red dots) 100 samples and plotted the corresponding mean of each sampled component.
The plotsfor i) (Fig. 1a) + Fig. 1b)) show versatile – several modes to the same context – and precise – smalldistance of end-effector (green dot) to goal (red dot), while avoiding obstacles– solutions while fullycovering the context space (Fig. 1c).
A trend of covering more modes with higher α can also be seenin (Fig. 1b)).
For the case without the responsibilities ii) (Fig. 1d)), the solutions are not preciseand invalid –reaching through the obstacles–. As Fig. 1f)) shows, the context distributions focus oneasy-to-solve context regions (top and bottom part) and do not cover the full context space, leadingto extrapolated solutions from components that are not trained for these contexts (red dots).
Comparison to HiREPS.
We pick the best performing model among 15 seeds/trials after hyperparameter optimization.
In Fig. 1e) the mean solutions of the sampled components –sampled in thesame way as before – can be seen.
HiREPS shows less diverse and qualitative solutions, where only3 of the 60 components were chosen in total. 5 3 · 10 6 − −−
NumberEpisodicSamples
M ea n R e w a r d s
OursHiREPSLaDiPS
[Figure]
NumberEpisodicSamples
M ea n R e w a r d s
OursHiREPSLaDiPS ( [Figure]
Rewards- TT − 0 .
3 − 0 .
2 − 0 .
3 − 2 .−− 1 .− 1 .− 1 .− 1 .
[Figure]
− 1 − 0 . 0 .1 − 0 .
8 − 0 .− 0 . 4 − 0 .0 0 .0 .
[Figure]
[Figure] [Figure]
[Figure]
Experiments. In the BP experiment –left of (h)– the robot has to throw the red ball into a cup on the table.
In the TT experiment –right of (h) – the robot hast to return the ball to the opponent’s table side. The learned mixture of expert (MoE) models show high quality of the skills, reﬂected in the high reward values (a + b) and dense coverage of the context spaces, i.e., the 2-dim. position of the cup (c) for BP and the 4-dim desired outgoing landing and incoming landing ball positions (d) for TT .
For BP we also compare the success rates (92% in average) of our approach (e) and HiREPS (75% in average) (f) throughout the context space, where we consider a trial successful if the ball goes into the cup.
For BP versatile skills induce versatile ball trajectories for a given context (g). Versatile strikes for the TT experiment are shown in Fig. 3) 4.2 Simulated Robotic Experiments We test and compare our algorithm on a Beer Pong and Table Tennis simulation environment in
MuJoCo [35], where we have chosen HiREPS [8] and LaDiPS [9] as baseline.
We believe thatboth methods are suitable algorithms to compare, since each of them is a contextual policy searchalgorithm and consider optimizing a mixture model. Throughout our experiments we choose probabilistic movement primitives (ProMP) [6] as policy representation, where depending on the weightsθ desired trajectories are generated and subsequently tracked with a PD-controller. In our experiments, given a context c , the components π ( θ | c , o ) adjust the weight vector and the length of thetrajectory, which are summarized in the vector θ . We consider non-markovian rewards, in whichthe reward depends on the history of state and actions.
This type of reward function is not applicable to common step-based RL methods which build on markovian properties.
As for analyzing theresults of the experiments, we focus on the questions i) how does our algorithm perform comparedto SOTA baselines, ii) are we able to cover the whole context space and iii) are we able to learnversatile skills?
Beer Pong.
The goal of the Barret WAM robot is to throw the red ball into the cup on the table.
The2-dim contexts resemble the position of the cup on the table.
We incrementally add 70 componentsin our method, while HiREPS and LaDiPS directly start with 70 components.
We have run all algorithms over 20 seeds/trials and report the performance in Fig. 2a) 1 , where we plot the mean rewardwith two times the standard error.
While HiREPS already converges after around half a million samples, we are able to quickly outperform it.
Since LaDiPS uses intra-option learning, it outperformsour method in terms of sample efﬁciency.
However, with increasing number of components we areable to achieve a higher end-reward indicating that we can learn more qualitative solutions.
Ouralgorithm allows to cover the whole context space with the learned component-wise distributions(see Fig. 2c)) resulting in a high end-reward.
This performance is reﬂected in Fig. 2e), where wehave divided the context space into a ﬁne grid and sampled 100 times for each context a componentfrom which we have executed the mean.
We repeated that for all 20 different models and plot the mean success rate of throwing the ball into the cup.
We did the same procedure for HiREPS in Fig.2f).
We can observe that HiREPS has much more darker rectangles, showing that the success ratein these context regions is low.
Although, versatility is encouraged in the joint space of the robot,different joint trajectories often yield different ball trajectories.
Given one context, in Fig.
2g) weshow the z-coordinates of the ball trajectories over time resulting from sampling 20 times from the MoE model.
Each component leads to different number of ”ball-jumps”.
In Appendix C.2.1 wequalitatively show that we can learn more versatile solutions than LaDiPS and report that we canachieve a much higher expected mixture entropy (Fig. 4), which indicates that our solutions are versatile. Table Tennis. The task of the robot is to return different incoming balls to desired targets on theopponent’s table side on different ways.
We consider a four dimensional context space, includingball’s initial serve position and ball’s target landing position (right and left half of table Fig. 2d)).
Forboth parts of the context, we ﬁx the z position and vary the x and y coordinates.
We incrementallyadd 50 components in our method, while HiREPS and LaDiPS directly start with 50 components.
We have run all algorithms over 20 seeds/trials and report the performance in Fig. 2b) 1 , in which themean reward with two times the standard error is plotted.
The component-wise context distributionsare spread among the context space (Fig. 2d)) and allow each component to locally specialize ona context region.
This high coverage of the context space allows for a high reward achievement,outperforming HiREPS and LaDiPS.
In Fig. 3), we show three different striking skills sampledfrom our trained MoE model, for a ﬁxed context, i. e. ﬁxed serving and desired ball position.
Theﬁrst two skills use the green side of the racket (forehand) while the third skill uses the red side of theracket (backhand).
Conclusion
We proposed a new objective for learning contextual and versatile Mixtures of Experts (MoE) models.
We based our objective on a maximum entropy formulation to increase the versatility of thesolutions and introduced a curriculum to allow the components to specialize.
Our formulation alsoallows for easy online adaptation of the model complexity during training.
We conducted an ablation to show the importance of the individual parts of our objective.
Further, we showed thatour method learns precise and versatile solutions, which outperform the baseline on sophisticatedsimulated robotic tasks.
This work aims at presenting a mathematically well-founded method anddemonstrates its general feasibility on various challenging tasks.
Currently, the major drawbackof our approach is sample efﬁciency, as we do not share experience between the components.
Weintend to address this issue in future work, e.g., by intra-option learning.
Another direction for future research is extending the approach to more complex models, such as non-linear mixtures ofexperts.
We expect to need fewer components to cover the whole context space with more complexcomponent model representations.