Strength Through Diversity: Robust Behavior Learning via Mixture Policies
Abstract: Efﬁciency in robot learning is highly dependent on hyperparameters.
Robot morphology and task structure differ widely and ﬁnding the optimal setting typically requires sequential or parallel repetition of experiments, strongly increasing the interaction count.
We propose a training method that only relies on a single trial by enabling agents to select and combine controller designs conditioned on the task.
Our Hyperparameter Mixture Policies (HMPs) feature diverse sub-policies that vary in distribution types and parameterization, reducing the impact of design choices and unlocking synergies between low-level components.
We demonstrate strong performance on continuous control tasks, including a simulated ANYmal robot, showing that HMPs yield robust, data-efﬁcient learning.
Introduction
The agent can seamlessly adapt its policy structure to the presented tasks by modulating component activations in a single trial.
Real-world autonomous robots require versatile controllers that continuously adapt behavior to changing environmental conditions and task speciﬁcations.
Reinforcement learning (RL) has driven success in modeling complex control strategies in games [1, 2], simulated robotics [3] and real-world systems [4, 5].
However, efﬁcient learning is often conditioned on good parameter selection and may require tuning for each task or domain [6].
Common approaches to hyperparameter optimization leverage parallel or sequential evaluation strategies.
While parallel strategies are computationally costly and need not improve on random search [7], sequential strategies [8, 9] are time-intensive with hybrid approaches trading-off one for the other [10].
Gradient-based methods enable online adaptation at the cost of requiring differentiable objectives.
Selecting parameters efﬁciently is essential to learning capable robot controllers.
The nature of motion planning problems further dictates suitable controller designs for learning.
Continuous control can represent intricate transitions in state-action space to yield highly-optimized behaviors through local exploration or generate smooth references for a low-level tracking controller.
Discrete control can leverage reduced resolution for coarse exploration and readily encodes bang-bang responses to switching dynamics.
Optimal controller selection does not have to be unimodal and can vary with different phases of a task and stages of the learning process.
It may then be advantageous to provide agents with a diverse set of controllers that differ in their parameterization.
This enables agents to select designs suitable for the presented task and unlock compositional synergies.
In order to support this type of compositionality, we extend the perspective of previous work on hierarchical reinforcement learning [11].
In this paper, we propose a hierarchical policy over a diverse mixture of low-level controllers to improve robustness and reduce the necessity for parameter tuning and related data requirements.
The low-level controllers are diverse in architecture, hyperparameters, and distribution characteristics to provide the robot with a rich set of controller designs.
Our approach then enables learning robots:
• to optimize a set of diverse controller designs concurrently for increased data-efﬁciency, • to self-select suitable controllers conditioned on the task for reduced human parameter tuning, • to compose behaviors from multiple controller designs for exploitation of emergent synergies.
We evaluate performance on a variety of torque-control tasks from the DeepMind Control Suite [12].
Additionally, we investigate learning of PD-control targets for the ANYmal robot in RaiSim [13], which was the foundation for Sim2Real transfer in Lee et al.
Throughout, we demonstrate that enabling agents to operate over a diverse set of controllers guards against individual failure modes and unlocks synergies between different controller designs.
While the high-level selector introduces its own hyperparameters, we demonstrate its robustness to loss of state information by modelling unconditional component selection and subjecting the selector to adversarial distractor components.
Related Work
The performance of deep RL algorithms is strongly tied to hyperparameter choices [6, 14, 15].
Commonly, hyperparameter optimization is performed in multiple experiments via sets of agents or tasks [16].
Simple parallel strategies include expert selection or grid-search [17, 18], and can be less efﬁcient than random search [7].
Bayesian Optimization provides more structure at the cost of sequential evaluations [9, 19, 20].
Evolutionary strategies [8, 21, 22] enable discontinuous optimization and can evolve parameters at different rates, but typically use sequential evaluation.
Population-Based Training (PBT) [10] alternatively evolves parameter variations online in parallel.
This requires populations of agents each with their own environment, leading to signiﬁcant data and computational requirements.
Our work is also related to neural architecture search [23].
In particular, similarities can be found to methods that render architecture search differential [24, 25], enabling direct optimization of the effectively-used architecture during a single experiment.
Optimizing hyperparameters during the lifetime of a single agent reduces these requirements.
Gradient-based optimization ([26, 27]) yields online adaptation when the objective is differentiable in the parameters.
HOOF [28] extends towards non-differential aspects and enables gradient-free off-policy training with hyperparameter schedules.
However, the optimized parameters need to directly affect the policy update, precluding e.g. application to architecture search.
Related methods have been used to learn architecture schedules [29].
With HMPs, we consider a single agent lifetime to reduce data and computational requirements.
We consider policies that vary in their parameters, architecture, and distributional characteristic.
Formulating a mixture over these diverse components allows us to evolve multiple controllers in parallel while the agent modulates activation based on expected performance.
The approach further enables combining controllers with different parameters.
The problem of hyperparameter choice can further be framed as a contextual bandit problem [30].
In comparison to mixture agents, this formulation does not natively share data across policies with different parameters.
Mixtures of specialized motion controllers naturally arise in form of motor primitives [31, 32].
The application of dynamical movement primitives [33] to robot control via a mixture library has been shown in [34].
Mixture distributions have a long-standing history to model diverse and multi-modal data [35].
In RL, they have been applied as a mixture of linear Gaussians in which component specialization is achieved by introducing entropy bounds [36] and provide an inference perspective to the options framework which models an agent via the separation into high-level controller and low-level behaviours [11, 37, 38].
Similarly, quality diversity algorithms evolve repertoires of diverse low-level skills which a high-level selector may act on [39, 40, 41].
We build on the training of mixture policies via weighted maximum likelihood optimization, which has previously been used in connection with information asymmetry to generate diverse, compositional skills [42].
Diversity in mixture policies has been explored to strengthen skill discovery by explicitly optimizing for diversity-related objectives [43, 44, 45, 46] with ﬁxed architectures and a single set of hyperparameters.
Preliminaries
We formulate controller optimization as a Markov Decision Process (MDP) deﬁned by the tuple {S , A , T , R , γ } , where S and A denote the state and action space, respectively, T : S ×
A → S represents the transition distribution, R : S ×
A → R the reward mapping, and γ ∈ [0 , 1) the discount factor.
We deﬁne s t and a t to be the state and action at time t , respectively.
Let π θ ( a | s ) denote a policy distribution with parameters θ and deﬁne the discounted inﬁnite horizon return
G t = (cid:80) ∞ t (cid:48) =
t γ t (cid:48) − t R ( s t (cid:48) , a t (cid:48) ) , where s t +1 ∼ T ( s t +1 | s t , a t ) and a t ∼ π θ ( a t
| s t ) .
Our goal is to learn the optimal policy maximizing G t under unknown dynamics and reward mappings.
This is typically done by modeling π θ ( a t | s t ) as a Gaussian distribution with a neural network predicting the mean and variance from s t .
In this work, we consider a more diverse class of policy distributions.
Hyperparameter Mixture Policies
Initialize: samples do for to target do
Sample batch of trajectories τ from memory B ,
N s actions from π θ k to estimate expectations
Compute gradients over batch for π θ , η , λ p , Q φ δ π ← −∇ θ (cid:80) s ∼D (cid:80) N s j =1 (cid:20) exp (cid:18) Q ( s,a j ) η (cid:19) · log π θ ( a j
| s ) (cid:21) + (cid:80) p λ p (cid:16) (cid:15)
p − E s ∼D (cid:2) KL ( π θ ||
π θ k ) (cid:3)(cid:17) (8) δ η ← ∇ η g ( η ) = ∇ η η(cid:15) + η (cid:80) s
∼D log 1 N s (cid:80) N s j =1 (cid:104) exp (cid:16) Q ( s,a j ) η (cid:17)(cid:105) (5) δ λ p ← ∇ λ p λ p (cid:16) (cid:15)
p − E s ∼D (cid:2) KL ( π θ ||
π θ k ) (cid:3)(cid:17) (8) δ Q ← ∇ φ (cid:80) ( s,a ) ∼D ( Q φ ( s, a ) − Q ret )(10)
Apply gradients to update π θ k +1 , η, λ p , Q φ
Update target networks π (cid:48) θ
= π θ , Q (cid:48)
φ =
Q φ
We propose Hyperparameter Mixture Policies (HMP) to train a hierarchical policy over diverse low-level controllers with distinct hyperparameters and distribution characteristics.
The resulting mixture is given by π θ ( a | s ) =
M (cid:88) i =
π Hθ ( i | s ) π Lθ ( a | s, i ) , (1) with π H ( i | s ) and π L ( a | s, i ) as the weight and probability density of component i .
Thus, π H is a high-level selector and π L a sub-policy from a diverse set of controller designs.
The agent then self-selects the most suitable controller for individual phases of a task or stages of the learning process.
This enables robust adaptation to a broad range of motion planning problems while reducing the necessity of manual parameter tuning and sequential experiment design.
4.1 Policy Improvement
We use an actor-critic algorithm where policy improvement relies on two stages as in [47].
First, a non-parametric policy q ( a | s ) is optimized on samples from the state-action value function Q π under the constraint of remaining close in expectation to the current parametric policy π θ .
The parametric policy is then updated to better approximate the non-parametric target.
By performing the actual policy improvement with a non-parametric policy, we bypass the need for gradient estimation via likelihood ratio [48] or reparametrization trick [49].
In addition, this perspective enables the optimisation of mixture distributions in reinforcement learning without continuous relaxation [50].
Step 1 - Fitting the Non-parametric Policy
As we do not have access to the ground-truth state-action value function Q we employ a learned approximation Q φ , parameterized by φ , and formulate the objective at iteration k as
[Equation]
where (cid:15) deﬁnes a trust-region around the current parametric policy, π θ k .
This can be solved to provide a closed-form relation in terms of the current parametric policy
[Equation]
where η is computed by minimizing the dual function
[Equation]
Step 2 - Updating the Parametric Policy
The parametric policy π θ is optimized to approximate the non-parametric policy q by minimizing their KL divergence as
[Equation]
Plugging in (4) and introducing an additional KL constraint to enable generalization beyond the sample distribution yields
[Equation]
The update proceeds via a Lagrangian relaxation of the KL constraint enabling the application of gradient-based optimization.
We further decouple the KL constraint and deﬁne independent constraints for each distributional parameter p in both the high-level selector and low-level control policies [47, 42].
We deﬁne these separately per diverse component to accommodate differences in the distributional parameter dynamics between low-level controllers (in comparison to e.g. Wulfmeier et al.
To enable training of diverse distributions with different constraints, we enforce the
KL constraints per component.
Additional changes to enable training of mixed continuous discrete policies are described in Section 4.3.
We obtain updated parameters θ k +1 as a solution of
[Equation]
where we sum over decoupled components, each only varying along its respective parameter p .
We also introduce component speciﬁc Lagrangian multipliers λ p and KL bounds (cid:15) p .
A two component mixture of e.g. a Categorical ( α 1 ) and a Gaussian ( µ 2 , Σ 2 ) would then yield p
=
{ α HL , α 1 , µ 2 , Σ 2 } .
4.2 Policy Evaluation
In order to stabilize off-policy learning of the state-action value function Q φ we leverage the Retrace algorithm [51].
Here, we truncate the inﬁnite series after 10 steps and bootstrap from the target state-action value network, with details provided in Appendix D. To increase efﬁciency, we further consider two-step transitions by squashing consecutive timesteps before adding them to memory.
4.3 Combining Continuous and Discrete Distributions
Continuous and discrete policies do not share the same support.
Furthermore, actions are subject to numerical cut-off errors.
In practice this can result in the action-likelihood of discrete policies being 0 most of the time.
To facilitate training with diverse mixtures, we approximate discrete components by piece-wise constant pseudo-densities for backpropagation.
Thus, out-of-distribution samples are mapped into the support for probability computation.
For query action a and a discrete mixture component i with ﬁnite support C i we obtain the corresponding piece-wise constant pseudo-density
[Equation]
where p i (˜ a | s ) is the probability of ˜ a in the original discrete distribution, 1 is the indicator function, and B δ (˜ a ) is a ball of radius δ around ˜
a (we use δ = 0 .
1 here).
This improves sharing of gradient information between continuous and discrete policies and enables discrete components to train on samples generated by continuous components to accelerate learning.
Experiments
We benchmark the performance of HMPs on continuous control in the DeepMind Control Suite [12], learning PD-control for ANYmal in RaiSim [52, 13], and manipulation tasks in Metaworld [53].
We further compare to the gradient-free hyperparameter optimizer HOOF [28] in OpenAI Gym [54].
To better isolate the effects of diversity, we consider a standard Gaussian policy and evaluate single parameter variations together with their composition into diverse HMPs.
We furthermore investigate
HMPs consisting of randomly sampled components and show that strong performance can be recovered.
Overall, we ﬁnd diversity to enable robust learning across tasks and to guard against failure modes of individual components.
Our ﬁgures visualize performance mean and standard deviation.
5.1 Qualitative Example
We provide an illustrative example of component specialization within a diverse policy in Figure 2.
The agent combines a localized Gaussian ( σ ini = 0 . 3 ) with a coarse Categorical ( n bin = 2 ) policy.
On a Cartpole swing-up task, the agent leverages bang-bang control for swing-up and continuous control for stabilization.
On a Cheetah locomotion task, continuous control coordinates the intricate contact phase and bang-bang control quickly retracts the limbs during the ﬂight phase.
Applying t-SNE dimensionality reduction yields consistent clustering across trajectories, indicating consistent component specialization.
This aligns with human intuition and highlights the promise of composition, further motivating HMPs and analysis of synergies in heterogeneous mixture policies.
5.2 Heterogeneous Distributions
Compositional Solutions We further evaluate synergies arising from heterogeneous mixtures by combining a narrow Gaussian policy ( σ ini = 0 . 3 ) with a Categorical policy (with n bin ∈ { 2 , 9 } ).
Figure 3 indicates that performance of the Gaussian signiﬁcantly improves in combination with a bang-bang policy ( n bin = 2 ) for torque-control (panels 1-5) (see also [55]).
Conversely, the Gaussian guards against the failure mode induced by bang-bang control on ANYmal (panel 6).
We can further replace the Gaussian with a more ﬁne-grained Categorical ( n bin = 9 ) to reach comparable performance on the Control Suite tasks.
However, the resulting mixture cannot compensate for bangbang signals on ANYmal as discrete control is not well-suited for position reference generation.
Diverse Distributions We broaden our analysis of diverse distributions and consider a Gaussian ( σ ini = 1 . 0 ), Kumaraswamy ( c ini = 1 . 0 ), Categorical ( n bin = 5 ) and Discrete Gaussian ( n bin = 5 ), as well as their combination into an HMP which we refer to as NKCD.
Figure 4 highlights that the
HMP is able to solve all tasks, guarding against individual component failure (e.g. K on Quadruped) or premature convergence (e.g. D on Reacher).
On ANYmal, the HMP leverages the Kumaraswamy policy to outperform the Gaussian policy, while the Kumaraswamy is suppressed on the Humanoid to reach strong performance.
This underlines the robust performance that diverse HMPs provide by evaluating multiple controller designs jointly, reducing environment-speciﬁc tuning and guarding from component failure.
We compare the NKCD HMP to the RHPO [42] and DDPG [56] agents.
RHPO leverages a homogeneous mixture policy consisting of 4 MPO-type Gaussians ( σ ini = 1 . 0 ).
Figure 4 shows that the HMP and RHPO outperform DDPG on all tasks.
The HMP performs competitively with RHPO throughout and outperforms RHPO on the real-world inspired ANYmal and manipulation domains.
This underlines HMP’s ability to enable data-efﬁcient learning by training multiple policy designs in parallel and transferring problem-speciﬁc controller selection to the agent.
Random components We consider sampling sub-policies with random hyperparameters.
This includes randomizing distribution type, initialization, architecture and activations of each component.
Figure 5 provides performance statistics across 10 random instances of an HMP with 10 random components.
We observe that random selection yields performance competitive with the optimized RHPO agent.
Beyond random selection, the engineer may restrict the space of available sub-policies to inject structural priors into the learning process.
Gradient-free optimization We compare the
NKCD HMP to HOOF [28], which introduced a method for gradient-free hyperparameter optimization and evaluated on OpenAI Gym [54]
Their results are provided in Figure 6 for reference.
We note that our diverse mixture displays competitive performance on these benchmarks without any ﬁne-tuning for Gym domains.
5.3 Hyperparameter Variations
In the following, we evaluate performance of different hyperparameter combinations on continuous control tasks from the DeepMind Control Suite and locomotion on the ANYmal robot in RaiSim.
We consider a standard MPO parameterization as the baseline and analyze the impact of combining potentially sub-optimal parameter variations.
To account for increased model capacity in mixture policies, we compare to a RHPO-type homogeneous mixture with standard MPO parameters.
Initialization We vary the initial standard deviation of Gaussian policy heads as this can signiﬁcantly impact the rate of convergence.
The bounded action space of the agent is a ∈ [ − 1 . 0 , +1 . 0] |A| .
We consider the initial values σ ini = { 0 . 3 , 1 . 0 , 3 . 0 } with the standard literature value σ sini = 1 . 0 .
Figure 7 (row 1) indicates that the Control Suite tasks favor large variance to drive exploration, while generating position targets on ANYmal requires low variance to avoid instability of the PD controller and subsequent falling.
Generally, we ﬁnd that a diverse policy improves performance over the weaker component, yielding a robust controller that can prevent individual failure modes.
This is evident for ANYmal Run, where the high variance policy fails but the diverse mixture succeeds.
Architecture We vary the layer structure with π l ∈ { [20] , [200] , [200 , 200] } and standard value π sl =
Figure 7 (row 2) indicates that performance is robust to architecture variations with increased capacity slightly improving performance.
The heterogeneous mixture performs slightly better on the Control Suite tasks while the homogeneous mixture is slightly better on ANYmal Run.
Activations We vary the policy activations with π a = { ELU, Leaky ReLU, Tanh } and π sa = ELU .
Figure 7 (row 3) shows that the heterogeneous mixture improves performance over both the homogeneous mixture and the individual components on the Control Suite tasks.
On ANYmal, combining
Leaky ReLU and Tanh activations initially causes quick episode termination and delayed learning.
Based on individual performance this is surprising and could indicate that ELU activations are better suited for composition on this task.
This is the only instance where we observe reduced performance.
Learning Rate We vary the policy learning rates such that α LR ∈ 2 × { 10 − 5 , 10 − 4 , 10 − 3 } and α sLR = 2 × 10 − 4 .
Figure 7 (row 4) indicates that smaller learning rates reduce efﬁciency.
However, pairing a fast with a slow head yields competitive performance, signiﬁcantly improving over the individual slow component, and outperforms a nominal mixture on the Finger and Quadruped tasks.
Conclusion
Finding the right hyperparameters has a considerable impact on performance when enabling robots to learn complex behaviors through interaction and recent progress in machine learning can often be traced back to better hyperparameter settings [57].
A sub-optimal algorithm with thoughtfully tuned hyperparameters easily outperforms a state-of-the-art approach that has not been tuned sufﬁciently.
Tuning requires both domain knowledge and experience with the underlying algorithm.
Even then, it still incurs a considerable computational cost that is particularly limiting when relying on real-world data.
Our work proposes the use of diverse mixture policies to effectively mitigate this challenge.
Moreover, we demonstrate the beneﬁts of combining different distribution types and policy parameterizations from a perspective of compositionality in skill learning.
Our Hyperparameter Mixture
Policies (HMPs) induce diversity that can help in component specialization during different phases of a task, e.g. where certain movements require either coarse or ﬁne-grained control.
It has also the potential to accelerate the learning process, e.g. where more extreme actions enable faster exploration.
The approach is easy to use and yields competitive performance across a range of common torque-control benchmark tasks, as well as for generating PD-control targets within a high-ﬁdelity simulation of the ANYmal quadruped, without extensive parameter tuning.
While learning algorithms can always beneﬁt from additional tuning, our approach increases robustness and helps to accelerate research in reinforcement learning for complex dynamic robots, in particular when there is no access to extensive computational resources. Acknowledgments
Tim Seyde, Wilko Schwarting, Igor Gilitschenski and Daniela Rus were supported in part by the
Ofﬁce of Naval Research (ONR) Grant N00014-18-1-2830, Qualcomm and Toyota Research Institute (TRI).
This article solely reﬂects the opinions and conclusions of its authors and not TRI,
Toyota, or any other entity.
We thank them for their support.
The authors further would like to thank
Lucas Liebenwein for assistance with cluster deployment and acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources.
We would also like to thank the reviewers and program chairs for their helpful feedback and suggestions for improvement