Low-Rank Modular Reinforcement Learning via Muscle Synergy
Abstract
Modular Reinforcement Learning (RL) decentralizes the control of multi-joint robots by learning policies for each actuator.
Previous work on modular RL has proven its ability to control morphologically different agents with a shared actuator policy.
However, with the increase in the Degree of Freedom (DoF) of robots, training a morphology-generalizable modular controller becomes exponentially difﬁcult.
Motivated by the way the human central nervous system controls numerous muscles, we propose a Synergy-Oriented LeARning (S OLAR ) framework that exploits the redundant nature of DoF in robot control.
Actuators are grouped into synergies by an unsupervised learning method, and a synergy action is learned to control multiple actuators in synchrony.
In this way, we achieve a low-rank control at the synergy level.
We extensively evaluate our method on a variety of robot morphologies, and the results show its superior efﬁciency and generalizability, especially on robots with a large DoF like Humanoid ++ and UNIMALs.
Introduction
Deep reinforcement learning (RL) has contributed signiﬁcantly to the sensorimotor control of both simulated [Heess et al., 2017, Zhu et al., 2020] and real-world
[Levine et al., 2016, Mahmood et al., 2018] robots.
Monolithic learning is a popular paradigm for learning control policies.
In this paradigm, a policy inferring a joint action for all limb actuators based on a global sensory state is learned.
Although monolithic learning has made impressive progress [Chen et al., 2020, Kuznetsov et al., 2020], it has two major shortcomings.
First, the input and output space is large.
For robots with more joints, learning controlling policies puts a heavy burden on the representational capacity of neural networks.
Second, the input and output dimensions are ﬁxed, making it inﬂexible to transfer the learned control policies to robots with different morphologies.
Modular reinforcement learning provides an elegant solution to these problems.
In this learning paradigm [Wang et al., 2018], the control policy is decentralized [Peng et al., 2021], and each limb actuator is controlled by a local policy.
Recent research efforts show that the local policies can learn high-performance and transferable control strategies by sharing parameters [Huang et al., 2020], communicating to each other by message passing [Huang et al., 2020], and adaptively paying attention to other actuators via graph neural networks [Kurin et al., 2020].
By exploiting the ﬂexibility and generalizability provided by modularity, a modular policy can now control robots of up to thousands of morphologies [Gupta et al., 2021a].
Despite the signiﬁcant progress, modular reinforcement learning is still limited in terms of the complexity of morphological structures that can be controlled and struggles on robots with many joints like Humanoid [Kurin et al., 2020].
The large degree of control freedom presents a major challenge for learning control policies.
A question is why humans can control hundreds of muscles with dexterity while the most advanced RL policy can only control less than ﬁfteen actuators.
Studies on muscle synergies [d’Avella et al., 2003] may provide an answer.
A human central nervous system decreases the control complexity by producing a small number of electrical signals and activating muscles in groups [Ting and McKay, 2007].
Muscle synergy is the coordination of muscles that are activated in synchrony.
With muscle synergies, the human nervous system achieves low-rank control over its actuators.
In this paper, we aim to use the inspiration of muscle synergies to reduce the control complexity and improve the learning performance of modular RL.
The ﬁrst challenge of incorporating muscle synergies into modular RL is to discover a synergy structure that can promote policy learning.
Neuroscience researchers factorize electrical signals [Saito et al., 2018, Falaki et al., 2017, Kieliba et al., 2018] to analyze the synergy structure, but policy signals are sub-optimal or even absent during reinforcement learning.
We thus exploit the functional similarity and morphological context of actuators and use a clustering algorithm to identify actuators in the same synergy.
The intuition is that muscles in a synergy typically serve the same functional purpose and have similar morphological contexts.
We quantify the functional similarity by the inﬂuence of an actuator’s action on the global value function, and the morphological structure is encoded as a distance matrix.
To use the two types of information simultaneously, we adopt the afﬁnity propagation algorithm
[Frey and Dueck, 2007].
The synergy structure is updated periodically during learning to promptly reﬂect changes in value functions.
To exploit the discovered synergy structure, we design a synergy-aware architecture for policy learning.
The major novelty here is that the policy learns action selection for each synergy, and the synergy actions are transformed linearly to get actuator actions.
Since the number of synergies is typically much smaller than actuators, we actually learn a low-rank control policy where the physical actions are a linear mapping from a low-dimensional action space.
Moreover, for better processing state information, the synergy-aware policy adopts a two-level transformer structure, which ﬁrst aggregates information within each synergy and then processes information across synergies.
We evaluate our Synergy-Oriented LeARning (S OLAR ) framework on two MuJoCo
[Todorov et al., 2012] locomotion benchmarks [Huang et al., 2020, Gupta et al., 2021b] and in multi-task to zeroshot learning, single-task settings.
S OLAR signiﬁcantly outperforms previous state-of-the-art algorithms in terms of both sample efﬁciency and ﬁnal performance on all tested settings, especially on robots with a large DoF like Humanoid ++ [Huang et al., 2020] and UNIMALs [Gupta et al., 2021b].
Performance comparison and the visualization of learned synergy structures strongly support the effectiveness of our synergy discovery method and synergy-aware transformer-based policy learning approach.
Our experimental results reveal the low-rank nature of multi-joint robot control signals.
Background
Modular RL .
Modular Reinforcement Learning decentralizes the control of multi-joint robots by learning policies for each actuator.
Each joint has its controlling policy and they coordinate with each other via various message passing schemes.
Modular RL usually needs to deal with agents with different morphologies.
To do so, Wang et al.
[2018] and Pathak et al.
[2019] represent the robot’s morphology as a graph and use GNNs as policy and message passing networks.
Huang et al.
[2020] uses both bottom-up and top-down message passing scheme through the links between joints for coordinating.
All of these GNN-like works show the beneﬁts of modular policies over a monolithic policy in tasks tackling different morphologies.
However, recently, Kurin et al.
[2020] validated a hypothesis that any beneﬁt GNNs can extract from morphological structures is outweighed by the difﬁculty of message passing across multiple hops.
They further propose a transformer-based method,
A MORPHEUS , that utilizes self-attention mechanisms as a message passing approach.
A MORPHEUS outperforms prior works and our work is based on A MORPHEUS .
Previous works mainly focused on effective message passing schemes, while our work aims at reducing learning complexities when the DoF of the robot is large
Muscle Synergy .
How the human central nervous system (CNS) coordinates the activation of a large number of muscles during movement is still an open question.
According to numerous studies, the CNS activates muscles in groups to decrease the complexity required to control each individual muscle [d’Avella et al., 2003, Ting and McKay, 2007].
According to muscle synergy theory, the
CNS produces a small number of signals.
The combinations of these signals are distributed to the muscles [Wojtara et al., 2014].
Muscle synergy is the term for the coordination of muscles that activate at the same time [Ferrante et al., 2016].
A synergy can include multiple muscles, and a muscle can belong to multiple synergies.
Synergies produce complicated activation patterns for a set of muscles during the performance of a task, which is commonly measured using electromyography (EMG)
[Tresch et al., 2002, Singh et al., 2018].
EMG signals are typically recorded as a matrix with a column for activation signals for a moment and a row for activation of a muscle [Rabbi et al., 2020].
Factorisation methods on the matrix are used to extract muscle synergies from muscle activation patterns.
Four most commonly used factorization methods are non-negative matrix factorisation
[Steele et al., 2015, Schwartz et al., 2016, Lee and Seung, 1999, Rozumalski et al., 2017, Shuman et al., 2016, Saito et al., 2018] , principal component analysis [Ting and Macpherson, 2005, Ting et al., 2015, Danion and Latash, 2010, Falaki et al., 2017], independent component analysis [Hyvärinen and Oja, 2000, Hart and Giszter, 2013], and factor analysis [Kieliba et al., 2018,
Saito et al., 2015].
In the ﬁeld of robot control, only a few works [Palli et al., 2014, Wimböck et al., 2011, Ficuciello et al., 2016] have exploited the idea of muscle synergy for dimensionality reduction to simplify the control.
However, these works usually ﬁrst use motion dataset from humans to obtain the synergy space and then learn to control in this synergy space.
In contrast, our work learns the synergy space simultaneously with the control policy in the synergy space.
Afﬁnity propagation [Frey and Dueck, 2007] is a clustering algorithm based on multi-round message passing between input data points.
It does not need to pre-deﬁne the number of clusters and proceeds by ﬁnding each instance an exemplar.
Data points that choose the same exemplar belongs to the same cluster.
Suppose { x i } ni =1 is a set of data points.
Deﬁne S ∈ R n × n as a similarity matrix.
When i ̸
= j , the element s i,j at i th row and j th column is the similarity between x i and x j , which can be measured as, for example, the negative squared distance of two data points.
When i =
j , the element s i,j represents how likely the corresponding instance is to become an exemplar.
The vector of diagonal elements, ( s 11 , s 22 , . . .
, s nn ) , is called preference .
Non-diagonal elements in S constitute the afﬁnity matrix.
The algorithm takes S as input and proceeds by updating two matrices: the responsibility matrix R whose values r i,j represent whether x j is well-suited to be the exemplar for x i ; the availability matrix A whose values a i,j quantify the appropriateness for x i picking x j as its exemplar [Frey and Dueck, 2007].
These two matrices are initialized to be zeroes and can be regarded as log-probability tables.
The algorithm then alternatives between two message-passing steps.
First, the responsibility matrix is updated:
[Equation]
Then, the availability matrix is updated:
[Equation]
Messages are passed until the clusters stabilize or the pre-determined number of iterations is reached.
Then the exemplar of i is arg max j r i,j + a i,j .
Method
In this section, we present our Synergy-Oriented LeARning (S OLAR ) scheme that incorporates the muscle synergy mechanism into modular reinforcement learning to reduce its learning complexity.
Our method has two major components.
The ﬁrst one is an unsupervised learning module that utilizes the morphological structure and value information to discover the synergy hierarchy.
The second is a novel attention-based policy architecture that supports synergy-aware learning.
Both of the components are specially designed to enable the control of robots with different morphologies.
We ﬁrst introduce the problem settings and then describe the details of the two components.
Problem settings.
We consider N robots, each with a unique morphology.
Agent n contains K n limb actuators that are connected together to constitute its overall morphological structure.
Exam- ples of such robots that are studied in this paper include Humanoid ++ and UNIMALs.
At each discrete timestep t , actuator k ∈ { 1 , 2 , . . .
, K n } of a robot n ∈ { 1 , 2 , . .
, N } receives a local sensory state s n,kt as input and outputs individual torque values a n,kt for the corresponding actuator.
Then the robot n executes the joint action a nt = { a n,kt } K n k =1 at time t , after which the environment returns the next state s nt +
= { s n,kt +1 } K n k
=1 corresponding to all limbs of the agent n and a collective reward for the whole morphology r nt ( s nt , a nt ) .
We learn a policy π θ to generate actions based on states.
The learning objective of the policy is to maximize the expected return on all the tasks:
[Equation]
where γ is a discount factor.
We adopt an actor-critic framework for policy learning.
The critic is shared among all tasks and estimates the Q-function for each robot n :
[Equation]
k l
I n t r aS y n e r g y S
e l f - A tt e n t i o n k l i
I n t e rS y n e r g y S
e l f - A tt e n t i
o
n i
A c t i o n
T r a n s f o r m a t
i o n k l
S y ne r g y
S t r u c t u r e
P e r a c t ua t
o r a c t i on
Preference: Δ𝑄# !,#
Affinity: exp (−𝑫 #,$ ) S
[Table]
[Figure]
i t
[Figure]
3.1 Discovering synergy structure
In Neuroscience, the muscle synergies are usually discovered by a factorization of electrical muscle signals during performing tasks [Todorov and Ghahramani, 2004, Rabbi et al., 2020].
Factorization is a method in hindsight – it statistically analyzes the optimal control policies of animals embodied in the electrical signals.
By contrast, in reinforcement learning, we do not have the optimal control policies in advance.
The synergy hierarchy learned from non-optimal policies is more likely incorrect, which will hamper policy learning.
Therefore, we propose learning the synergy hierarchy by an unsupervised learning method that incorporates morphological information besides learning information.
In this section, we describe our synergy hierarchy discovery methods.
Intuitively, actuators in the same synergy are activated simultaneously and they together ﬁnish a motion of an end effector.
This gives us a hint that actuators with similar functions are supposed to be in a synergy.
Formally, the function of an actuator (the k th actuator in robot n ) can be modelled by its inﬂuence to the value function:
[Equation]
where [ · , · ] combines two terms, a n,k is the actual action of actuator k , b n,k is a default action of actuator k ( b n,k = 0 in MuJoCo environments) and a n, - k is the actions of actuators on robot n except for the k th one.
In practice, we use a SoftMax function to regularize ∆ Q n,k : ∆ ˜ Q n,k = exp(∆ Q n,k ) /
∑ j exp(∆ Q n,j ) .
Ideally, ∆ ˜ Q n,k contains actuator clustering information, but the Q-values may be inaccurate during learning.
We propose to treat (∆ ˜ Q n, 1 , ∆ ˜ Q n, 2 , · · · , ∆ ˜ Q n,K n ) as the preference vector of the afﬁnity propagation clustering algorithm
(Sec. 2), i.e, the diagonal elements of similarity matrix, and in the next paragraph we further encode stable morphological information in the afﬁnity matrix of afﬁnity propagation, i.e, the non-diagonal elements of similarity matrix.
The advantage of this design is that (1) we can simultaneously consider functional similarity and morphological context when using clustering to discover synergies and (2) static morphological information may help stabilize the clustering results.
We now discuss how to encode morphological information in the afﬁnity matrix.
A robot morphology is usually organized in a tree structure with its torso as the root node [Huang et al., 2020].
We deﬁne the adjacency matrix A d ∈ { 0 , 1 } K n × K n to represent robot’s morphology, where A i,j =if actuators i, j are directly linked.
To measure the connectiveness of every actuator pair, we use
Floyd-Warshall algorithm
[Cormen et al., 2022] to compute the shortest distances between actuators based on A d , resulting in a distance matrix D ∈ N K n × K n .
The matrix exp( − D ) will serve as the afﬁnity matrix A of afﬁnity propagation.
After deﬁning the preference and afﬁnity for afﬁnity propagation, the clustering algorithms run n max rounds of message passing to determine actuator clusters.
We treat each cluster as a synergy.
In practice, we update the synergy structure periodically during the learning process.
While D remains unchanged, ∆ ˜ Q n,k changes as the Q-function is updated.
Each time we run the afﬁnity propagation with the new preference vector and then freeze the synergy structure for a while.
3.2 Learning synergy-aware low-rank policies
We now describe how we design a synergy-aware actor-critic learning framework that exploits the synergy hierarchy discovered in Sec. 3.1.
The core idea is that the policy generates control signals (actions) for synergies instead of actuators.
As the number of synergies is much fewer than actuators, the learning complexity is reduced.
The synergy actions will be combined linearly to generate physical actions for actuators.
In this way, we learn a low-rank control policy – the physical actions are actually a linear combination of actions in a low-dimensional space.
Suppose that we ﬁnd L n synergies { S n,i } L n i =1 for the n th robot.
We use a mask M n ∈ { 0 , 1 } K n × K n to represent the synergy structure.
The element at the i th row, j th column of M n is 1 if actuator i and j are in the same synergy.
The policy π θ consists of three components: an intra-synergy attention module, an inter-synergy attention module, and an action transformation matrix as in Figure 1.
The ﬁrst module aggregates information of actuators in a synergy.
Input to this module is the joint state s n , and this module outputs a hidden representation h n,i ∈
R d for each synergy S n,i .
In practice, we use a transformer with a two-head self-attention layer where the attentions between actuators belonging to different synergies are masked out.
The output of the self-attention layer is of the size K n × d .
We then use mean pooling to aggregate information of actuators in the same synergy.
In this way, we get a hidden state h n ∈ R L n × d , with each row being the aggregated information of a synergy.
The second module aggregates information from different synergies and outputs the synergy actions.
It has two multi-head self-attention layers.
We feed h n into this these layers and get vector-valued synergy actions a sn ∈ R S × L n × 1 .
The synergy actions are then transformed by learnable matrices T ∈ R S × K n × L n , and we obtain actions for actuators by a n
= mean ( T a sn ) dim =0 .
One question about the transformation matrix T is that L n may change during the learning process, requiring the dimension of T to be dynamic.
In Appendix A, we discuss how to deal with this problem.
The critic uses the same intra- and inter-synergy attention structure as the policy for processing state information.
The outputs of the inter-synergy attention module are fed into a fully-connected network to obtain the Q-value estimation.
In practice, we use TD3 [Fujimoto et al., 2018] for the training of the policy and the critic.
The intra- and inter-synergy transformer and the transformation matrix are updated in an end-to-end manner.
We provide more details of our learning framework in
Appendix A.
Experiments
In this section, we benchmark our method S OLAR on various MuJoCo
[Todorov et al., 2012] locomotion tasks.
we evaluate the effectiveness of S OLAR by asking the following questions: (1) Can our S OLAR outperforms other modular RL approaches when simultaneously trained on a large number of diverse agents with different morphologies?
(Sec. 4.2) (2) Can the learned policy in multi-task environments generalize to new tasks with unseen morphologies?
(Sec. 4.3) (3) How does S OLAR learn synergy clusters and how do synergy clusters facilitate learning?
(Sec. 4.4) (4) Can S OLAR scale to single-tasks with numerous joints?
(Sec. 4.5).
For qualitative results, please refer to the videos on our project website * .
And our code is available at GitHub * .
4.1 Experiment setup
We run experiments on two benchmarks, and report several results in this section.
For additional results, please refer to Appendix B. We test all methods with 4 random seeds and show the mean performance as well as 95% conﬁdence intervals.
Environments .
For multi-task and zero-shot evaluation, we adopt the widely-used modular MTRL benchmarks [Huang et al., 2020, Kurin et al., 2020, Hong et al., 2021], which are created based on Gym MuJoCo locomotion tasks by Huang et al.
In this section, we report results onin-domain settings: (1) 6 variants of walker
[ Walker ++] (2) 8 variants of humanoid [ Humanoid ++] (3) 3 variants of hopper [ Hopper ++]; and 1 cross-domain settings: all 6 variants of walker, allvariants of humanoids, and all 3 variants of hopper [ Walker - Humanoid - Hopper ++].
For single-task evaluation, we sample 3 morphologies from UNIMALS [Gupta et al., 2021b,a], which are obtained from evolution and have numerous joints, making it suitable to evaluate the performance of modular
RL policies in single-tasks.
For a detailed description of the environments please refer to Gupta et al.
[2021b] and Appendix A.
Baselines .
In this section, we compare our method S OLAR against state-of-the-art modular RL methods SMP
[Huang et al., 2020] and A MORPHEUS
[Kurin et al., 2020].
SMP passes messages along the limbs using a bottom-up and top-down scheme.
S OLAR and A MORPHEUS use transformerbased actors and critics, whose message passing schemes are realized by self-attention.
We also compare S OLAR with standard TD3-based non-modular RL: Monolithic.
Please refer to Appendix A for more details about baselines.
Ablations .
There are two contributions that characterize our method.
(1) An unsupervised learning module that utilizes the value information and the morphological contextual structure to discover the synergy hierarchy.
(2) A novel attention-based synergy-aware policy architecture that supports synergy-aware learning.
Our novelties are mainly about the discovery and utilization of synergies, and these two contributions closely rely on each other.
If we totally ablate synergies, we will get
A MORPHEUS .
Therefore, the effectiveness of synergy-based learning can be demonstrated by comparing S OLAR against A MORPHEUS .
We further design the following ablations: (1) A MORPHEUS w/ synergy mask .
Incorporate synergy-aware inter-cluster masks into A MORPHEUS to test the contribution of synergy hierarchy.
(2) S OLAR w/o preference .
Remove value information (the preference vector) from afﬁnity propagation of S OLAR .
In this case, afﬁnity propagation is based solely on morphological information, and we can test the effects of value information.
Implementations .
We use TD3 [Fujimoto et al., 2018] as the underlying reinforcement learning algorithm for training the policy over all baselines, ablations and our method for fairness.
We implement S OLAR in the A MORPHEUS codebase.
And like A MORPHEUS , there is no weight sharing between actor and critic.
S OLAR uses a traversal-based embedding for action transformation, which may incorporate structural information.
For a fair comparison, we concatenate these embeddings into the original observation vectors of each environment which is applied to all tested algorithms.
[Figure] [Figure]
We summarize the multi-task results in Figure 2.
S OLAR outperforms the previous state-of-theart algorithms
A MORPHEUS and SMP in all settings.
Furthermore, the performance gap between
S OLAR and A MORPHEUS is notably larger in Humanoid ++ which has the maximum number of joints among the tested settings.
We also plot the learning curves of six training variants (Figure 3) and two testing variants (Figure 4) of Humanoid ++.
These results consistently demonstrate the effectiveness of S OLAR in training robots with a large degree of freedom (DoF).
We speculate that the reason of the effectiveness is that S OLAR controls robots in a lower-rank action space than the original space and the action spaces of robots with a large DoF are more likely to be reduced via the synergy mechanism.
To further validate this speculation, we conduct extra evaluations on robots with much more joints in Sec. 4.5.
As for ablations, A MORPHEUS w/ synergy mask outperforms A MORPHEUS in most tasks in Figureand Figure 3, which indicates the effectiveness of the unsupervised learning method that discovers synergy clusters and synergy-aware masks in the policy.
However, S OLAR w/o preference performs comparable to or even worse than A MORPHEUS , which reﬂects the prominent usefulness of value information.
4.3 Zero-shot generalization
In this section, we benchmark S OLAR in a zero-shot learning setting where the policy is trained in multiple tasks and then generalized to unseen tasks.
We compare S OLAR with A MORPHEUS , SMP,
A MORPHEUS w/ synergy mask and S OLAR
w/o preference in Humanoid ++, which has the largest number of joints among the tested environments.
Humanoid ++ has eight variants of humanoids, where six of them are used as training tasks (see Figure 3) and the other two are used as testing tasks (Figure 4).
To obtain the results of testing tasks at each time step, we periodically load policy network parameters when training and evaluate the policy in these tasks.
The trajectories on testing tasks will be discarded and will not be used for learning.
[Table]
[Figure]
As shown in Figure 4, S OLAR outperforms all the baselines by a large margin in these unseen tasks.
S OLAR learns faster than other baselines and shows higher sample efﬁciency in Figure 3, showing that the knowledge obtained from training tasks is effectively used in unseen tasks.
4.4 Analysis of synergies
[Figure]
To investigate why S OLAR performs better than baselines, in this section, we visualize the synergy clusters learned by S OLAR and the evolutionary process of synergy clusters in Humanoid ++.
Afﬁnity propagation which we used as the method to discover synergy clusters can naturally outputs clustering results and the cluster centers in each cluster.
We ﬁrst visualize the synergy clusters of
S OLAR in Humanoid ++ at the end of training in Figure 5.
Here we use different colors and two shapes to mark each joint in robots.
Different colors represent different synergy clusters, and joints
[Table]
[Figure]
Mean Episode Reward of Amorpheus
To further investigate how these clusters evolve, we visualize the learning process of Left Knee
Humanoid in Figure 6.
According to the change of synergy clusters, the learning process can be divided into three phases.
In Phase 1, the episode reward is very low and ∆ ˜ Q may only provide noisy information, which results in all joints being clustered in a same cluster.
In Phase 2, S OLAR ﬁnds that the right shin is important and divides it into a separate cluster.
Then in Phase 3, S OLAR ﬁnally obtains a suitable clustering solution and the episode reward increases correspondingly.
The visualization results reﬂect two facts: (1) close joints are more likely to be in the same synergy cluster, and (2) joints near the torso may be more inﬂuential than those who are far from the torso, and are thus selected as the cluster centers.
These clusters can reduce learning and controlling complexities but are hard to discover through a standard learning mechanism.
Instead, S OLAR makes use of early, less accurate learning information ∆ ˜ Q and structural information via unsupervised learning to discover synergy clusters.
The synergy clusters then further facilitate learning.
4.5 Single-task with numerous joints
To consolidate the speculation in Sec.
4.2 that S OLAR is suitable for tasks with numerous joints, we test S OLAR in three single-robot tasks sampled from UNIMALS [Gupta et al., 2021b].
The average number of joints is about twice as many as that in Humanoid ++.
For more details about these tasks, please refer to Appendix A. We showcase the results in Figure 7.
S OLAR learns faster than state-ofthe-art A MORPHEUS , showing much higher sample efﬁciency, and again outperforms it in average episode rewards.
[Figure]
Conclusions
In this paper, we use the inspiration of muscle synergies from neuroscience to reduce the control complexity of modular reinforcement learning in tasks with a large number of degree of freedom.
We present S OLAR that leverages both structural information and policy learning information to learn synergy clusters.
Moreover, a stacked-transformer architecture is proposed for learning synergy actions which are combined linearly to produce low-rank actuator actions.
The evaluations of
S OLAR on various tasks and settings showcase its effectiveness.
The authors do not see obvious negative societal impacts of the proposed method.
Acknowledgments
This work is supported in part by Science and Technology Innovation 2030 New Generation Artiﬁcial Intelligence Major Project (No. 2018AAA0100904) and National Natural Science Foundation of China (62176135).
A Reproducibility A.1 Environment description
We run experiments on two benchmarks as described in Table 1.
All these benchmarks are implemented in MuJoCo
[Todorov et al., 2012].
Huang et al.
[2020] created Walker ++, Humanoid ++,
Hopper ++ and Walker - Humanoid - Hopper ++, by removing some limbs and joints from the original intact morphology of Gym MuJoCo robots [Todorov et al., 2012, Brockman et al., 2016].
In this benchmark, every robot has the ability to hop, walk, or run, i .
e ., only the robots that can move forward are left.
Following Huang et al.
[2020], Kurin et al.
[2020], Hong et al.
[2021], we use this benchmark for the evaluation of multi-task learning (Sec. 4.2) and zero-shot generalization ability (Sec. 4.3).
As for the single-task learning benchmark, we sampled some morphologies from UNIMALS [Gupta et al., 2021b,a].
Gupta et al.
[2021b] created UNIMALS by evolution on robots’ morphologies, kinematics, and dynamics.
At the end of the evolution, they had 100 task optimized robots.
We choose a subset of these robots that are optimized for locomotion in ﬂat terrain environment for single-task evaluation in Sec. 4.5.
We ensure that the selected robots have different morphologies from each other.
For multi-task evaluation, we use the whole 100 robots for training in Appendix B.1.
And for zero-shot evaluation in robots’ dynamic and kinematic properties in
Appendix B.1, we use the test set created by Gupta et al.
[2021a], which contains 2400 robots.
The reward of these tasks is given by the speed of the robot moving forward, and is penalized by the action norm.
An episode terminates when the robot’s height is low or the episode length exceeds 1000 time steps.
For all the environments, we can extract the morphology of the robot as a graph from the corresponding MuJoCo XML ﬁle.
Then the morphological information is concatenated in observations in baselines or is used to generate embeddings for action transformation in S OLAR .
The visualization of these robots can be found in synergy clustering results: Walker ++ as in Figure 11,
Humanoid ++ as in Figure 5, Hopper ++ as in Figure 10 and UNIMALS as in Figure 12.
The number of actuators we evaluated in this work varies from three to nine.
[Table]
A.2 Implementation details
We implement S OLAR based on A MORPHEUS [Kurin et al., 2020], which is built on Ofﬁcial PyTorch
Tutorial [Seq, 2020].
A MORPHEUS also shares the codebase with SMP [Huang et al., 2020].
Tableprovides the hyperparameters needed to replicate our experiments
[Table]
As in Figure
[Table]
there are three main components in our implementation, i
[Table]
e ., Intra-Synergy
Self-Attention , Inter-Synergy Self-Attention and Action Transformation .
To achieve Intra-Synergy
Self-Attention efﬁciently, we pass a synergy-based mask into PyTorch’s TranformerEncoderLayer
[Table]
through src_mask function argument.
The synergy-based mask is discussed in detail in Sec. 3.2.
Inter-Synergy Self-Attention is implemented by the normal TranformerEncoderLayer without masks.
As for Action Transformation , the main challenge is how to obtain a transformation matrix T ∈ R K n × L n , where K n and L n are the number of actuators and the number of synergies of robot n ∈ { 1 , 2 , · · · , N } .
L n may change during the learning process.
Intuitively, the matrix T is dependent on the robot’s morphology.
Here we use a traversal-based positional embedding [Hong et al., 2021] technique, and these embeddings together represent the robot’s morphological information.
To obtain the traversal-based positional embedding, we ﬁrst apply left-child-right-sibling representation to represent a general tree as a binary tree.
Then, we traverse the binary tree in pre-order, in-order and post-order, which forms a triple for each actuator consisting of its orders.
And these triples together are sufﬁcient to reconstruct the original tree, which indicates that each triple contains the structural information of the corresponding actuator.
The triple of each joint serves as indices into a embedding pool { r 1 , r 2 , . .
, r p } where p = max n K n is the maximum number of actuators in a robot, and each r i ∈
R s is a learnable vector.
The selected embeddings are processed by a network, and we concatenate the outputs to obtain a representation of the corresponding actuator.
Using dot product between representations of two actuators, we get a matrix H ∈ R K n × K n where H i,j indicates the relation between actuator i and actuator j .
Finally, we average the columns corresponding to actuators in the same synergy cluster and obtain the transformation matrix T ∈ R K n × L n .
The embedding pool and network are updated by backpropagating the RL loss.
Experiments are carried out on NVIDIA GTX 2080 Ti GPUs.
For Humanoid ++, our method requires approximately 10G of RAM and 5G of video memory, and takes about 23 hours to ﬁnish 2M timesteps of training.
The code for our method is included in the supplementary materials with sufﬁcient setup and running instructions.
The code follows the MIT license.
A.3 Baselines and ablations
We compare our method against various baselines and ablations.
Here we explain the implementations of these baselines and ablations.
A MORPHEUS .
We use the original implementation of A MORPHEUS released by Kurin et al.
For fair comparison, S OLAR uses the same value for those hyperparameters that are shared with
A MORPHEUS (Table 2).
SMP.
We use the implementation of SMP in the A MORPHEUS codebase, which is the same as the original implementation of SMP provided by Huang et al.
Monolithic.
Following the setup by Huang et al.
[2020], we choose TD3 as the standard monolithic
RL baseline.
The actor and critics of TD3 are implemented by fully-connected neural networks, which take the concatenation of observations of all actuators as input.
Since number of actuators varies in different robots, the dimension of observations is incompatible.
To overcome this issue, we zero-pad the observations and actions to the maximum dimension across all robots.
A MORPHEUS w/ synergy mask .
We passed a synergy-based mask to the ﬁrst layer of self-attention in A MORPHEUS , and the mask is learned in the same way as in our method.
The other two layers of
A MORPHEUS are not modiﬁed.
S OLAR w/o preference .
We simply set the preference vector of the afﬁnity propagation clustering algorithm to None, and the algorithm will use the median of the input afﬁnity matrix as a default preference vector.
B Additional experiments and discussions B.1 UNIMALS Locomotion task and Manipulation task
[Table]
[Figure]
To further analyze the scalability of our S OLAR when the number of robots and the number of actuators are large, we use the original UNIMALS [Gupta et al., 2021b]
Locomotion tasks in ﬂat terrain and Manipulation tasks in variable terrain.
In Locomotion tasks, the robot needs to go forward as fast as possible.
In Manipulation tasks, the robot needs to ﬁrst reach a box, and then push this box to a randomly generated goal.
The UNIMALS has 100 robots with different morphologies, kinematics, and dynamics for multi-task training.
We compared our method S OLAR against MetaMorph [Gupta et al., 2021a].
In this subsection, S OLAR is built upon MetaMorph.
Like MetaMorph, S OLAR incorporates structural information into A MORPHEUS and uses a dynamic replay buffer balancing technique to deal with the large number of robots.
S OLAR can be simply combined with MetaMorph by modifying its self-attention structure.
The training results are shown in Figure 8.
S OLAR outperforms MetaMorph by a large margin, which suggests that S OLAR is also effective with numerous different robots.
[Table]
In Sec. 4.3, we test the zero-shot performance of S OLAR on robots with different morphologies.
However, in reality, there are cases where the morphologies are not changed, but the dynamics (armature, density, damping, and motor gear) and kinematics (module shape parameters and joint angles) of robots are new in unseen tasks.
Thus we also benchmark the zero-shot performance of
S OLAR in these cases.
Gupta et al.
[2021a] created a test set to test zero-shot performance based on the 100 training robots of UNIMALS.
For each robot in the training set, they create 4 different variants for each property (regarding the dynamics and kinematics), leading to a total number of
[Table] [Table]
100 × 6 × 4 = 2400 test variants.
Here we reproduce from their paper the sampling ranges of test variants in Table 5 .
Please refer to Gupta et al.
[2021a] for more details.
Due to the limitation of computing resources, we reduce the batch size of S OLAR and MetaMorph from 5120 to
[Table]
and other hyperparameters follow Gupta et al.
[2021a]
[Table]
To evaluate the zero-shot generalization performance
[Table]
we load our models trained on multi-task UNIMALS and show the averaged performance (and variance) over 4 random seeds in Table 3 and Table 4 for Locomotion task and Manipulation task, respectively.
We ﬁnd that S OLAR exhibits strong generalization performance compared to the baseline, which indicates that the synergy clusters and action transformations learned on the training set are robust to dynamic and kinematic variations.
B.2 Compared with other baselines in single-task
[Figure]
In Sec. 4.5, we compare S OLAR with transformer-based baseline A MORPHEUS , here we also compare our S OLAR with the GNN-style baseline SMP in single-task evaluations in Figure 9 as additional results.
B.3 Synergy clustering results of more morphologies
In Figure 5, we visualize the synergy clustering results of Humanoid ++ to analyze the effect of synergy.
In this section, we provide visualizations of synergy clustering results for Hopper ++ (Figure 10), Walker ++ (Figure 11), and UNIMALS (Figure 12) as additional results.
[Figure] [Figure]
[Figure]
These additional results further consolidate our analysis in Sec. 4.4: (1) close joints are more likely to be in the same synergy cluster, and (2) joints near the torso may be more inﬂuential than those who are far from the torso, and are thus selected as the cluster centers.
B.4 Connection to conventional notion of muscle synergy
Mathematically, conventional muscle synergy studies typically use non-negative matrix factorization (Rabbi et al, 2020) and aim to solve the following optimization problem:
[Equation]
where U ∈ R N × T is a given matrix of observed electrical control signals.
The element at i th row and t th column of U is the control signal to muscle i at timestep t .
In this optimization problem, the num- ber of synergies, M (where M < N ), is typically pre-deﬁned or chosen according to a pre-deﬁned reconstruction error threshold.
By solving this problem, one can discover the synergy structure by observing matrix H .
Conventional muscle synergy studies that use other factorization methods (PCA, ICA, and FA) share the similar optimization problem with different matrix constraints.
We study a similar optimization problem but with additional constraints:
[Equation]
Here s t is the environment state at timestep t , and U t is the column t of matrix U , i.e., actions at timestep t .
And R ( s t , U t ) is the reward of choosing action U t at state s t .
The differences are:
We additionally maximize the expected return of muscle actions.
In our formulation, the number of synergies is not pre-deﬁned but is learned in an unsupervised manner.
The matrix U is also not given, but is generate by an attention-based policy.
This policy are optimized to maximize return as well as to minimize the decomposition loss (term 2).
In summary, we share a common optimization object with conventional muscle synergy studies, which is a decomposition loss.
However, we need to additionally learn the number of synergies and control signals which are inputs in the conventional studies.
Structurally, our framework gives an attention function class that covers a synergy decomposition solution which can minimize the decomposition loss while enable an efﬁcient control policy.
B.5 Limitations and future works
When generalizing to unseen robots with larger numbers of actuators than training robots, the embeddings of testing robots are not learned, which will hamper the zero-shot performance.
One possible future direction will involve designing a more scalable actuator embedding method.
Moreover, SOLAR is more suitable for robots with large number of actuators.
Our approach is able to reduce a great degree of control complexity for these robots.
But for robots with few actuators, SOLAR may only have a little positive impact and the learning of synergy-aware policy may even damage the performance.