DMAP: a Distributed Morphological Attention Policy for learning to locomote with a changing body s
AffiliationAddress
Abstract
Reinforcement learning typically seeks to learn control policies in stable environments.
Yet, real world scenarios require continuous adaptation.
In particular,learning to locomote when the length and the thickness of different body parts varyis challenging, as the policy is required to adapt to the current configuration tosuccessfully balance and advance the agent.
We study this problem in four classicalcontinuous control environments, augmented with morphological perturbations.
We show that a control policy based on the proprioceptive state performs poorlywith highly variable body configurations, while an (oracle) agent with access to alearned encoding of the perturbation performs significantly better.
We introduce
DMAP, a biologically-inspired, attention-based policy network architecture.
Itcombines a distributed policy, with individual controllers for each joint, and anattention mechanism, to dynamically gate sensory information from different bodyparts.
DMAP can be trained end-to-end in all the considered environments andperturbation intensities, overall matching or surpassing the performance of an oracle agent with access to the morphology information.
Thus DMAP, implementingprinciples of control drawn from the biological world, provides a strong inductivebias for learning challenging sensorimotor tasks.
Overall, our work corroboratesthe power of these principles in challenging locomotion tasks.
Introduction
Animals and humans are highly adaptive to changes in the external environment and in their ownbody.
For instance, animals can, after an early onset, locomote throughout development, when theirbody size and weight change dramatically.
They can also robustly deal with irregular surfaces andload changes [1, 2, 3, 4, 5].
Here, we focus on the problem of learning to walk when the bodyis subject to morphological perturbations, consisting in the change of length and thickness of itsparts.
Remarkably, animals can rapidly adapt to these kind of perturbations.
For example, desertants with elongated or shortened legs can immediately locomote, although misjudging the traveleddistance [6].
To challenge artificial agents on such skills, we adapted four different locomotionenvironments from OpenAI Gym [7] (Ant, Half Cheetah, Walker, Hopper), in which robots haveto learn how to walk as fast as possible.
In our adaptive setting, the agent’s body parameters aresampled from a morphological perturbation space at the beginning of each episode and we sought tofind reinforcement learning policies that could deal with these varying bodies.
We hypothesize that principles of the sensorimotor system provide a strong inductive bias to robustlylearn such a policy.
Specifically, we built on three well-known features: 1) Independent low-level
DMAP - Distributed Morphological Attention Policy
E p
i s o d e s
T i m e
Time
TCN
TCN
TCN Ω Ω Ω
Channel features
Distributed joint controllers
Action
Joint-channel attention
Proprioceptive state history
H
Independent representations
DMAP
IID Test
OOD Test
A
B
K
V
N A processing: Somatosensory inputs are first processed locally (e.g., by temporal filtering) beforethey are integrated across more distal body parts and represented in an orderly way [8, 9, 10]. 2)
Distributed control: Different body parts are controlled in a distributed fashion [9, 10, 5].
3) Gatedconnectivity:
Sensory information is dynamically gated in a task dependent manner [11], also beyondsomatosensation [12], and integrated with action and value signals in the brain [13, 10, 14].
Hence, we propose the Distributed Morphological Attention Policy (DMAP) architecture (Figure 1).
We show that it can robustly learn to locomote with changing bodies without access to their morphological parameters.
To assess the performance of this architecture, we compare it to several otheragents.
Experiments reveal that an agent with access only to the proprioceptive state, consisting of itsjoint angles and velocities ( Simple ), fails to learn an effective locomotion strategy for a large varietyof body configurations.
In contrast, augmenting the observation with the current body shape leads toa strong ceiling performance ( Oracle ).
Importantly, for all different environments, DMAP is as goodas Oracle.
We also compare DMAP to a recently proposed two-step training technique called rapid motoradaptation ( RMA )
This algorithm trains a temporal convolutional network to infer a contextencoding from a short history of proprioceptive state, imitating the Oracle’s morphology encoder.
While RMA can also reach Oracle performance, we show that end-to-end training of such networkdoes not succeed for more complicated environments (Ant and Walker).
In contrast, DMAP’s policy can be trained end-to-end without access to the morphological parameters and reaches or surpasses
Oracle performance for all four environments (Ant, Half Cheetah, Walker, Hopper).
This suggeststhat independent low-level processing, distributed control and dynamic gating mechanisms provide astrong inductive bias for sensorimotor control.
Finally, we analyze the information gating mechanismof DMAP, which establishes pose-dependent sensorimotor pathways.
The low dimensional embeddingof the gating mechanism exhibits rotational dynamics, robustly across different morphologies.
Related work
Benchmarks for zero-shot adaptation.
The setup of the experiments described in this paper can beframed as a zero-shot adaptation problem, as the agents are evaluated starting from the first momentin which they interact with an unseen context [16].
While classic RL benchmarks, such as Arcade[17] and rllab [18], assess the ability of an agent to maximize the cumulative reward in the trainingenvironment, more recent ones introduce perturbations in the dynamics [19, 20], in the observationfunction [21, 22] or in the reward function [23] at test time.
For a more complete list, we refer to[16].
Mann et al.
showed that the performance of RL agents under morphological distributional shiftscorrelates with the ability of internal models to generalize [24].
We took their design of environmentsas an inspiration for the design of morphological changes in our work.
Context encoding and adaptation.
Algorithms which enable learning in changing environments areimportant for the sim-to-real transfer problem, where domain randomization represents a successfulstrategy to bridge the reality gap [25, 26, 27].
A notable approach to solving the zero-shot transferproblem consists in encoding the context into a hidden state, e.g., by online system identification [28],or with a recurrent neural network (RNN) [18, 29].
However, while off-policy continuous controlalgorithms such as TD3 [30] and SAC [31] are the current state of the art among model-free RLalgorithms, both in sample efficiency and final performance [32], training a recurrent neural networkoff-policy presents relevant technical challenges [33].
The problem of adaptation to an unseen environmental conditions is central in the Meta-RL literature [34].
Both model-based (ReBAL, GrBal [35], MOLe [36]), and model-free (PEARL [37])algorithms prove capable to accelerate adaptation to unseen contexts.
However, they require anadaptation phase in the test environment, which can last a few episodes [37] or some hours [35].
Asthe focus of our research is on the development of a network architecture to enable robust locomotionin a zero-shot manner, we chose to compare it to RMA, a powerful system identification approach; Kumar et al. showed that a context encoding can be successfully inferred from a short sequence ofpast states and actions, and this context encoding can guide a robust policy [15].
Distributed control.
An agent can learn to generalize to multiple body shapes by using modularpolicies to control each joint independently, e.g., treating each joint [38] or groups of joints [39] asa node of the graph.
For example, Nervenet [40] achieves state-of-the-art performance on standard
RL benchmarks by leveraging different policies for each joint type.
However, it cannot generalizeto different designs in a zero-shot manner without fine-tuning.
On the other hand, Shared Modular
Policies (SMP) [41] employs the same policy for each joint, enabling an agent to deal with variableobservation and action space sizes.
Further solutions to handle bodies with variable connectivitygraphs include transformer-based solutions, such as Amorpheus [42] and the recently proposed
AnyMorph [43]. While these works study the problem of controlling bodies with a variable numberof segments and connectivity graph, we focus on perturbations which affect some morphologicalparameters, but without changing the body structure.
Hierarchical RL A common approach to solve complex RL tasks is to decompose them in a hierarchyof subproblems [44].
Walking with different body shapes could be interpreted as the union of along-horizon task (locomotion) and a short-horizon one (body shape identification).
Algorithmsto tackle multi-horizon control problems by
A biology-inspired architecture to handle morphological perturbations
Adaptation to morphological perturbations requires an agent to rapidly identify which joints areaffected and what compensatory torques are required.
We propose a Distributed Morphological
Attention Policy (DMAP).
With DMAP (Figure 1), we seek to facilitate this process by promotingthe formation of communication pathways from individual body parts to the joint controllers.
Thearchitecture is designed according to the previously mentioned principles of (biological) sensorimotorcontrol, implemented in the following way:
Independent low-level processing.
Each proprioceptive channel is first processed independently,in order to obtain a channel-specific representation.
This is inspired by the processing in theproprioceptive pathway [9, 10, 11].
Distributed control.
Independent networks control each of theagent’s joint, by outputting one single action scalar (corresponding to a joint’s torque).
This is inspiredby the architecture of the spinal cord and the motor homunculus [10, 8, 2].
Gated connectivity.
An attention mechanism assigns the connection weight between the control policy of a joint andthe features of each component of the proprioceptive state.
This is inspired by dynamic gatingmechanisms in the brain [11, 12, 13].
Based on these principles, we propose the following architecture (Figure 1):
A feature encoder receives in input a history of T proprioceptive states and actions (( s ( t − T ) , a ( t − T ) ) , ..., ( s ( t − 1) , a ( t − 1) )) , which we denote as H ( t )
∈ R ( N S + N A ) × T , N S and N A being the state and action space size, respectively.
Each row h ( t ) i ∈
R T of H ( t ) is processed independently by the same Temporal Convolutional Network (TCN). A linear transformation maps each temporally-filtered input channel into the vectors k ( t ) i ∈
R N A , representing the attention of each joint towards that input channel i , and v ( t )
i ∈ R N V , a value encoding vector of size N V a
hyperparameter of the network). The vectors k ( t ) i and v ( t ) i from different channels are stacked toform the matrices ˜ K ( t )
∈ R ( N S + N A ) × N A and V ( t )
∈ R ( N S + N A ) × N V . The matrix K ( t ) is obtainedby applying a softmax operation to each column of ˜ K ( t ) .
The context encoding vectors for each jointcontroller are the rows e ( t ) T of the matrix E ( t ) =
K ( t ) T V ( t ) , which are convex combinations of thefeatures extracted from each sensory modality, weighted by their attention score (Section A.1).
In thisway, each controller can focus on the relevant morphological information carried by specific sensorymodalities, while ignoring the irrelevant ones.
Furthermore, the attention matrix is conditioned to therecent transition history, making it adaptable both across episodes (and thus morphological states)and within a single episode.
Differently from standard policy networks, we adopt independent controllers
Ω i , i ∈ 1 , .., N A , foreach joint, implemented as distinct fully-connected networks.
Each network outputs the element a ( t ) iof the action vector a ( t ) based on the current state s ( t ) , the previous action a ( t − 1) and the contextencoding e ( t ) i .
Distributed action selection is a natural way to enable the emergence of sensorimotorpathways from the sensory data of different body parts to the controller of a joint.
As one controllerneeds to predict one single action value, we show in our experiments that smaller fully-connectednetworks are sufficient to successfully solve the considered tasks, limiting the complexity overheadstemming from distributed control (Section A.2).
Qualitatively, independent policy networks canmore easily adapt their behavior when a morphological perturbation affects a localized area of thebody, leaving the control policy of the other networks unaltered.
Morphological perturbation environments
We consider four locomotion environments of the PyBullet physics simulator [48]: Hopper, Walker,
Ant and Half-Cheetah, which are standard benchmarks for continuous control reinforcement learningalgorithms [49].
To test the ability of an agent to learn locomotion with a changing body, we introducea set of morphological perturbations for each of them.
The perturbations consist in modifying thesize and the length of one or more body parts (Section A.3).
The parameter σ controls the size of theperturbation space C σ , limiting the intensity of the perturbation between − σ and σ . The perturbation
[Figure] [Figure] is defined as the relative deviation of a morphological parameter from its original value.
For example,a perturbation of 0.5 applied to a leg thickness will make it 1.5 times thicker.
This construction resultsin widely varying body shapes for each of the four environments (Figure 2).
As in the unperturbed environments, the training is organized in episodes of at most 1000 time steps.
At the beginning of each episode, bodily parameters are sampled uniformly at random in the space ofadmissible perturbations (and fixed throughout this episode).
This setup defines a Contextual Markov
Decision Process (CMDP) [16] M = ⟨S , A , O , T , R , γ, C σ ⟩ , determined by the state space S , theobservation function O , the action space A , the transition dynamics T , the reward function R , thediscount factor γ and the context C σ .
The observation only includes proprioceptive information, in theform of angular position and velocity of the joints, besides other non-visual quantities (Section A.3).
It does not contain explicit information about the current context, as identical observations can comefrom bodies with different parameters.
On the other hand, the context influences the transitiondynamics p ( s t +1 | s t , a t , C σ ) , modifying the effect of an action depending on the body configuration.
This condition of partial observability makes the locomotion problem considerably harder.
However,as the parameters defining the context are constant throughout one episode, we show that a controlpolicy can adapt its locomotion strategy to it.
Experiments
We train all the agents with Soft Actor Critic (SAC) [31], using the open source library RLLib [50].
The experiments have been run on a local CPU cluster, for a total of approx.
100 000 cpu-hours.
Weuse similar hyperparameters to [32] (Section A.3), which provide state-of-the-art results in the baseversions of the four PyBullet environments considered in this work, and we run every experimentfor 5 random seeds.
We found that the training of the critic network benefits from augmenting theinput observation with the current morphological perturbation.
Therefore, we included it for allpolicy architectures, except for the Simple agent.
This addition does not pose any limitation to thedeployment of the proposed algorithms, as the policy network alone is necessary at that stage.
The experiments involve five different policy networks, which differ in architecture, required inputdata and training procedure (Table 1). [Table] [Table]
[Table]
I n c r e a s i n g c o m p l e x i t y During training, the morphology parameters for each episode are changed (for the complete episode).
To evaluate the agent’s ability to adapt to an unseen morphology, we consider the reward it accumulatesin the complete test episode (without changing the policy; zero-shot setting).
We do so for all thebody configurations in the test set.
During the testing phase, the agent does not observe the rewards itcollects.
The rewards in bold are significantly higher according to a paired t-test (p-value < 0.001).
We denote the tests as IID if the 100 body configurations were extracted from the morphologicalspace used during the training ( σ train ≥ σ test ), otherwise we denote them as OOD ( σ train < σ test ).
5.1 A morphology encoding boosts learning in strongly perturbed environments
To assess whether an agent can benefit from an encoding of its morphology when learning a locomotion policy, we compare the performance achieved when providing only the proprioceptive state(Simple) and when augmenting it with the raw perturbation (Oracle).
Specifically, this Oracle followsthe architecture of Kumar et al.
It extracts a morphology encoding from the raw perturbationvector and concatenates it to the observation vector, thus augmenting the input of the policy networkand breaking the partial observability of the environment.
We found that, while in mildly perturbedenvironments Simple performs competitively with Oracle, even outperforming it in Walker, learninga control policy solely based on the proprioceptive state becomes difficult in the larger perturbationspace (Figure 3 and Section A.4).
In all our experiments with σ ∈ { 0 . 3 , 0 . 5 } a morphology representation proved useful, which demonstrates, as expected, that learning an environment encoding is aviable option to deal with strongly perturbed environments.
5.2 The
morphology encoding can be regressed from experience
The morphology parameters C influence the transition dynamics p ( s t +1 | s t , a t , C ) by changing thephysical properties of the body.
Therefore, they are implicitly represented in a sequence of transitions,from which they might be inferred.
To test this possibility, we apply the RMA algorithm
[15] to ourmorphology-perturbed environments.
It leverages the latent representation of the body generatedby a trained Oracle, which has access to the contextual information, to train a TCN network toregress the context encoding from a short history of T proprioceptive states and actions H ( t ) =(( s ( t − T ) , a ( t − T ) ) , ..., ( s ( t − 1) , a ( t − 1) )) .
RMA achieves similar average reward to the Oracle (Table 2),from which we can deduce that a sequence of 30 transitions (approx.
0.5 s in our experiments), onlyincluding previous proprioceptive states and actions, is sufficient to extract an embedding of thecontext, proving this two-phase adaptation procedure viable for morphological perturbations. [Table] [Table]
5.3 DMAP enables end-to-end training of a morphology encoding network
It is unclear whether a TCN network requires a two-step training through imitation to learn howto extract a morphology encoding from a sequence of transitions.
In fact, if such encoding helpsmaximizing the reward, the TCN might successfully learn it end-to-end with reinforcement learning.
In our experiments, however, while in Hopper and in Half Cheetah an end-to-end training of a TCNencoder proves successful, in Ant and Walker it encounters strong instability and inconsistencyacross the random seeds (Figure 4).
Surprisingly, in these two latter environments the addition ofthe TCN encoder is even harmful for the overall performance, as Simple achieves stronger resultsin the same environments.
Further analysis (Section A.4) reveals that, for certain random seeds,the training process is disrupted before the agent can learn how to walk.
We hypothesize that thisphenomenon is due to the difficulty of extracting a meaningful encoding to directly optimize thereward without a suitable inductive bias, with the effect of the encoder acting as a confounder forthe policy network.
Indeed, replacing TCN with DMAP leads to better stability during the training(Figure 4) and to a general performance improvement, particularly in Ant and Walker (Section A.4).
When performing an ablation analysis, we found that the morphological encoding is fundamental,especially for the dynamic stability of the agents (Section A.5).
Differently from TCN, DMAPperforms on par with Oracle in all the considered environments, with the exception of Walker withσ = 0 .
Oracle and DMAP also perform similarly when tested OOD, with a slight advantage for
DMAP (Figure 5 and Section A.4).
Finally, we compared the adaptation speed of RMA and DMAPto unseen morphological perturbation.
We found that the two algorithms perform motor adaptationequally fast (Section A.6).
5.4 Attention gates the sensory input through information pathways
We hypothesized that DMAP enables the agent to learn which components of the input are relevantfor each joint’s controller.
Next, we verify that such pathways emerge by analyzing the values of theattention matrix K ( t ) , both in the steady and dynamic condition.
The steady attention matrix K represents the strength of each input to output (i.e., sensorimotor)connection.
We found morphology-specific patterns by averaging elements across time and episodes(Figure 6).
As in the spinal cord, the controllers typically attend to the sensory input of mechanicallylinked parts (e.g., foot controller to foot velocity, etc.).
Yet, this topography is not universal: forinstance in the hopper, all controllers focus on the leg angle and the foot velocity.
Furthermore,the attention patterns seem to reflect the bilateral symmetry of Ant and Walker and a front-backasymmetry due to the directionality of the locomotion task. [Figure] c k [Figure]
f r o n t H I P
[Figure]
a c k [Figure] f r o n t K N EE [Figure]
[Figure]
To analyze the temporal evolution of attention, we project the attention matrix K ( t ) onto a 3dimensional space learned with UMAP [51] (Figure 7).
Depending on the phase of the gait, theattention matrix K ( t ) strengthens or weakens specific sensorimotor pathways (Figure 7 A).
The graphsreveal a rotational dynamics of the gain modulation typical of biological, legged locomotion [52, 53,11].
Interestingly, the attention dynamics seem significantly less tangled than that of the proprioceptiveand action input (Figure 7 B).
We quantified this visual impression across the different environmentsand found a robust effect (Section A.8).
We speculate that this untangling mechanism might contributeto the robustness of the policy to morphological perturbations.
Thus, the connectivity patterns between input channels and joint controllers change throughout the different phases of the gait.
B. UMAP embeddings of the input observations
O input actions A
( t ) ∈ R N A × T and attention matrix K for 3 different morphologies of the Ant.
The change of body configuration causes a shift of the attention dynamics in the low-dimensional space.
The trajectories of the attention embedding seem less tangled than the corresponding input.
We quantified this with a tangling metric, Q (averaged over time and morphologies; large values correspond to highly tangled trajectories).
We refer to Section A.8 for the technical details.
Discussion
We proposed DMAP, a novel architecture that integrates computational principles from the sensorimotor system.
We find that DMAP enables the end-to-end training of a RL policy in environmentswith changing body parameters, without having access to the perturbation information.
It matchesor even surpasses the performance of an Oracle when tested in a zero-shot manner on unseen bodyconfigurations.
Interestingly, the attention mechanism of DMAP solves this challenging task byestablishing gait-phase-dependent gain modulation, robust across morphologies.
Notably, gains inthe nervous system also change in a phase-dependent way [52, 53, 11].
Limitations and broader impact.
One characteristic of our current architecture design is that eachjoint controller observes the full proprioceptive state.
A future direction will be to limit the input tothe local sensory data.
Such design, separating low-latency sensorimotor connection (reflexes) andslower connections to distal body parts, would be closer to the locomotor system of the animals [2].
Furthermore, in our experiments, we have used an oracle critic to learn an estimate of the cumulativereward and guide the training of the policy.
However, a suitable inductive bias, possibly inspired bythe neural circuits responsible for reward prediction error, could remove this limitation.
Our research aims to facilitate the training of control policies by transferring insights from neuroscience to artificial intelligence.
Powerful reinforcement learning algorithms might find applicationin robotics, potentially enabling, in the long term, a large-scale deployment of autonomous agents.
Automation comes with great opportunities, and ethical concerns [54].
A.1 DMAP detailed architecture
Distributed Morphological Attention Policy (DMAP) consists of three main components: independentlow-level proprioceptive processing, an attention-based morphology encoder and a distributed jointcontroller ( 8).
[Figure]
A.2 Hyperparameters
Table 3 lists the hyperparameters of SAC used for our experiments.
They are the same for all networkarchitectures, without specific fine tuning.
We also list the parameters of the neural networks of thedifferent agents.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[128, 32] [128, 32] [128, 32] [128, 32] Encoding size
[Table]
[128, 128] [128, 128] [128, 128] [128, 128] Q encoder hiddens
[256, 128] [256, 128] [256, 128] [256, 128] Q hiddens [128, 128] [128, 128] [128, 128] [128, 128] Activation
ReLU
ReLU ReLU ReLU
DMAP
Num TCN layers   3 Num
[Table] [Table]
size
[Table]
[5, 3, 3] [5, 3, 3] [5, 3, 3] [5, 3, 3] Stride
[Table]
[4, 1, 1] [4, 1, 1] [4, 1, 1] [4, 1, 1] Policy encoder hiddens Encoding size   4 Policy
[Table]
[32, 32] [32, 32] [32, 32] [32, 32] Num
[Table]
networks   3 Q encoder hiddens [256, 128] [256, 128] [256, 128] [256, 128] Q hiddens [128, 128] [128, 128] [128, 128] [128, 128] Activation
ReLU ReLU
ReLU ReLU
A.3 Perturbed environments specifications
The state and action components of the four perturbed environments (Ant, Half Cheetah, Walker,
Hopper) are the same as the ones of the environment they are derived from.
They are listed in Table 4,ordered as they appear in the state and action vectors.
The body parameters are sampled from auniform distribution at the beginning of each episode.
The body segments are perturbed in smallsubgroups, e.g., the two segments forming a limb of Ant, in which the perturbation is consistent.
Theparameters defining a perturbation space are detailed in Table 5.
During our experiments, we experienced an unusual behavior in the Half Cheetah environment forparticularly light limbs.
In this case, the Cheetah could move the legs at very high speed.
In thiscase, PyBullet allowed an unnatural locomotion, in which the agent could advance without touchingthe ground, seemingly ignoring gravity.
For this reason, we set a lower bound to the perturbationon the length and the thickness of the limbs, corresponding to − 0 .
3 (only for the Half Cheetah withσ = 0 . 5 ).
This clipping removed the unusual behavior of the simulator.
[Table]
knee angle
Back shin contact
Front right knee velocity
Back thigh contact
Back left foot contact Front left foot contact
Back right foot contact
Front right foot contact
Action
Back left hip torque
Front foot torque
Right thigh torque
Thigh torque
Back left knee torque
Front shin torque
Right leg torque
Leg torque
Front left hip torque
Front thigh torque
Right foot torque
Foot torque
Front left knee torque
Back foot torque
Left thigh torque Back right hip torque
Back shin torque
Left leg torque
Back right knee torque
Back thigh torque
Left foot torque
Front right hip torque Front right knee torque
A.4 Performance summary
Table 6 and Table 7 compare the IID performance of Simple-Oracle and of TCN-DMAP.
The numbersin bold denote a significant statistical difference between the two methods (p-value < 0 .
001 , pairedt-test).
We then list the IID ( 8) and OOD ( 9 10 11) test results of all the agents trained for this work.
Some negative values should not surprise the reader, as some agents, when tested way outside of the
[Table]
training distribution, fail to walk, collecting more penalization (e
[Table]
g., due to undesired contact force or
[Table]
excessive energy expenditure) than positive reward
[Table] [Table]
[Table] [Table]
We also show the graphs of the reward decay of Oracle, DMAP and TCN as a function of theperturbation intensity (Figure 9).
Finally, we visualize the distribution of the IID and OOD episodereward of DMAP, compared to TCN, across test body configurations, normalized by the Oraclereward (Figure 10 and 11).
[Table] [Table]
[Table] [Table]
[Table]
Hopper σ = 0 .
[Table]
Hopper σ = 0 [-.1683 ±
-]. [Table]
[Table]
[Table] [Table] [Table] [Table]
Hopper σ = 0 [-.
-].
[Table] [Table] [Table] [Table]
[Table] [Table]
Hopper σ = 0 [-.
-].
[Table] [Table] [Table] [Table]
[Table]
[Table]
To isolate the contribution of the morphology encoding on the performance of DMAP, we performedtest-time ablation by evaluating the trained DMAP distributed policies, while removing the morphology encoding.
We found a performance drop for all the environments and perturbation levels(Table 12). It is especially severe in Hopper and Walker (approx.
50% ), where the controller withablated encoding fails to stabilize the agents, causing the early termination condition to be met soonerand thus yielding lower episode reward. This suggests that the attention mechanism is fundamentalfor the dynamic stability of these agents.
One distinctive characteristic of RMA [15] is the short time it requires to perform motor adaptationto perturbations. We sought to compare the adaptation speed for RMA and DMAP.
We could not perform onlinechanges of the morphological parameters in PyBullet within an episode, therefore we comparedspeeds of RMA and DMAP when starting to run from the resting position with an unseen morphology
[Table]
[-Hopper σ = 0 .730 ±953 ±
-]
I n c r e a s i n g c
o
m p l e x i t y σ = 0 .σ
= 0 .
DMAP match or even surpass the Oracle performance and outperform TCN (except for Hopper with σ = 0 . 5 ) for all training environment.
[Table] in the Ant environment throughout all test episodes. We define the average speed as the norm ofthe projection of the velocity vector of the agent’s center of mass on the running surface and, sinceit varies during one gait cycle, we average it over a window of 30 transitions. We find that for allperturbation intensities DMAP and RMA reach the final speed after a similar number of timesteps(Figure 12).
This analysis suggests that the difference in the training procedure (imitation of the
Ant 0. Half
Walker 0.
Hopper
Perturbation intensity
N o r m a li z e d e p i s o d e r e w a r d
DMAP
TCN
Oracle’s encoder for RMA, end-to-end for DMAP) does not lead to a relevant changes in adaptationspeed. A.7 Robustness to a single leg length perturbation To quantify the robustness of DMAP to a specific morphological perturbation, we performed anexperiment where we progressively change the length of a limb in the ant (from 0 % perturbationintensity which is the normal limb to 100 % which corresponds to limb amputation). We investigated how it affects the episode reward.
We find that the reward monotonically decreasessimilarly for different limbs with an increase in robustness when DMAP is trained with higherperturbation intensity (Figure 13). A.8 Analysis of the attention dynamics We quantify the tangling of a trajectory with the measure introduced by Russo et al.
[Equation] where x t represents the trajectory at a specific time and x t ′ denotes the corresponding temporalderivative, and ϵ is a constant. We use this measure to study the the difference in trajectory tanglingbetween attention and action/observation (Figure 14). We find that the attention dynamics are less
Ant train =0. Ant train =0. [Table]
[Table] Ant train =0.
[Table]
Walker train =0.
Walker train =0. Walker train =0.
Hopper train =0. Hopper train =0.
[Figure]
N o r m a li z e d e p i s o d e r e w a r d
DMAP tangled than the inputs.
This suggests that DMAP achieves control across morphological parametersby regularizing. To gain further insights into this mechanism, we verified if the attentional dynamics become lessentangled over learning. Thus, we visualized how the attention dynamics evolves throughout thetraining (Figure 15). While at an early stage the characteristic rotational dynamics is absent, itemerges as the training proceeds, progressively untangling the trajectories.
[Figure]
Walker
[Figure]