DMAP: a Distributed Morphological Attention Policy for learning to locomote with a changing body
AffiliationAddress
Abstract
Reinforcement learning typically seeks to learn control policies in stable environments.
Yet, real world scenarios require continuous adaptation.
In particular,learning to locomote when the length and the thickness of different body parts varyis challenging, as the policy is required to adapt to the current configuration tosuccessfully balance and advance the agent.
We study this problem in four classicalcontinuous control environments, augmented with morphological perturbations.
We show that a control policy based on the proprioceptive state performs poorlywith highly variable body configurations, while an (oracle) agent with access to alearned encoding of the perturbation performs significantly better.
We introduce
DMAP, a biologically-inspired, attention-based policy network architecture.
Itcombines a distributed policy, with individual controllers for each joint, and anattention mechanism, to dynamically gate sensory information from different bodyparts.
DMAP can be trained end-to-end in all the considered environments andperturbation intensities, overall matching or surpassing the performance of an oracle agent with access to the morphology information.
Thus DMAP, implementingprinciples of control drawn from the biological world, provides a strong inductivebias for learning challenging sensorimotor tasks.
Overall, our work corroboratesthe power of these principles in challenging locomotion tasks.
Introduction
Animals and humans are highly adaptive to changes in the external environment and in their ownbody.
For instance, animals can, after an early onset, locomote throughout development, when theirbody size and weight change dramatically.
They can also robustly deal with irregular surfaces andload changes [1, 2, 3, 4].
Here, we focus on the problem of learning to walk when the body is subjectto morphological perturbations, consisting in the change of length and thickness of its parts.
Forthis purpose, we adapted four different locomotion environments from OpenAI Gym [5] (Ant, Half Cheetah, Walker, Hopper), in which robots have to learn how to walk as fast as possible.
In ouradaptive setting, the agent’s body parameters are sampled from a morphological perturbation space atthe beginning of each episode and we sought to find reinforcement learning policies that could dealwith these varying bodies.
We hypothesize that principles of the sensorimotor system provide a strong inductive bias to robustlylearn such a policy.
Specifically, we built on three well-known features: 1) Independent low-levelprocessing: Somatosensory inputs are first processed locally (e.g., by temporal filtering)
beforethey are integrated across more distal body parts and represented in an orderly way [6, 7, 8]. 2)
DMAP - Distributed Morphological Attention Policy
E p
i s o d e s
T i m e
Time
TCN
TCN
TCN Ω Ω Ω
Channel features
Distributed joint controllers
Action
Joint-channel attention
Proprioceptive state history
H
Independent representations
DMAP
IID Test
OOD Test
A
B
K
V
N A
Distributed control: Different body parts are controlled in a distributed fashion [7, 8, 4].
3) Gatedconnectivity:
Sensory information is dynamically gated in a task dependent manner [9], also beyondsomatosensation [10], and integrated with action and value signals in the brain [11, 8, 12].
Hence, we propose the Distributed Morphological Attention Policy (DMAP) architecture (Figure 1).
We show that it can robustly learn to locomote with changing bodies without access to their morphological parameters.
To assess the performance of this architecture, we compare it to several otheragents.
Experiments reveal that an agent with access only to the proprioceptive state, consisting of itsjoint angles and velocities ( Simple ), fails to learn an effective locomotion strategy for a large varietyof body configurations.
In contrast, augmenting the observation with the current body shape leads toa strong ceiling performance ( Oracle ).
Importantly, for all different environments, DMAP is as goodas Oracle.
We also compare DMAP to a recently proposed two-step training technique called rapid motoradaptation ( RMA )
This algorithm trains a temporal convolutional network to infer a contextencoding from a short history of proprioceptive state, imitating the Oracle’s morphology encoder.
While RMA can also reach Oracle performance, we show that end-to-end training of such networkdoes not succeed for more complicated environments (Ant and Walker).
In contrast, DMAP’s policycan be trained end-to-end without access to the morphological parameters and reaches or surpasses
Oracle performance for all four environments (Ant, Half Cheetah, Walker, Hopper).
This suggests that independent low-level processing, distributed control and dynamic gating mechanisms provide astrong inductive bias for sensorimotor control.
Finally, we analyze the information gating mechanismof DMAP, which establishes pose-dependent sensorimotor pathways.
The low dimensional embeddingof the gating mechanism exhibits rotational dynamics, robustly across different morphologies.
Related work
Benchmarks for zero-shot adaptation.
The setup of the experiments described in this paper can beframed as a zero-shot adaptation problem, as the agents are evaluated starting from the first momentin which they interact with an unseen context [14].
While classic RL benchmarks, such as Arcade[15] and rllab [16], assess the ability of an agent to maximize the cumulative reward in the trainingenvironment, more recent ones introduce perturbations in the dynamics [17, 18], in the observationfunction [19, 20] or in the reward function [21] at test time.
For a more complete list, we refer to[14].
Mann et al.
showed that the performance of RL agents under morphological distributional shiftscorrelates with the ability of internal models to generalize [22].
We took their design of environmentsas an inspiration for the design of morphological changes in our work.
Context encoding.
Algorithms which enable learning in changing environments are important for thesim-to-real transfer problem, where domain randomization represents a successful strategy to bridgethe reality gap [23, 24, 25].
A notable approach to solving the zero-shot transfer problem consists inencoding the context into a hidden state, e.g., by online system identification [26], or with a recurrentneural network (RNN) [16, 27].
However, while off-policy continuous control algorithms such as TD3 [28] and SAC [29] are the current state of the art among model-free RL algorithms, both insample efficiency and final performance [30], training a recurrent neural network off-policy presentsrelevant technical challenges [31].
As the focus of our work is on the development of a networkarchitecture to handle a changing body, we choose to follow the more straightforward approach tosystem identification proposed in [13], where the authors show that a context encoding is successfullyextracted from a short sequence of past states and actions through temporal convolutions.
Distributed control.
An agent can learn to generalize to multiple body shapes by using modularpolicies to control each joint independently, e.g., treating each joint [32] or groups of joints [33] asa node of the graph.
For example, Nervenet [34] achieves state-of-the-art performance on standard
RL benchmarks by leveraging different policies for each joint type.
However, it cannot generalizeto different designs in a zero-shot manner without fine-tuning.
On the other hand, Shared Modular
Policies (SMP) [35] employs the same policy for each joint, enabling an agent to deal with variableobservation and action space sizes.
While these works study the problem of controlling bodies with avariable number of segments, we focus on perturbations which affect some morphological parameters,but without changing the body structure.
A biology-inspired architecture to handle morphological perturbations
Adaptation to morphological perturbations requires an agent to rapidly identify which joints areaffected and what compensatory torques are required.
We propose a Distributed Morphological
Attention Policy (DMAP).
With DMAP (Figure 1), we seek to facilitate this process by promotingthe formation of communication pathways from individual body parts to the joint controllers.
Thearchitecture is designed according to the previously mentioned principles of (biological) sensorimotorcontrol, implemented in the following way:
Independent low-level processing.
Each proprioceptive channel is first processed independently,in order to obtain a channel-specific representation.
This is inspired by the processing in theproprioceptive pathway [7, 8, 9].
Distributed control.
Independent networks control each ofthe agent’s joint, by outputting one single action scalar (corresponding to a joint’s torque).
This isinspired by the architecture of the spinal cord and the motor homunculus [8, 6, 2].
Gated connectivity.
An attention mechanism assigns the connection weight between the control policy of a joint and [Figure] [Figure] the features of each component of the proprioceptive state.
This is inspired by dynamic gatingmechanisms in the brain [9, 10, 11].
[Figure]
A feature encoder receives in input a history of T proprioceptive states and actions(( s ( t − T ) , a ( t − T ) ) , ..., ( s ( t − 1) , a ( t − 1) )) , which we denote as H ( t )
∈ R ( N S + N A ) × T , N S and N Abeing the state and action space size, respectively.
Each row h ( t ) i ∈
R T of H ( t ) is processed independently by the same Temporal Convolutional Network (TCN), which maps the time series of eachstate and action component into the vectors k ( t ) i ∈
R N A , representing the attention of each jointtowards that input channel i , and v ( t )
i ∈ R N V , a value encoding vector of size N V .
The vectorsk ( t ) i and v ( t ) i from different channels are stacked to form the matrices K ( t ) ∈ R ( N S + N A )
× N A and V ( t ) ∈ R ( N S + N A )
× N V
The context encoding vectors for each joint controller are the rows e ( t ) Tof the matrix E ( t ) =
K ( t ) T V ( t ) , which are convex combinations of the features extracted fromeach sensory modality, weighted by their attention score (Section A.1).
In this way, each controllercan focus on the relevant morphological information carried by specific sensory modalities, whileignoring the irrelevant ones.
Furthermore, the attention matrix is conditioned to the recent transitionhistory, making it adaptable both across episodes (and thus morphological states) and within a singleepisode. 3.2 Distributed joint controllers [Figure]
Differently from standard policy networks, we adopt independent controllers
Ω i , i ∈ 1 , .., N A , foreach joint, implemented as distinct fully-connected networks.
Each network outputs the element a ( t ) iof the action vector a ( t ) based on the current state s ( t ) , the previous action a ( t − 1) and the contextencoding e ( t ) i .
Distributed action selection is a natural way to enable the emergence of sensorimotorpathways from the sensory data of different body parts to the controller of a joint.
As one controllerneeds to predict one single action value, we show in our experiments that smaller fully-connectednetworks are sufficient to successfully solve the considered tasks, limiting the complexity overheadstemming from distributed control (Section A.2).
Qualitatively, independent policy networks canmore easily adapt their behavior when a morphological perturbation affects a localized area of thebody, leaving the control policy of the other networks unaltered.
Morphological perturbation environments
We consider four locomotion environments of the PyBullet physics simulator [36]: Hopper, Walker,
Ant and Half-Cheetah, which are standard benchmarks for continuous control reinforcement learningalgorithms [37].
To test the ability of an agent to learn locomotion with a changing body, we introducea set of morphological perturbations for each of them.
The perturbations consist in modifying the size and the length of one or more body parts (Section A.3).
The parameter σ controls the size of theperturbation space C σ , limiting the intensity of the perturbation between − σ and σ .
The perturbationis defined as the relative deviation of a morphological parameter from its original value.
For example,a perturbation of 0.5 applied to a leg thickness will make it 1.5 times thicker.
This construction resultsin widely varying body shapes for each of the four environments (Figure 2).
As in the unperturbed environments, the training is organized in episodes of at most 1000 time steps.
At the beginning of each episode, bodily parameters are sampled uniformly at random in the space ofadmissible perturbations.
This setup defines a Contextual Markov Decision Process (CMDP) [14]
M = ⟨S , A , O , T , R , γ, C σ ⟩ , determined by the state space S , the observation function O , the actionspace A , the transition dynamics T , the reward function R , the discount factor γ and the context C σ .
The observation only includes proprioceptive information, in the form of angular position and velocityof the joints, besides other non-visual quantities (Section A.3).
It does not contain explicit informationabout the current context, as identical observations can come from bodies with different parameters.
On the other hand, the context influences the transition dynamics p ( s t +1 | s t , a t , C σ ) , modifying theeffect of an action depending on the body configuration.
This condition of partial observability makesthe locomotion problem considerably harder.
However, as the parameters defining the context areconstant throughout one episode, we show that a control policy can adapt its locomotion strategy to it.
Experiments
We train all the agents with Soft Actor Critic (SAC) [29], using the open source library RLLib [38].
The experiments have been run on a local CPU cluster, for a total of approx.
100 000 cpu-hours.
Weuse similar hyperparameters to [30] (Section A.3), which provide state-of-the-art results in the baseversions of the four PyBullet environments considered in this work, and we run every experimentfor 5 random seeds.
We found that the training of the critic network benefits from augmenting theinput observation with the current morphological perturbation.
Therefore, we included it for allpolicy architectures, except for the Simple agent.
This addition does not pose any limitation to thedeployment of the proposed algorithms, as the policy network alone is necessary at that stage.
The experiments involve five different policy networks, which differ in architecture, required inputdata and training procedure (Table 1).
[Table]
All the results refer to the cumulative reward obtained by zero-shot adaptation in 100 environmentscharacterized by body configurations not observed during the training.
The rewards in bold aresignificantly higher according to a paired t-test (p-value < 0.001).
We denote the tests as IID ifthe 100 body configurations were extracted from the morphological space used during the training( σ train ≥ σ test ), otherwise we denote them as OOD ( σ train < σ test ).
5.1 A morphology encoding boosts learning in strongly perturbed environments
To assess whether an agent can benefit from an encoding of its morphology when learning a locomotion policy, we compare the performance achieved when providing only the proprioceptive state(Simple) and when augmenting it with the raw perturbation (Oracle).
Specifically, this Oracle followsthe architecture of Kumar et al.
It extracts a morphology encoding from the raw perturbation
I n c r e a s i n g c
o m p l e x i t y [Table] vector and concatenates it to the observation vector, thus augmenting the input of the policy networkand breaking the partial observability of the environment.
We found that, while in mildly perturbedenvironments Simple performs competitively with Oracle, even outperforming it in Walker, learninga control policy solely based on the proprioceptive state becomes difficult in the larger perturbationspace (Figure 3 and Section A.4).
In all our experiments with σ ∈ { 0 . 3 , 0 . 5 } a morphology representation proved useful, which demonstrates, as expected, that learning an environment encoding is aviable option to deal with strongly perturbed environments.
5.2 The morphology encoding can be regressed from experience
The morphology parameters C influence the transition dynamics p ( s t +1 | s t , a t , C ) by changing thephysical properties of the body.
Therefore, they are implicitly represented in a sequence of transitions,from which they might be inferred.
To test this possibility, we apply the RMA algorithm
[13] to ourmorphology-perturbed environments.
It leverages the latent representation of the body generatedby a trained Oracle, which has access to the contextual information, to train a TCN network toregress the context encoding from a short history of T proprioceptive states and actions H ( t ) =(( s ( t − T ) , a ( t − T ) ) , ..., ( s ( t − 1) , a ( t − 1) )) .
RMA achieves very similar average reward to the Oracle(Table 2), from which we can deduce that a sequence of 30 transitions (approx.
0.5 s in ourexperiments), only including previous proprioceptive states and actions, is sufficient to extract anembedding of the context, proving this two-phase adaptation procedure viable for morphologicalperturbations.
5.3 DMAP enables end-to-end training of a morphology encoding network
It is unclear whether a TCN network requires a two-step training through imitation to learn howto extract a morphology encoding from a sequence of transitions.
In fact, if such encoding helpsmaximizing the reward, the TCN might successfully learn it end-to-end with reinforcement learning. I n c r e a s i n g c o
m p l e x i t y In our experiments, however, while in Hopper and in Half Cheetah an end-to-end training of a TCNencoder proves successful, in Ant and Walker it encounters strong instability and inconsistency acrossthe random seeds (Figure 4).
Surprisingly, in these two latter environments the addition of the TCNencoder is even harmful for the overall performance, as Simple achieves stronger results in the sameenvironments.
Further analysis (Section A.4) reveals that, for certain random seeds, the trainingprocess is disrupted before the agent can learn how to walk.
We hypothesize that this phenomenon isdue to the difficulty of extracting a meaningful encoding to directly optimize the reward without asuitable inductive bias, with the effect of the encoder acting as a confounder for the policy network.
Indeed, replacing TCN with DMAP leads to better stability during the training (Figure 4) and to ageneral performance improvement, particularly in Ant and Walker (Section A.4).
Differently from
TCN, DMAP performs on par with Oracle in all the considered environments, with the exceptionof Walker with σ = 0 . 1 .
Oracle and DMAP also perform similarly when tested OOD, with a slightadvantage for DMAP (Figure 5 and Section A.4). which receives the exact perturbation information. [Figure]
for each agent, trained with σ = 0 . We can observe several interpretable patterns, e.g., the bilateral symmetry in the Ant, as well as a front-back asymmetry (due to the fixed running direction). Also in the Ant, the hip controllers mainly look at the angular velocity of the front knees and of the back hips, while the ankle controllers do the opposite. In the Half Cheetah, too, the attention map develops a front-back asymmetry, with a preference for the input angular position, and each controller mainly attends to the state of the joint it controls.
To isolate the contribution of the morphology encoding on the performance of DMAP, we performedtest-time ablation by evaluating the trained DMAP distributed policies, while removing the morphology encoding. We found a performance drop for all the environments and perturbation levels(Table 3). It is especially severe in Hopper and Walker (approx.
50% ), where the controller withablated encoding fails to stabilize the agents, causing the early termination condition to be met soonerand thus yielding lower episode reward. This suggests that the attention mechanism is fundamentalfor the dynamic stability of these agents. [Table]
5.5 Attention gates the sensory input through information pathways
We hypothesized that DMAP enables the agent to learn which components of the input are relevantfor each joint’s controller.
Next, we verify that such pathways emerge by analyzing the values of theattention matrix K ( t ) , both in the steady and dynamic condition.
The steady attention matrix K represents the strength of each input to output (i.e., sensorimotor)connection.
We found morphology-specific patterns by averaging elements across time and episodes(Figure 6).
As in the spinal cord, the controllers typically attend to the sensory input of mechanicallylinked parts (e.g., foot controller to foot velocity, etc.).
Yet, this topography is not universal: forinstance in the hopper, all controllers focus on the leg angle and the foot velocity.
Furthermore,the attention patterns seem to reflect the bilateral symmetry of Ant and Walker and a front-backasymmetry due to the directionality of the locomotion task.
The attentional dynamics of K are cyclic, consistently with the periodic gait of the agents.
Thus, the connectivity patterns between input channels and joint controllers change throughout the different phases of the gait.
B. UMAP embeddings of the input observations O ( t ) ∈
R N S × T , input actions
A ( t ) ∈ R N A × T and attention matrix K for 3 different morphologies of the Ant.
The change of body configuration causes a shift of the attention dynamics in the low-dimensional space.
The trajectories of the attention embedding seem less tangled than the corresponding input.
We quantified this with a tangling metric, Q (averaged over time and morphologies; large values correspond to highly tangled trajectories).
We refer to Section A.5 for the technical details.
To analyze the temporal evolution of attention, we project the attention matrix K ( t ) onto a 3dimensional space learned with UMAP [39] (Figure 7). Depending on the phase of the gait, theattention matrix K ( t ) strengthens or weakens specific sensorimotor pathways (Figure 7 A). The graphsreveal a rotational dynamics of the gain modulation typical of biological, legged locomotion [40, 41,9]. Interestingly, the attention dynamics seem significantly less tangled than that of the proprioceptiveand action input (Figure 7 B).
We quantified this visual impression across the different environmentsand found a robust effect (Section A.5).
We speculate that this untangling mechanism might contributeto the robustness of the policy to morphological perturbations.
Discussion
We proposed DMAP, a novel architecture that integrates computational principles from the sensorimotor system.
We find that DMAP enables the end-to-end training of a RL policy in environmentswith changing body parameters, without having access to the perturbation information.
It matchesor even surpasses the performance of an Oracle when tested in a zero-shot manner on unseen bodyconfigurations.
Interestingly, the attention mechanism of DMAP solves this challenging task byestablishing gait-phase-dependent gain modulation, robust across morphologies.
Notably, gains inthe nervous system also change in a phase-dependent way [40, 41, 9].
Limitations and broader impact.
One characteristic of our current architecture design is that eachjoint controller observes the full proprioceptive state.
A future direction will be to limit the input tothe local sensory data.
Such design, separating low-latency sensorimotor connection (reflexes) andslower connections to distal body parts, would be closer to the locomotor system of the animals [2].
Furthermore, in our experiments, we have used an oracle critic to learn an estimate of the cumulativereward and guide the training of the policy.
However, a suitable inductive bias, possibly inspired bythe neural circuits responsible for reward prediction error, could remove this limitation.
Our research aims to facilitate the training of control policies by transferring insights from neuroscience to artificial intelligence.
Powerful reinforcement learning algorithms might find applicationin robotics, potentially enabling, in the long term, a large-scale deployment of autonomous agents.
Automation comes with great opportunities, and ethical concerns [42]. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6 for a discussion on
the scope and limitations of this work.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] In Section 6we discuss the borader impact of our research.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] In Section 4we introduce the experimental setup, specifying which information can be accessed by
the agents and what is not available.
(b) Did you include complete proofs of all theoretical results?
[N/A]
The results of this
work are computational.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] The code isincluded in the supplemental material, with instructions on how to train the agents to
reproduce the main results of the paper.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4 for a description of the training setup, Section A.3for the details about the environments and Section A.2 for the hypeprarameters of SACand of the architecture of the policies.
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[Yes] All the tables and the graphs of the learning curvesreport the mean value and the standard error of the mean, obtained by running each
experiment with 5 random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] In Section 5 we citethe two main libraries used in our work: PyBullet for the physics simulation and RLLibfor the reinforcement learning experiments.
(b) Did you mention the license of the assets?
[Yes] In Section 4 and Section 5 we specify
that the two libraries are open source.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
We included our source code with license in the supplemental material
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] No data from external sources was used for this study
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] No data from external sources was used for
this study
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
No human subject was involved in this study.
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
No human subject was involved in this
[Table]
study.
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
No human subject was involved in this study.
[Table]
A Appendix
[Figure] A.1 DMAP detailed architecture [Figure] Distributed Morphological Attention Policy (DMAP) consists of three main components: independentlow-level proprioceptive processing, an attention-based morphology encoder and a distributed jointcontroller ( 8).
[Figure]
A.2 Hyperparameters
Table 4 lists the hyperparameters of SAC used for our experiments.
They are the same for all networkarchitectures, without specific fine tuning.
We also list the parameters of the neural networks of thedifferent agents.
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
ReLU
ReLU
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
A.3 Perturbed environments specifications
The state and action components of the four perturbed environments (Ant, Half Cheetah, Walker,
Hopper) are the same as the ones of the environment they are derived from.
They are listed in Table 5,ordered as they appear in the state and action vectors.
The body parameters are sampled from auniform distribution at the beginning of each episode.
The body segments are perturbed in smallsubgroups, e.g., the two segments forming a limb of Ant, in which the perturbation is consistent.
Theparameters defining a perturbation space are detailed in Table 6.
During our experiments, we experienced an unusual behavior in the Half Cheetah environment forparticularly light limbs.
In this case, the Cheetah could move the legs at very high speed.
In thiscase, PyBullet allowed an unnatural locomotion, in which the agent could advance without touchingthe ground, seemingly ignoring gravity.
For this reason, we set a lower bound to the perturbationon the length and the thickness of the limbs, corresponding to − 0 .
3 (only for the Half Cheetah withσ = 0 . 5 ).
This clipping removed the unusual behavior of the simulator.
[Table]
Back shin contact
Front right knee velocity
Back thigh contact
Back left foot contact Front left foot contact
Back right foot contact
Front right foot contact
Action
Back left hip torque
Front foot torque
Right thigh torque
Thigh torque
Back left knee torque
Front shin torque
Right leg torque
Leg torque
Front left hip torque
Front thigh torque
Right foot torque
Foot torque
Front left knee torque
Back foot torque
Left thigh torque Back right hip torque
Back shin torque
Left leg torque
Back right knee torque
Back thigh torque
Left foot torque
Front right hip torque Front right knee torque
A.4 Performance summary
Table 7 and Table 8 compare the IID performance of Simple-Oracle and of TCN-DMAP.
The numbersin bold denote a significant statistical difference between the two methods (p-value < 0 .
001 , pairedt-test).
We then list the IID ( 9) and OOD ( 10 11 12) test results of all the agents trained for thiswork.
Some negative values should not surprise the reader, as some agents, when tested way outside
[Table]
of the training distribution, fail to walk, collecting more penalization (e
[Table]
g., due to undesired contact
[Table]
force or excessive energy expenditure) than positive reward
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
We also show the graphs of the reward decay of Oracle, DMAP and TCN as a function of theperturbation intensity (Figure 9)
[Table]
Finally, we visualize the distribution of the IID and OOD episodereward of DMAP, compared to TCN, across test body configurations, normalized by the Oraclereward (Figure 10 and 11). [Table]
[Table]
.248 ±289 ±171 ±219 ±267 ±221 ±
Hopper σ = 0 {+.1683 ±
+}
±729 ±1591 ±1618 ±1351 ±
Hopper σ = 0 {+.
+}
±1316 ±748 ±1368 ±1296 ±1121 ±
Hopper σ = 0 .532 ±729 ±456 ±672 ±666 ±562 ±
Hopper σ = 0 {+.
+}
±443 ±334 ±382 ±371 ±342 ±
We quantify the tangling of a trajectory with the measure introduced by Russo et al.
[Equation]
where x t represents the trajectory at a specific time and x t ′ denotes the corresponding temporalderivative, and ϵ is a constant.
We use this measure to study the the difference in trajectory tanglingbetween attention and action/observation (Figure 12).
We find that the attention dynamics are lesstangled than the inputs. This suggests that DMAP achieves control across morphological parametersby regularizing. To gain further insights into this mechanism, we verified if the attentional dynamics become lessentangled over learning.
Thus, we visualized how the attention dynamics evolves throughout thetraining (Figure 13).
While at an early stage the characteristic rotational dynamics is absent, itemerges as the training proceeds, progressively untangling the trajectories.
[Table]
±1002 ±
Hopper σ = 0 .961 ±1102 ±675 ±1161 ±1070 ±1036 ±
{+Hopper σ = 0 .730 ±953 ±
+}
±1017 ±919 ±863 ±
Hopper σ = 0 .428 ±591 ±342 ±643 ±558 ±576 ±
I n c r e a s i n g c
o
m p l e x i t y σ = 0 .σ
= 0 .
Ant 0.
Half Cheetah 0.
Walker 0.
Hopper
Perturbation intensity
N o r m a li z e d e p i s o d e r e w a r d
DMAP
TCN
Ant train =0.
Ant train =0.
Ant train =0.
Half Cheetah train =0. Half Cheetah train =0.
Half Cheetah train =0.
Walker train =0.
Walker train =0. Walker train =0.
Hopper train =0. Hopper train =0.
[Figure]
N o r m a li z e d e p i s o d e r e w a r d
DMAP
TCN
[Figure]
Walker
[Figure]