N ERVE N ET : L EARNING S TRUCTURED P OLICY WITH G RAPH N EURAL N ETWORKS
We address the problem of learning structured policies for continuous control.
In traditional reinforcement learning, policies of agents are learned by multi-layer perceptrons (MLPs) which take the concatenation of all observations from the environment as input for predicting actions.
In this work, we propose NerveNet to explicitly model the structure of an agent, which naturally takes the form of a graph.
Speciﬁcally, serving as the agent’s policy network, NerveNet ﬁrst propagates information over the structure of the agent and then predict actions for different parts of the agent.
In the experiments, we ﬁrst show that our NerveNet is comparable to state-of-the-art methods on standard MuJoCo environments.
We further propose our customized reinforcement learning environments for benchmarking two types of structure transfer learning tasks, i.e., size and disability transfer , as well as multi-task learning .
We demonstrate that policies learned by NerveNet are signiﬁcantly more transferable and generalizable than policies learned by other models and are able to transfer even in a zero-shot setting.
I NTRODUCTION
Deep reinforcement learning (RL) has received increasing attention over the past few years, with the recent success of applications such as playing Atari Games, Mnih et al.
and Go, (Silver et al., 2016; 2017).
Signiﬁcant advances have also been made in robotics using the latest RL techniques, e.g., Levine et al.
; Gu et al.
Many RL problems feature agents with multiple dependent controllers.
For example, humanoid robots consist of multiple physically linked joints.
Action to be taken by each joint or the body should thus not only depend on its own observations but also on actions of other joints.
Previous approaches in RL typically use MLP to learn the agent’s policy.
In particular, MLP takes the concatenation of observations from the environment as input, which may be measurements like positions, velocities of body and joints in the current time instance.
The MLP policy then predicts actions to be taken by every joint and body.
Thus the task of the MLP policy is to discover the latent relationships between observations.
This typically leads to longer training times, requiring more exposure of the agent to the environment.
In our work, we aim to exploit the body structure of an agent, and physical dependencies that naturally exist in such agents.
We rely on the fact that bodies of most robots and animals have a discrete graph structure.
Nodes of the graph may represent the joints, and edges represent the (physical) dependencies between them.
In particular, we deﬁne the agent’s policy using a Graph Neural Network, Scarselli et al.
(2009), which is a neural network that operates over graph structures.
We refer to our model as NerveNet due to the resemblance of the neural nervous system to a graph.
NerveNet propagates information between different parts of the body based on the underlying graph structure before outputting the action for each part.
By doing so, NerveNet can leverage the structure information encoded by the agent’s body which is advantageous in learning the correct inductive bias, and thus is less prone to
[Figure]
overﬁtting.
Moreover, NerveNet is naturally suitable for structure transferring tasks as most of the model weights are shared across the nodes and edges, respectively.
We ﬁrst evaluate our NerveNet on standard RL benchmarks such as the OpenAI Gym, Brockman et al.
(2016) which stem from MuJoCo.
We show that our model achieves comparable results to state-of-the-art MLP based methods.
To verify our claim regarding the structure transfer, we further introduce our customized RL environments which are based on the ones of Gym.
Two types of structure transfer tasks are designed, size transfer and disability transfer .
In particular, size transfer focuses on the scenario in which policies are learned for small-sized agents (simpler body structure) and applied directly to large-sized agents which are composed by some repetitive components shared with the small-sized agent.
Secondly, disability transfer investigates scenarios in which policies are learned for one agent and applied to the same agent with some components disabled.
Our experiments demonstrate that for structure transfer tasks our NerveNet is signiﬁcantly better than all other competitors, and can even achieve zero-shot learning for some agents.
For the multi-task learning tasks, NerveNet is also able to learn policies that are more robust with better efﬁciency.
The main contribution of this paper is the following: We explore the problem of learning transferable and generalized features by incorporating a prior on the structure via graph neural networks.
NerveNet permits powerful transfer learning from one structure to another, which goes well beyond the ability of previous models.
NerveNet is also more robust and has more potential in performing multi-task learning.
The demo and code for this project are released, under the project page of http://www.cs.toronto.edu/˜tingwuwang/nervenet.html .
N ERVE N ET
In this section, we ﬁrst introduce the notation.
We then explain how to construct the graph for each of our agents, followed by the description of the NerveNet.
Finally, we describe the learning algorithm for our model.
We formulate the locomotion control problems as an inﬁnite-horizon discounted Markov decision process (MDP).
To fully describe the MDP for continuous control problems which include locomotion control, we deﬁne the state space or observation space as S and action space as A .
To interact with the environments, the agent generates its stochastic policy π θ ( a τ | s τ ) based on the current state s τ ∈ S , where a τ ∈ A is the action and θ are the parameters of the policy function.
The environment on the other hand, produces a reward r ( s τ , a τ ) for the agent, and the agent’s objective is to ﬁnd a policy that maximizes the expected reward.
Shin
Root
Hip
Toe
Shin
Hip
Toe
Tail
Neck
Head ...
Shin
Root
Hip
Toe
Shin
Hip
Toe
Tail
Neck
Head
Joint’s Velocity
Joint’s Angle Theta
Agent’s Positional Information ...
ObservationVector ...
InputModel
PropagationModel ...
OutputModel
Controller’sOutput Vector
StateUpdate
2. G RAPH C ONSTRUCTION
In real life, skeletons of most robots and animals have a discrete graph structure, and are most often trees.
Simulators such as the MuJoCo engine by Todorov et al.
(2012), organize the agents using an
XML-based kinematic tree.
In our experiments, we will use the tree graphs as per MuJoCo.
Note that our model can be directly applied to arbitrary graphs.
In particular, we assume two types of nodes in our tree: body and joint .
The body nodes are abstract nodes used to construct the kinematic tree via nesting, which is similar to the coordinate frame system used in robotics, Spong et al.
The joint node represents the degrees of freedom of motion between the two body nodes.
Take a simple humanoid as an example; the body nodes Thigh and Shin are connected via the Knee , where Knee is a hinge joint .
We further add a root node which observes additional information about the agent.
For example, in the Reacher agent in MuJoCo, the root node has access to the target position of the agent.
We build edges to form a tree graph.
Fig.
1 illustrates the graph structure of an example agent, CentipedeEight .
We sketch the agent and its corresponding graph in the left and right part of the ﬁgure, respectively.
Note that for better visualization, we omit the joint nodes and use edges to represent the physical connections of joint nodes.
Different elements of the agent are parsed into nodes with different colors.
Further details are provided in the experimental section.
2. N ERVE N ET AS P OLICY N ETWORK
We now turn to NerveNet which parametrizes the policy with a Graph Neural Network.
Before delving into details, we ﬁrst introduce our notation.
We then specify the input model which helps to initialize the hidden state of each node.
We further introduce the propagation model that updates these hidden states.
Finally, we describe the output model.
We denote the graph structure of the agent as G =
( V, E ) where V and E are the sets of nodes and edges, respectively.
We focus on the directed graphs as the undirected case can be easily addressed by splitting one undirected edge into two directed edges.
We denote the out-going neighborhood of node u as N out ( u ) which contains all endpoints v with ( u, v ) being an edge in the graph.
Similarly, we denote the in-coming neighborhood of node u as N in ( u ) .
Every node u has an associated node type p u ∈ { 1 , 2 , . .
. , P } , which in our case corresponds to body , joint and root .
We also associate each edge ( u, v ) with an edge type c ( u,v ) ∈ { 1 , 2 , . . .
, C } .
Node type can help in cap- turing different importances across nodes.
Edge type can be used to describe different relationships between nodes, and thus propagate information between them differently.
One can also add more than one edge type to the same edge which results in a multi-graph.
We stick to simple graphs for simplicity.
One interesting fact is that we have two notions of “time” in our model.
One is the time step in the environment which is the typical time coordinate for RL problems.
The other corresponds to the internal propagation step of NerveNet.
These two coordinates work as follows.
At each time step of the environment, NerveNet receives observation from the environment and performs a few internal propagation steps in order to decide on the action to be taken by each node.
To avoid confusion, throughout this paper, we use τ to describe the time step in the environment and t for the propagation step.
2.2. I NPUT M ODEL
For each time step τ in the environment, the agent receives an observation s τ ∈ S .
The observation vector s τ is the concatenation of observations of each node.
We denote the elements of observation vector s τ corresponding to node u with x u .
From now on, we drop the time step in the environment to derive the model for simplicity.
The observation vector goes through an input network to obtain a ﬁxed-size state vector as follows:
[Equation]
where the subscript and superscript denote the node index and propagation step, respectively.
Here,
F in may be a MLP and h 0 u is the state vector of node u at propagation step 0 .
Note that we may need to pad zeros to the observation vectors if different nodes have observations of different sizes.
2.2. P ROPAGATION M ODEL
We now describe the propagation model of our NerveNet which mimics a synchronous message passing system studied in distributed computing, Attiya & Welch (2004).
We will show how the state vector of each node is updated from one propagation step to the next.
This update process is recurrently applied during the whole propagation.
We leave the details to the appendix.
Message Computation In particular, at propagation step t , for every node u , we have access to a state vector h tu .
For every edge ( u, v ) ∈ N out ( u ) , node u computes a message vector as below,
[Equation]
where M c ( u,v ) is the message function which may be an identity mapping or a MLP.
Note that the subscript c ( u,v ) indicates that edges of the same edge type share the same instance of the message function.
For example, the second torso in Fig.
1 sends a message to the ﬁrst and third torso , as well as the LeftHip and RightHip .
Message Aggregation Once every node ﬁnishes computing messages, we aggregate messages sent from all in-coming neighbors of each node.
Speciﬁcally, for every node u , we perform the following aggregation:
[Equation]
where A is the aggregation function which may be a summation, average or max-pooling function.
Here, ¯ m tu is the aggregated message vector which contains the information sent from the node’s neighborhood.
States Update We now update every node’s state vector based on both the aggregated message and its current state vector.
In particular, for every node u , we perform the following update:
[Equation]
where U is the update function which may be a gated recurrent unit (GRU), a long short term memory (LSTM) unit or a MLP.
From the subscript p u of U , we can see that nodes of the same node type share the same instance of the update function.
The above propagation model is then recurrently applied for a ﬁxed number of time steps T to get the ﬁnal state vectors of all nodes, i.e., { h T u | u ∈ V } .
2.2. O UTPUT M ODEL
In RL, agents typically use a MLP policy, where the network outputs the mean of the Gaussian distribution for each of the actions, while the standard deviation is a trainable vector, Schulman et al.
In our output model, we also treat standard deviation in the same way.
However, instead of predicting the action distribution of all nodes by a single network, we make predictions for each individual node.
We denote the set of nodes which are assigned controllers for the actuators as O .
For each such node, a MLP takes its ﬁnal state vectors h Tu ∈O as input and produces the mean of the action of the Gaussian policy for the corresponding actuator.
For each output node u ∈ O , we deﬁne its output type as q u .
Different sharing schemes are available for the instance of MLP O q u , for example, we can force the nodes with similar physical structure to share the instance of MLP.
For example, in Fig. 1, two LeftHip nodes have a shared controller.
Therefore, we have the following output model:
[Equation]
) where µ u ∈O is the mean value for action applied on each actuator.
In practice, we found that we can force controllers of different output types to share one uniﬁed controller, while not hurting the performance.
By integrating the produced Gaussian policy for each action, the probability density of the stochastic policy is calculated as
[Equation]
[Equation]
where a τ ∈ A is the output action, and σ u is the variable standard deviation for each action.
Here, θ represents the parameters of the policy function.
2. L EARNING A LGORITHM
To interact with the environments, the agent generates its stochastic policy π θ ( a τ | s τ ) after several propagation steps.
The environment on the other hand, produces a reward r ( s τ , a τ ) for the agent, and transits to the next state with transition probability P ( s τ +1 | s τ ) .
The target of the agent is to maximize its cumulative return
[Equation]
To optimize the expected reward, we use the proximal policy optimization (PPO) by Schulman et al.
In PPO, the agents alternate between sampling trajectories with the latest policy and performing optimization on surrogate objective using the sampled trajectories.
The algorithm tries to keep the KL-divergence of the new policy and the old policy within the trust region.
To achieve that,
PPO clips the probability ratio and adds an KL-divergence penalty term to the loss.
The likelihood ratio is deﬁned as r τ ( θ ; θ old ) =
π θ ( a τ | s τ ) /π
θ old ( a τ | s τ ) .
Following the notation and the algorithm of PPO, our NerveNet tries to minimize the summation of the original loss in Eq.
KL-penalty and the value function loss which is deﬁned as:
[Equation]
where ˆ A t is the generalized advantage estimation (GAE) calculated using algorithm from Schulman et al.
(2015b), and (cid:15) is the clip value, which we choose to be 0 .
Here, β is a dynamical coefﬁcient adjusted to keep the KL-divergence constraints, and α is used to balance the value loss.
Note that in Eq.
(8), V ( s t ) target is the target state value in accordance with the GAE method.
To optimize the ˜ J ( θ ) , PPO make use of the policy gradient in Sutton et al.
(2000) to do ﬁrst-order gradient descent optimization.
Value Network To produce the state value V θ
( s τ ) for given observation s τ , we have several alternatives: (1) using one GNN as the policy network and using one MLP as the value network (NerveNet-MLP); (2) using one GNN as policy network and using another GNN as value network (NerveNet-2) (without sharing the parameters of the two GNNs); (3) using one GNN as both policy network and value network (NerveNet-1).
The GNN for value network is very similar to the GNN for policy network.
The output for value GNN is a scalar instead of a vector of mean action.
We will compare these variants in the experimental section.
R ELATED W ORK
Reinforcement Learning Reinforcement learning (RL) has recently achieved huge success in a variety of applications.
Powered by the progress of deep neural networks, Krizhevsky et al.
(2012), agents are now able to successfully play Atari Games and beat the world’s best (human) players in the game of Go (Mnih et al., 2015; Silver et al., 2016; 2017).
Based on simulation engines like
MuJoCo, Todorov et al.
(2012), numerous algorithms have been proposed to train agents also in continuous control problems (Schulman et al., 2017; 2015a;
Heess et al., 2017; Metz et al., 2017).
Structure in RL Most approaches that exploit priors on structure of the problem fall in the domain of hierarchical RL, (Kulkarni et al., 2016; Vezhnevets et al., 2017), which mainly focus on modeling intrinsic motivation of agents.
In Hausknecht & Stone (2015), the authors extend the deep
RL algorithms to MDPS with parameterized action space by exploiting the structure of action space and bounding the action space gradients.
Graphs have been used in RL problems prior to our work.
In Metzen (2013)
; Mabu et al.
; Shoeleh & Asadpour (2017); Mahadevan & Maggioni (2007), the authors use graphs to learn a representation of the environment.
However, these methods are limited to problems with simple dynamical models like for example the task of 2d-navigation, and thus these problems are usually solved via model-based RL.
However, for complex multi-joint agents, learning the dynamical model as well as predicting the transition of states is time consuming and biased.
For problems of training model-free multi-joint agents in complex physical environments, relatively little attention has been devoted to modeling the physical structure of the agents.
Graph Neural Networks There have been many efforts to generalize neural networks to graphstructured data.
One line of work is based on convolutional neural networks (CNNs).
In (Bruna et al., 2014; Defferrard et al., 2016; Kipf & Welling, 2017), CNNs are employed in the spectral domain relying on the graph Laplacian matrix.
(Goller & Kuchler, 1996; Duvenaud et al., 2015) used hash functions in order to apply CNNs to graphs.
Another popular direction is based on recurrent neural networks (RNNs) (Goller & Kuchler, 1996;
Gori et al., 2005; Scarselli et al., 2009; Socher et al., 2011;
Li et al., 2015; Tai et al., 2015).
Among RNN based methods, many are only applicable to special structured graph, e.g., sequences or trees, (Socher et al., 2011; Tai et al., 2015).
One class of models which are applicable to general graphs are so-called graph neural networks (GNNs), Scarselli et al.
The inference procedure is a forward pass that exploits a ﬁxed-length propagation process which resembles synchronous message passing system in the theory of distributed computing, Attiya & Welch (2004).
Nodes in the graph have state vectors which are recurrently updated based on their history and received messages.
One of the representative work of GNNs, i.e., gated graph neural networks (GGNNs) by Li et al.
(2015), uses gated recurrent unit to update the state vectors.
Learning such a model can be achieved by the back-propagation through time (BPTT) algorithm or recurrent back-propagation, Chauvin &
Rumelhart (1995).
It has been shown that GNNs, (Li et al., 2015; 2017; Qi et al., 2017; Gilmer et al., 2017) have a high capacity and achieve state-of-the-art performance in many applications which involve graph-structured data.
In this paper, we model the structure of the reinforcement learning agents using GNNs.
Transfer and Multi-task Learning in RL Recently, there has been increased interest in transfer learning tasks for RL, Taylor & Stone (2009), which mainly focus on transferring the policy learned from one environment to another.
In Rajeswaran et al.
(2017b;a), the authors show that agents in reinforcement learning are prone to over-ﬁtting, and that the learned policies generalize poorly across environments.
In model-based RL, traditional control has been well studied for generalization properties, Coros et al.
Gupta et al.
(2017) try to increase the transferability via learning invariant visual features.
Efforts have also been made from the meta-learning perspective (Duan et al., 2016; Finn et al., 2017b;a).
In Wulfmeier et al.
(2017), the authors propose a method of transfer learning by using imitation learning.
Transferability comes naturally in our model by exploiting the (shared) graph structure of the agents.
Multi-task learning has also received a lot of attention, Wilson et al.
In Teh et al.
(2017), the authors use a distilled policy that captures common behaviour across tasks.
Wilson et al.
Oh et al.
; Andreas et al.
(2016) use a hierarchical approach, where multiple sub-policies are learnt.
In Yang et al.
; Parisotto et al.
, the authors exploited shared visual features
[Figure]
MLP, TreeNet and NerveNet on 8 continuous control benchmarks from the
Gym. to tackle multi-task learning.
In Ammar et al.
(2014), a multi-task policy gradient is proposed, while, Calandriello et al.
(2014) propose multi-task extensions of the ﬁtted Q-iteration algorithm.
Successor features in Barreto et al.
(2017) are used to boost the performance of multiple tasks.
Unlike ours, these methods do not exploit the physical graph structures of the agents.
E XPERIMENTS
In this section, we ﬁrst verify the effectiveness of NerveNet on standard MuJoCo environments in OpenAI Gym.
We then investigate the transfer abilities of NerveNet and other competitors by customizing some of those environments, as well as the multi-task learning ability and robustness.
4. C OMPARISON ON S TANDARD B ENCHMARKS OF M U J O C O
Baselines We compare NerveNet with the standard MLP models utilized by Schulman et al.
(2017) and another baseline which is constructed as follows.
We ﬁrst remove the physical graph structure and introduce an additional super node which connects to all nodes in the graph.
This results in a singly rooted depth- 1 tree.
We refer to this baseline as TreeNet.
The propagation model of TreeNet is similar to NerveNet where, however, the policy ﬁrst aggregates the information from all children and then feeds the state vector of the root to the output model.
This simpler model serves as a baseline to verify the importance of the graph structure.
We run experiments on 8 simulated continuous control benchmarks from the Gym, Brockman et al.
(2016), which is based on MuJoCo, Todorov et al.
In particular, we use
Reacher , InvertedPendulum , InvertedDoublePendulum , Swimmer , and four walking or running tasks: HalfCheetah , Hopper , Walker2d , Ant .
We set the maximum number of training steps to be 1 million for all environments as it is enough to solve them.
Note that for InvertedPendulum , different from the original one in Gym, we add the distance penalty of the cart and velocity penalty so that the reward is more consistent to the
InvertedDoublePendulum .
This change of design also makes the task more challenging.
[Table]
Results We do grid search to ﬁnd the best hyperparameters and leave the details in the Appendix 6.3.
As the randomness might have a big impact on the performance, for each environment, we run 3 experiments with different random seeds and plot the average curves and the standard deviations.
We show the results in Figure 3.
From the ﬁgures, we can see that MLP with the same setup as in Schulman et al.
(2017) works the best in most of tasks.
1 NerveNet basically matches the performance of MLP in terms of sample efﬁciency as well as the performance after it converges.
In most cases, the TreeNet is worse than NerveNet which highlights the importance of keeping the physical graph structure.
4. S TRUCTURE T RANSFER L EARNING
We now benchmark our model in the task of structure transfer learning by creating customized environments based on the existing ones from MuJoCo.
We mainly investigate two types of structure transfer learning tasks.
The ﬁrst one is to train a model with an agent of small size (small graph) and apply the learned model to an agent with a larger size, i.e., size transfer .
When increasing the size of the agent, observation and action space also increase which makes learning more challenging.
Another type of structure transfer learning is disability transfer where we ﬁrst learn a model for the original agent and then apply it to the same agent with some components disabled.
If one model overﬁts the environment, disabling some components of the agent might bring catastrophic performance degradation.
Note that for both transfer tasks, all factors of environments do not change except the structure of the agent.
Centipede We create the ﬁrst environment in which the agent has a similar structure to a centipede.
The goal of the agent is to run as fast as possible along the y -direction in the MuJoCo environment.
The agent consists of repetitive torso bodies where each one has two legs attached.
For two consecutive bodies, we add two actuators which control the rotation between them.
Furthermore, each leg consists of a thigh and shin , which are controlled by two hinge actuators.
By linking copies of torso bodies and corresponding legs, we create agents with different lengths.
Speciﬁcally, the shortest Centipede is CentipedeFour and the longest one is CentipedeFourty due to the limit of supported resource of MuJoCo.
For each time step, the total reward is the speed reward minus the energy cost and force feedback from the ground.
Note that in practice, we found that training a CentipedeEight from scratch is already very difﬁcult.
For size transfer experiments, we create many instances which are listed in Figure 4, like “4to06”, “6to10”.
For disability transfer , we create CrippleCentipede agents of which two back legs are disabled.
In Figure 4,
CrippleCentipede is speciﬁed as “Cp”.
Snakes We also create a snake-like agent which is common in robotics, Crespi & Ijspeert (2008).
We design the Snake environment based on the Swimmer model in Gym.
The goal of the agent is to move as fast as possible.
For details of the environment, please see the schematic ﬁgure 16.
4.2. E XPERIMENTAL S ETTINGS
To fully investigate the performance of NerveNet, we build several baseline models for structure transfer learning which are explained below.
NerveNet For the NerveNet, since all the weights are exactly the same for the small and the largeagent models, we directly use the old weights trained on the small-agent model.
When the large obtained better performances
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure] [Figure]
agent has repetitive structure, we further re-use the weights of the corresponding joints from the small-agent model.
MLP Pre-trained (MLPP)
For the MLP based model, while transferring from one structure to another, the size of the input layer changes since the size of the observation changes.
One straight-
[Figure] [Figure]
forward idea is to reuse the weights from the ﬁrst hidden layer to the output layer and randomly initialize the weights of the new input layer.
MLP Activation Assigning (MLPAA)
Another way of making MLP transferable is assigning the weights of the small-agent model to the corresponding partial weights of the large-agent model and setting the remaining weights to be zero.
Note that we do not add or remove any layers from the small-agent model to the large-agent except for changing the size of the layers.
By doing so, we can keep the output of the large-agent model to be same as the small-agent in the beginning,
i
[Figure]
e., keeping the same initial policy.
TreeNet TreeNet is similar as the model described before.
We apply the same way of assigning weights as MLPAA to TreeNet for the transfer learning task.
Random We also include the random policy which is uniformly sampled from the action space.
4.2. R ESULTS
Centipedes For the Centipedes environment, we ﬁrst run experiments of all models on
CentipedeSix and CentipedeFour to get the pre-trained models for transfer learning.
We train different models until these agents run equally well as possible, which is reported in Table 1.
Note that, in practice, we train TreeNet on CentipedeFour for more than 8 million time steps.
However, due to the difﬁculty of optimizing TreeNet on CentipedeFour , the performance is still lower.
But visually, the TreeNet agent is able to run in CentipedeFour .
We then examine the zero-shot performance where zero-shot means directly applying the model trained with one setting to the other without any ﬁne-tuning.
To better visualize the results, we linearly normalize the performance to get a performance score, and color the results accordingly.
The normalization scheme is recorded in Appendix 11.
The performance score is less than 1, and is shown in the parentheses behind the original results.
As we can see from Figure 4 (full chart in
Appendix 6.5), NerveNet outperforms all competitors on all settings, except in the 4toCp06 scenario.
Note that transferring from CentipedeFour is more difﬁcult than from CentipedeSix since the situation where one torso connects to two neighboring torsos only happens beyond 4 bodies.
TreeNet has a surprisingly good performance on tasks from CentipedeFour .
However, by checking the videos, the learned agent is actually not able to “move” as good as other methods.
The high reward is mainly due to the fact that TreeNet policy is better at standing still and gaining alive bonus.
We argue that the average running-length in each episode is also a very important metric.
By including the results of running-length, we notice that NerveNet is the only model able to walk in the zero-shot setting.
In fact, the performance of NerveNet is orders-of-magnitude better, and most of the time, agents from other methods cannot even move forward.
We also notice that if transferred from CentipedeSix , NerveNet is able to provide walkable pre-trained models on all new agents.
We ﬁne-tune for both size transfer and disability transfer experiments and show the training curves in Figure 5.
From the ﬁgure, we can see that by using the pre-trained model, NerveNet signiﬁcantly decreases the number of episodes required to reach the level of reward which is considered as solved.
By looking at the videos, we notice that the bottleneck of learning for the agent is “how to stand”.
When training from scratch, it can be seen that almost 0 .
5 million time steps are spent on a very ﬂat reward surface.
Therefore, the MLPAA agents, which copy the learned policy, are able to stand and bypass this time-consuming process and reach to a good performance in the end.
Moreover, by examining the result videos, we noticed that the “walk-cycle” behavior is observed for NerveNet but is not common for others.
Walk-cycle are adopted for many insects in the world, Biewener (2003).
For example, six-legged ants use a tripedal gait, where the legs are used in two separate triangles alternatively touching the ground.
We give more details of walk-cycle in
Section 4.5.
One possible reason is that the agent of MLP based method (MLPAA, MLPP) learns a policy that does not utilize all legs.
From CentipedeEight and up, we do not observe any MLP agents to be able to coordinate all legs whereas almost all policies learned by NerveNet use all legs.
Therefore,
NerveNet is better at utilizing structure information and not over-ﬁtting the environments.
Snakes The zero-shot performance for snakes is summarized in Figure 6.
As we can see, NerveNet has the best performance on all transfer learning tasks.
In most cases, NerveNet has a starting reward value of more than 300 , which is a pretty good policy since 350 is considered as solved for snakeThree .
By looking at the videos, we found that agents of other competitors are not able to control the new actuators in the zero-shot setting.
They either overﬁt to the original models, where the policy is completely useless in the new setting (e.g., the MLPAA is worse than random policy in
SnakeThree2SnakeFour ), or the new actuators are not able to coordinate with the old actuators trained before.
While for NerveNet, the actuators are able to coordinate to its neighbors, regardless of whether they are new to the agents.
We also summarize the training curves of ﬁne-tuning in Fig.
We can observe that NerveNet has a very good initialization with the pre-trained model, and the performance increases with ﬁne-tuning.
When training from scratch, NerveNet is less sample efﬁcient compared to the MLP model which might be caused by the fact that optimizing our model is more challenging than MLP.
Fine-tuning helps to improve the sample efﬁciency of our model by a large margin.
At the same time, although
[Figure]
Model
[Table]
Wolf
Horse
[Table]
[Figure]
[Table]
the MLPAA has a very good initialization, its performance progresses slowly with the increasing number of episodes.
In most experiments, the MLPAA and TreeNet did not match the performance of its non-pretrained MLP baseline.
4. M ULTI - TASK L EARNING
In this section, we show that NerveNet has a good potential of multi-task learning by incorporating structure prior into the network structure.
It is important to point out that multi-task learning represents a very difﬁcult, and more often the case, unsolved problem in RL.
Most multi-task learning algorithms, Teh et al.
; Andreas et al.
Oh et al.
; Yang et al.
(2017) have not been applied to domains as difﬁcult as locomotion for complex physical models, not to mention multi-task learning among different agents with different dynamics.
In this work, we constrain our problem domain, and design the Walker multi-task learning taskset, which contains ﬁve 2d-walkers.
We aim to test the model’s ability of multi-task learning, in particular, the ability to control multiple agents using one uniﬁed network.
The walkers are very different in terms of their dynamics, since they have very distinct structures, different types and numbers of controllers.
Walker-HalfHumanoid and Walker-Hopper are variants of
Walker2d and Hopper from the original MuJoCo Benchmarks, respectively.
Walker-Horse ,
Walker-Ostrich , Walker-Wolf on the other hand, are agents mimicking the natural animals.
Just like the real animals, some of the agents have tails or necks to help them to balance.
The detailed schematic ﬁgures are shown in the Appendix 6.8.
[Table]
E XPERIMENTAL S ETTINGS
To show the ability of multi-task learning of NerveNet, we design several baselines.
We use a vanilla multi-task policy update for all models.
More speciﬁcally, for each sub-task in the multi-task learning task-set, we use an equal number of time steps for each policy’s update and calculate the gradients separately.
Gradients are then aggregated and the mean value of gradients is applied to update the network.
To compensate for the additional difﬁculty in training more agents and tasks, we linearly increase the number of update epochs during each update in training, as well as the total number of time steps generated before the training is terminated.
The hyper-parameter setting is summarized in Appendix 6.7.
NerveNet For NerveNet, the weights are naturally shared among different agents.
More speciﬁcally, for different agents, the weight matrices for propagation and output are shared.
MLP Sharing For the MLP method, we shared the weight matrices between hidden layers.
MLP Aggregation
In the MLP Sharing approach, the total size of the weight matrices grows with the number of tasks.
For different agents, whose dimension of the observations are usually different, weights from observation to the ﬁrst hidden layer cannot be reused in the MLP Sharing approach.
Therefore in the MLP Aggregation method, we multiply each element of the observation vector separately by one matrix, and aggregate the resulting vectors from each element.
The size of this multiplying matrix is (1, dimension of the ﬁrst hidden layer).
TreeNet Similarly, TreeNet also has the beneﬁts that its weights are naturally shared among different agents.
However, TreeNet has no knowledge of the agents’ physical structure, where the information of each node is aggregated into the root node.
We also include the baseline of training single-task MLP for each agent.
We train the single-task
MLP baselines for 1 million time steps per agent.
In the Figure 8, we align the results of single-task
MLP baseline and the results of multi-task models by the number of episodes of one task.
As can be seen from Figure 8, NerveNet achieves the best performance in all the sub-tasks.
In
Walker-HalfHumanoid , Walker-Hopper , Walker-Ostrich , Walker-Wolf our NerveNet is able to out-perform other agents by a large margin.
In Walker-Horse , the performance of NerveNet and MLP Sharing are relatively similar.
For MLP Sharing, the performance on other four agents are relatively limited, while for Walker-Hopper , the improvement of performance is limited from half of the experiment onwards.
The MLP Aggregation and TreeNet methods are not able to solve the multi-task learning problem, with both of them stuck at a very low reward level.
In the vanilla optimization setting, we show that NerveNet has a bigger potential than the baselines.
From Table 2, one can observe that the performance of MLP drops drastically (42% performance drop) when switching from single-task to multi-task learning, while for NerveNet, there is no obvious drop in performance.
Our intuition is that NerveNet is better at learning generalized features, and learning of different agents can help in training other agents, while for MLP methods, the performance decreases due the competition of different agents.
[Figure]
[Figure]
of feature distribution and trajectory density.
As can be seen from the ﬁgure, NerveNet agent is able to learn shareable features for its legs, and certain walk-cycle is learnt during training.
4. R OBUSTNESS OF L EARNT P OLICIES
In this section, we also report the robustness of our policy by perturbing the agent parameters.
In reality, the parameters simulated might be different from the actual parameters of the agents.
Therefore, it is important that the agent is robust to parameters perturbation.
The model that has the better ability to learn generalized features are likely more robust.
We perturb the mass of the geometries (rigid bodies) in MuJoCo as well as the scale of the forces of the joints.
We use the pre-trained models with similar performance on the original task for both the
MLP and NerveNet.
The performance is tested in ﬁve agents from Walker task set.
The average performance is recorded in Table 3, and the speciﬁc details are summarized in Appendix 6.6.
The robustness of NerveNet’ policy is likely due to the structure prior of the agent instilled in the network, which facilitates overﬁtting.
[Figure]
4. I NTERPRETING THE L EARNED R EPRESENTATIONS
In this section, we try to visualize and interpret the learned representations.
We extract the ﬁnal state vectors of nodes of NerveNet trained on CentipedeEight .
We then apply 1- D and 2- D
PCA on the node representations.
In Figure 10, we notice that each pair of legs is able to learn invariant representations, despite their different position in the agent.
We further plot the trajectory density map in the feature map.
By recording the period of the walk-cycle, we plot the transformed features of the 6 legs on Figure 9.
As we can see, there is a clear periodic behavior of our hidden representations learned by our model.
Furthermore, the representations of adjacent left legs and the adjacent right legs demonstrate a phase shift, which further proves that our agents are able to learn the walk-cycle without any additional supervision.
4. C OMPARISON OF M ODEL V ARIANTS
We have several variants of NerveNet, based on the type of network we use for the policy/value representation.
We here compare all variants.
Again, we run experiments for each task three times.
The details of hyper-parameters are given in the Appendix.
For each environment, we train the network for one million time steps, with batch size 2050 for one update.
As we can see from Figure 11, the NerveNet-MLP and NerveNet-2 variants perform better than
NerveNet-1.
One potential reason is that sharing the weights of the value and policy networks makes the trust-region based optimization methods, like PPO, more sensitive to the weight α of the value function in equation 8.
Based on the ﬁgure, choosing α to be 1 is not giving good performance on the tasks we experimented on.
C ONCLUSION
In this paper, we aimed to exploit the body structure of Reinforcement Learning agents in the form of graphs.
We introduced a novel model called NerveNet which uses a Graph Neural Network to represent the agent’s policy.
At each time instance of the environment, NerveNet takes observations for each of the body joints, and propagates information between them using non-linear messages computed with a neural network.
Propagation is done through the edges which represent natural dependencies between joints, such as physical connectivity.
We experimentally showed that our
NerveNet achieves comparable performance to state-of-the-art methods on standard MuJoCo environments.
We further propose our customized reinforcement learning environments for benchmarking two types of structure transfer learning tasks, i.e., size and disability transfer .
We demonstrate that policies learned by NerveNet are signiﬁcantly better than policies learned by other models and are able to transfer even in a zero-shot setting.
A PPENDIX 6. D ETAILS OF N ERVE N ET
We use MLP to compute the messages which uses tanh nonlinearities as the activation function.
We do a grid search on the size of the MLP to compute the messages, the details of which are listed in Table 5, 4.
Throughout all of our experiments, we use average aggregation and GRU as the update function.
[Table] [Table]
G RAPH OF A GENT
In MuJoCo, we observe that most body nodes are paired with one and only one joint node.
Thus, we simply merge the two paired nodes into one.
We point out that this model is very compact, and is the standard graph we use in our experiments.
In the Gym environments, observation for the joint nodes normally includes the angular velocity, twist angle and optionally the torque for the hinge joint , and position information for the positional joint .
For the body nodes, velocity, inertia, and force are common observations.
For example in the centipede environment 1, the LeftHip node will receive the angular velocity (cid:36) j , and the twist angle θ j .
6. H YPERPARAMETER S EARCH
For MLP, we run grid search with the hidden size from two layers to three layers, and with hidden size from 32 to 256.
For NerveNet, to reduce the time spent on grid search, we constrain the propagation network and output network to be the same shape.
Similarly, we run grid search with the network’s hidden size, and at the same time, we run a grid search on the size of node’s hidden states from 32 to 64.
For the TreeNet, we run similar grid search on the node’s hidden states and output network’s shape.
For details of hyperparameter search, please see Table 5, 7, 6.
[Table] [Table]
NerveNet
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Use
[Table]
Penalty
[Table]
NerveNet Variants
NerveNet-1, NerveNet-2, NerveNet-MLP
Size of
[Table] [Table]
[Table]
Merge joint and body Node
Yes
[Table]
No
[Table]
Shared, Separate
Disable Edge Type
Yes
[Table]
No
Add Skip-connection from / to root
Yes
[Table]
No
6. S CHEMATIC F IGURES OF THE P ARSED A GENTS
In this section, we also plot the schematic ﬁgures of the agents for readers’ reference.
Graph structures are automatically parsed from the MuJoCo XML conﬁguration ﬁles.
[Figure] [Figure]
[Figure] [Figure]
[Figure]
T a b l e 8 : R e s u lt
s o f t
h e ze r os ho t l ea r n i ng o f
Centipede
[Table]
M
[Table]
p e d e F ou r 2 C e n ti p e d e T w e l v e -
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
C e n ti p e d e F ou r 2 C e n ti p e d e F ou r t ee n -
[Table]
[Table] [Table]
[Table]
C e n ti p e d e F ou r 2 C e n ti p e d e T w e n t y
[Table]
C e n ti p e d e F ou r 2 C e n ti p e d e T h
i r t y
[Table]
[Table] [Table]
[Table]
C e n ti p e d e F ou r 2 C e n ti p e d e F o r t y - 0 .
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
C e n ti p e d e S
i x2 C e n ti p e d e E
i gh t - 10 .
[Table] [Table]
[Table] [Table]
[Table]
C e n ti p e d e S
i x2 C e n ti p e d e F o r t y
[Table] [Table]
[Table] [Table]
[Table]
C e n ti p e d e F ou r 2 C p
C e n ti p e d e S
i x
[Table]
C e n ti p e d e F
[Table]
r 2 C p
C e n ti p e d e E i gh
[Table]
C e n ti p e d e F
[Table]
r 2 C p
C e n ti p e d e T e n
[Table]
C e n ti p e d e F ou r 2 C p
C e n ti p e d e T w e l v e - 6 .
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
C e n ti p e d e F ou r 2 C p
C e n ti p e d e F ou r t ee n - 11 .2 5 .- 24 .
C e n ti p e d e S
i x2 C p
C e n ti p e d e E
i gh t 0 .
C e n ti p e d e S
i x2 C p
C e n ti p e d e T e n - 6 .5 5 .- 29 .389 .1904 .
C e n ti p e d e S
i x2 C p
C e n ti p e d e T w e l v e - 6 .4 9 .- 34 .343 .1429 .
C e n ti p e d e S
i x2 C p
C e n ti p e d e F ou r t ee n - 11 .4- 27 .
D i s t a n ce M a x
D i s t a n ce S t d
D i s t a n ce A v e r a g e
T a s k
R a ndo m
M L P
AA
M L PP
T r ee N e t
N e r v e
N e t
R a ndo m
M L P
AA
M L PP
T r ee N e t
N e r v e
N e t
R a ndo m
M L P
AA
M L PP
T r ee N e t
N e r v e
N e t
C e n ti p e d e F ou r 2 C e n ti p e d e S
i x 63 .2082 .40 .
C e n ti p e d e F ou r 2 C e n ti p e d e E i gh t 70 .5 2 8 .77 .- 47 .676 .60 .
C e n ti p e d e F ou r 2 C e n ti p e d e T e n 29 .2 8 3 .104 .79 .580 .51 .
C e n ti p e d e F ou r 2 C e n ti p e d e T w e l v e 31 .
C e n ti p e d e F ou r 2 C e n ti p e d e F ou r t ee n 20 .1 8 8 .131 .
[Table]
[Table]
C e n ti p e d e F ou r 2 C e n ti p e d e T w e n t y
[Table]
[Table]
C e n ti p e d e
[Table]
r 2 C e n ti p e d e T
[Table]
i r t y
[Table]
C e n ti p e d e F ou r 2 C e n ti p e d e F o r t y
[Table]
[Table] [Table]
[Table] [Table]
C e n ti p e d e S
i x2 C e n ti p e d e E
i gh
t
[Table] [Table]
[Table] [Table]
C e n ti p e d e S
i x2 C e n ti p e d e T e n
[Table]
[Table] [Table]
[Table] [Table]
[Table]
C e n ti p e d e S
i x2 C e n ti p e d e T w e l v e 31 .1163 .-
[Table] [Table]
[Table] [Table]
C e n ti p e d e S
i x2 C e n ti p e d e F ou r t ee n 20 .1 2 1 .25 .-
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
C e n ti p e d e S
i x2 C e n ti p e d e T w e n t y 59 .1 3 0 .
[Table] [Table]
[Table] [Table]
[Table]
C e n ti p e d e S
i x2 C e n ti p e d e T h
i r t y - 7 .5 7 .- 38 .-
[Table] [Table]
[Table] [Table]
[Table] [Table]
C e n ti p e d e S
i x2 C e n ti p e d e F o r t y 166 .2158 .
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
C e n ti p e d e F ou r 2 C p
C e n ti p e d e S
i x 61 .1 2
C e n ti p e d e F ou r 2 C p
C e n ti p e d e E
i gh t 27 .4 9 .35 .-
[Table]
[Table] [Table]
[Table] [Table]
[Table]
C e n ti p e d e F ou r 2 C p
C e n ti p e d e T e n 18 .6 1 . - 53 .
[Table] [Table]
[Table]
C e n ti p e d e F ou r 2 C p
C e n ti p e d e T w e l v e 1 .1 5 .
[Table]
[Table]
C e n ti p e d e F ou r 2 C p
C e n ti p e d e F ou r t ee n - 1 .2 3 .43 .-
C e n ti p e d e S
i x2 C p
C e n ti p e d e E
i gh t 27 .7 8 8 .66 .- 23 .
[Table] [Table]
[Table] [Table]
[Table]
C e n ti p e d e S
i x2 C p
C e n ti p e d e T e n 18 .216 .
[Table] [Table]
[Table] [Table]
C e n ti p e d e S
i x2 C p
C e n ti p e d e T w e l v e 1 .2 0 9 .97 .
[Table] [Table]
[Table] [Table]
[Table] [Table]
C e n ti p e d e S
i x2 C p
C e n ti p e d e F ou r t ee n - 1 .1 4 8 .22 .
[Table] [Table]
[Table] [Table]
9 : R
[Table]
u
[Table]
ze
[Table]
o
[Table]
ho
[Table]
ea
[Table]
n
[Table]
ng
[Table]
Centiped
[Table]
ng
[Table]
nd
[Table]
hod
[Table]
r 2 C e n ti p e d e T h
i r t y
[Table] [Table]
[Table]
C e n ti p e d e F ou r 2 C e n ti p e d e F o r t y 2 .38 .127 .
[Table]
[Table] [Table]
C e n ti p e d e S
i x2 C e n ti p e d e E
i gh
t
[Table] [Table]
[Table] [Table]
[Table] [Table]
C e n ti p e d e S
i x2 C e n ti p e d e T e n 1 .
[Table]
C e n ti p e d e S
i x2 C e n ti p e d e T w e l v e
[Table]
[Table] [Table]
C e n ti p e d e S
i x2 C e n ti p e d e F ou r t ee n 47 .
[Table] [Table]
[Table] [Table]
C e n ti p e d e S
i x2 C e n ti p e d e T w e n t y 1 .38 .-
[Table]
C e n ti p e d e S
i x2 C e n ti p e d e T h
i r t y
[Table] [Table]
C e n ti p e d e S
i x2 C e n ti p e d e F o r t y 2 .37 .
C e n ti p e d e F ou r 2 C p
C e n ti p e d e
[Table]
In the MLP-Bind method, we bind the weights of MLP.
By doing this, the weights of the agent from the similar structures will be shared.
For example, in the centipede environment, the weights from observation to action of all the LeftHips are constrained to be same.
[Table] [Table]
[Table] [Table]
[Table]
As the scale of zero-shot results is very different, we normalize the results across different models for each transfer learning task.
For each task, we record the worst value of results from different models and the pre-set worst value V min.
we set the normalization minimun
value
as this worst value.
We calculate the normalization maximum value by (cid:98) max( V ) /
IntLen (cid:99) ∗ IntLen.
[Table]
D ETAILS
R OBUSTNESS RESULTS
[Table]
6. H YPERPARAMETERS FOR M ULTI - TASK L EARNING
[Table]
6. S CHEMATIC F IGURES OF W ALKERS A GENTS
We design Walker task-set, which contains ﬁve 2d walkers in the MuJoCo engine.
[Figure]
Root
Hip
Toe
Shin
Hip
Toe
Shin
Tail
Neck
Head
Walker-Wolf
Root
Thigh
Leg
Foot
Thigh
Leg
Foot
Thigh
Leg
Foot
Thigh
Leg Foot
Head
[Figure] [Figure]