N EURAL
G RAPH
E VOLUTION : A UTOMATIC
R OBOT
D ESIGN
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the recent success in robotic locomotion control, the design of robot relies heavily on human engineering.
Automatic robot design has been a long studied subject but with limited success due to the large combinatorial search space and the difﬁculty in evaluating the found candidates.
In this paper, We propose Neural
Graph Evolution (NGE) to address these two challenges.
We formulate automatic robot design as a graph search problem and perform evolution search in graph space.
NGE uses graph neural networks to parameterize the control policies that allows skill transfer from previously evaluated control policy to a new robot design.
The policy sharing and transfer greatly reducing the cost of re-training new candidates.
Similar to ES, NGE performs selection on current candidates and evolves new ones iteratively.
In addition, NGE applies Graph Mutation with Uncertainty by incorporating model uncertainty, which reduces the search space by balancing exploration and exploitation.
We show that NGE signiﬁcantly outperforms both random graph search (RGS) and ES by an order of magnitude.
As shown in experiments, NGE is the ﬁrst algorithm that can automatically discover complex robotic graph structures, such as a ﬁsh with two symmetrical ﬂat side-ﬁns and a tail, or a cheetah with athletic front and back legs.
Instead of using thousands of cores for weeks, NGE efﬁciently solves searching problem within a day on a single 64 CPU-core Amazon EC2 machine.
I NTRODUCTION
The goal of robot design is to search for robot body structures and their means of locomotion to best achieve a given objective.
Robot design often relies on careful human-engineering and expert knowledge.
Automatic robot design has been a long-studied subject but with limited success.
In (Sims, 1994), the authors evolved creatures with 3D-blocks.
Recently, soft robots are studied in (Joachimczak et al., 2014), which are evolved by adding small cells connected to the old ones, and also in (Cheney et al., 2014), where 3D voxels are treated as the minimum element of the robot.
Most evolutionary robots (Duff et al., 2001; Neri, 2010) require heavy engineering of the initial structures, evolving rules and careful human-guidance.
Further, none of the algorithms are able to evolve kinematically reasonable structures.
There are two major challenges in automatic robot design: 1) the search space of all possible design is large and combinatorial, and 2) the evaluation of each design requires learning or testing a separate optimal controller that is often expensive to obtain.
Due to the combinatorial nature of the problem, evolutionary strategy (ES) has been the de facto algorithm of automatic robot design in the pioneering works (Steels, 1993; Mitchell & Forrest, 1994; Langton, 1997).
The problem of large combinatorial search space and the difﬁculty in evaluation are also present in automatic neural network design or neural architecture search.
There have been several approaches to tackle these problems.
Bayesian optimization approaches (Snoek et al., 2012) primarily focus on ﬁnetuning the number of hidden units and layers from a predeﬁned set.
Reinforcement learning (Zoph &
Le, 2016) and genetic algorithms (Liu et al., 2017) are studied to evolve recurrent neural networks (RNNs) and convolutional neural networks (CNNs) from scratch to maximize the validation accuracy.
These approaches are computationally expensive because a large number of candidate networks have to be trained from scratch.
(Pham et al., 2018) and (Stanley & Miikkulainen, 2002) propose weight sharing among all possible candidates in the search space to effectively amortize the inner loop training time and thus speed up the architecture search.
Inspired by the recent progress on the neural architecture search, we propose an efﬁcient search method, “Neural Graph Evolution” (NGE), for automatic robot design in simulation that co-evolves the robot design and control policy simultaneously.
Unlike the recent deep reinforcement learning works (Mnih et al., 2013; Bansal et al., 2017; Heess et al., 2017) on learning locomotion control policy of speciﬁc robots carefully designed by human experts, we aim to adapt the robot design along with policy learning to maximize the agent’s performance.
NGE treats robot designs as a graph search problem.
It uses graph as the main backbone of rich design representation and graph neural networks (GNN) as the controller.
Similar to ES, NGE iteratively evolves new graphs and removes graphs based on the performance guided by the learnt GNN controller.
The speciﬁc contributions of this paper are as follows:
• We formulate the automatic robot design as a graph search problem.
• We utilize graph neural networks (GNNs) to share the weights between the controllers, which greatly reduces the computation time needed to evaluate each new robot design.
• We propose a population-based search algorithm over the robot design space that iterates over generations of candidates.
• To balance exploration and exploitation during the search, we developed a mutation scheme that incorporates model uncertainty.
• We show NGE automatically discovers robot designs that are comparable to the ones designed by human experts from stretch in MuJoCo (Todorov et al., 2012), while random graph search and naive evolutionary strategy fails to discover meaningful results on these tasks.
R ELATED W ORK 2. R EINFORCEMENT L EARNING
In reinforcement learning (RL), the problem is usually formulated as a Markov Decision Process (MDP).
The inﬁnite-horizon discounted MDP consists of a tuple of ( S , A , γ, P, R ), respectively the state space, action space, discount factor, transition function, and reward function.
The objective of the agent is to maximize the total expected reward J ( θ ) =
E π [ (cid:80) ∞ t =
γ t r ( s t , a t )] , where the state transition follows the distribution P ( s t +1 | s t , a t ) .
Here, s t and a t denotes the state and action at time step t .
In this paper, to evaluate each robot structure, we use PPO to train RL agents (Schulman et al., 2017; Heess et al., 2017).
PPO uses a neural network parameterized as π θ ( a t | s t ) to represent the policy, and adds a penalty for the KL-divergence between the new and old policy to prevent over-optimistic updates.
PPO optimize the following surrogate objective function instead:
[Equation]
We denote the estimate of the expected total reward given the current state-action pair, the value and the advantage functions, as Q t ( s t , a t ) , V ( s t ) and A t ( s t , a t ) respectively.
PPO solves the problem by iteratively generating samples and optimizing J PPO.
2. G RAPH N EURAL N ETWORK
Graph Neural Networks (GNNs) are very effective when the data is in in the form of graph (Bruna et al., 2014; Defferrard et al., 2016;
Li et al., 2015; Kipf & Welling, 2017; Duvenaud et al., 2015;
Henaff et al., 2015).
Recently, the use of GNNs in locomotion control has greatly increased the transferability of controllers (Wang et al., 2018).
A GNN operates on a graph whose nodes and edges are denoted respectively as u ∈ V and e ∈ E .
We consider the following GNN, where at timestep t each node in GNN receives an input feature and is supposed to produce an output at a node level.
Input Model : The input feature for node u is denoted as x tu .
x tu is a vector of size d , where d is the size of features.
In most cases, x tu is produced by the output of an embedding function used to encode information about u into d -dimensional space.
Propagation Model :
Within each timestep t , the GNN performs T internal propagations, so that each node has global (neighbourhood) information.
In each propagation, every node communicates
[Figure]
i
Add
[Figure]
Del
[Figure]
Graph
Pert
[Figure]
Graph
Mutation
[Figure]
Policy Sharing i
[Figure]
with its neighbours, and updates its hidden state by absorbing the input feature and message.
We denote the hidden state at the internal propagation step τ ( τ ≤ T ) as h t,τu .
Note that h t, 0 u is usually initialized as h t − 1 , T u , i.e., the ﬁnal hidden state in the previous time step.
h 0 , 0 is usually initialized to zeros.
The message that u sends to its neighbors is computed as
[Equation]
where M is the message function.
To compute the updated h t,τu , we use the following equations:
[Equation]
where R and U are the message aggregation function and the update function respectively, and N G ( u ) denotes the neighbors of u .
Output Model : Output function F takes input the node’s hidden states after the last internal propagation.
The node-level output for node u is therefore deﬁned as u t =
F ( h t, T u ) .
Functions M, R, U, F in GNNs can be trainable neural networks or linear functions.
For details of
GNN controllers, we refer readers to (Wang et al., 2018).
N EURAL G RAPH E VOLUTION
In robotics design, every component, including the robot arms, ﬁnger and foot, can be regarded as a node.
The connections between the components can be represented as edges.
In locomotion control, the robotic simulators like MuJoCo (Todorov et al., 2012) use an XML ﬁle to record the graph of the robot.
As we can see, robot design is naturally represented by a graph.
To better illustrate Neural
Graph Evolution (NGE), we ﬁrst introduce the following terminologies and summarize the algorithm.
Graph and Species .
We use an undirected graph G =
( V, E, A ) to represent each robotic design.
V and E are the collection of physical body nodes and edges in the graph respectively.
The mapping
A : V → Λ maps the node u ∈ V to its structural attributes A ( u ) ∈ Λ , where Λ is the attributes space.
For example, the ﬁsh in Figure 1 consists of a set of ellipsoid nodes, and vector A ( u ) describes the conﬁgurations of each ellipsoid.
The controller is a policy network parameterized by weights θ .
The combination of graph and policy is deﬁned as a species , denoted as Ω =
( G , θ ) .
Generation and Policy Sharing .
In the j th iteration, NGE evaluates a pool of species called a generation , denoted as P j = { ( G ji , θ ji ) , ∀ i = 1 , 2 , ..., N} , where N is the size of the generation.
Different from ES, in NGE, the search space includes not only graph space, but also the weight space of the policy network.
For better efﬁciency of NGE, we design a process called Policy Sharing (PS), where weights are reused from parent to child species.
Similar to ES, NGE performs optimization by iterating among mutation , evaluation and selection .
The objective and performance metric of NGE are introduced in Section 3.1.
In NGE, we randomly initialize the generation with N species.
For each generation, NGE trains each species and evaluates their ﬁtness separately, the policy of which is described in Section 3.2.
During the selection, we eliminate K species with the worst AF.
To mutate K new species from surviving species, we develop a novel mutation scheme called Graph Mutation with Uncertainty (GM-UC), described in Section 3.3, and efﬁciently inherit policies from the parent species by Policy Sharing, described in Section 3.4.
Our method is outlined in Algorithm 1.
3. A MORTIZED F ITNESS AND O BJECTIVE F UNCTION
Fitness represents the performance of a given G using the optimal controller parameterized with θ ∗ ( G ) .
However, θ ∗ ( G ) is impractical or impossible to obtain for the following reasons.
First, each design is computationally expensive to evaluate.
To evaluate one graph, the controller needs to be trained and tested.
Model-free (MF) algorithms take more than one million in-game timesteps to train a simple 6-degree-of-freedom cheetah (Schulman et al., 2017), while model-based (MB) controllers, even given ground-truth dynamics, can be 7x slower than real time to test, while usually having lower performance than MF methods (Tassa et al., 2012; Nagabandi et al., 2017).
Second, the search in robotic graph space easily or guaranteedly gets stuck in local-optima.
In robotic design, local-optima are difﬁcult to detect as it is hard to tell whether the controller has converged or has reached a temporary optimization plateau.
Therefore, to speed up and trade off between evaluating ﬁtness and evolving new species, we introduce amortized ﬁtness (AF), as deﬁned in the following equation,
[Equation]
AF spreads the computation cost across generations.
Within each iteration, the species update the policy weights θ for K times.
Essentially, we use a surrogate objective function, as shown in Figure 1.
With this surrogate objective function, NGE is able to co-adapt the controller θ and G simultaneously.
Naive ES and RGS algorithms use the estimated controller ˜ θ ( G ) after a ﬁxed number of updates, which is undertrained and biased, and therefore makes ES and RGS easy to overﬁt.
AF scores are used to rank the performance of all the species during the selection.
[Table] [Equation] intractable [Equation]
a biased estimation of
[Equation]
3. P OLICY R EPRESENTATION
Given a species with graph G , we train the parameters θ of policy network π θ ( a t | s t ) using reinforcement learning.
Similar to (Wang et al., 2018), we use a GNN as the policy network of the controller.
A graphical representation of our model is shown in Figure 1.
We follow the notations in Section 2.2.
For the input model , we parse the input state vector s t obtained from the environment into a graph, where each node u ∈ V fetches the corresponding observation o ( u, t ) from s t , and extracts the feature x O,tu with an embedding function Φ .
We also encode the attribute information A ( u ) into x Au with an embedding function denoted as ζ .
The input feature x tu is thus calculated as:
[Equation]
O
[Equation]
where [ . ] denotes concatenation.
We use θ Φ , θ ζ to denote the weights of embedding functions.
The propagation model is the standard model we described in Section 2.2.
We use summation as the aggregation function and a GRU (Chung et al., 2014) as the update function.
For the output model , we deﬁne the collection of controlling nodes as u ∈ F ⊆ V , and deﬁne Gaussian distributions on each node controller as follows:
[Equation]
where µ u and σ u are the mean and the standard deviation of the action distribution.
The weights of output function are denoted as θ F .
By combining all the actions produced by each node controller, we have the policy distribution of the agent:
[Equation]
We optimize the π ( a t
| s t ) with PPO, the details of which are provided in Appendix A.
3. G RAPH M UTATION WITH U NCERTAINTY
Between generations, the graphs of species evolve from parents to children.
Without heavy engineering of the mutation primitives, in particular, we allow the following primitive operations to the parent’s graph G :
M 1 , Add-Node : In the M 1 (Add-Node) operation, the growing of a new body part is done by sampling a node v ∈ V from the parent, and append a new node u to it.
We randomly initialize u ’s attributes from a uniform distribution in attribute space.
M 2 , Add-Graph : The M 2 (Add-Graph) operation allows for faster evolution by reusing the subtrees in the graph with good functionality.
We sample a sub-graph or leaf node G (cid:48)
=
( V (cid:48) , E (cid:48) , A (cid:48) ) from the current graph, and a placement node u ∈ V ( G ) to which to append G (cid:48) .
We randomly mirror the attributes of the root node in G (cid:48) to incorporate a symmetry prior.
M 3 , Del-Graph : The process of removing body parts is deﬁned as M 3 (Del-Graph) operation.
In this operation, a sub-graph G (cid:48) from G is sampled and removed from G .
M 4 , Pert-Graph : In the M 4 (Pert-Graph) operation, we randomly sample a sub-graph G (cid:48) and recursively perturb the parameter of each node u ∈ V ( G (cid:48) ) by adding Gaussian noise to A ( u ) .
We visualized a pair of example ﬁsh in Figure 1.
The ﬁsh on the top-right is mutated from the ﬁsh on the top-left by applying M 1 .
The new node (2) is colored magenta in the ﬁgure.
To mutate each new candidate graph, we sample the operation M and apply M on G as
G (cid:48)
=
M ( G ) , where M ∈ {M l , l = 1 , 2 , 3 , 4 } , P ( M = M l ) = p lm .
(9) p lm is the probability of sampling each operation with (cid:80) l p lm = 1 .
To facilitate evolution, we want to avoid wasting computation resources on species with low expected ﬁtness, while encouraging NGE to test species with high uncertainty.
We again employ a GNN to predict the ﬁtness of the graph G , denoted as ξ P ( G ) .
The weights of this GNN are denoted as θ G .
In particular, we predict the AF score with a similar propagation model as our policy network, but the observation feature is only x Au , i.e., the embedding of the attributes.
The output model is a graph-level output (as opposed to node-level used in our policy), regressing to the score ξ .
After each generation, we train the regression model using the L2 loss.
[Figure]
However, pruning the species greedily may easily overﬁt the model to the existing species since there is no modeling of uncertainty.
We thus propose Graph Mutation with Uncertainty (GM-UC) based on Thompson Sampling to balance between exploration and exploitation.
We denote the dataset of past species and their AF score as D .
GM-UC selects the best graph candidates by considering the posterior distribution of the surrogate P ( θ G | D ) :
[Equation]
Instead of sampling the full model with (cid:102)
θ G ∼ P ( θ G |D ) , we follow Gal & Ghahramani (2016) and perform dropout during inference, which can be viewed as an approximate sampling from the model posterior.
At the end of each generation, we randomly mutate C ≥ N new species from surviving species.
We then sample a single dropout mask for the surrogate model and only keep N species with highest ξ P .
The details of GM-UC are given in Appendix F.
3. R APID A DAPTING USING P OLICY S HARING
To leverage the transferability of GNNs across different graphs, we propose Policy Sharing (PS) to reuse old weights from parent species.
The weights of a species are as follows:
[Equation]
(11) where θ Φ , θ ζ , θ M , θ U , θ F are the weights for the models we deﬁned earlier in Section 3.2 and 2.2.
Since our policy network is based on GNNs, as we can see from Figure 1, model weights of different graphs share the same cardinality (shape).
A different graph will only alter the paths of message propagation.
With PS, new species are provided with a strong weight initialization, and the evolution will less likely be dominated by species that are more ancient in the genealogy tree.
E XPERIMENTS
In this section, we demonstrate the effectiveness of NGE on various evolution tasks.
In particular, we evaluate graph search from scratch in Section 4.1, and ﬁne-tuning from human-engineered species in Section 4.2.
We also provide an ablation study on GM-UC in Section 4.3, and ablation study on computational cost or generation size in Section 4.4.
Experiments are simulated with MuJoCo.
The code will be released upon acceptance.
We do a grid search on the hyper-parameters as summarized in Appendix E, and show the averaged curve of each method.
In the Random Graph Search (RGS) baseline, a large amount of graphs are generated randomly.
RGS focuses on exploiting given structures, and relies no evolution to generate new graphs.
In the Evolutionary Strategy (ES) baseline, no GM-UC is used during training.
Old species are allowed to continue their training across different generations (otherwise the controller will be under-trained).
For both RGS and ES algorithm, we use a multi-layer perceptron as the policy network.
We also introduce the test environments here:
Fish Env : In the fish environment, graph consists of ellipsoids.
The reward is the swimming-speed along the y -direction.
We denote reference human-engineered graph (Tassa et al., 2018) as G F .
Walker Env : We also deﬁne a 2D environment walker constructed by cylinders, where the goal is to move along x -direction as fast as possible.
We denote the reference human-engineered walker as
G W and cheetah as G C
(Tassa et al., 2018).
5/18/ localhost:8000/html_visual/expand_genealogy.html
http://localhost:8000/html_visual/expand_genealogy.html
R:7.
R:52.
R:50.
R:56.
R:46.
R:61.
R:1.
R:66.
R:34.
R:39.
R:42.15 R:68.37 R:67.
R:70.41 R:65.
R:72.02 R:70.05 R:62.
R:44.19 R:62.
R:63.
R:54.
R:70.
R:69.
R:56.
R:57.
R:64.
R:61.
R:61.66 R:49.
R:51.
R:52.63 R:51.
R:41.
R:28.
R:47.
[Figure]
4. E VOLUTION T OPOLOGY S EARCH
In this experiment, the task is to evolve the graph and the controller from scratch.
For both fish and walker , species are initialized as random ( G , θ ) .
Computation cost is often a concern among evolutionary algorithms.
In our comparison results, we allocate the same computation budget to all methods, which is approximately 12 hours on a EC2 m4.16xlarge cluster with 64 cores for one session.
A grid search on hyper-parameters is performed (details in Appendix E).
The averaged curves from different runs are shown in Figure 2.
In both fish and walker environments, NGE is the best model.
We ﬁnd RGS is not able to efﬁciently search the space of G even after evaluating 12 , 800 different graphs.
The performance of ES grows faster for the earlier generations, but is worse than our method in the end.
By looking at the generated species, ES overﬁts to local species that dominate the rest of generations.
To better understand the evolution process, we visualize the genealogy tree of ﬁsh using our model in
Figure 3.
Our ﬁsh species gradually generates three ﬁns with preferred { A ( u ) } , with two side-ﬁns symmetrical about the ﬁsh torso, and one tail-ﬁn lying on the middle line.
We obtain similar results for walker , as shown in Appendix C.
4. F INE - TUNING S PECIES
Evolving every species from scratch is costly in practice.
For many locomotion control tasks, we already have a decent human-engineered robot as a starting point.
In this ﬁne-tuning task, we study how our algorithm can improve upon the human-engineered design.
We examine the scenario in which ( V, E ) of graph is ﬁxed, and our model is tasked to ﬁne-tune (evolve) only the node attributes { A ( u ) } .
In the baseline models, the graph ( V, E, A ) is ﬁxed, and only the controllers are trained.
As we can see from Figure 4, when given the same wall-clock time, NGE can discover ﬁne adjustment that signiﬁcantly improves the initial human-engineered design.
The cheetah gradually transforms [Figure] the forefoot into a claw , the 3D-ﬁsh rotates the pose of the side-ﬁns and tail, and 2D-walker evolves bigger feet.
4. G REEDY S EARCH V . S . E XPLORATION UNDER U NCERTAINTY
We also investigate the performance of NGE with and without Graph Mutation with Uncertainty, whose hyper-parameters summarized in Appendix E.
In Figure 5a, we applied GM-UC to the evolution graph search task.
The ﬁnal performance of the GM-UC outperforms the baseline on both fish and walker environments.
The proposed GM-UC is able to better explore the graph space.
4. C OMPUTATION C OST AND G ENERATION S IZE
We also investigate how the generation size N affect the ﬁnal performance of NGE.
We note that as we increase the generation size and computing resources, NGE achieves marginal improvement on the simple Fish task.
A NGE session with 16-core m5.4xlarge ($0.768 per Hr) AWS machine could achieve almost the same performance with 64-core m4.16xlarge ($3.20 per Hr) in Fish environment in the same wall-clock time,
However, we do notice there is a trade off between computational resources and performance on the more difﬁcult task.
In general, NGE is effective even when the computing resources are limited and it signiﬁcantly outperforms RGS and ES by using only a small generation size of 16.
D ISCUSSION
In this paper, we introduced NGE, an efﬁcient graph search algorithm for automatic robot design that co-evolves the robot design graph and its controllers.
NGE greatly reduces evaluation cost by transferring the learned GNN-based control policy from previous generations, and better explores the search space by incorporating model uncertainties.
Our experiments show that searching for robotic graph is challenging where both random graph search and evolutionary strategy fail to discover meaning robot designs.
NGE outperforms the naive approaches in both ﬁnal performance and computation time by a order of magnitude, and is the ﬁrst algorithm that can discovers graphs similar to carefully hand-engineered design.
We believe this work is an important step towards automated robot design, and is inspiring to other graph search problems.
A D ETAILS OF N ERVE N ET ++
Similar to NerveNet, we parse the agent into a graph, where each node in the graph corresponds to the physical body part of the agents.
For the example, the fish in ﬁgure 1 could be parsed into a graph of ﬁve nodes, namely the torso (0), left-ﬁn (1), right-ﬁn (2), and tail-ﬁn bodies (3, 4).
By replacing MLP with NerveNet, the learnt policy has much better performance on robustness and transfer learning.
Since our GNN policy network is based on Wang et al.
(2018), we name our model
NerveNet++.
In the origin NerveNet, at every timestep, several propogation steps are performed so that every node could observe global information before producing the control signal.
It is time and memory consuming, and the minimum number of propogation steps is constrained by the depth of the graph.
Therefore in NerveNet++, we propose a propogation model with memory state, where node updates its hidden state by absorbing the input feature and message with time.
Since the episode of each game usually lasts for several hundred timesteps, it is computationally expensive and ineffective to build the full back-propagation graph.
Inspired by Mnih et al.
(2016), we employ the truncated graph back-propagation to optimize the policy.
NerveNet++ is suitable for evolutionary algorithm, as it brings speed-up in wall-clock time, and decreases the amount of memory usage.
The computational performance evaluation are provided in Appendix B.
The parameters of the NerveNet++ model is trained by PPO algorithm Schulman et al.
; Heess et al.
B O PTIMIZATION WITH T RUNCATED B ACKPROPAGATION
During training, the agent generates the rollout data by sampling from the distribution a t ∼ π ( a t | s t ) and store the training data of D = { a t , s t , { h t,τ =0 u }} .
For the training of reinforcement learning agents with memory, the original training objective is
[Equation]
where we denote the whole update model as H and
[Equation]
The memory state h u,t +1 is depend on previous actions, observations, and states.
Therefore, the full back-propagation graph will be the same length as the episode length, which is very computationally intensive.
The intuition from the authors in Mnih et al.
(2016) is that, for the RL agents, the dependency of the agents on timesteps that are far-away from the current timestep is limited.
And not
[Figure] [Figure]
[Figure]
much accuracy of the gradient estimator will be lost if we truncate the back-propagation graph.
We deﬁne a back-propagation length Γ , and optimize the following objective function instead:
[Equation]
, { h t,τ =0 u } )
[Equation]
, where
[Equation] [Equation]
Essentially this optimization means that we only back-propagate up to Γ timesteps, namely at the places where κ = 0 , we treat the hidden state as input to the network and stop the gradient.
To optimize the objective function, we follow same optimization procedure from Wang et al.
(2018), which is a variant of PPO Schulman et al.
(2017), where a surrogate loss J ppo ( θ ) is optimized.
We refer readers to these papers for algorithm details.
C F ULL NGE R ESULTS
Similar to the ﬁsh genealogy tree, in Fig.
8, the simple initial walking agent evolves into a cheetah-like structure, and is able to run with high speed.
We also show the species generated by NGE, ES and
RGS.
Note that we do not cherry-pick the results in Figure 9.
R:16.
R:46.
R:49.
R:50.
R:49.
R:22.
R:50.
R:54.
R:61.
R:66.61 R:4.
R:4.
RGS RGS
NTE
ES
D R ESETTING C ONTROLLER FOR F AIR C OMPETITION
[Figure]
Although amortized ﬁtness is a better estimation of the ground-truth ﬁtness, it is still biased.
Species that appear earlier in the experiment will be trained for more updates if it survives.
Indeed, intuitively, it is possible that in the real nature, species that appear earlier will dominate the generation by number, and new species are eliminated even if the new species has better ﬁtness.
Therefore, we design the experiment where we reset the weights for all species θ = ( θ Φ , θ ζ , θ M , θ U , θ F ) randomly.
By doing this, we are forcing the species to compete fairly.
From the ﬁgure 10, we notice that this method helps the exploration, which leads to higher reward in the end.
But it usually takes longer time for the algorithm to converge.
And therefore for the graph search task ﬁgure 2, we do not include the results with controller-resetting.
E H YPER - PARAMETERS S EARCHED
All methods are given equal amount of computation budget.
To be more speciﬁc, the number of total timestep generated by all species for all generation is the same for all methods.
For example, if we use 10 training epochs in one generation, each of the epoch with 2000 sampled timesteps, then the computation budget could let NGE evolve for 200 generation, where each generation has a species size of 64.
For NGE, RGS, ES models in Figure 11, we run a grid search on the hyper-parameters recorded in Table 2, and Table 3, and plot the curve with the best results respectively.
Since the number of generation for RGS baseline could be regarded as 1, its curve is plotted with the number of updates normalized by computation resource as x-axis.
Here we show the detail ﬁgures of six baselines, which are namely RGS-20, RGS-100, RGS-200, and ES-20, ES-100, ES-200.
The number attached to the baseline names is indicating the number of inner-loop policy training epochs.
In the case of RGS-20, where more than 12800 different graphs are searched, the average reward is still very low.
Increasing the number of inner-loop training of species to 100 and 200 does not help the ﬁnal performance signiﬁcantly.
[Figure]
To test the performance with and without GM-UC, we use 64-core clusters (generations of size 64).
Different from graph search experiment, we do not want to run excessive amount experiment with a grid search.
Therefore, the hyper-parameters are chosen to be the ﬁrst value available in Table 2 and
Table 3.
Items
Value Tried
Number of Iteration Per Update 10, 20, 100,
Number of Species per Generation 16, 32, 64,
Elimination Rate 0.15, 0.20, 0.
Discrete Socket
Yes, True
Timesteps per Updates 2000, 4000,
Target KL 0.
Learning Rate Schedule
Adaptive
Number of Maximum Generation
Prob of Add-Node , Add-Graph 0.
Prob of Pert-Graph 0.
Prob of Del-Graph 0.
Allow Mirrowing Attrs in Add-Graph
Yes, No
Allow Resetting Controller
Yes, No
Resetting Controller Freq
[Figure]
Items
Value Tried
Allow Graph-Add
True, False
Graph Mutation with Uncertainty
True, False
Pruning Temperature 0.01, 0.1,
Network Structure
NerveNet, NerveNet++
Number Candidates before Pruning 200,
F M ODEL BASED SEARCH USING T HOMPSON S AMPLING
Thompson Sampling is a simple heuristic search strategy that is typically applied to the multi-armed bandit problem.
The main idea is to select an action proportional to the probability of the action being optimal.
When applied to the graph search problem, Thompson Sampling allows the search to balance the trade-off between exploration and exploitation by maximizing the expected ﬁtness under the posterior distribution of the surrogate model.
Formally, Thompson Sampling selects the best graph candidates at each round according to the expected estimated ﬁtness ξ P using a surrogate model.
The expectation is taken under the posterior distribution of the surrogate P ( model | data ) :
[Equation]
F. S URROGATE MODEL ON GRAPHS
Here we consider a graph neural network (GNN) surrogate model to predict the average ﬁtness of a graph as a Gaussian distribution, namely P ( f ( G )) ∼ N (cid:0) ξ P ( G ) , σ 2 ( G ) (cid:1) .
We use a simple architecture that predicts the mean of the Gaussian from the last hidden layer activations,
h W ( G ) ∈ R D , of the GNN, where W are the weights in the GNN up to the last hidden layer.
Greedy search We denoted the size of dataset as N .
The GNN weights are trained to predict the average ﬁtness of the graph as a standard regression task:
[Equation]
where ξ P ( G n ) =
W Tout h W ( G n )
[Equation]
candidates 7: Update generation P j 8: for m < N do (cid:46) Train and evaluate each species 9: for k < maximum parameter updates do 10:
Train policy π G m 11: end for 12:
Evaluate the ﬁtness ξ ( G m , θ m ) 13: end for 14: end for
Thompson Sampling
In practice, Thompson Sampling is very similar to the previous greedy search algorithm.
Instead of picking the top action according to the best model parameters, at each generation, it draws a sample of the model and takes a greedy action under the sampled model.
Approximating Thompson
Sampling using Dropout Performing dropout during inference can be viewed as an approximately sampling from the model posterior.
At each generation, we will sample a single dropout mask for the surrogate model and rank all the proposed graphs accordingly.
1: Initialize generation P2: for j < maximum generations do 3:
Collect the ( ξ ki , G ki ) from previous k ≤ j generations (cid:46) dataset 4:
Train W and W out on { ( ξ ki , G ki ) }
Nn = (cid:46)
Train GM-UC 5:
Propose C new graph {G i } C i =1 , C >> M .
(cid:46)
Propose new candidates 6:
Sample a model from the posterior of the weights.
7: e.g. (cid:102) W, (cid:102) W out ∼ P
( W, W out |D )
≈ N
([ W, W out ] , [ W, W out ]) 8: (similar to DropConnect Wan et al.
Rank { ξ P ( G i | (cid:102) W, (cid:102) W out ) } C i =1 on the proposals and pick the top K 10: for m <
N do (cid:46) Train and evaluate each species 11: for k < maximum parameter updates do 12:
Train policy π G m 13: end for 14:
Evaluate the ﬁtness ξ ( G m , θ m ) 15: end for 16: end for
1: Initialize generation P2: for j < maximum generations do 3:
Collect the ( ξ ki , G ki ) from previous k ≤ j generations (cid:46) Update dataset 4:
Train W and W out on {G n , ξ ( G n ) }
Nn =1 using dropout rate 0.5 on the inputs of the fc layers.
Propose C new graph {G i } C i =1 , C >> M .
(cid:46)
Propose new candidates 6:
Sample a dropout mask
m i for the hidden units 7:
Rank { ξ P ( G i | W, W out , m i ) }
Ji =1 on the proposals and pick the top K 8: for m <
N do (cid:46) Train and evaluate each species 9: for k < maximum parameter updates do 10:
Train policy π G m 11: end for 12:
Evaluate the ﬁtness ξ ( G m , θ m ) 13: end for 14: end for