C URRICULUM - BASED C O - DESIGN OF M ORPHOLOGY AND C ONTROL OF V OXEL - BASED S OFT R OBOTS
Anonymous authors
Paper under double-blind review
A BSTRACT
Co-design of morphology and control of a Voxel-based Soft Robot (VSR) for solving a given task is challenging due to the bi-level optimization in the enormous combined design and policy space.
In this paper, we present a Curriculum-based
Co-design (CuCo) method for learning to design and control VSRs through an easy-to-difficult process.
Specifically, we expand the design space from a small size to the target size gradually through a predefined curriculum.
At each stage of the curriculum, we use reinforcement learning to simultaneously train the design and policy, which is enabled by incorporating the design process into the environment and using differentiable policy representations.
The converged morphology, the learned design and control policies from the last stage are inherited and serve as the starting point for the next stage.
In the empirical studies, we show that CuCo is more efficient in creating larger robots with better performance by reusing the practical design and control patterns learned within each stage, in comparison to prior approaches that learn from scratch in the space of target size (see video).
I NTRODUCTION
The philosophy of embodied cognition (Pfeifer & Bongard, 2006; Pfeifer et al., 2014) inspires the domain of robotics that a robot’s ability to interact with the environment depends both on its brain (control policy) and body (morphology), which are inherently coupled (Spielberg et al., 2019; Gupta et al., 2021).
However, finding an optimal robot morphology and its controller for solving a given task is often unfeasible.
The major challenge for this endeavor is the enormous combined design and policy space.
Firstly, the freedom to pick the number of multi-material modules and the ways they are connected makes it notoriously difficult to explore the design space (Medvet et al., 2022).
For instance, in a robot simulator (Liu et al., 2020), there are over 4 × 10 8 possible morphologies for a robot composed of only 12 modules.
Secondly, the evaluation of a morphology requires a separate training procedure for its unique controller.
In this work, we consider the co-optimization of design and control of Voxel-based Soft Robots (VSRs) (Bhatia et al., 2021), a form of modular soft robots composed of elastic, multi-material cubic blocks.
Unlike fragile fully-integrated robots, they can be easily disassembled and reassembled to adapt to a wide range of environments (Shah et al., 2020; Pigozzi et al., 2022).
For efficiently exploring the modular robot design space, prior approaches commonly rely on artificial evolution (Sims, 1994; Cheney et al., 2013; Medvet et al., 2021), which maintains a population of design prototypes and adopt a bi-level optimization loop, where the outer loop of morphology optimization is based on the fitness of individual controllers from the inner loop.
These methods, however, tend to learn from scratch in the target design space where there is a significant combinatorial explosion.
Thus, they spend a large amount of time on policy optimization and evaluation.
Additionally, the separate training procedures significantly hinder the experience of design and control to be shared across different robots.
In view of these challenges, we propose a Curriculum-based Co-design (CuCo) method for learning to optimize the morphologies and behaviour of VSRs (Figure 1).
CuCo draws inspiration from Curriculum Learning (CL) that starting with easier tasks can help the agent learn better when presented with more difficult tasks later on (Bengio et al., 2009).
The key to our approach is expanding the design space from a small size to the target size gradually through a predefined curriculum.
Precisely, at each stage of the curriculum, we learn practical design and control patterns via Reinforcement [Figure] [Figure] voxel
Rigid voxel
[Figure] voxel [Figure] actuator
Verticalactuator
[Figure]
r a d i e n tb a s e d R L t r a i n i ng [Figure] Learning (RL) (Schulman et al., 2017), which is enabled by incorporating the design process into the environment and using differentiable policy representations.
The converged morphology, the learned design and control policies from the last stage are inherited and serve as the starting point for the next stage.
Due to the exploitation of the previous knowledge of design and control, CuCo can quickly produce robust morphologies with better performance in larger dimensional settings.
To handle incompatible state-action spaces and make the control patterns transferable across various morphologies, we model the local observations of all voxels as a sequence and adopt the selfattention mechanism (Vaswani et al., 2017) in our control policy to capture the internal dependencies between voxels, which caters to the need for dynamically accommodating changes in morphologies.
We also add material information to each voxel’s observation, thus the control policy is conditioned on the robot’s morphology, which eliminates the need for a population of prototypes and enables experience to be shared across different robots, bringing better sample efficiency and flexibility.
We propose to utilize an indirect-encoding Neural Cellular Automata (NCA) (Mordvintsev et al., 2020) as the design policy, which takes multiple actions to grow a robot from an initial seed (morphology).
NCA encodes complex patterns in a neural network and generates different developmental outcomes while using a smaller set of trainable parameters.
In this paper, we make the following contributions: (1) We propose CuCo, a curriculum-based codesign approach for learning to design and control VSRs.
(2) We demonstrate that the practical design and control patterns learned in small design spaces can be exploited to produce high-performing morphologies in larger dimensional settings.
(3) We showcase that our NCA-based design policy and self-attention-based control policy can share design and control patterns across different sizes of morphologies, bringing better sample efficiency and flexibility.
(4) Comprehensive experimental studies show that CuCo is more efficient in creating larger robots with better performance, in comparison to prior approaches that learn from scratch in the target design space.
R ELATED W ORK
Brain-body Co
design.
Finding the optimal morphology and controller for a robot to solve a given task is a challenging subject (Sims, 1994;
Wang et al., 2019b; Nygaard et al., 2020; Medvet et al., 2021; Hejna et al., 2021).
Current works can be divided into two categories, gradient-free and gradient-based.
Most gradient-free methods rely on artificial evolution to explore the vast design space.
Cheney et al.
(2013) evolve VSRs with 3D voxels over materials and actuators.
Bhatia et al.
(2021) create a large-scale benchmark and several population-based co-design algorithms.
Walker & Hauser (2021) evolve a sculpting adaptation system that allows VSRs to change their bodies to adapt to the environment.
Talamini et al.
(2021) propose a task-agnostic approach for automatically designing morphologies.
While the population-based approaches are not easily trapped in local optima, they are more sample inefficient and computationally expensive.
Besides, most of these works rely on open-loop controllers, which simplify the control as a periodic sequence of actuation that depends only on the time (e.g., sinusoidal control signals for varying the volume of voxels periodically), preventing robots from learning non-periodic behaviour.
In contrast, gradient-based methods, specifically, reinforcement learning, have been conducted for optimizing rigid-bodied robots.
Ha (2019) uses a population-based policy gradient method to improve robot design.
Pathak et al.
(2019) construct a self-organized modular robotic system that can generalize to unseen morphologies and tasks.
Luck et al.
(2019) learn a critic-based surrogate model for evaluating morphologies.
Schaff et al.
(2019) maintain a distribution over morphological parameters.
Chen et al.
(2020) learn a hardware-conditioned policy.
Yuan et al.
(2022) learn a transform-and-control policy based on the Graph Neural Network (GNN) to optimize rigid-bodied robots in a fixed design space.
Our work falls into the second category, instead, we focus on a scalable modular robot design space assigned by a curriculum. Within each learning stage, we unify the design and control as a single decisionmaking process like (Yuan et al., 2022). However, both the design and control policies of our work are expanded across stages as they are not restricted by the robot’s morphological topology.
Curriculum Learning.
Curriculum learning follows the spirit that training the agent with a series of tasks organized according to different complexity can speed up the convergence of the learning process (Bengio et al., 2009).
It has achieved tremendous success in both RL and robotic communities (Li et al., 2020;
Iscen et al., 2021; Jiang et al., 2021b).
In RL, one of the keys to curriculum learning is task generation and assignment.
Dennis et al.
; Jiang et al.
(2021a); Parker-Holder et al. (2022) focus on Unsupervised Environment Design (UED) which automatically constructs a distribution over environment variations that adapts to the training process of RL agents. While
Matiisen et al. Portelas et al. (2020) investigate teacher algorithms that monitor the learning progress and generate curricula to guide student algorithms to learn in difficult environments. In robotics, Wang et al. (2019a; 2020) generate an endless progression of increasingly challenging environments to evolve robust robots.
Fang et al.
(2021) propose a method to learn skills via automated generation of diverse tasks.
Stensby et al.
(2021) co-design robots’ morphologies and controllers together with environments evolved open-endedly.
We adopt a fixed curriculum by increasing the size of the robot design space according to the learning progress, as the number of possible robot morphologies increases exponentially with the expansion of each dimension of the design space.
Learning a general Controller.
Another related line is learning control policies which can generalize across different morphologies.
Many such approaches utilize GNNs (Sanchez-Gonzalez et al., 2018;
Wang et al., 2019b; Huang et al., 2020; Yuan et al., 2022) that condition the policy input on a graph representation related to the robot’s morphology.
However, for modular robot systems
GNNs have the limitation of aggregating multi-hop information (Pigozzi et al., 2022), which makes the modules not completely interchangeable and the system could be influenced by information loss, greatly limiting their flexibility.
Instead, Transformer (Vaswani et al., 2017) can be characterized as a fully connected GNN with attentional aggregation (Kurin et al., 2021), which is not restricted by the input graph topology, bringing potential advantages for controlling modular robots.
It is shown that generalization can be further enhanced by modularization, due to the success in modeling dynamic structures via the self-attention mechanism (Gupta et al., 2022; Trabucco et al., 2022).
B ACKGROUND
Voxel-based Soft Robots.
We investigate VSRs with multi-material cubic blocks (e.g., soft voxels, rigid voxels and actuator voxels) which are organized in a 2 -D grid (Figure 1 (C)).
The connectivity describes the connection pairs of adjacent voxels.
During the simulation, voxels (except empty voxels) only sense locally, and based on the input sensory information, a controller outputs control signals to vary the volume of actuator voxels.
Similar to biological tissues, they can be easily expanded and contracted to complete a wide variety of tasks (Bhatia et al., 2021).
Reinforcement Learning.
In contrast to prior co-design works of VSRs which seldom involve the learning of control, we use reinforcement learning to simultaneously train the design and control policies.
In the domain of RL, the problem is usually modeled as a Markov Decision Process (MDP), which can be specified by a 5 -tuple ( S , A , P, r, γ ) .
With the state space S and the action space A , P : S × S × A → [0 , 1] is the transition function of the environment, r ( s, a ) : S ×
A → R is the reward function, and the discount factor γ ∈ (0 , 1] specifies the degree to which rewards are discounted over time.
The goal of the RL agent is to find a policy to maximize the total expected [Figure] reward E π [ (cid:80) ∞ t =
γ t r ( s t , a t )] , here the policy is represented by a deep neural network parameterized as π θ ( a t | s t ) , which maps from states to distributions over actions.
In this work, we employ Proximal
Policy Optimization (PPO) (Schulman et al., 2017), a popular RL algorithm base on Policy Gradient (PG).
PPO uses Kullback-Leibler (KL) divergence to penalize the change between the old and the current policy to avoid instabilities from drastic changes in the policy’s behaviour.
Neural Cellular Automata.
Since we incorporate the design process into the environment, our objective is to find a design policy that can deal with arbitrary VSRs’ morphologies and output a series of actions to modify them.
Typically, VSRs can be characterized as multi-cellular systems which develop from a small set of initial cells (voxels) (Horibe et al., 2021).
We represent our design policy as a Neural Cellular Automata (NCA) (Mordvintsev et al., 2020; Sudhakaran et al., 2022; Palm et al., 2022), which begins with some initial seeds and updates their states according to local rules parameterized by a multilayer perceptron, thus, NCA is also not restricted by the robot’s morphological topology, making it possible for scaling the learned rules to arbitrary design spaces.
Self-attention.
Self-attention was proposed due to the challenges in handling long sequences (Vaswani et al., 2017).
The goal is to dynamically track the relationships between components in the input sequence, which can be viewed as an adaptive weights mechanism.
Given the input vectors X , W Q , W K and W V are three trainable weight matrices.
Q, K and V are the query, key, and value vectors where Q = XW Q , K = XW K and V = XW V .
The importance scores are firstly computed by the dot product between Q and K and then scaled by a non-linear function.
Self-attention is typically integrated into the transformer block, which also contains a feed-forward neural network, each of these two submodules has a residual connection around it and is succeeded by a normalization layer.
Our work here specifically uses the transformer encoder, enabling our control policy to dynamically adapt to arbitrary dimensional observations due to changes in morphologies.
C U C O : L EARNING TO D ESIGN AND C ONTROL FROM E ASY TO D IFFICULT
Given a target design space (e.g., 7 × 7 ), exploring all possible combinations is unfeasible.
The key to our approach is to expand this design space from a small size to the target size gradually through a predefined curriculum, which divides the whole optimization process into several stages (Figure 1 (A)).
We maintain a design policy π θ D and a control policy π θ C from the beginning to the end, thus the overall policy is represented as π θ =
{ π θ D ( a dt | s dt , Ψ t ) , π θ C ( a ct | s ct , Ψ t ) } where Ψ t ∈ { 0 , 1 } is a binary value indicating whether the robot is being created or being controlled, s dt and s ct represent the state information received at each time step of design and control, respectively.
Within each stage, we simultaneously learn to design and control through reinforcement learning, which is enabled by unifying the two processes into a single MDP.
Specifically, at the beginning of each episode, the design policy π θ D ( a dt | s dt , Ψ t ) performs N D steps to develop a robot from an initial seed, and no reward is assigned during the design steps (Figure 1 (B)).
The resulting robot is then controlled by π θ C ( a ct | s ct , Ψ t ) which is conditioned on the morphological information.
Thus, the design policy will be informed by the future environmental rewards collected by the control policy under the designed morphology. After collecting the desired number of trajectories, the two policies are jointly updated using PPO (Schulman et al., 2017). From one stage to the next, the dimensions
[Figure]
4. D ESIGN P OLICY
Figure 2 demonstrates a single design step when developing a VSR.
The design space is surrounded by empty voxels and each voxel is represented as a discrete value that corresponds to its material property (e.g., empty voxel= 0 , soft voxel= 1 , rigid voxel= 2 , horizontal actuator= 3 and vertical actuator= 4 ).
The state vector s d i for each voxel is composed of its type and the types of its neighborhood (e.g., Neumann or Moore neighborhood) (Horibe et al., 2021).
Practically, let N be the size of the design space, the MDP state of a design step is represented by s dt =
{ s d 1 t , s d 2 t , ..., s d N t } .
Here, the design policy π θ D ( a dt | s dt , Ψ t ) is a multilayer perceptron parameterized by θ D , which maps s dt to a categorical action distribution created by the output logits.
The dimension of the input layer corresponds to the number of the cell’s neighbors (e.g, 9 in Figure 2) and the dimension of the output layer corresponds to the total number of material types (e.g, 5 in Figure 2). During training, the new cell state is determined by sampling from the distribution. While evaluating the agent, the design policy selects the action that corresponds to the highest logit value.
At the beginning of each episode of the first stage, a single voxel placed at the center of the design space is used as the initial seed, and the rest voxels’ states are set to zero.
After N D design steps, all voxels are updated according to the policy’s outputs.
In the next learning stage, the converged morphology from the last stage will serve as the new initial seed, which is also placed at the center of the design space as the initial state of each episode.
Meanwhile, the weights of the learned design policy are also inherited. Since the morphology defines the connectivity, here, we let the design policy learn the connectivity on its own.
No rewards are assigned to the design steps, and we terminate the episode if the resulting design does not meet the connectivity or has no actuators.
4. C ONTROL P OLICY
Once a robot is designed, it is consumed by the control policy π θ C ( a ct | s ct , Ψ t ) and presented in the simulator, during which the morphology is unchangeable.
Our policy is invariant to the dimensionality of observations and actions as we model the local observations of all voxels as a sequence.
The input state of the robot at time step t is represented as s ct =
{ s vt , s gt } , where s vt =
{ s v 1 t , s v 2 t , ..., s v N t } , s v i t is composed of each voxel’s local information which contains the relative position of its four corners with respect to the center of mass of the robot and its material information.
s gt is the task-related observation such as terrain information of the environment and goal-relevant information.
To handle the incompatible state-action spaces and dynamically accommodate changes in morphologies, we incorporate the self-attention mechanism into the control policy, which can control all the agents with a single deep neural network, bringing better sample efficiency than population-based methods.
Figure 3 illustrates the architecture of the control policy.
We apply a learnable position embedding layer after the linear projection layer.
The local observation, the output feature of the Transformer and the task-related observation are concatenated before passing them through a decoder.
In our work, the control policy outputs the mean µ of the continuous action, and we use a constant standard deviation (a fixed diagonal covariance matrix Σ ) to create the Gaussian distribution π θ C ( a ct | s ct , Ψ t ) =
N ( µ ( s ct , Ψ t ) , Σ) .
All actions are clipped within the range of [0 . 6 , 1 . 6] as the control signal of actuator voxels. During training, the action is randomly sampled from the Gaussian distribution. While evaluating the agent, only the mean action is exploited for control.
4. P OLICY U PDATE AND THE C URRICULUM S ETTING
We simultaneously train the design and control policies using Proximal Policy Optimization (PPO), which is based on the Actor-Critic architecture (Schulman et al., 2017).
In our work, the critic network has the same architecture as the control policy network.
It estimates the value function
E π θ [ (cid:80) Tt =
γ t r ] , which suggests a possible policy distribution.
Here, we use the average of the outputs of total voxels as the final predicted value.
With the policy gradient method, both design and control policies are updated in the direction of maximizing the predicted value.
When estimating the value of the MDP state of a design step, we set the environment state s c to zero.
We apply a predefined curriculum that adjusts the size of the design space according to the training progress.
We define the hard case according to the dimensions of design space as the number of possible robot designs increases exponentially with each additional block.
During training, we increase the dimension when the target number of policy iterations is reached.
E MPIRICAL E VALUATION
In this section, we demonstrate the effectiveness of CuCo on various tasks.
We seek to answer the following questions: (1) Does CuCo provide an effective mechanism for learning to design and control VSRs?
(2) How does CuCo compare to prior approaches which learn from scratch in the space of target size in terms of the final performance and the produced morphologies?
(3) What impact does each component of CuCo bring to the learning process?
5. E XPERIMENT S ETTINGS
We create a modular robot design space as described in Section 4 and 8 task-related environmentsbased on the Evolution Gym platform (Bhatia et al., 2021).
The environment details can be found in
Appendix A. We compare CuCo with a number of approaches, both RL-based and population-based.
Baselines are introduced as follows:
CuCo-NCU : CuCo-NCU utilizes the same architectures of the design and control policies as CuCo but eliminates the curriculum component.
It learns to design and control from scratch in the space of target size.
This method helps us to investigate the effectiveness of the curriculum.
Additionally, we compare CuCo with three evolutionary methods proposed in (Bhatia et al., 2021)
The inner loop optimization of control policies is driven by PPO
Based on the fitness values of individuals, the outer loop of morphology optimization is implemented by three kinds of state-of- the- art design optimization algorithms
GA: Genetic Algorithm (GA) directly encodes the VSR’s morphology as a vector where each element is tailored to the voxel’s material property in order.
It keeps a proportion of robots with high fitness as survivors at the end of each generation, and the survivors undergo mutation conducted by randomly changing each voxel of the robot with a probability to produce a new generation.
CPPN-NEAT: In this method, the morphology of VSR is indirectly encoded by a Compositional Pattern Producing Network (CPPN)
(Stanley, 2007; Cheney et al., 2013), which relies on having access to the whole design space.
The training of CPPN is accomplished by NeuroEvolution of Augmenting Topologies (NEAT) (Stanley & Miikkulainen, 2002). [Table]
: 7.26 R: 5.
R: 11. R: 12. R: 0. R: 6.
R: 8.
R: 6.87 R: 5.
R: 11.
R: 12.
BO: Bayesian Optimization (BO) is widely used in solving black-box problems.
It trains a surrogate model based on the Gaussian Process (GP) to alleviate the computational burden of expensive fitness evaluation (Kandasamy et al., 2018).
Following the convention in the literature (Cheney et al., 2013; Horibe et al., 2021), we consider a 2 -D design space where all dimensions are equal in size.
The curriculum defines three stages ( 3 × 3 → 5 × 5 → 7 × 7 ) for Walker, Pusher, Jumper and Lifter.
In the remaining tasks, the curriculum is set to ( 3 × 3 → 5 × 5 ).
We run all experiments for each task with the same number of time steps in the environment.
For CuCo, this amounts to 1000 policy updates within each stage.
We employ a population with a size of 12 for all population-based methods and report the performance of the best individual for each task.
Further implementation details can be found in Appendix D.
5. O VERALL P ERFORMANCE
In Table 1, we list the final performance of each method and the result is reported over 7 different runs per environment.
Learning curves for all methods and visualization results of converged morphologies can be found in Appendix B.1. We observe that within error margins, CuCo significantly outperforms the baselines in 6 of the 8 environments, while only CPPN-NEAT and BO can achieve limited success in Climber and Lifter, respectively. In Section 6, we discuss in detail these two cases.
     
[Figure]
      3 H U I R U P DQ F H 3XVKHU &X&R&X&R1$7&X&R1&     
  3ROLF\LWHUDWLRQV (×
[Figure]
)      3 H U I R U P DQ F H
:DONHU
Without the guidance of the curriculum, CuCo-NCU performs poorly on most tasks since learning to design and control from scratch in the high-dimensional design space is challenging.
We demonstrate the converged morphologies of all methods across three different runs and their corresponding performance in Pusher (Figure 4).
For CuCo, we visualize the converged design of each stage.
Direct-encoding methods like GA and BO prefer horse-like morphologies composed of randomly organized voxels.
However, voxels in the last two robots created by GA often collide with each other, preventing robots from learning efficient control policies.
CuCo-NCU fails to discover practical design and control policies with much lower performance.
In contrast, we can see that CuCo can develop beneficial homogeneous tissue structures by module reuse, which can also be observed in many creatures, such as insects and vertebrates with several repeated body segments determined by homologous genes in different species (Dellaert, 1995). When the robot is moving we can observe that groups of voxels are expanded or compressed simultaneously, leading to coordinated behaviour (video).
Interestingly, VSRs designed by CuCo obtain different skills for pushing a box.
The first rhino-like robot chooses to push the box with its “horn”.
The second robot makes full use of the vertical actuators to let the box roll forward, while the third one uses horizontal actuators to contract itself and then pop the box out.
As shown in this video, CuCo also produces robots which can keep the balance and find a better launch angle when throwing the box.
Although GA and BO produce VSRs with arms, they are not strong enough to throw the box further.
CPPN-NEAT generates much simpler patches based on the global information of the design space.
In Appendix B.2, we further investigate differences between CPPN and NCA when chosen as the design policy of CuCo.
In sum, CuCo holds great potential for exploiting design and control patterns learned in small spaces to produce high-performing robots in larger dimensional settings.
5. A BLATIONS
We perform ablation studies to take a closer look at the behaviour of CuCo.
Firstly, we demonstrate the learning curves of CuCo and CuCo-NCU (Figure 5) to show that following an easy-to-difficult curriculum, it is much more efficient than learning from scratch in the target design space.
Especially in Pusher, CuCo-NCU spends a large amount of time learning the connectivity.
Secondly, we remove the self-attention mechanism used in our control policy and construct a new ablation CuCo-NAT .
Evidently, by removing the self-attention mechanism, the control policy fails to dynamically adapt to morphological changes, resulting in much lower performance.
These two ablations indicate that the curriculum and the self-attention mechanism are indeed crucial for finding better solutions.
Another key feature of CuCo is the mechanism of inheritance.
Between the two learning stages, the converged morphology, the learned design and control policies from the last stage are inherited.
We perform additional ablation studies to measure the impact of the inheritance on the subsequent learning process.
We design the following ablations: (1) CuCo-NIM : No inheritance of the converged morphology from the last stage; (2) CuCo-NID : No inheritance of the design policy.
We reset the parameters of the design policy at the end of each stage; (3) CuCo-NIC : No inheritance of the control policy.
We reset the parameters of the control policy at the end of each stage; (4) CuCo-NIDC : No inheritance of the design and control policies.
We reset all network parameters at
    
  3ROLF\LWHUDWLRQV (×10 3 ) 
     3 H U I R U P DQ F H 3XVKHU &X&R&X&R1,0&X&R1,'&X&R1,&&X&R1,'&     
  3ROLF\LWHUDWLRQV (×10 3 ) 
    3 H U I R U P DQ F H
:DONHU the end of each stage. The learning curves of all ablations are shown in Figure 6.
For better visualization, we plot these curves from the second learning stage.
By comparing CuCo with CuCo-NID and CuCo-NIDC, it is worth noting that the inheritance of the design policy plays a critical role during the learning process. Even if the control policy is reset (CuCo-NIC), the performance is less affected.
Additionally, the inheritance of the converged morphology also contributes to the learning process, as new morphologies can be quickly built by expanding the old morphology.
D ISCUSSION AND C ONCLUSION
In this work, we proposed CuCo, a curriculum-based co-design method for learning to design and control voxel-based soft robots through an easy-to-difficult process.
Compared with previous approaches that learn from scratch in the target design space, our method is more efficient in producing larger robots with better performance as it follows a predefined curriculum and effectively exploits the knowledge of design and control obtained in small dimensional settings. Empirical evaluations on various robot control tasks show that CuCo outperforms prior methods in the final performance, producing intriguing robot designs with repeated body segments. However, We also observe that CuCo tends to produce more regular shapes than GA and BO, resulting in failures on some complex tasks (e.g., Climber and Lifter) which may require robots to have irregular body structures (e.g., thin limbs or fingers). We speculate that there are two reasons for the failure: (1) Design .
In our work, NCA is assigned no design-related reward and is encouraged to learn the connectivity on its own. Specifically, we check the connectivity at each design step and terminate the episode if the resulting design does not meet the connectivity or has no actuator voxels. Therefore, NCA becomes conservative in using empty voxels. Due to the mechanism of inheritance, this kind of design pattern would be further extended in larger design spaces, restricting the exploration of irregular morphologies.
(2) Control .
For some challenging tasks (e.g., Lifter), when the control policy is hard to explore and exploit the environmental reward, the design policy will also hardly see the future reward, as we formulate the design and control into a single MDP. Fortunately, the failure also suggests that we can add design rewards to guide NCA to explore some counterintuitive structures, and using more complex NCA architectures (Sudhakaran et al., 2022) may also help the exploration of more irregular designs.
Hence, an important future work is creating a more efficient mechanism for balancing the exploration and exploitation abilities of NCA. During the learning process, we do not constrain the number of modules for each type of material. Thus, the optimization is uncontrolled.
A future direction would be to investigate the constrained co-design method. We found that CuCo performs well under the fixed curriculum, a promising direction of future work will involve extending our method to automatic curriculum learning.
In CuCo, there are two kinds of developments at different timescales.
Within each stage, the robots grow under the restriction of body size.
From one stage to the next, robots will become bigger and stronger.
However, size is not the only morphological characteristic of living creatures.
An important line of exploration is to extend CuCo to help study the morphogenesis of virtual creatures.
Co-designing physical VSRs (Kriegman et al., 2019; 2020) using CuCo with sim2real techniques is expected to further extend the horizon.
A E NVIRONMENT D ETAILS
In this section, we provide details about the tasks simulated in Evolution Gym platform (Banarse et al., 2019).
Position.
p o is a 2-dim vector that represents the position of the center of mass of an object o in the simulation at time t .
p ox and p oy are x and y components of this vector, respectively.
p o is calculated by averaging the positions of all the point-masses that make up object o at time t .
Velocity.
v o is a 2-dim vector that represents the velocity of the center of mass of an object o in the simulation at time t .
v ox and v oy are x and y components of this vector, respectively.
v o is calculated by averaging the velocities of all the point-masses that make up object o at time t .
Orientation.
θ o is a 1-dim vector that represents the orientation of an object o in the simulation at time t .
Let p i be the position of point mass i of object o .
θ o is computed by averaging over all i the angle between the vector p i − p o at time t and time 0 .
This average is a weighted average weighted by || p i − p
o || at time 0 .
Other observations.
h ob ( d ) is a vector of length (2 d +1) that describes elevation information around the robot below its center of mass.
More specifically, for some integer x ≤ d , the corresponding entry in vector h ob ( d ) will be the highest point of the terrain which is less than p oy between a range of [ x, x + 1] voxels from p ox in the x -direction.
[Figure]
In this task, the robot is rewarded by walking as far as possible on flat terrain.
The target design space is 7 × 7 .
The task-specific observation is v robot , and the reward R is:
[Equation]
which rewards the robot for moving in the positive x -direction.
The robot receives a reward of 1 for reaching the end of the terrain.
The episode duration reaches a 500 timesteps.
[Figure]
In this task, the robot climbs up stairs of varying lengths.
The target design space is 5 × 5 .
The taskspecific observation is formed by concatenating vectors { v robot , θ robot , h robot b (5) } , and the reward
R is:
[Equation]
which rewards the robot for moving in the positive x -direction.
The robot also receives a one-time reward of 2 for reaching the end of the terrain, and a one-time penalty of − 3 for rotating more than 75 degrees from its originally orientation in either direction (after which the environment resets).
The episode duration reaches a 600 timesteps.
[Figure]
In this task, the robot pushes a box initialized in front of it.
The target design space is 7 × 7 .
The task-specific observation is formed by concatenating vectors { v box , p box − p robot , v robot } , and the reward R = R 1 + R 2 , where R 1 is:
[Equation]
which rewards the robot and box for moving in the positive x -direction, and R 2 is:
[Equation]
which penalizes the robot and box for separating in the x -direction.
The robot also receives a onetime reward of 1 for reaching the end of the terrain.
The episode duration reaches a 500 timesteps.
[Figure]
In this task, the robot jumps as high as possible in place on flat terrain.
The target design space is 7 × 7 .
The task-specific observation is formed by concatenating vectors { v robot , h robotb (2) } , and the reward R is:
[Equation]
which rewards the robot for moving in the positive y -direction and penalizes the robot for any motion in the x -direction.
The episode duration reaches a 500 timesteps. [Figure]
[Equation] In this task, the robot traverses a series of floating platforms at different heights. The target design space is 5 × 5 . The task-specific observation is formed by concatenating vectors { v robot , θ robot , h robot b (5) } , and the reward R is:
[Equation]
(6) which rewards the robot for moving in the positive x -direction, The robot also receives a one-time penalty of − 3 for rotating more than 90 degrees from its originally orientation in either direction or for falling off the platforms (after which the environment resets).
The episode duration reaches a 1000 timesteps.
[Figure]
In this task, the robot throws a box initialized on top of it.
The target design space is 5 × 5 .
The task-specific observation is formed by concatenating vectors { v box , p box − p robot , v robot } , and the reward R = R 1 + R 2 , where R 1 is:
[Equation]
which rewards the box for moving in the positive x -direction, and R 2 is:
[Equation]
which penalizes the robot for moving too far from x = 0 when throwing the box
The episode duration reaches a 300 timesteps.
A. C LIMBER
In this task, the robot climbs as high as possible through a flat, vertical channel.
The target design space is 5 × 5 .
The task-specific observation is v robot , and the reward R is:
[Equation]
9) which rewards the robot for moving in the positive y -direction.
The robot also receives a one-time reward of 1 for reaching the end of the channel.
The episode duration reaches a 400 timesteps.
A. L IFTER
In this task, the robot lifts a box out of a hole.
The target design space is 7 × 7 .
The task-specific observation is formed by concatenating vectors { p box − p robot , v robot , v box , θ box } , and the reward
R = R 1 + R 2 + R 3 , where R 1 is:
[Equation]
(10) which rewards the robot for moving the beam in the positive y -direction, and R 2 is:
[Equation] [Figure] which penalizes the robot for moving the box away from a goal x -position, g x . This ensures that the robot lifts the box straight up, and R 3 is:
[Equation] which penalizes the robot for falling below a threshold height t y (at which point the robot has fallen into the hole). The episode duration reaches a 300 timesteps.
B A DDITIONAL E XPERIMENTS B. L EARNING C URVES AND M ORPHOLOGY R ESULTS
Figure 15 plots the learning curves of all methods across environments. In Figure 16, we provide visualization results of morphologies constructed using CuCo and baselines. In Figure 17, we demonstrate the development of robot morphologies during CuCo’s learning process in UpStepper and Pusher, respectively . B. T HE C HOICE OF D ESIGN P OLICY
We perform an additional experiment to investigate the importance of using the NCA architecture as CuCo’s design policy. Here, we replace NCA with a Compositional Pattern Producing Network (CPPN) (Stanley, 2007) which takes the global information (e.g., x and y coordinates of the design space) as input and directly generates the robot design parameters. We refer to this method as
CuCo-CPPN . Note that the converged morphology and the design policy are not inheritable in CuCo-CPPN as the input dimension of the CPPN is related to the design space, only the control policy will be inherited.
Figure 18 shows the learning curves of CuCo-NCA and CuCo-CPPN in Pusher and Walker.
Evidently, CuCo-NCA outperforms CuCo-CPPN in terms of convergence speed and final performance. As shown in the video, NCA can encode practical design patterns and successfully expand them to larger design spaces, while CPPN does not significantly facilitate the learning process. [Table]
Walker UpStepper Jumper
[Table] Thrower PlatformJumper Climber
[Table] Lifter
C A TTENTION M ATRICES A NALYSIS
Self-attention brings better interpretability than multilayer perceptron.
In this section, we analyze the attention matrices that CuCo learns.
We use only one transformer encoder layer in this work, thus, we visualize the attention matrix after the input state passes through the attention layer.
Figure 19 demonstrates attention matrices of the critic network during the design process where voxel 9 is assigned with more attention scores.
Figure 20 demonstrates attention matrices of the policy network [Figure] [Figure] 
[Figure]
 [Figure] at two execution steps. The color of each attention score tells the strength of the compatibility between inputs and interprets what is driving the behaviour of the VSR.
When the robot’s front foot V- (voxel 9 ) or the rear foot (voxel 7 ) touches the ground, the corresponding voxels are assigned with greater wights, which is consistent with humans’ intuition and common sense.
D I MPLEMENTATION D ETAILS
We use PyTorch (Paszke et al., 2019) to implement our proposed method.
We took the official implementation of Transformer from Pytorch and added the position embedding.
All hyperparameters for CuCo are listed in Table 2.
For baseline algorithms, we use official implementations of GA,
CPPN-NEAT and BO from the Evolution Gym platform (Bhatia et al., 2021).
We employ a population with the size of 12 for all population-based methods.
Following the convention in the literature, we use multiple CPU threads to accelerate the training of PPO.
For all the environments used in this paper, it takes around 1 day to train our model on a computer with 12 CPU cores and an NVIDIA
RTX 3090 GPU.
[Table]