S OFT Z OO : A S OFT R OBOT C O - DESIGN B ENCHMARK F OR L OCOMOTION I N D IVERSE E NVIRONMENTS
A BSTRACT
While significant research progress has been made in robot learning for control, unique challenges arise when simultaneously co-optimizing morphology.
Existing work has typically been tailored for particular environments or representations.
In order to more fully understand inherent design and performance tradeoffs and accelerate the development of new breeds of soft robots, a comprehensive virtual platform — with well-established tasks, environments, and evaluation metrics — is needed.
In this work, we introduce SoftZoo, a soft robot co-design platform for locomotion in diverse environments.
SoftZoo supports an extensive, naturallyinspired material set, including the ability to simulate environments such as flat ground, desert, wetland, clay, ice, snow, shallow water, and ocean.
Further, it provides a variety of tasks relevant for soft robotics, including fast locomotion, agile turning, and path following, as well as differentiable design representations for morphology and control.
Combined, these elements form a feature-rich platform for analysis and development of soft robot co-design algorithms.
We benchmark prevalent representations and co-design algorithms, and shed light on 1) the interplay between environment, morphology, and behavior 2) the importance of design space representations 3) the ambiguity in muscle formation and controller synthesis and 4) the value of differentiable physics.
We envision that SoftZoo will serve as a standard platform and template an approach toward the development of novel representations and algorithms for co-designing soft robots’ behavioral and morphological intelligence.
Demos are available on our project page 1 .
I NTRODUCTION
The natural world demonstrates morphological and behavioral complexity to a degree unexplored in soft robotics.
A jellyfish’s gently undulating geometry allows it to efficiently travel across large bodies of water; an ostrich’s spring-like feet allow for fast, agile motion over widely varying topography; a chameleon’s feet allows for dexterous climbing up trees and across branches.
Beyond their comparative lack of diversity, soft robots’ designs are rarely computationally optimized in silico for the environments in which they are to be deployed.
The degree of morphological intelligence observed in the natural world would be similarly advantageous in artificial life.
In this paper, we present SoftZoo, a framework for exploring and benchmarking algorithms for codesigning soft robots in behavior and morphology, with emphasis on locomotion tasks.
Unlike pure control or physical design optimization, co-design algorithms co-optimize over a robot’s brain and body simultaneously, finding more efficacious solutions that exploit their rich interplay (Ma et al., 2021; Spielberg et al., 2021; Bhatia et al., 2021).
We have seen examples of integrated morphology and behavior in soft manipulation (Puhlmann et al., 2022), swimming (Katzschmann et al., 2018), flying (Ramezani et al., 2016), and dynamic locomotion (Tang et al., 2020).
Each of these robots was designed manually; algorithms that design such robots, and tools for designing the algorithms that design such robots have the potential to accelerate the invention of diverse and capable robots.
Project Page: https://sites.google.com/view/softzoo-iclr-
SoftZoo decomposes computational soft robot co-design into four elements: design representations (of morphology and control), tasks for which robots are to be optimized (mathematically, reward or objective functions), environments (including the physical models needed to simulate them), and codesign algorithms.
Representationally, SoftZoo provides an unified and flexible interface of robot geometry, body stiffness, and muscle placement that can take operate on common 3D geometric primitives such as point clouds, voxel grids, and meshes.
For benchmarking, SoftZoo includes a variety of dynamic tasks important in robotics, such as fast locomotion, agile turning, and path following.
To study environmentally-driven robot design and motion, SoftZoo supports an extensive, naturally-inspired material set that allows it to not only simulate hyperelastic soft robots, but also emulate ground, desert, wetland, clay, ice, and snow terrains, as well as shallow and deep bodies of water.
SoftZoo provides a differentiable multiphysics engine built atop the material point method (MPM) for simulating these diverse biomes.
Differentiability provides a crucial ingredient for the development of co-design algorithms, which increasingly commonly exploit model-based gradients for efficient design search.
This focus on differentiable multiphysical environments is in contrast to to previous work (Bhatia et al., 2021; Graule et al., 2022) which relied on simplified physical models with limited phenomena and no differentiability; this limited the types of co-design problems and algorithms to which they could be applied.
The combination of differentiable multiphysics simulation with the decomposition of environmentally-driven co-design into its distinct constituent elements (representation, algorithm, environment physics, task) makes SoftZoo particularly well suited to systematically understanding the influence of design representations, physical modeling, and task objectives in the development of soft robot co-design algorithms.
In summary, we contribute:
• A soft robot co-design platform for locomotion in diverse environments with the support of an extensive, naturally-inspired material set, a variety of locomotion tasks, an unified interface for robot design, and a differentiable physics engine.
• Algorithmic benchmarks for various representations and learning algorithms, laying a foundation for studying behavioral and morphological intelligence.
• Analysis of 1) the interplay between environment, morphology, and behavior 2) the importance of design space representations
the ambiguity in muscle formation and controller synthesis 4) the value of differentiable physics, with numerical comparisons of gradient-based and gradient-free design algorithms and intelligible examples of where gradient-based co-design fails.
This analysis provides insight into the efficacy of different aspects of state-of-the-art methods and steer future co-design algorithm development.
[Figure]
S OFT Z OO 2. O VERVIEW
SoftZoo is a soft robot co-design platform for locomotion in diverse environments.
It supports varied, naturally-inspired materials that can construct environments including ground, desert, wetland, clay, ice, shallow water, and ocean.
The task set consists of fast locomotion, agile turning, and path following.
The suite provides a seamless and flexible interface of robot design that specifies robot geometry, body stiffness, and muscle placement, and accepts common 3D representations such as point clouds, voxel grids, or meshes.
The robot can then be controlled according to muscle groups.
The underlying physics engine also supports differentiability that provides model-based gradients, which gains increasing attention in soft robot control and design.
In the following, we walk through high-level components of SoftZoo: simulation engine, environment setup, and locomotion tasks.
2. S IMULATION E NGINE
In this section, we briefly introduce the underlying simulation technique to facilitate later-on discussion.
SoftZoo is implemented using the Taichi library (Hu et al., 2019b), which compiles physical simulation code (and its reverse-mode autodifferentiated gradients) for efficient parallelization on GPU hardware.
Continuum mechanics in SoftZoo follows the discretization of Moving Least
Squares Material Point Method (MLS-MPM)
(Hu et al., 2018), a variant of B-spline MPM (Stomakhin et al., 2013) with improved efficiency.
Multimaterial simulation and signed-distance-functionbased boundary conditions for terrain are implemented to construct diverse environments.
Material Point Method (MPM).
The material point method is a hybrid Eulerian-Lagrangian method, where both Lagrangian particles and Eulerian grid nodes are used to transfer simulation state information back-and-forth.
At a high level, MPM is comprised of three major steps: particleto-grid transfer (P2G), grid operation, and grid-to-particle transfer (G2P).
Material properties, including position x p , velocity v p , mass m p , deformation gradients F p , and affine velocity field C p , are stored in Lagrangian particles that move through space and time.
MPM allows large deformation, automatic treatment of self-collision and fracture, and simple realization of multi-material interaction, and is hence well-suited for soft robots in diverse environments.
We show the governing equations used by MPM and more implementation details in Appendix D.
Contact Model.
The terrain in SoftZoo is represented as meshes that interact with either the robot body or other ground cover materials such as snow or sand.
We employ a grid-based contact treatment (Stomakhin et al., 2013) to handle particle terrain-particle collision, using a Coulomb friction model.
We compute the surface normal of the terrain to measure the signed distance function (SDF) and construct boundary conditions for velocity in grid space.
Multiphysical Materials.
We present a set of environments with multiphysical material support.
The materials cover a diverse set of physical phenomena, including hyperelasticity, plasticity, fluidity, and inter-particle friction.
These phenomena combine to model common real-world materials such as sand, snow, rubber, mud, water, and more.
We list all environments with their corresponding multiphysical materials in Appendix E.
In addition to categorical choices of environments, we also provide a set of elastic constitutive models ( e.g. , St. Venant-Kirchhoff, corotated linear, neoHookean, etc.) and expose their parameters so that the practitioners can easily fine-tune the material behaviors.
In our experiments, we use neo-Hookean material since it a) is known to be accurate for modeling silicone-rubber-like materials common in real-world soft robots, b) skips the costly calculation of singular value decomposition (SVD) and c) results in an improvement in simulation speed and numerical stability and reduces gradient instability caused by singular value degeneracy.
Actuation Model.
We define our actuation model as an anisotropic muscle installed along specified particles.
Each muscular element generates a directional force along a unit fiber vector f .
We realize this actuation model using the anisotropic elastic energy from Ma et al.
(2021) Ψ = s
∥ l − a ∥ 2 , where l = ∥
Ff ∥ 2 , F is the deformation gradient, s is a muscular stiffness parameter, and a is the actuation signal provided by a controller.
This energy is added to the constitutive material energy of particles along which the muscle fiber is installed.
2. E NVIRONMENT S ETUP
Initialization.
Environment construction requires three steps: terrain generation, robot placement, and material covering.
First, we procedurally generate terrain height-maps using Perlin noise with user-defined height range and other parameters for roughness.
The generated height map is converted to a mesh for ground surface.
We then instantiate a robot using specified design parameters.
A user may choose to have the robot placed randomly or at a fixed pre-defined position on the terrain.
Then, we compute an occupancy map in the Eulerian grid in MPM based on particles of robot body and terrain’s SDF.
Finally, we use the terrain’s surface normal to layer particles of specified terrain materials atop it, using the occupancy map to avoid particle placement in non-free space.
[Table]
A co-design algorithm consists of 1) a design optimizer that proposes a robot design at the start of a simulation trial (an “episode” in reinforcement learning terminology) and 2) a control optimizer that specifies a controller that determines robot actuation based on its observed state.
Accordingly, each task interfaces with the algorithm through the robot design interface , observation , action , and reward .
We introduce each element as follows.
Robot Design Interface.
Robot design involves specification of geometry, stiffness, and muscle placement.
We integrate these design specifications into Lagrangian particles in MPM simulation.
Geometry is modeled by mass m p ∈ R (clamping regions of sufficiently low mass to 0 ); stiffness is modeled by a the Young’s modulus in elastic material s p ∈ R .
We remove non-existing (zero-mass) particles to eliminate their effect.
(Low and zero mass particles cause numerical instabilities during simulation.)
For muscle placement r p ∈ R K on a robot with at most K actuators, we augment each actuated particle with a K -dimensional unit-vector where the magnitude of component i specifies the contribution of actuator i to that particle as well as a 3-dimensional unit-vector to specify muscle direction f p ∈ R 3 .
While a robot is represented as a particle set in the simulator, SoftZoo allows common 3D representations other than point clouds such as voxel grids or meshes.
We instantiate a set of particles in a bounding base shape ( e.g. , a box or ellipsoid), which defines the workspace of a robot design with position { x p } , velocity { v p } , and other attributes required in MPM.
For voxel grids, we utilize a voxelizer to transfer voxel occupancy to particle mass that resides in that voxel.
For meshes, we compute SDF for particles in the bounding base shape and assign non-zero mass to sub-zero level set (those with negative signed distance).
Observation.
Robot state observations, which are fed to controllers, are computed at every step in an episode, consisting of robot proprioceptive information, environmental information, and taskrelated information.
For robot proprioceptive information, considering states of all particles in robot body is unrealistic from both the perspective of sensor placement and computational tractability.
Instead, we compute the position and velocity centroids of the robot body, or of pre-defined body parts.
The environment is summarized as a semantic occupancy map ; from the MPM Eulerian grid, a 3D voxel grid is constructed in with each voxel indicates occupancy of terrain (below the
SDF boundary), terrain material particles, or the robot.
Finally, task-related information provides sufficient specifications in order to solve certain task, e.g. , target waypoints to be followed.
Action.
At each time step, the simulator queries the robot controller for an action vector u ∈ R K .
(Note that the time step is at the time scale of a robot controller, different from simulation steps often referred to as substeps.)
We use the same action vector for all simulation steps within an environment time step.
The returned action vector is of length as K , the maximal number of actuators.
The action space bound is set to achieve reasonable robot motion without easily causing numerical fracture of robot body.
We use [ − 0 .
3 , 0 . 3] as coordinate-wise bound in this paper.
Reward.
A robot’s task performance is quantitatively measured by a scalar-valued reward function.
The reward definition is task-specific and computed at every time step to provide feedback signal to the robot.
Please refer to Section 2.4 for detailed descriptions of reward for every tasks.
2. L OCOMOTION T ASKS
We only mention the high-level goal of each task.
Please refer to Section B for more details.
[Figure]
Movement Speed.
In this task, the robot aims to move as fast as possible along a target direction.
Turning.
In this task, the robot is encouraged to turn as fast as possible counter-clockwise about the upward direction of robot’s canonical pose.
Velocity Tracking.
In this task, the robot is required to track a series of timestamped velocities.
Waypoint Following.
In this task, the robot needs to follow a sequence of waypoints.
E XPERIMENTS 3. B IOLOGICALLY - INSPIRED M ORPHOLOGIES
To study the interplay between environment, morphology, and behavior, we first use a set of animal meshes (selected from https://www.davidoreilly.com/library) as robot bodies and label muscle locations.
Through optimizing their controllers, we observe that different designs exhibit betterperforming behavior in different environments.
Muscle Annotation.
We implement a semi-automatic muscle annotator that first converts a mesh into a point cloud, secondly performs K-means clustering with user-defined number of body groups, and finally applies principal component analysis on indepdendent clusters to extract muscle direction.
Users may then fine-tune the resulting muscle placement.
Animals.
We test four animals: Baby Seal , Caterpillar , Fish , and Panda .
These animals are chosen since they exhibit distinct strategies in nature.
We refer the readers to Section K for visualization of each animal-inspired design, including their muscle layouts, and detailed description.
Large-scale Benchmarking.
Table 1 shows the performance of the four animals when optimized for each locomotion task, in each environment.
We parameterize the controller based on a set of sine function bases with different frequencies, phases, and biases (see Section H).
We perform control optimization with differentiable physics and RL (Schulman et al., 2017), corresponding to the left/right values in every entry respectively.
We would like to emphasize that this experiment is not meant to draw a conclusion on superiority of differentiable physics or RL.
Rather, we highlight interesting emergent strategies of different morphologies in response to diverse environments.
A We find that the Baby Seal morphology is particularly well-suited for Ground and Ice , the Caterpillar morphology is well-suited for Wetland and Clay , the Fish morphology is well-suited for Desert and
Snow , and both the Caterpillar and Fish morphologies are well-suited for the Ocean .
Environmentally- and Morphologically-driven Motion.
Based on previous quantitative analysis, in Figure 2, we showcase several interesting emergent motion of different animals.
The Fish surprisingly demonstrates efficient movement on granular materials ( Desert ), by lifting its torso to reduce friction and pushing itself forward with its tail.
The Caterpillar produces a left-to-right undulation motion to locomote in Wetland ; the strategy allows it to disperse mud as it moves forward.
The
Panda ’s legs act in a spring-like fashion to perform galloping motion on flat terrain.
Finally, the
Baby Seal uses its tail muscle to create a flapping motion to move on the low-friction Ice terrain, where push-off would otherwise be difficult.
[Figure] [Figure]
3. T HE I MPORTANCE O F D ESIGN S PACE R EPRESENTATION
In this section
[Figure]
we demonstrate the importance of the design space representation for morphology optimization.
We apply gradient-based optimization methods across a wide variety of morphological design representations fixing the controller.
We focus on achieving fast movement speed in the
Ocean environment since aquatic creatures tend to manifest a more canonical muscle distribution, typically antagonistic muscle pairs along each side of the body.
The low failure rate of this task means that we achieve a wide spread of performances, determined by the design representation, which allows us to compare the effect of each on performance.
Baselines.
We implement a variety of design space representations.
We briefly mention the highlevel concept of each method as follows.
Please refer to Section I for more in-depth description.
Particle Representation.
The geometry, stiffness, and muscle placement is directly specified at the particle level, with each particle possessing its own distinct parameterization.
Voxel Representation.
This is similar to particle representation but specified at the voxel level.
Implicit Function (Mescheder et al., 2019).
We use a shared multi-layer perceptron that takes in particle coordinates and outputs the design specification for the corresponding particle.
Diff-CPPN (Fernando et al., 2016).
We adapt the differentiable CPPN, which provides a global mapping that converts particle coordinates to a design specification.
SDF-Lerp.
Given a set of design primitives (with design specification obtained by using the technique in Section 3.1), we compute SDF based on each design primitives for particles in robot design workspace.
For each particle, we then linearly interpolate the signed distances and set occupancy for those with negative values to obtain robot geometry.
We directly perform linear interpolation on stiffness and muscle placement of the primitive set.
For muscle direction, we use weighted rotation averaging with special orthogonal Procrustes orthonormalization (Br´egier, 2021).
Wass-Barycenter (Ma et al., 2021).
We compute Wasserstein barycenter coordinates based on a set of coefficients to obtain robot geometry.
We follow SDF-Lerp for stiffness and muscle placement.
Design Optimization.
In Table 2, we show the results of design optimization.
We can roughly categorize these design space representations into (1) no structural prior ( Particle and Voxel ) (2) preference for spatial smoothness ( Implicit Function and Diff-CPPN ) and (3) highly-structured design basis ( SDF-Lerp and Wass-Barycenter ).
We observe superior performance as increasing inductive prior is injected into the representation.
The use of design primitives casts the problem to composition of existing functional building blocks and greatly reduces the search space, leading to
[Figure]
effective optimization.
Further, structural priors, lead to designs with fewer thin or sharply changing features; such extreme designs have unwanted artifacts such as numerical fracture during motion.
Figure 3 presents optimized designs: meshes indicate geometry and colored point clouds indicate active muscle groups.
These results suggest the development of novel representation for robot design that facilitates improved optimization and learning.
3. A MBIGUITY B ETWEEN M USCLE S TIFFNESS A ND A CTUATION S IGNAL S TRENGTH
In this section, we investigate the ambiguity between muscle formation and actuation from the controller.
We aim to discover if there exists unidentifiability between design and control optimization.
We adopt trajectory optimization for control in this set of experiments since it has the best flexibility.
Stiffness as “Static” Actuation.
Here, we explore whether optimizing active actuation can reproduce motion of robots with different static muscle stiffness.
With a fixed robot geometry and muscle placement, we randomly sample 100 sets of controllers and muscle stiffness, and record trajectories of all particles for 100 frames.
We then try to fit a controller for robots with a different set of stiffness and measure how well they can match the last frame of the pre-collected dataset.
We use the
Earth Mover’s Distance (EMD) loss (Achlioptas et al., 2018) and differentiable physics for training.
Specifically, we use EMD to compare the difference between the motions of each particle set at the end of each trajectory.
In Figure 4a, we show matching error distribution across different stiffness multipliers (with respect to the original stiffness of the robot).
This experiment indicates we can roughly replicate the motion of robot with different stiffness by control optimization.
This aligns with the muscle model, where we can interpret stiffness as a static component of actuation.
Approximating Motion of Random Muscle Placement.
Here, we investigate if we can reproduce the motions of robots with different muscle placements by optimizing their controllers.
With fixed robot geometry and stiffness, we randomly sample 6 sets of controllers and muscle placements with 3 actuators.
We follow the same training procedure mentioned previously to fit controllers for robots with a different set of muscle placement.
In Figure 4b, we show relative matching error at different number of actuators with shaded area as confidence interval from the 6 random seeds.
Note that we adopt a fixed muscle direction and soft muscle placement for simplicity.
The plot suggests that with redundant ( > 3 in this case) actuators, robot motion can be roughly reproduced with control optimization even under different muscle placement.
Overall, we demonstrate the ambiguity between muscle construction and actuation that may induce challenges in co-design optimization.
However, these results also unveil the potential of casting muscle optimization to control optimization in soft robot co-design.
3. T HE G OOD AND B AD OF D IFFERENTIABLE P HYSICS FOR S OFT R OBOT C O - DESIGN
The Value of Differentiable Physics.
Gradient-based control optimization methods powered by differentiable simulation, are increasingly popular in computational soft robotics.
Such optimizers can efficiently search for optimal solutions and decrease the number of computationally-intensive simulation episodes needed to achieve optimal results in various computational soft robotics problems compared to model-free approaches such as evolutionary strategies or RL.
Despite its effectiveness, the local nature of gradient-based methods poses issues for optimization.
While recent research has rigorously studied differentiable physics in control (Suh et al., 2022), similar investigation is lacking in the context of design.
Here, we take a preliminary step to analyze gradient for design optimization.
In Figure 5a, we show the loss landscape of a single parameter of actuator placement.
We consider a voxel grid and smoothly (with soft actuator placement) transition the membership of the
[Table] [Table]
E v o l u t i o n
S t r a t e g y
D i ff .
P h
y s i
c s
[Figure]
actuator in a voxel.
We can observe multiple local optima, which may trap gradient-based optimization.
In Figure 5b, we show the discontinuity of loss resulted from shape changes.
A box with a missing corner (red) ends up at a very different location from that of a complete box (gray), as it topples when placed in metastable configurations.
These results present unique yet understandable challenges for differentiable physics in robot design, which suggests more future research to better leverage gradient information.
3. C O -O PTIMIZING S OFT R OBOTIC S WIMMERS
In this section, we demonstrate co-design results for a swimmer in the Ocean environment.
First, in Table 3, we compare between (1) optimizing control only (2) optimizing design only (3) cooptimizing design and control.
We use SDF-Lerp as design space representation and differentiable physics for optimization.
To generate meaningful results, for the control-only case, we use a fishlike design; and for the design-only case, we use a controller trained for a fish not included in the design primitive of SDF-Lerp .
The outcome verifies the effectiveness of generating superior soft robotic swimmer with co-design.
In Table 4, we compare co-design results from evolution strategy (ES) and differentiable physics.
We follow the above-mentioned setup for differentiable physics.
For evolution strategy, we adopt a common baseline: CPPN representation (Stanley, 2007) with HyperNEAT (Stanley et al., 2009); to make control optimization consistent with evolution strategy, we use CMA-ES (Hansen et al., 2003).
Furthermore, we show the optimized design and the corresponding motion sequence of both methods in Figure 6.
Interestingly, the better-performing codesign result much resembles the results of performing design optimization only, shown in Figure 3.
R ELATED W ORK
Co-Design of Soft Bodied Machines.
Evolutionary algorithms (EAs) have been used to design virtual agents since the pioneering work of Sims (1994); as these algorithms improved the problems they could be applied to grew in complexity.
Cheney et al.
(2014b) demonstrated EAs for codesigning soft robots over shape, materials, and actuation for open-loop cyclic controllers; follow-on work explored soft robots with circuitry (Cheney et al., 2014a) and those grown from virtual cells (Joachimczak et al., 2016).
Evolutionarily designed soft robots were demonstrated to be physically manufacturable using soft foams (Hiller & Lipson, 2011), and biological cells (Kriegman et al., 2020).
Virtually all such results have been powered by the neuroevolution of augmenting topologies (NEAT) algorithm (Stanley & Miikkulainen, 2002), and, typically, compositional pattern producing networks (CPPN)
(Stanley, 2007) for morphological design representation.
Recent work has explored other effectual heuristic search algorithms, such as simulated annealing, for optimizing locomoting soft robots represented by grammars (Van Diepen & Shea, 2019; van Diepen & Shea, 2022).
Recently, Bhatia et al.
(2021) provided an expansive suite of benchmark tasks for evaluating algorithms for co-designing soft robots with closed-loop controllers.
That work provides a baseline co-design method that employs CPPN-NEAT for morphological search and reinforcement learning for control optimization (Schulman et al., 2017).
Similar to (Bhatia et al., 2021), our work benchmarks EAs alongside competing methods.
We provide a suite of environmentally-diverse tasks in a differentiable simulation environment, which allows the use of efficient gradient-based search algorithms.
Further, we focus on a systematic decoupling of design space representation, control, task and environment, and search algorithms in order to distill the influence of each.
Differentiable Simulation for Soft Robot Optimization.
A differentiable simulator is one in which the gradient of any measurement of the system can be analytically computed with respect to any variable of the system, which can include behavioral (control) and physical (morphological and environmental) parameters.
Differentiability provides particular value for soft robotics; gradient-based optimization algorithms can reduce the number of (typically expensive) simulations needed to solve computational control and design problems.
Differentiable material point method simulation (MPM) has been used in co-optimizing soft robots over closed-loop controllers and spatially varying material parameters (Hu et al., 2019c; Spielberg et al., 2021), as well as proprioceptive models Spielberg et al.
Co-optimization procedures rooted in differentiable simulation have also been applied to more traditional finite element representations of soft robots.
Notably, (Ma et al., 2021) demonstrated efficient co-optimization of swimming soft robots’ geometry, actuators, and control, incorporating a learned (differentiable) Wasserstein basis for tractable search over high-dimensional morphological design spaces.
Follow-on work further showed that differentiable simulators can be combined with learned dynamics for co-design (Nava et al., 2022).
Such finite-element-based representations, however, have struggled with smoothly differentiating through rich contact.
Our work presents a rich comparison of differentiable and non-differentiable approaches for softbodied co-design, as well as a set of baseline methods.
We present a differentiable soft-bodied simulation environment capable of modeling multiphysical materials.
Our environment is based in
MPM, because of its ability to naturally and differentiably handle contact and multiphysical coupling.
Similar to other recent work, such as Suh et al.
and Huang et al.
(2021), we perform a thorough analysis (in our case, empirical) of the value and limitations of differentiable simulation environments, with an emphasis on cyberphysical co-design.
Environmentally-Driven Computational Agent Design.
Similar to their biological counterparts, virtual creatures have different optimized forms depending on their environment.
Auerbach & Bongard (2014) and Corucci et al.
(2017) analyzed real-world biological morphological features from the perspective of EAs; Cheney et al.
(2015) studied the interplay between of virtual terrain and robot geometry for soft-bodied locomotion locomotion.
In rigid robotics where simulation is less computationally expensive, online data-driven neural generative design methods are emerging.
These include methods that reason over parameterized geometries Ha (2019), topological structure Zhao et al.
Xu et al.
; Hu et al.
and shapeshifting behavior Pathak et al.
These methods currently require too much simulation-based data generation for soft robotics applications, but are useful as inspiration for future research directions.
C ONCLUSION
In this work, we introduced SoftZoo, a soft robot co-design platform for locomotion in diverse environments.
By using SoftZoo to investigate the interplay between design representations, environments, tasks, and co-design algorithms, we have found that 1) emergent motions are driven by environmental and morphological factors in a way that often mirrors nature 2) injecting design priors in design space representation can improve optimization effectiveness 3) muscle stiffness and controllers introduce redudnancy in design spaces, and 4) trapping local minima are common even in very simple morphological design problems.
Impact and Future Direction.
SoftZoo provides a well-established platform that allows for systematic training and evaluation of soft robot co-design algorithms; we hope it will accelerate algorithmic co-design development.
It also lays a cornerstone for studying morphological and behavioral intelligence under diverse environments.
The extensive experiments conducted using SoftZoo not only sheds light on the significance of co-design but also identifies concrete challenges from various perspectives.
Based on our findings, we suggest several future directions: 1) 3D representation learning to construct more effective and flexible design space representations 2) morphology-aware policy learning as an alternative formulation that more elegantly handles the inter-dependency between design and control optimization 3) principled approaches to combine differentiable physics and gradient-free methods like RL or EAs that marries priors from physics to optimization techniques less susceptible to pathological loss landscapes.
Overall, we believe SoftZoo paves a road to study morphological and behavioral intelligence, and bridges soft robot co-design with a variety of related research topics.
A N OTATIONS
Table A shows a list of notations used in the description of SoftZoo, where the subscript p indicates association with particles.
Note that actuation is defined per particle and action vector is defined as the output of controllers (which can be very low-dimensional such as R 10 ), i.e., a p = u · f p .
[Table]
B L OCOMOTION T ASKS B. M OVEMENT S PEED
In this task, the goal of the robot is to move as fast as possible along a pre-defined direction.
We compute the average velocity of all existing (non-zero-mass) particles on robot body and project it to the target direction to obtain a scalar estimate.
We thus write the metric per timestep as,
[Equation]
where m p is mass, v p is velocity, and t is a pre-defined target direction.
B. T URNING
In this task, the robot is encouraged to turn as fast as possible counter-clockwise about the upward direction of robot’s canonical pose.
We first compute the relative position of all particles on robot body with respect to its center of mass.
We then compute the cross product of the upward direction followed by unit-vector normalization to obtain tangential directions for all particle while turning.
Finally, we compute velocity projection of every particle along the tangent and return the average as the measure of turning performance.
We thus write the metric per timestep as,
[Equation]
where m p is mass, v p is velocity, and d is the up direction (defined manually based on robot canonical forward direction).
B. V ELOCITY T RACKING
In this task, the robot is required to track a series of timestamped velocities.
We use quintic polynomials formulated as a function of time for path parameterization since it generates smooth trajectories with easy access to different orders of derivative.
A path can be fully specified with start and target position x s , x t , velocity v s , v t , and acceleration a s , a t .
We can then obtain the quintic polynomial as,
[Equation]
where t is time and c 0 , c 1 , c 2 , c 3 , c 4 , c 5 ∈ R 3 are coefficients.
After setting a target state, we query the first-order derivative of the polynomial at each environment time step as the target velocity,
[Equation]
We write target velocity at a time instance as v target for the following.
Furthermore, due to the deforming nature of soft robot, we need to estimate the heading of the robot in order compare it with the target velocity.
We manually label particles corresponding to a head and a tail of the robot a priori p head , p tail based on its canonical heading direction.
[Equation]
During robot motion, we extract rotation from deformation gradient of these particles and inversely transform them back to material space to compute heading direction.
[Equation]
where F p is deformation gradient, U p , R Tp are the results of polar decomposition.
Velocities of all particles on robot body are then projected to the heading direction and averaged in order to compare to the target velocity.
We separate the measurement of magnitude and direction as this allows different weighting of the two terms.
We can then write the metric per timestep as,
[Equation]
where r
[Equation]
p = − ( |
[Equation]
|
[Equation]
proj
[Equation] [Equation]
where α mag , α dir are coefficients of magnitude and direction term respectively and we set α mag
= 0 .and
α dir = 0 .
We put more emphasis on the alignment of direction since it better indicates maneuverability.
B. W AYPOINT F OLLOWING
In this task, the robot needs to follow a sequence of waypoints.
The waypoints are generated by the above-mentioned quintic polynomial method x path ( t ) .
We compute root mean squared error between robot center of mass and the target waypoint for each time step.
We denote target position at a time instance as x target .
We can then write the metric per timestep as,
[Equation]
Remark that well-performing velocity tracking can induce large waypoint following error but trace out similar-shaped yet different-scaled trajectories.
Both serve a role as distinct aspects of evaluating path following.
C P LATFORM COMPARISON
A comprehensive comparison to existing soft robot platform is shown in Table 5.
D C ONTINUUM M ECHANICS S IMULATION
We formulate the continuum mechanics simulation in the framework of the moving least squares material point method (MLP-MPM)
(Hu et al., 2018), whose governing equations are characterized by:
[Equation]
[Table] [Table]
where ρ is the density of the material, v is the velocity, σ is the Cauchy stress of the energy [Table] and f ext is the external forces applied, which is the gravity in our cases.
We solve these equations for an equilibrium between different materials coupled in our environments.
We will not dive further into continuum mechanics and point the interested reader to Gonzalez & Stuart (2008) for more details.
In terms of the implementation of the differentiable physics-based simulation, we massively use DiffTaichi (Hu et al., 2019a) as the backbone.
E M ULTIPHYSICAL M ATERIALS
For result validation and visual entertainment, we present a diverse set of environments spanned by different material setups and tasks.
Here we illustrate the materials that each environment covered in Table 6.
Note that even though Desert , Clay , and Snow share the same composition of material types, we distinguish their elastoplasticity by imposing different parameters and models ( e
[Table]
g. , friction cone).
We will release the code-level implementation of all materials for reproducibility.
F G RADIENT C HECKPOINTING
Simulating environments like ocean, desert, etc requires a significant amount of particles.
This poses a challenge in differentiation as the computation graph needs to be cached for backward pass, leading to considerably high memory usage.
Accordingly, we implement gradient checkpointing that allows very large-scale simulation with gradient computation.
Instead of caching simulation state at every single step, we only store data every N steps.
When doing backward pass at i × N steps, we perform recomputation of forward pass from ( i − 1)
× N to i × N to reconstruct the computation graph in-between for reverse-mode automatic differentiation.
G O PTIMIZATION
In this section, we describe the implementation details of each optimization method.
G. D IFFERENTIABLE P HYSICS
Model-based gradient provides much accurate searching direction and thus considerably more efficient optimization.
However, gradient information is susceptible to local optimum and often leads to bad convergence without proper initialization.
Hence, we adopt a simple yet effective approach that samples 8 random seeds, performs optimization with differentiable physics for all runs, and picks the best result.
More formally we write as,
[Equation]
where θ are the model parameters, θ 0 is the initialization of a model, L is a cost function, F summarizes the optimization that follows the gradient update θ k +
=
θ k − ηh ( ∇L ) , η is the learning rate, and h is a function that specifies different gradient descent variants.
For the large-scale benchmark with biologically-inspired design (Table 1), we use learning rate 0 . 1 and training iterations 30 .
For all control-only and design-only optimization, we use learning rate 0 . 01 and training iterations 100 .
For co-design, we use learning rate 0 .
01 for both control and design with training iterations 250 .
We use Adam as the optimizer.
G. R EINFORCEMENT L EARNING
RL is only used in control optimization.
We use Proximal Policy Optimization (PPO)
(Schulman et al., 2017) with the following hyperparameters: number of timesteps 10 5 , buffer size 2048 , batch size 32 , GAE coefficient 0 .
95 , discounting factor 0 .
98 , number of epochs 20 , entropy coefficient 0 .
001 , learning rate 0 . 0001 , clip range 0 .
We use the same controller parameterization as all other experiments throughout the paper.
G. E VOLUTION S TRATEGY
We implement a fully-ES-based method as a co-design baseline.
The genome fitness function is set as the episode reward of the environment.
We pose a constraint on connected component of robot body.
For CPPN, we use a set of activation functions including sigmoid , tanh , sin , gaussian , selu , abs , log , exp .
The inputs of CPPN include x, y, z coordinates along with distance along xy, xz, yz planes and radius from the body center.
We use HyperNEAT (Stanley et al., 2009) for design optimization and CMA-ES (Hansen et al., 2003) for control optimization with initial standard deviation as 0 . 1 .
We don’t use an inner-outer-loop scheme for co-design.
Instead, HyperNEAT and
CMA-ES share the same set of population with population size as 10 .
We run ES for 100 generations for the co-design baseline.
H C ONTROLLER P ARAMETERIZATION
Locomotion often exhibits cyclic motion and thus control optimization can significantly benefit from considering periodic functions in controller parameterization.
Specifically, we use a set of sine functions with different frequency and phase (offset) as bases.
The controller is hence parameterized with a set of weights on these bases along with bias terms.
[Equation]
where ϕ ij are the bases, α ij and β ij are learnable weights and biases, x is controller inputs, ω i ( x ) , φ j ( x ) can be either learnable (as neural network) or pre-defined.
The control signal is then modulated by a tanh function multiplied by a pre-defined constant to ensure satisfication of control bound.
We use 4 different phases with frequency 20 and 80 rad/s throughout the paper.
While all experiments presented are not confined to using this sinewave basis controller, we empirically found it extremely efficient to generate reasonable results in comparison to other controller parameterizations like a generic neural network or trajectory optimization.
I D ESIGN S PACE R EPRESENTATION
In this section, we provide more implementation details of design space representations.
We first recall the notation of robot design interface.
m p ∈ R is geometry modeled by mass.
s p ∈ R is stiffness.
r p ∈ R K is muscle placement (muscle group assignment) with K as the maximal number of actuators/muscles.
f p ∈ R 3 is muscle direction represented in Euler angle.
τ m is the cutoff threshold of excluding low mass regions for numerical stability.
Note that the subscript p indicates attributes associated with a particle.
MLP a (cid:41) b ( · ; θ ) denotes a multi-layer perceptron with a inputs, b outputs, and θ as parameters (we omit hidden layers for clean notation and the output layer is linear).
m 0 , s 0 ∈ R are pre-defined and constant reference mass and stiffness respectively.
Geometry of all methods are processed by the following formula for numerical stability,
[Equation]
where 1 · is an indicator function.
ˆ x p ∈ R 3 is a centered and standardized position x p ∈ R 3 .
We denote the position of a base particle set { ˆ x p } as a point cloud representation that span the robot design workspace, which is a natural representation of continuum material in MPM.
Next, we describe each design space representation individually.
I. P ARTICLE - BASED R EPRESENTATION .
Given a base particle set { ˆ x p } , we instantiate two trainable scalars followed by sigmoid for geometry and stiffness, and a K -dimensional vector followed by softmax for muscle placement with a fixed muscle direction along the canonical heading direction.
[Equation]
where { ˜ m p ∈ R , ˜ s p ∈ R , ˜ r p ∈ R K } are learnable parameters associated with ˆ x p , and f (0) p is the fixed muscle direction along the canonical heading direction.
I. V OXEL - BASED R EPRESENTATION .
We voxelize the given base particle set to obtain a voxel grid and follows similar modelling technique to particle-based representation in voxel level.
[Equation]
where V2P is a mapping from voxel space to particle space (e.g., suppose we use a 2 × 2 ×voxel grid to represent 10 3 particles, we may associate the voxel coordinate (0 , 0 , 0) to first 5particles), { ˜ m v ∈ R , ˜ s v ∈ R , ˜ r v ∈ R K } are learnable parameters, and the muscle direction is of the same definition as particle-based representation f VBR p = f PBR p .
In comparison to particle-based representations, in practice the voxel-based representations have much fewer parameters that need to be learned.
I. I MPLICIT F UNCTION
We extend the idea of OccupancyNet (Mescheder et al., 2019) to predicting robot geometry, stiffness, and muscle placement.
It is modeled by a multi-layer perceptron (MLP) with 2 layers anddimensions for each.
We use tanh activation.
The MLP takes in x, y, z coordinates, distance along xy, xz, yz planes and radius from the body center.
The network outputs occupancy as geometry using sigmoid, stiffness multiplier using sigmoid, and a K -dimensional vector using softmax for muscle placement.
We describe in formula as,
[Equation]
where { θ m , θ s , θ r } are learnable parameters formulated as neural networks, and the muscle direction is of the same definition as particle-based representation f IF p = f PBR p .
This representation has inherent spatial continuity due to the use of smooth functions MLP with spatial coordinate ˆ x p as inputs.
I. D IFF -CPPN
Diff-CPPN is a differentiable version of Compositional Pattern Producing Networks (CPPN)
(Stanley, 2007), following similar concept in (Fernando et al., 2016).
CPPN is a graphical model
G = {N , E} composed of a set of activation functions Σ with interesting geometric properties (e.g., sine, tanh) that takes in particle or voxel coordinates and output occupancy or other properties.
Each node n i ∈ N has a set of input edges e i ∈
E that can be changed by evolution, and an activation function σ i ∈ (cid:80) .
The input-output relationship of a layer can be then written as,
[Equation]
It is originally designed to be optimized with varying graph topologies.
We use a meta graph to allow gradient flow and mimic the augmenting topolgies process in NEAT yet in a differentiable manner, i.e., the varying topology w j ∈ { 0 , 1 } is softened to w j ∈ R .
Remark the similarity of the above construction with a layer in regular MLP except for varying activation function across neurons.
We then can define,
[Equation]
where each layer of MLP CPPN follows the aforementioned input-output relationship, { θ m , θ s , θ r } are learnable parameters, and g ( · ) is a function of expanding position to additional spatial coordinates such as distance to the center (which is normally used in CPPN).
We use a fixed muscle direction.
The model takes in x, y, z coordinates, distance along xy, xz, yz planes and radius from the body center, and outputs occupancy as geometry using sigmoid, stiffness multiplier using sigmoid, and a K -dimensional vector using softmax for muscle placement.
We use sin and sigmoid activation functions with 3 hidden layers and 20 graph nodes in each layer.
I. SDF-L ERP
Given a base particle set { ˆ x p } , we compute the linear interpolation among a set of pre-defined design primitives { Ψ i } Ni =1 , where N is number of design primitives.
The shape of the robot design m p is then determined by a set of coefficients weighting the SDFs from design primitives.
In other words, the trainable parameters for robot geometry only construct a N -dimensional vector.
We then compute weighted sum of the SDF bases and extract robot body with the final SDF smaller or equal to zero.
We use a low-temperature sigmoid in implementation to keep gradient flow.
For stiffness s p , we can directly perform linear interpolation.
For muscle group membership r p , we use linear interpolation upon the one-hot vectors from design primitives and effectively realize a soft muscle group assignment.
For muscle direction f p , we adopt interpolation designed for rotation matrices (Br ´ egier, 2021).
Formally,
[Equation]
where { α ∈ R N , β ∈ R N , γ ∈ R N , κ ∈ R N } are learnable coefficients of the interpolation, Ψ SDF i , Ψ si , Ψ r i ,
Ψ f i are the SDF, stiffness, muscle placement, and muscle direction of a design primitive respectively, T is temperature (where we set to − 1000 to mimic SDF ≤ 0 ), E2M is conversion from an Euler angle to a rotation matrix, M2E is conversion from a rotation matrix to an Euler angle, || · || F is Frobenius norm.
The computation of muscle direction f SDF-Lerp p follows rotation
[Table]
matrix interpolation using special Procrustes method, detailed in (Br´egier, 2021).
In comparison to the above methods, SDF-Lerp leverages prior knowledge of design primitives Ψ i that can provide more structure to the optimization from design space representation.
I. W ASSERSTEIN B ARYCENTER
This method also uses design primitives { Ψ i } Ni =1 .
First, it adopts the same approach for stiffness and muscle placement as SDF-Lerp , s Wass p
[Equation]
We use a fixed muscle direction along the canonical heading direction.
The major difference is the way to represent robot geometry.
Following (Ma et al., 2021), we define a probability simplex (i.e., a set of coefficients with length as the number of design primitives) that serves as a weighting in the sense of Wasserstein distance among different shapes.
It can be written as,
[Equation]
where P α is the Wasserstein barycenter with coefficients α (intuitively, with some abuse of notation, we can view it as probability density function that determines the particle existence), defined as,
[Equation]
where W 2 is 2-Wasserstein distance and Ψ i is shape primitive.
This representation better preserves the volume from the shape of design primitives.
We refer the reader to the original paper for more details.
J F ULL RL R ESULTS
In Table 1, we show large-scale benchmark of biologically-inspired designs in SoftZoo.
The comparison includes optimization using differentiable physics and RL.
We use 5 random seeds for RL results yet only report average performance due to space limit.
In Table 7, we report the full RL results with both mean and standard deviation among all seeds.
K V ISUALIZATION OF B IOLOGICALLY - INSPIRED D ESIGN
In this section, we demonstrate the biologically-inspired designs used in this paper.
In Figure 7, we show the four animals used in the main paper, including Baby Seal , Caterpillar , Fish , and Panda .
In
Figure 8, we show the set of design primitives used in SDF-Lerp and Wasserstein Barycenter .
Baby Seal
Fish
Caterpillar
Panda
[Figure]
Orca
Shark