Paper: DittoGym: Learning to Control Soft Shape-Shifting Robots (ICLR-2024)

Reviewer comment (Experiments):
It is unclear whether the proposed simulator and benchmark is a good model of reality. Is it modeling a specific robot or class of robots? Could there be some experiments to show the validity of the simulator? I'm not necessarily referring to simulation to reality transfer, but rather to show that the simulator dynamics are correlated with the dynamics of a real robot.

Author response:
Thank you for pointing out the real world connection! We took great caution throughout the project to make sure everything is governed by real-world physical equations. Our physical simulation is based on MPM, a widely used simulation algorithm that’s designed to simulate real world materials like liquids and plastics. Our robot material is simulated in a way governed by equations of the real stress-strain curve of plastic-elasto materials. The actuation is governed by that of magnetic field (which is used to control the first real-world highly reconfigurable robot [26]), and follows the original implementation in Taichi that simulates point gravity attraction. The only change we made is to normalize the actuation to follow conservation of momentum because we are parameterizing internal forces only. This is done before feeding actuation into simulation and doesn’t break any physical equation itself.

Reviewer comment (Novelty):
The proposed hierarchical reinforcement learning approach is relatively standard. Its application to the problem of morphology-changing robots might be new, though.

Author response:
Thanks for mentioning HRL, a nice potential addition to our related work. While our method has a spatial resolution hierarchy, it’s quite different from the hierarchy in HRL. In HRL, high-level policy and low-level policies are designed to be disentangled, such that they never share the input-output behavior. Rather, the high-level policy’s output is usually fed to low-level policy as input. For example, The famous D4RL [27] environment “ant maze” is widely used in HRL [28][29], where a high-level policy outputs local goals and low-level policy outputs actuations. 

In contrast, our policy of different resolutions shares the same 2D input and output space. The action field of high-resolution policy and that of low-resolution are directly added algebraically (The weighting mask and feed of coarse action into fine policy are optional). This is because of our hierarchy in the sense of spatial hierarchy, like CNNs and Feature Pyramids in computer vision, rather than the hierarchy in HRL. We appreciate the relation you pointed out and would add explanations to our paper upon acceptance.

Reviewer comment (Experiments):
I don't understand the reasoning behind the proposed baselines. There seem to be other approaches that are more related to the problem (e.g. Phatak et al., Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity; Whitman et al., Learning Modular Robot Control Policies). Having a more detailed comparison could give a better idea of how difficult is the proposed benchmark for current algorithms.

Author response:
Thanks for bringing up these baselines. We’d like to clarify that suggested baselines aren’t directly applicable here while also providing results of the adopted version of suggested baseline [30].

* Clarification: highly-reconfigurable robots lack articulated joints, links and thus lack of explicit topological structure these baselines need, as we highlighted in general response. 

* **Additional experiments**: Despite that, we adopt the suggested Modular Policy baseline[28] by assigning topology with k-means clustering so GNN can be applied. We provide an additional baseline by letting an attention mechanism to figure out topology itself. We benchmark the baselines on 4 of the 8 tasks due to time constraints with the same number of seeds.

Details of both baselines and result curves are added to our [rebuttal website](https://sites.google.com/view/morphologicalmazerebuttal). Figure 1.1 in the website illustrates the cluster while figure 1.2 shows our method’s performance is still much higher than the added baselines. This is not surprising, as GNN is only suited for robots with clear articulation, which slime robots lack.

Reviewer comment (Experiments):
Why are these tasks selected? What exactly is it that they evaluate? Why is it important to have them instead of others?

Author response:
Our task selection follows both generic RL benchmark design principles and the specific needs of reconfigurable robots. We highlight the factors we hoped to cover in our benchmark and the corresponding factor vector of each environment:
* Tasks have to cover a range of difficulties (A=easy, B=hard)
* Some tasks evaluates robot itself, others requires the robot’s ability to interact with external objects (A=not require, B=require)
* Some tasks require just one morphology change, while others need changes of morphology multiple times (A=one change, B=multiple changes)
* Some have non-convex reward landscape that requires algorithm to be long-horizon (A=short-horizon, B=long-horizon)
* Some features softer material and some features less soft material (A=softer, B=less softer)

  | Task        | Factor Vector     |
  |-------------|-------------------|
  | SHAPE_MATCH | [A, A, A, A, A]   |
  | RUN         | [A, A, A, A, B]   |
  | GROW        | [A, A, A, A, A]   |
  | KICK        | [B, B, A, A, B]   |
  | DIG         | [B, B, A, A, B]   |
  | OBSTACLE    | [B, A, B, B, B]   |
  | CATCH       | [B, B, B, B, A]   |
  | SLOT        | [B, B, B, B, B]   |


	
Each task is designed to represent certain factor combinations that we deem important, and we believe they together can give users a comprehensive understanding that a lot of RL ablations do through choices of tasks.

We agree that more benchmarks would be helpful! However, our problem is the first of its kind. Reconfigurable robot benchmarks are not possible before us due to the lack of formalization and difficulty to make physical simulations, which is our main contribution. While we only have 8 environments in our paper, these two contributions make it much easier for the RL community to contribute more reconfigurable robot tasks. We plan to gradually add more tasks to the benchmark to achieve higher impact in the long term but we also believe that quality of benchmark is more significant in a first benchmark.

Reviewer comment (Novelty):
I am not sure I understand how the baseline of Neural Field Policy is implemented and whether this is a novel contribution of the paper or some other work proposed before for this problem.

Author response:
Sorry for the confusion. Neural Field Policy is designed to be a method that can take advantage of our infinitely dimensional action space by querying values in a coordinate-conditioned way. We recognize the confusion it caused, and will replace it in the camera-ready version with the two added baselines following your suggestion.

Reviewer comment (Presentation):
I don't think Fig. 6 is helpful, the concept it represents is trivial and easily explained in a sentence.

Author response:
Thank you for your suggestion! While the concept is simple, Fig. 6 is also designed to be a **visualization** of the resulting shape of different policies, just like how RL paper also demonstrates suboptimal runs of baselines in their videos to give viewers more insights. We will add more visualizations of different tasks in the camera-ready version.

Reviewer comment (Experiments):
It would be great if the authors could clarify why the proposed simulator is a good model of real robots, why the proposed tasks were selected, and clarify the experimental setup. In addition, it would be great to include more relevant benchmarks for this problem.

Author response:
Our task selection follows both generic RL benchmark design principles and the specific needs of reconfigurable robots. We highlight the factors we hoped to cover in our benchmark and the corresponding factor vector of each environment:
* Tasks have to cover a range of difficulties (A=easy, B=hard)
* Some tasks evaluates robot itself, others requires the robot’s ability to interact with external objects (A=not require, B=require)
* Some tasks require just one morphology change, while others need changes of morphology multiple times (A=one change, B=multiple changes)
* Some have non-convex reward landscape that requires algorithm to be long-horizon (A=short-horizon, B=long-horizon)
* Some features softer material and some features less soft material (A=softer, B=less softer)

  | Task        | Factor Vector     |
  |-------------|-------------------|
  | SHAPE_MATCH | [A, A, A, A, A]   |
  | RUN         | [A, A, A, A, B]   |
  | GROW        | [A, A, A, A, A]   |
  | KICK        | [B, B, A, A, B]   |
  | DIG         | [B, B, A, A, B]   |
  | OBSTACLE    | [B, A, B, B, B]   |
  | CATCH       | [B, B, B, B, A]   |
  | SLOT        | [B, B, B, B, B]   |


	
Each task is designed to represent certain factor combinations that we deem important, and we believe they together can give users a comprehensive understanding that a lot of RL ablations do through choices of tasks.

We agree that more benchmarks would be helpful! However, our problem is the first of its kind. Reconfigurable robot benchmarks are not possible before us due to the lack of formalization and difficulty to make physical simulations, which is our main contribution. While we only have 8 environments in our paper, these two contributions make it much easier for the RL community to contribute more reconfigurable robot tasks. We plan to gradually add more tasks to the benchmark to achieve higher impact in the long term but we also believe that quality of benchmark is more significant in a first benchmark.

Reviewer comment (Theory):
A potential weakness of the approach, especially when it should ultimately be transferred to real robots, is that the control of the robot's shape do not happen based on local interactions, i.e. the employed controller has to take into account the complete shape of the robot and its environment.

Author response:
Thank you for making the connection to real robots. First-person-view observation and third-person view observation are both common choices in real robot reinforcement learning papers. We focus on third-person-view exactly because the current real-world hardware implementations [20] [21] of the robot uses a third-person view setup. While we recognize that reconfigurable robot research may evolve in the long-run, we believe our setup and benchmark is best suited for current hardware.

Reviewer comment (Experiments):
"However, these search-based zeroth-order optimization methods are computationally demanding and inefficient” - is this shown in the paper? It's a common argument but it would be good to have some references here backing it up.

Author response:
Thanks for your suggestion. We will add more references to support this claim in camera-ready.

Reviewer comment (Novelty):
Importantly, what are the lessons for the larger machine learning community? e.g. for which other tasks could the course-to-fine (CFP) algorithm be useful? What about other hierarchical RL methods? The research is exciting from an alife/robot perspective but it could be better motivated for an ICLR audience.

Author response:
Coarse-to-fine has seen its success in many machine learning domains like  image and video generation [22][23]. However, it’s been largely ignored by the RL community because not many RL problems largely ignore the structures of action space. The lesson is that whenever resolutional hierarchy exists, don’t overlook it just because this is not traditionally used in your domain.

Reviewer comment (Evaluation):
Since this conference is about representation learning, it would be interesting to investigate further what type of representations the RL algorithms learn to control these soft robots. How do the coarse and fine-grained control interact to solve the tasks at hand?

Author response:
Thank you for defending the relevance of the conference. From the “About Us” page of ICLR, “The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but **generally referred to as deep learning**.” 

Therefore, as a deep reinforcement learning paper we naturally fit into the focus of ICLR.

Reviewer comment (Reproducibility):
Is the code available?

Author response:
Yes. We plan to release the code for both benchmark and algorithm upon acceptance.

Reviewer comment (Reproducibility):
How computationally expensive is the simulation environment?

Author response:
We made it really fast! The simulation is GPU based. We report the usage and speed of our most computation-intensive task DIG (because it contains the most particle numbers). Simulation alone, the environment runs at an average speed of 70.1 FPS on a RTX4090 GPU, with only 20-30% volatile utility usage and 15% memory usage. Therefore theoretically you can run 5 of such simulations on RTX4090 at similar speed.

Reviewer comment (Experiments):
How large is the action space?

Author response:
Although our formalization allows an infinite dimensional action space, our finest policy runs with an output of size 2*16*16 = 512 dims, which gets interpolated to this infinite dimensional action space.

Reviewer comment (Reproducibility):
How is the upsampling of coarse actions done?

Author response:
Sorry about the confusion. The upsampling of coarse action is done by the bicubic interpolation while the network details are shown below:

* Critic Network:
The 3-layer CNN based encoder can embed the input of 5x64x64 (3 channels for state, 2 channels for the upsampled action) images into a vector with 512 dims following a CNN architecture widely used in RL papers and frameworks [24][25]. Then it will get through a 3-layer MLP with 256 dims latent space to yield the Q value.
* Policy Network:
The similar 3-layer CNN based encoder can embed the input of 3x64x64 images into a embedding shaped 32x4x4 (still 512 dims). Then it will get through a 3-layer Deconv to yield grid-like action (2x8x8 or 2x16x16).

In the residual training stage, the policy network’s input channel will also be 5 (3 channels for state, 2 channels for the upsampled coarse action).

Reviewer comment (Reproducibility):
What are the network details (e.g. number of layers etc.)?

Author response:
Sorry about the confusion. The upsampling of coarse action is done by the bicubic interpolation while the network details are shown below:

* Critic Network:
The 3-layer CNN based encoder can embed the input of 5x64x64 (3 channels for state, 2 channels for the upsampled action) images into a vector with 512 dims following a CNN architecture widely used in RL papers and frameworks [24][25]. Then it will get through a 3-layer MLP with 256 dims latent space to yield the Q value.
* Policy Network:
The similar 3-layer CNN based encoder can embed the input of 3x64x64 images into a embedding shaped 32x4x4 (still 512 dims). Then it will get through a 3-layer Deconv to yield grid-like action (2x8x8 or 2x16x16).

In the residual training stage, the policy network’s input channel will also be 5 (3 channels for state, 2 channels for the upsampled coarse action).

Reviewer comment (Novelty):
Given the concern above, the theoretical novelty seems not high, as the main theoretical novelty of CFP is the introduction of adding residual action to coarse action.

Author response:
> Given the concern above, the theoretical novelty is not high, as the main theoretical novelty of CFP is the introduction of adding residual action to coarse action.

While our paper is not a ML theory paper, we still believe we have strong technical contributions that can inspire broader research from the community. First, formalize a novel control problem that has never been studied algorithmically into a Markov Decision Problem. This establishes a platform that future researchers can build on. To our knowledge, we are the first paper to let robots control its muscle to change its own morphology, not to mention the slime robot. Secondly, we overcome the challenges of simulation and provide the RL community with a set of fast benchmarks of this interesting but novel task. Our effort on the simulation is non-trivial and such novel benchmarks alone have been traditionally regarded as technical contributions in many top ML conferences [13] [14] [15]. Thirdly, CFP is one of the first RL algorithms that uses a 2D action space. Many design choices here such as fully convolutional network or coarse to fine are largely overlooked by the RL community, which traditionally focuses on unstructured action space while our insight brings attention to such 2D structure. We believe our formalization, benchmarks and algorithm together shall constitute technical contributions above the acceptance threshold.

Reviewer comment (Experiments):
The paper lacks comparisons with existing state-of-the-art methods. The experiments only evaluate the performance of two ablation baseline methods and one NFP method. More comparisons will be appreciated. Besides the evaluation metric on reward, if more metrics can be used to evaluate real-robot applications, it will be appreciated, such as the successful rate or time efficiency.

Author response:
> The paper lacks comparisons with existing state-of-the-art methods.

Thanks for your suggestion. We added two more related baselines and corresponding experiments following Reviewer UEFW’s suggestion, although we’d like to clarify that these baselines have to undergo significant adoption as we have a very novel task. 

We take state-of-the-art algorithms in the field of articulated reconfigurable robots, Modular Policy [16],  and adopt it for highly-reconfigurable robots by using k-means to assign topological structures to robot muscles. We provide an additional baseline by letting an attention mechanism to figure out topology itself. We benchmark the baselines on 4 of the 8 tasks due to time constraints with the same number of seeds.

Details of both baselines and result curves are added to our [rebuttal website](https://sites.google.com/view/morphologicalmazerebuttal). Figure 1.1 in the website illustrates the cluster while figure 1.2 shows our method’s performance is still much higher than the added baselines. This is not surprising, as GNN is only suited for robots with clear articulation, which slime robots lack.

Reviewer comment (Experiments):
As one of the main challenges proposed to address in this paper, the justification of lifetime adaptation is insufficient in the experiments. For example, if the approach can adapt to noise or actuator failures during the lifetime operation?

Author response:
> As one of the main challenges proposed to address in this paper, the justification of lifetime adaptation is insufficient in the experiments. For example, if the approach can adapt to noise or actuator failures during the lifetime operation?

Thanks for your suggestion. We follow your suggestion and add 2 additional sets of experiments that emulate actuator failure and observation failure. 

Regarding noise adaptation, we’d like to clarify that we already added gaussian noise to action output when getting our result in the main paper, to emulate noisy control. The SAC algorithm naturally optimizes expected return under high entropy (noisy) actions, which explains why our result is already good in the paper.

Our two added experiments randomly zeros out 20% of the action output / observation input respectively when evaluating our expert policy. This emulates actuator / sensor failures respectively. Figure 4 on our [rebuttal website](https://sites.google.com/view/morphologicalmazerebuttal) shows their performance as a ratio to original performance. As shown in the figure, our performance is negligibly impacted (<10%) by action failure except 1 out of the 8 tasks, DIG, which shows a performance drop of only 30%. The policy also seems to show some robustness to sensor failure from the plot, a less common setting.

Reviewer comment (Experiments):
Even though the simulation experiments are impressive, there is still a gap between applying the proposed work to real-world robots, as shown in the demo video. An explanation of how to extend the work from simulations to real-world robots will strengthen the paper significantly.

Author response:
> Even though the simulation experiments are impressive, there is still a gap between applying the proposed work to real-world robots, as shown in the demo video. An explanation of how to extend the work from simulations to real-world robots will strengthen the paper significantly.

Thanks for your comment. In our project, we tried to mimic the settings of real-world slime robots implementation in [19]. In this paper, the researchers use cameras and magnetic fields to demonstrate basic movements of the slime robot. In our benchmark, we adopt the gravity field simulation from Taichi to simulate magnetic fields and adopt the same real-world robot observation hardware researchers use. While we recognize the current framework has its assumptions, we believe our implementation is well-aligned with the hardware progress of real-world magnetic robots, which are bottlenecked by algorithms and benchmarks to which our paper aims to contribute.

Reviewer comment (Experiments):
There is no adequate comparison with classical robot design baselines, e.g., Bayesian optimization, genetic algorithms, etc.

Author response:
Thanks for your suggestion about the baselines. While co-design is closely related in our related work section, we’d like to clarify that we are working on a novel problem that is drastically **different** from co-design, rendering the co-design baselines unusable. As we mentioned in the general response, co-design problems **fix** a robot morphology during execution of the task once the algorithm finds an optimal morphology. In contrast, the goal of our project is to control a type of robot that does **not** have a fixed morphology. In our problem, the slime robot has to actively control and **change** its morphology throughout the task. In fact, our robot always begins with a circle or square shape in any task.

Despite this, we added two more related baselines and corresponding experiments following Reviewer UEFW’s suggestion. We take state-of-the-art algorithms in the field of articulated reconfigurable robots, Modular Policy [4],  and adopt it for highly-reconfigurable robots by using k-means to assign topological structures to robot muscles. We provide an additional baseline by letting an attention mechanism to figure out topology itself. We benchmark the baselines on 4 of the 8 tasks due to time constraints with the same number of seeds.

Details of both baselines and result curves are added to our [rebuttal website](https://sites.google.com/view/morphologicalmazerebuttal). Figure 1.1 in the website illustrates the cluster while figure 1.2 shows our method’s performance is still much higher than the added baselines. This is not surprising, as GNN is only suited for robots with clear articulation, which slime robots lack.

Reviewer comment (Experiments):
Is the action space or environment 2D or 3D? Why use 2D images as raw states for policy instead of using ground truth robot states (i.e., exact morphology parameterizations)?

Author response:
The action space is represented as a 2-D grid, although our simulator can be easily extended to 3D.

We choose image observations because pixels are the most general representation of tasks. e.g. Millions of video games are played by humans through pixels; typical real robot manipulation tasks in RL literatures. This has been widely recognized by the deep reinforcement community [5] [6], which considers control from pixels as a more challenging problem [7] [8]. 

A second reason we consider pixel observation is its potential for real-world transfer. Slime robots, for example the real-world implementation [9] [10], are highly unstructured and hard to obtain an intrinsic state. Pixel observation not only addresses the challenge for us but also organically capturess external world.