Paper: Curriculum Reinforcement Learning via Morphology-Environment Co-Evolution (ICLR-2024)

Reviewer comment (Novelty):
The significance of the paper's contributions is a bit unclear.

Author response:
> The significance of the paper's contributions is a bit unclear.
- To the best of our knowledge, MECE is the first paper to discuss the co-evoltuion between the agent's morphologies and training environments and use the co-evolution to create an efficient curriculum for training embodied agent for better generalization. Would you mind kindly providing papers that have previously examined this problem?
- Our baseline methods include the SOTA algorithms of training agent to adpat to diverse environments (POET), the SOTA morphology optimization method utilizing RL to evolve the agent's morphology in a fixed environment (Transform2Act), and a widely-studied method that models an agent's morphology as GNN (NGE).

Reviewer comment (Experiments):
How general is the proposed approach, beyond the tasks and environments considered in the experiments?

Author response:
> How general is the proposed approach, beyond the tasks and environments considered in the experiments?
- In MECE, the simulation is Mujoco, which is a widlely used simulator in RL.
- We did not design special constraints for environments because the learned $\pi_E$ automatically avoids generating unlearnable environments. Furthermore, tasks are not accompanied by any form of extrinsic reward signal. Hence, MECE has the potential to be applied to a wide range of tasks and circumstances, extending beyond the confines of experimental settings.

Reviewer comment (Experiments):
Is the proposed MECE method computationally efficient?

Author response:
> Is the proposed MECE method computationally efficient?
- MECE training is more computationally efficient than all the compared baselines. For all the test environments, MECE took around 44 hours to train on a standard server with 8 CPU cores and an NVIDIA RTX 3090 Ti GPU, while Transform2Act requires around 63 hours on the same server.

Reviewer comment (Experiments):
Have you encountered any scalability issues when applying MECE to more complex tasks or environments?

Author response:
> Have you encountered any scalability issues when applying MECE to more complex tasks or environments?
- The 3d-locomotion environment is a complex environment for RL navigation tasks. If we do not set any contraints on $\pi_e$, in earlier stages, the unmatural environment policy may change the environment at a large scale, which can make the training environments overly hard/simple. Hence, we limit the scale of changes by $\pi_e$ to environment in each step.

Reviewer comment (Experiments):
It is not clear to me how environments are produced and how the agents perform in your environment (Figure 4), do you have a video?

Author response:
> It is not clear to me how environments are produced and how the agents perform in your environment (Figure 4), do you have a video?
- We introdce how $\pi_E$ generates and controls the environments in Appendix A.
- We will release the source code with videos.

Reviewer comment (Experiments):
It seems that MECE's performance is not much better than Transform2Act, can you provide more results on different tasks?

Author response:
> It seems that MECE's performance is not much better than Transform2Act, can you provide more results on different tasks?
- We respectfully disagree with the assertion that this is "not much superior to Transform2Act." In Figure 2, MECE outperforms the modified Transform2Act (trained on diverse environments) by around 20% in terms of rewards and outperforms Transform2Act-Original by approximately 60%.

Reviewer comment (Evaluation):
As far as I understand from what is written in the introduction, the motivation of the morphology optimization in this paper is to obtain a morphology with which the agent can adapt its policy quickly to unseen tasks. It is also written in the second question of the experiments. However, it seems that the reported results in figures are average performances of the agent obtained at each training time step on randomly-selected environment. Therefore, the performance evaluated in this paper is the one for domain randomization. It is different from the motivation. The efficiency of the adaptation of the policy under the obtained morphology is not evaluated. My understanding might be wrong as the evaluation procedure was not clearly stated. Please clarify this point.

Author response:
> The setting of experiments is different from the motivation.
- Our evaluation is consistent with the motivation and the 2nd question at the beginning of the experiments: "Does our method create agents more adaptive and generalizable to diverse new environments?". As stated in the 2nd line of Section 5.2, we evaluated MECE and its learned morphology in 12 diverse and unseen environments (random variants of 2d-locomotion, 3d-locomoation, or Gap-crosser) under 6 random seeds with the morphology fixed. The experimental results show that the morhology evolved by MECE can generalize better to the tasks in diverse unseen environments.
- MECE does not require finetuning for the unseen environments in the test, unlike the Transform2Act and NGE methods, which still require this additional step. Furthermore, we present the performance of baselines without finetunes in the table below.
 
|Methods| Performance|
|:--|:--|
|MECE| 4252.26 $\pm$ 393.45|
|Transform2Act| 3644.28 $\pm$ 518.77 |
|Transform2Act without finetune|1990.54 $\pm$ 424.03| 
|NGE|1874.65 $\pm$ 816.51|
|NGE - without finetune|942.30 $\pm$ 566.22|

Reviewer comment (Theory):
It could be better if the design choices of the proposed approach is more elaborated. In particular, it is not clear how the reward functions (1) and (2) reflect the author’s hypotheses “a good morphology should improve the agent’s adaptiveness and versatility, i.e., learning faster and making more progress in different environments” and “a good environment should accelerate the evolution of the agent and help it find better morphology sooner”. It is also not clear why the authors want to train policies for morphology evolution and environment evolution instead of just optimizing the probability distributions over these spaces, despite the fact that these policies are not used afterwards and only the obtained morphology is used in the test phase.

Author response:
> How do $r_m$ and $r_E$ contribute to our statement on "a good morphology" and "a good environment"? + Why we need morphology policy $\pi_m$ and environment policy $\pi_e$?

- We explain the motivation of designing $r_m$ and $r_e$ in the 4th and 5th paragraphs of Section 3, in particular, the text above Eq. (1) and Eq. (2). They are in line with the statement of "a good morphology should make the agent learn faster and make more progress in different environments." (i.e., greater progress of control policy $\pi$ in Eq. (1)) and "a good environment should accelerate the evolution of the agent and help it find better morphology sooner" (i.e., greater progress of morphology policy $\pi_m$ in Eq. (2)). 

- The training and co-evolution of $\pi_e$ and $\pi_m$ is the key for the morphology optimization and gaining generaliation to diverse unseen environments. They jointly create a special adaptive curriculum for the training phase for the agent to evolve its morphology and generalization capability. In particular, $\pi_e$ creates a curriculum of environments for optimizing the morphology and control policy $\pi$, while $\pi_m$ creates a curriculum for the environment changing and control policy learning. They improve the efficency of exploration through (1) interactions with MDP using different morphologies and environments; (2) structural constraint and correlation captured by the GNN; (3) more directed evolution than random mutation.
- In our thorough ablation study-Ⅰ,Ⅱ,Ⅳ, we have extensively evaluated the importance and impacts of $r_E$ and $r_m$, $\pi_e$ and $\pi_m$ in morphology optimization and generalization to new environments.

Reviewer comment (Writing):
The clarity of the explanations could be improved. First, the notation inconsistencies makes it confusing. For example, r^m vs r_m, r^E vs r_e, and E and Env. If they are the same, please use the same notation. Algorithm 2 was also not very clear. How could pi_m be updated by using D where transition history doesn’t necessarily have a reward information r_m? The same applies for pi_e.

Author response:
> The clarity of the explanations could be improved.
- Thank you for this suggestion. We have updated all unclear notations and expressions following your suggestions.

Reviewer comment (Experiments):
Comparisons to other methods are a bit ad hoc, e.g. as the authors note, POET was not developed to deal with changing morphologies. In addition to randomly sampling environments here, I would suggest a slightly more advanced baselines that samples environments of increasing complexity

Author response:
> Comparison with POET.
- Yes, we agree that POET is not designed for changing morphologies. But this baseline was required by a previous reviewer. For ensure a fair comparison, we have made several modifications (detailed in point (3) of the ''Baseline'' paragraph in Section 5.1).

> "I would suggest a slightly more advanced baselines that samples environments of increasing complexity"
- Thank you for the suggestion! We already included such a baseline in our comparison (e.g., in Figure 2): Enhanced POET starts with a simple environment and then gradually creates and adds new environments with increasing diversity and complexity.
- As mentioned in the paper, MECE stands as the pioneering approach in discussing co-evolution between the agent's morphology and the training environment. Finding baselines that align properly with this settings might be challenging.

Reviewer comment (Experiments):
"When the control complexity is low, evolutionary strategies have been successfully applied to find diverse morphologies in expressive soft robot design space” -> how does the control complexity in this paper compare to the one by Cheney et al.? One could say the soft robots in Cheney et al. (2013) are more complex than the robots co-evolved in this paper.

Author response:
> The control complexity compared to Cheney et al..One could say the soft robots in Cheney et al. (2013) are more complex than the robots co-evolved in this paper.
- Whether a soft-bodied robot is more complex than a rigid-bodied robot is still an open problem with different opinions from different studies. In this paper, we only focus on the optimization in the space of rigid-bodied robots. We will make it clear in the new version.

Reviewer comment (Presentation):
It would be good to see some pictures of the evolved environments

Author response:
> "It would be good to see some pictures of the evolved environments."
- We will release the code with videos.