Paper: Meta-Evolve: Continuous Robot Evolution for One-to-many Policy Transfer (ICLR-2024)

Reviewer comment (Theory):
In section 3.2, the authors discuss kinematic tree matching between robots with similar kinematic structures. However, I am concerned that they are claiming that just because two robots share a similar kinematic tree, that they also share a similar expert policy. In my understanding, the kinematic tree matching would relate a pair of robots with 2 fingers and 3 fingers more closely than a pair of robots with 2 fingers and 4 fingers. However, it very well may be the case that a robot with 2 fingers will act more similarly to a robot with 4 fingers than one with 3 fingers.

Author response:
We apologize for the confusion. To clarify, we didn’t claim the relationship between similar kinematic trees and similar expert policies.

Our assumption is that, if two robots are similar in their *morphology and dynamics* (specified by their kinematic structures, physical parameters etc.), they may also be similar in their optimal policy and have small cost of policy transfer between them. It’s not just about the kinematic trees. We will include additional sentences in the next version of the paper to make it more clear.

Reviewer comment (Theory):
The convex hull of robot hardwares might be irrelevant if the robots close in hardware-space are far in policy space.

Author response:
We agree that it is possible that in some cases, transferring policy to robots closer in hardware space can actually be more expensive.

Our assumption is that the policy transfer cost is **locally** proportional to robot hardware space distance when the hardware difference is small. Globally, this assumption may not be true. 

Based on this assumption, we propose a heuristics to use physical parameter distance to estimate the cost of local policy transfer. The heuristics are used to construct evolution trees and are shown to be effective in our extensive experiments. As mentioned in the 2nd paragraph of Section 3.5, we leave the problem of finding the optimal way of estimating policy transfer cost and constructing evolution trees as future work:

“*Policy transfer through robot evolution relies on local optimization of the robot evolution. On the other hand, optimizing the evolution tree requires optimizing the robot evolution paths globally and needs an accurate “guess” of the future cost of policy transfer. In fact, our proposed heuristics can be viewed as using $L^p$ distance of evolution parameters to roughly guess the future policy transfer cost for constructing evolution tree. We leave the problem of finding the optimal evolution tree and meta robots as future work.*”

Reviewer comment (Reproducibility):
I am curious about the method the authors use to actually train the policies on the new robots. The description of generating new robots is clear but the method for actually transferring the policies is not.

Author response:
Sorry for the confusion. As mentioned in the 1st paragraph of Section 3.3 and line 4 of Algorithm 1, the policy is transferred by fine-tuning on the next intermediate robots, i.e. training the policy on the next intermediate robots using an RL algorithm. In our experiments, we use NPG algorithm as our RL algorithm.

Reviewer comment (Experiments):
Does the simple fact that the authors are fine-tuning the same policy more times on the baselines lead to poor performance?

Author response:
We did a sufficient amount of experiments on REvolveR, HERD and our Meta-Evolve, and we did not see performance degradation problems.

Reviewer comment (Evaluation):
Can you please describe quantitatively/qualitatively the quality of the learned target policies between the different methods?

Author response:
Our method aims at improving the overall policy transfer efficiency. After policy transfer is completed, the performance/quality of transferred policies can be further improved by continuing to train on the target robots with RL algorithms.

Given sparse task-completion reward, success rate is the best way to evaluate the policy quality. Quantitatively, transferred target policies reach the same success rate at the end of policy transfer, though it takes much smaller overall training/simulation cost for our Meta-Evolve to do so than HERD.

Qualitatively, as suggested, we visualize the transferred policies on Hammer and Relocate tasks using different methods in this video: https://drive.google.com/file/d/1Ln8oLZhU3g1hw_lZhqAu-CC0rW2WvSW_/view?usp=sharing (select 1080p for best quality). As shown in the video, policies transferred using different methods show slightly different behaviors, though they are all able to complete the tasks.

Reviewer comment (Novelty):
It is not obvious that the work has much to do with evolution, in either the biological or computational senses, except in its use of trees which are reminiscent of the tree of life.

Author response:
`It is not obvious that the work has much to do with evolution, in either the biological or computational senses`; `It is also not obvious why the word “meta” was chosen, especially given that it has other connotations in Reinforcement Learning`.

The word "evolution" has more than just biological or computational meaning. According to Webster's Dictionary (the most authoritative dictionary for modern English), another meaning of "evolution" is "a process of change in a certain direction" (see the second listed meaning in this link: https://www.merriam-webster.com/dictionary/evolution).

The current name for our method “Meta-Evolve” is a combination of “meta” (inspired by the idea of meta learning where the learning objective is across multiple tasks) and “evolve” (to show the connection to robot evolution).

We are open to changing the name to a better one if Reviewer NXFw could suggest a better name for our method.

Reviewer comment (Writing):
It is also not obvious why the word “meta” was chosen, especially given that it has other connotations in Reinforcement Learning.

Author response:
`It is not obvious that the work has much to do with evolution, in either the biological or computational senses`; `It is also not obvious why the word “meta” was chosen, especially given that it has other connotations in Reinforcement Learning`.

The word "evolution" has more than just biological or computational meaning. According to Webster's Dictionary (the most authoritative dictionary for modern English), another meaning of "evolution" is "a process of change in a certain direction" (see the second listed meaning in this link: https://www.merriam-webster.com/dictionary/evolution).

The current name for our method “Meta-Evolve” is a combination of “meta” (inspired by the idea of meta learning where the learning objective is across multiple tasks) and “evolve” (to show the connection to robot evolution).

We are open to changing the name to a better one if Reviewer NXFw could suggest a better name for our method.

Reviewer comment (Evaluation):
It is also not obvious why some of the thresholds in the evaluation were selected, and it would be interesting to see if a change from 80% to 70% or 90% would change the order of the methods. I suspect that it would not, but the choice of selecting a single threshold leaves the question open.

Author response:
`4. Why was the value of 80% selected?`; `it would be interesting to see if a change from 80% to 70% or 90% would change the order of the methods`

We explained this in the 3rd paragraph of Section E of the appendix:

“*We followed HERD (Liu et al., 2022a) and adopted the training overhead needed to reach 80% success rate as our evaluation metric. The core idea of using this evaluation metric is to compare the efficiency of the policy transfer when the difficulty of transferring each policy is unknown beforehand. Using an alternative success rate higher than 80% as the evaluation metric could also be feasible. However, since the success rate threshold for moving on to the next intermediate robot is 66.7%, the success rate increase from 80% to a higher one can only happen after the policy transfer is completed. Therefore, the remaining part of training for reaching a higher success rate is simply vanilla reinforcement learning on the target robots and is irrelevant to our problem of inter-robot policy transfer, so should not be included in the evaluation.*”

Reviewer comment (Novelty):
How common is the problem addressed by this paper, of transferring an effective policy from one robot morphology to multiple others at once?

Author response:
`1. How common is the problem addressed by this paper, of transferring an effective policy from one robot morphology to multiple others at once?`

The problem is common and will become even more popular in the future.

One example: recent advances in the robot learning community started to research on learning across multiple different robot embodiments, such as the recently released Open X-Embodiment benchmark [A] (https://robotics-transformer-x.github.io/) which includes **22** different robots.

[A] Padalkar, Abhishek, et al. "Open x-embodiment: Robotic learning datasets and rt-x models.". arXiv preprint arXiv:2310.08864 (2023). 2023 Oct 13.

Reviewer comment (Experiments):
Why was the only the 1-Steiner tree attempted in the door task?

Author response:
`2. Why was only the 1-Steiner tree attempted in the door task?`

This is because after ablation studies on Hammer and Relocate tasks, it already shows that 1-Steiner tree is the best design choice of our Meta-Evolve.

Reviewer comment (Experiments):
Why were so few seeds used for the results?

Author response:
`3. Why were so few seeds used for the results?`; `The number of training runs in the experiments was quite small, with only 5 random seeds.`

We conducted statistical analysis and found that with 5 seeds, our method Meta-Evolve is better than the baseline HERD with more than 99.99% confidence, on all tasks.

This means 5 random seeds are enough for our experiments.

Reviewer comment (Experiments):
Why was the value of 80% selected?

Author response:
`4. Why was the value of 80% selected?`; `it would be interesting to see if a change from 80% to 70% or 90% would change the order of the methods`

We explained this in the 3rd paragraph of Section E of the appendix:

“*We followed HERD (Liu et al., 2022a) and adopted the training overhead needed to reach 80% success rate as our evaluation metric. The core idea of using this evaluation metric is to compare the efficiency of the policy transfer when the difficulty of transferring each policy is unknown beforehand. Using an alternative success rate higher than 80% as the evaluation metric could also be feasible. However, since the success rate threshold for moving on to the next intermediate robot is 66.7%, the success rate increase from 80% to a higher one can only happen after the policy transfer is completed. Therefore, the remaining part of training for reaching a higher success rate is simply vanilla reinforcement learning on the target robots and is irrelevant to our problem of inter-robot policy transfer, so should not be included in the evaluation.*”

Reviewer comment (Evaluation):
Did the authors consider the relationship between the reduction in the total length of the paths in the graph (as compared to independent paths) and the reduction in the training time? Were these correlated?

Author response:
`5. Did the authors consider the relationship between the reduction in the total length of the paths in the graph (as compared to independent paths) and the reduction in the training time? Were these correlated?`

Yes, the two reductions are positively correlated.

Reviewer comment (Experiments):
Why wasn’t DAPG run to completion? How effectively was it able to perform the task, given that it did not complete in the time provided?

Author response:
`6. Why wasn’t DAPG run to completion? How effectively was it able to perform the task, given that it did not complete in the time provided?`

During exploration on new robots, DAPG has trouble finding successful trajectories with non-zero reward due to the huge robot difference. This means extremely low sample efficiency under sparse task-completion reward setting, and means that DAPG is not effective in directly transferring the expert policies to new robots.

Reviewer comment (Evaluation):
Do the authors have any comments on why (e.g. in the Hammer task, on path 8-10-11) their method outperformed HERD on one-to-one transfer?

Author response:
`7. Do the authors have any comments on why (e.g. in the Hammer task, on path 8-10-11) their outperformed HERD on one-to-one transfer?`

Sorry we don’t have a good explanation for that. We leave the detailed study of this phenomenon as a future work.

Reviewer comment (Theory):
As the name suggests, the method explores in the kinematic space of the robots. What happens if the dynamics differ drastically? An example would be having a robot that has the same kinematic structure but twice the masses. Can the method handle this? It seems that no. Isn't changing the dynamics but keeping the kinematics the same a different robot? I think this deserves more intuition and explanation.

Author response:
> `as the name suggests the method explores in the kinematic space of the robots. What happens if the dynamics differ drastically? An example would be, having a robot that has the same kinematic structure but twice the masses. Can the method handle this? It seems that no. Isn't changing the dynamics but keeping the kinematics the same a different robot?`
> `How can the proposed method handle significant dynamics differences between the source and target robots? Can it even do that?`

Sorry for the confusion. To clarify, our method also explores different robot dynamics because it also explores different robot physical parameters such as mass and inertia, as described in Section 3.2 and Equation (3).

Our method can handle the case where the source and target robot dynamics differ, but of course the difference cannot be unreasonably large. For example, we cannot expect a manipulation policy on a multi-fingered robot gripper to be transferred to a multi-legged locomotion robot dog.

As long as the dynamics difference is reasonable, our method is able to handle that. This is because our method interpolates physical parameters after kinematic tree matching, which is essentially interpolating robot dynamics.

As mentioned by Reviewer 6cdy, we conduct an additional experiment where we double the mass of the agile locomotion robots. Both HERD and our method can still successfully transfer the policy to the new target robots with double masses.

Reviewer comment (Experiments):
I am not sure how we should interpret the additional experiments on commercial robots. I cannot see the added value of those experiments and how they contribute towards convincing us that the method is applicable to real-world situations/robots. The robots are purely position-controlled and the behaviors are quite simple.

Author response:
> `I am not sure how we should interpret the additional experiments on commercial robots. I cannot see the added value of those experiments and how they contribute towards convincing us that the method is applicable to real-world situations/robots.`

The goal of our method is to transfer an expert policy from one robot to multiple other predefined robots.

In the additional experiments in Section A of the appendix, we verified that our method can transfer a manipulation policy from a five-finger dexterous robot hand to three different existing commercial robots (Jaco, Kinova3, IIWA) in simulation while outperforming baseline HERD by roughly 2$\times$ in terms of overall efficiency. In addition, we verified that the transferred policy on Kinova3 in simulation can be deployed on the real Kinova3 machine in a real-world situation.

Therefore, the additional experiments support our claim and further show that our method can be applied to real-world situations/robots.

Reviewer comment (Novelty):
The comments on Sim2Real are weak in my opinion. I do not see why the paper is not relevant to Sim2Real methods. The paper/method claims one-to-many policy transfer with *different dynamics* involved (per robot).

Author response:
> `the comments on Sim2Real are weak imho. I do not see why the paper is not relevant to Sim2Real methods.`
> `How does the proposed method fit inside the Sim2Real literature?`

Conceptually, sim-to-real transfer is not within the scope of our paper:
  - Our Meta-Evolve deals with transferring policy from the *source robot* to *target robots*, in *simulation*. 
  - Sim-to-real transfer deals with transferring the policies on *target robots* from *simulation* to *real machine*.

Therefore, it is actually OK to not even discuss sim-to-real transfer in our paper.

We still discussed it anyway in Section B (and also conducted additional real-robot experiments in Section A as experimental evidence), simply to address people’s (potential) concerns over deploying the transferred policies to real machines.

Reviewer comment (Writing):
The authors have chosen to consider a model-free approach to RL/transfer learning, while obviously the models are known (at least the kinematics part). This choice should be better motivated and highlighted in the text.

Author response:
> `The authors have chosen to consider a model-free approach to RL/transfer learning, while obviously the models are known (at least the kinematics part). This choice should be better motivated and highlighted in the text.`
> `How can the proposed method integrate model-based RL/learning? Why did the authors not experiment with this?`;

In general, models are not easily known in robotic tasks even if the robot kinematics and dynamics are known. This is because there are unknown robot-object interactions that affect both the robot states and the object states. Robot-object interactions are not only determined by the robots but also depend on the objects in the environment.

It’s easy to apply model-based RL algorithms in our method though. We can just instantiate the policy optimization part of line 4 of Algorithm 1 with a model-based RL algorithm. 

However, we believe our existing experiments and ablation studies are already sufficient to support the claims made in our paper.

Reviewer comment (Experiments):
One of the videos is not properly anonymized; we can clearly see the face of someone performing the experiments. Is this part of the video part of the DexYCB dataset? If this is the case, it should have been highlighted. If not (and the one performing the experiments is one of the authors), the AC should take a position on this as I am not sure if this is allowed by ICLR regulations.

Author response:
> `One of the videos is not properly anonymized; we can clearly see the face of someone performing the experiments. Is this part of the video part of the DexYCB dataset? If this is the case, it should have been highlighted.`

**This part of the video is from public dataset DexYCB** (other example videos of the dataset can be found in https://dex-ycb.github.io/). 

Also, this information has already been highlighted in the supplementary video. At the start of the video (0:00:00-0:00:03), we have already highlighted with text “**DexYCB Dataset Human Demonstration**”.

Reviewer comment (Theory):
How can the proposed method handle significant dynamics differences between the source and target robots? Can it even do that?

Author response:
> `as the name suggests the method explores in the kinematic space of the robots. What happens if the dynamics differ drastically? An example would be, having a robot that has the same kinematic structure but twice the masses. Can the method handle this? It seems that no. Isn't changing the dynamics but keeping the kinematics the same a different robot?`
> `How can the proposed method handle significant dynamics differences between the source and target robots? Can it even do that?`

Sorry for the confusion. To clarify, our method also explores different robot dynamics because it also explores different robot physical parameters such as mass and inertia, as described in Section 3.2 and Equation (3).

Our method can handle the case where the source and target robot dynamics differ, but of course the difference cannot be unreasonably large. For example, we cannot expect a manipulation policy on a multi-fingered robot gripper to be transferred to a multi-legged locomotion robot dog.

As long as the dynamics difference is reasonable, our method is able to handle that. This is because our method interpolates physical parameters after kinematic tree matching, which is essentially interpolating robot dynamics.

As mentioned by Reviewer 6cdy, we conduct an additional experiment where we double the mass of the agile locomotion robots. Both HERD and our method can still successfully transfer the policy to the new target robots with double masses.

Reviewer comment (Novelty):
How does the proposed method fit inside the Sim2Real literature?

Author response:
> `the comments on Sim2Real are weak imho. I do not see why the paper is not relevant to Sim2Real methods.`
> `How does the proposed method fit inside the Sim2Real literature?`

Conceptually, sim-to-real transfer is not within the scope of our paper:
  - Our Meta-Evolve deals with transferring policy from the *source robot* to *target robots*, in *simulation*. 
  - Sim-to-real transfer deals with transferring the policies on *target robots* from *simulation* to *real machine*.

Therefore, it is actually OK to not even discuss sim-to-real transfer in our paper.

We still discussed it anyway in Section B (and also conducted additional real-robot experiments in Section A as experimental evidence), simply to address people’s (potential) concerns over deploying the transferred policies to real machines.

Reviewer comment (Experiments):
How can the proposed method integrate model-based RL/learning? Why did the authors not experiment with this?

Author response:
> `The authors have chosen to consider a model-free approach to RL/transfer learning, while obviously the models are known (at least the kinematics part). This choice should be better motivated and highlighted in the text.`
> `How can the proposed method integrate model-based RL/learning? Why did the authors not experiment with this?`;

In general, models are not easily known in robotic tasks even if the robot kinematics and dynamics are known. This is because there are unknown robot-object interactions that affect both the robot states and the object states. Robot-object interactions are not only determined by the robots but also depend on the objects in the environment.

It’s easy to apply model-based RL algorithms in our method though. We can just instantiate the policy optimization part of line 4 of Algorithm 1 with a model-based RL algorithm. 

However, we believe our existing experiments and ablation studies are already sufficient to support the claims made in our paper.