From: ICLR.cc/2023/Conference/Paper529/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: Interesting paper about transferring policies between robots. The researchers investigate the problem of transferring an expert policy from a source robot to multiple different robots. They propose a method called Meta-Evolve that uses continuous robot evolution to efficiently transfer the policy to a newly defined meta robot, which is then transferred to each target robot. Experiments show that with three target robots, the proposed method is more efficient than the baseline of launching multiple independent policy transfers. 

The problem is somewhat novel, and the paper is well-written. Maybe the main issue is that experiments are done in simulation only and it is not entirely clear how well they will generalize. 
justification_for_why_not_higher_score: The method is not novel enough for a spotlight presentation.
justification_for_why_not_lower_score: The method works and is somewhat novel; the paper is substantially correct.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper529/Authors
title: Reply to Reviewer vZ7B
comment: We thank Reviewer vZ7B for further comments.

> `Transfer learning papers under different dynamics deal with the similar problem, I believe.`

**Again, which previous transfer learning paper has studied the same or similar problem before?**

If Reviewer vZ7B knows a previous paper that dealt with a similar problem, I would appreciate it if Reviewer vZ7B could tell us (e.g. title, author, year, venue, …?) so that we can compare our problem formulation against that paper.

If Reviewer vZ7B does not know such a previous paper, then we believe Reviewer vZ7B’s criticism on our problem formulation novelty is not supported, therefore is invalid.

> `Having multiple known target domains do not present an additional challenge in my opinion.`

Having multiple known target robots is not a challenge: we can always launch multiple independent HERD runs to do one-to-one policy transfer for each target robot. The goal of our Meta-Evolve method is to **speed up** this process by introducing a meta robot.

> `It is not likely that we have many known robots that are slightly different from each other in their dynamics`

Actually, there are a lot of known commercial robots that are slightly different from each other in their dynamics.

To name a few: Sawyer, Panda Franka, Kuka IIWA, Kinova3, Baxter, UR5e, ..., are all similar in their grippers and therefore their dynamics. We suggest that Reviewer vZ7B search and take a look at the images of these robots online.

> `and the challenge would be to learn a fixed representation specific to the target robots. `

Reviewer vZ7B has mentioned multiple times in their review and comment that our framework contains a *learned* meta-evolved representation or intermediate representation of robots -- **this is a huge misunderstanding**. 

To clarify, we never used a *representation* of robots: robots are always themselves in the simulator. And the meta robot is not learned: its robot parameters are determined by heuristics. The meta robot parameters have no relation to its own policy at all.

> `In that case, storing the learned policies may be a better alternative and choosing the starting policy as the one closest to the robot configuration space.`

**This is exactly what we did!!!** The policy of an intermediate robot is attached to that robot and has no relation to the evolution parameter $\mathbf{\theta}$ of the robot. The update of the policy is due to policy fine-tuning using the RL algorithm, not due to the change of evolution parameter $\mathbf{\theta}$.

> `When I say target robots are different, I mean target robots are more different from each other than as compared to the source robot. In that case, the meta-evolved representation may yield a local minima which is farther from each constituent robot.`

Our method is not omnipotent. Our method assumes the target robots are mutually more similar to each other than the source robot. This assumption is practically true: the most popular commercial robots (e.g. Sawyer, Panda Franka, Kuka IIWA, Kinova3, Baxter, UR5e, ...) are more similar to each other than human hand. We provided a detailed discussion on this in the first paragraph of Section 6. Please refer to Section 6 for more details.

> `I believe the paper needs more work in problem formulation and can benefit further from a more detailed experimental evaluation.`

In their first comment/official review released on Nov 5, Reviewer vZ7B did not request additional experiments. From the context of their second comment on Nov 30, it is unclear to us what experiments we are missing. We would appreciate it if Reviewer vZ7B could tell us what additional experiments we are supposed to conduct, what questions the additional experiments are trying to answer, and why the additional experiments are necessary for our paper.

For the problem formulation, **again, which previous paper has studied the same or similar problem before?**

If Reviewer vZ7B knows a previous paper that dealt with a similar problem, I would appreciate it if Reviewer vZ7B could tell us (e.g. title, author, year, venue, …?) so that we can compare our problem formulation against that paper.

From: ICLR.cc/2023/Conference/Paper529/Reviewer_vZ7B
title: response to authors
comment: Thank you for providing feedback and clarifying the responses.

Transfer learning papers under different dynamics deal with the similar problem, I believe. Having multiple known target domains do not present an additional challenge in my opinion. It is not likely that we have many known robots that are slightly different from each other in their dynamics, and the challenge would be to learn a fixed representation specific to the target robots. In that case, storing the learned policies may be a better alternative and choosing the starting policy as the one closest to the robot configuration space.

When I say target robots are different, I mean target robots are more different from each other than as compared to the source robot. In that case, the meta-evolved representation may yield a local minima which is farther from each constituent robot.

I believe the paper needs more work in problem formulation and can benefit further from a more detailed experimental evaluation.

From: ICLR.cc/2023/Conference/Paper529/Reviewer_DsD8
title: reply
comment: Re: However, we believe the five-finger to fewer-finger setting is the most practical one.

The title of the paper is "META-EVOLVE: CONTINUOUS ROBOT EVOLUTION FOR ONE-TO-MANY POLICY TRANSFER", not something more specific like "transfer policy from five finger hand to robot hands with less finger". So I believe a more complete evaluation is required, and the experiment should be simple to run, i.e., just take the policy that works for the 2 fingers hand and run your pipeline in the reverse direction.

From: ICLR.cc/2023/Conference/Paper529/Authors
title: Reply to Reviewer DsD8 and thank you for your time!
comment: We thank Reviewer DsD8 for their comments and the time they spent in the review of our paper. Regarding the comments and issues raised by Reviewer DsD8:

> `The only issue I have is a lack of investigation of how the proposed method performs in the reverse direction (simpler robot to more complicated one). This setting might even be more practical since it might be easier to train a policy for simpler robots at the beginning.`

We are sorry that we didn’t conduct such reverse-direction experiments so we cannot provide a concrete answer for this question now.

However, we believe the five-finger to fewer-finger setting is the most practical one. There are a lot of human daily activity data easily available (e.g. Ego4D [A] has 3,000 hours of egocentric data) which can act as a rich source of demonstrations for training good policies on five-finger hands. On the other hand, in practice, most popular commercial robot grippers have fewer fingers than human (e.g. Panda Franka, Sawyer, UR5, …).

> `I don't agree with the logic behind this sentence. "It shows that, transferring the policies to multiple related target through a meta robot can possibly help each robot improve their learning efficiency." If we only consider one single target robot, the two methods just provide different paths during evolution. And the path provided by the proposed approach is kind of arbitrary based on other randomly chosen target robots. But I agree it is interesting that a straight path is not optimal and worth future investigation.`

In fact, it has been shown in [B] that the optimal path is usually not straight but a path found impromptu during policy transfer. Please refer to [B] for more details. 

Therefore, we are not surprised by “*It shows that, transferring the policies to multiple related target through a meta robot can possibly help each robot improve their learning efficiency*”.



References:

[A] Grauman, Kristen, et al. "Ego4d: Around the world in 3,000 hours of egocentric video." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
 
[B] Liu, Xingyu, Deepak Pathak, and Kris M. Kitani. "HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration." 6th Annual Conference on Robot Learning. 2022.

From: ICLR.cc/2023/Conference/Paper529/Authors
title: Reply to Reviewer 1m9x and thank you for your time!
comment: We thank Reviewer 1m9x for their comments and the time they spent in the review of our paper. Regarding the comments and issues raised by Reviewer 1m9x:

> `The authors propose 3 methods to find good 'meta-robots', which from the evaluation seem to perform quite differently. It is not clear how one could find the best method to use between the 3 (and possibly more) without re-running the transfer procedure with all methods. Considering the gain in training time was found to be on average 2.4x for each approach, running the procedure with all methods would cancel any gains, and possibly take longer. It thus seems that in many cases, one-to-one transfers may still be as fast, without any added complexity to the procedure.`

Our paper is a research paper. To study the new problem raised in our paper, we propose three methods to explore ways of finding good meta robots. This is exactly what research should be: we authors spend effort on the exploration so that our readers can simply get the conclusion from us and don’t need to explore it again. In the case of our paper, after this problem is fully studied and explored, the users can simply pick the best one from our three proposed methods and don’t need to do all three experiments again.

> `It seems we should know a priori what target robots we will encounter.`

The target robots can and should be known.

Imagine we want to deploy the trained policies on the target robots in the real world. To do these real-world experiments, we need to first get these target robots (buy, borrow, rent etc.). Then at that time we will have access to everything about the target robots. It is impossible to deploy a policy on a real robot without knowing what the robot is.

> `I understand that some generalization is likely, when a new robot similar to training targets is encountered, but that is not guaranteed to occur.`

Indeed. Our Meta-Evolve is not guaranteed to work in all cases of target robots. In fact, we provided a discussion on the limitation of our method in Section 6 of the paper:

***When does Meta-Evolve Fail?** One core assumption behind our Meta-Evolve method is that the target robots are mutually similar while very different from the source robot. If the evolution goal is in opposite directions, our Meta-Evolve will not work. For example, given the source robot of a five-finger hand, if one target robot is a ten-finger hand while another target robot is a two-finger gripper, the optimal meta robot might simply be the source robot itself, because the evolution directions toward the target robots are opposite (growing multiple fingers vs. shrinking multiple fingers). However, in practice, most industrial robots are indeed mutually similar in morphology and kinematics while very different from the human hand, so our assumption still stands and Meta-Evolve can still be useful in such cases.*

> `The proposed method is relatively complex, which may make reproducibility difficult.`

Actually, Reviewers DuS3 and DsD8 both found our paper reproducible.

**Reviewer DuS3**: *Reproducibility: Given the straightforwardness of the method, the paper should be easily reproducible.*

**Reviewer DsD8**: *The paper is clearly written, and I believe is reproducible.*

From: ICLR.cc/2023/Conference/Paper529/Authors
title: Reply to Reviewer DuS3 and thank you for your time! (Part 2)
comment: > `Even under the strong correlation assumption mentioned above, wouldn't it be a more fair baseline for HERD to instead take the path of least euclidean distance (e.g. a Traveling Salesman path) over the tasks, rather than simply sum all (source-to-downstream) steps?`

HERD is a one-to-one policy transfer method. By definition, multiple robot evolution paths cannot be shared in HERD. The method that allows sharing part of the robot evolution paths (and the computations along the paths) is actually our proposed Meta-Evolve. Strictly speaking, a Traveling Salesman path is not HERD but a special case of our Meta-Evolve where the $i$-th meta robot is the $i$-th visited target robot along the path. 

We don’t expect the design choice of the Traveling Salesman path to be better than our proposed three design choices. The philosophy behind our Meta-Evolve is to make the meta robot a “general” robot that is later split into multiple “specialized” target robots. A Traveling Salesman path breaks this philosophy and requires policy transfers from one specialized target robot to another specialized target robot which could be much too hard if both are too specialized.

From: ICLR.cc/2023/Conference/Paper529/Authors
title: Reply to Reviewer DuS3 and thank you for your time! (Part 1)
comment: We thank Reviewer DuS3 for their comments and the time they spent in the review of our paper. Regarding the comments and issues raised by Reviewer DuS3:

> `As mentioned in the paper itself, the specific formulation of the meta-point (e.g. using geometric mean/median) is very adhoc, and relies on the assumption that there is a strong correlation between (simulation step distance) and (euclidean distance over robot parameter space).`

Regarding the assumption, we hereby provide additional explanation on the reasoning behind the assumption as follows. We included it as a new paragraph in Section 3.4 of the revised version of the paper. Please check it out.

*Note that by using geometric median, we assume that the training cost for transferring the policy from robot  $F(\mathbf{\alpha}\_{i,k})$ to robot $F(\mathbf{\alpha}\_{i,k+1})$ is proportional to $||\mathbf{\alpha}\_{i,k}-\mathbf{\alpha}\_{i,k+1}||\_2$.
We believe this is a reasonable assumption since the training cost should be proportional to the distribution difference $\mathcal{D}\_{KL}(F(\mathbf{\alpha}\_{i,k}), F(\mathbf{\alpha}\_{i,k+1}))$ of the MDPs of the two robots $F(\mathbf{\alpha}\_{i,k}),  F(\mathbf{\alpha}\_{i,k+1})$  and should be proportional to the robot hardware difference $||\mathbf{\alpha}\_{i,k}-\mathbf{\alpha}\_{i,k+1}||\_2$ when $||\mathbf{\alpha}\_{i,k}-\mathbf{\alpha}\_{i,k+1}||\_2 \rightarrow 0, \forall k$.*

> `Even though the experimental results luckily demonstrate that this sufficient for the robot hand-task, the method is very vulnerable to scaling/transformations over the robot parameter space (e.g. what if we used log(friction coefficient) instead of friction coefficient?).`

This is exactly the reason that we construct the convex hull of robot hardware parameters and normalize the evolution parameter $\alpha$ to $[0,1]$. The Euclidean distance is measured in the space of normalized $\alpha$. Please refer to Equations (2) and (3) for more details. After normalizing $\alpha$ to $[0,1]$, the scaling issue can be significantly mitigated. 

We don’t think our experiment results are due to luck. For example, one of the strategies we proposed for determining meta robot, “Min Dist to Cvx Hull”, is **irrelevant** to scaling/transformation of robot hardware parameters, so it is not vulnerable to scaling/transformation at all. At the same time, “Min Dist to Cvx Hull” yields similar performance as “Geom-Med” which **does rely** on scaling/transformation of robot hardware parameters (Tables 1, 2 & 3). This means the performance of our Meta-Evolve is robust to the selection of meta robot, and therefore also robust to scaling/transformations over the robot parameter space.

> `This is a core issue of the paper, and I believe this should at least be somewhat addressed, potentially with some form of learning (maybe a mapping from physical parameters to some latent space, etc.). Otherwise the method is very un-robust.`

Learning is not omnipotent. We don’t expect a learning-based latent representation method to be able to magically solve any hard problems, such as the problem dealt with in our paper.

Due to the huge complexity of robot control dynamics, it is almost impossible to use a learning-based method to predict the effect of a certain robot hardware parameter before fully testing it in simulation. For example, for a certain robot and a certain robotic task (e.g. sweeping objects off the table top), the mass may be very sensitive while the friction coefficient is not; and for the exact same robot but a different robotic task (e.g. grasping an object), the friction coefficient may be very sensitive while the mass is not. Unless the learning-based method is trained on **ALL** possible robots and **ALL** possible robotic tasks, the learning-based method would only be more vulnerable because it would be unpredictable on new robots and new tasks that are never seen during training. Therefore, we choose to not use learning in our method.

Besides, in Section 6 of the paper, we discussed the difficulty of using learning-based method from the perspective of global path optimization:

*We envision the learning or optimization of the meta robot being very challenging. Policy transfer through robot evolution relies on local optimization in the robot evolution space. On the other hand, the optimization of the meta robot requires optimizing the robot evolution path globally and needs an accurate “guess” of the results of the policy transfer that has not happened yet.*

From: ICLR.cc/2023/Conference/Paper529/Authors
title: Reply to Reviewer vZ7B and thank you for your time! (Part 2)
comment: > `\theta_u … \theta_l defines the convex hull of the set of robot physical parameters. Is this an assumption ?`

This is how we define the robot parameter space for robot evolution. Please refer to Section 3.2 (especially Equation (2)) for more details.

> `Results in Table are hard to interpret and transfer across other robots and environments. Can they be normalized or have more physical interpretation ?`

To interpret the results in the Tables:
  - For HERD, we report the number of simulation epochs and training iterations spent on the three paths: “source -> target 1”, “source -> target 2”, “source -> target 3”, and also the total numbers.
  - For our Meta-Evolve, we report the number of simulation epochs and training iterations spent on the four path parts: “source -> meta robot”, “meta robot -> target 1”, “meta robot -> target 2”, “meta robot -> target 3”, and also the total numbers.
  - We also report the total speedup of these numbers when using our Meta-Evolve compared to HERD.

`"Can they be normalized or have more physical interpretation?"`: The results in the tables are the number of simulation epochs and training iterations. We don’t think normalizing them makes any sense. These numbers represent the training overhead in each path or each part of the paths. There is no physical interpretation for it.

> `Comparisons with other approaches, apart from HERD would also be useful.`

In [B], HERD is compared against multiple other transfer learning baseline approaches and shows superior performance compared to all baselines. It is not necessary to compare to other transfer learning approaches again in our paper because it suffices to compare against and beat HERD in order to support the claims made in our paper.


References: 

[B] Liu, Xingyu, Deepak Pathak, and Kris M. Kitani. "HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration." 6th Annual Conference on Robot Learning. 2022.

From: ICLR.cc/2023/Conference/Paper529/Authors
title: Reply to Reviewer vZ7B and thank you for your time! (Part 1)
comment: We thank Reviewer vZ7B for their comments and the time they spent in the review of our paper. Regarding the comments and issues raised by Reviewer vZ7B:

> `Problem of transferring one to many policies under different dynamics is not entirely novel, as claimed in the contributions. It has been studied in the context of domain generalization, and reinforcement learning, apart from HERD and REvolveR.`

We respectfully disagree. To the best of our knowledge, we are the first to study the problem of one-to-many robot-to-robot policy transfer. Please note that our problem setting is different from traditional formulation of multi-task learning where the agents learn from scratch. Our problem setting focuses on policy **transfer** from an **existing** policy to multiple robots, which is novel.

Besides, **which previous paper has studied the same problem before?** If Reviewer vZ7B knows any prior works that have studied the exact same problem, we would appreciate it if Reviewer vZ7B could tell us. Otherwise, we believe our problem setting is entirely novel.

> `The main bottleneck of the paper is that the target robot configurations are assumed to be known. If intermediate features after policy training and/or pre-training strategy is generalizable across unknown robot configurations, it will make the approach more useful.`

The configurations of the target robots **can** and **should** be known. This is a completely reasonable and necessary assumption and is not a bottleneck of our paper.

Imagine we want to deploy the trained policies on the target robots in the real world. To do these real-world experiments, we need to first get these target robots (buy, borrow, rent etc.). Then at that time we will have access to everything about the target robots. **How can you deploy a policy on a real robot without knowing what the robot is?** 

Of course, as pointed out by Reviewer vZ7B, it would be great if our method could automatically generalize to any unknown robot configurations of target robots. However, this is an unnecessary requirement because it is easy to know the target robot configurations when deploying the policy on the real robots.

> `Would direct transfer need to work to ensure meta-evolution of robot policies from an intermediate representation ?`

Directly transferring the manipulation policies from source to target robots without going through intermediate robots usually does not work, as shown in [A] and [B]. 

> `It is not clear what happens if the target robots are different from each other. In that case, the meta-evolved representation may yield a local minima which is farther from each constituent robot.`

To clarify, In our paper, the target robots **ARE different** from each other (two-finger vs. three-finger vs. four finger).

Also, using our approach, the meta robots are usually determined to be overall **closer** to the target robots in terms of total distance in robot evolution space, not farther.

In practice, it is unlikely that the meta robot falls into a local minima during policy transfer. In our paper, we experimented on three robotic tasks each using three different strategies to determine the meta robot, and we never saw the meta robot fall into a local minima during policy transfer.

> `Moreover, the rationale of posing one to many policy transfer as an interpolation problem is not convincing, especially when the difference across robots is in the physical dynamics parameters that are non-linear to the robot control.`

Let’s first think about it in this way: is it convincing to pose **one-to-one** policy transfer as a robot interpolation problem? The answer is ``yes’’. As explained in [A], when the tasks are too difficult (such as the robotic manipulation tasks used in our paper), directly transferring the policy to another robot is too hard, so incrementally evolving the policy via robot interpolation is a good workaround.

Then, is it convincing to pose **one-to-many** policy transfer as a robot interpolation problem? Of course the answer is also ``yes’’: there is no reason that one-to-one policy transfer using robot interpolation is convincing while one-to-many policy transfer is not.

As for the relationship between dynamics parameters and robot control: robot control is almost never linear w.r.t. dynamics parameters. Even in one-to-one policy transfer, linear interpolation of dynamics parameters usually does not result in linear interpolation of control dynamics. But it is OK for the policy to be transferred through these nonlinear control dynamics paths, as shown in [B].

References:

[A] Liu, Xingyu, Deepak Pathak, and Kris M. Kitani. "REvolveR: Continuous Evolutionary Models for Robot-to-robot Policy Transfer." arXiv preprint arXiv:2202.05244 (2022).

[B] Liu, Xingyu, Deepak Pathak, and Kris M. Kitani. "HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration." 6th Annual Conference on Robot Learning. 2022.

From: ICLR.cc/2023/Conference/Paper529/Reviewer_vZ7B
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper presents a problem of transferring an expert policy from a source robot to multiple target robots by learning an intermediate meta representation. The objective is to save policy training time and exploration. Experiments on hand manipulation suite tasks show performance improvement over one-to-one policy transfer baseline.
strength_and_weaknesses: - Problem of transferring one to many policies under different dynamics is not entirely novel, as claimed in the contributions. It has been studied in the context of domain generalization, and reinforcement learning, apart from HERD and REvolveR. 

- The main bottleneck of the paper is that the target robot configurations are assumed to be known. If intermediate features after policy training and/or pre-training strategy is generalizable across unknown robot configurations, it will make the approach more useful.

- Would direct transfer need to work to ensure meta-evolution of robot policies from an intermediate representation ?

- It is not clear what happens if the target robots are different from each other. In that case, the meta-evolved representation may yield a local minima which is farther from each constituent robot.

- Moreover, the rationale of posing one to many policy transfer as an interpolation problem is not convincing, especially when the difference across robots is in the physical dynamics parameters that are non-linear to the robot control.

- \theta_u … \theta_l defines the convex hull of the set of robot physical parameters. Is this an assumption ? 

- Results in Table are hard to interpret and transfer across other robots and environments. Can they be normalized or have more physical interpretation ? Comparisons with other approaches, apart from HERD would also be useful.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written for the most part. The novelty of the paper is incremental, building upon HERD and REvolveR in a one-to-many policy transfer scenario. Experiments are done in simulation on mock set-ups.

summary_of_the_review: The paper tackles an important problem of learning generalized meta-representation that can transfer across robots in a sample-efficient manner. Results show performance improvement over a one-to-one policy transfer scenario to a target robot. Main concerns of the paper are in problem formulation, generalization scope and comparison with other transfer learning approaches. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper529/Reviewer_1m9x
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper tackles the problem of transferring a trained DRL policy from a source robot to a set of target robots (in the paper, evaluation was performed with a source 5-finger hand and 3 different target hands, with 2, 3, and 4 fingers).
Transferring a policy from to a single target is done in previous work, but remains computationally intensive, and the transfer to N target robots would scale linearly with N.

The authors present a novel method to find* (* albeit not automatically, but rather via hand-crafted heuristics) a 'meta-robot' (defined by a given kinematic chain and dynamic parameters) from which transferring a policy to the given target robots is more efficient. One-to-one policy transfer is then replaced to a one-to-one transfer to the meta-robot, followed by N efficient (?) transfers to the individual targets.

strength_and_weaknesses: Strengths:
- The problem tackled is interesting.
- The manuscript is well written and easy to understand.
- The approach is sensible.

Weaknesses:
- The authors propose 3 methods to find good 'meta-robots', which from the evaluation seem to perform quite differently. It is not clear how one could find the best method to use between the 3 (and possibly more) without re-running the transfer procedure with all methods. Considering the gain in training time was found to be on average 2.4x for each approach, running the procedure with all methods would cancel any gains, and possibly take longer. It thus seems that in many cases, one-to-one transfers may still be as fast, without any added complexity to the procedure.
- It seems we should know a priori what target robots we will encounter. I understand that some generalization is likely, when a new robot similar to training targets is encountered, but that is not guaranteed to occur.

clarity,_quality,_novelty_and_reproducibility: The manuscript is easy to read and the approach is novel. The proposed method is relatively complex, which may make reproducibility difficult.

summary_of_the_review: Overall it is a good paper and the ideas presented are worth reading by the community. However, I am not sure the degree of novelty is sufficient for ICLR.
As such, my adise is for "Borderline/Accept".

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper529/Reviewer_DuS3
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: In summary, the paper:
1. Provides previous background on a form of policy transfer (i.e. "HERD") which involves a series of interpolating training steps from a pretrained policy on a source robot with physical parameters $\theta_{S}$, to a target robot with physical parameters $\theta_{T}$.
2. Extends this formulation to a "MAML"-like [1] case, where instead, the goal is to first transfer to a "meta" robot $\theta_{meta}$ which can quickly transfer to multiple downstream tasks $\theta_{1},...,\theta_{N}$.
3. Proposes manually-described $\theta_{meta}$ points as e.g. geometric means / medians.
4. Evaluates the method experimentally on a fingered robot to show that the total number of simulation steps from (source-to-meta + all meta-to-downstream) is smaller than the total number from (all source-to-downstream).
strength_and_weaknesses: # Strengths
* The method is simple and intuitive to follow. I could understand the paper's motivations throughout, without much difficulty.
* The experiments on the robot hand validate the approach and make sense (i.e. we see a 2x environment sample efficiency over the baseline).

# Weaknesses
* As mentioned in the paper itself, the specific formulation of the meta-point (e.g. using geometric mean/median) is very adhoc, and relies on the assumption that there is a strong correlation between (simulation step distance) and (euclidean distance over robot parameter space). Even though the experimental results luckily demonstrate that this sufficient for the robot hand-task, the method is very vulnerable to scaling/transformations over the robot parameter space (e.g. what if we used log(friction coefficient) instead of friction coefficient?).
    * This is a core issue of the paper, and I believe this should at least be somewhat addressed, potentially with some form of learning (maybe a mapping from physical parameters to some latent space, etc.). Otherwise the method is very un-robust.
* Even under the strong correlation assumption mentioned above, wouldn't it be a more fair baseline for HERD to instead take the path of least euclidean distance (e.g. a Traveling Salesman path) over the tasks, rather than simply sum all (source-to-downstream) steps?
clarity,_quality,_novelty_and_reproducibility: **Clarity:** The paper is easy to follow and well-written, and its idea is conceptually intuitive.

**Novelty and Quality:** Currently, even though the method is simple (which is good) and is experimentally verified, the paper lacks novelty due to the unsolved vulnerability around the robustness of using purely euclidean distance (see Weaknesses section). This could lead a robotics practitioner to have reservations on applying the method to a different robot task (with completely different physical parameter mappings).

**Reproducibility:** Given the straightforwardness of the method, the paper should be easily reproducible.
summary_of_the_review: The method is clear, intuitive, simple and has been shown to work on a handed robot. However, as seen from the "Weaknesses" section, my biggest concern is its assumption that euclidean distance in physical parameter space is highly correlated with the sample efficiency of policy transfer. Thus I currently propose weak reject.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper529/Reviewer_DsD8
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper introduces a method to transfer a policy from one robot to multiple similar robots. This is achieved by first transferring the policy to a meta robot before finetuning it again for different source robots. Experiments on transferring policy from a five-finger robot hand to robot hands with fewer fingers show the effectiveness of the methods. Ablations are also done on how to determine the meta robot.


strength_and_weaknesses: Pro:
1. The problem is well-motivated, and the proposed method is intuitive.
2. In the proposed experiments,  the proposed method beats the baseline method by a large margin, with appropriate ablations.

Con:
1. All experiments are transferring policy from more complicated robots (5 fingers) to less complicated ones. What happens if we want to transfer from the simpler robot (2 fingers) to more complicated ones? Will the reverse direction be effective as well?

2. I don't agree with the logic behind this sentence. "It shows that, transferring the policies to multiple related target through a meta
robot can possibly help each robot improve their learning efficiency." If we only consider one single target robot, the two methods just provide different paths during evolution. And the path provided by the proposed approach is kind of arbitrary based on other randomly chosen target robots. But I agree it is interesting that a straight path is not optimal and worth future investigation. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, and I believe is reproducible.
summary_of_the_review: The problem in the paper is well motivated, the proposed method is intuitive and well executed and can serve as a strong baseline for future research in this direction. 

The only issue I have is a lack of investigation of how the proposed method performs in the reverse direction (simpler robot to more complicated one). This setting might even be more practical since it might be easier to train a policy for simpler robots at the beginning. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper