Paper: Leveraging Behavioral Cloning for Representation Alignment in Cross-Domain Policy Transfer (ICLR-2024)

Reviewer comment (Evaluation):
In Table 1, the proposed model commonly has a larger standard deviation than the compared methods. It would be good if the authors could provide more analysis of these results. Besides, I wonder if PLP (including PLP-disc) is sensitive to the hyperparameter choices.

Author response:
The score depends highly on the success of feature alignment or correspondence learning for domain translation. The scores are not distributed around the mean values but take either low or high values. So PLP or other methods have a larger distribution when successful (i.e., a higher one takes a closer value to 1.0).
Besides, we added an ablation study for hyperparameter choices to Appendix A.6 and Figure 9.
We observe that the performance was not highly sensitive to the MMD coefficient.

Reviewer comment (Evaluation):
In Table 2, the ablated variant of the proposed model, BC + MMD, outperforms PLP-full on three out of six tasks. This raises a question about the necessity of the TCC component in the overall framework. Further clarification on the role and impact of TCC in improving performance across various tasks would provide valuable insights.

Author response:
It is natural that the information on task progression brought by TCC is task-dependent.
TCC might not necessarily be included in PLP-full for all tasks, but we keep it in the method considering its contribution to alignment observed in Figure 3e and Figure 5 for this submission.

Reviewer comment (Experiments):
The method's effectiveness appears to be closely tied to the similarity between the source and target domains. However, the paper lacks an in-depth analysis of this issue concerning PLP's performance in various practical scenarios. It would be valuable to explore how the method performs when dealing with expert demonstrations solely from a highly dissimilar source domain, as well as when utilizing demonstrations exclusively from the target domain (where the source and target domains are identical). Additionally, the paper could benefit from further discussions regarding strategies for identifying a reliable source domain.

Author response:
This is true. As long as we try to perform offline cross-domain transfer without further exploration, the source domain and target domain have to be similar to some extent.

In this paper, we focused on the simplest situations where existing methods even struggle, but it would be interesting to examine the applicability of the method in more advanced settings. As long as the structures of tasks are similar and the proxy tasks cover necessary states, the method should work given a sufficient amount of data.

We do not get your point about this. Isn’t it a simple imitation or demonstration-guided learning within a single domain?

Reviewer comment (Presentation):
A minor issue: The description in the caption for Figure 4(e) appears to correspond to Figure 4(f).

Author response:
Thank you for pointing it out. We fixed it.

Reviewer comment (Novelty):
Novelty: learning a canonical representation space, as well as MMD and TTC, has already been applied in many transfer learning and cross-domain settings, not to mention that TCC actually proves to trivially contribute to performance improvement in experiement section. In the design of PLP, I simply see a combination of previous well-founded thoughts, however, expecting to see something more distinctively insightful to shed light on the avenue of cross-domain policy transfer where over recent years people have failed to deliver something new from a technical angle.

Author response:
Thank you for reviewing our paper.

>  expecting to see something more distinctively insightful to shed light on the avenue of cross-domain policy transfer where over recent years people have failed to deliver something new from a technical angle.

We agree with your point. We conjecture the relatively slow progress in this area can be partially attributed to the complicated structure of methods which makes them harder to extend. This paper aims to organize and simplify the necessary components to tackle various domain shifts by demonstrating that our minimalistic approach actually outperforms previous approaches, expecting future extensions added to the simple framework.

Reviewer comment (Theory):
Formula misalignment: the formulation of $\mathcal{L}\_{TCC}$ simply involves two encoded trajectories, which poses a mismatch with $\mathcal{L}\_{BC}$ and $\mathcal{L}\_{MMD}$ that are in expectation format over datasets.

Author response:
Thank you for reviewing our paper.

> Formula misalignment

> misclaim

> misspecification

Thank you for pointing the issues out. We fixed them.

Reviewer comment (Writing):
Shorthand misclaim: authors explains P2P and P2A in Figure 3 neither in the caption nor in the paragraph they first mention these shorthanded parlances in the main text (P5), but eventually in the experiment section, which would probably make readers confused at first.

Author response:
Thank you for reviewing our paper.

> Formula misalignment

> misclaim

> misspecification

Thank you for pointing the issues out. We fixed them.

Reviewer comment (Writing):
Seemingly symbol misclaim: inner optimization variable "q" appears to be missed to claim about what it means.

Author response:
Thank you for reviewing our paper.

> Formula misalignment

> misclaim

> misspecification

Thank you for pointing the issues out. We fixed them.

Reviewer comment (Writing):
Citation misspecification: "Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. XIRL: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning, pp. 537–546. PMLR, 2022." should be CoRL 2021 acceptance instead of 2022.

Author response:
Thank you for reviewing our paper.

> Formula misalignment

> misclaim

> misspecification

Thank you for pointing the issues out. We fixed them.

Reviewer comment (Experiments):
The paper posits that existing domain translation methods struggle with large discrepancies between domains and may not provide stable cross-domain alignment. However, this claim lacks empirical evidence or a theoretical foundation within the paper, which weakens the argument. A comparative analysis or quantitative evidence would strengthen the case for the proposed method's necessity and superiority. Besides, in general, we conduct the domain adaptation when there is no huge distribution shift. For example, in [1], the target shift is difficult to solve if there are no further assumptions. The examples shown in the introduction parts have less practical application significance, such as the adaption from the agent with no legs to other agents with multiple legs.

Author response:
> The paper posits that existing domain translation methods struggle with large discrepancies between domains and may not provide stable cross-domain alignment. However, this claim lacks empirical evidence or a theoretical foundation within the paper, which weakens the argument.

We extensively conduct comparison experiments between PLP and domain-translation-based methods (GAMA and CDIL), although we do not provide any theoretical support for it. 

> 1) Why do existing domain translation methods struggle with large discrepancies? 

Without supervision for correspondence, it can be difficult to find cross-domain correspondence via adversarial domain translation. It might be different in situations like style transfer. In this sense, it’s reasonable for GAMA and CDIL to work well in P2P. We have no good explanation for why these methods did not work well in R2R.

> 2) How can we verify this

GAMA and CDIL are based on domain translation, and we empirically show PLP’s superiority over these methods. In particular, GAMA and PLP-disc are architecturally close to each other and the main difference lies in the difference between domain translation vs representation alignment using multi-domain BC.

Reviewer comment (Experiments):
For this toy motivational example, despite acknowledging this issue, this paper doesn't involve it in their experimental evaluations. Including such a scenario, especially one that has been previously explored in the literature, would not only validate the method's effectiveness in extreme cases but also provide a direct comparison to existing solutions. This toy experiment has been explored in [2].

Author response:
> For this toy motivational example, despite acknowledging this issue, this paper doesn't involve it in their experimental evaluations.

In the example, we intended to represent the Point2Ant example, so we have comparison results in the experiment section. As for the experiments in [2], it’s not trivial to use it in our experiments since we need multiple tasks to construct proxy task datasets, though we find it useful for constructing multiple domains easily.

Reviewer comment (Novelty):
The paper could benefit from a more thorough discussion of other methods that tackle generalization in reinforcement learning. By comparing the proposed method to these existing approaches, the authors would better situate their work within the current research landscape and potentially highlight the unique advantages of their approach. For example, [2,3,4,5] also explores the generalization problem in the reinforcement learning field.

Author response:
>  4) What is the unique contribution beyond MMD and what are the advantages of this method over [2,3,4,5]?

First of all, the introduction of MMD is not the main contribution of our paper. The main point is the utilization of signals from multi-domain BC, which results in our simple architecture.  
As for the comparison to DBC-based methods [2,4] and state abstraction with dynamics modeling [3], one obvious difference seems to be that these methods rely on the identicality of action spaces.
PLP only utilizes expert demonstrations that should correspond to high-reward actions, whereas DBC-based ones need dense reward signals for every state and every task to find an important structure of state space. DBC also models transition dynamics and PLP might benefit from it as well. For [5], we do not find high relevance to our approach at the moment.

Reviewer comment (Novelty):
The potential novelty issue. This method is based on MMD, which is widely used in domain adaptation. What is the unique contribution to introducing MMD into reinforcement learning? Do the weaknesses in MMD still exist in this field, such as MMD's sensitivity to kernel choice and its performance in high-dimensional spaces?

Author response:
> What is the unique contribution to introducing MMD into reinforcement learning?

Unfortunately, we don’t have a clear answer to it. The community somehow preferred GAN-like domain-discriminative training over the years. This paper demonstrates that, at least in our use case of representation regularization on domain-shared latent space rather than on raw states and actions, MMD can help better in shaping a latent structure especially when the domains are not symmetric.

>  4) What is the unique contribution beyond MMD and what are the advantages of this method over [2,3,4,5]?

First of all, the introduction of MMD is not the main contribution of our paper. The main point is the utilization of signals from multi-domain BC, which results in our simple architecture.  
As for the comparison to DBC-based methods [2,4] and state abstraction with dynamics modeling [3], one obvious difference seems to be that these methods rely on the identicality of action spaces.
PLP only utilizes expert demonstrations that should correspond to high-reward actions, whereas DBC-based ones need dense reward signals for every state and every task to find an important structure of state space. DBC also models transition dynamics and PLP might benefit from it as well. For [5], we do not find high relevance to our approach at the moment.

Reviewer comment (Evaluation):
While the method claims to be computationally simpler, the paper does not provide a detailed analysis of computational costs or training times compared to other methods. Such an analysis would be beneficial for practical applications where computational resources and efficiency are critical factors.

Author response:
>  the paper does not provide a detailed analysis of computational costs or training times

We do not claim the computational efficiency of PLP. We claim PLP is much simpler than existing domain-translation methods in terms of the number of loss terms and accompanying coefficients and the number of necessary models. The simplicity of PLP makes it amenable to possible future extensions.

Reviewer comment (Novelty):
Relation to work in generative modelling: The proposed cross modal domain alignment architecture is very similar to “Latent Normalizing Flows for Many-to-Many Cross-Domain Mappings, ICLR 2020”. This work also includes domain specific encoders/decoders and a shared common latent space aligned using a INN.

Author response:
Thank you for sharing the interesting related paper. The architecture they proposed seems somewhat similar to ours, but in our case, we cannot utilize supervision of one-to-one correspondence between modalities, which makes the problem more difficult.

Reviewer comment (Experiments):
The paper should discuss the evaluation datasets/tasks in more detail. E.g. for the R2R-Lift task, the paper should explain in more detail the action spaces of the source and target domains. The paper only mentions that “a robot has to learn from another robot with a different number of joints and different grippers”. How much of a change in the action space / number of grippers can the proposed PLP method deal with?

Author response:
The details are described in Appendix C.1. For example, Sawyer and UR5e robots have a 32-dim state and 37-dim state, respectively with the same format of action space, although the outcome of action, especially that of “close gripper”, is similar but not identical because the shape of gripper differs (Figure 4c).

Reviewer comment (Evaluation):
In Figure 5, the paper should explain in more detail the factors that effect the alignment across tasks. E.g., why is there is a significant difference in the alignment between P2P-umaze and P2P-medium?

Author response:
It is actually not completely clear to us. On the difference in the alignment score between P2P-umaze and P2P-medium, the smaller number of proxy tasks might make alignment a little inaccurate. As for the downstream performance, we observe that it failed only when the target task was at the end of the maze. The agent successfully moved to the place right before the goal and failed to proceed to the goal. It is probably because the dataset for the alignment only contains trajectories of an agent moving from the goal place to the center as we remove the demonstrations for the target task in the alignment phase. The method might still not be very robust to out-of-distribution actions, although it can successfully handle such actions in V2V-Open.

Reviewer comment (Experiments):
The paper should also discuss the evaluation datasets/tasks in more detail.

Author response:
The details are described in Appendix C.1. For example, Sawyer and UR5e robots have a 32-dim state and 37-dim state, respectively with the same format of action space, although the outcome of action, especially that of “close gripper”, is similar but not identical because the shape of gripper differs (Figure 4c).

Reviewer comment (Experiments):
Expand the variety of domain transfers. This paper completed P2P and P2A. But we have never had a robot resembling a Point robot in the real world. Why not Ant2Hopper? Can you run more combinatorial options: Ant2Hopper, Ant2Walker2D? Can you include multiple source domains e.g., Ant, Hopper, Walker2D and HalfCheetah? With the promise of a general alignment method, it is necessary see how well the method perform on general and more realistic domains. While P2P and P2A transfers have been completed, the absence of a Point robot raises questions. Exploring transfers like Ant2Hopper or Ant2Walker2D and including multiple source domains (e.g., Ant, Hopper, Walker2D, HalfCheetah) would significantly showcase the generalizability of the method across more realistic domains.

Author response:
In these standard benchmarks such as HalfCheetah and Hopper, it’s not easy to create multiple distinct tasks as the agents are only moving forward if we understand correctly. One option would be using move forward and backward as a proxy task and target task, respectively.  

We added the results of Ant2Spider to the Appendix, although the morphologies are not largely different.

Reviewer comment (Experiments):
Reevaluate the manipulation task in the R2R-Lift scenario. The current evaluation R2R-Lift is completed on end effector delta space. So the task setup combines (1) tabletop manipulation task like lift where the robot barely needs to rotate the wrist, (2) in end effector delta space, (3) using low dimensional vector space representation. The current setup is too simplistic to conclusively determine the method's effectiveness. Introducing tasks that require wrist rotation, controlling manipulators over joint space, or running R2R in the visual space could provide a more robust assessment.

Author response:
Although P2A (and Ant2Spider added to the Appendix) includes control in a more complex action space, it would be valuable to confirm it in manipulation tasks as well.  
Control based on the pixel observation is covered by V2V experiments.

Reviewer comment (Experiments):
Increase the robustness of statistical claims by conducting more trials. Ideally, such simulator task should be evaluated with ≥3 different random seeds across ≥20 runs.

Author response:
We apologize for not clearly stating it. We mentioned the scores are averaged over nine runs, but in each seed, we calculated an average of 100 runs. The large standard deviations are not due to the small number of runs but the outcomes of genuine unstableness, unfortunately. The error of the mean estimation is much lower than the values shown in the tables. We added this to the paper.

Reviewer comment (Theory):
Discuss the assumption that multiple MDPs share a common latent structure. The paper should acknowledge the potential risks of aligning distributions that do not naturally overlap, and if possible, provide an argument against this critique.

Author response:
We think we cannot completely remove this risk as the other methods cannot. As we increase the coefficient of the MMD term, the method would enforce the overlap ignoring its effect on action prediction (i.e., BC term). We can mitigate the issue by encouraging more state-to-state alignment either in a supervised or unsupervised way such as TCC.