From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: notable-top-25%
metareview:_summary,_strengths_and_weaknesses: The paper aims to learn a single policy for different forms of agents to perform multiple different tasks. To this end, a benchmark, called MxT-Bench is proposed, that procedurally generates different morphology-task combinations. The paper also proposes control graphs as a generic representation of observations, actions, and tasks. 

Strengths:
- The paper motivates the approach very well and it nicely presents the ideas.
- The paper includes very interesting ideas such as policy distillation when there is a mismatch between state and action spaces across tasks and also representing the control graphs in a transformer architecture.
- The benchmark enables a systematic study of these approaches. The efforts in this domain are quite scattered. 
 - The experiments and the analysis are thorough and provide interesting insights.

Weakness:
- The embodiments and the tasks are quite simplistic. 

The reviewers are generally positive about the paper. The AC also finds the ideas very neat and the experiments solid. The only drawback is the simplicity of the embodiments and tasks, but that is fine since the paper proposes interesting ideas. In the future, more complex scenarios should be considered. The AC recommends acceptance.

justification_for_why_not_higher_score: The embodiments and tasks are somewhat simplistic. I would vote for Oral if the paper considered more realistic tasks and robot embodiments.
justification_for_why_not_lower_score: The paper presents interesting ideas, and the experiments are thorough. So, I believe choosing the paper as Spotlight will highlight the work and encourage the community to continue research in this domain. 
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper3487/Reviewer_UDt3
title: response to authors
comment: Thanks for the detailed response! I am still generally borderline. The concerns about W1, W2, and W4 largely remain. Thanks for the additional experiments in W3. Concatenating the previous steps seems a bit hard to scale to more complex tasks, but one could also imagine alternative recurrent architectures here.

From: ICLR.cc/2023/Conference/Paper3487/Authors
title: Followup Response
comment: We really appreciate the detailed suggestion during this discussion period. All the comments were helpful in improving our paper.

To clarify the terminology of "multi-task", we have added the footnote (in Section 1):

> Here, "task" means what each agent should solve. See Section 3 for the detailed definition. When considering the models, as slightly overloaded, it may imply morphological diversity as well.

We'd like to mention that, in Section 3 (Goal-conditional RL & Morphology-Task Generalization), we clearly state task $\psi$ defines reward $r_{\psi}$ and goal distribution $p_{\psi}$, and morphology $m$ defines state **$s^m$** and action **$a^m$**.

It would be great if you take these into consideration in the following discussion period.

From: ICLR.cc/2023/Conference/Paper3487/Reviewer_feBA
title: Response 
comment: Thanks for changing the title, I think it reflects the real contribution of the paper and intuitively clear now. 

Thanks for the goal clarification, now it makes sense to me. 

I will keep discussing the paper with other reviewers and do not have any questions to the authors.

From: ICLR.cc/2023/Conference/Paper3487/Authors
title: Third Response to Reviewer feBA
comment: We thank the reviewer feBA for the constructive and suggestive discussion. We answer your questions and concerns as follows:

**> Title**

We are now convincing that "control graph" might be an ambiguous term outside of the RL community. So, following your suggestions, we'd like to change the title to "**A System for Morphology-Task Generalization via Unified Representation and Behavior Distillation**", and also change "control graph (CG)" to "**morphology-task graph (MTG)**", which is a straightforward name to express that our graph representation is an extension of morphological graph for the multi-task (or multi-goal) settings. Besides, we have also added "Working memory graphs. Loynd et al., ICML 2020." as a reference.


**> Another example of morphology-task graph with extra goal E**

The setting is the same as the second response. We assume the agents should move body A to goal E (measured by the distance between xy coordinate). MTGv1 and MTGv2 are the following vectors:

MTGv1:
```
[
  [x_A, y_A, z_A, v_xA, v_yA, v_zA, 0, 0, x_E, y_E],
  [x_B, y_B, z_B, v_xB, v_yB, v_zB, 0, 0, 0, 0],
  [x_C, y_C, z_C, v_xC, v_yC, v_zC, x_D, y_D, 0, 0],
]
```

MTGv2:
```
[
  [x_A, y_A, z_A, v_xA, v_yA, v_zA, 0, 0, 1, 1],
  [x_B, y_B, z_B, v_xB, v_yB, v_zB, 0, 0, 0, 0],
  [x_C, y_C, z_C, v_xC, v_yC, v_zC, 1, 1, 0, 0],
  [x_D, y_D, z_D, v_xD, v_yD, v_zD, 1, 1, 0, 0],
  [x_E, y_E, z_E, v_xE, v_yE, v_zE, 0, 0, 1, 1],
]
```

> Is the assumption that multiplicative interaction is easier to learn to do this kind of operation? This should be made explicit in the paper.

We appreciate your suggestion. We have added an explanation about this (our assumption) in Section 4.3, "Transformer for MTG".

From: ICLR.cc/2023/Conference/Paper3487/Reviewer_feBA
title: response
comment: > Our results suggest that our unified IO may limit data representation it can ingest, but strongly preserves 3D geometric relationships that are crucial for any physics control problem, and we empirically show it outperforms naive tokenization in our control-focused dataset.

I now understand your point, thanks. I wonder if the difference in the performance because of the naive tokenisation or because some fundamental issue with it. I bet it's about the implementation and lack of proper positional embeddings.

> I agree that it might be possible that MLP also calculates the distance for each node from the goal, but we think it assumes the gradient descent and supervised learning objective (in Section 4.2) could implicitly induce the proper weights (1 & -1) for hidden units. In contrast, our formulation explicitly describes the local relations between goals and target nodes and embeds the global relation among the entire morphology via multi-head self-attention (or message passing).

Is the assumption that multiplicative interaction is easier to learn to do this kind of operation? This should be made explicit in the paper.

From: ICLR.cc/2023/Conference/Paper3487/Reviewer_feBA
title: response
comment: > Title

I quite disagree for two reasons. 

1) Control graph is quite confusing since it's unclear what does the graph control and why the input representation is 'control'. Moreover, it reduces the potential impact of your paper if applied out of context of continuous control. 
2) 'We believe control graph is worth to name as novel concept', I don't agree with this either. I do not think that the difference with prior work (NerveNet, SMP, Amorpheus) regarding goal representation is conceptually that big so that there should be a totally new name for this, especially when the name does not include the goal. That's why I think that the 'input representation' is a better choice, though it might not sound that flashy. There is actually a good name for this kind of input representation which is established in the community: 'factored observation'. See 'Working memory graphs' Loynd et al, ICML 2020 for more details (sorry I did not mentioned this very related paper earlier, I have just remembered about it).

> but in this paper, we treat them separately and multi-task just stands for multi-goal settings.”

I appreciate extending the related work, but I still find the solution to keep the existing terminology confusing. I will discuss this with other reviewers.

> Example of control graph v1/v2 a graph A->C<-B with a goal-node D

Thanks for adding these, I now fully understand the distinction. If you still have time to respond, could you give an example when we need an extra goal E in addition to the goal D already?

From: ICLR.cc/2023/Conference/Paper3487/Reviewer_zDig
title: Reply to Author
comment: Thanks for updating the title and Figure 1. I don't have further questions for the authors.

From: ICLR.cc/2023/Conference/Paper3487/Authors
title: Second Response to Reviewer feBA (2/2)
comment: > Sorry, I don't understand what this mean. Could you, please, elaborate or give an example?


As we mentioned earlier, our control graph shows better generalization than the tokenized variant (per every dimension by tokenizer module) similar to GATO [Reed et al. 2022] (Section 1, Section 5.1, Appendix E). While the current trend suggests such tokenization seems a more "transformer-friendly" representation than continuous values [Dosovitskiy et al., 2020, Alayrac et al., 2022, Reed et al. 2022, Janner et al., 2021], it loses some semantic information about the continuous value or 3D geometric information due to the discretization and embedding. Our results suggest that our unified IO may limit data representation it can ingest, but strongly preserves 3D geometric relationships that are crucial for any physics control problem, and we empirically show it outperforms naive tokenization in our control-focused dataset.


- [Reed et al., 2022] A generalist agent.
- [Dosovitskiy et al., 2020] An image is worth 16x16 words: Transformers for image recognition at scale.
- [Alayrac et al., 2022] Flamingo: a visual language model for few-shot learning.
- [Janner et al., 2021] Offline reinforcement learning as one big sequence modeling problem.


> I don't think I understand this. When I have a goal concatenated with the observational vector, an MLP can simply calculate the difference for each node with the goal for separate hidden units by learning a -1 weight for the goal observations and 1 for limb nodes. Or do you mean something else?


We agree that it might be possible that MLP also calculates the distance for each node from the goal, but we think it assumes the gradient descent and supervised learning objective (in Section 4.2) could implicitly induce the proper weights (1 & -1) for hidden units. In contrast, our formulation **explicitly** describes the local relations between goals and target nodes and embeds the global relation among the entire morphology via multi-head self-attention (or message passing).

From: ICLR.cc/2023/Conference/Paper3487/Authors
title: Second Response to Reviewer feBA (1/2)
comment: We are glad to see our response could address some of your concerns. We answer the remaining questions below. We hope these responses can also clarify your unclear points, but please let us know any remaining questions or concerns if you have.

**> Title**

Thank you for your feedback. For the following reasons, we still believe the revised title is a decent and better choice than before. We hope we reach some consensus, but please let us know if the title remains as a major issue for your evaluation. We will keep considering the title until the rebuttal period ends.

(1) The current title emphasizes that this paper proposes the entire system for morphology-task generalization, rather than the novel input representation (i.e. control graph) alone. Moreover, since "morphology" and "graph" often cooccur in RL, control graph might be fitting the literature.

(2) We believe control graph is worth to name as novel concept, because it was unclear how we could incoopolate goal information into the morphological graph, and because, inspired by scene graph, we could describe all in the scene into the single graph, which can bridge the community between morphology generalization and task (or goal) generalization.

(3) We clearly state the notion of control graph in the abstract and the caption of Figure 1. We think the current title is not so out of context.

Abstract:
> ... In order to align input-output (IO) interface among multiple tasks and diverse agent morphologies while preserving essential 3D geometric relations, we introduce control graph, which treats observations, actions and goals/task in a unified graph representation.

Figure 1:
> ... we convert stored transitions to the control graph representation
to align with unified IO interface (Section 4.3) ...

**> “multi-task” and “policy distillation”**

We appreciate your raising concerns on those. We added the extended related work in Appendix M and mentioned it in the main text (Section 2). For instance, “while, from a broader perspective, the notion of "multi-task" in RL may contain both morphological and goal diversity, but in this paper, we treat them separately and *multi-task* just stands for *multi-goal* settings.” We hope these descriptions would clarify the terminology of “multi-task” and “policy distillation” in our paper.


**> Example of control graph v1/v2 a graph A->C<-B with a goal-node D**

For convenience, we consider the node features that only include position and velocity. We also assume this task requests the agents should move body C to goal D (measured by the distance between xy coordinates). CGv1 representation gives us the following vectors:
```
[
  [x_A, y_A, z_A, v_xA, v_yA, v_zA, 0, 0],
  [x_B, y_B, z_B, v_xB, v_yB, v_zB, 0, 0],
  [x_C, y_C, z_C, v_xC, v_yC, v_zC, x_D, y_D],
]
```

The last two dimensions indicate which node is a target and where the goal is. In case of being interested in z coordinate, we just use [z_D, 0]. Also, if we have extra goal E, we prepare another two dimensions for [x_E, y_E].

Similarly, CGv2 represents the observation as follows:
```
[
  [x_A, y_A, z_A, v_xA, v_yA, v_zA, 0, 0],
  [x_B, y_B, z_B, v_xB, v_yB, v_zB, 0, 0],
  [x_C, y_C, z_C, v_xC, v_yC, v_zC, 1, 1],
  [x_D, y_D, z_D, v_xD, v_yD, v_zD, 1, 1],
]
```
The models can easily distinguish between morphological nodes and goal nodes because some dimensions of goal node features have different natures or are padded as 0 (e.g. the velocity of a fixed goal is always 0, z is always constant, etc).

> Is this for convenience of is it a limitation of your method? If that's a limitation, could you, please, add this discussion to the paper?

We have checked that our trained model could be generalized to the agents with an extra number of limbs (e.g. 7-leg ant or 8-body centipede, OOD samples in Table 4), but have not extensively increased the number of limbs until the model could not deal with them.
In Appendix D, we added the explanation that this might limit the morphology generalization in the open-ended evaluation.

**(Continue to the next page)**

From: ICLR.cc/2023/Conference/Paper3487/Reviewer_feBA
title: response
comment: Thanks for the clarifications! I have a couple of questions left.

> "**A System for Morphology-Task Generalization via Control Graph and Behavior Distillation**

I appreciate the authors changing the title, but I still believe that 'control graph' is a subpar choice since the reader has no idea what 'control graph' is before reading the paper. Since control graph is just a way to represent the inputs, I would stick to smth like 'Input representation and behaviour distillation for morphology-task generalisation'.

>  That is why we treat morphology and task (or goal) as separate concepts, rather than solely focusing on morphological diversity.

I am fine with treating morphology separately, but I would keep everything under the 'multi-task' umbrella and explicitly mention what is varied. For me, and I guess for other multi-task learning researchers, this is quite confusing and might bring additional confusion to the terminology in case your paper gets traction.

> Thank you for pointing out the additional literature. We have cited the suggested papers properly in the revised paper.
> To clarify the relationship, we also added the reference on "policy distillation" as you suggested.

Thanks for adding the related work.  Could you, please, point me to the place in the paper where you 'clarify the relationship?' I've seen the refs to Rusu etc, but I couldn't find the clarification of the relationship in the paper. Currently, 'behaviour distillation' looks as if it's something super novel, whereas in practive it is really not. I think it should be more clear for the reader who does not have enough context.

> **Details of Control Graph v1/v2**

I am a bit confused here. Do I get it correctly that for v2 you have some specific goal node and the specific dimensions of this node feature vector define for which morphological node this is the goal? If possible, could you, please, give an example, for instance for the graph A->C<-B with a goal-node D.

> we have chosen zero-shot agents that would not exceed the maximum number of nodes in the training sets for convenience

Is this for convenience of is it a limitation of your method? If that's a limitation, could you, please, add this discussion to the paper?

> but our results imply, the representations that maintain some features, might be better in some cases.

Sorry, I don't understand what this mean. Could you, please, elaborate or give an example?

> Control graph could naturally incorporate 3D relational information between the goal and the corresponding morphology nodes (as extra node features or node itself), while typical goal-conditioned policies often just concatenate them.

I don't think I understand this. When I have a goal concatenated with the observational vector, an MLP can simply calculate the difference for each node with the goal for separate hidden units by learning a -1 weight for the goal observations and 1 for limb nodes. Or do you mean something else?

From: ICLR.cc/2023/Conference/Paper3487/Authors
title: Summary of Author Response to All the Reviewers (2/2)
comment: **> Temporal memory**

Reviewer UDt3 raised a concern whether control graph can be applicable to the temporal-memory-required tasks. We have added the experiments on temporal memory baseline for ant-agent tasks in Appendix G, which concatenates the recent $H$-frame control graph as inputs while predicting one-step actions ($H=3$). The results show that incorporating temporal memory would be beneficial when the evaluation tasks are unseen, compared to the original control graph (no temporal information). Historical observation contributes to solving our tasks in MxT-Bench, and our control graph formulation could be naturally augmented with temporal information.

**> Notion of "multi-task"**

We had discussed the coverage of "multi-task" with the reviewer feBA. While, from a broader perspective, the notion of "multi-task" in RL may contain both morphological and goal diversity, **in this paper, we treat them separately, and "multi-task" just stands for "multi-goal" settings**.

The reason why we consider "morphology" and "task (or goal)" separately is that by considering morphology-task generalization, we could study the generalization in RL across all the MDP components (dynamics, reward, state space, action space). In the previous literature, "multi-task" RL in a broader sense has arisen from each MDP component; from different (1) dynamics, (2) reward, and (3) state/action space. We think multi-morphology RL covers (1) and (3), and multi-task (or multi-goal) RL does (2) and (1). In prior works, multi-morphology RL only solves a single task (running as fast as possible), and multi-task RL mainly considers a single-morphology agent that should maximize the different reward functions (defined by different goals) at the same time. We hope our "cartesian product" formulation (Figure 2) could bridge the community between morphology generalization and task (or goal) generalization.

Moreover, we would like to mention that, in Section 3 (Goal-conditional RL & Morphology-Task Generalization), we clearly state task $\psi$ defines reward $r_{\psi}$ and goal distribution $p_{\psi}$ and morphology $m$ defines state **$s^m$** and action **$a^m$**. 
We have also explicitly stated in Appendix M (extended related work) that "multi-task" means "multi-goal" in our setting.
Generally speaking, if we think of the neural network (or model), "multi-task" may include both morphology and goal differences, and if we focus on each robot itself (or agent), "multi-task" should be defined by "goals" (see footnote in Section 1). We think the concept of morphology-task generalization and our MxT -Bench reflect the latter appropriately.
We believe our presentation in the revised paper could properly avoid confusion for the readers.

**> Other changes in the paper**

- We have added the model-size scaling plot in Appendix H, which shows the performance can get better if we increase the number of parameters in Transformer.
- We have added a detailed description of CGv1 and CGv2 in Appendix A.
- We have added the plots of average normalized final distances of our datasets in Appendix C
- We have fixed minor typos or cited other related references.
- We have added the extended related work in Appendix M (update: 2022/11/16).
- We will release our code after the de-anonymization.

**Summary**

We thank all the reviewers again for the detailed and constructive review. We are glad that the reviewers found that (1) proposing a novel benchmark for morphology-task generalization, which is the underexplored area in RL, is a good contribution to the community, that (2) the behavior distillation pipeline would attract more attention to the scalability of RL, as an analogy of other ML fields such as NLP or Vision and Language, and that (3) our extensive "generalization" evaluation (through morphology-task graph and offline behavior distillation) provides some interesting insights and guidance for future directions. We hope our revision of the title and contents, as well as additional temporal memory results, can address all of your concerns. Let us know any remaining questions or concerns if you have.

From: ICLR.cc/2023/Conference/Paper3487/Authors
title: Summary of Author Response to All the Reviewers (1/2)
comment: We would like to appreciate the thoughtful comments from all the reviewers. We revised the manuscript based on your constructive feedback and suggestions (**highlighted as purple**). Our key responses are summarized below:


**> New Title**

As reviewers zDig and feBA suggested, we would like to change the title to improve the comprehensiveness about our contributions:

- **Before**: Control Graph as Unified IO for Morphology-Task Generalization

- ~~**After**: *A System for Morphology-Task Generalization via Control Graph and Behavior Distillation*~~

- **After: A System for Morphology-Task Generalization via Unified Representation and Behavior Distillation** (update: 2022/11/17)

**> Renamed Graph Representation**
(update: 2022/11/17)

After a constructive discussion with the reviewer feBA, we decided to rename our proposed graph representation to avoid ambiguity:

- **Before**: Control Graph (CG)

- **After: Morphology-Task Graph (MTG)** 

This is a more straightforward name to express the aspect that our graph representation is an extension of morphological graph for the multi-task (or multi-goal) settings. We are sorry that the reviewers should replace "control graph" with "morphology-task graph" in the previous discussion.

**> Structure of the paper**

As reviewer zDig suggested, we have added cross references to related sections to Figure 1, which could help better understand the connection among contents.

In addition, following the proposal from the reviewer feBA, we have moved the representation selection experiments (node features, Section 5.3) to the main text, while still keeping attention analysis in Appendix K. This change makes the paper reflect our entire proposal in Figure 1.


**> Coverage of MxT-Bench**

While reviewers UDt3 and JKD7 raised some concerns related to the coverage of MxT-Bench, we believe the current environments in MxT-Bench are complex enough to show the proof-of-concept and merits of control graph: for instance, the learned policy successfully solves **multi-goal** tasks such as twisters and also could deal with the interaction with movable objects by **maintaining a conservative momentum** (touch task). These constraints are directly connected to realistic requirements.

Towards more realistic settings beyond the skeleton of animals, we might realize them by curating diverse enough real robot assets on some hardware-accelerated simulators, such as IsaacGym. However, as stated in Section 6, one major bottleneck is actuator or physics incompatibility across different simulators, rather than control graph or behavior distillation. We will actively engage in such directions to realize a broader spectrum of generalization across different and diverse robots.


**> Behavior distillation**

Reviewers feBA and UDt3 raised concerns related to behavior distillation (importance, efficiency, terminology). We would like to clarify those as follows:

**Importance and Efficiency** 
Offline behavior distillation can be more computationally or time efficient than jointly trained (from scratch) off-policy RL [Huang et al., 2020, Kurin et al., 2021, Trabucco et al., 2022] for the iteration of the trials. Critically, offline training allows us efficient architecture and representation selection, which is often infeasible in multi-task online RL, so it is an important piece of our algorithm proposal and paper contribution. Recent work [Chen et al., 2021, CoRL Best Paper] also suggests the merits of such a pipeline (per-skill online RL generation, and all-skill offline distillation).


- [Huang et al., 2020] One policy to control them all: shared modular policies for agent-agnostic control.
- [Kurin et al., 2021] My body is a cage: the role of morphology in graph-based incompatible control.
- [Trabucco et al., 2022] Anymorph: Learning transferable policies by inferring agent morphology.
- [Chen et al., 2021] A system for general in-hand object re-orientation.


**Terminology** 
"Policy distillation" has been proposed and studied for a while, where a single student policy distills the knowledge from multiple teacher policies to obtain better multitask learners. In contrast, we call our protocol "behavior distillation", where the single student policy distills the knowledge from multi-source offline data (e.g. human teleoperation, scripted behavior, play data), not limited to RL policy, since recent RL or robotics research often leverage such a data-driven approach for scalability. To clarify the relationship, we have added references on "policy distillation", and a paragraph (Appendix M), in the revised paper.

From: ICLR.cc/2023/Conference/Paper3487/Authors
title: Response to Reviewer feBA (2/2)
comment: > What do you think the wider implications of the proposed input representation are? Can we draw any conclusions that can be used outside the continuous control setting?

As we showed in Table 1 and Appendix E, our control graph performs better generalization than the tokenized variant similar to GATO [Reed et al. 2022]. The current trend seems to be tokenization and treating it as a language [Dosovitskiy et al., 2020, Alayrac et al., 2022], but our results imply, the representations that maintain some features, might be better in some cases.

- [Reed et al., 2022] A generalist agent.
- [Dosovitskiy et al., 2020] An image is worth 16x16 words: Transformers for image recognition at scale.
- [Alayrac et al., 2022] Flamingo: a visual language model for few-shot learning.


> "policy distillation" is an established term, why do you use "behaviour distillation" instead?

Recent RL or robotics research often deals with multi-source offline data (human teleoperation, scripted, play data, etc.), not limited to RL policy for scalability. Considering those aspects, we prefer behavior distillation. To clarify the relationship, we also added the reference on "policy distillation" as you suggested.


> I do not understand the following sentence "which is significantly more efficient than multi-morphology training done in prior work" (Section 4.2)

Since we train a single-morphology single-task expert policy in a few minutes, that could be more computationally or time efficient than jointly trained (from scratch) off-policy RL [Huang et al., 2020, Kurin et al., 2021, Trabucco et al., 2022] for the iteration of the trials.

- [Huang et al., 2020] One policy to control them all: shared modular policies for agent-agnostic control.
- [Kurin et al., 2021] My body is a cage: the role of morphology in graph-based incompatible control.
- [Trabucco et al., 2022] Anymorph: Learning transferable policies by inferring agent morphology.


> "incorporates goal information while preserving the geometric structure of a task", what is "geometric structure of a task"?

Control graph could naturally incorporate 3D relational information between the goal and the corresponding morphology nodes (as extra node features or node itself), while typical goal-conditioned policies often just concatenate them.


> "we examine how control graph works well..." is unclear to me.

We have changed it to "*why* control graph works well" in the revision.


> Nit: In Section 3, when the Deep RL is introduced, the 'policy' term has not been introduced yet.

Thank you for pointing this out. We have fixed it in the revised version.

From: ICLR.cc/2023/Conference/Paper3487/Authors
title: Response to Reviewer feBA (1/2)
comment: We thank the reviewer for the careful reading and thoughtful comments. We address your concerns below and please let us know if there are further unclear points.


**> The positioning of the paper and title**

To clarify the connection between the title, introduction, and motivation, we would like to change the title to "**A System for Morphology-Task Generalization via Control Graph and Behavior Distillation**", because we believe that proposing (1) morphology-task generalization benchmark, (2) control graph to handle diverse morphology and task, and (3) behavior distillation pipeline for downstream tasks, such as fine-tuning or architecture/representation selection, are equally worth presenting to the community.

As you suggested, we have moved some parts of the representation selection experiments to the main text (while keeping attention analysis in Appendix K). We have also added cross references to related sections in Figure 1 as a guide for a storyline. Please see the revised paper.



**> Terminology of Multi-Task**

In the previous literature, “multi-task” has arisen from each MDP component: from different (1) dynamics [Hallak et al., 2015, Yu et al., 2019], (2) reward [Kaelbling, 1993, Andrychowicz et al., 2016], and (3) state/action space [Kurin et al., 2021, Gupta et al., 2022]. We think that multi-morphology RL covers (1) and (3), and multi-task (or multi-goal) RL covers (2) and (1). By considering Morphology-Task generalization as in our paper, we could study the generalization in RL across all the MDP components (dynamics, reward, state space, action space). That is why we treat morphology and task (or goal) as separate concepts, rather than solely focusing on morphological diversity.

- [Hallak et al., 2015] Contextual markov decision processes.
- [Yu et al., 2019] Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.
- [Kaelbling et al., 1993]  Learning to achieve goals.
- [Andrychowicz et al., 2016] Hindsight experience replay.
- [Kurin et al., 2021] My body is a cage: the role of morphology in graph-based incompatible control.
- [Gupta et al., 2022] Metamorph: learning universal controllers with transformers.


**> Reference**

Thank you for pointing out the additional literature. We have cited the suggested papers properly in the revised paper.

**> Details of Control Graph v1/v2**

Basically, we have padded the incompatible parts of goal information by 0. For both control graph v1/v2, we prepare the binary array that has indicators to the target (or goal) nodes. For v1, we obtain the product of the goal information and binary array, then use it as an extra node feature. For v2, we just add such binary arrays as extra node features (and include goals as extra nodes). We have added this description in Appendix A.

**> Other remaining questions and concerns**

> When you generate tasks, are they all solvable (whatever metric you choose for this). Can you provide plots with the return ranges for single-agent policies you use to collect the data? How did you select thresholds for goal-reaching?

The details of the expert data generation are described in Appendix C. We chose a dense task reward as a proxy metric for solvability and roughly checked whether the average normalized final distance of the datasets from single-agent policies is mostly less than 0.1. The thresholds (in Table 4, 5, 6, 7)  are manually designed from the try-and-error of the easy morphologies (e.g. ant_4, claw_4).  We added the plots of average normalized final distances of those datasets in Appendix C (Figure 10).

> Could you elaborate a bit more on the learnable position embeddings, how exactly is this implemented. When zero-shot transferred, how are position embeddings calculated?

We first compute the maximum number of nodes from all the environments (for both training/zero-shot test sets) we are interested in. Then, by following prior works [Huang et al., 2020, Gupta et al., 2022, Kurin et al., 2021], we have chosen zero-shot agents that would not exceed the maximum number of nodes in the training sets for convenience. If we try the agents that exceed the maximum number of nodes, we might implement the extra operation that repeats the last position's embedding as padding, or even if we just remove position embeddings, Transformer + control graph would work well (see Appendix F for the details of architecture selection).

> How consistent is the behaviour in Figure 6 across the seeds?

We confirmed the consistency across different seeds, while we have not extensively evaluated this across all possible morphology-task combinations.

From: ICLR.cc/2023/Conference/Paper3487/Authors
title: Response to Reviewer zDig
comment: We appreciate the reviewer for the careful reading and constructive suggestions. Please let us know if there are any remaining questions or concerns.


**> Title and overview of the story**

As you suggested, we have added cross references to related sections in Figure 1 to help the understanding of the entire structure of the paper. In addition, we would like to change the title to ~~"**A System for Morphology-Task Generalization via Control Graph and Behavior Distillation**"~~ "**A System for Morphology-Task Generalization via Unified Representation and Behavior Distillation** (update: 2022/11/17)", to improve the connection between the title, introduction, and motivation. We think that proposing (1) morphology-task generalization benchmark, (2) control graph to handle diverse morphology and task, and (3) behavior distillation pipeline for downstream tasks, such as fine-tuning or architecture/representation selection, are equally our contributions. Please see the revised paper.

From: ICLR.cc/2023/Conference/Paper3487/Authors
title: Response to Reviewer JKD7
comment: We thank the reviewer for the thoughtful review and comments. Please let us know any remaining questions or concerns if you have.

**> Attention analysis**

Because another reviewer suggests including the representation selection experiments in the main text, we have moved attention analysis to Appendix K. Figure 12 implies that CGv2 consistently focuses on goal nodes over time, and activates important nodes to solve the task; for instance, in centipede_touch (top), CGv2 pays attention to corresponding nodes (torso0 and goal0) at the beginning of the episode, and gradually sees other relevant nodes (torso1 and torso2) to hold the movable ball.

**> Benchmark across different platforms**

While we included the compositional or out-of-distribution zero-shot results in Table 1 (see Appendix D for the environment division), we also tested box pushing tasks, which are very different from reach, touch and twisters due to the object interaction and manipulation, in Appendix L, which shows some superiority of the control graph v2 as an inductive bias for downstream finetuning. In addition, as we stated in Section 6, one major bottleneck is actuator or physics incompatibility across different simulators (e.g. MuJoCo, Pybullet, Brax, IsaacGym, etc.). We will actively engage in such directions to realize a broader spectrum of generalization across different benchmarks.

From: ICLR.cc/2023/Conference/Paper3487/Authors
title: Response to Reviewer UDt3
comment: We appreciate the careful reading of our paper and detailed discussions. We address your concerns below, and please let us know if there are remaining questions or unclear points.


**> W1**

Regarding the agents’ complexity, we also tested recently proposed complex skeleton agents (called unimals, in Table1) [Gputa et al., 2022] in addition to ant, centipede, claw, and worm. Besides, the goals in our multi-task RL, are sampled from uniform distribution with a predefined range. We agree that it is always preferable to set up as complex environments as possible and see if the proposed method works. However, we believe the current environments are complex enough to show the proof-of-concept and merits of control graph: the learned policy successfully handles **multi-goal** tasks such as twisters and also could deal with the interaction with movable objects by **maintaining a conservative momentum** (touch task). These constraints are directly connected to realistic requirements.

Towards more realistic settings, we might realize them by curating diverse enough real robot assets on some hardware-accelerated simulators, such as IsaacGym. As we stated in Section 6, one major bottleneck is actuator or physics incompatibility across different simulators, rather than control graph or behavior distillation. We will actively engage in such directions to realize a broader spectrum of generalization across different and diverse robots.


- [Gputa et al., 2022] Metamorph: learning universal controllers with transformers.


**> W2**

We have proposed control graph as a unified interface for large scale behavior distillation. Offline behavior distillation can be more computationally or time efficient than jointly trained (from scratch) off-policy RL [Huang et al., 2020, Kurin et al., 2021, Trabucco et al., 2022] for the iteration of the trials. Critically, offline training allows us efficient architecture and representation selection, which is often infeasible in multi-task online RL, so it is an important piece of our algorithm proposal and paper contribution. Recent work [Chen et al., 2021, CoRL Best Paper] also suggests the merits of such a pipeline (per-skill online RL generation, and all-skill offline distillation).


- [Huang et al., 2020] One policy to control them all: shared modular policies for agent-agnostic control.
- [Kurin et al., 2021] My body is a cage: the role of morphology in graph-based incompatible control.
- [Trabucco et al., 2022] Anymorph: Learning transferable policies by inferring agent morphology.
- [Chen et al., 2021] A system for general in-hand object re-orientation.


**> W3**

We have added the experiments on temporal memory baseline for ant-agent tasks from MxT-Bench shown in Appendix G, which concatenates recent $H$ frames control graph as inputs while predicting one-step actions ($H = 3$). The results show that incorporating temporal memory would be beneficial when the evaluation tasks are unseen (i.e. Compositional (Task) and Out-of-Distribution settings), compared to the original one-step control graph. This is a natural trend since morphology-agnostic RL is usually solvable with one-step observation, and meta or multi-task RL often requires temporal information for task identification. Despite the seemingly simple settings, historical observation contributes to solving our tasks, and our control graph formulation could be naturally augmented with temporal information.


| | MLP | Transformer (CGv1) | Transformer (CGv2) | Transformer (CGv2-history) |
| ---- | ---- | ---- | ---- | ---- |
| **In-Distribution** | 0.3035 ± 0.03 |  0.1364 ± 0.04 | 0.0961 ± 0.02  | 0.1015 ± 0.04 |
| **Compositional (Morphology)** | 0.7480 ± 0.02 | 0.1477 ± 0.04 | 0.1010 ± 0.01 | 0.1070 ± 0.03 |
| **Compositional (Task)** | 0.4468 ± 0.11 | 0.2243 ± 0.03 | 0.2201 ± 0.04 | 0.2170 ± 0.01 |
| **Out-of-Distribution** | 0.6788 ± 0.01 | 0.3663 ± 0.05 | 0.3370 ± 0.03 | 0.3316 ± 0.05 |

**> W4**

Related to this, we added the model-size scaling plot in Appendix H. We aligned the number of parameters of the policy between MLP (default, about 2M) and Transformer (default, 1.6M) for a fair comparison. While the results imply Transformer still seems to underfit, Transformer with more parameters (4.8M) achieves better performance, and we may see some saturated performances by increasing the model size and tuning hyperparameters.

From: ICLR.cc/2023/Conference/Paper3487/Reviewer_UDt3
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors propose control graphs as a way to train agents with different morphologies and across different tasks. They develop MxT-Bench to train with different agent morphologies and across different tasks. They show a set of offline experiments that demonstrate the usefulness of their control graphs over MLPs and GNNs.
strength_and_weaknesses: Strengths:
- The paper is well presented and the motivation is very clear.
- The research question of designing architectures that generalize to different agent embodiments and tasks is very interesting, practically useful, and well-discussed.
- The idea for control graphs and setting them up to be used in a transformer is pretty neat!
- The analysis of the attention weights is well written and done.

Weaknesses:
- W1. The agent morphology and task setups are really limiting (agents being basic variants of ant, centipede, and claw agents, tasks being very limited to reach, touch, twister, and push tasks). It would be hard for me to accept the usefulness of control graphs without much more experimentation with different robotic setups and tasks. I would have liked to see something along the lines of trying control graphs with respect to parameterizable simulated robots and with respect to more real-world tasks. It doesn't seem like this work 
- W2. The work only deals with optimizing offline datasets.
- W3. It appears that the Transformer CGv1 and CGv2 do not encode any temporal information, that is, there is no information flow from the previous time steps to the current time step, which would make the current formulation not work well in any tasks that require memory.
- W4. In Table 1, performance for in-distribution seems lower than I would have expected. I would have expected a transformer to practically be able to overfit with near perfect training performance. Why isn't this happening?
clarity,_quality,_novelty_and_reproducibility: The tasks and setup were fairly easy to follow. There's been some recent work in morphology generalization, but not too much with respect to both different morphologies and to different tasks, which makes this work novel in that respect. I don't expect much of an issue with reproducibility as the tasks were relatively simple.
summary_of_the_review: The work is well-presented and easy to follow, and the direction of the work is quite fascinating. However, I'm concerned about the lack of demonstrated practical usefulness of control graphs. The experiments were only on relatively simple tasks with basic mujoco-styled agents. I would have liked to see more experiments where the agent requires temporal memory and with more real-world robotic embodiments. As the work stands now, it's hard for me to see the usefulness of using control graphs beyond pretty simple setups.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3487/Reviewer_JKD7
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors created an benchmark called "MxT-Bench", which study the cross product of morphology and task generalization. Moreover, the authors proposed a unified graph-based representation for multi-morphology and multi-task. The authors provide a wide range of experiments and showcase the benchmark and models.
strength_and_weaknesses: I want to preface this by saying that I'm not really familiar with the field of controls across morphologies. 

Strength:
- MxT bench is well designed and seems to be technically sound. This can be of interest to many different researchers.
- The experiment study is quite thorough and offers some interesting insights. 

Weakness:
- I feel like the analysis in 5.3 is not so strong. Especially figure 6, where I don't see much to be honest. 
- I wonder if a comparison of models learned across benchmarks can be useful. For example, does the cross morphology models learned in other benchmarks perform well in MxT?
clarity,_quality,_novelty_and_reproducibility: Great on all front.
summary_of_the_review: I like this paper and think this is above acceptance threshold. However, I'm waiting to see other reviewers's feedback (who probably are more familiar with the field than I am), before giving it a very strong endorsement.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper3487/Reviewer_zDig
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper focuses on multi-morphology and multi-task generalization in robotics. There are three main contributions to this paper. First, a comprehensive benchmark called MxT-Bench is developed for training and evaluating morphology-task generalization. This benchmark also supports the scalable procedural generation of both agents and tasks. Second, a new concept Control Graph is proposed as a universal interface representing observations, actions, and goals/tasks. Last, the authors study several offline supervised behavior distillation methods with different design choices and obtain conclusions about generalizable model design.

Compared to existing works of universal controller and morphology/controller co-design, this paper provides a more comprehensive and systematic design, which has the potential to unify the tools and standards used in this area. 

strength_and_weaknesses: ### **Strength**

* **A systematic platform.** Universal controllers for different robots and co-design of morphology and controller are prevalent topics in robotics. Different papers propose many different simulators and protocols, making the development and comparison between algorithms difficult. The benchmark proposed by this paper looks promising to me. I think it will greatly contribute to this area by unifying the tools and standards used in it.

* **Thorough analysis of experiments.** Beside the benchmark, this paper also explores multiple generalization settings, including multi-morphology, multi-task, and out-of-distribution downstream tasks. Some interesting results will definitely provide guidance for future work.


### **Weaknesses**

One weakness in my opinion is that an overview of the structure of this paper is missing. Since this paper proposes many new concepts, understanding the connections between them could be difficult.
Figure 1 tries to summarize all contents but still needs to be improved. For example, section numbers can be added to different components and main contributions can be highlighted.
The title can also be improved. I think the Control Graph is not the main contribution. A more precise title helps understand the content.

clarity,_quality,_novelty_and_reproducibility: ### **Clarity**

The writing is clear and easy to understand. An overview of the paper structure could help readers quickly understand the connections between all concepts.

### **Quality**

This paper has very high quality. All three contributions are important to this area.

### **Novelty**

The novelty of this paper comes from the new benchmark and unified representation. No novel algorithm is proposed.


### **Reproducibility**
I think the results are reproducible.

summary_of_the_review: I suggest accepting this paper. The morphology-task generalization topic becomes a prevalent trend recently. I believe the benchmark and unified interface proposed in this paper contribute a lot to this area.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper3487/Reviewer_feBA
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a continuous control benchmark for multitask reinforcement learning supporting three axes of change: creature morphology (supporting state and action spaces of different dimensions), task (e.g. reach some place or touch some point), and goal.
The paper further develops the graph-based representation pioneered by NerveNet, adding the support of the goal-directed setting. Finally, the authors investigate policy distillation behaviour on their own benchmark
strength_and_weaknesses: ## Strengths

* The proposed benchmark is a large benchmark supporting a variety of degrees of freedom: morphology, different mass/size for limbs, multiple tasks (using the authors' terminology), goal distribution.
* The paper investigates policy distillation in the incompatible setting (mismatching state and action space dimensions across tasks) which, I don't think, was done before.
* The benchmark seems to be super fast with single-agent PPO taking 5-30 minutes per environment

## Weaknesses

I think the positioning of the paper paper is its main weakness. I am not entirely sure what this paper is about. There are three potential storylines:

* The paper is about the benchmark 
* The paper is a policy distillation study in a novel setting (mismatching state/action spaces across agents)
* The paper is about the input representation

I think, all three stories are equally legit for this paper, but each of the story will define the requirements for experiments, and jumping around the stories will confuse the reader.

For instance, the title of the paper says that the paper is about control graph, i.e. input representation. However, the intro's motivational section speaks about the generalist agent and morphology generalisation. If the paper is about morphology generalisation, then Representation Selection, which is Appendix F now, should be in the main body of the paper etc. I think, the main body can accommodate more information if the transformer basics are moved to the Appendix.

I'm very confused about the terminology used throughout the paper: 'multi-task' is used in a very narrow meaning implying that the agents of the same morphology doing semantically different tasks (e.g. reaching some goal or moving limbs to specific positions). Usually, In RL, a task is a synonym of an MDP, and different morphology (even same morphology with different limb sizes/masses) constitute different tasks. As you mentioned, previous literature (e.g. Gupta et al., 2022, Metamorph) consider morphology/your definition of task separately and that 'broad generalisation over task and morphology jointly remains a long-standing problem'. Do you claim that there is a qualitative difference between generalising across morphologies and generalising across your definition of tasks? In other words, what is the motivation behind separating multitask into two separate groups?

Finally, I find it concerning, that the paper that does policy distillation does not cite policy distillation papers, including Parisotto's "Actor-Mimic" and Rusu's "Policy Distillation". The "Behaviour Distillation" section includes only 2020/21 papers with a single exception of Levine et al 2016. Task generalisation with GNN/Transformers work should include papers on combinatorial optimisation (see Cappart's 'Combinatorial optimization and reasoning with graph neural networks' for references). GNNs for continuous control work misses Blake's "Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezing".

Other questions:
* I am not sure I fully understand how the goals are encoded to either control graph v1 or v2. In v1, when you add the goal observation to the node features, are other nodes' features padded? Do you provide the task identifier somehow (e.g. if that's reach or touch or twisters). In v2, figure three shows the dashed edges from the limb nodes to goal nodes, how do you encode this information in the transformer case? Also, judging by Figure 3, the dimensionalities of the goal nodes for different tasks are different. How do you ensure that they are all the same before going to the GNN/transformer?
* When you generate tasks, are they all solvable (whatever metric you choose for this). Can you provide plots with the return ranges for single-agent policies you use to collect the data? How did you select thresholds for goal-reaching?
* Could you elaborate a bit more on the learnable position embeddings, how exactly is this implemented. When zero-shot transferred, how are position embeddings calculated?
* How consistent is the behaviour in Figure 6 across the seeds?
* What do you think the wider implications of the proposed input representation are? Can we draw any conclusions that can be used outside the continuous control setting?

clarity,_quality,_novelty_and_reproducibility: I found the paper often confusing and list my clarity concerns below:

* I find the definition of 'multitask' to be narrow and counterintuitive (more details in the 'weaknesses' section above).
* It is hard to understand the title without reading the paper, i.e. what is control graph? "Generalizable input representation for graph-based policy distillation" sounds more applicable from my perspective.
* "policy distillation" is an established term, why do you use "behaviour distillation" instead?
* I do not understand the following sentence "which is significantly more efficient than multi-morphology training done in prior work" (Section 4.2) 
* "incorporates goal information while preserving the geometric structure of a task", what is "geometric structure of a task"?
* "because the task structure may change over time", what is "task structure"?
* "we examine how control graph works well..." is unclear to me.
* Nit: In Section 3, when the Deep RL is introduced, the 'policy' term has not been introduced yet.


summary_of_the_review: I think the paper has interesting findings and has the potential, but issues with the positioning and clarity I pointed out made me choose the score of 5. I am ready to increase the score after the discussion period if my concerns are addressed. 

UPD after rebuttal: The authors were actively engaged in the discussion and updated the paper to address my concerns. I would give the paper 7, but the scoring allows only 6 or 8, and I will update the score to 8.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper