Paper: Compositional Interfaces for Compositional Generalization (ICLR-2024)

Reviewer comment (Novelty):
This paper seems to be solely an application of COIN architecture in the setting of compositional combinations of observation and action spaces. It is not clear what novelty or technical contribution it has.

Author response:
The primary contribution of the paper is the study of compositional generalization of multi-task multi-modal agents, specifically with respect to observation, action and instruction spaces. To the best of our knowledge, our work is the only such study to date. It is _not merely an application of a modular architecture, but rather a systematic evaluation of how much a modular architecture improves compositional generalization_ for such agents. We provided detailed experiments and analysis with respect to the number of training combinations, data size, and hold-out set difficulty. This form of compositional generalization is an important property of generalist agents: many real-world tasks  will involve training agents with a variety of sensors and actuators to accomplish a wide variety of objectives.

The remaining novel contributions of our work are as follows:
- An environment that enables such study by: (a) allowing easy composition of arbitrary observation, action and instruction spaces, (b) supporting a large number of observation, action and instruction spaces, (c) providing an automated way to generate training data
- An architecture that can handle learning in this setting through modular design, as well as incorporating new perception modalities without impacting performance on the tasks the agent was previously trained on.

Reviewer comment (Experiments):
The experiments are mainly on COIN architecture. No comparison with other approaches. It is hard to assess the effectiveness of this method.

Author response:
With respect to the choice of baselines, the literature has no standard off-the-shelf approaches applicable to our setting beyond our comparisons to single-task models.  The closest possible approaches are tokenized transformers such as those used in GATO (which is not open-sourced). With such model, however, "how to tokenize" would become a non-trivial question beyond the scope of our work, even if GATO was open-sourced.

Hence the most fair comparison is with respect to agents trained in a single-task setting or the expert data (in our case, the experts always complete the task). Note that for the same reasons, GATO also does not compare their model to alternative multi-task multi-modal architectures.

Reviewer comment (Experiments):
Why is there no comparison with non-COIN approaches?

Author response:
With respect to the choice of baselines, the literature has no standard off-the-shelf approaches applicable to our setting beyond our comparisons to single-task models.  The closest possible approaches are tokenized transformers such as those used in GATO (which is not open-sourced). With such model, however, "how to tokenize" would become a non-trivial question beyond the scope of our work, even if GATO was open-sourced.

Hence the most fair comparison is with respect to agents trained in a single-task setting or the expert data (in our case, the experts always complete the task). Note that for the same reasons, GATO also does not compare their model to alternative multi-task multi-modal architectures.

Reviewer comment (Novelty):
What is the novelty of this paper? It seems to be simply an application of COIN architecture on compositional combinations of observation and action spaces.

Author response:
The primary contribution of the paper is the study of compositional generalization of multi-task multi-modal agents, specifically with respect to observation, action and instruction spaces. To the best of our knowledge, our work is the only such study to date. It is _not merely an application of a modular architecture, but rather a systematic evaluation of how much a modular architecture improves compositional generalization_ for such agents. We provided detailed experiments and analysis with respect to the number of training combinations, data size, and hold-out set difficulty. This form of compositional generalization is an important property of generalist agents: many real-world tasks  will involve training agents with a variety of sensors and actuators to accomplish a wide variety of objectives.

The remaining novel contributions of our work are as follows:
- An environment that enables such study by: (a) allowing easy composition of arbitrary observation, action and instruction spaces, (b) supporting a large number of observation, action and instruction spaces, (c) providing an automated way to generate training data
- An architecture that can handle learning in this setting through modular design, as well as incorporating new perception modalities without impacting performance on the tasks the agent was previously trained on.

Reviewer comment (Experiments):
It would be great if noisy real-world data or more complex visual data is included in the testing scenario. It is unclear for now whether this design is robust enough to deal with the not-that-clean data regime.

Author response:
While we agree that results on noisier environments would be interesting, there were no such environments available that would allow us to study compositional generalization wrt observation, action and instruction spaces specifically. Moreover, in order to draw strong conclusions, it is often more useful to study the phenomena in a controlled, synthetic environment. By focusing on a simpler environment design, we were able to: 
(a) generate a very large number of environment combinations and individual spaces, 
(b) have a much better control of the factors of variation – in our case, we can only vary the observation modality, action space and instruction space – without introducing other confounding sources of variation that would be present in more complex environments.

Reviewer comment (Evaluation):
Could the authors provide insights or discussions about why the proposed method performs worse in the instruction space of `Sort by Property`? Is there a class of tasks that the proposed method fail to handle?

Author response:
While we can not tell with certainty, we suspect the performance on "Sort by Property" instructions are lower because of how different optimal policies are for this task compared to others. This is the only instruction in which the goal state is defined by distance between particular items – a feature that may be harder to learn when there are multiple other objectives competing for representational space.

Reviewer comment (Experiments):
For the `HARD HOLDOUTS` set-up, will the performance further improve if there are more easy combinations available for training? Are there any other possible solutions to address the easy-to-hard transfer problem?

Author response:
In Figure 6, we already vary the number of easier combinations used in training and find that by increasing those leads to improvement in performance. Note that we report only zero-shot performance on hard combinations, and expect that fine-tuning on some data from hard holdouts would further improve the performance.

Reviewer comment (Experiments):
There can be more possible compositional generalization problems, such as novel combinations of shape and color in observation.

Author response:
In this work, we chose to specifically focus on compositional generalization with respect to combinations of observations, actions and instructions, as those are particularly relevant for generalist (multi-task, multi-modal) agents. As the reviewer suggests, there is lots of prior work on other kinds of compositional generalization -- that is good too!  We of course neither attempt nor claim to cover every possible instance of "compositionality"; similarly most other works on compositionality do not attempt all instances.  Even _discussing_ (much less algorithmically treating) all possible generalization problems is beyond the scope of a monograph, much less a single ICLR submission.

Reviewer comment (Experiments):
The IDs of observation and action spaces are provided for each sample to select modules. They can be hidden in more general cases.

Author response:
We do only consider the case where the instruction and observations are provided separately, and the observation and action IDs are known. However, we do not think this is a significant limitation of our work in terms of its impact and contribution. Instruction and observation are typically provided together (in fact, this is a default assumption in the [instruction following](https://arxiv.org/abs/1906.03926) literature and goal-conditioned sequential decision making). Similarly, the observation and action IDs are often known – the action space typically needs to be known in order to enact the action in the environment, while the observation space can be determined from the format of the input.

Reviewer comment (Experiments):
The agent is trained with imitation learning, while another widely used algorithm is reinforcement learning.

Author response:
While adding comparison with other kinds of objectives would further strengthen our results, we argue that using only imitation learning is not a significant limitation of our work. Since the question of transfer and generalization is mostly related to the architecture and training regime, the results are also informative for the agents trained with reinforcement learning. The most related published works, such as [multi-game decision transformers](https://arxiv.org/abs/2205.15241) and [GATO](https://arxiv.org/abs/2205.06175), also train their agents with expert data only (either via offline RL or imitation learning).

Reviewer comment (Experiments):
In the experiment, the environment is small and synthetic. It also lacks strong baselines of existing embodied agent algorithms.

Author response:
While the larger scale and more realistic environments are often desirable, we argue that for the sake of drawing out clear and consistent empirical patterns, controlled environments such as the one developed in this paper are more useful. Such environment enabled us to: (a) generate a very large number of environment combinations and individual spaces, with which we can extract more signal from the noise, (b) have a much better control of the factors of variation – in our case, we can only vary the observation modality, action space and instruction space – without introducing other confounding sources of variation that would be present in more complex environments  

With respect to the choice of baselines, the literature has no standard off-the-shelf approaches applicable to our setting beyond our comparisons to single-task models.  The closest possible approaches are tokenized transformers such as used in GATO (which is not open-sourced).  In particular, "how to tokenize" would become a non-trivial question beyond the scope of this work, even if GATO was open-sourced. 

Hence the most fair comparison is with respect to agents trained in a single-task setting or to the performance in the expert data (in our case, the experts always complete the task). Note that for the same reasons, GATO also does not compare their model to alternative multi-task multi-modal architectures.