Paper: Compositional Instruction Following with Language Models and Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
The approach is incremental. Since components like WVFs were known before (Nangue Tasse et al., 2020), the main contribution is the usage of LLM. However, it is straightforward to apply LLM to parse the instruction especially since previous papers suggested LLM as a good parser (Shin et al., 2021).

Author response:
We utilize a learned semantic parser that uses in-context learning and feedback from the environment. To the best of our knowledge, both our semantic parsing method and the scale and diversity of simultaneously learned language-RL tasks is unique in the literature. It is not a priori certain that semantic parsing should work in the RL setting on this quantity of tasks. Our method learns the parser from environment feedback, a much weaker form of supervision than is typically assumed. Further, we use our method to solve language grounding tasks involving negation, which are under-explored (Pillai and Matuszek 2018). 

We evaluate our agent on a much larger set of simultaneous tasks than previous works using WVF, which instead evaluated on individual tasks with hand-specified compositions of value functions. These works assumed ground-truth Boolean expressions, which is a very strong assumption. Our method relaxes this assumption and instead learns these Boolean expressions from environment feedback which is non-trivial.

Our work does build on pre-existing methods. But many related works contain applications of existing methods, including SayCan (Ahn et al. 2022) which combines skill learning from demonstration with few-shot LLM planning, PaLM-E (Driess et al. 2023) which combines pre-existing vision and language representations to solve vision-language tasks. But we also go beyond the existing literature and demonstrate the utility of a novel semantic parsing method for RL settings and the advantage of compositionality at both the RL and language levels for solving a large set of language-RL tasks.

Reviewer comment (Novelty):
In terms of practicality, for BabyAI, it is easy to program a perfect parser, and thus, it is not necessary to use LLM in this task.

Author response:
The original BabyAI domain is intentionally structured such that a perfect parser for language instructions can be defined, and this makes it the right domain for developing and evaluating our method. Because we can define a parser, we are able to benchmark our method against an Oracle and validate our method by showing that our method converges to the Oracle’s performance. Many recent works published at top conferences have utilized BabyAI for evaluation for similar reasons including Carta et al. (2022) EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL and Li et al. (2022), Pre-Trained Language Models for Interactive Decision-Making. In future work we plan on generalizing our method to other domains.

Reviewer comment (Experiments):
The baseline is too primitive. It is not surprising that the proposed method outperforms the baseline because it has access to additional information (e.g. compositions of the instruction). Please consider other baselines on Baby AI such as those in Carta et al., (2023).

Author response:
We appreciate your feedback on our baseline. We are updating the baseline to utilize pretrained language representations: the SOTA all-mpnet-base-v2 sentence embedding model available in the SentenceTransformers library (Reimers and Gurevych 2019). Initial results show that this agent still requires more steps than our agent and does not generalize well to held-out tasks. We plan to share updated baseline results by the end of the discussion period. The similar performance of the new baseline is expected, as neural networks struggle with compositional generalization and to generalize systematically in this domain requires both compositional RL representations and language representations. While the new baseline agent may generalize better between language commands, it must still learn each task value function without the aid of compositional structure. To solve this many compositional tasks in a generalizable way requires compositional generalization at both the language and policy level, which our method enables.

Carta et al. (2023) does not appear to be an appropriate baseline as they only evaluate in a custom text-world environment with a significantly simpler observation space. Our method utilizes value function approximation from images and text, providing greater applicability and we evaluate in a more challenging observation space.

Reviewer comment (Evaluation):
The pre-training time can be an issue. In Figure 3, after the pertaining time of the proposed method, the baseline already converges to a reasonable performance. Although the baseline cannot improve further, its reasonable performance may outweigh the cost of additional training of the proposed method.

Author response:
We demonstrate that our method attains a higher success rate in fewer timesteps than the non-compositional baseline and also offers a simple means of interpretability which black-box methods like joint text-image value function learning do not. We utilize this interpretability to analyze the failure mode of the ablation using the less capable GPT-3.5 Agent. In terms of performance, our method significantly reduces (by more than 50%) the error rate over the baseline for an equal number of environment steps at convergence. Our method also generalizes better to held-out tasks as shown in Figure 4.

Reviewer comment (Writing):
The writing is hard to follow. More background on WVF is needed to understand the approach. The abstract also lacks information to capture the main idea of the paper (use LLM to parse instructions for WVFs). The method combines different components from different papers, so it is better to present an algorithm to describe the whole workflow in detail (especially the part that uses parsed symbols to select the WVF basis and the training of WVFs in the downstream task).

Author response:
We appreciate your feedback and will improve the explanation of the WVF learning for a camera-ready submission and include additional details in the appendix.

Reviewer comment (Evaluation):
How many runs are used to generate the plot in Figure 3? The variance of GPT 3.5 is high.

Author response:
We utilized 10 trials in all experiments in Figure 3. GPT 3.5 has lower performance and more variance than the other methods because it learns a weaker semantic parser.

Reviewer comment (Presentation):
Table 2, the first Assistant should not have ~?

Author response:
Thank you for pointing out this typo, it will be corrected.

Reviewer comment (Writing):
Can you summarize 18 goals in a table? This sentence is hard to understand "Each WVF is implemented using |G| = 18 CNN-DQN (Mnih et al., 2015) architectures".

Author response:
The goal space is the cross product of the colors and object types summarized in Appendix Table 4. We will add a new table in the appendix to make this explicit.

Reviewer comment (Experiments):
How did you generate 162 tasks? Are any criteria used?

Author response:
The 162 tasks are composed from the object types, colors, and task types in Table 1. We generate tasks from Boolean expressions over the task attributes, including negation of attributes. The background section (2. BabyAI Domain) provides an overview of our task space. We are happy to update this section for clarity based on your feedback.

Reviewer comment (Reproducibility):
As the authors have mentioned, the proposed method requires pretraining a set of WVFs as a basis for further language instruction parsing. Currently, what the set of WVFs is composed of requires an in-depth understanding of the domain. Automating the design of WVFs or allowing to dynamically modify WVFs will strengthen this work.

Author response:
We appreciate your comment and plan on investigating learning of WVFs without a curriculum in future work.

Reviewer comment (Experiments):
In this work, only the boolean expressions that lead to sufficiently high success rates are added to in-context samples. Does it mean that the correct boolean expression should be easy to explore for the LLMs? Is it possible to obtain more supervision from failed environmental rollouts?

Author response:
Our approach builds on similar in-context semantic parsing work like Toolformer, but goes beyond these methods as it is the first to use in-context learning from environment feedback. We have not been able to devise an effective method where the agent utilizes negative examples for in-context learning. We can however utilize negative samples in the fine-tuning setting through unlikelihood training (Welleck et al. 2019).

Reviewer comment (Experiments):
_Need for user-defined primitive tasks:_ The compositional approach presented in this paper requires manual specification of primitive tasks and training a Q-function for each primitive task. This set of primitive tasks might not be readily available; which makes this approach not easily applicable to realistic scenarios. Furthermore, the baseline approach does not have access to any such additional information.

Author response:
Need for user-defined primitive tasks:
Current robotics deployments require pre-defined primitive tasks and for human design of the low-level controllers. For example, SOTA language instruction following methods like SayCan (Ahn et al. 2022) rely on extensive human-design of both high level tasks and low-level control through manual demonstrations. This work is among the first to relax these requirements (the low-level controllers are learned) while providing the ability to compose these controllers, and is the largest set of such tasks learned simultaneously with language and RL.

Reviewer comment (Experiments):
_Unfair comparison to baseline:_ As mentioned above, the baseline does not have access to additional information that is being used to achieve the decomposition necessary for the approach presented in this paper. Furthermore, the baseline approach does not have access to any pre-trained LLM and has to learn the logical structure in the natural language instructions from scratch by interacting with the environment. It is worth considering a baseline that uses the output of a hidden layer of a pre-trained LLM as the encoding of the natural language specification.

Author response:
Unfair comparison to baseline:
We are updating the baseline to utilize pretrained language embeddings from the SOTA all-mpnet-base-v2 sentence embedding model from SentenceTransformers library (Reimers and Gurevych 2019) . Initial results show that this agent still requires more steps than our agent and does not generalize well to held-out tasks. We plan to share updated baseline results by the end of the discussion period. The similar performance of the new baseline is expected, as neural networks struggle with compositional generalization and to generalize systematically in this domain requires both compositional RL representations and language representations. While the new baseline agent may generalize better between language commands, it must still learn each task value function without the aid of compositional structure. To solve this many compositional tasks in a generalizable way requires compositional generalization at both the language and policy level, which our method enables.

Reviewer comment (Theory):
_Only Boolean combinations supported:_ The instructions are assumed to always correspond to a Boolean combination of primitive tasks. There is no support for temporal compositions such as doing one task after another. Furthermore, this reduces the problem to finding the mapping between keywords in the textual task description and the set of symbols used to denote the primitive tasks. In fact, I believe LLMs such as GPT3.5 can already construct Boolean formulas from such language instructions if this mapping is given. Although this mapping is being learned in the proposed approach, I am not sure if there are use cases where the set of primitive tasks is available but the mapping to the symbols is not.

Author response:
Only Boolean combinations supported:
Prior work in semantic parsing with LLM: to our knowledge, our method is the first to learn semantic parsing using in-context learning with feedback from environment feedback. This is a much weaker form of supervision than is typically assumed in semantic parsing and represents an important contribution to existing language instruction following literature that maps natural language to formal representations.

Even for cases where the mapping from task primitives to symbols is known, one challenge is that language is ambiguous and there is a many-to-one mapping of language instructions to individual tasks, and language symbols to task primitives. Indeed knowing only the semantics of the symbols is insufficient in our domain for learning to form valid Boolean expressions as our tasks involve combinations of primitive task negation, intersection, and conjunction. Lastly, Boolean combinations are able to express many interesting tasks. Because we support Boolean task specifications, our method also naturally generalizes to task negation. Negation is understudied in the grounded language literature (Pillai and Matuszek 2018) and to the best of our knowledge our paper also solves the largest and most diverse set of language-negation RL tasks. We leave temporal compositions of tasks to future work.

Reviewer comment (Evaluation):
Given the results in Figure 3, I do not think one can make the statement that the method is more sample efficient than the baseline. The baseline stops improving after the number of environment steps has reached the number of pretraining steps. To me, one method is more sample efficient than the other if it reaches the same performance with fewer samples. The conclusion that can be drawn from the plot is that the method outperforms the baseline in terms of final performance. It is not immediately clear why the baseline is limited in its final performance; would a larger network or more carefully tuned hyperparameters lead to better final performance?

Author response:
Sample Efficiency:
Your comment on the results of Figure 3 are well taken. We are claiming a sample efficiency gain and benchmark our model against the best performing non-compositional baseline DQN agent we could train. This baseline agent does continue to increase in performance, though at a decreasing rate of improvement. The x-axis scale change could also make this hard to see. Given that we are working in the function-approximation setting and learning value functions from text and images, the agent is not guaranteed to converge to the optimal action-value function. If we were evaluating in a tabular setting where convergence is guaranteed, it would be appropriate to run our baseline until convergence. Our setting is more challenging than previous similar compositional domains. To the best of our knowledge it contains the most simultaneously learned language-RL tasks. Since we have no guarantee of convergence, we instead run the baseline agent for many more steps than it takes for our method to converge to the Oracle agent’s performance. We thereby show that our method requires fewer steps than the baseline to reach the Oracle agent’s performance.

Reviewer comment (Experiments):
Why were there no pretrained language embeddings used for the baseline? That should have a large impact on at least the sample efficiency of the baseline.

Author response:
Baseline:
We are updating the baseline to utilize pretrained language embeddings from the SOTA all-mpnet-base-v2 sentence embedding model from SentenceTransformers library (Reimers and Gurevych 2019) . Initial results show that this agent still requires more steps than our agent and does not generalize well to held-out tasks. We plan to share updated baseline results by the end of the discussion period. The similar performance of the new baseline is expected, as neural networks struggle with compositional generalization and to generalize systematically in this domain requires both compositional RL representations and language representations. While the new baseline agent may generalize better between language commands, it must still learn each task value function without the aid of compositional structure. To solve this many compositional tasks in a generalizable way requires compositional generalization at both the language and policy level, which our method enables.

Reviewer comment (Evaluation):
I would present Figure 4 in the same way Figure 3 is presented. The authors state that because they look at generalization performance and not sample efficiency, a clear separation of the pretraining samples and the training samples is not necessary. However, the plot looks at generalization performance after a certain number of environment steps, and for a given point on the x-axis the current plot compares the generalization performance of two methods trained with different numbers of samples. If I understood it correctly, the number of steps for which the baseline was trained in that scenario is 10 million, so it is close to only half of all the pretraining steps. Since the generalization performance of the baseline is so low, I do not think it makes much difference in the conclusions that can be drawn, but it would result in a more accurate comparison.

Author response:
Formalities:
We will run the baseline for more steps for Figure 4 as suggested, to provide a more fair comparison of generalization performance at each timestep. We plan to update these results by the end of the discussion period.

Reviewer comment (Experiments):
The success rate that boolean expressions need to satisfy to be added to the database of boolean expressions that can be added to the prompt seems to be set by expected performance of the oracle parser. Did you perform any ablations on that hyperparameter? A lower threshold might lead to worse formulas being added but also more formulas early on and vice versa. It would be interesting to see how sample efficiency changes with that parameter.

Author response:
Questions:
1. Thank you for your suggestion. We arrived at that hyperparameter empirically but have not completed a full multi-trial evaluation of ablated values. We can include these in a camera-ready submission.

Reviewer comment (Experiments):
In the boolean expressions, abstract symbols were used for concepts such as "grey object" instead of the corresponding natural language labels. Would that not make parsing for the LLM easier?

Author response:
Questions:
2. We believe this helps demonstrate the applicability of our method to situations where the semantics of the value functions are either not easily expressed in language or are otherwise challenging to enumerate.

Reviewer comment (Experiments):
In paragraph 3.2.1 it says that during training a beam of 10 parses was looked at. During the evaluation of the agent, did the agent choose the most likely parsing and evaluate it for 100 episodes, or was the average evaluation of all 10 parses looked at?

Author response:
Questions:
3. During training each parse in the beam is evaluated for 100 episodes, until a high success rate parse is found, or all parses in the beam have been exhausted. The section will be updated in a camera-ready submission to make this clearer.