Paper: Learning Embodied Vision-Language Programming From Instruction, Exploration, and Environmental Feedback (ICLR-2024)

Reviewer comment (Experiments):
The data collection process relies on GPT-4, which takes as input language of systems message + environment message to output the required plan, code, and target state. This relies on the strong assumption that the systems + environment message fully captures the environment state, since the planning must be done without access to the view of the visual scene. I presume this means that the systems prompt must be elaborate, handcrafted, and task specific, such that GPT-4 can plan reasonably in the environment using possible objects at hand. Is there a robust way of designing such prompts for different/new tasks without tuning?

Author response:
We thank the reviewer for your valuable comments. You are right, we make sure the environment message fully captures the environment state since when we design a task, we manually inspect all the necessary objects that should be in the environment message. An automatic alternative is to put everything, no matter whether necessary or not, into the environmental message. However, we tried but we found that GPT-4 will not act properly if the environmental message is too long. However, since OpenAI keeps enlarging the input length support, maybe this issue could be addressed in the future. Besides, since this work is to have an initial exploration of the vision-language programming, which remains uncharted to date, we think some handcrafted work is required, and support our future work of making the training pipeline more flexible, such as iterative training, i.e., with an Octopus model that is trained with limited half-handcrafted environment message, the model itself can explore in the environment with visual input, and collect data for next training loop. However, we will keep this exploration in our future exploration.

Reviewer comment (Evaluation):
The results on Table 2 are not particularly convincing of this method’s success. Octopus should indeed outperform blind LLMs that do not take visual input. TAPA outperforms/is of equal performance in 2 of the 5 tasks. The paper lacks analysis on why this is the case. Where does TAPA fail? It is also hard to compare when the vision models are different; does OVD and CLIP-ViT perform similarly in terms of capturing information from the input scene?

Author response:
Actually, the input of the blind LLMs in Table 2 is the textual representation of the objects (and the relations) that are shown in the vision image, i.e, we put everything that exists in the robot's vision into the text to feed into LLMs, so although it is blind, it can "listen" what the environment has. So the higher task completion rate of Octopus does show the superiority of vision input and vision components in Octopus. In Table 2, Octopus reaches a good result on almost all the metrics except the seen environment, showing that Octopus, with the vision as the input can have a better generalization ability in planning. The pure language models, like TAPA, when given the familiar textual input that reflects the environment, seem to have a better result. But we believe task and environmental generalization is a more important aspect.

Reviewer comment (Writing):
Nit: Should add a figure describing Octopus model architecture; notations in methods section are not well defined.

Author response:
Thank you for pointing this! We draw the training pipeline in Figure 4, as the architecture is simply following the Flamingo framework (since Otter follows Flamingo), which can be accessed [here](https://github.com/lucidrains/flamingo-pytorch).

Reviewer comment (Reproducibility):
Is there a systematic way of generating the systems message able to generalize to new datasets? Or does it need to be hand-crafted and tuned for Omniverse & GTA & other datasets?

Author response:
For the system message, according to Appendix D, we think the stuff about the action list is definitely required to be revised for other environments. Other parts might only need minor revision.

Reviewer comment (Novelty):
The novelty of this paper is limited. Essentially, it uses GPT4 with the prompt engineer to collect data from two embodied environments and then trains a vision-language agent model. Moreover, the agent model does not have a specific framework diagram to show the detailed parts, making it difficult to see which part of the model is its innovation point. I suggest providing a framework diagram to clearly explain where the model is newly proposed in the paper and how it differs from existing methods.

Author response:
We sincerely thank the reviewer's valuable feedback and we add the diagram in Figure 4. Our core motivation is to develop a domain that remains uncharted to date, i.e., an embodied vision-language model to make reasonable plans and write executable programs. The novel technical part is that (1) it is the first attempt to have a VLM for code writing and (2) it designs a novel solution for planning improvement in the RLEF part, which collects the success/failure information when GPT-4 is collecting the vision-code pairs in the simulator.

Reviewer comment (Writing):
This paper appears to employ an intentional use of uncommon or less frequently used words in many sentences, substituting them for simpler, more common terms that could convey the message more clearly. As a result, the reading experience becomes somewhat disjointed, and the text may come across as rather weird to the reader.

Author response:
Thank you for the suggestion and we have revised the whole manuscript thoroughly.

Reviewer comment (Experiments):
The reason for using both OmniGibson and GTA-V environments to generate data seems not obvious. A more obvious comparison between the two environments is required, such as the visual comparison of the tasks.

Author response:
The reason of us building OctoVerse is to support the development of our embodied vision-language programmer. OctoVerse is generally based on the existing, most realistic simulators of GTA and OmniGibson, where we add the necessary adaptation to include **the instruction set of callable functions** and **carefully designed tasks**. We include the comparison between OctoVerse with other platforms in appendix C1. The general reason for selecting these two platforms is the consideration of their realistic modeling task designability and scalability. Admittedly, other simulators such as AI2THOR and VirtualHome might also be able to support the Octopus exploration if we also design an instruction set of callable functions and carefully designed tasks, we eventually selected OmniGibson and GTA for realistic modeling and task designability and scalability. The detailed task can be accessed [here](https://filetransfer.io/data-package/WDACj9TO#link) and in the appendix.

Reviewer comment (Reproducibility):
In Section Error Management, when the agent executes the wrong command, how does the method perform "error management" on it? It seems that this section only claims the cases under which a task is defined as failure, and does not show how to manage such failure.

Author response:
Yes, we only annotate commands with "success" or "failure" tags, and these tags are the supervision of the reward modeling in the RLEF part. According to Figure 4 in the paper, all the vision-code pairs in the (a) data collection pipeline will be used for the SFT training in Figure 4(b) Octopus training pipeline, since the GPT-4 does a good job on formatting, meaning that they can write reasonable code. And the "success" or "failure" tag will be used in the RLEF training.

Reviewer comment (Theory):
In Section ENVIRONMENTAL FEEDBACK, what if there are multiple erroneous states in a task sequence? Which are the positive states and which are the error states at this time? And it may not be said that if one of the states is wrong, then its previous states must all be negative.

Author response:
According to Figure 3, one error state is tagged if, after one step, post-execution states misalign their target states. Alternatively, if a task is not completed as intended, every state within that task is labeled as negative. Sorry for the previous confusion and we have clarified this part in Section 3.3.

Reviewer comment (Experiments):
The experimental baselines are unfair and unclear. For Blind LLMs, without visual input to GPT4, GPT4 cannot ground language into the visual environment, which will inevitably lead to worse results. As for TAPA, the reader cannot understand what kind of model it is and its workflow. Even the OVD are not introduced or cited.

Author response:
Sorry for the confusion it made. For Blind LLMs, we provide them with all the environment information in the textual format. Referring to Figure 3, we hope the Blind LLMs could perform as GPT-4 but internalize the system message. For TAPA, we make the implementation details clear in Appendix A, as the TAPA model utilizes the open-vocabulary detection (OVD) technique [Detecting twenty-thousand classes using image-level supervision, ECCV'22] to recognize objects within images. Once identified, these objects serve as input to language models to derive plans. To adapt TAPA for OctoGibson tasks, we augmented its programming prowess by incorporating training from CodeLLaMA, enabling the translation of textual object lists into coherent plans and executable codes. Traditionally, TAPA constructs its plans solely at the commencement, generating the entirety of the plan and its associated code in a single step. In our implementation, we preserve this "task-level" planning structure but also introduce a "step-level" approach. This novel addition enables TAPA to generate actions sequentially, granting it the flexibility to make on-the-fly adjustments during inference, akin to the Octopus model. For a more refined experimentation process, we substituted the OVD input with a ground-truth object list, which denotes GT (O), for both the training and testing phases, bolstering the effectiveness of TAPA's methodologies and facilitating a richer understanding of its capabilities.

Reviewer comment (Experiments):
What are the tasks of the four testing environments? The authors did not give a detailed introduction.

Author response:
We are deeply sorry for the unclear explanation in the submission, and revised it in the first part of Section 5. 
We first set up the OctoGibson to evaluate the performance of Octopus and other related models. Specifically, we are utilizing the metrics of goal task completion score to check whether the task is actually completed in the simulator and the plan score from human evaluation. We totally have 60 evaluation tasks, with 45 from the seen environment, and 15 that are unseen during training. We also have 45 routine tasks and 15 require reasoning. The detailed task can be accessed [here](https://filetransfer.io/data-package/WDACj9TO#link) and in the appendix.

Reviewer comment (Presentation):
Some titles and analyses of the experimental sections appear to be uninformative. For example, **LLMs Does Not Depend on Observation**, is this a conclusion or a statement? If it is a statement, the authors have already said before that this baseline has no visual input, and it is also contrary to the subsequent titles which are all conclusions. In addition, in the ablation experiments, larger models and more components trained can bring more performance improvements, which is common sense in sense, but the authors put them as ablation experiments alone and do not give any insights.

Author response:
Thank you for pointing out this we extensively revised the ablation study part, with the conclusion summarized below:
- **7B vs. 3B Model Size**: Smaller models like 3B show significantly reduced performance, but the improvement of RLEF is still valid on the smaller model.
- **Examining Training Components**: Adjusting both the connector and language decoder in training vision-language models slightly outperforms only adjusting the connector, as shown in Figure 5 (b).
- **Significance of Visual Inputs in Task Performance**: Disrupting the sequence of visual inputs in the Octopus model leads to a marked decline in task performance, emphasizing the importance of structured visual inputs, as depicted in Figure 5 (c).
- **Performance of GPT-4**: GPT-4, with purely textual input, successfully completes 31 out of 60 tasks, indicating significant potential for further advancements in the field.
- **Performance of GPT-4V**: GPT-4V shows promising results in generating code and task execution in image-based environments, though it falls short in specific tasks due to unfamiliarity with the environment and simulator discrepancies, as seen in Figure 5 (e).

Reviewer comment (Reproducibility):
How to train the reward model $r_{\phi}$? More details are needed.

Author response:
We have revised this part in Section 4.3 to make it clear. For reward model configuration, a single-modal CodeLLaMA-7B model is fine-tuned on a specific dataset to serve as a text-only reward model, which evaluates state transitions and helps the agent in determining the most rewarding actions for task execution and completion. It is worth mentioning that we use CodeLLaMA-7B as the reward model since when training the reward model, the input/output pair is only the code and the reward, which is in a pure language scenario. However, the Octopus is a vision-language model and the alignment of the vision and the language part seems to require extra effort to train if we use CodeLLaMA-7B, e.g. more vision-code pair for alignment. Alternatively, we use MPT-7B which is inherited from Otter, where VL alignment is greatly tuned via general vision-language instruct-response pairs. This discussion is also reported in Section 5.1 first paragraph.
For Policy Model Development, the initially supervised fine-tuned model is used as the initial policy model, with another duplicate model trained using Proximal Policy Optimization (PPO) to maximize response rewards. The loss function integrates expected rewards and a Kullback–Leibler (KL) penalty to balance response effectiveness and deviation from the initial policy model.

Reviewer comment (Presentation):
A visual example of task trees is required for better understanding.

Author response:
Thank you for the suggestion! The task list can be accessed [here](https://filetransfer.io/data-package/WDACj9TO#link) and in the supplementary material.

Reviewer comment (Novelty):
More related works in Section 2.1 are needed to help the reviewer identify the paper's contributions, like [1-4].

Author response:
We sincerely thank the reviewer for pointing out these important works, by comparing them we can better show the accurate standing of the Octopus work in the community. We add all these four works in the Table 1, and we show that the proposed Octopus distinguishes itself from other models as an end-to-end, unified vision-language model for both plan and code generation.

Reviewer comment (Experiments):
The authors term the simulation they used "OctoGibson" which is built upon OmniGibson. Can the authors give more details to elaborate the main difference between them, or did they just use that Simulator to collect the dataset?

Author response:
The OctoVerse is generally based on the existing realistic simulators of GTA and OmniGibson, where we add the necessary adaptation to include **the instruction set of callable functions** and **carefully designed tasks** that make the development of the embodied vision-language programmer training possible.

Reviewer comment (Presentation):
A better format is needed. Some lines need a reformat in the revised version. One example is "3.2 Instructions From Exploration", lines above and below seem to belong to the same paragraph.

Author response:
Thank you for pointing it out! Yes, we have drastically revised the whole draft several rounds and made sure it is easy to understand with good structure.

Reviewer comment (Experiments):
More experiments are needed. The author only conducts the experiments on the datasets they collected and lacks a direct comparison with more relative frameworks as discussed in Section 2.1.

Author response:
Yes, we have added several experiments in Section 5.1 and Table 2, where we add throughout comparison with "Blind LLM" including LLaMA, CodeLLaMA, TAPA, which can only "listen" to what is inside the environment. We also implement the EmbodiedGPT for comparison. These experiments have shown several conclusions including:
- Blind LLMs Struggle with Extended Input Content.
- CodeLLaMA Improves Coding but not Planning.
- Octopus Demonstrates Superior Task Generalization Ability as it integrates the visual input.
-  RLEF Enhances Octopus’s Planning Strategy, which is more significant than the components in EmbodiedGPT.

| Model             | Vision Model | Language Model | Seen Env | Unseen Env | Follow | Reason | All   |
|------------------|--------------|----------------|----------|------------|--------|--------|-------|
| LLaMA            | GT (O+R)     | LLaMA2-7B      | 0.07 / 0.11 | 0.13 / 0.13 | 0.11 / 0.16 | 0.00 / 0.00 | 0.08 / 0.12 |
| CodeLLaMA        | GT (O+R)     | CodeLLaMA-7B   | 0.09 / 0.20 | 0.20 / 0.40 | 0.16 / 0.31 | 0.00 / 0.07 | 0.12 / 0.25 |
| TAPA (task-level) | ~~OVD~~ GT (O) | CodeLLaMA-7B   | 0.09 / 0.36 | 0.13 / 0.33 | 0.11 / 0.36 | 0.06 / 0.33 | 0.10 / 0.35 |
| TAPA (step-level) | ~~OVD~~ GT (O) | CodeLLaMA-7B   | **0.16 / 0.42** | 0.13 / 0.27 | **0.18 / 0.38** | 0.07 / 0.40 | 0.15 / 0.38 |
| EmbodiedGPT       | CLIP-ViT     | MPT-7B         | 0.04 / 0.36 | 0.27 / **0.53** | 0.13 / 0.38 | 0.00 / 0.40 | 0.10 / 0.40 |
| Octopus (SFT Only) | CLIP-ViT   | MPT-7B         | 0.11 / 0.33 | 0.27 / 0.47 | 0.16 / 0.38 | 0.13 / 0.33 | 0.15 / 0.37 |
| Octopus (SFT + RLEF) | CLIP-ViT | MPT-7B         | 0.13 / 0.38 | **0.33 / 0.53** | **0.18 / 0.40** | **0.20 / 0.53** | **0.18 / 0.42** |

These experimental results show the promising direction of the end-to-end vision language models for planning and programming.

Reviewer comment (Experiments):
The author uses the GPT-4 to collect the training data, and one implicit assumption is that the performance of the GPT-4 is optimal or near-optimal. A comparison between the GPT-4 generated data sample and the human-collected sample would help. Or, did the author conduct some data quality control before the use of dataset?

Author response:
Thank you for the great question! Actually, when we use GPT-4 for the automatic data collection pipeline, the "environmental feedback" generally acts like the data quality controller. More specifically, when GPT-4 writes the plan, code, and target state, the code will be run in the simulator and check whether the target state is met and whether the final mission is satisfied. All these results of success/failure will be valuable resources for RLEF training. In fact, we intensively designed a very comprehensive system message for GPT-4 and it is good at following the structure with near-perfect code format (seldom syntax error occurs), and the syntax correctness can be identified by the simulator execution and we will remove these with syntax incorrectness when training SFT.

Reviewer comment (Experiments):
RLEF. It is very interesting to see the usage and effectiveness of RLEF. However, I am curious as to why you chose CodeLLaMA-7B as the reward model while using MPT-7B for the complete VLM?

Author response:
Thank you for pointing it out! Actually, we use CodeLLaMA-7B as the reward model since when training the reward model, the input/output pair is only the code and the reward, which is in a pure language scenario. However, the Octopus is a vision-language model and the alignment of the vision and the language part seems to require extra effort to train if we use CodeLLaMA-7B, e.g. more vision-code pair for alignment. Alternatively, we use MPT-7B which is inherited from Otter, where VL alignment is greatly tuned via general vision-language instruct-response pairs. This discussion is also reported in Section 5.1 first paragraph.

Reviewer comment (Evaluation):
In Table 2, there is a comparison between Octopus and MPT-7B. Also, the performance is not consistently superior, a further discussion is needed. And the metrics' definition is needed to help the understanding.

Author response:
In Table 2, Octopus reaches a good result on almost all the metrics except the seen environment, showing that Octopus, with the vision as the input can have a better generalization ability in planning. The pure language models, when given the familiar textual input that reflects the environment, seem to have a better result. But we believe task and environmental generalization is a more important aspect. For the detailed explanation of metrics, we fix this part at the beginning of section 5: We first set up the OctoGibson to evaluate the performance of Octopus and other related models. Specifically, we are utilizing the metrics of goal task completion score to check whether the task is actually completed in the simulator and the plan score from human evaluation. We totally have 60 evaluation tasks, with 45 from the seen environment, and 15 that are unseen during training. We also have 45 routine tasks and 15 require reasoning. Please note that models like Octopus might not always accurately identify specific object names as they appear in the simulator (e.g., ``water\_bottle\_189''). To address this, we implement a post-processing step for the generated code, substituting generic object references with their exact names from the simulator with simple string similarity matching.

Reviewer comment (Experiments):
Ablation: 3B: what is the 3B model?

Author response:
3B model compares to the 7B model with respect to the language model size. We clarify this part in the Figure 5 caption. We show that the RLEF component is effective on both 3B and 7B models.

Reviewer comment (Reproducibility):
The author states multiple times with "open-sourcing" in the main text, a link to the anonymous website would be helpful.

Author response:
Thank you for pointing it out! We add the open-source code in the appendix.