Paper: MultiReAct: Multimodal Tools Augmented Reasoning-Acting Traces for Embodied Agent Planning (ICLR-2024)

Reviewer comment (Experiments):
I am just curious if the CLIP reward can be leveraged to finetune the low-level policy.

Author response:
We sincerely thank the reviewer for their constructive feedback and insightful question regarding the potential use of CLIP reward in fine-tuning the low-level policy. We appreciate your recognition of our approach to ground LLM-based planning in real environments using video demonstrations and CLIP-based reward. Your understanding of the significance of our method in enhancing the detail and contextual relevance of sub-instructions is encouraging.

```
Q1: I am just curious if the CLIP reward can be leveraged to finetune the low-level policy.
```
Yes, the potential fine-tuning process could be:
**Generating Feedback**: After the low-level policy executes an action, the result (captured as an image or video frame) is fed into CLIP along with the textual description of the intended action or task. CLIP then evaluates how well the result aligns with the description.
**Reward Mechanism**: Based on CLIP's evaluation, a reward signal can be generated. If CLIP finds a high correlation between the action's outcome and the textual task description, a positive reward is given. Conversely, a low correlation results in a lower reward or a penalty.
**Policy Adjustment**: This reward signal is used to fine-tune the low-level policy's neural network.  RL algorithms such as PPO can be employed here, where the policy network's parameters are adjusted to maximize the received rewards over time.
Iterative Improvement: Over multiple iterations, the low-level policy learns to generate actions that increasingly align with the task descriptions, as validated by CLIP. The policy becomes more adept at interpreting instructions in the context of the visual environment it operates in.

We believe this integration would be significantly contribute to the field of embodied AI. Once again, thank you for your valuable feedback and the opportunity to consider this exciting direction for our research.

Reviewer comment (Novelty):
The novelty is limited as it essentially adds multimodal rewards with CLIP, which is more or less an existing idea (see, e.g., [1]), and video captioning for converting instruction and video demos into sub-instructions to the existing ReAct framework.

Author response:
We appreciate the reviewer's observations and critique regarding the perceived novelty of our approach and the breadth of our experimental evaluation. We understand the importance of distinguishing our work from existing literature and demonstrating its practical applicability in a range of scenarios.

> w1: novelty

**Multimodal Rewards with CLIP Integration**:

While integrating multimodal rewards with CLIP might be an existing idea. In our methodology, the Large Language Model (LLM) planner interacts with the CLIP model to dynamically generate both negative and positive conditions. This process is crucial for determining whether a sub-instruction has been successfully executed. Unlike static reward systems, our approach adapts the reward criteria based on the specific context of each task, leading to more nuanced and accurate evaluations.

Reviewer comment (Experiments):
Due to the point above, stronger experimental results in terms of the broadness of the tasks would be expected. The authors evaluate their framework only on the CLIPort tasks, which are relatively simple (e.g., the suction-based gripper abstracts away most of the low-level actions). Tasks involving more low-level controls (e.g., object manipulations) should be evaluated or at least discussed, where whether the CLIP or any existing 2D-based VLM is good enough for providing useful reward signals remains a question. For instance, the CALVIN benchmark. Alternatively, other popular embodied agent planning tasks involving only high-level actions such as ALFRED should be evaluated or at least discussed to give a better sense of the applicability of this framework.

Author response:
We appreciate the reviewer's observations and critique regarding the perceived novelty of our approach and the breadth of our experimental evaluation. We understand the importance of distinguishing our work from existing literature and demonstrating its practical applicability in a range of scenarios.

>w2: suction-based gripper abstracts away most of the low-level actions

Clarifying Research Motivation and Focus:

1. Emphasis on Embodied Planning:
Our primary objective is to enhance embodied planning capabilities within AI systems. This focus is specifically geared towards understanding and improving how AI systems can develop and execute plans based on abstract instructions. Our research is less about physical manipulation and more about the cognitive process of planning in response to given tasks.

2. Scope of Current Experiments:
The reason we chose CLIPort tasks for our initial evaluation is that they provide a controlled environment to test our framework's planning capabilities. While these tasks may appear simple due to the abstraction of low-level actions, they are quite effective in demonstrating the efficacy of our approach in generating and executing plans based on multimodal inputs.

3. Future Directions and Broader Applicability:
We acknowledge the importance of evaluating our framework in more diverse scenarios, including tasks that involve more intricate object manipulations (as in the CALVIN benchmark) or high-level action planning (such as in ALFRED). While these are valuable areas for future research, they were beyond the initial scope of this study. Our immediate goal was to establish a foundational understanding of how multimodal tools can enhance the planning process in AI systems.

Reviewer comment (Experiments):
The experiment setting is missing some key details (see questions).

Author response:
> W1: Experiment Fairness

To ensure the fairness of comparison we make the following setting
- **CLIPort:** During test time, CLIPort does not directly access video demonstrations. However, it is equipped with an oracle feature that accurately provides it with the necessary sub-instructions at each time step. This setup is intended to simulate a scenario where CLIPort can rely on high-quality, contextual guidance to execute tasks, without the direct need for visual demonstrations.
- **CaP and ReAct:** Both CaP (Comprehension and Planning) and ReAct (Cap) systems are provided with general instructions on how to fulfill the tasks. These instructions are comparable to what a human operator might receive.  For example, we tell LLMs ``a pyramid is something wide at the bottom and narrow on the top, for a 3 layer pyramid, it should have 3 blocks on the bottom, 2 blocks in the middle and 1 on the top.``  For a detailed view of these instructions, we refer you to Figure 4 in our paper.
- **MultiReAct**: Unlike CLIPort, which utilizes an oracle feature for sub-instruction guidance, or CaP and ReAct, which access general task instructions, MultiReAct relies exclusively on multimodal reasoning. This approach integrates visual data with abstract robot instructions, enabling the LLMs to infer appropriate actions without the need for explicit oracles or detailed guidelines.

Reviewer comment (Experiments):
I'm not sure if the main experiment comparison is fair. For example, do ReAct (Cap) and Cap also have access to demonstrations? These may be necessary for long-horizon tasks. Also I believe CLIPort doesn't get demonstration at test time. The authors should elaborate on the experiment settings.

Author response:
> W1: Experiment Fairness

To ensure the fairness of comparison we make the following setting
- **CLIPort:** During test time, CLIPort does not directly access video demonstrations. However, it is equipped with an oracle feature that accurately provides it with the necessary sub-instructions at each time step. This setup is intended to simulate a scenario where CLIPort can rely on high-quality, contextual guidance to execute tasks, without the direct need for visual demonstrations.
- **CaP and ReAct:** Both CaP (Comprehension and Planning) and ReAct (Cap) systems are provided with general instructions on how to fulfill the tasks. These instructions are comparable to what a human operator might receive.  For example, we tell LLMs ``a pyramid is something wide at the bottom and narrow on the top, for a 3 layer pyramid, it should have 3 blocks on the bottom, 2 blocks in the middle and 1 on the top.``  For a detailed view of these instructions, we refer you to Figure 4 in our paper.
- **MultiReAct**: Unlike CLIPort, which utilizes an oracle feature for sub-instruction guidance, or CaP and ReAct, which access general task instructions, MultiReAct relies exclusively on multimodal reasoning. This approach integrates visual data with abstract robot instructions, enabling the LLMs to infer appropriate actions without the need for explicit oracles or detailed guidelines.

Reviewer comment (Novelty):
The LLM is instructed to call the VLM and then iteratively call the actor and reward model. This is probably achievable with a simple program. The use of LLM here might be an overkill.

Author response:
> W2: Is LLMs overkill?

The LLMs is suitable for the planning tasks. For the following reasons:
1. **Complex Decision-Making and Integration:**
    The LLM's role extends far beyond that of a mere sequential call dispatcher. It is integral in orchestrating complex decisions that involve the integration of multimodal inputs and the interpretation of nuanced instructions within varying contexts. This sophisticated level of decision-making and integration, essential for our method, is beyond the capabilities of a simple program.
2. **Advanced Natural Language Processing:**
    LLMs excel in understanding and generating human-like text, a key feature that enhances the intuitiveness and effectiveness of interactions, particularly with abstract or complex instructions. Moreover, the plans generated by LLMs are marked by superior interpretability, facilitating clearer communication and execution of tasks.
3. **Scalability and Versatility:**
    The LLM-based planning pipeline offers remarkable scalability, adaptable to a wide array of robotics tasks. Unlike fixed programs, LLMs can adjust to new tasks with minimal modifications, demonstrating their versatility and applicability across diverse scenarios.
4. **Enhancing Reward Model Judgments:**
     LLMs contribute significantly to setting the judgment criteria for reward models. They generate context-specific positive and negative conditions, guiding the reward models to accurately assess whether a sub-instruction has been successfully fulfilled.

Reviewer comment (Experiments):
How many demonstrations are needed to fine-tune the VLM? How many are used to train CLIPort?

Author response:
> Q1: How many demonstrations are needed to fine-tune the VLM? How many are used to train CLIPort?
	
Both our VLM in the MultiReAct system and CLIPort are trained on a dataset comprising 100$T$ demonstrations. $T$ denotes the number of tasks.

Reviewer comment (Experiments):
Is the VLM fine-tuned for each individual task or on all tasks? Does it generalize to unseen tasks?

Author response:
> Q2: Is the VLM fine-tuned for each individual task or on all tasks? Does it generalize to unseen tasks?

Our VLM is fine-tuned across a broad spectrum of tasks, rather than on individual tasks.
**Incorporating Randomization:** We introduce randomization in the layout and arrangement within each task type. This strategy is crucial for the VLM to handle variations and complexities within the same task domain, ensuring robustness against overfitting to specific layouts or configurations.
- **In-Domain Generalization:** While our VLM does not generalize to entirely unseen tasks, it exhibits strong generalization capabilities within the domain of trained tasks. This means that the VLM can effectively deal with variations of the same tasks, such as different layouts and arrangements of objects. The model’s ability to adapt to these variations within its training domain is a significant strength. It can understand and process tasks that are similar in nature to the trained ones but differ in specific details, thereby demonstrating a level of flexibility and adaptability within its operational domain.

Reviewer comment (Experiments):
How are the Stack Block Pyramid and Tower of Hanoi tasks randomized? As far as I can tell, MultiReAct will try to replicate exactly what has been done in the demonstration. If this is the case, would the task fail if the colors and initial ring locations are swapped?

Author response:
> Q3: How are the Stack Block Pyramid and Tower of Hanoi tasks randomized? As far as I can tell, MultiReAct will try to replicate exactly what has been done in the demonstration. If this is the case, would the task fail if the colors and initial ring locations are swapped?
1. **Stack Block Pyramid:**
     In the Stack Block Pyramid task, randomization is introduced in terms of the block colors and starting positions. This approach ensures that while the overall task objective remains consistent, the specific scenario in each iteration is unique. MultiReAct is trained to understand the fundamental concept of building a pyramid, irrespective of these variations.
2. **Tower of Hanoi:**
	For the Tower of Hanoi task, we introduce randomization in the initial placement of the rings and occasionally in the color of the rings. The goal is to ensure that MultiReAct does not merely memorize a specific sequence of moves but understands the underlying logic and strategy required to solve the Tower of Hanoi under varying initial conditions.

Reviewer comment (Evaluation):
Why do you use CLIP reward score as performance measure in section 4.3 but not the ground-truth reward?

Author response:
> Q4: Why do you use CLIP reward score as performance measure in section 4.3 but not the ground-truth reward?

1. **Ground-Truth Reward Challenges:**
	In the context of our experiments, defining a precise ground-truth reward can be challenging. The tasks we are addressing involve complex interactions within dynamic environments, making the establishment of a fixed, objective ground-truth reward for each possible outcome impractical. The subjective nature of some tasks further complicates this issue.
 2. **CLIP Reward as a Proxy for Ground-Truth:**
	The CLIP reward score serves as a practical and effective proxy for ground-truth in our experiments. CLIP, trained on a vast range of images and text, is adept at evaluating the alignment between visual outcomes and textual descriptions. This alignment is crucial in tasks where success is defined by how well an action aligns with the given instructions or objectives.
 3. **Real-World Applicability:**
	 Using the CLIP reward score aligns with real-world scenarios where ground-truth may not be readily available or easily quantifiable. By leveraging CLIP's capabilities, we simulate a more realistic setting where the system must evaluate its performance based on available visual and textual cues, mirroring practical applications.