Paper: Building Cooperative Embodied Agents Modularly with Large Language Models (ICLR-2024)

Reviewer comment (Reproducibility):
Details regarding how CoELA cooperates with other agents are lacking. Sections 4 and 5 do not mention the mechanisms for cooperation between agents, such as how one MHP cooperates with another MHP or with CoELA.

Author response:
Both agents take the same environment observation as input, and output environment-acceptable primitive actions. Despite the inner working mechanism being different, CoELA can still model the MHP agent's state and plan cooperatively. We follow the design of MHP in [1], where MHP infers the other's intention and adapts its subgoal accordingly based on the observation of the other agent. We've added these details to the revised paper.

Reviewer comment (Evaluation):
The authors appear to be focused on exploring the potential of using Large Language Models (LLMs) in the Discrete Execution setting through communication. However, it's important to consider that communication may lead to challenges in certain scenarios, such as agents failing to reach a consensus. A discussion of such scenarios seems to be missing.

Author response:
Great question! Communication doesn't ensure consensus, and arguing back and forth can consume significant time, resulting in reduced efficiency. Interestingly though understandable, we did not observe such a phenomenon during our experiments. **CoELA is prone to cooperation** and coordinates plans without arguing back and forth which may be credited to **LLMs trained to follow instructions** and trust their cooperators. An interesting case where CoELA adapts plans considering others is shown in Figure 3a and Section C.4. Bob initially suggests a division of labor, but Alice disagrees and suggests another plan, Bob gives it a thought and accepts the new plan. This behavior is **beneficial for cooperation**, though it may lead to less efficiency when the cooperator is malicious. We've added this discussion to the paper as well.

Reviewer comment (Reproducibility):
How does CoELA determine the valid high-level plans at each state? Is it determined manually?

Author response:
CoELA proposes valid high-level plans **procedurally** given the current state and procedural knowledge retrieved from the memory module. We chose to organize them in a multi-choice format so the LLM can concentrate on the reasoning and make an executable plan **without any few-shot demonstrations**. More details are in the Appendix A.4

Reviewer comment (Novelty):
Have there been previous works using LLM for implementing embodied agents? Can more discussion or even experimental comparisons be made with these works? What are the core contributions and innovations of CoELA compared to them?

Author response:
Previous works using LLM for implementing embodied agents mainly focus on single-agent and simple tasks, while we address a more **challenging multi-agent cooperation problem** characterized by decentralized control, complex observations, costly communication, and long-horizon multi-objective tasks. To solve these new challenges, we design CoELA where we incorporate LLMs into a modular framework and enable **direct communication between each other**, which is both **effective** and **interpretable**. We've added a discussion section in the revised paper.

Reviewer comment (Experiments):
This method assumes a skill library that is manually defined for a specific domain, i.e. execution module. However, this limits its applicability in other domains where predefined skill libraries are not available.

Author response:
- Our low-level skills are obtained from motion planning, which is **common** in embodied AI benchmarks [1][2][3][4], given the focus is more on high-level reasoning and planning.
- Our framework is **flexible** thanks to our **modular design**, and can be **easily adapted** to various low-level control policies, including reinforcement learning and trajectory optimization, offering a distinct advantage over end-to-end methodologies

Reviewer comment (Experiments):
The pipeline appears to be quite complex and relies on several hand-defined modules, including perception modules, three memory modules, planning, and execution. Are all of these modules necessary? Conducting an ablation study would provide better understanding.

Author response:
- Thanks for the question! We do **have an ablation study** on the memory, communication, planning, and execution modules in the paper to have a better understanding of the effectiveness of each module, and the results are shown in Figure 4 \(c\) and discussed in section 5.4.
- The results show that **all modules contribute to the best performance** of CoELA when cooperating with other agents and humans.
    - Without the Memory Module, CoELA needs nearly doubled steps to finish the task
    - A strong LLM to drive the Planning and Communication module of CoELA is vital for better efficiency
    - Without the Execution Module, the agent struggles to make low-level control directly and fails most of the task
- It's not trivial to ablate on the perception module under the visual image observation setting, so we also conduct experiments with the oracle perception given, and the result is shown in Table 1. The performance gap between TDW-MAT and w/ Oracle Perception is rather close, **showing the Perception Model is effective** as well.
- On the other hand, our modular framework is not restricted to the number of modules and can be easily extended to deal with all kinds of challenging tasks.

Reviewer comment (Experiments):
The experimental design lacks breadth as it only considers one scenario. Including evaluations across multiple scenarios would strengthen the findings.

Author response:
This might not be true. We actually strive to conduct comprehensive experiments **on the two most challenging embodied multi-agent benchmarks**.
- Our experiments are conducted across **two simulation platforms** and different tasks with **varied observation and action space**, where C-WAH has **5 kinds of activities** and TDW_MAT features **12 different scenes**.
- We experiment under both scenarios of **collaborating with AI agents** (section 5.3.1) and **with humans** (section 5.3.2).
- The rearrangement task is a **canonical task** for evaluating embodied agents featuring various challenges including **perception**, **navigation** and **interaction**.[5]
- We didn't notice any better options for testing our cooperative embodied agents in challenging environments.
- Designing more suitable embodied multi-agent benchmarks for this field is also a valuable future avenue, though beyond the scope of this work.

Reviewer comment (Evaluation):
How to evaluate the communication cost? Is there any tradeoff study on communication cost and effectiveness?

Author response:
The communication cost is evaluated in metrics *Average Steps* and *Transport Rate*.
- Communication in our setting takes time as in real life, and therefore has an impact on the timesteps needed to carry out a task, which is reflected in the evaluation metric *Average Steps* and *Transport Rate*.
- As shown in B.1 and B.2 Action Space, the agent can send at most 500 characters in each frame, which means a communication action of sending a message of 600 characters will cost the agent 2 frames. The agents need to **reason over the pros and cons of each step of communication on its own**.
- An extreme case of the tradeoff study is where the communication takes no cost, then the agent can just send everything in their observation to the others and discuss their plan thoroughly before taking the next action, making the collaboration essentially less "de-centralized", and leading to a better efficiency regarding actions.

Reviewer comment (Experiments):
For turning left/right, what will happen if the object in interest is on the back of the agent? How is the visual-related textual input organized? Did the work try to use the oracle information of the whole environment, including objects and relations?

Author response:
We have both results from  Oracle Perception where the instance segmentation is given by the environment and Learned Perception where only  ego-centric RGBD image from the agent's camera is given and a neural perception model is deployed to segment the objects.
- Under both scenarios, there exists the **partial observation challenge**, if the object in interest is out of the agent's view, such as on the back of the agent, the agent need to actively explore the environment to find it, leading to a **navigation and exploration challenge**.
- Partial observation is a key feature in our challenging, making effective cooperation more challenging and lead to our key insights. We donot assume the oracle information of the whole environment.