Paper: Asking Before Acting: Gather Information in Embodied Decision-Making with Language Models (ICLR-2024)

Reviewer comment (Writing):
The writing could benefit from improvements, particularly typographical errors like "suboptimal beahviors" found in the second paragraph.

Author response:
Thanks! We have fixed the typos in our manuscript and these modifications are highlighted in blue for your convenience.

Reviewer comment (Experiments):
The authors' pursuit to tackle intricate issues in embodied AI, especially everyday tasks beyond mere embodied navigation, is praiseworthy. Nonetheless, showcasing the method's versatility across various realms, including embodied navigation, would enhance the paper's value.

Author response:
Thanks for pointing out these related works! We have added them to the related work section and highlighted the modifications in blue.

Reviewer comment (Novelty):
It would also be beneficial if the paper could cover works from pre-LLM era on embodied AI task that takes in help signal to guide its downstream task:  
1. Chi, T.C., Shen, M., Eric, M., Kim, S. and Hakkani-tur, D., 2020, April. Just ask: An interactive learning framework for vision and language navigation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 03, pp. 2459-2466).  
2. Zhang, J., Yu, S., Duan, J. and Tan, C., 2022. Good Time to Ask: A Learning Framework for Asking for Help in Embodied Visual Navigation. arXiv preprint arXiv:2206.10606.  
3. Singh, Kunal Pratap, Luca Weihs, Alvaro Herrasti, Jonghyun Choi, Aniruddha Kembhavi, and Roozbeh Mottaghi. "Ask4help: Learning to leverage an expert for embodied tasks." Advances in Neural Information Processing Systems 35 (2022): 16221-16232.

Author response:
Thanks for pointing out these related works! We have added them to the related work section and highlighted the modifications in blue.

Reviewer comment (Experiments):
Comparison with previous works: One of the major weaknesses is that several works with competitive baselines are omitted from the paper. For example, in AlfWorld, Reflexion (1) and AdaPlanner (2) achieve similar or better results on AlfWorld than ABA. For the robotics task, there is no comparison with works like Cliport (3). For the finetuned version of ABA, there is no comparision with finetuned models or imitation learning methods.

Author response:
**New baselines:**

We've carefully considered your recommendations, and here's our feedback:

1. *Reflexion [1] and Adaplanner [2]:*

    These methods focus on learning from previous mistakes through multiple rollouts, while ABA aims to perform tasks within a single shot. Therefore we are orthogonal and focus on different settings. We acknowledge the potential for future research to explore combinations with [1,2] to enhance performance.


2. *CLIPort [3]*

   We believe [3] is not suitable here since:

   - Different settings. Both ABA and ReAct are few-shot methods, while [3] needs to collect an expert dataset to do imitation learning.

   - In section 4.2, our tasks involve the agent asking questions or reasoning to gather information, a scenario not directly adaptable to imitation learning-based methods like [3].

3. *BUTLER in ALFWorld variants*

    We have already compared BUTLER in the original ALFWorld. We show ABA which use $K=2$ trajectories can significantly outperform BUTLER with $10^5$ training trajectories. For similar but more challenging environments, we believe it is not necessary to compare with it again.

Reviewer comment (Experiments):
A key part of ABA is receiving/guidance from the environment or external sources. However, in the paper, a second LLM is used as the 'human' or external source. There is no ablation showing that the external source is providing guidance and not just the answer.

Author response:
***Q3.  There is no ablation showing that the external source is providing guidance and not just the answer.***

In Section 3.3 and Appendix A, we showed how the human model is designed. It prompts the LLM with environmental information and is prompted to answer the questions. The human model does not have access to the oracle action, and it will not provide the answer.

Reviewer comment (Reproducibility):
What is the expert policy used to train ABA-FT?

Author response:
***Q5. What is the expert policy used to train ABA-FT?***

As outlined in Appendix E, we designed a rule-based policy according to the PDDL planning trajectories provided along with the environment to collect the data.

Reviewer comment (Experiments):
Is there any visual model used for the robotic control experiments?

Author response:
***Q6. Is there any visual model used for the robotic control experiments?***

As outlined in appendix J, we use a hand designed captioning module, which can access the simulator information and label the object type and its ground truth coordinates. As discussed in Huang et al. (2023), this module can be implemented through the depth image and a combination of object detection model, an object segmentation model, and a video tracker. But we temporarily leave this module hand-designed since it is not the key point of our work.

Reviewer comment (Novelty):
For embodied decision-making tasks, some related works need to be discussed. For example, previous works (e.g. [1]) explore the use of knowledge graph as external commonsense and extract pertinent information via GNN. Also, [2] extracts knowledge from VLM during the decision process. Both works aim to gather external information for improving navigation decisions in unseen environments, which is similar to the ABA idea.  
[1] Room-and-object aware knowledge reasoning for remote embodied referring expression.  
[2] Room-Object Entity Prompting and Reasoning for Embodied Referring Expression.

Author response:
We are grateful for your recommendation of additional relevant works. We have incorporated these insightful contributions into our related works section, alongside other vision-language navigation works like TD-STP, SEvol which we believe to be related to the embodied decision making setting. The modifications have been highlighted in blue.

Reviewer comment (Evaluation):
Given that the core of the ABA method is to obtain effective external assistance, and obtaining external assistance has costs such as time, resources, and human efforts. Therefore, how to balance the inquiry frequency and performance is a missing topic in this work.

Author response:
We appreciate your observation regarding the balance between performance and efficiency. In ABA, our key design focuses on prompting the agent to ask questions only when necessary, achieved through the injection of a proper prior. In the ABA-FT variant, we further enhance this efficiency by treating known information as a type of metadata, aiding the decision-making process on whether to ask questions.

To demonstrate the efficiency of these designs, we conducted an ablation study by counting the total number of questions asked in V7B ALFWorld successful trials, and the results are as follows (which is also added to Appendix L):

|                               | # Questions |
|-------------------------------|-------------|
| ALFWorld                      | $1.1$       |
| ALFWorld with ambiguous tasks | $1.39$      |
| Multiround ALFWorld           | $2.53$      |

The results indicate that the average number of questions closely aligns with the minimum necessary questions. This study substantiates our claim that ABA effectively minimizes the number of questions asked. We will leave minimizing the number of questions to our future work.

Reviewer comment (Writing):
There are still some typos, and the authors need to further polish the writing.  
- Page 4: Missing comma in the sixth line of subsection 3.2.1  
- Page 9: "open do- main image" in the conclusion section  
- Page 14: "Section ??"

Author response:
Thanks! We have promptly rectified these errors in our manuscript, and the modifications have been highlighted in blue for your convenience.

Reviewer comment (Theory):
The problem in the paper is formed as Contextual MDP, while I think it's better formulating the problem into a more suitable domain like POMDP, which is much well-accepted and clearly defined. Is there a necessity to use Contextual MDP? I hope the authors can further explain the motivation.

Author response:
***Q3. Why use Contextual MDP rather than POMDP:***

Although Contextual MDP is a type of POMDP, we opted for the former because:

- Contextual MDP provides a latent context $c$ which helps to model the task information, while POMDP does not. 

- $c$ helps better formulation: we can ask pertinent questions to infer the context $c$, the human model inside the environment $H$ will provide different answers to the same questions according to the different $c$. while POMDP can not.

- Contextual MDPs are well defined and widely accepted, especially in RL generalization theory, multi-task, and meta RL domains. Please refer to [9] for a thorough survey.

Reviewer comment (Novelty):
The paper's methodology heavily relies on the capabilities of Large Language Models (LLMs). I think the method itself does not make much sense. From my point of view, it seems just like the authors design some tasks with ambiguity and use the "ask before action" paradigm to help solve such ambiguity. But the model does not resolve such ambiguity by itself, i.e., through rapid trial-and-error. On the other hand, it still depends on the human to help it. So why not the human provide the complete information at the beginning? Besides, I don't see the proposed "ask before action" paradigm as a big contribution, as we have already seen many works related to the LLM agency in which LLMs are driven to finish more complex interactions and tasks.

Author response:
***Q1. about the setting and contribution: the "asking before action" paradigm***

After reading the review, we would like to clarify our setting in case of any potential misunderstandings:

Currently, LLM based agents are designed to solve tasks on their own without external interactions, which is identified to be inefficient in scenarios with limited information.
To address this concern, we concentrate on a special type of setting with human / external information sources in the loop (refer to Def 3.1), and propose the “asking before action” paradigm. This distinguishes our work from purely autonomous agents.

We believe human / external information in the loop is important and promising [1-6]. 
We are not primarily concerned with completing complex tasks through trial-and-error but with facilitating decision-making through interaction with humans. 

We take a significant step forward in the Human-in-the-Loop (HITL) setting. Previous works including [1-6] ask for oracle action, template questions, etc. ABA proposes gathering necessary information in natural language. This contributes to a novel setting, a set of newly designed benchmarks, and helps extend the boundary of current research.

Looking ahead, we foresee combining ABA with recent advances in purely autonomous works, such as [7] for navigation and [8] for pick-and-place tasks, to build more intelligent embodied agents.

**To summary, our contributions are:**

1. identify a novel setting (~Def 3.1) that can possibly contribute to building a more embodied AI in the real world.

2. design new methods which can mimic human behavior, and "representing a shift towards more intuitive, adaptive AI systems".

3. a set of benchmarks with a set of human model designs that can facilitate later research in this direction.

---

***Q2. about the scenarios: Why Not Provide Complete Information at the Beginning/Feed to Other Baselines?***

We appreciate your question about complete information. We believe providing complete information is impractical for real-world tasks due to scale differences between simulators and reality. 

While simulators may handle scenarios with dozens of objects, the real world poses challenges in enumerating all relevant information. To illustrate, consider seeking assistance in preparing lunch—an exhaustive list of utensils and ingredients would require significant human effort and might overlook unforeseen circumstances. Therefore, for real-world scenarios, providing complete information is not viable, and we have not supplied it to other baselines.

Reviewer comment (Evaluation):
There lacks error analysis conducted on instances where ABA underperformed or failed. What insights were derived from these analyses? Besides, I think additional QA in ABA actually imports more information compared with other baselines, which definitely leads to better performance. Maybe other baselines should also use information like “the second red block from the left” for an equal comparison.

Author response:
***Q4. Error analysis on ABA:***

We provided error analysis of a typical failure case in Appendix F, and we appended several new failure cases at the end of Appendix F (modifications are highlighted in blue). The analysis can be summarized as follows:

Two key failure types are identified:

- *undesirable answers from human models during evaluation*
    
    Note we use human models to mimic humans during evaluation to minimize human involvement, but this may sometimes provide undesirable answers.

- *mismatches between agent commonsense and the environment*

    For instance, the task is to collect eggs from the garbage can, but the agent believes the eggs are no longer suitable for eating and proposes to buy new ones. 

We believe incorporating real humans during deployment and refining prompting can mitigate these failures, leaving room for future research. For detailed information, please refer to Appendix F.

---

Once again, we sincerely appreciate your feedback, and we hope our responses address any concerns you may have. Your insights are crucial to the improvement of our work, and we welcome any additional feedback, and we genuinely hope for your reconsideration of the score. 

---

[1] Khanh Nguyen and Hal Daume ́ III. Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. arXiv preprint arXiv:1909.01871, 2019.

[2] Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12527–12537, 2019.

[3] Kunal Pratap Singh, Luca Weihs, Alvaro Herrasti, Jonghyun Choi, Aniruddha Kembhavi, and Roozbeh Mottaghi. Ask4help: Learning to leverage an expert for embodied tasks. Advances in Neural Information Processing Systems, 35:16221–16232, 2022b.

[4] Felipe Leno Da Silva, Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. Uncertainty- aware action advising for deep reinforcement learning agents. In Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pp. 5792–5799, 2020.

[5] Iou-Jen Liu, Xingdi Yuan, Marc-Alexandre Coˆte ́, Pierre-Yves Oudeyer, and Alexander Schwing. Asking for knowledge (afk): Training rl agents to query external knowledge using language. In International Conference on Machine Learning, pp. 14073–14093. PMLR, 2022.

[6] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022b.

[7] Shah, Dhruv, Błażej Osiński, and Sergey Levine. "Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action." Conference on Robot Learning. PMLR, 2023.

[8] Huang, Wenlong, et al. "Voxposer: Composable 3d value maps for robotic manipulation with language models." arXiv preprint arXiv:2307.05973 (2023).

[9] Kirk, Robert, et al. "A survey of generalisation in deep reinforcement learning." arXiv e-prints (2021): arXiv-2111.