Paper: LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents (ICLR-2024)

Reviewer comment (Experiments):
The feedback for replanning is based on metadata (i.e., oracle information about what is "wrong") of the simulated environments but this may not be always available beyond the benchmark for other downstream tasks (e.g., some tasks might not be able to provide why agents fail at actions). Can this direction be also useful for those cases without the oracle information?

Author response:
> **The feedback for replanning is based on metadata ... but this may not be always available beyond the benchmark for other downstream tasks ... Can this direction be also useful for those cases without the oracle information?**

In Section 5.2, we conducted experiments on feedback and replanning. As you mentioned, we utilized oracle information to articulate the reasons for failure in the current step in natural language, which is then fed into an LLM-based planner for replanning. These experiments primarily investigated the impact of NL feedback on planning performance, under the assumption that reasons for failure are accessible. However, in scenarios where oracle information is unavailable, our approach could potentially infer the causes of step failures using sensor data from embodied agents. For instance, feedback like “(this action failed: Robot is not holding any object)” could be derived by checking if the robot’s hand sensors detect an object. Similarly, “(this action failed: put down failed)” could be inferred using vision data by object detection. This possibility represents a separate research direction. Our paper primarily focuses on evaluating the effectiveness of NL feedback when it is available, demonstrating its potential to enhance high-level planning performance.

Reviewer comment (Evaluation):
The authors conduct experiments based on a subset (30%) of the original datasets but do not provide some significance tests. It might be better to include them in the results (e.g., Figure 2) with different subsets.

Author response:
> **The authors conduct experiments based on a subset (30%) of the original datasets but do not provide some significance tests. It might be better to include them in the results (e.g., Figure 2) with different subsets.**

Although we randomly selected a 30% subset, it might have some biases in selection. We fully agree with you that it would be better to include results with different subsets. We conducted additional experiments with three other subsets. Due to the limited time of this discussion period, we only included results for three models of GPT-Neo, LLaMA-1, and LLaMA-2 except for the large ones having more than 60B parameters. Our code was ready for this experiment; we only changed the random seed for subset selection on line 78 in *src/alfred/alfred_evaluator.py* in the supplementary code. 

The figure linked below shows the results with different subsets.

https://figshare.com/s/557079abb37b1712d074

The bold lines are the same as the previous results in Figure 2. The thin lines represent additional results with different subsets. The general trends that 1) bigger models work better and 2) LLaMA series are better than GPT-Neo are the same. However, we found some differences in the results. For example, the success rates range widely from 7.21 to 12.5 for GPT-NeoX-20B according to the selection of subsets. And LLaMA-1 worked better than LLaMA-2 for some subsets. As we found subset selection affect results to some extent, we will also include full set (using 100\% samples) results with all the models in the final paper (if we have the opportunity to prepare the final paper). Specifically, we will include two figures for the subset and the full set instead of Figure 2 (a).

We would like to mention that reporting results for a subset is still valuable since many researchers do not have enough computing resources, so we think it would be better to maintain a 30% subset as well so that it serves as a kind of small set for quick or affordable experiments.

This subset experiment is only for ALFRED. We already used the full set for WAH-NL.

Reviewer comment (Presentation):
The used font looks a bit different from the one used in the original ICLR format. It might be better to revert to the original font.

Author response:
> **The used font looks a bit different from the one used in the original ICLR format. It might be better to revert to the original font.**

Thank you for pointing this out. We used XeLaTeX for PDF compiling, and it gave slightly different fonts than the pdfLaTeX compiler. We corrected the compiling options and will use the corrected version when we upload the revised draft. It was not intentional, and the page count was the same.

Reviewer comment (Experiments):
**Lack of Visual Understanding**: The benchmark does not incorporate visual understanding, which is critical for low-level actions, especially in embodied AI where egocentric views play a significant role in interpreting and interacting with the environment.

Author response:
> **[W2] Lack of Visual Understanding**

In response to this concern, we wish to clarify that our LoTa-Bench is strategically designed to concentrate on high-level task planning, distinct from vision-based low-level control. The benchmarks’s primary goal is to provide a clear and focused evaluation of language-oriented high-level task planning performance, thus intentionally minimizing potential confounding influences from low-level controls.

However, we agree with your point regarding the importance of visual understanding. We are actively considering potential expansions to LoTa-Bench that would integrate visual information. This extension is designed not to exploring visual understanding for low-level control purposes but to enrich the capabilities of high-level planning. Potential directions include leveraging natural language feedback informed by visual information or integrating visual affordances into the planning process. Our objective with these future developments is to effectively bridge the gap between language-oriented task planning and visual perception.

Reviewer comment (Experiments):
**Domain Gap in Simulation**: There is a domain gap between the simulation environments used and the real world, leading to unrealistic assumptions. For example, the ALFRED simulator assumes an object is clean once it is placed in water, which may not always be true in real-world scenarios.

Author response:
> **[W3] Domain Gap in Simulation**  

> **[Q3] How do you plan to address the domain gap between simulation and real-world application, and do you have strategies for dealing with unrealistic assumptions like the one mentioned in ALFRED about objects being cleaned once put into water?**

As you pointed out, since LoTa-Bench evaluates planning performance in simulator environments, there is a domain gap with real world. The most prominent gap is ALFRED's assumption about cleaning. While other simulation tasks, such as picking and placing, heating using a microwave, and washing using a dishwasher, exhibit a smaller gap from real-world scenarios at a high-level planning stage. 

In response to this, one potential approach to expand the proposed benchmark is to enhance the AI2THOR simulator used in ALFRED to reduce the domain gap. It may become more realistic by adding conditions, such as requiring the target object to be touched with a sponge, particularly addressing unrealistic gaps like cleaning, and incorporating this into the object status update logic in the simulator. Additionally, cleaning using a dishwasher, as supported in WAH-NL, could also be added. For example, instead of simply placing a fork in the water, we can add steps, placing fork inside a dishwasher, close the dishwasher, and turn on the machine.

Reviewer comment (Experiments):
Absence of an in-depth investigation into several influential factors that affect the performance of task planners. These factors include the type and size of pre-trained language models, the number and strategy for selecting in-context examples, the ability of planners to replan based on natural language feedback, and the impact of fine-tuning the models. Addressing these factors is crucial for a thorough understanding of how to optimize task planners for real-world applications.

Author response:
> **[W4] Another limitation of the paper is the absence of an in-depth investigation into several influential factors that affect the performance of task planners. ... Addressing these factors is crucial for a thorough understanding of how to optimize task planners for real-world applications.** 

Our paper indeed addresses each of the factors you have mentioned, using our proposed LoTa-Bench framework. The details are described in Section 5 and 6 of our manuscript.

* **1) Impact of types and sizes of pre-trained language models**

    In our work, we utilized the LoTa-Bench framework to evaluate and analyze the task planning performance of various pre-trained LLMs, including GPT [7], GPT-Neo series [8, 9, 10], OPT [11], MPT [12], LLaMA 1 [13], and LLaMA 2 [14], and their sizes. Our experimental results, summarized in Figure 2 of the paper, show that task planning performance generally increases with the size of the language model. Specifically, GPT-3 175B demonstrates the best performance on both ALFRED and WAH-NL. Further details are provided in:
    * **Section 5.2**: “We evaluated the planning performance of the baseline planner described in Section 3. … Instruction- and chat-tuned models (dashed lines in Figure 2) did not perform better than their base models.” and **Figure 2**
    
* **2) Influence of the number of in-context examples**

    We explored the influence of the number of in-context examples using LLaMA 2 13B model. Our tests varied the number of examples from 0 to 30 in ALFRED and from 0 to 15 in WAH-NL. The results, depicted in Figure 3 of the paper, indicates that planning performance improves with an increased number of examples. More information can be found in:
    * **Section 5.2**: “We have investigated the impact of the number of examples in prompt with LLaMA 2 13B model that supports a longer context length of 4096. … (with an average of 13.61 and a standard deviation of 3.22) for the LLaMA 2 13B model.” and **Figure 3**
    
* **3) Strategy for selecting in-context examples**

    We introduced three strategies (Random Sampling, Task-Specific Sampling, Semantic Similarity Sampling) for selecting in-context examples from the train set in Section 6.1 and investigated their impact on planning performance. Notably, Semantic Similarity Sampling significantly enhances planning performance. More details are available in Section 6.1 and Figure 5.
    
* **4) Planners’ ability to replan based on natural language feedback**
    
    In Section 6.2, we discuss the effect of feedback and replanning. We observed that providing NL feedback about the reasons for action failure aids task planners in replanning when the size of language model is large. Further insights are available in Section 6.1 and Table 1.
    
* **5) Impact of fine-tuning on train set**

    Our investigation into the impact of fine-tuning on the train set is discussed in Section 6.3. We employed LoRA [15] for fine-tuning LLMs, which resulted in significant performance improvements. This is elaborated upon in: Section 6.2 and Figure 6.

We would like to emphasize the critical role of our LoTa-Bench framework in facilitating these comprehensive evaluations. Without such a benchmark suite, conducting in-depth analyses of task planner performance across the factors you have highlighted would be more challenging. Our framework not only enables these detailed evaluations but also provides substantial support in advancing the field of task planning.

[7] Language Models are Few-Shot Learners \
[8] GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow \
[9] GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model \
[10] Gpt-neox-20b: An open-source autoregressive language model \
[11] OPT: Open Pre-trained Transformer Language Models \
[12] Introducing mpt-7b: A new standard for open-source, commercially usable llms \
[13] LLaMA: Open and Efficient Foundation Language Models\
[14] Llama 2: Open Foundation and Fine-Tuned Chat Models \
[15] LoRA: Low-Rank Adaptation of Large Language Models

Reviewer comment (Novelty):
Missing references:  
- On Grounded Planning for Embodied Tasks with Language Models  
- Plan, Eliminate, and Track - Language Models are Good Teachers for Embodied Agents

Author response:
> **Minor one: Missing references**

Thank you for your pointing out these references. The first paper introduced a method for encoding an environment in the form of an object table to provide context to a language model-based planner. The second paper enhances LLM-based task planner with additional modules, eliminating irrelevant objects and receptacles from observation and tracking the completion of each step. Both works contribute to the field of LLM-based task planning by focusing on integrating contextual information to improve planning performance. We plan to add both papers to the Related Work section of our manuscript as follows:
“Moreover, integrating context into LLM-based task planners has been shown to enhance planning efficacy (Huang et al., 2023; Yao et al., 2023; Chen et al., 2023; Lin et al., 2023; Wu et al., 2023)”

Reviewer comment (Novelty):
The primary contribution of the paper seems limited to the integration of several existing benchmarks into a single platform.

Author response:
Thank you for your valuable feedback. We would like to respond to your comments on the weaknesses and questions.

> **The primary contribution of the paper seems limited to the integration of several existing benchmarks into a single platform. ... The paper does not make it clear how LoTa-Bench provides additional value over these existing benchmarks.**

First, we clarify the differences between LoTa-Bench and existing benchmarks.

* **1) LoTa-Bench vs. ALFRED + AI2THOR**

    ALFRED [1] built on AI2THOR [2] has both high-level and low-level instructions. One example is as follows.
      
    ```
    Task instruction: "Move a towel to the sink"
    Step-by-step action instruction:
    1. Move to the far left cupboard under the far left sink
    2. Open the cupboard and remove the towel inside, shut the cupboard
    3. Carry the towel to the left sink above the cupboard
    4. Place the towel in the sink
    ```
    
    As low-level instructions in natural language are given, the previous studies [3, 4] on ALFRED mostly focused on translating step instructions into robot actions with visual understanding, which overlooks the importance of high-level planning. Thus, we proposed LoTa-Bench designed to focus on high-level task planning by using only task instructions; no step-by-step instructions were given and no training on the dataset unlike previous studies to validate generalizability of the high-level task planner. Although we did not augment the ALFRED dataset itself, we utilized the dataset in a new way to propose a new benchmark problem to help the research community validate new task planning methods more easily and objectively.

* **2) LoTa-Bench vs. ActivityPrograms \& Watch-and-Help + VirtualHome**
 
    The ActivityPrograms dataset [5] provides natural language instructions, but it lacks goal conditions. Some language-oriented task planning studies that used this dataset relied on human evaluation for quantitative performance evaluation [6, 7]. LoTa-Bench, however, includes datasets with goal conditions, enabling automated quantitative performance measurement.
    
    The Watch-and-Help dataset [8] incorporates goal conditions but not NL instructions, limiting its use for language-oriented task planning. LoTa-Bench adapts the Watch-and-Help dataset for language-oriented task planning by modifying the goal conditions for autonomous agents and enriching it with NL instructions that we collected from crowdworkers.

Beyond providing benchmark suites, our comprehensive experimental analysis illuminates factors influencing LLM-based task planner performance, such as model size, type, and the number of in-context examples. We also investigate performance-enhancing extensions for LLM-based task planners within LoTa-Bench, including in-context example selection, feedback and replanning, and fine-tuning.

Together, we believe that these contributions uniquely position LoTa-Bench as a valuable asset in language-oriented task planning research.

[1] Alfred: A benchmark for interpreting grounded instructions for everyday tasks \
[2] Ai2-thor: An interactive 3d environment for visual ai \
[3] Episodic Transformer for Vision-and-Language Navigation \
[4] Agent with the big picture: Perceiving surroundings for interactive instruction following \
[5] Virtualhome: Simulating household activities via programs. \
[6] Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents \
[7] Parsel : Algorithmic Reasoning with Language Models by Composing Decompositions \
[8] Watch-and-help: A challenge for social perception and human-ai collaboration

Reviewer comment (Novelty):
There are existing works, as cited in the Related Work section, that already employ benchmarks like ALFRED and VirtualHome to test various planning and execution techniques.

Author response:
Thank you for your valuable feedback. We would like to respond to your comments on the weaknesses and questions.

> **The primary contribution of the paper seems limited to the integration of several existing benchmarks into a single platform. ... The paper does not make it clear how LoTa-Bench provides additional value over these existing benchmarks.**

First, we clarify the differences between LoTa-Bench and existing benchmarks.

* **1) LoTa-Bench vs. ALFRED + AI2THOR**

    ALFRED [1] built on AI2THOR [2] has both high-level and low-level instructions. One example is as follows.
      
    ```
    Task instruction: "Move a towel to the sink"
    Step-by-step action instruction:
    1. Move to the far left cupboard under the far left sink
    2. Open the cupboard and remove the towel inside, shut the cupboard
    3. Carry the towel to the left sink above the cupboard
    4. Place the towel in the sink
    ```
    
    As low-level instructions in natural language are given, the previous studies [3, 4] on ALFRED mostly focused on translating step instructions into robot actions with visual understanding, which overlooks the importance of high-level planning. Thus, we proposed LoTa-Bench designed to focus on high-level task planning by using only task instructions; no step-by-step instructions were given and no training on the dataset unlike previous studies to validate generalizability of the high-level task planner. Although we did not augment the ALFRED dataset itself, we utilized the dataset in a new way to propose a new benchmark problem to help the research community validate new task planning methods more easily and objectively.

* **2) LoTa-Bench vs. ActivityPrograms \& Watch-and-Help + VirtualHome**
 
    The ActivityPrograms dataset [5] provides natural language instructions, but it lacks goal conditions. Some language-oriented task planning studies that used this dataset relied on human evaluation for quantitative performance evaluation [6, 7]. LoTa-Bench, however, includes datasets with goal conditions, enabling automated quantitative performance measurement.
    
    The Watch-and-Help dataset [8] incorporates goal conditions but not NL instructions, limiting its use for language-oriented task planning. LoTa-Bench adapts the Watch-and-Help dataset for language-oriented task planning by modifying the goal conditions for autonomous agents and enriching it with NL instructions that we collected from crowdworkers.

Beyond providing benchmark suites, our comprehensive experimental analysis illuminates factors influencing LLM-based task planner performance, such as model size, type, and the number of in-context examples. We also investigate performance-enhancing extensions for LLM-based task planners within LoTa-Bench, including in-context example selection, feedback and replanning, and fine-tuning.

Together, we believe that these contributions uniquely position LoTa-Bench as a valuable asset in language-oriented task planning research.

[1] Alfred: A benchmark for interpreting grounded instructions for everyday tasks \
[2] Ai2-thor: An interactive 3d environment for visual ai \
[3] Episodic Transformer for Vision-and-Language Navigation \
[4] Agent with the big picture: Perceiving surroundings for interactive instruction following \
[5] Virtualhome: Simulating household activities via programs. \
[6] Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents \
[7] Parsel : Algorithmic Reasoning with Language Models by Composing Decompositions \
[8] Watch-and-help: A challenge for social perception and human-ai collaboration

Reviewer comment (Novelty):
The paper does not make it clear how LoTa-Bench provides additional value over these existing benchmarks.

Author response:
Thank you for your valuable feedback. We would like to respond to your comments on the weaknesses and questions.

> **The primary contribution of the paper seems limited to the integration of several existing benchmarks into a single platform. ... The paper does not make it clear how LoTa-Bench provides additional value over these existing benchmarks.**

First, we clarify the differences between LoTa-Bench and existing benchmarks.

* **1) LoTa-Bench vs. ALFRED + AI2THOR**

    ALFRED [1] built on AI2THOR [2] has both high-level and low-level instructions. One example is as follows.
      
    ```
    Task instruction: "Move a towel to the sink"
    Step-by-step action instruction:
    1. Move to the far left cupboard under the far left sink
    2. Open the cupboard and remove the towel inside, shut the cupboard
    3. Carry the towel to the left sink above the cupboard
    4. Place the towel in the sink
    ```
    
    As low-level instructions in natural language are given, the previous studies [3, 4] on ALFRED mostly focused on translating step instructions into robot actions with visual understanding, which overlooks the importance of high-level planning. Thus, we proposed LoTa-Bench designed to focus on high-level task planning by using only task instructions; no step-by-step instructions were given and no training on the dataset unlike previous studies to validate generalizability of the high-level task planner. Although we did not augment the ALFRED dataset itself, we utilized the dataset in a new way to propose a new benchmark problem to help the research community validate new task planning methods more easily and objectively.

* **2) LoTa-Bench vs. ActivityPrograms \& Watch-and-Help + VirtualHome**
 
    The ActivityPrograms dataset [5] provides natural language instructions, but it lacks goal conditions. Some language-oriented task planning studies that used this dataset relied on human evaluation for quantitative performance evaluation [6, 7]. LoTa-Bench, however, includes datasets with goal conditions, enabling automated quantitative performance measurement.
    
    The Watch-and-Help dataset [8] incorporates goal conditions but not NL instructions, limiting its use for language-oriented task planning. LoTa-Bench adapts the Watch-and-Help dataset for language-oriented task planning by modifying the goal conditions for autonomous agents and enriching it with NL instructions that we collected from crowdworkers.

Beyond providing benchmark suites, our comprehensive experimental analysis illuminates factors influencing LLM-based task planner performance, such as model size, type, and the number of in-context examples. We also investigate performance-enhancing extensions for LLM-based task planners within LoTa-Bench, including in-context example selection, feedback and replanning, and fine-tuning.

Together, we believe that these contributions uniquely position LoTa-Bench as a valuable asset in language-oriented task planning research.

[1] Alfred: A benchmark for interpreting grounded instructions for everyday tasks \
[2] Ai2-thor: An interactive 3d environment for visual ai \
[3] Episodic Transformer for Vision-and-Language Navigation \
[4] Agent with the big picture: Perceiving surroundings for interactive instruction following \
[5] Virtualhome: Simulating household activities via programs. \
[6] Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents \
[7] Parsel : Algorithmic Reasoning with Language Models by Composing Decompositions \
[8] Watch-and-help: A challenge for social perception and human-ai collaboration

Reviewer comment (Experiments):
Why was there no fine-tuning conducted with the WAH-NL dataset?

Author response:
In our initial approach, the primary goal of fine-tuning was to demonstrate the capability of our benchmark in facilitating a broad range of experiments for LLM-based planning. Consequently, we chose to present fine-tuning results solely within the ALFRED domain, considering it to be sufficiently illustrative of our benchmark's efficacy. Additionally, the WAH-NL domain offered a limited dataset of only 250 instances for fine-tuning, which we initially deemed inadequate for achieving significant performance improvements.

Nevertheless, inspired by your query, we have now undertaken a fine-tuning experiment using the WAH-NL dataset. This dataset comprises 250 training instances, and we maintained consistency in hyper-parameters and conditions as per our ALFRED dataset experiments. The outcomes are detailed in the table below.

**Table 1. Plan execution success rates of planners fine-tuned on the WAH-NL dataset. δ denotes changes in success rates with and without fine-tuning (expressed in percentage points, pp).**
|             | Baseline Success(%) | Fine-tuned Success (%) | δ (pp) |
|-------------|---------------------|------------------------|-------------|
| LLaMA 1 7B  | 28.65               | 21.36                  | -7.34       |
| LLaMA 1 13B | 23.70               | 29.47                  | +5.77       |
| LLaMA 1 30B | 32.22               | 33.78                  | +1.56       |

Although the improvements were modest and not consistent across all models, fine-tuning with the LLaMA 1 13B and 30B models did exhibit some performance enhancements (5.77 and 1.56 percent points).

The entire process, excluding the time spent on fine-tuning the models, took less than 7 hours, and the evaluation was fully automated using the proposed LoTa-Bench. We are confident that our benchmark suite can significantly accelerate research on various aspects of LLM-based planners, offering an efficient and versatile evaluation tool for the research community.